Hungry the day AFTER a big workout

Hi! I've read all of the posts about eating my exercise calories. Not a problem for me, usually. Often, I'm really hungry after a workout. But, there are other days when I am unable to eat back my calories, due to needing to work in the evening. I do my workouts in the afternoon since I am a school teacher, and there are nights when I need to be working in the evening and don't have time to eat.
Problem is, when this happens, I'm often ravenous the next day and eat the extra calories then.
Does this happen to anybody else? How important is it to stick to my calorie allotment day-to-day? Any ideas? Thanks for your help!