Good Progress...there go my daily calories!

I have dropped 17 lbs so far...And my daily calorie count is going down...BUMMER! The non workout days are getting harder to control the calls! I am working out four days now...Do I add a day so I can eat the calories? Don't want to overtrain!