Is it okay to eat below your recommended calorie intake?

I was just wondering, is it okay to fall below the ideal amount of calories you should eat each day? I work out almost every day and probably burn 360-420 calories a work out. I don't eat all of the calories back. I probably only eat about a quarter or half of those calories back after the fact.

Even on days when I don't work out, I'm very careful not to even get close to the limit of calories I should eat every day (1500-2000). I probably manage to stomach 1300 calories a day (not including the calories that I burn off during work outs. I don't know if the sudden drop in food intake is causing my body to hold on to weight or not, and I want to eat a little more, but I feel like a pig if I do.

So, is it okay that I don't hit the 2000 calorie intake mark? Is it prohibiting weight loss in some way or not?