General Calorie Questions

I have lost 110+ overall, with some ups and downs along the way and at this point really just want to drop 10-20 more lbs and lean out. In order to achieve this I have generally eating around 2000 calories a day, with some periods of time significantly under that. I am 32, 6'5 and around 255, with a body fat % around 25% atm. As of late I have been eating around 2300 calories a day and the weight loss has slowed. But my confusion revolves around how to approach eating my calories back. I tend to work out pretty hard for 30 minutes to 2 hours plus(depending on the workout for the day which varies between cardio to lifting free weights and finally Rushfit discs) at least 5 days a week. And I never eat any of my calories back. I usually feel pretty tired and my muscles don't seem to recover like they should, even on days off I feel sore when I don't think I should. I'm also not seeing the gains in the weight room that I feel I should(perhaps a plateau?) Furthermore I read an article on a MFP about finding the sweet spot that seems to state I need to eat the MFP calories back.

It seems for my size and exercise level my historical calorie intake is too low, but I am a little paranoid about upping my intake and eating back my MFP exercise calories back out of fear for the scale going back up. And yes I realize that this may just be unhealthy fear here. What I am really hoping for is some clarity from others that may have some experience with my current conundrum as well as pointers as how to kick start my weight loss again and dog some possible bad decisions. Should I eat back all or part of those calories burned? Should I go back to 2k or under calories? Feel free to view my diaries.

Thanks for any help!