I can't figure out the magic number...

So here is my journey so far. I started logging...really logging in November. I had figured out my TDEE was 2045, and with the deficit I should have been at around1600. I had eaten 1200 calories forever. I started slowly upping my calories...to 1400...then to 1600...then to 1800 and eventually by February I was at 2000. I had started the NROLFW, and was lifting 3 times a week. About a month ago, I decreased my calories to 1600 again, and life got busy, so because I was unable to get to the gym as often, I started working out at home to you tube videos...doing 30 -40 minutes a day 4 times a week. (Jillian Michaels, etc.) Through all of this my weight has remained mostly the same. I have fluctuated within 5 pounds, but never went below 130. My measurements dropped slightly...half an inch in waist, half an inch in hips.

I am so beyond frustrated to say the least. I cannot for the life of me figure out my "magic numbers". I eat relatively clean...log everything, and drink only water...all the time. I occasionally will have a glass of wine..but I always log it. I hate that I have to track every piece of food I put into my mouth, just to have no results. It's not worth it. I don't want to live like this. It's excruciating. There has to be a better way to live "normally". Eat what I want on certain days, and not have to worry. Does anybody have any suggestions? I am seriously open to just about anything at this point. I just want to lose 15-20 pounds. I want to feel better in my body, and want to wear shorts for the first time in 6 years this summer without feeling gross.

What am I missing? I feel like I should have shown some progress at this point...right?