Fat or lack of muscle?

but I feel flabby

I think you answered your own question. It's all about personal preference/goals, and your preference is pretty clear.

When I lose fat without exercise I look lousy. Your mileage my differ. If you look flabby to yourself, you need to exercise.

Have you taken measurements and found your body fat percentage?

If you still feel flabby, you should probably stop the body weight workouts and start with some real weights and lifting heavier. You aren't going to bulk up, you are going to increase the lean muscle you already have, which will help to increase your metabolism, which will help burn off the "flabby" bits and help you to lean out and lower your body fat percentage.

Stop caring about the scale and work on your body composition now.

The thing is, I DO exercise! Check out my exercise diary! I am afraid I might have BDD. All i see is flab...

I looked at your diary and judging by how little you eat and much you exercise, I would not be shocked if you are experiencing amenorrhea and missing your period.

Monday: Eat 1172 - Exercise 774 = 398 net
Tuesday: Eat 939 - Exercise 241 = 698 net
Wednesday: Eat 717 - Exercise 269 = 448 net
Thursday: Eat 867 - Exercise 0 = 867 net
Friday: Eat 814 - Exercise 314 = 500 net

This^

If you are not eating enough of healthy foods, I'm not sure that your body will ever look or feel good. Very low levels on most days of protein and fat. Not good. MFP defaults for protein and fat are already too low, don't go even lower. Please EAT.

Even just ditching the artificial sweeteners (for real food) and eating full fat products (instead of low or no fat) will probably help a lot.

but I feel flabby

I think you answered your own question. It's all about personal preference/goals, and your preference is pretty clear.

Really what I am asking is should I build my muscle more, or just lose more fat?

When I weighed 128, I was "skinny-fat". I had starved myself to get down to 135, and then I became very ill and was hospitalized (I won't bore you with the details). When I got out of the hospital, I weighed 128, but I was plenty flabby. I looked terrible---very pale and gaunt. The starvation diet had caused a lot of problems--among them, the loss of a lot of lean tissue. In the period following, I couldn't seem to stop piling on fat all over my body and I had absolutely no energy---total exhaustion. I should have gone on a moderate diet with plenty of exercise, but instead, I tried every diet that was ever in vogue at that time--each time, gaining back the weight I had lost plus more--and I was still exhausted all the time. Finally, my husband begged me to just stop all the silly diets and try to get healthier. My weight stabilized (too high though) ---but at least, I had gained my energy and health back to some degree. I have been afraid to get serious about reducing my body fat until fairly recently and THIS time I seem to be doing okay healthwise. This is a roundabout way of suggesting that you follow a sensible diet and increase your exercise and then let your body be what it wants to be. Please don't become a victim of the "dietocracy".

I just read your food diary and what I see is that you should eat "real" food instead of fake stuff. Just doing that will add enough calories to your diet that you should be OK. So basically stick with the same plan, just switch back to real foods. Real yogurt, real organic preserves with real sugar (not fake sweetener or hfcs), use brown sugar or organic cane sugar in your coffee, a piece of regular string cheese instead of weight watchers brand, stuff like that. You don't have to eat more, just try to eat the less processed forms of what you have on your food journal.

I would bet that you are getting some odd symptoms from your diet like headaches, nausea, brain fog, and fatigue. That all goes along with fake this, artificial that, and fat free stuff. That stuff is not healthy, but it WILL lead to migraines and other health problems down the road.

And go to the general weight loss thread - I can't remember the title, and read some of those articles for newbies that are pinned at the top. I know you aren't a newbie to dieting but I think those articles will help you understand why your mindset just needs to adjust a little. And I'm not suggesting that you're in starvation mode. Good grief that term is used to death around here and it seems you either believe in it or you don't. But what I think is YOU need to eat more REAL foods which will bring your calories up to a more reasonable amount for a given day.

You've got to feed your muscles and avoid the processed foods.