Sugars vs Calories

Hey guys!
I have a question about sugars and calories. So far, I have lost 24 pounds watching my calories and exercising. I have about 4 pounds more to lose and it is all in my pouch area below my belly button. I have a feeling I am going to have a hard time losing the last little bit. My question is should I be watching all the sugars I eat as well? I eat a lot of veggies and fruit and yogurt, so my sugars are always 10-30 grams over my daily allowance. Anybody have any ideas/advice?

Thanks!
Ky