Sugar Count

I'm constantly going over my sugar count but its literally only because I decent amounts of fruit and milk. I don't eat things such as candy, sugary breakfast cereal etc, etc. It's just because of the milk and fruit lol Is that bad? Should I cut down? It always fits into my calories. The reason I drink the milk (besides from the fact that I enjoy it) is actually to keep me from being too far under my calories because I get full quickly when I eat. Without it there are some days when I'd probably be 1000 calories.