Should you keep track of (Natural Sugar)?

Just wondering how many people don't track NATURAL SUGAR and still lost fat and weight?

Because I find that having a banana with my skim milk protein shake takes up a ton of sugar, but my research leads me to believe that you don't really need too. Tracking Table sugar should be what should be track!

Of course it doesn't mean to pig out on fruits all day long either

Just want to get everyones opinions on the matter .