Sugar

So I am trying to reduce my sugar intake... no more honey in tea, I have started to use Stevia in the raw on my oatmeal, and virutally no processed foods. As I am logging food each day, I see I am still WAY over my allotted daily sugar amount and I notice it is mostly from fresh fruit I am eating. Just really confused and if someone could shed some light on this, I would surely appreciate it.