Lactose and Fructose

I'm normally over on my Sugars. But 80% is from Milk and Fruit. Should I up my sugar goals or cut back on fruit and milk?