Fat Goal

If I do not hit my fat goal for maintaining weight (but building muscle) and I pass my protein limit or hit it usually, does that mean I will lose overall fat?

My weight is 166 and I'm 5'11" and I have 80 of fat for the day to intake to maintain - but I rarely hit more than 50 - 75 percent of what I should be, so sometimes I'll just use 30 out of 80 or 55 out of 80. Does this mean I will lose unwanted fat? Compared to hitting an going over my fat goal for the day?

Note I am working out a few times a week.