Food intolerance's affecting weight loss?

I have an intolerance to gluten and dairy. I am on a low carb diet so no gluten for me! But, i decided to start slowly introducing dairy back into my diet because I had already cut out so much crap food I was feeling deprived!

I lost a little weight being low carb, but my stomach is bigger! I would've thought the diet and exercise would've meant my tummy was the first to go. (as in the past that's what has happened)

Was wondering if anyone, with or without food intolerance's, has found eating dairy everyday affects your weight loss, makes you bloated, or anything like this?

Thanks everyone!