Loss but no Loss

smallfeet

5'6" I started at 123lbs which was really quite high for me, My average weight is around 107lbs. Because id been eating loads of crap i've completely sorted out my diet. I dropped down to a weight between 114-116lbs but i seem to have plateaued completely on the scale, while still getting smaller. (this is judged on clothes fit, back to a size 6-4 uk again)
Any ideas as to why the pounds arent actually dropping? I don't do enough exercise for it to be muscle gain...
Thanks