Losing inches instead of lbs. even though not much exercise?

I understand that when you workout you build muscle and muscle weighs more than fat so a person could lose inches and not lbs or inches faster than lbs. Thing is I didn't measure when I started this and keep forgetting to do it since starting. But this weekend I got into a size 12 jeans instead of the 14s I'd been wearing and today they are feeling a little loose (very little but I notice it when standing that I keep pulling them up). Now they are a different brand of jeans than the 14s and most likely a "stretch" jean as I think most of them are any more, so could have stretched a little from wearing them (I usually wear them a couple days before washing unless I get something on them.) I'm only down 4 lbs since starting MFP this time around and the scale has been hovering the last several days/week between 168 and 167.4 just keeps going back and forth. I work a desk job but I pretty much run around non-stop in the evenings (with a husband, 2 kids, a dog and a garden to tend to, not to mention the housework I have too much to do when I get home to sit down) but I don't "workout". I walk my dog or tend to my garden but I don't go to the gym or do DVDs or even use my elliptical machine I have in the dining room. Could I really be losing inches instead of lbs even though I'm not doing crunches or weights or major workouts? Seems odd to me but while I want to be a little frustrated with the scale not moving, getting out of size 14s seems like a success and I am both happy and proud of that so I posted this in the success stories for that reason. I'm going to try to remember next Sunday (my next official weigh-in) to take my measurements too so I can start keeping track of those too, if I'd done that to start with then I'd know the answer to my question now.