Wondering How This All Works

I've been here for quite some time and made awesome progress, but now I'm starting to question things...

How exactly does this whole weight loss/fat loss work? I know that 3,500 calories are supposed to equal a pound, but how does the body differentiate between food calories consumed that day and the fat your body holds? Does it go through the food calories first? And where does the fat go? I know the cells themselves never go away, but they shrink. Where does that "fat filling" go? Do we sweat it out? Does it pass through us as waste?

And I seem to lose weight much faster when I cut my calories severely and don't work out, rather than eat a healthy amount and work out like crazy. Why is this? If 3,500 calories equal a pound, shouldn't they both work equally as long as the deficit is the same?

Just curious, so don't give me the whole starvation mode lecture or tell me to cut out carbs or go paleo. And dear god, please don't even mention drinking nothing but shakes and doing insane workout dvds for a month. Please, I've heard it all before.