Burned calories and fitness levels

So this makes sense to me but I want to see if it's right.

I use an HRM and as my fitness levels are increasing, I'm burning fewer calories now than when I was less fit. With most of my exercise, I ramp it up each time, but my walk to and from work is unchanged. My HRM shows a lower number each week for that exercise.

Does that sound right?