Quick accuracy test: Calories from MFP vs HRM vs Power Meter
NorthCascades
Posts: 10,968 Member
Here's how the test happened. I rode a bike for about an hour with two Garmin units, one hooked up to a power meter, and the other to a chest strap. Then I entered the exercise into MFP to get its estimate. Both Garmins have the same user profile, and my weight gets synced from my Garmin account to my MFP account, so everybody is on the same page.
(1) Power meter was a Garmin Vector 2, dual sided, recorded with a GPS triathlete watch. A power meter is the gold standard for measuring energy use on a bike outside a laboratory.
(2) HRM is a Garmin Edge 800 bike computer, with an HRM-RUN chest strap. This is an older unit but what I had available. It has older Firstbeat energy expenditure software.
(3) MFP is MFP. I think it "cheated." I entered this after I did the ride, so Garmin had already fed it the right answer. It spit the correct answer back at me.
Here are the numbers.
PM: 11.32 miles, 54:46, 538 feet ele gain, 12.4 mph avg, 147 bpm avg, 150w avg, 493 kJ (~500 kCal)
HRM: 11.33 miles, 54:43, 535 feet ele gain, 12.4 mph avg, 148 bpm avg, 704 kCal
MFP: 492 kCal
Results: PM was the most accurate, MFP was second, HRM was the worst, significantly overestimating my calorie burn, by about 200 kCal per hour.
(1) Power meter was a Garmin Vector 2, dual sided, recorded with a GPS triathlete watch. A power meter is the gold standard for measuring energy use on a bike outside a laboratory.
(2) HRM is a Garmin Edge 800 bike computer, with an HRM-RUN chest strap. This is an older unit but what I had available. It has older Firstbeat energy expenditure software.
(3) MFP is MFP. I think it "cheated." I entered this after I did the ride, so Garmin had already fed it the right answer. It spit the correct answer back at me.
Here are the numbers.
PM: 11.32 miles, 54:46, 538 feet ele gain, 12.4 mph avg, 147 bpm avg, 150w avg, 493 kJ (~500 kCal)
HRM: 11.33 miles, 54:43, 535 feet ele gain, 12.4 mph avg, 148 bpm avg, 704 kCal
MFP: 492 kCal
Results: PM was the most accurate, MFP was second, HRM was the worst, significantly overestimating my calorie burn, by about 200 kCal per hour.
1
Replies
-
By the way, before you judge my slowness too harshly, please remember this was downtown Seattle, with traffic.0
-
Very interesting, @NorthCascades. You've got me seriously thinking about a power meter based on this and multiple other posts you've made.
But I'm only a recreational biker using a mountain bike in the countryside in warm weather. Do you have recommendations for decent power meters that won't cost more than my bike?1 -
Thanks, @NorthCascades - I know this was a follow-on to a discussion we had on another thread.
One question: You say that the PM was the most accurate. I'm assuming that's because it's the gold standard, and that for that conclusion you're relying on others' confirming research that somehow linked the PM results to measured calorie expenditure. Is that correct, or am I missing something?
Has anyone else out there in MFP-land kept any calorie comparisons between gym-machine & HRM, HRM & MFP, metered effort (like NorthCascades' PM or the C2 rower) and MFP or HRM, etc.?
There are so many assertions in the forums about calorie burns being incorrect or overstated, or about certain estimation methods being better in various circumstances - I've been wondering whether the differences are wide enough to have practical implications (as we always seem to assume), or not; and whether there's any pattern to discrepancies.
From memory only, I've found MFP closer to my HRM for MFP DB entries that inherently involve some measurement benchmarks (like "walking at X mph", at least when the terrain is kind of average), vs. for example, say, "aerobics" that can vary wildly in form and effort.
A silly example I posted last night (warm up for a longer interval activity) was 10 minutes, for which Concept 2 rowing machine said 96 calories (after weight adjustment per C2 website), and Polar HRM said 67 calories, though it seems as if I've seen closer results (on a percentage basis) for rowing workouts in the past.
I'll be on the lookout for a more rational C2 rowing machine workout I can post on this thread along with Polar HRM results - I'm a lazier and much weaker character than NorthCascades (
) so I'm not likely to do one just for the sake of Science. The one I posted last night seemed pretty lame because it was a too-short piece, and a warm-up for an activity to which I'm well-adapted, so it takes a while to get my HR up to what I'd consider a normal working level. 2 -
Very interesting, @NorthCascades. You've got me seriously thinking about a power meter based on this and multiple other posts you've made.
But I'm only a recreational biker using a mountain bike in the countryside in warm weather. Do you have recommendations for decent power meters that won't cost more than my bike?
There's been a price war in this market for a few years, and it's not over yet. It probably makes sense to wait a bit, but given where we're at today, and for a mountain bike, I'd think your best option would be some type of crank-based unit. I'd probably consider these:
* 4iiii
* Stages
* WatTeam Gen2 (maybe)
I personally have some reservations about all three of those. And with all the development going on in this field and constant price drop, it might be a good time to sit back and watch.2 -
This is a different thing, but I think it's in line with the other thread where this discussion started, so I hope it's OK if I post it here (slightly off topic for either thread).
Today, I took a brisk walk, which is not a usual form of exercise for me (bad knee). Per an old Garmin (speed & distance, no HR), I averaged 3.6mph.
My Polar HRM gave me 195 calories. MFP, via the standard "Walking, 3.5 mph, brisk pace" entry, would give me 183 calories. So, they're very close estimates. (This was level to slightly rolling terrain, on a roadside.)1 -
Thanks, @NorthCascades - I know this was a follow-on to a discussion we had on another thread.
One question: You say that the PM was the most accurate. I'm assuming that's because it's the gold standard, and that for that conclusion you're relying on others' confirming research that somehow linked the PM results to measured calorie expenditure. Is that correct, or am I missing something?
Has anyone else out there in MFP-land kept any calorie comparisons between gym-machine & HRM, HRM & MFP, metered effort (like NorthCascades' PM or the C2 rower) and MFP or HRM, etc.?
There are so many assertions in the forums about calorie burns being incorrect or overstated, or about certain estimation methods being better in various circumstances - I've been wondering whether the differences are wide enough to have practical implications (as we always seem to assume), or not; and whether there's any pattern to discrepancies.
From memory only, I've found MFP closer to my HRM for MFP DB entries that inherently involve some measurement benchmarks (like "walking at X mph", at least when the terrain is kind of average), vs. for example, say, "aerobics" that can vary wildly in form and effort.
A silly example I posted last night (warm up for a longer interval activity) was 10 minutes, for which Concept 2 rowing machine said 96 calories (after weight adjustment per C2 website), and Polar HRM said 67 calories, though it seems as if I've seen closer results (on a percentage basis) for rowing workouts in the past.
I'll be on the lookout for a more rational C2 rowing machine workout I can post on this thread along with Polar HRM results - I'm a lazier and much weaker character than NorthCascades ( ) so I'm not likely to do one just for the sake of Science. The one I posted last night seemed pretty lame because it was a too-short piece, and a warm-up for an activity to which I'm well-adapted, so it takes a while to get my HR up to what I'd consider a normal working level.
I am not an expert in power meters, but, assuming that they are accurate at measuring workload (and for the cost, they should be), a PM would almost by definition have to be more accurate than an HRM. Especially for cycling.
HRMs have three big issues: one, their algorithms are "best fit" based on a fairly limited sample size, so you have a decent error factor right there; two, they make too many assumptions about fitness level and HR max--now if one knew one's VO2 max and HRmax and could program those in, it would improve the accuracy for steady-state cardio--up to a point; and that leads to three, there are a number of factors that skew the already approximated HR/VO2 relationship that the HRM cannot begin to account for--e.g. Thermal stress, cardiovascular drift, body posture, amount of arm work or overhead work involved, fatigue, stress, etc, etc. It's actually a wonder that some HRMs can get within 20%-25% in certain cases.
Personally I think that when MFP and an HRM "agree", it is more random chance than an indication that either one is accurate.
2 -
Thanks, @NorthCascades - I know this was a follow-on to a discussion we had on another thread.
One question: You say that the PM was the most accurate. I'm assuming that's because it's the gold standard, and that for that conclusion you're relying on others' confirming research that somehow linked the PM results to measured calorie expenditure. Is that correct, or am I missing something?
Has anyone else out there in MFP-land kept any calorie comparisons between gym-machine & HRM, HRM & MFP, metered effort (like NorthCascades' PM or the C2 rower) and MFP or HRM, etc.?
There are so many assertions in the forums about calorie burns being incorrect or overstated, or about certain estimation methods being better in various circumstances - I've been wondering whether the differences are wide enough to have practical implications (as we always seem to assume), or not; and whether there's any pattern to discrepancies.
From memory only, I've found MFP closer to my HRM for MFP DB entries that inherently involve some measurement benchmarks (like "walking at X mph", at least when the terrain is kind of average), vs. for example, say, "aerobics" that can vary wildly in form and effort.
A silly example I posted last night (warm up for a longer interval activity) was 10 minutes, for which Concept 2 rowing machine said 96 calories (after weight adjustment per C2 website), and Polar HRM said 67 calories, though it seems as if I've seen closer results (on a percentage basis) for rowing workouts in the past.
I'll be on the lookout for a more rational C2 rowing machine workout I can post on this thread along with Polar HRM results - I'm a lazier and much weaker character than NorthCascades ( ) so I'm not likely to do one just for the sake of Science. The one I posted last night seemed pretty lame because it was a too-short piece, and a warm-up for an activity to which I'm well-adapted, so it takes a while to get my HR up to what I'd consider a normal working level.
I am not an expert in power meters, but, assuming that they are accurate at measuring workload (and for the cost, they should be), a PM would almost by definition have to be more accurate than an HRM. Especially for cycling.
HRMs have three big issues: one, their algorithms are "best fit" based on a fairly limited sample size, so you have a decent error factor right there; two, they make too many assumptions about fitness level and HR max--now if one knew one's VO2 max and HRmax and could program those in, it would improve the accuracy for steady-state cardio--up to a point; and that leads to three, there are a number of factors that skew the already approximated HR/VO2 relationship that the HRM cannot begin to account for--e.g. Thermal stress, cardiovascular drift, body posture, amount of arm work or overhead work involved, fatigue, stress, etc, etc. It's actually a wonder that some HRMs can get within 20%-25% in certain cases.
Personally I think that when MFP and an HRM "agree", it is more random chance than an indication that either one is accurate.
Two points, both minor:
As you know, some HRM, including the one I use, do know one's HRmax. (And I know mine, from an exercise max test - not a cardiac stress test. I had one of the latter, too, but those cowards made me stop running on the treadmill at only somewhat near my 220-age level, which all of us knew was not true max, because I had breath enough to argue with them about it while still running . . . but they had their cardiac data, so we stopped. ).
I read frequently on MFP, that the MFP database vastly overestimates calorie burn, that HRM are not accurate either, that power metering is substantially more accurate, and other assertions along similar lines. My interest is in seeing some sample points from various people, to loosely quantify what those differences are in real life.
Some things I can observe myself, like the amount by which my HR is higher for similar activity when it's very hot outside (same boat, roughly same speed, same distance), higher if I'm dehydrated, etc. But what I've seen myself in the way of differences is not vast, in most cases. It's often in the same percentage ballpark as the error I'd expect from food calories (20% difference allowed from package calorie label, IIRC).
It seems like it would be interesting to see various people's various data points in this comparative space, just to get a feel for what's experienced in real life as far as differences between the exercise estimating methods. Folks trying to quantify exercuse calories out could benefit from general information, IMO, even absent precise formulas. We're going to make estimates one way or another. What are best practices?
I personally haven't found much structured research that sheds light on this, but I'm not an expert in this area (as I know you are, Azdak), and probably don't have the best handle on how to find it if it exists.
1 -
Saturday's ride:
Distance - 212.7 km
Elevation - 2,123 m (6965 ft)
Moving Time - 11:40:27
Elapsed Time - 12:48:25
Speed - Avg: 18.2 km/h; Max: 50.4km/h
Heart Rate - Avg: 144bpm; Max: 176bpm
My estimate of 100 cal/5 km comes to 4260
MFP gave me 3927
Strava's comes to 3399
This HR calculator (linked below) gave me 6708
http://www.shapesense.com/fitness-exercise/calculators/heart-rate-based-calorie-burn-calculator.shtml2 -
Saturday's ride:
Distance - 212.7 km
Elevation - 2,123 m (6965 ft)
Moving Time - 11:40:27
Elapsed Time - 12:48:25
Speed - Avg: 18.2 km/h; Max: 50.4km/h
Heart Rate - Avg: 144bpm; Max: 176bpm
My estimate of 100 cal/5 km comes to 4260
MFP gave me 3927
Strava's comes to 3399
This HR calculator (linked below) gave me 6708
http://www.shapesense.com/fitness-exercise/calculators/heart-rate-based-calorie-burn-calculator.shtml
Very interesting: Thank you.
FWIW, that calculator gives me calorie estimates on the order of 60% above the ones my own HRM gives me.
409 for a spin class vs. 257
405 for a rowing machine workout vs. 260
733 for a short bike ride vs. 4660 -
Saturday's ride:
Distance - 212.7 km
Elevation - 2,123 m (6965 ft)
Moving Time - 11:40:27
Elapsed Time - 12:48:25
Speed - Avg: 18.2 km/h; Max: 50.4km/h
Heart Rate - Avg: 144bpm; Max: 176bpm
My estimate of 100 cal/5 km comes to 4260
MFP gave me 3927
Strava's comes to 3399
This HR calculator (linked below) gave me 6708
http://www.shapesense.com/fitness-exercise/calculators/heart-rate-based-calorie-burn-calculator.shtml
Very interesting: Thank you.
FWIW, that calculator gives me calorie estimates on the order of 60% above the ones my own HRM gives me.
409 for a spin class vs. 257
405 for a rowing machine workout vs. 260
733 for a short bike ride vs. 466
We only wish that were actually true!!
I'd be fighting to say out of the underweight range on the BMI chart if it were. 
I plugged in my husband's numbers too for that ride ... he rode with me. According to Strava and MFP, he burned a little bit more than I did which isn't a surprise (about 4000 or 4200 or so) ... and yet that heart rate calculator more than doubled his calorie burn too ... it gave him 9600 calories!!!0 -
This is a different thing, but I think it's in line with the other thread where this discussion started, so I hope it's OK if I post it here (slightly off topic for either thread).
Today, I took a brisk walk, which is not a usual form of exercise for me (bad knee). Per an old Garmin (speed & distance, no HR), I averaged 3.6mph.
My Polar HRM gave me 195 calories. MFP, via the standard "Walking, 3.5 mph, brisk pace" entry, would give me 183 calories. So, they're very close estimates. (This was level to slightly rolling terrain, on a roadside.)
What did Garmin suggest based on the GPS track?
0 -
One question: You say that the PM was the most accurate. I'm assuming that's because it's the gold standard, and that for that conclusion you're relying on others' confirming research that somehow linked the PM results to measured calorie expenditure. Is that correct, or am I missing something?
The snag with looking for research is, what discipline is relevant and what's going to generate the research?
Looking at this from a control engineering perspective, what you're talking about is how to take a measurement, then feed back based on that measurement and make decisions based on that. So there is a degree of accuracy, a degree of meaningfulness and a question of whether one is measuring what one wants to measure, or measuring something else and then extrapolating from that.
The latter is key when we're talking about convenient ways to measure calorie expenditure. Measuring calorie expenditure directly is time consuming, expensive and inconvenient. Measuring a proxy, and extrapolating from there is easier. The meaningfulness of the proxy depends on whether there is an established, consistent and predictable relationship.
In that sense there is a direct, linear relationship between power output and energy expenditure. So straight away you're in a good place. Using HR as a proxy the relationship is non linear. If HR is too low then the proportion of energy expended as exercise is too low to be meaningful, and if it's too high then HR isn't representing the chemical processes. Between the aerobic and lactate thresholds it's not too bad. Not quite linear, but close enough. Linearity breaks down between lactate and anaerobic thresholds and is unpredictable above.
There is also a degree of where in the process one is instrumenting. Essentially how close is the proxy to the metric we're interested in, how are we establishing that and how stable is the measure? Power meter implementations measure at a very predictable point; pedal, crank or hub. The running power meters in development measure push off and landing effects at the footstrike. An HRM measures the number of times the heart pumps, so variables include stroke volume, blood oxygen concentration, blood viscosity (which increases over training time), glycogen levels etc.
I'd add that you've got responsiveness of the measure. Power meters respond to changes in power output. Noting the point above about value in mountain biking, the nature of power transfer in mountain biking is less predictable than road biking, and I'd agree with the point about crank rather than pedal based, but the error isn't particularly material. HR doesn't respond immediately. Just think about the time taken to recover after stopping an exertion phase.
There is a lot of discussion on here of HR in absolute terms, rather than recognising that as a tool it has many weaknesses, but the importance of those varies depending on how it's used. HRMs are low cost, convenient, but need to be used with thought. Power meters remain disproportionately expensive, and require an expensive head-end to make the most of. As a method of calorie estimation that probably makes them unreasonable, but they are extremely capable training tools for cyclists. There is little point in recommending a power meter for a swimmer...
With all that in mind, is there scope for comparative research? I'd be very surprised. It's a fairly simple exercise to demonstrate that HRMs aren't a particularly meaningful tool, but in an industry that's about selling devices to people there is no demand for research that confirms that the high volume, low margin, device isn't a lot better than throwing two dice.
There is ongoing work on how to improve estimations, using HR as one of a number of inputs. You're not likely to see that work embodied in entry level devices, which is where the majority of discussions on here are focussed.
1 -
I am not an expert in power meters, but, assuming that they are accurate at measuring workload (and for the cost, they should be), a PM would almost by definition have to be more accurate than an HRM. Especially for cycling.
Power meters for bikes come with accuracy ratings. Mine has an allowable error of up to 2 %.
Mine is in the pedals, as "upstream" as possible. Meaning it's the best available measure of what I did, instead of what reached the bike. We could put a second meter in the rear wheel and it would read a few watts lower because of losses through the drive train.0 -
HRMs have three big issues: one, their algorithms are "best fit" based on a fairly limited sample size, so you have a decent error factor right there; two, they make too many assumptions about fitness level and HR max--now if one knew one's VO2 max and HRmax and could program those in, it would improve the accuracy for steady-state cardio--up to a point; and that leads to three, there are a number of factors that skew the already approximated HR/VO2 relationship that the HRM cannot begin to account for--e.g. Thermal stress, cardiovascular drift, body posture, amount of arm work or overhead work involved, fatigue, stress, etc, etc. It's actually a wonder that some HRMs can get within 20%-25% in certain cases.
For the record, these exist. My watch measurestimates my VO2max every time I run or bike. It uses the Garmin/Firstbeat software that requires HR+pace for running and HR+power for cycling. Keeps track of two separate VO2max values for me. I can see how they've changed, going back a couple of years. I trust it fairly well for Nordic skiing where I really don't have a better option.
But that's not the HRM I used for this test. I used a Garmin Edge 800. We can see how a Fenix 3 does as an HRM guesser when the Fenix 5 is delivered.0 -
NorthCascades wrote: »I am not an expert in power meters, but, assuming that they are accurate at measuring workload (and for the cost, they should be), a PM would almost by definition have to be more accurate than an HRM. Especially for cycling.
Power meters for bikes come with accuracy ratings. Mine has an allowable error of up to 2 %.
Mine is in the pedals, as "upstream" as possible. Meaning it's the best available measure of what I did, instead of what reached the bike. We could put a second meter in the rear wheel and it would read a few watts lower because of losses through the drive train.
One should note though that although the accuracy of the power meter is 2%, the biomechanical efficiency of turning calories into power has a range of 20-25%. I do not know how what the statistics are on how efficiencies vary among the population, (i.e. what the standard deviation for efficiency), but that still means even with a power meter the calorie calculations could be off by quite a bit.
Adding another data point to the HRM calorie calculations, I've been using an optical Scosche HRM for quite a while since the Garmin straps tend to die on me after 4-6 months. I recently got a new Garmin HRM-Run chest strap because the optical HRM was giving wacky reading when riding my bike in cooler weather. I've been using it for runs to test it out and see what kind of running dynamics I get, and my calorie counts from the HRM have been consistently 10-20% lower than with the optical HRM. I don't use the HRM calorie counts anyway, relying on the distance formula (0.63 * weight * distance). That formula has been reliable for me maintaining my race weight for coming on two years with mileage varying between 25-60 MPW.0 -
I think what some people are missing is that Watts = 859.85 Calories per hour. They are measuring the same thing. Watts on a bike are usually measured at the wheel or crank. The efficiency is pretty close to 100%. How many watts you put down gives an exact number of calories burnt for that activity. Running relies on efficency, weight, and speed. You can burn less calories by being a more efficient runner and the bio-mechanical efficiency of running is 20-25%. The bio-mechanical efficiency of swimming is even worse and can run in the 2-7% range that is why it is so hard to figure out calorie expenditure for these activities for a given distance.1
-
I use one of the left-hand crank mounted 4iiii power meters on my road bike and have been happy with it so far. It's probably the cheapest option at the moment.
Not a clue how it impacts calorie burned estimates though!1 -
One thing that's pretty neat about power meters and calories, is that they don't include BMR, all of the calories are legit. That's just a side effect of how they work. Power meters measure how much you bend part of your bike and how often; if you stayed on the couch it would say zero.1
-
I'd be interested to hear if others have done some checking in this regard.
0 -
Not real bike, but out on the fringe of calorie estimation: My (non-calibrate-able for weight/age/etc.) spin bike today said 568 calories, for a short (45-50 minute) spin class, for which my Polar HRM (calibrated for all of the above, including actual HRmax) said 231. Based on RPE alone, 568 would be nuts, but I'm small (5'5", 120s) so don't 'earn' much.
I've never looked at the spin bike calories before, but someone else in the class asked how it compared to my HRM estimate. Her bike, for the same class today, said something in the high 300s, and her fitness tracker (not sure which brand) gave her 250-something IIRC. She's only slightly older, but heavier & a little less generally active, plus on a beta-blocker so lowered HR, FWIW.
"Eat back 50%" could be generous . . . . 0
This discussion has been closed.
Categories
- All Categories
- 1.4M Health, Wellness and Goals
- 393.4K Introduce Yourself
- 43.8K Getting Started
- 260.2K Health and Weight Loss
- 175.9K Food and Nutrition
- 47.4K Recipes
- 232.5K Fitness and Exercise
- 426 Sleep, Mindfulness and Overall Wellness
- 6.5K Goal: Maintaining Weight
- 8.5K Goal: Gaining Weight and Body Building
- 153K Motivation and Support
- 8K Challenges
- 1.3K Debate Club
- 96.3K Chit-Chat
- 2.5K Fun and Games
- 3.7K MyFitnessPal Information
- 24 News and Announcements
- 1.1K Feature Suggestions and Ideas
- 2.6K MyFitnessPal Tech Support Questions