How accurate are our fitness devices? Published study.
Options
MelanieCN77
Posts: 4,047 Member
I haven't dove too far in yet, but I intend to read through it all during the day.
https://www.mdpi.com/2075-4426/7/2/3/htm
Abstract: The ability to measure physical activity through wrist-worn devices provides an opportunity for cardiovascular medicine. However, the accuracy of commercial devices is largely unknown. The aim of this work is to assess the accuracy of seven commercially available wrist-worn devices in estimating heart rate (HR) and energy expenditure (EE) and to propose a wearable sensor evaluation framework. We evaluated the Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, Mio Alpha 2, PulseOn, and Samsung Gear S2. Participants wore devices while being simultaneously assessed with continuous telemetry and indirect calorimetry while sitting, walking, running, and cycling. Sixty volunteers (29 male, 31 female, age 38 ± 11 years) of diverse age, height, weight, skin tone, and fitness level were selected. Error in HR and EE was computed for each subject/device/activity combination. Devices reported the lowest error for cycling and the highest for walking. Device error was higher for males, greater body mass index, darker skin tone, and walking. Six of the devices achieved a median error for HR below 5% during cycling. No device achieved an error in EE below 20 percent. The Apple Watch achieved the lowest overall error in both HR and EE, while the Samsung Gear S2 reported the highest. In conclusion, most wrist-worn devices adequately measure HR in laboratory-based activities, but poorly estimate EE, suggesting caution in the use of EE measurements as part of health improvement programs. We propose reference standards for the validation of consumer health devices (http://precision.stanford.edu/).
https://www.mdpi.com/2075-4426/7/2/3/htm
Abstract: The ability to measure physical activity through wrist-worn devices provides an opportunity for cardiovascular medicine. However, the accuracy of commercial devices is largely unknown. The aim of this work is to assess the accuracy of seven commercially available wrist-worn devices in estimating heart rate (HR) and energy expenditure (EE) and to propose a wearable sensor evaluation framework. We evaluated the Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, Mio Alpha 2, PulseOn, and Samsung Gear S2. Participants wore devices while being simultaneously assessed with continuous telemetry and indirect calorimetry while sitting, walking, running, and cycling. Sixty volunteers (29 male, 31 female, age 38 ± 11 years) of diverse age, height, weight, skin tone, and fitness level were selected. Error in HR and EE was computed for each subject/device/activity combination. Devices reported the lowest error for cycling and the highest for walking. Device error was higher for males, greater body mass index, darker skin tone, and walking. Six of the devices achieved a median error for HR below 5% during cycling. No device achieved an error in EE below 20 percent. The Apple Watch achieved the lowest overall error in both HR and EE, while the Samsung Gear S2 reported the highest. In conclusion, most wrist-worn devices adequately measure HR in laboratory-based activities, but poorly estimate EE, suggesting caution in the use of EE measurements as part of health improvement programs. We propose reference standards for the validation of consumer health devices (http://precision.stanford.edu/).
2
Replies
-
Don't want to be fussy, but why test these and not include Garmin in the mix? Seems really odd that the one typically touted as the best is excluded.4
-
I just noticed that. Garmin is one of the more popular ones recommended here all the time.0
-
Tacklewasher wrote: »Don't want to be fussy, but why test these and not include Garmin in the mix? Seems really odd that the one typically touted as the best is excluded.
Don't most of the big brands use the same formulas? So the variance would be more relative to accuracy monitoring HR than turning that HR (and other stats) into calories burned?0 -
So yet another study where the researchers failed to like actually research how the devices improve their ability to estimate after being on the wearer over a period of time.
Except for one I'm not familiar with - all the others require being on the person for a week or two to learn what the resting HR is, where the HR is when getting into exercise zone, ect.
All in an effort to NOT use HR-based calorie burn when it's not appropriate and the most inflated.
Then it would be interesting to see if improvement could have been made for the majority of time by using the function most have of setting a custom stride length based on actual measurements.
Again, if defaults are used, no wonder the walking is bad estimate.
Would be interesting for them to compare a user selected activity level from 5 choices for daily activity and exercise, and see how that computation for daily calories burned compares to reality.8 -
So yet another study where the researchers failed to like actually research how the devices improve their ability to estimate after being on the wearer over a period of time.
Except for one I'm not familiar with - all the others require being on the person for a week or two to learn what the resting HR is, where the HR is when getting into exercise zone, ect.
All in an effort to NOT use HR-based calorie burn when it's not appropriate and the most inflated.
Then it would be interesting to see if improvement could have been made for the majority of time by using the function most have of setting a custom stride length based on actual measurements.
Again, if defaults are used, no wonder the walking is bad estimate.
Would be interesting for them to compare a user selected activity level from 5 choices for daily activity and exercise, and see how that computation for daily calories burned compares to reality.
All of this. I'm truly boggled by how many people who have taken the time to calibrate their devices who have worn them for some time, who have tracked data, and who find them to be pretty much spot on whenever I see "studies" like this.2 -
Tacklewasher wrote: »Don't want to be fussy, but why test these and not include Garmin in the mix? Seems really odd that the one typically touted as the best is excluded.
Don't most of the big brands use the same formulas? So the variance would be more relative to accuracy monitoring HR than turning that HR (and other stats) into calories burned?
It would seem not, as the HR and EE variances are quite different across the devices. In the summary - "In contrast with low reported error for HR measurement, no device met our prespecified error criterion for energy expenditure."
I've found a balance with my AW that is probably to do with style of logging and type of exercise and whatever that works for me long term. The truth is that it'll all be individual and we need to remember these devices are imperfect consumer products and the tales they tell will need to be parsed through the evidence we see on the scale.1 -
I don't know about EE, but the Apple Watch HR tracking is pretty spot on. My son was in a motorcycle accident this weekend and was in ICU specifically because of his elevated heart rate. His apple watch was tracking pretty much exactly what was showing from the EKG.3
-
I've had 3 different models.
One was as "Oprah's pick" I saw on GMA. I was out shopping and I got the "shake" telling me to get up and move. I realized that having my hand on the shopping cart didn't record any steps! I had been to Target, Marshalls and the grocery store, the only steps being recorded where the ones back and forth to my car.
My son bought me an "off brand" one while in China. I was on a walking tour with a group of friends, all of their units recorded over 10,000 steps...mine recorded under 3,000.
I now have a fitbit. I take it's accuracy with a grain of salt. There are times I'm driving and it records steps.
I pretty much don't count on them, but it is nice to know when I get a message or my phone is ringing.0 -
mom23mangos wrote: »I don't know about EE, but the Apple Watch HR tracking is pretty spot on. My son was in a motorcycle accident this weekend and was in ICU specifically because of his elevated heart rate. His apple watch was tracking pretty much exactly what was showing from the EKG.
OMG... how awful. Is your son okay?1 -
I've had 3 different models.
One was as "Oprah's pick" I saw on GMA. I was out shopping and I got the "shake" telling me to get up and move. I realized that having my hand on the shopping cart didn't record any steps! I had been to Target, Marshalls and the grocery store, the only steps being recorded where the ones back and forth to my car.
My son bought me an "off brand" one while in China. I was on a walking tour with a group of friends, all of their units recorded over 10,000 steps...mine recorded under 3,000.
I now have a fitbit. I take it's accuracy with a grain of salt. There are times I'm driving and it records steps.
I pretty much don't count on them, but it is nice to know when I get a message or my phone is ringing.
Counting "steps" and giving you a calorie burn for them are two different things. If you do a lot of driving over very bumpy roads in a car with bad suspension, you might get an inflated burn from Fitbit because of the way it measures intensity, but for most people, they're not going to get much of a calorie adjustment during normal driving, and any adjustment would likely be mitigated by missed steps elsewhere in their day.
3 -
GottaBurnEmAll wrote: »mom23mangos wrote: »I don't know about EE, but the Apple Watch HR tracking is pretty spot on. My son was in a motorcycle accident this weekend and was in ICU specifically because of his elevated heart rate. His apple watch was tracking pretty much exactly what was showing from the EKG.
OMG... how awful. Is your son okay?
Yes, he amazingly walked away with no broken bones or internal bleeding. He's still in the hospital however. His lungs are bruised which is causing fluid to build up and making is hard for him to breath...which is causing the elevated heart rate. He's having to stay on oxygen. But other than that, he's incredibly fine.14 -
Tacklewasher wrote: »Don't want to be fussy, but why test these and not include Garmin in the mix? Seems really odd that the one typically touted as the best is excluded.
It does seem like an odd mix-but At first glance, these look like older models. Which makes some sense as it takes time to fund and do these studies-but also taints the results against the current market when improvements in technology and programming are occurring so regularly.
Garmin didn’t have any wrist HRM models when Fitbit produced the Surge. The Surge was in the era right at or slightly before Garmin’s very first Vivofit.4 -
I made an interesting observation with my fitbit with HR sensor: In my old home I used to go to work by car. Here I cycle. Both takes about the same amount of time. I 'burned more calories' from going by car compared to cycling according to my fitbit! I guess driving car is so stressful while cycling is relaxing. Odd one.1
-
Tacklewasher wrote: »Don't want to be fussy, but why test these and not include Garmin in the mix? Seems really odd that the one typically touted as the best is excluded.
Don't most of the big brands use the same formulas? So the variance would be more relative to accuracy monitoring HR than turning that HR (and other stats) into calories burned?
It would be more related to when they realize NOT to use HR-based calculations for calorie burn.
The vast majority of the day should not be using that method - only workouts (and for best accuracy, only steady-stated aerobic workouts - not lifting or intervals).
So for HR-based calculations - they likely use very similar formula - but it's the extra personal details that can make a difference - how do they determine a resting HR, how do they estimate a HRmax, what formula for estimating VO2max?
Then for day to day stuff, how do they determine the cut-off for exercise? - high HR along with many steps, how is that line determined?
Then for rest of the day best measured without HR, how do they pick default stride length - several formula's available and I've seen differences, what formula for distance to calories?
Agreed the reading of HR and the step impacts which are built into the chipset of the accelerometer isn't the difference - it's what they do with the data received.3 -
Tacklewasher wrote: »Don't want to be fussy, but why test these and not include Garmin in the mix? Seems really odd that the one typically touted as the best is excluded.
Don't most of the big brands use the same formulas? So the variance would be more relative to accuracy monitoring HR than turning that HR (and other stats) into calories burned?
Garmin and Suunto both license FirstBeat algorithms. Polar rolls their own. Fitbit probably has their own math too.1 -
I noticed one odd thing in the abstract: why would darker skin tone cause the wearable to be less accurate? (Not that I am concerned, I have the typical pink white skin tone of my Scottish and Irish ancestors)0
-
I noticed one odd thing in the abstract: why would darker skin tone cause the wearable to be less accurate? (Not that I am concerned, I have the typical pink white skin tone of my Scottish and Irish ancestors)
Darker skin tone (and tattoos, hair, etc) affect the ability of the sensors to “see” through the skin.
2 -
I noticed one odd thing in the abstract: why would darker skin tone cause the wearable to be less accurate? (Not that I am concerned, I have the typical pink white skin tone of my Scottish and Irish ancestors)
Earlier led sensors had more trouble "seeing" through darker skin (or when the hand is cold).
This has been remediated to varying extents by the various sensor manufacturers (extra and different color LEDs, better mechanical contact, improved processing, and what have you).
The watch models in the study are less than current cutting edge.
And other valid criticisms of the study (including the lack of more accurate alternatives for the consumer) have been raised.
No one on MFP has ever said that you should take watch burns as Gospel and avoid comparing your expected results to your actual results over 4 to 6 weeks so that you can adjust.
Utlimately it really does not matter how accurately your watch or your logs estimate you calories in and out.
What does matter is that the calculations are *consistent* (not all over the place) giving you an opportunity to adjust based on how the perceived caloric balances are affecting your weight trend.4 -
I noticed one odd thing in the abstract: why would darker skin tone cause the wearable to be less accurate? (Not that I am concerned, I have the typical pink white skin tone of my Scottish and Irish ancestors)
Earlier led sensors had more trouble "seeing" through darker skin (or when the hand is cold).
This has been remediated to varying extents by the various sensor manufacturers (extra and different color LEDs, better mechanical contact, improved processing, and what have you).
The watch models in the study are less than current cutting edge. And valid criticisms (including the lack of more accurate alternatives) have been raised.
No one on MFP has ever said that you should take watch burns as Gospel and avoid comparing your expected results to your actual results over 4 to 6 weeks so that you can adjust.
Utlimately it really does not matter how accurately your watch or your logs estimate you calories in and out.
What does matter is that the calculations are *consistent* (not all over the place) giving you an opportunity to adjust based on how the perceived caloric balances are affecting your weight trend.
I have been using a wearable for about 5 years (first a Fitbit Flex, then a Garmin Vivofit2) and have found them to be accurate enough for my purposes, and they got more accurate the longer I had them and there was more data to parse.0 -
ive had different versions of the fitbit off and on over the years - currently have the charge 3.
for me, it seems pretty accurate regarding calories burned/exercise. the older versions did not seem as accurate, if i remember correctly (again, based on ME)0
Categories
- All Categories
- 1.4M Health, Wellness and Goals
- 391.3K Introduce Yourself
- 43.4K Getting Started
- 259.6K Health and Weight Loss
- 175.6K Food and Nutrition
- 47.3K Recipes
- 232.3K Fitness and Exercise
- 387 Sleep, Mindfulness and Overall Wellness
- 6.4K Goal: Maintaining Weight
- 8.5K Goal: Gaining Weight and Body Building
- 152.7K Motivation and Support
- 7.8K Challenges
- 1.3K Debate Club
- 96.2K Chit-Chat
- 2.5K Fun and Games
- 3.2K MyFitnessPal Information
- 22 News and Announcements
- 913 Feature Suggestions and Ideas
- 2.3K MyFitnessPal Tech Support Questions