How accurate are our fitness devices? Published study.

I haven't dove too far in yet, but I intend to read through it all during the day.

https://www.mdpi.com/2075-4426/7/2/3/htm

Abstract: The ability to measure physical activity through wrist-worn devices provides an opportunity for cardiovascular medicine. However, the accuracy of commercial devices is largely unknown. The aim of this work is to assess the accuracy of seven commercially available wrist-worn devices in estimating heart rate (HR) and energy expenditure (EE) and to propose a wearable sensor evaluation framework. We evaluated the Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, Mio Alpha 2, PulseOn, and Samsung Gear S2. Participants wore devices while being simultaneously assessed with continuous telemetry and indirect calorimetry while sitting, walking, running, and cycling. Sixty volunteers (29 male, 31 female, age 38 ± 11 years) of diverse age, height, weight, skin tone, and fitness level were selected. Error in HR and EE was computed for each subject/device/activity combination. Devices reported the lowest error for cycling and the highest for walking. Device error was higher for males, greater body mass index, darker skin tone, and walking. Six of the devices achieved a median error for HR below 5% during cycling. No device achieved an error in EE below 20 percent. The Apple Watch achieved the lowest overall error in both HR and EE, while the Samsung Gear S2 reported the highest. In conclusion, most wrist-worn devices adequately measure HR in laboratory-based activities, but poorly estimate EE, suggesting caution in the use of EE measurements as part of health improvement programs. We propose reference standards for the validation of consumer health devices (http://precision.stanford.edu/).
«1

Replies

  • Tacklewasher
    Tacklewasher Posts: 7,122 Member
    Don't want to be fussy, but why test these and not include Garmin in the mix? Seems really odd that the one typically touted as the best is excluded.
  • MelanieCN77
    MelanieCN77 Posts: 4,047 Member
    I just noticed that. Garmin is one of the more popular ones recommended here all the time.
  • jjpptt2
    jjpptt2 Posts: 5,650 Member
    Don't want to be fussy, but why test these and not include Garmin in the mix? Seems really odd that the one typically touted as the best is excluded.

    Don't most of the big brands use the same formulas? So the variance would be more relative to accuracy monitoring HR than turning that HR (and other stats) into calories burned?
  • GottaBurnEmAll
    GottaBurnEmAll Posts: 7,722 Member
    heybales wrote: »
    So yet another study where the researchers failed to like actually research how the devices improve their ability to estimate after being on the wearer over a period of time.

    Except for one I'm not familiar with - all the others require being on the person for a week or two to learn what the resting HR is, where the HR is when getting into exercise zone, ect.

    All in an effort to NOT use HR-based calorie burn when it's not appropriate and the most inflated.

    Then it would be interesting to see if improvement could have been made for the majority of time by using the function most have of setting a custom stride length based on actual measurements.

    Again, if defaults are used, no wonder the walking is bad estimate.

    Would be interesting for them to compare a user selected activity level from 5 choices for daily activity and exercise, and see how that computation for daily calories burned compares to reality.

    All of this. I'm truly boggled by how many people who have taken the time to calibrate their devices who have worn them for some time, who have tracked data, and who find them to be pretty much spot on whenever I see "studies" like this.
  • MelanieCN77
    MelanieCN77 Posts: 4,047 Member
    jjpptt2 wrote: »
    Don't want to be fussy, but why test these and not include Garmin in the mix? Seems really odd that the one typically touted as the best is excluded.

    Don't most of the big brands use the same formulas? So the variance would be more relative to accuracy monitoring HR than turning that HR (and other stats) into calories burned?

    It would seem not, as the HR and EE variances are quite different across the devices. In the summary - "In contrast with low reported error for HR measurement, no device met our prespecified error criterion for energy expenditure."

    I've found a balance with my AW that is probably to do with style of logging and type of exercise and whatever that works for me long term. The truth is that it'll all be individual and we need to remember these devices are imperfect consumer products and the tales they tell will need to be parsed through the evidence we see on the scale.
  • mom23mangos
    mom23mangos Posts: 3,069 Member
    I don't know about EE, but the Apple Watch HR tracking is pretty spot on. My son was in a motorcycle accident this weekend and was in ICU specifically because of his elevated heart rate. His apple watch was tracking pretty much exactly what was showing from the EKG.
  • Allgaun
    Allgaun Posts: 222 Member
    I've had 3 different models.

    One was as "Oprah's pick" I saw on GMA. I was out shopping and I got the "shake" telling me to get up and move. I realized that having my hand on the shopping cart didn't record any steps! I had been to Target, Marshalls and the grocery store, the only steps being recorded where the ones back and forth to my car.

    My son bought me an "off brand" one while in China. I was on a walking tour with a group of friends, all of their units recorded over 10,000 steps...mine recorded under 3,000.

    I now have a fitbit. I take it's accuracy with a grain of salt. There are times I'm driving and it records steps.

    I pretty much don't count on them, but it is nice to know when I get a message or my phone is ringing.
  • GottaBurnEmAll
    GottaBurnEmAll Posts: 7,722 Member
    I don't know about EE, but the Apple Watch HR tracking is pretty spot on. My son was in a motorcycle accident this weekend and was in ICU specifically because of his elevated heart rate. His apple watch was tracking pretty much exactly what was showing from the EKG.

    OMG... how awful. Is your son okay?
  • GottaBurnEmAll
    GottaBurnEmAll Posts: 7,722 Member
    Allgaun wrote: »
    I've had 3 different models.

    One was as "Oprah's pick" I saw on GMA. I was out shopping and I got the "shake" telling me to get up and move. I realized that having my hand on the shopping cart didn't record any steps! I had been to Target, Marshalls and the grocery store, the only steps being recorded where the ones back and forth to my car.

    My son bought me an "off brand" one while in China. I was on a walking tour with a group of friends, all of their units recorded over 10,000 steps...mine recorded under 3,000.

    I now have a fitbit. I take it's accuracy with a grain of salt. There are times I'm driving and it records steps.

    I pretty much don't count on them, but it is nice to know when I get a message or my phone is ringing.

    Counting "steps" and giving you a calorie burn for them are two different things. If you do a lot of driving over very bumpy roads in a car with bad suspension, you might get an inflated burn from Fitbit because of the way it measures intensity, but for most people, they're not going to get much of a calorie adjustment during normal driving, and any adjustment would likely be mitigated by missed steps elsewhere in their day.

  • Duck_Puddle
    Duck_Puddle Posts: 3,237 Member
    edited January 2019
    Don't want to be fussy, but why test these and not include Garmin in the mix? Seems really odd that the one typically touted as the best is excluded.

    It does seem like an odd mix-but At first glance, these look like older models. Which makes some sense as it takes time to fund and do these studies-but also taints the results against the current market when improvements in technology and programming are occurring so regularly.

    Garmin didn’t have any wrist HRM models when Fitbit produced the Surge. The Surge was in the era right at or slightly before Garmin’s very first Vivofit.
  • yirara
    yirara Posts: 9,853 Member
    I made an interesting observation with my fitbit with HR sensor: In my old home I used to go to work by car. Here I cycle. Both takes about the same amount of time. I 'burned more calories' from going by car compared to cycling according to my fitbit! I guess driving car is so stressful while cycling is relaxing. Odd one.
  • heybales
    heybales Posts: 18,842 Member
    jjpptt2 wrote: »
    Don't want to be fussy, but why test these and not include Garmin in the mix? Seems really odd that the one typically touted as the best is excluded.

    Don't most of the big brands use the same formulas? So the variance would be more relative to accuracy monitoring HR than turning that HR (and other stats) into calories burned?

    It would be more related to when they realize NOT to use HR-based calculations for calorie burn.

    The vast majority of the day should not be using that method - only workouts (and for best accuracy, only steady-stated aerobic workouts - not lifting or intervals).

    So for HR-based calculations - they likely use very similar formula - but it's the extra personal details that can make a difference - how do they determine a resting HR, how do they estimate a HRmax, what formula for estimating VO2max?

    Then for day to day stuff, how do they determine the cut-off for exercise? - high HR along with many steps, how is that line determined?
    Then for rest of the day best measured without HR, how do they pick default stride length - several formula's available and I've seen differences, what formula for distance to calories?

    Agreed the reading of HR and the step impacts which are built into the chipset of the accelerometer isn't the difference - it's what they do with the data received.
  • NorthCascades
    NorthCascades Posts: 10,968 Member
    jjpptt2 wrote: »
    Don't want to be fussy, but why test these and not include Garmin in the mix? Seems really odd that the one typically touted as the best is excluded.

    Don't most of the big brands use the same formulas? So the variance would be more relative to accuracy monitoring HR than turning that HR (and other stats) into calories burned?

    Garmin and Suunto both license FirstBeat algorithms. Polar rolls their own. Fitbit probably has their own math too.
  • earlnabby
    earlnabby Posts: 8,171 Member
    I noticed one odd thing in the abstract: why would darker skin tone cause the wearable to be less accurate? (Not that I am concerned, I have the typical pink white skin tone of my Scottish and Irish ancestors)
  • Duck_Puddle
    Duck_Puddle Posts: 3,237 Member
    earlnabby wrote: »
    I noticed one odd thing in the abstract: why would darker skin tone cause the wearable to be less accurate? (Not that I am concerned, I have the typical pink white skin tone of my Scottish and Irish ancestors)

    Darker skin tone (and tattoos, hair, etc) affect the ability of the sensors to “see” through the skin.

  • PAV8888
    PAV8888 Posts: 14,139 Member
    edited January 2019
    earlnabby wrote: »
    I noticed one odd thing in the abstract: why would darker skin tone cause the wearable to be less accurate? (Not that I am concerned, I have the typical pink white skin tone of my Scottish and Irish ancestors)

    Earlier led sensors had more trouble "seeing" through darker skin (or when the hand is cold).

    This has been remediated to varying extents by the various sensor manufacturers (extra and different color LEDs, better mechanical contact, improved processing, and what have you).

    The watch models in the study are less than current cutting edge.

    And other valid criticisms of the study (including the lack of more accurate alternatives for the consumer) have been raised.

    No one on MFP has ever said that you should take watch burns as Gospel and avoid comparing your expected results to your actual results over 4 to 6 weeks so that you can adjust.

    Utlimately it really does not matter how accurately your watch or your logs estimate you calories in and out.

    What does matter is that the calculations are *consistent* (not all over the place) giving you an opportunity to adjust based on how the perceived caloric balances are affecting your weight trend.
  • earlnabby
    earlnabby Posts: 8,171 Member
    PAV8888 wrote: »
    earlnabby wrote: »
    I noticed one odd thing in the abstract: why would darker skin tone cause the wearable to be less accurate? (Not that I am concerned, I have the typical pink white skin tone of my Scottish and Irish ancestors)

    Earlier led sensors had more trouble "seeing" through darker skin (or when the hand is cold).

    This has been remediated to varying extents by the various sensor manufacturers (extra and different color LEDs, better mechanical contact, improved processing, and what have you).

    The watch models in the study are less than current cutting edge. And valid criticisms (including the lack of more accurate alternatives) have been raised.

    No one on MFP has ever said that you should take watch burns as Gospel and avoid comparing your expected results to your actual results over 4 to 6 weeks so that you can adjust.

    Utlimately it really does not matter how accurately your watch or your logs estimate you calories in and out.

    What does matter is that the calculations are *consistent* (not all over the place) giving you an opportunity to adjust based on how the perceived caloric balances are affecting your weight trend.

    I have been using a wearable for about 5 years (first a Fitbit Flex, then a Garmin Vivofit2) and have found them to be accurate enough for my purposes, and they got more accurate the longer I had them and there was more data to parse.
  • callsitlikeiseeit
    callsitlikeiseeit Posts: 8,626 Member
    ive had different versions of the fitbit off and on over the years - currently have the charge 3.

    for me, it seems pretty accurate regarding calories burned/exercise. the older versions did not seem as accurate, if i remember correctly (again, based on ME)