"Fitness trackers are largely inaccurate when counting calories, Stanford researchers say"
Options
Sunna_W
Posts: 744 Member
It seems like everyone has a tracker these days. How do you use yours and do you eat your calories back that are burned according to your tracking device? Will you change how you use your data based on the information provided in the article below?
I'll go first:
I don't eat back calories burned at all and this will not affect how I use my FitBit Alta HR. I use mine to track resting heart rate and sleep. My device says that I have walked 300+ steps / burned X number of calories while I am sleeping based upon my motion in bed, and if I get up during the night.
Calorie counting is a useful way to lose weight, but a new study suggests a fitness tracker could sabotage your efforts.
The devices are overwhelmingly popular. For instance, since its inception, the leading brand, Fitbit, has sold at least 30 million of them. The company promises on its website that the devices “track steps, distance, calories burned, floors climbed, active minutes & hourly activity.” Others, such as PulseOn, Apple Watch, Basis Peak, Samsung Gear S2 and Microsoft Band, promise the same.
A team of Stanford researchers, however, recently called foul after testing these trackers. The scientists said in a paper published Wednesday in the Journal of Personalized Medicine that though the devices purport to help users track their calories — daily energy expenditure — the number is often markedly incorrect.
The least accurate, PulseOn, was off by an average of 93 percent. The most accurate device, Fitbit Surge, was off by an average of 27 percent, the Guardian reported.
In a statement to NPR, PulseOn said the extremely high level of inaccuracy may “suggest that the authors may not have properly set all the user parameters on the device.”
The consequences of such large margins of error could, of course, be significant.
“People are basing life decisions on the data provided by these devices,” Euan Ashley, a professor of cardiovascular medicine at Stanford and co-author of the study, said in a news release.
Let’s say, as a hypothetical, some users check their device at the end of a long day and discover to their delight they burned 1,000 calories when they actually only burned 730. They might have an extra dessert or glass of wine since they think they’ve met their goal.
Over time, that adds up. In this scenario, that’s 1,890 extra calories each week the users don’t know about. Each pound of fat is composed of 3,500 calories.
“It’s just human nature,” Tim Church, professor of preventative medicine at Pennington Biomedical Research Center at Louisiana State University who wasn’t involved in the study, told NPR. “People are checking these inaccurate counts and they think they’ve earned a muffin or earned some ice cream and they’re sabotaging their weight-loss program.”
Of course, some margin of error when using a device like this is inevitable, but the scientists said it should be far lower.
“For a lay user, in a non-medical setting, we want to keep that error under 10 percent,” Anna Shcherbina, a Stanford graduate student and study co-author, said in a news release.
One of the key issues, Shcherbina hypothesized, was the difference in users’ body compositions.
“It’s very hard to train an algorithm that would be accurate across a wide variety of people because energy expenditure is variable based on someone’s fitness level, height and weight, etc.,” Shcherbina said.
The study participants included a “diversity of ages, male and female, and then also we looked at diversity of skin tone, and then size and weight to try and represent the population generally,” Ashley told the Guardian.
The devices proved most accurate for white women who were already fit, meaning “for those for whom it might matter the most, who are trying to lose weight, the error was actually greater,” Ashley told NPR, speculating that perhaps the companies only test the devices on a narrow group of people.
While the energy expenditure numbers were woefully off, Shcherbina pointed out that it’s much easier to assess heart rate, which can be measured directly and not through proxy calculations.
Indeed, Ashley said, “The heart rate measurements performed far better than expected.” Most were off by only about 5 percent.
There have long been hints that these devices aren’t useful for weight loss. A multiyear study published last September in JAMA split into two groups almost 500 people hoping to lose weight. One used fitness trackers, while the other did not.
Those with the trackers lost about 50 percent less weight than those without.
At the time, the study’s lead author, John Jakicic, a researcher of health and physical activity at the University of
Pittsburgh, thought it had to do with people incorrectly interpreting the fitness trackers.
“These technologies are focused on physical activity, like taking steps and getting your heart rate up,” Jakicic told NPR.
“People would say, ‘Oh, I exercised a lot today, now I can eat more.’ And they might eat more than they otherwise would have.”
The Stanford study, though, suggests that perhaps the participants were merely working with faulty data.
Ref: https://www.washingtonpost.com/news/morning-mix/wp/2017/05/25/fitness-trackers-are-largely-inaccurate-when-counting-calories-stanford-researchers-say/?tid=a_inl&utm_term=.7ee796173b4d
I'll go first:
I don't eat back calories burned at all and this will not affect how I use my FitBit Alta HR. I use mine to track resting heart rate and sleep. My device says that I have walked 300+ steps / burned X number of calories while I am sleeping based upon my motion in bed, and if I get up during the night.
_____________________________________________________________________________________________
Calorie counting is a useful way to lose weight, but a new study suggests a fitness tracker could sabotage your efforts.
The devices are overwhelmingly popular. For instance, since its inception, the leading brand, Fitbit, has sold at least 30 million of them. The company promises on its website that the devices “track steps, distance, calories burned, floors climbed, active minutes & hourly activity.” Others, such as PulseOn, Apple Watch, Basis Peak, Samsung Gear S2 and Microsoft Band, promise the same.
A team of Stanford researchers, however, recently called foul after testing these trackers. The scientists said in a paper published Wednesday in the Journal of Personalized Medicine that though the devices purport to help users track their calories — daily energy expenditure — the number is often markedly incorrect.
The least accurate, PulseOn, was off by an average of 93 percent. The most accurate device, Fitbit Surge, was off by an average of 27 percent, the Guardian reported.
In a statement to NPR, PulseOn said the extremely high level of inaccuracy may “suggest that the authors may not have properly set all the user parameters on the device.”
The consequences of such large margins of error could, of course, be significant.
“People are basing life decisions on the data provided by these devices,” Euan Ashley, a professor of cardiovascular medicine at Stanford and co-author of the study, said in a news release.
Let’s say, as a hypothetical, some users check their device at the end of a long day and discover to their delight they burned 1,000 calories when they actually only burned 730. They might have an extra dessert or glass of wine since they think they’ve met their goal.
Over time, that adds up. In this scenario, that’s 1,890 extra calories each week the users don’t know about. Each pound of fat is composed of 3,500 calories.
“It’s just human nature,” Tim Church, professor of preventative medicine at Pennington Biomedical Research Center at Louisiana State University who wasn’t involved in the study, told NPR. “People are checking these inaccurate counts and they think they’ve earned a muffin or earned some ice cream and they’re sabotaging their weight-loss program.”
Of course, some margin of error when using a device like this is inevitable, but the scientists said it should be far lower.
“For a lay user, in a non-medical setting, we want to keep that error under 10 percent,” Anna Shcherbina, a Stanford graduate student and study co-author, said in a news release.
One of the key issues, Shcherbina hypothesized, was the difference in users’ body compositions.
“It’s very hard to train an algorithm that would be accurate across a wide variety of people because energy expenditure is variable based on someone’s fitness level, height and weight, etc.,” Shcherbina said.
The study participants included a “diversity of ages, male and female, and then also we looked at diversity of skin tone, and then size and weight to try and represent the population generally,” Ashley told the Guardian.
The devices proved most accurate for white women who were already fit, meaning “for those for whom it might matter the most, who are trying to lose weight, the error was actually greater,” Ashley told NPR, speculating that perhaps the companies only test the devices on a narrow group of people.
While the energy expenditure numbers were woefully off, Shcherbina pointed out that it’s much easier to assess heart rate, which can be measured directly and not through proxy calculations.
Indeed, Ashley said, “The heart rate measurements performed far better than expected.” Most were off by only about 5 percent.
There have long been hints that these devices aren’t useful for weight loss. A multiyear study published last September in JAMA split into two groups almost 500 people hoping to lose weight. One used fitness trackers, while the other did not.
Those with the trackers lost about 50 percent less weight than those without.
At the time, the study’s lead author, John Jakicic, a researcher of health and physical activity at the University of
Pittsburgh, thought it had to do with people incorrectly interpreting the fitness trackers.
“These technologies are focused on physical activity, like taking steps and getting your heart rate up,” Jakicic told NPR.
“People would say, ‘Oh, I exercised a lot today, now I can eat more.’ And they might eat more than they otherwise would have.”
The Stanford study, though, suggests that perhaps the participants were merely working with faulty data.
Ref: https://www.washingtonpost.com/news/morning-mix/wp/2017/05/25/fitness-trackers-are-largely-inaccurate-when-counting-calories-stanford-researchers-say/?tid=a_inl&utm_term=.7ee796173b4d
1
Replies
-
That's a LOT of reading!
I will tell you my findings on having a tracker - had a Fitbit Flex and a One. No longer use them because they eventually broke and I was fed up feeling ruled by the amount of steps I needed to do in the day.
My tracker was very accurate at predicting my total calorie burn, I knew if I ate 10-20% less than the end of day figure Fitbit gave me I was on track to lose and I did.
I didn't like that I gave up doing other cardio workouts I loved because they didn't rack up steps for me. Since I have gone without Fitbit I'm still active but I do workouts I enjoy and am no longer ruled by getting X steps per day.
I have a lot of Fitbit friends and most of us find it an accurate gadget. However if we eat more than we burn then we wont lose. We can't blame the gadget only ourselves for that.4 -
My view of trackers are that they are primarily to make desk jockeys feel like they are more active than they are. They might even encourage a few people to take the stairs or to walk to the restroom on the other side of the building instead of the one nearest their desk. If it does that, accuracy isn't really that important. But if you are trying to use one to balance CICO, I think you are fooling yourself.3
-
Interesting that the story says that "Those with the trackers lost about 50 percent less weight than those without." Also shocking to see this: "The least accurate, PulseOn, was off by an average of 93 percent. The most accurate device, Fitbit Surge, was off by an average of 27 percent, the Guardian reported."
I don't use a tracker, not because I don't value them, but because I'm a cheapskate. I've recently purchased a GPS cycling computer to track my routes, times, and miles, which automatically syncs to Strava.
However, as a scientist, it makes sense -- the trackers are an indirect estimate of our activity. So, why not just measure our activity directly, instead of basing our activity measurements off of an estimate that could be +/- 27 to 93%.1 -
This research is valuable, since often the only information available comes from the marketing departments of companies that sell the devices.
This is a problem, because the average person is not trained in the arcane details of estimating energy expenditure (details which are also poorly understood by many fitness professionals). A number on a screen is much more convincing, even if the number is an absolute fabrication (see "HRM, strength training").
The original sin was committed by Polar 20+ years ago when they introduced the calorie-count feature. It was quickly accepted with little question, and it drove sales, so everyone since has felt compelled to follow suit.
Some of the remarks by the graduate assistant were amusingly naive. Quite frankly, a 25% error rate is about as good as you will ever get with a tracker device. And surprise, surprise, the companies "only test the devices on a small group of people". No duh. That type of testing is expensive and time-consuming--impossible to do when you have to come out with 2-3 new models a year or are a new startup. I doubt that most of the companies do any testing at all. I just assume they are copying their algorithms from other studies or other researchers.2 -
TimothyFish wrote: »My view of trackers are that they are primarily to make desk jockeys feel like they are more active than they are. They might even encourage a few people to take the stairs or to walk to the restroom on the other side of the building instead of the one nearest their desk. If it does that, accuracy isn't really that important. But if you are trying to use one to balance CICO, I think you are fooling yourself.
I think that most adults follow fairly habitual movement patterns. So, over time, it is possible one could observe patterns in the activity numbers that might be useful. But most aren't going to do that much analysis, so I agree taking the numbers at face value will likely hurt more than help.
1 -
I do not own a fitness tracker and have no intention of ever getting one in the future. To track my exercise I use the free version of Runtastic on my cellphone and I also accept that what it gives me for calories burned is not 100% accurate. I do not eat my exercise calories back.2
-
However, as a scientist, it makes sense -- the trackers are an indirect estimate of our activity. So, why not just measure our activity directly, instead of basing our activity measurements off of an estimate that could be +/- 27 to 93%.
Because the next question is how would you do that?2 -
Interesting that the story says that "Those with the trackers lost about 50 percent less weight than those without." Also shocking to see this: "The least accurate, PulseOn, was off by an average of 93 percent. The most accurate device, Fitbit Surge, was off by an average of 27 percent, the Guardian reported."
I don't use a tracker, not because I don't value them, but because I'm a cheapskate. I've recently purchased a GPS cycling computer to track my routes, times, and miles, which automatically syncs to Strava.
However, as a scientist, it makes sense -- the trackers are an indirect estimate of our activity. So, why not just measure our activity directly, instead of basing our activity measurements off of an estimate that could be +/- 27 to 93%.
You can -- you just have to live in one of these.
1 -
NorthCascades wrote: »However, as a scientist, it makes sense -- the trackers are an indirect estimate of our activity. So, why not just measure our activity directly, instead of basing our activity measurements off of an estimate that could be +/- 27 to 93%.
Because the next question is how would you do that?
Better yet, tell me how to do that so I can make some $$$$1 -
Its funny, I have a fitbit blaze but have never even looked at calorie burn. I do use it for heart rate, sleep. The steps are just for fun and i assome its a guess at best.0
-
They only work if you combine them with additional data. They are still only estimating based on averages, with a little math though they can be a really good tool.
I wear a Fitbit Blaze and eat at a 1000 calorie deficit based on its burn numbers, last week according to Trendweight, for me that was an actual of 638 calories. However, since then I figured out what food item I've been eating has inaccurate nutritional label information (a precooked frozen chicken breast). This week, my weight loss on the scale indicated I had an actual deficit of 759 calories. That's plenty for me. If I wanted a perfect 1000 calorie deficit, I could probably just eat 240 or so calories less than the Fitbit data says to achieve that. Who knows, the fitbit may not even be off by that whole 240 calories, I could have another food item that isn't accurate that I just haven't found yet.
I do work a desk job, when I go to one of those TDEE calculators, it says that my TDEE is only 2093 with my desk job. However, I work in a library, I get up to show people where books are all the time. So, I would have to eat 1093 calories to have a 1000 calorie deficit according to a TDEE calculator. However, it turns out that doing my desk job gets me like 10,000 steps a day. I burn more like 3000 calories per day, which would make my 1000 calorie a day diet WAY too extreme.
This difference between my actual burn and my estimated burn has just sabotaged me my whole life. I'm home today and I feel like I'm doing nothing, but I'm a fidgety person. I just looked and I have 5300 steps at 1 pm just puttering around the house.
Anyways, a lot of people use activity trackers incorrectly, which is a complete shame because with a little math they could be wonderful. I don't really see how that's the activity tracker's fault.. lol A lot of people try to make it count steps for activities that aren't step based too, which really screws it up. If you are putting your fitbit on your ankle to try to get it to count your recumbent bike steps you are going to have a higher calorie burn than your actual. I've seen all kinds of dumb stuff like that.
Mine has been a lifesaver. I can finally say, no, I'm an active person with a desk job. My weight loss indicates that I have a deficit of about 730 calories per day eating AT my estimated TDEE.
1 -
Some of the remarks by the graduate assistant were amusingly naive. Quite frankly, a 25% error rate is about as good as you will ever get with a tracker device. And surprise, surprise, the companies "only test the devices on a small group of people". No duh. That type of testing is expensive and time-consuming--impossible to do when you have to come out with 2-3 new models a year or are a new startup. I doubt that most of the companies do any testing at all. I just assume they are copying their algorithms from other studies or other researchers.
This has always bothered me a little bit. I realize what I'm about to say isn't entirely fair as a criticism but humor me.
When PowerTap says their meters have a maximum error of 1.5 %, I don't have to take their word for it. Plenty of universities have tested this independently and verified PT's claims. If I don't trust them, either, I can hang a known weight from the bike and verify for myself that the numbers are within spec.
I wouldn't expect that kind of accuracy from a Fitbit. To be honest, I have no idea how accurate a Fitbit is. I think Fitbit should tell their customers what to expect in terms of accuracy. In specific and not vague terms. As it is today, a fitness tracker can be wrong by 10,000 % and still within spec.
There are a lot of people on these and other forums swearing that Fitbits are very accurate for calories. Earlier today somebody claimed theirs was within 4 % for calories, but qualified that that's because he limits his exercise to thinks Fitbit understands. While it isn't Fitbit making these claims, it's confusing for people, and this is something that could and should be clarified.
Now I'll acknowledge that Fitbit puts out more new models in a year than PowerTap does in a decade. And costs less. Like I said, it's not completely fair as criticism. But I do think we should expect more from fitness trackers as consumers.0 -
NorthCascades wrote: »However, as a scientist, it makes sense -- the trackers are an indirect estimate of our activity. So, why not just measure our activity directly, instead of basing our activity measurements off of an estimate that could be +/- 27 to 93%.
Because the next question is how would you do that?
Measure your activity with things you can measure directly (as opposed to indirect estimates), such as the time it takes you to run a mile, how many push-ups you can do, how many squats in 1 minute, etc.
I'm not talking about measuring calories burned, that's an estimate too. I'm talking about using only metrics that you can measure directly and avoiding those that are indirectly measured, estimated, or approximated.0 -
It's pretty easy to understand that if you start using metrics that are directly measured that you'd have improved ability to track your progress in a weight loss or fitness program than using a device that that lacks precision in estimating your activity. That's what this article is about.0
-
@sporangia
I misunderstood your question, then. I thought you were asking for something more difficult. I'd still love for you to elaborate if you're up for it.
For example, when you say time to run a mile, I think a lot of fitness trackers will attempt that today based on counting your steps (which can be fooled eg by driving) and assuming your average stride length. Would you consider GPS an acceptable way of measuring? I assume the hard part is to know when you've gone a mile, not to know how much time has elapsed.
I think most of the fitness tracker market wants to know calories. Judging by how people here seem to use them, which maybe isn't a representative sample.0 -
I have a Fitbit Charge2 HR and a One...
Going into it I knew it wasn't going to be accurate for calorie burn or steps, but it gave me a baseline. They have helped me progress from very sedentary to very active. I am evolving. I started at a goal of the 10,000 steps and now average about 15,000 - 30,000 a day. The challenge to do a little bit more keeps me going outside everyday for my lunch time walk in rain snow and heat. I love walking and hiking. I am competitive so it works for me and I can just be competitive with myself.
I believe a fitbit is great for someone to build on and gets them (me) going in the right direction. True, some people will get bored and go back to old habits, which is the norm of diets and gym memberships too. There are some of us who will spark and won't look back. Would I recommend one to an avid marathon or triathlon runner, no. Will I use my fitbit forever? Probably not. I am now looking to move on to weights and yoga and other healthy lifestyles.7 -
NorthCascades wrote: »@sporangia
I misunderstood your question, then. I thought you were asking for something more difficult. I'd still love for you to elaborate if you're up for it.
For example, when you say time to run a mile, I think a lot of fitness trackers will attempt that today based on counting your steps (which can be fooled eg by driving) and assuming your average stride length. Would you consider GPS an acceptable way of measuring? I assume the hard part is to know when you've gone a mile, not to know how much time has elapsed.
I think most of the fitness tracker market wants to know calories. Judging by how people here seem to use them, which maybe isn't a representative sample.
I agree, the fitness tracker market wants to know that. It may not be a realistic expectation that our current technology can actually achieve that (and I didn't mean to suggest that measurements like running 1 mile can or should be converted to calories burned).
The article above states that people who do not use fitness trackers typically lose 50% more weight than those who use the trackers. Perhaps, as a society, we need to stop trying to track our every movement and calorie burnt and evaluate progress using other measurements, such as time for running 1 mile (using a track or some other place where the distance is known).0 -
Quite frankly I've had excellent results with my Garmin Vivosmart HR and Fenix 3 HR, neither of which were tested in this study. While I don't doubt the accuracy of many of those tested is much less than what is hoped by the majority of the population who "tracks steps" and hopes to magically lose weight simply by walking around. When a tracker with HR is put into the hands of someone who otherwise knows what to expect through research on TDEE, BMR and their own calorie intake/expenditure I bet the results are vastly superior.
User error + user expectations = garbage in and garbage out.1 -
NorthCascades wrote: »Some of the remarks by the graduate assistant were amusingly naive. Quite frankly, a 25% error rate is about as good as you will ever get with a tracker device. And surprise, surprise, the companies "only test the devices on a small group of people". No duh. That type of testing is expensive and time-consuming--impossible to do when you have to come out with 2-3 new models a year or are a new startup. I doubt that most of the companies do any testing at all. I just assume they are copying their algorithms from other studies or other researchers.
This has always bothered me a little bit. I realize what I'm about to say isn't entirely fair as a criticism but humor me.
When PowerTap says their meters have a maximum error of 1.5 %, I don't have to take their word for it. Plenty of universities have tested this independently and verified PT's claims. If I don't trust them, either, I can hang a known weight from the bike and verify for myself that the numbers are within spec.
I wouldn't expect that kind of accuracy from a Fitbit. To be honest, I have no idea how accurate a Fitbit is. I think Fitbit should tell their customers what to expect in terms of accuracy. In specific and not vague terms. As it is today, a fitness tracker can be wrong by 10,000 % and still within spec.
There are a lot of people on these and other forums swearing that Fitbits are very accurate for calories. Earlier today somebody claimed theirs was within 4 % for calories, but qualified that that's because he limits his exercise to thinks Fitbit understands. While it isn't Fitbit making these claims, it's confusing for people, and this is something that could and should be clarified.
Now I'll acknowledge that Fitbit puts out more new models in a year than PowerTap does in a decade. And costs less. Like I said, it's not completely fair as criticism. But I do think we should expect more from fitness trackers as consumers.
The problem is that the only machines that can actually tell you what you are burning exactly are machines that do expired gas analysis. I can't really picture people wanting to walk around with a mask on their face. Even those aren't perfect, some hospitals are using cheaper models now instead of medical grade, like medgem and New Leaf.
I was surprised how close the heart rates are, that was where I thought the error would be. Fitbit's devices vary in how many steps they count. I can scrub my shower for 10 minutes and not log any steps. Heck, if anything mine tends to undercount. It doesn't count steps when I'm pushing a shopping cart or stroller or holding something in my hand. If I walk holding my cup of coffee in my nondominant hand, I don't get steps.
I also kind of wonder if they didn't bother to setup the devices correctly, whether dominant or nondominant wrist, because it does make a difference. If you wanted the test to fail you could set it up incorrectly. They don't say anywhere in the study what settings they used for any of the devices.0 -
Those are also averages - meaning trackers work worse for some people, but may work better for others. The real message here is you need to find what works best for you and not just assume a tracker is it. It works for me, however.1
Categories
- All Categories
- 1.4M Health, Wellness and Goals
- 392K Introduce Yourself
- 43.6K Getting Started
- 259.8K Health and Weight Loss
- 175.7K Food and Nutrition
- 47.4K Recipes
- 232.3K Fitness and Exercise
- 402 Sleep, Mindfulness and Overall Wellness
- 6.4K Goal: Maintaining Weight
- 8.5K Goal: Gaining Weight and Body Building
- 152.8K Motivation and Support
- 7.9K Challenges
- 1.3K Debate Club
- 96.3K Chit-Chat
- 2.5K Fun and Games
- 3.4K MyFitnessPal Information
- 23 News and Announcements
- 998 Feature Suggestions and Ideas
- 2.4K MyFitnessPal Tech Support Questions