Why are they reporting correlations as percentages? That's incredibly misleading. If one wants a more intuitive measure of concordance than the correlation, one could use the coefficient of determination, which is the correlation squared.
This number tells you the percentage of the variation that is accounted for by the tracker. For a deeper introduction to correlations, see e.g. [1].
Let's rewrite the summary with the correct figures (and reduce the ridiculous number of sigfigs):
> The analysis reveals that commonly worn fitness trackers are the most accurate for measuring heart rate, providing a strong accuracy of 58 %. However, they are only moderately accurate for tracking step count and energy expenditure, 47 % and 32 %, respectively.
The highest level of accuracy is provided by the Apple Watch, which exhibits 74 % accuracy for heart rate and 50 % for energy expenditure. Garmin is the most accurate for tracking step count, as it offers 68 % accuracy.
> When it comes to energy expenditure, Apple is the only fitness tracker providing a strong accuracy of 71.02%. Jawbone, Fitbit, and Polar yield only moderate accuracy, ranging between 50.23% and 65.57%.
I don't buy it, no brand can be accurate on energy expenditure because it depends on a lot more things than what the watch can monitor.
Even in lab they have to put big mask on people while exercising to get something somewhat accurate.
On watch, calorie counter is a function of time and heart rate. Yet most watch brand can't even get heart rate right beside apple and Huawei.
Tried all the brand, went back to using a 10€ Casio and put a H10 monitor when doing sport.
>I don't buy it, no brand can be accurate on energy expenditure because it depends on a lot more things than what the watch can monitor
To be fair, the comment you excerpted noted that the best performer was still just 71% accurate. So they’re saying they’re far from perfect.
>On watch, calorie counter is a function of time and heart rate.
Well, we just don’t know. But let’s take Apple as an example. Apple knows your workout type. And they know your weight if you’ve entered it. And lots of other sensors as well (O2, if possible, temp, accelerometers, etc).
They’ve put a crazy amount of money into their testing lab where they do have extensive metabolic rate testers, VO2 Max, etc.
Will they get as good as that equipment for a dinky watch? No. But I also am glad I don’t have to strap a face mask on during my daily bike rides.
Can they, with insane volumes of data and hard research, make pretty good directional inferences using models that match their watch sensors to gold standard tests? It seems the answer to this (both logically and in the research data) is yes.
This video is a decade old so they’ve only become more advanced and have more data since then, but there are lots of models you can build to power these estimates: https://youtu.be/BceaTNT14Ao
I’m glad you’re happy with your Casio. Keep rocking what works for you. But I’m not sure what’s so hard to believe here when other researchers have actually produced the data on their accuracy.
At least in the case of Apple I have met a couple people who work on the fitness products and their accuracy. One of the people does real world testing so they are diving one day and coding the next. Great work if you can get it and your hobbies align.
The lab setup you describe is basically what they have to setup to correlate their proxies to energy. I’d expect the variability is going to come as a factor of body efficacy.
I used to be in much better shape (resting heart rate in the 50s) but I’m not much heavier and haven’t lost much of the muscle I gained (70ish percent of my former one rep max achievable). My day to day non-workout energy burn is measured higher now than it was when my heart rate was lower for all activity levels. I don’t think that it’s true I’m suddenly +15-20% energy consumption simply because I put on 10lbs but rather having a 10bpm increase in heart rate is probably what is being measured.
Short of that though it does a pretty good job. If I’m doing things that are strenuous but my arms are stationary it picks up the work. Very old wrist trackers needed to have an arm pumping or otherwise moving to do anything aside from a pre-calculated “you are alive” burn.
Given the level of effort for what we get, I’ll take it as a basic way to answer “did I actually get some exertion from watching my 2 year old today?”
> Given the level of effort for what we get, I’ll take it as a basic way to answer “did I actually get some exertion from watching my 2 year old today?”
That's what I initially started using it for but the best finally is just to do exercise everyday. Not everyone has time to run 2 hours or bike during 3 hours, but the bare minimum to stay healthy is 30 min of moderate to high intensity workout, which I think everyone could do.
If you stick to that, there isn't even any need to monitor yourself with a smartwatch.
> I don't buy it, no brand can be accurate on energy expenditure because it depends on a lot more things than what the watch can monitor.
You could say the same about virtually every modeling problem, yet the world runs on models that work adequately in practice. You measure what you can, infer what you can, and fill in the gaps with priors.
Not adding much to the conversation, but Google Pixel 3 is also quite accurate during running and cycling in terms of HR measurements, according to the already mentioned YouTuber 'Quantified Scientist'.
> no brand can be accurate on energy expenditure because it depends on a lot more things than what the watch can monitor.
I was going to quip that I can probably look at a person and state their energy expenditure with 70 % accuracy (whatever that means -- I'm going to assume coefficient of determination.)
Just someone's weight alone ought to have nearly a 70 % r² with energy expenditure. (A quick internet search suggests 71 % though I have not bothered verifying their methodology.[1])
Of course, they could mean 70 % across days for the same person, which is significantly more impressive as it takes out the strongest signals, but it's not at all clear what they actually mean and that's sort of my point.
Edit: I read the article closer now and when they say 70 % they actually mean a Pearson r of 0.7, i.e. a coefficient of determination of 50 %. Reporting r as a percentage seems confused at best, or possibly dishonest.
All reasonable fitness trackers use your weight as an input, and there's nothing in the article to suggest the devices weren't provided with this information even though it's not "something the watch can monitor".
On top of this inaccuracy in actual measurements, many fitness tracker brands invent their own metrics.
"How did you sleep?" I asked a friend who stayed over, - "I don't know" they replied as they raise their smartwatch to their face. "Pretty well, apparently! I got a sleep score of 84."
I've also heard of an acquaintance that they cancelled plans because their (Garmin™) Body Battery was too low.
Instinctively I flinch whenever people place such importance in what is essentially an arbitrary techno-horoscope. But I also find it hard to make an argument against it as long as it inspires healthy behaviour.
For a long time I was sort of baffled by the (bad) level-of-fitness and restedness metrics I was getting in a couple of popular apps.
At some point I decided to figure out what the deal was with it, and I saw that their calculations were based on resting heart rate, but because I only ever wear my Garmin running watch during actual runs, it was inferring my RHR from the lowest HR it picked up before or after my run, and reporting to every connected app that mine was like 120bpm (my real RHR is more like 50bpm). Everything was seemingly happy to just accept the garbage data and plug it into their formulas to produce garbage output. Really they probably should have just told me to see a doctor if they actually believed I had a 120bpm resting heart rate!
After that I turned off all of the automated metrics—it was a big realization about how easily messed up they can get (even if you wear your watch most of the day, you might have a fairly different RHR based on whether or not you sleep with it on). I can see this exact situation causing someone a lot of undue anxiety.
The garmin stuff kind of works as a realitive measure when most things are constant but completely break if you do something unusual like a long hike or a new cardio regiment
Well, sleep score still provides value to those with zero curiosity.
For everyone else, you can click into sleep score and see how much you actually slept vs think you slept which is useful data to anyone who wants to take sleep seriously.
I wear my Garmin 24/7 and generally trust it. I wouldn't cancel plans because of what it said but I can definitely tell when it's showing me data that reinforces how I feel (good or bad).
There is a YouTube channel named "The Quantified Scientist" which offers technical and more in-depth reviews of varied fitness trackers: https://www.youtube.com/@TheQuantifiedScientist. I would very much recommend it if you intend to buy a wearable and care about accuracy
That depends on what you're looking for. Apple Watch has a pretty good optical heart rate sensor but it's not great. Everyone knows that if you want accurate heart rate data you need to use a chest strap.
Other data point: he wears the Whoop as his daily wearable.
For the top quintile of gadgets, accuracy isn't what sets them apart but rather everything else (battery life, UI, data presentation, form factor, etc).
If your tracker shows you walked 12000 steps today as opposed to 8000 yesterday, even if you actually walked 10500 today and 8750 yesterday, that's still progress.
Yeah. I have noticed that the relative changes in numbers seem to be very correct. I can’t validate the absolute numbers, so I don’t really look at those so much. For the Apple Watch, I’ve noticed that the sleep tracker is quite accurate too, in absolute terms. That is, when I remember having long, wild dreams, I have a substantial amount of REM time. When I don’t remember my partner doing whatever at some hour, I was in deep sleep. If I toss and turn a lot, I see a lot of awake segments. So that part seems pretty accurate.
I recently started using one of these. The other day, I was working out on a rowing machine and received an alert (twice!) that my heart rate was extremely high. I was surprised because I wasn't even pushing myself that hard, and I am a fit person in general.
Upon research and asking around, I understood that it's more of a false positive. A more paranoid person may have reacted differently in such a scenario. So, consistency alone may not always suffice.
You need the band on your tracker to be very snug or this is likely to happen. You're actually seeing the tracker move, not the heart rate.
I've seen substantial mismatches between wrist and chest strap if my tracker had any ability to wiggle. With an infinitely adjustable metal band set so there's no slack but no pressure I see a good correspondence, although I haven't put it a test as hard as rowing (my activity of choice is hiking.) But my wife absolutely doesn't like such a band, wearing a soft, loose band and using her arms her "heart rate" has gone above 200.
I have also seen an environmental effect--these days I use the chest strap sensor because if my wrist is cold (conditions just below when I would add another layer) the wrist data can go totally bonkers.
In what fantasy past has cheap, personal activity tracking a) existed and b) been better?
Could it be better still? Sure, but I see no reason to assume that people are somehow lowering standards and not working on making it both cheaper and better all the time.
I use an Apple Watch, and while I don’t necessarily care about its accuracy in terms of steps walked or other specific details, I do value the insights it gives me into my daily activity. Even if it’s not entirely accurate, I get a reasonable estimation of how much I’ve moved, stood, and worked out. My watch motives me to me more active.
Trend lines are way more important for pretty much everyone. Anyone who cares about absolute numbers is either focusing on the wrong thing, or is doing vo2 max tests on the regular and knows the numbers on their watch are trends anyway
"The evaluations presented are based on general observations and should not be interpreted as definitive assessments of specific brands or their features."
The most recent paper used for this meta-study is still pretty old, and this technology changes. The study tells you almost nothing that could inform a buying decision regarding today's Apple Watch vs. Garmin Epix Pro (as examples). The sensors on the market today didn't exist and were not evaluated in any of the papers.
Readit News
This number tells you the percentage of the variation that is accounted for by the tracker. For a deeper introduction to correlations, see e.g. [1].
Let's rewrite the summary with the correct figures (and reduce the ridiculous number of sigfigs):
> The analysis reveals that commonly worn fitness trackers are the most accurate for measuring heart rate, providing a strong accuracy of 58 %. However, they are only moderately accurate for tracking step count and energy expenditure, 47 % and 32 %, respectively.
The highest level of accuracy is provided by the Apple Watch, which exhibits 74 % accuracy for heart rate and 50 % for energy expenditure. Garmin is the most accurate for tracking step count, as it offers 68 % accuracy.
----
[1]: https://entropicthoughts.com/the-surprising-richness-of-corr...
I don't buy it, no brand can be accurate on energy expenditure because it depends on a lot more things than what the watch can monitor.
Even in lab they have to put big mask on people while exercising to get something somewhat accurate.
On watch, calorie counter is a function of time and heart rate. Yet most watch brand can't even get heart rate right beside apple and Huawei.
Tried all the brand, went back to using a 10€ Casio and put a H10 monitor when doing sport.
To be fair, the comment you excerpted noted that the best performer was still just 71% accurate. So they’re saying they’re far from perfect.
>On watch, calorie counter is a function of time and heart rate.
Well, we just don’t know. But let’s take Apple as an example. Apple knows your workout type. And they know your weight if you’ve entered it. And lots of other sensors as well (O2, if possible, temp, accelerometers, etc).
They’ve put a crazy amount of money into their testing lab where they do have extensive metabolic rate testers, VO2 Max, etc.
Will they get as good as that equipment for a dinky watch? No. But I also am glad I don’t have to strap a face mask on during my daily bike rides.
Can they, with insane volumes of data and hard research, make pretty good directional inferences using models that match their watch sensors to gold standard tests? It seems the answer to this (both logically and in the research data) is yes.
This video is a decade old so they’ve only become more advanced and have more data since then, but there are lots of models you can build to power these estimates: https://youtu.be/BceaTNT14Ao
I’m glad you’re happy with your Casio. Keep rocking what works for you. But I’m not sure what’s so hard to believe here when other researchers have actually produced the data on their accuracy.
The lab setup you describe is basically what they have to setup to correlate their proxies to energy. I’d expect the variability is going to come as a factor of body efficacy.
I used to be in much better shape (resting heart rate in the 50s) but I’m not much heavier and haven’t lost much of the muscle I gained (70ish percent of my former one rep max achievable). My day to day non-workout energy burn is measured higher now than it was when my heart rate was lower for all activity levels. I don’t think that it’s true I’m suddenly +15-20% energy consumption simply because I put on 10lbs but rather having a 10bpm increase in heart rate is probably what is being measured.
Short of that though it does a pretty good job. If I’m doing things that are strenuous but my arms are stationary it picks up the work. Very old wrist trackers needed to have an arm pumping or otherwise moving to do anything aside from a pre-calculated “you are alive” burn.
Given the level of effort for what we get, I’ll take it as a basic way to answer “did I actually get some exertion from watching my 2 year old today?”
That's what I initially started using it for but the best finally is just to do exercise everyday. Not everyone has time to run 2 hours or bike during 3 hours, but the bare minimum to stay healthy is 30 min of moderate to high intensity workout, which I think everyone could do.
If you stick to that, there isn't even any need to monitor yourself with a smartwatch.
You could say the same about virtually every modeling problem, yet the world runs on models that work adequately in practice. You measure what you can, infer what you can, and fill in the gaps with priors.
I was going to quip that I can probably look at a person and state their energy expenditure with 70 % accuracy (whatever that means -- I'm going to assume coefficient of determination.)
Just someone's weight alone ought to have nearly a 70 % r² with energy expenditure. (A quick internet search suggests 71 % though I have not bothered verifying their methodology.[1])
Of course, they could mean 70 % across days for the same person, which is significantly more impressive as it takes out the strongest signals, but it's not at all clear what they actually mean and that's sort of my point.
Edit: I read the article closer now and when they say 70 % they actually mean a Pearson r of 0.7, i.e. a coefficient of determination of 50 %. Reporting r as a percentage seems confused at best, or possibly dishonest.
[1]: https://www.researchgate.net/figure/The-relation-between-bod...
"How did you sleep?" I asked a friend who stayed over, - "I don't know" they replied as they raise their smartwatch to their face. "Pretty well, apparently! I got a sleep score of 84."
I've also heard of an acquaintance that they cancelled plans because their (Garmin™) Body Battery was too low.
Instinctively I flinch whenever people place such importance in what is essentially an arbitrary techno-horoscope. But I also find it hard to make an argument against it as long as it inspires healthy behaviour.
At some point I decided to figure out what the deal was with it, and I saw that their calculations were based on resting heart rate, but because I only ever wear my Garmin running watch during actual runs, it was inferring my RHR from the lowest HR it picked up before or after my run, and reporting to every connected app that mine was like 120bpm (my real RHR is more like 50bpm). Everything was seemingly happy to just accept the garbage data and plug it into their formulas to produce garbage output. Really they probably should have just told me to see a doctor if they actually believed I had a 120bpm resting heart rate!
After that I turned off all of the automated metrics—it was a big realization about how easily messed up they can get (even if you wear your watch most of the day, you might have a fairly different RHR based on whether or not you sleep with it on). I can see this exact situation causing someone a lot of undue anxiety.
I wouldn't take that at face value. Sounds far more likely that they already wanted to flake and needed an excuse.
This sounds like someone who just feels like they need to provide tangible evidence for their reason being "I feel terrible"
For everyone else, you can click into sleep score and see how much you actually slept vs think you slept which is useful data to anyone who wants to take sleep seriously.
Health wearables give you some accountability.
For the top quintile of gadgets, accuracy isn't what sets them apart but rather everything else (battery life, UI, data presentation, form factor, etc).
If your tracker shows you walked 12000 steps today as opposed to 8000 yesterday, even if you actually walked 10500 today and 8750 yesterday, that's still progress.
Upon research and asking around, I understood that it's more of a false positive. A more paranoid person may have reacted differently in such a scenario. So, consistency alone may not always suffice.
I've seen substantial mismatches between wrist and chest strap if my tracker had any ability to wiggle. With an infinitely adjustable metal band set so there's no slack but no pressure I see a good correspondence, although I haven't put it a test as hard as rowing (my activity of choice is hiking.) But my wife absolutely doesn't like such a band, wearing a soft, loose band and using her arms her "heart rate" has gone above 200.
I have also seen an environmental effect--these days I use the chest strap sensor because if my wrist is cold (conditions just below when I would add another layer) the wrist data can go totally bonkers.
Could it be better still? Sure, but I see no reason to assume that people are somehow lowering standards and not working on making it both cheaper and better all the time.
I just don't care how accurate that the watch is (thus far anyway) because the perception of progress—delusional or no—is why I use it.
Generally, though, my perceived performance is reflected in the relative numbers. If I felt otherwise I too would be complaining about accuracy.
"The evaluations presented are based on general observations and should not be interpreted as definitive assessments of specific brands or their features."