Welcome to Debate Club! Please be aware that this is a space for respectful debate, and that your ideas will be challenged here. Please remember to critique the argument, not the author.
Peer reviewed studies are they the end all be all?
Options
Grimmerick
Posts: 3,344 Member
in Debate Club
I was reading an article and I thought with the amount of time MFP talks about peer reviewed studies (and we do seem to put a lot of stock in them) maybe we should really talk about peer reviewed studies, what makes them good, what makes them bad, how do you spot honest thorough studies and tell the difference between poorly set up or designed studies or even fraudulent ones Is there any good way to tell if you can truly trust even a peer reviewed study? Thought this might make a good debate.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3894865/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3894865/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/
4
Replies
-
I'm kinda surprised here. I thought with all the science minded people in here always wanting studies and peer reviewed ones especially, that this would be a really good topic?6
-
The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".10
-
Debate threads don't always take off right away. And many of the folks who have science backgrounds don't post constantly throughout the day, so you may have to give them some time to find it.
I learned (and continue to learn) about this sort of stuff from this community, so I don't have much to add to the discussion. Except to say I'd think that even if "peer-reviewed" isn't quite what some make it out to be, it sure seems better than the alternatives. And most people here consider "peer-reviewed" a starting point, not the be all and end all. It's like the bare minimum to determine if something is even worth the time to look at. You still actually have to dig into the study itself then.14 -
-
If one group of researchers came to a certain conclusion, another group or person will research and do a study to confirm it and if they come to the same conclusion, then you can say it was peer review, what is "it"? it's not that the researchers were peer reviewed but the topic at hand, whatever hypothesis it was they they were seeking to find the truth about, if enough researchers come to the same conclusion over and over, the hypothesis becomes a theory which is a nugget of truth which will probably be added to your kids science book as one sentence on page 51 which he probably won't remember.6
-
tirowow12385 wrote: »If one group of researchers came to a certain conclusion, another group or person will research and do a study to confirm it and if they come to the same conclusion, then you can say it was peer review, what is "it"? it's not that the researchers were peer reviewed but the topic at hand, whatever hypothesis it was they they were seeking to find the truth about, if enough researchers come to the same conclusion over and over, the hypothesis becomes a theory which is a nugget of truth which will probably be added to your kids science book as one sentence on page 51 which he probably won't remember.
Peer review doesn't refer to other scientists replicating the study (although having results that others can replicate is an important part of science). It's the process by which your peers evaluate your work for potential errors or issues before it is published.13 -
Debate threads don't always take off right away. And many of the folks who have science backgrounds don't post constantly throughout the day, so you may have to give them some time to find it.
I learned (and continue to learn) about this sort of stuff from this community, so I don't have much to add to the discussion. Except to say I'd think that even if "peer-reviewed" isn't quite what some make it out to be, it sure seems better than the alternatives. And most people here consider "peer-reviewed" a starting point, not the be all and end all. It's like the bare minimum to determine if something is even worth the time to look at. You still actually have to dig into the study itself then.
Very good point, I don't think it's always considered or treated as a "starting point" or the "bare minimum" on MFP though, you always see people demanding peer reviewed studies like that's the end of it. At least that's been what I have seen in some cases. Which made me wonder, how truthful and accurate can these studies be and how much stock we should actually put in them, without considering other things like how to tell a well designed one from a badly designed one and if there is anyone who stands to benefit from a certain outcome.2 -
I don't think anyone in the sciences would disagree with the problems posed in the two articles you posted. There are huge systemic concerns with peer review methodology. But that's why science is science--it's an ongoing debate that constantly has tens of thousands of experts kicking every tire that can possibly be kicked. It's basically a huge intellectual brawl, not some black and white cookie cutter process, and that is what makes it so valuable.
I certainly don't think that a given peer reviewed study is the be all and end all. I think a critical mass of peer reviewed studies with strong, reproducible methodology pointing in the same direction is pretty dang compelling, however. But you have to serve in the trenches for years and really know what you are doing to bring discernment to both the peer review process and the result.
It's not just peer review, but also replication that is a huge problem. Within social psychology, news recently came out that 75% (or some high number) of a certain segment of studies from some highly respected psych journals were not found to be reproducible. Social psychology in particular has been implicated in the shenanigans of "questionable research practices." There is a lot of gray area...and a lot of room to exploit it, or to, as they say, interpret the data. All sciences struggle with replication issues, however. Unfortunately in many fields there is a lot of reward for a new study, but little reward for putting in the time, expense and effort of direct replication of a previous study, so the quality control in that discipline suffers. There needs to be a big brawl and some butts kicked over the fact that not enough attention has been paid to replication in many disciplines, because a field as a whole loses credibility if they can't even reproduce their research, which is the freakin' BACKBONE of the scientific method.
Personally, I take every peer reviewed study with a grain of salt, but there are some I find to be much more persuasive than others based on the quality of their methods, who cites, them, and the caliber of those doing the citing (plus what they are saying about the original study).
But in almost every case, I have found that the best approach is to dig into the research, rather than relying on a newspaper article or a blog. There's plenty of charlatans (or just idiots) willing to trot out a peer reviewed study to dazzle the plebes, but many times the study doesn't even say what the charlatan/idiot is claiming. I have seen that many times, both in the general news, and here on MFP. The difference is, on MFP they are held accountable by people (in many cases, scientists) willing to bring, ahem, insightful critical analysis on what they are trying to promote.
Full disclosure, I am not a scientist, but have a strong background in it, and used to teach scientific research & communication methodology to majors in the sciences and social sciences. We spent a lot of time discussing critical analysis of sources, as you can imagine.14 -
tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
0 -
French_Peasant wrote: »I don't think anyone in the sciences would disagree with the problems posed in the two articles you posted. There are huge systemic concerns with peer review methodology. But that's why science is science--it's an ongoing debate that constantly has tens of thousands of experts kicking every tire that can possibly be kicked. It's basically a huge intellectual brawl, not some black and white cookie cutter process, and that is what makes it so valuable.
I certainly don't think that a given peer reviewed study is the be all and end all. I think a critical mass of peer reviewed studies with strong, reproducible methodology pointing in the same direction is pretty dang compelling, however. But you have to serve in the trenches for years and really know what you are doing to bring discernment to both the peer review process and the result.
It's not just peer review, but also replication that is a huge problem. Within social psychology, news recently came out that 75% (or some high number) of a certain segment of studies from some highly respected psych journals were not found to be reproducible. Social psychology in particular has been implicated in the shenanigans of "questionable research practices." There is a lot of gray area...and a lot of room to exploit it, or to, as they say, interpret the data. All sciences struggle with replication issues, however. Unfortunately in many fields there is a lot of reward for a new study, but little reward for putting in the time, expense and effort of direct replication of a previous study, so the quality control in that discipline suffers. There needs to be a big brawl and some butts kicked over the fact that not enough attention has been paid to replication in many disciplines, because a field as a whole loses credibility if they can't even reproduce their research, which is the freakin' BACKBONE of the scientific method.
Personally, I take every peer reviewed study with a grain of salt, but there are some I find to be much more persuasive than others based on the quality of their methods, who cites, them, and the caliber of those doing the citing (plus what they are saying about the original study).
But in almost every case, I have found that the best approach is to dig into the research, rather than relying on a newspaper article or a blog. There's plenty of charlatans (or just idiots) willing to trot out a peer reviewed study to dazzle the plebes, but many times the study doesn't even say what the charlatan/idiot is claiming. I have seen that many times, both in the general news, and here on MFP. The difference is, on MFP they are held accountable by people (in many cases, scientists) willing to bring, ahem, insightful critical analysis on what they are trying to promote.
Full disclosure, I am not a scientist, but have a strong background in it, and used to teach scientific research & communication methodology to majors in the sciences and social sciences. We spent a lot of time discussing critical analysis of sources, as you can imagine.
Wow Thank you for taking the time to write that!1 -
Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
I don't think you'll ever find a human process that always results in accurate and truthful information. You'll still see issues with deliberate deception (which a peer review may not catch), carelessness on the part of the reviewers, mistaken assumptions/understandings of the science in question, etc.
Peer review is just a process. It's an important one, but it's not perfect.8 -
Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
According to the more insightful poster above, there is, at least in the social psych field. I'm no scientist nor do I have a background in it, it will be better to direct your questions eslewhere.0 -
tirowow12385 wrote: »Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
According to the more insightful poster above, there is, at least in the social psych field. I'm no scientist nor do I have a background in it, it will be better to direct your questions eslewhere.
Thanks anyway0 -
janejellyroll wrote: »Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
I don't think you'll ever find a human process that always results in accurate and truthful information. You'll still see issues with deliberate deception (which a peer review may not catch), carelessness on the part of the reviewers, mistaken assumptions/understandings of the science in question, etc.
Peer review is just a process. It's an important one, but it's not perfect.
It was the Chiropractic care post that got me thinking about it. Then dry needling and acupuncture came into it and that really had me thinking, those are already difficult subjects to get a accurate study on since there are so many factors involved and so many people that have a stake in the outcome of the studies. Thanks for your perspective1 -
Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
Money quote from the second article you posted:
CONCLUSION
So peer review is a flawed process, full of easily identified defects with little evidence that it works. Nevertheless, it is likely to remain central to science and journals because there is no obvious alternative, and scientists and editors have a continuing belief in peer review. How odd that science should be rooted in belief.
Money quote from the 2nd citation in that article (which goes to an abstract of a systematic review of peer review):
CONCLUSIONS:
Editorial peer review, although widely used, is largely untested and its effects are uncertain.
I would expect more highly credible journals (Nature, Cell, Science, etc.) would have more rigorous peer review processes.1 -
Grimmerick wrote: »I'm kinda surprised here. I thought with all the science minded people in here always wanting studies and peer reviewed ones especially, that this would be a really good topic?
You might want to give it more than 46 minutes before expecting to be flooded with responses.
Peer-reviewed studies can have varying levels of value, according to the methodology used and the interpretation of the results. Often you see people quoting/linking peer-reviewed studies because they cherry-picked one sentence out of the study which they think proves their point, when actually the study, when taken in context, does the exact opposite.
Peer review at least adds a layer of accountability, but they're by no means infallible.8 -
Grimmerick wrote: »Debate threads don't always take off right away. And many of the folks who have science backgrounds don't post constantly throughout the day, so you may have to give them some time to find it.
I learned (and continue to learn) about this sort of stuff from this community, so I don't have much to add to the discussion. Except to say I'd think that even if "peer-reviewed" isn't quite what some make it out to be, it sure seems better than the alternatives. And most people here consider "peer-reviewed" a starting point, not the be all and end all. It's like the bare minimum to determine if something is even worth the time to look at. You still actually have to dig into the study itself then.
Very good point, I don't think it's always considered or treated as a "starting point" or the "bare minimum" on MFP though, you always see people demanding peer reviewed studies like that's the end of it. At least that's been what I have seen in some cases. Which made me wonder, how truthful and accurate can these studies be and how much stock we should actually put in them, without considering other things like how to tell a well designed one from a badly designed one and if there is anyone who stands to benefit from a certain outcome.
But they don't insist on peer reviewed studies because that's the end of it. They ask for peer reviewed studies because it's where you start, anything less than that could just as easily be propaganda or marketing or just some schmoe's blog. What usually happens is someone says - You can find tons of evidence on the internet that this is true. And when we ask for examples, we get blog posts and opinion pieces and websites that sell stuff. So then you ask for peer reviewed studies to actually try to see if there's any science.
I'd be curious where you've seen someone here say that a peer reviewed study is all you need to prove something.10 -
The part I struggle with is how many studies isolate variables in an attempt to study them, controlling the environment/circumstances to such a degree that I question how relevant those results are to a real-world scenario. Not that I doubt the validity of the results, but in a real-world scenario, there are (usually) FAR more factors at play. How does that all balance out? (rhetorical question, mostly).
Sorry if that sidetracks the conversation in a direction you didn't want to go.1 -
The part I struggle with is how many studies isolate variables in an attempt to study them, controlling the environment/circumstances to such a degree that I question how relevant those results are to a real-world scenario. Not that I doubt the validity of the results, but in a real-world scenario, there are (usually) FAR more factors at play. How does that all balance out? (rhetorical question, mostly).
Sorry if that sidetracks the conversation in a direction you didn't want to go.
Well, if you actually read the full papers of the studies, most of them do discuss the limitations of their models and possible confounding factors that they couldn't control for. Sometimes the results are only relevant to certain populations or certain bodily states (for example, some foods are only poisonous if you're underfat, young, and male). It's poor science journalism that leads the public to believe that just the one factor being studied is important because it's the only factor researched in the given study.5 -
Another point to look at for all studies, including peer- reviewed studies, is the sample size, and the demographics of the sample. I was reading an article on the MFP blog the other day that cited a study which found that lifting weights at the beginning of a workout is more beneficial than cardio first. However, when I looked at the abstract of the study, it was done on 10 males. Not only is that a very small sample size, it also may not be a correct generalization for other demographics. further, this study wasn't replicated (that I could find) by anyone else. Just because it's peer reviewed doesn't mean the results are applicable to me. I really think that authors should be critiquing studies when they cite them, so that readers are alerted to potential problems.0
Categories
- All Categories
- 1.4M Health, Wellness and Goals
- 391.3K Introduce Yourself
- 43.4K Getting Started
- 259.6K Health and Weight Loss
- 175.6K Food and Nutrition
- 47.3K Recipes
- 232.3K Fitness and Exercise
- 387 Sleep, Mindfulness and Overall Wellness
- 6.4K Goal: Maintaining Weight
- 8.5K Goal: Gaining Weight and Body Building
- 152.7K Motivation and Support
- 7.8K Challenges
- 1.3K Debate Club
- 96.2K Chit-Chat
- 2.5K Fun and Games
- 3.2K MyFitnessPal Information
- 22 News and Announcements
- 911 Feature Suggestions and Ideas
- 2.3K MyFitnessPal Tech Support Questions