Welcome to Debate Club! Please be aware that this is a space for respectful debate, and that your ideas will be challenged here. Please remember to critique the argument, not the author.
Peer reviewed studies are they the end all be all?
Grimmerick
Posts: 3,342 Member
I was reading an article and I thought with the amount of time MFP talks about peer reviewed studies (and we do seem to put a lot of stock in them) maybe we should really talk about peer reviewed studies, what makes them good, what makes them bad, how do you spot honest thorough studies and tell the difference between poorly set up or designed studies or even fraudulent ones Is there any good way to tell if you can truly trust even a peer reviewed study? Thought this might make a good debate.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3894865/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3894865/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/
4
Replies
-
I'm kinda surprised here. I thought with all the science minded people in here always wanting studies and peer reviewed ones especially, that this would be a really good topic?6
-
The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".10
-
Debate threads don't always take off right away. And many of the folks who have science backgrounds don't post constantly throughout the day, so you may have to give them some time to find it.
I learned (and continue to learn) about this sort of stuff from this community, so I don't have much to add to the discussion. Except to say I'd think that even if "peer-reviewed" isn't quite what some make it out to be, it sure seems better than the alternatives. And most people here consider "peer-reviewed" a starting point, not the be all and end all. It's like the bare minimum to determine if something is even worth the time to look at. You still actually have to dig into the study itself then.14 -
If one group of researchers came to a certain conclusion, another group or person will research and do a study to confirm it and if they come to the same conclusion, then you can say it was peer review, what is "it"? it's not that the researchers were peer reviewed but the topic at hand, whatever hypothesis it was they they were seeking to find the truth about, if enough researchers come to the same conclusion over and over, the hypothesis becomes a theory which is a nugget of truth which will probably be added to your kids science book as one sentence on page 51 which he probably won't remember.6
-
tirowow12385 wrote: »If one group of researchers came to a certain conclusion, another group or person will research and do a study to confirm it and if they come to the same conclusion, then you can say it was peer review, what is "it"? it's not that the researchers were peer reviewed but the topic at hand, whatever hypothesis it was they they were seeking to find the truth about, if enough researchers come to the same conclusion over and over, the hypothesis becomes a theory which is a nugget of truth which will probably be added to your kids science book as one sentence on page 51 which he probably won't remember.
Peer review doesn't refer to other scientists replicating the study (although having results that others can replicate is an important part of science). It's the process by which your peers evaluate your work for potential errors or issues before it is published.13 -
Debate threads don't always take off right away. And many of the folks who have science backgrounds don't post constantly throughout the day, so you may have to give them some time to find it.
I learned (and continue to learn) about this sort of stuff from this community, so I don't have much to add to the discussion. Except to say I'd think that even if "peer-reviewed" isn't quite what some make it out to be, it sure seems better than the alternatives. And most people here consider "peer-reviewed" a starting point, not the be all and end all. It's like the bare minimum to determine if something is even worth the time to look at. You still actually have to dig into the study itself then.
Very good point, I don't think it's always considered or treated as a "starting point" or the "bare minimum" on MFP though, you always see people demanding peer reviewed studies like that's the end of it. At least that's been what I have seen in some cases. Which made me wonder, how truthful and accurate can these studies be and how much stock we should actually put in them, without considering other things like how to tell a well designed one from a badly designed one and if there is anyone who stands to benefit from a certain outcome.2 -
I don't think anyone in the sciences would disagree with the problems posed in the two articles you posted. There are huge systemic concerns with peer review methodology. But that's why science is science--it's an ongoing debate that constantly has tens of thousands of experts kicking every tire that can possibly be kicked. It's basically a huge intellectual brawl, not some black and white cookie cutter process, and that is what makes it so valuable.
I certainly don't think that a given peer reviewed study is the be all and end all. I think a critical mass of peer reviewed studies with strong, reproducible methodology pointing in the same direction is pretty dang compelling, however. But you have to serve in the trenches for years and really know what you are doing to bring discernment to both the peer review process and the result.
It's not just peer review, but also replication that is a huge problem. Within social psychology, news recently came out that 75% (or some high number) of a certain segment of studies from some highly respected psych journals were not found to be reproducible. Social psychology in particular has been implicated in the shenanigans of "questionable research practices." There is a lot of gray area...and a lot of room to exploit it, or to, as they say, interpret the data. All sciences struggle with replication issues, however. Unfortunately in many fields there is a lot of reward for a new study, but little reward for putting in the time, expense and effort of direct replication of a previous study, so the quality control in that discipline suffers. There needs to be a big brawl and some butts kicked over the fact that not enough attention has been paid to replication in many disciplines, because a field as a whole loses credibility if they can't even reproduce their research, which is the freakin' BACKBONE of the scientific method.
Personally, I take every peer reviewed study with a grain of salt, but there are some I find to be much more persuasive than others based on the quality of their methods, who cites, them, and the caliber of those doing the citing (plus what they are saying about the original study).
But in almost every case, I have found that the best approach is to dig into the research, rather than relying on a newspaper article or a blog. There's plenty of charlatans (or just idiots) willing to trot out a peer reviewed study to dazzle the plebes, but many times the study doesn't even say what the charlatan/idiot is claiming. I have seen that many times, both in the general news, and here on MFP. The difference is, on MFP they are held accountable by people (in many cases, scientists) willing to bring, ahem, insightful critical analysis on what they are trying to promote.
Full disclosure, I am not a scientist, but have a strong background in it, and used to teach scientific research & communication methodology to majors in the sciences and social sciences. We spent a lot of time discussing critical analysis of sources, as you can imagine.14 -
tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
0 -
French_Peasant wrote: »I don't think anyone in the sciences would disagree with the problems posed in the two articles you posted. There are huge systemic concerns with peer review methodology. But that's why science is science--it's an ongoing debate that constantly has tens of thousands of experts kicking every tire that can possibly be kicked. It's basically a huge intellectual brawl, not some black and white cookie cutter process, and that is what makes it so valuable.
I certainly don't think that a given peer reviewed study is the be all and end all. I think a critical mass of peer reviewed studies with strong, reproducible methodology pointing in the same direction is pretty dang compelling, however. But you have to serve in the trenches for years and really know what you are doing to bring discernment to both the peer review process and the result.
It's not just peer review, but also replication that is a huge problem. Within social psychology, news recently came out that 75% (or some high number) of a certain segment of studies from some highly respected psych journals were not found to be reproducible. Social psychology in particular has been implicated in the shenanigans of "questionable research practices." There is a lot of gray area...and a lot of room to exploit it, or to, as they say, interpret the data. All sciences struggle with replication issues, however. Unfortunately in many fields there is a lot of reward for a new study, but little reward for putting in the time, expense and effort of direct replication of a previous study, so the quality control in that discipline suffers. There needs to be a big brawl and some butts kicked over the fact that not enough attention has been paid to replication in many disciplines, because a field as a whole loses credibility if they can't even reproduce their research, which is the freakin' BACKBONE of the scientific method.
Personally, I take every peer reviewed study with a grain of salt, but there are some I find to be much more persuasive than others based on the quality of their methods, who cites, them, and the caliber of those doing the citing (plus what they are saying about the original study).
But in almost every case, I have found that the best approach is to dig into the research, rather than relying on a newspaper article or a blog. There's plenty of charlatans (or just idiots) willing to trot out a peer reviewed study to dazzle the plebes, but many times the study doesn't even say what the charlatan/idiot is claiming. I have seen that many times, both in the general news, and here on MFP. The difference is, on MFP they are held accountable by people (in many cases, scientists) willing to bring, ahem, insightful critical analysis on what they are trying to promote.
Full disclosure, I am not a scientist, but have a strong background in it, and used to teach scientific research & communication methodology to majors in the sciences and social sciences. We spent a lot of time discussing critical analysis of sources, as you can imagine.
Wow Thank you for taking the time to write that!1 -
Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
I don't think you'll ever find a human process that always results in accurate and truthful information. You'll still see issues with deliberate deception (which a peer review may not catch), carelessness on the part of the reviewers, mistaken assumptions/understandings of the science in question, etc.
Peer review is just a process. It's an important one, but it's not perfect.8 -
Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
According to the more insightful poster above, there is, at least in the social psych field. I'm no scientist nor do I have a background in it, it will be better to direct your questions eslewhere.0 -
tirowow12385 wrote: »Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
According to the more insightful poster above, there is, at least in the social psych field. I'm no scientist nor do I have a background in it, it will be better to direct your questions eslewhere.
Thanks anyway0 -
janejellyroll wrote: »Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
I don't think you'll ever find a human process that always results in accurate and truthful information. You'll still see issues with deliberate deception (which a peer review may not catch), carelessness on the part of the reviewers, mistaken assumptions/understandings of the science in question, etc.
Peer review is just a process. It's an important one, but it's not perfect.
It was the Chiropractic care post that got me thinking about it. Then dry needling and acupuncture came into it and that really had me thinking, those are already difficult subjects to get a accurate study on since there are so many factors involved and so many people that have a stake in the outcome of the studies. Thanks for your perspective1 -
Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
Money quote from the second article you posted:
CONCLUSION
So peer review is a flawed process, full of easily identified defects with little evidence that it works. Nevertheless, it is likely to remain central to science and journals because there is no obvious alternative, and scientists and editors have a continuing belief in peer review. How odd that science should be rooted in belief.
Money quote from the 2nd citation in that article (which goes to an abstract of a systematic review of peer review):
CONCLUSIONS:
Editorial peer review, although widely used, is largely untested and its effects are uncertain.
I would expect more highly credible journals (Nature, Cell, Science, etc.) would have more rigorous peer review processes.1 -
Grimmerick wrote: »I'm kinda surprised here. I thought with all the science minded people in here always wanting studies and peer reviewed ones especially, that this would be a really good topic?
You might want to give it more than 46 minutes before expecting to be flooded with responses.
Peer-reviewed studies can have varying levels of value, according to the methodology used and the interpretation of the results. Often you see people quoting/linking peer-reviewed studies because they cherry-picked one sentence out of the study which they think proves their point, when actually the study, when taken in context, does the exact opposite.
Peer review at least adds a layer of accountability, but they're by no means infallible.8 -
Grimmerick wrote: »Debate threads don't always take off right away. And many of the folks who have science backgrounds don't post constantly throughout the day, so you may have to give them some time to find it.
I learned (and continue to learn) about this sort of stuff from this community, so I don't have much to add to the discussion. Except to say I'd think that even if "peer-reviewed" isn't quite what some make it out to be, it sure seems better than the alternatives. And most people here consider "peer-reviewed" a starting point, not the be all and end all. It's like the bare minimum to determine if something is even worth the time to look at. You still actually have to dig into the study itself then.
Very good point, I don't think it's always considered or treated as a "starting point" or the "bare minimum" on MFP though, you always see people demanding peer reviewed studies like that's the end of it. At least that's been what I have seen in some cases. Which made me wonder, how truthful and accurate can these studies be and how much stock we should actually put in them, without considering other things like how to tell a well designed one from a badly designed one and if there is anyone who stands to benefit from a certain outcome.
But they don't insist on peer reviewed studies because that's the end of it. They ask for peer reviewed studies because it's where you start, anything less than that could just as easily be propaganda or marketing or just some schmoe's blog. What usually happens is someone says - You can find tons of evidence on the internet that this is true. And when we ask for examples, we get blog posts and opinion pieces and websites that sell stuff. So then you ask for peer reviewed studies to actually try to see if there's any science.
I'd be curious where you've seen someone here say that a peer reviewed study is all you need to prove something.10 -
The part I struggle with is how many studies isolate variables in an attempt to study them, controlling the environment/circumstances to such a degree that I question how relevant those results are to a real-world scenario. Not that I doubt the validity of the results, but in a real-world scenario, there are (usually) FAR more factors at play. How does that all balance out? (rhetorical question, mostly).
Sorry if that sidetracks the conversation in a direction you didn't want to go.1 -
The part I struggle with is how many studies isolate variables in an attempt to study them, controlling the environment/circumstances to such a degree that I question how relevant those results are to a real-world scenario. Not that I doubt the validity of the results, but in a real-world scenario, there are (usually) FAR more factors at play. How does that all balance out? (rhetorical question, mostly).
Sorry if that sidetracks the conversation in a direction you didn't want to go.
Well, if you actually read the full papers of the studies, most of them do discuss the limitations of their models and possible confounding factors that they couldn't control for. Sometimes the results are only relevant to certain populations or certain bodily states (for example, some foods are only poisonous if you're underfat, young, and male). It's poor science journalism that leads the public to believe that just the one factor being studied is important because it's the only factor researched in the given study.5 -
Another point to look at for all studies, including peer- reviewed studies, is the sample size, and the demographics of the sample. I was reading an article on the MFP blog the other day that cited a study which found that lifting weights at the beginning of a workout is more beneficial than cardio first. However, when I looked at the abstract of the study, it was done on 10 males. Not only is that a very small sample size, it also may not be a correct generalization for other demographics. further, this study wasn't replicated (that I could find) by anyone else. Just because it's peer reviewed doesn't mean the results are applicable to me. I really think that authors should be critiquing studies when they cite them, so that readers are alerted to potential problems.0
-
The part I struggle with is how many studies isolate variables in an attempt to study them, controlling the environment/circumstances to such a degree that I question how relevant those results are to a real-world scenario. Not that I doubt the validity of the results, but in a real-world scenario, there are (usually) FAR more factors at play. How does that all balance out? (rhetorical question, mostly).
Sorry if that sidetracks the conversation in a direction you didn't want to go.
In addition to @Zodikosis reply, I think in a perfect world, those super-controlled studies would be followed up by other groups trying to replicate those results in different scenarios to get a better understanding. Whether or not that happens is a different story.
Ultimately, a preponderance of peer-reviewed studies, carried out by different parties, replicating results over a number of years is the actual goal I believe. One study is never enough, or even intended to be enough, to draw a solid conclusion.0 -
Be all and end all? No...but I'd rather start there than some rando's internet blog or some shaky article that cherry picks a whole bunch of information or some website that's selling a bunch of stuff.8
-
Putting my favorite read on this subject here:
http://callingbullshit.org/tools/tools_legit.html
I think this one is great for anyone to read regardless of science background. I am a scientist who has published before, and I think it does a good job of communicating how to pick out the quality of a study.8 -
In a general scientific sense, yes peer-reviewed is the end-all be-all. Peer-review does its best to make sure there are no glaring design flaws, that you haven't misinterpreted data or overlooked an important variable. The experiment may be extremely specific in its research but the research should be seen as solid.
The beauty of science is that it's a fluid field. If new research comes out that is at odds with the consensus, as long as the study seen to be legit, the consensus can change.1 -
Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
While peer review isn't perfect, it's probably the best resource we have. It means that the article has been checked out by others who are professionals in the field, and they thought it had value. It's kind of like looking at the star-rating of a product before you decide whether or not to buy it - except that peer review means that, supposedly, the people making the decision of whether or not to publish the article are people who can make an educated decision about it.
I equate it to, in fiction, the difference between traditional publishing and self-publishing. With traditional publishing, when a book is put out by like... Random House, or some other big-name publishing house, you know that there are a whole bunch of people who have looked the manuscript over and decided it was worth producing. However, with a self-pubbed book, there's no such filter. Are there some self-pub gems? Absolutely. Are there some trad-pubbed piles of poo? Definitely! It's not the ONLY criterion that should be considered. But it's an important one.
What peer review tells you is that a bunch of other people who have established careers in that field are willing to put THEIR reputations behind that article. And to me, that gives it a significant amount of credibility.
Edited to add this, because reasons:
12 -
clicketykeys wrote: »Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
While peer review isn't perfect, it's probably the best resource we have. It means that the article has been checked out by others who are professionals in the field, and they thought it had value. It's kind of like looking at the star-rating of a product before you decide whether or not to buy it - except that peer review means that, supposedly, the people making the decision of whether or not to publish the article are people who can make an educated decision about it.
I equate it to, in fiction, the difference between traditional publishing and self-publishing. With traditional publishing, when a book is put out by like... Random House, or some other big-name publishing house, you know that there are a whole bunch of people who have looked the manuscript over and decided it was worth producing. However, with a self-pubbed book, there's no such filter. Are there some self-pub gems? Absolutely. Are there some trad-pubbed piles of poo? Definitely! It's not the ONLY criterion that should be considered. But it's an important one.
What peer review tells you is that a bunch of other people who have established careers in that field are willing to put THEIR reputations behind that article. And to me, that gives it a significant amount of credibility.
Edited to add this, because reasons:
that meme is amazing!0 -
This content has been removed.
-
‘Peer reviewed’ doesn’t necessarily mean robust evidence.
Most research is evaluated also on the type of evidence. The study design or scope, comparisons to other similar studies etc.
Different research types can be evaluated broadly by referring to the level of evidence illustrated in this image (or similar pyramids/ranking tools used in universities around the world).
3 -
This content has been removed.
-
Grimmerick wrote: »janejellyroll wrote: »Grimmerick wrote: »tirowow12385 wrote: »The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".
So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.
I don't think you'll ever find a human process that always results in accurate and truthful information. You'll still see issues with deliberate deception (which a peer review may not catch), carelessness on the part of the reviewers, mistaken assumptions/understandings of the science in question, etc.
Peer review is just a process. It's an important one, but it's not perfect.
It was the Chiropractic care post that got me thinking about it. Then dry needling and acupuncture came into it and that really had me thinking, those are already difficult subjects to get a accurate study on since there are so many factors involved and so many people that have a stake in the outcome of the studies. Thanks for your perspective
Whilst peer review is a valuable tool, for reasons articulated above, it's just one tool. The value can vary by discipline.
My original training was in control systems engineering. Essentially how to measure a thing, frequently using the measurement of other things as a means to do so, then making decisions based on that and acting on it. That rather diminishes five years of training, but the end result for me was working in spacecraft operations, and weapon systems. Peer review in that space is pretty reliable, with the ability to rigourously assess the validity of assumptions, sample sizes, validity of proxies etc.
I'm now involved in business change, strategic communications and influence. To be perfectly honest I wouldn't put much value to peer review in this space. To many variables, regardless of how one segments a population. If I look at at something as well defined as my cohort in officer training college, there is no way that could be replicated. Any conclusions based on my cohort apply only to that cohort..
In that sense, in a health related environment as we are here, the value of peer review is more limited. We then get down to the point upthread about valuing the reviewer prepared to risk their reputation as much as the process itself. That is not to diminish the value of personal integrity, but you're now into a risk of groupthink, which is quite high in a reasonably risk averse discipline.3
This discussion has been closed.
Categories
- All Categories
- 1.4M Health, Wellness and Goals
- 393.6K Introduce Yourself
- 43.8K Getting Started
- 260.3K Health and Weight Loss
- 175.9K Food and Nutrition
- 47.5K Recipes
- 232.5K Fitness and Exercise
- 431 Sleep, Mindfulness and Overall Wellness
- 6.5K Goal: Maintaining Weight
- 8.6K Goal: Gaining Weight and Body Building
- 153K Motivation and Support
- 8K Challenges
- 1.3K Debate Club
- 96.3K Chit-Chat
- 2.5K Fun and Games
- 3.8K MyFitnessPal Information
- 24 News and Announcements
- 1.1K Feature Suggestions and Ideas
- 2.6K MyFitnessPal Tech Support Questions