Welcome to Debate Club! Please be aware that this is a space for respectful debate, and that your ideas will be challenged here. Please remember to critique the argument, not the author.

Peer reviewed studies are they the end all be all?

2

Replies

  • kimny72
    kimny72 Posts: 16,013 Member
    edited March 2018
    jjpptt2 wrote: »
    The part I struggle with is how many studies isolate variables in an attempt to study them, controlling the environment/circumstances to such a degree that I question how relevant those results are to a real-world scenario. Not that I doubt the validity of the results, but in a real-world scenario, there are (usually) FAR more factors at play. How does that all balance out? (rhetorical question, mostly).

    Sorry if that sidetracks the conversation in a direction you didn't want to go.

    In addition to @Zodikosis reply, I think in a perfect world, those super-controlled studies would be followed up by other groups trying to replicate those results in different scenarios to get a better understanding. Whether or not that happens is a different story.

    Ultimately, a preponderance of peer-reviewed studies, carried out by different parties, replicating results over a number of years is the actual goal I believe. One study is never enough, or even intended to be enough, to draw a solid conclusion.
  • makinemjellis
    makinemjellis Posts: 91 Member
    In a general scientific sense, yes peer-reviewed is the end-all be-all. Peer-review does its best to make sure there are no glaring design flaws, that you haven't misinterpreted data or overlooked an important variable. The experiment may be extremely specific in its research but the research should be seen as solid.

    The beauty of science is that it's a fluid field. If new research comes out that is at odds with the consensus, as long as the study seen to be legit, the consensus can change.
  • Hyacinth_Hippo
    Hyacinth_Hippo Posts: 51 Member
    clicketykeys wrote: »
    Grimmerick wrote: »
    tirowow12385 wrote: »
    The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".

    So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.

    While peer review isn't perfect, it's probably the best resource we have. It means that the article has been checked out by others who are professionals in the field, and they thought it had value. It's kind of like looking at the star-rating of a product before you decide whether or not to buy it - except that peer review means that, supposedly, the people making the decision of whether or not to publish the article are people who can make an educated decision about it.

    I equate it to, in fiction, the difference between traditional publishing and self-publishing. With traditional publishing, when a book is put out by like... Random House, or some other big-name publishing house, you know that there are a whole bunch of people who have looked the manuscript over and decided it was worth producing. However, with a self-pubbed book, there's no such filter. Are there some self-pub gems? Absolutely. Are there some trad-pubbed piles of poo? Definitely! It's not the ONLY criterion that should be considered. But it's an important one.

    What peer review tells you is that a bunch of other people who have established careers in that field are willing to put THEIR reputations behind that article. And to me, that gives it a significant amount of credibility.

    Edited to add this, because reasons:

    6a1rl5x3hvln.png


    that meme is amazing!
  • crackpotbaby
    crackpotbaby Posts: 1,297 Member
    ‘Peer reviewed’ doesn’t necessarily mean robust evidence.

    Most research is evaluated also on the type of evidence. The study design or scope, comparisons to other similar studies etc.

    Different research types can be evaluated broadly by referring to the level of evidence illustrated in this image (or similar pyramids/ranking tools used in universities around the world).

    beowiht4vlz9.jpeg
  • MeanderingMammal
    MeanderingMammal Posts: 7,866 Member
    Grimmerick wrote: »
    janejellyroll wrote: »
    Grimmerick wrote: »
    tirowow12385 wrote: »
    The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".

    So are there any flaws to peer reviews? Are they always completely accurate and truthful? This is where I have trouble.

    I don't think you'll ever find a human process that always results in accurate and truthful information. You'll still see issues with deliberate deception (which a peer review may not catch), carelessness on the part of the reviewers, mistaken assumptions/understandings of the science in question, etc.

    Peer review is just a process. It's an important one, but it's not perfect.

    It was the Chiropractic care post that got me thinking about it. Then dry needling and acupuncture came into it and that really had me thinking, those are already difficult subjects to get a accurate study on since there are so many factors involved and so many people that have a stake in the outcome of the studies. Thanks for your perspective

    Whilst peer review is a valuable tool, for reasons articulated above, it's just one tool. The value can vary by discipline.

    My original training was in control systems engineering. Essentially how to measure a thing, frequently using the measurement of other things as a means to do so, then making decisions based on that and acting on it. That rather diminishes five years of training, but the end result for me was working in spacecraft operations, and weapon systems. Peer review in that space is pretty reliable, with the ability to rigourously assess the validity of assumptions, sample sizes, validity of proxies etc.

    I'm now involved in business change, strategic communications and influence. To be perfectly honest I wouldn't put much value to peer review in this space. To many variables, regardless of how one segments a population. If I look at at something as well defined as my cohort in officer training college, there is no way that could be replicated. Any conclusions based on my cohort apply only to that cohort..

    In that sense, in a health related environment as we are here, the value of peer review is more limited. We then get down to the point upthread about valuing the reviewer prepared to risk their reputation as much as the process itself. That is not to diminish the value of personal integrity, but you're now into a risk of groupthink, which is quite high in a reasonably risk averse discipline.
  • dutchandkiwi
    dutchandkiwi Posts: 1,389 Member
    edited March 2018
    Peer review of course is flawed as it is most human processes. It is also always important to remember that peer review means evaluated on the basis of information given and a 'paper' process.
    If a researcher writes a paper and does not include all results then the peer review will nog have all information and that means that important parts can be missed. But a peer reviewer should look at what is given, balance the statistics, the sample (size and are they what was needed). The experimental set up and does it actually measure and prove what the paper say it does. By no means it is perfect. I too have found obvious errors, often editorial, in scientific papers when I was a student. Reviewers too are people with their own scientific bias.

    As far as I am aware the peer review does not include the underlying data set and raw data. Personally I think that is where the biggest flaw is, but it would also mean a lot more work for reviewers.But that is also the point where data/information may be lost due to not publishing of some.
    Leaving out of results may have very much an innocent reason; Total outliers in measurements, etc,

    Repeating of experiments/research is crucial and is not done often enough. There is simply more funding for new innovative research than there is for the more mundane confirmation studies and validation of studies. When I worked in the sciences, specifically measurement methodologies (longer ago than I want to admit) I concentrated on validation studies as I loved the puzzle that came with trying to repeat multiple times and determining performance of methods. But also how difficult it is to choose your statistical methodology and the interpretation that goes with statistics.

    Having said all that; peer review studies are better than none peer reviewed stuff. People just spouting a theory as the total truth without a shred of evidence (not in science, not in logic) Some of those things are simply dangerous garbage spread to desperate people and not seldom just for financial gain.
    Personally I go by: if the theory goes with a book and method that is expensive there is a significant change that it is woo.
    At least with a peer reviewed study I know knowledgable people have evaluated and balanced the research. It was judged but a jury of the researchers' peers.
  • jgnatca
    jgnatca Posts: 14,464 Member
    OP if you are looking for the absolute truth of a matter then you are going to be sorely disappointed. There are a few things that we reasonably know for sure. We all need to breathe or die in minutes. We all need enough water or we die in days. We all need to eat or fade away within weeks. We all sleep.

    I treat peer reviewed studies as the starting point of my trust. The farther away the claim is from those things I know are reasonably true, the less likely I am to trust it.

    If I don’t use a peer reviewed study as my starting point what am I left with? Tradition, rumour, anecdotes, Great Aunt Martha’s mustard plaster, and “miracle cures”.

    So what’s so wrong with tradition? It can seriously set you down the wrong path for a long time, making it harder to reach your goals.

    Let’s take the whole “purge the bowels” thing. These are all based on old cures before we had a full understanding of how the digestion system works. We now know that the liver and kidneys are responsible for filtering and removing harmful substances from our systems.

    https://www.mayoclinic.org/healthy-lifestyle/consumer-health/expert-answers/colon-cleansing/faq-20058435

    https://web.archive.org/web/20150424180208/http://www.cancer.org/Treatment/TreatmentsandSideEffects/ComplementaryandAlternativeMedicine/ManualHealingandPhysicalTouch/colon-therapy

    Why do colonics persist? In the absence of knowledge, colonic treatments are dramatic and visible. The patient is sure “something has been done” for them.
  • positivepowers
    positivepowers Posts: 902 Member
    janejellyroll wrote: »
    tirowow12385 wrote: »
    If one group of researchers came to a certain conclusion, another group or person will research and do a study to confirm it and if they come to the same conclusion, then you can say it was peer review, what is "it"? it's not that the researchers were peer reviewed but the topic at hand, whatever hypothesis it was they they were seeking to find the truth about, if enough researchers come to the same conclusion over and over, the hypothesis becomes a theory which is a nugget of truth which will probably be added to your kids science book as one sentence on page 51 which he probably won't remember.

    Peer review doesn't refer to other scientists replicating the study (although having results that others can replicate is an important part of science). It's the process by which your peers evaluate your work for potential errors or issues before it is published.

    This. I've had papers peer-reviewed. It just means that learned people in the same or relevant fields have checked the research, ran the numbers, and found no errors in logic, math, etc. They often make comments on grammar, format and things like that as well. It validates the research; it does not ensure the conclusions are correct.
  • Prime_Rib
    Prime_Rib Posts: 6 Member
    tirowow12385 wrote: »
    The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".

    I am grossly generalizing here, but it depends upon the peers. For example, the whole kerfuffle about fat being bad was the result of a study in the 1960s, peer reviewed, saying that fat is bad for you. Come to find out, this study was paid for by a sugar consortium and the result was meat got a bad rap. Now it is the opposite. Sugars and carbs are the enemy and fats are fine. Who the heck knows.

    It is important to know who these peers are and who is paying them. I would argue that pure science is rarely done anymore. Perhaps at CERN. Science is often at the behest of corporations looking to be seen in a better light.

    Climate science is the big debate now. Some say it is "settled science," as if such a thing possibly exists. (We learn every seven years that about 35% of our scientific assumptions are incorrect.) However, in academe, there is believed to be a consensus on the subject. So, if I write a paper and say that cows cause greenhouse gases and therefore we all must be vegetarian, it will be reviewed by like-minded people and it will be deemed correct and not challenged. And if you are a burgeoning scientist who could lose tenure if you have a contrary view, backed up with data, what are you going to? Be a pariah and not feed your family? Or are you going to toe the line?

    There are lies, damned lies, and statistics.

    So when it comes to dietary science, I'd first try and ascertain who is paying for the study. Peer review, academe in general, is a big circle jerk. University presses are the biggest rip-offs going. One line changed in a textbook so they can charge incoming students $350 a book.

    I am very distrustful of the current climate of peer review. So I tend to look at these things carefully and try to make up my own mind.

    I know I sound like a tin-foil hat guy. But, just to be clear, they are made of aluminum.
  • laur357
    laur357 Posts: 896 Member
    Librarian here. Peer-reviewed articles in respected scholarly journals are the go-to. Ideally you have a diverse editorial board of 'experts' with years of experience who review and comment on studies before publishing. They will hopefully find flaws in the study, logic errors, and poor communication styles before publication. However, the studies are being reviewed by humans, so there will always be flaws and biases AND no one is recreating the experiment to see if it's possible to replicate at the time of publication. Some journals are more well-respected than others, and there are ranking systems and impact factors that can point you to the better journals. They are generally good options for information.

    Systematic reviews are better for overarching evidence, and currently the ideal for evidence-based science. They consolidate information and look at many peer-reviewed studies, then draw conclusions from evidence coming from many sources. Medical guidelines are often based on systematic reviews, and having many articles and datasets to examine will hopefully weed out or give less weight to poorly designed studies or inaccurate results. Systematic reviews are not always available because years of work and publications are needed before information can be synthesized.

    Major problem in popular science reporting - journalists are often not scientists, and can't spot crap research. There's also some things you might not know if you don't read the article: extrapolation to humans from animal subjects, broad conclusions drawn and applied to humans, studies touting super-cool weight loss stuff that was tested on 9 people (4 of whom dropped out halfway through the study), experts generalizing the results to support their own personal research theories, and hidden conflict of interest statements (the report didn't tell you that the scientist who did a study on diet soda consumption is on the public health committee for the Pepsi corporation).
  • TR0berts
    TR0berts Posts: 7,739 Member
    jjpptt2 wrote: »
    The part I struggle with is how many studies isolate variables in an attempt to study them, controlling the environment/circumstances to such a degree that I question how relevant those results are to a real-world scenario. Not that I doubt the validity of the results, but in a real-world scenario, there are (usually) FAR more factors at play. How does that all balance out? (rhetorical question, mostly).

    Sorry if that sidetracks the conversation in a direction you didn't want to go.


    It's not that they're necessarily trying to replicate real-world scenarios. Rather, it's an attempt to figure out what really does what. Sometimes, it's an attempt to figure out what doesn't do what. Then further tests - maybe actual studies, but not necessarily - attempt to put previous results together to see what works "best" for a particular goal.

    Otherwise, you get silly things like what was once on the MFP blog regarding body recomposition. It had something like 7 things that were supposedly required to recomp. No, only 3 of those are required. The rest are nice and potentially helpful, but not required. Or the nonsense that pervaded (and sometimes still does) nutritional advice where one "needs to go low-carb to lose weight." Oh, you didn't actually track/control Calories while on low-carb? Then your conclusion is worthless.
  • crita50
    crita50 Posts: 28 Member
    Follow the money. The most important thing to note is who is paying for the study and what is their motivation for providing funding. What do they get out of it.
  • urloved33
    urloved33 Posts: 3,325 Member
    I happen to like independent reviews and independent research results.

Categories