Welcome to Debate Club! Please be aware that this is a space for respectful debate, and that your ideas will be challenged here. Please remember to critique the argument, not the author.

Peer reviewed studies are they the end all be all?

2»

Replies

  • dutchandkiwi
    dutchandkiwi Posts: 1,389 Member
    edited March 2018
    Peer review of course is flawed as it is most human processes. It is also always important to remember that peer review means evaluated on the basis of information given and a 'paper' process.
    If a researcher writes a paper and does not include all results then the peer review will nog have all information and that means that important parts can be missed. But a peer reviewer should look at what is given, balance the statistics, the sample (size and are they what was needed). The experimental set up and does it actually measure and prove what the paper say it does. By no means it is perfect. I too have found obvious errors, often editorial, in scientific papers when I was a student. Reviewers too are people with their own scientific bias.

    As far as I am aware the peer review does not include the underlying data set and raw data. Personally I think that is where the biggest flaw is, but it would also mean a lot more work for reviewers.But that is also the point where data/information may be lost due to not publishing of some.
    Leaving out of results may have very much an innocent reason; Total outliers in measurements, etc,

    Repeating of experiments/research is crucial and is not done often enough. There is simply more funding for new innovative research than there is for the more mundane confirmation studies and validation of studies. When I worked in the sciences, specifically measurement methodologies (longer ago than I want to admit) I concentrated on validation studies as I loved the puzzle that came with trying to repeat multiple times and determining performance of methods. But also how difficult it is to choose your statistical methodology and the interpretation that goes with statistics.

    Having said all that; peer review studies are better than none peer reviewed stuff. People just spouting a theory as the total truth without a shred of evidence (not in science, not in logic) Some of those things are simply dangerous garbage spread to desperate people and not seldom just for financial gain.
    Personally I go by: if the theory goes with a book and method that is expensive there is a significant change that it is woo.
    At least with a peer reviewed study I know knowledgable people have evaluated and balanced the research. It was judged but a jury of the researchers' peers.
  • jgnatca
    jgnatca Posts: 14,464 Member
    OP if you are looking for the absolute truth of a matter then you are going to be sorely disappointed. There are a few things that we reasonably know for sure. We all need to breathe or die in minutes. We all need enough water or we die in days. We all need to eat or fade away within weeks. We all sleep.

    I treat peer reviewed studies as the starting point of my trust. The farther away the claim is from those things I know are reasonably true, the less likely I am to trust it.

    If I don’t use a peer reviewed study as my starting point what am I left with? Tradition, rumour, anecdotes, Great Aunt Martha’s mustard plaster, and “miracle cures”.

    So what’s so wrong with tradition? It can seriously set you down the wrong path for a long time, making it harder to reach your goals.

    Let’s take the whole “purge the bowels” thing. These are all based on old cures before we had a full understanding of how the digestion system works. We now know that the liver and kidneys are responsible for filtering and removing harmful substances from our systems.

    https://www.mayoclinic.org/healthy-lifestyle/consumer-health/expert-answers/colon-cleansing/faq-20058435

    https://web.archive.org/web/20150424180208/http://www.cancer.org/Treatment/TreatmentsandSideEffects/ComplementaryandAlternativeMedicine/ManualHealingandPhysicalTouch/colon-therapy

    Why do colonics persist? In the absence of knowledge, colonic treatments are dramatic and visible. The patient is sure “something has been done” for them.
  • positivepowers
    positivepowers Posts: 902 Member
    janejellyroll wrote: »
    tirowow12385 wrote: »
    If one group of researchers came to a certain conclusion, another group or person will research and do a study to confirm it and if they come to the same conclusion, then you can say it was peer review, what is "it"? it's not that the researchers were peer reviewed but the topic at hand, whatever hypothesis it was they they were seeking to find the truth about, if enough researchers come to the same conclusion over and over, the hypothesis becomes a theory which is a nugget of truth which will probably be added to your kids science book as one sentence on page 51 which he probably won't remember.

    Peer review doesn't refer to other scientists replicating the study (although having results that others can replicate is an important part of science). It's the process by which your peers evaluate your work for potential errors or issues before it is published.

    This. I've had papers peer-reviewed. It just means that learned people in the same or relevant fields have checked the research, ran the numbers, and found no errors in logic, math, etc. They often make comments on grammar, format and things like that as well. It validates the research; it does not ensure the conclusions are correct.
  • Prime_Rib
    Prime_Rib Posts: 6 Member
    tirowow12385 wrote: »
    The whole idea behind people saying peer review is because they want to make it clear the study itself is legitimate and has been reviewed by other scholars in whatever the field. There's no vetting greater than "peer review".

    I am grossly generalizing here, but it depends upon the peers. For example, the whole kerfuffle about fat being bad was the result of a study in the 1960s, peer reviewed, saying that fat is bad for you. Come to find out, this study was paid for by a sugar consortium and the result was meat got a bad rap. Now it is the opposite. Sugars and carbs are the enemy and fats are fine. Who the heck knows.

    It is important to know who these peers are and who is paying them. I would argue that pure science is rarely done anymore. Perhaps at CERN. Science is often at the behest of corporations looking to be seen in a better light.

    Climate science is the big debate now. Some say it is "settled science," as if such a thing possibly exists. (We learn every seven years that about 35% of our scientific assumptions are incorrect.) However, in academe, there is believed to be a consensus on the subject. So, if I write a paper and say that cows cause greenhouse gases and therefore we all must be vegetarian, it will be reviewed by like-minded people and it will be deemed correct and not challenged. And if you are a burgeoning scientist who could lose tenure if you have a contrary view, backed up with data, what are you going to? Be a pariah and not feed your family? Or are you going to toe the line?

    There are lies, damned lies, and statistics.

    So when it comes to dietary science, I'd first try and ascertain who is paying for the study. Peer review, academe in general, is a big circle jerk. University presses are the biggest rip-offs going. One line changed in a textbook so they can charge incoming students $350 a book.

    I am very distrustful of the current climate of peer review. So I tend to look at these things carefully and try to make up my own mind.

    I know I sound like a tin-foil hat guy. But, just to be clear, they are made of aluminum.
  • laur357
    laur357 Posts: 896 Member
    Librarian here. Peer-reviewed articles in respected scholarly journals are the go-to. Ideally you have a diverse editorial board of 'experts' with years of experience who review and comment on studies before publishing. They will hopefully find flaws in the study, logic errors, and poor communication styles before publication. However, the studies are being reviewed by humans, so there will always be flaws and biases AND no one is recreating the experiment to see if it's possible to replicate at the time of publication. Some journals are more well-respected than others, and there are ranking systems and impact factors that can point you to the better journals. They are generally good options for information.

    Systematic reviews are better for overarching evidence, and currently the ideal for evidence-based science. They consolidate information and look at many peer-reviewed studies, then draw conclusions from evidence coming from many sources. Medical guidelines are often based on systematic reviews, and having many articles and datasets to examine will hopefully weed out or give less weight to poorly designed studies or inaccurate results. Systematic reviews are not always available because years of work and publications are needed before information can be synthesized.

    Major problem in popular science reporting - journalists are often not scientists, and can't spot crap research. There's also some things you might not know if you don't read the article: extrapolation to humans from animal subjects, broad conclusions drawn and applied to humans, studies touting super-cool weight loss stuff that was tested on 9 people (4 of whom dropped out halfway through the study), experts generalizing the results to support their own personal research theories, and hidden conflict of interest statements (the report didn't tell you that the scientist who did a study on diet soda consumption is on the public health committee for the Pepsi corporation).
  • TR0berts
    TR0berts Posts: 7,739 Member
    jjpptt2 wrote: »
    The part I struggle with is how many studies isolate variables in an attempt to study them, controlling the environment/circumstances to such a degree that I question how relevant those results are to a real-world scenario. Not that I doubt the validity of the results, but in a real-world scenario, there are (usually) FAR more factors at play. How does that all balance out? (rhetorical question, mostly).

    Sorry if that sidetracks the conversation in a direction you didn't want to go.


    It's not that they're necessarily trying to replicate real-world scenarios. Rather, it's an attempt to figure out what really does what. Sometimes, it's an attempt to figure out what doesn't do what. Then further tests - maybe actual studies, but not necessarily - attempt to put previous results together to see what works "best" for a particular goal.

    Otherwise, you get silly things like what was once on the MFP blog regarding body recomposition. It had something like 7 things that were supposedly required to recomp. No, only 3 of those are required. The rest are nice and potentially helpful, but not required. Or the nonsense that pervaded (and sometimes still does) nutritional advice where one "needs to go low-carb to lose weight." Oh, you didn't actually track/control Calories while on low-carb? Then your conclusion is worthless.
  • crita50
    crita50 Posts: 28 Member
    Follow the money. The most important thing to note is who is paying for the study and what is their motivation for providing funding. What do they get out of it.
  • urloved33
    urloved33 Posts: 3,323 Member
    I happen to like independent reviews and independent research results.
  • crackpotbaby
    crackpotbaby Posts: 1,297 Member
    Many people have not been taught skills in critical analysis and are unable to identify what is quality data or how it can be interpreted or applied.

    Some struggle to see the difference between an opinion and evidence.

    Regardless of the source, for an individual without the ability to interpret and apply that data, all information is basically ... meh.

    The quality of the research is critical but equally so is the analysis by the individual.

    Critical thinking, ftw.
This discussion has been closed.

Categories