Welcome to Debate Club! Please be aware that this is a space for respectful debate, and that your ideas will be challenged here. Please remember to critique the argument, not the author.

Do you read with a sense of skepticism?

13»

Replies

  • tbright1965
    tbright1965 Posts: 852 Member
    Just in time: https://www.facebook.com/CollegeHumor/videos/10155690434617807/

    Note, may not be safe for work as the f-bomb is dropped.
  • shaumom
    shaumom Posts: 1,003 Member
    ...When you read things, especially health and diet related, do you have a healthy sense of skepticism?

    Oh heck yeah. Although I tend to have skepticism of claims, but also certain aspects of research as well. I am one of those that checks for conflicts of interest when a study is completed, for the agenda of whoever is reporting the claims, of the potential for bias in studies.

    And while big established groups do often tend to have more well done research and I might often view them with more trust, they can also be more influenced by those with wealth and power, with 'the need to publish,' because the bigger groups tend to do studies that involve more potential gains in wealth by certain parties. There can also be more pressure on scientists to have findings that are the results their funding entity wants. And that same pressure can extend to those in the scientific publishing industry

    I really respect scientists, and what they do. But a number of famous scientists or people in the know have spoken up in the past few years about how broken the science publishing system is currently, to the point that it has put into question many studies, which is REALLY a shame, as there is some really important stuff we're discovering, and without trust in the system, then people feel justified in just saying 'that's fake news,' to whatever they don't like. *


    * a couple of the complaints/problems about the publishing system.
    - Article from 'The science council and institute of health' - https://www.acsh.org/news/2015/05/19/science-publication-is-hopelessly-compromised-say-journal-editors
    - article in scientist.com - https://www.the-scientist.com/uncategorized/is-peer-review-broken-47872
  • shaumom
    shaumom Posts: 1,003 Member
    That said, there is one thing that I run into , a LOT, where skepticism meets, hmmm, I'm gonna say a lack of understanding about the reality of human knowledge of health and medicine.

    Basically: there is a difference between 'currently proven to be impossible' and 'impossible as far as we know but new facts are putting that into question,' and 'never been studied.'

    There are some health claims made that fly in the face of all knowledge we have, and have no realistic facts to back up any new claims - I view those claims as 'previously known information says this is BS.'

    But there are health claims that have evidence to back them up, even when current knowledge would suggest these claims are wrong. If the evidence proves to be valid, then at that point, being skeptical, IMHO, means being skeptical about our CURRENT medical knowledge. We need to look at our current hypothesis about health and re-examine it, to see if we missed anything, and re-examine the new evidence, to see if the new evidence has some flaws.

    This kind of thing? I know a lot of people who will treat ANY information that contradicts previously well known science as 'woo,' even when a true scientist would be looking at both old and new information, as it's always a hypothesis, not an 'unchanging fact.'

    Admittedly, this one's personal. I have an auto-immune disorder that had symptoms patients reported for decades, that researchers insisted were impossible without even bothering to really ask questions of their patients to explore the subject, essentially. When it WAS finally studied, it showed that patients were absolutely correct; researchers' understanding of the scope of the disorder had been flawed. The researchers got trapped in that mind set of 'we now know everything about this subject, so you must be wrong,' and that's arrogance rather than skepticism, you know?


    And that last category, 'never been studied?' I think people don't realize how big a body of information that is, sometimes. Again, this one is personal. I have a rare disorder. It has REALLY not been studied. The majority of symptoms aren't yet listed for it because they have yet to be fully researched, so researchers just don't know. People with the disease will share information about treatments a doctor tried, just to see if it worked, so those can then share it with THEIR doctor, because there's so little research on treatments that the patients are telling doctors what helped as often as doctors are giving suggestions.

    Heck, we were excited because just last year, insurance companies finally recognized the disease and assigned it a code so doctors could actually BILL for it properly.

    But with something like this...it makes it really, really important to be skeptical about claims that X helped with Y symptom, but you also just have to look at what limited knowledge there is, and make an educated guess about the potential of the claims. There will be no studies verifying it for you. There will be nothing to confirm or deny it works but your own experience and what other lay people like you say.

    While this is more extreme that many illnesses, there are a surprising number of disorders and diseases where this still applies: a lot has not been researched and we don't know all of what helps or what hurts.

    I think that's hard to understand in today's world, where we have SO MUCH information around that it feels like we know the basics are almost everything. so there MUST be studies to back up claims or they have to be false. And that's...simply not true.

    I mean, sure, it could be the case - there's a LOT of BS out there, and many of it lacks studies to prove it. But some things are simply not yet studied. So a lack of supporting studies needs to be weighed against how much research there has been on a particular study, I think, to truly analyze any claims.
  • Wiseandcurious
    Wiseandcurious Posts: 730 Member
    edited January 2019
    I read with a healthy sense of scepticism... the things that I have no bias to agree with or have a bias to disagree with.

    I read, with the same rational mind, but with a lot greater predisposition to credibility and lesser predisposition to question things... the things that coincide with or support my own existing beliefs and attitudes.

    Really, I think almost everyone thinks they are rational and sceptical and capable of critical thinking but they mostly notice their healthy scepticism when it helps debunk something there wasn't much chance would fool them in the first place.

    My sense of scepticism is just that - a sense. What I think saves me most of the time is:
    - trying to be self-aware and call out my own biases as much as I can
    - using logical thinking and spotting logical falacies (there are a few very common ones that keep propping up on this forum, we should probably have a bingo card for those alone...)

    Eta: What I am trying to say is that I think the sense of scepticism is overrated, not in itself but because people feel it a lot less thatn they think and generally not in the situations where they would most need one, to counter their own bias. Practicing the other components of critical thinking, again and again in life, I think would be more helpful in separating truth from fiction.
  • earlnabby
    earlnabby Posts: 8,171 Member
    Skepticism to me basically means questioning. Is there already evidence available to prove or disprove the idea? Does it fly in the face of common sense? Is it too good to be true? Can you trust the expertise of the source?

    Anyone who comes to my door or contacts me by phone or email gets questioned. Who are you? What is the donation for? What are the issues you are campaigning on and where do you stand? Do you have a website or additional literature where I can find out more.

    I read health, weight loss, and nutrition information the same way. Always interested in new findings but tempered by questions
  • UltraVegAthlete
    UltraVegAthlete Posts: 667 Member
    I am skeptical about everything I read. There’s so much literature out there about how to live, how to eat, how to be a better person, that I think it’s up to our own judgements to make the call.
    The problem with the example with the cat and the milk is that milk contains something that makes the young come back to the mother. It’s addictive. I have a cat that I rescued when she was a kitten, abandoned too early, so now she likes to suck on ears and mews like a kitten. I don’t think milk is necessarily unhealthy, but I think species should stick to their own milk.
    Generally, my logic when it comes to what I eat, is if I can’t eat it raw without poisoning myself, or if my body reacts negatively to it (from allergies or something overpowering like garlic), I don’t eat it or I eat very little of it. I also don’t eat too much of something that I can’t physically get a lot of (nuts and seeds) or can’t get naturally (sugar and oil).
    I know I’m strict about my diet, but I’ve gotten used to it over the years.
  • shaumom
    shaumom Posts: 1,003 Member
    Aaron_K123 wrote: »
    shaumom wrote: »
    .
    Long story short though all this really amounts to is that you shouldn't take the conclusions of a publication are face value, even a scientific one. I dont think this is evidence that science is failing somehow, the reality will never be that every individual study is 100% reliable...that is just unrealistic. Scientific truths dont emerge from individual studies...they emerge from countless studies converging on the same answer until the entire field accepts them as truth. I think the public going to individual studies for what to believe instead of the overall consensus of the scientific community is the mistake.

    Oh, I would agree there's not evidence science is failing. I think it's more a problem with the current system as it has been slowly changing in the last decade or more. Not the system of research in terms of a single study, but the system as a whole, with what research is funded, and how, and how politics and funding are shaping research and publishing.

    As you said, scientific truths aren't found in one study. But currently, we're having some real problems with actually replicating the studies that are out there (https://www.bbc.com/news/science-environment-39054778). Some interesting, if frustrating, looks at studies in the past few years are finding very few repeated studies to see if the original study results were able to be replicated.

    I would agree that relying on a single study is a mistake, but it's one that is also encouraged by much of the layman's literature that repeat the findings of one study as though it is some newly discovered absolute truth. And then, if it's disproven, it simply reinforces worries about the unreliability of science, when in fact, it's more the 'misunderstanding' of science by sources that are then passing along that to the general public.

    ...and I hope this didn't get incoherent here, as it is so late I'm not really sure what I just wrote, LOL.
  • Aaron_K123
    Aaron_K123 Posts: 7,122 Member
    edited January 2019
    shaumom wrote: »
    Oh, I would agree there's not evidence science is failing. I think it's more a problem with the current system as it has been slowly changing in the last decade or more. Not the system of research in terms of a single study, but the system as a whole, with what research is funded, and how, and how politics and funding are shaping research and publishing.

    As you said, scientific truths aren't found in one study. But currently, we're having some real problems with actually replicating the studies that are out there (https://www.bbc.com/news/science-environment-39054778). Some interesting, if frustrating, looks at studies in the past few years are finding very few repeated studies to see if the original study results were able to be replicated.

    I would agree that relying on a single study is a mistake, but it's one that is also encouraged by much of the layman's literature that repeat the findings of one study as though it is some newly discovered absolute truth. And then, if it's disproven, it simply reinforces worries about the unreliability of science, when in fact, it's more the 'misunderstanding' of science by sources that are then passing along that to the general public.

    ...and I hope this didn't get incoherent here, as it is so late I'm not really sure what I just wrote, LOL.

    I think I do get what you are saying I just don't think that studies not being reproducible is somehow just a current issue and not something that has always been true. I don't think that the peer review system is suddenly just now failing, it is just that people in the public thought it was better than it actually is and are just shocked to hear that. The time in which someone becomes aware of an issue is not necessarily the time in which the issue began.

    The reason why individual studies are not treated as scientific truths within the scientific community is precisely because any individual study is likely to be flawed in some way and perhaps even critically flawed. It is only when there are mountains of studies all pointing to the same thing that you can feel some confidence that it is an accurate representation of reality because it is unlikely that ALL of the studies are somehow invalid and yet all point to the same thing.

    Scientists don't point to one study as evidence that something is absolutely true...the general public does that and the media does that and personally I don't think that is a good thing to do. Honestly I think there are a lot of the lay public who champion science so hard that they end up overselling it as somehow unimpeachable or that a "peer reviewed study" is just going to be true. I'm worried that there is going to be a backlash against the scientific community not based on what the scientific community itself claimed about its truths but rather about what the champions of science in the public realm claim about scientific truths that end up being on rather shaky ground of individual studies.

    I don't know, maybe not perhaps I'm being over dramatic there but I am not really that shocked to hear that many studies out there cannot be reproduced.
  • CSARdiver
    CSARdiver Posts: 6,257 Member
    Aaron_K123 wrote: »
    shaumom wrote: »
    Oh, I would agree there's not evidence science is failing. I think it's more a problem with the current system as it has been slowly changing in the last decade or more. Not the system of research in terms of a single study, but the system as a whole, with what research is funded, and how, and how politics and funding are shaping research and publishing.

    As you said, scientific truths aren't found in one study. But currently, we're having some real problems with actually replicating the studies that are out there (https://www.bbc.com/news/science-environment-39054778). Some interesting, if frustrating, looks at studies in the past few years are finding very few repeated studies to see if the original study results were able to be replicated.

    I would agree that relying on a single study is a mistake, but it's one that is also encouraged by much of the layman's literature that repeat the findings of one study as though it is some newly discovered absolute truth. And then, if it's disproven, it simply reinforces worries about the unreliability of science, when in fact, it's more the 'misunderstanding' of science by sources that are then passing along that to the general public.

    ...and I hope this didn't get incoherent here, as it is so late I'm not really sure what I just wrote, LOL.

    I think I do get what you are saying I just don't think that studies not being reproducible is somehow just a current issue and not something that has always been true. I don't think that the peer review system is suddenly just now failing, it is just that people in the public thought it was better than it actually is and are just shocked to hear that. The time in which someone becomes aware of an issue is not necessarily the time in which the issue began.

    The reason why individual studies are not treated as scientific truths within the scientific community is precisely because any individual study is likely to be flawed in some way and perhaps even critically flawed. It is only when there are mountains of studies all pointing to the same thing that you can feel some confidence that it is an accurate representation of reality because it is unlikely that ALL of the studies are somehow invalid and yet all point to the same thing.

    Scientists don't point to one study as evidence that something is absolutely true...the general public does that and the media does that and personally I don't think that is a good thing to do. Honestly I think there are a lot of the lay public who champion science so hard that they end up overselling it as somehow unimpeachable or that a "peer reviewed study" is just going to be true. I'm worried that there is going to be a backlash against the scientific community not based on what the scientific community itself claimed about its truths but rather about what the champions of science in the public realm claim about scientific truths that end up being on rather shaky ground of individual studies.

    I don't know, maybe not perhaps I'm being over dramatic there but I am not really that shocked to hear that many studies out there cannot be reproduced.

    It's the volume of data at play here. A loose application of Price's Law and the Pareto Principle. Quality is sacrificed for volume.

    I would be interested in seeing how this plays out in various disciplines. I would expect most physical science papers to be reproducible due to the abstract nature. Life/environmental sciences would be far less, and social sciences would be laughable.
  • Johnd2000
    Johnd2000 Posts: 198 Member
    I’ve reached that age where my starting point before reading anything is “this is probably nonsense, but here goes...”.

    Always ask yourself how the writer or publisher benefits from saying what they’re saying.
  • nrtauthor
    nrtauthor Posts: 159 Member
    I always read this stuff with a ton of skepticism.

    The questions I ask myself.

    1. What are they selling? They're almost ALWAYS selling something.

    2. What is their opinion on eggs? Remember when eggs were bad? Then good? Then bad again? Then good. Then bad...

    3. I need to see the actual scientific studies. Just referencing some vague unknowable study isn't enough. The actual study needs to be linked or I presume it's nonsense. Just like statistics, studies can be very biased and poorly conducted.

    One of my favorite past times used to be reading Women's Weekly. I know, trash right? But what amused me is every week they'd have a different stance on what helps your thyroid ( I have a thyroid condition so it interests me).

    Week one cauliflower is terrible for your thyroid! Avoid it at all costs!

    Week two cauliflower is GREAT for your thyroid! Go ahead and eat it.

    Week three cauliflower is horrible for thyroid problems! Don't eat it.

    Week four the verdict is out. Nobody knows if cauliflower is good or bad.

    It's such nonsense.
  • mbaker566
    mbaker566 Posts: 11,233 Member
    edited January 2019
    yes. most of my friends jump on bandwagons. i try not to shake my head at them
  • Silentpadna
    Silentpadna Posts: 1,306 Member
    Aaron_K123 wrote: »
    shaumom wrote: »
    ...When you read things, especially health and diet related, do you have a healthy sense of skepticism?

    Oh heck yeah. Although I tend to have skepticism of claims, but also certain aspects of research as well. I am one of those that checks for conflicts of interest when a study is completed, for the agenda of whoever is reporting the claims, of the potential for bias in studies.

    And while big established groups do often tend to have more well done research and I might often view them with more trust, they can also be more influenced by those with wealth and power, with 'the need to publish,' because the bigger groups tend to do studies that involve more potential gains in wealth by certain parties. There can also be more pressure on scientists to have findings that are the results their funding entity wants. And that same pressure can extend to those in the scientific publishing industry

    I really respect scientists, and what they do. But a number of famous scientists or people in the know have spoken up in the past few years about how broken the science publishing system is currently, to the point that it has put into question many studies, which is REALLY a shame, as there is some really important stuff we're discovering, and without trust in the system, then people feel justified in just saying 'that's fake news,' to whatever they don't like. *


    * a couple of the complaints/problems about the publishing system.
    - Article from 'The science council and institute of health' - https://www.acsh.org/news/2015/05/19/science-publication-is-hopelessly-compromised-say-journal-editors
    - article in scientist.com - https://www.the-scientist.com/uncategorized/is-peer-review-broken-47872

    I am a scientist and have been on both sides of peer review...both having my work reviewed for publication and reviewing others work for publication. It certainly is not a perfect system and there is certainly room for bias and laziness amongst reviewers and room for shoddy work to slip by.

    I think the public assumes peer review is a more rigorous process than it actually is. I get the sense that people think peer review means that the experts in a field attempt to reproduce the work or something and that certainly isn't the case. Frankly there is way way to much work put up for publication than there are people who have the time to do a real in depth critical review.

    Most times what happens is a paper is submitted to a journal, that journal has an editor who is familiar with the field that the journal specializes in and sends that manuscript out to usually three reviewers. These reviewers dont work for the journal, they are just other scientists who work in that field. Given that fields are so specialized often a manuscript can end up in the hands of a reviewer that is only moderately familiar with the field. These reviewers have full time jobs unrelated to reviewing and they dont get any extra time or pay for the act of reviewing. So really the best you can expect is for them to read the entire manuscript armed with the knowledge of the field they have which might not be all that detailed and then just provide critical commentary. That goes back to the editor who decides whether to reject the paper, accept the paper or go back to the authors and ask them to address the reviewers criticisms which can range from not liking a phrasing of a sentence to requesting that an entire new experiment be performed.

    All the reviewers have to go off of is what the authors say they did so there has to be some level of trust that the authors are correctly representing their work. Also any one person, even an expert in their field, is unlikely to know everything and can certainly miss dire problems with the work. For example a group might have a statistician who could spot an overly small sample size a mile away but the person from that group reviewing the paper isnt the statistician it's a biologist who can interpret the method used but may not recognize an issue with the sample size. Or maybe the statistician is the reviewer and can catch the issue with the sample size but doesn't catch that the buological method that was used has serious problems.

    It is to much to ask of any given reviewer to put in a ton of time to study up on all the specifics and history of the field that the paper they are reviewing relates to. I mean typically speaking a reviewer probably is going to spend maybe half a day on it... not weeks.

    Important to realize how many papers are published daily and how many many more are submitted for publication. There is no reward for doing a good job as a paper reviewer and no real punishment for doing a bad job. There is, however, a lot of pressure to finish the review to get back to your actual job. Not sure I see a way around this.

    The best analogy I can think of for the peer review system outside of science is jury duty. Science publications are peer reviewed and defendants have a jury of their peers. There aren't professional jurors anymore than there are professional reviewers...they are basically drafted. The quality of "peer" you get and how much time they are going to actually devote to the process is a pretty mixed bag and both probably just want to finish quickly so they can get back to their actual job.

    Long story short though all this really amounts to is that you shouldn't take the conclusions of a publication are face value, even a scientific one. I dont think this is evidence that science is failing somehow, the reality will never be that every individual study is 100% reliable...that is just unrealistic. Scientific truths dont emerge from individual studies...they emerge from countless studies converging on the same answer until the entire field accepts them as truth. I think the public going to individual studies for what to believe instead of the overall consensus of the scientific community is the mistake.

    As a natural skeptic, I am enjoying this train of thought. I think this piece is very even-handed. I applaud the fact that, as a scientist, you are not holding up peer review as the ultimate authority, nor are you denigrating it as meaningless.

    Having served on a jury recently in a murder trial, your reference to the "professional" side of that is interesting. As much as the lawyers and judge spoke about the intricacies of the law, ultimately the judge reminded us that the law was basically ours to interpret. And of course there are 12 different perspectives and reasoning capabilities in that room.

    I'm not fond of the "appeal to authority" logical fallacy that is not necessarily used by authors of published material, but by readers with an agenda. "Hey dude, it's 'settled' science. See this 'peer-reviewed' study?" But that is not to say that authority on a subject is meaningless.

    That said, I think peer-review is a valid concept. It's an imperfect one, but it's better than a lot of other possible ways to help discern truth.
  • Aaron_K123
    Aaron_K123 Posts: 7,122 Member
    edited January 2019
    Aaron_K123 wrote: »
    shaumom wrote: »
    ...When you read things, especially health and diet related, do you have a healthy sense of skepticism?

    Oh heck yeah. Although I tend to have skepticism of claims, but also certain aspects of research as well. I am one of those that checks for conflicts of interest when a study is completed, for the agenda of whoever is reporting the claims, of the potential for bias in studies.

    And while big established groups do often tend to have more well done research and I might often view them with more trust, they can also be more influenced by those with wealth and power, with 'the need to publish,' because the bigger groups tend to do studies that involve more potential gains in wealth by certain parties. There can also be more pressure on scientists to have findings that are the results their funding entity wants. And that same pressure can extend to those in the scientific publishing industry

    I really respect scientists, and what they do. But a number of famous scientists or people in the know have spoken up in the past few years about how broken the science publishing system is currently, to the point that it has put into question many studies, which is REALLY a shame, as there is some really important stuff we're discovering, and without trust in the system, then people feel justified in just saying 'that's fake news,' to whatever they don't like. *


    * a couple of the complaints/problems about the publishing system.
    - Article from 'The science council and institute of health' - https://www.acsh.org/news/2015/05/19/science-publication-is-hopelessly-compromised-say-journal-editors
    - article in scientist.com - https://www.the-scientist.com/uncategorized/is-peer-review-broken-47872

    I am a scientist and have been on both sides of peer review...both having my work reviewed for publication and reviewing others work for publication. It certainly is not a perfect system and there is certainly room for bias and laziness amongst reviewers and room for shoddy work to slip by.

    I think the public assumes peer review is a more rigorous process than it actually is. I get the sense that people think peer review means that the experts in a field attempt to reproduce the work or something and that certainly isn't the case. Frankly there is way way to much work put up for publication than there are people who have the time to do a real in depth critical review.

    Most times what happens is a paper is submitted to a journal, that journal has an editor who is familiar with the field that the journal specializes in and sends that manuscript out to usually three reviewers. These reviewers dont work for the journal, they are just other scientists who work in that field. Given that fields are so specialized often a manuscript can end up in the hands of a reviewer that is only moderately familiar with the field. These reviewers have full time jobs unrelated to reviewing and they dont get any extra time or pay for the act of reviewing. So really the best you can expect is for them to read the entire manuscript armed with the knowledge of the field they have which might not be all that detailed and then just provide critical commentary. That goes back to the editor who decides whether to reject the paper, accept the paper or go back to the authors and ask them to address the reviewers criticisms which can range from not liking a phrasing of a sentence to requesting that an entire new experiment be performed.

    All the reviewers have to go off of is what the authors say they did so there has to be some level of trust that the authors are correctly representing their work. Also any one person, even an expert in their field, is unlikely to know everything and can certainly miss dire problems with the work. For example a group might have a statistician who could spot an overly small sample size a mile away but the person from that group reviewing the paper isnt the statistician it's a biologist who can interpret the method used but may not recognize an issue with the sample size. Or maybe the statistician is the reviewer and can catch the issue with the sample size but doesn't catch that the buological method that was used has serious problems.

    It is to much to ask of any given reviewer to put in a ton of time to study up on all the specifics and history of the field that the paper they are reviewing relates to. I mean typically speaking a reviewer probably is going to spend maybe half a day on it... not weeks.

    Important to realize how many papers are published daily and how many many more are submitted for publication. There is no reward for doing a good job as a paper reviewer and no real punishment for doing a bad job. There is, however, a lot of pressure to finish the review to get back to your actual job. Not sure I see a way around this.

    The best analogy I can think of for the peer review system outside of science is jury duty. Science publications are peer reviewed and defendants have a jury of their peers. There aren't professional jurors anymore than there are professional reviewers...they are basically drafted. The quality of "peer" you get and how much time they are going to actually devote to the process is a pretty mixed bag and both probably just want to finish quickly so they can get back to their actual job.

    Long story short though all this really amounts to is that you shouldn't take the conclusions of a publication are face value, even a scientific one. I dont think this is evidence that science is failing somehow, the reality will never be that every individual study is 100% reliable...that is just unrealistic. Scientific truths dont emerge from individual studies...they emerge from countless studies converging on the same answer until the entire field accepts them as truth. I think the public going to individual studies for what to believe instead of the overall consensus of the scientific community is the mistake.

    As a natural skeptic, I am enjoying this train of thought. I think this piece is very even-handed. I applaud the fact that, as a scientist, you are not holding up peer review as the ultimate authority, nor are you denigrating it as meaningless.

    Having served on a jury recently in a murder trial, your reference to the "professional" side of that is interesting. As much as the lawyers and judge spoke about the intricacies of the law, ultimately the judge reminded us that the law was basically ours to interpret. And of course there are 12 different perspectives and reasoning capabilities in that room.

    I'm not fond of the "appeal to authority" logical fallacy that is not necessarily used by authors of published material, but by readers with an agenda. "Hey dude, it's 'settled' science. See this 'peer-reviewed' study?" But that is not to say that authority on a subject is meaningless.

    That said, I think peer-review is a valid concept. It's an imperfect one, but it's better than a lot of other possible ways to help discern truth.

    Science itself is predicated on the idea that human senses and intuitions are hopelessly biased and flawed and our best means of getting at truth is to conduct observational experiments designed to take the human element out of the equation as much as possible.

    Peer review itself is of course a human system by necessity and as such it would be miraculous if it wasnt biased and flawed. I actually doubt there are many scientists who would claim peer review is somehow unimpeachable....but I dont think anyone has come up with a better idea really. It is important to critically review work before it is published and it makes sense that the experts in the field be the ones to do it. But yeah...there is bias, there are mistakes sure and there is not enough time reasonably to truly vet a body of work that took an entire lab 4 years to produce over a cup of coffee with a paper printout.

    I mean what do people expect...that every submitted publication will have the experimental work repeated by a third party to confirm reproducibility before publication is accepted? In the parlance of the internet....ain't nobody got time for that.
  • Aaron_K123
    Aaron_K123 Posts: 7,122 Member
    edited January 2019
    Just looking through my email here is the last time I got a request to review a manuscript submitted for publication in case anyone is curious as to what that looks like. Not particularly formal...kind of like a "hey guy, want to do some work for us?" Redacted most of it so it isn't showing anything potentially confidential but figure you can still get the gist.

    1zgb9zs.jpg
  • Silentpadna
    Silentpadna Posts: 1,306 Member
    Aaron_K123 wrote: »
    Aaron_K123 wrote: »
    shaumom wrote: »
    ...When you read things, especially health and diet related, do you have a healthy sense of skepticism?

    Oh heck yeah. Although I tend to have skepticism of claims, but also certain aspects of research as well. I am one of those that checks for conflicts of interest when a study is completed, for the agenda of whoever is reporting the claims, of the potential for bias in studies.

    And while big established groups do often tend to have more well done research and I might often view them with more trust, they can also be more influenced by those with wealth and power, with 'the need to publish,' because the bigger groups tend to do studies that involve more potential gains in wealth by certain parties. There can also be more pressure on scientists to have findings that are the results their funding entity wants. And that same pressure can extend to those in the scientific publishing industry

    I really respect scientists, and what they do. But a number of famous scientists or people in the know have spoken up in the past few years about how broken the science publishing system is currently, to the point that it has put into question many studies, which is REALLY a shame, as there is some really important stuff we're discovering, and without trust in the system, then people feel justified in just saying 'that's fake news,' to whatever they don't like. *


    * a couple of the complaints/problems about the publishing system.
    - Article from 'The science council and institute of health' - https://www.acsh.org/news/2015/05/19/science-publication-is-hopelessly-compromised-say-journal-editors
    - article in scientist.com - https://www.the-scientist.com/uncategorized/is-peer-review-broken-47872

    I am a scientist and have been on both sides of peer review...both having my work reviewed for publication and reviewing others work for publication. It certainly is not a perfect system and there is certainly room for bias and laziness amongst reviewers and room for shoddy work to slip by.

    I think the public assumes peer review is a more rigorous process than it actually is. I get the sense that people think peer review means that the experts in a field attempt to reproduce the work or something and that certainly isn't the case. Frankly there is way way to much work put up for publication than there are people who have the time to do a real in depth critical review.

    Most times what happens is a paper is submitted to a journal, that journal has an editor who is familiar with the field that the journal specializes in and sends that manuscript out to usually three reviewers. These reviewers dont work for the journal, they are just other scientists who work in that field. Given that fields are so specialized often a manuscript can end up in the hands of a reviewer that is only moderately familiar with the field. These reviewers have full time jobs unrelated to reviewing and they dont get any extra time or pay for the act of reviewing. So really the best you can expect is for them to read the entire manuscript armed with the knowledge of the field they have which might not be all that detailed and then just provide critical commentary. That goes back to the editor who decides whether to reject the paper, accept the paper or go back to the authors and ask them to address the reviewers criticisms which can range from not liking a phrasing of a sentence to requesting that an entire new experiment be performed.

    All the reviewers have to go off of is what the authors say they did so there has to be some level of trust that the authors are correctly representing their work. Also any one person, even an expert in their field, is unlikely to know everything and can certainly miss dire problems with the work. For example a group might have a statistician who could spot an overly small sample size a mile away but the person from that group reviewing the paper isnt the statistician it's a biologist who can interpret the method used but may not recognize an issue with the sample size. Or maybe the statistician is the reviewer and can catch the issue with the sample size but doesn't catch that the buological method that was used has serious problems.

    It is to much to ask of any given reviewer to put in a ton of time to study up on all the specifics and history of the field that the paper they are reviewing relates to. I mean typically speaking a reviewer probably is going to spend maybe half a day on it... not weeks.

    Important to realize how many papers are published daily and how many many more are submitted for publication. There is no reward for doing a good job as a paper reviewer and no real punishment for doing a bad job. There is, however, a lot of pressure to finish the review to get back to your actual job. Not sure I see a way around this.

    The best analogy I can think of for the peer review system outside of science is jury duty. Science publications are peer reviewed and defendants have a jury of their peers. There aren't professional jurors anymore than there are professional reviewers...they are basically drafted. The quality of "peer" you get and how much time they are going to actually devote to the process is a pretty mixed bag and both probably just want to finish quickly so they can get back to their actual job.

    Long story short though all this really amounts to is that you shouldn't take the conclusions of a publication are face value, even a scientific one. I dont think this is evidence that science is failing somehow, the reality will never be that every individual study is 100% reliable...that is just unrealistic. Scientific truths dont emerge from individual studies...they emerge from countless studies converging on the same answer until the entire field accepts them as truth. I think the public going to individual studies for what to believe instead of the overall consensus of the scientific community is the mistake.

    As a natural skeptic, I am enjoying this train of thought. I think this piece is very even-handed. I applaud the fact that, as a scientist, you are not holding up peer review as the ultimate authority, nor are you denigrating it as meaningless.

    Having served on a jury recently in a murder trial, your reference to the "professional" side of that is interesting. As much as the lawyers and judge spoke about the intricacies of the law, ultimately the judge reminded us that the law was basically ours to interpret. And of course there are 12 different perspectives and reasoning capabilities in that room.

    I'm not fond of the "appeal to authority" logical fallacy that is not necessarily used by authors of published material, but by readers with an agenda. "Hey dude, it's 'settled' science. See this 'peer-reviewed' study?" But that is not to say that authority on a subject is meaningless.

    That said, I think peer-review is a valid concept. It's an imperfect one, but it's better than a lot of other possible ways to help discern truth.

    Science itself is predicated on the idea that human senses and intuitions are hopelessly biased and flawed and our best means of getting at truth is to conduct observational experiments designed to take the human element out of the equation as much as possible.

    Peer review itself is of course a human system by necessity and as such it would be miraculous if it wasnt biased and flawed. I actually doubt there are many scientists who would claim peer review is somehow unimpeachable....but I dont think anyone has come up with a better idea really. It is important to critically review work before it is published and it makes sense that the experts in the field be the ones to do it. But yeah...there is bias, there are mistakes sure and there is not enough time reasonably to truly vet a body of work that took an entire lab 4 years to produce over a cup of coffee with a paper printout.

    I mean what do people expect...that every submitted publication will have the experimental work repeated by a third party to confirm reproducibility before publication is accepted? In the parlance of the internet....ain't nobody got time for that.

    This is part of what I find refreshing. In my somewhat limited discussions with people on this topic, those appealing the most stringently to peer-reviewed sources aren't folks like you. It's folks who want information that validates things in accordance with existing biases.

    Even with that said, I tend to want to see those "vetted" (even if not perfectly) sources to be convinced, more so than anecdotal stuff.

    Keep it coming @Aaron_K123. Always enjoy your contributions to these threads.