Using science to argue your case

totem12

Please don't bite my head off...but here I go...

There are a few things get under my skin a bit when it comes to discussions/arguments like this:

1. There is more than likely a lot of unpublished or non-peer reviewed scientific articles out there in all disciplines that have very relevant information, because the findings in these studies go against the mainstream thought on that subject, or the results are not what the company contracting the study wants to see. I think this is the biggest issue with the GMO debate - the only peer-reviewed information says they're safe, and other studies that say otherwise are not published through reliable sources. Just because something is published through a reliable source doesn't necessarily mean it's 100% true, and just because something isn't published through a reliable source doesn't necessarily mean it's 100% false.

2. Scientists have the ability to alter time-lines and control groups in order to achieve certain results, or to make a result appear to be positive when it's not. I'm not saying all scientists do this, and I'm assuming that most don't, but the possibility is still there.

3. Conflict of interest. Again back to the GMO debate - if a company makes a new product and the same company is the one to test it to see if it's safe, is that not a terrible conflict of interest, and thus not entirely reliable? Third parties/independent studies I feel would be more reliable, but as mentioned before, if the results do not match the agenda, they can and most likely will be discredited.

4. Science can't prove everything, or at least not yet. Why do holistic approaches to medicine still work when there is no evidence that they do? Why do some children with autism that have changed to an organic diet have less problems with their autism? Maybe I just haven't done enough of my own research on stuff like that, but there is a lot of weird stuff in our world that science has not been able to prove and I don't know if it ever will.

I do agree that hard facts are important, but I think a lot more people are becoming increasingly wary of the "hard facts" that are presented because we are aware that we have been lied to in the past on what to believe. People want to be more aware, and people are aware that things are being hidden, and that's why they don't always trust science.

Some of these are definite problems with the scientific process and form part of what I was saying about learning to spot the warning signs! I disagree about the holistic approaches, only because for me personally I believe a trial trumps an anecdote (sometimes it is true, strange things happen and noone can tell you why, such is the random world we live in). I do however realise that these trials are often viewed the way you describe, as biased and conducted by someone with an agenda to push. Sometimes that is true, sometimes it is not.

Metastudies are often a good place to start, they are impartial, sift out as many badly designed studies as they can and draw conclusions about the entire breadth of data available.

Someone mentioned Ben Goldacre, he is certainly a good read! Recommend his books.

billsica

TL;DR


is this about Crossfit, Paleo or a sugar cleanse. In which case

Yes, Yes, Yes!

totem12

Most studies in peer reviewed journals can't ever be repeated.

You must not be familiar with how peer reviewed studies work.

I have published several of my own, so yes I am very familiar. There is no criterion in the process that says you have to prove the study could be independently replicated. A good peer review process simply demands that the information is there should anyone care to try.

UsedToBeHusky

1. There is more than likely a lot of unpublished or non-peer reviewed scientific articles out there in all disciplines that have very relevant information, because the findings in these studies go against the mainstream thought on that subject, or the results are not what the company contracting the study wants to see. I think this is the biggest issue with the GMO debate - the only peer-reviewed information says they're safe, and other studies that say otherwise are not published through reliable sources. Just because something is published through a reliable source doesn't necessarily mean it's 100% true, and just because something isn't published through a reliable source doesn't necessarily mean it's 100% false.

This is not true. For every peer-reviewed study, there is a hollistic advisor that has anecdotal evidence to the contrary. Or there are biased studies that are not peer-reviewed. The term "peer-reviewed" very specifically means that other scientists have recreated the parameters of the study and produced the same results.

2. Scientists have the ability to alter time-lines and control groups in order to achieve certain results, or to make a result appear to be positive when it's not. I'm not saying all scientists do this, and I'm assuming that most don't, but the possibility is still there.

This is not untrue. However, as previously stated, for a study to be considered to be "peer-reviewed," then other scientists have to recreate the parameters of the study and produce the same results. Not everyone is going to be in league with the original researcher. Particularly if a study is paid for by a food distributor, the FDA hires their own scientists to reproduce the results before claiming the original study has any basis.

3. Conflict of interest. Again back to the GMO debate - if a company makes a new product and the same company is the one to test it to see if it's safe, is that not a terrible conflict of interest, and thus not entirely reliable? Third parties/independent studies I feel would be more reliable, but as mentioned before, if the results do not match the agenda, they can and most likely will be discredited.

Just to further support the definition of "peer-reviewed," it means that scientists with equivalent qualifications as the original researcher performed the same study and got the same results. This does not mean that the peers that reviewed it were being paid by the same company or are otherwise in league with the original researcher. I'm not saying that this could never happen, but that doesn't mean that every study is laden with conspiracy and bias either.

4. Science can't prove everything, or at least not yet. Why do holistic approaches to medicine still work when there is no evidence that they do? Why do some children with autism that have changed to an organic diet have less problems with their autism? Maybe I just haven't done enough of my own research on stuff like that, but there is a lot of weird stuff in our world that science has not been able to prove and I don't know if it ever will.

Holistic approaches work because of faith. People believe they work and want to see evidence of it, therefore, they will recognize any positive result as proof that it works and disregard all else. Which is fine... if that works for some, then that is awesome. But that doesn't mean science has lost all credibility or that the findings of a controlled, peer-reviewed study should be completely disregarded because somebody might have tainted it with bias.

UsedToBeHusky

Most studies in peer reviewed journals can't ever be repeated.

You must not be familiar with how peer reviewed studies work.

I have published several of my own, so yes I am very familiar. There is no criterion in the process that says you have to prove the study could be independently replicated. A good peer review process simply demands that the information is there should anyone care to try.

I'm not sure why you think this, but you are wrong. The term "peer-reviewed" means the study was replicated and the results were repeated.

randomtai

TL;DR


is this about Crossfit, Paleo or a sugar cleanse. In which case

Yes, Yes, Yes!

Agreed... NERDS!! :P

ElliInJapan

One non-diet example of this for me is Dark Matter and Dark Energy. To me, it sounds like a cop-out... "Hmmm, the mass umbers are wrong on a galactic scale, but if we change them by adding in 'dark matter' that does nothing but add mass, then the numbers work! Therefore dark matter! Now we just have to find some."* However, it is the general consensus, and nobody has come up with a better explanation, so I will accept it for now. Were I an astrophysicist, I would probably be trying to figure out if there is an alternate explanation that makes more sense to me (but I may also have come to understand the equations and why dark matter is the best option).

*I admit that is a way-over-simplified view of the whole dark matter argument.

Haha, I understand how the terms sound somewhat fishy, but they're pretty descriptive. Without getting into many details, I'll say a few words, just because I like this area so much. There is evidence from multiple sources that the mass in the universe is more than the mass we "see" through telescopes etc, i.e. there is mass that only feels gravity and no other forces, but it's not clear what exactly it is. There are many (way too many) possible candidates and theories but the data are not conclusive up to now. So it's "dark matter" (i.e. non-radiative matter) until this settles. As for dark energy, it's more difficult to explain, but let's say the way the universe expands tells us there is some "background" energy that is neither mass nor radiation. And it's actually most of the universe's energy, about 70%. The thing is, it's very difficult and expensive to do experiments, and the theories we have up to now are incomplete and break down for the very early universe. There are a lot of possible explanations, way too many, but the problem is we cannot rule out most of them.

mrmagee3

Most studies in peer reviewed journals can't ever be repeated.

You must not be familiar with how peer reviewed studies work.

I have published several of my own, so yes I am very familiar. There is no criterion in the process that says you have to prove the study could be independently replicated. A good peer review process simply demands that the information is there should anyone care to try.

I'm not sure why you think this, but you are wrong. The term "peer-reviewed" means the study was replicated and the results were repeated.

I'm going to have to go with her on this one. I've been on both sides of this process, and it's never meant that. What field are you in? Perhaps that would explain the difference.

mrmagee3

One non-diet example of this for me is Dark Matter and Dark Energy. To me, it sounds like a cop-out... "Hmmm, the mass umbers are wrong on a galactic scale, but if we change them by adding in 'dark matter' that does nothing but add mass, then the numbers work! Therefore dark matter! Now we just have to find some."* However, it is the general consensus, and nobody has come up with a better explanation, so I will accept it for now. Were I an astrophysicist, I would probably be trying to figure out if there is an alternate explanation that makes more sense to me (but I may also have come to understand the equations and why dark matter is the best option).

*I admit that is a way-over-simplified view of the whole dark matter argument.

Haha, I understand how the terms sound somewhat fishy, but they're pretty descriptive. Without getting into many details, I'll say a few words, just because I like this area so much. There is evidence from multiple sources that the mass in the universe is more than the mass we "see" through telescopes etc, i.e. there is mass that only feels gravity and no other forces, but it's not clear what exactly it is. There are many (way too many) possible candidates and theories but the data are not conclusive up to now. So it's "dark matter" (i.e. non-radiative matter) until this settles. As for dark energy, it's more difficult to explain, but let's say the way the universe expands tells us there is some "background" energy that is neither mass nor radiation. And it's actually most of the universe's energy, about 70%. The thing is, it's very difficult and expensive to do experiments, and the theories we have up to now are incomplete and break down for the very early universe. There are a lot of possible explanations, way too many, but the problem is we cannot rule out most of them.

Your science is way more fun than my science.

KombuchaCat

I totally agree! Nothing bugs me more on this site than people refuting my opinion with something they just goggled. When I give an opinion it's just that, my opinion based on my life experience. I never said I had a degree in ????whatever. We are all allowed to have an opinion and express that on here if we like. However some people just want to be right or at least proove other's wrong. That being said I'm sure that people doing what you have said on here is not bound to change any time soon.

totem12

Most studies in peer reviewed journals can't ever be repeated.

You must not be familiar with how peer reviewed studies work.

I have published several of my own, so yes I am very familiar. There is no criterion in the process that says you have to prove the study could be independently replicated. A good peer review process simply demands that the information is there should anyone care to try.

I'm not sure why you think this, but you are wrong. The term "peer-reviewed" means the study was replicated and the results were repeated.

That is completely untrue. Peer review is the process by which your 'peers' in the field review your manuscript and add criticisms. Papers are revised according to the reviewer comments before the paper can be published. As I said, I have several published papers, and noone but me has ever carried out those experiments as far as I'm aware.

UsedToBeHusky

Most studies in peer reviewed journals can't ever be repeated.

You must not be familiar with how peer reviewed studies work.

I have published several of my own, so yes I am very familiar. There is no criterion in the process that says you have to prove the study could be independently replicated. A good peer review process simply demands that the information is there should anyone care to try.

I'm not sure why you think this, but you are wrong. The term "peer-reviewed" means the study was replicated and the results were repeated.

I'm going to have to go with her on this one. I've been on both sides of this process, and it's never meant that. What field are you in? Perhaps that would explain the difference.

I'm a business major. That is my understanding of "peer-reviewed". How can findings be accepted as fact if the experiment can't be recreated?

edited to add - Okay... I discussed this with a friend and I think I know where my confusion is. I'm actually referring to controlled studies that would not use human subjects.

It is true, that a study with human subjects could not be repeated with a great degree of accuracy.

BrianSharpe

Science is not trusted at all, which isn't that surprising when scientists have been very poor in communicating their subject to the public, and the media haven't bothered to try and report it accurately. Science comms often boils down to 'Clever scientist man said this so you should believe him'.

Surveys of the general public suggest that the most trusted professionals are doctors, and the least trusted are journalists. Yet still we get things like the MMR scare, and conspiracies abound regarding Big Pharma etc. It's all very strange.

Interesting point. I strongly suspect that the biggest reason for mistrust in science is the incredibly low level of scientific literacy, especially in North America. I don't know if it's a result of dumbing down of educational standards (a great way to improve test scores is to make the tests easier) but I've certainly observed a decline in the general knowledge of a high school graduate of my generation to more recent graduates ( a real eye opener was a grade 8 yest from 1911 that's been making the rounds on social media http://www.dailymail.co.uk/femail/article-2381482/Were-children-smarter-century-ago-Test-eighth-graders-Kentucky-dated-1912-ignites-debate-kids-intelligence-today.html)

BTW : I always though it would be lawyers and politicians considered the least trustworthy.....

totem12

Of course in an ideal world they can be! I would hope that mine could be replicated by anyone. Various factors contribute to why some cannot. Lots of experiments are long and difficult, technically challenging, and getting that final figure that is good enough quality to publish can be a miraculous feat of scientific magic that you'll never manage to pull off again. Scientists under pressure to publish can rush out a paper as soon as they manage to get the results they were hoping for and later find those results never show up again. It's terrible and it's flawed and unfortunately it's still the best we have.

AccioHotBod

Snark aside, I am a neuroscientist and I am genuinely curious why the general public does not trust science/peer review, and what you trust instead. It makes me sad to see that people basically think that my life's work is worthless.

Also, serious offer, if anyone on here wants help with reading and interpreting any particular research article, please PM me, I'm happy to help. Sometimes if you only read the abstract, that's not helpful. The most important sections are the methods and results.

I'm not going to speak for your field (because I've never read any peer-reviewed journals there or understand the publication process) but in many fields in the social sciences, for example, only articles that find a statistically significant effect (with the arbitrary p<.05) get published. So, authors are inclined to search for a hypothesis that will get them published, irrespective of any mounting evidence that refutes the current relationship they observe in their own study. The magnitude or size of these effects can also be overstated depending on model specification used to analyze the data, the observations that are removed from the study, etc. A lot of the time what we see is someone make an incredulous finding that gets them published, and then you will find a prompt backlash from the scientific community demonstrating that the effect shown is highly dependent on the factors specified- and we return to the status quo. Unfortunately, what makes headlines on TV are these “amazing” articles that get refuted later on, but that never makes it to the media…I think that might be why the public is leery of statistics or “science”- I think it’s more of a reflection of the media’s lack of follow-up and sensationalizing particular findings over others.

ElliInJapan

I'm not sure why you think this, but you are wrong. The term "peer-reviewed" means the study was replicated and the results were repeated.

You're wrong. There are results that take many months or even years to obtain. They cannot be replicated before publication. Reviewers read the work, ask questions and clarifications but do not repeat (in general) the work. I'm actually supposed to finish a review tonight (which is why I'm procrastinating in here). I've read the paper once, I've already decided it won't be published (it's not important enough for the particular journal) and I'll now read it once again and explain my decision. That's it.

totem12

BTW : I always though it would be lawyers and politicians considered the least trustworthy.....

I thought it was politicians as well!

UsedToBeHusky

I'm not sure why you think this, but you are wrong. The term "peer-reviewed" means the study was replicated and the results were repeated.

You're wrong. There are results that take many months or even years to obtain. They cannot be replicated before publication. Reviewers read the work, ask questions and clarifications but do not repeat (in general) the work. I'm actually supposed to finish a review tonight (which is why I'm procrastinating in here). I've read the paper once, I've already decided it won't be published (it's not important enough for the particular journal) and I'll now read it once again and explain my decision. That's it.

I corrected that. I was referring to a specific type of study.

UsedToBeHusky

Of course in an ideal world they can be! I would hope that mine could be replicated by anyone. Various factors contribute to why some cannot. Lots of experiments are long and difficult, technically challenging, and getting that final figure that is good enough quality to publish can be a miraculous feat of scientific magic that you'll never manage to pull off again. Scientists under pressure to publish can rush out a paper as soon as they manage to get the results they were hoping for and later find those results never show up again. It's terrible and it's flawed and unfortunately it's still the best we have.

I see your point. I only have experience in interpretation. Not actually performing the experiment.

bonniecarbs

Listen to your own body, it's your best advicer and decide for yourself what's best for you!

I understand this.