Using science to argue your case

Snark aside, I am a neuroscientist and I am genuinely curious why the general public does not trust science/peer review, and what you trust instead. It makes me sad to see that people basically think that my life's work is worthless.

I am in medical R&D and am intimately familiar with the peer review process as well, and I don't think criticism of it invalidates our work. There are legitimate concerns, especially with regard to an actual operational definition of peer review, that sometimes are glossed over:

1. Ask someone, especially a layperson, what peer review means, and you're likely to get a somewhat vague answer about other scientists looking over your paper and certifying it as "good science".
2. Try to define "peer" -- is it someone performing similar science to me? Similar field? An expert in experimental methodologies? The guy that the editor of Science played golf with that one time?
3. What constitutes a review? A quick read and nothing jumping out at me? An analysis of my experiment, data collection and analysis, conclusion? At what depth? The latter definition is not regularly employed.

People treat peer review as if there are a bunch of well respected scientists sitting around a table waiting for a paper to hit it, then pouring over it for days and finally giving it a thumbs-up or thumbs-down. This is rarely, if ever, what happens. If the general goal of peer review is to increase the quality of published work and identify any flaws in the paper...well, it might achieve those aims slightly better than no review process at all, but doing so at the risk of fostering the belief that if something is "peer reviewed", it must be good and methodologically sound. As a scientist, you are no doubt aware that there are absolute truckloads of papers published every year of dubious quality.

I think the bigger concerns, though, is how the public interprets the meaning of "peer review", as I mentioned above. The belief that "peer review" is analogous to saying "correct" is particularly damaging from the perspective of public policy. Proper interpretation of a paper, in my opinion, should view the findings narrowly. As people, and as researchers, often lose sight of this in favor of more grandiose conclusions -- and I think policy has suffered because of it.

There is a lot of crap in scientific journals too, and peer reviewed ones at that.

Quacks have their own journals with very convincing looking 'studies'.

Most studies in peer reviewed journals can't ever be repeated.

Yes. The public at large doesn't seem to grasp exactly how messed up the peer review process can be.

Yes, however at least if it's peer reviewed and confirmed independently you have a lot better idea if the information presented is valid. The larger problem is, most of the radical claims AREN'T SUPPORTED in any way shape or form by any science. They're based off blog posts that agree with what a person wants to believe, they never consider anything else could be right or if the science supports their argument... which is why you constantly see the common myths resurface here, don't eat late, gotta eat breakfast, more meals is better than fewer, "starvation mode", etc.

Totally agree!

Some areas of study have very strong scientific research, use the scientific method, actually rerun experiments to see if conclusions can be reproduced.... Diet has not really been like that. It's been what 50 years of 1. throw out butter and eat margarine and trans fats instead - followed by 'whoops' 2. Eggs are bad - followed by eggs are good, etc...

People do not trust diet science because we have gotten years of bad science or bad conclusion pushed on the public that did not work. That doesn't mean all science is bad.

One of the things that should be considered -- and I really glossed over it, is there should be lines drawn between conclusions that are drawn on the data, and the policy conclusions that are built onto those. There can be errors at any point in the equation. For a very simple example:

Let's say that I think egg intake is the cause of a disease state, CVD.
I design a cohort study that follows along a group of similar people for a period of time, and collect data on their eating habits and health outcomes.
I analyze this data, controlling for certain factors, and it turns out that people who ate more than 5 eggs per day had a 10% increased risk of developing CVD.
As a conclusion, I recommend that, in order to protect against CVD, people should lower egg intake below five eggs per day.

Seems reasonable, right? I think so. The question, though -- is what does this actually mean?

If you're a newspaper, it likely means "EGGS: THE SILENT KILLER".
If you're a normal person, it might mean, "Eggs can cause CVD!"

In my opinion, how you should approach this study is as follows:
1. Was the study done appropriate for the hypothesis that I was testing? Any inherent biases? Sample size issue? Randomization issues? Any other issues in the study design methodologies?
2. Are my data analysis methodologies sound and in keeping with standard acceptable practices?
3. What gaps exist in the conclusion?

Your takeaway from the study should be, "huh, we should do some more science to start looking into what exactly the mechanisms are that might cause eggs to correlate to a higher incidence of CVD, because this correlation bears some more studying."

What the public policy people do is have a congressional hearing in which they recommend that the average American eat 3 eggs a day or less -- that is, unless the egg lobby has more money than whatever lobby is pushing them to issue the recommendation in the first place. Then people come on MFP, make a post about how "eggs will kill you," link the study, and spike the ball with "science!"

I'd like to point out something that is definitely true for my area of expertise (physics) but perhaps in bio-medical sciences that's different, so please correct me if that's not the case. The frontline of research deals with things that are not well understood by scientists and there are conflicting theories and experiments and an ongoing scientific discussion about what's going on. These papers, even though they are important and sometimes ground breaking for the scientific world, are basically useless for general public. Well, the public has no interest for physics research, but high-tech industry is interested for sure. For example, a large part of today's research (in solid state physics) is related to / motivated by the physical realization of quantum computers. There's a huge number of papers published everyday, but there is really no point for people in industry following all these. Once things get clarified a bit, once there are some promising candidates then, yes, for sure. But until then the discussion stays within the scientific circles and for good reason: things are so unclear, there's no practical use of the work done up to now. I'd expect the same would be more or less true for other disciplines. The frontline of research is interesting because of the progress being made, but for practical uses, one needs to go one step back to knowledge that is well founded. If there is an ongoing discussion about an issue, you can't really pick one paper and draw conclusions out of it. You need to understand and follow the work done by the whole community and even then, that doesn't mean you can reach any conclusion.

(Edited for typos)

I'd like to point out something that is definitely true for my area of expertise (physics) but perhaps in bio-medical sciences that's different, so please correct me if that's not the case. The frontline of research deals with things that are not well understood by scientists and there are conflicting theories and experiments and an ongoing scientific discussion about what's going on. These papers, even though they are important and sometimes ground breaking for the scientific world, are basically useless for general public. Well, the public has no interest for physics research, but high-tech industry is interested for sure. For example, a large part of today's research (in solid state physics) is related to / motivated by the physical realization of quantum computers. There's a huge number of papers published everyday, but there is really no point for people in industry following all these. Once things get clarified a bit, once there are some promising candidates then, yes, for sure. But until then the discussion stays within the scientific circles and for good reason: things are so unclear, there's no practical use of the work done up to now. I'd expect the same would be more or less true for other disciplines. The frontline of research is interesting because of the progress being made, but for practical uses, one needs to go one step back to knowledge that is well founded. If there is an ongoing discussion about an issue, you can't really pick one paper and make conclusions out of it. You need to understand and follow the work done by the whole community and even then, that doesn't mean you can reach any conclusion.

I agree with this completely. I think one benefit of working in hard science research is that there's a certain amount of technical abstraction which would exclude the majority of the public from being able to properly digest the information in the first place. When one works in a "soft-science" realm, such as much of nutritional research and public policy research, that divide is not as great. It reads in a way that is approachable for a layperson, which can be problematic.

Specifically in regards to the state of nutritional research, I think the science has been generally poor over the last 50 or 60 years, so it's difficult to even step-back in that field.

Someone always worth a read is Dr Ben Goldacre, very interesting mid-point between Doctor and Journalist. He had a lot of views on certain things but he does show his workings at least.

I think (as someone who comes from a science background and does get involved in publishing papers), part of the problem is a attitudes (in my experience) at schools. when you're a kid you get told science is 'hard' maths is 'hard'. So if you can't immediately grasp some fairly out there concepts you get streamed to lower tiers do the basics then spend your adult life believing that science and maths are hard and not important. Then any sort of conflict in the scientific community justifies 'science is hard, and not important as no-one knows anyway'. Then triple that if you're a girl. One of my relatives told me I shouldn't do a science as I'll never find myself a 'proper' husband like that.

But, I know the plural of anecdote is not data so it's just an opinion :happy:

Listen to your own body, it's your best advicer and decide for yourself what's best for you!

This ^^

There are very few nutrition subjects for which there are not conflicting studies making it very easy to pick only those that back your case, so posting a few perfectly legit peer reviewed studies will prove nothing, other than you know how to search the internet. Also, there are few nutrition studies that yield the same results for 100% of the participants. The only way to know whether you would have been in the minority that did not meet the general results, is to get to know your own body.

And last, but certainly not least, is the fact that while physiologically something may be 100% backed up by science, doesn't make putting it into practice any easier. Weight loss is very often more about psychology than physiology.

Please don't bite my head off...but here I go...

There are a few things get under my skin a bit when it comes to discussions/arguments like this:

1. There is more than likely a lot of unpublished or non-peer reviewed scientific articles out there in all disciplines that have very relevant information, because the findings in these studies go against the mainstream thought on that subject, or the results are not what the company contracting the study wants to see. I think this is the biggest issue with the GMO debate - the only peer-reviewed information says they're safe, and other studies that say otherwise are not published through reliable sources. Just because something is published through a reliable source doesn't necessarily mean it's 100% true, and just because something isn't published through a reliable source doesn't necessarily mean it's 100% false.

2. Scientists have the ability to alter time-lines and control groups in order to achieve certain results, or to make a result appear to be positive when it's not. I'm not saying all scientists do this, and I'm assuming that most don't, but the possibility is still there.

3. Conflict of interest. Again back to the GMO debate - if a company makes a new product and the same company is the one to test it to see if it's safe, is that not a terrible conflict of interest, and thus not entirely reliable? Third parties/independent studies I feel would be more reliable, but as mentioned before, if the results do not match the agenda, they can and most likely will be discredited.

4. Science can't prove everything, or at least not yet. Why do holistic approaches to medicine still work when there is no evidence that they do? Why do some children with autism that have changed to an organic diet have less problems with their autism? Maybe I just haven't done enough of my own research on stuff like that, but there is a lot of weird stuff in our world that science has not been able to prove and I don't know if it ever will.

I do agree that hard facts are important, but I think a lot more people are becoming increasingly wary of the "hard facts" that are presented because we are aware that we have been lied to in the past on what to believe. People want to be more aware, and people are aware that things are being hidden, and that's why they don't always trust science.

I'll reference a study when it becomes clear that I'm talking to someone who is actually open to the idea that they might be wrong. I refuse to get into a pissing contest with someone who thinks that because they read something in Shape magazine, it must be true.

Most studies in peer reviewed journals can't ever be repeated.

You must not be familiar with how peer reviewed studies work.