Peer Review Process Doesn't Work!

kimny72

I think it's important to keep in mind that this article is an opinion piece. The original article was written by William A. Williamson, who as per his byline is a "Software Engineer".

And the piece was written on the "First Things" website. Clicking on their "About" page, they say it is published by The Institute on Religion and Public Life, which was founded to "confront the ideology of secularism".

So there's that.

Not to say he does or doesn't have a point, but it is always important to consider the source, whether the source is older established scientists invested in defending the status quo, or software engineers writing articles for a religious think tank.

GiddyupTim

I don't get it.
So.....peer review is not perfect. Yeah? Where's the news?
Of course, it is not perfect. People are not perfect. Studies are not perfect. That's why we have peer review. To increase the chances that a study is good and important. It doesn't say it is perfect. it doesn't say everyone will agree with the findings or conclusions. It just says that at least three experts in the same field (the number usually used) believe that the study published is meaningful and worth reading and considering.

FunkyTobias

GaleHawkins wrote: »

stevencloser wrote: »

Gale, tell me how the part where it says "If peer review is good at anything, it appears to be keeping unpopular ideas from being published." fits in with when you said that unreproducible studies are more often cited? More often than not, the studies everyone is talking about (and thus citing) are the ones that go against established knowledge (which happen to be often not reproducible and worthless).

Steve I did not say anything about "unreproducible studies are more often sited" that I can find. You will have to talk with the person that made that statement.

Sigh

It was in your original post

GaleHawkins wrote: »

The “bad” papers that failed to replicate were, on average, cited far more often than the papers that did!

If you can't be bothered to vet your sources, can you please actually read them before pasting them?

GaleHawkins

FunkyTobias wrote: »

GaleHawkins wrote: »

stevencloser wrote: »

Gale, tell me how the part where it says "If peer review is good at anything, it appears to be keeping unpopular ideas from being published." fits in with when you said that unreproducible studies are more often cited? More often than not, the studies everyone is talking about (and thus citing) are the ones that go against established knowledge (which happen to be often not reproducible and worthless).

Steve I did not say anything about "unreproducible studies are more often sited" that I can find. You will have to talk with the person that made that statement.

Sigh

It was in your original post

GaleHawkins wrote: »

The “bad” papers that failed to replicate were, on average, cited far more often than the papers that did!

If you can't be bothered to vet your sources, can you please actually read them before pasting them?

If you will read the OP linked article you will see your mistake. There was nothing to vet. If you have any sources that supports the author's position or distract from the author's position feel free to add to the subject of the debate.

JeromeBarry1

I began to doubt the peer-review process when I learned that peer reviewers are mostly college professors with full teaching, researching, and grant-writing plates who don't have the apparatus or inclination to check the data, methods, or conclusions of the original writers.

forwardmoving

tufel wrote: »

I don't get it.
So.....peer review is not perfect. Yeah? Where's the news?
Of course, it is not perfect. People are not perfect. Studies are not perfect. That's why we have peer review. To increase the chances that a study is good and important. It doesn't say it is perfect. it doesn't say everyone will agree with the findings or conclusions. It just says that at least three experts in the same field (the number usually used) believe that the study published is meaningful and worth reading and considering.

I find it fascinating how quick some people are to dismiss the peer review process yet rely on anecdotal stories and bloggers for medical advice.

FunkyTobias

stevencloser wrote: »

Gale, tell me how the part where it says "If peer review is good at anything, it appears to be keeping unpopular ideas from being published." fits in with when you said that unreproducible studies are more often cited? More often than not, the studies everyone is talking about (and thus citing) are the ones that go against established knowledge (which happen to be often not reproducible and worthless).

Still waiting for @GaleHawkins to answer this

_John_

JeromeBarry1 wrote: »

I began to doubt the peer-review process when I learned that peer reviewers are mostly college professors with full teaching, researching, and grant-writing plates who don't have the apparatus or inclination to check the data, methods, or conclusions of the original writers.

well, if it makes you feel any better, the post-docs and grad students in their lab may have done it since the professor is so busy...

stealthq

JeromeBarry1 wrote: »

I began to doubt the peer-review process when I learned that peer reviewers are mostly college professors with full teaching, researching, and grant-writing plates who don't have the apparatus or inclination to check the data, methods, or conclusions of the original writers.

How exactly would anyone be expected to check any of those things (for clinical trials in particular), regardless of the apparatus they have on hand or their inclination?

Reviewers can't be granted access to the full clinical data (HIPAA violation), so if anything in the paper required any of the data that can't be released, no verification can be done.

Neither the journal nor the reviewers see the full experimental data unless it is required to be publicly released. Even then, only data referenced in the paper must be released. Have data that counters the paper's premise? Don't have to share that. Omics data generally is released. The rest is not. Heck, for radioactive gels and the like you don't even see the original in the paper, but a cropped and re-assembled picture/scan.

Without access to the original biological samples, how would the reviewer even know the data was properly produced? Even WITH those samples, a fair amount of the data currently used in these papers is not reproducible in the way you may be thinking. RNASeq, for example, is a sampling technology. I can run it and get one sampling. You run with the same samples, same machine, same technician, same settings and get a different sampling. The samplings have a good probability of being similar, but there's no guarantee. They can be wildly different.

There just is no way. It is a shame, but this is not physics or computer science where you can provide the info and if you are accurate and thorough it's guaranteed that the next person is able to reproduce what you've done.

It is true that most reviewers are college professors. They also submit a good portion of the research, so ...

Those of us who have non-academic jobs find it hard to find the time to review in addition to our regular jobs where our bosses don't give two hoots if we review papers, whereas being a reviewer is often considered a positive thing in academia.

Personally, I find it more annoying that papers that are part of new fields of research often get reviewed by people who have no basis in that field. That's also not fixable - no qualified reviewers available, then you have to make do. But it is annoying to have one's paper stalled because of a couple of questions made in ignorance.