Changing How the World Thinks

An online magazine of big ideas

more

Science needs a radical overhaul

The lure of the illusion of discovery

Simine Vazire2

Myths of rigor and illusions of discovery are holding science back. It’s time for an overhaul, writes Simine Vazire.

 

Scientists have a problem. We are discovery junkies. The addiction metaphor is overused, but, well, I can’t resist. Scientists have a drive to discover the next big thing. This drive can be channeled in positive ways, but can do serious damage to science and to society when it goes unchecked. And what’s worse, the  journals we publish our science in are the enablers that pretend they are protecting us.

The problem runs deep. Science selects for people who are naturally curious, and then it heaps rewards and incentives that amplify their drive to find new things. Unfortunately, not all findings are equally gratifying, or equally valued – learning that your potential new cure (for cancer, or racism, or inequality) doesn’t work is just as informative as finding out that it does work, but it just doesn’t feel nearly as good. What’s more, the negative answers aren’t rewarded nearly as much as the positive ones. Journals don’t want to publish the negative results. Awards are rarely given out for rigorously testing a good idea that turned out to be wrong.  A track record of negative results is not going to get you a grant. 

A system that only publishes positive results and sweeps negative results under the rug would be bad enough, but it gets worse. There’s good reason to think that in some fields, many of the positive results aren’t real discoveries – they are quite likely to be false positives. In my field of psychology, the evidence of this problem is piling up, even if many of the leaders in the field still deny it. When we try to replicate influential studies that have been considered sound enough to put in textbooks, we often find that we can’t get the original finding. When we change the research process to put safeguards in place to reduce bias (for example, by getting authors to commit to their research plan in advance, using a mechanism called “Registered Reports”), we find that about half of the discoveries disappear. Even worse, when we simply try to reproduce the original study’s results using the original study’s own data, we often can’t do it. We know false discoveries are a big problem in psychology because we’ve looked. Most fields haven’t even begun to look.

We know false discoveries are a big problem in psychology because we’ve looked. Most fields haven’t even begun to look.

This seems like something that should worry scientists – if not the scientists making the false discoveries, at least the journals and editors that are being duped and publishing them. Unfortunately, in many sciences, no one in the system is particularly motivated to find out if a discovery is real or not. We all enjoy the warm glow of what feels like a discovery, and we all lose if we dig too deep and realize the discoveries aren’t real. So it’s not exactly that journals and editors are being duped – in fact, they’re a big part of the problem.

Indeed, after 10 years as a journal editor, seeing how things work behind the scenes, I’m convinced that journals and the people who run them (editors, publishers, societies) are a bigger culprit for the spread of bad science than are individual researchers. Journals compete to be the most prestigious, but the race for prestige is not determined by who provides the best quality control.  Instead, journals compete to publish the most attention-grabbing papers – the papers that are going to get the most clicks, media attention, and citations. In other words, journals are rewarding scientists for being flashy, for producing big, bold findings, and they are looking the other way when it comes to questions about whether those findings are reliable and whether the methods were rigorous. This reality is in stark contrast to the common myth about peer review – that journal-based peer review is a quality filter, and that the most prestigious journals have the most stringent filter. But the myth persists.

This misplaced faith in prestigious journals’ peer review system is doing serious damage to science. Scientists continue to chase the reward of getting published in prestigious journals (because their livelihoods often depend on it, and because they also buy into the myth that prestigious journals publish higher quality papers), so the research practices that are rewarded, and spread, are those that produce the flashy results that prestigious journals want. Scientists who – consciously or not – use methods that get extravagant and often false results get rewarded; they get jobs and grants and train future generations of scientists. Those who produce the more humble but accurate results get weeded out.

 

Why do we all – scientists, the public, funders – continue to buy into this myth of quality control?  Why don’t we refuse to trust journals, no matter how prestigious, until they show us that they care more about publishing high quality research than flashy findings? One reason is that we are too easily enamored of discoveries. Scientists love them. Journal editors and publishers love them. Journalists reporting on science love them. The public (at least those that are interested in science) love them. We’re all susceptible to the lure of scientific discovery – or the lure of the illusion of discovery. If we uncovered and retracted most false discoveries, it would feel like science was making slower progress (even though in fact we would be making faster progress). So a big part of the problem is our humanity – it takes an almost superhuman amount of self-control to wait for the much-rarer hit of true discovery rather than the immediate gratification of illusory discovery.

But the problems run deeper than that. Even if we wanted to do the right thing and evaluate scientific papers based on their quality, regardless of how flashy and exciting the claims of discovery may be, it’s not clear how we’d do that.

First, it’s not clear how to define “quality”. For a long time, the scientific community bought into the idea that “impact” (how much a paper gets cited, or how much attention it gets) is a pretty good marker of a paper’s quality. Many still believe this, and as long as we buy into that, we won’t fix this problem. But even if we all agree that impact is not a good measure of quality, that leaves unsolved the difficult question of how to measure quality.

We’re all susceptible to the lure of scientific discovery – or the lure of the illusion of discovery.

Luckily, we don’t need to wait for a definitive answer to “what is quality?” to start improving the quality of published work. For example, requiring authors to transparently report their methods, including materials and procedures, and, to the extent possible, their data and code, would already be an important step towards enabling reviewers and readers to evaluate the quality of the research (it’s hard to judge quality when you can’t scrutinize the process, reanalyze the data, etc.).  Similarly, conducting peer review before the research is carried out (i.e., on the proposed research question, design, methods, and planned analyses, as is done in Registered Reports) helps ensure that research is judged based on its rigor, rather than based on whether the findings are exciting.

Journals can also hire researchers to scrutinize their published papers, for example, by conducting re-analyses of the original data to test the reproducibility of the results, by conducting new replication studies to see whether the original result can be repeated, or by critiquing the work. 

Reviewers can be explicitly asked to judge the quality of the methods (in psychology, this means asking about things like the quality of the measurement or operationalizations, the soundness of causal inferences, the calibration of the conclusions to the evidence, etc.), and editors can weigh these factors more heavily than the flashiness of the results in their decisions. Importantly, journals need to make their peer review process transparent – by publishing the reviews and editors’ decision letters along with published papers – so that readers can evaluate what factors are emphasized during peer review. Are reviewers and editors paying more attention to the rigor of the methods, or are they encouraging the authors to exaggerate their finding and drop inconvenient results?

Finally, journals should also take seriously concerns raised about papers they have published. Investigations into credible allegations of errors or misconduct need to be handled transparently and with integrity. Too often, researchers who identify serious problems with published papers are ignored, or worse, by the very institutions charged with correcting the record - journal editors, publishers, and universities.

These measures aren’t enough – quality is more than just transparency, replicability, reproducibility, etc. In fact, these are extremely low bars, and we need to think bigger about what it means to evaluate the quality of a research paper (something my collaborators and I are working on for our field, and that will need to be done locally in each field). There is no shortcut – evaluating quality will thinking deeply and critically about tough concepts like validity and value. That’ll be hard. But the steps I’ve described here are all very simple steps journals can take to signal a commitment to quality – they are the bare minimum we should expect from journals that ask us for their trust. The most prestigious journals have no excuse not to implement these measures – they trade on their reputation for stringent quality control, and they make heaps of money off of that reputation. It’s time for them to earn their reputation, or lose it.

 

 

Latest Releases
Join the conversation

Sign in to post comments or join now (only takes a moment). Don't have an account? Sign in with Facebook, Twitter or Google to get started:

Albina Muro 14 April 2021

Thanks for providing recent updates regarding the concern, I look forward to read more.   Medical Billing Services

Albina Muro 14 April 2021

Thanks for providing recent updates regarding the concern, I look forward to read more.   <a href="http://byrneirishclan.com/index.php?route=forum/forum/view&post_id=75718&forum_id=38">Medical Billing Services</a>

Express Minicabs 8 April 2021

Express Minicabs offer Croydon minicabs to the airport and other places in the city to make your travel experience easy and affordable. call us on 02086862777. https://www.expresscouriercars.co.uk/

mshahid seo 8 April 2021

very interesting post.this is my first time visit here.i found so mmany interesting stuff in your blog especially its discussion..thanks for the post!   teardrop chair

mshahid seo 8 April 2021

very interesting post.this is my first time visit here.i found so mmany interesting stuff in your blog especially its discussion..thanks for the post!   <a href="https://betterhomesandgardensblog.com/teardrop-hanging-chair/">teardrop chair</a>

Alisah Wilson 5 April 2021

Allassignmenthelp.co.uk is a well-known online all assignment help provider in the UK, We offer the most trusted My Assignment Help UK by our expert academic writers. We know how important academic assessments to students for developing their career opportunities, Avail affordable and plagiarism-free assignment writing by the expert team.

mshahid seo 5 April 2021

Easily, the article is actually the best topic on this registry related issue. I fit in with your conclusions and will eagerly look forward to your next updates.   australia travel tips

mshahid seo 5 April 2021

Easily, the article is actually the best topic on this registry related issue. I fit in with your conclusions and will eagerly look forward to your next updates.   <a href="https://uk.search.yahoo.com/search?p=australia+itsallbee">australia travel tips</a>

mshahid seo 1 5 April 2021

Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work.   shepherd's pie

Monika Jacob 5 April 2021

Science means proving a theory through various experiments, how can we predict whether something is good or bad until it becomes in a real? For example, who would have thought that <a href="https://www.theacademicpapers.co.uk/dissertation-proposal-writing-service.php">Dissertation Proposal Writing Services</a> would become so important after a certain time?

mshahid seo 1 4 April 2021

Certain dissertation web sites over the internet courses currently have evidently unveiled while in the web-site.   what is 10 key

mshahid seo 1 4 April 2021

Certain dissertation web sites over the internet courses currently have evidently unveiled while in the web-site.   <a href="https://itjobdubai.com/">what is 10 key</a>

Anna Marle 3 April 2021

Pay for Essay Writing That Is Guaranteed to Meet Your Quality Standards Frequently in your student’s life, you will have to pay people to write essays on a specific subject or topic. This applies to students under all levels of education – whether you are a high-school or a university attendant.

mshahid seo 2 April 2021

That's the reason it is advisable that you need to right inspection earlier than writing. You can set up a lot better present with this.Jeff Bezos eye   https://itjobdubai.com/what-is-wrong-with-jeff-bezos-eye/

mshahid seo 2 April 2021

That's the reason it is advisable that you need to right inspection earlier than writing. You can set up a lot better present with this.   <a href="https://itjobdubai.com/what-is-wrong-with-jeff-bezos-eye/">Jeff Bezos eye</a>

Insta Dissertation 1 April 2021

I never use this type of services, especially online services because of the threat that they can be late in delivery of the writing, but after experiencing the InstaDissertation, I must say that the best paper writing service has a team of highly qualified writers who write with depth and insight full of humor and vivid language. <a href="https://instadissertation.co.uk/dissertation-writing-service">UK Dissertation Writing service</a>

stella mark 12 March 2021

Can nicely write on similar topics! Welcome to here you'll find out how it should look. Landscaper Designer in Geelong

stella mark 12 March 2021

I would recommend my profile is important to me.
Budget Landscaping

James john 10 March 2021

It has placed best articles to get registration from this schedule. Every time they are listing the
https://www.hireresearchers.co.uk/dissertation-writing-service

noah michael 9 March 2021

No doubt that you have writing a very good post thanks for sharing and also if you find dissertation help online in UK so feel free and get in touch and get 30%OFF on your first order! https://ukdissertationhelp.co.uk/

noah michael 9 March 2021

No doubt that you have writing a very good post thanks for sharing and also if you find <a href="https://ukdissertationhelp.co.uk/">dissertation help online UK</a> so feel free and get in touch and get 30%OFF on your first order!

Gerald Nelson 5 March 2021

It is a highly informative blog about science I like it and also shared with my writing team of <a href="https://www.writemyessays.org.uk">write my essay cheap UK</a> for getting help.

lifetime lifetime 3 March 2021

First You got a great blog .I will be interested in more similar topics. i see you got really very useful topics, i will be always checking your blog thanks. Motorama Association

lifetime lifetime 3 March 2021

Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon. Mountain Farm Lodge

lifetime lifetime 3 March 2021

Great Article it its really informative and innovative keep us posted with new updates. its was really valuable. thanks a lot. Mountain Lion International

lifetime lifetime 3 March 2021

I’ve read some good stuff here. Definitely worth bookmarking for revisiting. I surprise how much effort you put to create such a great informative website. Motor Passenger ASV Wing

lifetime lifetime 3 March 2021

I am a new user of this site so here i saw multiple articles and posts posted by this site,I curious more interest in some of them hope you will give more information on this topics in your next articles. Mesa Arizona Homes

lifetime lifetime 3 March 2021

I wish more writers of this sort of substance would take the time you did to explore and compose so well. I am exceptionally awed with your vision and knowledge. Murphys Home Styles

lifetime lifetime 3 March 2021

I would recommend my profile is important to me, I invite you to discuss this topic... Murseum Development

lifetime lifetime 3 March 2021

I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful. My Business Resources

lifetime lifetime 3 March 2021

I read your post and I found it amazing! thank! My Computer Guide

lifetime lifetime 3 March 2021

It should be noted that whilst ordering papers for sale at paper writing service, you can get unkind attitude. In case you feel that the bureau is trying to cheat you, don't buy term paper from it. My Old Coke County Home

lifetime lifetime 3 March 2021

If you don"t mind proceed with this extraordinary work and I anticipate a greater amount of your magnificent blog entries N1RCT

lifetime lifetime 3 March 2021

Amazing knowledge and I like to share this kind of information with my friends and hope they like it they why I do Nano Business Spring

lifetime lifetime 3 March 2021

I am impressed by the information that you have on this blog. It shows how well you understand this subject. National Incinerator

lifetime lifetime 3 March 2021

Very useful post. This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. Really its great article. Keep it up. National Computer Rental

lifetime lifetime 3 March 2021

This post is very simple to read and appreciate without leaving any details out. Great work! Nautlus 571

lifetime lifetime 3 March 2021

It is imperative that we read blog post very carefully. I am already done it and find that this post is really amazing. Navarro Gear

lifetime lifetime 3 March 2021

The worst part of it was that the software only worked intermittently and the data was not accurate. You obviously canot confront anyone about what you have discovered if the information is not right. NC State Good Sam

lifetime lifetime 3 March 2021

Very useful post. This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. Really its great article. Keep it up. Near Impact

lifetime lifetime 3 March 2021

Your site is truly cool and this is an extraordinary moving article. Network Security Computer

lifetime lifetime 3 March 2021

I have read your excellent post. This is a great job. I have enjoyed reading your post first time. I want to say thanks for this post. Thank you... Neuro Terrain

lifetime lifetime 3 March 2021

rgtjk

lifetime lifetime 3 March 2021

<a href="https://www.gghrfhle.com/">rgtjk</a>

cony john 1 March 2021

will be able to absolve them. In the meantime, it's anything but a major issue if a venture did not work out; after all the purpose was virtuous https://www.hireresearchers.co.uk/dissertation-writing-service

Victor Bradford 22 February 2021

Science will make you realize how this world is evolving around us! but you can get your science assignments done by assistance of best assignment writers so always stay a head with science!

Johnny Expert 18 February 2021

I agree with your post that discoveries are good. Even though I was clueless until I read some scientific posts on https://www.essayking.co.uk/ which modified my thinking and made me believe that changes are sometimes better.

noah michael 17 February 2021

Thank you for sharing such a useful post. Very Interesting Post! I regularly follow this kind of Blog. If you need the Best Dissertation Writing Help in UK, Get in touch.
https://www.myassignmenthelpau.com/uk/dissertation-help-uk

Michael Hussey 15 February 2021

I have enjoyed the topic and the way the writer wrote all the points in this blog was amazing as I understand each and everything on this article, I am a writet to at (https://www.assignmentmaster.co.uk/dissertation-writing/) and wanted to write on different topics like this one.

Vazia 8 February 2021

'peer review' is used like a magic word to either deflect criticism from any scientific theory or to dismiss one people do not agree with. it is not a magic bullet and we need a broader conversation about this.

Mandy Angelica 4 February 2021

Science does need a radical overhaul as they are in an illusion that medical discoveries will happen and no innovation and safe-guarding the public is secondary. For example the corona virus pandemic for MBA students was the worst as they had thesis’s to write up, jobs to fulfill, Co-ops to manage. With only the required help were we able to survive and come out of it unscathed.
https://thedissertationhelp.co.uk/top-10-dissertation-topics-on-corona-virus-pandemic-for-mba-students/