Changing How the World Thinks

An online magazine of big ideas

more

Under the Microscope

One of the cornerstones of science is under fire. A leading cosmologist explains why.

peer review

Heresy should be encouraged because that’s how breakthroughs happen.”

We often like to think we live in an age of reason. The fruits of modern science and technology are all around us: from smartphones and 3D printers to cures for some of history's most devastating diseases. We now have the highest life-expectancy our species has ever experience. Despite all this, a backlash seems to be emerging against certain aspects of science. Lately the revered process of peer review has been coming under fire – especially in the field of medical science. A recent piece in The Economist even argued that the internet means anyone can now appoint themselves a peer, thereby providing a challenge to the perceived power held by scientists.

In this forthright interview, cosmologist Carlos Frenk examines in detail one of the cornerstones of contemporary scientific process. He argues that peer review actually encourages unorthodox thinking, but warns, controversially perhaps, that science is a spectrum, and some disciplines may not be as rigorous as others.

Frenk is Director of the Institute for Computational Cosmology at Durham University, where he builds models in state-of-the-art supercomputers in an attempt to understand the evolution of the structures of our universe.

 

Do you think an inevitable part of the process of peer review is that it somehow prevents useful, fertile, if not correct ideas from being explored?

Let me first qualify my answer to this and to all of these questions – I speak from personal experience as a practitioner of a very particular branch of science which is astrophysics. I can probably speak for most of physics, but my experience is very much coloured by that.

Given my experience in physics, I'd say the idea that peer review tends to curtail heretical views is incorrect. In fact many, many heretical papers are continuously being published in the most reputable journals of physics precisely because the community recognises that heretical views that challenge the status quo are vital for the progress of science. Of course the really way-out crackpot ideas that are demonstrably incorrect, they get filtered out very quickly. If somebody wants to challenge the perceived, even the sacred code of physics, with arguments that are sound, they could be either theoretical or experimental.

The experimental case is quite simple: if the experiment is conducted correctly, and all the analyses of the data are correct, then I think the chances are the paper will be published. There are many examples, but one which was particularly spectacular was the discovery of the expansion of the universe, which was completely disruptive, even revolutionary. It was recognised that the set parameters of the universe were not what they had been held to be. It was a crisis, but the evidence was there and we had to accept it.

Physics is littered with completely novel ideas that challenge the orthodoxy, and they get published. Not only do they get published, but, for example, a paper of which I was an author was published in Nature and the subtitle of the paper was: “A Challenge to Cosmological Orthodoxy”. Nature were perfectly happy to use the phrase “challenge to orthodoxy” and the paper really was quite unorthodox.

I could go on and on. Most physicists think that Newton’s theory of gravity and Einstein’s of relativity are the correct description of the universe, but there are hundreds of papers challenging that view, seeking to debunk even sacred ideas like relativity. There's a whole branch of physics, an alternative theory to gravity called MOND (Modified Newtonian Dynamics), and these people publish papers.

By and large most people really want to challenge, because then you can become famous. If anything, I think there’s too much of that, because there’s a lot of rubbish being published by people not motivated to seek the truth, only to challenge the orthodoxy. There are a lot of pretty crummy papers which are actually wrong and get published anyway. Recently, I was corresponding with someone about the speed of light – one of the absolute blocks of modern physics is that the speed of light is a constant. Well, people come up with ideas that maybe it's not constant and they get published and you read them and you make your own judgement.

So I disagree that in my own field peer review vetoes novelty. Whether that’s because my field is by nature adventurous intellectually, so it can foster this idea of rebellion, or whether it's a intrinsic to science I cannot tell. My own personal view is that heresy should be encouraged because that’s how breakthroughs happen.

Isn't it a fine line between heretical ideas that are later proved to be true, and those that turn out to be nonsense?

It's not a fine line, it’s a very thick line actually, dividing heretical views that are well sounded and well reasoned from crackpot ideas. Clearly, we cannot really have a general school of crackpot ideas which are absurd from the beginning.

How do you judge that?

The principles with which you judge whether a piece of work is suitable for publication or not are not really based on scientific orthodoxy; the judgement is based on logic: whether or not the assumptions are based on something inspired by empirical observations and data, or by previous theoretical knowledge.

For example, if the mathematics is correct and if the experiment has been conducted following the canon for experimental science, which is very well laid out. Then, you look at whether the data analysis follows the rules of statistics. I think these are very general precepts that will be used to determine whether a paper should be published or not. The judgement does not really rely on accepting the orthodoxy; it relies on accepting basic scientific processes such as logic, mathematical inferences, deductions, experimental integrity and sound statistical analysis. So I would say that the establishment would not consciously stifle unorthodoxy at all. On the contrary.

A recent article in the Economist has argued that the internet means anyone can appoint themselves a peer and criticise work that appears in the public domain. Do you think this is true, and is it a good thing?

No, I think it’s a stupid thing. Science, whether we like it or not, is especially disciplined; it requires training which is very protracted and it takes a long time to be called a physicist, or a biologist, or a doctor or a chemist. You need skills to be able to judge whether a piece of scientific literature is valid or not, you need specialist knowledge, you have to have expertise in the subject. Unless it's something so banal that every educated person can judge whether it's wrong, which happens only a tiny amount of the time, the majority of articles that are published are specialist articles and they can only be understood by specialists. It’s as simple as that.

You don’t become a physicist overnight; it's not subjective. I can go to a museum of modern art and I can like or not like a particular painting and you can't accuse me of anything because it’s a matter of taste, it’s subjective. Whether you like something or not is an emotional, subjective judgement. Physics is not about that. Here there are techniques and a capability with mathematics, and a body of knowledge that builds up, and if you don’t have that body of knowledge, you’ll never be able to make a judgement as a peer; that’s why it's called “peer review”, because its done by peers. A review is one thing, a peer review is something else. If you want to avoid an explosion of crackpot ideas and publications you have to have some filter, otherwise science would become anarchy, so peer review has to occur by peers.

Having said that, the way we work at the moment has its flaws. It's like a democracy: a democracy has flaws but the alternative is worse. Peer review is like democracy – it's imperfect, it’s flawed in many ways but no alternative that anybody has come up with is better. So at the moment, a paper only gets published by a journal if it’s approved, typically, by one or two reviewers appointed by the editor. However that’s just the entry ticket. When your piece enters the public domain it can be criticised, and normally is criticised, by a whole bunch of people. Either people who read the paper and write to you, complaining about something you did wrong, or subsequently if your scientific result is important people will build on it, and people will find the flaws and so on.

So that kind of continual peer review is happening all the time informally. One common attribute of science is that we’re really nasty to each other, because we’re really critical. That is the cornerstone of scientific progress, that people are sceptical, and whenever we publish something that’s not routine you can be assured you’ll be eaten alive. And if you survive this gruelling process of criticism then the idea may start to find a base. Peer review is the apex of science, and it happens all the time informally. However, we do need something to filter what actually gets published.

Ultimately, I think the system of peer review, imperfect as it is, acts as a filter, a point of entry, which then gets opened to everyone to criticise through the literature. If we changed it, it would have to be for a good reason and certainly in my discipline I don’t see why you’d carry out a wholesale change of the way we do things. We could improve it, but we don’t need a revolution, though in other areas it might be different.

Which areas would you say could do with improvement?

Science is a spectrum and there are different levels of rigour in different scientific disciplines. Traditional classical natural sciences like physics and chemistry have very high standards of rigour and a tradition of very critical and pointed peer review. Biology is similar, as is genetics, but as you expand the remit of what you call 'science', then you come into some areas which don't really have the same levels of rigour, by nature of the subject.

Research happens in clinical or medical science, clinical research which is clearly very complicated, but which doesn't have the same level of rigour as, for example, physics. This research is essentially clinical and a lot of it relies on the quality of the experiments and statistical analysis, and the Economist and Atlantic article have taken to criticising – sometimes correctly, in my opinion – practices in these kind of fringe areas of science. But what the articles fail to recognise is that these are fringe areas of science, and make sweeping comments that physicists get really baffled by because they don’t recognise anything the criticism is talking about.

So we have to be careful about what kind of science we’re talking about. And I don’t want to be offensive or condescending to those who do clinical research, because what they do is intensely valuable and, if I’m going to have an operation, I'd much rather have the surgeon rely on clinical research than on quantum physics. Given that these are fringe areas of science, perhaps their peer review needs to have a different approach.

As the horizon of science expands, naturally things will come into the remit of science that before would not. And in those areas that are only now being established as science, perhaps the procedures in those areas need to be different from those at the core of science. The important point is that science is forever expanding: things that were magic in the past or philosophical can come back to science and when they become part of science, in the accepted body of science, then they have to subscribe to certain rules that have been developed over the centuries for judging the quality. But when they are in this ‘twilight zone’ of zone then things become a bit more complicated.

I think we’ve dealt with ideas that challenge the core, so what I'm talking about now is how the process of peer review has to be responsive to the fact some disciplines are very young. That’s not to say they stay very long in the twilight, but we’ve seen in the last twenty to thirty years completely new areas – neuroscience is probably the most famous one and my own subject, cosmology, was not really an accepted part of anything when I started my PhD. It lived in the fringes for a very short time and then it became, because of advances, part of the core. And so the same is true of neuroscience and nanotechnology, particularly in novel technological areas; these areas come in, but they have to go through a process of maturing.

So, are articles like the ones in The Economist and an older one from The Atlantic only applicable to certain areas of science?

Unfortunately I haven’t yet read the article in The Atlantic, but you can see in the title that it's about medical science. The medical sciences have come under a lot of criticism and a lot of it is understandable, given its weaknesses. But most of the criticism is because of a poor understanding of statistics, which I think is an endemic problem.

Statistics gets a very bad name, but only from people who don’t understand the subject. Statistics is rigorous: it’s a branch of mathematics and when done properly there’s no such thing as “lies, damn lies and statistics”, there’s only “lies, damn lies and stupidity”. If you understand statistics, it's a beautiful tool that has immense power to derive knowledge from what we call noisy data, and all data is noisy, meaning that there are fluctuations whose source is too complicated to understand. But what is remarkable is that, even when you have this noise you can still learn about the underlying processes and principles, and that is thanks to statistics. So I get really irritated when people use the lies phrase because all they are showing is ignorance of one of the pinnacles of the human intellect, which is the mathematical area of statistics.

When applied properly and understood properly, statistics is hugely powerful. Trouble is, in the medical sciences, the training that people in these disciplines have is of a different nature. They cure sick people and to fully understand statistics you need a very profound and solid grounding in mathematics, and that is something that takes skill and time to acquire and not everyone who is good at medicine has those skills. So what happens is that people who don’t really care about mathematical backgrounds therefore rely on kind of statistical packages and don’t fully understand them, or perhaps even care to, because they're interested in curing diseases for example.

So here we do run into a problem. What articles like The Economist fail to understand is that not all science is clinical science. It's just one tiny fraction of science, representing a tiny fraction of the scientific advances we have made, and they pick on it and generalise.

Do you think these articles are a wider reaction against science, or do you think its just a case of misinformation or a lack of understanding?

I think this is very complex question. There is no doubt that, paradoxically perhaps, in times where science has delivered some of the most spectacular results ever, there is nevertheless a reaction against science and a revival of superstition. This is really surprising because science has never had it so good and society has never benefited so much as its done in the last few decades from scientific development.

Everywhere you look, diseases that were incurable when I was a kid are now eradicated. That’s why people live longer, not because of any magical potions, but because of scientific advances, even in these areas that we've been criticising. Look around you. The world is so different from the one I grew up in. Take mobile phones. They weren't invented by businessmen or politicians, they were the result of scientific developments.

There still needs to be work done on the green revolution and people are still hungry in the world, but the progress is still spectacular. And yet there’s a kind of reaction against science, an arrival of superstition and a rejection of science. I can understand why, given that technological disasters like Chernobyl or Mad Cow Disease are very widely publicised. But these disasters are somehow contextualised as a reaction against science even with the huge amount of benefit which has been brought to everyday life.

Sometimes it's justified when medical research doesn’t really follow the correct scientific procedures, but on the other hand medical science has made enormous strides. I think it’s the media that’s partly to blame. They have a life of their own, they need to make a splash like scientists. I love the media, as it's key to democratic society to have debate and criticism and so on. I don’t blame the media, but they should apply the same rigour to their articles that is lacking in the scientists they criticise. Medical science is an easy target, and they take things out of context.

So I think it’s a very thorny question with many dimensions to it. The reaction against science is understandable and you can see where it’s coming from. It’s not stupid, it's not always unjustified; I just think its been taken out of context. But what I find most alarming is the rise of superstition, which is really puzzling. You think we live in a rational era and yet there’s much going in our society which is so irrational. For me, rationality is something that defines humans, but somehow, sometimes, it's left to one side.

Join the conversation

Sign in to post comments or join now (only takes a moment). Don't have an account? Sign in with Facebook, Twitter or Google to get started:

iai donation
iai donation
iai donation