A few years ago, Dr Craig Bennett and his colleagues carried out an experiment. They put a salmon in a working magnetic resonance imaging (fMRI) scanner, which monitors brain function. We should point out that the salmon was dead, bought at a fish market.
Once the fish was lying in the machine, the scientists showed it “emotionally ambiguous” photos (of people, not fish), and asked it to imagine itself in that situation. Later, in accordance with the rules of their art, they made calculations, and… on the image of the (dead) salmon’s brain, an area showed up in red as having been activated.
For this unusual experiment, the scientists received an Ig Nobel – a prize awarded for research that “first makes you laugh, and then makes you think”. Why? Because the whole point of the exercise was to demonstrate that these “rules of the art” – the statistical methods used to analyse magnetic resonance data – are flawed.
Unfortunately, this isn’t the only example of how scientists have problems with statistics. In another work, also from the field of neurobiology, three researchers examined more than 500 scientific articles from top journals. They were looking for a certain specific error; a particular way of confusing fundamental statistical concepts. It turned out that of 157 works where it was possible to make this error, it was in fact made in 79, which sometimes led to incorrect conclusions for the entire research.
As we can see, scientists (even the best ones) don’t know everything, and surprisingly often they don’t know statistics. And statistics are an important element of almost every scientific work, whether we like it or not.
Ignorance of statistics can even lead to unwitting fraud. As the Polish writer Stanisław Lem said, if you torture a gooseberry, you can extract a lot from it – the same is true of experimental data. More than one scientist has given in to the temptation to torture their data until they extract something. This wouldn’t be such a big problem, if attempts were made to get independent confirmation of the results in further experiments, but unfortunately it doesn’t always happen that way.
In 2015, scientists working on amyotrophic lateral sclerosis (ALS, also known as Lou Gehrig’s disease) wanted to explain why new medications were ineffective in clinical tests (meaning those performed on humans), while earlier published research on mice gave promising results. They started by repeating all the experiments on rodents, precisely as they had been described in the original publications, and it turned out that in these repeated experiments, none of the medications worked.
The problem of lack of replicability in scientific research came to be known as the replication crisis. It sounds a bit like something out of Lem, but it’s really just that many results can’t be replicated – not only in medical research, but also in psychology, systems biology, and many other fields. This is quite significant. For example, huge funds are designated for developing new drugs, but if even an experiment on mice can’t be replicated, how can we expect them to work on humans?
Ignorance of statistics is the first cause of the replication crisis. The second problem is more difficult, as it’s systemic: publication bias. It’s easier to publish something interesting and positive (‘Hake tea cures cancer!’) than something boring (‘Hake tea, like millions of other substances, doesn’t cure anything, it just smells bad’). Negative experimental results rarely make it into scientific journals.
This problem also applies to clinical research, and this is a very serious issue. More than half of clinical tests will never be published, and unpublished negative results can kill. In 1980, a small study was carried out on a potential arrhythmia medication. Nine members of the treatment group died. The scientists attempted several times to publish their results, but when their article was constantly rejected, they finally gave up. That’s too bad, because it could have saved the lives of many thousands of people.
To make matters worse, research that repeats earlier experiments isn’t easy to publish, as it’s not attractive. In addition, it’s most difficult to publish research that didn’t manage to repeat a sensational finding.
And here we arrive at a third problem: money. Scientists work mostly for money. To conduct research, to pay employees and students, even to publish their discoveries. But to get it, it’s best to do something spectacular; to impress not only specialists, but also laypeople, who often are the ones who control the money. That’s why there’s strong pressure to inflate the significance of your research; to colour it, present it beautifully, promise the media new drugs, vaccines, breakthrough discoveries; to overturn old theories. And all of this in good faith, because what scientist doesn’t believe in their work?
We’ve developed a culture of exceptionally aggressive competition. It’s precisely this culture that is largely responsible for the current crisis. Many Nobel laureates, such as Sydney Brenner, say they wouldn’t have been able to make their discoveries if they’d had to operate in the current scientific culture.
Considering all this, does faith in science still make sense?
Science has one important attribute that distinguishes it from other areas of our lives: it requires meticulous and publicly available documentation of each study, each experiment, each clinical test. In scientific work, it’s not enough to write that a dead salmon that’s shown a picture of a crying person displays mental activity. You have to describe the details precisely – how the experiment was done, how the data were processed, using what programme, within which parameters, what exactly the results were. Thus if somebody doesn’t believe in mental activity in dead fish, they can repeat the experiment.
It’s because of the scientific method that replication is possible at all, and it’s also the reason why the replication crisis was even noticed. Scientists themselves raised the alarm on the issue and proposed a series of solutions.
First of all, money must be found for scientists who want to repeat published experiments. Second, scientists must take a more serious approach to statistical calculations.
The most difficult thing will be to deal with the culture of competition that promotes sensational reports. For example, recently it made headlines that psychopaths prefer certain pieces of music (apparently what they like best is Blackstreet’s “No Diggity”). That’s how it was presented on many Polish news portals; some suggested that in general, listening to rap and heavy metal indicates psychopathic tendencies. Yet if the author of a press report is honest, they will give the source of the information and the name of the researcher. And in checking the sources, we learn that yes, there was such a study – it was conducted by Pascal Wallisch of New York University – but no diagnosed psychopaths took part. What’s more, the research still hasn’t been published anywhere, so it’s hard to say how it was conducted and to what degree we can trust it.
In fact, the author of the study himself warns against hasty interpretations of the results, stressing that he’s not 100% certain of them, and that before publication in a scientific journal, they will be repeated on a larger, more representative group.
So what can we do – we, the readers of popular scientific articles, constantly bombarded by reports of a new drug or about how drinking tea (e.g. hake tea) causes (or prevents) cancer (or high blood pressure)? How we can inoculate ourselves against scientific sensationalism is a subject for another, longer article. For now, let’s recall that in science every statement has a publicly available source, which you can check. In science nothing is taken on faith, so first of all, let’s be sceptical toward articles that don’t contain references. Because it may turn out that the source is a dead salmon in a fMRI machine.
Translated from the Polish by Nathaniel Espino
Read similar:
