Science in crisis: reproducible results trump novel nonsence

Science is based on irrefutable, reproducible results. That’s why we put so much faith in scientific evidence. We trust the results of published research have been rigorously tested and analysed, to ensure the information we are being told (and often basing many important ethical and wellbeing decisions on) is solid. Most methods used in scientific research are assumed to be repeatable, meaning the someone repeating the experiment would obtain the same results.

 

However, recently many people in the scientific community have been asking the same question: how repeatable are many trusted scientific methods?

 

This was the topic of debate (and eventually our study topic) between my peers and I on a recent field trip to the Strathbogie Ranges, over the mid semester break. Our initial plan was to repeat methods used in previous years of the subject, to allow for comparison between years and look for trends over time. Somehow, the topic of repeatability was brought up and we decided to complete a repeatability experiment, to test how reliable our method was and how valid our results might be.

 

This initiated a search for other studies that have solely tested the repeatability of commonly used scientific methods. Surprisingly, the search came up with limited results. Looking into this further, I discovered this has been a common topic in recent news.

 

Nature completed a study in 2016 to determine how many researchers in the scientific community were concerned with the repeatability of their (or others’) methods.  The results are astounding. Over 50% of researchers who took part in the survey admitted they had tried (and failed) to reproduce results from their own experiments, and over 70% had failed to reproduce the results of others. Above all other reasons, researchers that took part in the study indicated that the main cause for unreproducible results is selective result reporting. The pressure to produce unexpected and novel results pushes people to take findings out of context and leave the big picture behind. This often leads to the replicable results being left out completely

 

 

The need for sound scientific method still holds true. Major concerns have been raised, particularly in psychology and biomedical fields. As human health and wellbeing is highly important, it is constantly in the spotlight, it is unsurprising that reproducibility of results in these fields are often questioned.

But what about other fields of research? Ecology (my own chosen discipline) is often studied outdoors in highly variable environments. How do you create methods that are highly reproducible when there’s rarely one environment just like another? The large environmental scale and variability of ecology as a discipline seems to so far have saved it from the reproducibility debate. But aren’t human health concerns (and humans themselves) just as variable and large-scale?

 

Results can often be hard to reproduce in such variable field environments. Image authors own

 

So this leads me to the question: have we lost sight of what truly matters in scientific literature? Is publishing true, meaningful results still not the number one priority? Novel and unexpected may not always be the key, and the pressure to produce unique results should be reduced. Or perhaps in future more emphasis needs to be put on methodical design. Although large-scale repeatability experiments may not always be feasible, an aspect of tested reproducibility must be brought in to the scientific method, with researchers being held accountable for their work.


3 Responses to “Science in crisis: reproducible results trump novel nonsence”

  1. mbrian says:

    This is a topic that I find extremely frustrating and interesting. In my own experiment this year, an integral part of my experiment produced an extremely unpredicted result! While it was fascinating, it does affect a lot of my work! I would love for someone to come up with a solution!

  2. Murraya Lane says:

    Really interesting read! I too am in the field of ecology and I really enjoyed how you related these ideas to not only this field of study but also to ‘humans’ and the biomedical area. Having a repeatable study that gives reproducible results is so important; not just when trying to publish a paper but also in the real world! great job!!

  3. Kellen Lowrie says:

    A great thought provoking post! The repeatability issue is a big problem in the area I’m studying which involves new nanotechnology as drug carriers. Most of the researchers are focused on creating the next new, amazing drug carrier instead of making sure their results are repeatable. The “publish or perish” trend is very strong and it’s going to take a lot of communication from a large number of scientists to change how science in academia is performed.