Is it fair that negative results are not fully represented in published literature?

Standard

 “Everything we think we know may be wrong.  The correct results could be sitting in people’s file drawers because they can’t get them published”  David Lehrer.

The issue about whether negative results, otherwise known as non-significant results, should be published is a very controversial issue.  It has been suggested by The All Results Journal that approximately 60% of experiments conducted fail to produce the desired results and significant findings, however the proportion of these findings that are published is extremely low.  In some parts of the USA 95-100% of studies published are of positive results, indicating that very few negative results are published (Morgan, 2010).

Many of you may be wondering why I am implying this is bad.  We conduct experiments to try and find a difference between groups and these are the results that are being published so what’s the problem? Well, as this is a stats blog I’d better include a bit of maths in it.  If in studies we are looking for results where the alpha level is set at .05 then there is a 1 in 20 chance that a difference will occur due to chance.  So using this statistic we can assume that if 20 experiments are conducted then there is also a 1 in 20 chance that 1 of these will turn out to have significant findings due to chance.  Therefore, if a study finds positive results then it appears to be significantly valid when it stands alone, but if it was presented in context with the other 19 studies which have negative results (and therefore are unlikely to be published) then we would have a different outlook on it, as explained by Steffer (2011).  This demonstrates how the publication bias towards positive results influences perspective on positive results and creates an unbiased outlook.

Journal publishers say that both negative and positive results are equally considered when deciding what to publish, however research has shown that results take significantly longer to get published if they are negative rather than positive, if they get published at all, see Stern and Simes (1997).  The graph below, taken from their research, shows that the amount of results unpublished remains higher for negative results over increasing periods of time compared to positive results.

The awareness of the bias of publication is increasing and there are now journals which are dedicated to publishing negative results, such as the Journal of Articles in Support of the Null Hypothesis for psychology.  However, as a researcher planning a study are you likely to go and look for literature in a journal of negative results rather than a more reputable journal such as the Journal of Experimental Psychology which is more likely to publish positive results? I doubt it!

So what are the major implications of the bias of not publishing negative results?  Well for starters it can hinder scientific research.  If studies are not published then different researchers will continue to run almost identical studies and continue finding the same negative results which remain unpublished, which wastes a lot of time, effort and money.  Sometimes it can be just as important to know that one variable does not affect another, as it is to know that it does, and to show that there is obviously something wrong with the paradigm you are investigating.  Even if these results may not appear relevant to current research, future research can stem from this and may lead to other findings, whether positive or negative (Rice, 2011).  The increased publication of negative results may also take the pressure off researchers to find positive results in their data so may reduce the likelihood of exaggerated results and the manipulation of data by only reporting findings from certain participants or parts of the methodology, therefore creating a more realistic image of the findings.  But no matter what, the lack of publication of negative results creates a biased view of studies and can lead to a Type 1 error.

But let’s try and view this from another perspective, because other than positive results being far more exciting there must be other reasons why this bias exists.  Firstly, can you imagine the size of journals if they were to publish the 60% of negative results as well as the 40% of positive results they already publish (assuming all studies are reported)? It would be a nightmare for researchers, or students, to sift through all this literature to find the material that is relevant and useful for what they want.  Writing up a report to be published takes a lot of time, so would it be more productive to leave these studies the way they are and begin on a new investigation instead of wasting time writing up a failed one?  Negative results can also have psychological effects.  The reputation of a discipline can be greatly impacted on if the majority of the results published were negative.  People may no longer give money to a charity for research if it appears like it is being wasted on studies that are not finding anything significant.  This can reduce the motivation and faith in research, which can be especially important for many areas of research, such as treatments for cancer where the hope that research will find a cure is what can spur patients on.

So what can be done to reduce this issue? It has been suggested that researchers should enter their hypothesis and methodology into a database before conducting a study, and must therefore insert the data found afterwards regardless of the outcome, and even if it isn’t written up as a report (Schooler, 2011).  This method has already been used successfully in clinical and educational research fields in the US.

After the information presented above I hope as fellow students you can now appreciate how all findings are important and should have equal opportunities for publishing in order to create a more balanced and realistic view of research.  Negative results make up a large proportion of the data that is obtained however it is not adequately represented in literature so has a major impact on our perceptions, and can influence future research.

This article shows many varying views regarding whether negative results should be published or not.

Advertisements

5 responses »

  1. Pingback: Blog and comments for week 4 « psucd8

  2. Hey very good blog! It is very true that not publishing these studies is unfair. You have covered every aspect of the research and kept it very relevant. I can only expand on what you have said, its true we should push researchers to publish their work as their work could be vital to research in certain areas. But it is how we get these researchers to publish their work. It can’t be forced so it there another alternative?!

  3. Hey,
    well this really didnt leave much room for criticism or improvement haha very well explained and some very good points put across. I can only really expand on what it is youve said
    i found this view on negative results very interesting: The story of Thomas Edison’s long period of unsuccessfulness with experiments points out that though incredibly frustrating – when Edisons associate was ready to throw in the towel, Edison talked him out of it – Interview from 1921 “I cheerily assured him that we had learned something …We had learned for a certainty that the thing couldn’t be done that way, and that we would have to try some other way.”
    But if failure in research is itself not a problem, the communication of these failures is rather problematic – If the case is that we learn from negative findings, why is it that its so difficult to publish these results? as youve suggested its rather unlikely that people will purposely sieve through reports to find ones with negative results over positive ones. research of all kinds can provide us with applicaable or at the least interesting information.

    though not an academic reference, i found this blog had a good vew on negative results and asked some interesting questions: http://curt-rice.com/2011/07/21/negative-results-are-important-research-europe/

  4. Brilliant article I might add… I am not sure if you have heard about the Journal of Errology. The concept of this journal seems a tad bit better and instead of creating complete papers, they help researchers share the stumbles connected with their published paper. Since we all know that researchers are not the kind of people who just easily give up. I can be found at http://www.bioflukes.com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s