The Final comments of year 2 =]
The types of issues that a pilot study checks for include:
- Whether instructions are understandable for the participants
- Whether the researchers can conduct the experiment adequately to ensure there is no experimenter bias
- Whether the equipment operates correctly and is suitable for measuring the variables you intend to measure
- Whether the task is suitable for the cognitive or physical abilities of the participants
- Whether the levels of the intervention are appropriate
- Whether there are any adverse effects on the participants produced by the study
A discipline in which pilot studies are very effective and are highly recommended is in clinical psychology. These pilots are often known as clinical trials, or proof of concept studies (Thabane et al., 2010). These types of trials are used to assess the safety and implications of treatments and interventions, before reasoning as to whether to go ahead with the main study. Pilot studies are used to assess the power analysis of a study, which to us means the probability that a Type 1 error will occur (falsely reject H0), and can influence a researcher’s decision as to whether to continue with the original study or not (Halpern, Karlawish, & Berlin, 2002). Some researchers suggest that there is too much emphasis on the power analysis of a pilot study and that many studies have been aborted because of this when they could have actually found effects in the main study (Kraemer et al., 2006).
The use of a pilot study is very advantageous to researchers as it enhances the validity and reliability of their study. It provides them with an opportunity to use various methods of implementation in order to identify which is the most effective to use in the study (Woken, CTL). It also allows them to identify any problems within the experiment, such as the issues mentioned above, and to rectify these before the main study is conducted, which can save lots of time and money as it increases the likelihood that the study will be successful.
However, researchers often put too much reliance on the results of a pilot study, even though it is only based on a small sample of the population. The exact sample size needed for a pilot study is debatable depending on the purpose of the study (Johanson & Brooks, 2010), however they should fit the same criteria as the participants in the main study. A small sample size can result in data being unrepresentative of the target population and inaccurate predictions and assumptions being made based on this data. Pilot studies can also result in contamination of the main study if the results from this are included within the main study, or if the same participants are included, as they may have practice effects and may not show the same pattern of results as other participants (van Teijlingen & Hundley, 2001). Halpern et al. (2002) also suggested that there may be ethical issues involved in pilot studies as many people participate in research for altruistic reasons and because they believe that it is a way to help other people, however in pilot studies there is very little public benefit of the research, especially if the main study is then aborted.
Although pilot studies have been shown to be an effective tool to increase the reliability and validity of a study they are very rarely published in journals. This may be partly due to journals being biased towards publishing positive results, as discussed in a previous blog post, therefore pilot studies where methodologies may be inaccurate and where results are not significant do not look as appealing to journals to publish, regardless of how beneficial they may be.
So although care must be taken when using the results of pilot studies to generate hypotheses and determine whether to continue with a study, they are a very useful tool to evaluate the methodologies of an experiment in order to improve it and increase the likelihood that the results will be reliable and valid.
This chapter gives a broader view of why pilot studies should be used and what their values are. It also provides an example of a pilot study, and how they should be implemented.
On a final note, it’s nice to finally say goodbye to blogs, and goodluck to everyone with the remainder of years 2 and 3! =]
This blog has stemmed from an article I came across online from ‘The psychologist’ which critically analyses the film ‘The Woman In Black’. For anyone who hasn’t seen this film the basic plot can be found here. Green (2012) has analysed the behaviour of Arthur Kipps and tried relating this to the context of the film and Kipp’s situation, and has come to the conclusion that he has a case of psychosis. However, is this really a correct diagnosis? It could be suggested that the figure Kipps sees is just a delusion, one of the criteria set by the DSM to diagnose psychosis (p. 332), however as Kipps sees the woman before he hears the horror stories from the villagers is it more likely to actually be reality? When this is combined with the deaths of the children in the village, and previous encounters by others with the figure then how can it all be in his head? To me there appears to be too many linked variables for it just to be coincidence and for him to be diagnosed with psychosis. Also, does this superstition regarding the woman in black mean that all of the villagers should be diagnosed aswell, and that any form of belief about an unexplained event or being can be described as a case of psychosis?
So is too much psychological research regarding mental health disorders resulting in over diagnosis, and how does publicly available information impact on diagnosis?
According to the World Health Organisation (2003) mental health disorders make up for 12-15% of disabilities in the world, which is twice as many as cancer. But is it really possible to distinguish between over diagnosis and the potential reality of prevalence of disorders within society? Zimmerman et al. (2008) used 700 patients with psychological disorders and compared their previous diagnoses of BiPolar disorder with diagnosis using the current criteria. He found that less than half of the patients who had been diagnosed in the past would be diagnosed using the latest version of the DSM, suggesting that diagnoses are very inconsistent.
It has also been suggested that there are too many conditions that people could be diagnosed with, therefore the amount of ‘normal’ people is rapidly decreasing (Mayo Clinic, 2011). For example, there has recently been a disorder identified as the premenstrual dysmorphic disorder, where the symptoms are pretty much the same as what most woman have when approaching menstruation, such as tension and mood swings, so is this really a disorder or is it just a natural state?
Over diagnosis also results in overtreatment, which can have damaging side effects on people. Enhanced technology increases the likelihood that an abnormality will be identified when someone is examined (Hall, 2011), however the majority of these abnormalities will have no impact on the individual’s health so is it worth the diagnosis? It has been found that many forms of cancer and similar disorders actually reduce in size without treatment, therefore a diagnosis does not help the situation. It has also been suggested that the anxiety and increased amount of tests needed on an individual can impact on their health and may result in more damaging consequences than not knowing. On many occasions a certain diagnosis can be found in so many people that it could be considered normal, so where is the distinction between over diagnosis and reality?
So do the public benefit from information that is made available to them? The NHS have created self-help guides for certain disorders that an individual can use to modify their own behaviour, see here. These leaflets include information regarding the effects of the behaviour, criteria for diagnosis, ways to measure your own behaviour and identify a problem and ways to then modify this behaviour. Reports suggest that these help people understand their feelings more and realise that often something that may appear abnormal is actually normal. These guides have also been recommended by GPs to their patients. Publicly available information can also help reduce the stigma associated with certain mental health issues so that people who really do require help are not scared to get some, as only 9% of individuals who require counselling or therapy for common mental health issues actually receive any (Royal College of Psychiatrists).
As you can see from the information presented above over diagnosis is a big issue within the field of psychology and can have negative impacts on individuals. Making research about mental health issues available to the public may help reduce the amount of misdiagnoses and may encourage people to tackle their own problems without needing an official diagnosis.
Blog, due 19/2/12
Comments, due 22/2/12
“Everything we think we know may be wrong. The correct results could be sitting in people’s file drawers because they can’t get them published” David Lehrer.
The issue about whether negative results, otherwise known as non-significant results, should be published is a very controversial issue. It has been suggested by The All Results Journal that approximately 60% of experiments conducted fail to produce the desired results and significant findings, however the proportion of these findings that are published is extremely low. In some parts of the USA 95-100% of studies published are of positive results, indicating that very few negative results are published (Morgan, 2010).
Many of you may be wondering why I am implying this is bad. We conduct experiments to try and find a difference between groups and these are the results that are being published so what’s the problem? Well, as this is a stats blog I’d better include a bit of maths in it. If in studies we are looking for results where the alpha level is set at .05 then there is a 1 in 20 chance that a difference will occur due to chance. So using this statistic we can assume that if 20 experiments are conducted then there is also a 1 in 20 chance that 1 of these will turn out to have significant findings due to chance. Therefore, if a study finds positive results then it appears to be significantly valid when it stands alone, but if it was presented in context with the other 19 studies which have negative results (and therefore are unlikely to be published) then we would have a different outlook on it, as explained by Steffer (2011). This demonstrates how the publication bias towards positive results influences perspective on positive results and creates an unbiased outlook.
Journal publishers say that both negative and positive results are equally considered when deciding what to publish, however research has shown that results take significantly longer to get published if they are negative rather than positive, if they get published at all, see Stern and Simes (1997). The graph below, taken from their research, shows that the amount of results unpublished remains higher for negative results over increasing periods of time compared to positive results.
The awareness of the bias of publication is increasing and there are now journals which are dedicated to publishing negative results, such as the Journal of Articles in Support of the Null Hypothesis for psychology. However, as a researcher planning a study are you likely to go and look for literature in a journal of negative results rather than a more reputable journal such as the Journal of Experimental Psychology which is more likely to publish positive results? I doubt it!
So what are the major implications of the bias of not publishing negative results? Well for starters it can hinder scientific research. If studies are not published then different researchers will continue to run almost identical studies and continue finding the same negative results which remain unpublished, which wastes a lot of time, effort and money. Sometimes it can be just as important to know that one variable does not affect another, as it is to know that it does, and to show that there is obviously something wrong with the paradigm you are investigating. Even if these results may not appear relevant to current research, future research can stem from this and may lead to other findings, whether positive or negative (Rice, 2011). The increased publication of negative results may also take the pressure off researchers to find positive results in their data so may reduce the likelihood of exaggerated results and the manipulation of data by only reporting findings from certain participants or parts of the methodology, therefore creating a more realistic image of the findings. But no matter what, the lack of publication of negative results creates a biased view of studies and can lead to a Type 1 error.
But let’s try and view this from another perspective, because other than positive results being far more exciting there must be other reasons why this bias exists. Firstly, can you imagine the size of journals if they were to publish the 60% of negative results as well as the 40% of positive results they already publish (assuming all studies are reported)? It would be a nightmare for researchers, or students, to sift through all this literature to find the material that is relevant and useful for what they want. Writing up a report to be published takes a lot of time, so would it be more productive to leave these studies the way they are and begin on a new investigation instead of wasting time writing up a failed one? Negative results can also have psychological effects. The reputation of a discipline can be greatly impacted on if the majority of the results published were negative. People may no longer give money to a charity for research if it appears like it is being wasted on studies that are not finding anything significant. This can reduce the motivation and faith in research, which can be especially important for many areas of research, such as treatments for cancer where the hope that research will find a cure is what can spur patients on.
So what can be done to reduce this issue? It has been suggested that researchers should enter their hypothesis and methodology into a database before conducting a study, and must therefore insert the data found afterwards regardless of the outcome, and even if it isn’t written up as a report (Schooler, 2011). This method has already been used successfully in clinical and educational research fields in the US.
After the information presented above I hope as fellow students you can now appreciate how all findings are important and should have equal opportunities for publishing in order to create a more balanced and realistic view of research. Negative results make up a large proportion of the data that is obtained however it is not adequately represented in literature so has a major impact on our perceptions, and can influence future research.
This article shows many varying views regarding whether negative results should be published or not.
Blog, due Sunday 5th Feb
Comments, due Wednesday 8th Feb