Monthly Archives: March 2012

Come fly with me, let’s fly let’s fly away….Why Pilot studies are used in research.

A pilot study is a small version of an experiment which is conducted before the main study.  This acts like a prototype and aids the decision as to whether the main study should be conducted or not.  The main aim of a pilot study is to assess the effectiveness of an experiment and to identify any problems with it before the main study is conducted.

The types of issues that a pilot study checks for include:

  • Whether instructions are understandable for the participants
  • Whether the researchers can conduct the experiment adequately to ensure there is no experimenter bias
  • Whether the equipment operates correctly and is suitable for measuring the variables you intend to measure
  • Whether the task is suitable for the cognitive or physical abilities of the participants
  • Whether the levels of the intervention are appropriate
  • Whether there are any adverse effects on the participants produced by the study

(Altman et al., on behalf of NC3Rs, 2006)

A discipline in which pilot studies are very effective and are highly recommended is in clinical psychology.  These pilots are often known as clinical trials, or proof of concept studies (Thabane et al., 2010).  These types of trials are used to assess the safety and implications of treatments and interventions, before reasoning as to whether to go ahead with the main study.  Pilot studies are used to assess the power analysis of a study, which to us means the probability that a Type 1 error will occur (falsely reject H0), and can influence a researcher’s decision as to whether to continue with the original study or not (Halpern, Karlawish, & Berlin, 2002).  Some researchers suggest that there is too much emphasis on the power analysis of a pilot study and that many studies have been aborted because of this when they could have actually found effects in the main study (Kraemer et al., 2006).

The use of a pilot study is very advantageous to researchers as it enhances the validity and reliability of their study.  It provides them with an opportunity to use various methods of implementation in order to identify which is the most effective to use in the study (Woken, CTL).  It also allows them to identify any problems within the experiment, such as the issues mentioned above, and to rectify these before the main study is conducted, which can save lots of time and money as it increases the likelihood that the study will be successful.

However, researchers often put too much reliance on the results of a pilot study, even though it is only based on a small sample of the population.  The exact sample size needed for a pilot study is debatable depending on the purpose of the study (Johanson & Brooks, 2010), however they should fit the same criteria as the participants in the main study.  A small sample size can result in data being unrepresentative of the target population and inaccurate predictions and assumptions being made based on this data.  Pilot studies can also result in contamination of the main study if the results from this are included within the main study, or if the same participants are included, as they may have practice effects and may not show the same pattern of results as other participants (van Teijlingen & Hundley, 2001).  Halpern et al. (2002) also suggested that there may be ethical issues involved in pilot studies as many people participate in research for altruistic reasons and because they believe that it is a way to help other people, however in pilot studies there is very little public benefit of the research, especially if the main study is then aborted.

Although pilot studies have been shown to be an effective tool to increase the reliability and validity of a study they are very rarely published in journals.  This may be partly due to journals being biased towards publishing positive results, as discussed in a previous blog post, therefore pilot studies where methodologies may be inaccurate and where results are not significant do not look as appealing to journals to publish, regardless of how beneficial they may be.

So although care must be taken when using the results of pilot studies to generate hypotheses and determine whether to continue with a study, they are a very useful tool to evaluate the methodologies of an experiment in order to improve it and increase the likelihood that the results will be reliable and valid.

This chapter gives a broader view of why pilot studies should be used and what their values are.  It also provides an example of a pilot study, and how they should be implemented.

On a final note, it’s nice to finally say goodbye to blogs, and goodluck to everyone with the remainder of years 2 and 3! =]


Comments for Naomi for Blog 3


Has excess psychological research resulted in too many inappropriate diagnoses of mental health disorders?


This blog has stemmed from an article I came across online from ‘The psychologist’ which critically analyses the film ‘The Woman In Black’.  For anyone who hasn’t seen this film the basic plot can be found hereGreen (2012) has analysed the behaviour of Arthur Kipps and tried relating this to the context of the film and Kipp’s situation, and has come to the conclusion that he has a case of psychosis.  However, is this really a correct diagnosis? It could be suggested that the figure Kipps sees is just a delusion, one of the criteria set by the DSM to diagnose psychosis (p. 332), however as Kipps sees the woman before he hears the horror stories from the villagers is it more likely to actually be reality?  When this is combined with the deaths of the children in the village, and previous encounters by others with the figure then how can it all be in his head? To me there appears to be too many linked variables for it just to be coincidence and for him to be diagnosed with psychosis. Also, does this superstition regarding the woman in black mean that all of the villagers should be diagnosed aswell, and that any form of belief about an unexplained event or being can be described as a case of psychosis?

So is too much psychological research regarding mental health disorders resulting in over diagnosis, and how does publicly available information impact on diagnosis?

According to the World Health Organisation (2003) mental health disorders make up for 12-15% of disabilities in the world, which is twice as many as cancer.  But is it really possible to distinguish between over diagnosis and the potential reality of prevalence of disorders within society?  Zimmerman et al. (2008) used 700 patients with psychological disorders and compared their previous diagnoses of BiPolar disorder with diagnosis using the current criteria.  He found that less than half of the patients who had been diagnosed in the past would be diagnosed using the latest version of the DSM, suggesting that diagnoses are very inconsistent.

It has also been suggested that there are too many conditions that people could be diagnosed with, therefore the amount of ‘normal’ people is rapidly decreasing (Mayo Clinic, 2011).  For example, there has recently been a disorder identified as the premenstrual dysmorphic disorder, where the symptoms are pretty much the same as what most woman have when approaching menstruation, such as tension and mood swings, so is this really a disorder or is it just a natural state?

Over diagnosis also results in overtreatment, which can have damaging side effects on people.  Enhanced technology increases the likelihood that an abnormality will be identified when someone is examined (Hall, 2011), however the majority of these abnormalities will have no impact on the individual’s health so is it worth the diagnosis?  It has been found that many forms of cancer and similar disorders actually reduce in size without treatment, therefore a diagnosis does not help the situation.  It has also been suggested that the anxiety and increased amount of tests needed on an individual can impact on their health and may result in more damaging consequences than not knowing.  On many occasions a certain diagnosis can be found in so many people that it could be considered normal, so where is the distinction between over diagnosis and reality?

So do the public benefit from information that is made available to them? The NHS have created self-help guides for certain disorders that an individual can use to modify their own behaviour, see here.  These leaflets include information regarding the effects of the behaviour, criteria for diagnosis, ways to measure your own behaviour and identify a problem and ways to then modify this behaviour.  Reports suggest that these help people understand their feelings more and realise that often something that may appear abnormal is actually normal.  These guides have also been recommended by GPs to their patients.  Publicly available information can also help reduce the stigma associated with certain mental health issues so that people who really do require help are not scared to get some, as only 9% of individuals who require counselling or therapy for common mental health issues actually receive any (Royal College of Psychiatrists).

As you can see from the information presented above over diagnosis is a big issue within the field of psychology and can have negative impacts on individuals.  Making research about mental health issues available to the public may help reduce the amount of misdiagnoses and may encourage people to tackle their own problems without needing an official diagnosis.