Come fly with me, let’s fly let’s fly away….Why Pilot studies are used in research.

A pilot study is a small version of an experiment which is conducted before the main study.  This acts like a prototype and aids the decision as to whether the main study should be conducted or not.  The main aim of a pilot study is to assess the effectiveness of an experiment and to identify any problems with it before the main study is conducted.

The types of issues that a pilot study checks for include:

  • Whether instructions are understandable for the participants
  • Whether the researchers can conduct the experiment adequately to ensure there is no experimenter bias
  • Whether the equipment operates correctly and is suitable for measuring the variables you intend to measure
  • Whether the task is suitable for the cognitive or physical abilities of the participants
  • Whether the levels of the intervention are appropriate
  • Whether there are any adverse effects on the participants produced by the study

(Altman et al., on behalf of NC3Rs, 2006)

A discipline in which pilot studies are very effective and are highly recommended is in clinical psychology.  These pilots are often known as clinical trials, or proof of concept studies (Thabane et al., 2010).  These types of trials are used to assess the safety and implications of treatments and interventions, before reasoning as to whether to go ahead with the main study.  Pilot studies are used to assess the power analysis of a study, which to us means the probability that a Type 1 error will occur (falsely reject H0), and can influence a researcher’s decision as to whether to continue with the original study or not (Halpern, Karlawish, & Berlin, 2002).  Some researchers suggest that there is too much emphasis on the power analysis of a pilot study and that many studies have been aborted because of this when they could have actually found effects in the main study (Kraemer et al., 2006).

The use of a pilot study is very advantageous to researchers as it enhances the validity and reliability of their study.  It provides them with an opportunity to use various methods of implementation in order to identify which is the most effective to use in the study (Woken, CTL).  It also allows them to identify any problems within the experiment, such as the issues mentioned above, and to rectify these before the main study is conducted, which can save lots of time and money as it increases the likelihood that the study will be successful.

However, researchers often put too much reliance on the results of a pilot study, even though it is only based on a small sample of the population.  The exact sample size needed for a pilot study is debatable depending on the purpose of the study (Johanson & Brooks, 2010), however they should fit the same criteria as the participants in the main study.  A small sample size can result in data being unrepresentative of the target population and inaccurate predictions and assumptions being made based on this data.  Pilot studies can also result in contamination of the main study if the results from this are included within the main study, or if the same participants are included, as they may have practice effects and may not show the same pattern of results as other participants (van Teijlingen & Hundley, 2001).  Halpern et al. (2002) also suggested that there may be ethical issues involved in pilot studies as many people participate in research for altruistic reasons and because they believe that it is a way to help other people, however in pilot studies there is very little public benefit of the research, especially if the main study is then aborted.

Although pilot studies have been shown to be an effective tool to increase the reliability and validity of a study they are very rarely published in journals.  This may be partly due to journals being biased towards publishing positive results, as discussed in a previous blog post, therefore pilot studies where methodologies may be inaccurate and where results are not significant do not look as appealing to journals to publish, regardless of how beneficial they may be.

So although care must be taken when using the results of pilot studies to generate hypotheses and determine whether to continue with a study, they are a very useful tool to evaluate the methodologies of an experiment in order to improve it and increase the likelihood that the results will be reliable and valid.

This chapter gives a broader view of why pilot studies should be used and what their values are.  It also provides an example of a pilot study, and how they should be implemented.

On a final note, it’s nice to finally say goodbye to blogs, and goodluck to everyone with the remainder of years 2 and 3! =]


3 responses »

  1. I agree completely that pilot studies are very effective. But there has been a lot of controversy over how much reliance there is on this type of study! Even though a highly effective type of research it has 15% chance of turning out a type 1 or 2 error (Lancaster, 2004). This can have extreme consequences for those who take the results at face value.

  2. Hi there 🙂
    I agree that pilot studies are very good indicator for the main studies to see whether any improvements should be made or to see is there any problem/issue that can be avoid.
    Any issues that revealed in pilot studies would be helpful to improve the main studies, and it could be helpful to help improve the validity and reliability.
    And if there are too many problems, a second pilot study may need to be done in order to revise the main study whether the design of the main study having problem and if so, the main study may need to be abandoned.
    Therefore, pilot studies is indeed very effective in different areas of research.

  3. Pingback: Comments… :D the very last one! | The Blog :) by poey

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s