The types of issues that a pilot study checks for include:
- Whether instructions are understandable for the participants
- Whether the researchers can conduct the experiment adequately to ensure there is no experimenter bias
- Whether the equipment operates correctly and is suitable for measuring the variables you intend to measure
- Whether the task is suitable for the cognitive or physical abilities of the participants
- Whether the levels of the intervention are appropriate
- Whether there are any adverse effects on the participants produced by the study
A discipline in which pilot studies are very effective and are highly recommended is in clinical psychology. These pilots are often known as clinical trials, or proof of concept studies (Thabane et al., 2010). These types of trials are used to assess the safety and implications of treatments and interventions, before reasoning as to whether to go ahead with the main study. Pilot studies are used to assess the power analysis of a study, which to us means the probability that a Type 1 error will occur (falsely reject H0), and can influence a researcher’s decision as to whether to continue with the original study or not (Halpern, Karlawish, & Berlin, 2002). Some researchers suggest that there is too much emphasis on the power analysis of a pilot study and that many studies have been aborted because of this when they could have actually found effects in the main study (Kraemer et al., 2006).
The use of a pilot study is very advantageous to researchers as it enhances the validity and reliability of their study. It provides them with an opportunity to use various methods of implementation in order to identify which is the most effective to use in the study (Woken, CTL). It also allows them to identify any problems within the experiment, such as the issues mentioned above, and to rectify these before the main study is conducted, which can save lots of time and money as it increases the likelihood that the study will be successful.
However, researchers often put too much reliance on the results of a pilot study, even though it is only based on a small sample of the population. The exact sample size needed for a pilot study is debatable depending on the purpose of the study (Johanson & Brooks, 2010), however they should fit the same criteria as the participants in the main study. A small sample size can result in data being unrepresentative of the target population and inaccurate predictions and assumptions being made based on this data. Pilot studies can also result in contamination of the main study if the results from this are included within the main study, or if the same participants are included, as they may have practice effects and may not show the same pattern of results as other participants (van Teijlingen & Hundley, 2001). Halpern et al. (2002) also suggested that there may be ethical issues involved in pilot studies as many people participate in research for altruistic reasons and because they believe that it is a way to help other people, however in pilot studies there is very little public benefit of the research, especially if the main study is then aborted.
Although pilot studies have been shown to be an effective tool to increase the reliability and validity of a study they are very rarely published in journals. This may be partly due to journals being biased towards publishing positive results, as discussed in a previous blog post, therefore pilot studies where methodologies may be inaccurate and where results are not significant do not look as appealing to journals to publish, regardless of how beneficial they may be.
So although care must be taken when using the results of pilot studies to generate hypotheses and determine whether to continue with a study, they are a very useful tool to evaluate the methodologies of an experiment in order to improve it and increase the likelihood that the results will be reliable and valid.
This chapter gives a broader view of why pilot studies should be used and what their values are. It also provides an example of a pilot study, and how they should be implemented.
On a final note, it’s nice to finally say goodbye to blogs, and goodluck to everyone with the remainder of years 2 and 3! =]