.png){kind=icon}
Most of our book is concerned with what are essentially confirmatory studies. This involves a situation where a pilot study and/or reading of previous related work has allowed you to form one or more specific hypotheses and the study is designed specifically to provide a robust and powerful test of those hypotheses. This is called a confirmatory study. In an exploratory study less is known about the focal system and your study is aimed at throwing up hypotheses that future studies might usefully explore in greater depth.
In Section 2.3.3 of the supplementary material, we discuss a study of lifestyle factors associated with risk of stroke. This is unlikely to be such an exploratory study since a great deal is already known about risk of strokes and so from your reading of the existing literature you should be able to come up with specific hypotheses that should lead you to designing a confirmatory study focusing on factors that previous studies have indicated might well be important. However, if we were interested not in strokes but in some newly emerging disease then a study that investigated a very broad range of lifestyle factors might be a very worthwhile way of pointing us towards those factors that might be worthy of further specific and targeted study. Similarly, for a newly emerging bacterial infection we could imagine an in vitro study that investigated the effectiveness of a very wide range of antibiotics against this pathogen. Such an exploratory study might suggest the antibiotics that seem worthy of larger-scale in vivo studies of their efficacy in this context.
Thus, in scientific enquiry, exploratory studies tend to come before confirmatory ones and have much in common with how we have described pilot studies: they tend to measure a very broad range of factors and highlight those factors which might be worthy of more targeted investigation in follow-up confirmatory studies. We have these two types of studies because there is a trade-off in design: confirmatory studies should be designed to address a small number of hypotheses robustly and powerfully; exploratory studies address a much wider range of candidate hypotheses and so the design cannot be simultaneously optimized to give definitive results on all of them. Exploratory studies do not then in general give us definitive answers, they take a broad view and highlight those issues that might be worthy of further interest. Note the word ‘might’ here; exploratory studies should be designed to highlight potentially interesting avenues of research. As a consequence of this they may also through up “false positives” where they suggest a factor might be having an interesting effect when it in fact does not. In contrast, confirmatory studies take a much narrower focus and should give more definitive answers to a much more limited range of questions.
Thus, whether a study is exploratory or confirmatory will influence experimental design significantly, but it will also influence statistical analysis. As we discuss throughout the book, but especially in Chapter 6, confirmatory studies should be high-powered - this means that if the alternative hypothesis is true then the analysis of the experiment has a high chance of rejecting the null hypothesis. This means it has a low chance of failing to reject the null hypothesis when the null hypothesis is not true (a type II error). However, it is important in confirmatory studies to keep tight control over the other type of error: mistakenly rejecting the null hypothesis when it is actually true (a type I error). The analyses you carry out on your confirmatory study should be focussed on controlling the risk of such type 1 errors. However, for exploratory studies it is type II errors rather than type I errors that should concern us; we can live with making some type I errors (mistakenly thinking a factor might be important when it is not) because this error can be corrected by subsequent confirmatory study, whereas type II errors will not so naturally be corrected.
Returning to our example of screening antibiotics, imagine that we investigate 60 drugs of which only two (labelled 14 and 42) are effective. If our study highlights drugs 4, 14, 42, and 59 as being of interest then we have made two type I errors (identifying 4 and 59 as possibly being of interest), but we have also correctly identified the two effective drugs. A follow-up large-scale confirmatory study will investigate the effectiveness of all four and (if it is well designed) will provide strong evidence of the efficacy of 14 and 42 but not the other two. There has been a cost to our two false positives in the exploratory study because the follow-up confirmatory study was probably twice as big as it needed to be, but we have arrived at good evidence pointing to the two effective drugs against this pathogen. In contrast if the exploratory study throws up only drug 14, we have avoided any type I errors but made a type 2 error (failing to identify the effectiveness of drug 42); the follow-up confirmatory study will then provide even stronger evidence of the effectiveness of 14. But the type 2 error in the initial exploratory study has cost us; we have failed to identify one of the two drugs that could fight this pathogen. Thus, analyses of exploratory studies should be more liberal in acceptance of type I errors if this helps to minimize risk of type II errors, whereas confirmatory studies should be much more focussed on avoidance of type I errors. Chapter 6 provides much more consideration of power and type I and Type II errors.