Chapter 9 Review Questions

The two key characteristics of the experimental method are planned intervention by the researcher and random assignment. In contrast to observational studies, in which the researcher plays a passive role in collecting naturally occurring observations and looks for patterns after the fact, experiments involve an intervention by the researcher, who then tracks the consequences for the outcome of interest. This planned intervention is precisely what distinguishes the two types of data: whereas observational data are generated by forces beyond the researchers’ control, experimental data are generated from the researcher’s deliberate involvement. A second main component of the classic experimental method is random assignment. This element enables the researcher to create at least two groups: the experimental group that receives the treatment and the control group that does not. Assuming that the sample size is sufficiently large, and that treatment assignment is entirely determined by chance, we can be sure that the treatment and control groups are, on the aggregate, identical in terms of all attributes, both measured and unmeasured. Given that the groups are similar in all respects, we say that they are the same in expectation. That is to say, absent an exposure to the treatment, we would expect no real difference between the experimental groups in terms of all possible attributes, including the outcome of interest. Through the power of randomization, therefore, any observed differences in the two groups post-treatment may confidently be attributed to the experimental intervention.

Experiments may raise important ethical issues when they are implemented in practice. This is because, as opposed to playing a more passive role in observational studies, experimental researchers often manipulate aspects of an environment to see how human subjects react. In some studies, randomly withholding a program from part of the population for the purpose of making valid comparisons may be morally reprehensible on the grounds of equity and fairness. At times, this intervention implies some form of deception, which is necessary in order to get subjects to buy into the artificial environment created by the researcher and to limit interactions with other subjects, who may have received different conditions under the same experimental setting. Deception in experimental designs can range from the relatively benign withholding of information (as in a single- or double-blind study) to the somewhat more intrusive practice of deliberately misleading someone to believe that something fake is actually real.

In a between-subjects design, subjects are randomly assigned to various treatment and control groups and examined post-treatment (and sometimes pre-treatment, as well). In a within-subjects design, researchers evaluate subjects before and after they receive the treatment relative to the before and after scores of subjects who did not receive the treatment and infer causality based on any differences observed.

In some experimental designs, it is important that subjects remain unaware of the randomization process and of the existence of other groups exposed to different treatments. Knowledge of the design may induce some subjects to guess the study’s hypothesis or to share information with people assigned to other conditions, potentially biasing results. To guard against such contamination effects, researchers may decide to design their study as single- or double-blind. An experiment that blinds the subjects to the randomization process (i.e., the subject does not know if they are in the treatment or the control group, but the individual administering the treatment does know this information) is a single-blind design. For experiments with an individual administering the treatment, the gold standard is a double-blind design, in which both subjects and treatment administrators are unaware of the effect being tested. This second condition ensures that the administrator will not undeservedly induce subjects to react in a certain way or observe (consciously or subconsciously) some aspects of the subject’s reactions while ignoring other aspects.

Laboratory experiments are defined primarily by the location in which they occur. Subjects are recruited to a common place where the researcher exerts a relatively large degree of control over the experimental setting. Survey experiments are administered in the context of survey research; that is, respondents are randomly assigned to one or another version of the survey questionnaire. In light of experimentally manipulated variation in the survey instrument (the treatment), researchers look to identify differences in the responses given. In a field experiment, treatment assignment is randomized, but this time a researcher’s intervention takes place in a subjects’ natural environment. Conducting an experiment in this more authentic environment gives the researcher a more realistic assessment of treatment effects. The term natural experiment, or naturally occurring experiment, is used to describe a situation in which an exogenous change alters the natural course of events in the same way that a researcher might. When treatment assignment occurs naturally, as if by random chance, it may be possible for a researcher to exploit the strength of experimentation without actually designing and conducting an experimental intervention.