National Center for Technology Innovation
 

Experimental Study Design


An experimental study is a type of evaluation that seeks to determine whether a program or intervention had the intended causal effect on program participants. There are three key components of an experimental study design: (1) pre-post test design, (2) a treatment group and a control group, and (3) random assignment of study participants.

A pre-post test design requires that you collect data on study participants’ level of performance before the intervention took place (pre-), and that you collect the same data on where study participants are after the intervention took place (post). This design is the best way to be sure that your intervention had a causal effect.

To get the true effects of the program or intervention, it is necessary to have both a treatment group and a control group. As the name suggests, the treatment group receives the intervention. The control group, however, gets the business-as-usual conditions, meaning they only receive interventions that they would have gotten if they had not participated in the study. By having both a group that received the intervention and another group that did not, researchers control for the possibility that other factors not related to the intervention (e.g., students getting accustomed to a test, or simple maturation over the intervening time) are responsible for the difference between the pre-test and post-test results. It is also important that both the treatment group and the control group are of adequate size to be able to determine whether an effect took place or not. While the size of the sample ought to be determined by specific scientific methods, a general rule of thumb is that each group ought to have at least 30 participants.

Finally, it is important to make sure that both the treatment group and the control group are statistically similar. While no two groups will ever be exactly alike, the best way to be sure that they are as close as possible is having a random assignment of the study participants into the treatment group and control group. By randomly assigning participants, you can be sure that any difference between the treatment group and control group is due to chance alone, and not by a selection bias.

  • Study design
  • Flow chart starting with Target Population, arrow to Selection, arrow to Random Assignment, then a split into two separate parallel tracks. The top track arrows in from selection to Pretest, arrow to Intervention X, arrow to Posttest. The lower track arrows in from selection to Pretest, arrow to Business as Usual, arrow to Posttest.

  • What are the benefits of conducting an experimental study?
  • An experimental study is often considered the gold standard of research. Because of the pre-post tests, treatment and control groups, and group random assignment, experimental studies address more threats to internal validity than any other type of study. By having greater internal validity, an experimental study will have the best chance of determining whether or not a program or intervention had a causal effect on the treatment group. Furthermore, any findings from an experimental study can be applied to the population from which the study’s samples were drawn. For example, a robust study conducted in twelve fourth-grade math classes would probably represent the population of U.S. fourth graders and fourth-grade classrooms well enough for its findings to be considered applicable to all U.S. fourth-grade math classes.

  • When should I conduct an experimental study?
  • While experimental studies are considered to have the most internal validity, they are not always the most appropriate. As mentioned above, experimental studies are best used to address whether a program or intervention had the intended causal effect on program participants. Further, it is necessary that the program or intervention can be measured quantitatively in some fashion (through a knowledge test, observations, survey questions, etc.). The general form of a research question that an experimental study can answer is similar to a quasi-experimental study: “What is the effect of [specific program/intervention] on [a specific population]?

    Many other forms of research may be more appropriate for your needs, depending on your research question. For example, if one asked “How useful is x?” or “What is the market demand of y?”, experimental research would not be helpful.

    “The specific questions that the experiment is intended to answer must be clearly identified before carrying out the experiment. We should also attempt to identify known or expected sources of variability in the experimental units since one of the main aims of a designed experiment is to reduce the effect of these sources of variability on the answers to questions of interest. That is, we design the experiment in order to improve the precision of our answers.” (Taken from Valerie J. Easton and John H. McColl’s Statistics Glossary v1.1.)

  • What are the resources needed to conduct an experimental study?
  • Experimental studies are often difficult to implement because they can be so complex. Researchers must understand all the possible threats to the study’s validity, as well as the statistical methods needed to run accurate analyses. It is recommended that organizations use outside consultants or research organizations to run experimental studies. Not only does this allow the experimental study to be conducted by experts in study design, implementation, and analysis, but it also protects the results of the study from a perceived bias of the organization. For example, a company with a product that aims to improve literacy in children with learning disabilities might conduct a valid and bias-free study that shows that their product does in fact improve the literacy in children with learning disabilities. The reality is that school districts might not trust this result unless the study was conducted and analyzed by an unaffiliated organization. If you are working with an affiliated researcher, be sure to disclose the relationship up front so that consumers of the reports can judge any potential conflict of interest.

    Because of the intricacies of experimental studies—the design process, the random assignment, pre- and post-test development, and analysis—they tend to take more time than most other types of studies, lasting for years in some cases. For similar reasons, experimental studies tend to be much more expensive than other forms of research. Given that at least two full groups need to be recruited, these studies usually involve more participants and more settings, too. Before initiating an experimental study, you should be sure that you have the required amount of time and resources to complete it.

    See these resources for an experimental study:

    Wikipedia: Design of Experiments

    Experimental Research

    AAAS Office of Research Integrity Bibliography

  • Ethical considerations
  • Some ethical considerations apply specifically to conducting experimental research. The design of an experimental study dictates that there is both a treatment group, which receives the intervention, and a control group, which does not. In many cases denying the control group the intervention is ethical, since no harm is done to them, and they are as well off as they would have been had they not participated in the study. However, for vulnerable populations (e.g., students with disabilities), this is not as straightforward. If researchers have good reason to believe that an intervention will benefit their study participants, denying this intervention to a control group can be considered unethical. When you have an intervention meant to benefit vulnerable populations, you may consider another form of research design, such as a quasi-experimental research design.

    Elements of an experimental study

  • Study participant recruitment
  • Recruiting study participants can be difficult. First, it is important that the study participants are members of the population in which you hope your intervention will be effective. Second, study participants must agree to be in the study. Often, this involves getting parent/guardian approval by signing consent forms which describe the study and any risks and benefits that study participants may be exposed to. But just getting individuals to agree to participate is not enough. Any studies that are even partially funded by government agencies must have the study, data collection items, and even the consent forms approved by an Institutional Review Board before study participant recruitment begins. For more, see section on Institutional Review Boards (IRB).

  • What sorts of data should I collect?
  • For experimental studies, it is important to collect multiple forms of data. Ideally, a researcher will collect descriptive information, data on the fidelity of the study, data on the dosage of the intervention, and outcome data.

    Descriptive, or contextual, information will allow the researchers to understand the context and the details of the environment in which the study takes place. This may be background information on the participants in your study, or information about prior interventions they have received.

    Data on the fidelity of the study is information that may allow researchers to confirm that the study was conducted as planned. This may be confirmation that your treatment group—those people who received the intervention—actually did in fact get the full treatment.

    Data on the dosage of the intervention measures the quantity of the intervention. Dosage data is important to collect, not only because it may confirm your fidelity data, but because this data is also collected on the control group—those people in the study who did not receive your treatment. While your study will not offer treatment to the control group by design, it is important to know that the control group members did not seek alternate treatment or interventions outside of the study.

    Finally, it is essential in an experimental study that you collect outcome data. In order to be able to analyze your outcome data, it is important that this data is quantitative in nature. Some examples of quantitative data include test scores, observations (how many times each person did x), tracking/log data on how users interacted with a device or software, and survey responses (asking people to rate answers on a scale—e.g., strongly agree, agree, disagree, strongly disagree).

  • How should I analyze this data?
  • What data analysis you do depends on the research question. However, by and large, researchers will examine the descriptive information to explain the context of the study, analyze the fidelity and dosage implementation to determine if the study happened as planned, and then use statistical methods to analyze the outcome data. Typically, experimental studies use statistical equations called regressions. Statistical regressions are data equations that allow one to see the true numerical effect of the treatment status by controlling for participant characteristics and pretest scores. Other times, simple statistical tests, like t tests, may suffice. For more information on statistical tests, see these resources:

    Engineering Statistics Handbook: What Are Statistical Tests?

    Intuitive Biostatistics: Choosing a Statistical Test

    UCLA Academic Technology Services: Choosing the Correct Statistical Test

    Examples and additional resources

  • Real world example
  • Teaching Children with Autism Through Technology. This 2008 NCTI Tech in the Works award shows how collaborative research can sustain a challenging study in one of the nation’s most diverse school districts, Los Angeles Unified. In this experimental design study, researchers looked at the effects of a Computer-Assisted Instruction (CAI) program on preschoolers and first-graders with autism. The study looked at 50 children with autism (25 in the treatment group, 25 in the control group) and took place over a full school year, involving dozens of teachers and paraprofessionals, as well as the children’s parents.

    Final report from the study: http://www.nationaltechcenter.org/documents/TiW_FinalReport_TeachTown.pdf

  • Published articles using an experimental design
  • The National Study of the Effectiveness of Educational Technology Interventions

    This was a very large study with random assignments of control and treatment groups implementing educational technology, funded by the U.S. Department of Education. Findings were published in two waves, 2007 and 2009.