National Center for Technology Innovation
 

Quasi-Experimental Study


A quasi-experimental study is a type of evaluation which aims to determine whether a program or intervention has the intended effect on a study’s participants. Quasi-experimental studies take on many forms, but may best be defined as lacking key components of a true experiment. While a true experiment includes (1) pre-post test design, (2) a treatment group and a control group, and (3) random assignment of study participants, quasi-experimental studies lack one or more of these design elements.

Since the most common form of a quasi-experimental study includes a pre-post test design with both a treatment group and a control group, quasi-experimental studies are often an impact evaluation that assigns members to the treatment group and control group by a method other than random assignment. Because of the danger that the treatment and control group may differ at the outset, researchers conducting quasi-experimental studies attempt to address this in a number of other ways (e.g., by matching treatment groups to like control groups or by controlling for these differences in analyses). This section focuses on two forms of quasi-experimental studies: a pre-post test design study without a control group and a pre-post test design with a control group.

  • Pre-post test design study without a control group
  • A pre-post test design requires that you collect data on study participants’ level of performance before the intervention took place (pre-), and that you collect the same data after the intervention took place (post-). This study design only looks at one group of individuals who receive the intervention, which is called the treatment group. The pre-post test design allows you to make inferences on the effect of your intervention by looking at the difference in the pre-test and post-test results. However, interpreting the pre-test and post-test difference should be done with caution since you cannot be sure that the differences in the pre-test and the post-test are causally related to the intervention.

    Study Design

    Flow chart starting with Target Population, arrow to Selection, arrow to Pretest, arrow to Intervention X, arrow to Posttest.

  • Pre-post test design with a control group
  • While the pre-post test design will allow you to measure the potential effects of an intervention by examining the difference in the pre-test and post-test results, it does not allow you to test whether this difference would have occurred in the absence of your intervention. For example, perhaps the effect of improved academic achievement is due to the students getting used to taking a test rather than the use of educational software. To get the true effects of the program or intervention, it is necessary to have both a treatment group and a control group. As the names suggest, the treatment group receives the intervention. The control group, however, gets the business-as-usual conditions, meaning they only receive interventions that they would have gotten if they had not participated in the study. By having both a group that received the intervention and another group that did not, researchers control for the possibility that other factors not related to the intervention (e.g., students getting accustomed to a test, or simple maturation over the intervening time) are responsible for the difference between the pre-test and post-test results. It is also important that both the treatment group and the control group are of adequate size to be able to determine whether an effect took place or not. While the size of the sample ought to be determined by specific scientific methods, a general rule of thumb is that each group ought to have at least 30 participants.

    Study Design

    Flow chart starting with Target Population, arrow to Selection, a split into two separate parallel tracks. The top track arrows in from selection to Pretest, arrow to Intervention X, arrow to Posttest. The lower track arrows in from selection to Pretest, arrow to Business as Usual, arrow to Posttest.

  • What are the benefits of conducting a quasi-experimental study?
  • As mentioned above, the main difference between a quasi-experimental study and a true experimental study is that in an experimental study, the participants are assigned to a treatment group or a control group by random assignment. While doing so will allow you to get the best evidence of whether or not your intervention had the intended causal effect, random assignment is not always a practical step to take in the real world. For example, an organization may want to test the effects of an intervention on 4th grade special education students’ literacy. It is usually impractical to ask a school or school system to divide up students in their school into two separate classes through random assignment. Furthermore, it is unreasonable to ask a school to do this midyear. When random assignment is impractical, following a pre-post test design with a control group is the best route to go.

    Often it may not be practical to divide up study participants into both a treatment and control group at all. For example, if an organization approached a small elementary school to test the effects of an intervention on 4th grade special education students’ literacy, this school may only have one fourth-grade class. Dividing up the students into two groups may be impractical in the classroom setting. The pre-post test design, in this case, may be give you the best results with minimal classroom disruptions.

  • Ethical considerations
  • In many cases, withholding the intervention from the control group is ethical, since being in the study leaves them just as well off as they would have been had they not participated in the study. However, for vulnerable populations (e.g., students with disabilities) this may not be as straightforward. If researchers have good reason to believe that an intervention will benefit their study participants, denying this intervention to a control group may raise ethical questions. Conducting a pre-post test study addresses this ethical concern by offering the intervention to all study participants. For more, see section on ethical considerations.

  • When should I conduct a quasi-experimental study?
  • Quasi-experimental studies attempt to examine the effects of an intervention on a specific population. While you will not be able to make definitive causal inferences about the effects of the intervention through a quasi-experimental study, the general form of a research question that a quasi-experimental study can answer is “What is the effect of [specific program/intervention] on [a specific population]?”

    Many other forms of research may be more appropriate for your needs, depending on your research question. For example, if one asked “How useful is x?” or “What is the market demand of y?”, quasi-experimental research would not be helpful.

  • What are the resources needed to conduct a quasi-experimental study?
  • Quasi-experimental studies present their own set of challenges. Understanding all of the possible threats to the study’s validity, as well as the statistical methods needed to run accurate analyses is necessary. It is recommended that organizations use outside consultants or research organizations to run quasi-experimental studies. Not only does this allow the quasi-experimental study to be conducted by experts in study design, implementation, and analysis, but it also protects the results of the study from a perceived bias of the organization. For example, a company with a product that aims at improving literacy in children with learning disabilities might conduct a valid and bias-free study that shows that their product does, in fact, improve the literacy in children with learning disabilities. The reality is that school districts might not trust this result unless the study was conducted and analyzed by an unaffiliated organization. If you are working with an affiliated researcher, be sure to disclose the relationship up front so that consumers of the reports can judge any potential conflict of interest.

    Because of the intricacies of quasi-experimental studies—the design process, pre- and post-test development, and analysis—they tend to take approximately six months to a year. For similar reasons, quasi-experimental studies tend to be more expensive than other forms of research. Before initiating a quasi-experimental study, you should be sure that you have the required amount of time and resources to complete it.

    Other than the time and cost limitations of quasi-experimental studies, they often do a poor job controlling for the internal validity of the study. In studies with poor internal validity, researchers often will have to qualify their findings by leaving open the suggestion that the intervention was not the only factor in students’ success or lack thereof. For example, a finding might read “while we can’t be sure that this intervention improved results for reasons a, b, and c, the evidence shows that the intervention had positive effects on students’ performance such as……”

    Elements of a quasi-experimental study

  • Study participant recruitment
  • First, it is important that the study participants are members of the population in which you hope your intervention will be effective. Second, study participants must agree to be in the study. Often, this involves getting parent/guardian approval by signing consent forms which describe the study and any risks and benefits that study participants may be exposed to. However, just getting individuals to agree to participate is not enough. Any studies that are funded by government agencies must have the study, data collection items, and consent forms approved by an Institutional Review Board before study participant recruitment begins. For more, see section on Institutional Review Boards (IRB).

  • What sorts of data should I collect?
  • For quasi-experimental studies, it is important to collect multiple forms of data. Ideally, a researcher will collect descriptive information, data on the fidelity of the study, data on the dosage of the intervention, and outcome data.

    Descriptive, or contextual, information will allow the researchers to understand the context and the details of the environment in which the study takes place. This may be background information on the participants in your study, or information about prior interventions they have received.

    Data on the fidelity of the study is information that may allow researchers to confirm that the study was conducted as planned. This may be confirmation that your treatment group—those people who received the intervention—did in fact get the full treatment as planned.

    Data on the dosage of the intervention measures the quantity of the intervention. Dosage data is important to collect, not only because it may confirm your fidelity data, but because this data is also collected on the control group—those participants in the study who did not receive the treatment. While your study will not offer treatment to the control group by design, it is important to know that the control group members did not seek alternate treatment or interventions outside of the study.

    Finally, it is essential in a quasi-experimental study that you collect outcome data. In order to be able to analyze your outcome data, it is important that this data is quantitative in nature. Some examples of quantitative data include test scores, observations (how many times each person did x), tracking/log data on how users interacted with a device or software, and survey responses (asking people to rate answers on a scale—e.g., strongly agree, agree, disagree, strongly disagree).

    Because you cannot make definitive causal inferences from quasi-experimental quantitative data alone (e.g., “this device improves communication skills”), it is also important that your outcome data is supplemented with additional forms of qualitative data. Qualitative data, in the form of study participant interviews and focus groups, can give valuable information that may explain the quantitative results. For example, study participants may speak to why the intervention worked for them, or may clue researchers in to other factors outside of the study, that influenced their post-test results.

  • How should I analyze this data?
  • What data analysis you do depends on the research question. However, by and large, researchers will examine the descriptive information to explain the context of the study, analyze the fidelity and dosage implementation to determine if the study happened as planned, and then use statistical methods to analyze the outcome data. Statistical methods vary depending on the specific quasi-experimental design. Quasi-experimental studies rely primarily on simple statistical tests, like t tests. Typically, quasi-experimental studies with a control group use statistical equations called regressions. Statistical regressions are data equations that allow one to see the numerical effect of the treatment status by controlling for participant characteristics and pretest scores. For more information on statistical tests, see these resources:

    Engineering Statistics Handbook: What Are Statistical Tests?

    Intuitive Biostatistics: Choosing a Statistical Test

    UCLA Academic Technology Services: Choosing the Correct Statistical Test

    For each type of quasi-experimental study design, qualitative data analysis methods should be applied. The most common method in qualitative data analysis is “coding.” Coding is a process in which researchers read through the qualitative data and enter in agreed-upon short codes, which help enumerate and draw attention to themes. Codes will be unique to the study, depending entirely on the purpose of the study. It is important for multiple researchers to code the same data, to ensure inter-rater reliability. For example, in the study described below, Fraction Sense, the teacher’s instructional language was audio and video recorded. These tapes were then watched and key phrases or instructional moves were noted on a coding sheet for how often and to what degree teachers used the instructional language which they were taught during the professional development training. This allowed the researchers to determine whether the instruction was being given with fidelity and also cross-check whether students who received better instruction made more gains with the software.

    See more resources on quasi-experimental studies:

    Web Center for Social Research Methods: Quasi-Experimental Design

    Web Center for Social Research Methods: Other Quasi-Experimental Designs

    Colorado State University: Differences Between Experimental and Quasi-Experimental Research

    For more information on coding, and other qualitative data analysis methods, see the section on case study design.

    Examples and additional resources

  • Real world example
  • Fraction Sense Software Assists Math Students. This 2008 NCTI Tech in the Works award shows how collaborative research can overcome implementation challenges in public schools and technology can be developed to support all students. Using a multiple classrooms and a pre-post test design, this study shows how quasi-experimental research can be conducted in school environments.

    Final report of the study: http://www.nationaltechcenter.org/documents/FractionSenseFinalReport.pdf

  • Published articles using a quasi-experimental study
  • Shapley, K. S., Sheehan, D., Maloney, C. & Caranikas-Walker, F. (2010). Evaluating the implementation fidelity of technology immersion and its relationship with student achievement. Journal of Teaching, Learning, and Assessment, 19(4).

    This study shows how researchers can document fidelity and the difference high quality implementation can make for a program or initiative.