Threats to validity

Due by 11:59 PM on Thursday, February 22, 2024

Instructions

You need to complete the “Assessing validity” section below. Ideally you should type this in a Quarto file and render your document to HTML or Word or PDF, but you can also write in Word if you want (though your final project will need to be in Quarto, and this would give you practice).

I’ve created a Quarto template you can use here: threats-validity.zip. It’s also available on Posit.cloud.

Submit this assignment as a PDF or Word file on iCollege.

Internal validity scores

One helpful way to assess an evaluation’s internal validity is to systematically go through each possible threat and evaluate if the research design addresses it. For each of the 13 types of internal validity that we discussed in class, assign 1 point if the study addresses it and 0 points if it fails to do so. Add up the total points and assign the study a final internal validity score.

Not all types of validity will apply to every study. Testing, for example, is only an issue if there is a pre-test and it involves a skill that could feasibly be enhanced by practicing the test. If a threat doesn’t apply, don’t give it a score.

  • Omitted variable bias
    1. Selection
    2. Attrition
  • Trends
    1. Maturation
    2. Secular trends
    3. Seasonality
    4. Testing
    5. Regression to the mean
  • Study calibration
    1. Measurement error
    2. Time frame of study
  • Contamination
    1. Hawthorne effects
    2. John Henry effects
    3. Spillovers
    4. Intervening events

Assessing validity

Your textbook Impact Evaluation in Practice is full of short examples of real-world evaluations, experiments, and studies. For this assignment, you will assess the (1) internal validity, (2) external validity, and (3) construct validity for two of the examples from the book. If the summary in the book isn’t sufficient, you can skim through the original study for more details. The original studies can be found in the Content section of iCollege.

You will assess two (2) of these four studies. Pick whichever two seem the most interesting to you:

  1. The effect of conditional cash transfers on education in Mexico (Box 4.2, p. 70)
  2. The impact of Sesame Street on school readiness (Box 5.1, p. 91)
  3. The effects of police deployment on crime in Argentina (Box 7.2, p. 135)
  4. Early childhood development and migration in Jamaica (Box 9.5, p. 170)

For each of the two studies you choose, do the following:

  1. Go through the 13 types of internal validity and describe in 2–3 sentences how the study succeeds/fails to address each concern. Calculate a total internal validity score.
  2. Describe any threats to external validity and assess how generalizable the findings are. (≈75 words)
  3. Describe any threats to construct validity and assess if the researchers are measuring the thing they intended to measure. (≈75 words)

References

Di Tella, Rafael, and Ernesto Schargrodsky. 2004. “Do Police Reduce Crime? Estimates Using the Allocation of Police Forces After a Terrorist Attack.” American Economic Review 94 (1): 115–33. https://doi.org/10.1257/000282804322970733.
Gertler, Paul, James Heckman, Rodrigo Pinto, Arianna Zanolini, Christel Vermeersch, Susan Walker, Susan M. Chang, and Sally Grantham-McGregor. 2014. “Labor Market Returns to an Early Childhood Stimulation Intervention in Jamaica.” Science 344 (6187): 998–1001. https://doi.org/10.1126/science.1251178.
Kearney, Melissa, and Phillip Levine. 2015. “Early Childhood Education by MOOC: Lessons from Sesame Street.” w21229. Cambridge, MA: National Bureau of Economic Research. https://doi.org/10.3386/w21229.
Schultz, T. Paul. 2004. “School Subsidies for the Poor: Evaluating the Mexican Progresa Poverty Program.” Journal of Development Economics 74 (1): 199–250. https://doi.org/10.1016/j.jdeveco.2003.12.009.