Class 8: Diff-in-diff I

# In-person session 8

**February 29, 2024**

]

---

# Plan for today

.box-5.medium.sp-after-half[Models vs. designs]

.box-3.medium.sp-after-half[Interactions and regression]

.box-1.medium.sp-after-half[Simple diff-in-diff]

.box-6.medium.sp-after-half[Two-way fixed effects]

---

# Two quick things

.box-8.medium[Can you have a study or evaluation that is immune from all threats to validity?]

.box-8.small[And if not, how do you use evaluations for making policies?]

.box-8.medium.sp-before[Can we see more matching/IPW?]

---

layout: false
name: models-designs
class: center middle section-title section-title-5 animated fadeIn

# Models vs. designs

---

---

.center[
<figure>
 <img src="img/08-class/2021-nobel-winners.jpg" alt="2021 econ Nobel winners" title="2021 econ Nobel winners" width="55%">
</figure>
]

???

- Card (and Krueger): NJ/PA minimum wage + the beginning of this whole credibility revolution thing
- Angrist: MHE and MM and making causal inference accessible
- Imbens: A ton of CI stuff + attempting to bridge DAG world with situation-based world

- https://twitter.com/NobelPrize/status/1447502627114205187 - PA/NJ
- https://twitter.com/MaxCRoser/status/1447505582450151431
- https://twitter.com/Stanford/status/1447549033539637248

---

.center[
<figure>
 <img src="img/08-class/alan-krueger.jpg" alt="Alan Krueger" title="Alan Krueger" width="80%">
</figure>
]

???

Alan Krueger died by suicide in 2019

---

.center[
<figure>
 <img src="img/08-class/pa-nj-nobel.jpg" alt="Nobel PA/NJ" title="Nobel PA/NJ" width="57%">
</figure>
]

---

---

.box-5.large[Design-based vs. model-based inference]

---

.box-5.medium[How would you know when it is appropriate to use a quasi-experiment over an RCT?]

---

---

# Identification strategies

.box-inv-5.small.sp-after[The goal of *all* these methods is to isolate (or **identify**) the arrow between treatment → outcome]

.box-inv-5.less-medium[Model-based identification]

.float-left.center[.box-5[DAGs] .box-5[Matching] .box-5[Inverse probability weighting]]

.box-inv-5.less-medium.sp-before[Design-based identification]

.float-left.center[.box-5[Randomized controlled trials] .box-5[Difference-in-differences]]

.float-left.center[.box-5[Regression discontinuity] .box-5[Instrumental variables]]

---

# Model-based identification

.pull-left[
<figure>
 <img src="04-slides_files/figure-html/edu-earn-adjust-1.png" alt="Education earnings DAG" title="Education earnings DAG" width="100%">
</figure>
]

.box-inv-5.small[Everything that needs to be adjusted is measurable; no unobserved confounding]

.box-inv-5.small[**Big assumption!**]

.box-inv-5.tiny[This is why lots of people don't like DAG-based adjustment]
]

---

.center[
<figure>
 <img src="img/08-class/charles-ozzy.png" alt="King Charles and Ozzy Osbourne" title="King Charles and Ozzy Osbourne" width="50%">
</figure>
]

---

---

# Design-based identification

.box-inv-5.small[Use randomization to remove confounding]

.center[
<figure>
 <img src="05-slides_files/figure-html/experimental-dag-1.png" alt="RCT DAG" title="RCT DAG" width="60%">
</figure>
]
]

.box-inv-5.small[Use before/after & treatment/control differences to remove confounding]

.center[
<figure>
 <img src="08-slides_files/figure-html/min-wage-dag-1.png" alt="Diff-in-diff DAG" title="Diff-in-diff DAG" width="90%">
</figure>
]
]

---

---

.box-5.large[Which is better or more credible? RCTs, quasi experiments, or DAG-based models?]

---

.center[
<figure>
 <img src="img/08-class/causality-continuum.png" alt="The (wrong!) causality continuum" title="The (wrong!) causality continuum" width="90%">
</figure>
]

---

.box-5.huge[There's no hierarchy!]

---

layout: false
name: interactions
class: center middle section-title section-title-3 animated fadeIn

# Interactions and regression

---

.box-3.large[Can we talk more about interaction terms and how to interpret them?]

---

.box-3.large[Regression is just fancy averages!]

---

layout: false
name: diff-in-diff
class: center middle section-title section-title-1 animated fadeIn

# Simple diff-in-diff

---

.center[
<figure>
 <img src="img/08-class/lambeth-southwark-vauxhall.jpg" alt="Lambeth and Southwark-Vauxhall" title="Lambeth and Southwark-Vauxhall" width="70%">
</figure>
]

---

]

---

.center[
<figure>
 <img src="img/08-class/bedtime-math.png" alt="Bedtime math" title="Bedtime math" width="45%">
</figure>
]

---

.center[
<figure>
 <img src="img/08-class/bedtime-math-diff-diff.png" alt="Bedtime math diff-in-diff" title="Bedtime math diff-in-diff" width="100%">
</figure>
]

---

---

.box-1.medium[When doing your subtracting to get your differences in the matrix, is it better to do the vertical or horizontal subtractions?]

.box-1.medium[Are there situations where one is preferable to the other?]

---

.box-1.medium[Why are we learning two ways to do diff-in-diff? (2x2 matrix vs. `lm()`)]

---

.box-1.large[What happened to confounding??]

.box-1.medium[Now we're only looking at just two "confounders"?]

.box-1.medium[Should we still control for things?]

???

The parallel trends assumption takes care of that

---

.box-1.less-medium[What group level is best for comparison? For example, if we are looking at policy change in NJ, is it best to compare with just one or two similar states? How similar do the populations need to be?]

.box-1.medium.sp-after[Wouldn't matching be better?]

.box-1.less-medium[Do we have to think about balance when dealing with observational data in diff in diff?]

.box-inv-1[[Two-way fixed effects (TWFE)](https://www.andrewheiss.com/blog/2021/08/25/twfe-diagnostics/)]

???

- Multiple states/groups are possible - that's TWFE
- Wouldn't matching be better? Sure, if you're doing state-level stuff. But their data was restaurant level

- Balance: Maybe. With just two states/villages/countries/whatever, yes. With lots, the state/year fixed effects pick up those trends for you

---

.box-1.large[Minimum legal drinking age]

---

.center[
<figure>
 <img src="img/08-class/mm-fig-5-4.png" alt="Mastering Metrics Figure 5.4" title="Mastering Metrics Figure 5.4" width="65%">
</figure>
]

---

.center[
<figure>
 <img src="img/08-class/mm-fig-5-5.png" alt="Mastering Metrics Figure 5.5" title="Mastering Metrics Figure 5.5" width="65%">
</figure>
]

---

.center[
<figure>
 <img src="img/08-class/mm-fig-5-6.png" alt="Mastering Metrics Figure 5.6" title="Mastering Metrics Figure 5.6" width="65%">
</figure>
]

---

.box-inv-1.medium[MLDA reduction]

.box-1.medium[Two states: Alabama vs. Arkansas]

`$$\begin{aligned}
\text{Mortality}\ =&\ \beta_0 + \beta_1\ \text{Alabama} + \beta_2\ \text{After 1975}\ + \\
&\ \beta_3\ (\text{Alabama} \times \text{After 1975})
\end{aligned}$$`

---

.box-inv-1.medium[Organ donations]

.box-1.medium[Two states: California vs. New Jersey]

`$$\begin{aligned}
\text{Donation rate}\ =&\ \beta_0 + \beta_1\ \text{California} + \beta_2\ \text{After Q22011}\ + \\
&\ \beta_3\ (\text{California} \times \text{After Q22011})
\end{aligned}$$`

---

layout: false
name: twfe
class: center middle section-title section-title-6 animated fadeIn

# Two-way fixed effects (TWFE)

---

---

.box-6.medium[Two states: Alabama vs. Arkansas]

`$$\begin{aligned}
\text{Mortality}\ =&\ \beta_0 + \beta_1\ \text{Alabama} + \beta_2\ \text{After 1975}\ + \\
&\ \beta_3\ (\text{Alabama} \times \text{After 1975})
\end{aligned}$$`

---

.box-6.medium[All states: `Treatment == 1` if legal for 18-20-year-olds to drink]

`$$\text{Mortality}\ =\ \beta_0 + \beta_1\ \text{Treatment} + \beta_2\ \text{State} + \beta_3\ \text{Year}$$`

---

`$$\begin{aligned}
\text{Mortality}\ =&\ \beta_0 + \beta_1\ \text{Alabama} + \beta_2\ \text{After 1975}\ + \\
&\ \color{red}{\beta_3}\ (\text{Alabama} \times \text{After 1975})
\end{aligned}$$`

`$$\text{Mortality}\ =\ \beta_0 + \color{red}{\beta_1}\ \text{Treatment} + \beta_2\ \text{State} + \beta_3\ \text{Year}$$`

---

`$$\begin{aligned}
\text{Mortality}\ =&\ \beta_0 + \beta_1\ \text{Alabama} + \beta_2\ \text{After 1975}\ + \\
&\ \color{red}{\beta_3}\ (\text{Alabama} \times \text{After 1975})
\end{aligned}$$`

`$$\text{Mortality}\ =\ \beta_0 + \color{red}{\beta_1}\ \text{Treatment} + \beta_2\ \text{State} + \beta_3\ \text{Year}$$`

`$$\begin{aligned}
\text{Mortality}\ =\ & \beta_0 + \color{red}{\beta_1}\ \text{Treatment} + \beta_2\ \text{State} + \beta_3\ \text{Year}\ +\\
&\beta_4\ (\text{State} \times \text{Year})
\end{aligned}$$`

---

.center[
<figure>
 <img src="img/08-class/mm-tbl-5-2.png" alt="Mastering Metrics Table 5.2" title="Mastering Metrics Table 5.2" width="55%">
</figure>
]

---

`$$\begin{aligned}
\text{Donation rate}\ =&\ \beta_0 + \beta_1\ \text{California} + \beta_2\ \text{After Q22011}\ + \\
&\ \beta_3\ (\text{California} \times \text{After Q22011})
\end{aligned}$$`

$$
`\begin{aligned}
\text{Donation rate}\ =\ & \beta_0 + \color{red}{\beta_1}\ \text{Treatment}\ + \\
& \beta_2\ \text{State} + \beta_3\ \text{Quarter}
\end{aligned}`
$$

---

.box-6.large[What about this staggered treatment stuff?]

???

This is good for ethical reasons!

Blog post