class: center middle main-title section-title-3 # In-person<br>session 3 .class-info[ **January 25, 2024** .light[PMAP 8521: Program evaluation<br> Andrew Young School of Policy Studies ] ] --- name: outline class: title title-inv-8 # Plan for today -- .box-1.medium[Logic models] -- .box-5.medium[Regression FAQs] -- .box-4.medium[Regression with R] --- layout: false name: logic-models class: center middle section-title section-title-1 animated fadeIn # Logic models --- layout: true class: middle --- .box-1.large[Logic models as managerial tools] --- .box-1.large[Inputs vs. Activities vs.<br>Outputs vs. Outcomes] --- .box-1.large[Impact theory vs. logic model] --- layout: false class: title title-1 # Impact theory .box-inv-1[Ensure that the theory linking activities to the outcomes is sound!] .center[ <figure> <img src="img/03/psd-impact-theory-arrow.png" alt="PSD impact theory" title="PSD impact theory" width="100%"> </figure> ] --- .center[ <figure> <img src="img/01/psd-logic-model.png" alt="PSD logic model" title="PSD logic model" width="70%"> </figure> ] --- layout: false class: title title-1 # MPA/MPP at GSU .pull-left[ <figure> <img src="img/03/mpp-mission@4x.png" alt="MPP mission" title="MPP mission" width="130%"> </figure> ] .pull-right[ <figure> <img src="img/03/mpa-mission@4x.png" alt="MPA mission" title="MPA mission" width="100%"> </figure> ] ??? Inputs: - Students - Curriculum - Money (grants, tuition) - AYSPS itself - Faculty, staff - Technology (iCollege, PAWS) - Infrastructure - Transportation - State regulations - National accreditation standards - Faculty research Activities: - Classes - Group work - Exams - Internships - Job fairs - Commencement - Studying - Tutoring - Office hours - Advising - Attendance Outputs: - Assignments and projects - Grades - Degree - Jobs - Awards - Network - Debt Outcomes: - Leadership - AYS national ranking - Public service motivation - Critical thinking - Efficient workforce - Income - Better society --- layout: true class: middle --- .box-1.medium[Isn't it best to always<br>have an articulated theory?] .box-1.medium[Should implicit theory and articulated theory<br>be the same thing in most cases?] --- .box-1.medium[How much does this evaluation stuff cost?] .box-1.medium[Can you do scaled-down versions<br>of these evaluations?] --- .box-1.medium[What if a program exists already<br>and doesn't have a logic model?] --- .box-1.medium[Why would a program aim for final outcomes<br>that can't be measured?] --- .box-1.medium[What should you do if you find that your<br>theory of change (or logic model in general)<br>is wrong in the middle of the program?<br>Is it ethical to stop or readjust?] --- layout: false name: regression-faqs class: center middle section-title section-title-5 animated fadeIn # Regression FAQs --- layout: true class: middle --- .box-5.large.sp-after[Drawing lines through points] -- .center.sp-after[<https://evalsp24.classes.andrewheiss.com/slides/02-slides.html#17>] -- .box-5[<span style="color: #F6D645;">Lo</span>cally <span style="color: #F6D645;">e</span>stimated/<span style="color: #F6D645;">we</span>ighted <span style="color: #F6D645;">s</span>catterplot <span style="color: #F6D645;">s</span>moothing (LOESS/LOWESS)<br>is a common method (but not the only one!)] --- .center[ <video controls> <source src="img/10/loess_window.mp4" type="video/mp4"> </video> ] --- .box-5.large[Regression equations] .box-inv-5[And is the intercept ever useful,<br>or should we always ignore it?] ??? <https://evalsp24.classes.andrewheiss.com/slides/02-slides.html#38> <https://www.andrewheiss.com/blog/2022/05/20/marginalia/#regression-sliders-switches-and-mixing-boards> --- .box-5.medium[What does it mean to<br>hold something constant?] .box-5.medium[Why is one category always left out<br>when you use a categorical variable?] --- .center[ <figure> <img src="img/02-class/slider-switch-annotated-80.jpg" alt="Single slider and switch" title="Single slider and switch" width="100%"> </figure> ] --- .center[ <figure> <img src="img/02-class/mixer-board-annotated-80.jpg" alt="Mixer board" title="Mixer board" width="100%"> </figure> ] --- .box-5.medium.sp-after[Why use two steps to create a regression in R?<br>(i.e. assigning it to an object with `<-`?)] .box-5.medium[Why use `tidy()`<br>from the broom package?] ??? Show model with `lm()`; show t-test with `t.test()`; show both through `tidy()` Use **marginaleffects** --- .box-5.medium[How was the 0.05 significance<br>threshold determined?] .box-5.medium[Could we say something is significant<br>if p > 0.05, but just note that it is at<br>a higher p-value?<br>Or does it have to fall under 0.05?] --- .box-5.large[Why all this convoluted<br>logic of null worlds?] --- .center[ <figure> <img src="img/02-class/oatmeal-ratings.png" alt="Oatmeal ratings" title="Oatmeal ratings" width="80%"> </figure> ] --- .center[ <figure> <img src="img/02-class/oatmeal-ratings-1.png" alt="Oatmeal ratings" title="Oatmeal ratings" width="80%"> </figure> ] --- layout: false class: title title-5 # Different "dialects" of statistics .pull-left[ .box-5.medium[Frequentist] `$$P(\text{data} \mid H_0)$$` .box-inv-5["Regular" statistics;<br>what you've learned<br><small>(and are learning here)</small>] ] -- .pull-right[ .box-5.medium[Bayesian] `$$P(H \mid \text{data})$$` .box-inv-5[Requires lots of<br>computational power] ] --- layout: true class: middle --- .box-5.medium[Do we care about the actual coefficients<br>or just whether or not they're significant?] .box-5.medium[How does significance relate to causation?] .box-5.medium[If we can't use statistics to assert causation<br>how are we going to use this information<br>in program evaluation?] --- .box-5.large[What counts as a "good" R²?] --- .center[ <figure> <img src="img/02-class/nice-plot-1.png" alt="Euler diagram" title="Euler diagram" width="45%"> </figure> ] --- .center[ <figure> <img src="img/02-class/plot-diagram-prediction-1.png" alt="R2 prediction" title="R2 prediction" width="75%"> </figure> ] --- .center[ <figure> <img src="img/02-class/plot-diagram-estimation-1.png" alt="R2 estimation" title="R2 estimation" width="75%"> </figure> ] --- .center[ <figure> <img src="img/02-class/prediction-vs-estimation.jpg" alt="R2 estimation vs prediction" title="R2 estimation vs prediction" width="100%"> </figure> ] --- class: middle .box-5.large[Side-by-side regression tables] --- <table style="font-size:0.5em;border-bottom: 0; width: auto !important; margin-left: auto; margin-right: auto;" class="table"> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:center;">  (1) </th> <th style="text-align:center;">   (2) </th> <th style="text-align:center;">   (3) </th> <th style="text-align:center;">   (4) </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> (Intercept) </td> <td style="text-align:center;"> 362.307 </td> <td style="text-align:center;"> −5780.831*** </td> <td style="text-align:center;"> −5736.897*** </td> <td style="text-align:center;"> −5433.534*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (283.345) </td> <td style="text-align:center;"> (305.815) </td> <td style="text-align:center;"> (307.959) </td> <td style="text-align:center;"> (286.558) </td> </tr> <tr> <td style="text-align:left;"> bill_length_mm </td> <td style="text-align:center;"> 87.415*** </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 6.047 </td> <td style="text-align:center;"> −5.201 </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (6.402) </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> (5.180) </td> <td style="text-align:center;"> (4.860) </td> </tr> <tr> <td style="text-align:left;"> flipper_length_mm </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 49.686*** </td> <td style="text-align:center;"> 48.145*** </td> <td style="text-align:center;"> 48.209*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> (1.518) </td> <td style="text-align:center;"> (2.011) </td> <td style="text-align:center;"> (1.841) </td> </tr> <tr> <td style="text-align:left;"> sexmale </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 358.631*** </td> </tr> <tr> <td style="text-align:left;box-shadow: 0px 1.5px"> </td> <td style="text-align:center;box-shadow: 0px 1.5px"> </td> <td style="text-align:center;box-shadow: 0px 1.5px"> </td> <td style="text-align:center;box-shadow: 0px 1.5px"> </td> <td style="text-align:center;box-shadow: 0px 1.5px"> (41.572) </td> </tr> <tr> <td style="text-align:left;"> Num.Obs. </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 333 </td> </tr> <tr> <td style="text-align:left;"> R2 </td> <td style="text-align:center;"> 0.354 </td> <td style="text-align:center;"> 0.759 </td> <td style="text-align:center;"> 0.760 </td> <td style="text-align:center;"> 0.807 </td> </tr> <tr> <td style="text-align:left;"> R2 Adj. </td> <td style="text-align:center;"> 0.352 </td> <td style="text-align:center;"> 0.758 </td> <td style="text-align:center;"> 0.759 </td> <td style="text-align:center;"> 0.805 </td> </tr> <tr> <td style="text-align:left;"> AIC </td> <td style="text-align:center;"> 5400.0 </td> <td style="text-align:center;"> 5062.9 </td> <td style="text-align:center;"> 5063.5 </td> <td style="text-align:center;"> 4863.3 </td> </tr> <tr> <td style="text-align:left;"> BIC </td> <td style="text-align:center;"> 5411.5 </td> <td style="text-align:center;"> 5074.4 </td> <td style="text-align:center;"> 5078.8 </td> <td style="text-align:center;"> 4882.4 </td> </tr> <tr> <td style="text-align:left;"> Log.Lik. </td> <td style="text-align:center;"> −2696.987 </td> <td style="text-align:center;"> −2528.427 </td> <td style="text-align:center;"> −2527.741 </td> <td style="text-align:center;"> −2426.664 </td> </tr> <tr> <td style="text-align:left;"> F </td> <td style="text-align:center;"> 186.443 </td> <td style="text-align:center;"> 1070.745 </td> <td style="text-align:center;"> 536.626 </td> <td style="text-align:center;"> 457.118 </td> </tr> <tr> <td style="text-align:left;"> RMSE </td> <td style="text-align:center;"> 643.54 </td> <td style="text-align:center;"> 393.12 </td> <td style="text-align:center;"> 392.34 </td> <td style="text-align:center;"> 353.66 </td> </tr> </tbody> <tfoot><tr><td style="padding: 0; " colspan="100%">  + p &lt; 0.1, * p &lt; 0.05, ** p &lt; 0.01, *** p &lt; 0.001</td></tr></tfoot> </table> --- <table style="NAborder-bottom: 0; width: auto !important; margin-left: auto; margin-right: auto;" class="table"> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:center;">  (1) </th> <th style="text-align:center;">   (2) </th> <th style="text-align:center;">   (3) </th> <th style="text-align:center;">   (4) </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> (Intercept) </td> <td style="text-align:center;"> 362.307 </td> <td style="text-align:center;"> −5780.831*** </td> <td style="text-align:center;"> −5736.897*** </td> <td style="text-align:center;"> −5433.534*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (283.345) </td> <td style="text-align:center;"> (305.815) </td> <td style="text-align:center;"> (307.959) </td> <td style="text-align:center;"> (286.558) </td> </tr> <tr> <td style="text-align:left;"> bill_length_mm </td> <td style="text-align:center;"> 87.415*** </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 6.047 </td> <td style="text-align:center;"> −5.201 </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> (6.402) </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> (5.180) </td> <td style="text-align:center;"> (4.860) </td> </tr> <tr> <td style="text-align:left;"> flipper_length_mm </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 49.686*** </td> <td style="text-align:center;"> 48.145*** </td> <td style="text-align:center;"> 48.209*** </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> (1.518) </td> <td style="text-align:center;"> (2.011) </td> <td style="text-align:center;"> (1.841) </td> </tr> <tr> <td style="text-align:left;"> sexmale </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> 358.631*** </td> </tr> <tr> <td style="text-align:left;box-shadow: 0px 1.5px"> </td> <td style="text-align:center;box-shadow: 0px 1.5px"> </td> <td style="text-align:center;box-shadow: 0px 1.5px"> </td> <td style="text-align:center;box-shadow: 0px 1.5px"> </td> <td style="text-align:center;box-shadow: 0px 1.5px"> (41.572) </td> </tr> <tr> <td style="text-align:left;"> Num.Obs. </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 342 </td> <td style="text-align:center;"> 333 </td> </tr> <tr> <td style="text-align:left;"> R2 </td> <td style="text-align:center;"> 0.354 </td> <td style="text-align:center;"> 0.759 </td> <td style="text-align:center;"> 0.760 </td> <td style="text-align:center;"> 0.807 </td> </tr> <tr> <td style="text-align:left;"> R2 Adj. </td> <td style="text-align:center;"> 0.352 </td> <td style="text-align:center;"> 0.758 </td> <td style="text-align:center;"> 0.759 </td> <td style="text-align:center;"> 0.805 </td> </tr> <tr> <td style="text-align:left;"> AIC </td> <td style="text-align:center;"> 5400.0 </td> <td style="text-align:center;"> 5062.9 </td> <td style="text-align:center;"> 5063.5 </td> <td style="text-align:center;"> 4863.3 </td> </tr> <tr> <td style="text-align:left;"> BIC </td> <td style="text-align:center;"> 5411.5 </td> <td style="text-align:center;"> 5074.4 </td> <td style="text-align:center;"> 5078.8 </td> <td style="text-align:center;"> 4882.4 </td> </tr> <tr> <td style="text-align:left;"> Log.Lik. </td> <td style="text-align:center;"> −2696.987 </td> <td style="text-align:center;"> −2528.427 </td> <td style="text-align:center;"> −2527.741 </td> <td style="text-align:center;"> −2426.664 </td> </tr> <tr> <td style="text-align:left;"> F </td> <td style="text-align:center;"> 186.443 </td> <td style="text-align:center;"> 1070.745 </td> <td style="text-align:center;"> 536.626 </td> <td style="text-align:center;"> 457.118 </td> </tr> <tr> <td style="text-align:left;"> RMSE </td> <td style="text-align:center;"> 643.54 </td> <td style="text-align:center;"> 393.12 </td> <td style="text-align:center;"> 392.34 </td> <td style="text-align:center;"> 353.66 </td> </tr> </tbody> <tfoot><tr><td style="padding: 0; " colspan="100%">  + p &lt; 0.1, * p &lt; 0.05, ** p &lt; 0.01, *** p &lt; 0.001</td></tr></tfoot> </table> --- class: middle .box-5.large[See full documentation and<br>examples for `modelsummary()` [here](https://vincentarelbundock.github.io/modelsummary/articles/modelsummary.html)] --- class: middle .box-5.large[Make nicer tables [with {gt}](https://gt.rstudio.com/)] --- layout: false name: regression class: center middle section-title section-title-4 animated fadeIn # Regression with R