Outcome Modelling Methods-Applications

Hotz, V. J., Imbens, G. W., & Mortimer, J. H. (2005). Predicting the efficacy of future training programs using past experiences at other locations. Journal of Econometrics, 125(1–2), 241–270.

• We investigate the problem of predicting the average effect of a new training program using experiences with previous implementations. There are two principal complications in doing so. First, the population in which the new program will be implemented may differ from the population in which the old program was implemented. Second, the two programs may differ in the mix or nature of their components, or in their efficacy across different sub-populations.
• The first problem is similar to the problem of non-experimental evaluations. The ability to adjust for population differences typically depends on the availability of characteristics of the two populations and the extent of overlap in their distributions. The ability to adjust for differences in the programs themselves may require more detailed data on the exact treatments received by individuals than are typically available. This problem has received less attention, although it is equally important for the prediction of the efficacy of new programs.
• To investigate the empirical importance of these issues, we compare four experimental Work INcentive demonstration programs implemented in the mid-1980s in different parts of the U.S.
• We find that adjusting for pre-training earnings and individual characteristics removes many of the differences between control units that have some previous employment experience. Since the control treatment is the same in all locations, namely embargo from the program services, this suggests that differences in populations served can be adjusted for in this sub-population. We also find that adjusting for individual characteristics is more successful at removing differences between control group members in different locations that have some employment experience in the preceding four quarters than for control group members with no previous work experience. Perhaps more surprisingly, our ability to predict the outcomes of trainees after adjusting for individual characteristics is similar.
• We surmise that differences in treatment components across training programs are not sufficiently large to lead to substantial differences in our ability to predict trainees’ post-training earnings for many of the locations in this study. However, in the sub-population with no previous work experience there is some evidence that unobserved heterogeneity leads to difficulties in our ability to predict outcomes across locations for controls. [ABSTRACT FROM AUTHOR]

Hutcheon, J. A., & Liauw, J. (2022). Improving the external validity of Antenatal Late Preterm Steroids trial findings. Paediatric and Perinatal Epidemiology.

• BACKGROUND: The external validity of randomised trials can be compromised when trial participants differ from real-world populations. In the Antenatal Late Preterm Steroids (ALPS) trial of antenatal corticosteroids at late preterm ages, participants had systematically younger gestational ages than those outside the trial setting. As risk of respiratory morbidity (the primary trial outcome) is higher at younger gestations, absolute benefits of corticosteroids calculated in the trial population may overestimate real-world treatment benefits.
• OBJECTIVES: To estimate the real-world absolute risk reduction and number-needed-to-treat (NNT) for antenatal corticosteroids at late preterm ages, accounting for gestational age differences between the ALPS and real-world populations.
• METHODS: Individual participant data from the ALPS trial (which recruited 2831 women with imminent preterm birth at 34+0-36+5 weeks’) was appended to population-based data for 15,741 women admitted for delivery between 34+0 and 36+5 weeks’ from British Columbia, Canada, 2000-2013. We used logistic regression to calculate inverse odds of sampling weights for each trial participant and re-estimated treatment effects of corticosteroids on neonatal respiratory morbidity in ALPS participants, weighted to reflect the gestational age distribution of the population-based (real-world) sample.
• RESULTS: The real-world absolute risk reduction was estimated to be -2.2 (95% CI -4.6, 0.0) cases of respiratory morbidity per 100, compared with -2.8 (95% CI -5.3, -0.3) in original trial data. Corresponding NNTs were 46 in the real-world setting vs 35 in the trial. Our focus on absolute measures also highlighted that the benefits of antenatal corticosteroids may be meaningfully greater at 34 weeks vs. 36 weeks (eg risk reductions of -3.7 vs. -1.2 per 100 respectively).
• CONCLUSIONS: The absolute risk reductions and NNTs associated with antenatal corticosteroid administration at late preterm ages estimated in our study may be more appropriate for patient counselling as they better reflect the anticipated benefits of treatment when used in a real-world situation.