Identifying Papers Related to Generalizability
Applied researchers who aim to improve the generalizability of their impact studies would benefit from identifying relevant literature on the nature of the problem and methods for addressing it. To help applied researchers, we have assembled and lightly curated publications on generalizability that may be useful when conducting impact studies, particularly randomized trials.
We aimed to identify papers, books, and book chapters that offered:
- A conceptual understanding of generalizability, external validity, and transportability in impact studies, including the assumptions under which impact evidence will generalize.
- Quantitative evidence on the external validity, generalizability, or transportability of findings from impact studies.
- Quantitative methods for assessing or improving generalizability in impact studies, including methods for research design (e.g., sampling) and methods for data analysis (e.g., weighting and regression).
To assemble this literature, we conducted a literature search in February – April 2022. This search included:
Formal literature searches of several databases. This literature search used search strings to search databases to identify potentially relevant literature. The results from these searches were assessed for relevance by Olsen with review by Stuart.
Expert opinion. Relevant publications were identified by Olsen and Stuart based on their knowledge of the literature.
To identify relevant literature, we searched Google Scholar, Academic Search, Business Source, CINAHL, Criminal Justice, ERIC, ProQuest Central, Psychology, Sage Journals, and PubMed (see Publications – Appendix for more details).
The most effective way to identify relevant literature proved to be to search for articles with generalizability, transportability or external validity in the title and something more specific about generalizability evidence or methods in the abstract (see table below).
| Search String ID | Title Field (TI) String | Abstract Field (AB) String |
|---|---|---|
| 1 | (generaliza* or transportability or “external validity”) | (Site selection or Stratification or ((Probability or Random or Balanced or Nonprobability) and sampling)or “predictive model” or “regression trees” or “weighting” or (“propensity score” and (matching or weighting))) not “meta-analysis” |
| 2 | (generaliza* or transportability or “external validity”) | (“program evaluation” or “impact evaluation” or “impact study” or “randomized control* trial” or RCT or “randomized trial” or “randomized experiment” or “quasi-experiment” or “quasi-experimental design” or QED or “Regression Discontinuity Design” or RDD or “causal effect” or “treatment effect”) not (“meta-analysis” or “generalizability theory” or “survey experiment”) |