Transparency in Reporting

Andrade, C. (2015). Examination of participant flow in the CONSORT diagram can improve the understanding of the generalizability of study results. The Journal of Clinical Psychiatry, 76(11), e1469-1471.

• A fundamental principle in research is that the findings of a study can only be generalized to the population from which the sample of the study was drawn. What this population was can be discerned from an examination of the study selection criteria. Additional insights can sometimes be gleaned from the study flowchart or CONSORT diagram, which may show sample attenuation between subject screening and final recruitment. Such sample attenuation, if present, implies further limitation to the generalizability of the study outcomes.
• Two large, 2-year, randomized controlled maintenance therapy trials are described to illustrate sample attenuation that limits study generalizability, one in the context of mindfulness-based cognitive therapy versus antidepressant drugs for recurrent major depressive disorder and the other in the context of quetiapine versus placebo for bipolar disorder. Readers therefore need to examine both study selection criteria and the CONSORT diagram in order to better understand the extent to which study results apply to the patients whom they see.

Avellar, S. A., Thomas, J., Kleinman, R., Sama-Miller, E., Woodruff, S. E., Coughlin, R., & Westbrook, T. R. (2017). External validity: The next step for systematic reviews? Evaluation Review, 41(4), 283–325.

• Background: Systematic reviews—which identify, assess, and summarize existing research—are usually designed to determine whether research shows that an intervention has evidence of effectiveness, rather than whether an intervention will work under different circumstances. The reviews typically focus on the internal validity of the research and do not consistently incorporate information on external validity into their conclusions.
• Objectives: In this article, we focus on how systematic reviews address external validity. Methods: We conducted a brief scan of 19 systematic reviews and a more in-depth examination of information presented in a systematic review of home visiting research.
• Results: We found that many reviews do not provide information on generalizability, such as statistical representativeness, but focus on factors likely to increase heterogeneity (e.g., numbers of studies or settings) and report on context. The latter may help users decide whether the research characteristics—such as sample demographics or settings—are similar to their own. However, we found that differences in reporting, such as which variables are included and how they are measured, make it difficult to summarize across studies or make basic determinations of sample characteristics, such as whether the majority of a sample was unemployed or married.
• Conclusion: Evaluation research and systematic reviews would benefit from reporting guidelines for external validity to ensure that key information is reported across studies.

Braslow, J. T., Duan, N., Starks, S. L., Polo, A., Bromley, E., & Wells, K. B. (2005). Generalizability of studies on mental health treatment and outcomes, 1981 to 1996. Psychiatric Services (Washington, D.C.), 56(10), 1261–1268.

• OBJECTIVE: This study operationalized and measured the external validity, or generalizability, of studies on mental health treatment and outcomes published in four journals between 1981 and 1996.
• METHOD: MEDLINE was searched for articles on mental health treatment and outcomes that were published in four leading psychiatry and psychology journals between 1981 and 1996. A 156-item instrument was used to assess generalizability of study findings.
• RESULTS: Of more than 9,000 citations, 414 eligible studies were identified. Inclusion of community sites and patients from racial or ethnic minority groups were documented in only 12 and 25 percent of studies, respectively. Random or systematic sampling methods were rare (3 percent), and 75 percent of studies did not explicitly address sample representativeness. Studies with funding from the National Institute of Mental Health (NIMH) were more likely than those without NIMH funding to document the inclusion of patients from minority groups (30 percent compared with 20 percent). Randomized studies were more likely than nonrandomized studies to document the inclusion of patients from minority groups (28 percent compared with 17 percent), include patients with comorbid psychiatric conditions (31 percent compared with 19 percent), and attend to sample representativeness (28 percent compared with 15 percent). Modest improvements were seen over time in inclusion of patients from minority groups, inclusion of patients with psychiatric comorbidities, and attention to sample representativeness.
• CONCLUSIONS: Generalizability of studies on treatments and outcomes, whether experimental or observational, remained low and poorly documented over the 16-year period.

Dijkers, M. P. J. M., & SEDL, N. C. for the D. of D. R. (NCDDR). (2011). External Validity in Research on Rehabilitative Interventions: Issues for Knowledge Translation. FOCUS Technical Brief No. 33. In SEDL (SEDL. 4700 Mueller Boulevard, Austin, TX 78723. Tel: 800-476-6861; Tel: 512-476-6861; Fax: 512-476-2286; e-mail: info@sedl.org; Web site: http://www.sedl.org). SEDL; ERIC.

• This issue of “FOCUS” discusses external validity and what rehabilitation researchers can do to help practitioners answer the question “How far can we generalize this finding– is it applicable to other clients/patients, with different characteristics, in dissimilar settings treated by other clinicians?,” which clinicians and other practitioners ask whenever researchers publish evidence in support of a new or revised intervention.
• It is argued that there is no statistical basis for generalizing from the findings of a particular study, however well conducted, to a clinician’s next patient/client or even to the groups a health care policy maker is responsible for.
• Generalizing is done by the decision maker based on the similarity of the case or cases to be treated to the people who were included in the research, and based on the resemblance of the clinicians and rehabilitative settings involved to the treater(s) and setting(s) in the research. Therefore, it is the responsibility of rehabilitation researchers to provide extensive information on their intervention study in their reports, well beyond issues impacting its internal validity. Only by reading such detailed reports on the characteristics of a study’s patients/clients, treaters, treatments and outcomes, as well as on the wider health care and sociopolitical context of the study, can clinicians and their patients/clients begin to evaluate whether the same intervention might have a comparable feasibility and similar effect in their local situation.
• Unfortunately, reporting of issues relevant to external validity is much less common than of details relevant to internal validity, resulting in complaints by clinicians and other practitioners. This article provides and describes a checklist of 27 items that was designed to help rehabilitation researchers include more informative details in their research reports.

Eldridge, S., Ashby, D., Bennett, C., Wakelin, M., & Feder, G. (2008). Internal and external validity of cluster randomised trials: Systematic review of recent trials. BMJ (Clinical Research Ed.), 336(7649), 876–880.

• OBJECTIVES: To assess aspects of the internal validity of recently published cluster randomised trials and explore the reporting of information useful in assessing the external validity of these trials.
• DESIGN: Review of 34 cluster randomised trials in primary care published in 2004 and 2005 in seven journals (British Medical Journal, British Journal of General Practice, Family Practice, Preventive Medicine, Annals of Internal Medicine, Journal of General Internal Medicine, Pediatrics).
• DATA SOURCES: National Library of Medicine (Medline) via PubMed.
• DATA EXTRACTION: To assess aspects of internal validity we extracted data on appropriateness of sample size calculations and analyses, methods of identifying and recruiting individual participants, and blinding. To explore reporting of information useful in assessing external validity we extracted data on cluster eligibility, cluster inclusion and retention, cluster generalisability, and the feasibility and acceptability of the intervention to health providers in clusters.
• RESULTS: 21 (62%) trials accounted for clustering in sample size calculations and 30 (88%) in the analysis; about a quarter were potentially biased because of procedures surrounding recruitment and identification of patients; individual participants were blind to allocation status in 19 (56%) and outcome assessors were blind in 15 (44%). In almost half the reports, information relating to generalisability of clusters was poorly reported, and in two fifths there was no information about the feasibility and acceptability of the intervention.
• CONCLUSIONS: Cluster randomised trials are essential for evaluating certain types of interventions. Issues affecting their internal validity, such as appropriate sample size calculations and analysis, have been widely disseminated and are now better addressed by researchers. Blinding of those identifying and recruiting patients to allocation status is recommended but is not always carried out. There may be fewer barriers to internal validity in trials in which individual participants are not recruited. External validity seems poorly addressed in many trials, yet is arguably as important as internal validity in judging quality as a basis for healthcare intervention.

Fernandez-Hermida, J. R., Calafat, A., Becoña, E., Tsertsvadze, A., & Foxcroft, D. R. (2012). Assessment of generalizability, applicability and predictability (GAP) for evaluating external validity in studies of universal family-based prevention of alcohol misuse in young people: Systematic methodological review of randomized controlled trials. Addiction (Abingdon, England), 107(9), 1570–1579.

• AIMS: To assess external validity characteristics of studies from two Cochrane Systematic Reviews of the effectiveness of universal family-based prevention of alcohol misuse in young people.
• METHODS: Two reviewers used an a priori developed external validity rating form and independently assessed three external validity dimensions of generalizability, applicability and predictability (GAP) in randomized controlled trials.
• RESULTS: The majority (69%) of the included 29 studies were rated “unclear” on the reporting of sufficient information for judging generalizability from sample to study population. Ten studies (35%) were rated “unclear” on the reporting of sufficient information for judging applicability to other populations and settings. No study provided an assessment of the validity of the trial end-point measures for subsequent mortality, morbidity, quality of life or other economic or social outcomes. Similarly, no study reported on the validity of surrogate measures using established criteria for assessing surrogate end-points.
• CONCLUSIONS: Studies evaluating the benefits of family-based prevention of alcohol misuse in young people are generally inadequate at reporting information relevant to generalizability of the findings or implications for health or social outcomes. Researchers, study authors, peer reviewers, journal editors and scientific societies should take steps to improve the reporting of information relevant to external validity in prevention trials.

Flather, M., Delahunty, N., & Collinson, J. (2006). Generalizing results of randomized trials to clinical practice: Reliability and cautions. Clinical Trials (London, England), 3(6), 508–512.

• BACKGROUND: Well designed randomized controlled trials provide reliable evidence of treatment effects, but there is no consensus on how best to apply these results to clinical practice. The main concerns are that populations enrolled in trials are more selected than those treated in a clinical setting, and whether the treatment effects observed in trials will also be observed in clinical practice.
• METHODS: An informal literature review was undertaken to find studies analysing the issue of generalizing trial results (external validity) to clinical practice.
• RESULTS: Most of the studies focused on differences in patients characteristics (age, gender, severity of disease, concomitant treatments and so on) between the clinical trial population and a “real world” clinical population. None provided good evidence of a reduction in the treatment effect in the trial compared to what might happen in clinical practice for simple pharmacological treatments. However complex treatments like surgery or percutaneous interventional procedures, had a greater potential for variation. Extrapolating treatments to different health care settings from the trial can result in important variations in treatment effects.
• CONCLUSIONS: Complex therapies need careful consideration before they can be applied routinely from trials into practice, and applying results from one health care environment to a different one should be carried out with caution. Generalizing results from well conducted trials to clinical practice can mostly be carried out with confidence, especially for simple therapies with good evidence of benefit.

Gethin, G., Ivory, J. D., Connell, L., McIntosh, C., & Weller, C. D. (2019). External validity of randomized controlled trials of interventions in venous leg ulceration: A systematic review. Wound Repair & Regeneration, 27(6), 702–710. Academic Search Alumni Edition.

• We set out to evaluate quality of reporting of data related to external validity from randomized controlled trials (RCTs) assessing treatment interventions for active venous leg ulcers. Using a systematic review study design, we identified 144 full‐text RCTs of treatment interventions, where the wound was assessed and published in English from 1998 to 2018.
• We found that the median study sample size was 75.5. Weighted mean wound size was 13.22 cm2 and weighted mean wound duration was 22.20 months. Forty‐six (32%) reported numbers screened for eligibility and 27 (19%) reported the number who declined to participate; 19 (13%) reported on patient ethnicity; 60 (42%) reported comorbidities; and 5 (4%) reported current medication use. When reported, ⁶⁰⁄₁₀₂ (59%) excluded patients with an ankle‐brachial pressure index <0.8; ⁶⁸⁄₁₃₅ (50%) were conducted in Europe, ⁶⁄₁₃₅ (4%) in Asia, and ⁷⁴⁄₁₀₄ (71%) were conducted in outpatient facilities; 3 (2%) reported socioeconomic factors and 88 (61%) reported on adverse events.
• We concluded that there is inadequate reporting of data related to external validity in reports of RCTs assessing venous leg ulcers treatment interventions. Significant variability exists in the ankle‐brachial pressure index cutoff point for inclusion or exclusion, making generalizability difficult to assess. [ABSTRACT FROM AUTHOR]

Jonsson, U., Olsson, N. C., & Bölte, S. (2016). Can Findings from Randomized Controlled Trials of Social Skills Training in Autism Spectrum Disorder Be Generalized? The Neglected Dimension of External Validity. Autism: The International Journal of Research and Practice, 20(3), 295–305. ERIC.

• Systematic reviews have traditionally focused on internal validity, while external validity often has been overlooked. In this study, we systematically reviewed determinants of external validity in the accumulated randomized controlled trials of social skills group interventions for children and adolescents with autism spectrum disorder.
• We extracted data clustered into six overarching themes: source population, included population, context, treatment provider, treatment intervention, and outcome. A total of 15 eligible randomized controlled trials were identified. The eligible population was typically limited to high-functioning school-aged children with autism spectrum disorder, and the included population was predominantly male and Caucasian. Scant information about the recruitment of participants was provided, and details about treatment providers and settings were sparse.
• It was not evident from the trials to what extent acquired social skills were enacted in everyday life and maintained over time. We conclude that the generalizability of the accumulated evidence is unclear and that the determinants of external validity are often inadequately reported.
• At this point, more effectiveness-oriented randomized controlled trials of equally high internal and external validity are needed. More attention to the determinants of external validity is warranted when this new generation of randomized controlled trials are planned and reported. We provide a tentative checklist for this purpose.

Rothwell, P. M. (2005). External validity of randomised controlled trials:“to whom do the results of this trial apply?” The Lancet, 365(9453), 82–93.

[No Abstract]