BACKGROUND/RATIONALE:
Among the range of prevention programs in VHA, patients are likely to vary in their response to any given program. This variation is known as heterogeneity of treatment effects. To improve the effectiveness of new VA investments in prevention programs, it will be necessary to identify which Veterans have a better response to each program and which characteristics identify Veterans who might require alternative prevention programs. The purpose of this CREATE proposal is to systematically evaluate the heterogeneity of treatment effects (HTE) across the three prevention trials conducted as part of the Prevention CREATE Lab.



OBJECTIVE(S):
The purpose of the study was to determine the participant characteristics associated with response to treatment in the 3 interventions in this CREATE proposal (Aim 1) and to determine the relationship between VA expenditures and participant characteristics associated with response to treatment (Aim 2). We also proposed a secondary aim to determine the participant characteristics associated with increased per-protocol adherence in the 3 interventions. To date, no studies have applied multiple HTE approaches to the same trial and only recently have these methods been applied to behavioral intervention trials.

METHODS:
We obtained all data collected during the ACTIVATE trial (CRE 12-288) to examine HTE in the primary outcome of self-reported participation in a prevention program 6 months after enrollment via predictive risk modeling and via a data-driven approach. We also estimated 6-month VA total expenditures. In predictive risk modeling, patients are first grouped together within strata based on their risk from a pre-specified risk score (e.g., Framingham Risk Score [FRS] or an internally derived score constructed from a priori patient factors that have a plausible clinical relationship with the outcome). Treatment effects are then assessed within risk strata. This approach for describing treatment heterogeneity was first introduced in a landmark study showing that the clinical benefits accrued to patients randomized to receive carotid endarterectomy were entirely driven by 16% of the treatment group that was at highest risk for stroke.
Data-driven methods identify subgroups with similar responses to treatment whose treatment effects vary from other subgroups. These methods generally consider all available baseline covariates to classify patients into discrete, intuitive subgroups (e.g., men age >57, men 57, women age >57, women 57). Many data-driven methods are derived using statistical classification methods8 (recursive partitioning or decision trees) that are well suited to situations with many predictors with potentially complex interactions and little a priori knowledge concerning which subgroups may benefit most. The underlying search, optimization, and modeling algorithms vary by method, and, consequently, answer subtly different questions, often yielding varying results even when applied to the same dataset. For example, the simultaneous threshold interaction modeling algorithm searches for the subgroups of patients that yield the largest differential treatment effect upon the outcome; model-based recursive partitioning (MoB) searches for subgroups (defined by treatment by covariate interactions) that yield a better fitting model than the overall treatment effect model.

FINDINGS/RESULTS:
In the HTE analysis of prevention program participation, the PRM had discrimination (c-statistic) of 0.63 with 12 a priori chosen covariates, and the greatest treatment effect was in the second quartile, in which 54% (22 of 41) of intervention patients and 10% (5 of 50) of control patients reported prevention program enrollment. MoB identified 4 subgroups based on three of 28 covariates, with the greatest treatment effect among patients with lower mean numeracy, education less than a bachelor's degree, and diabetes, in which 54% (15 of 28) of intervention patients reported prevention program enrollment versus 7% (3 of 41) of control patients. The smallest effect was among those with high numeracy, with 38% (18 of 47) of intervention patients compared to 43% (23 of 53) of control patients.
In regression analysis of 6-month total VA expenditures, estimated mean expenditures were similar ($8,664 for HRA+coaching vs $9,900 for HRA-alone, p=0.25). In exploratory subgroup analysis, expenditures in the HRA+coaching group were higher than for HRA-alone among unemployed veterans with good sleep habits and fair or poor perceived health ($12,814 vs $17,318), but were lower among unemployed veterans with good sleep habits and good general health ($5,082 vs $11,612).

IMPACT:
Both preventive risk modeling and data-driven methods imply a potential decision rule for prioritizing who may benefit the most from HRA+coaching. In a setting of constrained resources, the data-driven method suggests priority should be given to patients with low numeracy, low education, and diabetes. The predictive risk model showed that those with lower overall risk of enrollment would have greater benefit from the intervention, so coaching resources should be directed to these patients. However, because risk was defined by an internally developed prediction model, identifying these patients would be more difficult to operationalize. Data-driven and predictive risk methods approach subgroup identification differently, answer related but slightly different questions, and differ in the heterogeneity observed in the effect of an intervention that combined an HRA with health coaching.
Randomized trials provide the strongest evidence about intervention effectiveness, but there is growing recognition that the average treatment effect (ATE) generated from a trial does not generalize to most patients eligible for the intervention. A principled approach to identifying heterogeneity of treatment effects (HTE) is needed. Historically, HTE was assessed by identifying subgroups stratified by one variable (e.g., male vs. female), which are easy to implement and intuitive to understand. However, this approach often does not fully characterize the multivariable risk and/or benefit of treatment; additionally, there is a risk of false negatives due to lack of statistical power in small samples and a risk of false positives as the number of stratified analyses grows. To avoid some of these pitfalls, multivariable (predictive risk, data-driven) approaches have been developed to more systematically discover and describe HTEs.

---

External Links for this Project

NIH Reporter

Grant Number: I01HX001065-01
Link: https://reporter.nih.gov/project-details/8483913

---

PUBLICATIONS:

Journal Articles

1. Olsen MK, Stechuchak KM, Hung A, Oddone EZ, Damschroder LJ, Edelman D, Maciejewski ML. A data-driven examination of which patients follow trial protocol. Contemporary clinical trials communications. 2020 Sep 1; 19:100631. [view]
2. Olsen MK, Stechuchak KM, Steinhauser KE. Comparing internal and external validation in the discovery of qualitative treatment-subgroup effects using two small clinical trials. Contemporary clinical trials communications. 2019 Sep 1; 15:100372. [view]
3. Shepherd-Banigan M, Smith VA, Maciejewski ML, Stechuchak KM, Hastings SN, Wieland GD, Miller KEM, Kabat M, Henius J, Campbell-Kotler M, Van Houtven CH. The Effect of Support and Training for Family Members on Access to Outpatient Services for Veterans with Posttraumatic Stress Disorder (PTSD). Administration and policy in mental health. 2018 Jul 1; 45(4):550-564. [view]

DRA: Health Systems
DRE: Epidemiology, Prevention
Keywords: Adherence, Best Practices, Decision-Making, Outcomes - Patient, Risk Factors, Symptom Management
MeSH Terms: none