Re-weighting the randomized controlled trial for generalization: finite-sample error and variable selection

Just published at JRSSA: An asymptotic and finite-sample sample analysis of external generalization from an RCT

A short explainer

Settings: one trial R, data from the target population T. Trial may not represent well the target population

Estimator: Reweighting (propensity score) can adjust for difference (IPSW estimator).

Question: which covariates to include in the IPSW model? Shifted covariate? Treatment-effect modifiers?

The covariable necessary for identifiability are those that are both treatment effect modifiers and shifted between the study population and target population.

But a detailed analysis of the error (bias & variance) of the ISPW reveals best choice of covariate: for reduced variance, use all treatment effect modifiers.

Detailed results

The error of the IPSW is function of n (# in trial) and m (# in target sample).

Studying the formula above shows that adding shifted variables increases dimensionality in the estimation of the propensity scores, thus inflating variance of the estimated transported effect.

Conversely, adding modifiers which are not shifted explains better the outcome as a function of the population traits, and thus decreases the variance of the estimated transported effect

This is a technical paper (it shows consistency of IPSW, which had not been established), but such mathematical work is useful to go beyond hand-waiving, and leads to very practical recommendations.

Preprint here