Butterfly Effect in Studies Using Claims Data? Some Small Perturbations in Study Design Lead to Differences in Causal Inference

Assessing post-market safety of some drugs in an observational study using claims data is a mandate of the Food and Drug Administration. Study design elements, or specifications, are usually tailored to the main inference question of interest of whether a drug exposure causes an adverse outcome. Some of these specifications are general design elements in any observational study, regardless of the data source (e.g., the time frame of the study), while others are unique to claims data (e.g., the pharmacy dispensing record stockpiling algorithm). Our study investigates whether minor perturbations of specifications, coming for example from different interpretations of the same published information, greatly impact causal inference. Our investigation follows the impact of each perturbation at different stages, from cohort composition to risk assessment, in a claims-based drug safety study. The results uncover some specifications requiring precise standardized definitions in drug safety protocols with claims. Standardization will help minimize potential bias in a study and facilitate replication of study findings.