self controlled case study

• Journal home
• Advance online publication
• About the journal
• J-STAGE home
• Annals of Clinical Epidemiolog ...
• Volume 3 (2021) Issue 3
• Article overview

Department of Health Services Research, Faculty of Medicine, University of Tsukuba Faculty of Epidemiology and Population Health, London School of Hygiene and Tropical Medicine

Department of Biostatistics, School of Public Health, Graduate School of Medicine, The University of Tokyo

Corresponding author

2021 Volume 3 Issue 3 Pages 67-73

• Published: 2021 Received: - Available on J-STAGE: July 01, 2021 Accepted: - Advance online publication: - Revised: -

(compatible with EndNote, Reference Manager, ProCite, RefWorks)

(compatible with BibDesk, LaTeX)

Self-controlled study designs, also known as case-only designs or Self-controlled Crossover Observational PharmacoEpidemiologic (SCOPE) studies, include case-crossover (CCO) and self-controlled case series (SCCS). These designs compare different time windows (i.e., lengths of time) within the same person. An SCCS compares the occurrence of an outcome (event) during periods with and without exposure in the same person, whereas a CCO compares periods with and without the outcome for exposure. The main strength of self-controlled study designs is that they can ignore confounding factors that do not change over time (e.g., sex, genetics, habitual healthy or unhealthy behaviors). The effect of these factors are canceled out through statistical analyses, even if they are unknown or unmeasured. However, self-controlled study designs cannot be used for all research questions. Assumptions specific to each study design are needed. In CCO, there should be no substantial changes in exposure trends during the study period, the exposure should be transient (intermittent), and the outcome should be abrupt (sudden). In SCCS, event rates should be constant within each defined period and events must be independently recurrent or rare. In addition, the occurrence of an event should not affect subsequent exposures. Self-controlled study designs may be particularly useful in studies using electronic health records, in which some (time-invariant) confounding factors may not have been recorded, provided that the research question meets the assumptions required for each study design.

First, we will briefly review traditional epidemiologic designs and then introduce self-controlled study designs to aid a better understanding of them.

Fig. 1 Graphical representation of cohort and case–control studies for an association between benzodiazepine and hip fracture

1.  Requena G, Huerta C, Gardarsdottir H, Logie J, González-González R, Abbing-Karahagopian V, et al. Hip/Femur Fractures Associated With The Use Of Benzodiazepines (Anxiolytics, Hypnotics And Related Drugs): A Methodological Approach To Assess Consistencies Across Databases From The PROTECT-EU Project. Pharmacoepidemiol Drug Saf 2016;25:66–78.

Fig. 2 Graphical representation of self-controlled case series and case-crossover designs for an association between benzodiazepine and hip fracture

2.  Requena G, Logie J, Martin E, Boudiaf N, González González R, Huerta C, et al. Do Case-Only Designs Yield Consistent Results Across Design And Different Databases? A Case Study Of Hip Fractures And Benzodiazepines. Pharmacoepidemiol Drug Saf 2016;25:79–87.

Traditional observational study designs and self-controlled study designs

3.  Cadarette SM, Maclure M, Delaney JAC, Whitaker HJ, Hayes KN, Wang SV, et al. Control Yourself: ISPE-Endorsed Guidance In The Application Of Self-Controlled Study Designs In Pharmacoepidemiology. Pharmacoepidemiol Drug Saf 2021;30:671–684.

4.  Nordmann S, Biard L, Ravaud P, Esposito-Farèse M, Tubach F. Case-Only Designs In Pharmacoepidemiology: A Systematic Review. PLoS One 2012;7:e49444.

An advantage of self-controlled study designs is that they can ignore factors that do not change throughout the study period (e.g., sex, risk genes, healthy or unhealthy behaviors within a short follow-up length) in the statistical analyses. Their effects are canceled out, even if they are unknown or unmeasured. However, statistical adjustments are still necessary for factors that change over time within an individual (e.g., age, some medications).

Despite the value of self-controlled study designs, caution is needed because they cannot be used for all research questions. CCO and SCCS designs require several assumptions specific to each study design (as explained later) to estimate the effect of an exposure on an outcome (e.g., odds ratio, rate ratio). When these assumptions are violated, self-control study designs result in biased estimates.

5.  Maclure M. The Case-Crossover Design: A Method For Studying Transient Effects On The Risk Of Acute Events. Am J Epidemiol 1991;133:144–153.

6.  Carracedo-Martínez E, Taracido M, Tobias A, Saez M, Figueiras A. Case-Crossover Analysis Of Air Pollution Health Effects: A Systematic Review Of Methodology And Application. Environ Health Perspect 2010;118:1173–1182.

7.  Consiglio GP, Burden AM, Maclure M, McCarthy L, Cadarette SM. Case-Crossover Study Design In Pharmacoepidemiology: Systematic Review And Recommendations. Pharmacoepidemiol Drug Saf 2013;22:1146–1153.

First, people with the outcome of interest should be identified. The index date is defined as the timing (e.g., day) when the first outcome occurred. Then, the time window (i.e., lengths of time) of the “case period” before the outcome occurred should be determined, assuming that the exposure during that period could cause the outcome. Then, one or several “control period(s)”, which are typically prior to the “case period” for the same person, should be defined. In a CCO, researchers can freely define the “control period(s)”, such as the number and length of “control period(s)” and length of interval between “case period” and “control period(s)”. Thus, showing how the results change using several different definitions of “control periods” in the sensitivity analyses is recommended. Finally, the “case period” and “control period(s)” should be compared for the odds of (presence of) exposure to estimate the odds ratio.

8.  Mostofsky E, Coull BA, Mittleman MA. Analysis of Observational Self-matched Data to Examine Acute Triggers of Outcome Events with Abrupt Onset. Epidemiology 2018;29:804–816.

9.  Suissa S. The Case-Time-Control Design. Epidemiology 1995;6:248–253.

10.  Wang S, Linkletter C, Maclure M, Dore D, Mor V, Buka S, et al. Future Cases As Present Controls To Adjust For Exposure Trend Bias In Case-Only Studies. Epidemiology 2011;22:568–574.

11.  Miyamoto Y, Iwagami M, Aso S, Matsui H, Doi K, Yasunaga H. Pregabalin And Injury: A Nested Case-Control And Case-Crossover Study. Pharmacoepidemiol Drug Saf 2020;29:558–564.

Fig. 3 Graphical representation of case-crossover design for the association between pregabalin prescription and injury

12.  Farrington CP. Relative Incidence Estimation From Case Series For Vaccine Safety Evaluation. Biometrics 1995;51:228–235.

13.  Gault N, Castañeda-Sanabria J, De Rycke Y, Guillo S, Foulon S, Tubach F. Self-Controlled Designs In Pharmacoepidemiology Involving Electronic Healthcare Databases: A Systematic Review. BMC Med Res Methodol 2017;17:25.

14.  Kwong JC, Schwartz KL, Campitelli MA, Chung H, Crowcroft NS, Karnauchow T, et al. Acute Myocardial Infarction After Laboratory-Confirmed Influenza Infection. N Engl J Med 2018;378:345–353.

15.  Shimada YJ, Tsugawa Y, Brown DFM, Hasegawa K. Bariatric Surgery And Emergency Department Visits And Hospitalizations For Heart Failure Exacerbation: Population-Based, Self-Controlled Series. J Am Coll Cardiol 2016;67:895–903.

First, people with the outcome of interest should be identified. Then, the study periods for cases should be defined arbitrarily (e.g., from January 2018 to December 2020, from 1 year before the event date to 1 year after the event date). The study periods of individuals are split into a “risk period(s)” (during which the risk of outcome is assumed to be increased because of the exposure) and a “baseline period(s)” for the remaining study periods. A person can have two or more “risk periods” if they are exposed several times during the study period. Finally, the “risk period(s)” are compared with “baseline period(s)” to determine the incidence of the outcome to estimate an incidence rate ratio. In an SCCS, there may be two or more occurrences of outcome in the same person.

16.  Petersen I, Douglas I, Whitaker H. Self Controlled Case Series Methods: An Alternative To Standard Epidemiological Study Designs. BMJ 2016;354:i4515.

17.  Ohbe H, Iwagami M, Sasabuchi Y, Yasunaga H. Increased Risk Of Infective Endocarditis After Traumatic Skin Wound. Heart 2021:heartjnl-2020-318632.

Fig. 4 Graphical representation of a self-controlled case series for the association between traumatic skin wound and the first incidence of infective endocarditis

18.  Hallas J. Evidence Of Depression Provoked By Cardiovascular Medication: A Prescription Sequence Symmetry Analysis. Epidemiology 1996;7:478–484.

19.  Takeuchi Y, Shinozaki T, Matsuyama Y. A Comparison Of Estimators From Self-Controlled Case Series, Case-Crossover Design, And Sequence Symmetry Analysis For Pharmacoepidemiological Studies. BMC Med Res Methodol 2018;18:4.

20.  Takeuchi Y, Kajiyama K, Ishiguro C, Uyama Y. Atypical Antipsychotics And The Risk Of Hyperlipidemia: A Sequence Symmetry Analysis. Drug Saf 2015;38:641–650.

21.  Pratt N, Chan EW, Choi NK, Kimura M, Kimura T, Kubota K, et al. Prescription Sequence Symmetry Analysis: Assessing Risk, Temporality, And Consistency For Adverse Drug Reactions Across Datasets In Five Countries. Pharmacoepidemiol Drug Saf 2015;24:858–864.

Fig. 5 Graphical representation of sequence symmetry analysis for the association between benzodiazepine and hip fracture

We provided an overview and some examples of self-controlled study designs, including CCO and SCCS designs. The main strength of self-controlled study designs, compared with traditional epidemiologic designs (cohort and case–control studies), is that time-invariant confounding factors can be canceled out, even if they are unknown or unmeasured. However, self-controlled study designs should be used carefully. If the aforementioned assumptions are not met, the results can be biased. Self-controlled study designs may be particularly useful in studies using electronic health records, in which some (time-invariant) confounding factors may not be recorded, provided that the research question meets assumptions required for each study design.

We would like to thank Dr. Yoshihisa Miyamoto in the Epidemiology and Prevention Group, Center for Public Health Sciences, National Cancer Centre, and Dr. Hiroyuki Ohbe in the Department of Clinical Epidemiology and Health Economics, School of Public Health, The University of Tokyo, for their critical reading of the manuscript and feedback. The English editing of the current paper was supported by a grant-in-aid from the Ministry of Health, Health, Labour and Welfare Policy Research Grants, Japan; (19AA2007).

Y.T. has received consultant fees from Pharmaceuticals and Medical Devices Agency and EPARK, Inc. Additionally, Y.T. has conducted a collaborative study with Pfizer inc., which is not associated with this paper. No other potential competing interests relevant to this paper are reported.

• 1.    Requena  G,  Huerta  C,  Gardarsdottir  H,  Logie  J,  González-González  R,  Abbing-Karahagopian  V, et al. Hip/Femur Fractures Associated With The Use Of Benzodiazepines (Anxiolytics, Hypnotics And Related Drugs): A Methodological Approach To Assess Consistencies Across Databases From The PROTECT-EU Project. Pharmacoepidemiol Drug Saf 2016; 25 :66–78.
• 2.    Requena  G,  Logie  J,  Martin  E,  Boudiaf  N,  González González  R,  Huerta  C, et al. Do Case-Only Designs Yield Consistent Results Across Design And Different Databases? A Case Study Of Hip Fractures And Benzodiazepines. Pharmacoepidemiol Drug Saf 2016; 25 :79–87.
• 3.    Cadarette  SM,  Maclure  M,  Delaney  JAC,  Whitaker  HJ,  Hayes  KN,  Wang  SV, et al. Control Yourself: ISPE-Endorsed Guidance In The Application Of Self-Controlled Study Designs In Pharmacoepidemiology. Pharmacoepidemiol Drug Saf 2021; 30 :671–684.
• 4.    Nordmann  S,  Biard  L,  Ravaud  P,  Esposito-Farèse  M,  Tubach  F. Case-Only Designs In Pharmacoepidemiology: A Systematic Review. PLoS One 2012; 7 :e49444.
• 5.    Maclure  M. The Case-Crossover Design: A Method For Studying Transient Effects On The Risk Of Acute Events. Am J Epidemiol 1991; 133 :144–153.
• 6.    Carracedo-Martínez  E,  Taracido  M,  Tobias  A,  Saez  M,  Figueiras  A. Case-Crossover Analysis Of Air Pollution Health Effects: A Systematic Review Of Methodology And Application. Environ Health Perspect 2010; 118 :1173–1182.
• 7.    Consiglio  GP,  Burden  AM,  Maclure  M,  McCarthy  L,  Cadarette  SM. Case-Crossover Study Design In Pharmacoepidemiology: Systematic Review And Recommendations. Pharmacoepidemiol Drug Saf 2013; 22 :1146–1153.
• 8.    Mostofsky  E,  Coull  BA,  Mittleman  MA. Analysis of Observational Self-matched Data to Examine Acute Triggers of Outcome Events with Abrupt Onset. Epidemiology 2018; 29 :804–816.
• 9.    Suissa  S. The Case-Time-Control Design. Epidemiology 1995; 6 :248–253.
• 10.    Wang  S,  Linkletter  C,  Maclure  M,  Dore  D,  Mor  V,  Buka  S, et al. Future Cases As Present Controls To Adjust For Exposure Trend Bias In Case-Only Studies. Epidemiology 2011; 22 :568–574.
• 11.    Miyamoto  Y,  Iwagami  M,  Aso  S,  Matsui  H,  Doi  K,  Yasunaga  H. Pregabalin And Injury: A Nested Case-Control And Case-Crossover Study. Pharmacoepidemiol Drug Saf 2020; 29 :558–564.
• 12.    Farrington  CP. Relative Incidence Estimation From Case Series For Vaccine Safety Evaluation. Biometrics 1995; 51 :228–235.
• 13.    Gault  N,  Castañeda-Sanabria  J,  De Rycke  Y,  Guillo  S,  Foulon  S,  Tubach  F. Self-Controlled Designs In Pharmacoepidemiology Involving Electronic Healthcare Databases: A Systematic Review. BMC Med Res Methodol 2017; 17 :25.
• 14.    Kwong  JC,  Schwartz  KL,  Campitelli  MA,  Chung  H,  Crowcroft  NS,  Karnauchow  T, et al. Acute Myocardial Infarction After Laboratory-Confirmed Influenza Infection. N Engl J Med 2018; 378 :345–353.
• 15.    Shimada  YJ,  Tsugawa  Y,  Brown  DFM,  Hasegawa  K. Bariatric Surgery And Emergency Department Visits And Hospitalizations For Heart Failure Exacerbation: Population-Based, Self-Controlled Series. J Am Coll Cardiol 2016; 67 :895–903.
• 16.    Petersen  I,  Douglas  I,  Whitaker  H. Self Controlled Case Series Methods: An Alternative To Standard Epidemiological Study Designs. BMJ 2016; 354 :i4515.
• 17.    Ohbe  H,  Iwagami  M,  Sasabuchi  Y,  Yasunaga  H. Increased Risk Of Infective Endocarditis After Traumatic Skin Wound. Heart 2021:heartjnl-2020-318632.
• 18.    Hallas  J. Evidence Of Depression Provoked By Cardiovascular Medication: A Prescription Sequence Symmetry Analysis. Epidemiology 1996; 7 :478–484.
• 19.    Takeuchi  Y,  Shinozaki  T,  Matsuyama  Y. A Comparison Of Estimators From Self-Controlled Case Series, Case-Crossover Design, And Sequence Symmetry Analysis For Pharmacoepidemiological Studies. BMC Med Res Methodol 2018; 18 :4.
• 20.    Takeuchi  Y,  Kajiyama  K,  Ishiguro  C,  Uyama  Y. Atypical Antipsychotics And The Risk Of Hyperlipidemia: A Sequence Symmetry Analysis. Drug Saf 2015; 38 :641–650.
• 21.    Pratt  N,  Chan  EW,  Choi  NK,  Kimura  M,  Kimura  T,  Kubota  K, et al. Prescription Sequence Symmetry Analysis: Assessing Risk, Temporality, And Consistency For Adverse Drug Reactions Across Datasets In Five Countries. Pharmacoepidemiol Drug Saf 2015; 24 :858–864.

Register with J-STAGE for free!

Already have an account? Sign in here

• Open access
• Published: 23 November 2021

Randomized and non-randomized designs for causal inference with longitudinal data in rare disorders

• Rima Izem   ORCID: orcid.org/0000-0002-4107-9793 1 &
• Robert McCarter 1  

Orphanet Journal of Rare Diseases volume  16 , Article number:  491 ( 2021 ) Cite this article

2894 Accesses

4 Citations

3 Altmetric

Metrics details

In the United States, approximately 7000 rare diseases affect 30 million patients, and only 10% of these diseases have existing therapies. Sound study design and causal inference methods are essential to demonstrate the therapeutic efficacy, safety, and effectiveness of new therapies. In the rare diseases setting, several factors challenge the use of typical parallel control designs: the small patient population size, genotypic and phenotypic diversity, and the complexity and incomplete understanding of the disorder’s progression. Repeated measures, when spaced appropriately relative to disease progression and exploited in design and analysis, can increase study power and reduce variability in treatment effect estimation. This paper reviews these longitudinal designs and draws the parallel between some new and existing randomized studies in rare diseases and their less well-known controlled observational study designs. We show that self-controlled randomized crossover and N-of-1 designs have similar considerations as the observational case series and case-crossover designs. Also, randomized sequential designs have similar considerations to longitudinal cohort studies using sequential matching or weighting to control confounding. We discuss design and analysis considerations for valid causal inference and illustrate them with examples of analyses in multiple rare disorders, including urea cycle disorder and cystic fibrosis.

Introduction

Millions are affected by rare disorders and have an urgent need for therapy to save or improve their quality of life. Over 7000 rare diseases, disorders, illnesses, or conditions impact 25 to 30 million people in the United States, most of them are children [ 1 ]. Many rare diseases cause death in infancy or early childhood, and about 90% have no approved treatment [ 2 ]. Despite the great need for effective therapies, evaluating evidence of efficacy and safety of novel therapies in the rare disease setting is challenging in the typical paradigm of parallel-group randomized studies [ 3 , 4 ]. In recognizing these challenges and the need for novel designs the US Food and Drug administration published several guidance documents in 2019 [ 5 , 6 , 7 ].

This paper gives an overview of study designs that rely on or exploit repeated measures for causal inference. These randomized or observational designs, when appropriately conducted and used, can alleviate some challenges in estimating treatment effects in rare disorders. Moreover, this review draws a parallel between randomized longitudinal designs and their less well known controlled observational studies counterpart. While randomized studies sit atop of the hierarchy of evidence, rigorously designed, well-controlled, and analyzed observational studies using causal inference methods can be adequate to assess the safety and effectiveness [ 8 ] of therapies in rare disorders [ 8 ]. In this paper,  the treatment effect is the measure of efficacy or safety in a randomized clinical trial and the measure of effectiveness or safety in a comparative observational study.

When possible, we will illustrate different considerations with existing clinical trials or observational studies in rare disorders, most often with urea cycle disorders (UCD) and cystic fibrosis (CF) disorders, as they represent a broad spectrum of rare disorders from ultra-rare to rare.

The case for longitudinal studies in rare disorders

We focus in this section on two challenges particularly salient in clinical study design in rare disorders: small population size and heterogeneity of clinical outcomes. Then, we discuss how planning and using repeated assessments of these outcomes has the potential to alleviate these challenges.

Challenges in clinical development in rare disorders related to sample size and heterogeneity of outcomes

Designing studies with sufficient statistical power to evaluate treatment effect is challenging in rare diseases because of limited population size and high variability of outcomes. Clinical study sample sizes in rare diseases typically range from a handful of subjects to less than a few hundreds, based on disease prevalence [ 9 ]. The prevalence threshold defining ‘rare’ varies by country or organization and is in the range of 40 to 50 cases per 100,000 people [ 10 ]. This range includes incidences of CF on the higher end, with one case for every 3000–4000 births among Caucasians [ 11 ], and of ultra-rare UCD mutations on the lower end, with one case in a million births [ 12 ].

Several factors contribute to between-subject variability in outcome measurements in clinical studies. First, at any given time potential participants in clinical studies represent a cross-section of the population varying in ages and stage of disease progression. Even if age is controlled in the study, the timing of diagnosis often varies with implications for variability of medical history. For example, whereas CF is likely to be diagnosed at birth, as part of newborn screening [ 13 ], the timing of diagnosis of UCD ranges from the first few days after birth to adulthood. The timing varies due to multiple factors, including family medical history (e.g., diagnosis of family members), and varying age of presentation of symptoms such as hospitalization for hyperammonemia. Treatment of UCD disorders starts after diagnosis, and delay in therapy has downstream effects on the brain and general health.

Also, variability in genetic, geographic, and environmental factors contributes to heterogeneity in clinical outcomes. In most rare disorders, a mutation disrupts a biochemical pathway, leading to various signs and symptoms downstream. The extent and clinical impact of these disruptions vary by mutation and environmental factors. For example, some genetic mutations in the UCD reduce while others eliminate enzyme activities related to nitrogen waste disposal in the urea cycle [ 14 , 15 , 16 ]. For the same mutation, disease progression varies by several geographic and calendar time factors including age at diagnosis, medical practice, diet, and access to health services. Thus, the clinical outcome in UCD vary in severity and over time from death shortly after birth, to different degrees of physical or cognitive impairment, to no symptoms until later in life.

Lastly, varying instruments used to assess biological, motor, or mental function across age groups can add, for each endpoint, between-instrument variability or instrument-specific measurement errors. For example, the UCD longitudinal study measures intelligence with several instruments, including the Wechsler preschool and primary scale intelligence [ 17 ] and the Wechsler Intelligence Scale for Children [ 18 ]. While age-sex standardization of each score ensures internal consistency, it does not guarantee comparability of scores between these two instruments in a study with preschoolers and adolescents.

The benefits of designs with repeated measures

The role of rare disorder natural history studies or registries in informing clinical trials is well established [ 7 ]. Understanding the symptomatology and management of a rare disease over time and its natural history, informs multiple critical study design attributes. These attributes include the population inclusion and exclusion criteria, the study endpoints, and the times of initiation of a new therapy. They also inform the pre-specification of meaningful treatment effect size, frequency and timing of outcome assessments, and potential duration of follow-up. For example, the US CF Foundation registry has over 30,000 subjects, with extended follow-up for up to 20 years since 1986 [ 19 ]. The UCD consortium has also collected rich longitudinal data since 1996 with historical and prospective data on over 800 subjects covering the period from birth to adulthood [ 20 ]. As annual reports from these registries indicate, these epidemiologic natural history studies have informed the design of multiple prospectively planned studies investigating new therapies in CF and UCD.

Beyond understanding the natural history, longitudinal data collection or repeated assessments on the same individual enhance the ability to evaluate a disorder's impact over time. In a longitudinal, repeated measures design, the unit of analysis, whether randomized or observational, is a time period or a time point within a subject. In contrast, in a typical parallel-arm clinical study, a subject is the typical unit of analysis. Thus, when the population size is limited, the accruing of units of analyses with repeated measures can substantively enhance statistical power relative to between-subject comparisons, as discussed and illustrated by many authors [ 21 , 22 , 23 , 24 , 25 , 26 ].

Outcome measures on the same subjects are typically less variable than across subjects when the repeated measures are suitably spaced relative to disease progression and time of exposure to novel therapies (see “ Considerations, advantages and limitations of longitudinal designs ” section). Thus, in many situations longitudinal studies incorporating within-subject comparisons can estimate treatment effects more precisely than between-subject comparison in a parallel-arm or comparative cohort design.

Lastly, longitudinal observational designs are less prone to unmeasured confounding bias because they can control for non-time varying confounding, whether measured or unmeasured. Such non-time varying confounding includes important risk factors such as genetic mutation and medical history before diagnosis. In contrast, adjusting for confounding in cohort studies is only possible for measured characteristics.

Randomized and observational study designs with repeated measure

This section reviews first those longitudinal designs relying solely on self-control to assess treatment effect, followed by designs augmenting external comparison with self-controlled comparisons. All these designs and analyses strategies are illustrated by examples in CF, UCD, or other rare disorders. For novel randomized or observational study designs, hypothetical examples are used.

Self-controlled designs: relying on within-subject comparisons to estimate the causal effect

A randomized, or observational, self-controlled study (Fig.  1 ) exploits time and controls for between-subject heterogeneity. These designs are feasible and meaningful when subjects can receive therapies in multiple distinct periods, and outcomes are responsive to change within a short time relative to treatment initiation.

Self-controlled study design

Randomized self-controlled studies are well-known in rare disorders [ 22 , 27 ]. In the randomized crossover trial , subjects contribute at least two time periods for outcome assessment and receive the novel therapy in one of these periods, in random order. For example, a pilot crossover study in CF compared pulmonary function improvements after treatment with different inhaled therapies, each for 3 weeks [ 28 ]. In the randomized N-to-1 time-series designs , one subject contributes multiple periods for outcome assessment, with a randomly assigned therapy in each period. For example, this design compared the efficacy of L-arginine capsules ingested weekly on reducing glutamine levels, a favorable outcome in UCD [ 29 ]. The observation periods in follow-up or look-back are of equal duration, typically. For example, in the CF case-crossover study, 3 weeks of treatment were separated by a 3-week washout period. In the UCDC 1-N study, periods were of the same duration of 1 week.

Observational studies using within-subject comparison are less well-known and potentially useful in rare disorders. The self-controlled case-series design is the non-randomized study parallel to the case-crossover or N-of-1 designs [ 30 ]. A self-controlled case-series is a relevant design for consideration to rare disease investigators because, compared to cohort studies, it has shown a remarkable ability to control for confounding in pharmacoepidemiology and comparative safety [ 31 , 32 ]. This design only uses subjects who received both treatment and comparators at different periods and anchors the observation period to a subject’s initial treatment period. Thus, hypothetically, one could investigate l -arginine capsules' impact on reducing glutamine levels in UCD with an observational study of a sample of subjects with intermittent treatment with L-arginine and glutamine measurements in on and off exposure periods.

The case-crossover design is another potentially useful observational study design for consideration in rare disorders. It is nested in a case-series design and is particularly useful and cost-effective in investigating causes for rare dichotomous outcomes [ 33 , 34 ]. This design investigates “the timing” of events rather than “the characteristics” of subjects with an event investigated in parallel-group designs. In this design, all subjects in the observational sample experienced the outcome, and exposure is ascertained in a hazard period immediately preceding the outcome and in control periods either preceding or following the hazard period. For example, hypothetically, if one wanted to investigate the benefit of a 4-week therapy in CF for prophylaxis of pulmonary exacerbations, one could recruit CF subjects after they experienced a pulmonary exacerbation and retrospectively collect exposure to therapies in a hazard period of 4 weeks preceding exacerbation compared to control periods, each of 4 weeks, before that.

Sequential designs: augmenting between-group comparison with within-group comparisons

Sequential designs adapt their operating characteristics, such as therapy initiation or discontinuation time, investigated therapies, or study stopping time. Adaptation happens at sequential looks, based on information available at each look. Among these designs, we discuss those that augment between-subject comparison with within-subject comparisons. Although these designs' flexibility and efficiency are promising, most are novel in rare diseases or not as well established. Thus, few case studies exist of their successful application.

Sequential treatment initiation designs (Fig.  2 ) include delayed treatment and stepped-wedged. In these designs, all subjects receive a new treatment, and the time of initiation is randomized. The delayed treatment design randomizes each subject to a specific initiation time. For example, the Phase III study of vestronidase alfa in patients with mucopolysaccharidosis VII [ 35 ] randomized participants to initiate treatments at week 1, week 9, week 17, or week 25 and followed subjects for 48 weeks. In a randomized sequential withdrawal (Fig.  2 ) study, all subjects initiate therapy, then discontinue therapy at a randomized time. For example, the Phase III study investigating the efficacy of pegvaliase in treating Phenylketonuria [ 36 ] randomized responders to therapy, determined after an assessment period, to either discontinue or continue therapy for 8 weeks.

Sequential designs

The stepped wedged design (Fig.  2 ) randomizes each cluster of subjects to consecutive initiation times, where the cluster groups similar participants (e.g., geographic-based similarity) [ 37 ]. Clusters in stepped-wedged designs typically simplify the logistics of randomization. For example, a cluster-randomized study investigated the impact of a clinic-based behavioral intervention on adherence to medication in CF patients using each CF center as a cluster for randomization purposes [ 38 ].

The sequential multiple assignment randomized trial (SMART) adapts therapies or doses, as information about each subject’s response to therapy accumulates, at pre-specified interim analyses. Examples of their use exist in oncology and behavioral research [ 39 , 40 ]. For example, the studies can start with multiple doses and change the investigated therapies at interim looks as information on subject’s response to treatment accumulates. Changes include dropping ineffective doses, adding a second-line therapy, or switching therapy of non-responders. Pre-planned sequential designs have the main advantage of allowing the study to stop early for efficacy or futility while calibrating these decisions for uncertainty at each look. One disadvantage of this approach is the increase in complexity of study planning and conduct.

The longitudinal cohort study can also leverage repeated measures on the same subject when the follow-up of each subject includes time on multiple treatments of interest, validated measures for the outcomes of interest, and time of initiation or duration of exposure or follow-up vary across subjects. When using this design for comparative safety or effectiveness, it can be re-imagined as the observational counterpart of a randomized study where the treatment decision is sequential, albeit not randomized [ 41 ].

For example, Li et al. [ 42 ] re-purposed the interstitial cystitis database as a design with sequential time from diagnosis to surgery initiation, cystoscopy, and hydrodistension. A recent comparative effectiveness study used this approach within the UCD Consortium longitudinal study to investigate liver transplantation's effectiveness on mortality and quality of life [ 43 ]. This approach divides the follow-up for each subject into multiple periods. Then, sequentially, in each period, subjects receiving the treatment are matched, on their medical history up to the start of each period, to eligible subjects who did not receive the treatment. Thus, a subject's off-therapy period can serve as the control for on-therapy periods.

Similarly, Hernan et al. re-imagined data from the Nurses’ Health Study [ 44 ] as a sequence of nested trials for hormone replacement therapy where the therapy initiation was sequential.

In this approach, subjects are dynamically weighted over time using the propensity score, defined as the probability of receiving treatment or control at a given time conditional on medical history at that time. Thus, the more similar the controls are to the treated subjects at a given time, the higher their weights. The treatment effect estimation incorporates these weights in the analysis.

Considerations for valid causal inference with longitudinal data

Causal inference, framework, and assumptions.

Causal inference provides the framework for quantifying a new therapy's effect by comparing the observed outcomes under treatment received to the potential outcomes had the same subjects received an alternate therapy. The gold standard design for estimating a new therapy's treatment effect, including in rare diseases, is the randomized clinical trial. Because randomization ensures that patient groups are comparable or exchangeable before randomization, any differences in outcomes observed at the end of the trial can be causally attributed to treatment.

In observational studies, causal inference methods aim to minimize bias and confounding to ensure that differences in outcomes result from differences in treatment [ 45 , 46 ]. These methods produce reliable treatment effect estimates under the assumptions of positivity (likelihood or propensity of receiving any treatment, is neither zero, nor one), exchangeability of treatments (comparability of characteristics between treatment groups), and consistency (equality of observed and potential outcomes). Practically, these assumptions apply to the units of analyses and ensuring that they are met or are plausible is design-specific as we describe in the next section.

Design and analyses considerations with longitudinal studies for valid causal inference

Considerations, advantages and limitations of longitudinal designs.

All studies considered in this paper have several advantages relative to the typical parallel-arm, placebo-controlled, randomized or cohort study. The main advantages of longitudinal studies, discussed in “ The benefits of designs with repeated measures ” section, are increasing the units of analyses, potentially reducing variability and confounding, and thus increasing study power to detect change. The longitudinal information can be in the follow-up or in the rich medical history. An additional advantage for randomized studies is that having all eligible subjects receive the new therapy reduces the ethical or recruitment concerns with having a placebo arm. While randomized studies can have strict inclusion and exclusion criteria, observational studies typically include a broader, more representative population with the potential for more generalizable findings. However, potential gains in study power from repeated measures are only possible when the causal inference assumptions above are met.

Of the three assumptions for valid causal inference, exchangeability of periods has broad implications on the feasibility and the specifications of longitudinal studies. In purely self-controlled studies, exchangeability is within-subjects, whereas, in longitudinal studies that combine person-time information across subjects and periods, exchangeability is relevant within and between subjects.

Randomization of subjects to different groups or therapies to different periods within the same subject guarantee positivity, some exchangeability, and lack of association of an outcome with future therapies. These three criteria are typically assumed in non-randomized studies comparisons and are more plausible in studies incorporating self-control than typical cohort studies. First, having a subset of subjects exposed to the treatment of interest and comparators guarantees positivity in observational studies. Second, by design, studies automatically control for non-time varying confounding, whether measured or unmeasured, when each subject serves as their control. Thus, the remaining threats to within-subject exchangeability include the carry-over effect of treatment in one period to the outcome on another period, time-varying confounding, time-varying treatments, or time-varying severity.

In practice, choosing short periods in longitudinal studies relative to age of participants and their disease progression, make the study feasible and exchangeability more plausible. For example, to investigate the effect of a novel therapy administered in the first few weeks of life in preventing brain injury, a study incorporating within-subject comparison would be more feasible for a brain function endpoint measured in the first few weeks of life (e.g., lab measurement or imaging) rather than later in life (e.g., neuropsychological tests after 3 years of age). Adding a short gap between treatment and control periods can also lessen concerns of the carry-over effect. For simplicity, periods are most often of the same duration. They typically start on the same day of the week and end on the same day of the week, or in the same season, to reduce the day of the week bias and seasonality.

Another challenge in using longitudinal observations for causal inference is determining a relevant index date for each subject that anchors pre-intervention medical history and post-intervention follow-up. An emerging approach in rare diseases is to use birth date or time of diagnosis as the index date. For example, in the earlier example investigating liver transplant effectiveness in patients with neonatal diagnosis in the UCD Consortium database [ 47 ], the index date was the birth date.

When selecting an observational database for longitudinal cohort studies, accuracy in timing is essential for multiple factors such as age, time of onset of symptoms, time of diagnosis, and developmental or therapeutic intervention milestones. The frequency of repeated measures is ideally compatible with exposure patterns, outcome natural history, and clinical visits pattern to increase adherence and minimize missing values in prospective studies.

Analysis considerations

Estimating a treatment effect in any of the above designs can vary in complexity based on the causal question and the study design. Reviews of analytical considerations in the rare diseases setting abound for randomized studies, for example in publications by these authors [ 48 , 49 ]. Similar analytical considerations apply for observational studies with the added complexity of controlling for confounding when necessary. We summarize these considerations in this section based on the following characteristics: whether the study solely uses self-control or also uses between-subject comparison to estimate treatment effect, what confounding is adjusted for by design, and whether any time-varying or sequential adjustments are needed.

In self-controlled designs such as the crossover, N-of-1, or case series designs, estimating the treatment effect involves comparing outcomes during the treatment periods to outcomes in control periods. In a case-crossover design, one estimates the treatment effect by comparing treatments received during the hazard period immediately preceding the outcome to treatments received during control periods [ 33 ]. Under exchangeability, analyses estimating the treatment effect in purely self-controlled designs are paired analyses. They include paired t-tests or an F-test for continuous outcomes and a McNemar’s test for dichotomous outcomes. More complex analyses, such as hierarchical mixed effect models or conditional regression models, can adjust for order effect, time-varying confounding such as age, or outcome change over time, under additional assumptions [ 30 , 50 ].

Sequential designs rely on longitudinal data collection to augment between-subject comparison with within-subject comparison from those that used more than one therapy. Randomized studies typically use hierarchical modeling or mixed effect modeling with a random effect accounting for correlation between repeated measures on the same subject [ 39 ]. With parametric models, use of hierarchical Bayesian models can also incorporate expert opinion and beliefs in prior distributions. Those update the model-based likelihood as data accrue and result in a treatment effect poster distribution [ 51 ].

In longitudinal cohort studies, analysis methods vary by the approach used to control for confounding including stratification, matching, weighting, or regression. They account for the correlation of multiple measurements from the same subject, weighting, and matching by including a random effect for subjects and using sandwich estimators, or bootstrap to derive standard errors. Using G-estimation or marginal structural models can handle time-varying treatment and control for time-varying confounding in the inference [ 52 , 53 ]. For example, these methods have been used in secondary analyses of previously collected randomized clinical trials to evaluate an intervention that was not randomized [ 54 ]. In the rare disease setting, marginal structural models were used to evaluate a new therapy's efficacy and safety in severe juvenile dermatomyositis [ 55 ].

This paper gives an overview of randomized and observational study designs that exploit repeated measures in rare diseases to answer causal inference questions evaluating new or existing therapies. This paper illustrated some considerations for using these designs with case studies in the rare disease areas. While the focus of this paper is on rare disorders, many considerations and challenges apply broadly to causal inference methods in small samples. With the advent of genetic testing and personalized medicine, understanding how we can best estimate causal effects in small population subsets will be broadly relevant.

The longitudinal study designs we reviewed include self-controlled design and sequential designs. Ideally, the follow-up for each subject in these longitudinal studies would be long enough for observing exposure to therapy, in a critical time for therapeutic intervention on the outcomes of interest, and short enough to lessen the concern of time-varying confounding.

These designs have different operating characteristics than simple parallel designs that can make their conduct more feasible or information collection more efficient. Several algorithms exist to identify which randomized study design in rare diseases is useful based on the outcome severity, the rapidity of response to therapy, and ethical considerations around using a placebo arm [ 27 , 56 ]. Based on our review in this paper, we argue that when it comes to using self-controlled observational studies or augmenting between-subject comparison with within-subject comparison, the same feasibility and adequacy principles developed for randomized studies apply for observational study designs. Additional complexities arise in observational studies to control for confounding between subjects and over time in the design and analysis. Control for confounding is achieved using weighting or matching methods developed for causal inference. Comparing how these methods perform in small samples based on different operating characteristics and, more specifically, the amount of confounding relative to within and between-subject variability would be valuable for rare diseases.

Rare disease networks show a great promise in accelerating our understanding of natural history and supporting the development of therapies in rare diseases [ 57 ]. The CF Foundation Therapeutics Development Network in the United States is the poster child of how such networks, when successful, can improve the lives of the patients they serve [ 23 ]. Several disease specific networks exist across the globe and include in the United States the National Institute of Health Rare Diseases Clinical Research Network [ 58 ] and the National Organization of Rare Diseases [ 59 ]. Novel sources for rare diseases data also include large electronic healthcare data networks such as the Patient-Centered Outcome Research Network [ 60 ]. Design considerations are possible to address in networks where the exchange of ideas and the economy of scale can lead to consensus clinical guidelines and standardizing data collection and capture.

Availability of data and materials

Not applicable.

National Institutes of Health. FAQs about rare diseases. 2020; Available from https://rarediseases.info.nih.gov/diseases/pages/31/faqs-about-rare-diseases .

IQVIA Institute for Human Data Science, Orphan Drugs in the United States: Exclusivity, Pricing and Treated Populations. December 2018. https://www.iqvia.com/insights/the-iqvia-institute/reports/orphan-drugs-in-the-united-states-exclusivity-pricing-and-treated-populations . Accessed Nov 2021.

Kempf L, Goldsmith JC, Temple R. Challenges of developing and conducting clinical trials in rare disorders. Am J Med Genet A. 2018;176(4):773–83.

Article   PubMed   Google Scholar  

Fonseca DA, et al. Orphan drugs: major development challenges at the clinical stage. Drug Discov Today. 2019;24(3):867–72.

The Food and Drug Administration. Pediatric rare diseases—a collaborative approach for drug development using Gaucher disease as a model; draft guidance for industry. 2017 Dec 2017; Available from https://www.fda.gov/regulatory-information/search-fda-guidance-documents/pediatric-rare-diseases-collaborative-approach-drug-development-using-gaucher-disease-model-draft .

The Food and Drug Administration. Rare diseases: common issues in drug development guidance for industry, draft guidance for industry. 2019; Available from https://www.fda.gov/regulatory-information/search-fda-guidance-documents/rare-diseases-common-issues-drug-development-guidance-industry .

The Food and Drug Administration. Rare diseases: natural history studies for drug development, guidance document. 2019; Available from https://www.fda.gov/regulatory-information/search-fda-guidance-documents/rare-diseases-natural-history-studies-drug-development .

The Food and Drug Administration. Framework for FDA's Real-World Evidence Program. 2018 Dec 2018; Available from https://www.fda.gov/media/120060/download .

Bell SA, Tudur SC. A comparison of interventional clinical trials in rare versus non-rare diseases: an analysis of ClinicalTrials.gov. Orphanet J Rare Dis. 2014;9:170.

Article   PubMed   PubMed Central   Google Scholar  

Richter T, et al. Rare disease terminology and definitions—a systematic global review: report of the ISPOR rare disease special interest group. Value Health. 2015;18(6):906–14.

O’Sullivan BP, Freedman SD. Cystic fibrosis. Lancet. 2009;373(9678):1891–904.

Summar ML, et al. The incidence of urea cycle disorders. Mol Genet Metab. 2013;110(1–2):179–80.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Merritt JL 2nd, et al. Newborn screening for proximal urea cycle disorders: current evidence supporting recommendations for newborn screening. Mol Genet Metab. 2018;124(2):109–13.

Article   CAS   PubMed   Google Scholar  

Brusilow SW, Valle DL, Batshaw M. New pathways of nitrogen excretion in inborn errors of urea synthesis. Lancet. 1979;2(8140):452–4.

Maestri NE, Clissold D, Brusilow SW. Neonatal onset ornithine transcarbamylase deficiency: a retrospective analysis. J Pediatr. 1999;134(3):268–72.

Batshaw ML, et al. Cerebral dysfunction in asymptomatic carriers of ornithine transcarbamylase deficiency. N Engl J Med. 1980;302(9):482–5.

Wechsler D. Wechsler preschool and primary scale of intelligence—fourth edition. San Antonio: The Psychological Corporation; 2012.

Google Scholar  

Wechsler D. WISC-III: Wechsler intelligence scale for children. San Antonio: Psychological Corporation, Harcourt Brace Jovanovich; 1991.

Cystic Fibrosis Foundation Patient Registry. Annual data report 2018. 2019. Bethesda.

Waisbren SE, et al. Improving long term outcomes in urea cycle disorders-report from the Urea Cycle Disorders Consortium. J Inherit Metab Dis. 2016;39(4):573–84.

Diggle P, et al. Analysis of longitudinal data. Oxford: Oxford University Press; 2002.

Korn EL, McShane LM, Freidlin B. Statistical challenges in the evaluation of treatments for small patient populations. Sci Transl Med. 2013;5(178):178sr3.

Smith CT, Williamson PR, Beresford MW. Methodology of clinical trials for rare diseases. Best Pract Res Clin Rheumatol. 2014;28(2):247–62.

Article   Google Scholar  

Chow S-C, Chang Y-W. Statistical considerations for rare diseases drug development. J Biopharm Stat. 2019;29(5):874–86.

Van der Lee J, et al. Efficient ways exist to obtain the optimal sample size in clinical trials in rare diseases. J Clin Epidemiol. 2008;61(4):324–30.

Senn S. Sample size considerations for n-of-1 trials. Stat Methods Med Res. 2019;28(2):372–83.

Cornu C, et al. Experimental designs for small randomised clinical trials: an algorithm for choice. Orphanet J Rare Dis. 2013;8:48.

Ballmann M, von der Hardt H. Hypertonic saline and recombinant human DNase: a randomised cross-over pilot study in patients with cystic fibrosis. J Cyst Fibros. 2002;1(1):35–7.

Hackett A, Gillard J, Wilcken B. n of 1 trial for an ornithine transcarbamylase deficiency carrier. Mol Genet Metab. 2008;94(2):157–61.

Whitaker HJ, et al. Tutorial in biostatistics: the self-controlled case series method. Stat Med. 2006;25(10):1768–97.

Schneeweiss S, Sturmer T, Maclure M. Case-crossover and case-time-control designs as alternatives in pharmacoepidemiologic research. Pharmacoepidemiol Drug Saf. 1997;6(Suppl 3):S51–9.

Madigan D, Schuemie MJ, Ryan PB. Empirical performance of the case-control method: lessons for developing a risk identification and analysis system. Drug Saf. 2013;36(Suppl 1):S73-82.

Maclure M. The case-crossover design: a method for studying transient effects on the risk of acute events. Am J Epidemiol. 1991;133(2):144–53.

Maclure M. “Why me?” versus ‘why now?’—differences between operational hypotheses in case-control versus case-crossover studies. Pharmacoepidemiol Drug Saf. 2007;16(8):850–3.

Harmatz P, et al. A novel Blind Start study design to investigate vestronidase alfa for mucopolysaccharidosis VII, an ultra-rare genetic disease. Mol Genet Metab. 2018;123(4):488–94.

Harding CO, et al. Pegvaliase for the treatment of phenylketonuria: a pivotal, double-blind randomized discontinuation phase 3 clinical trial. Mol Genet Metab. 2018;124(1):20–6.

Copas AJ, et al. Designing a stepped wedge trial: three main designs, carry-over effects and randomisation approaches. Trials. 2015;16:352–352.

Quittner AL, et al. Clustered randomized controlled trial of a clinic-based problem-solving intervention to improve adherence in adolescents with cystic fibrosis. J Cyst Fibros. 2019;18(6):879–85.

Kidwell KM. SMART designs in cancer research: past, present, and future. Clin Trials. 2014;11(4):445–56.

Kidwell KM, Hyde LW. Adaptive interventions and SMART designs: application to child behavior research in a community setting. Am J Eval. 2016;37(3):344–63.

Hernan MA, et al. Observation plans in longitudinal studies with time-varying treatments. Stat Methods Med Res. 2009;18(1):27–52.

Li YFP, Propert KJ, Rosenbaum PR. Balanced risk set matching. J Am Stat Assoc. 2001;96(455):870–82.

Ah Mew N, et al. Comparing treatment options for urea cycle disorders. Washington: Patient Centered Outcome Research; 2020.

Book   Google Scholar  

Hernan MA, et al. Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology. 2008;19(6):766–79.

Imbens G, Rubin D. Causal inference for statistics, social, and biomedical sciences: an introduction. Cambridge: Cambridge University Press; 2015.

Hernan MA, Robins JM. Causal inference: what if. Boca Raton: Chapman & Hall/CRC; 2020.

Patient Reported Outcome Research Institute. Comparing treatment options for urea cycle disorders. 2015; Available from https://www.pcori.org/research-results/2015/comparing-treatment-options-urea-cycle-disorders#:~:text=Urea%20cycle%20disorders%2C%20or%20UCDs,damage%2C%20disability%2C%20or%20death .

Hilgers RD, et al. Design and analysis of clinical trials for small rare disease populations. J Rare Dis Res Treatment. 2016;1(1):53–60.

Hilgers R-D, et al. Lessons learned from IDeAl—33 recommendations from the IDeAl-net about design and analysis of small population clinical trials. Orphanet J Rare Dis. 2018;13(1):1–17.

Maclure M, Mittleman MA. Case-crossover designs compared with dynamic follow-up designs. Epidemiology. 2008;19(2):176–8.

Hampson LV, et al. Bayesian methods for the design and interpretation of clinical trials in very rare diseases. Stat Med. 2014;33(24):4186–201.

Robins JM, Hernan MA, Brumback B. Marginal structural models and causal inference in epidemiology. Epidemiology. 2000;11(5):550–60.

Mansournia MA, et al. Handling time varying confounding in observational research. BMJ. 2017;359:j4587.

Farmer RE, et al. Application of causal inference methods in the analyses of randomised controlled trials: a systematic review. Trials. 2018;19(1):23.

Deakin CT, et al. Efficacy and safety of cyclophosphamide treatment in severe juvenile dermatomyositis shown by marginal structural modeling. Arthritis Rheumatol. 2018;70(5):785–93.

Gagne JJ, et al. Innovative research methods for studying treatments for rare diseases: methodological review. BMJ. 2014;349:g6802.

Jansen-Van Der Weide MC, et al. Rare disease registries: potential applications towards impact on development of new drug treatments. Orphanet J Rare Dis. 2018;13(1):1–11.

Krischer JP, et al. The Rare Diseases Clinical Research Network’s organization and approach to observational research and health outcomes research. J Gen Intern Med. 2014;29(Suppl 3):S739–44.

Putkowski S. National Organization for Rare Disorders (NORD): providing advocacy for people with rare disorders. NASN Sch Nurse. 2010;25(1):38–41.

Canterberry M, et al. The patient-centered outcomes research network antibiotics and childhood growth study: implementing patient data linkage. Popul Health Manag. 2020;23(6):438–44.

Download references

Acknowledgements

We thank our colleagues for helpful questions and discussions during our presentations of some of these designs at the annual meetings of the Urea Cycle Disorder Consortium in July 2019 and April 2020. We thank the rare diseases working group in the Berkeley forum for collaborative research for helpful discussions about challenges in randomized study designs in rare disorders.

Author’s time working on this manuscript was supported by funding from two grant awards by the National Institute Health: 5U54HD061221 (Principal Investigator: Andrea Gropman), and 5UL1TR001876-03 (Principal Investigator: Lisa Guay-Woodford).

Author information

Authors and affiliations.

Division of Biostatistics and Study Methodology, Children’s Research Institute at Children’s National Medical Center, The George Washington University, Washington, DC, USA

Rima Izem & Robert McCarter

You can also search for this author in PubMed   Google Scholar

Contributions

The ideas in this manuscript grew from several design and analyses collaborations between RI and RM with the Urea Cycle Disorder Consortium, as well as discussions between co-authors relative to study designs in rare disorders. RI wrote this review manuscript and is the corresponding author, RM provided comments and edits on this manuscript. RM also helped with revisions to the manuscript. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Rima Izem .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Izem, R., McCarter, R. Randomized and non-randomized designs for causal inference with longitudinal data in rare disorders. Orphanet J Rare Dis 16 , 491 (2021). https://doi.org/10.1186/s13023-021-02124-5

Download citation

Received : 01 February 2021

Accepted : 06 November 2021

Published : 23 November 2021

DOI : https://doi.org/10.1186/s13023-021-02124-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Orphanet Journal of Rare Diseases

ISSN: 1750-1172

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Self controlled case series methods: an alternative to standard epidemiological study designs

Affiliations.

• 1 Department of Primary Care and Population Health, University College London, London NW3 2PF, UK Department of Clinical Epidemiology, Aarhus University, Aarhus N, Denmark [email protected].
• 2 London School of Hygiene and Tropical Medicine, London, UK.
• 3 Department of Mathematics and Statistics, Open University, Milton Keynes, UK.
• PMID: 27618829
• DOI: 10.1136/bmj.i4515

Publication types

• Epidemiologic Research Design*
• Registries*
• Research Design / standards*

Grants and funding

• G0802403/MRC_/Medical Research Council/United Kingdom
• MR/L009005/1/MRC_/Medical Research Council/United Kingdom

• Diagnostics Lab Meeting Summaries
• Diagnostics Parallel Analysis Workgroup
• Therapeutics Lab Meeting Summaries
• Therapeutics Parallel Analysis Workgroup

COVID-19 Real-World Evidence Primer

Chapter 4: Examples of COVID-19 Real-World Evidence Studies

Self-controlled case series study.

Study citation: Katsoularis I, Fonseca-Rodríguez O, Farrington P, Lindmark K, Fors Connolly AM. Risk of acute myocardial infarction and ischaemic stroke following COVID-19 in Sweden: a self-controlled case series and matched cohort study. Lancet. 2021 Aug 14;398(10300):599-607. doi: 10.1016/S0140-6736(21)00896-5. Epub 2021 Jul 29. PMID: 34332652; PMCID: PMC8321431. 4

Study objective: As the science and understanding of COVID-19 has evolved, its effect on specific organ systems (e.g., vascular) requires study. This study sought to characterize the risk of both myocardial infarction and ischemic stroke for those infected with SARS-CoV-2.

PICO: This study used two methods for its analysis: a self-controlled case series (SCCS) and a matched cohort study of patients in Sweden who received a prior diagnosis of COVID-19, and later received diagnosis codes related to a myocardial infarction or ischemic stroke. At most time points of the study, prior laboratory confirmed SARS-CoV-2 infection was identified as a risk factor for these 2 vascular events.

Data source: Patient data were derived from SmiNet at the Swedish Public Health Agency (including all patients registered as having COVID-19 until September 14, 2020) and linked with registries administered by the Swedish National Board of Health and Welfare. Like many other public health databases, SmiNet has been used to help track COVID-19 in Sweden since February 2020, as COVID-19 has been a notifiable disease in Sweden, and diagnosed individuals are reported to SmiNet (Swedish Public Health Agency) daily. The source has suitable coverage for the population of interest.

Study period : February 1 to September 14, 2020

Key sources of error and how they were handled : All individuals in Sweden were eligible for inclusion, which is a particular strength of this nationally representative study, given the comprehensive population level data capture in this RWD source, and making it the largest study evaluating this association at the time of publication. Another strength was the use of 2 different designs (SCCS and conventional matched cohort), which rely on different sets of assumptions and address different types of biases. The use of the 2 designs contributed to the ability to control for bias using different assumptions about pathways for biases. This study also adjusted for income, education level, and country of origin, which are important demographic and socioeconomic factors to consider in real-world evidence studies of COVID-19 sequelae. Generalizability from a single country study may be limited, and RWD studies of COVID-19 outcomes will also benefit from understanding race and ethnicity when available. Additionally, the authors noted that there was a peak of vascular events on day 0 (date of vaccination). A sensitivity analysis that excluded patients with events on day 0 found similar results to the main analysis. This speaks to the robustness of the study results.

• Research article
• Open access
• Published: 08 February 2017

Self-controlled designs in pharmacoepidemiology involving electronic healthcare databases: a systematic review

• Nathalie Gault   ORCID: orcid.org/0000-0002-9913-9760 1 , 2 , 3 ,
• Johann Castañeda-Sanabria 2 , 4 ,
• Yann De Rycke 2 , 4 ,
• Sylvie Guillo 2 , 4 ,
• Stéphanie Foulon 5 , 6 &
• Florence Tubach 2 , 4 , 7  

BMC Medical Research Methodology volume  17 , Article number:  25 ( 2017 ) Cite this article

6213 Accesses

32 Citations

13 Altmetric

Metrics details

Observational studies are widely used in pharmacoepidemiology. Several designs can be used, in particular self-controlled designs (case-crossover and self-controlled case series). These designs offer the advantage of controlling for time-invariant confounders, which may not be collected in electronic healthcare databases. They are particularly useful in pharmacoepidemiology involving healthcare database. To be valid, they require the presence of some characteristics (key validity assumptions), and in such situations, these designs should be preferred. We aimed at describing the appropriate use and reporting of the key validity assumptions in self-controlled design studies.

Articles published between January 2011 and December 2014, and describing a self-controlled study design involving electronic healthcare databases were retrieved. The appropriate use (fulfilment of key assumptions) was studied in terms of major (abrupt onset event, rare or recurrent event, and intermittent exposure) and minor assumptions (those for which the design can be adapted).

Among the 107 articles describing a self-controlled design, 35/53 (66%) case-crossover studies, and 48/55 (87%) self-controlled case series fulfilled the major validity assumptions for use of the design; 4/35 and 14/48 respectively did not fulfill the minor assumptions. Overall, 31/53 (58%) case-crossover studies and 34/55 (62%) self-controlled case series fulfilled both major and minor assumptions. The reporting of the methodology or the results was appropriate, except for power calculation.

Conclusions

Self-controlled designs were not appropriately used in34% and 13% of the articles we reviewed that described a case-crossover or a self-controlled case series design, respectively. We encourage better use of these designs in situations in which major validity assumptions are fulfilled (i.e., for which they are recommended), accounting for situations for which the design can be adapted.

Peer Review reports

Pharmacoepidemiology aims at assessing the risk and benefit of pharmaceuticals in real-world populations [ 1 ]. Computerized medical databases are increasingly being used for real life post-marketing observational studies [ 2 ], and have several advantages: (i) the potential for studying a very large sample size, thereby allowing for study of rare events or exposures; (ii) the availability of data for older adults, children, patients with low resources, and nursing-home residents, who are most often under-represented in clinical trials [ 3 ]; (iii) the inclusion of off-label prescriptions; and (iv) data are prospectively collected [ 4 ]. However, data are usually collected for purposes other than research (administrative or healthcare management), so the databases frequently lack information on some potential confounding factors (e.g., genetics, body mass index, smoking status, alcohol consumption, or medical history and comorbidities) [ 2 ]. In this context, in which many confounders may not be collected, self-controlled designs are an interesting option in observational studies of pharmaceuticals [ 5 ]. Self-controlled designs are based only on cases, which then act as their own control (i.e., they consist in within-patient comparison between different periods of time). Their main advantage is that time-invariant confounders that act multiplicatively on the baseline rates are inherently controlled for. As Nordmann et al. reported in a systematic review [ 6 ], these designs mainly include the case-crossover design [ 7 , 8 ] and the self-controlled case-series [ 9 – 11 ] and were developed to study the short term effect of transient-exposures and abrupt onset events. Indeed, because of the self-matched design, the risk estimation includes only data for patients who switch their exposure status over time (i.e., from exposed to unexposed, or vice versa). With sustained exposure, the opportunity for exposed patients to become unexposed is reduced, therefore leading to a smaller number of patients with “discordant” exposure status, and reduced power. Moreover, the study of sustained exposures over longer periods is subject to time-varying confounding that needs to be addressed. Studies of insidious-onset events (e.g., depression, cancer, autism) are subject to misclassification bias because of uncertainty in the onset date of the outcome. Self-controlled designs require that some other validity assumptions be fulfilled. For case-crossover design, the opportunity for exposure should be the same during the case and control time periods (e.g., for car crashes or the risk of alcohol consumption, the control period should be the same day of the week, because driving or drinking behaviour may vary from weekdays to weekends [ 8 , 12 , 13 ]), and there should not be any time trend in exposure. For self-controlled case series, two consecutive events should be independent if they are recurrent; the probability of further exposure should not be affected by a previous event; and the event should not affect the short-term mortality probability. Because of methodological developments, these designs have become applicable in more situations, by weakening the assumptions they require, such as the possibility to study time trend in exposure [ 14 , 15 ], event-dependant exposure [ 16 , 17 ], inter-dependant recurrences [ 18 ], or event-dependant observation periods [ 16 , 19 ].

Several systematic reviews have examined the use and reporting of case-crossover designs only [ 20 ], self-controlled case series only [ 21 ] or self-controlled studies in general [ 6 ]. Recently, we showed that self-controlled designs are rarely used in pharmacoepidemiology [ 22 ]. We did not find any recent study focusing on the appropriate use of self-controlled designs in pharmacoepidemiology involving electronic healthcare databases, with regard to major and minor validity assumptions.

Here we aimed to assess the appropriate use of self-controlled designs, in terms of their validity assumptions in pharmacoepidemiology involving electronic healthcare databases and to update the Nordmann et al. review [ 6 ]. To achieve this goal, we performed a systematic review to describe whether the required characteristics for the use of the designs were fulfilled and adequately reported in published articles.

For the systematic review, we searched MEDLINE via Pubmed for English and French articles of self-controlled studies (case-crossover or self-controlled case series) involving electronic healthcare databases that were published from January 1, 2011 to December 31, 2014. The search keywords are reported in Additional file 1 : S1. Papers reporting the study of safety or efficacy of a medical product that used a self-controlled design and involving electronic healthcare databases were included. We excluded articles of studies not examining a drug or device, methodological articles (design development or study protocol), drug utilization or prescription studies (drug utilization or drug prescription patterns), descriptive or case-series studies (prevalence or incidence of disease), clinical practice evaluation (quality improvement in clinical practice), and articles not describing a self-controlled design or not involving electronic healthcare databases.

Data collection

The data were collected by three readers who used a standardized extraction form (Additional file 1 : S2) based on the STROBE recommendations [ 23 ] and on previous systematic reviews of designs in pharmacoepidemiology [ 6 , 22 ].

The collected data focused on the characteristics of exposures and events and on the study designs. The “exposure characteristics” section contained information about the type, prevalence and characteristic of the exposure ( i.e. one shot, such as vaccines; transient for a few days, such as antimicrobial therapy or analgesics; intermittent on a specified frequency, such as chemotherapy; or sustained, such as long term use of hypoglycaemic agents). In several cases, we reclassified the sustained exposures as transient, when the event of interest was explicitly hypothesised to occur shortly after the drug initiation (product initiation for incident user designs), or intermittent, when we considered that a high opportunity of switch from exposed to unexposed status (or vice versa) can be assumed over the observation period (especially when the risk period was equal and non-inferior Footnote 1 to the prescription period). When no classification was possible after consensus, the exposure classification was considered unclear. The “event characteristics” section contained information about the type, definition, prevalence or incidence, and characteristics of the event, such as the potential for recurrence (unique, such as hip fracture, or recurrent, such as seizures), and its onset (abrupt, such as car crash, or insidious, such as depression). An event was considered recurrent if it was likely to occur several times within the same patient during the observation period (because recurrence was clearly reported in the article or such an event is usually considered recurrent, such as febrile convulsions). An event was considered rare if its prevalence in the source study population was less than 5%. Subjective items ( i.e. type of exposure, event onset or recurrence) were discussed among the authors to reach consensus. The “design” section included information about the type (self-controlled case series or case-crossover) and characteristics of the study design: for case-crossover studies, definition of time windows; opportunity for exposure during the case and control periods, existence, reporting and consideration in the analysis of a time trend in exposure; and for self-controlled case series, independent recurrences in case of recurrent events, independence of exposure with respect to the event, and independence of the short-term mortality risk with respect to the event).

Major validity assumptions

For self-controlled designs ( i.e. required characteristics that must be fulfilled for the self-controlled design to be valid) were based on the methodological studies for the corresponding designs [ 7 – 10 ]. For case-crossover studies, these characteristics are a transient or intermittent exposure, an abrupt-onset event, and a rare event. For self-controlled case series, they are a transient or intermittent exposure, an abrupt-onset event, and a rare and/or recurrent event.

Minor validity assumptions

For self-controlled designs ( i.e. situations that can threaten the design validity but when the design can be adapted to be valid in light of recent methodological developments) were as follows: for case-crossover studies, the opportunity for exposure should be the same during the case and control time periods, and there should not be any time trend in exposure. We considered that a time trend existed if it was clearly reported by the authors or, when not clearly reported, if we considered that such trend could be suspected in the study setting (e.g., in the study of a drug shortly after market authorization (24) or the study of drugs during pregnancy especially when control periods are chosen before pregnancy because prescription patterns change after conception). For exposures with a time trend, we considered that the time trend was accounted for when the authors used a case-time control design or another appropriate design (such as bidirectional case-crossover design or case-case-time-control design) [ 14 , 15 , 24 , 25 ]. For self-controlled case series, two consecutive events should be independent if they are recurrent; otherwise the design should be adapted as appropriate, as in considering only the first event [ 10 , 18 ]. Also, the probability of further exposure should not be affected by a previous event ( i.e. event-independent exposure assumption) or the design should be adapted as appropriate, as in excluding person-times before exposure [ 11 , 16 , 17 ]. Finally, the event should not affect the short-term mortality probability, or the analysis should be adapted by involving the time interval between the event and end of the actual observation period [ 11 , 16 , 19 ].

All assumptions were mainly assessed on what the authors reported in their paper. With no mention of these assumptions in the article, the assessment was based on our own judgement, after consensus: in particular, the appropriateness of these assumptions can be deduced for particular studied events (e.g., myocardial infarction, falls, or febrile convulsions) by referring to several methodological studies with applications to such events [ 7 , 26 , 27 ]. When we could establish no clear conclusion, the assumption was considered unclear.

Of note, we did not consider that the design adaptation proposed by Wang et al. to study sustained exposures or insidious-onset outcomes (i.e., prolonged exposure windows) [ 28 ] allows for consider them minor validity assumptions in all situations. In fact, there is still a validity threat when there is a small probability of switching between exposure statuses within the observation window or when the hypothesised effect is of a cumulative nature with a delayed onset. In the former setting, this design adaptation could lead to loss of statistical power (increasing type II error and false negatives) because of too many rare discordant cases in case-crossover studies or unexposed cases in self-controlled case series. Also, the probability for the occurrence of time-varying within-person confounders is high, so self-controlled designs not recommended as compared to cohort-based approaches in such situations.

For articles with several outcomes, we considered the assumptions valid if they were fulfilled for at least one outcome. Finally, we considered that a study did not fulfil minor assumptions if at least one of the assumptions was violated.

Quality of reporting

We examined whether the recommendations from Nordmann et al. [ 6 ] were applied for quality of reporting. These recommendations include the reporting of the fulfilment of key assumptions (whether the setting was valid for the design implementation in terms of major and minor assumptions); the definition (number and duration) of the case period (for case-crossover) or risk period (for self-controlled case series) and the control periods; the appropriate statistical model ( i.e. conditional logistic regression model or conditional Poisson regression model for case-crossover [ 29 , 30 ], and conditional Poisson regression model, Cox’s stratified proportional hazards model or conditional logistic regression model for self-controlled case series [ 30 ]), an a priori sample size calculation (or power calculation, because sample size cannot be chosen in healthcare database research); the appropriate effect estimator ( i.e. odds ratio for case-crossover and incidence rate ratio for self-controlled case series) with a measure of variability (confidence interval or standard deviation); unadjusted and adjusted estimators; the person-time in each risk and control period for self-controlled case series; and finally, the reporting of any sensitivity analyses.

The data are described as number (%). Concerning the exposure characteristics, “one shot”, “transient” and “intermittent” exposure categories were grouped as “intermittent”. To describe the appropriate of use of the self-controlled designs, we determined whether the major validity assumptions were fulfilled. Furthermore, we determined whether fulfilment of the minor validity assumptions was reported in articles reporting valid major assumptions and also overall. When the fulfilment of the assumptions was unclear, we considered them valid in our main analysis. We further performed a sensitivity analysis considering such unclear assumptions as invalid. Another sensitivity analysis was performed considering only one article in case of several papers from the same author(s) with similar objectives. The quality of reporting was also reported. Analyses involved use of R 2.15.2 (The R Foundation for Statistical Computing). The results of our systematic review were reported according to the PRISMA guidelines [ 31 ].

We identified 107 articles describing self-controlled designs: 53 case-crossover and 55 self-controlled case series (one study used both designs, Fig.  1 ). The list of included articles is reported in Additional file 1 : S3. Their characteristics are reported in Table  1 . All papers were of drug safety.

Flow chart of the selection of articles

The fulfilment of the major and minor validity assumptions for the use of a self-controlled design is described in Table  2 and reported in details in Additional file 1 : S4a and S4b. Overall, 18 (34%) articles using a case-crossover design and 7 (13%) articles using a self-controlled case series did not fulfil all of the major validity assumptions for their use: most frequently, the studies examined the effect of a sustained exposure (e.g., antihypertensive drugs or platelet aggregation inhibitors in cardiovascular primary prevention), and 3 examined an event with an insidious onset (e.g., depression, chronic fatigue syndrome or congenital malformation). Classification of exposure was unclear in 8 articles using a case-crossover design and 1 article using a self-controlled case series design. Among the 35 case-crossover studies with valid major assumptions, 4 did not fulfil the minor ones: a time trend in exposure was not accounted for in 3, and the opportunity of exposure could vary between case and control periods in another. Among the 48 self-controlled case series with valid major assumptions, 14 did not fulfil the minor ones: 1 study examined an event with recurrences that were not independent (i.e., migraine), 6 studies violated the assumption of event-independent exposure and 8 studies examined an outcome that could censor the observation period (such as myocardial infarction, stroke, heart failure). An exposure temporal trend was unclear or not reported in 22 articles using a case-crossover design. Overall, all the assumptions were fulfilled in 31 (58%) case-crossovers studies and 34 (62%) self-controlled case series. Considering unclear assumptions as invalid led to smaller proportions of appropriate studies, especially for case-crossover designs (Additional file 1 : S5). Of note, a sensitivity analysis considering only one article when several from the same author(s) with similar objectives were included in the systematic review showed similar results (Additional file 1 : S6).

Considering the quality of reporting (Table  3 ), the assumptions for use of a self-controlled design were reported in 53 (50%) papers (at least one major assumption reported in 41, at least one minor assumption in 8, and both major and minor assumptions reported in 4). In 9 papers stating that the use of a self-controlled design was adequate with regard to major assumptions, the studied exposure was actually sustained. Overall, 94 (88%) papers reported a rationale for using a self-controlled design (i.e., to account for confounding factors that do not change over time). The definition of the control periods was unreported in 3 (6%) papers describing a case-crossover design and 6 (11%) a self-controlled case series. The model used was reported in 93% of articles and was appropriate in 100% of these. The sample size or post-hoc power calculation was reported in 12% of articles. The results were always displayed with a measure of variability, but in 37 (66%) of the self-controlled case series, the person-time in each period was not reported. Two thirds of articles reported adjusted estimates (with or without unadjusted estimates). Sensitivity analyses were described in half of the articles and well reported. Three articles reported all the recommended quality items.

In this review of articles, we quantified the appropriate use of self-controlled designs in pharmacoepidemiology involving electronic healthcare databases in terms of major and minor validity assumptions for the use of the designs. We focused on studies involving medical databases, in which self-controlled designs are particularly useful to adjust for time-invariant confounders that may not be collected. Self-controlled designs were not appropriately used in 34% and 13% of the articles we reviewed that described a case-crossover or self-controlled case series design, respectively. We encourage better use of these designs for situations in which major validity assumptions are fulfilled (i.e., for which they are recommended), accounting for situations for which the design can be adapted.

Our study updated the Nordmann et al. review [ 6 ] and is the first systematic review exploring the appropriate use of self-controlled designs in pharmacoepidemiology involving electronic healthcare databases, in terms of major and minor validity assumptions, in accordance with recent recommendations. The fulfilment of major assumptions is the minimum requirement for self-controlled designs to be valid, as they can be superior to designs with comparison groups in such situations (more powered and less biased) [ 5 ]. Moreover, recent recommendations state that the self-controlled designs should be preferred to designs with comparison groups in studies performed on healthcare databases, when key validity assumptions are fulfilled [ 4 ]. For articles that did not fulfil all of the major validity assumptions, it was essentially due to the study of sustained exposure (e.g., antihypertensive drugs or prophylaxis for cardiovascular events) or events with an insidious onset (depression or chronic fatigue). The intermittency of exposure is a requirement for both case-crossover and self-controlled case series to ensure that the number of patients with varying exposure statuses is not too small [ 4 ]. The acuteness of the event onset is a validity assumption that reduces the likelihood of misclassification bias [ 32 ]. However, some have proposed an adaptation of the case-crossover design for studying prolonged exposures and insidious-onset outcomes [ 28 ] and an adaptation of the self-controlled case series (towards the end of our observation period) for studying cumulative exposure [ 33 ]. The design adaptation Wang et al. proposed consists of lengthening exposure assessment windows [ 28 ]. Despite these adaptations, self-controlled designs are usually less powered than between-person comparisons when studying sustained exposures, because discordant pairs seldom arise when exposures are actually sustained and the observation period is short as compared to the risk and control period durations. Thus, both case-crossover and self-controlled case series designs would fail: the former because of a too-small number of discordant pairs and the latter because cases would be seldom unexposed. Hence, self-controlled designs are not recommended for situations of sustained exposure because they could lead to a loss of power [ 4 ]. In fact, few statistical power (or sample size) calculations were carried out in the included studies so the impact of studying long-term exposures when there is a low probability of switching is uncertain. In addition, lengthening exposure assessment windows increases the risk of bias due to within-person time-varying confounders (the absence of which is the main advantage of using self-controlled designs). Even if lengthening exposure assessment windows reduces misclassification bias, it does not answer the issue of reverse causality that could arise when an exposure occurs after the true time of outcome onset, thereby leading to a spurious association. More generally, failure to meet important assumptions of self-controlled designs has been associated with increased risk of discrepant results between case-only and cohort-based approaches (which can occur even in the absence of unmeasured confounders) [ 30 ].

For all these reasons, we considered that studies involving a self-controlled design were invalid in situations of sustained exposure when there was a small probability of switching between exposure status within the observation window or when the outcome has an insidious or delayed onset or results from a cumulative effect. Of note, we classified drugs that are usually used chronically as “intermittent exposures” when a high opportunity for a switch from exposed to unexposed status (or vice versa) can be assumed over the observation period. Several examples can be cited: methylphenidate in children with attention-deficit/hyperactivity disorder (the treatment is usually discontinued during holidays) and palivizumab, an anti-respiratory syncytial virus (RSV) monoclonal antibody for prophylaxis of severe lower respiratory tract infection in children (usually administered during the high-risk season of RSV infection).

In a sensitivity analysis, considering studies examining a sustained exposure as appropriate (without accounting for the probability of switching exposure status), 45 (85%) case-crossover and 53 (96%) self-controlled case series studies fulfilled all major assumptions. In this analysis, the most frequent validity threats were insidious or common events.

In terms of event frequency, we considered rare and/or recurrent events as appropriate for self-controlled case series and only rare events for case-crossover designs. However, in situations when the event is both non-recurrent and non-rare, a self-controlled design can still be used. Nevertheless, this use would imply that the number of strata (here, the number of cases) would increase but not their size (here, the number of periods within the same patient), thereby leading to poor estimation of the variance when using stratified models. Therefore, we considered the rare or recurrent event as a major assumption. Moreover, in the study of Pouwels et al., the rareness of the outcome was a factor associated with fewer discrepancies [ 30 ].

We found that minor assumptions were most often valid when major ones were valid. As a reminder, those assumptions were considered minor because the design can be adapted if they are not fulfilled, which allows for a self-controlled design. A small proportion of the self-controlled designs, 18 (16%), could have been improved by applying those adaptations. For instance, 3 studies with a case-crossover design did not adjust for a time trend in exposure, even though the paper clearly stated that such a trend existed. It has been shown that lack of adjustment for exposure time-trends in case-crossover studies led to biased estimations [ 14 , 25 , 34 ], and hence several extensions of the case-crossover design have been developed to take into account a temporal trend in exposure [ 14 , 15 , 24 , 25 ]. Of note, in 22 additional papers from our systematic review, the existence of an exposure time-trend was not discussed by the authors nor could be assessed from the reported information, but we still considered them appropriate. Thus, the proportion of case-crossover studies that could have used the design more adequately may be underestimated. Researchers must keep in mind that the exploration and reporting of such a trend in case-crossover studies is crucial for design validity. Concerning the event-independent exposure assumption, we found that it was fulfilled in 88% of articles involving a self-controlled case series. A simulation study reported that relative incidence is almost always overestimated when the event-independent exposure assumption is violated in self-controlled case series studies (except for the situation of extreme dependence), but the bias is corrected when the design is adapted [ 35 ]. The corresponding methodological developments were published in the late 2000s [ 11 , 16 ], perhaps too recently to be applied in the studies we reviewed.

Of note, a simulation study explored the validity of the case-time-control design in situations of within-individual exposure dependency over several control periods but showed that the method is robust to deviation of this assumption [ 36 ]. We did not explore this assumption the studies included in our review.

The previous systematic review by Nordmann et al. reported the validity assumptions of self-controlled designs in pharmacoepidemiology between 1995 and 2010 (before the development of the previously cited recommendations) [ 6 ]. The authors reported an inappropriate use of self-controlled designs: validity assumptions were not fulfilled for 76% of the articles describing a case-crossover design and 60% self-controlled case series. Concerning major assumptions, our review, which covered healthcare database studies published recently, shows that these data have improved. Moreover, major and minor validity assumptions were not distinguished in the Nordmann et al. review. Nevertheless, we found the same main reasons for the inappropriate use of these designs (i.e., the study of sustained exposure and the absence of considering exposure time-trend).

Self-controlled designs can control for intra-individual time-invariant confounders. Many design extensions that weaken the validity assumptions have been developed, and these designs still are under development, such as for the study of multiple exposures [ 37 ], or the study of recurrent events when recurrences are not independent [ 27 ]. However, the designs are subject to several biases (e.g., residual confounding due to unmeasured within-person time-varying factors or misclassification of exposure [ 32 ]). Moreover, self-controlled designs explore the triggers that precede abrupt-onset events, and answer the questions “Why now?” or “What happened just before?”, which is slightly different from the question raised with between-person comparisons (“Why me?”) [ 38 ]. Nevertheless, they are complementary to cohort-based approaches [ 4 ], and both designs should be applied, especially when one or more assumptions are not fulfilled [ 30 ].

Regarding the quality of reporting, we found 9 studies examining sustained exposures (e.g., antihypertensive treatments or low-dose aspirin for secondary prevention of cardiovascular events), which indicates the reporting of the design being appropriate to study abrupt-onset outcome and transient drug exposure. This high number underlines that authors and reviewers should be aware of the design’s validity assumptions, recommendations for use and the need to report validity assumptions fulfilled (or not). Indeed, the minor assumptions were rarely reported in the papers we reviewed. In addition, the sample size or power calculation was rarely reported, with no improvement compared to a previous review [ 6 ]. However, studies involving electronic healthcare databases usually have very large sample sizes and perhaps the sample size calculation is not needed in this context, because the sample size cannot be chosen. However post-hoc power calculation in the database study sample is important to decide which healthcare database should be used and to interpret the absence of a statistical association, especially in the context of a very rare event. Post-hoc power calculation indicates how easily an effect that is fixed a priori can be shown (accounting for the observed number of patients/cases and the observed variability). Even if confidence intervals (which represent how accurate the results are estimated) are reported, the power calculation is more related to the number of observed events, and is information that is easier for the reader to understand: in case of non-significant associations, power can be quite difficult to interpret on the basis of the sole confidence interval.

With respect to other elements of reporting quality, a measure of variability of the estimate was always reported, which we considered adequate if the number and duration of different periods were also reported. The effect estimator was not always appropriately reported, but reported statistical models were all considered as appropriate. Valid models other than the conditional logistic regression or the conditional Poisson regression can still be applied, such as the Cox stratified proportional-hazards model for case-crossover studies and the Cox stratified or conditional logistic regression models for self-controlled case series [ 30 ], even if unusual. However, it has been shown that in case-crossover studies, for instance, conditional logistic regression model and conditional Poisson model give identical estimates [ 29 ]. In case-crossover studies, times to event are the same for case and control periods (because periods are defined similarly within strata) and even if a stratified Cox model can be used, it does not assess and compare a time to event. Therefore, the estimator computed (even if called a “hazard ratio”) still represents an “odds ratio” and should be interpreted as such. One third of the included papers reported both adjusted and unadjusted estimates, which is quite low when they allow for assessing the importance of bias. Nordmann et al. recommended that the number of discordant pairs in case-crossover designs (i.e., number of patients who crossed from unexposed in the control period to exposed in the case period, or vice versa [ 8 ]), or the count of events in the different time periods for self-controlled case series should be reported [ 6 ]. We considered that these items need to be reported, except if a measure of variability of the estimate is reported along with a clear description of the number and duration of different periods. However, only a small number of self-controlled case series reported the duration of control period along with the person-times in risk and control periods, but this was appropriately reported for case-crossover studies. Consequently, the quality of reporting in self-controlled studies can still be improved, in accordance with the recommendations provided by Nordmann et al. [ 6 ].

Our study has some limitations. First, a potential paper selection bias could exist, which we tried to limit with a comprehensive literature search using keywords, title and abstract terms, and few limits, that were already used in previous systematic reviews [ 6 , 30 , 39 ]. Moreover, the definition of abrupt versus insidious onset of the event is somewhat subjective, as is transient versus sustained exposure, or some minor assumptions for the self-controlled case series. We tried to limit this issue by consensus. Some strengths of this study are worth noting. We focused on studies involving medical databases because of a growing interest in the use of “big data” in healthcare research [ 40 ]. We updated Nordmann et al. review [ 6 ] up to 2014, and to ensure that no indexing issue in PubMed can be suspected, we updated our literature search in July 2016.

Self-controlled designs have many advantages, including their ability to inherently adjust for time-invariant factors, which is important when using electronic healthcare databases, where some confounding factors are usually not collected or nor available. We found that in terms of fulfilling the major assumptions of the designs, the designs for one-third of the case-crossover studies and less than one-fifth of self-controlled case series we reviewed were not appropriately used in these pharmacoepidemiology studies involving electronic healthcare databases. We encourage a better justification of the design validity in terms of major and minor assumptions in accordance with recommendations for their use and more accurate reporting of self-controlled case series. Addressing these issues will contribute to a wiser use of these self-controlled designs, which is advantageous for pharmacoepidemiology involving large healthcare databases.

In some articles of self-controlled case series, the risk period was defined arbitrarily in the same way for all patients without accounting for the actual duration of prescription. For drugs used chronically, this situation can result in misclassification bias, because the subject is considered unexposed when a prescription period was still ongoing. Such exposures were considered sustained and the design was considered as inappropriate in these situations. In contrast, when risk period was equal to the prescription period and when the observation period was large enough to observe a switch in exposure status, we considered the exposure as transient even if it was used chronically

Abbreviations

Preferred reporting items for systematic reviews and meta-analyses

Respiratory syncytial virus

Strengthening the reporting of observational studies in epidemiology

Strom BL. What is pharmacoepidemiology? In: Strom BL, editor. Pharmacoepidemiology, vol. 6. 4th ed. Chichester, West Sussex: John Wiley & Sons; 2005. p. 3–15.

Google Scholar  

Strom BL. Overview of automated databases in phamacoepidemiology. In: Strom BL, editor. Pharamcoepidemiology. 4th ed. Chichester, West Sussex: John Wiley & Sons; 2005. p. 219–22.

Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. 2005;58(4):323–37.

Article   PubMed   Google Scholar  

Gagne JJ, Fireman B, Ryan PB, Maclure M, Gerhard T, Toh S, Rassen JA, Nelson JC, Schneeweiss S. Design considerations in an active medical product safety monitoring system. Pharmacoepidemiol Drug Saf. 2012;21 Suppl 1:32–40.

Maclure M, Fireman B, Nelson JC, Hua W, Shoaibi A, Paredes A, Madigan D. When should case-only designs be used for safety monitoring of medical products? Pharmacoepidemiol Drug Saf. 2012;21 Suppl 1:50–61.

Nordmann S, Biard L, Ravaud P, Esposito-Farese M, Tubach F. Case-only designs in pharmacoepidemiology: a systematic review. PLoS One. 2012;7(11):e49444.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Maclure M. The case-crossover design: a method for studying transient effects on the risk of acute events. Am J Epidemiol. 1991;133(2):144–53.

CAS   PubMed   Google Scholar  

Maclure M, Mittleman MA. Should we use a case-crossover design? Annu Rev Publ Health. 2000;21:193–221.

Article   CAS   Google Scholar  

Farrington CP. Relative incidence estimation from case series for vaccine safety evaluation. Biometrics. 1995;51(1):228–35.

Article   CAS   PubMed   Google Scholar  

Whitaker HJ, Farrington CP, Spiessens B, Musonda P. Tutorial in biostatistics: the self-controlled case series method. Stat Med. 2006;25(10):1768–97.

Whitaker HJ, Hocine MN, Farrington CP. The methodology of self-controlled case series studies. Stat Methods Med Res. 2009;18(1):7–26.

Marshall RJ, Jackson RT. Analysis of case-crossover designs. Stat Med. 1993;12(24):2333–41.

Redelmeier DA, Tibshirani RJ. Association between cellular-telephone calls and motor vehicle collisions. N Engl J Med. 1997;336(7):453–8.

Suissa S. The case-time-control design. Epidemiology. 1995;6(3):248–53.

Suissa S. The case-time-control design: further assumptions and conditions. Epidemiology. 1998;9(4):441–5.

Farrington CP, Whitaker HJ, Hocine MN. Case series analysis for censored, perturbed, or curtailed post-event exposures. Biostatistics. 2009;10(1):3–16.

Kuhnert R, Hecker H, Poethko-Muller C, Schlaud M, Vennemann M, Whitaker HJ, Farrington CP. A modified self-controlled case series method to examine association between multidose vaccinations and death. Stat Med. 2011;30(6):666–77.

Farrington CP, Hocine MN. Within-individual dependence in self-controlled case series models for recurrent events. J R Stat Soc C-Appl. 2010;59:457–75.

Article   Google Scholar  

Farrington CP, Anaya-Izquierdo K, Whitaker HJ, Hocine MN, Douglas I, Smeeth L. Self-controlled case series analysis with event-dependent observation periods. J Am Stat Assoc. 2011;106(494):417–26.

Consiglio GP, Burden AM, Maclure M, McCarthy L, Cadarette SM. Case-crossover study design in pharmacoepidemiology: systematic review and recommendations. Pharmacoepidemiol Drug Saf. 2013;22(11):1146–53.

Weldeselassie YG, Whitaker H, Farrington CP. Use of the self-controlled case-series method in vaccine safety studies: review and recommendations for best practice. Epidemiol Infect. 2011;139(12):1805–17. doi: 10.1017/S0950268811001531 .

Gault N, Castaneda-Sanabria J, Guillo S, Foulon S, Tubach F. Underuse of self-controlled designs in pharmacoepidemiology in electronic healthcare databases: a systematic review. Pharmacoepidemiol Drug Saf. 2016;25(4):372–7.

von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP, Initiative S. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007;370(9596):1453–7.

Navidi W. Bidirectional case-crossover designs for exposures with time trends. Biometrics. 1998;54(2):596–605.

Wang S, Linkletter C, Maclure M, Dore D, Mor V, Buka S, Wellenius GA. Future cases as present controls to adjust for exposure trend bias in case-only studies. Epidemiology. 2011;22(4):568–74.

Article   PubMed   PubMed Central   Google Scholar  

Farrington P, Pugh S, Colville A, Flower A, Nash J, Morgan-Capner P, Rush M, Miller E. A new method for active surveillance of adverse events from diphtheria/tetanus/pertussis and measles/mumps/rubella vaccines. Lancet. 1995;345(8949):567–9.

Luo X, Sorock GS. Analysis of recurrent event data under the case-crossover design with applications to elderly falls. Stat Med. 2008;27(15):2890–901.

Wang PS, Schneeweiss S, Glynn RJ, Mogun H, Avorn J. Use of the case-crossover design to study prolonged drug exposures and insidious outcomes. Ann Epidemiol. 2004;14(4):296–303.

Armstrong BG, Gasparrini A, Tobias A. Conditional Poisson models: a flexible alternative to conditional logistic case cross-over analysis. BMC Med Res Methodol. 2014;14:122.

Pouwels KB, Mulder B, Hak E. Moderate concordance was found between case-only and parallel group designs in systematic comparison. J Clin Epidemiol. 2016;71:18–24.

Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

van Staa T-P, Abenhaim L, Leufkens HGM. A study of the effects of exposure misclassification due to the time-window design in pharmacoepidemiologic studies. J Clin Epidemiol. 1994;47(2):183–9.

Schuemie MJ, Trifiro G, Coloma PM, Ryan PB, Madigan D. Detecting adverse drug reactions following long-term exposure in longitudinal observational data: The exposure-adjusted self-controlled case series. Stat Methods Med Res. 2016;25(6):2577–2592. doi: 10.1177/0962280214527531 .

Wang SV, Schneeweiss S, Maclure M, Gagne JJ. “First-wave” bias when conducting active safety monitoring of newly marketed medications with outcome-indexed self-controlled designs. Am J Epidemiol. 2014;180(6):636–44.

Hua W, Sun G, Dodd CN, Romio SA, Whitaker HJ, Izurieta HS, Black S, Sturkenboom MCJM, Davis RL, Deceuninck G, et al. A simulation study to compare three self-controlled case series approaches: correction for violation of assumption and evaluation of bias. Pharmacoepidemiol Drug Saf. 2013;22(8):819–25.

Jensen AK, Gerds TA, Weeke P, Torp-Pedersen C, Andersen PK. On the validity of the case-time-control design for autocorrelated exposure histories. Epidemiology. 2014;25(1):110–3.

Simpson SE, Madigan D, Zorych I, Schuemie MJ, Ryan PB, Suchard MA. Multiple self-controlled case series for large-scale longitudinal observational databases. Biometrics. 2013;69(4):893–902.

Maclure M. ‘Why me?’ versus ‘why now?’ Differences between operational hypotheses in case–control versus case-crossover studies. Pharmacoepidemiol Drug Saf. 2007;16(8):850–3.

Carnahan RM, Moores KG. Mini-Sentinel’s systematic reviews of validated methods for identifying health outcomes using administrative and claims data: methods and lessons learned. Pharmacoepidemiol Drug Saf. 2012;21(S1):82–9.

Murdoch TB, Detsky AS. The inevitable application of big data to health care. JAMA. 2013;309(13):1351–2.

Download references

Acknowledgements

Not applicable.

Availability of data and materials

The dataset supporting the conclusions of this article is included within the article’s Additional file 1 .

Authors’ contributions

Study conception NG and FT, data collection instrument NG and FT, search for articles NG JCS and SG, data collection NG, SF and JCS, data analysis NG, results interpretation NG JCS YDR and FT, writing the article NG JCS and FT, approval of the final version of the article NG, SF, YDR, SG, JCS and FT.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Ethics and consent to participate, author information, authors and affiliations.

APHP, Département d’Epidémiologie Biostatistiques et Recherche Clinique, Hôpital Bichat, 75018, Paris, France

Nathalie Gault

Université Paris Diderot, Sorbonne Paris Cité, UMR 1123 ECEVE, 75018, Paris, France

Nathalie Gault, Johann Castañeda-Sanabria, Yann De Rycke, Sylvie Guillo & Florence Tubach

INSERM CIC-EC 1425, Hôpital Bichat, 75018, Paris, France

APHP, Département Biostatistiques Santé Publique et Information Médicale, Centre de Pharmaco-épidémiologie de l’AP-HP, Hôpital Pitié-Salpétrière, 75013, Paris, France

Johann Castañeda-Sanabria, Yann De Rycke, Sylvie Guillo & Florence Tubach

Biostatistics unit, Gustave Roussy, 94800, Villejuif, France

Stéphanie Foulon

CESP, Université Paris-Sud, UVSQ, INSERM, Université Paris-Saclay, 94800, Villejuif, France

Université Pierre et Marie Curie, Sorbonne Universités, 75013, Paris, France

Florence Tubach

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Nathalie Gault .

Additional file

Additional file 1: s1..

MEDLINE search algorithm. S2. Standardized extraction form and specific validity assumptions. S3. Articles included in the review. S4. Validity assumptions for each article describing a case-crossover (4a) or a self-controlled case-series design (4b). S5. Validity assumptions for the use of a self-controlled design. Sensitivity analysis when unclear assumptions are considered invalid. S6. Validity assumptions for the use of a self-controlled design. Sensitivity analysis after excluding articles for the same series. (DOC 797 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Gault, N., Castañeda-Sanabria, J., De Rycke, Y. et al. Self-controlled designs in pharmacoepidemiology involving electronic healthcare databases: a systematic review. BMC Med Res Methodol 17 , 25 (2017). https://doi.org/10.1186/s12874-016-0278-0

Download citation

Received : 07 June 2016

Accepted : 15 December 2016

Published : 08 February 2017

DOI : https://doi.org/10.1186/s12874-016-0278-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Pharmacoepidemiology
• Self-controlled designs
• Observational studies
• Systematic review

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Hum Vaccin Immunother
• v.19(2); 2023
• PMC10402862

Risk of all-cause and cardiac-related mortality after vaccination against COVID-19: A meta-analysis of self-controlled case series studies

Greg marchand.

a Marchand Institute for Minimally Invasive Surgery, Mesa, AZ, USA

Ahmed Taher Masoud

b Faculty of Medicine, Fayoum University, Fayoum, Egypt

c Swedish Medical Center, Englewood, CO, USA

Self-controlled case series (SCCS) is a novel study design uniquely equipped to ethically quantify the safety of vaccination. We sought out to perform a meta-analysis on all SCCS assessing mortality associated with COVID-19 vaccination in the immediate post-vaccination period. We included SCCS investigating the safety of COVID-19 vaccination and reporting all-cause and cardiac-related mortality. Three SCCS were located, totaling approximately 750,000 patients. The pooled hazard ratio (HR) revealed no significant association of COVID-19 vaccination with all-cause mortality (HR = 0.89, 95% CI [0.71, 1.10], p  = .28). Regarding cardiac-related mortality, the pooled HR suggests that COVID-19 vaccination is associated with an increased risk of cardiac-related mortality (HR = 1.06, 95% CI [1.02, 1.11], p  = .007). Subgroup analysis showed that the male gender is significantly associated with an increased incidence of cardiac-related deaths (HR = 1.09, 95% CI [1.02, 1.15], p  = .006). In conclusion, COVID-19 vaccination may be associated with a small increase in cardiac-related mortality, especially among males. Prospero Prospective Registration Number : CRD42022372256

Introduction

Severe Acute Respiratory Syndrome/Coronavirus 2 (SARS-CoV-2) vaccination is a crucial prevention method that has helped to control the COVID-19 pandemic. The first such vaccine was licensed by the US Food and Drug Administration (FDA) for emergency use in December 2020. 1 , 2 Since then, there have been several COVID-19 vaccinations developed with different developmental technologies arising essentially all over the globe, including products from AstraZeneca, Janssen, Sputnik, Pfizer-BioNTech, Moderna, and Sinopharm. Although these vaccines show a great benefit in the prevention of COVID-19 infection and the reduction of hospitalization and mortality rates, 3–5 they have also been associated with many different reported adverse events. Some of the most common of these events include injection site tenderness pain, fatigue, headache, myalgia, and chills. 6–8 More serious events, including severe reactions and even death have been reported. 9–11 Previous studies showed that mRNA vaccines, such as Moderna and Pfizer-BioNTech may be related to myocarditis and myopericarditis in certain patient populations, 12 while other studies have suggested that the ChAdOx1 nCoV-19 vaccine (AstraZeneca) may be associated with an increased incidence of thromboembolic events. 13

The investigation and assessment of the vaccine safety profile are crucial aspects of any vaccination program. 14 In the case of response to the rapidly evolving COVID-19 pandemic, many vaccines have been approved for emergency usage without full FDA approval, as an appropriate response to the urgent need to protect the at-risk population. 15

In the presence of such a rapidly changing and dangerous pandemic, purposely exposing patients to the deadly pathogen or wide-scale inoculation with untested vaccination formulations is not feasible secondary to ethical and logistical limitations. Therefore, limited testing prior emergency use authorization was used in almost all cases (15.)

Self-controlled case series (SCCS) is a relatively newly developed study design and statistical methodology that is utilized in evaluating vaccine safety. 16 SCCS is developed to estimate the relative incidence of acute adverse events in a specific period after vaccination which is supposed to be the highest risk period compared to all other times, which represents the control period (observation period). Comparisons are made between the included participants, and only individuals who report the event are included in this study design; thus, participants act as their own control. 17

In response to a relative wealth of new studies on the topic of COVID-19 vaccine complications in this format, we sought out to perform a meta-analysis of all SCCS studies available on this topic. Our goal was to evaluate the all-cause mortality and cardiac-related mortality risk in the immediate period following COVID-19 vaccination.

We performed this systematic review and meta-analysis in accordance with the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement and the Cochrane Handbook of systematic reviews of interventions. 18 , 19 This study was registered to PROSPERO with the registration number: CRD42022372256. A PRISMA flow chart of our literature search can be found in Figure 1 .

PRISMA flow chart of the literature search.

Eligibility criteria

We included self-controlled case series (SCCS): (1) investigating the safety of COVID-19 vaccination and (2) reporting all-cause and cardiac-related mortality outcomes. The SCCS method is mainly used to evaluate vaccine safety as it calculates the relative incidence (RI) by performing a comparison between the incidence of an event in a specific period that follows exposure (high-risk period) with the incidence during a control period (which is all time in the follow-up period that is not the risk period). The most noted point of strength of the SCCS method is that fixed-time confounders, such as health-related risk factors, are controlled for. 15 , 16 , 20

We searched PubMed, Cochrane CENTRAL, Medline, ClinicalTrials.Gov, SCOPUS, and Web of Science for published articles that matched our inclusion criteria. We searched from each database’s inception until November 1, 2022, which was the last day of our search.

Ultimately, three self-controlled case studies were included in the final quantitative synthesis, with a total of approximately 750,000 patients. The first case study was performed in the United States, and included residents aged >18 years old receiving COVID-19 vaccination. 21 Included participants were followed up for 25 weeks after vaccination, and the risk period was defined as 28 days post-vaccination. The second study included young people in England 22 followed up for 12 weeks with a risk period defined as the 6 weeks post-vaccination. The third study was conducted in Italy 23 and defined their risk period as the first 30 days after vaccination. Unfortunately, none of the studies included more specific data for deeper analysis of exactly how soon after vaccination events occurred, and which vaccines were involved in these events.

Data analysis

We extracted outcome endpoints regarding all-cause mortality and cardiac-related mortality. We performed the meta-analysis of this study using Review Manager Software. 24 The effect estimates of the studies were pooled as hazard ratios (HRs) and 95% confidence interval (CI). Although this meta-analysis was limited to SCCS, as with any meta-analysis, we were extremely concerned about the heterogeneity of the data presented. As a meta-analysis, by definition, includes the combining of data from different performed studies, if that data shows similar results, thenthe data are homogeneous, and the combination of those studies is considered very strong evidence for the outcome the studies all portray. On the other hand, in cases where studies do not show the same outcomes, those results are considered heterogeneous, and care must be taken to consider why the different studies are showing different results. One example of this is an “outlier” study, which can skew data. We analyzed homogeneous data under the fixed-effects model and heterogeneous data under the random-effects model. We assessed the heterogeneity among studies using the I 2 and the p-value of the Chi-square tests. 19 Values of P  < .1 or I 2 >50% were significant indicators of the presence of heterogeneity. In cases where heterogeneity persists without explanation, the results of the meta-analysis are generally thought to be weaker than an analysis where the heterogeneity can be reduced using accepted strategies, or at least explained by the researchers.

Results of the meta-analysis

All-cause mortality.

The pooled hazard ratio (HR) revealed no significant association of COVID-19 vaccination with mortality (HR = 0.89, 95% CI [0.71, 1.10], p  = .28). Subgroup analysis by age revealed that, in the 18–24 age group, COVID-19 vaccination was not associated with an increased risk of all-cause mortality (HR = 1.04, 95% CI [0.81, 1.35], p  = .74). The pooled analysis was heterogeneous (I 2  = 94%), as seen in Figure 2a . Heterogeneity was resolved by excluding Stivanello et al., 23 and homogeneous results still failed to reach significant p values ( p  = .32), as seen in Figure 2b .

(a and b) All-cause mortality between the at-risk and control periods. Figure 1a represents the initial analysis, and Figure 1b shows the same analysis after excluding Stivanello et al. to solve the heterogeneity using the “leave-one-out” method.

Cardiac-related mortality

The pooled hazard ratio (HR) suggests that COVID-19 vaccination is associated with an increased risk of cardiac-related mortality (HR = 1.06, 95% CI [1.02, 1.11], p  = .007). Subgroup analysis showed that male gender is significantly associated with increased risks of cardiac mortality (HR = 1.09, 95% CI [1.02, 1.15], p  = .006). Subgroups of the female gender and 18–24 age groups showed no significant associations, as seen in Figure 3 .

Cardiac-related mortality between the at-risk and control periods, including subgroups by age and gender.

The analysis shows no connection between COVID-19 vaccination and an increased risk of all-cause mortality. However, we found a small, but statistically significant association between the vaccine and cardiac-related mortality. Subgroup analysis was performed to explore the possible risk factors identified male gender as the most important risk factor.

The results are similar to the evidence provided in the literature thus far. 25–27 There is now an increasing body of evidence that COVID-19 vaccination does not increase all-cause mortality in large populations. Some examples of this include Bardenheier et al. 25 in 2021 that found that vaccinated residents of nursing homes are significantly less likely to develop vaccine-related adverse events than, they were from actual COVID-19 infection. This study went on to show that the risk of mortality was significantly higher in unvaccinated residents than those who received vaccines. 25 Another example includes the cohort study Xu et al. 26 also in 2021, that analyzed mortality from non-COVID-19 causes. Again, lower mortality rates were found in the vaccinated group. 26 Although included in their selected outcomes, they did not find an association with increased cardiac risk, as was shown in our analysis.

The association between COVID-19 vaccination and increased incidences of cardiac-related disorders such as myocarditis has been described by several studies. 12 , 28 , 29 Goddard et al. 30 conducted a large study evaluating the incidence of cardiac side effects of the vaccine among 7 million people in the United States. They reported an incidence of 320 cases 1–98 days after approximately 7 million vaccine doses. 30

Some studies subgrouping by age showed a higher incidence of cardiac side effects in males compared with females. Katsoularis et al. for example, the highest incidence in males aged 18–25. 31 This study also reported an increased risk of myocardial infarction and stroke following actual COVID-19 infection. A large surveillance study on the US vaccinated individuals with myocarditis found that most individuals were considered recovered by healthcare providers after 3 months of the onset of myocarditis and that the outcomes on quality of life were comparable to those in age-matched pre-pandemic populations. 32

Our findings of increased cardiac-related deaths, especially in males, may have limited clinical utility, as the decision for vaccination should be individualized to each patient. We agree that a patient’s primary care physician should take into account each patient’s current and past medical history and consideration of each patient’s risk of serious disease or death from the virus they are receiving vaccination against. Although we hope this data is helpful in this calculus, we make no other specific recommendations.

As COVID-19 vaccination remains a major political and health concern in the United States, our researchers encountered many opinions, scientific and political, regarding the SCCS produced by Ladapo et al. 21 and published through the Florida Department of Health. We have thoroughly reviewed the opinions and calls for withdrawal from this study, and take these opinions as seriously as we can take any accusation against a data set we have incorporated into one of our reviews. We find no evidence challenging either the validity of the data in Ladapo et al. 21 or challenging the quality or bias of the data set contained therein. As for the interpretation of the data made by that group and their subsequent recommendations, we have no opinion, as our task was to compose a meta-analysis of all SCCS studies on this topic, and we believe we maintain a healthy “arm’s length” from these discussions. We have utilized the data and have no evidence that the data are incorrect. Further, we have author consensus that a publication officially produced by a Department of Health in the United States easily meets any reasonable definition of “published and peer reviewed,” in the same way that publications from the Center for Disease Control, (Atlanta, Georgia, USA) are essentially universally seen as such.

As for the strengths of our study, the design is an important factor as limiting our included studies to the SCCS study methodology completely controls for most fixed-time confounders. This methodology also allows for the inclusion of a large sample size, which we have included in this study. In addition to being the first meta-analysis to pool results from only SCCS study designs, another point of strength is that our analysis was very homogeneous.

As far as limitations, the major limitation is that although the studies themselves were quite large, there were a limited number of SCCS studies available for this analysis, at only three. There are also additional limitations inherent to the design of the SCCS study itself. The use of this design in COVID-19 vaccination programs violates the assumption that an event does not affect subsequent exposure (which could be especially true for mRNA vaccines), and this could in theory lead to a source of bias. As there is currently no described tool for assessing the bias inherent to SCCS studies, the authors see the risk of bias as a significant limitation of the present study.

Another major limitation is that the above 60 age group may lead to confounding in cardiac-related mortality. In the Ladapo et al. Study, 21 this group was the majority of all cardiac-related death rates. More importantly, when the authors removed this group from the analysis, the pooled risk incidence (RI) reported no significant association between mortality and COVID-19 post-vaccination (RI = 1.15, 95% CI = 0.99–1.34), mRNA vaccination (RI = 1.17, 95% CI = 1.00–1.37), and males with mRNA vaccination (RI = 1.09, 95% CI = 0.89–1.34).

The main finding of this meta-analysis is the lack of a connection between COVID-19 vaccination and an increased risk of all-cause mortality, when using all available data from self-controlled case series currently published on this topic. Additionally, in subgroup analysis, we found a statistically significant increase in cardiac-related death, especially in males. These findings, although statistically significant and backed by large sample sizes, were reached by the analysis of self-controlled case series studies, which may invite bias and therefore may be considered a lower level of evidence than analyses of RCTs or cohort trials. Further studies in this area will be necessary to judge the true risk of vaccine-related mortality, and more data will be needed to differentiate precisely which vaccine types and regiments this risk is most related to.

Acknowledgments

The Marchand Institute for Minimally Invasive Surgery would like to acknowledge the efforts of all the students, researchers, residents, and fellows at the institute who put their time and effort into these projects without compensation, only for the betterment of women’s health. We firmly assure them that the future of medicine belongs to them.

Funding Statement

The author(s) reported that there is no funding associated with the work featured in this article.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Authors’ contributions

All authors attest to significant contributions to this work.

Commitment to diversity

The Marchand Institute remains committed to diversity and tolerance in its research and actively maintains a workplace free of racism and sexism. We proudly state that the authors of this study represent diverse backgrounds and under-represented ethnic groups.

Consent to publish

Data used was exempt from consent to participate or publish secondary to the nature of the study being a systematic review, retrospectively looking at previously published data.

Ethics approval and consent to participate

The manuscript has been reviewed by the institutional IRB board at Marchand Institute and was found to be exempt from the IRB review (October 2022). Data used was exempt from consent to participate or publish secondary to the nature of the study being a systematic review, retrospectively looking at previously published data.

Patient consent

Not applicable to systematic review.