Prospective and retrospective sequential meta-analysis
One can use RTSA for both prospective and retrospective sequential meta-analysis.
To make a prospective meta-analysis using RTSA, a design is required. One can design a meta-analysis using the RTSA() function by specifying that the type of analysis is design:
RTSA(type = "design",...)Storing the RTSA design object and using it as a argument in the RTSA() function when data has been collected ensures that the design of the meta-analysis is not dependent on the data.
pro_design <- RTSA(type = "design",...)
pro_analysis <- RTSA(type = "analysis", design = pro_design,
data = trials)In the retrospective meta-analysis, there is no pre-existing design and the RTSA() function is provided with all design arguments such as alpha (type-I-error level), beta (type-II-error level), side (whether the analysis is one- or two-sided), es_alpha (the error spending function for alpha), futility (if any futility - which one), and more.
To run a retrospective meta-analysis in RTSA one sets the type to design. The difference between a retrospective and prospective analysis is that the retrospective analysis does not require a design.
retro_analysis <- RTSA(type = "analysis", alpha = alpha_level,
beta = beta_level, data = trials, ...)We will now provide two examples of respectively a prospective sequential meta-analysis and a retrospective sequential meta-analysis.
Prospective sequential meta-analysis
Let \(X\) be the treatment difference between treatment \(A\) and \(B\) such as a mean difference. Suppose that the research question of interest is: “Is X different from 0?”. Our null hypothesis is then that \(X=0\). Consider the scenario that 0.5 is a minimal clinical value of interest for the difference between treatment \(A\) and \(B\) and we expect a standard deviation of 1. We want to test at a 5% significance level at 90% power.
No trials on the topic exists, however three research centers have planned to run close to identical trials investigating the treatment difference between treatment \(A\) and \(B\). Each center can not individually recruit a sufficient number of participants to have a well-powered trial, hence they want to combine their trials in a meta-analysis. They decide that the number of participants should be split such that the first center accounts for 50% of the sample, whereas the remaining two account for 25%.
The centers want to stop and allocate the patients to the most effective treatment as soon as possible if early findings of efficacy of any of the treatments are found. They choose a Lan and DeMets’ version of O’Brien-Fleming boundaries for enabling early stopping. They are not interested in stopping for futility.
Regarding heterogeneity and the interpretation of the meta-analysis, they decide to use a fixed-effect meta-analysis based on the trials are close to homogeneous. However, presence of heterogeneity will be investigated.
We can then use RSTA to plan the prospective sequential meta-analysis. One can perform a prospective meta-analysis in RTSA by creating a design:
library(RTSA)
design_pma <- RTSA(type = "design", outcome = "MD", alpha = 0.05, beta = 0.1,
mc = 0.5, sd_mc = 1,side = 2, timing = c(0.5,0.75,1),
es_alpha = "esOF", fixed = TRUE, weights = "IV")
design_pma## Design with Trial Sequential Analysis was computed with the following settings:
##
## Boundaries for a 2-sided design with a type-I-error of 0.05, and type-II-error of 0.1.
## Futility is set to: none. Alpha-spending function: esOF.
## Beta-spending function: .
##
## The required information size is not adjusted by heterogeneity. The required information size is further increased with 2 percent due to the sequential design. The total required information size is 173.
##
## Timing, and boundaries:
## sma_timing upper lower
## 0.509 2.963 -2.963
## 0.764 2.359 -2.359
## 1.018 2.014 -2.014
## sma_timing is the ratio of the required sample for a sequential meta-analysis to a non-sequential meta-analysis sample size.
##
## Sample size calculation for standard meta-analysis:
## This is a prospective meta-analysis sample size calculation.
## The sample size calculation assumes a 2-sided test, equal group sizes,
## a type-I-error of 0.05 and a type-II-error of 0.1.
## The minimum clinical relevant value is set to: 0.5 for outcome metric MD.
## Additional parameters for sample size are:
## Standard deviation of the mean (standard error): 1.
##
## Fixed-effect required information size:
## 169 participants in total.
##
## For more information about the sample size calculation see vignette:
## 'Calculating required sample size and required number of trials'.
##
## Sample size calculation for sequential meta-analysis:
## Fixed-effect: 173 participants.
##
## Please note the following warnings:
## - The RTSA function is used for design. Boundaries are computed but sequential inference will not be calculated. Use the metaanalysis() function if interested in meta-analysis results.The output provides the required number of participants and boundaries for testing in a sequential meta-analysis. Hence for the three trials, the samples can be split as follows: 87, 43.5, 43.5.
Suppose that the first trial is done. We can then update the prospective meta-analysis by using the design in the call:
set.seed(0702)
treatA <- rnorm(n = ceiling(87/2), mean = 1.0, sd = 1)
treatB <- rnorm(n = ceiling(87/2), mean = 0.5, sd = 1)
trial1 <- data.frame(mI = mean(treatB), mC = mean(treatA), sdI = sd(treatB),
sdC = sd(treatA), nI = ceiling(87/2), nC = ceiling(87/2))
x <- RTSA(type = "analysis", design = design_pma, data = trial1)
x## Trial Sequential Analysis was computed with the following settings:
##
## Boundaries for a 2-sided design with a type-I-error of 0.05, and type-II-error of 0.1.
## Futility is set to: none. Alpha-spending function: esOF.
## Beta-spending function: .
##
## The required information size is not adjusted by heterogeneity. The required information size is further increased with 2 percent due to the sequential design. The total required information size is 173.
##
## Timing, boundaries, and test statistic:
## sma_timing upper lower fut_upper fut_lower z_fixed z_random
## 0.509 2.964 -2.964 NA NA -1.661 -1.661
## 1.018 1.969 -1.969 NA NA NA NA
## sma_timing is the ratio of the required sample for a sequential meta-analysis to a non-sequential meta-analysis sample size.
##
## Timing, outcomes, and confidences intervals for fixed-effect and random-effects models:
## sma_timing MD_fixed TSA_0.95lci_fixed TSA_0.95uci_fixed
## 0.509 -0.339 -0.944 0.266
## 1.018 NA NA NA
## sma_timing MD_random TSA_0.95lci_random TSA_0.95uci_random
## 0.509 -0.339 -0.944 0.266
## 1.018 NA NA NA
## lci is the lower limit of the confidence interval. uci is the upper limit of the confidence interval.
##
## Meta-analysis results:
## Fixed pooled effect (MD): -0.34 (95% TSA-adjusted CI: -0.94;0.27), naive p-value: 0.0967
##
## Heterogeneity results:
## tau^2: 0.00; I^2: 0.0%; D^2: 0.0%; Heterogeneity p-value: 1.0000
##
## Please note the following warnings:
## - The order of the Trial Sequential Analysis will be based on the order of the studies in the data-set. Please add a 'order' column in the data-set to specify the order.Type-I-error in sequential meta-analysis
The type-I-error will be controlled in a prospective sequential meta-analysis for at least fixed-effect meta-analyses, when first being designed and then updated using the same design. Changes to the original design such as having an additional trial will not impact type-I-error however it can impact the power of the meta-analysis. This is also true for unaccounted for presence of heterogeneity. If there is presence of heterogeneity then the trial will be under-powered but again it will not affect type-I-error control.
A paper reviewing some of these scenarios where we diverge from the design is under way. Both for the prospective meta-analysis and the retrospective sequential meta-analysis.
Retrospective sequential meta-analysis
The sequential retrospective meta-analysis can for many reasons be biased by the knowledge of results of trials used in the meta-analysis. Previous knowledge can affect the research hypothesis, which of the trials to be included in the meta-analysis, and more. Furthermore, the set of trials available might be overly positive due to publication bias. Most of these biases will cause a potential increase in type-I-error which cannot be handled by TSA.
Another cause of increase in type-I-error is the potential adaption of the method to the data of retrospective meta-analysis. These include leveraging information from the trials intended to use in the sequential meta-analysis. Such information could be the estimate of heterogeneity, or for binary data the probability of event in the control group. For good reasons, one might want to use the estimate of heterogeneity in the sample size calculation of the retrospective meta-analysis. However, the effect of being data-adaptive and not sticking to a pre-specified design can affect the control of type-I-error.
If one can accept not to use p-values as decision markers, a sequential retrospective meta-analysis is still of value. The method will provide the users with:
- Control of type-I-error in the scenario that the current state is the truth.
- A reminder user that the hypothesis has been tested multiple times and the type-I-error rate is affected.
- Quantification whether the meta-analysis is sufficiently powered or more trials are required to reach the wanted level of power.
An example of a retrospective meta-analysis is the eds data from the RTSA package. Here there is no design, so the function requires all of the relevant design settings such as setting the spending functions, type-I- and type-II-error levels, etc.
eds <- eds[order(eds$year),]
ex_retro_rtsa <- RTSA(type = "analysis", data = eds, side = 2, outcome = "MD",
alpha = 0.05, beta = 0.1, futility = "none",
fixed = TRUE, es_alpha = "esOF", mc = -1,
ana_times = c(3,6,9))## Warning in ris(outcome = outcome, mc = mc, side = side, alpha = alpha, beta =
## beta, : There is presence of tau2 (heterogeneity) and hence I2 and/or D2 is not
## 0. Consider changing fixed to FALSE.plot(ex_retro_rtsa)
Another example is the coronary data from the package.
ex2_retro_rtsa <- RTSA(type = "analysis", data = coronary, side = 2, outcome = "OR",
alpha = 0.05, beta = 0.1, futility = "non-binding",
fixed = FALSE, es_alpha = "esOF", es_beta = "esPoc", mc = 0.9)plot(ex2_retro_rtsa)