Simulated power for an APE/AIE claim at a given sample size

Simulates studies from the pinned DGP, fits the index model by maximum likelihood, computes the APE (or, for moderated DGPs, the AIE) with a delta-method Wald CI, and evaluates the requested confidence-interval claim (Riesthuis, 2024): "minimum" (CI lower bound above the SESOI – the default and the package's point), "detect" (CI excludes 0, directional), or "equivalence" (CI within +/- SESOI). Also reports the full outcome distribution.

Usage

ape_power(
  dgp,
  n,
  claim = c("minimum", "detect", "equivalence"),
  sesoi = NULL,
  alpha = 0.05,
  conf = NULL,
  nsim = 1000,
  seed = NULL,
  se = c("model", "robust"),
  keep_draws = TRUE,
  separation = c("fail", "keep")
)

Arguments

dgp: A powerape_dgp after set_ape() or set_aie().
n: Total sample size of the simulated study.
claim: "minimum" (default), "detect", or "equivalence".
sesoi: Smallest effect size of interest, in APE units. Required for "minimum" and "equivalence"; optional for "detect" (if supplied, the outcome table is still broken out against it).
alpha: The claim's error rate (default 0.05): two-sided for "detect", one-sided for "minimum", TOST for "equivalence". May also be a sample-size rule: a function of n returning the error rate to use at that n, for example function(n) alphaN::alphaN(n, BF = 3), the Bayes-factor calibration of Wulff and Taylor (2024) that lowers alpha as n grows. The searching functions evaluate the rule at every candidate n, so ape_n() designs jointly over the pair (alpha(n), n); results store the rule and the level it produced.
conf: Optional override: the interval level used for the claim itself (1 - alpha for detection, 1 - 2 alpha otherwise). Supplying conf = 0.95 for a minimum-effect or equivalence claim reproduces the pre-1.8.0 behavior (one-sided error rate 2.5%).
nsim: Number of simulation replications.
seed: Optional seed (the caller's RNG state is preserved).
se: Standard errors for the exogenous cross-sectional routes: "model" (default, expected-information ML) or "robust" (heteroskedasticity-robust HC0 sandwich, as in the sandwich package). Panel designs always use unit-clustered SEs and IV designs the stacked method-of-moments robust sandwich; se is ignored there.
keep_draws: Keep the per-replication estimates, standard errors, and convergence flags in the result (default TRUE; a few kilobytes). They let power_at() and ape_alpha() re-evaluate the claim at any error rate without simulating again.
separation: How simulated studies with separation in an identifying cell are scored: "fail" (default; counted as failed, as the field's reference analysis refuses them) or "keep" (retained with their degenerate Wald intervals). See the section 'Separation and sparse cells'.

Value

A powerape_power object: power, Monte Carlo standard error, outcome distribution, the failed-fit count (failures count against power, conservatively), the share of simulated studies with separation (separated) and the average smallest identifying-cell count of events or non-events (min_cell, NA without binary cells), the separation convention used, the error rate used (alpha, and alpha_rule when it came from a sample-size rule), the stored draws, and the embedded DGP spec for reproducibility and power_statement().

Details

Error-rate convention. Every claim is tested at alpha (default 0.05) in its conventional form. Detection is the two-sided test at alpha, read off the 1 - alpha (95%) interval (only correctly signed rejections count, which costs no power). The minimum-effect claim is a one-sided test at alpha and equivalence a two-one-sided-tests (TOST) procedure at alpha; both read off the 1 - 2 alpha (90%) interval, the convention of Lakens (2017), Lakens et al. (2018), TOSTER, and Riesthuis (2024) for equivalence. The outcome distribution uses both intervals accordingly. To reproduce the more conservative choice of a 95% interval for the minimum-effect test (Riesthuis, 2024) pass conf = 0.95, which sets that claim's one-sided error rate to 2.5%. Two routes to an error rate that is chosen rather than inherited are described in vignette("justified-alpha"): a sample-size rule passed as alpha, and ape_alpha() for the error-cost optimum of Maier and Lakens (2022), computed from this function's stored draws.

Separation and sparse cells

With rare outcomes, small samples, or a small treated group, a simulated study can contain a cell that identifies the effect – a level of a binary focal variable, or for AIE designs a focal-by-moderator cell – with no events or no non-events. The maximum-likelihood effect does not exist there (quasi-complete separation): Stata's probit/logit drop the perfect predictor and margins reports the effect as not estimable, whereas R's glm() stops silently at a large finite coefficient whose delta-method standard error collapses to the other cell's binomial SE, so the replication usually counts as a detection. By default (separation = "fail") such replications count as failed, against every claim, like any other replication without a usable estimate; separation = "keep" retains them with their degenerate Wald intervals (the R glm() + marginaleffects analysis, and Stata's asis option; the behavior before powerape 1.11.0). Either way the share is reported (separated), and the function warns when it exceeds 1%, or when the smallest identifying cell averages fewer than 10 events (or non-events) per simulated study: there the Wald test's one-sided error rates can run up to about twice their nominal level in the direction in which the sparse cell has the lower rate, and score, Fisher, or Firth analyses have different power than the Wald analysis powered here.

Examples

# \donttest{
d <- ape_dgp(focal = pa_var("treat", "binary", p = 0.5), baseline = 0.30)
d <- set_ape(d, target = 0.10)
ape_power(d, n = 800, claim = "minimum", sesoi = 0.03, nsim = 500, seed = 1)
#> powerape -- minimum-effect claim (CI lower bound > 0.030)
#>   probit, n = 800, assumed true APE +0.1000, 90% CI (alpha = 0.05), nsim = 500
#>   power = 0.678 (MCSE 0.021)
#>   outcomes: minimum 0.678 | detect-only 0.176 | inconclusive 0.146 | equivalence 0.000 | failed 0.000 
# }