Statistical & Methodological Rigor

23.1 Purpose and Scope

  • This section defines minimum standards for study design, analysis, and reporting across our portfolio. It applies to quantitative, qualitative, and mixed-methods research. The goal is to make claims credible, transparent, and reproducible.

23.2 Study Design First

  • Question → Design → Analysis alignment: The analytical plan must match the research question and design (experimental, quasi-experimental, observational, qualitative, or mixed).
  • Protocol/preregistration: Provide time-stamped protocols and analysis plans where feasible; justify deviations.
  • Construct validity: Define exposures/interventions and outcomes precisely; explain measurement tools, calibration, and validation evidence.
  • Bias mapping: Identify likely biases (selection, information, confounding, measurement, survivorship) and address them in design/analysis.

23.3 Sampling & Power

  • Sample-size justification: Provide power or precision calculations with all inputs (effect size, variance, alpha, power, ICC for clusters, expected attrition).
  • Feasibility & stopping: State planned accrual, interim looks, and stopping rules where applicable.
  • Representativeness: Describe the sampling frame, inclusion/exclusion criteria, and any weighting.

23.4 Randomization, Masking, and Allocation (if applicable)

  • Sequence generation & concealment described in detail (method, blocking/stratification).
  • Blinding: Who was blinded and how blinding integrity was checked; if not possible, explain mitigation (objective outcomes, centralized assessment).

23.5 Data Quality & Preprocessing

  • Data provenance: Sources, linkage, and cleaning rules (outlier handling, winsorization, transformations).
  • QC checks: Missingness patterns, duplicate detection, range checks; predefine exclusion criteria.
  • Measurement reliability: Report instrument reliability/validity (test–retest, inter-rater, Cronbach’s alpha/omega where relevant).

23.6 Analysis Transparency

  • Primary vs. secondary analyses: Label clearly; identify exploratory work.
  • Model specification: Provide full model formulas, variable definitions, coding schemes, and inclusion criteria.
  • Software & versions: Name packages and versions; share environment files or containers.

23.7 Effect Sizes, Uncertainty & Inference

  • Report effect sizes with 95% confidence intervals (or credible intervals for Bayesian analyses) and exact p-values where used.
  • Avoid “statistical significance” as a binary proxy for truth; interpret magnitude, uncertainty, and context.
  • Multiplicity control: Predefine family-wise or FDR procedures for multiple comparisons; report adjustments.

23.8 Missing Data

  • Describe the mechanism (MCAR/MAR/MNAR) and rates by group/variable.
  • Use principled methods (e.g., multiple imputation, IPW) where appropriate; perform sensitivity analyses to assess robustness.

23.9 Assumptions & Diagnostics

  • Report checks for linearity, homoscedasticity, independence, distributional assumptions, influence/leverage, and collinearity.
  • Provide model fit indices and calibration where relevant (e.g., Hosmer–Lemeshow limits, calibration plots, Brier score).

23.10 Causal Inference in Non-Randomized Studies

  • Design first: Use DAGs or equivalent reasoning to justify adjustment sets.
  • Methods may include propensity scores (matching/weighting), instrumental variables, regression discontinuity, difference-in-differences, synthetic controls.
  • Test parallel trends/overlap/weak instrument risks and provide robustness & falsification checks; consider E-values or bias-sensitivity analyses.

23.11 Prediction & Machine Learning

  • Data splits: Separate train/validation/test; avoid leakage (e.g., scaling within CV folds only).
  • Cross-validation: Nested CV for model selection; report hyperparameters and tuning grids.
  • Performance: Discrimination (AUC/PR-AUC), calibration, and decision-centric metrics; include confidence intervals via resampling.
  • External validation: Preferably on out-of-distribution data; discuss dataset shift and generalizability.
  • Fairness & subgroup performance: Report metrics across salient subgroups; address imbalance and harms of misclassification.
  • Interpretability: Provide feature importance/SHAP or model cards with limitations and intended use.

23.12 Time-to-Event & Longitudinal Data

  • Survival: Proportional hazards checks; alternatives when violated (time-varying effects, AFT, flexible hazards).
  • Longitudinal/multilevel: Specify random effects, correlation structures, and intra-class correlations; justify missing-data handling.

23.13 Qualitative Rigor

  • Specify methodology (e.g., grounded theory, phenomenology, thematic analysis).
  • Sampling: Rationale and saturation.
  • Credibility & trustworthiness: Triangulation, member checking, audit trails, reflexivity statements.
  • Provide codebooks/frameworks and (where ethical) de-identified excerpts.

23.14 Mixed-Methods Integration

  • Describe design type (convergent, explanatory, exploratory), integration points, and how qualitative and quantitative strands inform each other (meta-inferences).

23.15 Robustness & Sensitivity

  • Predefine robustness checks: alternative specifications, influence diagnostics, bandwidth/placebo tests for quasi-experiments, alternative outcome definitions.
  • Report all material sensitivity results, not only those that favor conclusions.

23.16 Graphical & Tabular Reporting

  • Use clear axes, units, and scales; avoid deceptive truncation.
  • Show distribution-level information (e.g., scatter/violin/box) rather than mean ± SE alone.
  • Provide n per group and the definition of error bars.
  • Supply machine-readable tables or data files for plotted results.

23.17 Reproducible Computation

  • Share analysis code and minimal runnable examples reproducing key tables/figures.
  • Include seeds for stochastic procedures; document randomization schemes in simulations.
  • Prefer scripted workflows (Makefiles, notebooks with execution order, or pipelines) over manual steps.

23.18 Interim, Sequential, and Adaptive Analyses

  • Declare interim looks and alpha-spending; use appropriate boundaries (e.g., O’Brien-Fleming).
  • For adaptive designs, document adaptation rules and simulation-based operating characteristics.

23.19 Outliers & Influential Observations

  • Predefine rules for detection/handling; report sensitivity with and without influential points.
  • Never delete data solely for “improving significance.”

23.20 Pre-Spec vs. Post-Hoc

  • Label clearly. Distinguish pre-specified hypotheses from exploratory analyses.
  • Exploration is welcome when transparently reported and not reframed as confirmatory evidence.

23.21 Negative, Null, and Replication Studies

  • Welcome and evaluate on methodological quality, not outcome direction.
  • Replications should clarify direct vs. conceptual and align materials and protocols; report deviations.

23.22 Decision Thresholds & Claims

  • Avoid overstating certainty. Claims must reflect evidence weight, limitations, and generalizability.
  • Where policy/clinical thresholds are used, justify them and present decision-relevant trade-offs.

23.23 Editorial & Specialist Review

  • Editors may request statistical/methods review for complex analyses.
  • Authors must provide data/code and image originals on request; inability to verify may affect editorial decisions.

23.24 Checklists & Author Responsibilities

  • Submit a methods checklist covering items above (design, power, missing data, diagnostics, multiplicity, robustness, code availability).
  • Ensure all figures/tables match the reported analyses and that materials are accessible.

23.25 Sanctions for Non-Compliance

  • Pre-publication: revise-and-resubmit or rejection for inadequate rigor/transparency.
  • Post-publication: correction, expression of concern, or retraction if results are unreliable.