23.1 Purpose and Scope
- This section defines minimum standards for study design, analysis, and reporting across our portfolio. It applies to quantitative, qualitative, and mixed-methods research. The goal is to make claims credible, transparent, and reproducible.
23.2 Study Design First
- Question → Design → Analysis alignment: The analytical plan must match the research question and design (experimental, quasi-experimental, observational, qualitative, or mixed).
- Protocol/preregistration: Provide time-stamped protocols and analysis plans where feasible; justify deviations.
- Construct validity: Define exposures/interventions and outcomes precisely; explain measurement tools, calibration, and validation evidence.
- Bias mapping: Identify likely biases (selection, information, confounding, measurement, survivorship) and address them in design/analysis.
23.3 Sampling & Power
- Sample-size justification: Provide power or precision calculations with all inputs (effect size, variance, alpha, power, ICC for clusters, expected attrition).
- Feasibility & stopping: State planned accrual, interim looks, and stopping rules where applicable.
- Representativeness: Describe the sampling frame, inclusion/exclusion criteria, and any weighting.
23.4 Randomization, Masking, and Allocation (if applicable)
- Sequence generation & concealment described in detail (method, blocking/stratification).
- Blinding: Who was blinded and how blinding integrity was checked; if not possible, explain mitigation (objective outcomes, centralized assessment).
23.5 Data Quality & Preprocessing
- Data provenance: Sources, linkage, and cleaning rules (outlier handling, winsorization, transformations).
- QC checks: Missingness patterns, duplicate detection, range checks; predefine exclusion criteria.
- Measurement reliability: Report instrument reliability/validity (test–retest, inter-rater, Cronbach’s alpha/omega where relevant).
23.6 Analysis Transparency
- Primary vs. secondary analyses: Label clearly; identify exploratory work.
- Model specification: Provide full model formulas, variable definitions, coding schemes, and inclusion criteria.
- Software & versions: Name packages and versions; share environment files or containers.
23.7 Effect Sizes, Uncertainty & Inference
- Report effect sizes with 95% confidence intervals (or credible intervals for Bayesian analyses) and exact p-values where used.
- Avoid “statistical significance” as a binary proxy for truth; interpret magnitude, uncertainty, and context.
- Multiplicity control: Predefine family-wise or FDR procedures for multiple comparisons; report adjustments.
23.8 Missing Data
- Describe the mechanism (MCAR/MAR/MNAR) and rates by group/variable.
- Use principled methods (e.g., multiple imputation, IPW) where appropriate; perform sensitivity analyses to assess robustness.
23.9 Assumptions & Diagnostics
- Report checks for linearity, homoscedasticity, independence, distributional assumptions, influence/leverage, and collinearity.
- Provide model fit indices and calibration where relevant (e.g., Hosmer–Lemeshow limits, calibration plots, Brier score).
23.10 Causal Inference in Non-Randomized Studies
- Design first: Use DAGs or equivalent reasoning to justify adjustment sets.
- Methods may include propensity scores (matching/weighting), instrumental variables, regression discontinuity, difference-in-differences, synthetic controls.
- Test parallel trends/overlap/weak instrument risks and provide robustness & falsification checks; consider E-values or bias-sensitivity analyses.
23.11 Prediction & Machine Learning
- Data splits: Separate train/validation/test; avoid leakage (e.g., scaling within CV folds only).
- Cross-validation: Nested CV for model selection; report hyperparameters and tuning grids.
- Performance: Discrimination (AUC/PR-AUC), calibration, and decision-centric metrics; include confidence intervals via resampling.
- External validation: Preferably on out-of-distribution data; discuss dataset shift and generalizability.
- Fairness & subgroup performance: Report metrics across salient subgroups; address imbalance and harms of misclassification.
- Interpretability: Provide feature importance/SHAP or model cards with limitations and intended use.
23.12 Time-to-Event & Longitudinal Data
- Survival: Proportional hazards checks; alternatives when violated (time-varying effects, AFT, flexible hazards).
- Longitudinal/multilevel: Specify random effects, correlation structures, and intra-class correlations; justify missing-data handling.
23.13 Qualitative Rigor
- Specify methodology (e.g., grounded theory, phenomenology, thematic analysis).
- Sampling: Rationale and saturation.
- Credibility & trustworthiness: Triangulation, member checking, audit trails, reflexivity statements.
- Provide codebooks/frameworks and (where ethical) de-identified excerpts.
23.14 Mixed-Methods Integration
- Describe design type (convergent, explanatory, exploratory), integration points, and how qualitative and quantitative strands inform each other (meta-inferences).
23.15 Robustness & Sensitivity
- Predefine robustness checks: alternative specifications, influence diagnostics, bandwidth/placebo tests for quasi-experiments, alternative outcome definitions.
- Report all material sensitivity results, not only those that favor conclusions.
23.16 Graphical & Tabular Reporting
- Use clear axes, units, and scales; avoid deceptive truncation.
- Show distribution-level information (e.g., scatter/violin/box) rather than mean ± SE alone.
- Provide n per group and the definition of error bars.
- Supply machine-readable tables or data files for plotted results.
23.17 Reproducible Computation
- Share analysis code and minimal runnable examples reproducing key tables/figures.
- Include seeds for stochastic procedures; document randomization schemes in simulations.
- Prefer scripted workflows (Makefiles, notebooks with execution order, or pipelines) over manual steps.
23.18 Interim, Sequential, and Adaptive Analyses
- Declare interim looks and alpha-spending; use appropriate boundaries (e.g., O’Brien-Fleming).
- For adaptive designs, document adaptation rules and simulation-based operating characteristics.
23.19 Outliers & Influential Observations
- Predefine rules for detection/handling; report sensitivity with and without influential points.
- Never delete data solely for “improving significance.”
23.20 Pre-Spec vs. Post-Hoc
- Label clearly. Distinguish pre-specified hypotheses from exploratory analyses.
- Exploration is welcome when transparently reported and not reframed as confirmatory evidence.
23.21 Negative, Null, and Replication Studies
- Welcome and evaluate on methodological quality, not outcome direction.
- Replications should clarify direct vs. conceptual and align materials and protocols; report deviations.
23.22 Decision Thresholds & Claims
- Avoid overstating certainty. Claims must reflect evidence weight, limitations, and generalizability.
- Where policy/clinical thresholds are used, justify them and present decision-relevant trade-offs.
23.23 Editorial & Specialist Review
- Editors may request statistical/methods review for complex analyses.
- Authors must provide data/code and image originals on request; inability to verify may affect editorial decisions.
23.24 Checklists & Author Responsibilities
- Submit a methods checklist covering items above (design, power, missing data, diagnostics, multiplicity, robustness, code availability).
- Ensure all figures/tables match the reported analyses and that materials are accessible.
23.25 Sanctions for Non-Compliance
- Pre-publication: revise-and-resubmit or rejection for inadequate rigor/transparency.
- Post-publication: correction, expression of concern, or retraction if results are unreliable.