Performance & Telemetry calculator

A/B testing statistical significance calculator

Calculate statistical significance for web, app, and backend A/B tests covering conversion proportions (z-test / Chi-square) and continuous metrics (Welch t-test / Mann-Whitney U), with p-value, confidence interval, and sample size power evaluation.

Run this test live in StatLab npm i @statlab/core

When to use it

Use whenever comparing two variant groups in product feature experiments or conversion rate optimization.

Required Inputs

  • Control & Treatment sample sizes (N_A, N_B)
  • Conversions / Success counts (or metric means & SDs)
  • Confidence level
  • Minimum detectable effect (MDE)

Mathematical Formula

z = (p̂_B - p̂_A) / √[ p̂(1 - p̂)(1/n_A + 1/n_B) ], where p̂ = (x_A + x_B)/(n_A + n_B)

Reporting Cautions

  • Avoid continuous peeking at p-values without sequential testing or alpha-spending corrections.
  • Ensure sample ratio mismatch (SRM) checks pass before evaluating significance.

Code Snippets (Python, R, TypeScript)

Python (SciPy / Statsmodels) 
from statsmodels.stats.proportion import proportions_ztest
count = [1410, 1240]
nobs = [25000, 25000]
z, p = proportions_ztest(count, nobs)
print(f"z={z:.4f}, p={p:.4f}")
R Language 
prop.test(c(1410, 1240), c(25000, 25000))
TypeScript (@statlab/core) 
import { abTestSignificance } from '@statlab/core';
const res = abTestSignificance({ control: { n: 25000, conv: 1240 }, variant: { n: 25000, conv: 1410 } });

Developer Use Cases & Production Integrations

  • Automating continuous A/B experiment evaluation inside VoxelPulse feedback telemetry loops.
  • Determining statistical stopping rules for feature flag rollouts.
  • Measuring backend algorithm yield and conversion lift in production.

VoxelPulse Telemetry Control Plane

Continuous A/B testing, statistical telemetry analysis, and automated feedback loops for live backend services.

Explore VoxelPulse →

VoxelAssurance Release Readiness

Fixed-price AI release-readiness sprints, automated regression testing, and reliability statistical benchmarks.

Explore VoxelAssurance →

Related Statistical Test Calculators

Explore static calculators across the StatLab inference engine suite.