Performance & Telemetry calculator

LLM evaluation benchmark significance calculator

Test whether output quality, accuracy, toxicity, or rag-relevance score differences between LLM prompts or models are statistically significant using McNemar test (binary evaluation) or paired Wilcoxon / Bootstrap CI (continuous scoring).

Run this test live in StatLab npm i @statlab/core

When to use it

Use when comparing two LLM versions, system prompts, or RAG pipelines on identical evaluation benchmark test suites.

Required Inputs

  • Model A scores / pass-fail array
  • Model B scores / pass-fail array
  • Evaluation metric type (Binary Pass/Fail or Continuous 1-5 scale)

Mathematical Formula

McNemar χ² = (|b - c| - 1)² / (b + c), where b = A_pass/B_fail, c = A_fail/B_pass

Reporting Cautions

  • Always evaluate on paired prompt test sets to eliminate prompt difficulty confounding.
  • Account for LLM output non-determinism by running multiple temperature seeds per prompt.

Code Snippets (Python, R, TypeScript)

Python (SciPy / Statsmodels) 
from statsmodels.stats.contingency_tables import mcnemar
# table = [[both_pass, A_pass_B_fail], [A_fail_B_pass, both_fail]]
res = mcnemar([[745, 35], [90, 130]], exact=False, correction=True)
print(f"p={res.pvalue:.5f}")
R Language 
mcnemar.test(matrix(c(745, 90, 35, 130), nrow=2))
TypeScript (@statlab/core) 
import { llmEvalSignificance } from '@statlab/core';
const result = llmEvalSignificance(scoresModelA, scoresModelB);

Developer Use Cases & Production Integrations

  • Evaluating LLM release candidates in VoxelAssurance AI quality verification sprints.
  • Testing whether prompt engineering modifications yield statistically genuine accuracy gains.
  • Benchmarking RAG retrieval accuracy across vector index parameters.

VoxelPulse Telemetry Control Plane

Continuous A/B testing, statistical telemetry analysis, and automated feedback loops for live backend services.

Explore VoxelPulse →

VoxelAssurance Release Readiness

Fixed-price AI release-readiness sprints, automated regression testing, and reliability statistical benchmarks.

Explore VoxelAssurance →

Related Statistical Test Calculators

Explore static calculators across the StatLab inference engine suite.