AI & Machine Learning calculator

Cohen's Kappa and inter-annotator agreement calculator

Calculate Cohen's Kappa (κ), Weighted Kappa (linear / quadratic), and percentage agreement for multi-annotator or LLM-judge evaluation alignment.

Run this test live in StatLab npm i @statlab/core

When to use it

Use when measuring inter-rater or judge agreement corrected for chance.

Required Inputs

  • Rater 1 classification categories
  • Rater 2 classification categories
  • Weighting scheme (Unweighted, Linear, Quadratic)

Mathematical Formula

κ = (p_o - p_e) / (1 - p_e)

Reporting Cautions

  • Kappa is sensitive to marginal category prevalence (prevalence paradox).
  • Use Weighted Kappa for ordinal rating scales.

Code Snippets (Python, R, TypeScript)

Python (SciPy / Statsmodels) 
from sklearn.metrics import cohen_kappa_score
kappa = cohen_kappa_score(rater1, rater2, weights='quadratic')
print(f"kappa={kappa:.4f}")
R Language 
library(irr)
kappa2(data.frame(rater1, rater2))
TypeScript (@statlab/core) 
import { cohensKappa } from '@statlab/core';
const k = cohensKappa(rater1, rater2);

Developer Use Cases & Production Integrations

  • Evaluating agreement between human evaluators and automated LLM-as-a-judge scoring in VoxelAssurance.
  • Measuring annotation consistency in supervised dataset labeling pipelines.

VoxelPulse Telemetry Control Plane

Continuous A/B testing, statistical telemetry analysis, and automated feedback loops for live backend services.

Explore VoxelPulse →

VoxelAssurance Release Readiness

Fixed-price AI release-readiness sprints, automated regression testing, and reliability statistical benchmarks.

Explore VoxelAssurance →

Related Statistical Test Calculators

Explore static calculators across the StatLab inference engine suite.