AI & Machine Learning calculator

Fleiss' Kappa calculator for 3+ raters

Calculate Fleiss' Kappa (κ) to measure inter-rater reliability across 3 or more fixed raters or LLM evaluators classifying items into categorical ratings.

Run this test live in StatLab npm i @statlab/core

When to use it

Use when 3 or more raters assign items into mutually exclusive categories.

Required Inputs

  • Rating count matrix (N items x K categories)
  • Category names

Mathematical Formula

κ = (P̄ - P̄_e) / (1 - P̄_e)

Reporting Cautions

  • Assumes raters are fixed and randomly sampled from a pool of equivalent judges.
  • Sensitive to overall category frequency distributions.

Code Snippets (Python, R, TypeScript)

Python (SciPy / Statsmodels) 
from statsmodels.stats.inter_rater import fleiss_kappa
kappa = fleiss_kappa(counts_matrix)
print(f"kappa={kappa:.4f}")
R Language 
library(irr)
kappam.fleiss(matrix)
TypeScript (@statlab/core) 
import { fleissKappa } from '@statlab/core';
const k = fleissKappa(matrix);

Developer Use Cases & Production Integrations

  • Measuring consensus agreement across multi-prompt LLM judge ensembles in VoxelAssurance.
  • Validating multi-annotator dataset quality.

VoxelPulse Telemetry Control Plane

Continuous A/B testing, statistical telemetry analysis, and automated feedback loops for live backend services.

Explore VoxelPulse →

VoxelAssurance Release Readiness

Fixed-price AI release-readiness sprints, automated regression testing, and reliability statistical benchmarks.

Explore VoxelAssurance →

Related Statistical Test Calculators

Explore static calculators across the StatLab inference engine suite.