Is your LLM eval lying to you?
sigeval — pytest for LLMs that isn't flaky.
A prompt scores 0.85 on your evals. Ship it? Depends entirely on how many samples that was. 17/20 passing tells you almost nothing. Enter your numbers and find out what your eval actually proves.
Stop guessing. Put this in your CI.
This calculator is one function from sigeval — the eval library that runs it on every commit, so a borderline result blocks the merge instead of shipping a coin-flip.
pip install sigeval
from sigeval import assert_eval
assert_eval("on_topic", scorer, sample=x, n_samples=20, threshold=0.8)
# PASSES only if the TRUE pass-rate is significantly above 0.8
Star on GitHub
Why "just check the average" fails
LLMs are non-deterministic — the same prompt scores differently run to run. A single pass-rate is a point estimate with real uncertainty around it. sigeval reasons about the confidence interval: it only returns PASS when the true rate is significantly above your bar, FAIL when significantly below, and INCONCLUSIVE when you simply haven't sampled enough to know. That's the difference between a green build you can trust and one that flips red next commit.