Hallucination Detection for Legal LLM Input/Output
Hallucination scoring for contract review and legal research
⚠️ Research tool — not legal advice. This tool detects statistical patterns that correlate with hallucination. It does not verify case citations, confirm statute numbers, or validate contract terms against any authoritative legal database. A "Grounded" result means the response is semantically consistent with your source — not that it is legally correct. Always verify AI-generated legal analysis with a qualified attorney before acting on it.
How it works
- Select a scenario from the dropdown — or paste your own contract clause, statute, or case excerpt into the source document field.
- Toggle the response type — correct or hallucinated — and click Generate & Evaluate.
- Or paste any AI response manually and click Evaluate to score it directly.
Detection methods
Two independent detectors run on every evaluation and must both be considered together.
| Detector | Method | Speed |
|---|---|---|
| CERT (geometry) | Measures whether the response moved toward the source document in embedding space, or drifted away from it | ~5–50 ms |
| HHEM (classifier) | Reads source and response as text and checks for semantic contradiction | ~100–200 ms |
When both agree, confidence is high in either direction.
When they disagree, the response is geometrically in the correct topic region — it uses the right legal vocabulary in the right context — but likely contains a specific factual error: a fabricated case citation, a clause term that was never in the contract, a statute number that does not exist. This is what the research literature classifies as a Type III hallucination: factually wrong within a semantically correct frame. It is the most dangerous failure mode in legal AI and the hardest to catch automatically. Treat any disagreement as a flag for manual review.
Why geometry detects hallucination
LLM responses exist as vectors in a high-dimensional embedding space φ: T → ℝᵈ. A response that genuinely engages with a source document — a contract clause, a statute, a case holding — will be geometrically displaced toward that document's representation. A hallucinated response tends to remain anchored near the original question rather than moving toward the source.
Semantic Grounding Index (SGI) quantifies this as a distance ratio:
SGI(q, c, r) = ‖φ(r) − φ(q)‖ / ‖φ(r) − φ(c)‖
where q is the query, c is the source document, and r is the LLM response. A grounded response satisfies SGI ≥ 0.95 — it moved closer to the source than to the question. No trained classifier required. One embedding call, one ratio.
Directional Grounding Index (DGI) applies when no source document is present. It computes the displacement vector Δ = φ(r) − φ(q) and measures its alignment with μ̂ — the mean displacement direction of verified correct legal answers across eight calibrated domains:
DGI(q, r) = (Δ / ‖Δ‖) · μ̂
A score below 0.30 indicates the response trajectory is anomalous relative to verified legal reasoning patterns — a geometric signal of confabulation even without a reference document to compare against.
This geometric layer is a fast, model-agnostic first-pass filter. It catches where the response went in the embedding space. HHEM's learned classifier catches what the response says relative to the source. The two signals are orthogonal — running both is the point.
Built by Tony Sodano
Geometry: arXiv:2512.13771 · arXiv:2602.13224 · arXiv:2603.13259