Engineering case study
From a Gemini demo to a self-hosted legal reasoning platform
LegalAI started as a thin wrapper around one external model. It is now a provider-agnostic system where the domain quality comes from retrieval, a knowledge graph, fine-tuned heads, and a verifier — not from model scale — and where the deployment can run completely disconnected.
Why it was built
Contract review is high-stakes, repetitive, and full of text that a law firm or in-house team cannot send to a third-party API. The interesting version of this product isn't "call a big model" — it's "get genuinely useful analysis while the privileged text never leaves the building." That constraint is the whole project.
The vision
A legal reasoning platform an organisation runs on its own hardware — laptop, single server, or air-gapped cluster — where the quality comes from structure, retrieval, a knowledge graph, fine-tuned classifiers, and a verifier that can say "I'm not sure." An external API is allowed only where a cutover gate proves it wins, and only for documents whose tier permits it.
Design philosophy
Rule-based first, neural where it earns its place. Every degradation path is honest — the knowledge graph no-ops when Memgraph is down, the NLI head falls back to lexical overlap, generation raises a clear error rather than guessing. Nothing is auto-applied: redlines, overrides, and flagged analyses all route through a human.
Architecture decisions
Five choices that shaped the system, and the problem each one solved.
Provider-agnosticism is enforced, not intended
No module outside the Model Router's provider package may import a vendor SDK — a CI-gated import-linter contract. Services name a task; the router picks the provider. This is what keeps the air-gapped build actually air-gapped.
Sensitivity is a first-class property
Every document carries one of four tiers, assigned on upload. The Class-C gate keys on it, the router fails closed for confidential/privileged, and a PII redaction gate runs before any external dispatch. Before this existed, every call site passed "internal" implicitly and the gate protected nothing.
The verifier is the release gate
Every agent summary and grounded answer is entailment-checked against its sources. A fabricated citation, a knowledge-graph conflict, or an unsupported claim sets needs_human_review and drops the run into a real queue.
Model choices are gated by evaluation
A task routes to a self-hosted model by default only when a graded cutover gate shows it meets or beats the external baseline. "We chose not to ship this model" is a CI-enforced invariant with a model card explaining why.
Honest scope over impressive scope
The classical sensitivity and risk classifiers were trained, evaluated, and not promoted because they lost to the rules. That result is documented in a model card rather than hidden.
Research inspiration
Where the five novelty directions came from.
Deontic graph attention
Graph-attention networks over an obligation graph for cross-document conflict detection — inspired by the gap between string-match conflict detection and true semantic conflict.
Temporal obligation simulation
Auto-deriving downstream dates from "N days after <trigger>" conditional patterns, distinct from absolute-date extraction.
Legal-semantic fingerprinting
Contrastive clause embeddings trained on verified hard negatives — a modal-verb swap or a negation is textually tiny but legally opposite.
Adaptive negotiation playbooks
Procedural memory of an organisation's negotiation history, beyond static preferences.
Deontic-structure-aware ablation
Explaining a prediction by perturbing one legally-meaningful span at a time — a modal marker, an entity, a defined term.
Technologies used
Everything self-hostable; nothing that phones home by default.
Backend
FastAPI · Pydantic · SQLAlchemy · Postgres / SQLite · Redis
AI orchestration
LangGraph · DBOS durable execution · a declarative Model Router · import-linter
Models
Qwen3-8B/14B · bge-m3 + bge-reranker on TEI · DeBERTa NLI · GLiNER NER
Retrieval & graph
Hybrid RAG with RRF · rank-bm25 · FAISS · Memgraph / embedded KùzuDB
NLP / CV
regex-first pipeline · dateparser · PyMuPDF · OpenCV · Tesseract OCR fallback
Frontend
Next.js · TypeScript · Tailwind · TanStack Query · React Flow · Recharts
Eval & training
Inspect AI · LegalBench · MLflow · DVC · Argilla
Observability & governance
OpenTelemetry · SHA-256 egress audit trail · per-call routing logs · RBAC
Development journey
Eight phases, each one leaving the system deployable. Recorded in detail in a 60-plus-entry append-only engineering journal.
V1 hardening, then a FastAPI re-platform: Postgres/Redis, org-scoped auth, document persistence, and the provider-agnostic Model Router seam.
CPU NLP + CV pipelines turn flat text into a structured ClauseObject graph — segmentation, defined terms, deontic tagging, temporal normalisation, ambiguity detection.
Knowledge Graph (Memgraph) + hybrid RAG (dense + sparse + graph, RRF fusion) replace a hardcoded knowledge base.
A planner-driven LangGraph agent pipeline with a real NLI faithfulness verifier gating every summary.
The Model Router becomes genuinely provider-agnostic: embeddings and reranking self-hosted, Gemini demoted to an optional Class C plugin, an import-linter contract enforcing the boundary.
Self-hosted generation target (Qwen3-8B/14B), a graded eval harness with a cutover gate, GLiNER NER, the NLI head.
Document sensitivity tiering enforced end-to-end, a PII redaction gate + egress audit trail, per-user identity + RBAC, on-prem/air-gapped packaging, a collapsed data layer (KùzuDB), durable execution (DBOS), a three-tier memory service, and the first Next.js SPA.
Bitemporal graph versioning, cross-document consistency + simulation baselines, a negotiation drafting agent, the risk dashboard + knowledge-graph explorer, classical training runs (eval-gated, not promoted), MLflow + DVC, and five research prototypes from NOVELTY.md.
By the numbers
8
delivery phases, each deployable
400+
backend tests, green
40+
API endpoints, all typed
4
sensitivity tiers, enforced end-to-end
3
knowledge-graph / data-layer backends
0
model credentials required to run
Future work
What isn't done is either the next phase or explicitly blocked — on a GPU or on a person. Neither is hidden.
- Multi-tenant scale, SOC 2 and GDPR workflows
- Streaming agent trace over a session WebSocket
- Negotiation Studio — an interactive drafting surface
- Alembic migrations replacing the interim column shim
- Fine-tuned clause-type and deontic-tagger heads (pipeline verified via dry-run / smoke)
- The contrastive legal-clause embedding model (122 verified triplets built, CPU-side)
- Deontic-GAT training (architecture designed in DEONTIC_GAT_DESIGN.md)
- A transformer sensitivity classifier to beat the rules
- Legal-expert adjudication of weak labels (Argilla server runs; no expert in this environment)
- Formal prior-art / patent-counsel review of the five research directions
- Eval-gated promotion wired into CI to block a merge on a failed cutover gate
The through-line
Every phase moved inference toward the organisation’s own hardware and away from any external dependency. The bounded GPU reality — one 16 GB card — forced the architecture to be honest: an 8–14 B model plus strong retrieval and fine-tuned classifiers, with an external API kept only where a cutover gate proves it wins.