Abstract
Lawyer Assistant is an open-source, local-first retrieval-augmented generation (RAG) system for legal documents. It combines a hybrid retrieval pipeline — dense embeddings from BGE-M3, sparse BM25, and cross-encoder reranking with BGE-Reranker-v2-M3 — with a plan-based intent router and a human-in-the-loop agentic fallback to answer natural-language questions about contracts and case files with page-level citations. A separate clause-level compliance scanner classifies document sections and flags risky provisions against configurable playbooks. The system is engineered for fully offline operation: models are cached locally, GPU acceleration is provisioned automatically at install time (CUDA-aware PyTorch selection with force-swap of mismatched builds), and every document, embedding, and answer stays on the user's machine unless an optional cloud API mode is explicitly enabled. This report documents the architecture, retrieval methodology, orchestration, deployment engineering, and evaluation of the system as of release v1.1.1 (August 2026).
Keywords: retrieval-augmented generation; legal technology; hybrid retrieval; cross-encoder reranking; local-first AI; compliance analysis; document understanding
1. Introduction
Legal work is document work. Contracts, pleadings, regulatory filings, and case files accumulate in volumes that outgrow manual review, yet the consequences of missing a clause — or of trusting a fabricated citation — are severe. General-purpose AI assistants offer fluent answers but cannot be trusted with private documents: they require uploading the corpus to a third-party server, and they are known to hallucinate case law, a failure mode that has already produced sanctioned filings in real litigation (see §2.3).
Lawyer Assistant addresses both problems with a single design constraint: the system must run entirely on the user's machine, with no cloud dependency, while matching the answer quality of hosted legal AI tools. The contributions of this project are:
- A hybrid retrieval pipeline (dense + sparse + cross-encoder rerank) that grounds answers in the user's own documents with exact page and section citations (§4).
- A plan-based intent router with an explicit abstain behavior, ensuring the model answers only from retrieved evidence (§5).
- A clause-level compliance scanner that classifies contract sections and flags risky provisions against rule-based playbooks (§6).
- An install-time GPU provisioning system that selects and force-swaps PyTorch builds to match the user's hardware, making local acceleration reliable for non-expert users (§7).
- A workspace isolation model in which each project folder owns its own vector index, BM25 index, and history (§3.2).
2. Background and Related Work
2.1 Retrieval-augmented generation
RAG [1] grounds LLM outputs in an external corpus by retrieving relevant passages and conditioning generation on them. Dense retrieval encodes queries and documents into a shared vector space; multilingual models such as BGE-M3 [2] produce dense, sparse, and multi-vector representations and are specifically designed for high-quality cross-lingual retrieval. Hybrid systems combine dense and lexical (BM25) signals; reciprocal rank fusion (RRF) [3] is a standard, parameter-light method for merging result lists, and is one of the fusion strategies supported here.
2.2 Cross-encoder reranking
First-stage retrieval (dense and sparse) favors recall, returning a broad candidate set (here, top-50 from each). Second-stage reranking applies a cross-encoder — BGE-Reranker-v2-M3 [4] — that jointly encodes query and candidate and assigns a relevance score, improving precision on the final shortlist (here, top-5). This two-stage pattern is the dominant design in production RAG systems.
2.3 Hallucination risk in legal AI
The cost of hallucination in legal AI is not hypothetical. In Mata v. Avianca, Inc. (S.D.N.Y. 2023), counsel were sanctioned for submitting a brief containing fabricated case citations produced by a generative model [5]. Subsequent sanction orders — including Couvrette v. Wisnovsky (D. Or. 2025), which resulted in fines and fee awards exceeding $110,000 for two attorneys [6] — have established a clear regulatory trajectory: in 2026 the Florida Supreme Court adopted a rule addressing AI-hallucinated citations in court filings [7], and the American Bar Association's Formal Opinion 512 addresses the lawyer's duty of competence in using generative AI [8]. Systems that surface fabricated authorities therefore create direct professional-liability exposure. Lawyer Assistant's answer is architectural: the model cannot invent sources because generation is conditioned on retrieved chunks, every claim must cite a file, page, and section that exists in the user's corpus, and a relevance-verification step filters retrieved material before generation (§5.4).
2.4 Contract-understanding benchmarks
Public datasets such as CUAD (Contract Understanding Atticus Dataset) [9] and the subsequent golden-evaluation variants provide standard question sets for evaluating clause-level understanding; Lawyer Assistant's benchmark harness includes both (§8.2).
3. System Overview and Architecture
3.1 Process topology
The application is a three-process local stack orchestrated by an Electron shell:
| Process | Technology | Role |
|---|---|---|
| Shell | Electron (main process) | Window management; spawns backend and (in dev) the Vite server; IPC bridge |
| Frontend | Vite + React + TypeScript | Chat UI, sources panel, compliance scanner, visual pipeline editor; Zustand state stores |
| Backend | Python + FastAPI (uvicorn, port 8765) | All retrieval, orchestration, scanning, and streaming endpoints |
localhost:11434; embeddings and reranking use locally cached ONNX/PyTorch models under models/. All state persists locally: chat history in browser localStorage (Zustand), pipeline layout in localStorage['pipeline-layout-v1'], scan flags in backend SQLite, and agent thread state in data/conversations.db via LangGraph's AsyncSqliteSaver.
The backend exposes a single API surface, with streaming implemented uniformly as Server-Sent Events (SSE): chat tokens, thinking traces, tool calls, node status, scan progress, and flags all stream to the UI, so no user-facing operation blocks on a long-running request.
3.2 Workspace isolation
A workspace is a user-selected project folder. With an active workspace, the vector index, BM25 index, processed files, and history.json all live under
3.3 Data flow
Ingestion (/api/workspace/ingest/stream, SSE with progress) parses each supported document, chunks it, embeds it, stores it in the per-workspace ChromaDB index, and updates the BM25 index. Query time (/api/chat/stream) routes the user's message through the intent router, which plans, calls the retrieval pipeline, verifies relevance, and streams a cited answer. A background folder-watcher keeps the index in sync with files added to the workspace.
4. Retrieval Methodology
4.1 Ingestion: parsing and chunking
Documents are parsed with Docling, configured for English OCR at 150 DPI with page images and table-structure extraction. Parsed text is segmented by a structure-aware chunker with a target size of 480 tokens (ChunkingConfig.chunk_size = 480), a minimum of 32 tokens, and respect_sections=True, meaning headings and section boundaries are preserved as chunk delimiters rather than splitting mid-section. Each chunk carries provenance metadata — file name, page, and section — which is what enables page-level citation downstream.
4.2 Embeddings
Queries and chunks are embedded with BGE-M3 (BAAI/bge-m3), producing 1024-dimensional, L2-normalized vectors. BGE-M3 is a multilingual embedding model, so non-English source text remains searchable even though the UI and prompts are English. Embedding models are resolved from a local cache (models/bge-m3) before any network lookup, preserving offline operation after first-run setup.
4.3 Vector and sparse indexes
Dense vectors are stored in ChromaDB (collection legal_chunks, cosine distance, HNSW index) at workspace/bm25_index) provides lexical retrieval, which is essential for exact-phrase queries such as statutory citations and defined terms that dense retrieval can blur.
4.4 Query pipeline
query
├─ embed (BGE-M3, 1024-dim, L2-normalized)
├─ dense search — ChromaDB, top_k_retrieval = 50
├─ sparse search — BM25, top 50
├─ fusion — combine_and_dedup (default) | RRF (k=60) | boost_only
├─ rerank — BGE-Reranker-v2-M3 cross-encoder → top_k_final = 5, score ≥ 0.0
└─ response — {query, results[{file_name, page, section, score, text}], latency_ms}
Both first-stage retrievers return the top 50 candidates; the two lists are merged (by default with combine_and_dedup; reciprocal rank fusion with k=60 and a boost-only strategy are configurable), and the merged list is reranked by the cross-encoder to a final shortlist of 5 with a minimum score of 0.0. All tunables live in typed dataclasses (ModelConfig, ChunkingConfig, StorageConfig, SearchConfig, IngestionConfig) and can be overridden via PLR_* environment variables or, at runtime, through the visual pipeline editor (§7.4).
4.5 Retrieval modes
Three query modes are exposed: rag (default) runs the full intent-router orchestration and answers from documents; retrieval_only returns a formatted list of results with no LLM generation; direct generates from the LLM alone with no retrieval. This separation is deliberate — it makes the retrieval layer independently testable and lets power users verify what the system actually found.
5. Conversational Orchestration
5.1 Intent routing
The default path is a plan-based intent router. The LLM is given a strict prompt ("you MUST NOT answer directly; pick a tool"), classifies the user's intent, writes a one-to-two-sentence plan, and the router detects the tool to execute. The main search path calls search_documents with the user's exact query (never rephrased), then verify_relevance, then writes the answer from the retrieved chunks. Streaming events (intent, plan, tool_call, tool_result, sources, token, done) give the UI a full transparency trace of what the model did.
5.2 Agentic fallback with human-in-the-loop
A legacy agentic router — a LangGraph ReAct agent with a three-node StateGraph (agent → approval → tools) — serves retrieval_only/direct modes and conversation resume. Crucially, the approval node calls LangGraph's interrupt() before any tool executes; the UI presents an approval bar, and /api/chat/resume streams the decision back, resuming from the exact interrupt point. If the user rejects, the agent is instructed to answer without tools. This human-in-the-loop guardrail is the difference between an assistant that merely recommends actions and one that takes them.
5.3 Tool set
| Tool | Purpose |
|---|---|
search_documents(query, top_k=5, skip_rerank) | Runs the full retrieval pipeline; returns ranked chunks with file/page/section/score |
verify_relevance(query, documents) | LLM check for entity/topic match; returns per-document verified flags |
scan_document(document_id, playbook) | Runs the compliance scanner over an indexed document; persists flags |
ingest_file(path) | Parse → chunk → embed → store; makes a file searchable and scannable |
5.4 Honesty guardrails
Three mechanisms enforce answer integrity. Relevance verification filters retrieved chunks before generation. Abstention: the system is prompted to say it does not know when the documents do not contain the answer, and an abstain signal is separately tested. Citation validation: the frontend and backend validate that every cited source resolves to an indexed chunk, and citation/source parsing has dedicated test suites (§8.1). These mechanisms directly target the hallucinated-authority failure mode documented in §2.3.
6. Compliance Scanning
The compliance scanner treats contract review as a two-stage classification-and-rule task rather than free-form summarization:
document → chunk (with page/section metadata)
→ classify clause type per chunk (embedding classifier over BGE-M3 references)
→ for each rule in the active playbook: rule check (clause type, severity, keywords/LLM)
→ emit flags → persist to SQLite
Each playbook is a set of rules keyed to clause types (e.g., liability, termination, renewal), with severity levels and keyword/LLM conditions. Flags are surfaced in a dedicated UI and support a resolution workflow (resolved / dismissed / escalated). Streaming scan events (chunk_classified, rule_check, flag_found, scan_done) make long scans auditable in real time.
7. Deployment Engineering
7.1 Offline first-run
A first-run launcher pre-downloads Docling's layout/table models alongside BGE-M3, points HUGGINGFACE_HUB_CACHE at the local models/ directory, disables hf_xet/torch.compile, and runs an import smoke check — so a fresh install actually indexes PDFs instead of silently failing on a missing model or a network call.
7.2 CUDA-aware PyTorch provisioning
GPU acceleration is the single largest source of local-AI install failures. The setup procedure therefore: (1) probes the machine and installs the PyTorch wheel matching the hardware — CUDA 12.4 on NVIDIA GPUs, CPU-only otherwise — before installing the requirements file; (2) probes the resulting environment via torch.version.cuda; (3) force-swaps a mismatched pre-existing build (pip uninstall first) so an already-installed CPU torch cannot silently defeat GPU indexing; and (4) verifies torch.cuda.is_available() and logs the outcome.
7.3 GPU manager
A runtime GPU manager tracks device, free/total VRAM, per-model placement and residency, and resident Ollama models, with user-selectable residency profiles persisted per project. This turns GPU memory — a scarce, contended resource on consumer hardware — into an explicit, observable allocation.
7.4 Visual pipeline editor
A ReactFlow canvas exposes the pipeline as editable nodes (intent, planner, dense, sparse, rerank, ingest, scan, answer). The exported layout is not cosmetic: since /api/pipeline/config, it reconfigures the Python backend at runtime — toggling reranking, switching search mode, and overriding top_k and dense/sparse fusion weights via pipeline_config.py, which the retrieval pipeline reads on construction.
7.5 Update integrity
The installer/launcher writes a version manifest into the shipped payload; on launch, the launcher compares it with the copy in the user's state directory and re-copies the application when the payload is newer, rather than trusting a stale "already copied" marker. This closed the update loop in which an installed app could keep running stale code after an upgrade (a failure class that had shipped the white-screen fix to no one until this mechanism existed).
8. Evaluation
8.1 Test suite
The backend test suite is organized by system phase (boot, data model, chunking, embeddings, vector store, retrieval, end-to-end, GPU manager, pipeline config, workspace, citation validation, source parsing, abstain/temperature, post-tool gate) and currently comprises 255 passing tests, including 37 GPU-manager tests. Citation validation, source parsing, and abstention each have dedicated suites, reflecting the project's focus on answer integrity.
8.2 Retrieval benchmarks
A benchmark harness (backend/benchmark/) measures three dimensions — retrieval quality, component performance, and end-to-end accuracy — with standard metrics (Recall@K, MRR, Precision@K) over a 200-question dataset, a golden evaluation set, and CUAD [9]. Failure analysis categorizes and diagnoses misses, and per-component benchmarks cover chunking, embedding throughput, reranker speed-vs-quality, and retrieval accuracy.
8.3 Qualitative attributes
- Privacy: the default configuration performs no network calls after first-run setup; cloud API mode is opt-in.
- Explainability: every answer surfaces file/page/section sources, and SSE traces expose the model's plan and tool use.
- Cross-lingual retrieval: BGE-M3's multilingual representations make non-English corpora searchable despite an English UI.
9. Discussion
Several design decisions are worth highlighting. One FastAPI process owns both chat and scanning, with models warmed in a background thread at startup, which keeps the first query fast at the cost of a single-process failure domain. SSE everywhere trades a bit of protocol ceremony for a uniform streaming UX across chat, ingest, and scan. Hybrid retrieval as default reflects the observation that legal search is adversarial: exact terms and statutory citations matter as much as semantics, and BM25 alone or dense alone demonstrably misses a class of queries the other catches. Human-in-the-loop agentic routing accepts a small interaction cost in exchange for eliminating unauthorized tool execution. The intent router's insistence on the exact user query (never rephrased) for retrieval is a deliberate anti-drift measure: rephrasing by the model is a known source of retrieval degradation.
10. Limitations and Future Work
Current limitations include: English-centric OCR and prompts (retrieval is multilingual, but document parsing quality is strongest for English text); a dependence on a locally installed Ollama for generation (mitigated by the optional cloud API mode); a single-vector-store backend (ChromaDB) without query-time sharding for very large corpora; and benchmarks that are project-internal rather than published against external leaderboards. Future work targets multi-workspace cross-referencing, structured clause extraction beyond flagging, published benchmark results, and broader non-English document support.
11. Conclusion
Lawyer Assistant demonstrates that a privacy-preserving, fully local legal AI is not a compromise product: a hybrid dense/sparse/cross-encoder retrieval pipeline, a plan-based router with abstention, and a clause-level compliance scanner deliver grounded, citable answers without uploading a single document. Its install-time GPU provisioning and update-integrity mechanisms address the operational failures that typically kill local AI adoption. The project is open source under the MIT license, and its architecture — workspace isolation, streaming-first interfaces, honest-answer guardrails — is intended as a reference design for local-first legal technology.
References
- Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. https://arxiv.org/abs/2005.11401
- Chen, J., et al. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. 2024. https://arxiv.org/abs/2402.03216
- Cormack, G., Clarke, C., Buettcher, S. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR 2009.
- BAAI. BGE Reranker v2.0. 2024. https://huggingface.co/BAAI/bge-reranker-v2-m3
- Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y. June 22, 2023) (sanction order).
- Couvrette v. Wisnovsky, D. Or., 2025 WL 4109655 (Dec. 12, 2025) (sanction order).
- Florida Supreme Court, Rule 2.515 (adopted May 28, 2026; effective June 15, 2026).
- American Bar Association, Formal Opinion 512 (2024).
- Hendrycks, D., et al. CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. NeurIPS 2021. https://arxiv.org/abs/2103.06298
- GGUF Loader, GGUF Loader: Open-Source & Offline Local Model Runtime. https://github.com/GGUFloader/gguf-loader