AI Legal Case Preparation Software Evaluation Guide

Definitions

Field definitions

Controlled vocabulary guidance

use_case_class

Examples: Matter organization, Source-grounded summarization, Chronology or timeline assembly, Issue and fact extraction, Question or interview preparation, Research organization, Exhibit or bundle preparation, Drafting assistance, Restricted or prohibited use

Governance: Choose the narrowest class that describes the evaluated workflow. Do not use a broad class to conceal different source, review, privilege, or consequence boundaries.

source_status

Examples: Approved, Pending review, Disputed, Superseded, Unavailable, Restricted, Out of scope

Governance: Status describes the source or proposition in the declared workflow. It is not a conclusion about truth, legal effect, or admissibility.

review_decision

Examples: Accepted for defined use, Accepted with limitation, Correct and re-review, Reject and hand off, Insufficient evidence, Access denied, Not assessed

Governance: Require a reason, reviewer, scope, and next action. A review decision must not be interpreted as a universal quality or legal result.

error_type

Examples: Unsupported output, Source misattribution, Material omission, Wrong entity or matter, Wrong date or version, Scope drift, Privilege or access issue, Unsafe continuation, Integration or workflow failure, Formatting or usability defect

Governance: Record one primary error type and any secondary types. Preserve the affected source, output, reviewer rationale, severity, and correction or handoff.

operating_state

Examples: Design, Evaluation, Pilot with named users, Approved for defined use, Restricted, Paused, Retired

Governance: Operating state is a governance status for the defined boundary. It does not represent model accuracy, legal compliance, or suitability outside the approved scope.

Practical workflow

1. Define the decision and use-case boundary

   Name the preparation activity, user, matter type, jurisdiction, source universe, intended audience, and decision the output may support. Separate low-consequence organization tasks from legal analysis, filing, advice, settlement, discovery, privilege, or client communication. State what the system is prohibited from doing and the exact human or alternate-process handoff.

2. Inventory candidate case-preparation uses

   List the actual tasks to evaluate, such as matter intake, chronology assembly, issue spotting, document summarization, witness preparation, research organization, exhibit indexing, deadline extraction, question drafting, or source comparison. For each task record inputs, outputs, reviewer, expected effort, failure consequence, source boundary, and whether the output can be regenerated.

3. Declare the source and matter boundary

   Specify which matters, repositories, folders, document types, metadata, messages, transcripts, orders, public authorities, and integrations are in scope. Record excluded sources, unsupported formats, jurisdiction limits, date cutoffs, duplicate handling, access conditions, and whether a result may cross matter, client, office, or ethical-wall boundaries.

4. Map the source-grounding path

   Trace how a source becomes available to the workflow: ingestion, synchronization, indexing, retrieval, context selection, transformation, prompt or instruction, model call, output, citation, and review. Require stable document or passage references where the use case needs them. Record what happens when the source is missing, stale, conflicting, inaccessible, or outside the declared matter boundary.

5. Specify output and review obligations

   Define the output format, required citations or source references, materiality flags, uncertainty wording, reviewer role, review depth, approval step, and storage location. Distinguish review of factual support, legal characterization, privilege, confidentiality, completeness, formatting, and downstream use. Do not accept a generic statement that a user should review everything without specifying the review record.

6. Design a representative reference test set

   Build a versioned set from de-identified or otherwise authorized matters and include ordinary, sparse, long, contradictory, multilingual, scanned, handwritten, duplicate, superseded, privileged, restricted, date-sensitive, and adversarial examples as relevant. Include cases where the correct behavior is to abstain, ask for clarification, identify a gap, or hand off rather than produce an answer.

7. Define expected review criteria

   For every test item, record the task scope, eligible sources, expected facts or fields, acceptable citations, prohibited inferences, required exclusions, reviewer rubric, severity if wrong, and pass or fail rule. Use a qualified reviewer for legal interpretation and a separate technical or operations reviewer when the workflow depends on retrieval, access, or integration behavior.

8. Test grounding and provenance

   Run the reference set with fixed inputs and capture the retrieved sources, source passages or identifiers, prompt or configuration, model or feature version, output, citations, reviewer decision, and corrections. Check whether a reviewer can reproduce the path from output to source and distinguish source text, system transformation, user instruction, and human judgment.

9. Measure errors with named denominators

   Report task-specific measures such as supported statement rate, citation traceability rate, material omission rate, wrong-entity rate, wrong-date rate, unsupported-output rate, abstention or handoff rate, reviewer correction rate, and severity-weighted error count. Publish the tested population, exclusions, unit, adjudication method, confidence or uncertainty treatment, and whether results are sample-based or census-based.

10. Evaluate human review ergonomics

    Observe whether reviewers can inspect the source, compare the output, see uncertainty and limitations, correct or reject a result, record a reason, escalate a concern, and prevent unreviewed downstream use. Test workload, interruption, conflicting reviewer decisions, missing citations, stale source versions, and the behavior when a reviewer cannot complete the gate.

11. Review privilege, confidentiality, and access

    Map matter, client, office, team, role, ethical-wall, document, field, prompt, output, administrator, support, and integration access. Test least-privilege behavior, tenant separation, export and sharing controls, service accounts, logs, administrator visibility, training or improvement use, subprocessors, and access removal. Require an authorized decision for privileged or restricted material and record the boundary.

12. Set retention, deletion, and hold behavior

    Record retention owners, source and output classes, version history, temporary context, prompts, logs, caches, backups, exports, derived indexes, and model-provider storage. Test ordinary deletion, legal hold, matter closure, correction, supersession, access withdrawal, backup expiry, and provider-side deletion or isolation commitments. Preserve the evidence required to reconstruct a material review without retaining data beyond the approved basis.

13. Inspect audit and change records

    Confirm that the system records identity, matter, source scope, retrieval, prompt or instruction, model or feature version, output, citation, reviewer action, correction, approval, export, integration event, failure, and administrative change when applicable. Test time ordering, tamper evidence, search, export, access to logs, clock treatment, and whether a record can distinguish automated activity from human action.

14. Test integration contracts

    Evaluate identity, case management, document management, email, calendar, research, billing, e-discovery, storage, messaging, and reporting integrations only where the use case requires them. Record fields, identifiers, permissions, sync direction, event timing, retries, duplicate handling, conflict resolution, version behavior, error visibility, rate limits, and rollback or manual fallback. Do not treat a connector listing as proof of a working workflow.

15. Exercise failure modes and fail-closed paths

    Inject missing or conflicting sources, stale indexes, retrieval failure, prompt injection, malicious or misleading documents, unsupported languages or formats, wrong matter selection, duplicate versions, timeout, partial sync, unauthorized access, model or feature change, and unavailable reviewers. Verify that the system labels the condition, preserves useful evidence, blocks unsafe continuation, and routes to a named owner or alternate process.

16. Document acceptance and operating ownership

    Create an acceptance record for each use case with scope, test-set version, results, exceptions, unresolved limitations, access decision, retention basis, integration evidence, review design, rollback path, owner, approver, training, monitoring cadence, change triggers, and sunset condition. Approve the workflow only for the tested boundary. Re-test after material source, model, prompt, feature, policy, integration, matter, or jurisdiction changes.

Comparison

Limitations and exceptions

• This guide is an organization-designed evaluation framework. It does not certify a vendor, product, model, workflow, security control, legal service, or result.
• Test-set results describe the defined population, task, source set, configuration, reviewers, and evaluation period. They do not establish behavior for every matter, document, language, jurisdiction, user, model version, or future change.
• A citation, retrieval trace, or provenance record shows a relationship between an output and recorded inputs. It does not prove that the source is complete, authentic, current, privileged, legally sufficient, or correctly interpreted.
• Human review is a required governance activity for the defined workflow, not a guarantee that a reviewer will detect every omission, unsupported statement, access error, privilege issue, or harmful downstream use.
• Rates for unsupported outputs, omissions, corrections, abstentions, or other errors depend on the taxonomy, denominator, adjudication rule, sample design, task scope, and severity treatment. Do not compare rates across products without aligned definitions.
• Privilege, confidentiality, records, privacy, security, retention, and professional-responsibility duties depend on the matter, client, jurisdiction, agreement, provider architecture, and applicable policy. Qualified owners must make the governing decisions.
• An integration that transfers data or displays an output does not establish that identifiers, permissions, versions, audit events, retries, deletions, or failure states are preserved correctly.
• Acceptance is limited to the stated boundary and evidence. Reopen it after material source, model, prompt, feature, policy, integration, access, incident, matter, or jurisdiction changes.

Primary sources

Methodology

This is an organization-designed evaluation method checked against the cited NIST and ABA primary sources on August 13, 2026. Start with a narrow case-preparation use case and declare the matter, client, jurisdiction, source, user, output, decision, and access boundary. Map the complete path from source ingestion or synchronization through indexing, retrieval, prompt or instruction, model or feature processing, output, citation, human review, correction, approval, export, retention, and deletion. For every use case, define permitted and prohibited actions, required source references, abstention and handoff conditions, reviewer role, severity of failure, and downstream use. Build a versioned reference set from authorized or de-identified material and include ordinary, sparse, contradictory, multilingual, scanned, duplicate, superseded, privileged, restricted, adversarial, and missing-source cases when relevant. For every test item record eligible sources, expected facts or fields, acceptable citations, prohibited inferences, reviewer rubric, and pass rule. Report metrics with explicit populations, units, denominators, exclusions, adjudication method, severity, and sample or census basis. Useful organization-designed measures include supported statement rate = material output statements with an approved source reference and reviewer confirmation / all material output statements reviewed x 100; source traceability rate = material output statements for which a reviewer can locate the cited source passage or record in the declared source set / all material output statements requiring grounding x 100; unsupported-output rate = reviewed output statements that lack support in the approved source set or exceed the task boundary / all material output statements reviewed x 100; material omission rate = material expected facts or fields absent or materially incomplete in the output / all material expected facts or fields in the test oracle x 100; wrong-entity rate = outputs assigning a fact, date, document, person, or event to the wrong matter or entity / all tested assignments x 100; reviewer correction rate = outputs requiring a recorded substantive correction before the approved downstream use / all outputs sent to human review x 100; abstention or handoff rate = test cases correctly stopped or routed because the source, access, scope, or review condition was insufficient / all test cases requiring abstention or handoff x 100; and provenance completeness = required provenance events present and ordered for a tested run / all required provenance events for that run x 100. These are measurement definitions, not universal thresholds or claims about product quality. Keep unsupported output, omission, misattribution, wrong date, privilege or access issue, unsafe continuation, and integration failure as distinct error types. Test allowed and denied access, matter and client isolation, ethical walls, administrator and provider visibility, temporary context, prompts, logs, indexes, backups, exports, holds, deletion, and access withdrawal. Test integrations for identifiers, permissions, versions, timing, retries, duplicates, conflicts, partial completion, and rollback. Record failure evidence and use fail-closed handoffs where the defined policy requires them. Acceptance should name the tested boundary, reference-set version, results, unresolved limitations, exceptions, operating owner, approver, training, monitoring cadence, change triggers, and sunset condition. A material source, model, prompt, feature, access, policy, integration, incident, matter, or jurisdiction change reopens evaluation. Human reviewers and supervising counsel remain responsible for the legal judgments and authorized downstream actions within the applicable professional and organizational rules.

FAQs

Related CaseDocker capabilities

Related guides and resources

• Legal AI citation verification workflow
• Legal AI prompt and output audit logs
• Legal RAG knowledge base governance
• AI legal chronology generation controls
• AI contract obligation extraction accuracy testing