Legal AI Pilot Success Metrics

Definitions

Field definitions

Controlled vocabulary guidance

Quality result

Examples: Pass, Partial, Fail, Not assessable, Disputed, Withdrawn

Governance: Define each state in the rubric, preserve the original output, and require adjudication for disputed or multiple-defensible answers.

Error severity

Examples: Critical, High, Moderate, Low, Informational

Governance: Use organization-approved anchors tied to decision impact, affected population, exposure, detectability, and required containment; do not infer severity from frequency alone.

Abstention result

Examples: Appropriate abstention, Inappropriate abstention, Unsupported answer, Human escalation, Not an abstention opportunity

Governance: Maintain a task-level expected action and explain why the observed response did or did not satisfy the boundary.

Citation status

Examples: Valid, Unsupported, Fabricated, Stale, Inaccessible, Ambiguous, Not checked

Governance: Check source identity, version, location, proposition support, and reviewer access. Keep unchecked citations out of the valid numerator.

Pilot decision

Examples: Stop, Contain, Revise, Hold, Limited go, Go within scope

Governance: Tie each decision to versioned criteria, evidence, residual risk, authority, conditions, expiry, and next review; a go decision is not a performance guarantee.

Failure disposition

Examples: Open, Contained, Remediation planned, Accepted with owner, Re-test required, Closed with evidence

Governance: Preserve the original failure and require an owner, due date or trigger, compensating control, and closure evidence.

Practical workflow

1. State the pilot decision

   Write the decision the pilot must support: refine the workflow, expand to a named cohort, contain use, stop the test, or hold for more evidence. Identify the accountable decision owner and the authority for access, client use, data handling, and production release.

2. Declare the eligible population

   Define eligible users, roles, matters, document sets, requests, or benchmark tasks with a stable query or roster, effective date, jurisdiction, access rule, reporting period, and owner. Count eligible, ineligible, unknown, unavailable, and not-applicable records before interpreting a rate.

3. Set the pilot boundary

   Record permitted use cases, excluded use cases, data classes, jurisdictions, matter types, model or configuration version, human-review requirement, retention rule, escalation route, and prohibited decisions. A pilot boundary is part of the denominator and interpretation contract.

4. Build representative benchmark tasks

   Create task IDs across routine, difficult, rare, high-impact, ambiguous, multilingual, incomplete, and adversarial examples that are relevant to the declared workflow. Preserve source references, expected evidence, risk tier, evaluator instructions, and the reason each task is representative or intentionally stress-oriented.

5. Define the reference and rubric

   For every task, state the expected answer or acceptable range, required citations, material-error categories, abstention conditions, privacy or security checks, and reviewer adjudication rule. Permit a documented unresolved or multiple-defensible-answer state instead of forcing false certainty.

6. Freeze instrumentation and versions

   Record the model, prompt or template, retrieval source, index or knowledge-base version, configuration, user role, data snapshot, timestamp, latency clock, evaluator version, and metric version. Do not compare outputs across silent changes in task mix, source corpus, model, or rubric.

7. Measure quality and errors

   Evaluate each eligible task against the rubric and preserve pass, partial, fail, not assessable, and disputed states. Report quality pass rate, material-error rate, error categories, severity, affected task IDs, reviewer, and adjudication. Do not hide a failed task because a reviewer corrected it.

8. Measure abstention and escalation

   Separate appropriate abstentions, inappropriate abstentions, unsupported answers, human escalations, and tasks that should not have been attempted. Use the eligible abstention-opportunity population for the abstention measure and retain the task-level reason, expected action, and final disposition.

9. Validate citations and source grounding

   Sample or census citations at the citation level. Check that each citation resolves, refers to the intended source and version, supports the proposition, and is available to the authorized reviewer. Record missing, stale, inaccessible, contradictory, fabricated, or overbroad citations as failures or limitations.

10. Measure reviewer effort and rework

    Define review start and stop events, pause rules, reviewer role, and unit. Track median and percentile review time, outputs requiring material correction, correction categories, reruns, escalations, and unresolved work. Keep model output time separate from human review time and queue delay.

11. Measure adoption and workflow completion

    Count eligible users who perform the declared qualifying action and eligible workflow instances that reach the accepted terminal state. Report access, activation, repeat use, completion, and non-use separately by role, cohort, matter type, office, and configuration version.

12. Monitor security, privacy, and support

    Record confirmed and suspected incidents, near misses, unauthorized data paths, access failures, retention exceptions, policy violations, support contacts, severity, response time, affected population, and unresolved risk. Report counts and rate bases without treating a zero observed incident count as proof of zero risk.

13. Measure cost and latency with units

    Publish direct pilot spend, usage or inference units, cost per completed qualifying task, and cost categories separately. Measure request-to-first-output and request-to-final-output latency in seconds or minutes using p50, p95, maximum, timeout, and missing-clock counts. Do not convert these measures into promised savings.

14. Review the failure register

    Reconcile all benchmark tasks, production-like tasks, rejected inputs, abstentions, incidents, support cases, timeouts, citation defects, reviewer corrections, unknowns, and excluded records. Every failure needs a stable ID, category, severity, owner, containment or remediation, status, and decision impact.

15. Apply stop or go governance

    Compare the complete evidence set with approved gates. Stop or contain for a defined critical failure, unresolved privacy or security issue, unsafe material-error pattern, unsupported use, or missing authority. Go only for the stated scope when evidence meets the approved criteria and residual risks, limitations, owners, review cadence, and rollback route are accepted.

Comparison

Limitations and exceptions

• Pilot results are conditional on the declared population, tasks, sources, model, configuration, reviewer rubric, period, and operating controls. They do not establish performance for every legal workflow or future release.
• Quality, citation, adoption, cost, and time measures can be distorted by task mix, reviewer disagreement, missing events, access limits, learning effects, concurrent process changes, and selective participation.
• A measured association between the pilot and an outcome does not prove that the AI system caused the outcome. Do not claim causal improvement, avoided cost, savings, or ROI without an appropriate design and validated accounting treatment.
• A low observed incident count can reflect limited exposure, weak monitoring, incomplete reporting, or a short pilot. Preserve suspected events and near misses, and state detection and coverage limits.
• Benchmark tasks can become stale, overfit, or unrepresentative. Version task sets, refresh them after workflow or source changes, and include difficult, ambiguous, excluded, and failure cases.
• A valid citation does not by itself make an output legally correct, complete, privileged to disclose, or suitable for a client, court, regulator, or business decision.
• Stop or go criteria are governance controls designed by the organization. They are not universal legal thresholds, certifications, warranties, or permission to remove qualified human review.

Primary sources

Methodology

Use a versioned metric contract for every pilot measure. The contract must state the decision purpose, unit of analysis, eligible population, reporting period, time zone, source systems, event definitions, exclusions, unknown handling, reviewer or adjudicator, formula, owner, and interpretation limits. Freeze the eligible population before calculating rates. Keep population counts, task counts, citation counts, output counts, user counts, incident counts, support counts, currency, and time units visible. Quality pass rate = benchmark tasks with a rubric result of Pass / eligible benchmark tasks evaluated x 100. Publish Partial, Fail, Not assessable, Disputed, Withdrawn, and missing-rubric counts separately; do not move corrected failures into the Pass numerator. Material error rate = evaluated tasks with at least one declared material error / eligible benchmark tasks evaluated x 100. Also report error count by category and severity because a task-level rate can hide multiple errors. If the decision is citation-sensitive, citation validity rate = citations verified as resolving to the claimed source, version, location, and supported proposition / citations checked x 100. A citation that is not checked is not valid for the numerator. Report unsupported, fabricated, stale, inaccessible, ambiguous, and not-checked citations separately. Appropriate abstention rate = tasks with an expected abstention or escalation where the model abstained or escalated in the approved manner / eligible abstention-opportunity tasks evaluated x 100. Inappropriate abstention rate = abstentions that occurred where the task was supported and within scope / eligible supported tasks attempted x 100. Keep unsupported answers and tasks that should not have been attempted as separate failure categories. Review time per output = reviewer active minutes / outputs reviewed, with queue delay, model latency, pause time, reruns, and missing clocks reported separately. Publish median, p75, p95, maximum, count, and unit when the sample supports those summaries. Rework rate = outputs requiring one or more material corrections or reruns before acceptance / outputs reviewed x 100. Preserve every correction and count repeated rework separately from first-pass failure. Qualifying adoption rate = eligible pilot users who complete at least one declared permitted action / eligible pilot users provisioned and in scope x 100. Workflow completion rate = eligible workflow instances reaching the accepted terminal event with required evidence / eligible applicable workflow instances x 100. Report non-use, blocked access, canceled work, unknown actor, and not-applicable workflow counts rather than treating them as successful or failed completion. Incident rate may be reported as confirmed or suspected incidents / eligible pilot users, eligible tasks, or pilot days, multiplied by a declared scale such as 1,000; publish the raw count, coverage, detection method, severity, and unresolved cases because a rate is not evidence that unobserved risk is absent. Direct cost per completed qualifying task = observed direct pilot currency / completed qualifying tasks, with currency, period, cost categories, and excluded or unallocated spend stated. Do not subtract forecasts, hypothetical avoided cost, internal time estimates, or benefits unless they are separately labeled and supported; this guide does not establish ROI or causation. Latency is measured from a declared request event to a declared output event in milliseconds, seconds, or minutes; report p50, p95, maximum, timeout count, and missing-clock count. Support contact rate = support contacts linked to the pilot / eligible active pilot users or completed qualifying tasks, using the denominator that matches the question and publishing both raw contacts and repeat contacts. Define organization-designed stop or go gates before inspecting the result. Example gates may require no unresolved Critical security or privacy incident, no prohibited-data use, citation validity above an approved local threshold for citation-required tasks, material-error and appropriate-abstention results within the approved risk-tier bands, complete failure-register ownership, review-time evidence for the intended workflow, and an approved rollback or containment route. These examples are not universal thresholds. Apply stricter gates to high-impact or client-facing tasks, retain every failed or disputed case, and require authorized review of exceptions. A limited go means only the named population, task family, data class, configuration, human-review step, and period may proceed. Reassess after model, prompt, retrieval, source, policy, workflow, user, jurisdiction, or data changes.

FAQs

Related CaseDocker capabilities

Related guides and resources

• Legal AI citation verification workflow
• Legal AI vendor security checklist
• Legal AI data residency and privacy requirements
• Legal AI prompt and output audit logs
• Case management portfolio