AI Contract Obligation Extraction Accuracy Testing Guide

Definitions

Field definitions

Controlled vocabulary guidance

Match status

Examples: exact, partial, unmatched, abstained, not-applicable, unresolved, parser-failure

Governance: Keep exact and partial outcomes separate; every partial rule must name the field, evidence requirement, normalization version, and approved partial floor.

Error taxonomy

Examples: missed-obligation, hallucinated-obligation, wrong-party, wrong-action, wrong-date, wrong-amount, wrong-condition, wrong-citation, boundary, normalization, duplicate, merge-split, cross-reference, unsupported-source, permission-provenance, parser, reviewer-disagreement

Governance: Allow one primary cause and optional secondary causes; review taxonomy coverage after each material model or schema change.

Confidence and review priority

Examples: high-confidence, medium-confidence, low-confidence, abstained, high-risk, novel-pattern, contradictory, mandatory-review

Governance: Define confidence bins and review routing in the evaluation plan; do not use a label as a universal automation permission.

Drift disposition

Examples: no-action, investigate, expand-sample, retest, restrict-automation, rollback, retire

Governance: Record baseline, current population, trigger evidence, decision owner, due date, and whether the disposition changes the approved use.

Practical workflow

1. Define the intended extraction decision

   State whether the system is discovering possible obligations, creating a review queue, populating a governed obligation register, supporting renewal work, or authorizing downstream action. Define the contract populations, jurisdictions, languages, clause families, users, consequences of error, automation boundary, and cases that must always receive human review.

2. Specify the obligation schema

   Define the unit of analysis and required fields before sampling: obligation text or proposition, subject party, counterparty, action, object, trigger, due date or date range, amount, currency, frequency, condition, exception, source citation, confidence, abstention reason, and review state. Distinguish fields that may be unknown from fields that may be omitted because no obligation exists.

3. Create a representative sampling frame

   Build a frame of eligible contracts and clauses with stable identifiers, source location, contract type, language, jurisdiction, execution state, amendment status, page and token counts, scan or text quality, clause family, obligation risk, and permission class. Record excluded populations and why they are excluded. Do not let convenience sampling stand in for the intended deployment population.

4. Allocate the gold-set sample

   Use a versioned stratified allocation. A planning example is sample_units_for_stratum = ceil(stratum_population / total_population x target_sample_units), followed by an approved minimum for rare or high-risk strata. Record the sampling frame, randomization method, replacement policy, target size, finite-population or clustering adjustment, and the final count in every stratum.

5. Write annotation instructions

   Define what counts as an obligation, how to label permissions and prohibitions, how to identify parties and roles, how to normalize dates and amounts, how to preserve conditions and exceptions, how to cite source text, and how to handle cross-references, definitions, tables, schedules, amendments, incorporated terms, ambiguous language, and missing evidence. Include positive, negative, and abstention examples.

6. Annotate independently

   Use at least two qualified annotators for the pilot or for a documented sample of production-like records. Keep annotators blind to model output when creating the reference. Store source span offsets or stable citations, field-level labels, rationale, uncertainty, annotator identity, instruction version, and timestamp. Keep the original contract and the answer key immutable after adjudication.

7. Adjudicate disagreements

   Resolve disagreements with a named adjudicator or panel using the published instructions. Preserve both original labels, the disagreement reason, the adjudicated value, the source evidence, and any instruction change. Report agreement before adjudication by field and stratum; do not hide ambiguous language by forcing a false single answer.

8. Freeze evaluation versions

   Assign versions to the contract sample, gold labels, schema, normalization rules, model, prompt or configuration, retrieval context, parser, post-processing code, and evaluation script. Separate development, tuning, locked acceptance, and post-release monitoring samples. Record hashes or immutable references so a result can be rerun without silently changing the inputs.

9. Run extraction under controlled conditions

   Run the exact model and configuration against the locked sample with permissions, context windows, retrieval sources, temperature or sampling controls, and timeout behavior fixed. Save raw output, structured output, citations, confidence, abstention reason, latency, token or page usage, parser status, and any retry or truncation event. Do not repair outputs manually before scoring.

10. Normalize without erasing evidence

    Publish deterministic normalization for whitespace, case, punctuation, quotation marks, Unicode, party aliases, date formats, time zones, amount separators, currency symbols, units, frequency, citation punctuation, and clause numbering. Keep raw and normalized values side by side. A normalization rule may make values comparable, but it must not convert an unsupported inference into a gold fact.

11. Score exact and partial matches

    Score at both obligation-record and field levels. Use exact equality after the published normalization. For text, calculate token overlap precision, token overlap recall, and token F1; for dates, amounts, parties, conditions, and citations, compare declared components and provenance. A partial-match rule must be versioned with its field, unit, minimum evidence, and organization-approved partial floor; it is not a universal legal standard.

12. Report precision, recall, and F1

    Report micro and macro results separately and state the unit. For a declared match rule, precision = matched predicted obligation records / all predicted obligation records x 100%, recall = matched gold obligation records / all gold obligation records x 100%, and F1 = 2 x precision x recall / (precision + recall) when both are non-zero. Publish results by stratum, field, confidence band, and exact versus partial match.

13. Measure field accuracy and missingness

    For each field, field_accuracy = correctly matched eligible field instances / eligible gold field instances x 100%, with exact and partial variants reported separately. Also report missing or malformed predictions, unsupported values, duplicate records, false-positive fields, and fields that are not applicable. Keep the denominator visible because a model can appear accurate by abstaining or by scoring only easy fields.

14. Measure abstention and confidence

    Report abstention_rate = abstained eligible extraction opportunities / all eligible extraction opportunities x 100%, with the unit defined as obligation records or field instances. Report coverage = non-abstained eligible opportunities / all eligible opportunities x 100%. For calibration, bin confidence, publish bin counts, mean confidence, observed correctness, and calibration error = sum over bins of (bin_count / total_count) x absolute(observed_accuracy - mean_confidence), in percentage points or a 0 to 1 score.

15. Classify errors and route human review

    Use a controlled taxonomy: missed obligation, hallucinated obligation, wrong party, wrong action or object, wrong date or trigger, wrong amount or currency, wrong condition or exception, wrong citation or span, boundary error, normalization error, duplicate, merge or split, cross-reference failure, unsupported source, permission or provenance failure, parser failure, and reviewer disagreement. Route high-risk, low-confidence, novel, contradictory, and abstained records to qualified reviewers.

16. Review calibration and human workload

    Compare confidence with observed correctness by field, stratum, and release. For binary correctness labels, Brier_score = sum((confidence_i - correctness_i)^2) / number_of_scored_predictions, in unitless 0 to 1 units. Measure reviewer queue volume, review time per record, agreement, adjudication rate, correction rate, escalation rate, and unresolved ambiguity. A lower model error rate is not enough if review capacity or correction burden is unsafe.

17. Test drift against a baseline

    Repeat the same scoring protocol on a time-based monitoring sample and compare current and baseline populations. Report delta_precision, delta_recall, and delta_F1 in percentage points; field-distribution change as current field proportion minus baseline field proportion in percentage points; abstention and calibration changes in percentage points; and error-taxonomy changes by count and share. Investigate new contract forms, clauses, languages, vendors, jurisdictions, model versions, and workflow changes.

18. Approve risk-tiered acceptance

    Set acceptance rules for the declared use, not for legal AI generally. The approval record should name the sample and versions, exact and partial rules, numerator and denominator, units, strata, exclusions, confidence evidence, abstention policy, human-review capacity, error examples, drift baseline, residual risks, waivers, rollback trigger, and owner. Use no universal threshold; require explicit risk-tiered approval and retesting after material change.

Comparison

Limitations and exceptions

• There is no universal legal AI extraction threshold. Accuracy depends on the contract population, language, clause complexity, schema, normalization, field definitions, risk tolerance, and whether the workflow requires discovery, review support, or automated action.
• A gold set can encode annotator disagreement, missing context, interpretation choices, and sampling bias. Preserve uncertainty and adjudication history rather than presenting every label as an objective legal truth.
• Exact and partial matching answer different questions. A generous partial rule can hide boundary or meaning errors, while an exact rule can penalize harmless formatting differences. Report both with the rule and denominator visible.
• Confidence scores are not calibrated probabilities by default. Calibration can change by field, language, clause family, model version, prompt, retrieval context, and time period.
• High aggregate F1 can conceal dangerous failures in low-volume, high-risk, rare-language, amended, cross-referenced, or permission-restricted clauses. Require stratified results and targeted human review.
• Drift metrics are signals for investigation, not proof that a model is legally unsafe or safe. A stable metric can coexist with a changed risk population or a new error type.
• The cited governance and measurement sources provide risk-management and measurement context; they do not prescribe a universal contract-obligation schema, matching rule, or acceptance number.

Primary sources

Methodology

This guide uses an organization-designed evaluation and acceptance framework reviewed against the cited NIST and EUR-Lex sources as of 2026-08-13. Freeze a representative gold set with contract, clause, language, jurisdiction, source quality, amendment, obligation-risk, and permission strata. For each gold obligation record, preserve source span or stable citation, obligation text, subject party, action or object, trigger, due date or range, amount and currency, condition or exception, and field applicability. Use deterministic normalization for comparison while retaining raw values. Exact record match requires the normalized record identity and required fields to agree under the published schema. Partial text matching uses overlap = predicted tokens intersected with gold tokens; token_precision = overlap / predicted tokens; token_recall = overlap / gold tokens; token_F1 = 2 x token_precision x token_recall / (token_precision + token_recall), with zero defined when both terms are not positive. For structured fields, publish component rules for party aliases, dates, amounts, conditions, and citations, and version an organization-approved partial-match floor; there is no universal floor. At record level, precision = matched predicted obligation records / all predicted obligation records x 100%, recall = matched gold obligation records / all gold obligation records x 100%, and F1 = 2 x precision x recall / (precision + recall) when precision plus recall is greater than zero. For a field, field_accuracy = correctly matched eligible field instances / eligible gold field instances x 100%; report exact, partial, missing, malformed, unsupported, and false-positive field counts separately. Abstention_rate = abstained eligible opportunities / all eligible opportunities x 100%, coverage = non-abstained eligible opportunities / all eligible opportunities x 100%, and review_rate = records routed to human review / eligible records x 100%, with the opportunity or record as the declared unit. For confidence calibration, partition scored predictions into versioned bins, report bin count, mean confidence, observed correctness, calibration error = sum over bins of (bin count / total count) x absolute(observed accuracy - mean confidence), and Brier_score = sum((confidence_i - correctness_i)^2) / number of scored predictions; calibration error may be reported as a proportion from 0 to 1 or percentage points, while Brier score is unitless from 0 to 1. For drift, report current minus baseline precision, recall, F1, field accuracy, abstention, review, and calibration in percentage points, plus changes in population and error-taxonomy shares; publish counts and denominators. Compute micro metrics by pooling eligible instances and macro metrics as the unweighted mean of stratum metrics, and report confidence intervals or uncertainty methods when the decision requires them. Keep a controlled error taxonomy and review a blind sample of accepted, abstained, low-confidence, high-risk, novel, and corrected records. Acceptance must be risk-tiered and workflow-specific: define the intended automation boundary, mandatory human-review classes, abstention handling, residual-error budget, evidence quality, reviewer capacity, drift trigger, rollback path, and approval owner. Example gates may be organization-designed, but no threshold in this guide is universal or a substitute for legal judgment. Re-run the frozen evaluation after material changes to the model, prompt, retrieval context, schema, normalization, source population, permissions, parser, or workflow, and retain raw outputs, labels, formulas, exclusions, errors, waivers, approvals, and monitoring evidence.

FAQs

Related CaseDocker capabilities

Related guides and resources

• Contract management
• Contract lifecycle management information
• Contract obligation register template
• Legal workflow playbooks