From spot fixes to measured gains: improving document extraction accuracy with ai_extract and MLflow Prompt Registry

That approach becomes unreliable as fields and document formats multiply. A change may fix one field while introducing regressions elsewhere, work for one layout but fail on another, or appear better simply due to normal variation across LLM runs.

We saw this on a recent project that extracted roughly 40 structured fields from partner-submitted order documents across dozens of PDF layouts. Reviewers corrected and approved extracted values before they moved downstream, giving us human-reviewed ground truth for evaluation.

The challenge was not getting ai_extract to return the fields. It was improving extraction without degrading the system overall.