2026-07-15 –, Johnson Great Room
Automated data enrichment, filling missing fields in structured records from unstructured sources, is the canonical case for pointing an autonomous agent at a database and letting it fill every blank. In high-stakes data that instinct is dangerous. A confidently wrong value is worse than a blank, and retrieval-grounded extraction reduces but does not remove the tendency to assert what the source never stated. An LLM can extract these fields; this paper asks where agentic judgment earns its place and where it becomes a liability.
We study this on the Texas Justice Initiative's police shooting databases, where nearly two thousand records are missing the weapon, the subject's race, or the outcome, whose fields volunteers typically recover by hand, fifteen to thirty minutes each. Our LangGraph pipeline, deterministic in its control flow, searches, validates, extracts, and escalates hard cases to a human. It completes 92% of officer and 70% of civilian records and invents zero facts across twenty fabricated incidents. The recovery itself came from a deterministic prompt fix without any agent. An autonomous agent pointed at the same fabricated incidents, with more freedom, commits a wrong-article fabrication the pipeline escalates.
If the deterministic core does the recovering, the agentic layer earns its place by making those recovered values trustworthy. Agency lives only in this thin judgment layer above extraction, and its components act in one of two ways. One acts on the pipeline's control flow: a relevance judge reads the retrieved articles and, when none actually report this incident, routes the record to a human instead of completing it. The other two pass judgment on what extraction produced: one deletes a value the source never states, and the other explains to the reviewer why the sources disagree on a value. Extraction calls an LLM too, but because it only proposes values for these judges to rule on, we do not count it as agentic. Every judge had to clear a reward-hacking-resistant evaluation gate before it shipped. The main contribution of this paper is a discipline, an "earn-it" protocol, for drawing the line between what a high-stakes pipeline should settle deterministically and where it is worth granting agentic judgment.
The Data Quality Problem in Public Accountability
In Texas, the Attorney General is required by law to collect a report on every officer-involved shooting, but the published summaries are high-level. The Texas Justice Initiative (TJI), a nonprofit, re-publishes the underlying incident records with the granularity that makes independent analysis possible. Yet those records are incomplete: across 1,956 incidents (2014–2024), 57% of civilian records are missing the weapon, 22.5% are missing the subject's name, and 39% of officer records are missing the officer's name. Volunteers recover these gaps by hand from news archives, fifteen to thirty minutes per record, because journalists routinely name people and describe circumstances that mandatory government filings omit. The author has volunteered with TJI since 2019 and co-authored its published report on officer-involved shootings.
An Agentic Enrichment Pipeline
The system is a seven-node LangGraph pipeline that searches the web, extracts fields, and escalates hard cases to a human. But it is mostly a deterministic workflow, not an autonomous agent, and that restraint is the point. After a Load node pulls a record from PostgreSQL, a deterministic Coordinator hub routes every transition through Search (Tavily), Validate (rule-based date/location/name checks), and Synthesize (LLM extraction), ending at Complete or Escalate. The single largest recovery gain came from a deterministic, dataset-aware extraction prompt, with no agent involved; a reasoning-and-acting loop tested on the same failures recovered nothing more. What ships as genuinely agentic is a thin faithfulness layer of three bounded LLM judges with graduated authority: a relevance judge that can block a record whose articles describe a different shooting, a race verifier that nulls a race the source never explicitly states, and a conflict annotator that only advises a human reviewer. The pipeline never writes back to the source database; escalation is a first-class outcome, and every committed value carries a confidence label and its source URLs.
Evaluation
Evaluation is treated as first-class engineering, not an afterthought. On held-out samples (100 records per dataset, with ground-truth fields hidden from the pipeline and compared only afterward), it completes 92% of officer records and 70% of civilian records, with exact-match field accuracy of 71–77% (86–89% under fuzzy match); officers complete more often because their shootings draw denser coverage. On a deliberately adversarial probe of 20 fabricated incidents (invented names placed in real Texas cities on real dates, six engineered as traps so that real articles about the wrong person would pass date and location checks), the pipeline invented zero facts and escalated all twenty. Run head-to-head on those same twenty incidents, a fully autonomous agent with free-text search, open web access, and none of the pipeline's guards declined most but completed one fabricated record the pipeline escalates, reproducibly and with no signal to warn a reviewer, at several times the cost. A generic instruction to cite sources is not the same as a mechanism with the authority to act on it. The safeguard behind every change is a reward-hacking-resistant, multi-objective evaluation gate: a pure function over two saved reports that scores completion, enforces a hard zero-hallucination veto, and checks field-level correctness on a stable cohort, so a completion gain can never launder a correctness loss. End to end, a record costs roughly $0.20 (about $400 for the full archive), set against the hundreds of volunteer-hours the manual workflow would take. Per-race completion is reported as a non-gating diagnostic, kept visible to a human reviewer rather than acted on automatically at these group sizes.
Design Principles and Broader Applicability
Four principles, each a stance on a tradeoff, shaped the system: do everything deterministic first, because every place a model may choose is a place it can choose wrong; calibrate each component's authority to how sharply it can decide (block, null, or advise); prefer faithfulness over coverage, because in an accountability database a wrong value is worse than a blank; and distrust the headline metric, because "complete more records" is trivially gamed by accepting weak extractions. No agentic component shipped on intuition; each had to clear an offline "earn-it" gate, and four of the seven ideas tried were gated out, failed, deferred, or declined, which is as much the contribution as the three that shipped. The discipline rests on three domain-general preconditions rather than on TJI specifics: a held-out signal to score against, a hard-veto safety metric that no other gain may override, and decisions that can be ranked by stakes. Where those hold (public-health surveillance, environmental incident tracking, historical archives, and other domains where structured databases have gaps that scattered public sources could fill), the pattern should carry.
What Attendees Will Learn
- The workflow-versus-agent design axis: where an LLM earns its place, and where a deterministic rule or prompt quietly beats one, shown by a head-to-head in which an unconstrained autonomous agent fabricates a record the bounded pipeline escalates
- How to build a reward-hacking-resistant, multi-objective evaluation gate that cannot be satisfied by trading correctness for completion
- Calibrating component authority to stakes (block / null / advise), and designing human-in-the-loop escalation as a first-class outcome rather than a failure
- An "earn-it" protocol for admitting agentic components only after they clear an offline gate, including the null results that kept components out
- Testing and mocking patterns for pipelines that depend on external web-search APIs and LLMs
- Using LangGraph deliberately narrowly: typed state, deterministic routing, and a clean seam to inject or mock every model call
Target audience
Data scientists, ML/AI engineers, and scientific Python users interested in applied agentic AI, LLM evaluation, data quality, or civic tech.
Source code
github.com/hongsupshin/police-data-intelligence (open source, MIT, with tests and CI via GitHub Actions)
Related publication
Officer-Involved Shootings in Texas: 2016-2019
Speaking experience
The author has presented at academic conferences and industry events on data science and machine learning topics. Video recordings are available from a Texas Justice Initiative presentation and an Austin Python Meetup community meetup talk.
Hongsup Shin is a Senior AI & LLM Engineer at NVIDIA's Silicon Co-design Group, building agentic systems and automation for silicon engineering workflows. His work spans production RAG infrastructure, multi-agent systems, learning-to-rank for hardware verification, and human-in-the-loop system design. He has published at IEEE SOCC (2022, 2024) and previously presented at SciPy 2019 on ML applications for hardware failure detection. He currently serves as SciPy Conference Proceedings Chair, founded the Austin ML Journal Club, and has volunteered with the Texas Justice Initiative since 2019, where he authored TJI's first data analytics report on officer-involved shootings in Texas. He holds a Ph.D. in Neuroscience from Baylor College of Medicine.