Automation Program Manager, Legal and Compliance Operations

Hill-climb reliability (prompt, iterate and improve), correctness, precision and recall on assigned automations across Legal and Compliance Workflows

Iterate on LLM components, including prompts, task decomposition, routing, and abstention and fallback behavior

Diagnose and improve rules-based components, including coverage, ordering and precedence, conflicts between rules, rules firing on the wrong population, and rules that stop matching after a policy or schema change

Tune signal-based components, including thresholds, signal freshness and availability, and the precision cost of upstream signal drift

Investigate regressions to isolate whether the cause is model, rule, signal, upstream data, policy change, or a shift in the incoming population, and distinguish real degradation from volume mix shift

Drive regressions to a measured recovery that holds across a following measurement period, and document what degraded, why it was not caught earlier, and what detection would have caught it

Build and maintain offline eval and gold standard data sets, coordinating expert-review sampling with the reviewers who hold the ground truth

Build regression detection that surfaces degradation before the next reporting cycle, and close measurement coverage gaps so automations that are live are also measured

Build scripts, notebooks and internal tooling so program managers and operations leads can run measurement independently

Document what was tried, what moved the metric, and what did not, so subsequent workflows start further along

2+ years of relevant experience in data annotation, labeling, quality assurance, or content review operations, including time spent close to automated decision output

Experience with guideline accuracy and inter-rater agreement, including diagnosing why qualified reviewers disagree

Working knowledge of Python and SQL, with experience pulling a dataset, computing precision and recall, and segmenting results to locate where loss is concentrated

Experience reading and reasoning about conditional logic to trace how a decision was reached

Experience working with large operational datasets in a notebook environment

Investigative approach to metric movement, identifying cause rather than the explanation that fits

Experience communicating results clearly, including reporting a metric that has not moved or a regression that is not yet explained

Experience in adversarial or fraud-adjacent domains where the reviewed population actively shifts

Experience with rules engines, decision tables, routing logic or classifier thresholds

Experience coordinating with vendor or cross-functional reviewer pools

Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies

Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements)

Experience or expertise in legal and compliance operations, or trust and safety

Experience iterating on prompts with measured before-and-after results

Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews)