DFAH-Bench (Determinism–Faithfulness Assurance Harness) compares recorded decisions and tool paths across replays. Start with two runs of the same case, then explore what the banking study adds about evidence, completed actions, task outcomes and cost.
read_status → read_ruleread_rule → read_statusThe tool order changed. The review policy flags the difference.
19.5% of 627 retrospective replay groups with a unanimous decision changed their recorded tool sequence.
Choose a set of recorded runs. Compare its decisions and tool calls across replays, then see whether one review policy flags the change. These are saved runs, not live model calls.
| Case / replay | Decision | Tool order |
|---|
Try this in Lab 9 → · Example guide · Export verified results · Versioned source
Four plain-language terms behind the measurement.
Replay tells us whether a decision and path repeat. The banking study asks whether the decision was supported by evidence, whether the action completed, and whether the task succeeded. It also tracks what the runs cost.
Known successful episodes in the primary banking study; 270 scheduled per setup.
Structural checks alone led this retained schedule. Assigning every unknown outcome in favor of either gate would not reverse the ordering.
All 1,080 scheduled episodes across three cohorts reached terminal artifacts; 1,033 outcomes are known and 47 remain unknown. In the primary cohort, the unknown counts are 10 for structural checks, 7 for the prose gate and 7 for the typed gate. The planned statistical tests could not run, so these are descriptive comparisons of the retained schedule.
Three separate cohorts use Sierra’s τ-Knowledge banking environment: an open-weight DeepSeek v4.1 Flash primary generator, a frontier Gemini 3.8 Flash replication, and a DeepSeek retrieval variant. Gemini also supplied simulation and grading. Each cohort compares structural checks alone, a prose-model gate and a typed-choice gate.
The typed-choice gate repeated its decision more often than the prose-model gate: 98.7% versus 93.1%. But it matched fewer constructed policy labels: 54.5% versus 82.6%.
In one referral case, it repeatedly permitted action even though required current-time evidence was missing.
Agreement averages 105 complete three-repeat state groups for the typed gate and 101 for the prose gate. Policy-label matches use 175/321 and 261/316 qualified decisions, respectively. These descriptive probes concern selected contexts and constructed labels; native task outcomes are measured separately.
The Python package separates a completed tool call from one that was proposed, rejected, or left unresolved. It also distinguishes an empty call list from missing evidence.
Lab 10: Evidence Before Execution →
Lab 10 uses six synthetic records, with three completed invocations. It runs without
provider calls. A returned tool call establishes invocation completion; its business
outcome requires its own evidence. The 0.1.3 package exposes this through
execution_summary.
On 77 matched primary tasks, the typed gate cost $0.0043 less per episode than the prose gate. That saving came alongside fewer native successes on the retained schedule.
The three banking cohorts used $27.25 in reconciled API cost across all roles and attempts.
OpenRouter routed generator, simulator, native grader and prose-gate requests. Jev supplied the typed-choice gate through its SystemOne endpoint. Matched cost comparisons use complete task pairs and preserve each cohort’s coverage. These hosted API costs have a different basis from the local Ollama walkthrough and illustrative hardware sizing. The study guide connects paired costs to native success, admission limits and recovery behavior.
Read the methods and full comparisons → · Published paper and figures
Each point compares how often a decision repeats (DAR) with how often the ordered tool-name path repeats (TARseq). Points below the diagonal changed paths more often than decisions.
In 627 replay groups, every run reached the same decision:
122/627 (19.5%) changed sequence: 17 changed only order, 58 changed call multiplicity while retaining the same tool-name set, and 47 changed the set itself. Order can matter when a control depends on which tool runs first.
Path agreement measures repeatability. Policy compliance and task success need separate evidence.
| Coverage | Configuration | Replays | Groups | Episodes | DAR | TARseq | Gap |
|---|
Task-weighted over synthetic compliance and financial-DataOps cases. Complete rows cover 100 groups; limited rows cover shorter contiguous prefixes. Gaps are paired by case before rounding.
Across these two API configurations, decision agreement was 94.2–95.1%; full-path agreement was 45.0–51.5%.
| Configuration | Groups | Episodes | Decision | Tool names | Full path | Result only |
|---|---|---|---|---|---|---|
| gpt-5.6-terra | 96/100 | 288/300 | 95.1% | 69.4% | 51.5% | 54.3% |
| claude-sonnet-5 | 94/100 | 282/300 | 94.2% | 66.9% | 45.0% | 56.9% |
Full path means ordered tool names, canonical arguments and deterministic result hashes. The displayed percentages retain the paper’s exact configuration-specific coverage.
All 600 scheduled episodes reached a terminal state; ten lacked a required channel, making ten groups ineligible and leaving 570 eligible episodes across 190 groups. One Sonnet/DataOps stratum retained 44 groups against a predeclared minimum of 45. These rows are a diagnostic comparison with their coverage shown, not a provider ranking.
Gemma 4 E4B retained 400/400 episodes. Qwen 3.5 retained 392/400 after eight identical parse failures made one group ineligible. Across the 99 eligible groups, the required four-tool path repeated exactly.
The 100% observation belongs to this fixed synthetic harness and checks its capture and replay mechanics.
Move the sliders to see how changes in decisions or tool calls affect replay agreement. Keep decision variation at 0%, then raise tool-path variation to see how the same answer can come from different steps. This is a simulation.
Connect to a model running through Ollama on your machine. This page runs a synthetic compliance case with deterministic mock tools, then compares decisions and tool paths across live replays. Model requests go to the Ollama URL you enter; you will need Ollama and a downloaded model.
1 · Start Ollama with this page allowed as an origin, and pull a model:
OLLAMA_ORIGINS="https://ibm-client-engineering.github.io" ollama serve ollama pull qwen2.5:7b-instruct
The agent gets 3 tools — check_sanctions, get_customer_profile,
calculate_risk_score — with deterministic mock outputs, then must decide
escalate / dismiss / investigate. This mirrors
econometrics/benchmarks/run_unified_benchmark.py. If the connection fails, check the
OLLAMA_ORIGINS step (browsers require it for cross-origin calls to localhost).
Install the package, check that a demo agent can run, then inspect a saved replay. This built-in example makes no model calls or network requests.
python -m venv .venv-dfah source .venv-dfah/bin/activate python -m pip install 'dfah-bench==0.1.3'
dfah check-agent --agent dfah.demo:toy_agent \ --episode-timeout-s 5 dfah run --agent dfah.demo:toy_agent \ --replays 3 --episode-timeout-s 5 \ --out .dfah/runs/quickstart dfah analyze .dfah/runs/quickstart \ --report .dfah/runs/quickstart/report.html dfah inspect .dfah/runs/quickstart --case CASE-001
The toy run checks the integration. Next, try a synthetic suite that reflects your tool schema, then sampled shadow replays. Blocking runs require an explicit policy. See the package quickstart and Lab 9 for the versioned review loop, then Lab 10 for execution evidence in 0.1.3.
Regenerate the public retrospective analysis from its sanitized replay fixture. It retains 4,157 episodes from configurations with observed tool use across 719 comparable groups. No API keys or model downloads are needed:
git clone https://github.com/ibm-client-engineering/output-drift-financial-llms cd output-drift-financial-llms pip install -r requirements.txt make test-bench # offline test suite make reproduce-paper # regenerates v2 analysis and validates its manifest
Use Lab 8 for a guided walkthrough and REPRODUCIBILITY.md for the exact environment. The study guide explains how the public replay fixture, offline execution lab, and hosted banking study fit together.
The revised v2 eligibility rules and their provenance are recorded in the manifest.
The ledger lineage is 8,129 raw records, 8,127 in the legacy analysis, 5,501 after
task/configuration exclusions, and 4,157 in configurations with observed tool use.
make reproduce-paper-v1 runs the archived v1 pipeline. Prospective provider
extensions are public as aggregate-only, checksum-pinned artifacts; raw captures require
separate approval. The hosted τ collection runner and retained provider logs for the
banking analyses are outside this public package.
Extend to your own domain with python examples/domain_extension_medical.py.