A final answer is only one part of an agent run. DFAH-Bench checks whether repeated decisions preserve the recorded tool path and, when available, the argument and result identities behind it. Explore the reproduced measurements here, then qualify your own integration before spending on a larger replay.
DAR (Decision Agreement Rate) asks: do N replays reach the same decision? TARseq asks: do they take the same ordered tool-name path? Points below the diagonal are configurations whose stable answers conceal a less stable recorded path. Coverage is shown explicitly; this is a replay measurement, not a model leaderboard.
The corrected core contains 627 groups whose replays all reached the same decision:
122/627 (19.5%) changed sequence: 17 changed only order, 58 changed call multiplicity while retaining the same tool-name set, and 47 changed the set itself. The logs do not establish that order-only changes are harmless.
This conditional view asks what the final decision alone misses. It does not determine whether either path was correct or safe.
| Coverage | Configuration | Replays | Groups | Episodes | DAR | TARseq | Gap |
|---|
Task-weighted over synthetic compliance and financial-DataOps cases. Complete rows cover 100 groups; limited rows are retained contiguous prefixes and are not directly comparable as a ranking. Gaps are paired by case and computed before displayed values are rounded.
| Configuration | Groups | Episodes | DAR | Names | Name + args | Result only |
|---|---|---|---|---|---|---|
| gpt-5.6-terra | 96/100 | 288/300 | 95.1% | 69.4% | 51.5% | 54.3% |
| claude-sonnet-5 | 94/100 | 282/300 | 94.2% | 66.9% | 45.0% | 56.9% |
All 600 scheduled episodes reached a terminal state; ten lacked a required channel, making ten groups ineligible and leaving 570 eligible episodes across 190 groups. One Sonnet/DataOps stratum retained 44 groups against a predeclared minimum of 45, so this remains a diagnostic extension, not a provider ranking.
Gemma 4 E4B retained 400/400 episodes. Qwen 3.5 retained 392/400 after eight identical parse failures made one group ineligible. Across the 99 eligible groups, the required four-tool path repeated exactly.
That is 100% observed agreement inside one fixed synthetic harness. It validates capture and replay mechanics; it is not evidence of general model determinism, financial accuracy, or production fitness.
A simulated compliance-triage agent replays one alert N times. Turn the instability knobs and watch the replay measures respond. The interesting zone: decision instability at 0, trajectory instability up. The agent looks perfect to an outcome-only evaluator while its process drifts.
This page can drive a real tool-using agent loop against a model running on your machine via
Ollama — the same compliance case, deterministic mock tools, temperature 0, seed 42 —
and compute decision, name-path, and strong-path agreement on the live replays.
Nothing leaves your machine; the browser talks to localhost.
1 · Start Ollama with this page allowed as an origin, and pull a model:
OLLAMA_ORIGINS="https://ibm-client-engineering.github.io" ollama serve ollama pull qwen2.5:7b-instruct
The agent gets 3 tools — check_sanctions, get_customer_profile,
calculate_risk_score — with deterministic mock outputs, then must decide
escalate / dismiss / investigate. This mirrors
econometrics/benchmarks/run_unified_benchmark.py. If the connection fails, check the
OLLAMA_ORIGINS step (browsers require it for cross-origin calls to localhost).
The package starts with a bounded conformance check, then writes versioned replay artifacts that can be resumed and inspected. The built-in agent below is deterministic and makes no network calls.
python -m venv .venv-dfah source .venv-dfah/bin/activate python -m pip install dfah-bench
dfah check-agent --agent dfah.demo:toy_agent dfah run --agent dfah.demo:toy_agent \ --replays 3 --out .dfah/runs/quickstart dfah analyze .dfah/runs/quickstart \ --report .dfah/runs/quickstart/report.html dfah inspect .dfah/runs/quickstart --case CASE-001
A perfect toy score confirms the integration contract, not real-world model quality. Continue with a synthetic suite that reflects your own tool schema, then use sampled shadow replays before considering a blocking gate. See the package quickstart.
The repository preserves the 8,129-record lineage and ships a sanitized replay fixture. The corrected analysis retains 4,157 episodes from configurations with observed tool use across 719 comparable groups. No API keys or model downloads are needed for retrospective regeneration:
git clone https://github.com/ibm-client-engineering/output-drift-financial-llms cd output-drift-financial-llms pip install -r requirements.txt make test-bench # offline test suite make reproduce-paper # regenerates corrected v2 and validates its manifest
The ledger lineage is 8,129 raw records, 8,127 in the legacy analysis,
5,501 after task/configuration exclusions, and 4,157 in configurations with observed
tool use. Use make reproduce-paper-v1 only for the archived v1 pipeline.
Prospective provider extensions are public as aggregate-only, checksum-pinned artifacts;
their approval-gated raw captures are not part of the public reproduction target. See
REPRODUCIBILITY.md
for the exact environment and
Lab 8 for the guided walkthrough. Extend to your own domain with zero metric-code changes:
python examples/domain_extension_medical.py.