DFAH-Bench · Interactive Explorer

Same decision. Different path.

DFAH-Bench measures whether tool-using LLM agents that agree on decisions also agree on how they got there — trajectories, evidence, tool paths. Outcome-only evaluation misses all of it. Everything on this page runs in your browser with the paper's actual metric formulas, and every number below regenerates from the public repo with one command: make reproduce-paper.

8,127
replay episodes (8,129 raw − 2 single-replay groups)
21.8%
of decision-stable case groups hide trajectory divergence
10
models · 3 financial tasks · 1,338 case groups
53/53
reproduction checks pass from a fresh clone

Results explorer — the DAR–TAR gap

DAR (Decision Agreement Rate) asks: do N replays reach the same decision? TAR (Trajectory Agreement Rate) asks: do they take the same tool path? Points below the diagonal are agents whose stable answers conceal unstable process. Click a model to highlight it.

Pattern matcher Stable executor Trajectory diverger diagonal = perfect process stability (DAR = TAR)

Kill criterion (§4.4)

912 case groups where decisions agree (DAR ≥ 0.9) and trajectories are observable:

19.4%
78.2% stable

■ 177 strong divergence (TAR < 0.7)   ■ 22 moderate (0.7–0.9)  = 199 / 912 = 21.8%

Diverger rates among stable cases

Claude Sonnet 455.6%
Gemini 2.5 Pro56.6%

Source: results/dfah_kill_criterion*.csv — regenerated by scripts/compute_dfah_metrics.py from raw replay logs.

ProfileModelDARTARGapDCBECDAcc

Task-averaged over compliance / portfolio / dataops. “—” = model made no tool calls (trajectory channel absent — reported as missing, never as agreement). DCB = decision concentration bias (1 − H/log K, cross-case); ECD = evidence-contact divergence (mean pairwise Jaccard distance). Full table with κ and bootstrap CIs in the paper.

Replay playground — feel the gap

A simulated compliance-triage agent replays one alert N times. Turn the instability knobs and watch the paper's metrics respond — computed live with the same formulas as bench/metrics/. The interesting zone: decision instability at 0, trajectory instability up. The agent looks perfect to an outcome-only evaluator while its process drifts.

The N replays

Live — replay a real case against your own model

This page can drive a real tool-using agent loop against a model running on your machine via Ollama — the same compliance case, deterministic mock tools, temperature 0, seed 42 — and compute DAR/TAR/ECD on the live replays. Nothing leaves your machine; the browser talks to localhost.

1 · Start Ollama with this page allowed as an origin, and pull a model:

OLLAMA_ORIGINS="https://ibm-client-engineering.github.io" ollama serve
ollama pull qwen2.5:7b-instruct

The agent gets 3 tools — check_sanctions, get_customer_profile, calculate_risk_score — with deterministic mock outputs, then must decide escalate / dismiss / investigate. This mirrors econometrics/benchmarks/run_unified_benchmark.py. If the connection fails, check the OLLAMA_ORIGINS step (browsers require it for cross-origin calls to localhost).

Reproduce the paper — one command

The full replay corpus (8,129 episodes) ships in the repo. No API keys, no model downloads:

git clone https://github.com/ibm-client-engineering/output-drift-financial-llms
cd output-drift-financial-llms
pip install -r requirements.txt
make test-bench        # 143 offline tests
make reproduce-paper   # regenerates every paper number; fails loudly on mismatch

Bootstrap CIs use B=10,000 with seed 42. See REPRODUCIBILITY.md for the exact environment and the disclosed caveats, and Lab 8 for the guided walkthrough. Extend to your own domain with zero metric-code changes: python examples/domain_extension_medical.py.