Skip to content

Introduction

Output Drift in Financial LLMs Workshop

Welcome to the Output Drift in Financial LLMs Workshop! This hands-on workshop teaches you how to measure and analyze non-determinism in large language model (LLM) outputs for financial applications.

Why This Matters

Financial institutions deploying AI systems need evidence for:

  • Governance Review: Reconstructable decisions and execution records
  • Risk Management: Predictable behavior in production
  • Trust & Reliability: Stakeholder confidence in AI-driven recommendations

This workshop is based on research showing that LLM outputs can drift even at temperature=0.0. In the tested tasks, that variation reached 35%, which complicates replay, change review, and monitoring.

Explore the results

Compare recorded decisions and tool paths, browse the study findings, then try the replay playground: Open the interactive explorer.

Try an offline lab

See the difference between a proposed tool call and a completed one using six synthetic records: Start Lab 10.

What You'll Learn

By the end of this workshop, you will:

  • Understand output drift and its implications for financial AI systems
  • Set up and run reproducible LLM experiments across multiple providers
  • Measure drift using industry-standard metrics (consistency, Jaccard similarity, schema violations)
  • Analyze cross-provider reliability patterns
  • Implement replay and observability checks for AI deployments

Workshop Structure

Lab Description Duration
Lab 0: Workshop Pre-work Install prerequisites and set up your environment 15 min
Lab 1: Understanding Output Drift Learn the theory and see real examples of drift 20 min
Lab 2: Setting Up Your Environment Configure API keys and run environment tests 15 min
Lab 3: Running Your First Experiment Execute experiments and understand the framework 30 min
Lab 4: Analyzing Drift Metrics Interpret results and generate visualizations 25 min
Lab 5: Cross-Provider Testing Compare reliability across different AI providers 30 min
Lab 6: Extending the Framework Add custom tasks and integrate with your workflows 30 min
Lab 7: Replayable Financial Agents Run agent benchmarks from the ICLR 2026 paper 30 min
Lab 8: DFAH-Bench — Replay Measurement Test the package, then reproduce the paper from the checked-in replay corpus 30 min
Lab 9: Replay, Review, and Retest Evaluate a correction with a fixed policy and export verified results using 0.1.2 25 min
Lab 10: Evidence Before Execution Inspect final invocation records and missingness using the 0.1.3 source 15 min

Total Duration: Approximately 4.5–5 hours. Labs 8 (package track), 9 and 10 can also be taken independently, without a model service or API key.

Research Foundation

This workshop is based on three papers from the same research line:

"DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making" arXiv:2607.20491 | DOI | The v2 replay analysis reproduces with make reproduce-paper; v3 source, results and reproduction scope

"Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents" ICLR 2026 FinAI Workshop (The 2nd ICLR Workshop on Advances in Financial AI) | arXiv:2601.15322

"LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows" Presented at the AI4F Workshop 2025 | arXiv:2511.07585

Key Findings (v1, Output Drift): - Even at temperature=0.0, frontier models exhibit 5.5-35% output variance - Several tested 7-20B configurations repeated all sampled outputs at T=0.0 - RAG tasks show the highest drift (56.25% consistency at temperature=0.2) - Structured output tasks (SQL, summarization) maintain better determinism

Earlier Replayable Agents study (4,705 runs): - Decision determinism and the historical task-label match were not detectably correlated (r = -0.11, p = 0.63) - This descriptive result does not identify model strategy or hidden reasoning - The portfolio fixture and its dependent task-label matches are not evidence in the DFAH-Bench replay analysis

DFAH-Bench replay analysis (arXiv v2): - The primary slice contains 4,157 episodes from configurations with observed tool use across 719 groups, eight configurations, and two synthetic tasks - Among 627 unanimous-decision groups, 122 (19.5%) change tool sequence and 47 (7.5%) change the tool-name set - A separate 570-episode API diagnostic finds 94.2–95.1% decision agreement but only 66.9–69.4% exact name-path agreement - Missing or malformed required channels make a replay group ineligible; they are not treated as agreement

Community reports and findings: - A November 17, 2025 contributor report preserved four of six stated results but not the materials needed for independent reproduction - The bounded observations vary by model/task combination; they do not establish a size law or isolate architecture, provider, task, or harness effects - Public issue #2, reported July 20 and fixed August 2, 2026, hardened the legacy validators against absent outputs, unknown task types, and missing evaluations - See the dated community ledger for the evidence limits, anonymized review themes, and resolution links

Banking and evidence extension (arXiv v3): - 1,080 native banking episodes reached terminal artifacts; 1,033 have known outcomes and 47 remain unknown across separate generator/retrieval cohorts. - The primary typed-choice bundle cost less per episode on complete pairs, while its task-success count remained below the prose-model gate under every assignment of the schedule's unknown outcomes. - Higher repeated gate agreement coexisted with lower constructed policy-label match in fixed-state probes. Evidence sufficiency needs direct assessment. - See the study guide for denominators, unavailable planned tests, cost scopes and the distinction between building the paper and reproducing the new empirical analyses.

Prerequisites

Required: - Python 3.11+ - Basic command line proficiency - Understanding of APIs and environment variables

Recommended: - Familiarity with LLMs and prompt engineering - Basic knowledge of financial concepts - Experience with data analysis (pandas, visualization)

Model access for provider experiments (not needed for Labs 8–10): - Ollama (free, local) - IBM watsonx.ai (trial available) - OpenAI, Anthropic, or other providers

Target Audience

This workshop is designed for:

  • AI/ML Engineers building production LLM systems
  • Risk & Compliance Officers evaluating AI deployments
  • Financial Technologists integrating AI into workflows
  • Researchers studying LLM reliability and non-determinism
  • Product Managers planning AI-powered financial products

Getting Help

If you encounter issues or have questions:

  1. Check the Troubleshooting Guide
  2. Review the API Reference
  3. Ask workshop facilitators or teaching assistants
  4. Open an Issue on GitHub
  5. Submit a Pull Request with improvements

Repository Structure

output-drift-financial-llms/
├── run_evaluation.py       # Main experiment orchestrator (v1 Output Drift)
├── run_dfah_demo.py        # DFAH demo (no LLM needed)
├── Makefile                # make reproduce-paper / make test-bench
├── REPRODUCIBILITY.md      # Exact environment, commands, disclosed caveats
├── COMMUNITY_FINDINGS.md   # Stable pointer to the dated community ledger
├── docs/                   # Workshop documentation (labs 0-10) and explorer
├── bench/                  # Historical replay metrics and fixtures
│   ├── metrics/            # DAR/TAR, ECD, DCB, SCDR implementations
│   ├── spec/               # Replay episode schema + task ontologies
│   ├── provenance/         # Hash-chained, Ed25519-signed audit bundles
│   └── stats/              # Bootstrap CIs, significance tests
├── src/dfah/               # Prospective pip-installable replay package
├── harness/                # v1 framework core
│   ├── deterministic_retriever.py
│   ├── task_definitions.py
│   └── cross_provider_validation.py
├── providers/              # LLM providers (watsonx, anthropic, gemini)
├── econometrics/           # Replayable Agents (v2) - benchmarks & metrics
│   ├── benchmarks/         # 3 financial agent benchmarks (50 cases each)
│   │   └── results/run_logs/  # Raw replay corpus: 8,129 episodes
│   └── agentic/            # Trajectory determinism & faithfulness metrics
├── scripts/                # Replay analysis + reproduce_paper.py
├── tests/                  # Offline research and package tests
├── results/                # Historical reference CSVs and v2 manifest
├── data/                   # Test datasets & generators
├── examples/               # Audit trails + domain-extension example
└── requirements.txt        # Python dependencies

Reproducibility & Citations

The v1 drift experiments are pinned at release v0.1.0 (commit c19dac5):

git clone https://github.com/ibm-client-engineering/output-drift-financial-llms
git checkout v0.1.0

The sanitized DFAH-Bench replay fixture is checked in. The default target regenerates the v2 retrospective slice under its revised eligibility rules and verifies the aggregate-only extensions and manifest:

make reproduce-paper
make reproduce-paper-v1  # archived v1 lineage only

If you use this framework in your research, please cite:

@article{khatchadourian2026replayable,
  title={Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents},
  author={Khatchadourian, Raffi},
  journal={arXiv preprint arXiv:2601.15322},
  year={2026},
  eprint={2601.15322},
  archivePrefix={arXiv},
  doi={10.48550/arXiv.2601.15322}
}

@inproceedings{khatchadourian2025output,
  title={LLM Output Drift: Cross-Provider Validation \& Mitigation for Financial Workflows},
  author={Khatchadourian, Raffi and Franco, Rolando},
  booktitle={AI4F Workshop},
  year={2025},
  eprint={2511.07585},
  archivePrefix={arXiv},
  doi={10.48550/arXiv.2511.07585}
}

@article{khatchadourian2026dfahbench,
  title={DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making},
  author={Khatchadourian, Raffi},
  journal={arXiv preprint arXiv:2607.20491},
  year={2026},
  eprint={2607.20491},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  doi={10.48550/arXiv.2607.20491},
  url={https://arxiv.org/abs/2607.20491}
}

DFAH-Bench: arXiv:2607.20491 | Replayable Agents: arXiv:2601.15322 | Output Drift: arXiv:2511.07585

License

This project is licensed under the MIT License. See the repository license for details.

Contributors & Acknowledgments

This workshop and framework were developed by Raffi Khatchadourian and Rolando Franco in IBM Financial Services in collaboration with researchers focused on responsible AI deployment in regulated industries.

Special thanks to the open-source community and the contributors who helped build and test this framework.

Ready to Begin?

Start with Lab 0: Workshop Pre-work to set up your environment!