Lab 2: Setting Up Your Environment

Overview

In this lab, you'll configure API keys, test provider connectivity, and run your first replay evaluation to understand the framework's core components.

Duration: ~15 minutes

Historical workshop scope

The model labels in this lab describe repeatability observed in the original workshop runs. They do not certify correctness, compliance, or deployment fitness. Requalify the exact model, provider, prompt, tools, and suite before relying on a result.

Learning Objectives

By the end of this lab, you will:

• Configure API keys for at least one provider (Ollama recommended)
• Understand the DeterministicRetriever and its role in reproducible retrieval
• Test framework components with a simple evaluation
• Generate your first replay record

Prerequisites

• Completed Lab 0: Workshop Pre-work
• At least one provider configured (Ollama, watsonx.ai, or others)

Step 1: Verify Ollama Installation

If using Ollama (recommended for getting started):

# Check if Ollama is running
curl http://localhost:11434/api/tags

If not running, start Ollama:

ollama serve

Pull the recommended model (if not already done):

ollama pull qwen2.5:7b-instruct

Why Qwen2.5:7B?

Qwen2.5:7B produced identical outputs in the original bounded workshop runs at T=0.0, so it is a convenient local starting point. That result is an integration check, not a safety or compliance determination.

Step 2: Configure Environment Variables

Create or edit your .env file in the repository root:

# Navigate to repository root
cd /path/to/output-drift-financial-llms

# Create .env file
touch .env

Add your API configuration:

# Ollama (local, free)
OLLAMA_BASE_URL=http://localhost:11434

# IBM watsonx.ai (optional but recommended for cross-provider validation)
WATSONX_API_KEY=your_api_key_here
WATSONX_PROJECT_ID=your_project_id_here
WATSONX_URL=https://us-south.ml.cloud.ibm.com

# Anthropic (optional)
ANTHROPIC_API_KEY=your_anthropic_api_key_here

# Google Gemini (optional)
GEMINI_API_KEY=your_gemini_api_key_here

Sensitive Data

Never commit .env to Git! It's already in .gitignore.

Step 3: Generate Synthetic Financial Database

Our framework uses a synthetic financial database for SQL generation tasks:

python data/generate_toy_finance.py

Expected output:

🏦 Generating synthetic financial database...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Created tables:
  ✅ customers (100 records)
  ✅ accounts (150 records)
  ✅ transactions (500 records)
  ✅ loans (75 records)

Database: data/toy_finance.sqlite (45 KB)
✅ Generation complete!

This creates data/toy_finance.sqlite containing realistic financial data for testing.

Step 4: Test Framework Components

Let's test the core framework components to ensure everything is working.

Test 1: DeterministicRetriever

The DeterministicRetriever (harness/deterministic_retriever.py) makes SEC 10-K retrieval order explicit and reproducible for downstream review.

Create test_retriever.py:

from harness.deterministic_retriever import create_retriever_from_files

# Initialize retriever from SEC filings directory
retriever = create_retriever_from_files(
    corpus_path="data/sec",  # SEC 10-K filings
    chunk_size=200,
    overlap=50
)

# Test query
query = "What were net credit losses in 2023?"
results = retriever.retrieve(query, k=5)

print("Deterministic Retrieval Test")
print("=" * 50)
for i, (snippet_id, text, metadata) in enumerate(results, 1):
    print(f"\nChunk {i}:")
    print(f"  Snippet ID: {snippet_id}")
    print(f"  Text: {text[:100]}...")

print("\nRetrieval order is stable in the pinned exercise!")

Run it:

python test_retriever.py

Why Multi-Key Ordering?

The retriever uses multi-key ordering (score↓, section_priority↑, snippet_id↑, chunk_idx↑) so ties resolve predictably. This is a reproducibility control; whether it satisfies a workflow requirement is a separate governance decision.

Test 2: Simple Drift Evaluation

Now let's run a minimal drift test with 5 runs using the OpenAI client:

Create test_simple_drift.py:

#!/usr/bin/env python3
"""Simple drift evaluation using Ollama via OpenAI client."""
from openai import OpenAI
from collections import Counter

# Initialize Ollama client
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # Not used by Ollama
)

# Simple prompt
prompt = "What is the sum of 2 + 2? Answer with just the number."

print("🧪 Running 5 identical queries at T=0.0")
print("=" * 50)

responses = []
for i in range(1, 6):
    response = client.chat.completions.create(
        model="qwen2.5:7b-instruct",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
        seed=42
    )
    answer = response.choices[0].message.content
    responses.append(answer)
    print(f"Run {i}: {answer}")

# Measure modal exact-output agreement
counts = Counter(responses)
unique_responses = set(counts)
modal_count = counts.most_common(1)[0][1]
consistency_pct = modal_count / len(responses) * 100

print("\n" + "=" * 50)
print(f"Unique responses: {len(unique_responses)}")
print(f"Modal exact-output agreement: {consistency_pct:.0f}%")

Run it:

python test_simple_drift.py

Expected output for Tier 1 models (Qwen2.5:7B, Granite-3-8B):

🧪 Running 5 identical queries at T=0.0
==================================================
Run 1: 4
Run 2: 4
Run 3: 4
Run 4: 4
Run 5: 4

==================================================
Unique responses: 1
Consistency: ✅ 100%

Exact agreement in this run

The tested outputs reached 100% consistency at T=0.0. Exact agreement is useful replay evidence, but it does not establish correctness or compliance.

Step 5: Understanding Task Definitions

The framework defines three core financial tasks in harness/task_definitions.py:

# View task definitions
cat harness/task_definitions.py

The three core tasks:

Task	File Reference	Observed Tier 1 consistency	Purpose
SQL	harness/task_definitions.py:20-45	100%	Text-to-SQL generation
Summarize	harness/task_definitions.py:47-72	100%	JSON summarization with schema
RAG	harness/task_definitions.py:74-99	93.75%	Retrieval-augmented Q&A

Each task includes: - System prompts intended to constrain output variation - Temperature=0.0 and seed=42 defaults - Validation schemas (JSON schema for summarization, SQL syntax checker) - Citation requirements (for RAG tasks)

Step 6: Review Sample Audit Trail

The framework generates JSONL (JSON Lines) replay records with legacy governance-mapping labels. These labels are metadata for review, not evidence that a requirement was met. Let's examine the sample provided:

# View sample audit trail entry
head -n 1 examples/sample_audit_trail.jsonl | python -m json.tool

Example audit trail entry:

{
  "timestamp": "2025-11-01T14:23:45Z",
  "model": "granite-3-8b-instruct",
  "provider": "watsonx.ai",
  "temperature": 0.0,
  "seed": 42,
  "top_p": 1.0,
  "prompt": "What were JPMorgan's net credit losses in 2023?",
  "prompt_hash": "a3d8f9c2e1b4d7f8",
  "response_hash": "b2c1e7a9f3d8c5b1",
  "response": "JPMorgan reported net credit losses of $X billion in 2023 [jpm_2024_10k].",
  "citations": ["jpm_2024_10k"],
  "compliance_metrics": {
    "citation_accuracy": 1.0,
    "schema_valid": true,
    "decision_flip": false
  },
  "latency_ms": 1240,
  "concurrency": 1,
  "corpus_version": "sec_2024_q4"
}

Captured Timestamp

This historical record contains one event timestamp. It is not bi-temporal: transaction time and system-valid time are not recorded separately. Add those fields if the intended recordkeeping design requires them.

Understanding Framework Components

1. DeterministicRetriever

File: harness/deterministic_retriever.py

from harness.deterministic_retriever import create_retriever_from_files

retriever = create_retriever_from_files(
    corpus_path="data/sec",
    chunk_size=200,
    overlap=50
)

Purpose: Makes SEC 10-K retrieval order stable and inspectable within the pinned exercise.

Features: - Multi-key ordering (score, section priority, snippet ID, chunk index) - Stable chunk IDs for reproducibility - Section-aware retrieval (prioritizes financial statement sections)

2. Task Definitions

The framework includes 3 core task types:

Task	Description	Tier 1 Consistency
SQL	Text-to-SQL generation from natural language	100%
Summary	JSON summarization of financial data	100%
RAG	Retrieval-augmented Q&A over SEC 10-Ks	93.75%

Context for the observed 100% SQL and Summary agreement: - Structured output formats - Deterministic syntax - Narrow output space

3. Cross-Provider Validation

File: harness/cross_provider_validation.py

from harness.cross_provider_validation import CrossProviderValidator

validator = CrossProviderValidator(
    providers=["ollama", "watsonx"],
    tolerance_pct=5.0  # illustrative, task-specific tolerance
)

# Validate pre-collected outputs from different providers
outputs = {"ollama": ollama_result, "watsonx": watsonx_result}
results = validator.validate(outputs, task_type="sql")
print(f"Consistent: {results['consistent']}")
print(f"Similarity: {results['similarity_scores']}")

Purpose: Compare pre-collected outputs from local (Ollama) and cloud (watsonx.ai) configurations.

Numeric tolerance: This lab uses a configurable ±5% example. It is not a universal GAAP materiality threshold; production settings require task-specific approval.

Troubleshooting

Ollama Connection Failed

# Check if Ollama is running
curl http://localhost:11434/api/tags

# If not, start it:
ollama serve

Model Not Found

# List available models
ollama list

# Pull the model if missing
ollama pull qwen2.5:7b-instruct

Database Not Found

# Regenerate the database
python data/generate_toy_finance.py

Import Errors

# Ensure virtual environment is activated
source venv/bin/activate  # macOS/Linux
# or
venv\Scripts\activate  # Windows

# Reinstall dependencies
pip install -r requirements.txt

Key Takeaways

1. Tier 1 configurations: Qwen2.5 and Granite-3-8B reached 100% exact-output agreement in the bounded 480-run matrix; GPT-OSS-20B did so in a separate workshop check
2. DeterministicRetriever: Applies stable ordering to the pinned SEC 10-K corpus
3. Replay records: JSONL captures one event timestamp plus configuration and result fields
4. Task types: SQL and summarization reached 100% agreement in the tested runs; RAG varied more
5. Cross-provider comparison: Measures whether outputs differ between captured local and cloud configurations

Quiz: Test Your Understanding

Why use multi-key ordering in DeterministicRetriever?

Answer: To make retrieval order stable and reproducible. If chunks have the same relevance score, the additional keys resolve ties consistently for later inspection.

What did Tier 1 mean in the original workshop?

Answer: It denoted 100% observed output consistency in the bounded test conditions. It was not a regulatory or deployment certification.

What does the ±5% tolerance mean in this exercise?

Answer: It is an illustrative, configurable numeric comparison value, not a universal GAAP materiality threshold.

Next Steps

Now that your environment is configured and you understand the framework components:

1. Proceed to Lab 3: Running Your First Experiment to run drift evaluations
2. Review task definitions in harness/task_definitions.py
3. Examine the DeterministicRetriever implementation in harness/deterministic_retriever.py
4. Study the CrossProviderValidator code in harness/cross_provider_validation.py

---

Lab 2 Complete!

Your environment is configured and tested. Ready to run experiments? Move on to Lab 3: Running Your First Experiment!