Alina Khay · Financial LLM evaluation

Earnings Input Reliance Explorer

Change one supplied financial input and see how a model’s recorded answer preference changes. Explore earnings versus expectations, price momentum, and opinion tone independently.

6 checkpoint conversions24 numerical blocksSynthetic announcementsRecorded scores · no live inference
Explore the benchmark dataset ↗
Matched input change

score change · percentage points
Baseline positive-answer score
Changed-input positive-answer score
Inspect the exact supplied contexts and answer instructions

Shared system instruction: “You are a financial analyst. Follow the requested answer format.” Revenue meets consensus, guidance is unchanged, and commentary is explicitly described as opinion containing no additional facts.

Baseline

Changed input

Momentum and tone together

Select a cell to make it the baseline. EPS surprise, section order, checkpoint, and answer encoding stay fixed.

Across the complete primary benchmark

Signed input effects, averaged over both A/B mappings and all 24 numerical blocks. These averages use the full primary design and do not change with the selected individual case.

Six-conversion primary effect comparison
ConversionMomentum · ppTone · ppEPS · ppBeat/miss accuracy

The final column is a separate numerical diagnostic using natural “beat”/“miss” answer prefixes. It is not next-day stock-return accuracy.

What the explorer measures

Input reliance

The matched comparison reverses one supplied cue while holding the other inputs and numerical block fixed.

Answer preferences

Scores compare the likelihoods of the two requested answer prefixes. A/B mean averages the original and reversed letter mappings.

Conditional evidence

These are short synthetic cases evaluated using six specific Q8 conversion files. There are no observed market returns and no fine-tuning.

Scores are not calibrated probabilities of a stock rising. The experiment does not establish trading profitability, an internal reasoning mechanism, or properties invariant to checkpoint conversion and numerical precision. The paper reports broader controls and uncertainty estimates.