An automated LLM review pipeline (dspy signature reviewing user-authored financial calculators) graded a correct, current real-world constant as a 'defect' because the value post-dated the model's training data: the view used the Amex Platinum's $895 annual fee (raised from $695 effective 2025-09-18), and the reviewer flagged "the standard US annual fee is $695... change - 895 to - 695", capping the star rating and telling readers to mentally 'correct' the numbers toward the stale figure. A second finding in the same review assumed a generic input ('monthly travel spend') mapped to a specific rewards bonus category ('3x on flights booked directly') and proposed rewriting a correct formula.
Root cause: the severity rubric defined 'defect' as "demonstrably wrong math" without distinguishing what is demonstrable from the artifact (algebraic wiring, double-counting, sign/unit errors, internal label-vs-formula inconsistency) from what is only checkable against the world (prices, fees, program multipliers, rate constants). The model treats disagreement with its own memory as demonstration.
Mitigation pattern: add an explicit 'external facts vs. math' carve-out to the rubric — (1) real-world constants embedded by the human author are presumed more current than the model's knowledge (humans react to provider changes faster than model retraining); disagreement with remembered values grades at the lowest, non-penalizing severity at most; (2) external constants stay defect-eligible only for internal inconsistency or face-value sign/unit/order-of-magnitude implausibility; (3) never let review prose instruct readers to adjust results toward the model's remembered figures; (4) on the authoring side, have generators attach as-of dates to real-world constants so reviewers can see currency is deliberate. Applies to any LLM QA/moderation pass over human-maintained content containing time-sensitive facts.