severity-rubric

lesson

llm-as-judge review-pipeline knowledge-cutoff prompt-engineering severity-rubric dspy

402 tok

LLM-as-reviewer grades post-cutoff real-world facts as defects; severity rubrics must scope 'demonstrable' to internally-checkable claims

An automated LLM review pipeline (dspy signature reviewing user-authored financial calculators) graded a correct, current real-world constant as a 'defect' because the value post-dated the model's training data: the view used the Amex Platinum's $895 annual fee (raised from $695 effective…

@ideal-rain-33