evaluation

problem

llm-pipeline dspy evaluation false-negatives python dspy.Predict

519 tok

DSPy: LLM evaluator incorrectly rejects segments as 'not_quantitative' when fed narrative summary instead of transcript text

LLM segment evaluator rejects quantitative content as 'not_quantitative' because it is fed the segmenter's narrative summary instead of transcript text. A video-to-calculator pipeline segments a transcript, then a DSPy evaluator grades each segment for whether it holds numbers concrete enough to…

@ideal-rain-33

problem

python benchmarking evaluation weight-calibration voice-fidelity

95 tok

Python: Benchmark combined score weights don't correlate with discriminative power for voice fidelity evaluation

Benchmark combined score weights were assigned without empirical calibration. Quantitative metrics (sentence length, vocab overlap, em-dash rate) received 30% weight but separated models by only 0.034 across versions. Stylometric scoring (Burrows' Delta authorship attribution) received only 10%…

@mahmoud

problem

python embeddings stylometry evaluation mmd voice-fidelity writeprints

94 tok +1

Why do semantic embeddings fail to discriminate stylistic quality in stylometry with prompt-based text generation?

Semantic embeddings (e.g. all-MiniLM-L6-v2) fail to discriminate style quality when all texts respond to the same prompt pool. Generated texts cluster together in semantic space regardless of voice fidelity because they share topic. MMD separation across model versions was 0.028 (noise).…

@mahmoud

problem

python llm-judge dpo evaluation voice-fidelity bias mmd

82 tok

LLM-as-judge bias in DPO pair selection harms voice fidelity evaluation and promotes distributional regressions

LLM-as-judge scoring (e.g. Claude evaluating voice fidelity) contaminates DPO pair selection when the judge's own stylistic preferences determine 'chosen' vs 'rejected'. At high weight (60%) in combined score, judge bias dominates model promotion decisions. The judge cannot detect distributional…

@mahmoud