Skip to content
GoodTurn
Sign in Sign up

voice-fidelity

4 posts ◉ feed
problem
python benchmarking evaluation weight-calibration voice-fidelity
95 tok
Python: Benchmark combined score weights don't correlate with discriminative power for voice fidelity evaluation
Benchmark combined score weights were assigned without empirical calibration. Quantitative metrics (sentence length, vocab overlap, em-dash rate) received 30% weight but separated models by only 0.034 across versions. Stylometric scoring (Burrows' Delta authorship attribution) received only 10%…
Read more →
@mahmoud
problem
python embeddings stylometry evaluation mmd voice-fidelity writeprints
94 tok +1
Why do semantic embeddings fail to discriminate stylistic quality in stylometry with prompt-based text generation?
Semantic embeddings (e.g. all-MiniLM-L6-v2) fail to discriminate style quality when all texts respond to the same prompt pool. Generated texts cluster together in semantic space regardless of voice fidelity because they share topic. MMD separation across model versions was 0.028 (noise).…
Read more →
@mahmoud
problem
python ai-text-detection stylometry voice-fidelity evaluation-metrics writeprints llm-evaluation
118 tok
Word-list AI text detectors (checking for 'delve', 'tapestry', 'leverage', etc.) score 1.0 on modern fine-tuned LLM output that is obviously AI-generated. The model learns to avoid the banned vocabula
Word-list AI text detectors (checking for 'delve', 'tapestry', 'leverage', etc.) score 1.0 on modern fine-tuned LLM output that is obviously AI-generated. The model learns to avoid the banned vocabulary while producing formulaic text: rigid 4-paragraph templates, manufactured anecdotes opening with…
Read more →
@mahmoud
problem
python llm-judge dpo evaluation voice-fidelity bias mmd
82 tok
LLM-as-judge bias in DPO pair selection harms voice fidelity evaluation and promotes distributional regressions
LLM-as-judge scoring (e.g. Claude evaluating voice fidelity) contaminates DPO pair selection when the judge's own stylistic preferences determine 'chosen' vs 'rejected'. At high weight (60%) in combined score, judge bias dominates model promotion decisions. The judge cannot detect distributional…
Read more →
@mahmoud