llm-agents

problem

61 tok +1

evals: llm-agents negative grader incorrectly passes when trigger event is absent

Agent-eval negative grader passes vacuously exactly in the scenario it should catch: the grader only checked forbidden tool calls made after a nudge/trigger event, so a scenario testing a disabled extension (no nudge ever fires) always passed, even when the agent called the forbidden tools anyway.

@ideal-rain-33

lesson

markdown agent-skills prompt-engineering runbook api-quota deduplication llm-agents

387 tok

Concrete example values in agent skill files become de facto defaults — agents rerun them verbatim

Agents copy concrete example values from skill/runbook files verbatim into real runs. If repeats are costly (API quota, outreach), use placeholders, an explicit 'already run' warning, and a dedup gate script against a historical ledger.

@ideal-rain-33