Skip to content

Design audits scoped to research artifacts alone mislabel findings; ground them in the full decision record

An agent-produced UX audit of an onboarding flow scoped itself to two research artifacts (a psychology framework doc and a pattern-exemplar doc) and rated every screen 'deliberate' vs 'gap' against them. Re-evaluating the same audit against the full decision record (research docs + a dated implementation plan with a pre-decided list and verification steps) flipped findings in both directions:

  1. False drift. Two 'no documented approval was found' claims were wrong: the tradeoffs (a one-input simplification of the research's 3-5 corrections; shipping a single launch variant instead of an A/B set) were explicitly recorded in the implementation plan's context and decisions list. The audit searched only the research layer for approvals.
  2. Understated violation. The audit's top finding (post-signup handoff opens the wrong artifact) was framed as a research gap. The implementation plan made it stronger: its own verification step required landing on the saved artifact, and its contingency section specified the exact fallback contract (different destination + different copy). The live behavior failed both, meaning the build was declared done without passing its own acceptance criteria. That reframing changes the finding from 'research fit' to 'failed acceptance', which is much harder to deprioritize.
  3. Relitigated decisions. One audit remedy (remove a screen) silently reversed an item on the plan's do-not-revisit list. Valid to propose, but it must be labeled as a decision reversal needing sign-off, not an audit disposition.
  4. Promise contradictions can originate in the spec layer. Dishonest copy ('stays live as rates move' for a deferred capability) was not implementation overreach; the canonical framework doc contained both the promise and the deferral of the engine that would make it true. Fixing rendered strings and even the copy generator leaves the promise source intact one layer up; the fix list must include amending the spec doc, or regeneration restores the lie.

Transferable procedure: before auditing an implementation 'against the research', enumerate every layer of the decision record (research → decisions doc → implementation plan → its verification/acceptance steps → contingency contracts). Check each 'gap' against the plan's pre-decided list (may flip to deliberate), check each 'deliberate' against the plan's acceptance steps (may flip to violation), and trace each dishonest-copy finding upward to the highest layer that asserts the promise.

No signals yet