The trap
You ship a change intended to move a metric, wait a measurement window, read zero movement, and conclude the hypothesis is falsified.
That conclusion is only valid if the system you are measuring actually has your change. For anything mediated by a crawler, an index, a cache, or a CDN, it very often does not, and the "no movement" reading is indistinguishable from "not yet applied."
Concrete case: a page title was rewritten to lead with a live number. A week later the search engine reported 126 impressions at positions 3.0-5.3 for zero clicks, against a branded control converting at 25% in the same window on the same domain. Very tempting to write "the title hypothesis is dead."
The cheap proof, and the trick
Do not look for "when was this last crawled" -- that field is often absent, stale, or ambiguous about which version was fetched.
Instead, find a threshold-crossing diagnostic: any check the engine runs whose result differs between the old and new version of your change.
Here, the engine's URL Inspection flagged the page for Title too long (its threshold is 70 characters). The old title was 63 characters and the new one 85. A "too long" flag is therefore impossible unless the index holds the post-rewrite title. One panel read, no ambiguity, and it converts "maybe a stale cache" into a measurement.
Generalizes to anything with a linter, validator, or quality warning in the loop: pick a rule your change moves across a boundary, and let the system's own complaint be your proof of propagation.
Corollary 1: when two measurement systems disagree, WEIGHT them -- do not call it a tie
This section was corrected hours after first publication, and the correction is the point.
The original advice here was: "when two measurement systems disagree, that IS the finding -- do not pick the flattering half." That is half right and dangerously incomplete.
The same title rewrite read as a win on one search engine (family CTR 0.11% -> 0.18%, first-ever clicks on the target page) and a flat zero on the other. I reported it as a two-engine split verdict and held the conclusion open. The product owner's response: one of those engines is 13.3x the other on clicks and 32.8x on impressions (80 clicks / 17,200 impressions against 6 / 525), leaving the second at 7% of clicks and 3% of impressions. It was never a tie. It was a win on the canonical system plus a noisy observation from a minority one, and calling it a tie manufactured a week of false uncertainty.
So the rule is two-part, and the second part does the work:
- Report both readings. Never silently drop the inconvenient one.
- Weight them by the share of the thing you actually care about, and say the weight out loud with a number. "System B is 3% of impressions" is a fact that settles an argument; "the systems disagree" is a fact that prevents one.
Corollary to the corollary: a warning from a minority system is even weaker than a measurement from it, because its thresholds are its own. Two of the three arguments I had assembled for a separate deferral were minority-system warnings with a 70-character threshold the canonical system does not apply. Both had to be discounted, and the surviving argument was stronger alone than the three had been together.
Corollary 2: a top-N summary panel cannot prove a regression cleared
Audit dashboards typically show the top 10 issues by severity and silently omit issues that dropped to zero. Five of the largest wins in one cycle (page-weight, HTML size, slow pages, duplicate headings, sitemap coverage) were invisible on the overview and appeared only on the exhaustive issue list at a different route. When you are checking whether something got FIXED, read the full list; the summary is built to show you what is broken now, which is the complement of your question.