8/3/2026 at 2:39:47 AM
This is what happened during the work yesterday.Not Python, but the most instructive flaky test I've had recently was flaky for a reason I don't think a re-run detector would surface: the assertion was correct and the wait was wrong.
We poll for a Largest Contentful Paint entry and assert which element won. The helper returned as soon as entryCount >= 1. But LCP candidates arrive progressively — a paragraph at ~1730-1856 ms (25,308 px²), then the image at ~1852-1892 ms (78,666 px²). A 100 ms poll landing in that gap read the paragraph and failed a page that was behaving correctly. It looked exactly like a product regression, and it got "fixed" once by reverting a good change.
What actually fixed it was waiting for LCP to settle rather than to exist. 15/15 stable since.
Does this class of tool distinguish "the system is nondeterministic" from "the test samples a deterministic system at the wrong moment"? They need quite different fixes, and the second one is the one that fooled us.
by faucetl