When Not to Run a Test: The Readability Gate

Do not run a test when the result cannot change your decision or cannot be read at all. That covers four common cases: the page has a clear defect that needs fixing regardless of any test, the page collects so few conversions that random swings will swamp any plausible effect, the business has already decided to keep the change for legal, brand or operational reasons, and the coming weeks contain a sale, a launch or a seasonal shift that will rewrite the baseline mid-test. In each case, do the sensible thing directly: fix it, keep it, or gather different evidence. A refused test is not a missed opportunity. It is time returned to work that can actually be judged.

If a form drops submissions, a price displays incorrectly, delivery costs appear only after payment details, or a page errors on a common phone, the correct action is repair, not experimentation. Testing a defect against a fix wastes half the traffic on a version you already know is wrong, and it delays the repair while the defect keeps costing real visitors money. The same applies to missing information a buyer plainly needs: sizes, compatibility, service area, total cost. Add it, log it, and watch the step rate recover. Tests are for choices between two defensible versions. A defect is not a defensible version, and treating it as one confuses the archive as well as the visitors.

Readability is set by conversions on the specific page, not by total store visits. A busy homepage feeding a quiet product page does not make the product page testable. Ask two questions. How many conversions did this page or template collect last month? And how large would an improvement need to be before it mattered to the business? If detecting that improvement would need more conversions than several stable months can supply, the gate closes. This is not pessimism; it is arithmetic described in the pillar guide. The honest alternatives are a bigger change on a busier page, a before-and-after read, or direct observation of visitors.

Some changes are decisions, not questions. A legal correction, an accessibility repair, a price restructure the business has committed to, or a redesign already bought and launched for strategic reasons will stay whatever a small test says. Testing them performs rigour while producing a result nobody will act on, and a negative-looking noise result can then haunt a sound decision for months. Keep the change, log the reasoning plainly, and watch the relevant step rates afterwards for genuine harm. The archive entry should say kept on judgement, not tested and proven. That distinction is what keeps the whole testing programme believable when a real question does arrive.

A test needs a stable background. Starting one just before a promotion, a stock shortage, a press mention, a price change or the seasonal peak means the two versions will be judged inside conditions that are themselves changing, and any difference can be credited to the calendar instead of the page. Either run well before the unstable period, with time to finish inside stable weeks, or wait until trading settles. Time-based designs in sequential and time-based tests have their own narrow uses, but they do not rescue a test launched into a known disruption. Patience here is method, not delay.

Before any test, write five lines in the log. What is the change, in one sentence? Is either version defective in a way that should simply be fixed? How many conversions did the page collect last month? What effect size would change the decision, and can this traffic read it in whole stable weeks? What will happen on each outcome: keep, revert, or extend? If any line cannot be answered, the test is not ready. This takes minutes and prevents the most expensive failure in small-store testing, which is not a losing variant. It is a month spent producing a number nobody can trust, followed by an argument about what it meant.

A refusal is a routing decision, not a bin. Record the idea in the backlog with the reason it failed the gate: too thin, defect first, unstable weeks, or decided anyway. Thin ideas can return when traffic grows or be merged into a larger change with a plausible swing, which is the subject of testing offers, not cosmetics. Defects leave the backlog immediately as fixes. Ideas refused for timing get a calendar note for the next stable period. Kept this way, the gate builds a better queue instead of a pile of disappointments, and the next test you do run starts with a cleaner question and a fair chance of being read.