The Test Archive: Recording Losses at Full Size

The test archive is the single file, folder or log where every test, before-and-after read and significant judgement change is recorded in the same format: the hypothesis, the dates, the conditions, the numbers, the decision, and the result at full size, including losses and inconclusive reads. Its value is not administrative. It is what stops the business re-running a losing idea eighteen months later under a new name, what keeps claimed wins honest when memory starts editing them, and what turns scattered experiments into cumulative knowledge about your visitors. An archive that holds only successes lies by selection, and everyone who works from it inherits the lie.

Keep the format short enough that you will actually complete it. Date range and page. The hypothesis, copied from the log entry written before the test, not paraphrased after. What changed, with a screenshot or a link to the saved version. The primary measure and its numbers for both versions or both windows, with the segment stated. Guard measures. Trading notes: promotions, stock, traffic shifts, defects found. The decision, and the confidence grade in plain words. That is one entry. Anything less and a future reader cannot re-judge the result; anything much more and entries will start being skipped on busy weeks, which is how archives quietly die.

A loss recorded at full size names the prediction, the size of the shortfall and the best current explanation, labelled as explanation rather than fact. It does not get softened into mixed results unless the measures genuinely split, and it does not get buried under the next win. Losses are the archive's most valuable entries because they close doors: the guarantee framing that did nothing, the bundle visitors disliked, the headline style this audience ignores. Each closed door saves a future test slot, which at low traffic is the scarcest resource the programme has. The discipline in reading results without self-deception ends here, in whether the written record matches what the dashboard actually said.

Record inconclusive reads with the same care, including the counts achieved and the effect size the test could and could not have detected. That last detail is what makes the entry useful later: if traffic doubles next year, the archive tells you the question is now readable and worth re-running with the same hypothesis. Without it, the idea looks merely failed and gets discarded, or merely untested and gets re-run at the same unreadable size. Label the follow-up decision too: retest larger, retest later, or drop. An inconclusive entry with a routing decision is a completed piece of work, not an embarrassment to be filed quietly.

Make one habit the archive's reason to exist: before any test or major change is approved, search it for the page, the element and the idea. Three outcomes are possible. The idea was tried and lost, so it needs new evidence to return. The idea was tried inconclusively at a smaller traffic level, so it may now be readable. Or the page has a history that changes the hypothesis, such as two losses on price framing that point toward risk, not cost, as the binding hesitation. That search takes minutes and routinely saves months. It also disciplines proposals in a healthy way: a new idea arrives stronger when its proposer can say how it differs from what the archive already holds.

The best format is the one inside the tools you already open: a page in your notes system, a spreadsheet with one row per test and a linked document per entry, or the change log itself extended with result fields. Avoid a dedicated testing platform whose reports live apart from the log and die when the subscription lapses. Review the archive at the same monthly sitting as the funnel review, and write a three-line summary of what the programme currently believes about its visitors: the offers they respond to, the hesitations that matter, the ideas that are closed. That living summary, refreshed from full-size records, is the closest thing a small business has to an institutional memory.

A solo operator has no team politics, but has a subtler enemy: the wish to feel that the month's work produced wins. The archive counters it by making honesty the path of least resistance: same template, same sitting, losses written in the same tone as wins. If you work with a partner, a freelancer or an agency, the archive does heavier work: it is the shared record that survives staff changes and the document any claimed result can be checked against. Either way, the rule from the pillar guide holds. Twelve honest entries a year compound. A cabinet of edited trophies compounds nothing except confidence in stories.