Multivariate and Other Overkill: What to Skip at Your Scale

Skip multivariate testing, multi-armed designs, personalisation variants and most fancy sequential schemes until your traffic can feed a simple two-version test comfortably, which for many small stores means skipping them indefinitely. Each additional version divides the same small pool of conversions into thinner slices, so every comparison gets noisier and the run gets longer, while the interactions these designs exist to detect are usually smaller than the main effects you have not measured yet. The overkill is seductive because the software makes it easy to launch. Ease of launching is not readability. One big question, two versions, read patiently, remains the design that fits your numbers, as the pillar guide sets out.

A multivariate test varies several elements at once in combination: two headlines by two images by two buttons is eight versions. The promise is efficiency and insight into interactions. The cost is that each version receives an eighth of your conversions, and interaction effects, the difference between a combination and the sum of its parts, are typically the smallest effects in the room. A store that cannot read a two-version test in a month would need seasons to read eight versions, by which time the page, the range and the season have all changed. If several elements genuinely need deciding, test them one at a time in order of expected swing, largest first, or bundle them into one large version and accept the coarser lesson.

Showing different versions to new and returning visitors, or to different sources, multiplies the same problem by audience: each cell needs its own readable count, and small stores rarely have one readable audience, let alone four. Personalisation also adds failure modes a simple test does not have: visitors seeing the wrong version, members of a household sharing a device, and a page whose behaviour you can no longer describe to a customer who phones. There are narrow exceptions, such as a returning-buyer page whose traffic is genuinely large, but the default answer at this scale is no. Write one page that serves the main audience well, with honest signposting for the others, and spend the complexity budget on the offer instead.

The same suspicion applies to bandit algorithms that shift traffic automatically toward an early leader, since early leaders at low counts are usually lucky rather than better, and the algorithm banks the luck. It applies to testing platforms sold on dashboards and significance badges your traffic cannot support, covered in tools for testing on a budget. And it applies to elaborate test calendars that queue twenty experiments a quarter for a store that can read three. Overkill is not only statistical. Every running design consumes attention, the scarcest resource in a one-person programme, and attention spent babysitting an unreadable design is attention not spent watching visitors or fixing defects.

Three exceptions deserve naming. A genuinely high-traffic page, by your store's standards, with hundreds of monthly conversions can support a three-version test if each version is a large, distinct concept. A design decision that will be templated across hundreds of pages justifies more rigour than a one-page choice, because the effect multiplies by every page it touches. And a platform-provided test of a platform feature, such as a checkout extension, may run on the platform's pooled mechanics in ways that change the arithmetic. Each exception should be argued in the log against the readability gate, not assumed. If the argument needs the word synergy, it is not an exception.

Replace multivariate with sequence: order the candidate elements by plausible swing and decide them one at a time, using tests where readable and judgement where not. Replace personalisation with clarity: one page, one main job, with the minority audiences routed honestly rather than detected secretly. Replace bandits with patience: equal split, whole weeks, a pre-written read date. Replace the twenty-test calendar with the monthly rhythm of the pillar: one question, the right instrument, a full archive entry. These replacements look less sophisticated in a screenshot and produce more knowledge per month, because every design in the programme is one your traffic can actually complete.

When a tool, a freelancer or a course proposes a more elaborate design, ask one question: how many conversions per version, per month, does this design need, and how many do we have? Make the proposer do the arithmetic with your real counts from the last quarter. Designs that survive that arithmetic deserve consideration; designs that change the subject to engagement, impressions or best practice have answered the question. Sophistication in testing is earned by traffic, the way a larger workshop is earned by orders. Until the counts arrive, the craft is restraint: big questions, two versions, honest reads, and an archive that grows more valuable every month precisely because it contains nothing inflated.