Experiments are Bricks

I was listening to an episode of Lenny’s Podcast with the Head of Growth at Shopify. In it, they talked about how 30% - 40% of all experiments fail to show long term impact. I’m not doubting the numbers, but I think this misses the point. Experiments around growth aren’t meant to last forever. They’re bricks, there to provide the support for the experiment that comes next. If you’re experiment count is low enough that you’re still thinking about the change you make to the CTA text, you probably aren’t thinking about growth experiments the right way.

Given enough time, almost any A/B test style growth experiment you run will probably look meaningless. Even if you measure statistical significance, over a long time horizon that significance tends to fade. You shouldn’t expect every little change in copy and colors to continue to deliver uplift forever.

Experiments, especially the ‘test a small change’ kind, aren’t meant supposed to be tasks to complete and then forget about. They’re bricks. Each one placed down so that the next one can sit on top. No single experiment is ever going be the key to unlocking growth, but lay down enough of them, each building on the last and you have a good shot at getting there.

This is partly why these experiments lose their significance over time. The controlled conditions you had at the time aren’t there any more after you’ve made another two dozen changes. Especially when you have worked to optimise each change to push key numbers higher.

That isn’t to say that none of these changes really were just statistical noise, there are plenty of ways to mess up experimentation. Unless you’re confidence threshold is absurdly high you’ll let a few of those slip through. The reality is though that once all is said and done you’ll have a hard time telling them apart. Dwelling too much on trying to identify if an experiment you ran last year really made a difference is almost always a waste of time.

I think this also speaks to the temptation to wipe the board clean and start again from zero. Treated the wrong way, small changes can feel like a waste of time and you can trick yourself into thinking that a clean slate is the answer. This might be a brand new marketing website or a rewrite of your product. Both of these, more often than not, are not going to be the solve.

After the little steps we took have faded into insignificance it can be easy to forget about them. Once forgotten the prospect of starting from zero seems appealing. It can’t be that hard can it?

Before long, any rebuild we attempt is going to reach an impasse. Those bricks we laid over months and years of small experiments add up to a lot in their totality. Each brick built on the last and created something through repetition. Adding change upon change and allowing for momentum. When you’re trying to grow, momentum is invaluable.

Growth can be a grind. I’m yet to meet a founder, PM or growth manager who got everything right first time around. Building a successful product requires a lot of iteration. There may well be secrets out there but you’re very unlikely to find them through thinking. The message, feature and price point that lead you from zero to product market fit are discovered through moving fast and taking lots of small steps.

You rarely have the time or the money to bet big on one idea and follow it blindly to the end. Moving fast means correcting course as you go. Each experiment you run should be helping you nudge just that one little step forward in the right direction. A business is the sum of its parts, both the ones that are there today and all the parts that have come before.