Empty pocket testing by Buffer

The following are a number of Lean Startup validation case studies. Some will already be well known; some will be completely unknown. A lot of landing page testing has happened since The Lean Startup was being pieced together by Eric Reis. These are retrospective reconstructions of what happened using landing pages as vehicles for minimum viable products.

For example, Buffer did empty pocket testing with a landing page before building their product. Just for your context, Buffer is a social media sharing tool, allowing you to publish tweets or social media posts on a pre-defined schedule.

While you may have heard of some of the lean landing page case studies before, there is a lot of nuance in exactly what each test actually tested. They are typically not “traditional” A/B split tests, where they were testing whether a new variation of an ad or landing page beat the old one.

In order to help make it more explicit, I’ve tried reformatting the experiments to be lightweight. Lean Startup experiments are generally not about testing the landing page or the product, but the business ideas they represent.

Hypothesis: The target audience wants this product

Test Type: Value hypothesis, confirm the problem exists and people want a “hands off” way to tweet

Success Criteria: Emails gathered > 0

Traffic Source: Social media, word of mouth

[image:Buffer]

Result: Pass. A few people used it to give founder Joel Gascoigne their email. He used these to get some useful feedback and initiate a conversation with prospects.

Step back: Potential users had left their email address at a random web page promising them help with this particular problem. This meant the idea itself was valuable, and there was potentially unmet demand for this particular idea. I would be careful to use only # of emails gathered as the primary metric in all cases.

For a consumer facing product, this is probably good enough, assuming you have enough traffic. It would be better to also include some kind of a target number of sessions, to make sure that you have enough “attempts to convert” to make your metric meaningful.

Hypothesis: The target audience is willing to pay for the product

Test Type: Value hypothesis, confirm declared willingness to pay for “a way to automate their tweeting”

Success Criteria: People would click-through the additional pricing page, and still leave their email.

Traffic Source: Social media, word of mouth

[image:Buffer]

Result: Pass. People were still clicking through this additional step. Joel was able to gather useful information about the suggested pricing plans, in order to figure out pricing.

Step back: Potential users weren’t put off by the blatant pitch, and still kept leaving an email address at the far end. What Joel hadn’t tested was whether people would actually buy; however, he was able to complete a functional prototype within seven weeks, and tested this hypothesis with a functional system. He actually got his first paying customer 4 days after the “rough-around-the-edges” product launch.

I just wanted to thank Joel for contributing this fantastic case study to the Lean Startup community. It’s quite a well known one. As a result, I really wanted to cover it as an example of a line of thinking that’s worth following.

Empty Pocket Testing

At the core, the Buffer landing page MVP test was meant to address a major question for founders: will “they” buy it, if I create it? Before getting caught up in theoretical debates about what is and isn’t an MVP, Buffer just did an experiment. It just happened to be using a landing page to address a major risk factor for a new product business.

In this particular case, checking for whether early adopters

1. had the budget and
2. were willing to spend it

de-risked spending more time and money on the solution significantly. Even though they were asking theoretically, this helped to validate their sense that their target early adopters would be willing to pay.

[image:Dan Moyle]

If you’d like to see a number of case studies like the above, grab Launch Tomorrow. I’m updating it in an upcoming version with a lot of in-depth experiments that have been run.