Is premature optimization a warranted concern?

Here’s a wonderfully geeky question from a forum I’m on, related to lean startup:

I have a product that I want to test. I’ve built the one-pager website and set up some Facebook advertising to figure out the messaging / features that makes people sign up to determine the direction I will eventually build towards.

I have three ads: A, B, A+B
and three website versions: A, B, A+B

My question is whether I should test the advertising AND the website versions at the same time. I want to test quickly, but I’m worried that if someone clicks advertisement A and then lands on website B, it won’t be relevant and it will skew my data.

Is this a warranted concern? Should I test the advertising first, then the website version? Or should I do both and just figure that, with enough users, the data will smooth itself out faster then if I test one at a time? Any insight would be greatly appreciated.

Couple of things:

1. Yes it is a warranted concern. When you send traffic from ad A to landing page B, you lose them. You can set up ad A to point to landing page A, ad B to landing page B, and so on. Then by rotating through all ads, you’ll implicitly rotate through all of your landing pages.This “scent” which a prospect follows is really important. You may be losing people. They land on a LP which doesn’t correspond to their post-click expectations. This may skew your results.The goal here isn’t necessarily to convert, i.e. get a lot of signups. It’s to measure user behavior and preferences. Discover where you have latent/unmet demand.
2. The right test depends on what you’re trying to learn and what your hypothesis is exactly. For example, why are you even split testing in the first place? If you start split testing too early, it ends up being premature optimization. Understanding why you are split testing will help you formulate a meaningful hypothesis. Split testing only tells you if option A is relatively better than option B.You’re better off first choosing a threshold for conversion, then running ads to see if you hit that threshold. It’s a different type of statistical test and way of thinking about the problem. Then you’re also learning more about demand.
3. You will achieve statistical significance faster focusing only on user behavior around ads. Clicks. Based on that you can “test before you test”, and find out what larger/full tests are worth doing in a full environment.

As a rule of thumb, understanding your users better will usually be more profitable than understanding your product idea better.

If you want to go into greater depth about using landing pages rigorously to learn about your market, then check out Launch Tomorrow. It clears up a lot of the confusion around new products and landing page MVPs. Why blunder your way through? Launch Tomorrow‘ll take care of ya.