What Conversion Experts Wished You Knew About Optimization

Conversion optimization, whilegrowing in popularity, is still largely misunderstood.

As I see it, there are a small amount of people who really understand conversion rate optimization (CRO). Then there’s another small group of people who really don’t understand it, but are still publishing blog posts about it.

This creates confusion for the vast majority of people – those who are interested in increasing their revenue through CRO, but have no idea how to do it because of misinformation and conflicting advice.

Being atConversionXL, I get to meet and chat with the most experienced optimizers in the world. Through our agency, I also get to see where the gap is between what we (the optimization specialists) know about CRO, and what we wish business owners and executives knew.

Bridging this communication gap can only be a positive thing – creating a more efficient client/agency relationship, education in-house optimization teams, and as a result, creating more wealth for companies willing to invest in a CRO process.

In addition to my own experience at ConversionXL, I talked to some of the most knowledgeable optimizers I know for this article.

Here are 6 things optimizers wish business owners knew about conversion optimization.

1. CRO Case Studies Are Lying to You

Look, if an A/B testing case study opens the door to CRO for you, then I appreciate that value. But just know, almost allconversion optimization tips and case studies are BS.

Andrew Anderson, Head of Optimization atRecovery Brands, put it even more bluntly, saying “case studies are almost all frauds.” As he explains:

“As is the case with most disciplines, the case studies you see thrown about are just noise and empty marketing."

The same can be said about the loudest industry "experts" as well. While we appreciate that you get interested in optimization from them, following them blindly is the fastest way to fail.

Why, you may ask, do you have to be skeptical of case studies?

Most don’t supply full numbers, so there’s no way of knowing the statistical rigor of the test. I’d bet a great deal of them are littered withlow sample sizesandfalse positives. It’s deceptive, and it may not even be intentional; the author just doesn’t know what they don’t know.

If the results seem implausible, many times it’s because they are (source)

And of those that are legit –- how many losing tests did they run before they got that result? Often it’s the insights gained from the losing tests that lead to a strong winner, however, because these case studies are only reporting on wins you’re looking at a pure example ofsurvivorship bias.

Image via MrMoneyMoustache

Conversion optimization is also highly dependent on context.

Even if the test was based on good statistics,you’re ignoring context. Different traffic sources, different branding, and different customer segments behave differently and respond to different strategies. One thing that works for a case study might tank your conversion rates, simply because you’re working toward the goal without understanding the process that leads to it.

Thebest case scenario for CRO case studiesis that you:

• Remain skeptical on the validity of the results.
• Use them only as inspiration, not as an assumption to carry to your own tests.

Another point (and something we’re working on atConversionXL Institute) is that the process behind the A/B test (the resources required, the methodology, the research involved, and the iterations attempted) are more valuable than the end result.

2. You Probably Won’t Get Many Massive Wins (Unless Your Site Is Really Awful)

At conferences, when someone gets on stage and talks about their last 400% win - usually from something silly like changing a button color –those in the audience who know better just roll their eyes.

It’s really a misconception that conversion optimization is that sexy. Sure, it is a process that systematically improves the health of your business (a sexy idea to me), but it’s no silver bullet, and you almost certainly aren’t going to triple your revenue with one A/B test.

These success stories are ridiculously self-serving, and mostly serve as public relations for the startup or agency that is publishing them.

Experiment Engine data showsthat, instead, you should “get used to ties,” because they’re much more common than the outlandish uplifts.

Image via ExperimentEngine

“This backs the anecdotal feedback that most tests are flat,” says the company. “Which means getting one of those can’t believe it, incredible stories of increasing your conversion rate by 300% is a much less common occurrence than you’d be led to believe.”

When your site is terrible, the wins are larger and easier to attain. When you get to a certain point, the optimization becomes much more incremental – which isn’t a bad thing. Those incremental gains add up, and the net effect is similar to compound interest.

As Andrew Anderson put it,

“[optimization] is not sexy – doing optimization right means going away from what you think is important or what looks good...

...It’s about the grind of finding what matters most, giving it to users, and then continuing to go down the right paths. It means doing a thousand small things and not some big project or just some simple tweaks.”

Which leads me to my next point: conversion optimization is iterative.

3. Optimization Is Iterative

Iterative testingis the name of the game.

AsMarie Polli, Senior Conversion Strategist atConversionXL, says, “It’s highly unlikely that the first variation tested is the best option out there. For every hypothesis, there are a ton of different executions. They shouldn’t give up just because the first variation lost to the original.”

To illustrate, let’s say you create an experiment. Through some combination of heuristic analysis, analytics, and user feedback, it seems customers are concerned about the quality of the product and the lack of social proof. So you add some testimonials and run a test.

The result: no difference. So – does that mean your hypothesis was wrong?

有一个chance, but the real question you should ask yourself is this: how many different ways are there to add social proof to a page? How many different ways could we design a treatment addressingsocial proof?

The answer: infinite.

No one has a crystal ball, so it takes iteration to reach results.

On the same note, those new to optimization often get super disappointed with inconclusive test results. However, it’s necessary to push through and realize there’s no silver bullet.

Image via FindingBetterAgencies

Brian Massey, co-founder ofConversion Sciences, says one of the biggest myths in optimization is that “If thine testing worketh not in the first months, thy site is not worthy to be optimized.”

In addition, he argues:

A large part of optimization is the discovery of what actually matters. When you think of it that way, inconclusive tests are much more valuable than they first seem.

Andrew Anderson’s Discipline Based Testing Methodology同样强调测试发现。他的美联社proach really takes the pressure off of a first-test victory and places the value in discovery, efficiency, and different executions:

Of course, if your test hypothesis is based on a guess or “let’s just try it,” then move on to testing something else. But if you’ve done theresearch, it might just take more patience and iteration.

Further reading:What Do You Do With Inconclusive A/B Test Results?

4. All Wins Are Perishable

A win today mightregress to the meantomorrow. Why? The world changes. As much as we’d like to control for all factors, you can’t predict how the weather, pop culture, social media, or public relations will change your conversion rates.

I love howMatt Gershoff, CEO ofConductrics, put it in theDigital Analytics Power Hour podcast: All Ideas Are Perishable.

That essentially means that optimization never ends (or asBrian Balfour put it, “growth is never done.”)

“The need for CRO is ongoing,” says Marie Polli. “As competitors change, the marketplace evolves and so do your customers needs. It’s ok to take a break every now and then, but the mindset should be set to make CRO part of the ongoing workflow in the company.”

You might also know this asTactic Fatigueor theLaw of Shitty Click-Throughs. Essentially, over time, growth tactics fade in effectiveness, so you always need to be looking for that edge. You always need to be experimenting.

Image via Brian Balfour

“[Optimization] is not something you just do once,” according to Andrew Anderson. “It takes an ongoing attitude and a very strong discipline to do the right thing to get results. Do it right and it will be the leading driver of your company’s growth. Do it as just a test or two, and it will barely be a blip on your radar.”

Ton Wesseling once told methat one of the biggest mistakes a company can make is thinking optimization is more optional than other marketing spends. The perception is that it will only freeze ROI, where other acquisition channels (PPC, SEO, etc.) actively drop your traffic.

The thing is, optimization is a program – a discipline – not a tactic. If you invest in it for the long term, and build a strongculture of experimentation, you’ll reap the rewards.

5. Keep Calm and Test On

According to Marie Polli, “business owners are too hasty, which is understandable, but can result in a situation where tests get closed too early and false positives get implemented.”

Or asJudah Phillips put it, “like fine wine, conversion optimization takes some time to mature.”

We know that you want to move quickly. Optimizers want to move quickly, too. But math doesn’t bend itself to the demands of a marketing manager or a CEO (if only).

If you run a test poorly – stopping it early, for instance – you’re taking the very real risk that your data is inaccurate. And why are you running controlled experiments, anyway, if you don’t care about the integrity of the data?

A/B tests are notoriously fickle at the beginning. They change rapidly, so it’snot a good ideato call it quits before the numbers stable out.

When the results are wild at the beginning, do as Dr. Julia Freese ofKonversionKraftsaid and “keep calm for now and continue testing.”

In addition, you can’t just stop a test when you hit statistical significance (which is acommon misconception).

While not going too far into details, the big problem with only looking at significance is you’re not accounting for a representative sample.

Let me explain.

If you start a test on Monday and end it (with significance) on Friday, you didn’t account for weekend buyers and weekend buying habits, which might be much different than any given day of the week.

The correct way to go about it is to pre-calculate your minimum needed sample size (never a penalty for a bigger sample size). Then, test for one (better still two) full business cycles – so the sample would include all weekdays, weekends, various sources of traffic, your blog publishing schedules, newsletters, phases of the moon, what Donald Trump is tweeting about, and everything else that might influence the outcome.

Or if you want to read more in-depth on how long to run a test, readRonny Kohavi’s brilliant answer on this Quora thread.

6. Distance Yourself from Opinion and Ego

When it comes to optimization (and much of life),ego is the enemy.

According to Marie Polli, one of the biggest impediments to client work is that, “they treat their website like it’s their baby.”

Likewise, “if they really want to give optimization a chance, they should take a step back and allow for numbers to determine which variation performs better.”

Image via EnricDurany

When you distance yourself from the idea of a specific outcome – even though I know you want variation B to win because your expertise tells you itwill–you open up the possibilities for a discovery based program that leads you places you might not have imagined.

AsTon Wesseling, founder ofTesting.Agency, says, “they are calledexperiments!”

He argues that clients should know the value of the long-term insights from experiments and use this as a great driver for product innovation and larger changes within the current business (models).

“Don't be rigid when experiments don't follow guidelines or design guides,” says Wesseling. “If we want to learn if a certain change creates impact, we should be able to change it out-of-the-box – or even disruptively.”

Conversion optimization is a powerful tool for your entire business if you let it be. As Matt Gershoff once told me, optimization is about “gathering information to inform decisions,” and in that way, it’s really more about decision optimization.

According to Stephen Pavlovich fromConversion.com, optimization represents a much larger scope if you can think outside of simple cosmetic changes:

“Conversion optimization isn't about putting lipstick on the pig – we're not tweaking layout and text. Testing is a hugely powerful tool that can answer fundamental questions to your business. At Conversion.com, we work with our clients to test everything from brand messaging to pricing, from product features to market potential.”

Paul Rouke, founder ofPRWD, agrees, saying “Forget testing button colours – there is a full spectrum of testing possible which can impact your overall strategy and value proposition –iterative, innovative and strategic.

“Never underestimate the opportunity that conversion optimization has to impact your whole business,” he says.

This goes back to one ofAndrew Anderson’s central tenets of testing for discovery: anything that limits the way you think about testing (namely opinions and ego), essentially limits the potential, effectiveness, and efficiency of the program at large.

In my experience, this is the hardest part of optimization. Sometimes the datatells us things we don’t want to hear. What separates mediocre optimization programs from the great ones is that the great programs value experimentation, where the mediocre ones are mired in politics and ego-protection.

Recommended reading:6 Clever Nudges To Build a Culture of Experimentation

Conclusion

Conversion optimization is one of the most effective ways to grow your business.

As Paul Rouke said, “If you want to truly become a "customer-centric" business with huge growth potential, then intelligent, customer-insight-driven conversion optimisation is the only way you will effectively achieve this.”

While it’s still misunderstood by many people, if we bridge the gap in understanding, we all benefit:

• Agencies and consultants are less frustrated.
• Clients are less disappointed.
• Optimization becomes more effective, which increases revenue.

We’re trending in the right direction, and have seen much more mainstream adoption of optimization in the last few years.

So if you’re planning ondoing conversion optimization(orhiring conversion optimization services) in the future, keep some of the above things in mind to get more out of the program.

---

About The Author

Alex Birkett is a Growth Marketer at ConversionXL. He moved to Austin, Texas after graduating from the University of Wisconsin. Follow him onTwitter.