Day 7: Set Up Your A/B Testing Software

Conversion optimization is a process that requires continuous testing, refining, and scaling. Even the smartest marketers aren’t going to get their landing pages right on the first go. There is always room for improvement, and even the difference of 1% in conversion rates can make a world of difference.

Let’s say you bring in 1,000 visitors, and a conversion event is valued at $100.

At a 1% conversion rate, you’ll make $1,000. At a 2% conversion rate, you’ll make double — $2,000. It goes without saying that every incremental improvement is extremely important for your company.

What is A/B Testing?

The process of A/B testing ensures that you’re always driving these incremental improvements — and that the discovery process never comes to a standstill.

By definition, A/B tests are a type of experiment. You test two (or more) variations of something against each other against a randomized, statistically valid sample. Then, you measure the results to see which variant drives the best ROI.

What Can You A/B Test?

Anything. That’s why you need to be extremely strategic. If you A/B test everything, you won’t get anything done. You’ll be testing all day without driving real results. As we mentioned yesterday, you need to prioritize your educated guesses.

Here are some examples of elements that may make sense to A/B test:

  • Page layout

  • Communication/value proposition

  • Explainer videos

  • Visuals

  • Calls to action

  • Sign-up forms

  • Personalization

  • Copy length

  • Headings

  • Subheadings

  • Customer testimonials

  • Client logos

Here is an A/B testing case study from Underwater Audio, a company that sells waterproof iPods and audio accessories.

Underwater Audio’s primary goal is to sell products. Users are likely to be exploring items, researching options for products, and potentially reading reviews too. For that reason, webpage scanning patterns are extremely important. Underwater Audio suspected that their pages were not designed as optimally as possible.

Underwater Audio wanted to test out a concept that we discussed earlier this week, the F-Shaped browsing pattern:

Image Source

Typically, companies have just a few seconds to capture their audience’s attention. That’s why the Underwater Audio tam decided to test the visual hierarchy on their product pages.

Here is what the original page looked like:

The CTA and testimonial bubble were moved to become the center of attention. The company hypothesized that these elements were important sales tools that would inspire user action.

Here’s what happened:

The company’s hypothesis was correct. The new page outperformed the original version by a 35.6% bump in sales. The testimonial from a professional swimmer, more prominent CTA, larger font size, and clearly communicated value proposition were key conversion drivers.

How to A/B Test

Don’t do it manually. If you run on in-house IT resources and Excel, you’ll end up taking forever. Rely on software to simplify the process as much as possible.

A/B testing software will help you set-up experiments, randomize web traffic samples, and track results in one dashboard. Don’t worry about building something custom. You can get up and running tomorrow, if you want.

And that brings us to your homework for the week…

Today’s Homework

Today’s assignment is to get set up with A/B testing software. Here are two options worth checking out. Try demos for both to figure out which functionality you’ll need. Here is a Quorathread that can help you research the features available in both options — most of the reviewers seem to think that Optimizely is the simpler of the two options.

Visual Website Optimizer

This tool comes with more than 100 features that let you split website traffic by geography, run multivariate tests, segment user behavior, and optimize for mobile. You can run tests in just a few moments and track analytics within the same interface.


This tool can help you test website elements and track results on the fly with a single line of code. You can change copy, colors, images, and CTAs.

Unbounce, a tool that we introduced you to earlier this week for creating landing pages, also integrates with both of these platforms. You can also connect both tools to popular CMS platforms as well as Google Analytics.

The first week is over, and boy, did time fly by quickly. Congratulations on making it through. Here’s to an awesome Week 2.

Day 28: Check Your A/B Test Results

Two weeks have passed since you launched your A/B tests. Today is the day that you’ll check your data.

In future weeks, as you boost your website traffic, you’ll start to see statistically significant results more quickly. If you’re still getting your website up and running, have patience. Results take time. It’s not uncommon for marketers to keep A/B tests going for months at a time.

If you’re new to testing, the results can seem overwhelming. You’re probably buried in data, wondering how to interpret your results. The following step-by-step guidelines will help you navigate this process.

Step 1: Gather Your Data

When you set up your A/B tests, we asked you to pick exactly one conversion metric to watch. Here are some examples of what you might have been tracking:

  • Product sign-ups

  • Ecommerce transactions

  • Email list sign-ups

  • Tree trial sign-ups

  • Demo completions

  • Ebook downloads

  • Video plays

  • Social media shares

  • Blog post comments

  • Revenue per visitor

…The list goes on. Prioritize the conversion events that grow your business.

Step 2: Revisit Your Hypothesis

Before analyzing your results, turn back to your original research question. Statistical significance means that results couldn’t have happened by choice. There will be an explanation for the trends you observe.

When you analyze results from your A/B test, you’ll have a strong understanding of what happened. Your hypothesis is a crucial step for understanding ‘why’.

What user behavior to expect to observe and why? This crucial question will help you translate quantitative insights into strategic best practices.

As an example, take a look at the following A/B testing customer story from Optimizely and Backcountry, an online retailer that sells outdoor gear, clothing, and accessories.

The company wanted to optimize its upcoming shipping strategy for the holidays. The company decided to test a series of shipping deals during other high-traffic holidays. Shipping deals often help entice customers to make a final purchase.

Backcountry’s product team hypothesized that offering free, two-day shipping during its annual Fourth of July sale would increase revenue per visitor on its site.


Lower shipping costs would incentivize customers to buy more.

Here are the original and variation shopping carts:

Step 3: Check Your Results

Backcountry did not share the results of this study. Can you guess what the outcome might have been?

Remember that the goal of this test was to determine which shipping option would boost revenue per website visitor.

For discussion’s sake, let’s make a guess that the variation shipping option won.

Keep the Process Going

A/B testing is a continuous process. Don’t stop with the first set of results.

The Fourth of July test was the Backcountry team’s first step. At the time of Optimizely’s customer story, the team planned to test other high-traffic holidays for insights into Christmas and Thanksgiving.

Today’s Homework

  1. Interpret your A/B test results by following the steps that we just outlined.

  2. After analyzing your data and confirming whether results were statistically significant or not (i.e. whether or not they were valid), put together an action plan for the next set of tests. Don’t let the momentum stop, and continuously strive for improvement.

Day 29: Do Qualitative Research…Again

We had you set-up your first qualitative studies on day 3, at the beginning of this conversion optimization process. Today, you’re going to do it again. Why?

Over the last month, you’ve made significant changes to your website optimization strategy. There could be blind spots that your testing, product, and marketing teams are overlooking.

The changes you make to your website will affect your users. Keep a pulse on what your users are thinking and feeling.

Validate Your A/B Tests

Yesterday, when you ran your A/B tests, you made some educated guesses about what your users were thinking and feeling.

Qualitative research can help you validate those insights.

Validation comes from talking with a number of people, until you start to see a pattern.

Don’t rush these conversations. In all honesty, today’s homework assignment should probably take you a couple of days — maybe up to a week. Devote enough time and attention to collecting in-depth insights and identifying patterns.

Today’s Homework

Go back to day 3 and repeat those exact steps. Compare your findings today with your findings back then. What’s changed? What hasn’t? How do your conversations compare with the results from your A/B tests?

Day 30: Conclusion – Future Proof Your Testing Framework

Conversion optimization is not a one-time deal. For your strategy to succeed and continuously produce results, you need to create a testing culture within your organization.

What does that mean?

  • Put someone in charge of overseeing testing

  • Make testing a part of your processes

  • Don’t jump to conclusions about user behavior

  • Make it a habit to test assumptions, always

  • Always be driven by data rather than inferences

  • Make the data collection process extremely quick

  • Build relationships with vendors so that testing is a continuous process

Conversion optimization should feel like a habit rather than a special project. Hopefully, with 30 days of analysis under your belt, you and your marketing team are comfortable with creating a routine for your business.

If there’s anything you should have learned over these last 30 days, it’s that the process of conversion optimization needs structure. You need to outline a strategic plan and roadmap for the goals you want to achieve.

Start with the big-picture objectives, and reverse-engineer the process.

Today’s Homework

In these first 30 days, you likely came away with some very tangible insights and conclusions. The results of your A/B tests may have pointed you to a high-converting landing page.

Now, it’s time to start planning the next 30 days.

How does conversion optimization fit into the scope and framework of your marketing goals?

Today, you’re going to start figuring out the answer to that question. But don’t worry, you don’t need to do it alone. Get your team involved. Build a plan together, and get the entire team involved. Testing is a collaborative effort. You’re not alone.

What do you want to learn?
What do you think your company should be doing better?

These two questions are powerful motivators through the testing process.

It’s not easy. Sometimes, it will be fun, and sometimes, it will be tough. Just remember that your investment is well-worth it in pushing your organization forward.

Always be testing. Always be learning.Day 3: Run Qualitative Research Studies

On day 2, we promised that qualitative research would be an important topic. This technique is so valuable to your conversion optimization strategy, that we’ve devoted an entire day to it.

More often than not, marketers get obsessed with metrics. They’re buried in numbers without fully-understanding the why and how.

Qualitative research is extremely important for making sense of your research. But unless you’ve taken social science or MBA classes, you’ve probably never even heard of this term.

What Is Qualitative Research?

Qualitative research is an exploratory practice that researchers and marketers deploy when they’re not quite sure what to expect. Generally, researchers enter the process with a completely blank slate and open mind.

At the beginning stages of qualitative research, marketers and business leaders may not yet have a thorough understanding of the problem — qualitative research can help define it. Your strategy is to look for connections between concepts and ideas (ones that you didn’t see before — the point of qualitative research is to identify new perspectives).

A common misconception about qualitative research is that you’re not collecting data. This is a myth. It is important to understand that data collection means more than generating statistical analyses and numbers.

Common methods of collecting data include focus groups, triads, in-depth interviews, uninterrupted observation, and ethnographic participation. Methods are typically semi-structured and casual as opposed to rigid and formal.

Image Source: Wikimedia Commons

Don’t Waste Time: Turn to Existing Data

One way to kickstart your qualitative research process is to turn to the conversion optimization tools that you already have in place. If you’re running live chat software or are working with a full-fledged customer service team, you’re in luck. You probably have a bunch of data that is already going to waste.

Here are three places to turn:

  1. Your LiveChat transcripts and

  2. Your customer service records

  3. FAQs in your knowledge center (via UserVoice)

The strategy you’re using here is observation. Research customer feedback in a natural, uninterrupted setting.

The strategy is to sift through these records until you start to see a clear pattern in customer activity. What are some clear pain points? What do your customers value, and what are they looking for? Why are customers interested in your business?

Jump into these resources with an open mind, but also make sure to maintain a sense of focus. Pick 2-3 questions that most directly impact your business’s ROI. For instance, you might want to research which website elements are posing challenges to conversions.

Run Customer Interviews

The word “interview” is enough to scare anybody. Before you get research-happy and whip out your tape recorder, take a moment to remember your most valuable learning moments. These are casual conversations, not structured, research-driven Q&As.

So do just that. Make an effort to talk to more of your customers 1:1, especially if you are running your business and part of the marketing team. Don’t expect your sales and customer service reps to do this leg-work for you. Offer to listen in on some calls, or handle your calls yourself.

You might want to create a script of questions you want answered, or you may want to keep the conversation completely open-ended.

Whatever you do, keep the dialogue casual. Don’t make it look like you’re fishing for information — that’s a sure way to put people on edge.

If you’re struggling to come up with what questions to ask, just look to some of the concepts that you already want to know:

  1. What’s stopping people from converting?

  2. What are some pain points that people experience during the conversion process?

  3. What is your company’s core value proposition?

  4. What could your company be doing better?

  5. What would compel your customers to refer your company to a friend?

  6. What would it take for your customers to do business with your company again?

Get External Opinions

First impressions are extremely important. If you’re only surveying existing customers, you’ risk missing out on what’s outside your comfort zone. To keep your company’s perspectives fresh, it’s important to talk to people who have nothing to do with your brand.

When you only have a day for this exercise, it doesn’t make sense to go plan a focus group or full-fledged market research study. The solution isn’t to talk to random people on the street, either.

Rely on the power of technology to get you the answers that you need.

  1. UserTesting

    This platform provides user research results in under an hour. You can set up a test on your website with instructions for exercises that you’d like completed. UserTesting creates a recording of your screen, collects written answers, and records what respondents are thinking and feeling along the way.

    You can select panel participants by demographic data, geographic information, and experience level with the Internet.


    Clarity is a matchmaking platform that connects advice seekers with subject matter experts. The majority of Clarity users are entrepreneurs seeking business advice, but you can also use the platform to recruit perspectives from your target customer base.

    Clarity is also helpful if you are looking to gather feedback from a niche B2B audience, something that may be more challenging on a platform like UserTesting. When you set up calls on Clarity, you’ll pay by the minute.

    If Clarity or UserTesting aren’t viable options for you, just reach out to your friends, family, and professional network. Offer up a gift card for peoples’ time.

    Whatever your approach, be sure to be extremely appreciative, and take your conversations extremely seriously. You never know what piece of advice will help you fix an otherwise mission-critical conversion element.

Today’s Homework

  1. Create a list of questions that are important to your company’s conversion optimization strategy. These should be actionable points of research that translate directly influence your company’s bottom line.

  2. Start talking to potential customers, existing customers, and people unrelated to your brand. Choose a mix of approaches that are outlined for Day 3. In case you forgot, here they are again:

    • Collection and analysis of insights from live chat transcripts and knowledge center forums

    • Listening to customer service and sales calls

    • Calling up existing customers and prospects

    • Personally participating in customer service calls

    • Running a UserTest

    • Calling up an expert via

    • Talking to friends and family

Day 4: Fix Low-Hanging Problems

Yesterday, we asked you to run qualitative research studies on your site. Fancy jargon aside, the goal was to start learning from your target customers — to help you diagnose usability challenges that you would otherwise overlook.

What you’ll probably be surprised to see is how much low-hanging fruit opportunities exist for improving conversions. In many situations, it’s simple stuff that bogs down your customers.

Focus on the Low-Hanging Fruit

Find opportunities to make changes without the help of your development or IT teams. Here are some examples of opportunities that other website owners typically find:

  • Make your web copy shorter — take what you have, cut it in half, and cut it in half again.

  • Adjust your calls to action (CTAs) — make them easier to understand and more prominent on your webpage for users to see.

  • Make your web forms shorter — a lengthy sign-up process can turn off your users from wanting to work with your company.

  • Clarify your company’s value proposition — the benefits of your product or service may be difficult to understand.

  • Boost the incentives — offer up a promotion or giveaway to encourage prospects to engage with your brand.

  • Create step-by-step instructions — find opportunities to guide your customers through the sales process.

  • Improve trust — ­ensure that you aren’t scaring your customers and prospects away; implement trust signals via social media details, membership stats, testimonials, or client logo.

Dig Deep Where User Psychology Meets ROI

The heart of conversion optimization isn’t math or science. It’s psychology. Your strategy needs to solve a specific user need. Start by identifying specific user needs and reverse engineer the solution to meet that goal.

Now comes the fun part — prioritizing your time. Your research may end up presenting a laundry list of needs to address. There are only so many hours in the day, and your marketing team (if you are lucky enough to have one) is likely limited on bandwidth.

The way to focus is to look for patterns. Rather than solving every target customer’s pain points 1:1, find opportunities to solve problems in batches. Then, align this list of pain points with your highest ROI opportunities.

Here are some example challenges that could happen on CrazyEgg:
(based on interviews with 14 people)

  • Ten prospective customers don’t understand the software

  • One person thinks the sign-up process is too long

  • Eight people want an easier way to subscribe to the blog

  • There is demand from 6 people for a different, complementary product

  • People (14 of them) were afraid to enter their credit card details

Out of this list, which do you think are the most immediate options to tackle? They are:

  1. The fact that prospects are confused by the software

  2. The fact that people are afraid to enter their credit card details

    • The blog isn’t the biggest priority inhibiting sales

    • Just one person was bogged down by the sign-up process; not enough to freak out

    • The demand for a complementary product is something that requires extensive time and dedication to address — outside the scope of a quick fix.

The sign-up form issue was a challenge that CrazyEgg recently experienced. Visitors didn’t want to include their credit card details to sign up for a free trial. CrazyEgg added an explainer to the company’s checkout page that visitors would not be charged for the trial.

CrazyEgg quantified the difference in performance, and found that the new page yielded a 116% increase in sign-ups.

Today’s Homework

  1. Clean up your website copy. Simplify the messaging as much as possible. Cut it down in half if you can, and get rid of paragraphs.

  2. Of the challenges that you discovered on Day 3, pick 2-3 that you can address right now. Make those changes on your website. If you can’t fit it into one day (and do it well), it’s probably too big an issue for the scope of this exercise.

Conversion optimization is a marathon, not a sprint. BUT, you need to eliminate the hurdles that will inevitably stop you from winning business. That’s exactly what you did today.


Emily Anderson