How do you communicate and report conversion lift to stakeholders?

Choosing the right page to test usually rests on one single criteria: 1. Is the page a place you want people to take a specific action? If the answer to that is yes, it's probably worth testing. For smaller websites, this might mean you're testing either your home page or a specific landing page. However, for websites with more pages, it's going to help you HUGELY if you use the 80/20 principle to determine which pages to test first. That is: Which pages offer the highest ROI? Or Which pages get the most traffic? Or Which pages had the most recent conversions (I usually take the last 120 days into account). With your target page identified, now you can tap into data like heatmaps, click maps, or user feedback to create a hypothesis.

If you're new to conversion rate optimisation a good place to start is to make sure that you have Google Analytics and Google Tag Manager setup (both free to use). As you progress there are lots of other tools to check out, like Microsoft Clarity and Hotjar to name a few. You'll also want to check that you're tracking key conversion points like different forms and your chatbot in analytics. Having this set up correctly will mean you can see exactly which forms are working for you and which need some attention.

There are initial considerations for your variable selection; what you want to test and what you can test aren't always possible. You need to calculate your total conversions, conversion rate, and the length of the test required in order to detect a statistically significant effect. Prioritize based on speed and business impact. From a marketers perspective, the cause and effect of losing a significant amount of IDFAs is imperfect measurement. Thus, more often than not the variable you're testing is the "true impact of marketing". Your baseline is your control, what would happen if you did nothing, and your variable is the specific channel or some combination of your marketing mix.

Also, define your signals. These will be a clear indicator to show you are heading in the right or wrong direction. I always say that if you can't show results - you need at least to show insights. This to a business is key!

Choosing variables involves weighing desires against possibilities, considering conversion numbers, rates, and necessary test lengths for significant results. Marketers often grapple with the challenge of accurately measuring the impact of marketing strategies, especially with the loss of IDFAs affecting precision. Your baseline sets the stage for understanding the pure effect of your actions, while your variables explore the influence of specific channels or marketing mix combinations.

Simplify Metrics: Present data in clear terms, like percentage increase in conversions or ROI improvement. Use Visuals: Incorporate graphs or dashboards to showcase trends and results. Highlight Key Wins: Emphasize the specific actions that led to the lift. Provide Context: Compare against past performance or industry benchmarks for relevance. Outline Next Steps: Suggest optimizations or follow-up campaigns based on the results.

Does that button color turn people OFF buying? Do people understand your headline? Is your form too simple? You can answer these types of questions by running A/B or even A/B/C tests. Understanding the answer helps you drive users further up your sales funnel. Once you've identified the part of your page that needs testing, set up your test using any of the tools set up just for this purpose. Google Optimize used to be a good place to start for people on a budget (I used that when I first started and had no budget for testing), though its features were limited, and Google has now sunsetted it. So, I recommend tools like VWO—which I now use every day—and see if they work for you from a pricing and ease-of-use perspective. Happy testing

To successfully run an A/B test you need a good volume of users, as basically half of your users will be shown version A, and half will be shown version B. Therefore you need both these user groups to be big enough to give you an accurate view within the given timeframe. If you don't have enough users, or don't have the time to wait to reach a viable number of users, don't be afraid of making changes and tracking the difference from the previous period. This way you can move more quickly even with a small audience, the caveat as that you will need to track your changes and keep an eye on the conversion rate.

If you don't have much traffic you can still run successful AB tests. Just think that by enabling yourself into experimentation you're already doing more than many of your competitors. Focus on qualitative testing (i.e. user testing/interviews). This will give you enough information to improve your conversions by at least 50% within a few weeks of data.

Discover what works best on your site - be it button colour, headline clarity, or form complexity - through A/B testing. This clarity propels users up the sales funnel. Now, I would probably recommend VWO for its comprehensive features and user-friendliness. Dive into testing and see the difference for yourself.

When choosing between A/B and MVT (Multivariate Testing), it’s key to understand your objective and data capacity. A/B tests are ideal when comparing two variations, like testing a new homepage design against the current one, where you need clear, singular insights. For instance, testing two different CTA buttons across a large audience. MVT, on the other hand, should be used when you’re dealing with multiple variables. For example, if you want to test headline, image, and CTA combinations simultaneously. This is powerful for understanding how elements interact, but it requires more traffic (statistical power) and complex data analysis to pinpoint which variations drive the best performance.

In addition to calculating the lift, you should have your Guardrail metrics defined. Is it "good" to lower your Cost Per User if Average Revenue Per User also drops? Is an increase in conversion rate "good" if it slows down your app considerably? And how much is okay? Choose the vital business metrics you monitor with the cohort and ensure you fully understand all implications of the variant.

Besides calculating lift, set your guardrail metrics. Assess if a lower Cost Per User is beneficial when Average Revenue Per User drops, or if a higher conversion rate is worth a slower app. Determine key metrics and understand all variant implications.

If the thought of an equation scares you don't be put off! Google Analytics and other leading tools will do the calculation for you, as long as the conversion point is set up and tracking correctly.

Defining the winner of an A/B test starts with good Exploratory Data Analysis (EDA). Begin with boxplots to spot outliers and visualize the distribution, then use scatter plots to identify trends. Once you have a feel for the data, create a histogram to understand the distribution type. If the data is normally distributed, you can apply a t-test (Student’s t-test) to determine significance. However, if the distribution is not normal, use a non-parametric test like Mann-Whitney. By tailoring your analysis to the data’s characteristics, you ensure your conclusions are statistically sound and reliable.

As much as possible, tie results to the company's larger goals. A test isn't valuable unless it's important to the business in some way. While it's true that some tests are harder to measure the impact of, do your best, state all the assumptions you must, but above all else - be useful to the business. Use the same verbiage. Get skin in the game.

It's difficult to define success without a relative understanding of what the lift means. What's the business impact? An example may include equating what the total conversions means in terms of revenue for the business. Equally important to get across is that this as a benchmark. Conversion lift is not static over time. It must be repeatedly tested to detect changes.

Align test results with the company's broader goals to ensure relevance. While measuring some tests' impact may be challenging, clarify assumptions and strive to be valuable to the business. Use common language and actively engage to show your commitment.

Stakeholders are mainly interested in results and less so the steps taken to achieve them. When presenting, start with the net performance increase from your testing and then work back to a summary of why the winning test version performed better.

The most effective way to report to stakeholders is in money. For example: - We achieved the same metrics with a lower budget, so we saved the company $$$ - We brought more customers in with this ad, we know it was this ad as we can trace it. We brought $$$ into the business. - Our efforts take less time, so we saved $$$ in company hours and improved efficiency by %

When communicating conversion lift to stakeholders, it’s essential to approach it with empathy and clarity. Understand that not everyone speaks the technical language of data. Start by building trust—translate your findings into business impact. For instance, rather than focusing on raw lift percentages, connect those numbers to revenue, customer satisfaction, or efficiency improvements. Visualize the data using intuitive charts and dashboards in tools like Looker, but also humanize the story. Technical details, when communicated well, can have a transformative effect, helping stakeholders see beyond the data to the true potential of optimization efforts.