Вячеслав Потапов’s Post

There's a dirty secret in the A/B testing industry that not enough people know about. Achieving statistical significance on a result with 5% lift is a huge win for most, but how much traffic do you need to detect it? If you want to detect a 5% relative improvement at a 95% significance level and 80% statistical power, you need ~120K users per variant, or ~240K users for a two-tailed A/B Test 🤯 You'll need dramatically more traffic if you want to detect a smaller improvement, or if you run more than two variants Such high traffic requirements make it challenging if not impossible for smaller companies to utilize A/B testing effectively without substantial traffic or unacceptably long experiment run-times. And don't get me started on sample ratio mismatch that could invalidate your results. Original credit to Ron Kohavi: https://lnkd.in/gcr3_a_N A deeper dive: https://lnkd.in/g8b5Ufcn Thanks Deborah O'Malley for the tip!

Day 8/30: The Power of A/B Testing Let’s talk about testing (and how I apparently love saying it). 🧪 1.Test your headlines – catch their attention or lose them at hello. 2.Run the test long enough – 3/4 days minimum, patience is a virtue! 3.Analyze the results – the data never lies. 4.Use the winner and, you guessed it, keep testing. 🌀 If you didn’t count how many times I said “testing,” go back and watch again. Spoiler: I’m obsessed. Testing makes perfect!

I need some beta testers who are cold calling in the US and have a big list I am building a pickup prediction algorithm within Scriptify. So far, we have successfully been able to eliminate 50% of the numbers that are wrong, dead or disconnected, gatekeepers etc. But this is getting users anywhere between 10-30% connect rates. Some users also report lower connect rates than this btw. Now I am building on top of the current algorithm that can identify the top 10% in your list who actually answer the phone. This new algorithm should also ideally eliminate the massive variability in the reported connect rates and bring some consistency to it. Feel free to shoot me a dm or comment below and send me your list. Requirements - It should have decently data quality (example - mobile numbers from seamless.ai etc.) Size should be around 500 What you get - I will validate your list and give you back 50-100 folks. You can call them and you should get connected every 1-4 dials. Now before anyone comments - Parallel dialing has its own place, and pickup prediction + auto-dialing has its own place. I am just looking out for what my users want.

𝗪𝗵𝗮𝘁 𝗔𝗿𝗲 𝗦𝗼𝗺𝗲 𝗖𝗼𝗺𝗺𝗼𝗻 𝗣𝗶𝘁𝗳𝗮𝗹𝗹𝘀 𝗜𝗻 𝗔/𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴? A/B testing is a powerful tool for optimizing performance and user experience, but it comes with several potential pitfalls that researchers and practitioners should be aware of. Here are some common pitfalls in A/B testing:

A/B tests often mislead you: Because they're likely to: - Give false positives - Only provide short-term results - Don't result in big improvements And they're costly: - Setting up the A/B test ($2,000 + 3-7 days) - Running the A/B test ($3,000 + 2-5 weeks) - Analyzing the results ($1,500 + 1-3 days) Around $6500/test for tiny improvements. Instead, consider qualitative research: - Live user sessions - Understand user problems - Uncover behavior drivers - Solve biggest issues - Reveal hidden insights Because big problems are more valuable. And once you solve the biggest problem: ↳ Use A/B tests to solve them better Don't incrementally improve a solution to the wrong problem Qualitative before quantitative. P.S. Do you A/B test?

Did you know most AB testing tools don't compute revenue confidence? If you're using AB testing and focusing on revenue as your main metric, you might be making a big mistake. Never rely on an AB testing tool's statistics engine for revenue metrics. Here's why: Most tools only show confidence based on conversions, not actual revenue. They often make it seem like they can compute revenue confidence, but they can't. To calculate revenue confidence, you need to consider the Mean and Standard Deviation (STDEV). If this is new to you, some of your AB test results might be completely wrong. Not sure if you're doing it right? Hit me up or comment "stat check please." I'm offering free consultations and explanations on a first-come, first-served basis!

Alpha testing vs.Betatesting Alpha and beta tests are both types of acceptance tests. During alpha testing, data is not real and typically, the data set is very small in order to make debugging and root cause analysis easer. Beta test participants are potential customers who have agreed to test a possibly unstable application. The users create their own data sets and the test focus changes to usability and evaluating real-life performance with multiple users using their own hardware. #Alpha testing #Beta testing

The hidden complexity of A/B testing. 🧪 By Ruby Stoddart “While model performance metrics are valuable, nothing speaks louder than a successful A/B test.” But real-world testing isn't always straightforward. To learn how to… → Design truly independent tests → Use cluster-based randomization → Implement sequential testing correctly …see our complete guide 👉 https://bit.ly/4dkROmF #DataScience

🚀 Excited about A/B testing? Here’s an example of how I can help. I was given the task of figuring out which page is more effective based on the A/B testing results. To tackle this, I calculated the average time users spent on each page and visualized it with a bar chart. A small task, but one image is worth a thousand rows.  Dataset: Landing Page A/B Testing Dataset https://lnkd.in/ekqVn2mK