Product Manager - CRO & Growth | Organizer of the first CRO Meetup in Berlin (CROdashians) Driving Sales Growth | Optimizing Digital Experiences | Experimentation Enthusiast
𝐂𝐑𝐎 𝐭𝐢𝐩𝐬 𝐈 𝐰𝐢𝐬𝐡 𝐈 𝐤𝐧𝐞𝐰 𝐚𝐭 𝐭𝐡𝐞 𝐬𝐭𝐚𝐫𝐭 𝐨𝐟 𝐦𝐲 𝐜𝐚𝐫𝐞𝐞𝐫 𝐓𝐢𝐩 #1: Don’t Call It “Significant” after 2 days! Maybe some of you know the feeling: You run your first A/B test, check the data every hour, and after two days, the tool shows green numbers. You do a little happy dance inside yourself and excitedly tell stakeholders, “It’s significant!” But actually... it isn’t. 😅 Here’s what I learned: 𝐏𝐚𝐭𝐢𝐞𝐧𝐜𝐞 𝐢𝐬 𝐤𝐞𝐲 𝐢𝐧 𝐞𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧. Short test durations can be misleading because: 👉 Small sample sizes create random fluctuations. 👉 Early results are often skewed by novelty effects. 👉 External factors (like weekdays vs. weekends) can distort the data. Wait for the test to reach a solid sample size and communicate the timeline with stakeholders upfront so everyone knows when results will be reliable. #cro #experimentation
"Don't call it significant after two days" - I think that's what my wife said after our first two dates 🤣
Designer, Developer, and Optimizer
1dIt’s really, really important to understand how sample size, significance level, and MDE are interconnected and how they relate to determining test durration