Daniela Carzaniga

Stop me if you heard it before: You release a new feature and during the test, it delivers a great uplift. However, that is nowhere to be seen a few weeks after the final release. What gives? How to prevent it? Being a Product Manager with almost a decade of experience, while also creating relevant content, makes it that I don't get to learn a lot of new things about my chosen career. That being said, a few months ago something completely blew my mind and changed how I look at the A/B tests. It was just a throw-away sentence by a colleague: "Yeah, but the A/B test uplift might only be due to the novelty of the new feature to the users. We don't know if it will contribute to our goals in the long run." My whole career passed in front of my eyes. So many A/B tests showed a massive spike in metrics that were nowhere to be found shortly after the full release. "Maybe something happened that negatively impacted the metrics and the new release compensated" - we hypothesized. I was so blind and ignorant. Of course: users will notice something new you highlight. users will flock to a new feature out of curiosity. the metrics will jump due to playing around with the new stuff. once the novelty wears off, the actual impact will be visible. it will be smaller than the initial numbers. the numbers proclaimed are wrong. I was a fool to believe in them. I was impatient, I was ignorant. I was wrong. I always believed that statistical significance is the fool-proof mechanism to determine whether the A/B test is stellar or wonky. But it's simply not enough on a test that takes around 2 weeks, as you need to rush other tests for different features. And yet, the determined impact of those is as imaginary and fake, as the impact estimates in the initial Product Development Document (PRD) or User Story. So are all A/B tests wrong? Is there a way to know the truth? Yes, fortunately, yes. There are two ways: 1) Run the test until the novelty factor is no longer there. How long does this take? Your guess is as good as mine. Plus, as mentioned, nobody has time to wait. There is a better way: 2) Extract the "New Users" cohort in your A/B test. That's right! If everything is new for everyone, then the impact is real. If the metric uplift is similar in the general and new users cohort alike (and you have the statistical significance) only then you can safely proclaim a successful experiment. The best part? Amplitude, the partner of this post, agrees with me and they have decided to bring this approach to their new, A/B test template dashboard! This way you can easily incorporate this post's learning to your work! So, if you want to make sure you are drawing the right conclusions, check out the template in the comments section :) So, was this as new and enlightening as it was for me? What was your most surprising A/B test? Sound off in the comments! #productmanagement #productmanager #abtest  

209

14 commenti

📚 𝗣𝗿𝗼𝗱𝘂𝗰𝘁 𝗕𝗼𝗼𝗸 𝗥𝗲𝘃𝗶𝗲𝘄 𝗦𝗵𝗼𝘄 #1 📚 I can’t always just read novels, so in the last weeks, I’ve also been diving into some great reads about product discovery, UX, and people management. Have a look at my recommendations: 1. 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗗𝗶𝘀𝗰𝗼𝘃𝗲𝗿𝘆 𝗛𝗮𝗯𝗶𝘁𝘀 by Teresa Torres Score: ⭐⭐⭐⭐⭐ (5/5) 𝗧𝗟;𝗗𝗥: 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘁𝗵𝗲 𝘂𝘀𝗲𝗿𝘀 𝗮𝗹𝗹 𝘁𝗵𝗲 𝘁𝗶𝗺𝗲. 𝗕𝘂𝘆 𝘁𝗵𝗶𝘀 𝗯𝗼𝗼𝗸. • 𝗦𝘂𝗺𝗺𝗮𝗿𝘆: A guide to implementing continuous discovery practices in your product development process. What does that actually mean? It provides a hands-on approach to uncovering opportunities during your discovery work, which will be key for your development and short-term strategy. • 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆𝘀: This book is a masterclass in actionable advice. The examples are highly relevant and can be easily implemented in your daily work. The opportunity solution tree is a fantastic tool to use—one colleague even called it “the tree game.” I only wish there were more examples in the book. The Netflix focus is interesting but doesn’t always work for me. • 𝗠𝗶𝗻𝗶𝗺𝗮𝗹 𝗙𝗹𝘂𝗳𝗳: Perfect mix of anecdotes and takeaways. 2. 𝗨𝗫 𝗳𝗼𝗿 𝗟𝗲𝗮𝗻 𝗦𝘁𝗮𝗿𝘁𝘂𝗽𝘀 by Laura Klein Score: ⭐⭐⭐⭐☆ (4/5) 𝗧𝗟;𝗗𝗥: 𝗬𝗼𝘂𝗿 𝗵𝗮𝗻𝗱𝘆 𝘁𝗼𝗼𝗹𝗯𝗼𝘅 𝗳𝗼𝗿 𝗟𝗲𝗮𝗻 𝗨𝗫. • 𝗦𝘂𝗺𝗺𝗮𝗿𝘆: This book covers the trifecta of UX: research, validation, and iteration. Laura Klein tackles all kinds of topics (MVPs, A/B tests, interviews, etc.) through numerous examples. It may not be ideal for seasoned UX designers or Product Managers with a knack for UX because much of it might already be known to them. • 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆𝘀: This book is a toolbox for UX research and validation. It offers a thorough overview of all kinds of methods. For deep dives into specific topics, you might need additional resources—A/B testing alone could fill an entire book. • 𝗠𝗶𝗻𝗶𝗺𝗮𝗹 𝗙𝗹𝘂𝗳𝗳: Very little fluff, sticking mostly to relevant UX advice you can actually use. And you better like sarcasm. 3. 𝗖𝗼𝗺𝗽𝗲𝘁𝗶𝗻𝗴 𝗔𝗴𝗮𝗶𝗻𝘀𝘁 𝗟𝘂𝗰𝗸 by Clayton Christensen Score: ⭐⭐⭐☆☆ (3/5) 𝗧𝗟;𝗗𝗥: 𝗙𝗿𝗼𝗺 𝗖𝗼𝗽𝗲𝗿𝗻𝗶𝗰𝘂𝘀 & 𝗠𝗶𝗹𝗸𝘀𝗵𝗮𝗸𝗲𝘀. • 𝗦𝘂𝗺𝗺𝗮𝗿𝘆: The “Jobs to be Done” (JTBD) theory suggests that customers hire products to perform specific tasks. Christensen popularized this framework. This book helps you find the JTBD for your product and innovate in your business accordingly. • 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆𝘀: JTBD is legendary in product and marketing realms. The book’s stories, including the famous milkshake story, illustrate the concept well. Ultimately, it adds another tool to your design thinking arsenal. • 𝗠𝗶𝗻𝗶𝗺𝗮𝗹 𝗙𝗹𝘂𝗳𝗳: Clayton Christensen’s academic background is evident. The JTBD concept could be conveyed in a much shorter format. There’s quite a bit of filler. What are your favorite PM books?

10

Sometimes your product’s users just want to be “hugged”! Not literally though 🙂 Interesting talk by Charles Duhigg on the different types of conversations people have or at least try to have. In product management, we are all too familiar with having a hard time understanding some users' feedback. In my view, it’s often due to the user and the PM not having the same type of conversation: the user wants to have an emotional conversation about his pain points while the PM is having a practical one about their global importance, which solutions could solve them and how he could allocate resources to implement those. Charles’ example about a patient requesting unnecessary surgery reminds me of the all too common interviews where the user insists the PM implement the solutions he “envisioned”. It’s often due to a feeling of not being heard and this frustration can be so powerful that it can result in very rational people requesting things that are detrimental to their own well-being like poorly designed features or even worse, unnecessary surgery! The solution is to ask “deep questions” that will help you figure out what type of conversation the user is trying to have and by doing so identify the underlying problem you need to address. So next time you’re in a user interview and someone complains about an extra-click they have to perform, try finding out how this makes them feel and what that represents in their lives rather than just creating a new entry in the complaints list. Asking these “deep questions” will also allow you to create a long-lasting connection with your users that will help you go beyond their words and understand who they really are, what really matters to them and how you can best improve their lives with your product.

1

The problem with unchecked and unintentional complexity is that it gradually narrows the 'solution space' — the range of future improvements, adjustments, or innovations that can be realistically pursued. Picture the solution space as a playground where designers and developers have room to explore new ideas and tackle emerging challenges. As complexity accumulates without careful management, walls begin to form in this playground, limiting the freedom to experiment and improve. Changes become riskier, and small tweaks can lead to a cascade of unintended consequences. The more convoluted the system, the fewer opportunities there are to introduce meaningful, impactful updates without further complicating the product or alienating users. Read more in my latest blog post linked below. #ProductManagement #Complexity #Innovation #TechLeadership #DigitalTransformation #Scalability #TechDebt #IntentionalComplexity

3

Managing a product is not an easy job. But imagine that you multiply all challenges a few times and now you have a few PMs - each of them having their products, their own market dynamics with specific issues, and their own teams so you now have engineers complaining times x :P. This can cause a real headache, I've been there, I know. Therefore, the 𝗖𝗲𝗿𝘁𝗶𝗳𝗶𝗲𝗱 𝗣𝗿𝗼𝗱𝘂𝗰𝘁 𝗣𝗼𝗿𝘁𝗳𝗼𝗹𝗶𝗼 𝗠𝗮𝗻𝗮𝗴𝗲𝗿 program has been created - to help Heads of Products and CPOs set priorities in the multi-product environment or manage complex multi-module platforms. It is also good for PMs who would like to step up and take up more responsibilities. The 2nd edition is behind us https://lnkd.in/des-6_xS The 3rd edition is on 𝟮𝟲-𝟮𝟳 𝗦𝗲𝗽𝘁𝗲𝗺𝗯𝗲𝗿 Link https://lnkd.in/dC2t6S3R Rokanan Group | Product Trainings #portfolio #productstrategy

7

2 commenti

This new article is for PMs and POs to demistify couple of concepts related to product management like priorisation, strategy, risk management and research. Often thought of very complex things we are afraid to touch, while there are ways to fear not, and get started. Hope you'll find those helpful. #productmanagement #prioritization #frameworks #research #riskmanagement #backlog #strategy

7

1 commento

Hey Product Managers: To 𝘀𝘂𝗽𝗲𝗿𝗰𝗵𝗮𝗿𝗴𝗲 𝘆𝗼𝘂𝗿 𝗶𝗺𝗽𝗮𝗰𝘁 consider learning Systems Thinking! 𝗧𝗵𝗲 𝗯𝗲𝗻𝗲𝗳𝗶𝘁𝘀: • Become more influential. • Make better product decisions and generate stronger ideas. • Develop a perspective that leaders and executives value. Short on time? Start with these 𝗾𝘂𝗶𝗰𝗸, 𝗶𝗺𝗽𝗮𝗰𝘁𝗳𝘂𝗹 𝘃𝗶𝗱𝗲𝗼𝘀: 1. Russ Ackoff (𝗕𝗼𝗻𝘂𝘀: 𝗛𝗲'𝘀 𝗵𝗶𝗹𝗮𝗿𝗶𝗼𝘂𝘀!)   • "Systems Thinking" (12 min): https://t.ly/bVEEN 2. Andrew Yu (Product Consultant, ex-LinkedIn)   • "The Product Ecosystem" (5 min, then next link): https://t.ly/v9h5f   • "Airbnb Case Study" (4 min): https://t.ly/f1Dui Shoutout to exceptional systems thinkers in Product that I've had the pleasure to work with: Matthew Nelson Jenna Eubank A few that I don't know (yet) but learn from every day: Shreyas Doshi Lenny Rachitsky Peter Yang Elizabeth Stone ----- #ProductManagement #SystemsThinking #ProfessionalDevelopment #Leadership

17

2 commenti

Ever read "Inspired" by Marty Cagan? It's full of great product management ideas. I am going to share the key concepts/frameworks he discussed in short and easy posts with real-world examples. Whether you are new or experienced, there's something here for you. Let's learn how to create products people actually want to use! Through this series, I will finally do what I love but always put off - Reading! Stay tuned for our first post. It'll be fun! 🚀 #inspired #productmanagement #productbooks #MartyCagan #productlearning

15

Did you know every Product Manager is doomed to fail? You need to be prepared! Here is how to deal with 10 potential failures:: 1) Too many stakeholders will want something ASAP This is how our position simply works. Accept you can’t do everything, focus on selected ones, and collect data to show why you prioritised what you did if you get questioned. 2) The new release will be a disaster Tracking might not work, new bugs may appear, the feature was not correctly set in production, and more. These things happen. Make sure you or your team always have a roll-out plan, good monitoring, and learn from your mistakes 3) You will get sick/need to take some time off Take it. You are not good sick, depressed, or exhausted. Rest is fine. Others will pick up where you left (but you need to train them to be ready!) 4) Your team will get sick or need to take time off All your carefully crafted estimations are going to waste. Don’t bother with those. Just make sure you have a feasible goal at the beginning of the sprint. One that can be achieved, even if one person gets sick. 5) Your Product will stop working Learn every time this happens and set up alerts/monitoring to catch it quicker next time. There is no point in beating yourself that it could have easily been prevented it. 6) Important metrics will go down Breathe. Count to 10! Pet a dog, cat, horse, whatever. There are a lot of things you can do to see what happened. I have a full post on this coming up :) 7) Key players will leave your team. People will come and go. It’s the circle of hiring. Support your leaving colleagues, wish them well, and make sure it’s not your fault. Adjust your plans to exclude the missing team member and communicate that well with stakeholders. 8) Important aspects of the work won’t be ready Sorry, it happened to you. Just make sure you have a good epic before the release checklist. Make sure the team fulfills all the acceptance criteria. Better to be late than subpar! 9) Your product will get (potentially) hacked You can't control bad actors. Comply with the law and fix the vulnerability that opened you for damage. Especially if the hack is only potential. Better to be late with new features than explain to users and authorities why your product’s user list with credit card number is now auctioned on eBay… 10) You will be sick of it all — It’s a lot to take in. Nothing a single human will ever manage in every aspect. Don't make rush judgements. If needed, take time off to figure stuff out. You are likely to suffer from the same at other companies and products! Make sure you are making your best career choice. How do you feel about those? Will they help you? What is your biggest failure story and how did you recover? Sound off in the comments! #productmanagement #productmanager #agile P.S. To be ready for all PM challenges, check out my courses on www. drbartpm. com :) Use the code “linkedinfollower” on checkout to receive a 30% discount on all of them :)

628

28 commenti

Did you know every Product Manager is doomed to fail? You need to be prepared! Here is how to deal with 10 potential failures:: 1) Too many stakeholders will want something ASAP This is how our position simply works. Accept you can’t do everything, focus on selected ones, and collect data to show why you prioritized what you did if you get questioned. 2) The new release will be a disaster Tracking might not work, new bugs may appear, the feature was not correctly set in production, and more. Breathe. These things happen. Make sure you or your team always have a roll-out plan, good monitoring, and learn from your mistakes 3) You will get sick/need to take some time off Take it. You are not good sick, depressed, or exhausted. Rest is fine. Others will pick up where you left off (but you need to train them to be ready!) 4) Your team will get sick or need to take time off All your carefully crafted estimations of scope and effort required are going to waste. Don’t bother with those. Just make sure you have a feasible goal at the beginning of the sprint. One that can be achieved, even if one person gets sick. 5) Your Product will stop working Learn every time this happens and set up alerts/monitoring to catch it quicker next time. There is no point in beating yourself that it could have easily been prevented it. 6) Important metrics will go down Breathe. Count to 10! Pet a dog, cat, horse, whatever. There are a lot of things you can do to see what happened. I have a full post on this coming up :) 7) Key players will leave your team. People will come and go. It’s the circle of hiring. Support your colleagues who are leaving, wish them well, and make sure it’s not your fault. Adjust your plans to exclude the missing team member and communicate that well with stakeholders. 8) Important aspects of the work won’t be ready Sorry, it happened to you. Just make sure you have a good epic before the release checklist. Make sure the team fulfills all the acceptance criteria. Better to be late than subpar! 9) Your product will get (potentially) hacked You can't control bad actors. Comply with the law and fix the vulnerability that opened you for damage. Especially if the hack is only potential. Better to be late with new features than explain to users and authorities why your product’s user list with credit card number is now auctioned on eBay… 10) You will be sick of it all — It’s a lot to take in. Nothing a single human will ever manage in every aspect. Don't make rush judgments. If needed, take time off to figure stuff out. You are likely to suffer from the same at other companies and products! Make sure you are making your best career choice. How do you feel about those? Will they help you? What is your biggest failure story and how did you recover? Sound off in the comments! #productmanager #productmanagement #mentalhealth   P.S. To be ready for all PM challenges, check out my courses on www. drbartpm. com :)

481

23 commenti

There are only 2 true tests of product-market fit. Nothing else matters. The tests: → Will customers pay for it? → If you took it away, will they clamor for it back? That's pretty much it. So, to get to PMF faster, simply ask: → What would you change that would make you want to buy it? → What would you do if you didn't have this? #productmarketfit *** ♻️ Like this? Please repost. ➡️ Follow me for daily executive coaching.

16

15 commenti

Good Read

2

PRODUCTHEAD: Deconstructing Duolingo’s growth » Dissect your monolithic metrics to drive growth in more easily influenced segments » Don’t kill the golden goose of growth by oversaturating a channel » If growth is your goal, have a sense of urgency Subscribe at producthead.news #ProductManagement #prodmgmt

1

Why do we always feel the need to add more features? As product builders, it’s tempting to cater to everyone and keep our options open. This usually stems from a poorly defined Ideal Customer Profile, making product decisions even harder or, in some cases, lazier. Find the one use case that’s a massive problem for your customers. Go deep, solve it, and make it insanely simple. Once you nail that, it becomes your foundation to expand into other valuable areas. Focusing on simplicity not only improves the user experience but also accelerates product development and reduces maintenance costs. It creates a clear value proposition, making it easier to communicate and market your product. When your product solves a major problem effortlessly, it stands out in a crowded market and helps you rise above the noise. Simplifying is powerful.

26

Couldn't make it to Marty Cagan's talk in Berlin? Or maybe you were there but too focused on his insights about product transformation to write things down? No worries - I've got your back! 😉 I've captured all the key learnings in my latest article here:👉https://lnkd.in/eCeyeRE2

9

The concept of an MVP (Minimum Viable Product) has drifted away from its original intent. Today, it’s more about delivering a technically feasible product rather than a truly usable one. MVPs are often treated as a measure of output—how fast a feature can be built—rather than the outcome it creates for users. This is where many technical product teams go wrong. They focus more on the complexity of the build and delivering features than on solving real customer problems. Frameworks like the 2x2 prioritization matrix are useful for managing engineering resources and iteration cycles, but they don’t necessarily guide teams toward building the right product. The truth is, creating a product that truly meets users’ needs might be complex, and that’s okay. The problem is that many teams are more influenced by technical difficulty than by the outcome they're trying to achieve. This often happens because teams lack a deep understanding of the problem they’re solving and what their users truly need. It’s hard to build the right product when you’re not talking to customers regularly. Let's focus on three things when aiming for the right product outcome: How broad is the outcome? How narrow is the outcome? Is the outcome the right fit to solve the problem? This leads to what I call the Minimum Viable Outcome (MVO). Unlike technical MVPs, which prioritize shipping features, the MVO is all about delivering the right outcome that solves the customer’s problem. With the right feedback, constant experimentation, and incremental iteration, we can move towards the North Star—building a product that truly fits the users’ needs.

59

4 commenti

[Re-post in English] Speculative Future: Will Product Manager Interviews Ask This Question? "How would you design a product with AI capabilities?" Here’s my current framework for tackling this: 1️⃣ Define product goals and constraints. 2️⃣ Identify target users. 3️⃣ Uncover user needs and pain points. 4️⃣ Explore potential solutions. 5️⃣ Prioritize features. 6️⃣ Establish success metrics. 7️⃣ Plan next steps after hypothesis validation. Sounds like a typical Product Design question, right? But the true complexity lies in the details. Here are the critical nuances to consider—still a work in progress, and I’d love your feedback! 💰 Budget constraints: AI isn’t cheap—what’s the cost ceiling? 🎯 Accuracy expectations: What level of precision is required to deliver value? 👙 Non-AI alternatives: Can the problem be solved without AI? Avoid shiny tech traps. 🗺️ User experience impact: Does AI simplify or complicate the UX? 🤼 Team collaboration: What roles are needed to support development? 🧹 Data preparation: How do we source, clean, and label training data? 🆚 Model selection: Which model fits the problem? What are the trade-offs? 🤯 Post-launch challenges: If results fall short, is it the feature or the AI? ♻️ Continuous optimization: How do we hypothesize and validate for improvements? 📏 Privacy concerns: How do we ensure compliance when retraining on user data? Still think AI projects are sleek and sexy? 😉 Let me know your thoughts!

1

2 commenti

Biggest learnings from 10 months as a Fractional CPO and Product Coach (number 4 will surprise you). 1. User research is seen as an overhead (cost & time), rather than an investment. This is fascinating, but not surprising. Most companies envision a desperate PM, clutching the collar of an unsuspecting user, pleading ‘please, please, tell me what to build, I beg of you. Engineers are going to be free tomorrow, hungry for work and I have nothing.’ On the contrary, it’s an investment with a substantial yield and should be done weekly. Proper training is advised; just ‘doing it’ is not enough, there’s a meticulous mastery to candidate recruitment and bias minimisation. 2. Data is limited, highly fragmented, or absent. Making decisions without understanding platform usage is like trying to find your way in a city you’ve never visited with your eyes closed and no GPS. Utter mayhem. Data is power. Make sure you’re tracking and adding all metrics into one repository. If client data is in one place, and usage in another with no way of connecting them, it’s useless for decision-making. 3. Companies tend to think in solutions, not problems. Best founders and PMs are in love with the PROBLEM they’re solving, not the solution. The first question I ask when a new ‘solution’ is presented to me is ‘what problem does it solve?’. It requires a mindset shift to start thinking this way. The key you’re showing me is super shiny and beautiful, but what door does it unlock, and more importantly, why do we want to unlock that door? 4. Product Vision does not exist. Sing it with me: ‘To a ship without a destination, no wind is favourable.’ (Young Seneca). Zackly. How can we decide if this or that problem is more important to solve? If we don’t understand or have a compelling vision statement, and compelling it must be, then there’s no way of knowing if it’s taking us closer to the future we’re creating. So, set a clear Vision. Understand what problems stand in your way to get to that vision. Quantify and deeply scope those problems (data and user interviews), and you’re already ahead of the curve. Hope this was helpful. Any surprises?

13