This has absolutely been my experience building prod LLM-based apps. It requires a change in mindset from ‘this passes all the tests’ to ‘this passes enough of the tests often enough and is valuable enough that we’ll work around the times it fails’. It also creates some weird dynamics in terms of SDLC because as Brian correctly points out creating an exciting prototype is child’s play and whereas for normal apps the prototype might represent 50% or more of the work required to ship with LLMs the prototype can do maybe 80% of what you want but the effort remaining to get it to 99% effectiveness is 100x what it took to create the POC. We’re all having to learn new ways of thinking about dev if we want to work productively in this space.
Product Leader | Building Great Products | Building Great Teams | Startups | AI/ML | Cyber Security
Musings on building a Generative AI product. LinkedIn has published an exceptional blog post on their experience of rolling out Generative AI and I love the honesty: “The explosion of generative AI made us pause and consider what was possible now that wasn’t a year ago. We tried many ideas which didn’t really click” and “building on top of generative AI wasn’t all smooth sailing, and we hit a wall in many places.” There's a crucial takeaway from a Product perspective: Early wins building LLM Products can be deceptive. Building the pipeline was the easier part (Routing and Retrieval) but Generation was harder: Generation: precision-oriented step that sieves through the noisy data retrieved, filters it and produces the final response. Generation followed the Pareto principle: “Getting it 80% was fast, but that last 20% took most of our work”. The theme of getting 80% of the way there quickly appears again, and this initial success can cause problems: “The team achieved 80% of the basic experience we were aiming to provide within the first month and then spent an additional four months attempting to surpass 95% completion of our full experience - as we worked diligently to refine, tweak and improve various aspects. We underestimated the challenge of detecting and mitigating hallucinations, as well as the rate at which quality scores improved—initially shooting up, then quickly plateauing. For product experiences that tolerate such a level of errors, building with generative AI is refreshingly straightforward. But it also creates unattainable expectations, the initial pace created a false sense of ‘almost there,’ which became discouraging as the rate of improvement slowed significantly for each subsequent 1% gain.” AI Product: Don’t underestimate the last 20% when forecasting timelines and setting expectations. #llms #productmanagement #aiapplication
Brian Greene
7mo
A joke I heard about software development is more true now than ever… The first 80% of the project takes 80% of the time. The remaining 20% takes the other 80% of the time. My friend Thomas Busch said this a long time ago with a perfectly straight face. Flawless delivery. Who knew it would be so useful with AI? 🤣
Co-founder at QuantumZero ► modern tools to monetize data at speed and scale ► unlock data value while enforcing rigorous controls, regardless of location ► Enjoy ice cream 🍦
7moBrian Greene completely agree. However, we need to think about LLMs differently than regular software applications! We have been taught that computers don’t make errors, because they predictably and accurately execute on what they are told … LLMs by definition are probabilistic and hence a degree of “error” or deviation from the expected output is inherent in the architecture. You can RAG till the Moo 🐮 comes home, but that deviation will never go away. The users of LLMs need to be comfortable with a degree of deviation/error - and decide whether that is good enough or improvement needs more investment.