Relevance tuning sprint methodology for improving search relevance never goes out of style 1. Pick 10 queries that show a use-case you want to improve 2. Hand label - 25-50 labels per query, distributed across ratings from like 1-5 so you know what's good/bad 3. Tune+measure - devs dork around with search, measuring as they go, trying to shuffle good stuff to the top 4. GOTO 1 (remeasure previous sprints, make sure nothing breaks) (Quepid is a great tool for this)
Search relevance folks spend their days building forests. It’s rare that they get to examine individual trees. And individual information needs don’t always resonate when your goal is increasing net customer happiness. But Bizz has an itch (often someone in product, bizdev, customer success) that something is wrong. Product, scrum, principals etc.. have batted away their complaints for months , for lack of evidence. Now Bizz come back bearing a couple dozen queries exhibiting the problem and are able to bend Tekk’s ear. The night before sprint-planning and now Tekk has the itch too! Can’t sleep and pulls a random sample of queries (100-200 if manually labeling), more if they can write a script or prompt an LLM to identify the issue. An hour or two of work and they have built additional “napkin-evidence” but with (a little) higher predictive power. Now there’s a case to slip an investigative spike into the sprint. But just for Tekk and Bizz. The rest of the team has forests to tend.
Generative AI Solutions Architect | RAG Specialist | AI Search Practitioner | Creative Technologist | Builder at Heart | Award-Winning Graphic Designer | Seeker with Abundance of Curiosity
19hInteresting insight Doug Turnbull does this typically occur on a 2 week sprint cycle or 3 week from your experience?