Wubbleyou’s Post

“Can we trust AI which gives us a different answer every time?” It’s a great question – if an accounting report gave us a different number for last month’s revenue every time we asked, we would not trust the data. But, there is a difference between true facts in a statement and the statement itself. If I asked someone in my team our revenue last month, they might say: “£X, with a net profit of £Y”, or “Revenue was £X, and we were Z% over budget on compute, but under budget on travel”, or simply “£X” So, the question becomes, “would we accept AI that presents the same facts in a different way?” We accept it in people. As long as… They actually present the facts correctly. However, LLMs are known to hallucinate. The trouble with this is, LLMs are a black box – you can’t just open them up and see what caused it to respond like this – they are very fancy prediction engines always trying to predict the next best word to say. So the final question might be, would we accept AI that presents the same facts in a different way every time you asked, as long as the core truths presented are validated as correct before they’re presented? Yes, this seems sensible to me. So how do we do that? LLMs can be tuned to be more or less ‘deterministic’ via a temperature setting, where deterministic means asking the same question more than once will create an identical response every time you ask, creating the more reliable predictability, we can combine that with instruction on how to present a response, to get more predictable responses. We can also review the 'facts' in the response, and sense check these before the LLM is allowed to use them. This could be conventional software or more AI. Where LLMs are used to create actionable insight, it makes sense to: ✅ Introduce a monitor, to check key facts as correct/incorrect before they are presented, then once these facts are validated, present these back to the LLM for presentation ✅ Be very specific with the expected structure of the response, and configure to be more deterministic ✅ Be specific with prompting to only use the facts presented to create output Great discussion with Dynamo North East, expertly hosted by David Dunn, with insight from Deborah Hardoon. Good to also hear from Sarah Glancey, Al (Aleasser) Alzein, Esther Gillespie, Michael Stirrup, Jon Leighton, Paula Harrison, Jon Saunders & plenty more. #Newcastle #Scaleups #AI