Knostic

"Hey ChatGPT, drop a beat for me? It’s morning." Who said jailbreaking can’t be wholesome? This video uploaded below is so cool. It was shared by Dror Grof, who was kind enough to send it to me. Found any interesting example yourself? How do you see guardrails play into your own productivity in the future? Now, let's dig deeper. LLM voice mode is definitely susceptible to attack, and what’s more - it’s the perfect demonstration of how guardrails are imperfect, how they can be bypassed, and how versatile - or less so - they make ChatGPT and other LLMs. Any change in technology affects us, and is often seen as new, But, we've nearly always seen it before elsewhere. Remember the attack hiding a jailbreak in a file, uploaded to the LLM? Or the one targeting its' GIF library? It's Reflections on Trusting Trust (Ken Thompson), threat modeling, and attack patterns we've seen before - as always when a new technology steps in the fray. Example: bypassing an anti-virus twenty years ago by archiving the virus executable in a zip. And then in three layers of zip. And then rar on top. On a personal note, this jailbreak once again demonstrates how need-to-know controls like those we build Knostic are so important for safe and secure deployment of LLMs, to make sure that regardless of limitations of guardrails, systems like Microsoft O365 Copilot of Glean won't overshare, or leak, beyond what a user is supposed to know and do. Example: The new attack class we released, Flowbreaking, here: https://lnkd.in/d26YyAEx #jailbreaking #chatgpt #ai #artificialintelligence #hacking #cybersecurity #informationsecurity #riskmanagement #dropabeat #goodmorning

Have you ever seen LLMs delete an answer? One second it was there, and the next - poof. We asked ourselves: is this reproducible and exploitable? The answer was yes. And, as a result, a widely used LLM provided our researcher, masquerading as a girl, with potential instructions on "self-harm." You can read the research blog, here: https://lnkd.in/dhhM9mmj We’re disclosing two new LLM attacks, "Second Thoughts" and "Stop and Roll", and propose that they represent a new class of attacks, Flowbreaking, which complements Prompt Injection and Jailbreaking. Flowbreaking takes advantage of the fact that AI/ML systems such as LLM applications and agents are more complex than we often think, where internal components (i.e., guardrails) can be targeted individually, as well as by manipulating the interplay between them. We can observe beyond the closed box’s input and output perimeter to test them, and exploit them. Impact-wise, when it comes to enterprise adoption of LLM search tools such as Microsoft O365 Copilot and Glean, organizations would need to make sure that answers aren't streamed, but provided in full, and implement an access control capability, making sure that regardless of any attack, the user would only get information based on their own need-to-know and business context. The "self-harm" issue was discovered after we released the report, and we're in the process of reporting it to the relevant LLM provider. We will continue to release research on LLM security and safety, and invite you to follow Knostic here on LinkedIn, or visit us at https://lnkd.in/eYebEzij. #informationsecurity #riskmanagement #LLM #AI #aisecurity #hacking #artificialintelligence #copilot #chatgpt #openai #jailbreaking #promptinjection