Keynotes
Title: On the Security and Privacy Risks of Generative AI Systems
Alina Oprea, Professor @ Northeastern University
Alina Oprea is a Professor at Northeastern University in the Khoury College of Computer Sciences. She joined Northeastern University in Fall 2016 after spending 9 years as a research scientist at RSA Laboratories. Her research interests in cyber security are broad, with a focus on AI security and privacy, ML-based threat detection, cloud security, and applied cryptography. She is the recipient of the Technology Review TR35 award for her research in cloud security in 2011, the Google Security and Privacy Award in 2019, the Ruth and Joel Spira Award for Excellence in Teaching in 2020, and the CMU Cylab Distinguished Alumni Award in 2024. Alina served as Program Committee co-chair of the flagship cyber security conference, the IEEE Security and Privacy Symposium in 2020 and 2021. She also served as Associate Editor of the ACM Transactions of Privacy and Security( TOPS) journal and the IEEE Security and Privacy Magazine. Her work was recognized with Best Paper Awards at NDSS 2005, AISEC in 2017, and GameSec in 2019..
In the last few years, we have seen tremendous progress on the capabilities of generative AI and large language models (LLMs). As model sizes have reached hundreds of billions of parameters, training models from scratch has become infeasible. Consequently, system developers typically leverage pre-trained LLMs, and later fine-tune them or augment them with external content to specialize them to new tasks. In this talk, we pose the question if these complex LLM deployment pipelines introduce new security and privacy risks for users. We discuss a new privacy attack on fine-tuned LLMs and a new poisoning attack for LLMs utilizing Retriever Augmented Generation (RAG). We also discuss the challenges of developing mitigations and highlight several open problems in securing AI systems.
Title: Challenges and Threats in Generative AI: Misuse and Exploits
Lea Schönherr, Tenure-track Faculty @ CISPA Helmholtz Center for Information Security
Lea Schönherr is a tenure-track faculty at CISPA Helmholtz Center for Information Security since 2022. Her research focuses on information security, particularly adversarial machine learning, trustworthy generative AI, and ML security applications. She is especially interested in language as an interface to machine learning models, including their cognitive representations and code generation with LLMs. She has published several papers on threat detection and defense of speech recognition systems, generative models, and preventing the misuse of generative AI. She obtained her PhD from Ruhr-Universität Bochum, Germany, in 2021 and is a recipient of two fellowships from UbiCrypt (DFG Graduate School) and Casa (DFG Cluster of Excellence).
Generative AI (genAI) is becoming more integrated into our daily lives, raising questions about potential threats within genAI systems and their outputs. In this talk, we will examine the resulting challenges and security threats associated with generative AI. In the first part of the talk, we look at threat scenarios in which generative models are utilized to produce content that is impossible to distinguish from human-generated content. This fake content is often used for fraudulent and manipulative purposes. As generative models evolve, the attacks are easier to automate and require less expertise, while detecting such activities will become increasingly difficult. This talk will provide an overview of our current challenges in detecting fake media in human and machine interactions. The second part will cover exploits of LLMs to disrupt alignment or to steal sensitive information. Existing attacks show that content filters of LLMs can be easily bypassed with specific inputs and that private information can be leaked. Also, established methods in the adversarial machine learning field cannot be easily transferred to generative models. From an alternative perspective, we show that an alternative way to protect intellectual property is by obfuscating prompts. We demonstrate that with only some overhead, we can achieve similar utility while protecting confidential data. The final part of the presentation will discuss the use of generative models in security applications. This includes benchmarking and fixing vulnerable code and understanding these models' capabilities by investigating their code deobfuscation abilities.
Slides
Title: A threat-centric look at Privacy-Preserving Machine Learning
Giovanni Cherubin, Senior Researcher @ Microsoft
Giovanni Cherubin is a Senior Researcher at Microsoft in Cambridge, working with the Microsoft Security Response Centre (MSRC). Before joining Microsoft, Giovanni held research positions at the Alan Turing Institute and EPFL. He obtained a PhD in Machine Learning and Cyber Security from Royal Holloway University of London. His research focuses on the privacy and security properties of machine learning models, as well as the theoretical and empirical study of their information leakage. Additionally, Giovanni works on distribution-free uncertainty estimation for machine learning, such as Conformal Prediction. He has received multiple awards for his contributions to security, privacy, and distribution-free inference.
Privacy-preserving Machine Learning (PPML) has the rare privilege among security research fields of having defences that are both practical and theoretically robust, thanks to over 20 years of progress. However, deployment of these defenses often sparks heated debates over how to tune their parameters. This is partially because these defences are typically designed to counter "any" attack, which can lead to overlooking the specific threats relevant to a particular deployment. This talk will cover the key advancements in PPML research through the principle of "first consider the threats, then pick a defence." By deliberately defining which attacks we consider to be a threat (and which ones we don't) before deploying a model, we can more effectively select concrete parameters for our defences, and better communicate the extent and limitations of the protection we've achieved.