Large language models (LLMs) have become mainstream. Recent estimates suggest there are over 1.1 billion ChatGPT users. LLMs and AI agents have become a part of every aspect of our lives, from writing emails to coding and everything in between.

Yet, this scenario is a cybersecurity incident waiting to happen. Many developers aren’t used to treating LLMs as part of their official stack, so they fall outside official cybersecurity systems. Luckily, OWASP has released the OWASP LLM Top 10, detailing LLMs’ ten most glaring security vulnerabilities.

Below, we’ll review each risk on the top ten list for LLMs. We’ll consider how to mitigate each to make sure your LLMs, and how you use them, are as secure as possible.

LLM01: Prompt Injection

Prompt injection is an LLM vulnerability where an attacker is able to get the AI to ignore its original instructions to carry out the attacker’s instructions instead. This can be either through instructions provided to the LLM or as a result of content from third-party sources consulted by the AI.

Prompt injection can result in everything from exposed data to manipulated decisions or the AI performing actions that should be restricted. In the full version of the OWASP LLM Top 10, it lays out nine potential scenarios that could be caused by prompt injection, like a cyberattacker attaching a seemingly nonsensical string of characters at the end of a prompt to cause something dangerous to happen.

To prevent prompt injection, OWASP recommends keeping content separate from instructions provided by users, keeping backend access by AIs to a minimum, and requiring human approval for sensitive actions.

LLM02: Insecure Output Handling

Insecure output handling can take place when an application blindly feeds the output of an AI into another application without checking it first. An LLM might generate executable code or a database query, for example, which then gets executed without scrutiny by the second application. This can result in everything from cross-site scripting to remote code execution to sensitive data getting leaked.

Luckily, preventing insecure output handling is relatively simple. Validating and sanitizing AI output will virtually eliminate LLM02. Recommendations include adding parameters to database queries, providing escape content — which is when sensitive characters are replaced with safer counterparts — and avoiding passing AI output directly into a terminal or command prompt.

LLM03: Training Data Poisoning

Training data poisoning occurs when attackers intentionally insert dangerous content into an LLM’s training data. This can be anything from malicious code to offensive data, resulting in the LLM returning unsafe or compromised responses. This was already causing issues with AI as far back as ten years ago with the Microsoft Tay fiasco, when a chatbot trained on Twitter data started spewing hateful content within 24 hours of being launched.

To avoid LLM03, organizations should carefully vet where their training data is coming from and keep meticulous records of how it was collected and modified. Adopting different training data for different use cases instead of relying on one monolithic dataset also helps to keep risks to a minimum. Monitoring a model using established benchmarks will help your system catch any malicious behavior if your LLM becomes compromised.

LLM04: Model Denial of Service

Model Denial of Service is almost identical to a distributed denial-of-service (DDoS) attack. It happens when an attacker overloads a model with requests or consumes an excessive amount of a model’s resources, resulting in slow responses, outages, or excessively large bills. This can be caused by everything from excessively long prompts to expensive reasoning tasks to simply flooding the zone with extensive requests.

Implementing usage limits helps avoid Model Denial of Service. Rate limiting helps prevent a model from becoming overloaded, while limiting input size stops long prompts from causing harm. Monitoring usage patterns also helps prevent Model Denial of Service from happening, as attackers often probe a system for vulnerabilities before they attack.

LLM05: Supply Chain Vulnerabilities

LLMs rely on a lot of dependencies, often requiring multiple libraries, datasets, pre-trained models, plugins, and cloud services to complete their tasks. This can result in compromised tools like the PyTorch vulnerability of 2022, where users subscribed to the library’s nightly builds ended up installing malicious software.

To avoid supply chain vulnerabilities, try to use tools from trusted third-party vendors as often as possible. You should also make an inventory of all of your system’s dependencies, which can then be incorporated into your security system. Monitoring and updating your AI should be every bit as important as keeping up with your other software.

LLM06: Sensitive Information Disclosure

Sensitive information disclosure is when an LLM is tricked into divulging important assets like personal information, confidential business documents, or proprietary code. This can be due to the model being trained on sensitive data, overly permissive access controls, or carefully crafted user prompts. This can cause issues like when Samsung employees pasted proprietary code into an AI, asking the tool to check their work for errors, inadvertently leaking sensitive data in the process.

Samsung’s response to the 2023 security incident is a good example of how to prevent LLM06. It implemented a 1,024-byte-per-user limit on uploads. Another strategy is to implement the principle of least privilege to limit what a model is able to access as much as possible. Prompt testing should be performed consistently, as well, to make sure that sensitive information isn’t being leaked.

LLM07: Insecure Plugin Design

Many LLMs extend their functionality using third-party plugins. A poorly designed plugin can become an attractive target for attackers, as it often has access to a system’s most sensitive data but with only a modicum of the scrutiny given to more official systems. This oversight can allow attackers to inject their own malicious content or exfiltrate sensitive information using queries like SQL WHERE. Problems with plugin design often arise when developers accept unscripted input or fail to perform proper authorization checks.

To prevent insecure plugin design, developers should only allow third-party plugins to execute very specific functions rather than allowing them to simply accept freeform input. Sensitive actions should be protected behind an additional layer of human approval, as well. Security testing, parameter validation, and strong authentication between plugins and downstream services also help to significantly reduce possible attack surfaces before vulnerabilities are allowed to reach production.

LLM08: Excessive Agency

Excessive agency is when an AI is given too much autonomy to make its own decisions and execute its own actions. The danger’s not so much the AI itself running amok but the scope of its access, which can have devastating consequences if a malicious actor misuses the service. Without additional protection, excessive agency allows attackers to send emails, modify files, or interact with external services using prompt injection.

Restricting AI systems’ access to strictly the resources they need is the first and best way to prevent excessive agency. Sensitive actions like sending communications, modifying records, or spending money in any fashion should always require human intervention, as well. Keeping meticulous records of every action performed by an LLM, limiting the tools it can access, and rate limiting all help to prevent excessive agency from becoming a major issue if there is an incident.

LLM09: Overreliance

Overreliance occurs when users treat everything output by an AI as automatically correct. This can become a major issue if it’s being used for anything technical, legal, or medical and is blindly trusted instead of verified. This can result in incidents like Mata v. Avianca, where an airline company’s lawyer submitted legal precedents in a trial without verifying them first, resulting in a $5,000 fine against the lawyers.

A simple piece of time-honored wisdom can help eliminate overreliance: trust but verify. AI should always be treated as an assistant, at best, rather than an authority. Critical outputs should always be assessed against trustworthy benchmarks, as well. Finally, sensitive output like AI-generated code, documentation, or recommendations should always be assessed by a human before official release.

LLM10: Model Theft

Model theft is when one company steals another company’s AI, allowing it to reproduce years of research without needing to spend its own money. This can occur by stealing the files directly, reconstructing the model by repeatedly querying the API, or exploiting weaknesses in the system protecting the model. This can result in cases like the Amazon Machine Learning data breach, where attackers were able to reverse engineer a machine learning model by repeatedly querying the API.

To prevent model theft, developers should strictly monitor API usage, require strong authentication, and impose rate limiting, all of which make automated querying more difficult. Developers should also make sure their model’s security is as robust as possible, regularly audit their models, and watermark their models to detect an anomaly if it occurs.

Final Thoughts on the OWASP LLM Top 10

If LLMs are going to be more than just another gimmick, we’ve got to think of them like any other tool. They deserve the same level of careful planning and scrutiny as our databases, APIs, and cloud-based services. LLMs also bring their own security risks, introducing new cyberattacks like prompt injection or Model Denial of Service. Many of their solutions, like requiring human authorization for sensitive operations and robust monitoring, have been best practices for cybersecurity for some time, but they’re especially important if you want to make sure your LLMs are secure.