The Security Risks of Large Language Models

Most organisations did not decide to adopt large language models. They discovered, usually months late, that adoption had already happened: a support team pasting customer emails into a chat assistant, a finance analyst uploading a management pack to summarise, a developer wiring a model into an internal tool over a weekend. The governance question is rarely "should we use this?" It is "what has already been sent, to whom, under what terms, and can anyone show it?"

That matters because a language model behaves unlike the systems your controls were built around. It accepts instructions from anywhere in the text it is shown, cannot reliably tell an instruction from data, and produces output that downstream systems tend to trust. Almost every serious incident pattern published so far turns on those three properties rather than on anything exotic about the model.

The risk sits in the plumbing, not the model

Treating this as a model-safety problem sends teams down the wrong path — benchmarking hallucination rates while the real exposure is a service account with standing write access to the CRM. Treat the model as an untrusted component in the middle of a trusted pipeline: everything flowing in may be attacker-controlled, everything flowing out may be attacker-shaped. Your controls belong at the boundaries.

Prompt injection, direct and indirect

A model reads its instructions and the content it is given as one stream of text. If your assistant retrieves a web page, a PDF, a supplier invoice or an inbound email, text hidden in that content can redirect its behaviour. This is not a bug awaiting a patch. No system prompt reliably prevents it, and rules written into a prompt are not controls — they are requests. Enforcement has to happen in code, after generation.

Data leaving your control

Prompts and outputs are generally logged by the provider, sometimes retained for abuse monitoring, and on consumer tiers may be used to improve the service. Whether that is lawful for you depends on the contract you are actually on, not the one on the marketing page.

Insecure handling of model output

Where generated text is rendered as HTML, executed as SQL or shell, or passed to another system as a parameter, the model becomes an injection vector into your own estate. Escape and validate output exactly as you would user-submitted input.

Excessive agency

Tools that send email, update records or move money multiply the blast radius of a successful injection. The control is narrow, short-lived credentials scoped to one purpose, plus human approval before anything irreversible or externally visible.

Retrieval that ignores permissions

An index built by crawling a shared drive inherits none of that drive's access controls. Unless retrieval is filtered by the requesting user's entitlements at query time, the assistant becomes an efficient search engine for documents that person was never meant to see.

Controls that change the outcome, and the evidence they produce

Failure mode	Control that works	Evidence to retain
Indirect prompt injection	Treat retrieved content as untrusted; enforce authorisation outside the prompt	Architecture note showing where authorisation is checked
Uncontrolled data egress	Enterprise tenancy with training opt-out; approved-tool list	Signed terms, processor agreement, sub-processor list
Insecure output handling	Escape and validate output before rendering or executing	Code review record; injection test cases
Excessive agency	Scoped credentials; approval gate on irreversible actions	Permission matrix; approval log with actor and timestamp
Permission bleed in retrieval	Entitlement filtering applied at query time	Access test results per user role
Shadow AI	A sanctioned tool that is easier than the unsanctioned one	Usage telemetry; policy acknowledgements

What the law actually asks of you

Two regimes apply at once, and they ask different questions. Under UK and EU GDPR, sending personal data to a model provider is processing like any other: it needs a lawful basis, a purpose that has not quietly drifted, a processor contract, a transfer mechanism where the provider sits outside the UK or EEA, and security appropriate to the risk under Article 32. Where use is likely to result in high risk to individuals — profiling, decisions affecting people, large-scale special category data — a data protection impact assessment is required before you start. If model use causes a breach, the Article 33 clock still runs, and 72 hours is not long when nobody can say what was in the prompts. Maximum penalties reach 4% of global annual turnover.

The EU AI Act adds obligations keyed to role and risk. Most mid-size organisations are deployers rather than providers, which is the lighter position — but fine-tuning a model, putting your own name on a system, or materially changing its intended purpose can move you into provider obligations. Deployers of high-risk systems are expected to assign competent human oversight, use input data relevant to the purpose, keep logs and inform affected workers. The Act also expects staff using these systems to have a sufficient level of AI literacy, which in practice means documented role-specific training rather than a policy circulated by email. Maximum penalties are 7% of global turnover for prohibited practices and 3% for most other obligations.

Alongside the law, ISO/IEC 42001 offers a certifiable management-system structure for AI, ISO/IEC 27001 remains the home for the underlying security controls, and the OWASP Top 10 for LLM Applications is the most practical engineering checklist available. Auditors increasingly ask which of these you map to.

A pre-deployment checklist

• A named owner for the use case, and a named owner for the service itself.
• Contract and tier confirmed in writing, including whether inputs are used for training and how long they are retained.
• Sub-processors and hosting locations identified, with a transfer mechanism recorded where relevant.
• A written statement of what data may enter prompts and what must never — with the "never" list short enough to be remembered.
• Retrieval sources enumerated, with entitlement filtering tested per role rather than assumed.
• Every tool the system can invoke listed, with the credential scope for each.
• An approval gate on anything irreversible, externally visible or financial.
• Logging of who invoked the system, what it did and what it returned, retained long enough to investigate.
• Injection test cases in the test suite, including instructions hidden in retrieved documents.
• Incident routing agreed with the security team, so an AI-related breach lands in the existing process.

Where this guide stops

This is a security and governance overview, not legal advice, and it cannot resolve questions that turn on your specific facts. Whether a system falls into the EU AI Act's high-risk categories, whether fine-tuning has made you a provider, and whether a particular international transfer stands up all need assessing against your own processing and contracts — take qualified legal advice on those. It does not cover model evaluation, bias testing or accuracy assurance, which are separate disciplines. Regulated sectors carry additional supervisory expectations, and entities in scope of NIS2 or equivalent UK rules have incident-reporting duties on top of everything here. Guidance continues to develop, so revisit any assessment on a fixed cycle rather than filing it.