There’s a tempting idea in AI: write a good enough system prompt, and the model will behave. Tell it “never go over the customer’s credit limit,” “always check stock first,” “never sell below cost” — and you’re done. You’re not. And in a real business system, that assumption is exactly what turns an impressive demo into a production failure.
The problem
An AI model is probabilistic. It doesn’t follow a rule — it’s nudged toward an answer. So however carefully you word the prompt, sooner or later, on some odd phrasing, it does the wrong thing: approves an order it shouldn’t, gives a bigger discount than allowed, promises stock that isn’t there.
Not often. But “not often” isn’t good enough when the rule can never break. A credit limit is exactly that kind of rule.
And it fails without anyone attacking it. No hacker, no clever exploit — just the model being helpful and getting it wrong. (Air Canada’s chatbot once invented a refund policy, and the airline was held liable. Nobody attacked it. The model just made it up.)
The fix
Stop asking the prompt to enforce rules. Put the hard rules in code instead — around the model, not inside its instructions.
- Before the model runs, filter the data so it only ever sees valid options. It can’t offer a wrong price if the wrong price never reaches it.
- After the model runs, check its answer before anything acts on it. This catches a bad order even if the model slips one through.
- Around what it can do, give it the least access it needs. A read-only assistant gets read-only keys — so a hallucinated “delete this” has nowhere to go.
The model keeps doing what it’s genuinely good at: conversation, reasoning, ranking. The things it must never get wrong are simply no longer left to it.
What OWASP says
OWASP — the Open Worldwide Application Security Project, the community behind the widely-used security standards for web and application security — says it in its report on LLM Applications. Here’s the full list, in their order:
LLM01 — Prompt Injection: handle sensitive functions in code, not in the prompt; give the app its own least-privilege access; validate the model’s output with code; require human sign-off for risky actions; mark untrusted input clearly.
LLM02 — Sensitive Information Disclosure: stop the model from leaking personal, business, or credential data; sanitize inputs and limit what it can reach.
LLM03 — Supply Chain: vet the models, libraries, and training data you build on; a compromised component compromises everything downstream.
LLM04 — Data and Model Poisoning: protect training and fine-tuning data from tampering that plants biases or backdoors.
LLM05 — Improper Output Handling: treat the model as untrusted; validate and sanitize everything it returns before anything acts on it.
LLM06 — Excessive Agency: give the model the fewest tools and permissions it needs; run actions as the actual user; enforce authorization in your own systems, not in the model.
LLM07 — System Prompt Leakage: keep secrets and rules out of the prompt; don’t use the prompt as a security control; put guardrails outside the model.
LLM08 — Vector and Embedding Weaknesses: secure your RAG data; retrieval from embeddings is probabilistic, so it can’t carry a hard guarantee on its own.
LLM09 — Misinformation: ground answers in real data; cross-check important output; keep a human in the loop for high-stakes decisions.
LLM10 — Unbounded Consumption: cap usage and cost; stop runaway queries from draining resources or racking up bills.
The common thread across the ones that matter most here — LLM01, 05, 06, 07, 09: every rule that must never break lives in code, and the model is trusted only with the reasoning it’s actually good at.
The bottom line
Soft reasoning goes to the model. Hard rules go in code. That’s the line most “AI agent” tools never draw — and it’s the difference between a demo and something you can actually run in an enterprise application.
Based on OWASP Top 10 for LLM Applications (2025).
