Force the model to answer in JSON. Constrain the decision field to an enum. Reject anything that doesn't parse. This is good advice, it is cheap, and most teams still don't do it. But it comes with a failure mode that is easy to miss and expensive to learn: schema validation constrains the shape of an answer, never its truth.
A structured output that is perfectly well-formed can still be exactly what an attacker wanted. {"action": "approve"} is legal JSON, a valid enum member, and a catastrophe if the model was steered into it by injected content. The schema was never asked whether the decision was right — only whether it was the correct shape.
What structured output actually buys you
Start with the win, because it is real. When you move a model from free-text replies to tool-mode or constrained JSON, you delete an entire category of problem. The model can no longer return a paragraph where you expected a value. It can't bury an instruction in prose your downstream code will read. Enum-constrained decision fields mean the set of things the model can decide is fixed by you, not negotiated with the input.
This kills the high-volume, low-effort injection: the malformed payload, the free-text directive, the response that doesn't parse. Discard anything outside the schema and those attacks fail closed. If you are persisting model output anywhere and you are not doing this, do it first.
The gap: form is not semantics
Here is where teams get lulled. A schema is a proof about structure. It says the output has the right keys, the right types, an enum member from your list. It says nothing about whether that member is correct. An attacker who can influence the model's reasoning — through a retrieved document, a tool result, a poisoned field — doesn't need to break your schema. They need the model to pick the valid value that serves them. A well-formed answer that is simply wrong passes every check you wrote.
The sharpest version of this is the fabricated entity. Ask a model to cite the finding it's acting on and it will return {"finding_id": "F-4021"} — right prefix, right digit count, no referent anywhere in your database. It parses. It validates against the pattern. It points at nothing. Injected content is remarkably good at coaxing a model into inventing plausible identifiers, and a regex-shaped schema check will wave every one of them through.
Grounding: check the referent, not the shape
The fix is grounding. Before you trust any value that references an entity — a component, a finding, a user, a record ID — confirm that entity actually exists in your data. Not that it matches a format. That it has a referent. A valid-looking ID with nothing behind it is a fabrication, and the only thing that catches it is a lookup against ground truth.
This is the line most schema-first pipelines never draw. They validate the shape and persist. Grounding says the model's claim about the world must be checked against the world before it becomes state. It is the difference between “this is a well-formed finding reference” and “this is a finding that exists.”
Reject at storage, not downstream
Where you reject matters as much as whether you reject. The storage boundary is the one place to fail closed, because once a value is persisted it is trusted by definition — every downstream consumer inherits the trust you didn't withhold. Validate and ground at the write. If you push the check three services downstream, you have already spent the value as state, and now you are trying to claw back a decision the rest of the system believes.
And close the last channel: free-text fields. Every un-constrained notes or reason string is a way for injected directives to reach your database, your UI, or the next model's context. Enumerate what you can, length-bound and sanitize what you can't, and never let a reason field become an instruction.
What we tell clients
Treat everything the model returns as untrusted until it clears validation — the same way you'd treat input from a hostile client. Structured output and enum constraints are the cheap first gate; they stop malformed injection and they belong on every path where model output becomes state. But they constrain form, not meaning. A well-formed value the attacker biased, or a fabricated ID with the right prefix, is invisible to them.
So ground every referenced entity against real data, reject at the storage boundary, and leave no free-text field un-constrained. Schema validation tells you the answer is shaped right. Grounding is the only thing that tells you it's true — and truth is the property you were actually trying to protect.
This is one pattern in our Defense Architecture catalog. The pattern page for Structured Output & Validation has the trust boundaries, threat surface, and architecture diagram in full.