Untrusted Until It Clears Validation
Everything the model returns is untrusted until it clears validation. Force a schema, discard anything outside it, and ground every referenced entity against real data before you persist a byte.
The Conventional Framing
The advice is now standard: don't let the model reply in free text when a decision has to be machine-consumed. Force JSON or tool-mode output, constrain the decision fields to an enum, and reject anything that doesn't parse against the schema. A model that can only return {"action": "approve" | "reject" | "escalate"} can't smuggle a paragraph of injected instructions into a field your code will act on.
This is real, and it is cheap. Schema enforcement kills an entire class of failure — malformed output, free-text decisions, prose where you expected a value. It turns “parse whatever the model said and hope” into “accept a fixed shape or fail closed.” If you are persisting model output anywhere, this is table stakes.
Why Schema Validation Is Necessary — and Not Sufficient
Here is the trap. Schema validation constrains form, not semantics. A well-formed value that is simply wrong passes every check you wrote. {"action": "approve"} is valid JSON, a legal enum member, and exactly what an attacker who steered the model wanted. The schema was never asked whether the decision was correct — only whether it was shaped right.
The second failure is fabricated entities. Ask a model to cite the finding it's acting on and it will happily return {"finding_id": "F-4021"} — an ID with the right prefix, the right digit count, and no referent in your database. It parses. It validates. It points at nothing. Injected content is very good at getting a model to invent plausible identifiers.
The failure modes that survive a naive schema check:
- Well-formed-but-wrong values. An enum member the attacker biased the model toward. Structurally perfect, semantically hostile. No schema catches this.
- Fabricated entities. Component IDs, finding IDs, user references that match the shape but don't exist. The schema validates the pattern, never the referent.
- Injected directives in free-text fields. Leave one
notesorreasonstring un-constrained and it becomes a channel — into your database, your UI, the next model's context.
Architecture
Components:
- Constrained decoder— forces JSON/tool-mode output with enum-constrained decision fields; free text is not an option the model has
- Schema validator— rejects anything outside the declared shape — wrong types, extra keys, non-enum values, unparseable output
- Grounding check— confirms every referenced entity (component, finding, user ID) actually exists in your data before anything is trusted
- Storage boundary— the single gate where validated-and-grounded output is persisted; rejection happens here, not downstream
Trust Boundaries
- Untrusted input → LLM — injection enters here; constraining the decoder limits what the model can emit, but not what it can decide
- LLM → schema validation — the form gate — malformed output and non-enum values are discarded, never parsed permissively
- Schema → grounding check — the critical boundary — a well-formed value is still untrusted until every referenced entity is confirmed against ground truth
- Grounding → storage — reject here, at persistence, so a fabricated or biased value never becomes state that downstream code trusts
Threat Surface
| Threat | Vector | Impact |
|---|---|---|
| Well-formed-but-wrong | Injection biases the model toward a valid enum value (e.g. 'approve') | Attacker-chosen decision passes every schema check and persists |
| Fabricated entity | Model returns a plausible finding/component ID with no referent in the data | Persisted state points at nothing; downstream logic acts on a ghost |
| Free-text leakage | An un-constrained notes/reason field carries injected directives | Injection reaches the database, the UI, or the next model's context |
| Permissive parsing | Code coerces or salvages malformed output instead of rejecting it | The schema boundary is bypassed; garbage becomes trusted state |
The ZIVIS Position
- •Everything the model returns is untrusted.Treat model output like user input from a hostile client. It has cleared nothing until validation says so. Well-formed is not the same as safe.
- •Constrain form, then verify semantics.Schema and enums stop malformed injection — the cheap, high-volume attacks. They do not stop a well-formed lie. Grounding does.
- •Ground every referenced entity.Before you persist anything referencing a component, finding, or user, confirm that entity exists in your data. A valid ID with no referent is a fabrication, not a fact.
- •Reject at storage, not downstream.The storage boundary is the one place to fail closed. Once a bad value is state, every consumer inherits the trust you failed to withhold.
- •Leave no free-text field un-constrained.Every open string is a channel. Enumerate what you can, length-bound and sanitize the rest, and never let a reason field become an instruction.
What We Tell Clients
Do this on every path where model output becomes state. It is low-to-medium cost and it removes an entire failure class in an afternoon: force tool-mode or JSON, constrain decision fields to enums, and discard anything outside the schema. If you are only doing this much, you are ahead of most teams — and still exposed.
The part that actually holds the line is grounding. Schema validation proves the output is shaped right; it says nothing about whether it is true. A well-formed decision the attacker biased, or a fabricated finding ID with the right prefix, sails straight through. So cross-check every referenced entity against real data before you persist, and reject at the storage boundary — not three services downstream where the value is already trusted. Form is the first gate. Ground truth is the one that matters.
Related Patterns
- Structured Output— the prompting-side view of forcing schema-constrained model responses
- Plan-Then-Execute & Action-Selector— constraining the set of actions before untrusted content is in scope
- Output Filtering— sanitizing what leaves the system once output has been validated
- Guardrail Scanners— probabilistic detection layered on top of deterministic validation
References
Frequently Asked Questions
How does structured output and validation defend against prompt injection?
It forces the model into JSON or tool-mode output with decision fields constrained to enums, and rejects anything that does not parse against the schema. A model that can only return an enum like approve, reject, or escalate cannot smuggle a paragraph of injected instructions into a field your code will act on. It is cheap and removes an entire class of malformed-output failures.
Is schema validation enough to stop prompt injection?
No. Schema validation constrains form, not semantics. A well-formed value that is simply wrong, such as an approve decision the attacker biased the model toward, passes every check you wrote. The schema was only ever asked whether the output was shaped right, never whether the decision was correct.
What is entity grounding and why does it matter here?
Grounding cross-checks every referenced entity, such as a component, finding, or user ID, against real data before you trust it. Injected content is very good at getting a model to invent plausible identifiers that have the right prefix and digit count but no referent in your database. A valid ID with no referent is a fabrication, not a fact, so grounding is what actually holds the line where the schema cannot.
Where in the pipeline should invalid model output be rejected?
At the storage boundary, before anything is persisted. Reject there rather than three services downstream, because once a bad value becomes state, every consumer inherits the trust you failed to withhold. Treat all model output as untrusted input from a hostile client, and never leave a free-text field un-constrained, since every open string is a channel into your database, UI, or the next model's context.