Patterns/Prompting Patterns

JSON Doesn't Sanitize Content

Why output format constraints don't protect against malicious content in valid structures

The Conventional Framing

Structured output constrains model responses to specific formats like JSON, XML, or defined schemas. This ensures outputs are parseable and integrate cleanly with downstream systems.

The pattern is essential for programmatic consumption of model outputs.

Why Format Is Not Validation

Structured output ensures the format is correct—valid JSON, matching schema, proper types. It says nothing about whether the content within that structure is safe.

A perfectly structured JSON response can contain SQL injection in a string field, XSS in a text value, or instructions that cause harm when the JSON is processed downstream.

The nested content problem:

Structured formats often contain free-text fields. These fields can contain anything—including adversarial payloads that escape or attack whatever processes the structured output.

Architecture

Components:

  • Schema definition— specifies output structure
  • Format enforcement— ensures valid structure
  • Field population— model fills in content
  • Downstream processing— systems consume output

Trust Boundaries

Structured output (valid JSON): { "user_name": "Alice'; DROP TABLE users;--", "bio": "<script>stealCookies()</script>", "action": "delete all files in /important/" } Format: ✓ Valid JSON Schema: ✓ Matches expected structure Content: ✗ Contains SQL injection, XSS, dangerous instruction Downstream systems process this "valid" output.
  1. Model → Structured output — content is model-generated
  2. Format validation → Processing — format valid ≠ content safe
  3. Output → Downstream — malicious content reaches consumers

Threat Surface

ThreatVectorImpact
Injection in structured fieldsSQL, XSS, command injection in string fieldsDownstream systems execute malicious content
Schema-valid attacksMalicious content that matches expected schemaValidation passes but content is harmful
Nested structure exploitationAttacks hidden in deeply nested fieldsSurface validation misses deep content
Type coercion attacksValues that change meaning when parsedUnexpected behavior in downstream systems

The ZIVIS Position

  • •
    Format is not sanitization.Valid JSON can contain injection payloads. Structured output validates syntax, not safety.
  • •
    Sanitize at the boundary.Every field that will be used in SQL, HTML, commands, etc. must be sanitized for that context, regardless of source.
  • •
    Schema validation is necessary but insufficient.Validate schema AND sanitize content. Schema says 'this is a string'; sanitization says 'this string is safe for use in HTML'.
  • •
    Trust nothing from model output.Treat model output like user input: valid-looking but potentially malicious. Defense in depth applies.

What We Tell Clients

Structured output solves parsing problems, not security problems. A valid JSON response can contain injection attacks in every string field.

Sanitize model output for whatever context it will be used in—SQL queries, HTML rendering, command execution. Don't assume that because the format is valid, the content is safe.

Related Patterns