LLM Prompt Engineering for Structured Outputs: What I Stopped Doing

Most of my prompt engineering time in 2023 went toward coercing LLMs into valid JSON. In 2024, JSON mode, structured outputs (OpenAI API response_format: { type: "json_schema" }), and tool/function calling replaced 80% of that work. What remains genuinely hard is worth documenting — the rest was ritual.

What I stopped doing

"Return ONLY valid JSON with no markdown fences"

Gone. With GPT-4o (2024-08-06) and Claude 3.5 Sonnet structured output modes, fence leakage dropped from ~8% to ~0.1% in our eval set (2,400 extraction prompts on internal support tickets).

Before:

You are a JSON API. Output ONLY a JSON object. Do not include explanations.
Schema: {"category": string, "priority": "low"|"medium"|"high", "summary": string}

After: API-level schema enforcement. Prompt becomes task description only.

Few-shot examples for schema compliance

Three-shot examples teaching key ordering and enum values — removed once schema mode shipped. Examples now focus on domain edge cases (ambiguous ticket categorization), not syntax.

Token savings: ~400 tokens per request on our support triage pipeline. At 50k requests/day, non-trivial cost reduction.

Regex post-processing pipelines

We had a Python module json_salvage.py — 240 lines extracting {...} from model rambling, fixing trailing commas, unescaping smart quotes. Still in repo for legacy model paths (gpt-4-turbo without structured outputs). Default route no longer calls it.

Deletion candidate after Q3 when legacy model deprecation completes.

Chain-of-thought inside the structured payload

Bad pattern we used: "reasoning": string field in schema, hoping it improved accuracy. Added 200+ tokens, increased schema violation rate when reasoning contained unescaped quotes. Moved reasoning to separate non-structured call when needed for audit; primary extraction stays minimal schema.

What still requires careful prompting

Nested ambiguity and business rules

Schema enforces shape, not semantics. Example: expense report line items — model returns valid JSON with amount: 0 for missing values vs null. We now:

• Use JSON Schema required + nullable types explicitly
• Add validation layer (Pydantic v2) after LLM — never trust without business rules
• Prompt with 2–3 counterexamples ("when receipt is missing, set amount null, not zero")

Multi-step extraction with dependencies

Structured output is single-shot. Extracting invoice header then line items with cross-field validation (line totals sum to header) still needs:

1. Pass 1: header schema
2. Pass 2: line items with header context injected
3. Deterministic validator; single retry with error feedback on failure

Function calling helps when pass 2 is a different "tool" — model selects tool, we execute, return result. Not always cheaper than two structured calls.

Model-specific schema limits

OpenAI structured outputs reject some JSON Schema features (e.g., oneOf with >16 branches in our testing). Claude tool use has different limits. We maintain lowest-common-denominator schemas for multi-vendor routing.

Eval caught: anyOf for address formats (US vs EU) failed on one provider — flattened to optional fields.

Long-context retrieval + structured answer

RAG vs fine-tuning decisions apply here. Structured output at end of 8k-token retrieved context — model still hallucinates fields not in source. Prompt must say "use null when not present in context" and validator must flag unsupported claims.

Current production stack

• Primary: GPT-4o-mini for high-volume extraction (cost/latency)
• Fallback: Claude 3.5 Sonnet for messy OCR-derived text
• Schema registry: internal repo, versioned JSON Schema, CI validates against provider compatibility matrix
• Eval: 1,200 golden pairs, regression on schema pass rate + field-level F1

Structured output pass rate target: 99.5%. Field F1 on category: 94% — prompt work lives there, not syntax.

Retry-with-error-feedback pattern

When Pydantic validation fails post-LLM, we send one retry with compact error string:

messages.append({"role": "user", "content": f"Validation errors: {err.json()}. Fix and return valid JSON only."})

Structured mode + single retry recovered 78% of first-pass failures without human review. Two retries showed diminishing returns (+4%) at 2× latency — capped at one retry in production.

Important: error feedback must not include PII from the failed payload in logs — we hash field names only in audit trail.

Cost/latency trade on structured vs function calling

For extraction with 12 fields, benchmark on GPT-4o-mini:

Approach	p95 latency	Cost/1k requests
JSON schema mode	890 ms	$0.42
Function calling single tool	920 ms	$0.44
Legacy prompt + salvage	1100 ms	$0.51

Function calling wins when the model must choose among tools (route to different schemas). Pure extraction: schema mode simpler, fewer moving parts.

Quantization interaction

When we tested quantized local models (Llama 3 8B Q4) for offline extraction, schema compliance dropped to 91% even with grammar-constrained decoding (llama.cpp json_schema sampler). Acceptable for draft mode; not for auto-commit.

What I'd do next

1. Schema diff in CI when product changes field definitions — notify downstream consumers
2. Automatic prompt minimization — A/B test removing instruction sentences when schema mode enabled (ongoing; 30% of prompts still have redundant preambles)
3. Investigate partial streaming JSON for UX on long generations — structured mode currently blocks until complete

Stop fighting the model's formatting instincts. Use platform features for syntax; spend prompt budget on domain disambiguation and validation after the fact.