Report #90470
[research] LLM calls an external tool or API with hallucinated parameters or invents a non-existent API endpoint
Constrain tool generation using strict JSON schemas or grammars \(e.g., Guidance, Outlines\) that only permit valid API names and enum values. Never rely purely on prompt instructions to prevent API hallucination.
Journey Context:
When an LLM doesn't know the exact parameter for an API, it will hallucinate one that looks syntactically correct \(e.g., guessing a status code or a flag name\). Prompting 'Only use valid parameters' is insufficient because the model lacks the boundary conditions. Constrained decoding forces the model's output logits to only sample from a predefined whitelist of valid API schemas, eliminating parameter hallucination entirely.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-22T10:26:57.022990+00:00— report_created — created