Instruction Hierarchy

Instruction hierarchy is a training technique that gives a large language model an explicit ranking of whose instructions to follow when they conflict, rather than treating all text in its context window as equally authoritative. Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel of OpenAI introduced it in "The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions" (arXiv:2404.13208, April 2024). The paper argues that a root cause of prompt injection and jailbreak attacks is that models often give a system prompt written by an application developer the same weight as text pasted in by an untrusted user or fetched from a third-party document. This means an attacker's embedded instructions can simply overwrite the developer's original intent.

The proposed fix ranks instruction sources by privilege: system prompt above developer message above user message above third-party content such as tool outputs or retrieved documents. It also defines how the model should behave when a lower-privileged instruction conflicts with a higher-privileged one, generally by ignoring or refusing the lower-privileged instruction rather than complying with it. To teach this behavior, the authors built an automated data generation method that creates training examples of conflicting instructions at different privilege levels, paired with the response that respects the hierarchy, and fine-tuned GPT-3.5 on this data.

The authors reported that the resulting models became more resistant to prompt injection and jailbreak attacks, including attack types not seen during training, while showing only minor drops on standard capability benchmarks. Instruction hierarchy has since become a standard part of OpenAI's model specification and safety stack, deployed alongside other techniques such as chain-of-thought monitoring. It is commonly cited as one of the first systematic training-time defenses against prompt injection in production language models.