Instruction Hierarchy

# The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

## Overview

This paper proposes a formal hierarchy for instructions with different privilege levels, specifically focusing on system versus user prompts. The authors show how training with this hierarchy improves defense against prompt injection attacks while maintaining standard capabilities.

## Algorithm

The instruction hierarchy framework defines priority levels for different types of instructions:

1. **Priority 0 (Critical)**: System Messages (developer-provided)
2. **Priority 10 (High)**: User Messages (end-user provided)  
3. **Priority 20 (Medium)**: Messages in images/audio
4. **Priority 30 (Low)**: Tool outputs and third-party content

The training process involves:
1. Curating datasets with instructions at different hierarchy levels
2. Training models to assign appropriate attention weights to instructions based on their priority level
3. Implementing conflict resolution mechanisms that prioritize higher-level instructions
4. Evaluating the model's ability to resist prompt injection attacks while maintaining functionality

## Key Findings

1. **Improved Security**: Models trained with the instruction hierarchy showed a 63% improvement in defense against system prompt extraction and over 30% improvement in jailbreak resistance.

2. **Maintained Capabilities**: Despite the increased security, models maintained their standard capabilities, showing that the hierarchy approach doesn't compromise functionality.

3. **Hierarchical Processing**: The approach enables more sophisticated processing where models can accommodate user requests within system constraints while maintaining behavioral consistency.

## Comparison with Other Work

**vs. System Prompt Design**: While papers like "System Prompts in Large Language Models" focus on the content and structure of system prompts, this work focuses on the processing hierarchy and how different instruction types should be prioritized during model execution.

**vs. Prompt Engineering a Prompt Engineer (PE²)**: PE² focuses on automating prompt generation, while this work focuses on how to process and prioritize different types of prompts. Both are important for robust prompt engineering systems, but address different aspects.

**vs. Attention Mechanisms Research**: Papers like "Inside the Attention Mechanism" explain how attention works in transformers, while this work builds on that understanding to propose a specific hierarchy for processing different instruction types.

**vs. Prompt Report**: The Prompt Report surveys various prompting techniques but doesn't specifically address the security aspects of prompt processing that this paper tackles.

## Impact and Future Directions

This work has significant implications for the security and robustness of LLM applications:

1. **Security Framework**: It provides a formal framework for thinking about prompt security that can be implemented across different model architectures.

2. **Standardization**: The priority-based approach offers a standard way to think about instruction processing that could be adopted by different LLM providers.

3. **Application Design**: It informs how developers should structure their applications to properly separate system-level instructions from user inputs.

Future research directions include:
- Extending the hierarchy to more granular levels
- Developing better conflict resolution mechanisms
- Adapting the approach to multimodal models
- Creating evaluation benchmarks specifically for instruction hierarchy compliance