Rlprompt

# RLPrompt: Reinforcement learning optimization

**Date**: May 25, 2022  
**Institution**: Carnegie Mellon University, UC San Diego

## Algorithm

RLPrompt addressed discrete prompt optimization interpretability by formulating prompt generation as a reinforcement learning problem. Used a parameter-efficient policy network (multi-layer perceptron inserted into frozen pre-trained LM) trained with reward signals from downstream task performance, implementing reward stabilization for complex stochastic signals.

Core innovation: First RL-based approach for discrete prompt optimization with transferable prompts across different language models. Works with both masked (BERT) and autoregressive (GPT) architectures while maintaining parameter efficiency.

Performance results: Superior performance on few-shot classification and unsupervised text style transfer, though generated prompts often appeared as ungrammatical "gibberish" text while retaining effectiveness.

## Comparison with Other Work

RLPrompt differs from gradient-based methods like AutoPrompt by using reinforcement learning rather than gradient approximation for discrete optimization. Unlike continuous methods like Prefix-Tuning and Prompt Tuning, it optimizes discrete tokens rather than continuous embeddings. Compared to LLM-as-optimizer methods like APE and OPRO, RLPrompt uses policy gradient optimization rather than natural language generation. It's more parameter-efficient than full fine-tuning while providing better interpretability than black-box methods. Unlike evolutionary methods like EvoPrompt, it uses gradient-based policy optimization rather than population-based search. The "gibberish" prompts distinguish it from natural language optimization methods.