Acoustic Prompt Tuning

# Acoustic Prompt Tuning for audio-language models

**Date**: November 30, 2023, Updated February 18, 2025  
**Institution**: Multiple institutions

## Algorithm

Extended LLMs to audio domain understanding without compromising domain-specific capabilities through soft prompting architecture. Injects audio embeddings as soft prompts with instruction-aware audio aligner generating prompts conditioned on both text and sounds.

Cross-modal breakthrough: Competitive results compared to expert models across audio tagging, automated captioning, query-based sound event detection, temporal event retrieval, and sound event counting. Successfully extends frozen VLMs to audio domain without fine-tuning.

Methodological innovation: Curriculum learning strategy with interleaved token architecture and Natural Language Audio Reasoning (NLAR) for comparative analysis of audio clips.

## Comparison with Other Work

Acoustic Prompt Tuning extends continuous methods like Prefix-Tuning and Prompt Tuning to audio domains rather than text-only optimization. Unlike discrete methods like AutoPrompt and EvoPrompt, it works with continuous audio embeddings. Compared to LLM-as-optimizer methods like OPRO, it uses domain-specific prompting rather than general optimization. Unlike reinforcement learning methods like RLPrompt and TEMPERA, it uses supervised learning with curriculum strategies. It's similar to multimodal extensions of other methods but focuses specifically on audio rather than vision. Unlike text-only methods, it requires handling both audio and text modalities in the prompt architecture.