PromptInjection Theory Technology

| Paper Name                                                       | Link                                     | Summary |
|------------------------------------------------------------------|------------------------------------------|----------|
| 《Ignore Previous Prompt: Attack Techniques For Language Models》 | [link](https://openreview.net/pdf?id=qiaRo_7Zmug) | Transformer-based large language models (LLMs) provide a powerful foundation for natural language tasks in large-scale customer-facing applications. However, studies that explore their vulnerabilities emerging from malicious user interaction are scarce. By proposing PROMPTINJECT, a prosaic alignment framework for mask-based iterative adversarial prompt composition, we examine how GPT3, the most widely deployed language model in production, can be easily misaligned by simple handcrafted inputs. In particular, we investigate two types of attacks – goal hijacking and prompt leaking – and demonstrate that even lowaptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3’s stochastic nature, creating long-tail risks.  <br><br> <img width="986" alt="image" src="https://github.com/user-attachments/assets/e82049db-9e92-4b09-951f-c99a3cbb7481" /> <br><br> <img width="978" alt="image" src="https://github.com/user-attachments/assets/5d59424b-900e-42e7-bf8c-0fad7010437d" />| 
| 《Breaking ReAct Agents: Foot-in-the-Door Attack Will Get You In》| [link](https://arxiv.org/pdf/2410.16950) | <img width="1055" alt="image" src="https://github.com/user-attachments/assets/144d0ad6-fda9-44f7-a2af-b798e22c5ce1" /> | 
| 《Making LLMs Vulnerable to Prompt Injection via Poisoning Alignment》| [link](https://arxiv.org/abs/2410.14827) | In a prompt injection attack, an attacker injects a prompt into the original one, aiming to make the LLM follow the injected prompt and perform a task chosen by the attacker. Existing prompt injection attacks primarily focus on how to blend the injected prompt into the original prompt without altering the LLM itself. Our experiments show that these attacks achieve some success, but there is still significant room for improvement. In this work, we show that an attacker can boost the success of prompt injection attacks by poisoning the LLM’s alignment process. Specifically, we propose PoisonedAlign, a method to strategically create poisoned alignment samples. When even a small fraction of the alignment data is poisoned using our method, the aligned LLM becomes more vulnerable to prompt injection while maintaining its foundational capabilities. |
| 《HIJACKRAG: Hijacking Attacks against Retrieval-Augmented Large Language Models》| [link](https://arxiv.org/pdf/2410.22832) | <img width="662" alt="image" src="https://github.com/user-attachments/assets/129c27a3-df1b-4a75-9a78-8d39fa7ab2ff" /> <img width="658" alt="image" src="https://github.com/user-attachments/assets/ebe2ab8e-91d4-49bf-a816-ce65e3c3a001" /> |
| 《Second-order Jailbreaks: Generative Agents Successfully Manipulate Through an Intermediary》| [link](https://openreview.net/pdf?id=HPmhaOTseN) | As the capabilities of Large Language Models (LLMs) continue to expand, their application in communication tasks is becoming increasingly prevalent. However, this widespread use brings with it novel risks, including the susceptibility of LLMs to "jailbreaking" techniques. In this paper, we explore the potential for such risks in two- and three-agent communication networks, where one agent is tasked with protecting a password while another attempts to uncover it. Our findings reveal that an attacker, powered by advanced LLMs, can extract the password even through an intermediary that is instructed to prevent this. Our contributions include an experimental setup for evaluating the persuasiveness of LLMs, a demonstration of LLMs’ ability to manipulate each other into revealing protected information, and a comprehensive analysis of this manipulative behavior. Our results underscore the need for further investigation into the safety and security of LLMs in communication networks. | 
| 《SAFEGUARD IS A DOUBLE-EDGED SWORD: DENIAL-OF-SERVICE ATTACK ON LARGE LANGUAGE MODELS》| [link](https://arxiv.org/pdf/2410.02916) | <img width="743" alt="image" src="https://github.com/user-attachments/assets/7ec7a078-5e6d-4bd0-a147-bbee05c2aedf" /> |