Tensorrt Llm.prompt

# TensorRT-LLM Optimization

## Overview

TensorRT-LLM is NVIDIA's open-source library for compiling and optimizing Large Language Models for inference on NVIDIA GPUs. It combines TensorRT's graph optimization and kernel fusion capabilities with LLM-specific optimizations including paged KV cache, inflight batching, multi-GPU tensor parallelism, and quantization to INT8/INT4/FP8. TensorRT-LLM typically delivers 2-8x throughput improvement over vanilla PyTorch inference on the same hardware.

The library supports most popular LLM architectures (LLaMA, GPT, Falcon, MPT, Baichuan, Qwen, Mistral, Mixtral, and many more) and provides a high-level API for building optimized engines from HuggingFace models. The compilation step is the investment — once compiled, the engine delivers consistently high performance with minimal overhead.

## Key Concepts

- **Engine compilation**: Convert a HuggingFace model to a TensorRT engine with optimized kernels, graph fusion, and memory planning
- **Inflight batching**: Continuously schedule requests at the token level rather than waiting for entire sequences to complete
- **Paged KV cache**: Allocate KV cache memory in pages to reduce fragmentation and support longer sequences
- **Tensor parallelism**: Split model layers across multiple GPUs for models that don't fit on a single GPU
- **Quantization**: Reduce model precision to FP8, INT8, INT4 (GPTQ, AWQ) for lower memory and higher throughput
- **Plugin system**: Custom TensorRT plugins for LLM-specific operations like FlashAttention, rotary embeddings
- **Speculative decoding**: Use a smaller draft model to predict multiple tokens, verified by the large model in one pass

## Implementation Patterns