Load Balancing.prompt

# Load Balancing for LLM Serving

## Overview

Load balancing distributes inference requests across multiple model instances or GPUs to maximize throughput and minimize latency. For LLM serving, load balancing must consider GPU utilization, queue depths, model loading states, and request characteristics (prompt length, expected output length). Simple round-robin distribution is suboptimal because LLM requests have highly variable resource requirements.

## Key Concepts

- **Request Routing**: Directing requests to optimal instances
- **Queue Management**: Managing request queues across instances
- **Health Checking**: Monitoring instance health and availability
- **Auto-scaling**: Adjusting instance count based on load
- **Session Affinity**: Routing related requests to the same instance
- **Weighted Routing**: Distributing load based on instance capacity

## Implementation Patterns

### LLM-Aware Load Balancer