Mixture of Experts Training.prompt

# Mixture-of-Experts Training

## Overview

Mixture-of-Experts (MoE) models train a gating network to route each token to a subset of expert networks, achieving the quality of much larger dense models at a fraction of the computational cost. Training MoE models presents unique challenges: load balancing (ensuring experts are used evenly), routing stability (preventing expert collapse), and communication overhead (distributing experts across GPUs). Modern MoE architectures like Mixtral, DeepSeek-V2, and Qwen-MoE have demonstrated that MoE can match dense model quality with 3-4x less compute.

## Key Concepts

- **Expert Collapse**: When most tokens route to a small subset of experts
- **Load Balancing Loss**: Auxiliary loss encouraging even expert utilization
- **Top-K Routing**: Selecting K experts per token (typically K=1 or K=2)
- **Capacity Factor**: Maximum tokens per expert per batch
- **Expert Parallelism**: Distributing experts across GPUs
- **Shared Experts**: Experts always activated regardless of routing

## Implementation Patterns

### MoE Training Loop