Autograd Extensions Distributed Context Distributed Operations PyTree Sharding Typing Extensions

Datasets

Determinism Gradient Synchronization Pipeline State Management Pipelining Internals Distributed Profiling Experiment Tracking

Piecewise Scheduler Visualization

Metrics

Model State I/O Model State Mapper

Model Design Horizontal Parallelism Pipeline Parallelism Qwen3 MoE

Attention Layers Embeddings Feed Forward Networks (FFN) Model Heads Hidden States Aggregation Mixture of Experts (MoE) Positional Embeddings

Stochastic Optimizers

PEFT Overview Full Fine-Tuning LoRA Method Stacking