Skip to content

Table of Contents

🏁 Getting Started

Your first job.

  • Installation: Install d9d from PyPI or from a checkout, and the optional extras.
  • Quickstart: Train a small model on one GPU and learn the parts of a d9d job.

💡 Concepts

The ideas behind d9d.

  • How d9d Works: The parts of a job, the model build, parameter placements, packs and pipeline programs, the training loop, checkpoints and events.
  • Choosing Parallelism: Where to learn the parallelism strategies and how to apply them in d9d.
  • d9d and Other Frameworks: How the design of d9d compares with Megatron-Core, torchtitan, DeepSpeed, Accelerate and others, and when to choose each.

🍳 Guides

Solutions for concrete tasks.

  • Write Your Own Job: Turn the Quickstart script into a job for your own model, data and loss.
  • Running Jobs: Launch on one GPU, several GPUs or several nodes. Checkpoints, resuming and export.
  • Fine-Tune a Hugging Face Model: Load Qwen3 from a Hugging Face checkpoint, fine-tune all weights or LoRA adapters and export it back.
  • Troubleshooting: Reading the logs of a multi-rank job.

🌐 Distributed Core

The foundational primitives managing the cluster.

  • Distributed Context: The source of truth for topology, and the DeviceMesh domains (regular, dense, expert, batch, flat).
  • Distributed Operations: Utilities for gathering var-length tensors and objects.
  • State Offloading: Releasing GPU training state to host memory for colocated RL (the sleep/wake primitives).
  • PyTree Traversal: Dataclass-aware recursive mapping and flattening over nested tensor structures.
  • Typing Extensions: Python type annotations for common objects and structures.

🚀 Execution Engine

How to configure and run jobs.

  • Training Loop: The lifecycle of the Trainer, dependency injection, and execution flow.
  • Inference Loop: The lifecycle of distributed Inference and forward-only execution.
  • Checkpointing: What a checkpoint holds, when the job saves, how it resumes and where it must be stored.
  • Configuration: Pydantic schemas for configuring jobs, scheduling, and logging.
  • Interfaces: How to inject your custom Model, Data, and Step logic (Train & Infer).

💾 Data and State

Managing data loading and model checkpoints.

  • Model State Mapper: The graph-based transformation engine for checkpoints (transform architectures on-the-fly).
  • Model State I/O: Streaming reader/writers for checkpoints.
  • Datasets: Distributed-aware dataset wrappers and length bucketing.

🧠 Modeling and Architecture

Building blocks for LLMs.

  • Model Catalogue: Models available directly in d9d.
  • Model Design: Principles for creating compatible models.
  • Modules: Building blocks for implementing compatible models.

⚡ Parallelism

Strategies for distributing computations.

  • Horizontal Parallelism: Data parallelism, Fully Sharded Data Parallel (FSDP), expert parallelism and tensor parallelism.
  • Pipeline Parallelism: Vertical scaling, schedules (1F1B, Zero Bubble), and cross-stage communication.

🔧 Fine-Tuning (PEFT)

Parameter-Efficient Fine-Tuning framework.

📈 Optimization and Metrics

Metrics, learning rate schedules and optimizers.

📊 Experiment Tracking

Where the loss and the metrics go.

  • Experiment Tracking: Choosing a tracker (the log, W&B, TensorBoard, MLflow, Aim), what the loop logs and adding a new tracker.

⚙️ Internals

Deep dive into the engine room.