Pipelining Internals
About
This page explains the internals of the d9d.pipelining package. It is for developers who want to add new stage layouts or schedules, or change the execution engine.
Internal API
If you use the standard d9d training loop, you do not need to call this package. d9d sets up pipelining from its configuration. This page is for users who extend d9d.
Architecture
The Idea
d9d separates the schedule structure from the runtime execution.
- You write a builder (e.g.
1F1B,DualPipe) that produces a list of logical actions per rank (e.g.Forward(Stage=0, MB=1),Backward(Stage=0, MB=0)). You can let d9d addSend/Recvactions to your compute-only schedule. It derives them from data dependencies and orders them so the schedule does not deadlock. - A simple virtual machine iterates over the action list and executes each action.
With this split, a research schedule such as Zero Bubble or DualPipeV is a list of actions. You do not write a state machine or recursive calls.
Core Components
PipelineStage (stage/stage.py)
Wraps a user nn.Module. It does not decide when to run. It gives the actions and the executor the atomic operations of a pipeline stage, such as the forward and backward passes.
It consists of:
- Computation handlers:
ForwardComputeHandler: Runs the forward pass and caches inputs and outputs for the backward pass.BackwardComputeHandler: Runs the backward pass. It can split the backward pass intobackward_input(dI) andbackward_weight(dW) for Zero Bubble schedules.
- Communication handlers: Own the P2P buffers for the forward and backward passes. Buffer sizes are inferred per microbatch, so the microbatches in a pack can differ in shape.
Actions (runtime/action.py)
The atomic instructions for the pipeline virtual machine.
ForwardComputeAction: Runs the forward pass for a microbatch.BackwardFullInputComputeAction: Runs the backward pass. It computes gradients for the inputs only, or for the inputs and the weights.BackwardWeightComputeAction: Computes the deferred weight gradients (used in Zero Bubble schedules).ForwardSendAction/ForwardReceiveAction/BackwardSendAction/BackwardReceiveAction: Network I/O.ComposeAction: Groups several actions into one. Schedules such as DualPipeV use it to pair a forward action and a backward action for overlap. The executor currently runs the grouped actions in order.
Actions are declarative and immutable.
Programs
A program is a dict[int, list[ActionBase]]: it maps each rank to its sequential list of actions.
The d9d.pipelining.program.builder package holds the PipelineProgramBuilder base class and the passes that help to build programs. The d9d.pipelining.program.catalogue package holds the ready programs: looped BFS (also used for GPipe and inference), interleaved 1F1B, Zero Bubble V and DualPipeV.
A program depends only on the microbatch count and the fixed pipeline topology. The PipelineProgramCache composes a program once per microbatch count and reuses it across steps. For each program, it keeps the action list of this rank and whether the program contains backward work.
Executor (runtime/executor.py)
The PipelineScheduleExecutor is the runtime engine.
Each step, it receives a pack (a sequence of ready per-microbatch inputs) and a step handler. The handler opens a context around each forward and backward pass of a stage and processes the outputs of each microbatch. Then it:
- Gets the program for the pack length from the
PipelineProgramCache. - Configures the stage buffers for the pack. It sizes the P2P buffers of each microbatch separately, so microbatches can differ in shape. It skips this step when the shapes match the previous step.
- Iterates over the action list of this rank and applies each action.
- Waits for all pending sends to complete.
- Releases the forward cache and the receive buffers of every stage.
Comparison with PyTorch
The d9d pipelining implementation borrows concepts from the torch.distributed.pipelining API, such as the Zero Bubble implementation. It restructures the code for clarity, type safety and modularity.
The main differences are a strict separation of concerns and composition over inheritance:
-
Decomposed stage logic:
- PyTorch: A single
_PipelineStageBaseclass manages P2P buffer allocation, gradient accumulation state, and forward/backward execution. - d9d: The
PipelineStageclass is a thin orchestrator. It delegates the work to dedicated handlers.
- PyTorch: A single
-
Polymorphic actions vs. enumeration:
- PyTorch: Represents schedule instructions with one generic
_ActionNamedTuple and an Enum (_ComputationType.FORWARD,_ComputationType.SEND_F, etc.). - d9d: Uses a class hierarchy for actions (
ForwardComputeAction,ForwardSendAction,ComposeAction). Each action implementsapply, so the executor does not branch on an enum. Compiler passes such asadd_communication_opsusematch/caseon the action classes. Each action carries its own fields (e.g., thefull_backwardflag), and the type checker covers them.
- PyTorch: Represents schedule instructions with one generic
-
Builders vs. schedule classes:
- PyTorch: Often couples the schedule definition with the runtime object. For example, the
Schedule1F1Bclass both generates the order and executes it. - d9d: Separates the program builder, which generates the actions, from the executor, which runs them. You can inspect a schedule before execution, or swap the scheduling algorithm without changing the executor.
- PyTorch: Often couples the schedule definition with the runtime object. For example, the
Building Custom Schedules
To build a new schedule, implement a PipelineProgramBuilder.
Implement the Builder
Run the Builder
Pass the builder to build_schedule instead of a built-in one:
The Trainer builds its schedule from the configuration, so it runs only the built-in schedules.
API Reference
d9d.pipelining.stage
Pipeline stage that runs the forward and backward passes of a model part.
PipelineStage
Bases: Generic[TPipelineInput, TStageTransfer, TSharedInput, TPipelineOutput]
A single stage of a pipelined model.
It combines the P2P handlers (StageReceiver and StageSender) with the compute handlers
(ForwardComputeHandler and BackwardComputeHandler). It hides buffer management, P2P
communication and gradient computation from the schedule.
info
property
The position of this stage in the pipeline.
__init__(info, module, group, stage_to_host_topology)
Constructs the PipelineStage object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
info
|
PipelineStageInfo
|
The position of the stage in the pipeline. |
required |
module
|
Module
|
The module that runs this stage. |
required |
group
|
ProcessGroup
|
The pipeline-parallel process group. |
required |
stage_to_host_topology
|
dict[int, int]
|
Mapping from stage index to the pipeline-parallel rank that hosts it. |
required |
Raises:
| Type | Description |
|---|---|
TypeError
|
If the pipeline has more than one stage and the module does not implement
|
backward_one_chunk(microbatch_index, loss=None, full_backward=True)
Runs the backward pass for one microbatch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
microbatch_index
|
int
|
The microbatch index. |
required |
loss
|
Tensor | None
|
The loss tensor. Only the last stage uses it, and there it must be set. |
None
|
full_backward
|
bool
|
If |
True
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If the stage is not configured for backward passes, or if |
backward_weight_one_chunk(microbatch_index)
Runs the deferred weight backward for one microbatch.
backward_one_chunk(..., full_backward=False) must run first for this microbatch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
microbatch_index
|
int
|
The microbatch index. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the stage is not configured for backward passes. |
configure_buffers(has_backward, pipeline_inputs_per_microbatch)
Initializes the communication handlers and buffers for the stage.
It must run before the stage executes any microbatch. Transfer shapes are inferred per microbatch, so the microbatches in a pack can differ in shape.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
has_backward
|
bool
|
Whether the stage must prepare for a backward pass. |
required |
pipeline_inputs_per_microbatch
|
tuple[TPipelineInput, ...]
|
A |
required |
forward_one_chunk(microbatch_index, pipeline_inputs, pipeline_shared)
Runs the forward pass for one microbatch.
The first stage reads pipeline_inputs. Other stages read the received transfer. The
result is cached for the backward pass.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
microbatch_index
|
int
|
The microbatch index. |
required |
pipeline_inputs
|
TPipelineInput
|
The |
required |
pipeline_shared
|
TSharedInput
|
The |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the stage is not configured. |
get_bwd_recv_ops(microbatch_index)
get_bwd_send_ops(microbatch_index)
get_fwd_recv_ops(microbatch_index)
Returns P2P ops to receive forward inputs for the given microbatch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
microbatch_index
|
int
|
The microbatch identifier. |
required |
Returns:
| Type | Description |
|---|---|
list[P2POp]
|
The list of P2P operations (empty if this stage has no forward receiver). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the stage is not configured. |
get_fwd_send_ops(microbatch_index)
Returns P2P ops to send forward outputs for the given microbatch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
microbatch_index
|
int
|
The microbatch identifier. |
required |
Returns:
| Type | Description |
|---|---|
list[P2POp]
|
The list of P2P operations (empty if this stage has no forward sender). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the stage is not configured. |
get_pipeline_output(microbatch_index)
Returns the PipelineOutput this (last) stage produced, to hand to the callback.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
microbatch_index
|
int
|
The microbatch identifier. |
required |
Returns:
| Type | Description |
|---|---|
TPipelineOutput
|
The produced |
Raises:
| Type | Description |
|---|---|
ValueError
|
If called on a non-last stage, which produces a |
get_produced_transfer(microbatch_index)
Returns the StageTransfer this (non-last) stage produced, to hand to the next stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
microbatch_index
|
int
|
The microbatch identifier. |
required |
Returns:
| Type | Description |
|---|---|
TStageTransfer
|
The produced |
Raises:
| Type | Description |
|---|---|
ValueError
|
If called on the last stage, which produces a |
pop_local_bwd_output(microbatch_index)
Retrieves and releases the input gradients to hand to the previous stage on the same rank.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
microbatch_index
|
int
|
The microbatch identifier. |
required |
Returns:
| Type | Description |
|---|---|
TStageTransfer
|
The input gradients, shaped like the incoming |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the stage is not configured for backward passes. |
reset()
Releases the forward cache and live receive buffers of this stage.
set_local_bwd_input(inputs, microbatch_index)
Sets the output gradients from the next stage on the same rank.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs
|
TStageTransfer
|
The gradients of the |
required |
microbatch_index
|
int
|
The microbatch identifier. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the stage is not configured for backward passes, or has no backward receiver (last stage). |
set_local_fwd_input(inputs, microbatch_index)
Sets the forward inputs from the previous stage on the same rank.
V-shape schedules use it, because the previous stage can live on the same rank.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs
|
TStageTransfer
|
The |
required |
microbatch_index
|
int
|
The microbatch identifier. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the stage is not configured, or has no forward receiver (first stage). |
d9d.pipelining.runtime
Runtime that executes pipeline schedule programs.
ActionBase
Bases: ABC
Abstract base class for all pipeline schedule actions.
An action is an atomic unit of work in a pipeline schedule, such as computing a microbatch or sending a tensor.
has_backward_work
abstractmethod
property
Whether this action is part of the backward pass.
work_type
abstractmethod
property
The type of work this action performs.
__str__()
abstractmethod
Returns a short string form of the action for logs and visualization.
apply(ctx)
abstractmethod
Executes the action logic using the provided context.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ctx
|
ActionContext[TPipelineInput, TStageTransfer, TSharedInput, TPipelineOutput]
|
The runtime context containing stages, data, and communication handlers. |
required |
BackwardFullInputComputeAction
dataclass
Bases: ActionBase
Action that runs the backward pass of a microbatch, for the inputs and optionally the weights.
Attributes:
| Name | Type | Description |
|---|---|---|
stage_idx |
int
|
The index of the pipeline stage. |
microbatch_idx |
int
|
The index of the microbatch. |
full_backward |
bool
|
If |
BackwardReceiveAction
dataclass
Bases: ActionBase
Action that starts receiving the output gradients of a microbatch.
Attributes:
| Name | Type | Description |
|---|---|---|
stage_idx |
int
|
The index of the receiving pipeline stage. |
microbatch_idx |
int
|
The index of the microbatch. |
BackwardSendAction
dataclass
Bases: ActionBase
Action that starts sending the input gradients of a microbatch.
Attributes:
| Name | Type | Description |
|---|---|---|
stage_idx |
int
|
The index of the sending pipeline stage. |
microbatch_idx |
int
|
The index of the microbatch. |
BackwardWeightComputeAction
dataclass
Bases: ActionBase
Action that runs the deferred weight backward of a microbatch.
Attributes:
| Name | Type | Description |
|---|---|---|
stage_idx |
int
|
The index of the pipeline stage. |
microbatch_idx |
int
|
The index of the microbatch. |
ComposeAction
dataclass
Bases: ActionBase
Composite action that runs several sub-actions in order.
Schedules use it to mark a forward pass and a backward pass for overlap.
Attributes:
| Name | Type | Description |
|---|---|---|
actions |
tuple[ActionBase, ...]
|
A tuple of sub-actions to be executed sequentially. |
ForwardComputeAction
dataclass
Bases: ActionBase
Action that runs the forward pass of a microbatch.
Attributes:
| Name | Type | Description |
|---|---|---|
stage_idx |
int
|
The index of the pipeline stage. |
microbatch_idx |
int
|
The index of the microbatch. |
ForwardReceiveAction
dataclass
Bases: ActionBase
Action that starts receiving the forward inputs of a microbatch.
Attributes:
| Name | Type | Description |
|---|---|---|
stage_idx |
int
|
The index of the receiving pipeline stage. |
microbatch_idx |
int
|
The index of the microbatch. |
ForwardSendAction
dataclass
Bases: ActionBase
Action that starts sending the forward outputs of a microbatch.
Attributes:
| Name | Type | Description |
|---|---|---|
stage_idx |
int
|
The index of the sending pipeline stage. |
microbatch_idx |
int
|
The index of the microbatch. |
OfflinePipelineExecutor
Bases: PipelineSchedule[Any, Any, Any]
Executor that runs the model immediately without pipeline parallelism.
This schedule treats the model as a single stage. It runs the forward pass, and optionally the backward pass, for every microbatch in the pack. It serves single-device runs through the pipeline API.
PipelineScheduleExecutor
Bases: PipelineSchedule[Any, Any, Any]
Executor that runs a pipeline schedule as its sequence of actions.
__init__(pp_group, stages, program_builder)
Constructs the PipelineScheduleExecutor object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pp_group
|
ProcessGroup
|
The pipeline-parallel process group. |
required |
stages
|
list[PipelineStage]
|
List of stages managed by this executor. |
required |
program_builder
|
PipelineProgramBuilder
|
Builder that composes the per-rank action program for a microbatch count. |
required |
d9d.pipelining.program.builder
Building blocks and compiler passes that generate pipeline schedule programs.
PipelineProgramBuilder
Bases: ABC
Abstract base class for builders of pipeline execution schedules.
has_backward
abstractmethod
property
Whether the programs of this builder run backward passes.
num_stages_per_rank
abstractmethod
property
The number of model stages hosted on each rank.
topology_style
abstractmethod
property
The topology style used to assign stages to ranks.
compose(num_microbatches, pp_size)
abstractmethod
Generates the execution program for all ranks in the pipeline.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
num_microbatches
|
int
|
Number of microbatches per step. |
required |
pp_size
|
int
|
Number of pipeline-parallel ranks. |
required |
Returns:
| Type | Description |
|---|---|
dict[int, list[ActionBase]]
|
A dictionary mapping rank indices to their list of sequential actions. |
ScheduleStyle
Bases: StrEnum
Strategies for mapping logical stages to physical ranks.
Attributes:
| Name | Type | Description |
|---|---|---|
loop |
Assigns stages round-robin: stage |
|
v |
Assigns stages in a zig-zag V shape: rank 0 hosts the first and the last stage. Used by the Zero Bubble V and DualPipeV schedules. |
add_communication_ops(compute_actions, stage_to_rank, num_stages)
Injects communication actions into a computation-only schedule.
This function simulates the compute schedule. When a compute action produces a result that another rank needs, it adds a send and receive pair. Each receive comes before the computes that depend on it, so the schedule does not deadlock.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
compute_actions
|
dict[int, list[ActionBase]]
|
Initial schedule containing only compute operations. |
required |
stage_to_rank
|
dict[int, int]
|
Mapping from stage index to rank index. |
required |
num_stages
|
int
|
Total number of pipeline stages. |
required |
Returns:
| Type | Description |
|---|---|
dict[int, list[ActionBase]]
|
A new schedule dictionary including both compute and communication actions. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the schedule simulation deadlocks. |
build_stage_to_host_rank_topology(pp_size, num_stages, style)
Constructs the mapping from stage index to rank index.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pp_size
|
int
|
Number of pipeline-parallel ranks. |
required |
num_stages
|
int
|
Total number of model stages. |
required |
style
|
ScheduleStyle
|
The topology style to use for assignment. |
required |
Returns:
| Type | Description |
|---|---|
dict[int, int]
|
A dictionary mapping stage indices to rank indices. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the style is unknown, or if the style is |
invert_stage_to_host_rank_topology(stage_to_host)
Inverts the topology mapping to list execution stages per rank.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
stage_to_host
|
dict[int, int]
|
Mapping from stage index to rank index. |
required |
Returns:
| Type | Description |
|---|---|
dict[int, list[int]]
|
A dictionary mapping each rank index to the list of stage indices it hosts. |
d9d.pipelining.program.catalogue
Catalogue of ready pipeline schedule programs.
DualPipeVPipelineProgramBuilder
Bases: PipelineProgramBuilder
Builder for the DualPipeV pipeline parallelism schedule.
DualPipeV is a bidirectional pipeline schedule. It hosts exactly 2 stages per rank in a V shape. It splits some backward passes into input-gradient and weight-gradient parts to fill pipeline bubbles.
References
__init__()
Constructs the DualPipeVPipelineProgramBuilder object.
Interleaved1F1BPipelineProgramBuilder
Bases: PipelineProgramBuilder
Builder for interleaved pipeline parallelism schedules.
This builder supports:
- Standard Interleaved 1F1B: Hosts several stages per rank and prefers depth-first execution. (See https://arxiv.org/abs/2104.04473)
- Interleaved Zero Bubble (ZB1P): Extends 1F1B by splitting the backward pass into input-gradient and weight-gradient parts. Weight gradients are deferred to fill pipeline bubbles. (See https://arxiv.org/abs/2401.10241)
__init__(num_stages_per_rank, enable_zero_bubble=False)
compose(num_microbatches, pp_size)
Generates the execution program for all ranks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
num_microbatches
|
int
|
Number of microbatches per step. It must be divisible by the number of
rounds, |
required |
pp_size
|
int
|
Number of pipeline ranks. |
required |
Returns:
| Type | Description |
|---|---|
dict[int, list[ActionBase]]
|
A dictionary mapping rank indices to their list of sequential actions. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
LoopedBFSPipelineProgramBuilder
Bases: PipelineProgramBuilder
Builder for the Looped Breadth-First pipeline parallelism schedule.
This schedule first runs the forward pass of all microbatches for the local stages. For training, it then runs the backward passes in reverse order.
References
__init__(num_stages_per_rank, inference_mode=False)
ZeroBubbleVPipelineProgramBuilder
Bases: PipelineProgramBuilder
Builder for the Zero Bubble V (ZBV) pipeline schedule.
This schedule hosts exactly two stages per rank in a V shape. It splits the backward pass into input-gradient and weight-gradient parts to fill pipeline bubbles.
References
Zero Bubble Pipeline Parallelism, Section 6
__init__()
Constructs the ZeroBubbleVPipelineProgramBuilder object.