Split Backward
About
The d9d.autograd.split_backward package splits a backward pass into two phases: the gradients of the inputs and the gradients of the weights. Split-backward pipeline schedules, such as Zero Bubble, run the two phases at different times. The package has two parts. The split backward functions run the phases. The global grad context tells custom autograd functions which gradients the current phase needs.
The Two Phases
A pipeline stage sends the gradients of its inputs to the previous stage. The weight gradients are needed only at the optimizer step. So a schedule computes the input gradients first and fills the idle time of the pipeline with the weight gradients later.
stage_backward_input(): Computes the gradients of the inputs and defers the weight gradients. It captures the gradients at the nodes of the autograd graph where the weight gradients branch off.stage_backward_weight(): Replays the captured gradients and accumulates the weight gradients.stage_backward_full(): Runs both phases in one backward pass.
The PendingWeightBackward from stage_backward_input() keeps the autograd graph alive. Keep it until you call stage_backward_weight().
The Global Grad Context
Why
Split-backward pipeline schedules compute activation gradients and weight gradients at different times. They run a backward pass for a subset of tensors, e.g. torch.autograd.backward(..., inputs=[activations]).
For built-in operations such as torch.matmul, PyTorch then skips the weight gradients.
Custom torch.autograd.Function implementations do not get this behavior. PyTorch sets ctx.needs_input_grad to True for every input with requires_grad=True. It does so even if the current backward() call does not compute that edge.
So a custom operation, such as a grouped GEMM, computes all its gradients in every partial backward pass. This wastes compute in split-backward schedules.
For more details, see PyTorch Issue #174017.
How It Works
d9d adds the GlobalGradContext to work around this limitation. It is shared state through which the training loop tells custom operations which gradients it needs.
- Training loop: Sets the enabled gradient directions, e.g. "input gradients only".
- Operation: The custom
backwardcallsneeds_input_grad(). It computes a gradient only if autograd needs it and the context enables its direction.
By default, GLOBAL_GRAD_CONTEXT enables both input and weight gradients.
Usage
In Custom Autograd Functions
When you write a custom operation, assign a GradDirection to each input. Call needs_input_grad() instead of ctx.needs_input_grad before you compute a gradient.
In Training Loops
The d9d pipelining schedules run the split backward pass for you. If you use them, directly or through the Trainer, you do not need to do anything.
To split the backward pass in your own schedule, call the two phases. They set the global grad context themselves:
If you run the backward pass without these functions, set the context yourself:
API Reference
d9d.autograd.split_backward
Split backward passes for input and weight gradients, and the context that custom autograd functions check.
GLOBAL_GRAD_CONTEXT = GlobalGradContext()
module-attribute
The singleton GlobalGradContext.
Custom autograd functions check it in their backward pass through needs_input_grad().
GlobalGradContext
Global state that controls gradient computation in custom autograd functions.
PyTorch sets ctx.needs_input_grad to True for every edge that requires grad in a custom
torch.autograd.Function. It does so even in a partial backward pass, such as
torch.autograd.backward(inputs=...). See the
related issue.
This class works around the limitation:
- Training code sets which gradient edges (inputs or weights) to compute now.
- Module code checks whether it must compute a gradient edge, and skips the work otherwise.
__init__()
Constructs the GlobalGradContext object with both directions enabled.
check_direction(direction)
Checks whether gradient computation for the given direction is enabled.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
direction
|
GradDirection | None
|
The direction to check. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
|
with_directions(*directions)
Enables only the given gradient directions inside the context.
The previous directions are restored when the context exits, even if the block raises an exception.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*directions
|
GradDirection
|
The gradient directions to enable. |
()
|
GradDirection
Bases: StrEnum
Gradient edges that a custom autograd function can compute.
Custom autograd functions use it to skip gradient work during split backward passes.
Attributes:
| Name | Type | Description |
|---|---|---|
inputs |
The gradient edges to the module inputs (activations). |
|
weight |
The gradient edges to the module parameters (weights). |
ParamGroup
dataclass
Group of parameters and their dependency intermediates in the autograd graph.
The split backward pass uses it to find the intermediate nodes through which gradients flow to a set of parameters.
Attributes:
| Name | Type | Description |
|---|---|---|
params |
set[Node]
|
The gradient accumulator nodes of the parameters. |
intermediates |
list[Node] | None
|
The nodes through which gradients enter these parameters, or |
grads |
list[tuple[Tensor | None, ...] | None] | None
|
The gradients captured at each intermediate node during the input backward. Each entry holds one gradient per output of the node. |
PendingWeightBackward
dataclass
The weight backward pass that stage_backward_input deferred.
Pass it to stage_backward_weight once. It keeps the autograd graph alive until then.
Attributes:
| Name | Type | Description |
|---|---|---|
weights |
list[Parameter]
|
The weights to accumulate the gradients into. |
param_groups |
list[ParamGroup]
|
The parameter groups with the gradients captured for the weight backward. |
independent_output_grads |
list[tuple[GradientEdge, Tensor]]
|
The gradient edges and the gradients of the outputs that do not depend on the inputs. |
outputs |
list[Tensor]
|
The outputs of the forward pass. |
done |
bool
|
Whether |
needs_input_grad(ctx, index, direction)
Checks whether the backward pass of a custom autograd function must compute the gradient of an input.
Use it in backward instead of ctx.needs_input_grad, so that split backward passes skip the gradients of
the other direction.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ctx
|
BackwardCFunction
|
The autograd context passed to |
required |
index
|
int
|
The position of the input in the arguments of |
required |
direction
|
GradDirection
|
The gradient direction of the input. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
|
stage_backward_full(outputs, output_grads, inputs)
Runs a full backward pass for a pipeline stage.
It computes the gradients of the inputs and accumulates the weight gradients.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
outputs
|
list[Tensor]
|
The output tensors of the forward pass. |
required |
output_grads
|
list[Tensor] | None
|
The gradients for |
required |
inputs
|
list[Tensor]
|
The input tensors of the forward pass to return gradients for. |
required |
Returns:
| Type | Description |
|---|---|
list[Tensor | None]
|
The gradients of |
list[Tensor | None]
|
gradient. |
stage_backward_input(outputs, output_grads, inputs, weights)
Runs the first phase of a split backward pass: the input gradients.
This function computes the gradients of inputs and defers the gradients of weights. It
captures the gradients at the intermediate nodes where the weight gradients branch off. The
second phase, stage_backward_weight, replays them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
outputs
|
list[Tensor]
|
The output tensors of the forward pass. |
required |
output_grads
|
list[Tensor] | None
|
The gradients arriving for the outputs. |
required |
inputs
|
list[Tensor]
|
The input tensors from the forward pass. |
required |
weights
|
Iterable[Parameter]
|
The model parameters (weights). |
required |
Returns:
| Type | Description |
|---|---|
list[Tensor | None]
|
The gradients of |
PendingWeightBackward
|
input that got no gradient. |
stage_backward_weight(pending)
Runs the second phase of a split backward pass: the weight gradients.
It accumulates the gradients of the weights that stage_backward_input deferred.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pending
|
PendingWeightBackward
|
The weight backward pass from |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |