Skip to content

Split Backward

About

The d9d.autograd.split_backward package splits a backward pass into two phases: the gradients of the inputs and the gradients of the weights. Split-backward pipeline schedules, such as Zero Bubble, run the two phases at different times. The package has two parts. The split backward functions run the phases. The global grad context tells custom autograd functions which gradients the current phase needs.

The Two Phases

A pipeline stage sends the gradients of its inputs to the previous stage. The weight gradients are needed only at the optimizer step. So a schedule computes the input gradients first and fills the idle time of the pipeline with the weight gradients later.

  • stage_backward_input(): Computes the gradients of the inputs and defers the weight gradients. It captures the gradients at the nodes of the autograd graph where the weight gradients branch off.
  • stage_backward_weight(): Replays the captured gradients and accumulates the weight gradients.
  • stage_backward_full(): Runs both phases in one backward pass.

The PendingWeightBackward from stage_backward_input() keeps the autograd graph alive. Keep it until you call stage_backward_weight().

The Global Grad Context

Why

Split-backward pipeline schedules compute activation gradients and weight gradients at different times. They run a backward pass for a subset of tensors, e.g. torch.autograd.backward(..., inputs=[activations]).

For built-in operations such as torch.matmul, PyTorch then skips the weight gradients.

Custom torch.autograd.Function implementations do not get this behavior. PyTorch sets ctx.needs_input_grad to True for every input with requires_grad=True. It does so even if the current backward() call does not compute that edge.

So a custom operation, such as a grouped GEMM, computes all its gradients in every partial backward pass. This wastes compute in split-backward schedules.

For more details, see PyTorch Issue #174017.

How It Works

d9d adds the GlobalGradContext to work around this limitation. It is shared state through which the training loop tells custom operations which gradients it needs.

  1. Training loop: Sets the enabled gradient directions, e.g. "input gradients only".
  2. Operation: The custom backward calls needs_input_grad(). It computes a gradient only if autograd needs it and the context enables its direction.

By default, GLOBAL_GRAD_CONTEXT enables both input and weight gradients.

Usage

In Custom Autograd Functions

When you write a custom operation, assign a GradDirection to each input. Call needs_input_grad() instead of ctx.needs_input_grad before you compute a gradient.

import torch
from d9d.autograd.split_backward import GradDirection, needs_input_grad


class MyCustomOp(torch.autograd.Function):
    @staticmethod
    def forward(ctx, inputs, weight):
        ctx.save_for_backward(inputs, weight)
        return torch.matmul(inputs, weight)

    @staticmethod
    def backward(ctx, grad_output):
        inputs, weight = ctx.saved_tensors
        grad_input = grad_weight = None

        if needs_input_grad(ctx, 0, GradDirection.inputs):
            grad_input = torch.matmul(grad_output, weight.t())

        if needs_input_grad(ctx, 1, GradDirection.weight):
            grad_weight = torch.matmul(inputs.t(), grad_output)

        return grad_input, grad_weight

In Training Loops

The d9d pipelining schedules run the split backward pass for you. If you use them, directly or through the Trainer, you do not need to do anything.

To split the backward pass in your own schedule, call the two phases. They set the global grad context themselves:

1
2
3
4
5
6
7
from d9d.autograd.split_backward import stage_backward_input, stage_backward_weight

# Phase 1: the input gradients. Send them to the previous stage, and keep `pending` until phase 2.
input_grads, pending = stage_backward_input(outputs, output_grads, inputs, weights=model.parameters())

# Phase 2, later in the schedule: the weight gradients.
stage_backward_weight(pending)

If you run the backward pass without these functions, set the context yourself:

1
2
3
4
5
import torch
from d9d.autograd.split_backward import GLOBAL_GRAD_CONTEXT, GradDirection

with GLOBAL_GRAD_CONTEXT.with_directions(GradDirection.inputs):
    torch.autograd.backward(outputs, grad_tensors=output_grads, inputs=stage_inputs, retain_graph=True)

API Reference

d9d.autograd.split_backward

Split backward passes for input and weight gradients, and the context that custom autograd functions check.

GLOBAL_GRAD_CONTEXT = GlobalGradContext() module-attribute

The singleton GlobalGradContext.

Custom autograd functions check it in their backward pass through needs_input_grad().

GlobalGradContext

Global state that controls gradient computation in custom autograd functions.

PyTorch sets ctx.needs_input_grad to True for every edge that requires grad in a custom torch.autograd.Function. It does so even in a partial backward pass, such as torch.autograd.backward(inputs=...). See the related issue.

This class works around the limitation:

  1. Training code sets which gradient edges (inputs or weights) to compute now.
  2. Module code checks whether it must compute a gradient edge, and skips the work otherwise.
__init__()

Constructs the GlobalGradContext object with both directions enabled.

check_direction(direction)

Checks whether gradient computation for the given direction is enabled.

Parameters:

Name Type Description Default
direction GradDirection | None

The direction to check.

required

Returns:

Type Description
bool

True if the direction is enabled or direction is None, False otherwise.

with_directions(*directions)

Enables only the given gradient directions inside the context.

The previous directions are restored when the context exits, even if the block raises an exception.

Parameters:

Name Type Description Default
*directions GradDirection

The gradient directions to enable.

()

GradDirection

Bases: StrEnum

Gradient edges that a custom autograd function can compute.

Custom autograd functions use it to skip gradient work during split backward passes.

Attributes:

Name Type Description
inputs

The gradient edges to the module inputs (activations).

weight

The gradient edges to the module parameters (weights).

ParamGroup dataclass

Group of parameters and their dependency intermediates in the autograd graph.

The split backward pass uses it to find the intermediate nodes through which gradients flow to a set of parameters.

Attributes:

Name Type Description
params set[Node]

The gradient accumulator nodes of the parameters.

intermediates list[Node] | None

The nodes through which gradients enter these parameters, or None after the weight backward consumed them.

grads list[tuple[Tensor | None, ...] | None] | None

The gradients captured at each intermediate node during the input backward. Each entry holds one gradient per output of the node.

PendingWeightBackward dataclass

The weight backward pass that stage_backward_input deferred.

Pass it to stage_backward_weight once. It keeps the autograd graph alive until then.

Attributes:

Name Type Description
weights list[Parameter]

The weights to accumulate the gradients into.

param_groups list[ParamGroup]

The parameter groups with the gradients captured for the weight backward.

independent_output_grads list[tuple[GradientEdge, Tensor]]

The gradient edges and the gradients of the outputs that do not depend on the inputs.

outputs list[Tensor]

The outputs of the forward pass.

done bool

Whether stage_backward_weight already ran it.

needs_input_grad(ctx, index, direction)

Checks whether the backward pass of a custom autograd function must compute the gradient of an input.

Use it in backward instead of ctx.needs_input_grad, so that split backward passes skip the gradients of the other direction.

Parameters:

Name Type Description Default
ctx BackwardCFunction

The autograd context passed to backward.

required
index int

The position of the input in the arguments of forward, without ctx.

required
direction GradDirection

The gradient direction of the input.

required

Returns:

Type Description
bool

True if autograd needs the gradient of the input and GLOBAL_GRAD_CONTEXT enables its direction.

stage_backward_full(outputs, output_grads, inputs)

Runs a full backward pass for a pipeline stage.

It computes the gradients of the inputs and accumulates the weight gradients.

Parameters:

Name Type Description Default
outputs list[Tensor]

The output tensors of the forward pass.

required
output_grads list[Tensor] | None

The gradients for outputs from the next pipeline stage. If None, outputs must be scalars, and autograd seeds them with an implicit unit gradient.

required
inputs list[Tensor]

The input tensors of the forward pass to return gradients for.

required

Returns:

Type Description
list[Tensor | None]

The gradients of inputs, in order. The entry is None for an input that got no

list[Tensor | None]

gradient.

stage_backward_input(outputs, output_grads, inputs, weights)

Runs the first phase of a split backward pass: the input gradients.

This function computes the gradients of inputs and defers the gradients of weights. It captures the gradients at the intermediate nodes where the weight gradients branch off. The second phase, stage_backward_weight, replays them.

Parameters:

Name Type Description Default
outputs list[Tensor]

The output tensors of the forward pass.

required
output_grads list[Tensor] | None

The gradients arriving for the outputs.

required
inputs list[Tensor]

The input tensors from the forward pass.

required
weights Iterable[Parameter]

The model parameters (weights).

required

Returns:

Type Description
list[Tensor | None]

The gradients of inputs, in order, and the deferred weight backward pass. The entry is None for an

PendingWeightBackward

input that got no gradient.

stage_backward_weight(pending)

Runs the second phase of a split backward pass: the weight gradients.

It accumulates the gradients of the weights that stage_backward_input deferred.

Parameters:

Name Type Description Default
pending PendingWeightBackward

The weight backward pass from stage_backward_input.

required

Raises:

Type Description
ValueError

If pending already ran, or if no gradient was captured for an intermediate node of a group.