Gradient Synchronization
About
The d9d.parallel.grad_sync package synchronizes gradients of DTensor parameters in distributed training. PyTorch DistributedDataParallel applies one communication strategy to the whole model. This package instead supports the mixed layouts of ND parallelism, which combines data, tensor, context and pipeline parallelism. It reads the DTensor placements of each parameter to find the mesh dimensions that need an all-reduce. Then it groups the parameters into communication buckets.
Training loop
The d9d training loop synchronizes the gradients itself. Call this package directly when you train without the loop.
Bucketing and Flattening
When many small tensors are reduced, latency dominates the communication cost. So GradientSynchronizer groups parameters into buckets.
A SyncGradientBucket stores the gradients of all its parameters in one contiguous buffer. Its reduction is a single all_reduce per process group on this buffer, instead of hundreds of small operations.
Parameters share a bucket only if they have the same:
- Optimizer parameter group
- Device
- Gradient dtype
- Reduce mesh: The mesh dimensions where the parameter has a
Replicateplacement.
The bucket_size_mb argument limits the size of a bucket in MiB. The limit counts the local shard of each gradient in the gradient dtype. Parameters are bucketed in reverse order, because the backward pass produces gradients roughly in that order. Parameters with no Replicate placement need no reduction. They go to local buckets that do not communicate.
Asynchronous Reduction
Training often accumulates gradients over several microbatches before an optimizer step. This package manages the DTensor gradients during the accumulation. You do not need a no_sync context manager.
-
Local accumulation: During the backward pass of the first \(N-1\) microbatches, local gradients accumulate in the bucket buffer. The parameter
DTensorisReplicate, so the gradient also has aReplicateplacement across the data parallel mesh. Its data still differs between ranks at this point. -
Automatic trigger: Each bucket counts the gradient accumulations of its parameters. The
all_reducestarts only when all parameters of the bucket reach therequire_accumulationscount. The trigger runs inside the backward hook of the last microbatch. So the communication overlaps with the backward pass of the remaining layers. The communication runs on a separate CUDA stream. You must wait for it before you use the gradients on your default stream. -
Fixed order: Collectives are matched by call order, so all ranks must start them in the same order. The backward pass can make buckets ready in a different order on each rank. So a ready bucket starts its
all_reduceonly after all buckets before it have started theirs. Buckets are ordered by their first parameter in reverse parameter order across all parameter groups, which is the expected backward order. The gradients of one bucket can be written on several CUDA streams, e.g. by several FSDP roots. Theall_reducewaits for the writes of all these streams. -
Synchronization: When the asynchronous reduction completes, the flat buffer holds the globally summed gradient. The gradients have the same
Replicateplacement as their parameters. So the optimizer can use them without any further synchronization.
Limits
- Owned gradients: After
bind(), each gradient of a parameter that needs a reduction is a view into the buffer of its bucket. Do not set gradients toNone, e.g. withoptimizer.zero_grad(). Callzero_grad()of the synchronizer instead. A backward pass into a replaced gradient raises an error. - Exact accumulation count: Each parameter must get exactly
require_accumulationsgradients per step. An extra backward pass raises an error. A missing one makeswait()raise an error. - No unused parameters: Every parameter that requires a gradient must get one in each backward pass. Otherwise its bucket never starts the reduction and
wait()raises an error. If only some ranks skip the parameter, the other ranks wait in theall_reduceuntil the collective timeout. - No shared parameters: A parameter can appear only once in the parameter groups. Tied weights in two parameter groups raise an error in
bind(). - Replicate and Shard only: A
DTensorparameter with another placement, e.g.Partial, raises an error inbind().
API Reference
d9d.parallel.grad_sync
Gradient bucketing and asynchronous gradient reduction for DTensor parameters.
It works like DistributedDataParallel, but reduces over the replicated dimensions of the placements.
GradientSynchronizer
Gradient synchronizer for replicated parameters. It reduces the gradients during the backward pass.
It splits the parameters into buckets, allocates flat gradient buffers and runs asynchronous all-reduce operations.
__init__(param_groups, bucket_size_mb)
bind()
Builds the buckets, allocates their gradient buffers and registers the backward hooks.
It must be called before the backward pass.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the synchronizer is already bound, if a parameter appears more than once in the
parameter groups, or if a |
scale_grads(factor)
Multiplies the gradients of all managed parameters in place.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
factor
|
Tensor
|
The scalar to multiply the gradients by. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the synchronizer is not bound. |
set_required_accumulations(require_accumulations)
Sets the accumulation count of the current step for all buckets.
The count can change from step to step, because the pack length varies. Later binds, for example after offload and onload, reuse the latest count.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
require_accumulations
|
int
|
The number of accumulations required before the gradients are reduced. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
unbind()
Destroys the buckets, frees their buffers and removes the hooks.
wait()
Makes the current stream wait for all asynchronous reductions and marks the buckets as synchronized.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the synchronizer is not bound, or if some bucket has not started its reduction. |
zero_grad()
Resets gradients and accumulation counters for all managed parameters.