Skip to content

Distributed Context

About

The d9d.dist_context package holds the distributed state of a process. Its DistributedContext knows the rank, the GPU of the rank, the process groups and the device meshes of the domains. Every component that needs these facts gets the context through its constructor. Do not read raw os.environ variables or create ad-hoc process groups: then the components can disagree about the topology. DistributedContext requires CUDA GPUs.

Creating and Closing the Context

init_dist_context() creates the context from DeviceMeshParameters, once per process. In a distributed job, every rank calls it with the same arguments. It takes the world size and the rank from the default process group if one exists, and otherwise from WORLD_SIZE and RANK of the launcher. Without a launcher, the job has one rank and runs with python. The product of the four parallelism degrees must equal the world size.

init_dist_context() is a context manager, and the context is valid only inside its with block. On exit, it destroys the groups that it created, including the default process group. If the block raises, the groups stay, so the job fails instead of hanging.

In a distributed job, the block also replaces the DTensor RNG tracker of the process. The default tracker of PyTorch broadcasts the generator state of rank 0, which would overwrite the seeds of the other pipeline stages. The previous tracker comes back on exit, also after an error.

A job takes the context and never closes it. TrainingConfigurator and InferenceConfigurator take it as dist_context, so one context can serve several jobs of a process, e.g. training and then evaluation.

GPU Assignment

The degrees describe the ranks, not the GPUs. The same configuration runs on 8 GPUs and on 2, so the context takes the GPU of each rank from the launch:

  • Local rank: It comes from LOCAL_RANK, which torchrun sets. It is the index of the rank on its node, whatever its GPU.
  • GPU: The node runs LOCAL_WORLD_SIZE ranks on torch.cuda.device_count() visible GPUs. If the ranks do not outnumber the GPUs, local rank i runs on GPU i. Otherwise each GPU runs an equal block of consecutive local ranks. E.g. with 8 ranks on 2 GPUs, local ranks 4 to 7 run on GPU 1. The context raises if the counts do not divide.

Several ranks on one GPU let you check a parallel layout on fewer GPUs than it targets. They need NCCL_MULTI_RANK_GPU_ENABLE=1 from the launcher. CONTRIBUTING.md shows the setup for the tests of d9d.

Parallelism Degrees

The work of one step splits in four ways. DeviceMeshParameters holds one degree for each:

  • pipeline_parallel: It splits the layers into stages.
  • data_parallel: It splits the samples.
  • context_parallel: It splits one sample into parts, e.g. a text into chunks of the sequence, a graph into subgraphs or an image into tiles.
  • tensor_parallel: It splits the matrix multiplications inside one sample.

Other kinds of parallelism split the same ranks differently, so they do not add to the world size. Each built-in domain that does so has its own section:

  • dense: The sizes of the shard groups of the dense layers, data_parallel_shard and context_parallel_shard. They default to 1, so the dense layers do not shard, as in DDP. The number of replicas follows from the degree, so a shard size that fits the memory of a GPU stays when the job grows.
  • expert: The degree of expert parallelism, expert_parallel. Without this section, the job builds no expert domain.

The shard sizes must divide data_parallel and context_parallel, and expert_parallel must divide data_parallel * context_parallel.

Domains

Different parts of a model need different parallelism, e.g. dense layers and Mixture-of-Experts (MoE) layers. d9d describes each arrangement of the ranks as a domain. All domains hold the same ranks, and each arranges them into its own device mesh. The context keeps each mesh under its domain object: dist_context.mesh_for(DENSE_DOMAIN).

Domain Dimensions Purpose
BASE_DOMAIN BaseDim: PP, DP, CP, TP Pipeline stages, data and the coordinates of a rank
DENSE_DOMAIN DenseDim: PP, DP_REPLICATE, DP_CP_SHARD, CP_REPLICATE, TP Dense layers
EXPERT_DOMAIN ExpertDim: PP, EP_REPLICATE, EP_SHARD, TP Expert layers
METRIC_DOMAIN MetricDim: PP_DP_CP, TP Sums of the metrics and the loss weights
  • Base domain: Its dimensions are the four degrees. Their order defines how the ranks lie in the world: the ranks of one tensor-parallel group are neighbors.
  • Dense domain: It splits dp into DP_REPLICATE and a shard, and cp into a shard and CP_REPLICATE, as the dense section says. DP_CP_SHARD merges the two shards.
  • Expert domain: The context builds it only if the expert section is set. Otherwise mesh_for(EXPERT_DOMAIN) raises and asks to set the section. EP_REPLICATE and EP_SHARD split the ranks of dp * cp. So an expert-parallel group never holds two ranks of one tensor-parallel group, which hold the same tokens.
  • Metric domain: PP_DP_CP holds all ranks with their own tokens or pipeline stage. The gradient manager and the metrics sum over it, so the ranks of one tensor-parallel group count once.

Each domain names its dimensions with a StrEnum, e.g. DenseDim.DP_REPLICATE, so the type checker catches a dimension of another domain.

Coordinates and Communication

DistributedContext answers four kinds of questions. Only communication needs process groups:

Kind Members In a job without parallelism
Rank rank, world_size, is_main_process, local_rank, is_local_main_process, current_device Rank 0 of 1
Coordinates coordinate(), dim_size() Coordinate 0, size 1
Communication is_distributed, mesh_for(), wait_world(), main_process_first(), local_main_process_first(), set_timeout() is_distributed is False and mesh_for() raises. set_timeout() does nothing. The others only wait for the GPU.

Code that communicates checks is_distributed before it calls mesh_for().

Extra Domains

A new kind of parallelism adds a domain that splits the ranks differently. For example, an image model can split the cp ranks into height and width. Subclass DeviceMeshDomain and return the dimensions of the mesh from mesh_dims(), outermost first. The context lays out the ranks along them row-major, as init_device_mesh() does. So dimensions that keep the order of the base dimensions group the ranks as they do. A domain takes its own degrees in its constructor.

Pass the domain in extra_domains of init_dist_context(). The sizes of its dimensions must be positive and multiply to the world size, and the names of its dimensions must be unique. The names of all domains must be unique too. mesh_for() looks the mesh up by the domain object, so a domain object that was not passed raises, even if its name matches.

Usage

Configuration

A job with pipeline parallelism, HSDP inside each node and expert parallelism:

1
2
3
4
5
6
7
mesh:
  pipeline_parallel: 2
  data_parallel: 16
  dense:
    data_parallel_shard: 8
  expert:
    expert_parallel: 8

Creating the Context

from d9d.dist_context import DeviceMeshParameters, init_dist_context
from d9d.loop.run import TrainingConfigurator

params = DeviceMeshParameters(data_parallel=8)

with init_dist_context(params) as dist_context:
    trainer = TrainingConfigurator(
        dist_context=dist_context,
        # The parameters and the providers of the job.
    ).configure()
    trainer.train()

Coordinates and Meshes

from d9d.dist_context import BASE_DOMAIN, DENSE_DOMAIN, BaseDim, DenseDim, DistributedContext

dist_context: DistributedContext = ...

# Works with and without parallelism.
dp_rank = dist_context.coordinate(BASE_DOMAIN, BaseDim.DP)
dp_size = dist_context.dim_size(BASE_DOMAIN, BaseDim.DP)

if dist_context.is_distributed:
    shard_mesh = dist_context.mesh_for(DENSE_DOMAIN)[DenseDim.DP_CP_SHARD]

An Extra Domain

from enum import StrEnum

from d9d.dist_context import DeviceMeshDomain, DeviceMeshParameters, MeshDim, init_dist_context


class DataContextParallelDim(StrEnum):
    PP = "pp"
    DP_CP = "dp_cp"
    TP = "tp"


class DataContextParallelDomain(DeviceMeshDomain[DataContextParallelDim]):
    @property
    def name(self) -> str:
        return "dp_cp"

    def mesh_dims(self, params: DeviceMeshParameters) -> tuple[MeshDim[DataContextParallelDim], ...]:
        return (
            MeshDim(DataContextParallelDim.PP, params.pipeline_parallel),
            MeshDim(DataContextParallelDim.DP_CP, params.data_parallel * params.context_parallel),
            MeshDim(DataContextParallelDim.TP, params.tensor_parallel),
        )


DP_CP_DOMAIN = DataContextParallelDomain()

with init_dist_context(params, extra_domains=(DP_CP_DOMAIN,)) as dist_context:
    shard_index = dist_context.coordinate(DP_CP_DOMAIN, DataContextParallelDim.DP_CP)

Ordering the Ranks

1
2
3
4
5
6
7
8
9
# Only one process per node downloads the file.
with dist_context.local_main_process_first():
    if dist_context.is_local_main_process:
        download_dataset()
    # Other ranks enter the block after the download.
# All ranks continue together.

# Wait for all ranks.
dist_context.wait_world()

API Reference

d9d.dist_context

The distributed context of a job and the device meshes of its domains.

BASE_DOMAIN = _BaseDomain() module-attribute

The domain of the four parallelism degrees. The order of its dimensions defines how the ranks lie in the world.

DENSE_DOMAIN = _DenseDomain() module-attribute

The domain of the dense layers, which splits the data-parallel and context-parallel ranks into replicas and shards.

EXPERT_DOMAIN = _ExpertDomain() module-attribute

The domain of the expert layers. The context builds it only if the expert section of the configuration is set.

METRIC_DOMAIN = _MetricDomain() module-attribute

The domain that sums the metrics and the loss weights, so that the tensor-parallel ranks count once.

BaseDim

Bases: StrEnum

The dimensions of the mesh of BASE_DOMAIN.

Attributes:

Name Type Description
PP

The pipeline stages.

DP

The data-parallel ranks, which take different samples.

CP

The context-parallel ranks, which take different parts of one sample.

TP

The tensor-parallel ranks, which split the matrix multiplications of one sample.

DenseDim

Bases: StrEnum

The dimensions of the mesh of DENSE_DOMAIN.

Attributes:

Name Type Description
PP

The pipeline stages.

DP_REPLICATE

The data-parallel ranks that hold the same shards of the dense parameters.

DP_CP_SHARD

The data-parallel and context-parallel ranks that shard the dense parameters.

CP_REPLICATE

The context-parallel ranks that hold the same shards of the dense parameters.

TP

The tensor-parallel ranks.

DenseDomainConfig

Bases: BaseModel

Configuration for the dense domain, which shards the parameters of dense layers.

Attributes:

Name Type Description
data_parallel_shard PositiveInt

The number of data-parallel ranks that shard a parameter.

context_parallel_shard PositiveInt

The number of context-parallel ranks that shard a parameter.

DeviceMeshDomain

Bases: ABC, Generic[DimT]

Abstract base class for a domain: one arrangement of all ranks into a device mesh.

name abstractmethod property

The name of this domain, unique within a context.

mesh_dims(params) abstractmethod

Returns the dimensions of the mesh of this domain, outermost first.

The context lays out the ranks along them row-major, as init_device_mesh() does. So a dimension that keeps the order of the dimensions of BASE_DOMAIN groups the ranks as they do.

Parameters:

Name Type Description Default
params DeviceMeshParameters

The parallelism degrees and the sections of the built-in domains.

required

Returns:

Type Description
tuple[MeshDim[DimT], ...]

The dimensions of the mesh.

DeviceMeshParameters

Bases: BaseModel

Configuration for the parallelism degrees of a job.

Attributes:

Name Type Description
pipeline_parallel PositiveInt

The degree of pipeline parallelism.

data_parallel PositiveInt

The degree of data parallelism.

context_parallel PositiveInt

The degree of context parallelism.

tensor_parallel PositiveInt

The degree of tensor parallelism.

dense DenseDomainConfig

The configuration of the dense domain.

expert ExpertDomainConfig | None

The configuration of the expert domain. None means that the job builds no expert domain.

DistributedContext

The distributed state of one process: its rank, its GPU, its process groups and the meshes of its domains.

Create it with init_dist_context(). It is valid only inside the with block of init_dist_context().

current_device property

The GPU of this rank, with its index.

is_distributed property

Whether the job has more than one rank.

is_local_main_process property

Whether the current process is local rank 0 on its node.

is_main_process property

Whether the current process is global rank 0.

local_rank property

The index of this rank on its node.

rank property

The global rank of this process.

world_size property

The number of ranks in the job.

__init__(rank, world_size, local_rank, device, domain_dims, meshes, groups)

Constructs the DistributedContext object.

Call init_dist_context() instead, which builds the arguments.

Parameters:

Name Type Description Default
rank int

The global rank of this process.

required
world_size int

The number of ranks in the job.

required
local_rank int

The index of this rank on its node.

required
device device

The GPU of this rank.

required
domain_dims Mapping[DeviceMeshDomain, tuple[MeshDim, ...]]

The dimensions of the mesh of every domain of the context.

required
meshes Mapping[DeviceMeshDomain, DeviceMesh]

The mesh of every domain. Empty in a job without parallelism.

required
groups list[ProcessGroup]

The process groups of the meshes, without the default process group.

required
coordinate(domain, dim)

Returns the coordinate of this rank along a dimension of the mesh of a domain.

Parameters:

Name Type Description Default
domain DeviceMeshDomain[DimT]

A domain of this context.

required
dim DimT

The name of a dimension of the domain.

required

Returns:

Type Description
int

The coordinate, 0 in a job without parallelism.

Raises:

Type Description
ValueError

If the context has no such domain or the domain has no such dimension.

dim_size(domain, dim)

Returns the number of ranks along a dimension of the mesh of a domain, 1 in a job without parallelism.

Parameters:

Name Type Description Default
domain DeviceMeshDomain[DimT]

A domain of this context.

required
dim DimT

The name of a dimension of the domain.

required

Returns:

Type Description
int

The number of ranks along the dimension.

Raises:

Type Description
ValueError

If the context has no such domain or the domain has no such dimension.

local_main_process_first()

Runs the block on the local main processes first.

Other ranks wait before entering the block. Local main processes wait after leaving it, so all ranks continue together.

main_process_first()

Runs the block on the global main process first.

Other ranks wait before entering the block. The global main process waits after leaving it, so all ranks continue together.

mesh_for(domain)

Returns the device mesh of a domain.

Parameters:

Name Type Description Default
domain DeviceMeshDomain

A domain of this context.

required

Returns:

Type Description
DeviceMesh

The device mesh of the domain.

Raises:

Type Description
ValueError

If the job has no parallelism or the context has no such domain.

set_timeout(timeout_seconds)

Sets the timeout of the default process group and of all process groups of the context.

Does nothing in a job without parallelism.

Parameters:

Name Type Description Default
timeout_seconds float

The new timeout in seconds.

required
wait_world()

Blocks until all ranks reach this point and the GPU of this rank finishes its queued work.

ExpertDim

Bases: StrEnum

The dimensions of the mesh of EXPERT_DOMAIN.

Attributes:

Name Type Description
PP

The pipeline stages.

EP_REPLICATE

The data-parallel and context-parallel ranks that hold the same experts.

EP_SHARD

The data-parallel and context-parallel ranks that split the experts of a layer.

TP

The tensor-parallel ranks.

ExpertDomainConfig

Bases: BaseModel

Configuration for the expert domain.

Attributes:

Name Type Description
expert_parallel PositiveInt

The number of ranks that split the experts of a layer.

MeshDim dataclass

Bases: Generic[DimT]

One dimension of a device mesh.

Attributes:

Name Type Description
name DimT

The name of the dimension.

size int

The number of ranks along the dimension.

MetricDim

Bases: StrEnum

The dimensions of the mesh of METRIC_DOMAIN.

Attributes:

Name Type Description
PP_DP_CP

All ranks with their own tokens or pipeline stage.

TP

The tensor-parallel ranks, which share their tokens.

build_dist_logger(dist_context, level)

Configures the d9d logger to write to stdout and returns it.

Each line starts with the coordinates of this rank in BASE_DOMAIN, e.g. [d9d] [pp:0-dp:3-cp:0-tp:0], so that ranks can be told apart in distributed logs. Existing handlers of the d9d logger are removed.

Parameters:

Name Type Description Default
dist_context DistributedContext

The distributed context of this process.

required
level int

The log level.

required

Returns:

Type Description
Logger

The configured d9d logger.

init_dist_context(params, extra_domains=())

Builds the distributed context of this process for the with block and sets its current GPU.

In a distributed job, every rank must call it with the same params and the same extra_domains.

On exit, it restores the DTensor RNG tracker. If the block did not raise, it also destroys the process groups that it created, including the default one.

Parameters:

Name Type Description Default
params DeviceMeshParameters

The parallelism degrees and the sections of the built-in domains.

required
extra_domains Sequence[DeviceMeshDomain]

The domains to build after the built-in domains.

()

Yields:

Type Description
DistributedContext

The distributed context.

Raises:

Type Description
ValueError

If the degrees do not match the world size or the GPUs, if a domain is invalid, or if the ranks pass different domains.