Distributed Context
About
The d9d.dist_context package holds the distributed state of a process. Its DistributedContext knows the rank, the GPU of the rank, the process groups and the device meshes of the domains. Every component that needs these facts gets the context through its constructor. Do not read raw os.environ variables or create ad-hoc process groups: then the components can disagree about the topology. DistributedContext requires CUDA GPUs.
Creating and Closing the Context
init_dist_context() creates the context from DeviceMeshParameters, once per process. In a distributed job, every rank calls it with the same arguments. It takes the world size and the rank from the default process group if one exists, and otherwise from WORLD_SIZE and RANK of the launcher. Without a launcher, the job has one rank and runs with python. The product of the four parallelism degrees must equal the world size.
init_dist_context() is a context manager, and the context is valid only inside its with block. On exit, it destroys the groups that it created, including the default process group. If the block raises, the groups stay, so the job fails instead of hanging.
In a distributed job, the block also replaces the DTensor RNG tracker of the process. The default tracker of PyTorch broadcasts the generator state of rank 0, which would overwrite the seeds of the other pipeline stages. The previous tracker comes back on exit, also after an error.
A job takes the context and never closes it. TrainingConfigurator and InferenceConfigurator take it as dist_context, so one context can serve several jobs of a process, e.g. training and then evaluation.
GPU Assignment
The degrees describe the ranks, not the GPUs. The same configuration runs on 8 GPUs and on 2, so the context takes the GPU of each rank from the launch:
- Local rank: It comes from
LOCAL_RANK, whichtorchrunsets. It is the index of the rank on its node, whatever its GPU. - GPU: The node runs
LOCAL_WORLD_SIZEranks ontorch.cuda.device_count()visible GPUs. If the ranks do not outnumber the GPUs, local rankiruns on GPUi. Otherwise each GPU runs an equal block of consecutive local ranks. E.g. with 8 ranks on 2 GPUs, local ranks 4 to 7 run on GPU 1. The context raises if the counts do not divide.
Several ranks on one GPU let you check a parallel layout on fewer GPUs than it targets. They need NCCL_MULTI_RANK_GPU_ENABLE=1 from the launcher. CONTRIBUTING.md shows the setup for the tests of d9d.
Parallelism Degrees
The work of one step splits in four ways. DeviceMeshParameters holds one degree for each:
pipeline_parallel: It splits the layers into stages.data_parallel: It splits the samples.context_parallel: It splits one sample into parts, e.g. a text into chunks of the sequence, a graph into subgraphs or an image into tiles.tensor_parallel: It splits the matrix multiplications inside one sample.
Other kinds of parallelism split the same ranks differently, so they do not add to the world size. Each built-in domain that does so has its own section:
dense: The sizes of the shard groups of the dense layers,data_parallel_shardandcontext_parallel_shard. They default to 1, so the dense layers do not shard, as in DDP. The number of replicas follows from the degree, so a shard size that fits the memory of a GPU stays when the job grows.expert: The degree of expert parallelism,expert_parallel. Without this section, the job builds no expert domain.
The shard sizes must divide data_parallel and context_parallel, and expert_parallel must divide data_parallel * context_parallel.
Domains
Different parts of a model need different parallelism, e.g. dense layers and Mixture-of-Experts (MoE) layers. d9d describes each arrangement of the ranks as a domain. All domains hold the same ranks, and each arranges them into its own device mesh. The context keeps each mesh under its domain object: dist_context.mesh_for(DENSE_DOMAIN).
| Domain | Dimensions | Purpose |
|---|---|---|
BASE_DOMAIN |
BaseDim: PP, DP, CP, TP |
Pipeline stages, data and the coordinates of a rank |
DENSE_DOMAIN |
DenseDim: PP, DP_REPLICATE, DP_CP_SHARD, CP_REPLICATE, TP |
Dense layers |
EXPERT_DOMAIN |
ExpertDim: PP, EP_REPLICATE, EP_SHARD, TP |
Expert layers |
METRIC_DOMAIN |
MetricDim: PP_DP_CP, TP |
Sums of the metrics and the loss weights |
- Base domain: Its dimensions are the four degrees. Their order defines how the ranks lie in the world: the ranks of one tensor-parallel group are neighbors.
- Dense domain: It splits
dpintoDP_REPLICATEand a shard, andcpinto a shard andCP_REPLICATE, as thedensesection says.DP_CP_SHARDmerges the two shards. - Expert domain: The context builds it only if the
expertsection is set. Otherwisemesh_for(EXPERT_DOMAIN)raises and asks to set the section.EP_REPLICATEandEP_SHARDsplit the ranks ofdp * cp. So an expert-parallel group never holds two ranks of one tensor-parallel group, which hold the same tokens. - Metric domain:
PP_DP_CPholds all ranks with their own tokens or pipeline stage. The gradient manager and the metrics sum over it, so the ranks of one tensor-parallel group count once.
Each domain names its dimensions with a StrEnum, e.g. DenseDim.DP_REPLICATE, so the type checker catches a dimension of another domain.
Coordinates and Communication
DistributedContext answers four kinds of questions. Only communication needs process groups:
| Kind | Members | In a job without parallelism |
|---|---|---|
| Rank | rank, world_size, is_main_process, local_rank, is_local_main_process, current_device |
Rank 0 of 1 |
| Coordinates | coordinate(), dim_size() |
Coordinate 0, size 1 |
| Communication | is_distributed, mesh_for(), wait_world(), main_process_first(), local_main_process_first(), set_timeout() |
is_distributed is False and mesh_for() raises. set_timeout() does nothing. The others only wait for the GPU. |
Code that communicates checks is_distributed before it calls mesh_for().
Extra Domains
A new kind of parallelism adds a domain that splits the ranks differently. For example, an image model can split the cp ranks into height and width. Subclass DeviceMeshDomain and return the dimensions of the mesh from mesh_dims(), outermost first. The context lays out the ranks along them row-major, as init_device_mesh() does. So dimensions that keep the order of the base dimensions group the ranks as they do. A domain takes its own degrees in its constructor.
Pass the domain in extra_domains of init_dist_context(). The sizes of its dimensions must be positive and multiply to the world size, and the names of its dimensions must be unique. The names of all domains must be unique too. mesh_for() looks the mesh up by the domain object, so a domain object that was not passed raises, even if its name matches.
Usage
Configuration
A job with pipeline parallelism, HSDP inside each node and expert parallelism:
Creating the Context
Coordinates and Meshes
An Extra Domain
Ordering the Ranks
API Reference
d9d.dist_context
The distributed context of a job and the device meshes of its domains.
BASE_DOMAIN = _BaseDomain()
module-attribute
The domain of the four parallelism degrees. The order of its dimensions defines how the ranks lie in the world.
DENSE_DOMAIN = _DenseDomain()
module-attribute
The domain of the dense layers, which splits the data-parallel and context-parallel ranks into replicas and shards.
EXPERT_DOMAIN = _ExpertDomain()
module-attribute
The domain of the expert layers. The context builds it only if the expert section of the configuration is set.
METRIC_DOMAIN = _MetricDomain()
module-attribute
The domain that sums the metrics and the loss weights, so that the tensor-parallel ranks count once.
BaseDim
Bases: StrEnum
The dimensions of the mesh of BASE_DOMAIN.
Attributes:
| Name | Type | Description |
|---|---|---|
PP |
The pipeline stages. |
|
DP |
The data-parallel ranks, which take different samples. |
|
CP |
The context-parallel ranks, which take different parts of one sample. |
|
TP |
The tensor-parallel ranks, which split the matrix multiplications of one sample. |
DenseDim
Bases: StrEnum
The dimensions of the mesh of DENSE_DOMAIN.
Attributes:
| Name | Type | Description |
|---|---|---|
PP |
The pipeline stages. |
|
DP_REPLICATE |
The data-parallel ranks that hold the same shards of the dense parameters. |
|
DP_CP_SHARD |
The data-parallel and context-parallel ranks that shard the dense parameters. |
|
CP_REPLICATE |
The context-parallel ranks that hold the same shards of the dense parameters. |
|
TP |
The tensor-parallel ranks. |
DenseDomainConfig
Bases: BaseModel
Configuration for the dense domain, which shards the parameters of dense layers.
Attributes:
| Name | Type | Description |
|---|---|---|
data_parallel_shard |
PositiveInt
|
The number of data-parallel ranks that shard a parameter. |
context_parallel_shard |
PositiveInt
|
The number of context-parallel ranks that shard a parameter. |
DeviceMeshDomain
Abstract base class for a domain: one arrangement of all ranks into a device mesh.
name
abstractmethod
property
The name of this domain, unique within a context.
mesh_dims(params)
abstractmethod
Returns the dimensions of the mesh of this domain, outermost first.
The context lays out the ranks along them row-major, as init_device_mesh() does. So a dimension that
keeps the order of the dimensions of BASE_DOMAIN groups the ranks as they do.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
params
|
DeviceMeshParameters
|
The parallelism degrees and the sections of the built-in domains. |
required |
Returns:
| Type | Description |
|---|---|
tuple[MeshDim[DimT], ...]
|
The dimensions of the mesh. |
DeviceMeshParameters
Bases: BaseModel
Configuration for the parallelism degrees of a job.
Attributes:
| Name | Type | Description |
|---|---|---|
pipeline_parallel |
PositiveInt
|
The degree of pipeline parallelism. |
data_parallel |
PositiveInt
|
The degree of data parallelism. |
context_parallel |
PositiveInt
|
The degree of context parallelism. |
tensor_parallel |
PositiveInt
|
The degree of tensor parallelism. |
dense |
DenseDomainConfig
|
The configuration of the dense domain. |
expert |
ExpertDomainConfig | None
|
The configuration of the expert domain. |
DistributedContext
The distributed state of one process: its rank, its GPU, its process groups and the meshes of its domains.
Create it with init_dist_context(). It is valid only inside the with block of init_dist_context().
current_device
property
The GPU of this rank, with its index.
is_distributed
property
Whether the job has more than one rank.
is_local_main_process
property
Whether the current process is local rank 0 on its node.
is_main_process
property
Whether the current process is global rank 0.
local_rank
property
The index of this rank on its node.
rank
property
The global rank of this process.
world_size
property
The number of ranks in the job.
__init__(rank, world_size, local_rank, device, domain_dims, meshes, groups)
Constructs the DistributedContext object.
Call init_dist_context() instead, which builds the arguments.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
rank
|
int
|
The global rank of this process. |
required |
world_size
|
int
|
The number of ranks in the job. |
required |
local_rank
|
int
|
The index of this rank on its node. |
required |
device
|
device
|
The GPU of this rank. |
required |
domain_dims
|
Mapping[DeviceMeshDomain, tuple[MeshDim, ...]]
|
The dimensions of the mesh of every domain of the context. |
required |
meshes
|
Mapping[DeviceMeshDomain, DeviceMesh]
|
The mesh of every domain. Empty in a job without parallelism. |
required |
groups
|
list[ProcessGroup]
|
The process groups of the meshes, without the default process group. |
required |
coordinate(domain, dim)
Returns the coordinate of this rank along a dimension of the mesh of a domain.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
domain
|
DeviceMeshDomain[DimT]
|
A domain of this context. |
required |
dim
|
DimT
|
The name of a dimension of the domain. |
required |
Returns:
| Type | Description |
|---|---|
int
|
The coordinate, 0 in a job without parallelism. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the context has no such domain or the domain has no such dimension. |
dim_size(domain, dim)
Returns the number of ranks along a dimension of the mesh of a domain, 1 in a job without parallelism.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
domain
|
DeviceMeshDomain[DimT]
|
A domain of this context. |
required |
dim
|
DimT
|
The name of a dimension of the domain. |
required |
Returns:
| Type | Description |
|---|---|
int
|
The number of ranks along the dimension. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the context has no such domain or the domain has no such dimension. |
local_main_process_first()
Runs the block on the local main processes first.
Other ranks wait before entering the block. Local main processes wait after leaving it, so all ranks continue together.
main_process_first()
Runs the block on the global main process first.
Other ranks wait before entering the block. The global main process waits after leaving it, so all ranks continue together.
mesh_for(domain)
Returns the device mesh of a domain.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
domain
|
DeviceMeshDomain
|
A domain of this context. |
required |
Returns:
| Type | Description |
|---|---|
DeviceMesh
|
The device mesh of the domain. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the job has no parallelism or the context has no such domain. |
set_timeout(timeout_seconds)
Sets the timeout of the default process group and of all process groups of the context.
Does nothing in a job without parallelism.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeout_seconds
|
float
|
The new timeout in seconds. |
required |
wait_world()
Blocks until all ranks reach this point and the GPU of this rank finishes its queued work.
ExpertDim
Bases: StrEnum
The dimensions of the mesh of EXPERT_DOMAIN.
Attributes:
| Name | Type | Description |
|---|---|---|
PP |
The pipeline stages. |
|
EP_REPLICATE |
The data-parallel and context-parallel ranks that hold the same experts. |
|
EP_SHARD |
The data-parallel and context-parallel ranks that split the experts of a layer. |
|
TP |
The tensor-parallel ranks. |
ExpertDomainConfig
Bases: BaseModel
Configuration for the expert domain.
Attributes:
| Name | Type | Description |
|---|---|---|
expert_parallel |
PositiveInt
|
The number of ranks that split the experts of a layer. |
MeshDim
dataclass
MetricDim
Bases: StrEnum
The dimensions of the mesh of METRIC_DOMAIN.
Attributes:
| Name | Type | Description |
|---|---|---|
PP_DP_CP |
All ranks with their own tokens or pipeline stage. |
|
TP |
The tensor-parallel ranks, which share their tokens. |
build_dist_logger(dist_context, level)
Configures the d9d logger to write to stdout and returns it.
Each line starts with the coordinates of this rank in BASE_DOMAIN, e.g. [d9d] [pp:0-dp:3-cp:0-tp:0], so
that ranks can be told apart in distributed logs. Existing handlers of the d9d logger are removed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dist_context
|
DistributedContext
|
The distributed context of this process. |
required |
level
|
int
|
The log level. |
required |
Returns:
| Type | Description |
|---|---|
Logger
|
The configured |
init_dist_context(params, extra_domains=())
Builds the distributed context of this process for the with block and sets its current GPU.
In a distributed job, every rank must call it with the same params and the same extra_domains.
On exit, it restores the DTensor RNG tracker. If the block did not raise, it also destroys the process groups
that it created, including the default one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
params
|
DeviceMeshParameters
|
The parallelism degrees and the sections of the built-in domains. |
required |
extra_domains
|
Sequence[DeviceMeshDomain]
|
The domains to build after the built-in domains. |
()
|
Yields:
| Type | Description |
|---|---|
DistributedContext
|
The distributed context. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the degrees do not match the world size or the GPUs, if a domain is invalid, or if the ranks pass different domains. |