Determinism
About
The d9d.internals.determinism package seeds the random number generators (RNG) of all distributed processes. set_seeds seeds Python, NumPy, PyTorch and the DTensor RNG. By default, ranks in different pipeline stages get different seeds. Ranks along all other mesh dimensions share a seed.
Internal API
If you use the standard d9d training or inference loop, you do not need to call this package. d9d seeds the RNGs at startup. This page is for users who extend d9d.
API Reference
d9d.internals.determinism
Seeding of random number generators across distributed processes.
set_seeds(dist_context, seed, distinct_seed_mesh_dim='pp')
Sets the random seeds of Python, NumPy and PyTorch.
The seed is the base seed plus the rank in distinct_seed_mesh_dim, e.g. the pipeline-parallel rank.
So different pipeline stages get different random states, while ranks along other dimensions share one.
In a distributed setup, the function also seeds the DTensor random generator over the other dimensions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dist_context
|
DistributedContext
|
The distributed context. |
required |
seed
|
int
|
The base random seed. |
required |
distinct_seed_mesh_dim
|
str
|
The name of the mesh dimension along which seeds differ, e.g. |
'pp'
|