Random Number Generation
About
The d9d.rng package derives seeds from a base seed and keys, and seeds the global generators from them. A SeedScope is a node in a tree of seeds. The root seed is a hash of the base seed. A child seed is a hash of the parent seed and the keys of the child. So the same keys give the same seed in every process, on every rank and in every data loader worker.
How Seeds Are Derived
Keys are strings and integers. Keys of another type or another split give another seed, e.g. "1" and 1, or ("ab", "c") and ("a", "bc"). The hash does not depend on PYTHONHASHSEED, so the seeds do not change between processes, Python versions and releases of d9d.
A seed has 64 bits, the full range of torch.Generator.manual_seed().
Forks of the Global Generators
The global generators are the torch CPU generator and the CUDA generator of the current device. Code that passes no generator draws from them, e.g. torch.rand() and dropout.
fork_rng(scope) seeds the global generators from a scope while its context is open. On exit, it restores the states they had before, also on an exception. Forks nest. fork_rng_child(*keys) forks the child of the innermost open fork, so you need no scope at hand. Outside a fork, it raises an error.
The d9d loop runs every call of user code in a fork (see Random Numbers in the Loop). Inside such a call, fork_rng_child() makes a draw depend only on its keys, not on the draws before it.
Forks have these limits:
- One thread: The global generators belong to the process. A fork on another thread, e.g. in a dataset on the prefetch thread, races with the forks of the main thread. Seed a
torch.Generatorfrom a scope there instead. - No CUDA graph capture: PyTorch forbids setting the generator state during capture.
- torch only: A fork does not seed Python
randomor NumPy.
Usage
Seed a generator from a scope and the keys that identify the draw:
In a d9d job, you get the scopes from the loop, e.g. seed_scope of the data provider context. See Random Numbers in the Loop.
Inside a call of the loop, fork a child to give each item its own numbers, whatever was drawn before it:
Outside the loop, open the fork yourself:
API Reference
d9d.rng
Seeds of random number generators, derived from a base seed and keys, and forks of the global generators.
SeedScope
A node in the tree of seeds derived from the base seed of a job.
seed
property
The seed of this scope, in [0, 2**64). APIs that take a 32-bit seed must use it modulo 2**32.
__init__(seed)
Constructs the SeedScope object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
seed
|
int
|
The seed of the scope, in |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
child(*keys)
fork_rng(scope)
Seeds the global generators from a seed scope while the context is open.
The global generators are the torch CPU generator and the CUDA generator of the current device. On exit, the global generators return to the states they had before the context.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scope
|
SeedScope
|
The seed scope to take the seed from. |
required |
fork_rng_child(*keys)
Forks the global generators from a child of the seed scope of the innermost open fork_rng().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*keys
|
str | int
|
The keys of the child. |
()
|
Returns:
| Type | Description |
|---|---|
AbstractContextManager[None]
|
A context manager that holds the fork while it is open. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If no |