Skip to content

Random Number Generation

About

The d9d.rng package derives seeds from a base seed and keys, and seeds the global generators from them. A SeedScope is a node in a tree of seeds. The root seed is a hash of the base seed. A child seed is a hash of the parent seed and the keys of the child. So the same keys give the same seed in every process, on every rank and in every data loader worker.

How Seeds Are Derived

Keys are strings and integers. Keys of another type or another split give another seed, e.g. "1" and 1, or ("ab", "c") and ("a", "bc"). The hash does not depend on PYTHONHASHSEED, so the seeds do not change between processes, Python versions and releases of d9d.

A seed has 64 bits, the full range of torch.Generator.manual_seed().

Forks of the Global Generators

The global generators are the torch CPU generator and the CUDA generator of the current device. Code that passes no generator draws from them, e.g. torch.rand() and dropout.

fork_rng(scope) seeds the global generators from a scope while its context is open. On exit, it restores the states they had before, also on an exception. Forks nest. fork_rng_child(*keys) forks the child of the innermost open fork, so you need no scope at hand. Outside a fork, it raises an error.

The d9d loop runs every call of user code in a fork (see Random Numbers in the Loop). Inside such a call, fork_rng_child() makes a draw depend only on its keys, not on the draws before it.

Forks have these limits:

  • One thread: The global generators belong to the process. A fork on another thread, e.g. in a dataset on the prefetch thread, races with the forks of the main thread. Seed a torch.Generator from a scope there instead.
  • No CUDA graph capture: PyTorch forbids setting the generator state during capture.
  • torch only: A fork does not seed Python random or NumPy.

Usage

Seed a generator from a scope and the keys that identify the draw:

1
2
3
4
5
6
7
import torch

from d9d.rng import SeedScope

scope = SeedScope.root(42).child("augment")
generator = torch.Generator().manual_seed(scope.child(sample_index).seed)
noise = torch.randn(16, generator=generator)

In a d9d job, you get the scopes from the loop, e.g. seed_scope of the data provider context. See Random Numbers in the Loop.

Inside a call of the loop, fork a child to give each item its own numbers, whatever was drawn before it:

1
2
3
4
5
from d9d.rng import fork_rng_child

for prompt_idx, prompt in enumerate(prompts):
    with fork_rng_child("rollout", prompt_idx):
        completions.append(sample(model, prompt))

Outside the loop, open the fork yourself:

1
2
3
4
from d9d.rng import SeedScope, fork_rng

with fork_rng(SeedScope.root(42).child("eval")):
    run_evaluation(model)

API Reference

d9d.rng

Seeds of random number generators, derived from a base seed and keys, and forks of the global generators.

SeedScope

A node in the tree of seeds derived from the base seed of a job.

seed property

The seed of this scope, in [0, 2**64). APIs that take a 32-bit seed must use it modulo 2**32.

__init__(seed)

Constructs the SeedScope object.

Parameters:

Name Type Description Default
seed int

The seed of the scope, in [0, 2**64).

required

Raises:

Type Description
ValueError

If seed is out of range.

child(*keys)

Returns the seed scope below this one, with a seed hashed from this seed and the keys.

Parameters:

Name Type Description Default
*keys str | int

The keys of the child, e.g. the name of a call and its counters.

()

Returns:

Type Description
SeedScope

The child seed scope.

root(base_seed) staticmethod

Returns the root seed scope of a job, with a seed hashed from the base seed.

Parameters:

Name Type Description Default
base_seed int

The base seed of the job. Any integer is valid.

required

Returns:

Type Description
SeedScope

The root seed scope.

fork_rng(scope)

Seeds the global generators from a seed scope while the context is open.

The global generators are the torch CPU generator and the CUDA generator of the current device. On exit, the global generators return to the states they had before the context.

Parameters:

Name Type Description Default
scope SeedScope

The seed scope to take the seed from.

required

fork_rng_child(*keys)

Forks the global generators from a child of the seed scope of the innermost open fork_rng().

Parameters:

Name Type Description Default
*keys str | int

The keys of the child.

()

Returns:

Type Description
AbstractContextManager[None]

A context manager that holds the fork while it is open.

Raises:

Type Description
RuntimeError

If no fork_rng() is open on this thread.