State Offloading
About
The d9d.offload package moves GPU-resident training state to host (CPU) memory and back. It is the base of the sleep and wake API of the Trainer. Sleep frees the GPUs for a colocated workload, for example a rollout engine that shares the GPUs in colocated reinforcement learning. The user entry points Trainer.sleep(), Trainer.wake() and Trainer.is_sleeping() are documented on the Training Loop page. This page covers the primitives they are built on.
Building Blocks
The package defines two things:
- The
Offloadableprotocol: The contract for a subsystem that owns GPU memory and can release and restore it. - The tensor primitives
offload_tensorandonload_tensor: they move the storage of one tensor to the host and back. The tensor object, and theDTensorwrapper, stay the same.
The Round-Trip Guarantee
An offload followed by an onload changes nothing observable. Across the round trip:
- Parameters and buffers keep their object identity.
- The optimizer keeps its state dict keys and the tensor objects they map to.
DTensorwrapper instances keep theirdevice_mesh,placements, globalshape,strideanddtype.
Only the device storage is allocated again. So external references stay valid after wake-up. Gradient hooks, optimizer state keyed by parameter and a frozen reference model held by a task all point at the same objects.
The primitives swap the storage in place instead of creating new tensors. For a DTensor, only the storage of the local shard moves. The distributed metadata stays on the wrapper.
Usage
Offloading a Tensor
offload_tensor returns an OffloadedTensor handle. Keep it until you call onload_tensor, and pass the same tensor object to both calls.
For a DTensor, pass the wrapper. Only its local shard moves:
Implementing Offloadable
The Trainer offloads a fixed set of subsystems. You cannot add your own subsystem to Trainer.sleep(), but you can implement the protocol for your own state and call it from an EVENT_TRAIN_SLEEP_PRE handler (see Order of Operations). A typical implementation does three things:
- Keep the handles in a mirror.
- Wait for the asynchronous copies with
torch.cuda.synchronize. - Reject a second offload or onload in a row.
The built-in implementations follow the same pattern: TrackedModules (model parameters and buffers), PipelinedOptimizer (optimizer state) and GradientManager (gradient buckets and the residual loss accumulator).
API Reference
d9d.offload
Offloading of GPU training state to host memory and back.
OffloadContext
dataclass
Offloadable
Bases: Protocol
Protocol for subsystems that own GPU-resident state and can release it to host memory.
An offload followed by an onload must not change anything observable. Parameter identities,
optimizer state keys, DTensor instances, placements and dtypes stay the same. Only the device storage
is allocated again.
is_offloaded()
Reports whether this subsystem currently has its state in host memory.
Returns:
| Type | Description |
|---|---|
bool
|
|
offload(ctx)
Releases the GPU memory owned by this subsystem, moving its state to host memory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ctx
|
OffloadContext
|
Context for this operation. |
required |
onload(ctx)
Moves the state that offload released back to GPU memory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ctx
|
OnloadContext
|
Context for this operation. |
required |
OffloadedTensor
dataclass
Handle to a tensor whose local storage was moved to host memory.
offload_tensor produces it and onload_tensor consumes it. Subsystems hold it as an opaque handle
between an offload and the matching onload.
Attributes:
| Name | Type | Description |
|---|---|---|
host |
Tensor
|
The host buffer that holds the local storage of the offloaded tensor. |
OnloadContext
dataclass
Context passed to Offloadable.onload.
Attributes:
| Name | Type | Description |
|---|---|---|
device |
device
|
The device to move the state back to. |
offload_tensor(tensor, *, pin_memory)
Moves the local storage of tensor to host memory, in place.
The tensor object stays the same, so external references to it stay valid. For a DTensor, only the
storage of the local shard moves. The wrapper keeps its device_mesh, placements, global shape and
global stride.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
tensor
|
Tensor
|
The device tensor to offload. Can be a plain tensor or a |
required |
pin_memory
|
bool
|
Whether to allocate the host buffer in pinned memory. |
required |
Returns:
| Type | Description |
|---|---|
OffloadedTensor
|
A handle holding the host buffer. Pass it to |
onload_tensor(tensor, offloaded, *, device)
Moves the local storage of tensor from the host buffer back to device, in place.
The tensor object, and the DTensor wrapper, stay the same instances as before the offload. Only the
device storage is allocated again.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
tensor
|
Tensor
|
The tensor that was passed to |
required |
offloaded
|
OffloadedTensor
|
The handle returned by |
required |
device
|
device
|
The device to move the local storage to. |
required |