Skip to content

Data

About

The d9d.data package provides PyTorch Dataset wrappers and helpers for distributed training. It covers length-based bucketing, data-parallel sharding, padding, token pooling masks and the streams of microbatch packs.

Why d9d Does Not Wrap Datasets Automatically

d9d provides explicit wrappers that you compose yourself. It does not inject samplers or wrap datasets behind your back, as some other frameworks do.

  • Order of composition: A data pipeline behaves differently depending on the order of its wrappers. For example, bucketing before sharding is not the same as sharding before bucketing. When you stack the wrappers yourself, you control this order.
  • Per-dataset configuration: Datasets have different physical limits. A dataset on network storage can need contiguous reads (ShardIndexingMode.chunked), while an in-memory dataset can use round-robin access (ShardIndexingMode.sequential). Explicit wrappers expose these options instead of hiding them in global trainer arguments.

Bucketing

In sequence processing, the items of a batch often have different lengths. With random sampling, every batch is padded to its longest sequence, so compute is spent on padding tokens.

BufferSortedDataset reads a buffer of items, sorts it by a sort key (usually the length) and splits it into packs. Items in one pack, e.g. one microbatch, have similar lengths, which reduces padding. It then shuffles the packs and the items within each pack, so the data order is not strictly sorted. All ranks must use the same seed, so that their shards cover every item exactly once.

The underlying dataset must implement DatasetImplementingSortKeyProtocol, i.e. it must have a sort_key(index) method.

Sharding

With data parallelism, each GPU processes a subset of the data. ShardedDataset is a deterministic view of one shard of the data, selected by the shard index (usually the data-parallel rank).

It supports:

  • Sequential sharding: Round-robin distribution (0, 4, 8, ... for rank 0 of 4).
  • Chunked sharding: Contiguous blocks (0, 1, 2, ... for rank 0).
  • Optional padding: All shards get the same length by repeating the last item. Without it, ranks with fewer items finish early, and the other ranks hang in collectives.

shard_dataset_data_parallel builds a ShardedDataset for the data-parallel dimension of a DistributedContext.

A ShardedDataset must be restored from a checkpoint with the same number of shards.

Padding

Variable-length sequences must be padded to the same length before they can be stacked into a batch. pad_stack_1d pads and stacks 1D tensors. Use it to write a collate_fn.

Token Pooling

token_pooling_mask_from_attention_mask builds a mask of the tokens to pool from an attention mask. TokenPoolingType selects the first token, the last non-padding token or all non-padding tokens. The last mode assumes right padding.

Microbatch Packing

FixedCountMicrobatchPacker, PinMemoryMicrobatchPackStream and num_microbatches_for_global_batch build the stream of microbatch packs that the training loop consumes. See Data Loading for how to use them.

build_device_pack_stream hands the packs out on the device. With a positive prefetch factor, it copies the next packs on a side CUDA stream while the current step runs. See Data Prefetching.

Usage

Bucketing

from torch.utils.data import Dataset
from d9d.data import BufferSortedDataset

class MyTextDataset(Dataset):
    def __init__(self, data: list[str]):
        self.data = data

    def __len__(self):
        return len(self.data)

    def __getitem__(self, index):
        return self.data[index]

    # Required by BufferSortedDataset.
    def sort_key(self, index):
        return len(self.data[index])

# Shuffle the base dataset globally beforehand if possible.
raw_data = ["short", "very very very long phrase", "tiny", "medium size"] * 100
base_ds = MyTextDataset(raw_data)

# - buffer_size=100: sort 100 items at a time to find similar lengths
# - pack_size=4: group them into batches of 4.
sorted_ds = BufferSortedDataset(
    base_dataset=base_ds,
    buffer_size=100,
    pack_size=4,
    seed=42
)

Sharding for Data Parallelism

import torch
from torch.utils.data import TensorDataset
from d9d.dist_context import DistributedContext
from d9d.data import shard_dataset_data_parallel, ShardIndexingMode

# The data-parallel size and rank come from the DistributedContext.
context: DistributedContext

base_ds = TensorDataset(torch.randn(100, 10))

sharded_ds = shard_dataset_data_parallel(
    dataset=base_ds,
    dist_context=context,
    # Optional parameters.
    indexing_mode=ShardIndexingMode.chunked,
    pad_to_equal_size_across_shards=True
)

print(f"I see {len(sharded_ds)} items.")

Manual Sharding

import torch
from torch.utils.data import TensorDataset
from d9d.dist_context import BASE_DOMAIN, BaseDim, DistributedContext
from d9d.data import ShardedDataset, ShardIndexingMode

# Take the data-parallel size and rank from the DistributedContext. Without parallelism, they are 1 and 0.
context: DistributedContext
dp_size = context.dim_size(BASE_DOMAIN, BaseDim.DP)
dp_rank = context.coordinate(BASE_DOMAIN, BaseDim.DP)

base_ds = TensorDataset(torch.randn(100, 10))

sharded_ds = ShardedDataset(
    dataset=base_ds,
    total_shards=dp_size,
    current_shard=dp_rank,
    indexing_mode=ShardIndexingMode.chunked,
    # Prevents hangs in collectives when shards have different lengths.
    pad_to_equal_size_across_shards=True
)

print(f"I am rank {dp_rank}, I see {len(sharded_ds)} items.")

Padding

import torch
from d9d.data import pad_stack_1d, PaddingSide1D

items = [
    torch.tensor([1, 2, 3]),
    torch.tensor([4]),
    torch.tensor([5, 6])
]

# 1. Right padding.
batch = pad_stack_1d(items, pad_value=0, padding_side=PaddingSide1D.right)

# 2. Left padding.
batch_left = pad_stack_1d(items, pad_value=0, padding_side=PaddingSide1D.left)

# 3. Padding to a multiple of 8, e.g. for kernel requirements or context-parallel sharding.
batch_aligned = pad_stack_1d(
    items,
    pad_value=0,
    pad_to_multiple_of=8
)

API Reference

d9d.data

Dataset wrappers and data utilities for distributed training.

CollateFn = Callable[[Sequence[TDataTree]], TDataTree] module-attribute

Type alias for a function that collates a sequence of samples into a batch.

The function receives a sequence of samples (PyTrees) and stacks or merges them into one batched PyTree.

MicrobatchPack = Sequence[TDataTree] module-attribute

Type alias for one step's worth of data: a sequence of microbatches.

BufferSortedDataset

Bases: Dataset[_T_co], Stateful

Dataset wrapper that serves items sorted within buffers, with local shuffling.

Items of similar length end up in the same pack, which reduces padding in variable-length training. The shuffling keeps enough randomness in the order of updates.

Algorithm:

  1. Select a range of buffer_size indices.
  2. Build sort keys: (base_dataset.sort_key(index), random_tie_breaker).
  3. Sort the indices by these keys.
  4. Split the sorted list into packs of pack_size items.
  5. Shuffle the order of the packs.
  6. Shuffle the items within each pack.
  7. Flatten and serve.
__init__(base_dataset, buffer_size, pack_size, seed=0)

Constructs the BufferSortedDataset object.

Parameters:

Name Type Description Default
base_dataset DatasetImplementingSortKeyProtocol[_T_co]

The underlying dataset.

required
buffer_size int

The number of items sorted together.

required
pack_size int

The size of a group of similar items, usually the batch or microbatch size.

required
seed int

The seed of the shuffling. All copies of the dataset, e.g. on all ranks and in all workers, must use the same seed, so that they serve the same item at the same index.

0

DataLoaderProtocol

Bases: Protocol

Protocol for a sized, stateful stream of single microbatches.

A conforming loader yields one collated microbatch at a time and reports its length in microbatches. It supports checkpointing through the Stateful interface (state_dict and load_state_dict).

A torchdata StatefulDataLoader conforms to it.

__iter__()

Returns an iterator over single collated microbatches.

Returns:

Type Description
Iterator[PyTree]

An iterator yielding one collated microbatch at a time.

__len__()

Returns the number of microbatches this loader yields.

Returns:

Type Description
int

The number of microbatches.

load_state_dict(state_dict)

Restores the loader's state from a previously produced state dict.

Parameters:

Name Type Description Default
state_dict dict[str, Any]

The state dict to restore from.

required
state_dict()

Returns the loader's checkpointable state.

Returns:

Type Description
dict[str, Any]

A dictionary representing the loader's state.

DatasetImplementingSortKeyProtocol

Bases: Protocol[_T_co]

Protocol for datasets that return a sort key for an item without loading it.

It is typically used for length-based bucketing, where the dataset exposes the length of an item.

__getitem__(item)

Returns the item at the given index.

__len__()

Returns the total number of items in the dataset.

sort_key(index)

Returns a value used for sorting the dataset at the given index.

Parameters:

Name Type Description Default
index int

The index of the item.

required

Returns:

Type Description
Any

A comparable value used for sorting, e.g. the item length.

DevicePackStream

Bases: ABC, Stateful

Abstract base class for streams that hand microbatch packs to the loop on the device.

Its state is the data state stored in checkpoints.

__iter__() abstractmethod

Yields the stream's packs on the device, ready for use on the current CUDA stream.

Closing the generator stops any work done ahead and releases the packs prepared for later steps.

DirectDevicePackStream

Bases: DevicePackStream

Device pack stream that copies each pack to the device on the current CUDA stream when it is handed out.

__init__(stream, device)

Constructs the DirectDevicePackStream object.

Parameters:

Name Type Description Default
stream MicrobatchPackStream

The stream of host-side packs.

required
device Device

The device to copy the packs to.

required

FixedCountMicrobatchPacker

Bases: MicrobatchPackStream

Microbatch pack stream that groups the microbatches of a loader into packs of microbatches_per_step.

It is the default MicrobatchPackStream. A short trailing pack (drop_last=False) is consistent across ranks only when every rank sees the same number of microbatches, e.g. when the dataset was sharded with pad_to_equal_size_across_shards.

Its total_steps is derived from the loader's length, and it delegates its state to the loader (the checkpoint boundary).

total_steps property

The number of packs (steps) this packer yields.

It includes a trailing short pack unless drop_last is set.

__init__(loader, microbatches_per_step, drop_last=True)

Constructs the FixedCountMicrobatchPacker object.

Parameters:

Name Type Description Default
loader DataLoaderProtocol

The microbatch stream to group.

required
microbatches_per_step int

The number of microbatches in each pack.

required
drop_last bool

Whether to drop the trailing incomplete pack instead of yielding it short.

True

Raises:

Type Description
ValueError

If microbatches_per_step is not positive.

__iter__()

Iterates the loader, grouping microbatches into packs.

Yields:

Type Description
MicrobatchPack

A full pack, or a shorter trailing pack when drop_last is unset.

MicrobatchPackStream

Bases: Protocol

Protocol for a stateful stream of microbatch packs that the loop drives.

One pack holds exactly one step's worth of microbatches. The stream supports checkpointing through the Stateful interface (state_dict and load_state_dict). It is the only checkpoint boundary of the data pipeline.

The stream yields CPU tensors, which can be in pinned memory. The loop moves each pack to the device. When prefetching, the loop iterates the stream on a background thread.

total_steps property

The number of steps (packs) this stream yields, or None if it is not known ahead of time.

A streaming or data-dependent source returns None. In that case, the job duration must come from JobScheduleConfig.

__iter__()

Returns an iterator over microbatch packs.

Returns:

Type Description
Iterator[MicrobatchPack]

An iterator yielding one microbatch pack (one step's worth of microbatches) at a time.

load_state_dict(state_dict)

Restores the stream's state from a previously produced state dict.

Parameters:

Name Type Description Default
state_dict dict[str, Any]

The state dict to restore from.

required
state_dict()

Returns the stream's checkpointable state.

When prefetching, the loop calls this method after every pack. Keep it cheap, and do not return objects that later iteration mutates.

Returns:

Type Description
dict[str, Any]

A dictionary representing the stream's state.

PaddingSide1D

Bases: StrEnum

Side on which 1D sequences are padded.

Attributes:

Name Type Description
left

Pad on the left side.

right

Pad on the right side.

PinMemoryMicrobatchPackStream

Bases: MicrobatchPackStream

Microbatch pack stream wrapper that copies every tensor of each pack into pinned host memory.

Pinned memory lets the loop copy packs to the device asynchronously. Unlike the pin_memory option of torch.utils.data.DataLoader, the traversal uses d9d.pytree, so tensors nested in dataclasses are pinned as well. Pinning runs in the iterating thread.

Its total_steps and state are those of the wrapped stream.

total_steps property

The number of packs (steps) the wrapped stream yields, or None if it is unknown.

__init__(inner)

Constructs the PinMemoryMicrobatchPackStream object.

Parameters:

Name Type Description Default
inner MicrobatchPackStream

The wrapped stream that owns the data and its position.

required
__iter__()

Iterates the wrapped stream, pinning each pack.

Yields:

Type Description
MicrobatchPack

The next pack with all of its tensors in pinned host memory.

PrefetchingDevicePackStream

Bases: DevicePackStream

Device pack stream that copies packs to the device on a side CUDA stream ahead of the steps that use them.

A background thread pulls the packs from the stream, so their loading, pinning and copy launches stay off the loop's critical path. Prefetching runs the stream's state ahead of the job, so state_dict returns the stream state as of the last handed out pack. This is the state of the last step, since checkpoints are only taken between steps.

__init__(stream, device, prefetch_factor)

Constructs the PrefetchingDevicePackStream object.

Parameters:

Name Type Description Default
stream MicrobatchPackStream

The stream of host-side packs.

required
device Device

The device to copy the packs to.

required
prefetch_factor int

The number of packs copied ahead.

required

Raises:

Type Description
ValueError

If prefetch_factor is not positive.

ShardIndexingMode

Bases: StrEnum

Ways to split a dataset across shards.

The examples show 14 items split across 4 shards.

Attributes:

Name Type Description
sequential

Round-robin distribution.

shard0: 0, 4, 8, 12 shard1: 1, 5, 9, 13 shard2: 2, 6, 10 shard3: 3, 7, 11

chunked

Contiguous blocks.

shard0: 0, 1, 2, 3 shard1: 4, 5, 6, 7 shard2: 8, 9, 10, 11 shard3: 12, 13

ShardedDataset

Bases: Dataset[_T_co], Stateful

Dataset wrapper that exposes only one shard of a dataset.

Use it for data-parallel training, where each process sees only a subset of the data. Optional padding gives all shards the same length, so no rank waits forever in a collective.

__getitem__(index)

Returns the item at an index relative to this shard.

If the index points past the end of the dataset, which happens only with padding, returns the last item of the dataset.

Parameters:

Name Type Description Default
index int

The index relative to this shard.

required

Returns:

Type Description
_T_co

The data item.

__init__(dataset, total_shards, current_shard, indexing_mode, pad_to_equal_size_across_shards)

Constructs the ShardedDataset object.

Parameters:

Name Type Description Default
dataset Dataset[_T_co]

The underlying dataset to shard.

required
total_shards int

The total number of shards, e.g. the number of data-parallel ranks.

required
current_shard int

The index of the current shard, e.g. the current data-parallel rank.

required
indexing_mode ShardIndexingMode

How indices are assigned to shards.

required
pad_to_equal_size_across_shards bool

If True, all shards report the same length. Padding repeats the last item of the dataset.

required

Raises:

Type Description
ValueError

If dataset does not implement __len__.

__len__()

Returns the number of items in this shard.

If pad_to_equal_size_across_shards is True, returns the maximum shard length.

Returns:

Type Description
int

The number of items in the shard.

Raises:

Type Description
ValueError

If the indexing mode is unknown.

TokenPoolingType

Bases: StrEnum

Supported token pooling strategies.

Attributes:

Name Type Description
first

Selects the first token of the sequence, e.g. the [CLS] token.

last

Selects the last non-padding token of the sequence, e.g. for decoder-only models.

all

Selects all non-padding tokens, e.g. for mean pooling.

build_device_pack_stream(stream, device, prefetch_factor)

Builds the device pack stream for the given prefetch factor.

Parameters:

Name Type Description Default
stream MicrobatchPackStream

The stream of host-side packs.

required
device Device

The device to copy the packs to.

required
prefetch_factor int

The number of packs copied ahead; 0 disables prefetching.

required

Returns:

Type Description
DevicePackStream

A prefetching stream for a positive prefetch_factor, a direct one for 0.

Raises:

Type Description
ValueError

If prefetch_factor is negative.

num_microbatches_for_global_batch(dist_context, global_batch_size, microbatch_size)

Computes the number of microbatches per step required to reach a target global batch size.

The global batch is spread across the data-parallel ranks, and each rank processes microbatch_size samples per microbatch. So each rank processes global_batch_size / (dp_size * microbatch_size) microbatches per optimizer step.

Parameters:

Name Type Description Default
dist_context DistributedContext

The distributed context.

required
global_batch_size int

The total effective batch size across all replicas and accumulation.

required
microbatch_size int

The number of samples in a single microbatch on a single rank.

required

Returns:

Type Description
int

The number of microbatches per step (the gradient-accumulation factor).

Raises:

Type Description
ValueError

If global_batch_size is not divisible by dp_size * microbatch_size.

pad_stack_1d(items, pad_value, padding_side=PaddingSide1D.right, pad_to_multiple_of=None)

Pads 1D tensors to the same length and stacks them into a batch.

All tensors are padded to the length of the longest one.

Parameters:

Name Type Description Default
items Sequence[Tensor]

The 1D tensors to stack.

required
pad_value int

The value used for padding.

required
padding_side PaddingSide1D

The side on which to pad.

right
pad_to_multiple_of int | None

If set, the padded length is rounded up to a multiple of this value.

None

Returns:

Type Description
Tensor

The stacked tensor. Shape: (batch, seq_len).

Raises:

Type Description
ValueError

If items is empty or pad_to_multiple_of is not positive.

shard_dataset_data_parallel(dataset, dist_context, indexing_mode=ShardIndexingMode.sequential, pad_to_equal_size_across_shards=True)

Wraps a dataset into a ShardedDataset over the data-parallel ranks.

The shard count and index come from the "dp" dimension of the batch domain mesh. Without distributed training, the dataset has a single shard.

Parameters:

Name Type Description Default
dataset Dataset[_T_co]

The source dataset to shard.

required
dist_context DistributedContext

The distributed context.

required
indexing_mode ShardIndexingMode

How indices are assigned to shards.

sequential
pad_to_equal_size_across_shards bool

If True, all shards have the same length.

True

Returns:

Type Description
Dataset[_T_co]

A dataset instance representing the local shard.

token_pooling_mask_from_attention_mask(attention_mask, pooling_type)

Builds a binary mask of the tokens to pool for the given strategy.

last assumes right padding.

Parameters:

Name Type Description Default
attention_mask Tensor

A binary mask of valid tokens (1) and padding (0). Shape: (batch, seq_len).

required
pooling_type TokenPoolingType

The strategy for selecting tokens.

required

Returns:

Type Description
Tensor

A mask with 1 at the positions to pool and 0 elsewhere. first and last return a torch.long

Tensor

mask. all returns attention_mask itself. Shape: (batch, seq_len).

Raises:

Type Description
ValueError

If pooling_type is not supported.