Skip to content

Decoder with Heads

About

The d9d.models package composes a decoder backbone, such as Qwen3 Dense, with task heads into one model that supports pipeline parallelism. The d9d.models.io package holds the types that this model passes between pipeline stages.

Composition

The backbone maps token ids to hidden states. A head turns the hidden states into the output of a task, e.g. a loss or logits. The model attaches the heads only on the last pipeline stage.

  • One head: DecoderWithHead stores the head as head and returns its output as it is. DecoderForCausalLM, DecoderForClassification and DecoderForEmbedding build the head themselves.
  • Several heads: DecoderWithHeads stores the heads in heads by name and returns a mapping from the name to the output. Each head gets its own shared input under its name.

build_decoder_head builds a built-in head from its configuration. It takes the hidden size and the vocabulary splits from the backbone. You can also pass your own TaskHead to the model.

The backbone and the heads are public attributes. So the model provider parallelizes each of them separately, e.g. with parallelize_causal_lm_head, and the state mappers address them by their prefixes model., head. and heads.<name>..

Usage

from d9d.models import (
    CausalLMHeadConfig,
    ClassificationHeadConfig,
    DecoderWithHeads,
    build_decoder_head,
    parallelize_causal_lm_head,
    parallelize_classification_head,
)

# In ModelProvider.initialize_model_stage.
backbone = ...  # E.g. a Qwen3DenseModel.
heads = {
    "lm": build_decoder_head(CausalLMHeadConfig(), backbone=backbone),
    "cls": build_decoder_head(ClassificationHeadConfig(num_labels=2), backbone=backbone),
}
model = DecoderWithHeads(backbone, heads, context.stage)

# In ModelProvider.parallelize_model_stage, after the backbone.
if context.stage.is_current_stage_last:
    parallelize_causal_lm_head(context.model.heads["lm"], context.dist_context)
    parallelize_classification_head(context.model.heads["cls"], context.dist_context)

API Reference

d9d.models

Ready-to-use models: the decoder composed with task heads, and the Qwen3 models with their state mappers.

AnyHeadConfig = Annotated[CausalLMHeadConfig | ClassificationHeadConfig | EmbeddingHeadConfig, Field(discriminator='kind')] module-attribute

Closed, discriminated union of the built-in head configurations.

SINGLE_HEAD_PREFIX = 'head.' module-attribute

FQN prefix of the task head in a single-head decoder (any DecoderWithHead).

CausalLMHeadConfig

Bases: BaseModel

Configuration for a causal language modeling head.

The head takes its hidden size and split-vocabulary layout from the backbone, so this config has no fields besides its discriminator.

Attributes:

Name Type Description
kind Literal['causal_lm']

Discriminator field. Always "causal_lm".

ClassificationHeadConfig

Bases: BaseModel

Configuration for a sequence/token classification head.

Attributes:

Name Type Description
kind Literal['classification']

Discriminator field. Always "classification".

num_labels int

The number of output classes.

dropout float

The dropout probability applied before the projection.

DecoderBackbone

Bases: ModuleLateInit, ModuleSupportsPipelining[SequenceInput, SequenceTransfer[Tensor], SequenceShared, SequenceTransfer[Tensor]], Protocol

Protocol for a decoder backbone that maps stage inputs to hidden states.

A backbone reports its hidden_size and its split-vocabulary layout, and supports late init and pipelining. The decoder compositions call only this surface. The reported dimensions are fixed at construction, and task heads derive their shapes from them.

hidden_size property

The size of the backbone hidden states.

split_vocab_order property

The order in which vocabulary segments are concatenated.

split_vocab_size property

The mapping of vocabulary segment names to their sizes.

__call__(inputs, shared)

Runs the backbone stage through nn.Module.__call__, so module hooks apply.

Parameters:

Name Type Description Default
inputs SequenceInput | SequenceTransfer[Tensor]

SequenceInput on the first stage; the incoming SequenceTransfer otherwise.

required
shared SequenceShared

The backbone shared input broadcast to every stage.

required

Returns:

Type Description
SequenceTransfer[Tensor]

The produced SequenceTransfer.

forward(inputs, shared)

Runs this stage.

Parameters:

Name Type Description Default
inputs TPipelineInput | TStageTransfer

PipelineInput on the first stage; StageTransfer otherwise.

required
shared TSharedInput

The value broadcast to every stage.

required

Returns:

Type Description
TStageTransfer | TPipelineOutput

PipelineOutput on the last stage; StageTransfer otherwise.

reset_parameters()

Resets the module parameters (i.e. performs random initialization).

stage_transfer_spec(pipeline_input, boundary)

Describes the SequenceTransfer crossing the given boundary of this stage.

Parameters:

Name Type Description Default
pipeline_input SequenceInput

A representative SequenceInput microbatch. Only its shapes are read.

required
boundary StageBoundary

Which inter-stage edge to describe.

required

Returns:

Type Description
SequenceTransfer[TensorSpec]

A SequenceTransfer of TensorSpec.

DecoderForCausalLM

Bases: DecoderWithHead[TBackbone, SplitLanguageModellingHead, SequenceCausalLMHeadShared, SequenceCausalLMOutput]

A decoder backbone composed with a single causal language modeling head.

It builds the head itself, so composing the model is one call. Its I/O types are fixed to that head's shared input and output.

__init__(backbone, stage)

Constructs the DecoderForCausalLM object.

Parameters:

Name Type Description Default
backbone TBackbone

The decoder backbone, exposed as self.model (FQN model.*).

required
stage PipelineStageInfo

Pipeline stage information for this instance.

required
forward(inputs, shared)

Runs the backbone and, on the last stage, the attached head.

Parameters:

Name Type Description Default
inputs SequenceInput | SequenceTransfer[Tensor]

SequenceInput on the first stage; the incoming SequenceTransfer otherwise.

required
shared SequenceHeadShared[THeadShared]

The backbone shared input plus the head's shared input.

required

Returns:

Type Description
SequenceTransfer[Tensor] | THeadOutput

The produced SequenceTransfer on non-last stages, or the head's output on the last stage.

reset_parameters()

Resets the parameters of the backbone and of the head.

stage_transfer_spec(pipeline_input, boundary)

Describes the SequenceTransfer crossing the given boundary, as the backbone does.

The head only runs on the last stage, which has no outgoing transfer, so the backbone alone defines the transfer.

Parameters:

Name Type Description Default
pipeline_input SequenceInput

A representative SequenceInput microbatch. Only its shapes are read.

required
boundary StageBoundary

Which inter-stage edge to describe.

required

Returns:

Type Description
SequenceTransfer[TensorSpec]

A SequenceTransfer of TensorSpec.

DecoderForClassification

Bases: DecoderWithHead[TBackbone, ClassificationHead, SequencePoolingHeadShared, SequenceClassificationOutput]

A decoder backbone composed with a single classification head.

It builds the head itself, so composing the model is one call. Its I/O types are fixed to that head's shared input and output.

__init__(backbone, config, stage)

Constructs the DecoderForClassification object.

Parameters:

Name Type Description Default
backbone TBackbone

The decoder backbone, exposed as self.model (FQN model.*).

required
config ClassificationHeadConfig

Configuration of the classification head to build.

required
stage PipelineStageInfo

Pipeline stage information for this instance.

required
forward(inputs, shared)

Runs the backbone and, on the last stage, the attached head.

Parameters:

Name Type Description Default
inputs SequenceInput | SequenceTransfer[Tensor]

SequenceInput on the first stage; the incoming SequenceTransfer otherwise.

required
shared SequenceHeadShared[THeadShared]

The backbone shared input plus the head's shared input.

required

Returns:

Type Description
SequenceTransfer[Tensor] | THeadOutput

The produced SequenceTransfer on non-last stages, or the head's output on the last stage.

reset_parameters()

Resets the parameters of the backbone and of the head.

stage_transfer_spec(pipeline_input, boundary)

Describes the SequenceTransfer crossing the given boundary, as the backbone does.

The head only runs on the last stage, which has no outgoing transfer, so the backbone alone defines the transfer.

Parameters:

Name Type Description Default
pipeline_input SequenceInput

A representative SequenceInput microbatch. Only its shapes are read.

required
boundary StageBoundary

Which inter-stage edge to describe.

required

Returns:

Type Description
SequenceTransfer[TensorSpec]

A SequenceTransfer of TensorSpec.

DecoderForEmbedding

Bases: DecoderWithHead[TBackbone, EmbeddingHead, SequencePoolingHeadShared, SequenceEmbeddingOutput]

A decoder backbone composed with a single embedding head.

It builds the head itself, so composing the model is one call. Its I/O types are fixed to that head's shared input and output.

__init__(backbone, config, stage)

Constructs the DecoderForEmbedding object.

Parameters:

Name Type Description Default
backbone TBackbone

The decoder backbone, exposed as self.model (FQN model.*).

required
config EmbeddingHeadConfig

Configuration of the embedding head to build.

required
stage PipelineStageInfo

Pipeline stage information for this instance.

required
forward(inputs, shared)

Runs the backbone and, on the last stage, the attached head.

Parameters:

Name Type Description Default
inputs SequenceInput | SequenceTransfer[Tensor]

SequenceInput on the first stage; the incoming SequenceTransfer otherwise.

required
shared SequenceHeadShared[THeadShared]

The backbone shared input plus the head's shared input.

required

Returns:

Type Description
SequenceTransfer[Tensor] | THeadOutput

The produced SequenceTransfer on non-last stages, or the head's output on the last stage.

reset_parameters()

Resets the parameters of the backbone and of the head.

stage_transfer_spec(pipeline_input, boundary)

Describes the SequenceTransfer crossing the given boundary, as the backbone does.

The head only runs on the last stage, which has no outgoing transfer, so the backbone alone defines the transfer.

Parameters:

Name Type Description Default
pipeline_input SequenceInput

A representative SequenceInput microbatch. Only its shapes are read.

required
boundary StageBoundary

Which inter-stage edge to describe.

required

Returns:

Type Description
SequenceTransfer[TensorSpec]

A SequenceTransfer of TensorSpec.

DecoderWithHead

Bases: Module, ModuleLateInit, ModuleSupportsPipelining[SequenceInput, SequenceTransfer[Tensor], SequenceHeadShared[THeadShared], THeadOutput], Generic[TBackbone, THead, THeadShared, THeadOutput]

Model that composes one decoder backbone with exactly one prebuilt task head.

This is the single-head counterpart of DecoderWithHeads, not a special case of it. With one head there is nothing to key. The head is self.head (FQN head.*), its shared input is a field, and the model returns the head's output instead of a one-entry mapping. For example, a causal LM task reads ctx.pipeline_results.cross_entropy directly.

The backbone (self.model) and the head (self.head) are public, so a provider can parallelize and checkpoint-map each of them separately. The head is attached only on the last pipeline stage.

Class Type Parameters:

Name Bound or Constraints Description Default
TBackbone

The backbone type, kept precise so parallelize_* accepts self.model.

required
THead

The head type, kept precise so parallelize_*_head accepts self.head.

required
THeadShared

The shared input accepted by the composed head.

required
THeadOutput

The output produced by the composed head.

required
__init__(backbone, head, stage)

Constructs the DecoderWithHead object.

Parameters:

Name Type Description Default
backbone TBackbone

The decoder backbone, exposed as self.model (FQN model.*).

required
head THead

The prebuilt task head, attached on the last stage as self.head (FQN head.*).

required
stage PipelineStageInfo

Pipeline stage information for this instance.

required
forward(inputs, shared)

Runs the backbone and, on the last stage, the attached head.

Parameters:

Name Type Description Default
inputs SequenceInput | SequenceTransfer[Tensor]

SequenceInput on the first stage; the incoming SequenceTransfer otherwise.

required
shared SequenceHeadShared[THeadShared]

The backbone shared input plus the head's shared input.

required

Returns:

Type Description
SequenceTransfer[Tensor] | THeadOutput

The produced SequenceTransfer on non-last stages, or the head's output on the last stage.

reset_parameters()

Resets the parameters of the backbone and of the head.

stage_transfer_spec(pipeline_input, boundary)

Describes the SequenceTransfer crossing the given boundary, as the backbone does.

The head only runs on the last stage, which has no outgoing transfer, so the backbone alone defines the transfer.

Parameters:

Name Type Description Default
pipeline_input SequenceInput

A representative SequenceInput microbatch. Only its shapes are read.

required
boundary StageBoundary

Which inter-stage edge to describe.

required

Returns:

Type Description
SequenceTransfer[TensorSpec]

A SequenceTransfer of TensorSpec.

DecoderWithHeads

Bases: Module, ModuleLateInit, ModuleSupportsPipelining[SequenceInput, SequenceTransfer[Tensor], SequenceHeadsShared[THeadShared], SequenceHeadsOutput[THeadOutput]], Generic[TBackbone, THeadShared, THeadOutput]

Model that composes one decoder backbone with a mapping of prebuilt named task heads.

For a model with a single head, use DecoderWithHead instead. With one head there is nothing to key, and this class would make every caller name the only head.

The backbone (self.model) and the heads (self.heads) are public, so a provider can parallelize and checkpoint-map each of them separately. This class does not build heads: use build_decoder_head for built-in heads, or pass custom heads in the same way. Heads are attached only on the last pipeline stage.

For heads of one kind, the head I/O type parameters are that kind's shared input and output. For heads of several kinds, they are the unions of those types.

Class Type Parameters:

Name Bound or Constraints Description Default
TBackbone

The backbone type, kept precise so parallelize_* accepts self.model.

required
THeadShared

The shared input accepted by the composed heads.

required
THeadOutput

The output produced by the composed heads.

required
__init__(backbone, heads, stage)

Constructs the DecoderWithHeads object.

Parameters:

Name Type Description Default
backbone TBackbone

The decoder backbone, exposed as self.model (FQN model.*).

required
heads Mapping[str, TaskHead[THeadShared, THeadOutput]]

A mapping of head names to prebuilt task heads, attached on the last stage as self.heads (FQN heads.<name>.*).

required
stage PipelineStageInfo

Pipeline stage information for this instance.

required
forward(inputs, shared)

Runs the backbone and, on the last stage, every attached head.

The backbone receives shared.sequence. Each head receives the entry of shared.heads stored under its name.

Parameters:

Name Type Description Default
inputs SequenceInput | SequenceTransfer[Tensor]

SequenceInput on the first stage; the incoming SequenceTransfer otherwise.

required
shared SequenceHeadsShared[THeadShared]

The backbone shared input plus the per-head shared inputs.

required

Returns:

Type Description
SequenceTransfer[Tensor] | SequenceHeadsOutput[THeadOutput]

The produced SequenceTransfer on non-last stages, or each head's output keyed by head name on the last stage.

reset_parameters()

Resets the parameters of the backbone and of every head.

stage_transfer_spec(pipeline_input, boundary)

Describes the SequenceTransfer crossing the given boundary, as the backbone does.

Heads only run on the last stage, which has no outgoing transfer, so the backbone alone defines the transfer.

Parameters:

Name Type Description Default
pipeline_input SequenceInput

A representative SequenceInput microbatch. Only its shapes are read.

required
boundary StageBoundary

Which inter-stage edge to describe.

required

Returns:

Type Description
SequenceTransfer[TensorSpec]

A SequenceTransfer of TensorSpec.

EmbeddingHeadConfig

Bases: BaseModel

Configuration for a dense embedding head.

Attributes:

Name Type Description
kind Literal['embedding']

Discriminator field. Always "embedding".

embedding_dim int | None

Size of the output embedding. None for no extra projection.

normalize bool

Whether to apply L2 normalization to the final embeddings.

build_decoder_head(config, *, backbone)

Builds a task head from its configuration and the decoder backbone it attaches to.

The head takes hidden_size and the split-vocabulary layout from the backbone, so a config holds only task-specific fields. A custom head does not go through this factory: pass its TaskHead instance to the decoder directly.

Parameters:

Name Type Description Default
config AnyHeadConfig

Task head configuration selecting the head type and its task-specific fields.

required
backbone DecoderBackbone

The decoder backbone the head attaches to. It provides the shared dimensions.

required

Returns:

Type Description
TaskHead

An instantiated task head.

Raises:

Type Description
ValueError

If the head configuration type is unknown.

parallelize_causal_lm_head(head, dist_context)

Applies Hybrid Sharded Data Parallel (HSDP) to a causal language modeling head.

Parameters:

Name Type Description Default
head SplitLanguageModellingHead

The language modeling head to parallelize.

required
dist_context DistributedContext

The distributed context that provides the device meshes.

required

parallelize_classification_head(head, dist_context)

Applies Hybrid Sharded Data Parallel (HSDP) to a classification head.

Parameters:

Name Type Description Default
head ClassificationHead

The classification head to parallelize.

required
dist_context DistributedContext

The distributed context that provides the device meshes.

required

parallelize_embedding_head(head, dist_context)

Applies Hybrid Sharded Data Parallel (HSDP) to an embedding head.

Parameters:

Name Type Description Default
head EmbeddingHead

The embedding head to parallelize.

required
dist_context DistributedContext

The distributed context that provides the device meshes.

required

d9d.models.io

Typed pipeline I/O dataclasses for the model catalogue.

SequenceHeadsOutput = Mapping[str, THeadOutput] module-attribute

The output of a backbone composed with named task heads: each head's output, keyed by head name.

Two heads of the same type have different names, so their outputs cannot collide. The type parameter is the output of the composed heads. It is a single type for heads of one kind (e.g. SequenceHeadsOutput[SequenceCausalLMOutput]), or a union for heads of several kinds.

SequenceHeadShared dataclass

Bases: Generic[THeadShared]

The shared input a backbone composed with exactly one task head consumes on every stage.

This is the single-head counterpart of SequenceHeadsShared. The head input is a field, not a named entry, and the model returns the head's output unwrapped.

Class Type Parameters:

Name Bound or Constraints Description Default
THeadShared

The shared input accepted by the composed head.

required

Attributes:

Name Type Description
sequence SequenceShared

The backbone shared input.

head THeadShared

The head's own shared input. Read on the last stage only.

SequenceHeadsShared dataclass

Bases: Generic[THeadShared]

The shared input a backbone composed with named task heads consumes on every stage.

The type parameter is the shared input of the composed heads. It is a single type for heads of one kind (e.g. SequenceHeadsShared[SequenceCausalLMHeadShared]), or a union for heads of several kinds.

Class Type Parameters:

Name Bound or Constraints Description Default
THeadShared

The shared input accepted by the composed heads.

required

Attributes:

Name Type Description
sequence SequenceShared

The backbone shared input.

heads Mapping[str, THeadShared]

Each head's own shared input, keyed by head name. Read on the last stage only.

SequenceInput dataclass

The input of a sequence transformer: the token ids fed to the first stage.

Attributes:

Name Type Description
input_ids Tensor

Indices of the input tokens. Shape: (batch, seq_len).

SequenceShared dataclass

The shared input the transformer backbone consumes on every stage.

Attributes:

Name Type Description
position_ids Tensor

The position of each token in the position embeddings. Shape: (batch, seq_len).

hidden_states_agg_mask Tensor | None

Mask of the tokens to aggregate into hidden state snapshots, or None if snapshotting is disabled. Shape: (batch, seq_len).

SequenceTransfer dataclass

Bases: Generic[TLeaf]

The object moved between adjacent stages of a sequence transformer.

Attributes:

Name Type Description
hidden_states TLeaf

The output of the last layer of the sending stage. Shape: (batch, seq_len, hidden_size).

hidden_states_snapshot TLeaf | None

The aggregated hidden states of the embeddings and of all layers so far, or None if snapshotting is disabled. Shape: (num_layers, batch, hidden_size).