Decoder with Heads
About
The d9d.models package composes a decoder backbone, such as Qwen3 Dense, with task heads into one model that supports pipeline parallelism. The d9d.models.io package holds the types that this model passes between pipeline stages.
Composition
The backbone maps token ids to hidden states. A head turns the hidden states into the output of a task, e.g. a loss or logits. The model attaches the heads only on the last pipeline stage.
- One head:
DecoderWithHeadstores the head asheadand returns its output as it is.DecoderForCausalLM,DecoderForClassificationandDecoderForEmbeddingbuild the head themselves. - Several heads:
DecoderWithHeadsstores the heads inheadsby name and returns a mapping from the name to the output. Each head gets its own shared input under its name.
build_decoder_head builds a built-in head from its configuration. It takes the hidden size and the vocabulary splits from the backbone. You can also pass your own TaskHead to the model.
The backbone and the heads are public attributes. So the model provider parallelizes each of them separately, e.g. with parallelize_causal_lm_head, and the state mappers address them by their prefixes model., head. and heads.<name>..
Usage
API Reference
d9d.models
Ready-to-use models: the decoder composed with task heads, and the Qwen3 models with their state mappers.
AnyHeadConfig = Annotated[CausalLMHeadConfig | ClassificationHeadConfig | EmbeddingHeadConfig, Field(discriminator='kind')]
module-attribute
Closed, discriminated union of the built-in head configurations.
SINGLE_HEAD_PREFIX = 'head.'
module-attribute
FQN prefix of the task head in a single-head decoder (any DecoderWithHead).
CausalLMHeadConfig
Bases: BaseModel
Configuration for a causal language modeling head.
The head takes its hidden size and split-vocabulary layout from the backbone, so this config has no fields besides its discriminator.
Attributes:
| Name | Type | Description |
|---|---|---|
kind |
Literal['causal_lm']
|
Discriminator field. Always |
ClassificationHeadConfig
DecoderBackbone
Bases: ModuleLateInit, ModuleSupportsPipelining[SequenceInput, SequenceTransfer[Tensor], SequenceShared, SequenceTransfer[Tensor]], Protocol
Protocol for a decoder backbone that maps stage inputs to hidden states.
A backbone reports its hidden_size and its split-vocabulary layout, and supports late init and
pipelining. The decoder compositions call only this surface. The reported dimensions are fixed at
construction, and task heads derive their shapes from them.
hidden_size
property
The size of the backbone hidden states.
split_vocab_order
property
The order in which vocabulary segments are concatenated.
split_vocab_size
property
The mapping of vocabulary segment names to their sizes.
__call__(inputs, shared)
Runs the backbone stage through nn.Module.__call__, so module hooks apply.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs
|
SequenceInput | SequenceTransfer[Tensor]
|
|
required |
shared
|
SequenceShared
|
The backbone shared input broadcast to every stage. |
required |
Returns:
| Type | Description |
|---|---|
SequenceTransfer[Tensor]
|
The produced |
forward(inputs, shared)
Runs this stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs
|
TPipelineInput | TStageTransfer
|
|
required |
shared
|
TSharedInput
|
The value broadcast to every stage. |
required |
Returns:
| Type | Description |
|---|---|
TStageTransfer | TPipelineOutput
|
|
reset_parameters()
Resets the module parameters (i.e. performs random initialization).
stage_transfer_spec(pipeline_input, boundary)
Describes the SequenceTransfer crossing the given boundary of this stage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pipeline_input
|
SequenceInput
|
A representative |
required |
boundary
|
StageBoundary
|
Which inter-stage edge to describe. |
required |
Returns:
| Type | Description |
|---|---|
SequenceTransfer[TensorSpec]
|
A |
DecoderForCausalLM
Bases: DecoderWithHead[TBackbone, SplitLanguageModellingHead, SequenceCausalLMHeadShared, SequenceCausalLMOutput]
A decoder backbone composed with a single causal language modeling head.
It builds the head itself, so composing the model is one call. Its I/O types are fixed to that head's shared input and output.
__init__(backbone, stage)
Constructs the DecoderForCausalLM object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
backbone
|
TBackbone
|
The decoder backbone, exposed as |
required |
stage
|
PipelineStageInfo
|
Pipeline stage information for this instance. |
required |
forward(inputs, shared)
Runs the backbone and, on the last stage, the attached head.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs
|
SequenceInput | SequenceTransfer[Tensor]
|
|
required |
shared
|
SequenceHeadShared[THeadShared]
|
The backbone shared input plus the head's shared input. |
required |
Returns:
| Type | Description |
|---|---|
SequenceTransfer[Tensor] | THeadOutput
|
The produced |
reset_parameters()
Resets the parameters of the backbone and of the head.
stage_transfer_spec(pipeline_input, boundary)
Describes the SequenceTransfer crossing the given boundary, as the backbone does.
The head only runs on the last stage, which has no outgoing transfer, so the backbone alone defines the transfer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pipeline_input
|
SequenceInput
|
A representative |
required |
boundary
|
StageBoundary
|
Which inter-stage edge to describe. |
required |
Returns:
| Type | Description |
|---|---|
SequenceTransfer[TensorSpec]
|
A |
DecoderForClassification
Bases: DecoderWithHead[TBackbone, ClassificationHead, SequencePoolingHeadShared, SequenceClassificationOutput]
A decoder backbone composed with a single classification head.
It builds the head itself, so composing the model is one call. Its I/O types are fixed to that head's shared input and output.
__init__(backbone, config, stage)
Constructs the DecoderForClassification object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
backbone
|
TBackbone
|
The decoder backbone, exposed as |
required |
config
|
ClassificationHeadConfig
|
Configuration of the classification head to build. |
required |
stage
|
PipelineStageInfo
|
Pipeline stage information for this instance. |
required |
forward(inputs, shared)
Runs the backbone and, on the last stage, the attached head.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs
|
SequenceInput | SequenceTransfer[Tensor]
|
|
required |
shared
|
SequenceHeadShared[THeadShared]
|
The backbone shared input plus the head's shared input. |
required |
Returns:
| Type | Description |
|---|---|
SequenceTransfer[Tensor] | THeadOutput
|
The produced |
reset_parameters()
Resets the parameters of the backbone and of the head.
stage_transfer_spec(pipeline_input, boundary)
Describes the SequenceTransfer crossing the given boundary, as the backbone does.
The head only runs on the last stage, which has no outgoing transfer, so the backbone alone defines the transfer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pipeline_input
|
SequenceInput
|
A representative |
required |
boundary
|
StageBoundary
|
Which inter-stage edge to describe. |
required |
Returns:
| Type | Description |
|---|---|
SequenceTransfer[TensorSpec]
|
A |
DecoderForEmbedding
Bases: DecoderWithHead[TBackbone, EmbeddingHead, SequencePoolingHeadShared, SequenceEmbeddingOutput]
A decoder backbone composed with a single embedding head.
It builds the head itself, so composing the model is one call. Its I/O types are fixed to that head's shared input and output.
__init__(backbone, config, stage)
Constructs the DecoderForEmbedding object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
backbone
|
TBackbone
|
The decoder backbone, exposed as |
required |
config
|
EmbeddingHeadConfig
|
Configuration of the embedding head to build. |
required |
stage
|
PipelineStageInfo
|
Pipeline stage information for this instance. |
required |
forward(inputs, shared)
Runs the backbone and, on the last stage, the attached head.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs
|
SequenceInput | SequenceTransfer[Tensor]
|
|
required |
shared
|
SequenceHeadShared[THeadShared]
|
The backbone shared input plus the head's shared input. |
required |
Returns:
| Type | Description |
|---|---|
SequenceTransfer[Tensor] | THeadOutput
|
The produced |
reset_parameters()
Resets the parameters of the backbone and of the head.
stage_transfer_spec(pipeline_input, boundary)
Describes the SequenceTransfer crossing the given boundary, as the backbone does.
The head only runs on the last stage, which has no outgoing transfer, so the backbone alone defines the transfer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pipeline_input
|
SequenceInput
|
A representative |
required |
boundary
|
StageBoundary
|
Which inter-stage edge to describe. |
required |
Returns:
| Type | Description |
|---|---|
SequenceTransfer[TensorSpec]
|
A |
DecoderWithHead
Bases: Module, ModuleLateInit, ModuleSupportsPipelining[SequenceInput, SequenceTransfer[Tensor], SequenceHeadShared[THeadShared], THeadOutput], Generic[TBackbone, THead, THeadShared, THeadOutput]
Model that composes one decoder backbone with exactly one prebuilt task head.
This is the single-head counterpart of DecoderWithHeads, not a special case of it. With one
head there is nothing to key. The head is self.head (FQN head.*), its shared input is a
field, and the model returns the head's output instead of a one-entry mapping. For example, a
causal LM task reads ctx.pipeline_results.cross_entropy directly.
The backbone (self.model) and the head (self.head) are public, so a provider can
parallelize and checkpoint-map each of them separately. The head is attached only on the last
pipeline stage.
Class Type Parameters:
| Name | Bound or Constraints | Description | Default |
|---|---|---|---|
TBackbone
|
The backbone type, kept precise so |
required | |
THead
|
The head type, kept precise so |
required | |
THeadShared
|
The shared input accepted by the composed head. |
required | |
THeadOutput
|
The output produced by the composed head. |
required |
__init__(backbone, head, stage)
Constructs the DecoderWithHead object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
backbone
|
TBackbone
|
The decoder backbone, exposed as |
required |
head
|
THead
|
The prebuilt task head, attached on the last stage as |
required |
stage
|
PipelineStageInfo
|
Pipeline stage information for this instance. |
required |
forward(inputs, shared)
Runs the backbone and, on the last stage, the attached head.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs
|
SequenceInput | SequenceTransfer[Tensor]
|
|
required |
shared
|
SequenceHeadShared[THeadShared]
|
The backbone shared input plus the head's shared input. |
required |
Returns:
| Type | Description |
|---|---|
SequenceTransfer[Tensor] | THeadOutput
|
The produced |
reset_parameters()
Resets the parameters of the backbone and of the head.
stage_transfer_spec(pipeline_input, boundary)
Describes the SequenceTransfer crossing the given boundary, as the backbone does.
The head only runs on the last stage, which has no outgoing transfer, so the backbone alone defines the transfer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pipeline_input
|
SequenceInput
|
A representative |
required |
boundary
|
StageBoundary
|
Which inter-stage edge to describe. |
required |
Returns:
| Type | Description |
|---|---|
SequenceTransfer[TensorSpec]
|
A |
DecoderWithHeads
Bases: Module, ModuleLateInit, ModuleSupportsPipelining[SequenceInput, SequenceTransfer[Tensor], SequenceHeadsShared[THeadShared], SequenceHeadsOutput[THeadOutput]], Generic[TBackbone, THeadShared, THeadOutput]
Model that composes one decoder backbone with a mapping of prebuilt named task heads.
For a model with a single head, use DecoderWithHead instead. With one head there is nothing
to key, and this class would make every caller name the only head.
The backbone (self.model) and the heads (self.heads) are public, so a provider can
parallelize and checkpoint-map each of them separately. This class does not build heads: use
build_decoder_head for built-in heads, or pass custom heads in the same way. Heads are
attached only on the last pipeline stage.
For heads of one kind, the head I/O type parameters are that kind's shared input and output. For heads of several kinds, they are the unions of those types.
Class Type Parameters:
| Name | Bound or Constraints | Description | Default |
|---|---|---|---|
TBackbone
|
The backbone type, kept precise so |
required | |
THeadShared
|
The shared input accepted by the composed heads. |
required | |
THeadOutput
|
The output produced by the composed heads. |
required |
__init__(backbone, heads, stage)
Constructs the DecoderWithHeads object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
backbone
|
TBackbone
|
The decoder backbone, exposed as |
required |
heads
|
Mapping[str, TaskHead[THeadShared, THeadOutput]]
|
A mapping of head names to prebuilt task heads, attached on the last stage
as |
required |
stage
|
PipelineStageInfo
|
Pipeline stage information for this instance. |
required |
forward(inputs, shared)
Runs the backbone and, on the last stage, every attached head.
The backbone receives shared.sequence. Each head receives the entry of shared.heads
stored under its name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
inputs
|
SequenceInput | SequenceTransfer[Tensor]
|
|
required |
shared
|
SequenceHeadsShared[THeadShared]
|
The backbone shared input plus the per-head shared inputs. |
required |
Returns:
| Type | Description |
|---|---|
SequenceTransfer[Tensor] | SequenceHeadsOutput[THeadOutput]
|
The produced |
reset_parameters()
Resets the parameters of the backbone and of every head.
stage_transfer_spec(pipeline_input, boundary)
Describes the SequenceTransfer crossing the given boundary, as the backbone does.
Heads only run on the last stage, which has no outgoing transfer, so the backbone alone defines the transfer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pipeline_input
|
SequenceInput
|
A representative |
required |
boundary
|
StageBoundary
|
Which inter-stage edge to describe. |
required |
Returns:
| Type | Description |
|---|---|
SequenceTransfer[TensorSpec]
|
A |
EmbeddingHeadConfig
Bases: BaseModel
Configuration for a dense embedding head.
Attributes:
| Name | Type | Description |
|---|---|---|
kind |
Literal['embedding']
|
Discriminator field. Always |
embedding_dim |
int | None
|
Size of the output embedding. |
normalize |
bool
|
Whether to apply L2 normalization to the final embeddings. |
build_decoder_head(config, *, backbone)
Builds a task head from its configuration and the decoder backbone it attaches to.
The head takes hidden_size and the split-vocabulary layout from the backbone, so a config
holds only task-specific fields. A custom head does not go through this factory: pass its
TaskHead instance to the decoder directly.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
AnyHeadConfig
|
Task head configuration selecting the head type and its task-specific fields. |
required |
backbone
|
DecoderBackbone
|
The decoder backbone the head attaches to. It provides the shared dimensions. |
required |
Returns:
| Type | Description |
|---|---|
TaskHead
|
An instantiated task head. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the head configuration type is unknown. |
parallelize_causal_lm_head(head, dist_context)
Applies Hybrid Sharded Data Parallel (HSDP) to a causal language modeling head.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
head
|
SplitLanguageModellingHead
|
The language modeling head to parallelize. |
required |
dist_context
|
DistributedContext
|
The distributed context that provides the device meshes. |
required |
parallelize_classification_head(head, dist_context)
Applies Hybrid Sharded Data Parallel (HSDP) to a classification head.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
head
|
ClassificationHead
|
The classification head to parallelize. |
required |
dist_context
|
DistributedContext
|
The distributed context that provides the device meshes. |
required |
parallelize_embedding_head(head, dist_context)
Applies Hybrid Sharded Data Parallel (HSDP) to an embedding head.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
head
|
EmbeddingHead
|
The embedding head to parallelize. |
required |
dist_context
|
DistributedContext
|
The distributed context that provides the device meshes. |
required |
d9d.models.io
Typed pipeline I/O dataclasses for the model catalogue.
SequenceHeadsOutput = Mapping[str, THeadOutput]
module-attribute
The output of a backbone composed with named task heads: each head's output, keyed by head name.
Two heads of the same type have different names, so their outputs cannot collide. The type
parameter is the output of the composed heads. It is a single type for heads of one kind (e.g.
SequenceHeadsOutput[SequenceCausalLMOutput]), or a union for heads of several kinds.
SequenceHeadShared
dataclass
Bases: Generic[THeadShared]
The shared input a backbone composed with exactly one task head consumes on every stage.
This is the single-head counterpart of SequenceHeadsShared. The head input is a field, not a
named entry, and the model returns the head's output unwrapped.
Class Type Parameters:
| Name | Bound or Constraints | Description | Default |
|---|---|---|---|
THeadShared
|
The shared input accepted by the composed head. |
required |
Attributes:
| Name | Type | Description |
|---|---|---|
sequence |
SequenceShared
|
The backbone shared input. |
head |
THeadShared
|
The head's own shared input. Read on the last stage only. |
SequenceHeadsShared
dataclass
Bases: Generic[THeadShared]
The shared input a backbone composed with named task heads consumes on every stage.
The type parameter is the shared input of the composed heads. It is a single type for heads of
one kind (e.g. SequenceHeadsShared[SequenceCausalLMHeadShared]), or a union for heads of
several kinds.
Class Type Parameters:
| Name | Bound or Constraints | Description | Default |
|---|---|---|---|
THeadShared
|
The shared input accepted by the composed heads. |
required |
Attributes:
| Name | Type | Description |
|---|---|---|
sequence |
SequenceShared
|
The backbone shared input. |
heads |
Mapping[str, THeadShared]
|
Each head's own shared input, keyed by head name. Read on the last stage only. |
SequenceInput
dataclass
The input of a sequence transformer: the token ids fed to the first stage.
Attributes:
| Name | Type | Description |
|---|---|---|
input_ids |
Tensor
|
Indices of the input tokens. Shape: |
SequenceShared
dataclass
The shared input the transformer backbone consumes on every stage.
Attributes:
| Name | Type | Description |
|---|---|---|
position_ids |
Tensor
|
The position of each token in the position embeddings.
Shape: |
hidden_states_agg_mask |
Tensor | None
|
Mask of the tokens to aggregate into hidden state snapshots, or |
SequenceTransfer
dataclass
Bases: Generic[TLeaf]
The object moved between adjacent stages of a sequence transformer.
Attributes:
| Name | Type | Description |
|---|---|---|
hidden_states |
TLeaf
|
The output of the last layer of the sending stage.
Shape: |
hidden_states_snapshot |
TLeaf | None
|
The aggregated hidden states of the embeddings and of all layers so far, or
|