Skip to content

Event Bus and Hooks

About

d9d does not use a fixed set of lifecycle methods, such as on_step_start or on_post_optimizer. Instead, a typed event bus lets you extend both the training and the inference loops. Any user component (TrainTask, ModelProvider, etc.) can subscribe to specific points of the execution. You subscribe only to the events you need, so your code does not depend on the internal execution order.

How It Works

The system has three core concepts:

  1. Event[TContext]: A typed descriptor of a specific moment in the lifecycle.
  2. Contexts: Data classes (e.g. EventStepContext) that hold the state relevant to the event.
  3. EventBus: The dispatcher that passes contexts to the subscribed handlers.

Note

The event bus does not catch exceptions. If a handler raises, the loop stops immediately.

Registering Handlers

BaseTask (and so TrainTask and InferenceTask) and ModelProvider have a register_events hook. The loop calls it during configuration and passes a context that holds the EventBus.

Mark your methods with the @subscribe decorator. Then call subscribe_annotated to register all of them at once.

from d9d.loop.control import RegisterTaskEventsContext, TrainTask
from d9d.loop.event import subscribe, subscribe_annotated
from d9d.loop.event.catalogue.common import EventModelStagesReadyContext, EventStepContext
from d9d.loop.event.catalogue.train import EVENT_TRAIN_MODEL_STAGES_READY, EVENT_TRAIN_STEP_POST


class CustomTrainTask(TrainTask):
    def __init__(self):
        self._modules = []

    def register_events(self, context: RegisterTaskEventsContext) -> None:
        # Registers every method of this instance marked with @subscribe.
        subscribe_annotated(context.event_bus, self)

    @subscribe(EVENT_TRAIN_MODEL_STAGES_READY)
    def _on_model_ready(self, ctx: EventModelStagesReadyContext) -> None:
        self._modules = ctx.modules

    @subscribe(EVENT_TRAIN_STEP_POST)
    def _on_step_post(self, ctx: EventStepContext) -> None:
        print(f"Step {ctx.schedule.current_step} of {ctx.schedule.total_steps} completed.")

    def build_forward_inputs(self, ctx):
        ...  # Task logic

    def compute_loss(self, ctx):
        ...  # Task logic

Manual Registration

You can also call the EventBus directly. This is useful for simple lambda callbacks or handlers created at runtime.

from d9d.loop.control import RegisterTaskEventsContext, TrainTask
from d9d.loop.event.catalogue.train import EVENT_TRAIN_OPTIMIZER_READY


class CustomTrainTask(TrainTask):
    def register_events(self, context: RegisterTaskEventsContext) -> None:
        context.event_bus.subscribe(
            EVENT_TRAIN_OPTIMIZER_READY,
            lambda event_ctx: print(f"Optimizer loaded: {event_ctx.optimizer}"),
        )

    def build_forward_inputs(self, ctx):
        ...  # Task logic

    def compute_loss(self, ctx):
        ...  # Task logic

Custom Events

You can define and trigger your own events in custom logic.

import dataclasses
from pathlib import Path

from d9d.loop.event import Event, EventBus


@dataclasses.dataclass(kw_only=True)
class CheckpointContext:
    step: int
    path: Path


# Define a new event.
EVENT_CHECKPOINT_SAVED = Event[CheckpointContext](id="user.checkpoint_saved")


# Trigger it from your task.
def process_something(bus: EventBus):
    bus.trigger(
        EVENT_CHECKPOINT_SAVED,
        CheckpointContext(step=1000, path=Path("/checkpoints/step_1000")),
    )

API Reference

Core Components

d9d.loop.event

Event bus and subscription API for the training and inference loops.

Event dataclass

Bases: Generic[TContext]

Typed event descriptor.

Each event has a unique string identifier and an associated context type. Events are defined as module-level constants and used as keys for subscription and triggering.

Attributes:

Name Type Description
id str

Unique identifier for the event.

EventBus

Registry of handlers for typed events that calls them when an event is triggered.

__init__()

Constructs the EventBus object.

bounded(event_pre, event_post, context)

Context manager that triggers a pre-event on entry and a post-event on exit.

Parameters:

Name Type Description Default
event_pre Event[TContext]

The event to trigger immediately before yielding.

required
event_post Event[TContext]

The event to trigger immediately after the block completes successfully.

required
context TContext

The context object passed to both events.

required
subscribe(event, handler)

Registers a handler function to be executed when a specific event occurs.

Parameters:

Name Type Description Default
event Event[TContext]

The event descriptor to subscribe to.

required
handler Callable[[TContext], None]

The callback function to execute when the event is triggered.

required
trigger(event, context)

Dispatches an event to all its registered handlers with the given context.

Parameters:

Name Type Description Default
event Event[TContext]

The event descriptor to trigger.

required
context TContext

The data associated with the event to pass to the handlers.

required
subscribe(event)

Marks a method to be subscribed to an event.

The decorator does not register the method. Call subscribe_annotated on the instance to register it.

Parameters:

Name Type Description Default
event Event[TContext]

Event descriptor to bind this method to.

required

Returns:

Type Description
Callable[[Callable[[TContext], None]], Callable[[TContext], None]]

The decorated function.

subscribe_annotated(bus, target)

Subscribes all methods of the target object that are decorated with @subscribe.

Parameters:

Name Type Description Default
bus EventBus

The event bus to register the handlers on.

required
target object

The initialized class instance containing the decorated methods.

required

Common Events

d9d.loop.event.catalogue.common

Event contexts shared by the training and inference loops.

EventConfigurationStartedContext dataclass

Context provided when the loop configuration starts.

Attributes:

Name Type Description
dist_context DistributedContext

The initialized distributed execution context.

EventDataStreamReadyContext dataclass

Context provided when the microbatch pack stream has been fully initialized.

Attributes:

Name Type Description
stream MicrobatchPackStream

The microbatch pack stream instance.

EventModelStagesReadyContext dataclass

Context provided when the model stages are initialized and parallelized.

Attributes:

Name Type Description
modules list[Module]

The references to the model stages.

EventStepContext dataclass

Context providing step information during iterative execution.

Attributes:

Name Type Description
schedule JobSchedule

The schedule that tracks the current step and the total number of steps.

Training Events

d9d.loop.event.catalogue.train

Events of the training loop.

EVENT_TRAIN_CONFIG_STARTED = Event[EventConfigurationStartedContext](id='train.configuration.start') module-attribute

Triggered when the training configuration process begins. Provides access to the distributed context.

EVENT_TRAIN_DATA_STREAM_READY = Event[EventDataStreamReadyContext](id='train.configuration.data_stream') module-attribute

Triggered when the training microbatch pack stream has been fully initialized.

EVENT_TRAIN_FINISHED = Event[EventTrainFinishedContext](id='train.finished') module-attribute

Triggered when the entire training execution loop finishes successfully.

EVENT_TRAIN_FORWARD_BACKWARD_POST = Event[EventStepContext](id='train.forward_backward.post') module-attribute

Triggered immediately after all forward and backward passes for the current step have finished.

EVENT_TRAIN_FORWARD_BACKWARD_PRE = Event[EventStepContext](id='train.forward_backward.pre') module-attribute

Triggered immediately before the sequence of forward and backward passes begins.

EVENT_TRAIN_LR_SCHEDULER_READY = Event[EventLRSchedulerReadyContext](id='train.configuration.lr_scheduler') module-attribute

Triggered when the learning rate scheduler has been configured.

EVENT_TRAIN_MODEL_STAGES_READY = Event[EventModelStagesReadyContext](id='train.configuration.model_stages') module-attribute

Triggered when the model stages are initialized and parallelized.

EVENT_TRAIN_OPTIMIZER_READY = Event[EventOptimizerReadyContext](id='train.configuration.optimizer') module-attribute

Triggered when the optimizer has been built and is ready for use.

EVENT_TRAIN_OPTIMIZER_STEP_POST = Event[EventStepContext](id='train.optimizer_step.post') module-attribute

Triggered immediately after the optimizer has updated the model parameters but before gradients are zeroed.

EVENT_TRAIN_OPTIMIZER_STEP_PRE = Event[EventStepContext](id='train.optimizer_step.pre') module-attribute

Triggered right before the optimizer updates the model parameters, after gradients are scaled and clipped.

EVENT_TRAIN_READY = Event[EventTrainReadyContext](id='train.ready') module-attribute

Triggered right before the main loop starts, after configuration is complete and checkpoints are loaded.

EVENT_TRAIN_SLEEP_POST = Event[EventSleepContext](id='train.sleep.post') module-attribute

Triggered after every selected subsystem has been offloaded and the device cache has been emptied.

EVENT_TRAIN_SLEEP_PRE = Event[EventSleepContext](id='train.sleep.pre') module-attribute

Triggered before any subsystem is offloaded. The training state is still fully resident on GPU.

EVENT_TRAIN_STEP_POST = Event[EventStepContext](id='train.step.post') module-attribute

Triggered at the end of a training step, after all its work except checkpointing.

EVENT_TRAIN_STEP_PRE = Event[EventStepContext](id='train.step.pre') module-attribute

Triggered at the start of a training step.

EVENT_TRAIN_WAKE_POST = Event[EventSleepContext](id='train.wake.post') module-attribute

Triggered after every selected subsystem has been restored to GPU.

EVENT_TRAIN_WAKE_PRE = Event[EventSleepContext](id='train.wake.pre') module-attribute

Triggered before any subsystem is restored to GPU.

EventLRSchedulerReadyContext dataclass

Context provided when the learning rate scheduler has been instantiated.

Attributes:

Name Type Description
lr_scheduler LRSchedulerProtocol

The learning rate scheduler instance.

EventOptimizerReadyContext dataclass

Context provided when the optimizer has been instantiated.

Attributes:

Name Type Description
optimizer OptimizerProtocol

The optimizer instance wrapping the model parameters.

EventSleepContext dataclass

Context provided for a sleep or wake lifecycle transition.

Attributes:

Name Type Description
tags frozenset[SleepTag]

The set of subsystem tags involved in this transition.

EventTrainFinishedContext dataclass

Context provided when the entire training loop has completed successfully.

EventTrainReadyContext dataclass

Context provided when training is fully ready to begin and the checkpoint is loaded.

Attributes:

Name Type Description
writer TrackerWriter

The writer that logs values, including GPU tensors, to the experiment tracker at the current step.

Inference Events

d9d.loop.event.catalogue.inference

Events of the inference loop.

EVENT_INFERENCE_CONFIG_STARTED = Event[EventConfigurationStartedContext](id='inference.configuration.start') module-attribute

Triggered when the inference configuration process begins. Provides access to the distributed context.

EVENT_INFERENCE_DATA_STREAM_READY = Event[EventDataStreamReadyContext](id='inference.configuration.data_stream') module-attribute

Triggered when the inference microbatch pack stream has been fully initialized.

EVENT_INFERENCE_FINISHED = Event[EventInferenceFinishedContext](id='inference.finished') module-attribute

Triggered when the entire inference execution loop finishes successfully.

EVENT_INFERENCE_FORWARD_POST = Event[EventStepContext](id='inference.forward.post') module-attribute

Triggered immediately after the forward pass sequence for the current step has finished.

EVENT_INFERENCE_FORWARD_PRE = Event[EventStepContext](id='inference.forward.pre') module-attribute

Triggered immediately before the forward pass sequence begins.

EVENT_INFERENCE_MODEL_STAGES_READY = Event[EventModelStagesReadyContext](id='inference.configuration.model_stages') module-attribute

Triggered when the model stages are initialized for inference.

EVENT_INFERENCE_READY = Event[EventInferenceReadyContext](id='inference.ready') module-attribute

Triggered right before the main inference loop starts, after configuration is complete and checkpoints are loaded.

EVENT_INFERENCE_STEP_POST = Event[EventStepContext](id='inference.step.post') module-attribute

Triggered at the end of an inference step, before checkpointing.

EVENT_INFERENCE_STEP_PRE = Event[EventStepContext](id='inference.step.pre') module-attribute

Triggered at the start of an inference step.

EventInferenceFinishedContext dataclass

Context provided when the entire inference loop has completed successfully.

EventInferenceReadyContext dataclass

Context provided when inference is fully ready to begin and the checkpoint is loaded.