Metric Collection
About
AsyncMetricCollector processes a metric without blocking the training loop. The Metric interface is synchronous. The collector wraps a metric and runs its synchronization and computation on a side CUDA stream. So the training loop continues at once and does not wait for the metric all-reduce. Collecting the results makes the current stream wait for the side stream, but not the host: the results stay tensors on the GPU.
The Trainer collects its metrics this way. You need the collector only in your own training loop or logging.
Usage
API Reference
d9d.metric.AsyncMetricCollector
Asynchronous metric collector that synchronizes and computes a metric on a side CUDA stream.
The distributed reduction and the computation run on the side stream, off the main training stream.
__init__(metric)
Constructs the AsyncMetricCollector object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
metric
|
Metric
|
The metric to collect. |
required |
bind()
Moves the metric to CUDA and creates the side stream.
collect_results()
Returns the results of the asynchronous computation and resets the metric.
The current CUDA stream waits for the computation, the host does not.
Returns:
| Type | Description |
|---|---|
PyTree[Tensor]
|
A PyTree with the structure of the metric output and tensors on the device of the metric as leaves. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the collector is not bound, or if |
schedule_collection(group)
Schedules the metric synchronization and computation on the side stream.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
group
|
ProcessGroup | None
|
The process group to synchronize the metric over before the computation. |
required |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the collector is not bound. |
unbind()
Releases the reference to the side stream.