API reference: MosaicML Composer integration#

The NeptuneLogger class of the MosaicML Composer library captures metadata that is generated when training models with Composer.

You can use its .neptune_run property to access the associated Neptune run, and the .base_handler property to assign custom metadata directly to the base namespace used for the metadata tracked by the logger.

`NeptuneLogger`#

Creates a Neptune logger for tracking metadata during training with Composer.

Parameters

Name	Type	Default	Description
`project`	`str`, optional	`None`	The name of your Neptune project, in the form `workspace-name/project-name`. If you leave it empty, the value of the `NEPTUNE_PROJECT` environment variable is used.
`api_token`	`str`, optional	`None`	Account's Neptune API token. If `None`, the value of the `NEPTUNE_API_TOKEN` environment variable is used. To keep your token secure, avoid placing it in the source code. Instead, save it as an environment variable.
`rank_zero_only`	`bool`, optional	`True`	Whether to log only on the rank-zero process.
`upload_checkpoints`	`bool`, optional	`False`	Whether to upload model checkpoints to Neptune.
`base_namespace`	`str`, optional	`training`	The name of the namespace where metadata logged by the Neptune logger will be stored.
`**neptune_kwargs`	`Dict[str, Any]`, optional	-	Additional keyword arguments to be passed directly to the `init_run()` function, such as `description` and `tags`.

Examples

If you have your Neptune credentials saved as environment variables, the following enables the Neptune logger with default settings:

from composer import Trainer
from composer.loggers import NeptuneLogger

neptune_logger = NeptuneLogger()

trainer = Trainer(
    ...,
    loggers=neptune_logger,
)

trainer.fit()

How do I save my credentials as environment variables?

Set your Neptune API token and full project name to the NEPTUNE_API_TOKEN and NEPTUNE_PROJECT environment variables, respectively.

Linux macOS Windows Jupyter Notebook

export NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

export NEPTUNE_PROJECT="ml-team/classification"

export NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

export NEPTUNE_PROJECT="ml-team/classification"

setx NEPTUNE_API_TOKEN "h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

setx NEPTUNE_PROJECT "ml-team/classification"

You can also navigate to Settings → Edit the system environment variables and add the variables there.

%env NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

%env NEPTUNE_PROJECT="ml-team/classification"

To find your credentials:

API token: In the bottom-left corner of the Neptune app, expand your user menu and select Get your API token. If you need the token of a service account, go to the workspace or project settings and enter the Service accounts settings.
Project name: Your full project name has the form workspace-name/project-name. You can copy it from the project menu ( → Edit project details).

If you're working in Google Colab, you can set your credentials with the os and getpass libraries:

import os
from getpass import getpass
os.environ["NEPTUNE_API_TOKEN"] = getpass("Enter your Neptune API token: ")
os.environ["NEPTUNE_PROJECT"] = "workspace-name/project-name"

You can also pass more options to the run created by the logger:

neptune_logger = NeptuneLogger(
    project="workspace-name/project-name", # (1)!
    upload_checkpoints=True,
    name="mellow-panda",
    description="Quick training run with Composer",
    tags=["training", "composer"],
    dependencies="infer",
)

The full project name. For example, "ml-team/classification".

To copy it to your clipboard, navigate to the project settings in the top-right () and select Edit project details.

Show init_run() parameters list

Name	Type	Default	Description
`project`	`str`, optional	`None`	Name of a project in the form `workspace-name/project-name`. If `None`, the value of the `NEPTUNE_PROJECT` environment variable is used.
`api_token`	`str`, optional	`None`	Your Neptune API token (or a service account's API token). If `None`, the value of the `NEPTUNE_API_TOKEN` environment variable is used. To keep your token secure, avoid placing it in source code. Instead, save it as an environment variable.
`with_id`	`str`, optional	`None`	The Neptune identifier of an existing run to resume, such as "CLS-11". The identifier is stored in the object's `sys/id` field. If omitted or `None` is passed, a new tracked run is created.
`custom_run_id`	`str`, optional	`None`	A unique identifier that can be used to log metadata to a single run from multiple locations. Max length: 36 characters. If `None` and the `NEPTUNE_CUSTOM_RUN_ID` environment variable is set, Neptune will use that as the `custom_run_id` value. For details, see Set custom run ID.
`mode`	`str`, optional	`async`	Connection mode in which the logging will work. Possible values are `async`, `sync`, `offline`, `read-only`, and `debug`. If you leave it out, the value of the `NEPTUNE_MODE` environment variable is used. If that's not set, the default `async` is used.
`name`	`str`, optional	`"Untitled"`	Custom name for the run. You can use it as a human-readable ID and add it as a column in the runs table (`sys/name`).
`description`	`str`, optional	`""`	Editable description of the run. You can add it as a column in the runs table (`sys/description`).
`tags`	`list`, optional	`[]`	Must be a list of `str` which represent the tags for the run. You can edit them after run is created, either in the run information or runs table.
`source_files`	`list` or `str`, optional	`None`	List of source files to be uploaded. Must be list of `str` or a single `str`. Uploaded sources are displayed in the Source code section of the run. If `None` is passed, the Python file from which the run was created will be uploaded. When resuming a run, no file will be uploaded by default. Pass an empty list (`[]`) to upload no files. Unix style pathname pattern expansion is supported. For example, you can pass `".py"` to upload all Python source files from the current directory. Paths of uploaded files are resolved relative to the calculated common root of all uploaded source files. For recursion lookup, use `"**/.py"` (for Python `3.5` and later). For details, see the glob library.
`capture_stdout`	`Boolean`, optional	`True`	Whether to log the standard output stream. Is logged in the `monitoring` namespace.
`capture_stderr`	`Boolean`, optional	`True`	Whether to log the standard error stream. Is logged in the `monitoring` namespace.
`capture_hardware_metrics`	`Boolean`, optional	`True`	Whether to track hardware consumption (CPU, GPU, memory utilization). Logged in the `monitoring` namespace.
`fail_on_exception`	`Boolean`, optional	`True`	If an uncaught exception occurs, whether to set run's `Failed` state to `True`.
`monitoring_namespace`	`str`, optional	`"monitoring"`	Namespace inside which all monitoring logs will be stored.
`flush_period`	`float`, optional	`5` (seconds)	In asynchronous (default) connection mode, how often Neptune should trigger disk flushing.
`proxies`	`dict`, optional	`None`	Argument passed to HTTP calls made via the Requests library. For details on proxies, see the Requests documentation.
`capture_traceback`	`Boolean`, optional	`True`	In case of an exception, whether to log the traceback of the run.
`git_ref`	`GitRef` or `Boolean`	`None`	`GitRef` object containing information about the Git repository path. If `None`, Neptune looks for a repository in the path of the script that is executed. To specify a different location, set to `GitRef(repository_path="path/to/repo")`. To turn off Git tracking for the run, set to `GitRef.DISABLED` or `False`. For examples, see Logging Git info.
`dependencies`	`str`, optional	`None`	Tracks environment requirements. If you pass `"infer"` to this argument, Neptune logs dependencies installed in the current environment. You can also pass a path to your dependency file directly. If left empty, no dependency file is uploaded.
`async_lag_callback`	`NeptuneObjectCallback`, optional	`None`	Custom callback function which is called if the lag between a queued operation and its synchronization with the server exceeds the duration defined by `async_lag_threshold`. The callback should take a `Run` object as the argument and can contain any custom code, such as calling `stop()` on the object. Note: Instead of using this argument, you can use Neptune's default callback by setting the `NEPTUNE_ENABLE_DEFAULT_ASYNC_LAG_CALLBACK` environment variable to `TRUE`.
`async_lag_threshold`	`float`, optional	`1800.0` (seconds)	Duration between the queueing and synchronization of an operation. If a lag callback (default callback enabled via environment variable or custom callback passed to the `async_lag_callback` argument) is enabled, the callback is called when this duration is exceeded.
`async_no_progress_callback`	`NeptuneObjectCallback`, optional	`None`	Custom callback function which is called if there has been no synchronization progress whatsoever for the duration defined by `async_no_progress_threshold`. The callback should take a `Run` object as the argument and can contain any custom code, such as calling `stop()` on the object. Note: Instead of using this argument, you can use Neptune's default callback by setting the `NEPTUNE_ENABLE_DEFAULT_ASYNC_NO_PROGRESS_CALLBACK` environment variable to `TRUE`.
`async_no_progress_threshold`	`float`, optional	`300.0` (seconds)	For how long there has been no synchronization progress. If a no-progress callback (default callback enabled via environment variable or custom callback passed to the `async_no_progress_callback` argument) is enabled, the callback is called when this duration is exceeded.

`.base_handler`#

Gets a namespace handler for the base logging namespace.

Use the handler to log extra metadata to the run and organize it under the base namespace (default: training).

You can operate on the handler like a Run object: Access a path inside the handler and assign metadata to it with = or other logging methods from the Neptune client library. For some suggestions, see Essential logging methods.

from composer.loggers import NeptuneLogger

neptune_logger = NeptuneLogger()
trainer = Trainer(loggers=neptune_logger, ...)
trainer.fit()

neptune_logger.base_handler["some_metric"] = 1
trainer.close()

Result

The value 1 is organized under training/some_metric in the run structure.

To learn more about namespace handlers, see Setting a base namespace.

`.neptune_run`#

Gets the Neptune Run object of the NeptuneLogger instance.

You can then access fields inside the run and use any logging methods from the Neptune client library. For some suggestions, see Essential logging methods.

from composer.loggers import NeptuneLogger

neptune_logger = NeptuneLogger()
trainer = Trainer(loggers=neptune_logger, ...)
trainer.fit()

neptune_logger.neptune_run["your/metadata/structure"] = some_metadata
neptune_logger.neptune_run["some_file"].upload("path/to/some/file")

trainer.close()

Structurally, this will bypass the base namespace set for the logger.

By default, the metadata is logged in the namespace <run root>/training, but the below code would log scores directly at the run root and not under training.

neptune_logger.neptune_run["scores"] = ...

To log under the base namespace instead, see .base_handler property.

API reference: MosaicML Composer integration#

NeptuneLogger#

.base_handler#

.neptune_run#

`NeptuneLogger`#

`.base_handler`#

`.neptune_run`#