Experiments

An experiment is a fundamental concept in Neptune – every program you run, every Notebook you use within Neptune is an experiment. Each experiment consists of source code and its metadata, which represent its execution. If you are interested in running an experiment with different parameters please read about hyperparameter optimization.

Experiment’s Metadata

Experiment’s metadata is divided in three groups:

  • metadata defined by the user before creation of the experiment include things like experiment’s name, parameters, or environment files which should be accessible to experiment in the runtime;
  • metadata defined in code include channels, actions, tags and properties (key-value pairs that you can use to store additional information);
  • finally, there is some metadata set by Neptune itself, like experiment’s state.

Experiment States

Each experiment can be in one of the states listed below:

  • Creating – the experiment is being created, experiment sources are being uploaded;
  • Waiting – the experiment is registered in Neptune queue and is waiting for execution in the cloud;
  • Initializing – the experiment is preparing for execution (installing pip requirements etc.);
  • Running – experiment code execution has started;
  • Cleaning – experiment code execution has finished, Neptune is cleaning up (uploading results etc.);
  • Succeeded – the experiment has successfully finished;
  • Failed – the experiment has finished with an error;
  • Aborted – the experiment was aborted on user’s request;
  • Crashed – the experiment has crashed (Out of memory error etc.).

Each running experiment can occasionally become “not responding”. When an experiment is “not responding”, it means that there are communication problems between the experiment and Neptune. The communication problems can be temporary, meaning that after some time the experiment can become responsive again.

Parameters

The experiment can have a set of parameters associated with it. Each parameter has a name, value and an optional description. Parameters tracked by Neptune are available in UI (leaderboard and experiment’s dashboard).

Neptune supports passing custom parameters to your experiment code in two ways: via ctx.params or via sys.argv.

Parameters Passed via ctx.params

You can access parameters values in your code using the params object from neptune.Context.

import neptune
ctx = neptune.Context()
learning_rate = ctx.params['lr']

You can pass parameter values to ctx.params either via config file or command line argument.

Config:

parameters:
  lr:
    value: 0.005
    description: "Learning rate"

Command line:

You can pass parameters to neptune run and neptune send commands. Use arguments: [PARAMETER [PARAMETER ...]] where PARAMETER has one of forms:

  • -p|--parameter [name]:[value/values]
  • -p|--parameter [name]:[value/values]:[description]
  • -p|--parameter [name]::[description].
neptune send --parameter 'lr:0.005:Learning rate'

Examples

Overwriting parameter values from config file by command line

Config:

parameters:
  lr:
    description: Learning rate of my experiment
  epoch:
    value: 10
    description: Number of epochs to run

Command line:

neptune send --parameter lr:0.1 --parameter epoch:100

Result:

Parameter Value Description
lr 0.1 Learning rate of my experiment
epoch 100 Number of epochs to run
Merging Parameters From Config File and Command Line

Config:

parameters:
  batch:
    value: 10
    description: Size of a single batch

Command line:

neptune send --parameter 'model:svm:Model to use in experiment'

Result:

Parameter Value Description
batch 10 Size of a single batch
model svm Model to use in experiment

Parameters Passed via sys.argv

You do not have to use Neptune Client Library to take advantage of Neptune parameters. Neptune can pass parameter values to your code using sys.argv. You just need to pass parameters via command line after execution command or via config file.

Note for Windows users

The following examples use single quotes for quoting command line parameters. On Windows, you cannot quote command line arguments with single quotes. Use double quotes instead. To put a double quote inside a quoted string, write \" instead of just ". For example:

neptune send main.py --x %5.0 --y %foo --z "%\"string with spaces\""

Named Parameters

Preceding the parameter values with % tells Neptune to track the parameters:

neptune send main.py --x %5.0 --y %foo

Parameters passed without % are not tracked:

neptune send main.py --x 5.0 --y foo

Positional Parameters

Positional parameters can be tracked too. In this case, you need to put the parameter name preceded by : after the value:

neptune send main.py %5.0:x %foo:y

Quoting Parameter Values and Names

String parameters containing whitespace or one of the following characters: % : ( ) [ ] " need to be quoted:

neptune send main.py --x '%"parameter with spaces"'

Only double quotes are supported in parameter quoting syntax. To put a double quote in a quoted string, you need to use two double quotes instead of one. In the example below, a value "Hamlet" by William Shakespeare is passed as x:

neptune send main.py --x '%"""Hamlet"" by William Shakespeare"'

Parameter names can be quoted too:

neptune send main.py '%"value with spaces":"parameter with spaces"'

Custom Parameter Names

If you want to have a named parameter tracked under a custom name, you can put the name preceded by : after the value:

neptune send main.py --lr '%5.0:"learning rate"'

In Detail

Command Line Arguments

Neptune removes the elements of parameter syntax before passing command line arguments, so you don’t have to handle them in your code.

Examples

The table below contains different variants of parameter syntax along with command line arguments passed to the code (i.e. python main.py ...) and the parameters being tracked.

Neptune command sys.argv Tracked parameters
neptune send main.py --x 5.0 --y foo --x 5.0 --y foo (none)
neptune send main.py --x %5.0 --y %foo --x 5.0 --y foo x=5.0, y=foo
neptune send main.py %5.0:x %foo:y 5.0 foo x=5.0, y=foo
neptune send main.py --x '%"with spaces"' --x "with spaces" x=with spaces
neptune send main.py --x '%"""Hamlet"""' --x "\"Hamlet\"" x="Hamlet"
neptune send main.py '%"spaced value":"spaced param"' "spaced value" spaced param=spaced value
neptune send main.py --lr '%5.0:"learning rate"' --lr 5.0 learning rate=5.0

Mixed Approach

Parameters passed either via sys.argv or ctx.params are representing the same entity – experiment parameters so it is natural to mix both approaches.

Documenting Parameters Passed via sys.argv

You want to pass parameters to your experiment via sys.argv, but you want to have a description for every parameter.

Config:

parameters:
  model:
    description: Model to use in experiment
  batch:
    value: 10
    description: Size of a single batch
  epoch: 100

Command line:

neptune send main.py %svm:model --b %:batch --epoch %1

Result:

Parameter sys.argv ctx.params Description
model svm ctx.params.model == svm Model to use in experiment
batch --b 10 ctx.params.batch == 10 Size of a single batch
epoch --epoch 100 ctx.params.epoch == 100

Tags

Tags are one of the ways you can organize your experiments in Neptune. A tag is a word that contains only lowercase letters, numbers, underscores and dashes. Tags can be assigned to an experiment or removed from an experiment anytime throughout experiment’s lifetime. When listing experiments, it is possible to search by tags. Tags can be set from:

tags: [tag1, tag2, tag3, tag4]
  • the experiment’s code
ctx.tags.append('tag-to-add')
ctx.tags.remove('tag-to-remove')
  • and from command line
neptune send --tag foo --tag bar

Working Directory

Each experiment or notebook will be executed in /neptune directory. Working directory will contain all source files and directories uploaded by neptune send command.