Get started: run Python script as Neptune experiment

estimated time to complete: 10 minutes

What will you learn?

This tutorial will guide you through the installation and basic usage of Neptune. The overall goal is present Neptune’s features related to running machine learning experiments. We will start with minimal example in keras and build on it, towards more complex and fun use cases.

Upon completion of this tutorial you will know how to:

  • Install Neptune,
  • Run experiment using regular Python script,
  • Browse experiment’s results in the Neptune web application,
  • Browse and manage Experiments view,
  • Use basic methods from Python client library.

Before we start: what is experiment anyways?

Experiments are at the heart of any machine learning project. In Neptune, “Experiment” is a single execution of the Python script. Every time you type in terminal: python model_training.py to train your model you can run the same script (with just few additional lines of code) with Neptune. This tutorial will show you how to do it.

When started, an experiment is visible in the web application (example here) with tons of accompanying information. Experiments are associated with projects - more about them later. For now, it is sufficient to think of projects as a collection of experiments.

There is another type of experiment: Jupyter Notebook experiment that will be covered in further tutorials.

Part I: Registration and installation guide

Registration

Sign up to Neptune by using this link: https://neptune.ml/register. Registration steps require email address confirmation. Please make sure to do it. You will receive $5 bonus for experiments, upon completion.

Installation

Prerequisites

  1. Neptune account (see registration above)
  2. Python 2.7, 3.5 or 3.6
  3. Linux-based OS

Install Neptune

  1. Open terminal
  2. Create working directory (we will work in this directory the entire time)

    1
    2
    3
    cd ~  # go to home directory
    mkdir neptune-tutorials  # create working directory
    cd neptune-tutorials  # go to neptune-tutorials
    

    If you want to work in different directory, make sure that you have writing permissions in that directory.

  3. Create virtualenv and activate it (this step is recommended but not necessary)

    1
    2
    virtualenv neptune_venv
    source neptune_venv/bin/activate
    

  4. Install Neptune package using pip

    1
    pip install neptune-cli
    

  5. Sign in to your account
    1
    neptune account login
    
    This command opens a new tab in your default browser and ask for sign-in credentials.

Now, you are ready to run your first experiment!

Part II: Run first experiment

We will train simple keras model (with TensorFlow backend) on MNIST dataset. We are going to start with minimalist approach and then build on top of that in order to learn more of Neptune’s features. Let’s jump right in!

Get minimal example code

Create directory for Python files

1
2
mkdir tutorial-1
cd tutorial-1
(Your current working directory should now be: ~/neptune-tutorials/tutorial-1)

Below is the training code. Simply copy and paste it to the 1-minimal-example.py file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
import neptune
import keras
from keras import backend as K
from keras.callbacks import Callback

ctx = neptune.Context()

# prepare data
mnist = keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
n_batches = x_train.shape[0] // 64 + 1

# prepare model
model = keras.models.Sequential([
  keras.layers.Flatten(),
  keras.layers.Dense(512, activation=K.relu),
  keras.layers.Dropout(0.2),
  keras.layers.Dense(10, activation=K.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# prepare keras callback to monitor training progress
class NeptuneMonitor(Callback):
    def __init__(self):
        super().__init__()
        self.current_epoch = 0

    def on_batch_end(self, batch, logs=None):
        x = (self.current_epoch * n_batches) + batch
        ctx.channel_send(name='batch end accuracy', x=x, y=logs['acc'])
        ctx.channel_send(name='batch end loss', x=x, y=logs['loss'])

    def on_epoch_end(self, epoch, logs=None):
        ctx.channel_send(name='epoch end accuracy', x=epoch, y=logs['acc'])
        ctx.channel_send(name='epoch end loss', x=epoch, y=logs['loss'])
        self.current_epoch += 1

# fit the model to data
model.fit(x_train, y_train,
          epochs=5, batch_size=64,
          callbacks=[NeptuneMonitor()])

Run the code

There are two ways of running experiment: in the cloud or on the local hardware. Below both options are presented. You can proceed with any one of them, because source code is the same.

In the cloud

Cloud option uses Google Compute Platform or GCP in short. This option requires no additional effort, like creating account on GCP, or setting up virtual machine. Neptune handles everything end-to-end. Below is a command that starts an experiment on Google Cloud. Do not worry about the parameters - they will be explained in further tutorials.

1
neptune send --worker xs-k80-preemptible --environment keras-2.2-gpu-py3 1-minimal-example.py

You just started your first Neptune experiment

New tab in your default browser is opened. There you can track progress of keras model training in real time. Take some time and explore tabs on the left - each contains different information.

Example link: app.neptune.ml/neptune-ml/neptune-tutorials/e/TT-3/charts

On local hardware

You can run all computations on your local hardware. In this option, local machine does all heavy lifting, while Neptune receives experiment tracking information that you can browse and analyse online. In order to start an experiment locally, you need to install additional requirements first:

1
pip install keras tensorflow
(assuming that virtualenv with Neptune is activated)

Start experiment

1
neptune run 1-minimal-example.py

You just started your first Neptune experiment

New tab in your default browser is opened. There, you can track progress of keras model training in real time. Take some time and explore tabs on the left - each contains different information.

Example link: app.neptune.ml/neptune-ml/neptune-tutorials/e/TT-2/charts

Minimal example analysis

Working directory

Once you started an experiment (either cloud-based or local hardware-based) the contents of the tutorial-1 directory are uploaded to Neptune in order to save your code snapshot and let you browse it online. From now on, Python file 1-minimal-example.py is called entrypoint.

important note about code directory

If your code is super private you can prevent Neptune from uploading it. Simply use additional parameter --exclude [*] in neptune send/run command.

Example: neptune run --exclude [*] 1-minimal-example.py

See also: How to keep my code private? for more information.

Source code

Neptune client is a lightweight Python library that integrates your code with Neptune. You have installed this package by running pip install neptune-cli, as described in the Installation section.

First part of 1-minimal-example.py imports Neptune and creates Context - your single point of integration with Neptune backend.

1
2
3
4
5
6
import neptune
import keras
from keras import backend as K
from keras.callbacks import Callback

ctx = neptune.Context()

Another interesting part is keras callback named NeptuneMonitor. In four lines you can see method: channel_send(...), which is the most important part of integration with Neptune. It lets you track your metrics, losses and any other type of experiment tracking data. Charts in Neptune are created automatically from numeric channels.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# prepare keras callback to monitor training progress
class NeptuneMonitor(Callback):
    def __init__(self):
        super().__init__()
        self.current_epoch = 0

    def on_batch_end(self, batch, logs=None):
        x = (self.current_epoch * n_batches) + batch
        ctx.channel_send(name='batch end accuracy', x=x, y=logs['acc'])
        ctx.channel_send(name='batch end loss', x=x, y=logs['loss'])

    def on_epoch_end(self, epoch, logs=None):
        ctx.channel_send(name='epoch end accuracy', x=epoch, y=logs['acc'])
        ctx.channel_send(name='epoch end loss', x=epoch, y=logs['loss'])
        self.current_epoch += 1

Remember

ctx = neptune.Context() creates an object that integrates your code with Neptune.

ctx.channel_send() method sends User-defined values to Neptune experiment. Numeric channels are visualized on charts automatically.

Part III: Let’s go deeper

In this part we will build on top of minimal example. There are a few fundamental concepts that are worth mentioning at the beginning of learning how to use Neptune. First are channels and second are properties.

More about channels

You can think of channels as a way to track all metrics, values, losses that you care about in your machine learning experiment. You can create unlimited number of channels (some public projects on Neptune have more that 100 channels). In general there are three types of channels: numeric, text and image. First one is the most common, so we are going to add another one to the 1-minimal-example.py.

Adding numeric channel

You will add an innovative metric that Jakub (our Senior Data Scientist) have prepared. In depth analysis of its performance is very promising, so we are considering NeurlIPS paper submission next year:) Below is code excerpt:

1
2
3
4
5
6
7
8
# (...)
def on_epoch_end(self, epoch, logs=None):
    ctx.channel_send(name='epoch end accuracy', x=epoch, y=logs['acc'])
    ctx.channel_send(name='epoch end loss', x=epoch, y=logs['loss'])
    innovative_metric = logs['acc'] - 2 * logs['loss']
    ctx.channel_send(name='innovative_metric', x=epoch, y=innovative_metric)
    self.current_epoch += 1
# (...)
As mentioned before, charts for numeric channels are created automatically. See charts tab, to check the performance of this innovative metric.

Adding text channel

Let’s assume, that at the end of each epoch we want to have textual information about accuracy and loss. We will create two text channels that will satisfy this requirement. Below is code excerpt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
def on_epoch_end(self, epoch, logs=None):
    ctx.channel_send(name='epoch end accuracy', x=epoch, y=logs['acc'])
    ctx.channel_send(name='epoch end loss', x=epoch, y=logs['loss'])
    innovative_metric = logs['acc'] - 2 * logs['loss']
    ctx.channel_send(name='innovative_metric', x=epoch, y=innovative_metric)
    msg_acc = 'End of epoch {}, accuracy is {:.4f}'.format(epoch, logs['acc'])
    ctx.channel_send(name='accuracy information', x=epoch, y=msg_acc)
    msg_loss = 'End of epoch {}, categorical crossentropy loss is {:.4f}'.format(epoch, logs['loss'])
    ctx.channel_send(name='loss information', x=epoch, y=msg_loss)
    self.current_epoch += 1

Run experiment with new channels included

At this point it is worth to run experiment with additional channels. In the box below is modified minimal-example that you copy & paste to new file: 2-example-with-more-channels.py

Expand, copy and paste
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
import neptune
import keras
from keras import backend as K
from keras.callbacks import Callback

ctx = neptune.Context()

# prepare data
mnist = keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
n_batches = x_train.shape[0] // 64 + 1

# prepare model
model = keras.models.Sequential([
  keras.layers.Flatten(),
  keras.layers.Dense(512, activation=K.relu),
  keras.layers.Dropout(0.2),
  keras.layers.Dense(10, activation=K.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# prepare keras callback to monitor training progress
class NeptuneMonitor(Callback):
    def __init__(self):
        super().__init__()
        self.current_epoch = 0

    def on_batch_end(self, batch, logs=None):
        x = (self.current_epoch * n_batches) + batch
        ctx.channel_send(name='batch end accuracy', x=x, y=logs['acc'])
        ctx.channel_send(name='batch end loss', x=x, y=logs['loss'])

    def on_epoch_end(self, epoch, logs=None):
        ctx.channel_send(name='epoch end accuracy', x=epoch, y=logs['acc'])
        ctx.channel_send(name='epoch end loss', x=epoch, y=logs['loss'])
        innovative_metric = logs['acc'] - 2 * logs['loss']
        ctx.channel_send(name='innovative_metric', x=epoch, y=innovative_metric)
        msg_acc = 'End of epoch {}, accuracy is {:.4f}'.format(epoch, logs['acc'])
        ctx.channel_send(name='accuracy information', x=epoch, y=msg_acc)
        msg_loss = 'End of epoch {}, categorical crossentropy loss is {:.4f}'.format(epoch, logs['loss'])
        ctx.channel_send(name='loss information', x=epoch, y=msg_loss)
        self.current_epoch += 1

# fit the model to data
model.fit(x_train, y_train,
          epochs=5, batch_size=64,
          callbacks=[NeptuneMonitor()])

When new code is ready under this directory: ~/neptune-tutorials/tutorial-1/2-example-with-more-channels.py, you are ready to start an experiment. Similarly to previous experiments, you can either use cloud or local version.

1
2
3
4
5
# cloud
neptune send --worker xs-k80-preemptible --environment keras-2.2-gpu-py3 2-example-with-more-channels.py

# local
neptune run 2-example-with-more-channels.py

Results

Example experiment with additional channels. Note that numeric channels are green, text channels are yellow. Click on any box with channel to see its data. You can download raw data in the csv format - just click on the download icon on the right side of the data view.

All numeric channels have corresponding charts in the charts tab.

Charts are interactive!

You can do many actions with them:

  1. Click on chart and slide (up-down or left-right) to zoom in. Double click on the chart to zoom out.
  2. Click on the legend (below chart) to hide it’s data.
  3. Click on the eye icon (top right side of each chart) to hide it. Hidden charts are placed at the bottom of the page.
  4. Slider on the right side is for smoothing the curve.

Properties

Common data structure for machine learning experiments are key-value pairs. They store single values that are important to model training, such as performance on test data or various properties of the model. In this example, we will add two crucial values to the experiment, that is loss and accuracy on test data. In Neptune you can store key-value pairs using the concept of properties. Its usage is straightforward and will be presented in this section.

We build on top of 2-example-with-more-channels.py. In order to add key-value pairs we need to append few lines of code. In this example we make use of keras model.evaluate() method. It returns loss and metrics for the model in test mode. We report these metrics as properties in Neptune. Here is the code:

1
2
3
4
5
# evaluate model on test data
names = model.metrics_names
values = model.evaluate(x_test, y_test)  # evaluate model on test data
ctx.properties[names[0]] = values[0]
ctx.properties[names[1]] = values[1]
That’s it! Properties will be automatically added to your experiment.

Remember

Use ctx.properties[key] = value to add key-value pair to Neptune experiment.

Start experiment with properties added

In the box below you can find code with model evaluation added. Copy & paste it to the ~/neptune-tutorials/tutorial-1/3-example-with-properties.py

Expand, copy and paste
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
import neptune
import keras
from keras import backend as K
from keras.callbacks import Callback

ctx = neptune.Context()

# prepare data
mnist = keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
n_batches = x_train.shape[0] // 64 + 1

# prepare model
model = keras.models.Sequential([
  keras.layers.Flatten(),
  keras.layers.Dense(512, activation=K.relu),
  keras.layers.Dropout(0.2),
  keras.layers.Dense(10, activation=K.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])


# prepare keras callback to monitor training progress
class NeptuneMonitor(Callback):
    def __init__(self):
        super().__init__()
        self.current_epoch = 0

    def on_batch_end(self, batch, logs=None):
        x = (self.current_epoch * n_batches) + batch
        ctx.channel_send(name='batch end accuracy', x=x, y=logs['acc'])
        ctx.channel_send(name='batch end loss', x=x, y=logs['loss'])

    def on_epoch_end(self, epoch, logs=None):
        ctx.channel_send(name='epoch end accuracy', x=epoch, y=logs['acc'])
        ctx.channel_send(name='epoch end loss', x=epoch, y=logs['loss'])
        innovative_metric = logs['acc'] - 2 * logs['loss']
        ctx.channel_send(name='innovative_metric', x=epoch, y=innovative_metric)
        msg_acc = 'End of epoch {}, accuracy is {:.4f}'.format(epoch, logs['acc'])
        ctx.channel_send(name='accuracy information', x=epoch, y=msg_acc)
        msg_loss = 'End of epoch {}, categorical crossentropy loss is {:.4f}'.format(epoch, logs['loss'])
        ctx.channel_send(name='loss information', x=epoch, y=msg_loss)
        self.current_epoch += 1


# fit the model to data
model.fit(x_train, y_train,
          epochs=5, batch_size=64,
          callbacks=[NeptuneMonitor()])


# evaluate model on test data
names = model.metrics_names
values = model.evaluate(x_test, y_test)
ctx.properties[names[0]] = values[0]
ctx.properties[names[1]] = values[1]

Similarly to previous examples you can run this experiment in the cloud or on local hardware. Both options are presented below:

1
2
3
4
5
# cloud
neptune send --worker xs-k80-preemptible --environment keras-2.2-gpu-py3 3-example-with-properties.py

# local
neptune run 3-example-with-properties.py

Results

Example result of running this experiment can be seen here (you are encouraged to run your own training!). In the details tab, you can see additional section Properties with name and value columns.

Also, you can see and analyze properties in the experiments view - next fundamental concept in Neptune covered in the next section.

Summary

In this tutorial you trained keras model on the MNIST data. You installed Neptune and ran several experiments. Each experiment consists of several channels (either numeric or text) that lets you track and analyze model training dynamics. You started experiment in the cloud or on your local hardware, where both options are equally simple to do. Columns on the experiment view consist of both channels and properties.

Code examples are available on Github and on Neptune. Latter resource also contains experiments.

Tutorial in a nutshell

Terminal

1
2
3
4
5
6
7
8
pip install neptune-cli  # install Neptune 
neptune account login  # sign in to Neptune

# run experiment in the cloud
neptune send --worker xs-k80-preemptible --environment keras-2.2-gpu-py3 1-minimal-example.py

# run experiment locally
neptune run 1-minimal-example.py

Python

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
import neptune  # import Python package

# create Context - an object that integrates your code with Neptune
ctx = neptune.Context()

# send new (x, y) pair to channel 'metric name'.
ctx.channel_send(name='metric name', x=x_value, y=y_value)

# create key-value pair as Neptune property.
ctx.properties[key] = value