Get started: run Python script as Neptune experiment

estimated time to complete: 10 minutes

What will you learn?

This tutorial will guide you through the installation and basic usage of Neptune. The overall goal is present Neptune’s features related to running machine learning experiments. We will start with minimal example in keras and build on it, towards more complex and fun use cases.

Upon completion of this tutorial you will know how to:

  • Install Neptune,
  • Run experiment using regular Python script,
  • Browse experiment’s results in the Neptune web application,
  • Browse and manage Experiments view,
  • Use basic methods from Neptune client library.

Before we start: what is experiment anyways?

Experiments are at the heart of any machine learning project. In Neptune, “Experiment” is a single execution of the Python script. Every time you type in terminal: python model_training.py to train your model you can run the same script (with just few additional lines of code) with Neptune. This tutorial will show you how to do it.

When started, an experiment is visible in the web application (example here) with tons of accompanying information. Experiments are associated with projects - more about them later. For now, it is sufficient to think of projects as a collection of experiments.

Part I: Registration and installation guide

Registration

Sign up to Neptune by using this link: https://neptune.ml/register. Registration steps require email address confirmation. Please make sure to do it.

Installation

Prerequisites

  1. Neptune account (see registration above)
  2. Python 2.7, 3.5 or 3.6
  3. Linux-based OS

Install Neptune

  1. Open terminal
  2. Create working directory (we will work in this directory the entire time)

    1
    2
    3
    cd ~  # go to home directory
    mkdir neptune-tutorials  # create working directory
    cd neptune-tutorials  # go to neptune-tutorials
    

    If you want to work in different directory, make sure that you have writing permissions in that directory.

  3. Create virtualenv and activate it (this step is recommended but not necessary)

    1
    2
    virtualenv neptune_venv
    source neptune_venv/bin/activate
    

  4. Install Neptune package using pip

    1
    pip install neptune-client
    

Now, you are ready to run your first experiment!

Part II: Run first experiment

We will train simple keras model (with TensorFlow backend) on MNIST dataset. We are going to start with minimalist approach and then build on top of that in order to learn more of Neptune’s features. Let’s jump right in!

Get minimal example code

Create directory for Python files

1
2
mkdir tutorial-1
cd tutorial-1
(Your current working directory should now be: ~/neptune-tutorials/tutorial-1)

Below is the training code. Simply copy and paste it to the 1-minimal-example.py file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
import neptune
import keras
from keras import backend as K
from keras.callbacks import Callback

# This function assumes that NEPTUNE_API_TOKEN environment variable is defined.
neptune.init(project_qualified_name='username/sandbox')

neptune.create_experiment()

# prepare data
mnist = keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
n_batches = x_train.shape[0] // 64 + 1

# prepare model
model = keras.models.Sequential([
  keras.layers.Flatten(),
  keras.layers.Dense(512, activation=K.relu),
  keras.layers.Dropout(0.2),
  keras.layers.Dense(10, activation=K.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# prepare keras callback to monitor training progress
class NeptuneMonitor(Callback):
    def __init__(self):
        super().__init__()
        self.current_epoch = 0

    def on_batch_end(self, batch, logs=None):
        x = (self.current_epoch * n_batches) + batch
        neptune.send_metric(name='batch end accuracy', x=x, y=logs['acc'])
        neptune.send_metric(name='batch end loss', x=x, y=logs['loss'])

    def on_epoch_end(self, epoch, logs=None):
        neptune.send_metric(name='epoch end accuracy', x=epoch, y=logs['acc'])
        neptune.send_metric(name='epoch end loss', x=epoch, y=logs['loss'])
        self.current_epoch += 1

# fit the model to data
model.fit(x_train, y_train,
          epochs=5, batch_size=64,
          callbacks=[NeptuneMonitor()])

Run the code

When you run the code, Neptune receives experiment tracking information that you can browse and analyse online. In order to start an experiment locally, you need to install additional requirements first:

1
pip install keras tensorflow
(assuming that virtualenv with Neptune is activated)

Start experiment

1
python 1-minimal-example.py

You just started your first Neptune experiment

New tab in your default browser is opened. There, you can track progress of keras model training in real time. Take some time and explore tabs on the left - each contains different information.

Example link: app.neptune.ml/neptune-ml/neptune-tutorials/e/TT-2/charts

Minimal example analysis

Working directory

Once you started an experiment, the 1-minimal-example.py file is uploaded to Neptune in order to save your code snapshot and let you browse it online. From now on, this file is called entrypoint.

important note about code directory

If your code is super private you can prevent Neptune from uploading it. Simply use upload_source_files=[] parameter.

neptune.create_experiment(upload_source_files=[])

See also: How to keep my code private? for more information.

Source code

Neptune client is a lightweight Python library that integrates your code with Neptune. You have installed this package by running pip install neptune-client, as described in the Installation section.

First part of 1-minimal-example.py imports Neptune and creates experiment.

1
2
3
4
5
6
7
8
9
import neptune
import keras
from keras import backend as K
from keras.callbacks import Callback

# This function assumes that NEPTUNE_API_TOKEN environment variable is defined.
neptune.init(project_qualified_name='username/sandbox')

neptune.create_experiment()

Another interesting part is keras callback named NeptuneMonitor. In four lines you can see method: send_metric(...), which is the most important part of integration with Neptune. It lets you track your metrics, losses and any other type of experiment tracking data. Charts in Neptune are created automatically from these metrics.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# prepare keras callback to monitor training progress
class NeptuneMonitor(Callback):
    def __init__(self):
        super().__init__()
        self.current_epoch = 0

    def on_batch_end(self, batch, logs=None):
        x = (self.current_epoch * n_batches) + batch
        neptune.send_metric(name='batch end accuracy', x=x, y=logs['acc'])
        neptune.send_metric(name='batch end loss', x=x, y=logs['loss'])

    def on_epoch_end(self, epoch, logs=None):
        neptune.send_metric(name='epoch end accuracy', x=epoch, y=logs['acc'])
        neptune.send_metric(name='epoch end loss', x=epoch, y=logs['loss'])
        self.current_epoch += 1

Remember

neptune.init('username/sandbox') authorizes you to interact with Neptune in given project.

neptune.send_metric method sends User-defined values to Neptune experiment. Sent metrics are visualized on charts automatically.

Part III: Let’s go deeper

In this part we will build on top of minimal example. There are a few fundamental concepts that are worth mentioning at the beginning of learning how to use Neptune. First are send methods and second are properties.

More about send methods

You can think of send methods as a way to track all metrics, values, losses, images and data that you care about in your machine learning experiment. You can create unlimited number of different pieces of information (some public projects on Neptune have more that 100). In general there are four types of data that you can send to Neptune: numeric, text, image, artifact. First one is the most common, so we are going to add another one to the 1-minimal-example.py.

Adding send_metric

You will add an innovative metric that Jakub (our Senior Data Scientist) have prepared. In depth analysis of its performance is very promising, so we are considering NeurlIPS paper submission next year:) Below is code excerpt:

1
2
3
4
5
6
7
8
# (...)
def on_epoch_end(self, epoch, logs=None):
    neptune.send_metric(name='epoch end accuracy', x=epoch, y=logs['acc'])
    neptune.send_metric(name='epoch end loss', x=epoch, y=logs['loss'])
    innovative_metric = logs['acc'] - 2 * logs['loss']
    neptune.send_metric(name='innovative_metric', x=epoch, y=innovative_metric)
    self.current_epoch += 1
# (...)
As mentioned before, charts for are created automatically from metric values. See charts tab, to check the performance of this innovative metric.

Adding text

Let’s assume, that at the end of each epoch we want to have textual information about accuracy and loss. We will create two text pieces of information that will satisfy this requirement. Below is code excerpt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
def on_epoch_end(self, epoch, logs=None):
    neptune.send_metric(name='epoch end accuracy', x=epoch, y=logs['acc'])
    neptune.send_metric(name='epoch end loss', x=epoch, y=logs['loss'])
    innovative_metric = logs['acc'] - 2 * logs['loss']
    neptune.send_metric(name='innovative_metric', x=epoch, y=innovative_metric)
    msg_acc = 'End of epoch {}, accuracy is {:.4f}'.format(epoch, logs['acc'])
    neptune.send_text(name='accuracy information', x=epoch, y=msg_acc)
    msg_loss = 'End of epoch {}, categorical crossentropy loss is {:.4f}'.format(epoch, logs['loss'])
    neptune.send_text(name='loss information', x=epoch, y=msg_loss)
    self.current_epoch += 1

Run experiment with new data included

At this point it is worth to run experiment with additional data. In the box below is modified minimal-example that you copy & paste to new file: 2-example-with-more-data.py

Expand, copy and paste
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
import neptune
import keras
from keras import backend as K
from keras.callbacks import Callback

# This function assumes that NEPTUNE_API_TOKEN environment variable is defined.
neptune.init(project_qualified_name='username/sandbox')

neptune.create_experiment()

# prepare data
mnist = keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
n_batches = x_train.shape[0] // 64 + 1

# prepare model
model = keras.models.Sequential([
  keras.layers.Flatten(),
  keras.layers.Dense(512, activation=K.relu),
  keras.layers.Dropout(0.2),
  keras.layers.Dense(10, activation=K.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# prepare keras callback to monitor training progress
class NeptuneMonitor(Callback):
    def __init__(self):
        super().__init__()
        self.current_epoch = 0

    def on_batch_end(self, batch, logs=None):
        x = (self.current_epoch * n_batches) + batch
        neptune.send_metric(name='batch end accuracy', x=x, y=logs['acc'])
        neptune.send_metric(name='batch end loss', x=x, y=logs['loss'])

    def on_epoch_end(self, epoch, logs=None):
        neptune.send_metric(name='epoch end accuracy', x=epoch, y=logs['acc'])
        neptune.send_metric(name='epoch end loss', x=epoch, y=logs['loss'])
        innovative_metric = logs['acc'] - 2 * logs['loss']
        neptune.send_metric(name='innovative_metric', x=epoch, y=innovative_metric)
        msg_acc = 'End of epoch {}, accuracy is {:.4f}'.format(epoch, logs['acc'])
        neptune.send_text(name='accuracy information', x=epoch, y=msg_acc)
        msg_loss = 'End of epoch {}, categorical crossentropy loss is {:.4f}'.format(epoch, logs['loss'])
        neptune.send_text(name='loss information', x=epoch, y=msg_loss)
        self.current_epoch += 1

# fit the model to data
model.fit(x_train, y_train,
          epochs=5, batch_size=64,
          callbacks=[NeptuneMonitor()])

When new code is ready under this directory: ~/neptune-tutorials/tutorial-1/2-example-with-more-data.py, you are ready to start an experiment. Similarly to previous experiments:

1
python 2-example-with-more-data.py

Results

Example experiment with additional channels. Note that numeric channels are green, text channels are yellow. Click on any box with channel to see its data. You can download raw data in the csv format - just click on the download icon on the right side of the data view.

All numeric channels have corresponding charts in the charts tab.

Charts are interactive!

You can do many actions with them:

  1. Click on chart and slide (up-down or left-right) to zoom in. Double click on the chart to zoom out.
  2. Click on the legend (below chart) to hide it’s data.
  3. Click on the eye icon (top right side of each chart) to hide it. Hidden charts are placed at the bottom of the page.
  4. Slider on the right side is for smoothing the curve.

Properties

Common data structure for machine learning experiments are key-value pairs. They store single values that are important to model training, such as performance on test data or various properties of the model. In this example, we will add two crucial values to the experiment, that is loss and accuracy on test data. In Neptune you can store key-value pairs using the concept of properties. Its usage is straightforward and will be presented in this section.

We build on top of 2-example-with-more-data.py. In order to add key-value pairs we need to append few lines of code. In this example we make use of keras model.evaluate() method. It returns loss and metrics for the model in test mode. We report these metrics as properties in Neptune. Here is the code:

1
2
3
4
5
# evaluate model on test data
names = model.metrics_names
values = model.evaluate(x_test, y_test)  # evaluate model on test data
neptune.set_property(names[0], values[0])
neptune.set_property(names[1], values[1])
That’s it! Properties will be automatically added to your experiment.

Remember

Use neptune.set_property('new_key', 'some_value') to add key-value pair to Neptune experiment.

Start experiment with properties added

In the box below you can find code with model evaluation added. Copy & paste it to the ~/neptune-tutorials/tutorial-1/3-example-with-properties.py

Expand, copy and paste
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
import neptune
import keras
from keras import backend as K
from keras.callbacks import Callback

# This function assumes that NEPTUNE_API_TOKEN environment variable is defined.
neptune.init(project_qualified_name='username/sandbox')

neptune.create_experiment()

# prepare data
mnist = keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
n_batches = x_train.shape[0] // 64 + 1

# prepare model
model = keras.models.Sequential([
  keras.layers.Flatten(),
  keras.layers.Dense(512, activation=K.relu),
  keras.layers.Dropout(0.2),
  keras.layers.Dense(10, activation=K.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# prepare keras callback to monitor training progress
class NeptuneMonitor(Callback):
    def __init__(self):
        super().__init__()
        self.current_epoch = 0

    def on_batch_end(self, batch, logs=None):
        x = (self.current_epoch * n_batches) + batch
        neptune.send_metric(name='batch end accuracy', x=x, y=logs['acc'])
        neptune.send_metric(name='batch end loss', x=x, y=logs['loss'])

    def on_epoch_end(self, epoch, logs=None):
        neptune.send_metric(name='epoch end accuracy', x=epoch, y=logs['acc'])
        neptune.send_metric(name='epoch end loss', x=epoch, y=logs['loss'])
        innovative_metric = logs['acc'] - 2 * logs['loss']
        neptune.send_metric(name='innovative_metric', x=epoch, y=innovative_metric)
        msg_acc = 'End of epoch {}, accuracy is {:.4f}'.format(epoch, logs['acc'])
        neptune.send_text(name='accuracy information', x=epoch, y=msg_acc)
        msg_loss = 'End of epoch {}, categorical crossentropy loss is {:.4f}'.format(epoch, logs['loss'])
        neptune.send_text(name='loss information', x=epoch, y=msg_loss)
        self.current_epoch += 1

# fit the model to data
model.fit(x_train, y_train,
          epochs=5, batch_size=64,
          callbacks=[NeptuneMonitor()])

# evaluate model on test data
names = model.metrics_names
values = model.evaluate(x_test, y_test)  # evaluate model on test data
neptune.set_property(names[0], values[0])
neptune.set_property(names[1], values[1])

Similarly to previous examples you run an experiment by executing Python script:

1
python 3-example-with-properties.py

Results

Example result of running this experiment can be seen here (you are encouraged to run your own training!). In the details tab, you can see additional section Properties with name and value columns.

Also, you can see and analyze properties in the experiments view - fundamental concept in Neptune covered in the next section.

Summary

In this tutorial you trained keras model on the MNIST data. You installed Neptune and ran several experiments. Each experiment consists of several send methods (either numeric or text) that lets you track and analyze model training dynamics. You started experiment as regular Python script. Columns on the experiment view consist of metrics, text and properties.

Code examples are available on Github and on Neptune. Latter resource also contains experiments.

Tutorial in a nutshell

Terminal

1
2
3
pip install neptune-client  # install Neptune

python 1-minimal-example.py  # run experiment

Python

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import neptune  # import Python package

# initialize neptune
# authorizes you to interact with Neptune in given project
# This function assumes that NEPTUNE_API_TOKEN environment variable is defined.
neptune.init(project_qualified_name='username/sandbox')

# create experiment
neptune.create_experiment()

# send new metric value
neptune.send_metric('log_loss', 0.753)

# set new key-value pair as Neptune property.
neptune.set_property('new_key', 'some_value')