Track

How to log keras metrics?

I have a training script written in keras. How do I adjust it to log metrics to Neptune?

Keras neptune.ml integration

Solution

Step 1

Say your training script looks like this:

import keras
from keras import backend as K

mnist = keras.datasets.mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = keras.models.Sequential([
  keras.layers.Flatten(),
  keras.layers.Dense(512, activation=K.relu),
  keras.layers.Dropout(0.2),
  keras.layers.Dense(10, activation=K.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)

Step 2

Now let’s use Keras Callback

from keras.callbacks import Callback

class NeptuneMonitor(Callback):
    def on_epoch_end(self, epoch, logs={}):
        innovative_metric = logs['acc'] - 2 * logs['loss']
        neptune.send_metric('innovative_metric', epoch, innovative_metric)

Step 3

Instantiate it and add it to your callbacks list:

with neptune.create_experiment():
    neptune_monitor = NeptuneMonitor()
    model.fit(x_train, y_train, epochs=5, callbacks=[neptune_monitor])

All your metrics are now logged to Neptune:

image

How to log PyTorch metrics?

I have a training script written in PyTorch. How do I adjust it to log metrics to Neptune?

Pytorch neptune.ml integration

Solution

Say your training script looks like this:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms

DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
ITERATIONS = 10000

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.conv2 = nn.Conv2d(20, 50, 5, 1)
        self.fc1 = nn.Linear(4*4*50, 500)
        self.fc2 = nn.Linear(500, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2, 2)
        x = x.view(-1, 4*4*50)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data',
                   train=True,
                   download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))])
                   ),
    batch_size=64,
    shuffle=True)

model = Net().to(DEVICE)

optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

for batch_idx, (data, target) in enumerate(train_loader):
    data, target = data.to(DEVICE), target.to(DEVICE)
    optimizer.zero_grad()
    output = model(data)
    loss = F.nll_loss(output, target)
    loss.backward()
    optimizer.step()

    if batch_idx == ITERATIONS:
        break

Add a snippet to the training loop, that sends your loss or metric to Neptune:

import neptune

neptune.init('shared/onboarding')
neptune.create_experiment()
...
for batch_idx, (data, target) in enumerate(train_loader):
    ...
    neptune.send_metric('batch_loss', batch_idx, loss.data.cpu().numpy())

Your loss is now logged to Neptune:

image

How to log LightGBM metrics?

I have a training script written in LightGBM. How do I adjust it to log metrics to Neptune?

lightGBM neptune.ml integration

Solution

Say your training script looks like this:

import lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_wine

data = load_wine()

X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.1)
lgb_train = lgb.Dataset(X_train, y_train)
lgb_eval = lgb.Dataset(X_test, y_test, reference=lgb_train)

params = {
    'boosting_type': 'gbdt',
    'objective': 'multiclass',
    'num_class': 3,
    'num_leaves': 31,
    'learning_rate': 0.05,
    'feature_fraction': 0.9,
}

gbm = lgb.train(params,
                lgb_train,
                num_boost_round=500,
                valid_sets=[lgb_train, lgb_eval],
                valid_names=['train','valid'],
                )

Now, you need to use lightGBM callbacks to pass log metrics to Neptune:

Step 1

Take this callback:

import neptune

neptune.init('shared/onboarding')
neptune.create_experiment()

def neptune_monitor():
    def callback(env):
        for name, loss_name, loss_value, _ in env.evaluation_result_list:
            neptune.send_metric('{}_{}'.format(name, loss_name), x=env.iteration, y=loss_value)
    return callback

Step 2

Pass it to lgb.train object via callbacks parameter:

gbm = lgb.train(params,
                lgb_train,
                num_boost_round=500,
                valid_sets=[lgb_train, lgb_eval],
                valid_names=['train','valid'],
                callbacks=[neptune_monitor()],
                )

All your metrics are now logged to Neptune

image

How to log matplotlib figure to Neptune?

How to log charts generated in matplotlib, like confusion matrix or distribution in Neptune?

matplotlib neptune.ml integration

Solution

Step 1

Create matplotlib figure

import matplotlib.pyplot as plt
import seaborn as sns

fig = plt.figure()
sns.distplot(np.random.random(100))

Step 2

Save it to file. You can use a temporary file if you want to.

fig.savefig('figure.png')

Step 3

Send it to Neptune!

neptune.create_experiment()
neptune.log_image('distplot', 'figure.png')

Step 4

Explore it in the browser:

image image

How to save experiment output?

I can run my experiment but I am struggling to save the model weights and the csv file with the results when it completes. How can I do that in Neptune?

Solution

Save everything as you go! For example:

with neptune.create_experiment() as exp:
    exp.send_artifact('/path/to/model_weights.h5')
    ...
    exp.send_artifact('/path/to/results.csv')

Your results will be available for you to download in the Output section of your experiment.

image

How specify experiment parameters?

I saw that Neptune logs experiment parameters.

image

But I don’t know how to specify parameters for my experiments.

Solution

You define your parameters at experiment creation, like this:

import neptune

# This function assumes that NEPTUNE_API_TOKEN environment variable is defined.
neptune.init('username/my_project')

# check params argument
with neptune.create_experiment(name='first-pytorch-ever',
                               params={'dropout': 0.3,
                                       'lr': 0.01,
                                       'nr_epochs': 10}):
# your training script

Where params is standard Python dict.

How to log images to Neptune?

I generate model predictions after every epoch. How can I log them as images to Neptune?

Solution

Log single image to Neptune

Create PIL image that you want to log. For example:

import imgaug as ia
from PIL import Image

img = ia.quokka()
img_pil = Image.fromarray(img)

Log it to Neptune:

import neptune

# This function assumes that NEPTUNE_API_TOKEN environment variable is defined.
neptune.init(project_qualified_name='shared/onboarding')

with neptune.create_experiment() as exp:
    exp.send_image('quokka', img_pil)

As a result, quokka image is associated with the experiment

image

Log multiple images to neptune

You can log images in a loop. For example, you can augment your image and log it to Neptune:

from imgaug import augmenters as iaa

aug_seq = iaa.Affine(scale={"x": (0.8, 1.2), "y": (0.8, 1.2)},
                     translate_percent={"x": (-0.2, 0.2), "y": (-0.2, 0.2)},
                     rotate=(-25, 25),
                     )

exp2 = neptune.create_experiment()
for run in range(20):
    img_aug= aug_seq.augment_image(img)
    img_pil_aug = Image.fromarray(img_aug)
    exp2.send_image('quokka_version_{}'.format(run), img_pil_aug)

exp2.close()
image

How to log metrics to Neptune?

How to track multiple metrics (loss, scores) in the experiment?

Solution

Step 1: Log

In order to log metrics to Neptune, you simply need to:

import neptune

neptune.init('shared/onboarding')
with neptune.create_experiment():
    # 'log_loss' is User defined metric name
    neptune.send_metric('log_loss', 0.753)
    neptune.send_metric('AUC', 0.95)

Another option is to log key: value pair like this:

neptune.set_property('model_score', '0.871')

Note

You can create as many metrics as you wish.

Step 2: Analyze

Browse and analyse your metrics on the dashboard (example) or in the particular experiment (example experiment).

How to version datasets?

When working on a project, it is not unusual that I change the datasets on which I train my models. How can I keep track of that in Neptune?

Solution

Under many circumstances it is possible to calculate a hash of your dataset. Even if you are working with large image datasets, you have some sort of a smaller metadata file, that points to image paths. If this is the case you should:

Step 1

Create hashing function. For example:

import hashlib

def md5(fname):
    hash_md5 = hashlib.md5()
    with open(fname, "rb") as f:
        for chunk in iter(lambda: f.read(4096), b""):
            hash_md5.update(chunk)
    return hash_md5.hexdigest()

Step 2

Calculate the hash of your training data and send it to Neptune as text:

TRAIN_FILEPATH = 'PATH/TO/TRAIN/DATA'
train_hash = md5(TRAIN_FILEPATH)

neptune.send_text('train_data_version', train_hash)
...

Step 3

Add data version column to your project dashboard:

image

Note

If your dataset is too large for fast hashing you could think about rearranging your data to have a light-weight metadata file.

How to keep my code private?

My code is proprietary, so I do not want to send any sources to Neptune, while training locally. How to do it?

Solution

All you need to do it to pass empty list [] to the upload_source_files parameter, like this:

import neptune

# This function assumes that NEPTUNE_API_TOKEN environment variable is defined.
neptune.init(project_qualified_name='shared/onboarding')

with neptune.create_experiment(upload_source_files=[]) as exp:
    ...

As a result you will not send sources to Neptune, so they will not be available in the Source Code tab in the Web app.

How to upload notebook checkpoint?

I want to add Notebook checkpoint to my project. How to do it?

Solution

Go to your Jupyter, where you can see two Neptune buttons:

  • n button is for configuration changes

  • Upload button is for making checkpoint in Neptune

image

Click Upload, whenever you want to create new checkpoint in Neptune. You will see tooltip with link as a confirmation.

image

Note

You can use Upload as many times as you want.

How to setup Neptune-enabled JupyterLab on AWS?

I would like to run Neptune and track experiments that I run on AWS cloud. How do I do that?

AWS neptune.ml integration

Solution

Register to AWS

Follow the registration instructions from official webpage to create your AWS account.

Start EC2 instance

Start a new EC2 instance. Select ubuntu as your instance type and choose a worker type you need. You can go with t2.micro just to test it out.

ssh to your instance

Connect to your instance by going to the terminal and running:

ssh -i /path_to_key/my_key.pem ubuntu@public_dns_name

(make sure that you put correct key and public_dns_name)

Install docker

Create a new file install_docker.sh:

nano install_docker.sh

Copy the following commands to it:

sudo apt-get update
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

Run the installation script:

source install_docker.sh

Define your secrets

Go to Neptune web app, get your NEPTUNE_API_TOKEN and copy it. Then, create a password for your JupyterLab server.
Set those two secrets to your environment variables NEPTUNE_API_TOKEN and JUPYTERLAB_PASSWORD, like below:
export NEPTUNE_API_TOKEN='your_api_token=='
export JUPYTERLAB_PASSWORD='difficult_password'

Build docker image

Create a new file Dockerfile:

nano Dockerfile

Copy insights of the following Dockerfile to your newly created file:

# Use a miniconda3 as base image
FROM continuumio/miniconda3

# Installation of jupyterlab and extensions
RUN pip install jupyterlab==0.35.6  && \
    pip install jupyterlab-server==0.2.0  && \
    conda install -c conda-forge nodejs

# Installation of Neptune and enabling neptune extension
RUN pip install neptune-client  && \
    pip install neptune-notebooks  && \
    jupyter labextension install neptune-notebooks

# Setting up Neptune API token as env variable
ARG NEPTUNE_API_TOKEN
ENV NEPTUNE_API_TOKEN=$NEPTUNE_API_TOKEN

# Adding current directory to container
ADD . /mnt/workdir
WORKDIR /mnt/workdir
(If you want to run on GPU make sure to change your `Dockerfile` to start from nvidia docker images).

Run following command to build your docker image:

sudo docker build -t jupyterlab --build-arg NEPTUNE_API_TOKEN=$NEPTUNE_API_TOKEN .

Start JupyterLab server

Spin up JupyterLab server with docker:

sudo docker run --rm -v `pwd`:/work/output -p 8888:8888 jupyterlab:latest \
/opt/conda/bin/jupyter lab --allow-root --ip=0.0.0.0 --port=8888 --NotebookApp.token=$JUPYTERLAB_PASSWORD

Forward ports via ssh tunnel

Open a new terminal on your local machine and run:

ssh -L 8888:localhost:8888 ubuntu@public_dns_name &

(make sure that you put correct public_dns_name)

Open JupyterLab server in your browser

Go to localhost:8888 and enjoy your JupyterLab server with Neptune!

Final result

Neptune extensions are enabled and NEPTUNE_API_TOKEN is already in the environment variable so you can work with Notebooks and run experiments with no problems.

How to setup Neptune-enabled AWS SageMaker Jupyter instance?

I would like to use Neptune to track experiments that I run on AWS SageMaker. How do I do that?

Amazon SageMaker neptune.ml integration

Solution

Register to AWS

Follow the registration instructions from official website to create your AWS account.

Create Lifecycle configuration

Go to SageMaker Lifecycle configurations and click on Create configuration.

image

You can choose whatever name you want just make sure to remember it.

Now, you should modify the Create notebook script to run it only once at creation of your SageMaker notebook instance.

image

Copy and paste the script below to your Create notebook tab. Choose in which environments you want to install neptune-client in the PARAMETERS section.

#!/bin/bash

set -e

sudo -u ec2-user -i <<'EOF'

# PARAMETERS
ENVS=(
  python3
  pytorch_p36
)

# Install Neptune client library
for env in $ENVS; do
    source /home/ec2-user/anaconda3/bin/activate $env
    pip install neptune-client
    source /home/ec2-user/anaconda3/bin/deactivate
done

# Install jupyter extension
source /home/ec2-user/anaconda3/bin/activate JupyterSystemEnv
pip install neptune-notebooks
jupyter nbextension enable --py neptune-notebooks --sys-prefix
jupyter labextension install neptune-notebooks
source /home/ec2-user/anaconda3/bin/deactivate

EOF

Create notebook instance

Go to SageMaker Notebook instances and click on Create notebook instance.

image

Select an instance name and click on Additional configuration to add the Lifecycle configuration you have just created:

image

You can now click Create notebook instance instance at the bottom to start your instance.

image

Start notebook

If everything went well your AWS SageMaker instance should be InService and you can now open Jupyter Notebook or Jupyter lab with Neptune notebook-versioning enabled!

image

You can now version your notebooks and track experiments in Amazon SageMaker with Neptune!

image

How to track Google Colab experiments with Neptune?

I would like to run my experiments on Google Colab and track them with Neptune. How do I do that?

Google Colaboratory neptune.ml integration

Solution

Install Neptune client

Go to your first cell in Google Colab and install neptune-client:

! pip install neptune-client

Set Neptune API token

Go to Neptune web app and get your API token. Set it to the environment variable NEPTUNE_API_TOKEN:

% env NEPTUNE_API_TOKEN='your_private_neptune_api_token=='

Delete this cell.

Warning

It is very important that you delete this cell not to share your private token with anyone.

That’s it. Run your training script with Neptune.

import neptune
neptune.init('USER_NAME/PROJECT_NAME')

with neptune.create_experiment():
    neptune.send_metric('auc', 0.92)