Storage

Neptune comes with a cloud-based storage to store data about experiments: source code and results (channel values, logs, output files, e.g. trained models). On top of that, you can upload any data to the project’s storage and use it in your experiments.

Directory Structure of a Project’s Storage

Each project has an independent storage content. In particular, you cannot use data stored in the project A to run experiment within the project B.

The data within a project is structured in the following way:

uploads/
├── uploaded_file.txt
├── uploaded_dir/
│   └── [...]
└── [...]
{shortId}/
├── backup/
│   └── [...]
├── output/
│   └── [...]
├── images/
│   └── [...]
├── channels/
│   └── [...]
├── stdout
├── stderr
│
└── source/
    └── [...]
{shortId}/
└── [...]

Absolute and Relative Paths

Whenever you’re working with Neptune’s storage, be it via the CLI, or in Neptune UI, the default working directory is /uploads. If you want to access data from your experiments, you need to type the path starting from /{shortId}/....

Data and Experiments

Using Data as an Experiment’s Input

Files and directories on Neptune’s storage can be used in experiments. You can use flag --input to make any file or directory accessible for experiment during runtime in the /input directory . For example

neptune send \
  --input input.txt:data.txt \
  --input extracted.tgz/001.txt \
  --input /SAN-7/output/weights.txt:weights.txt \
  --input my_images_directory:images \
  mnist_cnn_neptune.py --kernel_size 3

The command above makes respective files visible from experiment’s code under /input/data.txt, /input/001.txt and /input/weights.txt. The directory my_images_directory is available as /input/images.

By default, files and directories are available under their own names, but you can override this behavior using the actual_name:desired_name syntax.

You can use outputs from one experiment as an input to another experiment. Take a close look at the example above: /SAN-7 is a directory of a previous experiment.

Saving Experiment’s Outputs

Output Directory

Besides using stored data as input, your experiment can output any files and directories to directory /output/. They end up in /{shortId}/output/. In the example above output of the experiment SAN-7 is used as input of the new experiment.

Using --backup Parameter

In addition to using the /output directory, you can save the side effects of experiment. You can use flag --backup to make any file or directory be saved in /{shortId}/backup directory. For example

neptune send \
  --backup "/tmp" \
  --backup "main.py"

The command above will copy all files and directories from /tmp directory and file main.py form working directory to /{shortId}/backup directory after experiment ends.

The --backup flag can be also used when running experiment on your own computer as shown in the example below:

neptune run \
  --environment keras-2.2-gpu \
  --backup "/tmp" \
  --backup "main.py"

Using Public Datasets

Neptune provides some popular datasets, e.g. from Kaggle contests. Now you don’t have to upload them, they are available in /public read-only directory. You can find detailed information about datasets here:

Managing Storage Content

Uploading Data To Storage

You can upload any files and directories to the shared storage of your project using Neptune CLI’s neptune data upload command.

Example of CLI usage:

  • neptune data upload /local/path/to/file.txt uploads the file file.txt to /uploads/file.txt;
  • neptune data upload /local/path/to/file.txt path/to/uploaded uploads the file file.txt to /uploads/path/to/uploaded/file.txt;
  • neptune data upload -r my_directory uploads local directory along with its content to /uploads/my_directory.

Browsing Stored Data

You can browse your files and directories on the storage via Browse files tab of Neptune UI.

images browse storage

Additionally, you can always use Neptune CLI’s neptune data ls command to list files and directories in your project.

Example of CLI usage:

  • neptune data ls lists all the files and directories in /uploads;
  • neptune data ls dirname lists all the files and directories in /uploads/dirname;
  • neptune data ls /dirname lists all the files and directories in /dirname.

Downloading Data

You can download files and directories from Neptune storage via Neptune CLI’s neptune data download command. Additionally, you can always download singular files via Neptune UI.

Example of CLI usage:

  • neptune data download file.txt downloads the file /uploads/file.txt and saves locally under file.txt;
  • neptune data download /SAN-7/output/weights.txt downloads the file /SAN-7/output/weights.txt and saves it locally under weights.txt;
  • neptune data download file.txt /local/my_file.txt downloads file /uploads/file.txt and saves it locally under /local/my_file.txt;
  • neptune data download my_directory downloads /uploads/my_directory along with it content.

Removing Data from Storage

You can remove any uploaded files or directories via Neptune CLI neptune data rm command.

Example of CLI usage:

  • neptune data rm my_file.txt removes file /uploads/my_file.txt from the shared storage.
  • neptune data rm -r my_directory removes directory /uploads/my_directory along with its content from the shared storage.