How to use outputs from one experiment in another experiment?

Problem

She ran successful experiment in the Neptune cloud and saved output data or model weights. Now, she wants to use this data in another experiment. How to do it in the Neptune cloud?

Solution

Step 1

Perform your experiment in the cloud, for example via:

1
neptune send --worker 2xl-8k80 --environment keras-2.2-gpu run_exp_1.py

and save something, for example some metadata in the /output/metadata.csv.

In the script run_exp_1.py it was something along this line:

1
2
3
...
metadata_df.to_csv('/output/metadata.csv', index=None)
...

You can check, that your data is located in the Output tab in the experiment view (example).

Step 2

In order to use metadata.csv in another experiment you need to check what is the id of the first experiment. In this example it was: SAL-580.

Pass it to the --input parameter in new experiment command:

1
neptune send --input /SAL-580/output/metadata.csv -w 2xl-8k80 -e keras-2.2-gpu run_exp_2.py

You can now access metadata.csv in the code, like this:

1
2
3
4
5
import pandas as pd

metadata = pd.read_csv('/input/metadata.csv')

...

Re-using results in a nutshell

  • Re-using results:
    • in the preceding experiment, save results to /output/MY_RESULTS
    • in the following experiment:
      • use input parameter: neptune send --input /EXP-ID/output/MY_RESULTS ...
      • use data in script: results = load(MY_RESULTS)

See also