Anaconda Distribution ¶

While some pre-built modules that contains the required Python dependencies might already be available in the cluster where the user plans to execute the HYPPO software, it is very likely some dependencies might have to be installed separately from the module (see this section ). One easy solution is to create your own anaconda environment . In this page, we give a brief tutorial on how to set up your own conda environment.

New conda environment ¶

Create & activate ¶

The procedure is pretty simple. A new environment can be initialized using the conda create command as follows:

            conda create -n hyppo python=3.10

           

The above will create a new folder in the HOME directory named hyppo which will contain the entire custom anaconda environment built using python3.6. In order to execute a program using this environment, the user needs to activate the environment which can be done using the source activate command:

            conda activate hyppo

           

Note

For execution on allocated nodes in a cluster, the environment will have to be activated inside the SLURM script that is to be submitted.

Installing packages ¶

Once activated and to make the HYPPO software work, we can install all the dependencies using the requirements.txt file as follows:

            pip install -r requirements.txt

           

We note that the default PyTorch package will be the one with CUDA enabled, which is what we want to do distributed machine learning training. The GNU parallel should also be installed and can be done as follows:

            conda install -c conda-forge parallel

           

Ready-to-build environment ¶

Environment file ¶

The list of libraries installed within the currently active environment can be saved into a YAML file, e.g. environment.yml , as follows:

            conda env export > environment.yml

           

This will facilitate other users to build an identical environment in their own machine and/or repository:

            conda env create -n hyppo -f environment.yml

           

The above command will create a new environment repository usually under the .conda/envs cached path within the home directory. To see a full list of all of the environments in the system, the following command can be used (here under Cori):

            >>> conda info --envs
# conda environments:
#
base                  *  /global/common/cori_cle7/software/jupyter/cori/21-03
software                 /global/homes/v/vdumont/.conda/envs/software

           

HYPPO environment ¶

The anaconda environment file linked below ( software.yml ) can be used to create a workable Python environment to execute the HYPPO software as is. If one needs to install additional libraries which could be required by some other packages, the libraries can be easily installed within the activated environment by using the pip install or conda install command.

Download environment file

Cloning & customization ¶

If one already has a custom environment set up and listed when typing conda info --envs , or if one created an environment from the available environment files listed above, it is possible to further customize the environment by activating it and install new libraries. The user might decide to not alter the existing environment and create a separate version, this can be achieved using the --clone option. For example, in order to create an separate environment to compile the documentation, the following command could be executed:

            conda create --name documentation --clone software

           

Danger

Do not attempt to just copy the environment’s repository using the cp -r command and rename the copied directory as conda will miserably fail to recognize the new environment.

Warning

Generating several conda environments may take up significant memory, avoid creating redundant environments.

This above command will clone the software environment and create a copy called documentation . One can then activate the cloned environment and install other libraries needed for what the new environment is intended to be used. For instance, to compile the documentation all the Sphinx-related libraries need to be installed which can be done using the requirements.txt file for the documentation as follows:

            conda activate documentation
pip install requirements.txt
conda install -c conda-forge pandoc

           

Finally, in case the user wishes to remove an existing anaconda environement, the following command could be used:

            conda env remove --name environment_name

           