Make new package for panda HPO task submission available on swan

pmtuan · August 22, 2021, 6:04am

Hi everyone,

I am developing as a part of my qualification task for ATLAS authorship a python package to submit hyperparameter optimization jobs for machine learning to the CERN Grid from a Jupyter notebook using interactive and graphical tools. To make the package widely available for end-users, I am wondering if it can be integrated as a package to run on SWAN. Please let me know. Thanks.

etejedor · August 23, 2021, 7:58am

Hello,

I think for now the best option would be to ask users to install it on their CERNBox:

github.com

swan-cern/help/blob/master/docs/advanced/install_packages.md

# Install packages in CERNBox

LCG releases on CVMFS incorporate new packages quite frequently, so if you think there is a missing package that can be 
potentially useful for a significant number of users, please [let the SWAN team know](mailto:swan-talk@cern.ch) and 
[contact the librarians directly](https://sft.its.cern.ch/jira/projects/SPI).

On the other hand, you can install packages on your CERNBox and, if necessary, configure your environment to pick them 
up in SWAN. 

## Python

A typical case is the installation of Python packages, which requires to run pip from a SWAN terminal:

   `pip install --user package_name`

If this fails because you are trying to install an updated version of a package that already exists in CVMFS, you will need to add the `--upgrade` flag.

Then, it would be necessary to add the local installation path to `PYTHONPATH`, by creating a bash startup script that configures that variable (don't forget to call this startup script in the session configuration menu):

``` python

This file has been truncated. show original

I understand this package is now under active development and new releases will appear rather often. For such cases, installing on CVMFS (the distributed file system that provides software for SWAN) is not ideal, since new software releases on CVMFS appear every few months. On the other hand, users that install it on their CERNBox can update it as often as they want.