Anyway to use Tensorflow 1.6.0 in SWAN?

suho · May 13, 2021, 9:36pm

Dear ML (Tensorflow) experts…
I am using Keras-Tensorflow for ML training and have been using TF code well in cmslpc cluster with
Keras version 2.2.4 and TF version 1.6.0 (python 2.7.14+).

To run the jobs in VM, I started using the CERN SWAN, which uses Keras version 2.4.0 and TF version 2.3.0 (Python 3.7.0)
The TF code (made for 1.6.0) were not initially compatible with SWAN VM (TF 2.3.0), but most were resolved with importing tf as tensorflow.compat.v1. and it gave my .pb file sucessfully.
However, the problem rises again when I try to use the .pb file completed with SWAN back into my CMSLPC, which uses TF 1.6.0.
When I loop over my ntuples to obtain its discriminator vlaue from the TF, it crashes.

Long story short, I would love to use the SWAN VM with just Keras and TF version that’s identical to CMSLPC version, so my CMSLPC code can be used in SWAN and back into cmslpc easily. Can anyone help me to use KERAS, Tensorflow, AND PYTHON in SWAN to shift from (2.4.0,2.3.0,3.7.0) to CMSLPC version (2.2.4, 1.6.0, 2.7.14+) with some shell PATH commands?? Thank you!

etejedor · May 17, 2021, 8:01am

Hello,

Regarding the Python version, when starting your SWAN session, you can select a Software stack that is based on Python2 (e.g. 99 Python2).

Once you have created that session, you can install (e.g. with pip) the versions of Keras and Tensorflow that you wish to use on your CERNBox, create an environment script to modify the PYTHONPATH as described here:

github.com

swan-cern/help/blob/master/docs/advanced/install_packages.md

# Install packages in CERNBox

LCG releases on CVMFS incorporate new packages quite frequently, so if you think there is a missing package that can be 
potentially useful for a significant number of users, please [let the SWAN team know](mailto:swan-talk@cern.ch) and 
[contact the librarians directly](https://sft.its.cern.ch/jira/projects/SPI).

On the other hand, you can install packages on your CERNBox and, if necessary, configure your environment to pick them 
up in SWAN. 

## Python

A typical case is the installation of Python packages, which requires to run pip from a SWAN terminal:

   `pip install --user package_name`

If this fails because you are trying to install an updated version of a package that already exists in CVMFS, you will need to add the `--upgrade` flag.

Then, it would be necessary to add the local installation path to `PYTHONPATH`, by creating a bash startup script that configures that variable (don't forget to call this startup script in the session configuration menu):

``` python

This file has been truncated. show original

then restart your session and select that user environment script in the web form.

We are planning to make this easier and allow the user to define custom environments per project with the versions of packages they desire, but this is not in production yet.