Missing configurations using miniconda

sfonseca · February 26, 2021, 2:36pm

Dear Swan experts,

We are developing the notebook at Swan but we are getting the ERROR [0 ] after run the following commands. Can you clarify this error msg for us?

Best regards,

Sandro Fonseca and Eliza Melo

#%%capture

!pip install --user uproot

!pip install --user coffea

!pip install --user numpy

!pip install --user matplotlib

[0] Successfully built numba

ERROR: pyarrow 3.0.0 has requirement numpy>=1.16.6, but you’ll have numpy 1.16.4 which is incompatible.

ERROR: distributed 1.28.1 has requirement dask>=0.18.0, but you’ll have dask 0+unknown which is incompatible.

Installing collected packages: llvmlite, numba, cloudpickle, pyarrow

WARNING: The script plasma_store is installed in ‘/eos/user/s/sfonseca/.local/bin’ which is not on PATH.

Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

#%%capture

!wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

!bash Miniconda3-latest-Linux-x86_64.sh -b -u

!~/miniconda3/bin/conda init

!conda deactivate ; conda deactivate

HTTP request sent, awaiting response… 200 OK

Length: 94235922 (90M) [application/x-sh]

Saving to: ‘Miniconda3-latest-Linux-x86_64.sh’

100%[======================================>] 94,235,922 8.38MB/s in 11s

2021-02-26 14:53:28 (8.10 MB/s) - ‘Miniconda3-latest-Linux-x86_64.sh’ saved [94235922/94235922]

PREFIX=/eos/user/s/sfonseca/miniconda3

Unpacking payload …

Collecting package metadata (current_repodata.json): - done

Solving environment: done

All requested packages already installed.

installation finished.

WARNING:

You currently have a PYTHONPATH environment variable set. This may cause

unexpected behavior when running the Python interpreter in Miniconda3.

For best results, please verify that your PYTHONPATH only points to

directories of packages that are compatible with the Python interpreter

in Miniconda3: /eos/user/s/sfonseca/miniconda3

Python path configuration:

PYTHONHOME = ‘/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt’

PYTHONPATH = ‘/usr/local/lib/swan/extensions/:/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib:/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages’

program name = ‘/eos/user/s/sfonseca/miniconda3/bin/python’

isolated = 0

environment = 1

user site = 1

import site = 1

sys._base_executable = ‘/eos/user/s/sfonseca/miniconda3/bin/python’

sys.base_prefix = ‘/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt’

sys.base_exec_prefix = ‘/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt’

sys.executable = ‘/eos/user/s/sfonseca/miniconda3/bin/python’

sys.prefix = ‘/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt’

sys.exec_prefix = ‘/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt’

sys.path = [

‘/usr/local/lib/swan/extensions/’,

‘/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib’,

‘/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages’,

‘/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python38.zip’,

‘/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python3.8’,

‘/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python3.8/lib-dynload’,

]

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding

Python runtime state: core initialized

ModuleNotFoundError: No module named ‘encodings’

Current thread 0x00007f02dac73740 (most recent call first):

/usr/bin/sh: conda: command not found

/usr/bin/sh: /root/miniconda3/etc/profile.d/conda.sh: Permission denied

ozapatam · February 26, 2021, 3:37pm

Hello @sfonseca

Let me explain what is happening

When you are inside SWAN, you are using software provide by LCG stacks that already have
the python interpreter and all the required environment variables in the system run properly the software from CVMFS.
When you put conda over this enviromnet you are mixing two different python setups with different versions producing a collition.

I found this recipe privide from one of our users

may this can help to isolate the conda enviroment.

But other alternative is to install the package that you need in your home without any conda enviroment,
using pip from the LCG stack.

github.com

swan-cern/help/blob/master/docs/advanced/install_packages.md

# Install packages in CERNBox

LCG releases on CVMFS incorporate new packages quite frequently, so if you think there is a missing package that can be 
potentially useful for a significant number of users, please open a ticket to the [EP-SFT librarians](https://sft.its.cern.ch/jira/projects/SPI)
and ask them to install the package in the LCG releases.

On the other hand, you can install packages on your CERNBox and, if necessary, configure your environment to pick them 
up in SWAN. 

## Python

A typical case is the installation of Python packages, which requires to run pip from a SWAN terminal:

   `pip install --user package_name`

If this fails because you are trying to install an updated version of a package that already exists in CVMFS, you will need to add the `--upgrade` flag.

Then, it would be necessary to add the local installation path to `PYTHONPATH`, by creating a bash startup script that configures that variable (don't forget to call this startup script in the session configuration menu):

``` python

This file has been truncated. show original

We are planning to allow projects have conda environments attached, but the feature is still under development.

Best Regards!
Omar.