CERN Accelerating science

Bleeding edge python3 not working

#1

Dear SWAN team,

First, I wish you all a very happy and fruitful new year !

I write this topic because I found since yesterday some instabilities in the package Bleeding Edge Python3. I know those instabilities are possible in bleeding edge (that is the point of this software stack), but I wanted to make some feedback on its usage since it was not working even yesterday.

To clarify the issue, the notebook seem to be connected to a kernel at first, but anytime I try to launch any kind of cell, the kernel dies and restart showing the error message “kernel has died, it will be relaunched automatically” (translated from French so might be not exact). A reproducing snippet of code is not necessary since it happen launching any cell from any notebook, even the simplest example notebook.

Maybe this issue is known and under investigation, in this case, apologies for spamming :slight_smile:

Best regards,
Brian

#2

Hi @vbrian This issue is being investigated currently, we are developing and deploying the fix that resolves this issue. Thank you

regards,
Prasanth

#3

Yes, if you select bleeding edge dev3/latest, go to Terminal, and execute

python
> import ROOT

This will show you the error. There is some problem with ROOT in newest build. We found it with our test pipelines and notified relevant people.

#4

Relevant ticket:
https://sft.its.cern.ch/jira/browse/ROOT-10506

we expect to merge a fix between today and tomorrow.

#5

we see our testing pipelines for PyRDF and Spark on Development Bleeding Edge started succeeding https://gitlab.cern.ch:8443/db/swan-spark-notebooks/-/jobs/6778352. @vbrian ROOT seems to work back in that development release

#6

Yes, all should be back to normal in bleeding edge, this PR was merged:

#7

Dear everyone,

Thanks a lot for your reactivity on this issue !

I think however that another problem is causing again the same issue ; not being able to connect a notebook to a bleeding edge python3 kernel.
This issue may be under investigation but I wanted to make it known here because it has been here since Monday as far as I remember.

Thanks a lot again !

Cheers,
Brian

#8

Hi Brian,

Yes, LCG Dev3 is for now unavailable, we received support tickets too. For the future, we will try to update here when we know the LCG Dev3 is investigated.

Current status:

04-02-2020 17:46:21 - Diogo Castro
Bleeding edge was updated to Python 3.7 last week. This seems to break compatibility with Python provided in the docker image. We apologize for this, but we need to understand how to completely isolate the notebook environment from the docker image. This might take some time, but we’ll try our best to fix this as soon as possible.

#9

@vbrian and others. We have a temporary fix in swan-spare003.cern.ch . If you have no access, let us know in swan-admins@cern.ch

#10

Dear SWAN team,

I would like to use a notebook with Bleeding Edge python3.

When I connect to swan.cern.ch i get redirected to swan006.cern.ch.
When I open a notebook with this configuration (bleeding edge, swan006) I get connection failed error alert and the notebook is not usable.

When opening the same notebook bleeding edge python 3 from swan-spare003.cern.ch, it is able to connect to the kernel just fine, but

import ROOT

returns

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-ee8dc4376aa8> in <module>
----> 1 import ROOT

/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3python3/Tue/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages/ROOT/__init__.py in <module>
     26 environ['CPPYY_NO_ROOT_FILTER'] = '1'
     27 
---> 28 import cppyy
     29 import ROOT.pythonization as pyz
     30 

/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3python3/Tue/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages/cppyy/__init__.py in <module>
     69     from ._pypy_cppyy import *
     70 else:
---> 71     from ._cpython_cppyy import *
     72 
     73 

/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3python3/Tue/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages/cppyy/_cpython_cppyy.py in <module>
     16 # first load the dependency libraries of the backend, then pull in the
     17 # libcppyy extension module
---> 18 c = loader.load_cpp_backend()
     19 import libcppyy as _backend
     20 _backend._cpp_backend = c

/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3python3/Tue/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages/cppyy_backend/loader.py in load_cpp_backend()
     72 
     73     if not c:
---> 74         raise RuntimeError("could not load cppyy_backend library")
     75 
     76     return c

RuntimeError: could not load cppyy_backend library

Could this problem be related to the one described almost a month ago in this thread?
Have I chosen the wrong configuration on the wrong machine?
Is it there some more information I could provide?

Best Regards

Dario

#11

Dear Dario,
The environment of the ROOT installation on bleeding edge is not configured correctly at the moment. We expect to have a fix this week.
Please use LCG96 if you can for the moment, sorry for the inconveniences.
Enric