PyRDF issues

@etejedor @vbrian this is quick fix

  • Create bash pyrdf_fix.sh script with
source /cvmfs/sft.cern.ch/lcg/views/dev3python3/Wed/x86_64-slc6-gcc8-opt/setup.sh
  • go to swan006.cern.ch
  • select Bleeding Edge Python3, Cluster Kubernetes, and $CERNBOX_HOME/pyrdf_fix.sh
  • connect and run

You have thus time until this Wednesday until the broken package is fixed. @etejedor can you help us raise issue to Bleeding Edge maintainers?

Hi,
Do you have more information about the error? It is hard to tell what is going on.

When you say there is a broken package, you mean PyRDF? How do you know it is PyRDF? No new version should have been published on bleeding edge for PyRDF, no update has happened in the past few weeks.

@etejedor I don’t know what is broken exactly

Brian has Caused by: java.io.EOFException which is essentially meaning some stage in the middle exploded and did not return result, I get some crazy segmentation faults from ROOT

 *** Break *** segmentation violation
 Generating stack trace...
 0x00007f45df11bcfc in <unknown> from /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Mon/x86_64-centos7-gcc8-opt/lib/libCling.so
 0x00007f45df11c3f6 in <unknown> from /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Mon/x86_64-
....
/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Mon/x86_64-centos7-gcc8-opt/lib/libpython2.7.so.1.0
 0x00007f45fc3a1445 in __libc_start_main + 0xf5 from /lib64/libc.so.6
 0x000000000040066e in <unknown> from python 

I can give to some developer detailed logs of what happened if needed. We can maybe take this offline.

Dear Piotr,

If you don’t mind, and if possible, I would be glad to be in some mailing loop. Even if I won’t help anyone, but just by curiosity :blush:

@vbrian did the quickfix with additional source unblock you? Probably the one from Thursday/Friday will also work :slight_smile: Yes, as soon as we know the root-cause probably we will post here.

@pmrowczy I tried but no, it seems that it does not override the environment as I would have expected

i.e. same error and

echo $LCG_VIEW
/cvmfs/sft.cern.ch/lcg/views/dev3python3/latest/x86_64-centos7-gcc8-opt

@vbrian @etejedor ok, I made it work by changing to this in $CERNBOX_HOME/pyrdf_fix.sh and used any PyRDF Notebook:

export LD_LIBRARY_PATH=/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3python3/Thu/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/torch/lib:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3python3/Thu/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/tensorflow:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3python3/Thu/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/tensorflow/contrib/tensor_forest:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3python3/Thu/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/tensorflow/python/framework:/cvmfs/sft.cern.ch/lcg/releases/java/8u222-884d8/x86_64-centos7-gcc8-opt/jre/lib/amd64:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3python3/Thu/x86_64-centos7-gcc8-opt/lib64:/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3python3/Thu/x86_64-centos7-gcc8-opt/lib:/cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/lib:/cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/lib64:/cvmfs/sft.cern.ch/lcg/releases/binutils/2.30-e5b21/x86_64-centos7/lib:/usr/local/lib/:/cvmfs/sft.cern.ch/lcg/releases/R/3.5.3-877c1/x86_64-centos7-gcc8-opt/lib64/R/library/readr/rcon

I have no idea what is wrong with ROOT in dev3python3/Mon

Hi Brian,
We are deploying a new container image for the k8s Spark clusters that should solve the issue. We’ll let you know when you can try again.
Cheers,
Enric

Dear Enric,

Thanks a lot, good news ! At least Piotr’s workaround is working fine (and I have no urgent time constraint right now…). By curiosity, have you spotted the issue ?

Cheers,
Brian

Hi Brian,
The crash was due to a newly introduced dependency of ROOT with glibc-headers, which was not installed in the k8s image.
Cheers,
Enric

Dear @vbrian

The new image (swan:v2) for spark executors with glibc-headers is deployed now. Please can you give it a try without the workaround and my recreating your spark session. Thank you

regards,
Prasanth

Dear Prasanth, Dear Piotr, Dear Enric, Dear everyone,

It seems to work ! Great job :smiley:
Thanks a lot for your reactivity and help on that

Best regards,
Brian