Problem using PyTimber.pagestore on SWAN

1

Hello,

I want to use the PyTimber pagestore function to save data files through SWAN on to CERNBox. When I try to run the pagestore function, the files however do not seem to be saved correctly. It does work OK when I run it locally on my PC. Similarly, if I try to read these pagestore files, the data does not seem to be read correctly, even if I try to read data that I have already confirmed should work. Any ideas…?

Some code that uses said function and saves the data (this exact code runs without problem on my local machine but not on SWAN!):
https://gitlab.cern.ch/abt-projects/pulsed-power/agnieszka-vacuum-scripts/-/blob/master/import_data.py

Thank you,
Oskar

2

Hello Oskar,

I can not access the link to take a look at your code,
can you may post some minimal example and some instructions to reproduce the problem?

Best
Omar.

3

Hi Omar,

Sorry, I didn’t realize you couldn’t access it.

The following code seems to reproduce the problem as well - let me know if you need anything else! Thank you.

Oskar

import pytimber
from pytimber import pagestore
import time, calendar
#from numba import jit
%matplotlib notebook
ldb = pytimber.LoggingDB()

mydbPressureMDB = pagestore.PageStore('lhcPressure.db', './')

t1 = "2012-07-01 00:00:00"
t2 = "2012-07-01 23:59:59"
ts1 = calendar.timegm(time.strptime(t1,"%Y-%m-%d %H:%M:%S")) - 0 * 3600
ts2 = calendar.timegm(time.strptime(t2,"%Y-%m-%d %H:%M:%S")) - 0 * 3600

keys_beam1 = ['MKI.D5L2.B1:PRESSURE', 'VGPB.118.5L2.C.PR', 'MKI.B5L2.B1:PRESSURE', 'MKI.A5L2.B1:PRESSURE', 'VGPB.137.5L2.B.PR', 'VGPB.14.5L2.B.PR', 'VGPB.176.5L2.B.PR', 'VGPB.192.5L2.B.PR', 'VGPB.59.5L2.B.PR', 'VGPB.98.5L2.B.PR', 'LHC.BCTFR.A6R4.B1:BEAM_INTENSITY', 'MKI.A5L2.B1:PRESSURE_INT', 'MKI.B5L2.B1:PRESSURE_INT', 'MKI.C5L2.B1:PRESSURE_INT', 'MKI.D5L2.B1:PRESSURE_INT']
keys_beam2 = ['MKI.D5R8.B2:PRESSURE', 'MKI.C5R8.B2:PRESSURE', 'MKI.B5R8.B2:PRESSURE', 'MKI.A5R8.B2:PRESSURE', 'VGPB.138.5R8.R.PR', 'VGPB.14.5R8.R.PR', 'VGPB.176.5R8.R.PR', 'VGPB.192.5R8.R.PR', 'VGPB.59.5R8.R.PR', 'VGPB.98.5R8.R.PR', 'LHC.BCTFR.A6R4.B2:BEAM_INTENSITY', 'MKI.A5R8.B2:PRESSURE_INT', 'MKI.B5R8.B2:PRESSURE_INT', 'MKI.C5R8.B2:PRESSURE_INT', 'MKI.D5R8.B2:PRESSURE_INT']

def Store_data(keys):
    start_time = time.time()
    timber_data = ldb.getAligned(keys, ts1, ts2)
    end_time = time.time()
    for key in keys:
        mydbPressureMDB.store_variable(key, timber_data['timestamps'], timber_data[key]);

print('Aligning and storing data from ',t1,' to ',t2)
Store_data(keys_beam1+keys_beam2)
4

Hello Oskar,

I did it, I ran it in SWAN and in my computer using LCG_97a.
But I can not see the difference in ‘lhcPressure.db’

I did a small code to see the values in the table pages in sqlite in SWAN swan001.cern.ch

image
image1650×777 284 KB

and I ran it in my local host

image
image1658×545 176 KB

which is your error?

Best,
Omar.

5

Hi Omar,

How do you mean “can’t see the difference in lhcPressure.db’?”

The error that I am getting is

FileNotFoundError: [Errno 2] No such file or directory: ‘./1.idx’

I find this strange because this file 1.idx is one of the files that the program will output if I run it on my local machine.

6

Hi Oskar,

I can see that files too, can you tell me which stack are you using and which SWAN node?

image
image1370×641 34.2 KB

7

Hi Omar,

I am just running it on the default stack, 97a. How do I see which SWAN Node I am using?

Thank you!

8

Hi Oskar,

in the url in the browser, in the previous post, you can see the image where the url
is swan001.cern.ch.

Cheers
Omar.

9

Hi Omar,

I tried just now on node 001 and the problem remains the same. I’m pretty sure that I’ve used other nodes too.

10

Hello Oskar,

I was able to reproduce the problem,
The notebook on EOS is not producing the files, I created a notebook in /tmp using a symlink and it works.

I am going to ask our experts on EOS.
Best
Omar.

11

Hi @objorkqv ,

can you please try to run your code again?
we think that the problem was solved.

Cheers
Omar.

12

Hello Omar,

I’m very sorry that I am responding so slowly - the SWAN community page is not sending me any notifications for some reason.

I tried running it again but the error message was the same:

[Errno 2] No such file or directory: ‘/eos/…/1.idx’

13

Hi Omar,

Do you have any news on this topic? Many thanks.

Oskar

14

Hi Oskar,

Yes, I talked today with a colleage that is a expert on EOS,
we think may some cache is causing the problem, our plan is to restart the EOS services,
to see if this solves the problem, otherwise I think we need to open a ticket to the EOS experts.

As soon as we do the restart of the services I will let you know to do the test.

Apologize for the late reply, this is wierd issue and we are trying to figure out what is happening.

Best Regards
Omar.

15

Hi @objorkqv
can you please try your code again in the machine swan005.cern.ch, it was already rebooted
and the code works for me

image
image818×870 122 KB

Cheers
Omar.

16

Hi @ozapatam

I tried going directly to swan005.cern.ch but it says ‘This machine is scheduled for maintenance.
Please, always use swan.cern.ch instead of going directly to a specific machine.’

Cheers,
Oskar

17

Hello Oskar,

Can you please try again?

It should be fine now :smiley:
Omar.

18

Hi Omar,

I could access swan005.cern.ch now but the scripts still does not run. Sorry!

Cheers,
Oskar

19

Hi @objorkqv

In that case please open a ticket for the EOS project experts

please add me in the watch list to see if they need something from SWAN side.

Best
Omar.

20

Hi Omar,

Will do. Thanks for your efforts to help us out anyway.

Kind regards,
Oskar

1 Like