Spark monitoring from notebook

Spark
1

Dear developpers,

I would like to report a small bug about the spark monitoring window.
Indeed I guess that this spark monitoring display is supposed to appear just right after the cell asking for a spark job to be done. However, from time to time I observe that this spark display appear after an other random cell…(usually around the end of the whole notebook, and I have a lot of cell if that might be related…).
I also would like to say that this is not a critical issue, but just to mention this small issue, since it happens from time to time (for instance this issue was not present until monday or tuesday…). Moreover, this might be related to the fact that I use the LC96 bleeding edge software stack ?

Bests regards,
Brian

2

Dear Brian,
Thank you for the report.

So you say this is something that you noticed recently (Monday, Tuesday)? @pkothuri @pmrowczy can that give us a hint of what could be going on, maybe some recent development for the Spark monitor?

Brian, you think you can come up with a reproducer we can try or it is really intermittent?

3

Dear Enric,

This is something I noticed (since) this week for sure, and if I remember well, that it was at the beginning of the week…
However as for the reproducer, if you want I think I can share the notebook I run (+ the inputs…), but this cannot be called a “simple” reproducer since there are many cells and one has to dig in the many subsections to get where the display has been drawn…But tell me to who I should share it if you are willing to search for the display (and btw if you want to burn your eyes…)
Moreover I will keep you in touch next week if this has magically disappeared or not.

Thanks a lot for your reactivity !
Best regards,
Brian

4

Hi,
Sure, can you share a project on SWAN with me, Prasanth and Piotr (I mentioned their usernames in my previous comment). If possible, please try to make it self-contained (so that we can just open and run). What Spark cluster are you running this on? The LCG release is bleeding edge, then?

5

Dear @vbrian, this could be a regression introduced by last update in spark monitor extension we added, that you probably triggered just by having lots of cells last week and previous weeks you probably had less cells. We will try to reproduce it. Thanks!