Plots generated with %matplotlib widget are not being saved when converting to HTML file

smazenou · February 12, 2024, 5:54pm

Hello,

I am trying to create notebook reports for archiving purposes, but it seems that the widget output is not saved.

Steps to reproduce:

102 with nxcals pro, in a notebook called Untitled:

import matplotlib.pyplot as plt
import os
import getpass
import sys

%matplotlib widget
plt.plot([0, 1, 2, 3, 4], [0, 3, 1, 2, 5])
plt.show()

os.system(f'{sys.executable} -m jupyter nbconvert --to html Untitled.ipynb')
os.system(f'mv Untitled.html /eos/user/s/{getpass.getuser()}/')

Thanks in advance for your precious help.

pesteves · March 4, 2024, 1:24pm

Dear Simon,

You can save the plot by using the following code:

import matplotlib.pyplot as plt
import os
import getpass
import sys

# Create the plot
plt.plot([0, 1, 2, 3, 4], [0, 3, 1, 2, 5])

# Save the plot as an image
plt.savefig('/eos/user/s/YOURUSER/plot.png')

# Show the plot
plt.show()

# Convert notebook to HTML
os.system(f'{sys.executable} -m jupyter nbconvert --to html Untitled.ipynb')
os.system(f'mv Untitled.html /eos/user/s/{getpass.getuser()}/')

Let me know if this works for you. I tested it and I could see the plot in the html file.

Best,
Pedro

smazenou · March 4, 2024, 1:54pm

Dear Petro,

Thank you for this answer. However, I might not have been clear explaining my problem. I am developing notebooks for the MP3 group, and they would like to have both interactive notebooks (created with the magic command %matplotlib widget) and html reports of the notebook (created with the jupyter nbconvert command). The problem I’m facing is that interactive plots are not being saved in the html report, even though this behaviour works in vscode or in a “classic” jupyter environment.

My quick fix was to add this comment to every plot:

%matplotlib inline
# To display plots dynamically run the following line (don't forget to put again inline and run the cell again for the plot to appear in the report)
# %matplotlib widget

However, this solution is far for being ideal given that matplotlib widget is not the default and that one has to rerun the cell to be able to zoom, and that one need to remember to rerun the cell with matplotlib inline for the plot to appear in the report.

I hope those explanations were clearer and that you will be able to solve this issue

etejedor · March 6, 2024, 1:49pm

Dear Simon,

So if I understand correctly, you want the generated HTML to have interactive plots? Or just a static image would do?

Cheers,
Enric

smazenou · March 6, 2024, 2:11pm

Dear Enric,

I would like to have the images being static in the generated report indeed. Having interactive plots in the generated HTML report would be a nice to have, but it’s out of the scope of the issue.

Cheers,
Simon

etejedor · March 6, 2024, 2:16pm

Dear Simon,

It might be an issue with the nbconvert version that is in SWAN, I see this issue:

github.com/jupyter/nbconvert

exporting matplotlib plots to html when using %matplotlib widget or notebook

opened 08:03AM - 12 Oct 21 UTC

closed 09:00PM - 14 Nov 21 UTC

Cube707

question

# exporting matplotlib plots to html when using `%matplotlib widget` or `noteboo…k` I am sorry if this is a duplicate, but I have read a bunch of other issues and still don't understand if this should work or if it is correct behaviour. # Problem: I am trying to export a notebook that contains matplotlib plots and using the magic command `%matplotlib widget` to get interactive plots. See example: ```python %matplotlib widget from matplotlib import pyplot as plt from math import sin x = list(range(100)) y = [sin(x) for x in x] plt.plot(x,y) ``` Reading [this](https://github.com/jupyter/nbclient/pull/75#issue-627213467), this way should work and be the desierd way. Now when exporting to html no images appear in the resulting file. I have tried these commands: ```bash jupyter-nbconvert --to html notebook.ipynb jupyter-nbconvert --to html --execute notebook.ipynb ``` Using `--execute` I get parts of the widget (the buttons), but still not visible image. This also causes an issue downstream, where the pdf-exporter relies on the html exporter and there for [has the same problem](https://github.com/betatim/notebook-as-pdf/issues/34). ## Possible workarounds: One could just search and replace all magic commands and rerun. This will produce static images which will export just fine. But I feel like this is an unnecessary step and could be added as a feature. I personally don't care about the interactive features and could live with just a static image export. ## Some more analysis: The behaviour differs a lot depending on the tools used and in which order, which is part of why I am so confused. It seems like some temporary data is stored somewhere, and a reboot is needed to get rid of it sometimes. Here are some attempts of untangling this: - Running `%matplotlob widget` in VScode, then using `jupyter-nbconvert --to html notebook.ipynb` - Running `%matplotlob widget` in Jupyter Lab and than using `jupyter-nbconvert --to html notebook.ipynb` - Running `%matplotlob widget` with `jupyter-nbconvert --execute notebook.ipynb --inplace & jupyter-nbconvert notebook.ipynb --to html` With all of this the output datatype is set to `"application/vnd.jupyter.widget-view+json"`, which seems to be required [as described here](https://github.com/jupyter/nbconvert/issues/1081#issuecomment-528087715). ```json { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "version_major": 2, "version_minor": 0, "model_id": "c4687722682d4bbbb1fad0aa8e6e0d26" }, "image/png": "loads of base64....", "text/plain": [ "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …" ] }, ``` Most of the times this gets some javascript generated and displayes the wiget buttons, but no image. But the image is present as base64, which I assume should work, but doesn't? Acationally this just doesn't work and has no javascript inside the resulting html file. - Running `%matplotlob notebook` in VScode, then using `jupyter-nbconvert --to html notebook.ipynb`: not supported, doesn't work - Running `%matplotlob notebook` in Jupyter Lab and than using `jupyter-nbconvert --to html notebook.ipynb`: This is the only instance that works!! A static image gets displayed. This seems to be because there is a `<img>` tag with base64 present that is used as a fallback: ```json "outputs": [ { "data": { "application/javascript": [ "so much JS......." ], "text/plain": [ "<IPython.core.display.Javascript object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<img src=\"bounch of base64.....\" width=\"640\">" //this seems to be what works ], "text/plain": [ "<IPython.core.display.HTML object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[<matplotlib.lines.Line2D at 0x25824ca5dc0>]" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ] ``` - Running `%matplotlob notebook` with `jupyter-nbconvert --execute notebook.ipynb --inplace` and `jupyter-nbconvert --execute notebook.ipynb --to html`: No result is shown. Once again a bunch of JS is generated, but doesn't seem to work inside the html. ```json "outputs": [ { "data": { "application/javascript": [ "loads of JS, once again...." ], "text/plain": [ "<IPython.core.display.Javascript object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<div id='ad87f4e9-85ac-493f-b215-f7bbecc29d08'></div>" // this has no imagedata attached ], "text/plain": [ "<IPython.core.display.HTML object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "[<matplotlib.lines.Line2D at 0x1e6ec5d7940>]" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], ```

Are you running this from the SWAN Classic UI or from the new JupyterLab (Alma9)? I believe it’s worth trying in JupyterLab since there we have newer versions of all Jupyter packages.

Cheers,
Enric

smazenou · March 6, 2024, 2:31pm

Given that our notebooks requires NXCALS accees, we are stuck using 102b and CC7.

etejedor · March 6, 2024, 2:35pm

Can you check if it works with %matplotlib notebook + jupyter nbconvert as suggested in the issue linked above?

smazenou · March 6, 2024, 2:42pm

%matplotlib notebook used to be working, but we had to pin it to version matplotlib==3.1.3 due to another bug (see Bug in interactive matplotlib plots in SWAN notebooks)

The problem is that we wanted to remove that old dependancy, so we switched to the notebook backend, created new plots in our notebook and now it would be a great effort to migrate everything back.

smazenou · March 6, 2024, 2:44pm

Also, fyi, the ipympl package (i.e. the one responsable for the %matplotlib widget magic command) is not installed on the JupyterLab environment.

etejedor · March 6, 2024, 2:48pm

Can you please check if the conversion to HTML itself works well (with matplotlib widget) if you select LCG 105a (not NXCALS), CentOS7?

If that is so, we are now in the process of creating a new NXCALS software stack that will be based on LCG 105a, so we would just wait until that is available and the problem would be solved.

smazenou · March 6, 2024, 3:03pm

It looks like I’m unable to generate %matplotlib widget plots with 105b somehow. However, if I update ipympl to latest and I restart the kernel, then it looks like everything is working as espected. Therefore, I think nxcals LCG 105a would be solving my issue.

etejedor · March 6, 2024, 3:18pm

Ok, good to hear. You can also request the SPI librarians (https://its.cern.ch/jira/projects/SPI/issues) to update ipympl in the LCG releases. That won’t make it into 105a, but it will be there from 106 on (and therefore in the next NXCALS stack, whenever it’s created).