Install opencv-python in SWAN

Software
1

Hi,

I recently moved from my own jupyter lab running on a VM to SWAN since large notebooks seem to run better. At the expense of not having conda environments.

Now I have to do some image processing for which I would like to use python-opencv.
However installation (pip install --user opencv-python) fails with

ERROR: Could not build wheels for opencv-python which use PEP 517 and cannot be installed directly

I tried to upgrade via pip install --user --upgrade pip setuptools wheel however is still fails.

How do we install opencv to be used in SWAN?

Thanks,
Stefan

2

Sadly your pip install --user --upgrade pip is unlikely to have any impact at all to the default pip (you could verify by running pip --version and seeing that it is still the old version). It is believable that running ~/.local/lib/python3.?/site-packages/bin/pip instead might work, but it isn’t a given due to the way Python has been configured to pickup the various site-package directories.

If you really want to be able to influence your environment properly, I’d personally recommend using virtual environments. I posted about using them reasonably effectively (though at the cost of having some ugly setup code at the beginning of each notebook) in Avoiding the use of ``--user`` and ``.local`` for pip installations. Given the sluggish home filesystem though it can take a while to first install a sizeable virtual environment (after that it is much quicker and cached in your $HOME/python/environments directory).

Note: I’m not a SWAN maintainer and there may be simpler approaches to solving this problem which don’t involve the above suggestion (I didn’t look too hard).

Cheers!

3

Hello Stefan

looks like a complicated package,
please try with this in the SWAN terminal.

export CCACHE_DISABLE=1
export CMAKE_GENERATOR="Unix Makefiles"
pip install opencv-python --user

Best
Omar.

1 Like
4

@pelson
Thanks for pointing that out.

I checked your approach already before. But I didn’t like the default location + opencv is not available from the acc-py index. Disabling it did the trick.

I now used your approach, modified it slightly to install to my preferred location and it worked fine.

Just to clarify the ‘virtual envs’ created with your helper script are not truly independent (as they still need they jupyter kernel), thus they can also access the default packages installed along with it?!
Since you insert the site package path to the virtual environment packages just before the one to the default packages, we can happily ignore incompatibility error messages like

ERROR: itkwidgets 0.25.3 has requirement ipywidgets>=7.5.1, but you’ll have ipywidgets 7.4.2 which is incompatible.

@ozapatam
Thanks, I just tried your suggestion as well and it works just fine as well.

Best,
Stefan

2 Likes
5

Merge requests welcome for adding the environment location flag to the script :wink: (currently hard-coded at run_in_venv.py · master · Philip Elson / swan-run-in-venv · GitLab, but potentially configurable as an argument like the others such as disabling the Acc-Py index).

opencv is not available from the acc-py index

That’s interesting. It is supposed to be a full proxy of the public PyPI. There is an opencv at http://acc-py-repo.cern.ch/repository/py-thirdparty-remote/packages/opencv-python/4.4.0.46/opencv_python-4.4.0.46-cp36-cp36m-manylinux2014_x86_64.whl, but we have seen issues with Nexus (the proxy) not properly invalidating caches, which could be what you are seeing here.

That error you mention looks a lot like pip telling you that you have some other packages installed in the system that aren’t compatible with the package(s) you’ve just installed, I think. I turn on system-site-packages when creating the venv to avoid duplicating what is already installed, and I don’t think pip will automatically upgrade stuff if it has already been installed, hence the message. It would be interesting to check your venv actually has ipywidgets in it to validate this - I think it shouldn’t have been installed by pip.

6

I cannot push to your repo, I think you need to give me access. In any case I just requested access.

Indeed, there is no ipywidgets in the venv.

Interestingly, in a fresh install I also had some issues, i.e. pip wasn’t able to find a matching version. After I upgraded pandas and matplotlib it worked.

7

Even though I removed --system-site-packages from the venv installer I still seem to get interference with installed packages, either in .local or from default packages (when I check sys.path .local is not there but various lcg site-packages).
That is probably why I had weird behaviour of packages installed via venv (dask, opencv in particular).

In any case I started with a fresh install and removed all venvs and user installed packages (by removing the respective folders in my CERNBOX).

Installation of opencv-python went fine with specifying the variables given above.
However there is clearly something wrong. No images are loaded (without triggering an error).
I can however open the image with e.g pillow.

I tried also tried the same opencv version installed via conda on my jupyter lab setup, there everything went fine.

Any ideas?

8

Another update…
It seems that only the interaction with the filesystem is somehow compromised, functions like imread and imwrite are not properly working (without giving an error).

For others searching for a solution:
The current workaround is to load the image via PIL and convert it to opencv format. (Remember though that opencv uses BGR to represent an image contrary to ‘standard’ RGB)