Use of --ignore-installed in TensorFlow's step_install

[agijsberts]

TensorFlow's EasyBlock installs the produced wheel package via
pip install --ignore-installed

easybuild-easyblocks/easybuild/easyblocks/t/tensorflow.py

Line 372 in 4f81668

cmd = "pip install --ignore-installed --prefix=%s %s" % (self.installdir, whl_paths[0])

As a consequence, it pulls in binary wheels of all dependencies, including for example NumPy. So loading the TensorFlow module will replace the NumPy as part of the Python package (linked against MKL in my case) with a precompiled NumPy linked against a pre-shipped OpenBLAS. Is this intended behavior?

At least on our systems, removing --ignore-installed leads to desired behavior, i.e. pip does not install those dependencies that were already satisfied via dependent modules.

[boegel]

@agijsberts That's by no means intended, we overlooked this completely... Thanks a lot for reporting this!

The --ignore-installed was added in the original implementation of the TensorFlow easyblock to try and ensure the specified *.whl is installed, cfr. #1287 .

We probably also need to use --no-deps to avoid that pip pulls in all dependencies, there should be no need for that, everything should be in place already (and if not, that should be fixed).

cc @akesandgren

[agijsberts]

Thank you for confirming my suspicion.

Mind that the pip command does pull in tensorboard, which is obviously an important dependency. I guess you mean that tensorboard (and maybe a couple of other libraries) should be pulled in via the extensions mechanism (exts_list)?

[boegel]

@agijsberts Indeed, we'll need to fix this in the existing TensorFlow easyconfigs, but that should be just a matter of including everything that is needed via exts_list, indeed...

[boegel]

Fixed in #1537, where --no-deps is used if extensions are included in the TensorFlow easyconfig file (i.e. if exts_list is not empty).

See also updated easyconfigs for TensorFlow 1.10.x in easybuilders/easybuild-easyconfigs#6940