Parsing Cray easyconfigs fails when using Hierarchical Module Naming Scheme (HMNS)

[seb45tian]

When processing Cray specific easyconfigs in combination with an Hierarchical Module Naming Scheme parsing of the easyconfig file fails. (I actually encountered this when using the "--list-installed-software" option which in turn is parsing all easyconfigs. Not trying this on a Cray machine.)

Steps to reproduce:

eb HPL-2.3-CrayCCE-19.06.eb -D --module-naming-scheme HiearchicalMN

Traceback Error Message:

  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Applic.HPC/Easybuild/ivybridge/2019a/software/EasyBuild/4.1.2/lib/python2.7/site-packages/easybuild/main.py", line 502, in <module>
    main()
  File "/Applic.HPC/Easybuild/ivybridge/2019a/software/EasyBuild/4.1.2/lib/python2.7/site-packages/easybuild/main.py", line 350, in main
    easyconfigs, generated_ecs = parse_easyconfigs(paths, validate=not options.inject_checksums)
  File "/Applic.HPC/Easybuild/ivybridge/2019a/software/EasyBuild/4.1.2/lib/python2.7/site-packages/easybuild/framework/easyconfig/tools.py", line 413, in parse_easyconfigs
    easyconfigs.extend(process_easyconfig(ec_file, **kwargs))
  File "/Applic.HPC/Easybuild/ivybridge/2019a/software/EasyBuild/4.1.2/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1791, in process_easyconfig
    ec = EasyConfig(spec, build_specs=build_specs, validate=validate, hidden=hidden)
  File "/Applic.HPC/Easybuild/ivybridge/2019a/software/EasyBuild/4.1.2/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 496, in __init__
    self.full_mod_name = mns.det_full_module_name(self)
  File "/Applic.HPC/Easybuild/ivybridge/2019a/software/EasyBuild/4.1.2/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 2334, in det_full_module_name
    require_result=require_result)
  File "/Applic.HPC/Easybuild/ivybridge/2019a/software/EasyBuild/4.1.2/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 2306, in _det_module_name_with
    mod_name = mns_method(ec)
  File "/Applic.HPC/Easybuild/ivybridge/2019a/software/EasyBuild/4.1.2/lib/python2.7/site-packages/easybuild/tools/module_naming_scheme/hierarchical_mns.py", line 87, in det_full_module_name
    return os.path.join(self.det_module_subdir(ec), self.det_short_module_name(ec))
  File "/Applic.HPC/Easybuild/ivybridge/2019a/software/EasyBuild/4.1.2/lib/python2.7/site-packages/easybuild/tools/module_naming_scheme/hierarchical_mns.py", line 139, in det_module_subdir
    tc_comp_name, tc_comp_ver = self.det_toolchain_compilers_name_version(tc_comps)
  File "/Applic.HPC/Easybuild/ivybridge/2019a/software/EasyBuild/4.1.2/lib/python2.7/site-packages/easybuild/tools/module_naming_scheme/hierarchical_mns.py", line 110, in det_toolchain_compilers_name_version
    res = (tc_comps[0]['name'], self.det_full_version(tc_comps[0]))
TypeError: 'NoneType' object has no attribute '__getitem__'

So far I could narrow it down to this:

• det_toolchain_compilers() from tools/module_naming_scheme/toolchain.py is returning a list with a NoneType object --> [None] (instead of just None?)
• The list with None comes from the call to det_toolchain_element_details(). I don't know if this is what's wong, but there is a mismatch between tc_ec[name] ("CrayCCE") and elem ("PrgEnv-cray") --> (l. 77 in toolchain.py) Should these be the same here?

[akesandgren]

There are a sequence of problems.
EXTERNAL_MODULE that don't exist doesn't get name and version set in _parse_dependency in framework/easyconfig/easyconfig.py when the external module data doesn't exist.

If I add a patch that sets them:

        if dependency['external_module']:
            # check whether the external module is hidden
            if dependency['full_mod_name'].split('/')[-1].startswith('.'):
                dependency['hidden'] = True

            # AS:
            if dependency['name'] is None:
                dependency['name'] = dependency['short_mod_name']
                dependency['version'] = ''

it still crashes later on with:

ERROR: Failed to process easyconfig /hpc2n/eb/software/Core/EasyBuild/4.2.0/easybuild/easyconfigs/h/HPL/HPL-2.3-CrayCCE-19.06.eb: Empty list of MPI libs for {'full_mod_name': 'Core/CrayCCE/19.06', 'versionsuffix': '', 'version': '19.06', 'toolchain': {'version': '', 'name': 'system'}, 'name': 'CrayCCE', 'short_mod_name': 'CrayCCE/19.06', 'hidden': False, 'parsed': True, 'system': True} toolchain definition: {'LAPACK': [], 'SCALAPACK': [], 'BLAS': [], 'COMPILER': ['PrgEnv-cray'], 'BLACS': [], 'MPI': []}

class CrayMPICH: should maybe set MPI_MODULE_NAME = None instead of empty list.
If I do that it works.

So I'll create a PR for @boegel to look at...