Harmonization of aiida.common, aiida.control, aiida.manage, aiida.tools, aiida.utils

[ltalirz]

There are currently a number of top-level folders in aiida, whose meaning is not well defined:

• aiida.common
• aiida.control
• aiida.manage
• aiida.tools
• aiida.utils

This leads to confusion among new developers, and has also led to different interpretations of previous developers. We should

1. decide what these top-level folders should contain
2. document this in the __init__.py of the corresponding folders
3. move code that is currently in the wrong place to where it belongs

[ltalirz]

@giovannipizzi @sphuber and me have discussed this issue, initially on Nov 16th 2018, and now again on Dec 6 2018.

We propose the following:

• aiida.control => aiida.manage.external : controlling system software needed for AiiDA to run (postgres, rabbitmq) [no load_dbenv required]
  • Actual content: Postgres
  • profile => aiida.manage.profile
  • code => aiida.orm.utils.builders.code
  • computer => aiida.orm.utils.builders.computer
• aiida.tools: tools to operate on the AiiDA ORM [may require load_dbenv]
  • When to put functionality in ORM classes and when to put them in tools?
    Interfaces of ORM classes should provides functionality needed to set up & access the data structure. Advanced operations on the data types that may be useful but should not clutter the namespace go into tools.
  • Current content is fine
  • To open issue: Think about how to expose this functionality in a more visible way (e.g. in a .tools, with an aiida.tools.data entrypoint, see Implement Data.tools.* to access methods defined by a plugin #2327 )
• aiida.manage: controlling the configuration of aiida (profile, database or repository): e.g. validation, migration of config.json, vacuum cleaner of repository... [may require load_dbenv]
  • Current content is fine
• aiida.common: definitions of common data structures that do not require the DB, link types, exceptions + common utilities [no load_dbenv required]
  • common.orbital => moved to aiida.tools.data.orbital
  • common.additions.config_migrations => aiida.manage.config.migrations
  • common.additions.backup_script => aiida.manage.backup
  • common.example_helpers => remove (including test)
  • common.caching => aiida.manage.caching
  • common.ipython => aiida.tools.ipython
  • common.graph => aiida.tools.graphviz
  • common.log => (partly) into aiida.orm.utils.log (SPH: some functionality needs to stay in aiida.common because needs to be loadable without backend)
  • common.profile => aiida.manage.profile
  • common.setup => aiida.manage.setup
  • common.utils
    • Classproperty, abstractclassmethod, abstractstaticmethod => common.lang
    • Xyz_parser_iterator => tools.orm.data.structures
    • EmptyContextManager => common.lang
    • Fortran-stuff => moved to aiida-quantumespresso
    • Ask_question, query_yes_no, query_string: move to backup_setup file (only place where it is used)
    • Combomethod => common.lang
    • Prettifier, pretty_labels, join_labels => remains in common.utils for the moment
    • Get_mode_string, md5, sha1... => common.files
    • HiddenPrints => merge with Capturing
    • type_check => common.lang
    • get_regex_pattern_from_sql, sql_string_matchm escape_for_bash => common.escaping
• aiida.utils: will be removed entirely (mainly in favor of common)
  • ascii_vis => aiida.cmdline.utils.ascii_vis
  • capturing => aiida.common.utils.capturing
  • cli: (sebastiaan: )remove
  • delete_nodes => aiida.manage.database.delete.nodes + issue
  • email => aiida.common.utils
    • If possible, remove validate_email dependency (rather use from django)
  • error_accumulator => common.utils
  • find_folder => aiida.common.folders
  • fixtures => aiida.manage.fixtures
  • json => aiida.common.json
  • queries => deleted, not used
  • serialize => aiida.common.serialize
  • timezone => common.timezone
  • which => common.files

[ltalirz]

In terms of how to proceed:

• we wait until provenance_redesign is merged back into develop
• releasing 1.0.0b1 before fixing this is acceptable
• ideally, this should be fixed before releasing 1.0.0

After discussion with @sphuber and @giovannipizzi , we can start now.

[sphuber]

Fixed in #2357