✨ Redesign logging: routing options, advanced flag, log level audit

[agoscinski]

This was produced by an agentic workflow from the design doc below. Please note that the design doc is not anymore up to date with the current commits. I further added a commit 03f2d5b to handle logging during a Python finalization. In this case certain functionalities of the logger are not available like styling so a reduced form of logging is taking place. Also a client logfile has been added in 01b3f81 (when interacting with aiida through python or verdi). When dogfooding we realized that the client logfile also contains the verdi status outputs. We might not want to include it. I am not sure exactly how to exclude it but lets see if the logfiles become to big because of this.

---

Context

AiiDA's current logging exposes 11 individual logging.*_loglevel options to all users. Most users don't need this granularity. Terminal output and daemon log output are fully coupled — there is no way to write more detail to the daemon log file while keeping the terminal clean. The REPORT level (workflow progress via self.report()) is stored in the DB via DBLogHandler; this is correct but not well communicated to workflow developers.

Goals:

1. Simplify config UX: show only 2 routing options by default, hide the rest behind --advanced
2. Add independent routing filters: terminal channel vs daemon-file channel
3. Promote 3 CalcJob INFO messages to WARNING so they appear in verdi process report by default
4. Audit log levels: demote verbose INFO calls to DEBUG across the engine and schedulers

---

Design

Two separate concerns, clearly separated:

Source control (what messages are generated) → existing advanced options
logging.aiida_loglevel, logging.plumpy_loglevel, etc.
These gate what messages exist. Unchanged semantics, just hidden from default view.

Routing control (where messages go) → two new simple options
logging.terminal_loglevel — additional handler-level filter for all terminal output
logging.system_loglevel — additional handler-level filter for daemon log file output

These are handler-level filters, not source-level. If aiida_loglevel=REPORT (default) and system_loglevel=INFO, no extra INFO messages appear in the daemon log because the logger already blocked them. An advanced user who wants INFO in the daemon log sets BOTH aiida_loglevel=INFO (source) AND system_loglevel=INFO (routing), which cleanly routes INFO to file while terminal_loglevel=REPORT keeps the terminal uncluttered.

logging.terminal_loglevel is the persistent equivalent of verdi --verbosity. The -v flag remains a per-command override.

---

Scenario Trace (verified)

Scenario	Source level	Handler	Result
Python script default	aiida_loglevel=REPORT	console at terminal_loglevel=REPORT	REPORT+ shown ✓
verdi process list	aiida_loglevel=REPORT	cli at terminal_loglevel=REPORT	REPORT+ shown ✓
verdi -v debug process list	aiida/verdi overridden to DEBUG; cli handler overridden to DEBUG	DEBUG+ shown ✓
Daemon (default)	aiida_loglevel=REPORT	daemon_log_file at system_loglevel=REPORT	REPORT+ to file ✓
Daemon (advanced: aiida=INFO, system=INFO)	aiida_loglevel=INFO	daemon_log_file at system_loglevel=INFO	INFO+ to file ✓
Verdi same advanced setup	aiida_loglevel=INFO	cli at terminal_loglevel=REPORT	REPORT+ shown, INFO blocked ✓

Prototype validity confirmed (PoC run 2026-04-03):

• evaluate_logging_configuration() recurses into all Mapping objects including handler configs, so lambdas inside handler dicts ARE evaluated. ✓ (verified by running the exact function with a test config — lambda inside handlers.console.level was called and resolved to 'REPORT')
• Python's dictConfig supports a level key on handlers (calls handler.setLevel()). ✓ (verified: logger set to DEBUG, handler set to WARNING via dictConfig — INFO messages blocked, WARNING messages passed)
• Pydantic 2.12.5 Field(json_schema_extra={'advanced': True}) propagates advanced: True into model_json_schema()['properties'][field_name]. ✓ (verified: the key appears in the schema dict and is absent for fields without it)
• Import order in daemon block: get_config_option is imported inside the with_orm block (line 236 of log.py), which runs after the daemon handler is constructed (line 198). Phase 2b must add its own from aiida.manage.configuration import get_config_option at the top of the daemon block — the plan already shows this. ✓

---

Phase 1: New Config Options + Advanced Flag

1a. Add two new fields to ProfileOptionsSchema

File: src/aiida/manage/configuration/config.py

Add before the existing logging fields:

logging__terminal_loglevel: LogLevels = Field(
    'REPORT',
    description=(
        'Minimum log level needed for outputting a log into the terminal'
    ),
)
logging__system_loglevel: LogLevels = Field(
    'REPORT',
    description='Minimum log level written to the daemon log files.',
)

Both default to REPORT → no behavior change for existing installations.

1b. Mark all 11 existing individual logging options as advanced

File: src/aiida/manage/configuration/config.py

Add json_schema_extra={'advanced': True} to each of the 11 logging__*_loglevel fields:

logging__aiida_loglevel: LogLevels = Field(
    'REPORT',
    description='Minimum level to log to daemon log and the `DbLog` table for the `aiida` logger.',
    json_schema_extra={'advanced': True},
)
# Same for: verdi, disk_objectstore, db, plumpy, kiwipy, paramiko,
#           alembic, sqlalchemy, circus, aiopika

Also update the description of logging__verdi_loglevel: it currently says 'Minimum level to log to console when running a verdi command', which implies it controls terminal output. After this change, terminal routing is controlled by terminal_loglevel; verdi_loglevel is a source filter only. Change to: 'Minimum source log level for the verdi logger.'

Note: logging__circus_loglevel is a pre-existing dead option — no circus logger is configured in get_logging_config(). Mark it advanced like the others; do not wire it up (out of scope).

1c. Add advanced property to Option

File: src/aiida/manage/configuration/options.py — add after the global_only property (line 53):

@property
def advanced(self) -> bool:
    """Return True if this option should be hidden from default ``verdi config list`` output."""
    return self._schema.get('advanced', False)

_schema is the dict from GlobalOptionsSchema.model_json_schema()['properties'][option_name] (line 103 in options.py). Pydantic propagates json_schema_extra fields into that dict, so self._schema.get('advanced', False) returns True for any field marked with json_schema_extra={'advanced': True}.

1d. Add --advanced flag to verdi config list

File: src/aiida/cmdline/commands/cmd_config.py

verdi_config_list is defined at line 35. Three changes:

1. Add a click option decorator before the existing @click.pass_context at line 34:

   @click.option('--advanced', is_flag=True, help='Show all options including advanced per-logger settings.')

   And add advanced: bool to the function signature.

2. Add option.advanced and not advanced to both list comprehensions (lines 60–64 and 67–71) that build the table. option_values is a dict of {name: (Option, source, value)}, so add:

   if name.startswith(prefix) and not (c.advanced and not advanced)

3. After echo.echo(tabulate(...)) at line 76, print a hint when advanced options are hidden:

   if not advanced:
       echo.echo_info('Use `verdi config list --advanced` to show all per-logger options.')

Tests: tests/cmdline/commands/test_config.py — any test that asserts logging options appear in the default verdi config list output will fail after this change. Update those assertions to either pass --advanced or check for the footer hint instead.

---

Phase 2: Routing Filters

File: src/aiida/common/log.py

Background: two terminal handlers, one daemon handler

AiiDA uses three output destinations, each with its own Python logging handler:

Handler name	Handler class (in log.py line 88–94)	Active when
console	logging.StreamHandler → stdout	Python script imports aiida directly (not via verdi)
cli	aiida.cmdline.utils.log.CliHandler → styled stdout	verdi command is running (CLI_ACTIVE is True)
daemon_log_file	logging.handlers.RotatingFileHandler → log file	AiiDA daemon is running

console and cli are not two loggers — they are two handlers for the same loggers. configure_logging() lines 218–223 switch every logger from console to cli when a verdi command is active.

Currently none of these handlers has a level set, so they pass every record through without filtering. The routing feature adds handler-level filtering so the destination can be independently controlled.

2a. (NEW FEATURE) Add terminal_loglevel as a handler-level filter on console and cli

In get_logging_config() (src/aiida/common/log.py lines 88–94), both terminal handlers currently have no level key. Add one to each:

'handlers': {
    'console': {
        'class': 'logging.StreamHandler',
        'formatter': 'halfverbose',
        'level': lambda: get_config_option('logging.terminal_loglevel'),  # ← add
    },
    'cli': {
        'class': 'aiida.cmdline.utils.log.CliHandler',
        'formatter': 'cli',
        'level': lambda: get_config_option('logging.terminal_loglevel'),  # ← add
    },
},

The lambda is evaluated by evaluate_logging_configuration() (src/aiida/common/log.py lines 149–166), which recursively evaluates all lambdas in the config dict — including inside the handlers sub-dict. The resolved string (e.g. 'REPORT') is then passed to dictConfig, which calls handler.setLevel('REPORT').

2b. (NEW FEATURE) Add system_loglevel as a handler-level filter on the daemon log handler

In configure_logging() (src/aiida/common/log.py lines 197–206), the daemon file handler is constructed directly in Python code (not inside the config dict), so we call get_config_option directly rather than using a lambda. Line 199 currently hardcodes 'level': 'DEBUG'; replace it:

Behavior change: The daemon file handler previously accepted everything (DEBUG+). After this change it defaults to REPORT. In practice the source-level default was already REPORT, so no messages are lost for default configurations. However, advanced users who had lowered only aiida_loglevel (e.g. to INFO) to see more detail in daemon logs will now also need to set system_loglevel=INFO. This should be noted in the commit message and changelog.

# add this import at the top of the daemon block (same pattern as the with_orm block at line 236):
from aiida.manage.configuration import get_config_option

config['handlers'][daemon_handler_name] = {
    'level': get_config_option('logging.system_loglevel'),  # ← was hardcoded 'DEBUG'
    'formatter': 'halfverbose',
    'class': 'logging.handlers.RotatingFileHandler',
    'filename': daemon_log_file,
    'encoding': 'utf8',
    'maxBytes': 10000000,
    'backupCount': 10,
}

2c. (BUG FIX required by 2a) Make --verbosity also override the cli handler level

verdi --verbosity DEBUG process list sets CLI_LOG_LEVEL = 'DEBUG'. The existing block at src/aiida/common/log.py lines 229–232 lowers the source logger levels for aiida, verdi, and disk_objectstore to DEBUG — so those loggers now emit DEBUG records. But after Phase 2a, the cli handler has terminal_loglevel (default 'REPORT') set on it, which will now block those DEBUG records before they reach the terminal.

Fix: add one line to the existing CLI_LOG_LEVEL block to also lower the cli handler level:

if CLI_LOG_LEVEL is not None:
    for name, logger in config['loggers'].items():
        if name in ['aiida', 'verdi', 'disk_objectstore']:
            logger['level'] = CLI_LOG_LEVEL
    if 'cli' in config['handlers']:
        config['handlers']['cli']['level'] = CLI_LOG_LEVEL  # ← add: match handler to logger

No other changes to configure_logging().

---

Phase 3: Promote CalcJob Diagnostic INFO → WARNING

File: src/aiida/engine/processes/calcjobs/calcjob.py

These 3 messages go through the CalcJob node logger (with dbnode_id). Currently INFO so they miss the DB at db_loglevel=REPORT default and don't appear in verdi process report. Promoting to WARNING fixes this:

• Line 854: 'could not parse scheduler output: the detailed_job_info attribute is missing' → WARNING
• Line 856: 'could not parse scheduler output: return value of detailed_job_info is non-zero' → WARNING
• Line 884: f'{scheduler.class.name} does not implement scheduler output parsing' → keep INFO (feature notice, not a problem)

db_loglevel stays REPORT default, stays advanced. The threshold is correct: REPORT (23) ≥ captures REPORT + WARNING + ERROR.

---

Phase 4: Log Level Audit — INFO → DEBUG

Demote all verbose operational INFO calls to DEBUG. These messages are internal machinery detail (transport steps, scheduler interactions, state transitions) that users don't need to see by default.

Before changing any call: grep for the message string in the file to locate the current line. Do not rely on memorised line numbers — they may have shifted. Read the surrounding context to confirm the call is routine (not an error, warning, or significant operational event) before demoting.

Move to DEBUG

Grep for logger.info( in each file and demote all calls that describe routine internal steps — not errors, warnings, or significant operational events:

• src/aiida/engine/processes/calcjobs/tasks.py (~20 calls): all transport task step messages (upload, submit, update, retrieve, stash, kill progress messages)
• src/aiida/engine/processes/calcjobs/manager.py: 'waiting for transport', 'successfully retrieved status of active jobs'
• src/aiida/engine/processes/calcjobs/monitors.py: 'Monitor returned: ...'
• src/aiida/engine/processes/calcjobs/calcjob.py: scheduler timing info notice
• src/aiida/engine/processes/process.py: 'Loaded process from saved state', 'no controller/connection to kill child'
• src/aiida/engine/processes/workchains/workchain.py: 'received callback that awaitable has terminated'
• src/aiida/engine/processes/workchains/restart.py: 'output was already attached, skipping'
• src/aiida/engine/runners.py: 'adding subscriber for broadcasts'
• src/aiida/schedulers/plugins/slurm.py: job submission and status poll messages
• src/aiida/schedulers/plugins/sge.py: job submission and status poll messages
• src/aiida/schedulers/plugins/pbsbaseclasses.py: job submission and status poll messages
• src/aiida/schedulers/plugins/direct.py: job submission and status poll messages
• src/aiida/schedulers/plugins/lsf.py: job submission and status poll messages
• src/aiida/brokers/rabbitmq/defaults.py: broker connection step messages
• src/aiida/parsers/plugins/diff_tutorial/parsers.py: tutorial parser step messages
• src/aiida/orm/nodes/data/code/portable.py: portable code copy step message
• src/aiida/storage/psql_dos/backend.py: one internal step message (keep maintenance operation messages as INFO)

Keep as INFO (significant operational events worth surfacing)

• src/aiida/engine/daemon/worker.py: daemon lifecycle (start/stop/signal)
• src/aiida/engine/runners.py: process confirmed terminated by polling
• src/aiida/engine/processes/process.py: kill request received, kill signal unable to reach child
• src/aiida/engine/processes/control.py: continue message received
• src/aiida/engine/processes/launcher.py: process already terminated
• src/aiida/storage/psql_dos/backend.py: maintenance operations (backup, vacuum, migrate)
• src/aiida/schedulers/plugins/direct.py: configuration notice
• src/aiida/transports/ssh_async.py: auth script success

---

Files Modified (Summary)

File	Change
src/aiida/manage/configuration/config.py	Add 2 new options; mark 11 as advanced
src/aiida/manage/configuration/options.py	Add advanced property
src/aiida/cmdline/commands/cmd_config.py	Add --advanced flag; filter options
src/aiida/common/log.py	Handler-level lambdas in config; daemon handler uses system_loglevel; --verbosity also overrides cli handler level
src/aiida/engine/processes/calcjobs/calcjob.py	2x INFO→WARNING; 1x INFO→DEBUG
src/aiida/engine/processes/calcjobs/tasks.py	INFO→DEBUG (transport task step messages)
src/aiida/engine/processes/calcjobs/manager.py	INFO→DEBUG (transport wait / job status messages)
src/aiida/engine/processes/calcjobs/monitors.py	INFO→DEBUG (monitor return message)
src/aiida/engine/processes/process.py	INFO→DEBUG (state reload / kill child messages)
src/aiida/engine/processes/workchains/workchain.py	INFO→DEBUG (awaitable terminated callback)
src/aiida/engine/processes/workchains/restart.py	INFO→DEBUG (output already attached)
src/aiida/engine/runners.py	INFO→DEBUG (broadcast subscriber message)
src/aiida/schedulers/plugins/slurm.py	INFO→DEBUG (submission / poll messages)
src/aiida/schedulers/plugins/sge.py	INFO→DEBUG (submission / poll messages)
src/aiida/schedulers/plugins/pbsbaseclasses.py	INFO→DEBUG (submission / poll messages)
src/aiida/schedulers/plugins/direct.py	INFO→DEBUG (submission / poll messages)
src/aiida/schedulers/plugins/lsf.py	INFO→DEBUG (submission / poll messages)
src/aiida/brokers/rabbitmq/defaults.py	INFO→DEBUG (broker connection step)
src/aiida/parsers/plugins/diff_tutorial/parsers.py	INFO→DEBUG (tutorial parser steps)
src/aiida/orm/nodes/data/code/portable.py	INFO→DEBUG (portable code copy step)
src/aiida/storage/psql_dos/backend.py	INFO→DEBUG (one internal step; keep maintenance ops as INFO)

---

Commit Plan

Four commits, one per phase. Each is self-contained and bisectable.
Follows project conventions: emoji prefix, imperative mood, 50/72 rule.

Commit 1 — Phase 1

✨ Add routing log options, `--advanced` flag

Add `logging.terminal_loglevel` and `logging.system_loglevel`
to `ProfileOptionsSchema` (both default REPORT, no behavior
change). Mark all 11 existing per-logger options as advanced
via `json_schema_extra`. Add `Option.advanced` property and
`--advanced` flag to `verdi config list` with a footer hint.
Update `verdi_loglevel` description to clarify it is a
source-level filter only.

Files: config.py, options.py, cmd_config.py, test_config.py

Commit 2 — Phase 2

✨ Add handler-level log routing filters

Add `terminal_loglevel` lambda to `console` and `cli` handler
dicts in `get_logging_config()`. Replace hardcoded `DEBUG` in
the daemon file handler with `system_loglevel` from config.
Extend `CLI_LOG_LEVEL` block to also override the `cli` handler
level so `verdi --verbosity` still works.

Note: the daemon file handler level changes from DEBUG to
REPORT (default). Users who had lowered `aiida_loglevel`
alone to get more detail in daemon logs will now also need
to set `system_loglevel` to match.

Files: src/aiida/common/log.py

Commit 3 — Phase 3

🐛 Promote scheduler parse failures to WARNING

Two messages about unparseable scheduler output were logged at
INFO and silently missing from `verdi process report`. At
WARNING they are captured by the default `db_loglevel=REPORT`
threshold and stored in the `DbLog` table.

Files: src/aiida/engine/processes/calcjobs/calcjob.py

Commit 4 — Phase 4

♻️ Demote verbose engine/scheduler INFO to DEBUG

Internal step messages (transport upload/submit/retrieve,
scheduler poll, broker connection steps) are machinery detail.
Demoting to DEBUG reduces noise at the default REPORT level.

Files: all Phase 4 files listed in Files Modified table

---

Verification

1. verdi config list shows exactly 2 logging options: logging.terminal_loglevel and logging.system_loglevel
2. verdi config list --advanced shows all 13 logging options
3. Footer hint appears when advanced options are hidden
4. verdi -v debug process list still works (debug messages appear in terminal)
5. Setting aiida_loglevel=INFO + system_loglevel=INFO routes INFO to daemon log; terminal still shows REPORT+
6. verdi process report <pk> shows the promoted WARNING messages for scheduler output parse failures
7. pytest tests/cmdline/commands/test_config.py (update assertions for hidden options)
8. pytest tests/common/test_logging.py

[codecov]

Codecov Report

❌ Patch coverage is 83.72093% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.18%. Comparing base (06968d4) to head (e21e584).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/aiida/schedulers/plugins/pbsbaseclasses.py	0.00%	4 Missing ⚠️
src/aiida/schedulers/plugins/sge.py	20.00%	4 Missing ⚠️
src/aiida/schedulers/plugins/slurm.py	0.00%	3 Missing ⚠️
src/aiida/cmdline/utils/log.py	90.00%	2 Missing ⚠️
src/aiida/common/log.py	90.91%	2 Missing ⚠️
src/aiida/engine/processes/process.py	33.34%	2 Missing ⚠️
src/aiida/schedulers/plugins/lsf.py	0.00%	2 Missing ⚠️
src/aiida/engine/processes/calcjobs/calcjob.py	66.67%	1 Missing ⚠️
src/aiida/engine/processes/workchains/restart.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7323      +/-   ##
==========================================
- Coverage   80.26%   80.18%   -0.07%     
==========================================
  Files         577      577              
  Lines       45497    45548      +51     
==========================================
+ Hits        36514    36519       +5     
- Misses       8983     9029      +46     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.