Skip to content

Refactored health check logic for MultiDBClient#3994

Merged
vladvildanov merged 34 commits intomasterfrom
vv-multidb-healthcheck-refactor
Mar 19, 2026
Merged

Refactored health check logic for MultiDBClient#3994
vladvildanov merged 34 commits intomasterfrom
vv-multidb-healthcheck-refactor

Conversation

@vladvildanov
Copy link
Copy Markdown
Collaborator

@vladvildanov vladvildanov commented Mar 9, 2026

Description of change

Refactoring this PR brings:

  1. Configuration on Health check level. Currently, the configuration for health check probes, delay, timeout etc. is done on the level of Health check policy which eliminates configuration of each specific health check.
  2. Add health check timeout configuration, to control how long the health check operation may take (including all probes).
  3. Concurrent health check execution. Currently, health checks per database are executed sequentially, whereas the idea is to refactor it to make them concurrently executable and independent of each other.
  4. Timeout exception handling refactor. Currently, if health check exceeds timeout we propagate TimeoutError
    to the end user, but it makes more sense to continue operating and mark the specific database as unhealthy instead.
  5. Only async health checks are used across both clients, to have an alignment for health check logic.
  6. Health checks now use dedicated connections instead of using shared pool

Pull Request check-list

Please make sure to review and check all of these items:

  • Do tests and lints pass with this change?
  • Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
  • Is the new or changed code fully tested?
  • Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
  • Is there an example added to the examples folder (if applicable)?

Note

Medium Risk
Refactors core MultiDBClient health-check execution (sync + asyncio), including new timeout/concurrency behavior and dedicated connection management; regressions could affect failover/circuit state transitions under load. Changes are well-covered by updated/added tests but touch critical availability logic.

Overview
MultiDBClient health checks are reworked to be async-only across both sync and asyncio clients, with per-health-check configuration (health_check_probes, health_check_delay, new health_check_timeout) moved onto AbstractHealthCheck and propagated via config defaults.

Health checks now run concurrently (across checks and databases) with per-check asyncio.wait_for timeouts; timeouts/exceptions are treated as an unhealthy database (UnhealthyDatabaseException) rather than bubbling interval timeouts to callers. Policies now manage dedicated cached health-check clients per database and expose close(); both clients ensure these pools are closed (MultiDBClient.close() / async client aclose()), and the sync client uses an enhanced BackgroundScheduler with a shared background event loop to run async health checks safely.

Docs and tests are updated to reflect the new async custom health-check interface/signature, new pytest marker (no_mock_connections), and expanded coverage for timeout behavior, concurrency, and client lifecycle.

Written by Cursor Bugbot for commit acde146. This will update automatically on new commits. Configure here.

@vladvildanov vladvildanov requested a review from Copilot March 9, 2026 09:00
@vladvildanov vladvildanov added async experimental techdebt Things that can be improved or refactored labels Mar 9, 2026
@jit-ci
Copy link
Copy Markdown

jit-ci Bot commented Mar 9, 2026

🛡️ Jit Security Scan Results

CRITICAL HIGH MEDIUM

✅ No security findings were detected in this PR


Security scan by Jit

Comment thread redis/multidb/healthcheck.py
Comment thread redis/multidb/healthcheck.py Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors MultiDBClient health checking to support per-health-check configuration (probes/delay/timeout), adds a configurable health-check timeout, and updates execution to run health checks concurrently, with updated tests and docs to match the new behavior.

Changes:

  • Move probes/delay/timeout configuration onto individual HealthCheck implementations (via AbstractHealthCheck), and wire config defaults into PingHealthCheck.
  • Add DEFAULT_HEALTH_CHECK_TIMEOUT / health_check_timeout and enforce timeouts during health check execution.
  • Update sync + asyncio clients/tests/docs to reflect concurrent health check execution and new configuration surface.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
redis/multidb/healthcheck.py Refactors health check interfaces/policies, adds per-check timeout and concurrent execution (threads).
redis/multidb/config.py Adds health_check_timeout to config and passes health-check params into default PingHealthCheck.
redis/multidb/client.py Updates policy instantiation and adjusts background health-check round execution handling.
redis/asyncio/multidb/healthcheck.py Async equivalent of per-check config + timeout and concurrent execution (tasks).
redis/asyncio/multidb/config.py Adds health_check_timeout and passes params into default async PingHealthCheck.
redis/asyncio/multidb/client.py Updates policy instantiation and changes background health-check gathering behavior.
tests/test_multidb/conftest.py Updates health-check fixture to include probes/delay/timeout properties.
tests/test_multidb/test_healthcheck.py Updates unit tests for new policy constructors, parallel behavior, and timeout cases.
tests/test_multidb/test_client.py Adjusts expectations around health-check call counts and timing to reflect concurrency.
tests/test_multidb/test_pipeline.py Updates pipeline tests for concurrent/background health-check behavior.
tests/test_asyncio/test_multidb/conftest.py Async fixture updates for probes/delay/timeout properties.
tests/test_asyncio/test_multidb/test_healthcheck.py Async unit tests updated for new constructors/parallelism/timeout behavior.
tests/test_asyncio/test_multidb/test_client.py Async client tests updated for concurrent/background health-check behavior and timing.
tests/test_asyncio/test_multidb/test_pipeline.py Async pipeline tests updated to match concurrent background health checks.
docs/geographic_failover.rst Documents new health_check_timeout and updated custom health check example.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/test_asyncio/test_multidb/test_healthcheck.py Outdated
Comment thread docs/geographic_failover.rst Outdated
Comment thread redis/multidb/healthcheck.py Outdated
Comment thread redis/multidb/healthcheck.py Outdated
Comment thread redis/multidb/client.py Outdated
Comment thread redis/asyncio/multidb/client.py
Comment thread tests/test_multidb/test_healthcheck.py Outdated
vladvildanov and others added 2 commits March 9, 2026 11:16
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Comment thread tests/test_multidb/test_healthcheck.py Outdated
@vladvildanov vladvildanov requested a review from uglide March 9, 2026 10:07
Comment thread redis/multidb/config.py
Comment thread redis/multidb/healthcheck.py Outdated
Comment thread redis/multidb/healthcheck.py Outdated
Comment thread redis/multidb/healthcheck.py Outdated
Comment thread redis/multidb/healthcheck.py Outdated
Comment thread redis/multidb/healthcheck.py Outdated
Copy link
Copy Markdown
Collaborator

@petyaslavova petyaslavova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread redis/asyncio/multidb/healthcheck.py
Comment thread redis/multidb/client.py Outdated
Comment thread redis/multidb/client.py Outdated
Comment thread redis/multidb/client.py Outdated
Comment thread redis/multidb/client.py
Comment thread redis/asyncio/multidb/healthcheck.py
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Comment thread redis/background.py Outdated
Comment thread redis/multidb/client.py Outdated
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Comment thread redis/background.py Outdated
Comment thread redis/background.py Outdated
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Comment thread redis/asyncio/multidb/healthcheck.py Outdated
Comment thread redis/multidb/client.py
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Comment thread redis/background.py Outdated
Comment thread redis/background.py Outdated
Comment thread redis/multidb/client.py
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Comment thread redis/background.py
Comment thread redis/asyncio/multidb/client.py Outdated
Comment thread redis/asyncio/multidb/healthcheck.py Outdated
Comment thread redis/asyncio/multidb/healthcheck.py Outdated
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Comment thread redis/asyncio/multidb/client.py
Comment thread redis/asyncio/multidb/healthcheck.py Outdated
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Comment thread redis/asyncio/multidb/client.py
Comment thread redis/multidb/client.py
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Comment thread redis/asyncio/multidb/client.py
Comment thread redis/asyncio/multidb/healthcheck.py
Copy link
Copy Markdown
Collaborator

@petyaslavova petyaslavova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@vladvildanov vladvildanov merged commit f00b9db into master Mar 19, 2026
64 checks passed
@vladvildanov vladvildanov deleted the vv-multidb-healthcheck-refactor branch March 19, 2026 14:40
petyaslavova added a commit that referenced this pull request Mar 23, 2026
* Refactored sync health check

* Refactored async multidb health check

* Removed double exception handling

* Update docs/geographic_failover.rst

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/test_asyncio/test_multidb/test_healthcheck.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/test_multidb/test_healthcheck.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Refactored health checks to be completion-based

* Codestlye fixes

* Fixed argument name

* Fixed redundant timeout

* Refactored health check to be async-only

* Refactored event loop execution in sync client

* Fixed proper handling of failures and updated docs

* Fixed correct initial health check execution

* Fixed event loop issue

* Fixed flacky test

* Fixed coroutine execution

* Fixed flacky test

* Fixed correct connection cleanup

* Fixed 3.12 compatibility issue

* Fixed race condition

* Updated exception handling

* Fix avoiding indefinite hangs

* Refactor health checks to use clients instead of pools

* Removed unused if statement

* Updated to broaded exception

* Fixed passing not supported arguments

* Updated docs, added new properties to health check cluster client

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: petyaslavova <petya.slavova@redis.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

async experimental techdebt Things that can be improved or refactored

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants