The Operator may stop at the step of leave-joint-state

[HunDunDM]

Bug Report

What did you do?

• Start a cluster with 4 tikv
• Kill tikv-2, and wait for it to become Down

What did you expect to see?

replace-down-peer will be done soon.

What did you see instead?

Some Operators stay at the leave-joint-state step until timeout.

What version of PD are you using (pd-server -V)?

v6.4.0

Note

Region

» region 10
{
  "id": 10,
  "start_key": "748000FFFFFFFFFFFFFC00000000000000F8",
  "end_key": "748000FFFFFFFFFFFFFD00000000000000F8",
  "epoch": {
    "conf_ver": 10,
    "version": 4
  },
  "peers": [
    {
      "id": 11,
      "store_id": 1,
      "role_name": "Voter"
    },
    {
      "id": 18,
      "store_id": 4,
      "role": 1,
      "role_name": "Learner",
      "is_learner": true
    },
    {
      "id": 23,
      "store_id": 5,
      "role_name": "Voter"
    },
    {
      "id": 204,
      "store_id": 6,
      "role_name": "Voter"
    }
  ],
  "leader": {
    "id": 11,
    "store_id": 1,
    "role_name": "Voter"
  },
  "down_peers": [
    {
      "down_seconds": 609,
      "peer": {
        "id": 18,
        "store_id": 4,
        "role": 1,
        "role_name": "Learner",
        "is_learner": true
      }
    }
  ],
  "pending_peers": [
    {
      "id": 18,
      "store_id": 4,
      "role": 1,
      "role_name": "Learner",
      "is_learner": true
    }
  ],
  "written_bytes": 0,
  "read_bytes": 0,
  "written_keys": 0,
  "read_keys": 0,
  "approximate_size": 1,
  "approximate_keys": 0
}

log

[2022/12/15 18:26:28.439 +08:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=10] [step="leave joint state, promote learner peer 204 on store 6 to voter, demote voter peer 18 on store 4 to learner"] [source="active push"]