import into failed with error "lock held by current process" after injection pd leader network partition

[D3Hunter]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

import into failed after simulated pd leader io delay 1s

import into failed when simulated pd io delay

./tidb-server -V
Release Version: v8.5.5
Edition: Community
Git Commit Hash: 0c53c76
Git Branch: HEAD
UTC Build Time: 2025-12-25 08:52:50
GoVersion: go1.25.5
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2025-12-29T12:07:12.638+0800

operatorLogs:

[2025-11-29 22:06:15] ###### start import into
import into user_data1 from 's3://qe-testing/brie/lightning-mhy-100G-csv/sysbench.user_data1.000001*.csv?access-key=xxxxxx&secret-access-key=xxxxxx&endpoint=http%3a%2f%2fks3-cn-beijing-internal.ksyuncs.com&force-path-style=false&region=Beijing&provider=ks' WITH DETACHED,thread=8,skip_rows=1
[2025-11-29 22:06:15] ###### wait for import job to finish
[2025-11-29 22:24:46] ###### wait for import job to finish failed
select id, table_schema, table_name, step, status,parameters from mysql.tidb_import_jobs where start_time >= '2025-11-29 22:06:15' and table_schema = 'importtest' and table_name = 'user_data1'
jobId: 120002, table_schema: importtest, table_name: user_data1, step: importing, status: failed

./tidb-server -V
 Release Version: v9.0.0-beta.2.pre-834-g5bcc229
Edition: Community
Git Commit Hash: 5bcc2299e457bb44d8d7a769f7a4162b310605b6
Git Branch: HEAD
UTC Build Time: 2025-11-28 10:30:24
GoVersion: go1.23.12
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Kernel Type: Classic
2025-11-29T22:06:00.991+0800

[2025/11/29 22:24:19.012 +08:00] [INFO] [task_executor.go:508] ["run subtask failed"] [task-id=120002] [task-key=ImportInto/120002] [subtaskID=120006] [step=import] [takeTime=15.765984ms] [error="lock held by current process"] [errorVerbose="lock held by current process\n(1) attached stack trace\n  -- stack trace:\n  | github.com/cockroachdb/pebble/vfs.defaultFS.Lock\n  | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.4-0.20250120151818-5dd133a1e6fb/vfs/file_lock_unix.go:50\n  | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Lock\n  | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.4-0.20250120151818-5dd133a1e6fb/vfs/disk_health.go:679\n  | github.com/cockroachdb/pebble.LockDirectory\n  | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.4-0.20250120151818-5dd133a1e6fb/open.go:1068\n  | github.com/cockroachdb/pebble.Open\n  | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.4-0.20250120151818-5dd133a1e6fb/open.go:116\n  | github.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngineDB\n  | \t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:229\n  | github.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngine\n  | \t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:235\n  | github.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).OpenEngine\n  | \t/workspace/source/tidb/pkg/lightning/backend/local/local.go:833\n  | github.com/pingcap/tidb/pkg/lightning/backend.EngineManager.OpenEngine\n  | \t/workspace/source/tidb/pkg/lightning/backend/backend.go:270\n  | github.com/pingcap/tidb/pkg/executor/importer.(*TableImporter).OpenIndexEngine\n  | \t/workspace/source/tidb/pkg/executor/importer/table_import.go:513\n  | github.com/pingcap/tidb/pkg/disttask/importinto.(*importStepExecutor).RunSubtask\n  | \t/workspace/source/tidb/pkg/disttask/importinto/task_executor.go:237\n  | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask.func1\n  | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:504\n  | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n  | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:505\n  | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n  | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:379\n  | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).startTaskExecutor.func2\n  | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/manager.go:357\n  | github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithLog.func1\n  | \t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:181\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) lock held by current process\nError types: (1) *withstack.withStack (2) *errutil.leafError\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20250523034308-74f78ae071ee/juju_adaptor.go:15\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngineDB\n\t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:230\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngine\n\t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:235\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).OpenEngine\n\t/workspace/source/tidb/pkg/lightning/backend/local/local.go:833\ngithub.com/pingcap/tidb/pkg/lightning/backend.EngineManager.OpenEngine\n\t/workspace/source/tidb/pkg/lightning/backend/backend.go:270\ngithub.com/pingcap/tidb/pkg/executor/importer.(*TableImporter).OpenIndexEngine\n\t/workspace/source/tidb/pkg/executor/importer/table_import.go:513\ngithub.com/pingcap/tidb/pkg/disttask/importinto.(*importStepExecutor).RunSubtask\n\t/workspace/source/tidb/pkg/disttask/importinto/task_executor.go:237\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask.func1\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:504\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:505\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:379\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).startTaskExecutor.func2\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/manager.go:357\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithLog.func1\n\t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:181\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"]
[2025/11/29 22:24:19.012 +08:00] [INFO] [retry.go:80] ["meet un-retryable error"] [error="lock held by current process"] [info="type: *errors.withStack, value: lock held by current process"]
[2025/11/29 22:24:19.013 +08:00] [WARN] [task_executor.go:755] ["subtask failed"] [task-id=120002] [task-key=ImportInto/120002] [error="lock held by current process"] [errorVerbose="lock held by current process\n(1) attached stack trace\n  -- stack trace:\n  | github.com/cockroachdb/pebble/vfs.defaultFS.Lock\n  | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.4-0.20250120151818-5dd133a1e6fb/vfs/file_lock_unix.go:50\n  | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Lock\n  | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.4-0.20250120151818-5dd133a1e6fb/vfs/disk_health.go:679\n  | github.com/cockroachdb/pebble.LockDirectory\n  | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.4-0.20250120151818-5dd133a1e6fb/open.go:1068\n  | github.com/cockroachdb/pebble.Open\n  | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.4-0.20250120151818-5dd133a1e6fb/open.go:116\n  | github.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngineDB\n  | \t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:229\n  | github.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngine\n  | \t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:235\n  | github.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).OpenEngine\n  | \t/workspace/source/tidb/pkg/lightning/backend/local/local.go:833\n  | github.com/pingcap/tidb/pkg/lightning/backend.EngineManager.OpenEngine\n  | \t/workspace/source/tidb/pkg/lightning/backend/backend.go:270\n  | github.com/pingcap/tidb/pkg/executor/importer.(*TableImporter).OpenIndexEngine\n  | \t/workspace/source/tidb/pkg/executor/importer/table_import.go:513\n  | github.com/pingcap/tidb/pkg/disttask/importinto.(*importStepExecutor).RunSubtask\n  | \t/workspace/source/tidb/pkg/disttask/importinto/task_executor.go:237\n  | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask.func1\n  | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:504\n  | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n  | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:505\n  | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n  | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:379\n  | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).startTaskExecutor.func2\n  | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/manager.go:357\n  | github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithLog.func1\n  | \t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:181\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) lock held by current process\nError types: (1) *withstack.withStack (2) *errutil.leafError\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20250523034308-74f78ae071ee/juju_adaptor.go:15\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngineDB\n\t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:230\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngine\n\t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:235\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).OpenEngine\n\t/workspace/source/tidb/pkg/lightning/backend/local/local.go:833\ngithub.com/pingcap/tidb/pkg/lightning/backend.EngineManager.OpenEngine\n\t/workspace/source/tidb/pkg/lightning/backend/backend.go:270\ngithub.com/pingcap/tidb/pkg/executor/importer.(*TableImporter).OpenIndexEngine\n\t/workspace/source/tidb/pkg/executor/importer/table_import.go:513\ngithub.com/pingcap/tidb/pkg/disttask/importinto.(*importStepExecutor).RunSubtask\n\t/workspace/source/tidb/pkg/disttask/importinto/task_executor.go:237\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask.func1\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:504\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:505\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:379\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).startTaskExecutor.func2\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/manager.go:357\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithLog.func1\n\t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:181\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"]
[2025/11/29 22:24:19.022 +08:00] [ERROR] [task_executor.go:384] ["run subtask failed"] [task-id=120002] [task-key=ImportInto/120002] [error="lock held by current process"] [errorVerbose="lock held by current process\n(1) attached stack trace\n  -- stack trace:\n  | github.com/cockroachdb/pebble/vfs.defaultFS.Lock\n  | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.4-0.20250120151818-5dd133a1e6fb/vfs/file_lock_unix.go:50\n  | github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Lock\n  | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.4-0.20250120151818-5dd133a1e6fb/vfs/disk_health.go:679\n  | github.com/cockroachdb/pebble.LockDirectory\n  | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.4-0.20250120151818-5dd133a1e6fb/open.go:1068\n  | github.com/cockroachdb/pebble.Open\n  | \t/root/go/pkg/mod/github.com/cockroachdb/pebble@v1.1.4-0.20250120151818-5dd133a1e6fb/open.go:116\n  | github.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngineDB\n  | \t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:229\n  | github.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngine\n  | \t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:235\n  | github.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).OpenEngine\n  | \t/workspace/source/tidb/pkg/lightning/backend/local/local.go:833\n  | github.com/pingcap/tidb/pkg/lightning/backend.EngineManager.OpenEngine\n  | \t/workspace/source/tidb/pkg/lightning/backend/backend.go:270\n  | github.com/pingcap/tidb/pkg/executor/importer.(*TableImporter).OpenIndexEngine\n  | \t/workspace/source/tidb/pkg/executor/importer/table_import.go:513\n  | github.com/pingcap/tidb/pkg/disttask/importinto.(*importStepExecutor).RunSubtask\n  | \t/workspace/source/tidb/pkg/disttask/importinto/task_executor.go:237\n  | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask.func1\n  | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:504\n  | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n  | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:505\n  | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n  | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:379\n  | github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).startTaskExecutor.func2\n  | \t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/manager.go:357\n  | github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithLog.func1\n  | \t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:181\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (2) lock held by current process\nError types: (1) *withstack.withStack (2) *errutil.leafError\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20250523034308-74f78ae071ee/juju_adaptor.go:15\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngineDB\n\t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:230\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).openEngine\n\t/workspace/source/tidb/pkg/lightning/backend/local/engine_mgr.go:235\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).OpenEngine\n\t/workspace/source/tidb/pkg/lightning/backend/local/local.go:833\ngithub.com/pingcap/tidb/pkg/lightning/backend.EngineManager.OpenEngine\n\t/workspace/source/tidb/pkg/lightning/backend/backend.go:270\ngithub.com/pingcap/tidb/pkg/executor/importer.(*TableImporter).OpenIndexEngine\n\t/workspace/source/tidb/pkg/executor/importer/table_import.go:513\ngithub.com/pingcap/tidb/pkg/disttask/importinto.(*importStepExecutor).RunSubtask\n\t/workspace/source/tidb/pkg/disttask/importinto/task_executor.go:237\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask.func1\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:504\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:505\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:379\ngithub.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).startTaskExecutor.func2\n\t/workspace/source/tidb/pkg/disttask/framework/taskexecutor/manager.go:357\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithLog.func1\n\t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:181\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)