dlt version

1.24.0

Describe the problem

A RunnableLoadJob can publish state == "completed" before its _on_completed(...) callback has finished persisting the pending-transition marker in the package directory.

Because the loader uses a single shared _done_event for the whole package, completion of any worker wakes the main loop, which then polls all running jobs. If another job already reports "completed" but is still inside _on_completed(...), the loader may:

• remove that job from started_jobs
• clear its pending transition
• finalize the package
• rename normalized/<load_id> to loaded/<load_id>

while the worker thread is still writing files inside that same package directory.

Expected behavior

No response

Steps to reproduce

/

Operating system

macOS

Runtime environment

Local

Python version

3.13

dlt data source

No response

dlt destination

No response

Other deployment details

No response

Additional information

No response