refactor: replace multiprocessing.Pool with ThreadPoolExecutor in process_single_position

[aofei-liu]

Summary

• Replace multiprocessing.Pool (spawn context) with concurrent.futures.ThreadPoolExecutor in process_single_position
• Remove multiprocessing import, add os and ThreadPoolExecutor

Motivation

Transform functions are I/O-bound, not GIL-bound

All downstream callers in biahub pass transform functions that release the GIL:

Caller	func passed	GIL-bound?	Details
deskew.py	_czyx_deskew_data	No	PyTorch/MONAI + scipy
deconvolve.py	deconvolve	No	PyTorch FFT
concatenate.py	copy_n_paste	No	numpy slicing
segment.py	segment_data	No	Cellpose (PyTorch)
process_data.py	binning_czyx	Negligible	Small Python loop over channels, heavy work is numpy
stabilize.py	apply_stabilization_transform	No	ANTsPy (C++)
register.py	apply_affine_transform	No	ANTsPy or scipy.ndimage

Since the actual computation happens in C extensions (numpy, scipy, ANTsPy) or PyTorch, threads achieve the same parallelism as processes here.

spawn-based multiprocessing has significant overhead

The previous implementation used mp.get_context("spawn") to work around tensorstore#61. spawn starts a fresh Python interpreter per worker, which means:

• Re-importing all libraries (PyTorch, tensorstore, etc.) in each child process
• Serializing/pickling func and all arguments across process boundaries
• Higher memory usage from duplicated process state

Threads share the parent process memory and avoid all of this overhead. The tensorstore fork-safety issue is also irrelevant with threads since there is no fork.

Better compatibility with Nextflow orchestration

In production, biahub runs process_single_position inside Nextflow tasks. Nextflow already parallelizes at the position level, so spawning child processes within each task:

• Creates resource accounting issues — Nextflow allocates CPUs/memory for the task, but spawned child processes compete outside Nextflow's awareness
• Wastes resources on redundant library imports in each child process

With ThreadPoolExecutor, the Nextflow task is a single process with threads — cpus and memory directives map cleanly to actual resource usage.

Test plan

• Existing test_process_single_position hypothesis tests pass
• Verify in a biahub Nextflow pipeline run (e.g. deskew or deconvolution)

[Copilot]

Pull request overview

Refactors process_single_position to use a ThreadPoolExecutor instead of multiprocessing.Pool, aiming to reduce spawn/pickling overhead and improve compatibility when running under external orchestrators (e.g., Nextflow).

Changes:

• Replaced multiprocessing.Pool usage with concurrent.futures.ThreadPoolExecutor for per-(time, channel) parallelism.
• Swapped multiprocessing import for os + ThreadPoolExecutor.
• Updated logging messages to reflect thread-pool usage.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[srivarra]

Just 2 small changes.

[srivarra]

Looks good, the only thing is if we can wrap this new test in the previous one together. maybe set a new variable in @given for thread_count or something like that.

[srivarra]

Looks great! thank you!

[ieivanov]

This is a big change with a lot of downstream consequences.

@srivarra please always get review from team members running reconstructions before merging structural changes to iohub. So far, using the ThreadPoolExecutor I see proportional drop in performance for deskew and OOM errors. I'll look a bit close to make sure I'm not doing something stupid, and then I'll ask you to debug for a solution that works for both biahub and nextflow.

[edyoshikun]

this will fail in the BRUNO. It will try to call the node's num of CPUs~128 depending on the node.

[edyoshikun]

see https://github.com/mehta-lab/VisCy/blob/0c5b83434f2740bc12fec8d8221fcf46c7634b72/packages/viscy-data/src/viscy_data/triplet.py#L45