Unify algorithms for on-the-fly analysis and hardware control in a repo to facilitate `smart microscopy`

[edyoshikun]

Currently, the different teams at Biohub have created multiple tools and image-based hardware tunning to do the same task. This creates code duplication and inefficient project development as there is no need to completely reinvent the wheel. One suggestion that we proposed during our last copylot meeting was to find a repo/place where we can unify these pieces of code that can be easily discovered (i.e autofocus, autotracking, autoexposure, centroid detection, plate map position generator, etc).

These algorithms should be simple, perform a single task or functionality, and break down complex and repetitive tasks such that we can integrate these easily into other projects. For example, one algorithm is finding the focused slice given a 3D volume. and dragonfly's plate interpolation here which can take a 3D array and return a suggestion or take a list of positions and return a plane estimate respectively. In compmicro, we would like to use algorithms like autoexposure and autofocus by integrating this through the pycromanger hook functions, which basically allow on-the-fly processing and tunning of the microscope hardware.

A proposed solution:

• Unify these algorithms in a common repo (i.e copylot/algorithms) used among most groups in the Biohub. This will hopefully surface these pieces of code that are unknown to other groups and reduce code duplication. Additionally, keeping them in one repo facilitates access, code-review and adds momentum for other people to contribute.

This should also be discussed on the SIGs to see where do we see the best fit.

What are your thoughts?
@i-jey @talonchandler @ieivanov @rachel-banks @ziw-liu @AhmetCanSolak @YangLiujames @keithchev @JoOkuma @gregcourville

Please tag anyone else who might find this suggestion useful.

[ziw-liu]

I'm all for reusing the infra whenever possible so we don't do the packaging job again if we don't have to. One problem I can think of is dependency management when accumulating scientific/CV python libraries (especially those with compiled extensions like CuPy and OpenCV). This can be mitigated with cautious planning and optional dependencies though.

[ieivanov]

I am also supportive of this idea. It will be helpful for everyone to have these algorithms in a common repo with a common structure, such that they can be easily reused.

I think we'll need to define which algorithms belong here and which don't. I think we should strive for these algorithms to be independent of the underlying microscope acquisition engine and control software. For example, autofocus algorithms which take a series of images and return the correct focal plane would be a good fit here. This algorithm will be independent of how the images are acquired and will not be responsible for moving the z stage to the correct focal plane - this will be done by the microscope control software based on the output of the autofocus algorithm.

Let's explicitly list out algorithms that may fit here, with their input and output, such that we can as a group decide what belongs and what doesn't. Based on the discussion so far:

• image-based autofocus algorithms
  • input: image stack at different focal planes
  • output: index of the image at the best focal plane
• autoexposure algorithms
  • input and output as above
• single-shot and iterative autofocus and autoexposure algorithms
  • input: image and associated metadata (z position or exposure time)
  • output: instructions on what to try next (i.e. new z plane or exposure time settings); user can repeat the algorithm or quit if the output value (say z plane) is close enough to the starting one
• autotracking
  • same idea as autofocus but applied in xy(z)
• FOV selection
  • input: set of images
  • output: score for each image, may be boolean (yes/no) or float (rank best to worst)

Algorithms I'm less confident that they belong here:

• sample (e.g. plate) position interpolation - probably?
  • input: set of known points and set of query points
  • output: interpolation at query points. this seems to be pretty straightforward interpolation; the useful bit on the dragonfly microscope is the interaction between the user and the algorithm, which may be hard to replicate entirely in copylot
• hardware-based autofocus - probably not?
  • I think it would strongly depend on the underlying hardware
• centroid finding - not sure?
  • is this just a helper function to autotracking? how is it better / different than algorithms in opencv?

Anything other algorithms you think may or may not belong here? We're looking for your input. If we house these algorithms in copylot they would nicely integrate with image acquisition and hardware control through copylot. However, they would also nicely integrate with micro-manager / pycromanager - controlled microscopes if we write them to be independent of the acquisition and hardware control software.

[rachel-banks]

I am also in support of this idea, and agree with what Ivan has listed above.

Sample position interpolation I think would be applicable here.

Centroid finding is just a helper function for autotracking. We have a few helper functions written that are different methods of performing the tracking. I'm not familiar with the opencv version, @ieivanov do you know if it can do 3D? Or only 2D?

[JoOkuma]

I really like the idea.

I think packing these algorithms into a library, and a single conda package that includes our other libraries (e.g. iohub) would be very useful because installing python packages is very complicated, especially on Windows.

IMO, the functions should have a somewhat standardized API and they could be implemented such that it works both with cupy and numpy without depending on cupy.

How often is opencv used?
From my experience is worth the effort to reimplement the opencv functions in python than deal with its installation, especially when using other qt-based packages (e.g. napari).

[AhmetCanSolak]

Thank you all for your input here and in offline conversations. I like that there is a general agreement on such collection of algorithm implementations. Here, I would like to summarize the main points everyone seems to agree on so far (if not, please comment below):

• We want to implement these algorithms agnostic to any acquisition engine and/or hardware, @AhmetCanSolak @keithchev @ieivanov
• Implementations should accept ArrayLikes and other required generic-typed parameters when needed, and should return ArrayLike, @ieivanov , @ziw-liu , @AhmetCanSolak , @JoOkuma
• coPylot packaging won't be compromised by introducing new default dependencies with compiled extensions (open to have such dependencies(cupy, etc.) as optional) @ziw-liu , @AhmetCanSolak , @JoOkuma
• List of initially desired algorithms: @ieivanov , @rachel-banks , @edyoshikun
• • image-based autofocus algorithms
• • autoexposure algorithms
• • single-shot and iterative autofocus and autoexposure algorithms
• • autotracking
• • FOV selection
• • sample (e.g. plate) position interpolation

I am curious about how often OpenCV is used, too. I have similar experience to @JoOkuma 's experience.

[ieivanov]

It's exciting that we all agree on this.

I mentioned OpenCV as an example of standard image analysis pipeline which some of the algorithms here may depend on. I'll let @talonchandler @edyoshikun and @rachel-banks say what dependencies the algorithms they've written may bring along to copylot

[talonchandler]

I'm in favor! I think it's a good decision to make these copylot/algorithms "arrays in and arrays out" so that they can be used by different hardware and acquisition software.

No major dependencies on my end beyond numpy. I support avoiding an opencv dependency if possible.

[ziw-liu]

I am curious about how often OpenCV is used, too. I have similar experience to @JoOkuma 's experience.

I used OpenCV as an example not because I am in favor of including it, but because I know it will cause problems and should be avoided if possible among other potentially problematic dependencies.

[rachel-banks]

We use opencv only with the template matching method for autotracking, because it is much faster than the skimage version. If someone, (@JoOkuma maybe?) has a suggestion for another alternative, I'm happy to avoid it.

skimage is I think the only other major dependency for autotracking.

[JoOkuma]

@rachel-banks I don't know any alternatives. From a quick search, it seems they use the same algorithm (depending on what mode you're using on OpenCV).
I would try speeding skimage implementation, by processing directly in 3D (OpenCV is 2D only) and if gpu is available it could use cucim (skimage with cupy).

[mattersoflight]

This is exciting! I think the algorithms summarized above cover a lot of "decisions" that need to be made during high throughput imaging, imaging developing embryos, and maintaining focus over long time series.

Another set of algorithms we could unify here is registration algorithms. The methods to compute registration (e.g., phase correlation, template matching) are currently used across microscopes and can benefit from collaborative algorithmic/numerical optimizations. I think that our near term needs are:

• register volumes across time - autotracker.
• register volumes across channels - multiple microscopes (dragonfly, falcon, hummingbird, mantis).

[rafa-gomez-s]

I strongly vote to make a separate repo/module to collect generic analysis algorithms. We should NOT mix generic analysis stuff with hardware control, unless a particular part of the hardware control code requires some very specific analysis.

[mattersoflight]

@rafa-gomez-s and I were talking about this github issue. I agree with his vote that a small analysis library will be easier to build, manage, and reuse across different acquisition repos.