Single-shot self-supervised particle tracking on ArXiv

LodeSTAR tracks the plankton Noctiluca scintillans. (Image by the Authors of the manuscript.)
Single-shot self-supervised particle tracking
Benjamin Midtvedt, Jesús Pineda, Fredrik Skärberg, Erik Olsén, Harshith Bachimanchi, Emelie Wesén, Elin K. Esbjörner, Erik Selander, Fredrik Höök, Daniel Midtvedt, Giovanni Volpe
arXiv: 2202.13546

Particle tracking is a fundamental task in digital microscopy. Recently, machine-learning approaches have made great strides in overcoming the limitations of more classical approaches. The training of state-of-the-art machine-learning methods almost universally relies on either vast amounts of labeled experimental data or the ability to numerically simulate realistic datasets. However, the data produced by experiments are often challenging to label and cannot be easily reproduced numerically. Here, we propose a novel deep-learning method, named LodeSTAR (Low-shot deep Symmetric Tracking And Regression), that learns to tracks objects with sub-pixel accuracy from a single unlabeled experimental image. This is made possible by exploiting the inherent roto-translational symmetries of the data. We demonstrate that LodeSTAR outperforms traditional methods in terms of accuracy. Furthermore, we analyze challenging experimental data containing densely packed cells or noisy backgrounds. We also exploit additional symmetries to extend the measurable particle properties to the particle’s vertical position by propagating the signal in Fourier space and its polarizability by scaling the signal strength. Thanks to the ability to train deep-learning models with a single unlabeled image, LodeSTAR can accelerate the development of high-quality microscopic analysis pipelines for engineering, biology, and medicine.

Microplankton life histories revealed by holographic microscopy and deep learning on ArXiv

Tracking of microplankton by holographic optical microscopy and deep learning. (Image by H. Bachimanchi.)
Microplankton life histories revealed by holographic microscopy and deep learning
Harshith Bachimanchi, Benjamin Midtvedt, Daniel Midtvedt, Erik Selander, and Giovanni Volpe
arXiv: 2202.09046

The marine microbial food web plays a central role in the global carbon cycle. Our mechanistic understanding of the ocean, however, is biased towards its larger constituents, while rates and biomass fluxes in the microbial food web are mainly inferred from indirect measurements and ensemble averages. Yet, resolution at the level of the individual microplankton is required to advance our understanding of the oceanic food web. Here, we demonstrate that, by combining holographic microscopy with deep learning, we can follow microplanktons throughout their lifespan, continuously measuring their three dimensional position and dry mass. The deep learning algorithms circumvent the computationally intensive processing of holographic data and allow rapid measurements over extended time periods. This permits us to reliably estimate growth rates, both in terms of dry mass increase and cell divisions, as well as to measure trophic interactions between species such as predation events. The individual resolution provides information about selectivity, individual feeding rates and handling times for individual microplanktons. This method is particularly useful to explore the flux of carbon through micro-zooplankton, the most important and least known group of primary consumers in the global oceans. We exemplify this by detailed descriptions of micro-zooplankton feeding events, cell divisions, and long term monitoring of single cells from division to division.

Geometric deep learning reveals the spatiotemporal fingerprint of microscopic motion on ArXiv

Input graph structure including a redundant number of edges. (Image by J. Pineda.)
Geometric deep learning reveals the spatiotemporal fingerprint of microscopic motion
Jesús Pineda, Benjamin Midtvedt, Harshith Bachimanchi, Sergio Noé, Daniel Midtvedt, Giovanni Volpe, Carlo Manzo
arXiv: 2202.06355

The characterization of dynamical processes in living systems provides important clues for their mechanistic interpretation and link to biological functions. Thanks to recent advances in microscopy techniques, it is now possible to routinely record the motion of cells, organelles, and individual molecules at multiple spatiotemporal scales in physiological conditions. However, the automated analysis of dynamics occurring in crowded and complex environments still lags behind the acquisition of microscopic image sequences. Here, we present a framework based on geometric deep learning that achieves the accurate estimation of dynamical properties in various biologically-relevant scenarios. This deep-learning approach relies on a graph neural network enhanced by attention-based components. By processing object features with geometric priors, the network is capable of performing multiple tasks, from linking coordinates into trajectories to inferring local and global dynamic properties. We demonstrate the flexibility and reliability of this approach by applying it to real and simulated data corresponding to a broad range of biological experiments.

Presentation by H. Bachimanchi at Prof. Metzler’s group at the University of Potsdam, 4 February 2022

Tracking of the planktons. (Image by H. Bachimanchi.)
Characterising plankton behaviours using deep learning powered inline holography
Harshith Bachimanchi
Presentation at Prof. Ralf Metzler’s Theoretical Physics group at University of Potsdam (Online)
4 February 2022, 14:15 CET

Digital holographic microscopy is a powerful label-free imaging technique for studying biological specimens. The complex optical fields of microscopic objects can be stored in the form of interference patterns and can be reconstructed by using the principles of holography. Recently, we have developed a digital inline holographic microscope with a deep learning powered analysis to track planktons through generations, and continuously measure their three-dimensional position and dry mass. By bringing planktons of different trophic levels together, we were able to perform a quantitative assessment of trophic interactions between planktons such as feeding events, biomass transfer from cell to cell, etc. In this talk, I will be giving a short overview of our method and present some of our recent results.

Presentation by H. Bachimanchi at M2C2, Weizmann Institute, Israel, 5 May 2021

Classification of phytoplankton (blue) and microzooplankton (orange) by holography + deep learning: Schematic of the experimental setup (left). (Image by Harshith Bachimanchi.)
Microzooplankton classification and their feeding patterns by digital holographic microscopy and deep learning
Harshith Bachimanchi
Presentation at Marine Microbial Chemical Communication (M2C2) webinar series
(online) at Weizmann institute of science, Israel
5 May 2021, 15:45 CEST

Phytoplankton and zooplankton are the foundation of the marine food chain. Being an autotrophic primary producer, phytoplankton can generate their own source of energy through photosynthesis. During this process, phytoplankton populations all over the world absorb about 65 Gt (gigatons) of carbon from the atmosphere and thereby equivalently produce the largest amount of oxygen on the earth. The main consumers of this absorbed carbon are the heterotrophic microzooplankton, occupying the next level in the hierarchy of the marine food chain, consuming about two-thirds of the total production (39 Gt). This is likely the largest transition of biological carbon on Earth. Despite being fundamental for our understanding of the carbon cycle and the earth’s climate, the standard estimates leave many questions unanswered at a single microplankton level. Here, we demonstrate that machine learning can be used to estimate the amount of carbon consumed at a single plankton level. We use digital holographic microscopy powered by deep learning to classify planktons by their species and track the biomass of the plankton during individual feeding events. We use the planktonic species, Dunaliella tertiolecta, and Oxyrrhis marina, for our experiments which belong to classes of phytoplankton and microzooplankton respectively. With the help of artificial neural networks, we manage to estimate the carbon consumption and native carbon content at an individual microzooplankton level. Furthermore, we demonstrate the advantages of the approach and compare the results with standard ensemble estimates.

Soft Matter Lab presentations at the SPIE Optics+Photonics Digital Forum

Seven members of the Soft Matter Lab (Saga HelgadottirBenjamin Midtvedt, Aykut Argun, Laura Pérez-GarciaDaniel MidtvedtHarshith BachimanchiEmiliano Gómez) were selected for oral and poster presentations at the SPIE Optics+Photonics Digital Forum, August 24-28, 2020.

The SPIE digital forum is a free, online only event.
The registration for the Digital Forum includes access to all presentations and proceedings.

The Soft Matter Lab contributions are part of the SPIE Nanoscience + Engineering conferences, namely the conference on Emerging Topics in Artificial Intelligence 2020 and the conference on Optical Trapping and Optical Micromanipulation XVII.

The contributions being presented are listed below, including also the presentations co-authored by Giovanni Volpe.

Note: the presentation times are indicated according to PDT (Pacific Daylight Time) (GMT-7)

Emerging Topics in Artificial Intelligence 2020

Saga Helgadottir
Digital video microscopy with deep learning (Invited Paper)
26 August 2020, 10:30 AM
SPIE Link: here.

Aykut Argun
Calibration of force fields using recurrent neural networks
26 August 2020, 8:30 AM
SPIE Link: here.

Laura Pérez-García
Deep-learning enhanced light-sheet microscopy
25 August 2020, 9:10 AM
SPIE Link: here.

Daniel Midtvedt
Holographic characterization of subwavelength particles enhanced by deep learning
24 August 2020, 2:40 PM
SPIE Link: here.

Benjamin Midtvedt
DeepTrack: A comprehensive deep learning framework for digital microscopy
26 August 2020, 11:40 AM
SPIE Link: here.

Gorka Muñoz-Gil
The anomalous diffusion challenge: Single trajectory characterisation as a competition
26 August 2020, 12:00 PM
SPIE Link: here.

Meera Srikrishna
Brain tissue segmentation using U-Nets in cranial CT scans
25 August 2020, 2:00 PM
SPIE Link: here.

Juan S. Sierra
Automated corneal endothelium image segmentation in the presence of cornea guttata via convolutional neural networks
26 August 2020, 11:50 AM
SPIE Link: here.

Harshith Bachimanchi
Digital holographic microscopy driven by deep learning: A study on marine planktons (Poster)
24 August 2020, 5:30 PM
SPIE Link: here.

Emiliano Gómez
BRAPH 2.0: Software for the analysis of brain connectivity with graph theory (Poster)
24 August 2020, 5:30 PM
SPIE Link: here.

Optical Trapping and Optical Micromanipulation XVII

Laura Pérez-García
Reconstructing complex force fields with optical tweezers
24 August 2020, 5:00 PM
SPIE Link: here.

Alejandro V. Arzola
Direct visualization of the spin-orbit angular momentum conversion in optical trapping
25 August 2020, 10:40 AM
SPIE Link: here.

Isaac Lenton
Illuminating the complex behaviour of particles in optical traps with machine learning
26 August 2020, 9:10 AM
SPIE Link: here.

Fatemeh Kalantarifard
Optical trapping of microparticles and yeast cells at ultra-low intensity by intracavity nonlinear feedback forces
24 August 2020, 11:10 AM
SPIE Link: here.

Note: the presentation times are indicated according to PDT (Pacific Daylight Time) (GMT-7)

Harshith Bachimanchi joins the Soft Matter Lab

Harshith Bachimanchi starts his PhD at the Physics Department of the University of Gothenburg on 20th January 2020.

Harshith has a Master degree in physics from the Indian Institute of Science Education and Research, Pune, India, where he submitted a Master thesis in optics, whose results can be found here.

In his PhD, he will focus on microscopy and deep learning.

Seminar on controlled generation of high power optical vortex arrays by Harshith Bachimanchi from IISER Pune, Faraday, 18 September 2019

Controlled generation of high power optical vortex arrays, and their frequency-doubling characteristics
Seminar by Harshith Bachimanchi from the Indian Institute of Science Education and Research, Pune (IISER Pune).

Optical vortices, beams carrying orbital angular momentum (OAM) per photon are of supreme interest in recent times for their wide variety of applications in quantum information, micro-manipulation, and material lithography [1, 2, 3]. Due to a helical phase variation in propagation, and an undefined phase at the centre, these beams have a phase singularity in their wavefront, resulting in the doughnut-shaped intensity distribution. Though the vortex beams have been widely explored in the past, the recent advancements on multiple particle trapping, single-shot material lithography, and multiplexing in quantum information [4] demand an array of optical vortices in a simple experimental scheme.

While the majority of the existing mode converters transform the Gaussian beam into a single vortex beam, the intrinsic advantage of the dynamic phase modulation through holographic technique allow the spatial light modulators (SLMs) to generate vortex arrays directly from a Gaussian beam. However, the low damage threshold of SLMs restricts their usage for high power vortex array applications.

Here, we elaborate a simple experimental scheme to generate high power, ultrafast, higher order optical vortex arrays. Simply by using a dielectric Microlens array (MLA) and a plano-convex lens we generate an array of beams carrying the spatial property of the input beam. Though we’ve verified the technique for the case of optical vortices, it holds true for a useful subset of structured optical beams. Considering the MLA as a 2D sinusoidal phase grating, we have numerically calculated the intensity pattern of the array beams in close agreement with the experimental results. We have also theoretically derived the parameters controlling the intensity pattern, size and the pitch of array and verified experimentally. The single-pass frequency doubling of the vortex array at 1064 nm in a 1.2 mm BiBO crystal produced green vortex arrays of orders as high as lsh = 12, twice the order of the pump array beam, with a conversion efficiency as high as ∼3.65% [5].

References:

  1. Grier, D. G. A revolution in optical manipulation. Nature 424, 810 (2003)
  2. Mair, A., Vaziri, A., Weihs, G. & Zeilinger, A. Entanglement of the orbital angular momentum states of photons. Nature 412, 313 (2001).
  3. Scott, T. F., Kowalski, B. A., Sullivan, A. C., Bowman, C. N. & McLeod, R. R. Two-color single-photon photoinitiation and photoinhibition for subdiffraction photo-lithography. Science 324, 913–917 (2009).
  4. Omatsu, T. et al. Metal microneedle fabrication using twisted light with spin. Opt. Express 18, 17967–17973 (2010).
  5. Harshith, B.S., Samanta, G.K. Controlled generation of array beams of higher order orbital angular momentum and study of their frequency-doubling characteristics. Sci Rep 9, 10916 (2019).

Place: Faraday room, Fysik Origo, Fysik
Time: 18 September, 2019, 15:00