Presentation by J. Pineda at LoG Meetup Sweden, 12 February 2025

MIRO employs a recurrent graph neural network to refine SMLM point clouds by compressing clusters around their center, enhancing inter-cluster distinction and background separation for efficient clustering. (Image by J. Pineda.)
Relational Inductive Biases as a Key to Smarter Deep Learning Microscopy
Jesús Pineda
Learning on graphs and geometry meetup at Uppsala University
Date: 12 February 2025
Time: 11:15
Place: Lecture hall 4101, Ångströmlaboratoriet, Uppsala, Sweden

Geometric deep learning has revolutionized fields like social network analysis, molecular chemistry, and neuroscience, but its application to microscopy data analysis remains a significant challenge. The hurdles stem not only from the scarcity of high-quality data but also from the intrinsic complexity and variability of microscopy datasets. This presentation introduces two groundbreaking geometric deep-learning frameworks designed to overcome these barriers, advancing the integration of graph neural networks (GNNs) into microscopy and unlocking their full potential. First, we present MAGIK, a cutting-edge framework for analyzing biological system dynamics through time-lapse microscopy. Leveraging a graph neural network augmented with attention-based mechanisms, MAGIK processes object features using geometric priors. This enables it to perform a range of tasks, from linking coordinates into trajectories to uncovering local and global dynamic properties with unprecedented precision. Remarkably, MAGIK excels under minimal data conditions, maintaining exceptional performance and robust generalization across diverse scenarios. Next, we introduce MIRO, a novel algorithm powered by recurrent graph neural networks. MIRO pre-processes Single Molecule Localization (SML) datasets to enhance the efficiency of conventional clustering methods. Its ability to handle clusters of varying shapes and scales enables more accurate and consistent analyses across complex datasets. Furthermore, MIRO’s single- and few-shot learning capabilities allow it to generalize effortlessly across scenarios, making it an efficient, scalable, and versatile tool for microscopy data analysis. Together, MAGIK and MIRO address critical limitations in microscopy data analysis, offering innovative solutions for multi-scale data analysis and advancing the boundaries of what is currently achievable with geometric deep learning in the field.

Reference

Pineda, Jesús, Benjamin Midtvedt, Harshith Bachimanchi, Sergio Noé, Daniel Midtvedt, Giovanni Volpe, and Carlo Manzo. Geometric deep learning reveals the spatiotemporal features of microscopic motionNat Mach Intell 5, 71–82 (2023).

Pineda, Jesús, Sergi Masó-Orriols, Joan Bertran, Mattias Goksör, Giovanni Volpe, and Carlo Manzo. Spatial Clustering of Molecular Localizations with Graph Neural Networks.  arXiv: 2412.00173 (2024).

Invited talk by J. Pineda at the CMCB Lab on 27 January 2025

MIRO employs a recurrent graph neural network to refine SMLM point clouds by compressing clusters around their center, enhancing inter-cluster distinction and background separation for efficient clustering. (Image by J. Pineda.)
Spatial clustering of molecular localizations with graph neural networks
Jesús Pineda
Date: 27 January 2025
Time: 10:00
Place: SciLifeLab Campus Solna, Sweden

Single-molecule localization microscopy (SMLM) generates point clouds corresponding to fluorophore localizations. Spatial cluster identification and analysis of these point clouds are crucial for extracting insights about molecular organization. However, this task becomes challenging in the presence of localization noise, high point density, or complex biological structures. Here, we introduce MIRO (Multimodal Integration through Relational Optimization), an algorithm that uses recurrent graph neural networks to transform the point clouds in order to improve clustering efficiency when applying conventional clustering techniques. We show that MIRO supports simultaneous processing of clusters of different shapes and at multiple scales, demonstrating improved performance across varied datasets. Our comprehensive evaluation demonstrates MIRO’s transformative potential for single-molecule localization applications, showcasing its capability to revolutionize cluster analysis and provide accurate, reliable details of molecular architecture. In addition, MIRO’s robust clustering capabilities hold promise for applications in various fields such as neuroscience, for the analysis of neural connectivity patterns, and environmental science, for studying spatial distributions of ecological data.

Reference
Pineda, Jesús, Sergi Masó-Orriols, Joan Bertran, Mattias Goksör, Giovanni Volpe, and Carlo Manzo. Spatial Clustering of Molecular Localizations with Graph Neural Networks.  arXiv: 2412.00173

Spatial clustering of molecular localizations with graph neural networks on ArXiv

MIRO employs a recurrent graph neural network to refine SMLM point clouds by compressing clusters around their center, enhancing inter-cluster distinction and background separation for efficient clustering. (Image by J. Pineda.)
Spatial clustering of molecular localizations with graph neural networks
Jesús Pineda, Sergi Masó-Orriols, Joan Bertran, Mattias Goksör, Giovanni Volpe and Carlo Manzo
arXiv: 2412.00173

Single-molecule localization microscopy (SMLM) generates point clouds corresponding to fluorophore localizations. Spatial cluster identification and analysis of these point clouds are crucial for extracting insights about molecular organization. However, this task becomes challenging in the presence of localization noise, high point density, or complex biological structures. Here, we introduce MIRO (Multimodal Integration through Relational Optimization), an algorithm that uses recurrent graph neural networks to transform the point clouds in order to improve clustering efficiency when applying conventional clustering techniques. We show that MIRO supports simultaneous processing of clusters of different shapes and at multiple scales, demonstrating improved performance across varied datasets. Our comprehensive evaluation demonstrates MIRO’s transformative potential for single-molecule localization applications, showcasing its capability to revolutionize cluster analysis and provide accurate, reliable details of molecular architecture. In addition, MIRO’s robust clustering capabilities hold promise for applications in various fields such as neuroscience, for the analysis of neural connectivity patterns, and environmental science, for studying spatial distributions of ecological data.

Cross-modality transformations in biological microscopy enabled by deep learning published in Advanced Photonics

Cross-modality transformation and segmentation. (Image by the Authors of the manuscript.)
Cross-modality transformations in biological microscopy enabled by deep learning
Dana Hassan, Jesús Domínguez, Benjamin Midtvedt, Henrik Klein Moberg, Jesús Pineda, Christoph Langhammer, Giovanni Volpe, Antoni Homs Corbera, Caroline B. Adiels
Advanced Photonics 6, 064001 (2024)
doi: 10.1117/1.AP.6.6.064001

Recent advancements in deep learning (DL) have propelled the virtual transformation of microscopy images across optical modalities, enabling unprecedented multimodal imaging analysis hitherto impossible. Despite these strides, the integration of such algorithms into scientists’ daily routines and clinical trials remains limited, largely due to a lack of recognition within their respective fields and the plethora of available transformation methods. To address this, we present a structured overview of cross-modality transformations, encompassing applications, data sets, and implementations, aimed at unifying this evolving field. Our review focuses on DL solutions for two key applications: contrast enhancement of targeted features within images and resolution enhancements. We recognize cross-modality transformations as a valuable resource for biologists seeking a deeper understanding of the field, as well as for technology developers aiming to better grasp sample limitations and potential applications. Notably, they enable high-contrast, high-specificity imaging akin to fluorescence microscopy without the need for laborious, costly, and disruptive physical-staining procedures. In addition, they facilitate the realization of imaging with properties that would typically require costly or complex physical modifications, such as achieving superresolution capabilities. By consolidating the current state of research in this review, we aim to catalyze further investigation and development, ultimately bringing the potential of cross-modality transformations into the hands of researchers and clinicians alike.

Crystallization and topology-induced dynamical heterogeneities in soft granular clusters published in Physical Review of Research

Scheme of the microfluidic system for the production of clusters of a soft granular medium, and Snapshots of the cluster at different times corresponding to different sections of the channel. (Image by the Authors of the manuscript.)
Crystallization and topology-induced dynamical heterogeneities in soft granular clusters
Michal Bogdan, Jesus Pineda, Mihir Durve, Leon Jurkiewicz, Sauro Succi, Giovanni Volpe, Jan Guzowski
Physical Review of Research, 6, L032031 (2024)
DOI: 10.1103/PhysRevResearch.6.L032031
arXiv: 2302.05363

Soft-granular media, such as dense emulsions, foams or tissues, exhibit either fluid- or solidlike properties depending on the applied external stresses. Whereas bulk rheology of such materials has been thoroughly investigated, the internal structural mechanics of finite soft-granular structures with free interfaces is still poorly understood. Here, we report the spontaneous crystallization and melting inside a model soft granular cluster—a densely packed aggregate of N~30-40 droplets engulfed by a fluid film—subject to a varying external flow. We develop machine learning tools to track the internal rearrangements in the quasi-two-dimensional cluster as it transits a sequence of constrictions. As the cluster relaxes from a state of strong mechanical deformations, we find differences in the dynamics of the grains within the interior of the cluster and those at its rim, with the latter experiencing larger deformations and less frequent rearrangements, effectively acting as an elastic membrane around a fluidlike core. We conclude that the observed structural-dynamical heterogeneity results from an interplay of the topological constrains, due to the presence of a closed interface, and the internal solid-fluid transitions. We discuss the universality of such behavior in various types of finite soft granular structures, including biological tissues.

Book “Deep Learning Crash Course” published at No Starch Press

The book Deep Learning Crash Course, authored by Giovanni Volpe, Benjamin Midtvedt, Jesús Pineda, Henrik Klein Moberg, Harshith Bachimanchi, Joana B. Pereira, and Carlo Manzo, has been published online by No Starch Press in July 2024.

Preorder Discount
A preorder discount is available: preorders with coupon code PREORDER will receive 25% off. Link: Preorder @ No Starch Press | Deep Learning Crash Course

Links
@ No Starch Press

Citation 
Giovanni Volpe, Benjamin Midtvedt, Jesús Pineda, Henrik Klein Moberg, Harshith Bachimanchi, Joana B. Pereira, and Carlo Manzo. Deep Learning Crash Course. No Starch Press.
ISBN-13: 9781718503922

Presentation by J. Pineda at SPIE-ETAI, San Diego, 23 August 2023

Input graph structure including a redundant number of edges. (Image by J. Pineda.)
MAGIK: Microscopic motion analysis through graph inductive knowledge
Jesús Pineda
Date: 23 August 2023
Time: 2:30 PM PDT

Characterizing dynamic processes in living systems provides essential information for advancing our understanding of life processes in health and diseases and for developing new technologies and treatments. In the past two decades, optical microscopy has undergone significant developments, enabling us to study the motion of cells, organelles, and individual molecules with unprecedented detail at various scales in space and time. However, analyzing the dynamic processes that occur in complex and crowded environments remains a challenge. This work introduces MAGIK, a deep-learning framework for the analysis of biological system dynamics from time-lapse microscopy. MAGIK models the movement and interactions of particles through a directed graph where nodes represent detections and edges connect spatiotemporally close nodes. The framework utilizes an attention-based graph neural network (GNN) to process the graph and modulate the strength of associations between its elements, enabling MAGIK to derive insights into the dynamics of the systems. MAGIK provides a key enabling technology to estimate any dynamic aspect of the particles, from reconstructing their trajectories to inferring local and global dynamics. We demonstrate the flexibility and reliability of the framework by applying it to real and simulated data corresponding to a broad range of biological experiments.

Reference
Pineda, J., Midtvedt, B., Bachimanchi, H. et al. Geometric deep learning reveals the spatiotemporal features of microscopic motionNat Mach Intell 5, 71–82 (2023)

Poster by J. Pineda at SPIE-ETAI, San Diego, 21 August 2023

The proposed method allows for robust detection, segmentation, and tracking of soft granular clusters. (Image by J. Pineda.)
Unveiling the complex dynamics of soft granular materials using deep learning
Jesús Pineda
Date: 21 August 2023
Time: 5:30 PM PDT

Soft granular materials, comprising closely packed grains held together by a thin layer of lubricating fluid, display intricate many-body dynamics resulting in complex flows and rheological behavior, including plasticity and viscoelasticity, memory effects, and avalanches. Despite their widespread presence in nature and industrial applications, the structural mechanics and microscale dynamics of soft granular clusters still need to be better understood, especially those under strong confinement or surrounded by free interfaces. This work aims to bridge the gap in understanding the internal dynamics of finite-sized soft granular media by introducing a deep learning approach to characterize the shapes and movements of deformable grains in the material. We demonstrate the reliability and versatility of the method by studying the dynamics of soft granular clusters that self-organize under external flow in various physically relevant scenarios.

Soft Matter Lab members present at SPIE Optics+Photonics conference in San Diego, 20-24 August 2023

The Soft Matter Lab participates to the SPIE Optics+Photonics conference in San Diego, CA, USA, 20-24 August 2023, with the presentations listed below.

Giovanni Volpe is also co-author of the presentations:

  • Jiawei Sun (KI): (Poster) Assessment of nonlinear changes in functional brain connectivity during aging using deep learning
    21 August 2023 • 5:30 PM – 7:00 PM PDT | Conv. Ctr. Exhibit Hall A
  • Blanca Zufiria Gerbolés (KI): (Poster) Exploring age-related changes in anatomical brain connectivity using deep learning analysis in cognitively healthy individuals
    21 August 2023 • 5:30 PM – 7:00 PM PDT | Conv. Ctr. Exhibit Hall A
  • Mite Mijalkov (KI): Uncovering vulnerable connections in the aging brain using reservoir computing
    22 August 2023 • 9:15 AM – 9:30 AM PDT | Conv. Ctr. Room 6C

J. Pineda was awarded the Young Investigator Poster Award at the XVII International Congress of the Spanish Biophysical Society, Castelldefels, 30 Jun 2023

Jesús Pineda receives the Young Investigator Poster Award. (Photo by S. Masò Orriols.)
Jesús Pineda was awarded the Young Investigator Poster Award on 30 June 2023 for its poster MAGIK: Microscopic motion analysis through graph inductive knowledge presented at the XVII International Congress of the Spanish Biophysical Society in Castedefells.

Here the link to the poster.