Roadmap on Deep Learning for Microscopy on ArXiv

Spatio-temporal spectrum diagram of microscopy techniques and their applications. (Image by the Authors of the manuscript.)
Roadmap on Deep Learning for Microscopy
Giovanni Volpe, Carolina Wählby, Lei Tian, Michael Hecht, Artur Yakimovich, Kristina Monakhova, Laura Waller, Ivo F. Sbalzarini, Christopher A. Metzler, Mingyang Xie, Kevin Zhang, Isaac C.D. Lenton, Halina Rubinsztein-Dunlop, Daniel Brunner, Bijie Bai, Aydogan Ozcan, Daniel Midtvedt, Hao Wang, Nataša Sladoje, Joakim Lindblad, Jason T. Smith, Marien Ochoa, Margarida Barroso, Xavier Intes, Tong Qiu, Li-Yu Yu, Sixian You, Yongtao Liu, Maxim A. Ziatdinov, Sergei V. Kalinin, Arlo Sheridan, Uri Manor, Elias Nehme, Ofri Goldenberg, Yoav Shechtman, Henrik K. Moberg, Christoph Langhammer, Barbora Špačková, Saga Helgadottir, Benjamin Midtvedt, Aykut Argun, Tobias Thalheim, Frank Cichos, Stefano Bo, Lars Hubatsch, Jesus Pineda, Carlo Manzo, Harshith Bachimanchi, Erik Selander, Antoni Homs-Corbera, Martin Fränzl, Kevin de Haan, Yair Rivenson, Zofia Korczak, Caroline Beck Adiels, Mite Mijalkov, Dániel Veréb, Yu-Wei Chang, Joana B. Pereira, Damian Matuszewski, Gustaf Kylberg, Ida-Maria Sintorn, Juan C. Caicedo, Beth A Cimini, Muyinatu A. Lediju Bell, Bruno M. Saraiva, Guillaume Jacquemet, Ricardo Henriques, Wei Ouyang, Trang Le, Estibaliz Gómez-de-Mariscal, Daniel Sage, Arrate Muñoz-Barrutia, Ebba Josefson Lindqvist, Johanna Bergman
arXiv: 2303.03793

Through digital imaging, microscopy has evolved from primarily being a means for visual observation of life at the micro- and nano-scale, to a quantitative tool with ever-increasing resolution and throughput. Artificial intelligence, deep neural networks, and machine learning are all niche terms describing computational methods that have gained a pivotal role in microscopy-based research over the past decade. This Roadmap is written collectively by prominent researchers and encompasses selected aspects of how machine learning is applied to microscopy image data, with the aim of gaining scientific knowledge by improved image quality, automated detection, segmentation, classification and tracking of objects, and efficient merging of information from multiple imaging modalities. We aim to give the reader an overview of the key developments and an understanding of possibilities and limitations of machine learning for microscopy. It will be of interest to a wide cross-disciplinary audience in the physical sciences and life sciences.

Active matter in space published in npj Microgravity

Effect of gravity on matter: Sedimentation and creaming. Fv and Fg represent the viscous force and gravitational force, respectively. (Image by Authors.)
Active matter in space
Giorgio Volpe, Clemens Bechinger, Frank Cichos, Ramin Golestanian, Hartmut Löwen, Matthias Sperl and Giovanni Volpe
npj Microgravity, 8, 54 (2022)
doi: 10.1038/s41526-022-00230-7

In the last 20 years, active matter has been a highly dynamic field of research, bridging fundamental aspects of non-equilibrium thermodynamics with applications to biology, robotics, and nano-medicine. Active matter systems are composed of units that can harvest and harness energy and information from their environment to generate complex collective behaviours and forms of self-organisation. On Earth, gravity-driven phenomena (such as sedimentation and convection) often dominate or conceal the emergence of these dynamics, especially for soft active matter systems where typical interactions are of the order of the thermal energy. In this review, we explore the ongoing and future efforts to study active matter in space, where low-gravity and microgravity conditions can lift some of these limitations. We envision that these studies will help unify our understanding of active matter systems and, more generally, of far-from-equilibrium physics both on Earth and in space. Furthermore, they will also provide guidance on how to use, process and manufacture active materials for space exploration and colonisation.

Enhanced force-field calibration via machine learning featured in AIP SciLight

The article Enhanced force-field calibration via machine learning
has been featured in: “Machine Learning Outperforms Standard Force-Field Calibration Techniques”, AIP SciLight (November 6, 2020).

Scilight showcases the most interesting research across the physical sciences published in AIP Publishing Journals.

Scilight is published weekly (52 issues per year) by AIP Publishing.

Enhanced force-field calibration via machine learning published in Applied Physics Reviews

Representation of a particle in a force field
Enhanced force-field calibration via machine learning
Aykut Argun, Tobias Thalheim, Stefano Bo, Frank Cichos, Giovanni Volpe
Applied Physics Reviews 7, 041404 (2020)
doi: 10.1063/5.0019105
arXiv: 2006.08963

The influence of microscopic force fields on the motion of Brownian particles plays a fundamental role in a broad range of fields, including soft matter, biophysics, and active matter. Often, the experimental calibration of these force fields relies on the analysis of the trajectories of these Brownian particles. However, such an analysis is not always straightforward, especially if the underlying force fields are non-conservative or time-varying, driving the system out of thermodynamic equilibrium. Here, we introduce a toolbox to calibrate microscopic force fields by analyzing the trajectories of a Brownian particle using machine learning, namely recurrent neural networks. We demonstrate that this machine-learning approach outperforms standard methods when characterizing the force fields generated by harmonic potentials if the available data are limited. More importantly, it provides a tool to calibrate force fields in situations for which there are no standard methods, such as non-conservative and time-varying force fields. In order to make this method readily available for other users, we provide a Python software package named DeepCalib, which can be easily personalized and optimized for specific applications.


ERC-founder H2020 European Research Council (ERC) Starting Grant ComplexSwimmers (677511).

Machine learning for active matter published on Nature Machine Intelligence

Neural net with input layer (left), dense internal layers, and output layer (right).

Machine learning for active matter
Frank Cichos, Kristian Gustavsson, Bernhard Mehlig & Giovanni Volpe
Nature Machine Intelligence 2(2), 94–103 (2020)

The availability of large datasets has boosted the application of machine learning in many fields and is now starting to shape active-matter research as well. Machine learning techniques have already been successfully applied to active-matter data—for example, deep neural networks to analyse images and track objects, and recurrent nets and random forests to analyse time series. Yet machine learning can also help to disentangle the complexity of biological active matter, helping, for example, to establish a relation between genetic code and emergent bacterial behaviour, to find navigation strategies in complex environments, and to map physical cues to animal behaviours. In this Review, we highlight the current state of the art in the application of machine learning to active matter and discuss opportunities and challenges that are emerging. We also emphasize how active matter and machine learning can work together for mutual benefit.

CECAM Workshop “Active Matter and Artificial Intelligence Location”, Lausanne, Switzerland 30 September – 2 October 2019

Active Matter and Artificial Intelligence
Location : CECAM-HQ-EPFL, Lausanne, Switzerland
September 30, 2019 – October 2, 2019

Frank Cichos (Universität Leipzig, Germany)
Klaus Kroy (Universität Leipzig, Germany)
Fernando Peruani (Université Nice Sophia Antipolis, France)
Giovanni Volpe (University of Gothenburg, Sweden)


Biological active matter is composed of self-propelling agents, such as molecular motors, cells, bacteria and animals [1,2], which can perform tasks and feature emergent collective behaviors thanks to their capability of sensing their environment, processing this information and exploiting it through feedback cycles [3]. These processes are intrinsically noisy [4] both at the microscale (e.g. thermal noise [5]) and at the macroscale (e.g. turbulence [6]). Therefore, through millions of years, biological systems have evolved powerful strategies to accomplish specific tasks and thrive in their environment – strategies that are encoded in their shape, biophysical properties, and signal processing networks [13].

Artificial active matter is now being explored as a powerful means to address the big challenges that our society is facing [7]: from new strategies for targeted drug delivery, to the decontamination of polluted soils, to the extraction of energy from naturally occurring out-of-equilibrium conditions. In this context, biological active matter provides an ideal source of tested ideas and approaches [8,9], which we are now trying to exploit to develop artificial systems [10,11].

However, in biological systems, there is only a limited possibility to reduce complexity and introduce controllable perturbations. Therefore, the development of computational models and of proof-of-principle experiments provides an ideal test bench to explore the origin of complexity in biological systems and to harness it for the development of new applications. For example, tuning of sensorial delays yield different behaviors in gradient fields relevant for cellular systems [12], and, inspired by neuronal networks, relevant past experience is harnessed to predict the evolution of complex systems.

In this process, we have been led to the application of machine learning to active matter. Machine learning is an abstraction of the adaption processes found in biological active matter and researchers have recently started to explore these algorithms in active matter in some pioneering works. For example, reinforcement learning [14], which reflects a type of learning based on rewards, has been used to steer the motion of microscopic particles [15,16], to understand how birds can exploit turbulent thermal air flows to soar [6], to control the motion of artificial microswimmers in complex flow patterns [17] as well as in collective field taxis [18].

We are now at a critical crossroad in the development of active matter research where biological and artificial active matter are meeting with machine learning. The specific aim of this workshop is to bring together researchers from the fields of physics, biology, mathematics and machine learning to lay the groundwork of a scientific network to address the pressing questions:

1. What can machine learning do for biological active matter? Can we gain new insight into how powerful strategies have evolved? Can we understand the role of information processing, feedback cycles and sensorial delay in these strategies?

2. What can machine learning do for artificial active matter? Can we learn new approaches towards high-impact applications? For example, how can signaling and feedback be introduced into artificial active matter?

3. What insights can active matter provide for machine learning? Can we get some insight from the natural strategies optimized by evolution?


[1] Ramaswamy, S., The mechanics and statistics of active matter. Annu. Rev. Condens. Matter Phys. 1, 323–345 (2010).

[2] Marchetti, M. C. et al., Hydrodynamics of soft active matter. Rev. Mod. Phys. 85, 1143–1189 (2013).

[3] Katz, Y., Tunstrøm, K., Ioannou, C. C., Huepe, C., Couzin, I. D., Inferring the structure and dynamics of interactions in schooling fish. Proc. Natl. Acad. Sci. USA 108, 18720–18725 (2011).

[4] Yates, C. A. et al., Inherent noise can facilitate coherence in collective swarm motion. Proc. Natl. Acad. Sci. USA 106, 5464–5469 (2009).

[5] Kromer, J. A., Märcker, S., Lange, S., Baier, C., Friedrich, B. M., Decision making improves sperm chemotaxis in the presence of noise. PLoS Comput. Biol. 14, e1006109–15 (2018).

[6] Reddy, G., Celani, A., Sejnowski, T. J., Vergassola, M., Learning to soar in turbulent environments. Proc. Natl. Acad. Sci. USA 113, E4877–84 (2016).

[7] Bechinger, C. et al., Active particles in complex and crowded environments. Rev. Mod. Phys. 88, 045006 (2016).

[8] Pearce, D. J. G., Miller, A. M., Rowlands, G., Turner, M. S., Role of projection in the control of bird flocks. Proc. Natl. Acad. Sci. USA 111, 10422–10426 (2014).

[9] Bierbach, D. et al., Insights into the social behavior of surface and cave-dwelling fish (Poecilia mexicana) in light and darkness through the use of a biomimetic robot. Front. Robot. AI 5, 15 (2018).

[10] Buttinoni, I. et al., Dynamical clustering and phase separation in suspensions of self-propelled colloidal particles (2017).

[11] Qian, B., Montiel, D., Bregulla, A., Cichos, F., Yang, H., Harnessing thermal fluctuations for purposeful activities: the manipulation of single micro-swimmers by adaptive photon nudging. Chem. Sci. 4, 1420–1429 (2013).

[12] Mijalkov, M., McDaniel, A., Wehr, J., Volpe, G., Engineering sensorial delay to control phototaxis and emergent collective behaviors. Phys. Rev. X 6, 011008 (2016).

[13] Palmer, S. E., Marre, O., Berry, M. J., Bialek, W., Predictive information in a sensory population. Proc. Natl. Acad. Sci. USA 112, 6908–6913 (2015).

[14] Sutton, R. S., Barto, A. G., Reinforcement learning: an introduction. MIT Press, Cambridge (1998).

[15] Colabrese, S., Gustavsson, K., Celani, A., Biferale, L., Flow navigation by smart microswimmers via reinforcement learning. Phys. Rev. Lett. 118, 158004 (2017).

[16] Muiños-Landin, S., Ghazi-Zahedi, K., Cichos, F., Reinforcement learning of artificial microswimmers. arXiv 1803.06425v2 (2018).

[17] Gustavsson, K., Biferale, L., Celani, A., Colabrese, S., Finding efficient swimming strategies in a three-dimensional chaotic flow by reinforcement learning. Eur. Phys. J. E Soft Matter 40, 313–7 (2017).

[18] Palmer, G., Yaida, S., Optimizing collective fieldtaxis of swarming agents through reinforcement learning. arXiv 1709.02379 (2017).

Colloquium on artificial microswimmers by Frank Cichos, PJ Lecture Hall, 8 Nov 2018

Information Controlled Structure Formation in Artificial Microswimmer Systems
General Physics Colloquium by Frank Cichos, University of Leipzig, Germany

Abstract: Self-organization is the generation of order out of local interactions in non-equilibrium. It is deeply connected to all fields of science from physics, chemistry to biology where functional living structures self-assemble and constantly evolve all based on physical interactions. The emergence of collective animal behavior, of society or language are the result of self-organization processes as well though they involve abstract interactions arising from sensory inputs, information processing, storage and feedback resulting in collective behaviors as found, for example, in crowds of people, flocks of birds, schools of fish or swarms of bacteria.
We introduce such information based interactions to the behavior of self-thermophoretic microswimmers. A real-time feedback of swimmer positions is used as the information to control the swimming direction and speed of other swimmers. The emerging structures reveal frustrated geometries due to confinement to two dimensions. They diffuse like passive clusters of colloids, but posses internal dynamical degrees of freedom that are determined by the feedback delay and the noise in the system. As the information processing in the feedback loops can be designed almost arbitrarily complex systems with mixed feedback delays and noise will give rise to new emergent dynamics of the self-organized structures. The presented control schemes further allow the integration of machine learning algorithms to introduce an adaptive behavior of swimmers.

Place: PJ Lecture Hall

Microscopic Critical Engine featured in Phys.Org

Microscopic engine powered by critical demixing

Our recent article Microscopic engine powered by critical remixing
by Falko Schmidt, Alessandro Magazzù, Agnese Callegari, Luca Biancofiore, Frank Cichos & Giovanni Volpe, published in Physical Review Letters 120(6), 068004 (2018) has been featured in “Tiny engine powered by demixing fluid” Phys.Org (February 12, 2018)

Microscopic Critical Engine featured in Optics & Photonics News

Microscopic engine powered by critical demixing

Our recent article Microscopic engine powered by critical remixing
by Falko Schmidt, Alessandro Magazzù, Agnese Callegari, Luca Biancofiore, Frank Cichos & Giovanni Volpe, published in Physical Review Letters 120(6), 068004 (2018) has been featured in “Laser + Critical Liquid = Micro-Engine”, Optics & Photonics News (February 12, 2018)

Optics & Photonics News (OPN) is The Optical Society’s monthly news magazine. It provides in-depth coverage of recent developments in the field of optics and offers busy professionals the tools they need to succeed in the optics industry, as well as informative pieces on a variety of topics such as science and society, education, technology and business. OPN strives to make the various facets of this diverse field accessible to researchers, engineers, businesspeople and students. Contributors include scientists and journalists who specialize in the field of optics.

Microscopic Critical Engine featured in APS Physics

Microscopic engine powered by critical demixing

Our recent article Microscopic engine powered by critical remixing
by Falko Schmidt, Alessandro Magazzù, Agnese Callegari, Luca Biancofiore, Frank Cichos & Giovanni Volpe, published in Physical Review Letters 120(6), 068004 (2018) has been featured in “Focus: A Tiny Engine Powered by Light and Liquid Physics”, Physics 11, 16 (February 9, 2018)

Physics provides daily online-only news and commentary about a selection of papers from the APS journals collection. It is aimed at the reader who wants to keep up with highlights of physics research with explanations that don’t rely on complex technical detail.

The category Physics: focus stories features only a few number of articles each week selected among the set of articles published on all the APS journals.
Research articles that have an interdisciplinary character are usually selected, and their explanations are geared toward students and non-experts. Features are written by a journalist for an audience with a general interest in physics.