Soft Materials | Course
Advanced ML Techniques Applied to Microscopy Data Analysis

Course recaps

• October 2022: First session recap
• November 2022: Interview with course participant Carolina van Baalen
• March 2023: Course conclusion

Course Description & Motivation

Microscopy images, irrespective of the specific imaging technique, e.g. optical, electron or atomic force microscopy, are an extremely rich source of quantitative data. With the ever increasing push to enhance spatial and temporal resolution, as well as with the increase of storage and computing power, very large amounts of data are easily generated and require automation for data extraction. From the familiar case of particle tracking, to more complex tasks in image segmentation and feature recognition, machine-learning (ML) methods are rapidly taking the scene. This course, aimed at doctoral students, has the goal to guide attendees through a progression from basic ML methods, through the extension of those to increasingly complex analyses all the way to offering the students the possibility to directly apply the concepts learned during the course to their own data. The course will combine lectures with hands-on exercises in concentrated blocks across the semester. Students have the possibility to select different blocks, for instance if they already have basic ML programming knowledge. The students will also be able to work on a project related to their research where they apply ML to some imaging data. As a prerequisite, basic programming knowledge in Python is required.

Structure of the Course

3 modules (3 full days each; morning lectures – afternoon exercises) distributed across the fall semester 2022 (5-7 Oct, 2-4 Nov & 14-16 Dec) with final presentations in March 2023; possibly organised as a final off-site event with project presentations and discussions.

Module 1: Introduction to ML algorithms (5-7 October 2022)

• Day 1: Deep Learning and Dense Neural Networks
  Basic dense neural networks and backpropagation. Introduction to deep learning and feature engineering. Introduction to the standard neural-networks packages, TensorFlow/Keras and PyTorch.
• Day 2: Recurrent Neural Networks and Transformers
  Analysis of time series and text using recurrent neural networks (e.g., long short-term memory neural networks). Introduction of the concept of attention.
• Day 3: Convolutional Neural Networks
  Analysis of images using convolutional layers. Introduction to advanced architectures such as autoencoders for the compression of images and U-Nets for the analysis of biomedical images.

On Day 3, we will also have a brief introduction of the research problems each student is interested in solving.

Module 2: ML for image analysis I (2-4 November 2022)

• Day 4: Image Classification
  We introduce the problem of microscopy image classification. We will demonstrate how to classify microscopy images using standard convolutional neural networks (with an example on classification of apoptotic cells), as well as the more recent attention-based neural networks (with examples on chest x-ray images, histopathologic classification ). We also discuss potential challenges, such as non-informative image regions and distributed-information content.
• Day 5: Semantic Segmentation
  We introduce the problem of semantic segmentation of microscopy images. We will show how to extract a semantic segmentation of microscopy images using standard convolutional encoder-decoders and U-Nets (with examples from blood smears, cellular organelles, and histology slides) as well as attention-based neural networks (revisiting the previous examples). We also discuss common challenges, such as overlapping or densely packed objects, and particularly difficult morphologies.
• Day 6: Particle Tracking
  We introduce the problem of object tracking in microscopy images. We will show supervised and unsupervised techniques for detecting and locating objects in microscopy images, using convolutional neural networks (with an example with particles trapped using optical tweezers), convolutional encoder-decoders (cell nuclei) as well as LodeSTAR (lineage estimation in a growing culture of yeast cells). We will also discuss common challenges, such as specificity and object separation.

On Day 6, we will also have a brief follow-up on the student research projects with a discussion of possibilities and alternatives.

Module 3: ML for image analysis II (14-16 December 2022)

• Day 7: Particle Tracing
  We introduce the problem of frame-to-frame detection linking. We will demonstrate techniques using LSTM-networks, attention-networks, transformer-style networks, and graph neural networks (with examples based on the tracing of Brownian particles as well as motile cells). We also discuss potential challenges, such as information-constrained systems, incomplete/incorrectly annotated data, highly dynamic object representations, and poor spatial and temporal resolution. The examples in this chapter will be based on datasets from the cell tracking (2D and 3D time-lapse cells video sequences) and particle tracking challenges (2D and 3D fluorescent images of receptors and viruses).
• Day 8: Generative Models
  We cover the fundamental techniques behind deep learning-based generative modeling. We will show how to generate microscopy data using state-of-the-art generative models, such as variational autoencoder and generative adversarial neural networks (for example we will employ them to generate realistic images of biological tissues from their corresponding segmentations). We will further demonstrate how these generated images can be used to increase the amount of data available to train neural networks. We will discuss different challenges of generative modeling (e.g., mode collapse, training stability) and provide practical tools to overcome them, especially when generating microscopy images.
• Day 9: Cross-modality Transformations
  We introduce the problem of transforming images between different imaging modalities, and how this enables the creation of images that are not even experimentally possible (i.e., they cannot be acquired on a real setup because of physical limitations). We will demonstrate U-Nets and  conditional generative models (used for virtual staining of cell sub-structures as well as histological slides). We will also discuss how trustworthy the obtained images are in terms of quantitative information content as well as high-frequency information preservation.

On Day 9, we will also have a second follow-up on the student research projects with a discussion of possibilities and alternatives.

Project presentations (end of February 2023)

• Day 10: Presentations by the students of their projects in the frame of a small retreat. Discussion of the achieved results, problems, and possible alternative approaches.