Imaging large samples at the resolution offered by electron microscopy is typically achieved by sequentially recording overlapping tiles that are later combined to seamless mosaics. Mosaics of serial sections are aligned to reconstruct three-dimensional volumes. To achieve this, image distortions and artifacts as introduced during sample preparation or imaging need to be removed.

In this chapter, we will discuss typical sources of artifacts and distortion, and we will learn how to use the open source software TrakEM2 to correct them.

Optical and electron microscopy have made tremendous inroads toward understanding the complexity of the brain. However, optical microscopy offers insufficient resolution to reveal subcellular details, and electron microscopy lacks the throughput and molecular contrast to visualize specific molecular constituents over millimeter-scale or larger dimensions. We combined expansion microscopy and lattice light-sheet microscopy to image the nanoscale spatial relationships between proteins across the thickness of the mouse cortex or the entire Drosophila brain. These included synaptic proteins at dendritic spines, myelination along axons, and presynaptic densities at dopaminergic neurons in every fly brain region. The technology should enable statistically rich, large-scale studies of neural development, sexual dimorphism, degree of stereotypy, and structural correlations to behavior or neural activity, all with molecular contrast.

Natural physical, chemical, and biological dynamical systems are often complex, with heterogeneous components interacting in diverse ways. We show that graph neural networks can be designed to jointly learn the interaction rules and the structure of the heterogeneity from data alone. The learned latent structure and dynamics can be used to virtually decompose the complex system which is necessary to parameterize and infer the underlying governing equations. We tested the approach with simulation experiments of moving particles and vector fields that interact with each other. While our current aim is to better understand and validate the approach with simulated data, we anticipate it to become a generally applicable tool to uncover the governing rules underlying complex dynamics observed in nature.

The most sophisticated existing methods to generate 3D isotropic super-resolution (SR) from non-isotropic electron microscopy (EM) are based on learned dictionaries. Unfortunately, none of the existing methods generate practically satisfying results. For 2D natural images, recently developed super-resolution methods that use deep learning have been shown to significantly outperform the previous state of the art. We have adapted one of the most successful architectures (FSRCNN) for 3D super-resolution, and compared its performance to a 3D U-Net architecture that has not been used previously to generate super-resolution. We trained both architectures on artificially downscaled isotropic ground truth from focused ion beam milling scanning EM (FIB-SEM) and tested the performance for various hyperparameter settings.

Our results indicate that both architectures can successfully generate 3D isotropic super-resolution from non-isotropic EM, with the U-Net performing consistently better. We propose several promising directions for practical application.

Neurobiologists address neural structure, development, and function at the level of "macrocircuits" (how different brain compartments are interconnected; what overall pattern of activity they produce) and at the level of "microcircuits" (how connectivity and physiology of individual neurons and their processes within a compartment determine the functional output of this compartment). Work in our lab aims at reconstructing the developing Drosophila brain at both levels. Macrocircuits can be approached conveniently by reconstructing the pattern of brain lineages, which form groups of neurons whose projections form cohesive fascicles interconnecting the compartments of the larval and adult brain. The reconstruction of microcircuits requires serial section electron microscopy, due to the small size of terminal neuronal processes and their synaptic contacts. Because of the amount of labor that traditionally comes with this approach, very little is known about microcircuitry in brains across the animal kingdom. Many of the problems of serial electron microscopy reconstruction are now solvable with digital image recording and specialized software for both image acquisition and postprocessing. In this chapter, we introduce our efforts to reconstruct the small Drosophila larval brain and discuss our results in light of the published data on neuropile ultrastructure in other animal taxa.

Determining the spatial organization and morphological characteristics of molecularly defined cell types is a major bottleneck for characterizing the architecture underpinning brain function. We developed Expansion-Assisted Iterative Fluorescence In Situ Hybridization (EASI-FISH) to survey gene expression in brain tissue, as well as a turnkey computational pipeline to rapidly process large EASI-FISH image datasets. EASI-FISH was optimized for thick brain sections (300 μm) to facilitate reconstruction of spatio-molecular domains that generalize across brains. Using the EASI-FISH pipeline, we investigated the spatial distribution of dozens of molecularly defined cell types in the lateral hypothalamic area (LHA), a brain region with poorly defined anatomical organization. Mapping cell types in the LHA revealed nine spatially and molecularly defined subregions. EASI-FISH also facilitates iterative reanalysis of scRNA-seq datasets to determine marker-genes that further dissociated spatial and morphological heterogeneity. The EASI-FISH pipeline democratizes mapping molecularly defined cell types, enabling discoveries about brain organization.

Anatomy of large biological specimens is often reconstructed from serially sectioned volumes imaged by high-resolution microscopy. We developed a method to reassemble a continuous volume from such large section series that explicitly minimizes artificial deformation by applying a global elastic constraint. We demonstrate our method on a series of transmission electron microscopy sections covering the entire 558-cell Caenorhabditis elegans embryo and a segment of the Drosophila melanogaster larval ventral nerve cord.

Determining the spatial organization and morphological characteristics of molecularly defined cell types is a major bottleneck for characterizing the architecture underpinning brain function. We developed Expansion-Assisted Iterative Fluorescence In Situ Hybridization (EASI-FISH) to survey gene expression in brain tissue, as well as a turnkey computational pipeline to rapidly process large EASI-FISH image datasets. EASI-FISH was optimized for thick brain sections (300 µm) to facilitate reconstruction of spatio-molecular domains that generalize across brains. Using the EASI-FISH pipeline, we investigated the spatial distribution of dozens of molecularly defined cell types in the lateral hypothalamic area (LHA), a brain region with poorly defined anatomical organization. Mapping cell types in the LHA revealed nine novel spatially and molecularly defined subregions. EASI-FISH also facilitates iterative re-analysis of scRNA-Seq datasets to determine marker-genes that further dissociated spatial and morphological heterogeneity. The EASI-FISH pipeline democratizes mapping molecularly defined cell types, enabling discoveries about brain organization.

In order to study anatomy of organisms with high-resolution there is an increasing demand to image large specimen in three dimensions (3D). Confocal microscopy is able to produce high-resolution 3D images, but these are limited by its relatively small field of view compared to the size of large biological specimens. To overcome this drawback, motorized stages moving the sample are used to create a tiled scan of the whole specimen. The physical coordinates provided by the microscope stage are not precise enough to allow reconstruction (”Stitching”) of the whole image from individual image stacks.
We developed an algorithm, as well as an ImageJ plug-in, based on the Fourier Shift Theorem that computes all possible translations (x, y, z) between two 3D images at once, yielding the best overlap in terms of the cross correlation measure. Apart from the obvious gain in computation time it has the advantage that it cannot be trapped in local minima as it simply computes all possible solutions. Computing the overlap between two adjacent image stacks is fast (12 seconds for two 512x512x89 images on a Intel ® Core2Duo with 2.2GHz) making it suitable for real time use, i.e. computing the output image during acquisition of the individual image stacks.
To compensate the possible shading- and brightness differences we apply a smooth linear intensity transition between the overlapping stacks. Additionally we extended the to generic 3D registration using gradient based rotation detection on top of the phase correlation method. We demonstrate the performance of our 3D stitching plug-in on several tiled confocal images and show an example of its application for 3D registration.

Fiji is a distribution of the popular open-source software ImageJ focused on biological-image analysis. Fiji uses modern software engineering practices to combine powerful software libraries with a broad range of scripting languages to enable rapid prototyping of image-processing algorithms. Fiji facilitates the transformation of new algorithms into ImageJ plugins that can be shared with end users through an integrated update system. We propose Fiji as a platform for productive collaboration between computer science and biology research communities.