The body of an animal influences how its nervous system generates behavior1. Accurately modeling the neural control of sensorimotor behavior requires an anatomically detailed biomechanical representation of the body. Here, we introduce a whole-body model of the fruit fly Drosophila melanogaster in a physics simulator. Designed as a general-purpose framework, our model enables the simulation of diverse fly behaviors, including both terrestrial and aerial locomotion. We validate its versatility by replicating realistic walking and flight behaviors. To support these behaviors, we develop new phenomenological models for fluid and adhesion forces. Using data-driven, end-to-end reinforcement learning we train neural network controllers capable of generating naturalistic locomotion along complex trajectories in response to high-level steering commands. Additionally, we show the use of visual sensors and hierarchical motor control, training a high-level controller to reuse a pre-trained low-level flight controller to perform visually guided flight tasks. Our model serves as an open-source platform for studying the neural control of sensorimotor behavior in an embodied context.

Preprint: www.biorxiv.org/content/early/2024/03/14/2024.03.11.584515

The body of an animal influences how the nervous system produces behavior. Therefore, detailed modeling of the neural control of sensorimotor behavior requires a detailed model of the body. Here we contribute an anatomically-detailed biomechanical whole-body model of the fruit fly Drosophila melanogaster in the MuJoCo physics engine. Our model is general-purpose, enabling the simulation of diverse fly behaviors, both on land and in the air. We demonstrate the generality of our model by simulating realistic locomotion, both flight and walking. To support these behaviors, we have extended MuJoCo with phenomenological models of fluid forces and adhesion forces. Through data-driven end-to-end reinforcement learning, we demonstrate that these advances enable the training of neural network controllers capable of realistic locomotion along complex trajectories based on high-level steering control signals. We demonstrate the use of visual sensors and the re-use of a pre-trained general-purpose flight controller by training the model to perform visually guided flight tasks. Our project is an open-source platform for modeling neural control of sensorimotor behavior in an embodied context.Competing Interest StatementThe authors have declared no competing interest.

Animals are often bombarded with visual information and must prioritize specific visual features based on their current needs. The neuronal circuits that detect and relay visual features have been well studied. Much less is known about how an animal adjusts its visual attention as its goals or environmental conditions change. During social behaviours, flies need to focus on nearby flies. Here we study how the flow of visual information is altered when female Drosophila enter an aggressive state. From the connectome, we identify three state-dependent circuit motifs poised to modify the response of an aggressive female to fly-sized visual objects: convergence of excitatory inputs from neurons conveying select visual features and internal state; dendritic disinhibition of select visual feature detectors; and a switch that toggles between two visual feature detectors. Using cell-type-specific genetic tools, together with behavioural and neurophysiological analyses, we show that each of these circuit motifs is used during female aggression. We reveal that features of this same switch operate in male Drosophila during courtship pursuit, suggesting that disparate social behaviours may share circuit mechanisms. Our study provides a compelling example of using the connectome to infer circuit mechanisms that underlie dynamic processing of sensory signals.

The Importance Weighted Auto Encoder (IWAE) objective has been shown to improve the training of generative models over the standard Variational Auto Encoder (VAE) objective. Here, we derive importance weighted extensions to Adversarial Variational Bayes (AVB) and Adversarial Autoencoder (AAE). These latent variable models use implicitly defined inference networks whose approximate posterior density qφ(z|x) cannot be directly evaluated, an essential ingredient for importance weighting. We show improved training and inference in latent variable models with our adversarially trained importance weighting method, and derive new theoretical connections between adversarial generative model training criteria and marginal likelihood based methods. We apply these methods to the important problem of inferring spiking neural activity from calcium imaging data, a challenging posterior inference problem in neuroscience, and show that posterior samples from the adversarial methods outperform factorized posteriors used in VAEs.

The mouse embryo has long been central to the study of mammalian development; however, elucidating the cell behaviors governing gastrulation and the formation of tissues and organs remains a fundamental challenge. A major obstacle is the lack of live imaging and image analysis technologies capable of systematically following cellular dynamics across the developing embryo. We developed a light-sheet microscope that adapts itself to the dramatic changes in size, shape, and optical properties of the post-implantation mouse embryo and captures its development from gastrulation to early organogenesis at the cellular level. We furthermore developed a computational framework for reconstructing long-term cell tracks, cell divisions, dynamic fate maps, and maps of tissue morphogenesis across the entire embryo. By jointly analyzing cellular dynamics in multiple embryos registered in space and time, we built a dynamic atlas of post-implantation mouse development that, together with our microscopy and computational methods, is provided as a resource.