fMRI signals were traditionally seen as slow and sampled in the order of seconds, but recent technological advances have enabled much faster sampling rates. We hypothesized that high-frequency fMRI signals can capture spontaneous neural activity that index brain states. Using fast fMRI (TR=378ms) and simultaneous EEG in 27 humans drifting between sleep and wakefulness, we found that fMRI spectral power increased during NREM sleep (compared to wakefulness) across several frequency ranges as fast as 1Hz. This fast fMRI power was correlated with canonical arousal-linked EEG rhythms (alpha and delta), with spatiotemporal correlation patterns for each rhythm reflecting a combination of shared arousal dynamics and rhythm-specific neural signatures. Using machine learning, we found that alpha and delta EEG rhythms can be decoded from fast fMRI signals, in subjects held-out from the training set, showing that fMRI as fast as 0.9Hz (alpha) and 0.7Hz (delta) contains reliable neurally-coupled information that generalizes across individuals. Finally, we demonstrate that this fast fMRI acquisition allows for EEG rhythms to be decoded from 3.8s windows of fMRI data. These results reveal that high-frequency fMRI signals are coupled to dynamically varying brain states, and that fast fMRI sampling allows for more temporally precise quantification of spontaneous neural activity than previously thought possible.

The brain exhibits rich oscillatory dynamics that play critical roles in vigilance and cognition, such as the neural rhythms that define sleep. These rhythms continuously fluctuate, signaling major changes in vigilance, but the widespread brain dynamics underlying these oscillations are difficult to investigate. Using simultaneous EEG and fast fMRI in humans who fell asleep inside the scanner, we developed a machine learning approach to investigate which fMRI regions and networks predict fluctuations in neural rhythms. We demonstrated that the rise and fall of alpha (8-12 Hz) and delta (1-4 Hz) power-two canonical EEG bands critically involved with cognition and vigilance-can be predicted from fMRI data in subjects that were not present in the training set. This approach also identified predictive information in individual brain regions across the cortex and subcortex. Finally, we developed an approach to identify shared and unique predictive information, and found that information about alpha rhythms was highly separable in two networks linked to arousal and visual systems. Conversely, delta rhythms were diffusely represented on a large spatial scale primarily across the cortex. These results demonstrate that EEG rhythms can be predicted from fMRI data, identify large-scale network patterns that underlie alpha and delta rhythms, and establish a novel framework for investigating multimodal brain dynamics.

Predictive coding is a theoretical framework that can explain how animals build internal models of their sensory environments by predicting sensory inputs. Predictive coding may capture either spatial or temporal relationships between sensory objects. While the original theory by Rao and Ballard, 1999 described spatial predictive coding, much of the recent experimental data has been interpreted as evidence for temporal predictive coding. Here we directly tested whether the “mismatch” neural responses in sensory cortex are due to a spatial or a temporal internal model. We adopted two common paradigms to study predictive coding: one based on virtual-reality and one based on static images. After training mice with repeated visual stimulation for several days, we performed multiple manipulations, including: 1) we introduced a novel stimulus, 2) we replaced a stimulus with a novel gray wall, 3) we duplicated a trained stimulus, or 4) we altered the order of the stimuli. The first two manipulations induced a substantial mismatch response in neural populations of up to 20,000 neurons recorded across primary and higher-order visual cortex, while the third and fourth ones did not. Thus, a mismatch response only occurred if a new spatial – not temporal – pattern was introduced.

Spatial multiomic profiling has been transforming the understanding of local tumor ecosystems. Yet, the spatial analyses of tumor-immune interactions at systemic levels, such as in liquid biopsies, are challenging. Within the last 10 years, we have longitudinally collected nearly 3,000 patient blood samples for multiplexing imaging of circulating tumor cells (CTCs) and their interactions with white blood cells (WBCs). Multicellular CTC clusters exhibit enhanced metastatic potential. The detection of CTCs and characterization of tumor immune ecosystems are constrained by (1) low frequency of CTCs in blood samples; (2) specific lineages of immune cells are not recognized by limited channels of current imaging methods, (3) reliance on labor-intensive manual analysis slows down the discovery of biomarkers for predicting therapy response and survival in cancer patients. We hypothesize that an AI-powered platform will accelerate the lineage and spatial characterization of tumor immune ecosystems for prognostic evaluations.

Artificial neural networks (ANNs) have been shown to predict neural responses in primary visual cortex (V1) better than classical models. However, this performance often comes at the expense of simplicity and interpretability. Here we introduce a new class of simplified ANN models that can predict over 70% of the response variance of V1 neurons. To achieve this high performance, we first recorded a new dataset of over 29,000 neurons responding to up to 65,000 natural image presentations in mouse V1. We found that ANN models required only two convolutional layers for good performance, with a relatively small first layer. We further found that we could make the second layer small without loss of performance, by fitting individual "minimodels" to each neuron. Similar simplifications applied for models of monkey V1 neurons. We show that the minimodels can be used to gain insight into how stimulus invariance arises in biological neurons.

Preprint: https://www.biorxiv.org/content/early/2024/07/02/2024.06.30.601394

Representation learning in neural networks may be implemented with supervised or unsupervised algorithms, distinguished by the availability of instruction. In the sensory cortex, perceptual learning drives neural plasticity1-13, but it is not known whether this is due to supervised or unsupervised learning. Here we recorded populations of up to 90,000 neurons simultaneously from the primary visual cortex (V1) and higher visual areas (HVAs) while mice learned multiple tasks, as well as during unrewarded exposure to the same stimuli. Similar to previous studies, we found that neural changes in task mice were correlated with their behavioural learning. However, the neural changes were mostly replicated in mice with unrewarded exposure, suggesting that the changes were in fact due to unsupervised learning. The neural plasticity was highest in the medial HVAs and obeyed visual, rather than spatial, learning rules. In task mice only, we found a ramping reward-prediction signal in anterior HVAs, potentially involved in supervised learning. Our neural results predict that unsupervised learning may accelerate subsequent task learning, a prediction that we validated with behavioural experiments.

Preprint: https://www.biorxiv.org/content/early/2024/02/27/2024.02.25.581990

All brain functions in animals rely upon neuronal connectivity that is established during early development. Although the activity-dependent mechanisms are deemed important for brain development and adult synaptic plasticity, the precise cellular and molecular mechanisms remain however, largely unknown. This lack of fundamental knowledge regarding developmental neuronal assembly owes its existence to the complexity of the mammalian brain as cell-cell interactions between individual neurons cannot be investigated directly. Here, we used individually identified synaptic partners from Lymnaea stagnalis to interrogate the role of neuronal activity patterns over an extended time period during various growth time points and synaptogenesis. Using intracellular recordings, microelectrode arrays, and time-lapse imaging, we identified unique patterns of activity throughout neurite outgrowth and synapse formation. Perturbation of voltage-gated Ca channels compromised neuronal growth patterns which also invoked a protein kinase A mediated pathway. Our findings underscore the importance of unique activity patterns in regulating neuronal growth, neurite branching, and synapse formation, and identify the underlying cellular and molecular mechanisms.

Modern algorithms for biological segmentation can match inter-human agreement in annotation quality. This however is not a performance bound: a hypothetical human-consensus segmentation could reduce error rates in half. To obtain a model that generalizes better we adapted the pretrained transformer backbone of a foundation model (SAM) to the Cellpose framework. The resulting Cellpose-SAM model substantially outperforms inter-human agreement and approaches the human-consensus bound. We increase generalization performance further by making the model robust to channel shuffling, cell size, shot noise, downsampling, isotropic and anisotropic blur. The new model can be readily adopted into the Cellpose ecosystem which includes finetuning, human-in-the-loop training, image restoration and 3D segmentation approaches. These properties establish Cellpose-SAM as a foundation model for biological segmentation.

Circulating tumor cells (CTCs) are critical biomarkers for predicting therapy response and survival in breast cancer patients. Multicellular CTC clusters exhibit enhanced metastatic potential, yet their detection and characterization are constrained by low frequency in blood samples and reliance on labor-intensive manual analysis. Advancing these methods could significantly improve prognostic evaluation and therapeutic strategies.Leveraging FDA-approved CellSearch technology and single-cell sequencing, we analyzed 2, 853 blood specimens, longitudinally collected from 1358 patients with advanced cancer (breast, prostate, etc) and other diseases. Integrating machine learning and deep learning tools, we developed a novel CTCpose platform to automate detection and analysis of CTCs, immune cells, and their interactions. Using artificial intelligence (AI)-driven image analysis, we extracted over 270 cellular and nuclear features including intensity, morphometry, fourier shape, gradient/edge, and haralick of cytokeratin, CD45, and DAPI expression patterns, enabling precise characterization of CTCs, white blood cells (WBCs), CTC clusters, and their interactions with immune cells (WBCs).The CTCpose platform enabled automated identification of CTCs, WBCs, homotypic CTC clusters, heterogenous CTC-WBC clusters, and immune cell clusters, providing comprehensive insights into cell morphology, biomarker expression, and spatial organization. These features correlated with patient survival, disease progression, and treatment response. Our findings highlight the clinical significance of CTC-immune cell interactions and dynamic alterations of CTCs (singles and clusters) and underscore their potential in stratifying patients into distinct risk categories.This study demonstrates the transformative potential of deep learning in overcoming limitations of traditional CTC detection methods and integrating imaging data with large cohorts of patient data. By automating and enhancing the analysis of CTC-immune cell interactions, we present a robust framework for developing predictive models with direct clinical relevance. This work opens avenues for personalized treatment strategies, underscoring the impact of AI in advancing precision oncology.Yuanfei Sun, Joshua R. Squires, Andrew Hoffmann, Youbin Zhang, Allegra Minor, Anmol Singh, David Scholten, Chengsheng Mao, Yuan Luo, Deyu Fang, William J. Gradishar, Massimo Cristofanilli, Carsen Stringer, Huiping Liu. Deep learning enables automated detection of circulating tumor cell-immune cell interactions with prognostic insights in cancer [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 2420.

Cognitive maps confer animals with flexible intelligence by representing spatial, temporal and abstract relationships that can be used to shape thought, planning and behaviour. Cognitive maps have been observed in the hippocampus1, but their algorithmic form and learning mechanisms remain obscure. Here we used large-scale, longitudinal two-photon calcium imaging to record activity from thousands of neurons in the CA1 region of the hippocampus while mice learned to efficiently collect rewards from two subtly different linear tracks in virtual reality. Throughout learning, both animal behaviour and hippocampal neural activity progressed through multiple stages, gradually revealing improved task representation that mirrored improved behavioural efficiency. The learning process involved progressive decorrelations in initially similar hippocampal neural activity within and across tracks, ultimately resulting in orthogonalized representations resembling a state machine capturing the inherent structure of the task. This decorrelation process was driven by individual neurons acquiring task-state-specific responses (that is, 'state cells'). Although various standard artificial neural networks did not naturally capture these dynamics, the clone-structured causal graph, a hidden Markov model variant, uniquely reproduced both the final orthogonalized states and the learning trajectory seen in animals. The observed cellular and population dynamics constrain the mechanisms underlying cognitive map formation in the hippocampus, pointing to hidden state inference as a fundamental computational principle, with implications for both biological and artificial intelligence.