Local goodness-of-fit measures for neural decoding

State-space models are widely used to relate neural activity to internal states that cannot be observed directly, but assessing whether a given model is appropriate for a dataset remains difficult. Current approaches, such as cross-validation, summarize model fit across an entire recording. They can reveal that a model is imperfect, but not when during the recording it fits poorly. We therefore lack tools for targeted model refinement. Here, we develop local goodness-of-fit measures that ask whether each spike is consistent with the model’s prediction based on all preceding spikes. Each spike, through the tuning of the neuron that fired, points to a range of likely states, just as the prediction does. A spike that narrows a broad prediction is consistent with it; one that points to states the prediction considers unlikely is not. Because the comparison needs only the prediction and the states each spike points to, it also applies to models that decode from unsorted spike waveforms. Using simulations in which model errors are known, we show that two consistency measures, based on the overlap of high-probability regions and on predictive checks, flag such inconsistent spikes, whereas the standard Kullback-Leibler divergence also flags spikes that merely narrow a broad prediction. We then apply the measures to a hippocampal recording from a rat performing a spatial foraging task. In these data, a model assuming continuous movement is expected to fail during immobility, when the hippocampal representation can jump to distant locations, so we compare it with a model that also allows such jumps. As expected, the overlap measure flags more spikes during immobility than during movement under the continuous model, and allowing jumps removes most of the continuous model’s flags. The consistency measures thus show when during a recording a state-space model fits poorly, providing tools for its targeted refinement.