Branson Lab

We can think of video of animals behaving as a raw measure of their behavior. Clearly, much processing is necessary for this data to be scientifically interpretable. Tracking the animals - converting these videos into trajectories of their positions in each frame - goes a long way toward compressing the data into a more interpretable form. Simple statistics such as the average speed of an animal or the fraction of time spent in a certain part of the environment are illuminating, but there is more information in the data than this. What is a good representation derived from these trajectories to illustrate the behavioral effects of a given manipulation? What is a general representation that will efficiently and completely describe the behavioral effects of many scientific experiments? This question is related to the question, what is the behavioral vocabulary of a given model organism?

Our research focuses on using machine vision and learning to answer these questions. We are developing machine vision software for automatically tracking animals and developing algorithms for learning behavior detectors from manually segmented trajectories. Using these, we are encoding ethologists' knowledge of observed behavioral phenotypes. We are also using machine learning to automatically discover new behavioral phenotypes and statistics. Our data mining approaches use other signals of the neural state of an animal as a semi-supervised labeling of behavior, favoring correlations with, e.g., neuronal expression patterns in thousands of transgenic lines, neural recordings in freely behaving animals, and stimuli (or other measures of the environment of the animals) presented to the animals.

For the algorithms we develop, we are also focused on producing software that can be used by biologists without an in-depth knowledge of computer vision techniques. These algorithms must also generalize to many experiments and be accurate and robust enough to form the basis of scientific results.