Visual Perception as Retrospective Bayesian Decoding from High- to Low-level Features in Working Memory

When looking at a scene, we typically have a quick and accurate perceptual understanding of its high-level category, for example, a home, office, street, or jungle. We rarely pay attention to the scene's low-level properties such as luminance values at various spots unless we are asked to report them, and even then, we are not very accurate about them. The fact that higher-level properties of a scene are more relevant to our behavior than low-level properties has informed global precedence theories of perception. However, experimental studies of the brain have established that lower-level features in a scene are detected before higher-level features; this result somehow led to the commonly used, but rarely tested, assumption that visual perception follows the same low-to-high-level hierarchy of feature detection. This project attempts to resolve this apparent contradiction by separating feature detection and perception, and by integrating visual perception and working memory, the brain's short-term storage of relevant visual information. The project will provide a new computational framework for understanding perception and memory which challenges traditional theories.

Technically, vision can be viewed as involving both encoding and decoding. Encoding refers to how visual stimuli evoke sensory responses in the brain whereas decoding concerns how these responses eventually lead to the subjective perception of the stimuli. A common assumption of many existing models is that decoding follows the same low-to-high-level hierarchy as encoding, but this was never rigorously tested. Additionally, under natural viewing conditions, the small fovea and frequent saccades introduce delays between sensory encoding of different parts of a scene and perceptual integration of the whole scene, suggesting that working memory must be involved in perceptual decoding; yet previous decoding models do not consider working memory. This project aims to address these issues using psychophysical and computational methods, with the specific goal of elucidating the nature of decoding hierarchy in light of working-memory properties. Specifically, compared with lower-level stimulus features, higher-level features are more invariant and categorical, thus requiring less information to specify and permitting more stable maintenance in noisy working memory. The brain should therefore prioritize decoding of reliable higher-level features and then use them to constrain and improve the decoding of unstable lower-level features in memory (when necessary). The project will test some surprising predictions of this retrospective Bayesian decoding theory and develop a neural network implementation of the theory.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.