Towards machines that see like us: human eye movements for robust deep recurrent neural networks

In recent years, Deep Neural Networks (DNNs) have become the gold standard in computer vision. In neuroscience, DNNs have been suggested as a modelling framework for information processing in the brain, and provide the best models to date for predicting biological neural responses (Khaligh-Razavi & Kriegeskorte, 2014; Yamins et al., 2014). Although the origins of DNNs were inspired by neuroscience, they largely abstract away from biology and therefore miss many important aspects of neural computation. Which of these aspects are relevant for understanding computations in the brain, however, is an open empirical question. In this research proposal, we aim to narrow the gap between artificial and biological networks by integrating two key aspects into DNN models: recurrence, and information sampling via eye-movements.One large difference between current state of the art DNNs and the brain’s neural networks is that the former are mostly feedforward while the latter make heavy use of recurrent computations. The brain’s recurrent backbone of computation allows it to integrate information over time and to flexibly recycle computational resources. Many studies have shown these recurrent computations to be crucial for brain processing (Kietzmann et al., 2019; Lamme & Roelfsema, 2000). Despite this consensus, it remains poorly understood what the exact functional roles of recurrent computations are, and how their power can be harnessed in artificial neural networks.A second important difference is that humans and artificial systems adopt very different strategies to sample inputs. While DNNs commonly process static images in a single feedforward sweep, humans actively sample their environment for information via eye movements at a rate of ~3Hz. These eye movements are not random and serve a crucial computational role: as the environment is explored, evidence is integrated across several eye fixations (Kietzmann, Geuter, & König, 2011). The latter exemplifies the natural synergy between recurrent networks and eye movements, as recurrent connectivity allows integrating information across time points. The integration of biological aspects, here recurrence and eye movements, in state of the art DNNs has two interdisciplinary benefits. First, it will yield better models of information processing in the brain, as the information processing characteristics in artificial and biological systems will be better aligned. Second, it may help alleviate open problems in computer vision, as human vision is highly robust. For example, small image perturbations, unnoticeable to the human eye, can lead DNNs to misclassify inputs (Szegedy et al., 2013). These adversarial examples are an active topic of research in machine learning, and recurrent processing combined with eye movements may be key factors to avoid them. Harnessing recurrence and eye movements may not only provide better models of the human visual system, but also close an important vulnerability in computer vision.We expect our research to impact both neuroscience, via better models of information processing in the brain, and computer vision, by creating more robust computer vision applications. To quantify to what extent our networks improve upon previous models of brain function, we will compare the behaviour and dynamics of information processing of our networks with human data previously collected across a wide range of tasks. To quantify if our networks provide more robust computer vision system, we will test their performance on adversarial examples. The overall project will be carried out in Prof. Kietzmann’s group at the Donders Institute, Nijmegen, Holland, with a three months stay in Prof. Kriegeskorte’s group at Columbia University, New York, USA.