I am a Technical Lead Manager at Facebook Reality Labs Research, leading a research program exploring human cognition and its intersections with futuristic technologies, with a focus on wearable augmented reality (AR) devices.
At FRL, we use basic and applied research to understand how futuristic technologies might extend people’s capabilities. Our work focuses on developing computational interactions that drive adaptive input and user interfaces for future AR devices. You can find a high-level summary of our approach to the problem space in our recent publication on AI for HCI (Jonker et al., 2020).
The Challenge for AR
Wearable AR devices provide the opportunity for an entirely new era of personal computing – one that provides persistent “always-on” assistance. This assistance can integrate seamlessly into a user’s physical world, personalize its behaviour to their goals, and support without being disruptive. However, given that wearable AR devices merge digital content with the real world, these devices run the risk of overwhelming the user, presenting irritating or irrelevant content, or disengaging the user from the real world. To build wearable AR that is truly assistive, we must understand how people attend to, process, and remember information, and design the system around these cognitive abilities.
A Cognitive Approach
Human cognition is a profoundly complex and multidimensional system. Studying this system is challenging because human behaviours are the emergent properties of many interacting cognitive subprocesses. For example, our ability to learn new information is determined by multiple factors, including attention to various features, whether we’ve learned something similar in the past, the strategy we use, fatigue, level of interest, and so on.
To make progress in understanding cognition, we take a multi-pronged approach to research, studying cognition both in the laboratory and under naturalistic conditions using a variety of techniques, including behavioural, eye-tracking, and neuroimaging methodologies. In our work, we focus on building computational models that represent a user’s cognitive processes during interaction with AR, and we leverage cognitive science to study system input and interaction prototypes and improve them. Together, these models and evaluative studies are used to put human cognition at the center of AR development.
Our North Star is to invent a future that naturally extend your ability to think, reason, and ideate without breaking your immersion in the real world.
My research focuses on attention and memory and has employed various methodologies including fMRI, EEG, wearable cameras, and virtual reality with eye-tracking. Throughout my career, I have focused on the role of context in cognitive processing, context being defined as the collection of environmental and internal features that are peripheral to the focal task and goals.
I completed my Masters and PhD at the University of Waterloo, where I studied episodic memory retrieval, associative encoding, and mind-wandering. During my PhD, we proposed and tested a new theory of a well-studied phenomenon, retrieval-induced forgetting. This theory, published in Psychological Review, challenged the dominant approach of memory suppression (Anderson, Bjork, & Bjork, 1994), and placed contextual processing at the center of retrieval-based forgetting phenomena (for a summary, see Jonker, Seli, & MacLeod, 2015).
During my postdoc at the University of California, Davis, I continued focusing on the role of context in memory retrieval. We found evidence that retrieval can strengthen other memories that share contextual features. Our work, published in Proceedings of the National Academy of Sciences, demonstrated that neural regions known for processing contextual information (e.g., parahippocampal gyrus) play a major role in retrieval-based memory enhancements (Jonker et al., 2018). This work complemented my PhD studies by highlighting that contextual information can drive both memory enhancements (retrieval-induced facilitation) and memory impairments (retrieval-induced forgetting) depending on how it is used during retrieval.
At Facebook Reality Labs, my current work continues to center on context and cognition, with a primary focus on visual attention, working memory, and cognitive load. As described above, the bulk of our research examines how human attention and working memory operate in complex AR systems and naturalistic environments. Specifically, we aim to build models of human cognition, which can be used to improve human-computer interaction, and we leverage cognitive science to evaluate systems and improve their design to extend cognitive capabilities.