Next-level virtual reality: bringing in brain signal feedback

Srdjan Lesaja, biomedical engineering Ph.D. student.
Srdjan Lesaja, biomedical engineering Ph.D. student.

Dean Krusienski, Ph.D., professor in the Department of Biomedical Engineering in the Virginia Commonwealth University College of Engineering, is investigating how to make virtual reality (VR) environments responsive, in real-time, to the user’s state of mind.

His project to develop techniques that incorporate feedback from brain signals into VR interactions has received a two-year grant from the National Science Foundation and its Cyber-Human Systems and Integrative Strategies for Understanding Neural and Cognitive Systems programs.

“We are trying to estimate a user’s cognitive state while they are performing in an interactive virtual reality environment,” he said. “The ultimate goal is to manipulate the VR experience based on our estimation of the cognitive state.”

For example, someone using VR to relax could experience their environment adjusting to help them reach a deeper state of relaxation.

Another application would be in training, such as for air traffic controllers. In a skilled task training situation, he said, if the VR system could determine the user’s level of stress or workload, it could adjust the difficulty level of the task to make the most efficient use of the training time. “If the task is too hard for them, we could slow it down or make the objects they’re interacting with larger or smaller. If it’s too easy, we can add challenges to the environment.”

He is focusing on collecting feedback from brain signals through electroencephalogram (EEG) monitoring as a way to assess the user’s mental workload or emotional state. As a tool for diagnosing medical conditions, an EEG uses electrodes attached to the scalp to detect electrical activity in the brain. Krusienski is also exploring how to optimally configure EEG sensors so they can be incorporated into VR headsets.

He is collaborating with Yusuke Yamani, Ph.D., assistant professor in the Department of Psychology, Human Factors, at Old Dominion University.

Krusienski, who is also the graduate program director for the Department of Biomedical Engineering, was trained in electrical engineering and has been studying brain-signal analysis since he was a postdoctoral fellow.

Being able to sense a user’s state of mind could provide benefits in a host of situations, such as helping people improve their performances in driving or in health care.

In gaming, he said, if a VR system could receive immediate feedback on the player’s level of excitement, stress or even fear, it could “react in a way that better engages them and helps them enjoy the experience.”

The project is part of his lab’s goal to achieve a greater understanding of the brain. “We try to keep this research as fundamental as possible to increase the likelihood that they’ll be more broadly applicable,” he said.