Communication with a robot using brain activity from a human collaborator could provide a direct and fast feedback loop that is easy and natural for the human, thereby enabling a wide variety of intuitive interaction tasks.
Using brain signals to control robots could offer exciting possibilities for intuitive human-robot interaction. Although capturing and identifying such signals represents a considerable challenge given current technology, recent research has shown that the error-related potential (ErrP) signal is generated by the brain in response to observing or making a mistake. If this signal could be leveraged to facilitate human-robot control even in a restricted class of situations, it would enable many new applications of natural human-robot collaboration. For example, humans could remotely supervise robots on factory floors and communicate “stop” instantaneously when the robot makes a mistake without needing to type a command or push a button.
Reliably detecting this error-related potential could enable communication via a signal that occurs naturally in the brain during interaction with, or observation of, a collaborating robot. This could potentially alleviate the extensive user training, extra cognitive load, or constant visual stimuli often required by common brain-computer interface (BCI) systems. Due to the inherent difficulty of quickly extracting such signals from a subject’s brain activity using electroencephalography (EEG), studies involving error-related potentials are often performed in controlled settings and for simulated or open-loop tasks. However, robotic applications demand closed-loop scenarios in real-world environments; this paper therefore explores the applicability of EEGmeasured ErrP signals to real-time closed-loop robotic tasks. Towards this end, a feedback system is developed for human-robot collaboration that hinges upon online identification of error-related potentials. In particular, a Rethink Robotics Baxter robot performs object selection while being observed by a human. The human operator’s EEG signals are collected and decoded in real time; if an ErrP signal is detected, the robot immediately corrects its trajectory.
An important result obtained by these experiments is the observation and analysis of interactive error-related potentials, namely secondary errors, generated in online closedloop experiments due to the human’s active participation. They naturally occur in response to a real-time misclassification of the EEG signal, i.e. when the robot does not properly obey the human’s feedback. These secondary errors are often significantly easier to classify than the initial errorrelated potential (primary error). This signal can therefore improve system performance and greatly aid the development of EEG-based closed-loop controllers for robotic tasks; this paper provides preliminary offline analysis towards this goal.
A. Human-Robot Interaction using ErrP Communication
There are many ways that a human can interact with a robot using ErrP signals for communication. The human can be actively included or excluded from the control loop, and the computation can be performed in real time or after the experiment is completed. Specifically:
Closed-Loop implies that the robot and human directly affect each other throughout the task; the behavior of one affects the other. For example, the EEG classifier detects an ErrP and communicates a trajectory correction to Baxter. This adjustment is immediately observed by the human, affecting their behavior and EEG signals, completing the closed loop. Open-Loop implies that the robot performs the task without any feedback from the human. The robot is still observed by the human and EEG data is acquired, but it is not decoded in real time to command the robot. The human’s role may seem similar to the closed-loop scenario, but the lack of collaboration significantly affects the subject’s engagement. Online Performance implies that the EEG classification system operates in real time, as is necessary for closed-loop feedback. To achieve this, the system must acquire EEG data for less than one second and then make a classification decision within milliseconds; longer latencies would deteriorate the effectiveness of closed-loop feedback.
Offline Performance is obtained by running the EEG classification system on pre-recorded EEG data after an open-loop or closed-loop experiment is completed. Offline analysis has no constraint on computation time, so it generally achieves better performance by optimizing the classification pipeline.
Due to the nature of real-world robotic applications, this work focuses on developing an online closed-loop system. EEG data is acquired for brief windows during the task and decoded in a few hundred milliseconds to provide immediate feedback to the robot. 12 subjects participated, only one of which had ever used an EEG system before, and none had been previously trained for the task. One of the subjects was in a meditative state, and the data of that subject was excluded from the present analysis for consistency. The classifier used in online sessions was trained on a single experiment basis using only 50 initial trials. Offline analysis is also presented, with optimal classification parameters using all available data; with advances in computation and hardware, similar results could eventually be achieved online.
B. Paper Contributions
This paper develops and tests an online feedback system for an object selection task where human directives are communicated to the robot via an EEG classification system. In particular, this work presents the following:
- Demonstration of the existence and applicability of error-related potentials to robotic collaboration tasks
- Closed-loop ErrP classification, allowing a human to influence a robot in real time via natural thought patterns and the robot to influence the human’s EEG activity
- System performance results for online and offline analysis of both open-loop and closed-loop scenarios
- Demonstration of secondary ErrPs in closed-loop scenarios, and their offline utilization to boost performance
This work therefore makes progress towards the goal of intuitive human-robot interaction by exploring methods of applying EEG data to real-time robotic control.
Conclusion and Future Work
By focusing on the detection of naturally occurring error-related potential signals, an online closed-loop EEG system has been developed that enables intuitive human-robot interaction even for general population subjects that have not been previously trained on the task or EEG systems. The existence of ErrPs for a real-world robotic application is demonstrated, and a classification pipeline is developed to decode the EEG activity. Once trained offline using a small sample of openloop trials, the pipeline can decode brain signals fast enough to be used online.
Offline analysis also demonstrates the existence of a secondary ErrP when the human observes that the robot has incorrectly interpreted their feedback. This signal is typically easier to classify than the original error, and can thus be used to improve the performance accuracy. In the future, this signal can be incorporated into the online scenario to boost closed-loop performance as well. This also suggests that new paradigms can be designed that exploit the secondary error. For example, the robot can perform motions that are designed to elicit error potentials from the human user to acquire feedback at crucial decision points, when choosing between many options, or even during a continuous trajectory.
In this way, the presented system moves closer towards the goal of creating a framework for intuitive human-robot interaction in real-world tasks.
Acknowledgments
This work was funded in part by the Boeing Company, the NSF Graduate Research Fellowship number 1122374, and CELEST, a NSF Science of Learning Center (NSF SMA0835976), for which the authors express thanks. Gratitude is also expressed towards Bianca Homberg, whose time and effort while laying a foundation for the Baxter interface are much appreciated.
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)