Brain-computer interface (BCI) is a technology that controls external devices by interpreting EEG signals. In recent years, BCI technology has shown great potential in helping patients with neuromuscular diseases to regain motor function. Such patients often lose the ability to move their upper limbs due to diseases, and being able to control external devices through their minds will undoubtedly greatly improve their quality of life. However, there are still many challenges in the practical application of the traditional BCI system, such as the accuracy of command recognition, the response speed of the system, and the user's operation experience.

In order to overcome the above challenges, this paper proposes a 3D robot arm sharing control system based on asynchronous BCI and computer vision. The system enables the user to select fifteen commands by combining SSVEPs and blink-related electrooculogram (EOG) signals, enabling precise control of the robotic arm in three-dimensional space. The system also uses computer vision technology to assist in identifying and locating target objects, which further improves the efficiency and accuracy of task completion. The experimental results showed that the average accuracy of the system in ten subjects was more than 92%, and all subjects were able to successfully complete the "grab-drink" task, showing better performance than the traditional BCI system.

Research Methods:

System Description

Figure 1. Schematic diagram of the BCI robot control system

The system consists of three main subsystems: a hybrid asynchronous BCI model, a robotic arm, and a computer vision model.

1. Hybrid asynchronous BCI model: This subsystem combines SSVEPs and EOGs and runs in asynchronous mode, allowing users to select corresponding commands to operate the robot arm. The user interface is a 3×5 stimulus matrix, with each button corresponding to a command. The buttons blink at a specific frequency and phase to induce SSVEPs, while the EOG signal is elicited by reducing the button size in a random order, thus synchronizing with the SSVEP stimulus. In order to connect the BCI commands with the movements of the robot arm in 3D space, a new strategy called 3D vector synthesis was designed.

2. Robot arm: The 7-degree-of-freedom (DOF) humanoid robot arm Gen3 produced by Kinova Robotics is used. The working space of the robot arm is a cube with a side length of 0.5 meters, and the hand orientation is fixed in Cartesian space towards the end effector. In order to avoid collision with the desktop, visual boundaries are set.

3. Computer Vision: Capture RGB-D images at 640 resolution × 480 using an Intel RealSense D435 camera fixed to the wrist of the robot arm. The camera is connected to a computer via USB 3.0. When the robot arm successfully moves to the capture area of the target under BCI guidance, the camera detects the QR code (QR code) on the target and provides the position information required by the robot arm subsystem.

Figure 2. (a) The main user interface of the proposed system, with the numbers next to the icons representing the corresponding index for each button. The values displayed on the interface indicate the frequency and phase used by each button during SSVEP stimulation. (b) Diagram of the relationship between the BCI command and the 3D orientation of the end effector of the robot arm.

Figure 3. (a) Diagram of the camera's field of view (FOV) used to detect an object. (b) Schematic diagram of the ROS architecture and flowchart for visually guided robotic arm control.

Experimental setup and data processing

The experiments are divided into offline and online parts, which aim to verify the practicability of the proposed system.

Figure 4. EEG and EOG signal data processing flowchart. At the end of each trial stimulus, these two different data are acquired, and the corresponding processing procedures are executed simultaneously. The FBCCA method consists of a subband filter and a CCA (Typical Correlation Analysis) process, and blink-related waveform detection is performed on the pretreated SSVEP and EOG signals, respectively. By selecting buttons for both data types, decision making can be used to determine the final output.

Offline experiments: Subjects are first trained through offline experiments to familiarize themselves with the system interface and operation methods. In the offline experimental phase, subjects are required to gaze at the flickering stimulus on the screen and blink for a specific period of time. SSVEP and EOG signals were recorded in the experiment, and offline datasets were generated for training classifiers through signal processing and feature extraction. The signal processing steps include:

Pre-processing: Pre-processing of acquired EEG and EOG signals, including denoising, filtering, and normalization.

Feature extraction: Subband filtering and canonical correlation analysis (CCA) methods are used to extract features from SSVEP signals, while blink-related features are extracted from EOG signals by blink detection algorithms.

Classification: A Support Vector Machine (SVM) classifier is trained to classify the extracted features to identify the subject's intent.

Figure 5. A 6-fold cross-validation of the confusion matrix for all subjects using a 2-sec length of data segment in offline training. The color bar indicates the classification accuracy, and the number on the diagonal represents the correct output.

Online experiment: In the online experiment phase, the subject controls the robot arm through a real-time BCI system. The online lab process consists of the following steps:

1. Command selection: The subject fixes on the target command button while blinking at preset intervals. The system collects and processes EEG and EOG signals in real time, recognizes the subject's intent through the trained classifier, and generates corresponding control commands.

2. Robot control: The generated control command is sent to the robot arm subsystem through the TCP/IP protocol, and the robot arm is driven to perform the corresponding actions. The computer vision subsystem monitors the position of the target object in real time and provides position feedback to the robot arm subsystem to ensure that the robot arm can accurately grasp and operate the target object.

3. Task execution: The subject controls the robot arm to complete the "grab-drink" task through the BCI system. Tasks include multi-step operations such as positioning the cup, grabbing the cup, moving to the mouth position, and simulating the action of drinking water. The success of each step is monitored and fed back in real time through a computer vision system.

Analysis of the results of the study

System performance

1. Accuracy and trajectory efficiency: The experimental results showed that ten subjects achieved an average accuracy rate of more than 92% and a high robot motion trajectory efficiency in the experiment. All subjects were able to successfully complete the "grab-and-drink" task using the proposed method, with fewer erroneous commands and shorter completion time than direct BCI control. Here are some of the results:

Table I Results of an online free spelling test using a hybrid asynchronous BCI subsystem

Figure 6. Subject S9 completed the BCI-controlled robotic 3D arrival task in the stepper motion control mode. Three consecutive snapshots (from 1 to 3) show that Subject S9 successfully guided the Gen3 robotic arm from the starting position (1) to the approaching target ping-pong ball (2) and touched the ball (3).

Figure 7. TE (task efficiency) for each subject to complete the robot 3D arrival task. The theoretical value of TE is 1. The one-way repeated-measure ANOVA test showed no significant difference between the three different starting positions (F(2, 57) = 0.25, p = 0.78).

2. System performance comparison: Compared with the traditional direct control method based on BCI, the proposed shared control method has significant advantages in command error rate and task completion time. This suggests that the combination of hybrid asynchronous BCI and computer vision can significantly improve the efficiency and accuracy of complex tasks.

Figure 8. Subject S9 uses a Gen3 robotic arm to drink water. Six consecutive snapshots (1 to 6) show the entire process of the arrival-grab-drink task, in which he successfully guides the robotic arm closer to the bottles (1 and 2), selects the target bottles (3 and 4) by itself, drinks with a straw with the help of computer vision (5), and automatically puts the bottle back on the table by the robot (6).

3. User experience: Through questionnaires and subjective evaluations, the subjects generally believe that the system is easy to operate, low learning cost, and has good stability and reliability in actual use.

It is only used for academic sharing, if there is any infringement, please leave a message and delete the infringement immediately!

This article is from Xinzhi self-media and does not represent the views and positions of Business Xinzhi.If there is any suspicion of infringement, please contact the administrator of the Business News Platform.Contact: system@shangyexinzhi.com