Brain Control: Human-computer Integration Control

Based on Brain-computer Interface Approach

1 1 1 1
Xing-Yu WANG Jing JIN Yu ZHANG Bei WANG
Abstract: Recently, a new system called brain control system has been developed rapidly. Brain control system is
a human-computer integration control system based on brain-computer interface (BCI), which relies on human0 s ideas and
thinking. Brain control system has been successfully applied in wide fields, assisting disabled patients daily life, training
patients with stroke or limb injury, monitoring the state of human operator, as well as entertainment and smart house etc.
In this paper, the background, basic principle, system structure and developments are firstly introduced briefly. The current
research status focusing on the problems of electroencephalograph (EEG) signal pattern, control signal transfer algorithm and
system application is summarized and analyzed in detail. The further research direction and problems are discussed. Finally,
the future development of brain control is analyzed and prospects are given.
Keywords: Brain control, brain-computer interface (BCI), human-computer integration control, electroencephalogram
(EEG)

Recently, a new system based on brain signal has been ity injury, monitoring the mental status for human opera-
developed rapidly. Comparing with the usual systems con- tors, but also in offering new techniques for entertainment,
trolled by computer, this system is controlled by “human smart home, and military applications.
brain”. That is the brain signal is utilized to realize the
system control based on brain-computer interface (BCI)[1] .
The system of such human-computer integration control
1 Brain control system
by brain-computer interface is called brain control system.
“Brain control” relates to several fields including neuro- Human0 s ideas or thinking activity is not in physical form
science, cognitive science, control science, medicine, com- but belonging to psychology. Therefore, it cannot be used
puter science, and psychology etc. It represents a new fron- directly for actual system control. In order to achieve brain
tier as an interdisciplinary research direction and attracts control, a communication and control technique has to be
wide attentions of researchers[2] . developed between human brain and computer or other
The original motivation of brain control is to meet the electronic devices. This technique must have two basis: sci-
requirement of rehabilitation for the patients with amy- entific evidence and technical support.
otrophic lateral sclerosis (ALS), spinal cord injury (SCI) From the control-science point of view, brain is the con-
and other motion function disorders. The brain control sys- trol center of all the functions such as sports and language,
tem helps these patients to operate a wheelchair or robot and gives instructions through peripheral nerves and mus-
by using their brain signal in order to realize some essential cles to the body. Scientists of neuroscience had found
movement function. Therefore, the patients can be partly that even if the peripheral nerves and muscles are bro-
independent from daily care[3] . Besides, the development in ken, the function of brain still could work normally. The
the fields of neuroscience, information science, biomedical instruction performed by brain can be reflected into the
engineering, and especially brain signal and brain function electroencephalograph (EEG). It was also found that EEG
detection techniques, gives scientific evidence and techni- showed some corresponding and rhythmic pattern related to
cal support for brain control. During the past thirty years, human0 s thought or evoked by external stimulus[4] . There-
brain control has been developed rapidly and applied to fore, it is possible to translate the abstract and virtual form
wide fields, not only in assisting daily life for disabled pa- of human intention into actual and physical form of EEG
tients, helping rehabilitation training for stroke and extrem- signal. EEG becomes the “bridge” connecting human brain
and outside devices. The achievements in the field of neu-
roscience reveal the scientific evidence and give the working
Manuscript received November 2, 2012; accepted November 8, 2012
Supported by National Natural Science Foundation of China principle for brain control.
(61074113, 60674089, 60543005, 61203127) and Fundamental Re- In 1875, Richard et al. discovered EEG using rabbit0 s
search Funds for the Central Universities (WH1114038)
Recommended by Academician Lin HUANG scalp. In 1924, Berger recorded the EEG from human0 s
Citation: Xing-Yu Wang, Jing Jin, Yu Zhang, Bei Wang. Brain con- scalp, which is the milestone of the technologies based on
trol: human-computer integration control based on brain-computer
interface approach. Acta Automatica Sinica, 2013, 39 (3): 208−221 EEG. The effective brain activities are always covered in
1. Key Laboratory of Advanced Control and Optimization for EEG, which lead to low signal-to-noise ratio (SNR). It is
Chemical Processes, Ministry of Education, School of Information
science and Technology, East China University of Science and Tech- hard to extract the information of cognitive task from raw
nology, Shanghai 200237, China EEG. In the middle of 20 th century, scientist averaged the
 Xing-Yu WANG et al./ Brain Control: Human-computer Integration Control Based on · · · 209

EEG and extracted the time-locked brain response evoked for BCI. However, due to the cost of the device and the per-
by cognitive task, which was called event-related potential formance of the signal, EEG is mostly used for BCI system.
(ERP). When human received the special external stim- In this paper, we will give a review for EEG-based BCI.
ulus through his or her sensory system, ERP would be At present, the brain control system mainly contains
evoked. In 1962, Galambos et al. reported the averaged three aspects. The first aspect is the relationship between
ERP obtained by computer technique[5] . In 1964, British human0 s thinking or ideas and the EEG signal. We should
psychologist Walter presented the first ERP component know what kind of EEG pattern corresponds to one certain
evoked by cognitive task called contingent negative vari- thought or stimulus. The second is the design and construc-
ation (CNV)[6] . ERP was regarded as the window which tion of brain-computer interface. The third is the develop-
could show the brain0 s activity and has been widely used in ment of application system. The proper systems should be
brain control system. designed and developed according to different tasks or envi-
On the other hand, the development of BCI strengthens ronments. The structural diagram of brain control system
the technique foundation for brain control. BCI is a channel is shown in Fig. 1.
between brain and outside world without using peripheral
nerves and muscles. BCI could translate the brain pattern 2 Overview of BCI development
containing the “mind” of human into control command to
control the external device. The control command reflects In 1973, Vidal et al. from University of California, Los
the information from original brain activity and the inter- Angeles (UCLA) did a research on the communication be-
active feedback between brain and computer. As a commu- tween brain and computer, which is the first step for real
nication system, BCI mainly contains the following parts: BCI system[3] . In 1991, Wolpaw et al., reported that the
signal recording, signal processing and interface between amplitude of mu rhythm could be controlled by subjects
computer and external device. and could be used to control external device (i.e., brain
control)[7] . In 1995, the number of BCI teams was not more
than six. And it became more than twenty in 1999, more
than forty in 2002, more than one hundred in 2007 and even
more until now[8] . There are two important conferences for
BCI, one is held by Jonathan R Walpow every three years
in America called “BCI meeting”. The first conference of
“BCI meeting” was held in June 1999[1] and the second one
in June 2002. There were about thirty BCI teams and one
hundred researchers who joined this meeting[9] . Another
one is held by Gert Pfurtscheller every two years in Graz
Austria. Its fifth conference was held in 2011.
With the development of brain control, some important
research results were obtained. In 1999, Nature published
the paper on BCI for the first time. This work presented
the BCI spelling system based on SCP[10] . In 2003, Na-
ture reported an experiment from Duke University, which
Fig. 1 Structural diagram of brain control system was done by Nicolelis[11] . In this experiment, they guided
a monkey to control robot using its brain signal. In the
There are two kinds of ERP components used for brain same year, Pfurtscheller from Graz University of Technol-
computer interface, which are called exogenous component ogy made a non-invasive restoration of hand grasp function
and endogenous component, respectively. Exogenous com- in a tetraplegic patient by EEG-recording and functional
ponent is evoked by physical stimuli (sound, light, and electrical stimulation (FES) using surface electrodes. After
electric) while endogenous component is related to men- five months0 training, the patient could use this device to
tal activity. BCI could be categorized based on the way of catch a cup and drink[12] .
EEG recordings from the human brain (invasive and non- This kind of brain control system is based on EEG.
invasive). Invasive BCI needs a medical doctor to put elec- The brain patterns recently used for BCI system contain
trodes on the surface of cortex and record the electrocor- slow cortical potential (SCP), motor imagery (MI), steady
ticogram (ECoG), which will involve ethical issues. Non- state visual evoked potential (SSVEP) and so on. SCP
invasive BCI is to put electrodes on the scalp to record and event-related (de)synchronization (ER(D)S) are spon-
the EEG. However, the EEG is usually ten times weaker taneous brain activities related to specific mental tasks,
than ECoG and requires more work to extract the features while SSVEP and P300 are evoked potential caused by ex-
of brain patterns. Beside ECoG and EEG, functional mag- ternal stimuli. SCPs are negative or positive polarizations
netic resonance imaging (fMRI), near-infrared spectroscopy of the EEG or magnetic field changes in the MEG that
(NIRS) and magnetoencephalogram (MEG) were also used usually last from 300 ms to several seconds. Subjects could
210 Acta Automatica Sinica, 2013, Vol. 39, No. 3

be trained to control the SCP by biofeedback. In 1957, reports about BCI. For example, Discovery magazine re-
Brookhartd et al. found the SCP for the first time[13] . In garded BCI as one of the seven most important technologies
1990, Birbaumer et al. described the SCP in detail[14] . In in the world in 2006. Time magazine also reported that BCI
1999, Birbaumer et al. first applied the SCP to BCI and was one of the fifty most valuable inventions in the world
built a spelling system based on the SCP[10] . in 2009.
P300 potential is evoked around 300 ms after low proba-
bility stimulus. If a rare stimulus happened in a sequence of 3 Current developments and main
similar stimuli (i.e. oddball paradigm), a positive deflection
called P300 potential would be evoked at central parietal
problems
area for around 300 ms after this stimulus. P300 mainly ap-
From the view of control science, the problems that need
pears at central vision area (Pz). In 1965, Sutton first found
to be solved for brain control bring new challenges. At
P300 potential[15] . In 1988, Farwell and Donchin developed
present, brain control is still in its primary stage and faces
the first P300 spelling system[16] . A 6 × 6 matrix consisting
many problems that are hard to be solved including ba-
of characters and numbers was used as a stimulus layout, in
sic science and technique problems. The following is the
which every row and column would be flashed once in each
analysis based on current developments and existing main
run. P300 has been widely applied to BCI system. For ex-
problems.
ample, driving based on traffic light[17] , dialling telephone
number[18] , robot arm control[19] , mouse control[20−21] ,
3.1 Research on brain patterns
wheelchair control[22] , robot control[23] , household electri-
cal device control[24] and so on. As mentioned above, several brain patterns were used
In 1977, Professor Pfurtscheller found the phenomenon to build the BCI. However, it is hard to decode ideas and
of ERD by asking the subject to press the button using thoughts from EEG and only a few brain patterns could be
right and left thumbs. This experiment explained the ERD successfully applied to the BCI. Therefore, the research on
phenomenon of mu component in alpha band at the cen- surveying the new brain pattern is very important. Now,
ter of brain area[25] . In 1993, the BCI system based on there are mainly three ways to solve this problem, i.e. im-
ERD was developed and obtained the accuracy of 85 %[26] . proving the evoking paradigm of existing brain patterns,
In 1999, Professor Pfurtscheller found the phenomenon of combining several brain patterns called “hybrid”, and find-
ERS[27] , which mainly occurred at central motion region ing new brain pattern based on neuroscience and cognitive
(C3, C4, and CZ). Now, ERD(S) has been used to develop science.
the system for assisting limb injury patients[12] , controlling Slow cortical potential (SCP) is one of the brain patterns
keyboard[28] , controlling virtual wheelchair[29] and so on. used in BCI. However, the users need to be trained for sev-
Steady state visual evoked potentials (SSVEPs) are nat- eral months before he/she can control the amplitude of SCP.
ural responses to visual stimulus at specific frequencies be- It may be impractical for the real application of BCI. The
tween 3.5 Hz and 75 Hz. When the retina is excited by a ERD (S) or MRPs based on motor imagery is one of the hot
visual stimulus, the brain generates electrical activity at research topics in BCI, which is related to the physiological
the same (or harmonic) frequency of the visual stimulus. phenomenon and has advantage in direction control. The
In 1973, Vidal first introduced SSVEP to BCI[3] . Now, ERD (S) or MRPs can be used to build the BCI system to
SSVEP-based BCI has been mainly applied to computer train the patients who are recovering from the stroke[37−39] .
game[30−31] , mouse control[32] , speller[33] and robot arm However, this kind of signals has drawbacks: unstable, low
control[34] . Movement related potentials (MRPs) appeared recognizing speed, few patterns, long training time, narrow
when subject did the motor imagery of left hand, right group of users and so on.
hand, foot, or tongue. Birch and Levine applied MRPs The advantages of visual evoked potentials (VEP) are
to offline BCI system, respectively[35−36] . rich coded commands, stable pattern, high classification ac-
Since BCI is interdisciplinary technology, researchers on curacy and high information transfer rate, which make VEP
brain control are with different background and do the re- become a hot research topic in BCI. Bin and Liu et al. did
search with different perspective. Researchers who are ma- much work on SSVEP and presented many new methods
jor in neurophysiological, neural information engineering, for SSVEP-based BCI. For example, they applied canonical
rehabilitation medicine, biomedical engineering, control sci- correlation analysis (CCA) to SSVEP recognition[40−41] and
ence and engineering, electronic engineering, instrumenta- used SSVEP-based BCI to dial telephone number. Zhang
tion, machine learning, mathematics commit themselves to et al., surveyed new paradigm for SSVEP and developed
the research of brain control which has been a hot topic. a system using coded multi-frequency with few frequencies
Ten years ago, our team with the background of control to realize multi-target control[42] . Jin et al., combined the
science and engineering started to do the research on brain time and frequency to control multi-target with few frequen-
control. Now, beside some famous Universities and insti- cies and decreased the interference of neighbor stimuli[43] .
tutes, there are also some big companies that invested for However, SSVEP may be affected by the form, configura-
developing BCI system. Furthermore, there are many news tion and sequence of stimulus, and also by the human0 s
 Xing-Yu WANG et al./ Brain Control: Human-computer Integration Control Based on · · · 211

sensory capacity. The VEP is not suitable for blind users. bor flashes and adaptive strategy was used to increase the
Although auditory sense and somatosensory could be used information transfer rate. The method for avoiding dou-
by blind users, they are relatively weak and hard to be ble flashes and the method for decreasing the interference
classified[44−45] . from neighbor flashes were integrated together[55] . The op-
P300 potential has wide group of users, could be classi- timized interval of flashes was used to improve P300-based
fied with high accuracy, has plenty of coded orders and need BCI system and increase the quality of evoked potential,
no training. However, there are still some unsolved prob- which would decrease the overlap of the potentials[66] . Jin
lems for P300-based BCI. The paradigm for evoking P300 et al. developed the Chinese characters input BCI system
potential will affect the quality of evoked potential, and with T9 Chinese input method to realize the communica-
hence classification accuracy. Now, the researches mainly tion between computer and mobile phone[67] .
focus on the delay of flash time[46−52] , the form of stim- Multi-ERP and Hybrid BCI could improve the stability
uli (stimuli matrix[53−54] , configuration of stimuli[51, 55−56] ), and flexibility of BCI, and also enlarge the group of users.
the characteristics of stimuli [46, 57−58] and the sequence of In 2011, Kaufmann et al.[68] used the face picture as stim-
stimuli[59−61] and so on. ulus and their results showed that the familiar face could
In 2002, Gonsalvez and Polich found that target to improve the classification accuracy of BCI. In 2010, Alli-
target interval (TTI) will affect the amplitude of P300 son et al. combined the motor imagery and SSVEP to im-
potential[47] . Based on previous work, Hill et al.[48] , in 2008 prove the online performance of BCI[69] . In 2011, Brunner
found that using the same TTI without traditional row and compared the SSVEP-based BCI with the hybrid BCI of
column pattern could not obtain the accuracy as high as SSVEP and motor imagery[70] . In the same year, Horki et
that using traditional row and column pattern. They also al. applied the hybrid BCI of SSVEP and motor imagery to
presented a code book to show the interval and sequence robot arm control[71] . In 2012, Allsion et al. applied hybrid
of flashes. Martens et al., found that the overlap of P300 BCI of SSVEP and motor imagery to mouse control using
wave was stronger when using short TTI than using long synchronization control mode[72] .
TTI. If the TTI for the same target is shorter than 500 ms, The Tsinghua University BCI team used moving stimu-
repetition blindness would happen[49−51] . In order to solve lus to replace flashing stimulus and reported that this kind
these problems, reference [52] presented a method to avoid of BCI could be better than P300-based BCI. In 2012, Jin
double flashes. In 2010, McFarland et al.[62] , did the ex- et al. presented a method combining the flash and move-
periment which used different flash frequencies to find the ment characteristics, which showed that it could improve
best flash frequency. The configuration of flashes affects the performance of BCI further[76] . In 2011, Kaufmann et
the information transfer rate for P300-based BCI. The non- al.[68] used familiar face as stimulus and their results showed
target flashes around target flashes will distract the atten- that familiar face could evoke large N400 which would in-
tions of subjects, when the interval between target flashes crease the classification accuracy a lot. Professor Li Yuan-
and non-target flashes is less than 500 ms, which is called Qing combined the P300 and motor imagery and succeeded
attentional blink[50−51] . However, we can decrease the non- in controlling the direction and speed of wheelchair[77] . In
target flashes around target flashes to avoid the attentional 2012, Zhang et al.[78] presented that inverse face could evoke
blink[55] . In 2010, Liu et al.[56] developed the gaze indepen- larger vertex positive potential, (VPP), which was com-
dent P300-based BCI. bined with N170 and P300 to improve the performance
The research on the stimulus sequence is a new topic in of BCI system. After that, Jin et al.[79] compared the
BCI. In 2006, Gerson researched the problem where it was paradigm of shaking face with the paradigm of stable face
required from the subject to search the different picture and their results showed that late ERP could be classified
from a sequence of similar pictures, which would evoke ERP with high accuracy, while it could not improve the classi-
when subject caught the target picture. This paradigm fication accuracy further when it was combined with early
is called rapid serial visual presentation (RSVP)[60] . This ERP.
kind of paradigm will be suitable for the patients who could
not move their eyes. In 2011, Treder et al. improved this 3.2 Control signal translation algorithm
method and applied it to the speller through which the Another critical issue in the BCI researches is how to
severely locked-in patients also could use speller BCI[61] . translate the EEG signals into corresponding control com-
The research team of East China University of Science mands. Algorithm realizing the translation is called con-
and Technology has done a lot of work on BCI[63] . In view trol signal translation algorithm. Brain activities contain
of above problems, Jin et al. used math combination to the information of human0 s ideas and thought. Translation
reduce the flash time for each flash run and increase the algorithm is an important tool to detect these brain activi-
information transfer rate[64] . The goal of this work is to ties. If the BCI is referred as a pattern recognition system,
optimize the large matrix to obtain the best flash time for it will mainly consist of three research aspects: preprocess-
each run and solve the “double flashes” problem without de- ing, feature extraction and classification. Feature extrac-
creasing the speed of BCI system[65] . Optimized stimulus tion and classification are the two crucial factors for BCI
matrix was used to decrease the interference from neigh- performance. In principle, most methods that have been
212 Acta Automatica Sinica, 2013, Vol. 39, No. 3

introduced for feature extraction and classification in ma- tering algorithms. CSP extracts features having the max-
chine learning could be applied to the BCI. However, the imal variance difference between two classes; FC extracts
application of these methods should be fit to the partic- features having the maximal discriminant information be-
ularities of the EEG signals and the BCI system. Such tween two classes; CCA extracts features having the max-
particularities are mainly reflected by the three facts: 1) imal correlation within the same class. Note, the features
the EEG signals are very weak and easy to be affected by extracted by the aforementioned algorithms may not be the
other electrical signals, such as electrocardiogram (ECG), real source signals in EEG, but the features fitting to the
electromyography (EMG) and electrooculography (EOG); constraints by these algorithms.
2) the EEG signals are usually varied among different sub- The spatial filtering algorithms mentioned above are
jects since they are not only closely related to the thoughts basically matrix factorization methods based on two-
or intents, but also affected by the mental state of the in- dimensional subspace. The EEG signals used in a brain
dividuals; 3) the extracted EEG features are closely related control system contain typically multi-way data informa-
to the channel configuration. Therefore, the translation al- tion, such as time domain, frequency domain, spatial (chan-
gorithm for control command should be developed by con- nel) domain, variability among trials, and specificity among
sidering the above particularities in EEG signals. subjects. Simultaneous analysis for the multi-dimensional
3.2.1 Feature extraction algorithm information could be effective to excavate some underlying
characteristics in the EEG signals, and hence assist to clas-
The aim of feature extraction is to extract the informa-
sify features more accurately, which is, however, hard to
tion reflecting certain brain activity. For an EEG-based
be implemented using these matrix factorization methods.
BCI, we should first consider how to extract the most effec-
Thus researches on multiway analysis algorithm are very
tive features and what the characteristics of such features
important for improving the BCIs.
are, so that we can design an appropriate classification al-
A class of multiway algorithms based on tensor analy-
gorithm to classify the features with high performance. At
sis was designed to extract collaboratively more effective
present, the commonly used features in BCI are: temporal
features from multi-dimensional information of the EEG
amplitude, power spectral density (PSD), adaptive autore-
signals. Such multiway analysis process provides neu-
gressive (AAR) parameter, time-frequency representation,
rophysiologically meaningful interpretation for extracted
etc. According to different types of BCIs, the features re-
feature[91−93] . The multi-way algorithms which have been
flecting the most strongly exploited EEG pattern should be
applied to feature extraction in BCIs mainly include: mul-
extracted for the subsequent classification.
tilinear PCA[94] and multilinear CSP[95] for ERD/ERS
The temporal features are usually extracted by using a feature extraction, multiway CCA for SSVEP feature
specific filter with down-sampling that removes the tempo- extraction[96] , etc. Tensor discriminant analysis was in-
ral noises to enhance the signal-to-noise ratio (SNR). The troduced to motor-imagery based BCI and improved the
most popular temporal features in BCIs are amplitude and pattern recognition performance[97] . Zhang et al.[96] devel-
band power (BP). The commonly used filtering methods oped a multiway CCA-based SSVEP recognition algorithm,
contain band-pass filtering[21] , Kalman filtering[80] , moving- which not only optimized the spatial features but also re-
average filtering[78] , etc. In addition, continuous/discrete duced effects resulted from variability among trials, through
wavelet transform (CWT/DWT) is also usually adopted to maximizing alternatively the correlation between the spa-
extract the temporal features from the EEG within a short tial dimension as well as trial dimension of the EEG signals
duration[81−82] . and the reference signals.
Spatial feature extraction, also called spatial filtering, These multiway extension versions outperform the tra-
aims to extract the features with higher SNRs through ditional matrix factorization algorithms in varying degree
multiple channels optimization to improve the classifica- for feature extraction of the corresponding BCIs. Superior
tion performance. Due to the effects of volume conduc- results indicate that the multiway algorithms will likely ex-
tion, the EEG signals recorded from the scalp usually have tract more discriminative features to improve classification
poor spatial resolution[86−87] . Common average reference performance. However, such multiway algorithms still have
(CAR) and Laplace filter are the two simplest and most not been widely applied to BCIs. More systematic stud-
used spatial filtering methods. The spatial filtering algo- ies are required to investigate the advantages and disad-
rithms used frequently for feature extraction include prin- vantages of the multiway algorithms for the EEG feature
ciple component analysis (PCA)[88] , independent compo- extraction further.
nent analysis (ICA)[89] , common spatial pattern (CSP)[90] ,
3.2.2 Classification algorithm
Fisher0 s criterion (FC)[86, 91] , canonical correlation analysis
(CCA)[40−41] , etc. Both PCA and ICA are unsupervised After extracting effective features from the EEG signals,
spatial filtering algorithms. PCA extracts the features that another critical issue is how to select or design an appropri-
are orthogonal to each other with the minimal energy loss, ate classifier for classification. Currently, the classification
while ICA extracts the features that are independent to algorithms which have been used in BCIs mainly are: lin-
each other. CSP, FC, and CCA are supervised spatial fil- ear discriminant analysis (LDA), regularized LDA (RLDA),
 Xing-Yu WANG et al./ Brain Control: Human-computer Integration Control Based on · · · 213

support vector machine (SVM), Bayesian classifier, hidden classes. However, the selected hyperplane in the SVM is the
Markov model (HMM), neural network (NN), and ensemble one that maximizes the margins between the two classes,
classifier, etc. This section will give a brief review for these which is also named maximum margin hyperplane. The
classification algorithms, and provide a basic knowledge for SVM are known to be insensitive to overtraining and have
further analysis. good generalization capacity in small sample size. Cur-
Linear discriminant analysis (LDA)[98−100] is to learn rently, the SVM has been widely applied to various BCIs
an optimal hyperplane separating the features from two and achieved good classification results[108−109] . In addi-
classes. Currently, the LDA has been widely adopted in tion, several variants of the SVM have also been developed
various BCI systems[100−102] since it has relatively low com- and introduced to classification in BCIs, such as the sensor
putational requirement and usually provides good classifi- weighting SVM (sw-SVM) for P300 classification[110] and
cation results. Unfortunately, the LDA is hard to classify fuzzy theory-based SVM for ERD/ERS classification[111] ,
some complex and nonlinear EEG signals accurately, and etc.
also will most probably result in poor generalization perfor- Bayesian classifier assigns to a feature vector the class it
mance for the classifier due to the curse-of-dimensionality belongs to with the highest posterior probability[98] . The
in small sample size setting. Bayesian classification is equivalent to the standard LDA
Regularized LDA (RLDA) tries to enhance the gener- when the two classes are Gaussian with the same covari-
alization capacity of classifier by imposing specified con- ance, while it forms a Bayesian quadratic when the two
straint to penalize the training process in the traditional classes have different Gaussian distributions. Although the
LDA. Several typical RLDAs used in BCIs are Stepwise Bayesian classifier has still not been widely adopted in
LDA (SWLDA)[16] , Shrinkage LDA (SKLDA)[87] , Bayesian BCIs, it has been successfully applied to the ERD/ERS
LDA (BLDA)[24] , and LASSO regression model[103] . The classification[112] .
SWLDA performs feature dimensionality reduction through Hidden Markov model (HMM) is a kind of probabilistic
a combination of forward and backward stepwise analy- automaton that can provide the probability of observing
sis in which the discriminant model is estimated by it- a given sequence of feature vectors, where the probability
eratively adding and removing features from the model usually is Gaussian mixture model for BCI[113] . The HMM
based on statistical tests until the maximal number of has recently been applied to the classification of tempo-
features is achieved. The SWLDA has recently been a ral sequences of BCI features since EEG components used
state-of-the-art classification algorithm used in the P300- to drive BCI have specific time courses[113] . However, the
based BCIs[104−105] . The SKLDA adjusts the estimated HMM still has not been widely applied to BCIs.
covariance matrix to the real one using a shrinkage Multilayer perceptron (MLP) is probably the most popu-
technique[106] to improve the generalization capacity of the lar one neural network algorithm for BCIs. The MLP is uni-
trained classifier. The analytical solution of the shrinkage versal approximator which can approximate any continuous
parameter can be computed according to one method in- function when composed of enough neurons and layers, and
troduced in literature[87] . The SKLDA has been applied to hence has been successfully used in various BCIs[114−115] .
the motor-imagery based BCI[107] and ERP-based BCI[87] However, the MLP is sensitive to overtraining, especially
with outstanding classification performance for small train- with such noisy and non-stationary data as EEG, thereby
ing sample size. The BLDA prevents overfitting to high deteriorating the generalization capacity of the trained clas-
dimensional and possibly noisy datasets using regulariza- sifier to some extent[116] . Thus structures of neurons should
tion whose degree is estimated automatically and quickly be carefully designed when applying the MLP to BCIs.
from the framework of Bayesian analysis[24] . Recently, the At present, most classification schemes adopted in BCIs
BLDA has been successfully applied to the P300-based BCI are based on one single classifier, which may, however,
and outperformed the traditional LDA[3, 67] . The LASSO is hardly achieve the optimal performance for a specific classi-
to penalize the regression weight vector by imposing the l1 - fication issue due to the intrinsic defects of the used classi-
norm (sparsity) regularization on a standard linear regres- fier. To overcome this problem, ensemble learning has been
sion model[103] . Larger regularization parameter will result considered as a feasible approach. The ensemble methods
in more zero entries in the regression weight vector. Zhang used in BCIs are either combination of multiple classifiers
et al.[103] designed a LASSO-based frequency component that have the same type but different model parameters[117] ,
recognition model for SSVEP by exploiting such sparsity. or combination of several types of classifiers[118] . The en-
The LASSO-based model learns the optimal discriminant semble classifier may help to remedy the defects of each
vector to recognize directly the SSVEP component, by im- single classifier by combining multiple classifiers with good
plementing simultaneously penalized linear regression be- diversity, and hence to improve the classification perfor-
tween the EEG signals and reference signals corresponding mance for EEG signals.
to different stimulus frequencies. In summary, although a few characteristic classification
Support vector machine (SVM) is also to find an op- algorithms have been proposed for translation of control sig-
timal hyperplane separating features from different two nal, the most challenging problems, i.e. curse of dimension-
214 Acta Automatica Sinica, 2013, Vol. 39, No. 3

ality, overfitting and low adaptability still have not been ef- successful application in game entertainment may be the
fectively solved. The number of training EEG data acquired most attractive, since the users can operate the mouse, key-
from the calibration stage is usually limited when taking board and control stick directly without muscle activity to
into account the practicability of the BCI system. The realize the collaboration and competition in the game.
EEG feature vectors are generally high dimensional since In terms of practicability, the current BCI development
the EEG data are recorded from multi-point and multi- also faces some other technical problems, beside the afore-
channel. This will most probably result in the so-called mentioned issues about EEG pattern investigation and
curse of dimensionality. That is, it is hard to learn good translation algorithm design. These problems can be sum-
classifier with high generalization capacity using insufficient marized as:
but high-dimensional features. On the other hand, the rel- 1) Improvement of the EEG recording devices. The most
atively complicated classifiers (e.g., MLP) outperform some important problem for BCI is to record the EEG signals
simple classifier (e.g., LDA) in EEG classification with non- effectively, which is the one of key points deciding whether
linear features, whereas they are sensitive to noise and be- the system can work efficiently. EEG recording devices,
come overfitting to the training data with poor generaliza- such as sensors and amplifier, etc. could be innovated to
tion capability. In addition, the EEG features are usually prevent interference from the non-EEG signals and effects of
subject-specific and also vary along with time to some ex- the skull on the EEG signals, and hence enhance the quality
tent. Although the commonly used LDA and SVM gener- of recorded EEG signals. In addition, the practicability
ally work well with low variance, they can hardly keep good and portability of the devices need further improvement.
performance after long time use of BCI due to their poor Recently, several companies have developed portable EEG
adaptability. recording devices[120−121] that make important progress in
We think there are two important aspects which should the practicability of BCI, although these portable devices
be mentioned for designing the translation algorithm: 1) In still require further improvement.
order to improve the practicability of the BCI, the system
2) Security of the brain control system. The personal
calibration time should be kept as short as possible with
safety is the most important factor since the brain control
minimal classification performance degeneration. This re-
system is a high-level human-machine integration system.
quires an effective algorithm to learn a classifier with good
In recent years, a concept of “brain switch” has been pro-
generalization capacity from limited training sample set; 2)
posed to meet the requirement of security, which could play
During long time use of the BCI, the EEG feature signals
an important role in BCI to avoid invalid commands re-
may vary to some extent along with time. The classifier
ceived at the idle state. For instance, some mistakes and
trained at the initial calibration stage will hardly provide
unnecessary commands may bring unexpected danger to the
good classification results for the feature signals recorded
users when they are operating the BCI-based wheelchair or
after long time use. Thus the designed classifier should up-
prosthesis control system. Such things are particularly dan-
date adaptively its model parameters based on the variety
gerous to disabled users since it is difficult for them to stop
of EEG features to achieve accurate classification. To this
the control system immediately by physical motion. Thus,
end, we could either design an ensemble classifier by com-
how to choose a safe and reliable EEG pattern as brain
bining multiple existing classifiers to improve the robust-
switch signal for specific subject is worth researching[122] .
ness to variety of EEG features, or introduce the adaptive
learning[119] into classifier design to enhance the adaptabil- 3) Human-machine adaptability of the brain control sys-
ity and prevent the effects resulting from variety of EEG tem. a) The performance of BCI is usually subject-specific,
features. which should be addressed to improve the universality of
the BCI; b) The status of the users should also be taken
3.3 Application system into account to improve the performance of BCI. The sta-
tus and feelings of the users may affect the performance of
So far, the researches of the brain control system rarely
BCI, since the BCI system is inherently based on human.
focus on the actual application, but still mostly are at the
Therefore, the human factor should be considered when de-
laboratory study stage. However, compared to the early
signing the paradigm and selecting the parameters of BCIs,
studies of offline BCIs, the real-time control and feedback
to develop a highly user-friendly brain control system.
of many BCI systems have been realized recently, and some
4) A unified standard for BCI performance evaluation.
actual application systems have also been developed. The
There is still no complete theoretical framework and reg-
actual applications of BCI are mainly related to medical
ularity for BCI performance evaluation, since the current
field and non-medical field. In the medical field, the BCI
researches on BCIs are still mostly at the laboratory study
aims to re-establish communicative and environmental con-
stage. The current used evaluation standards mainly in-
trol approaches, such as intelligent wheelchair, neural pros-
clude information transfer rate (ITR), system delay and
thesis, virtual speller and humanoid robot, etc. for disabled
response time, mental workload and user friendliness, etc.
users with normal brain function. In the non-medical field,
However, in terms of commercialization, more scientific
the BCI has been applied to the detection of operator func-
standard for BCI performance evaluation is required.
tion state, game entertainment and smart home, etc. The
 Xing-Yu WANG et al./ Brain Control: Human-computer Integration Control Based on · · · 215

4 Prospect ibration method works successfully for most subjects and

avoids the time-consuming procedure of offline calibration
After nearly four decades, the research of brain control for individual effectively.
system has been developed rapidly. The brain control, as an 3) Intelligence of brain control system. The close com-
interdisciplinary new research field, not only has the broad bination between human brain and computer in the brain
development prospects but it also faces many challenges. control system is reflected by the integration of biological
The followings are the prospects of future development for intelligence and artificial intelligence. Hence, the brain con-
brain control system. trol system could be a medium for researches on both the
1) Investigation of novel EEG patterns. a) A feasible biological intelligence and artificial intelligence. Revealing
method for improving BCI is to elicit more discrimina- the mechanism of biological intelligence and exploring the
tive brain activities by optimizing or combining the exist- limitation of artificial intelligence are always the challenges
ing brain patterns; b) Simultaneous evoking through vi- to science. The study on intelligence of brain control system
sual, auditory, and tactile approaches may elicit brain ac- may contribute to solve the aforementioned problems, and
tivity with higher discriminability, thereby improving the provide a new insight in the intelligent control and artificial
BCI performance further; c) Another important approach intelligence machine. On the other hand, improvement and
is to discover novel EEG patterns based on psychophysi- optimization for the brain control system may also benefit
ology, in which psychological factor is regarded as the in- from the intelligence study. For instance, most of the ex-
dependent variable, physiological factor is regarded as the isting BCIs simply implement tasks step-by-step with poor
dependent variable and human is the control object. Af- execution efficiency and low safety. Intellectualization of
ter 50 years research, scientists have discovered various control device could optimize the control procedure to re-
ERP components[123] that are related to attention, per- alize the required tasks from the user with only a few com-
ception, decision and working memory updating, etc., re- mands, and enhance the overall performance of the brain
spectively. Although only several ERP components have control system.
been exploited for BCI development, other more ERP com- 4) Networking of brain control system. Currently, the
ponents also would be applied to BCI in the near future. brain control systems have mainly been developed for indi-
Thus, the research on the psychophysiology could assist to vidual operation. However, a network system is required for
investigate and discover new EEG patterns for developing joint control or remote control. The networking integrates
new BCl paradigm. human brain, machine and network into a control system,
2) Study on new translation algorithms. Effective trans- which may lead to some network control problems on feed-
lation algorithm with capabilities of multiway analysis and back, stability, robustness and delay, and hence affect the
adaptation should be designed by exploiting the new the- status of the user to yield poor classification performance.
ories and methods of the pattern recognition and machine Such complex control system will bring more difficult but
learning[124−125] , to overcome problems caused by the curse valuable research issues.
of dimensionality, overfitting and low adaptability, etc. The 5) Expanded new application fields. The current appli-
multiway analysis methods (consist of feature extraction cation of BCI technique is mainly in the field of rehabilita-
and classification) could provide more real and natural tion. Recently, there is also a direct brain controlled robotic
structure representation for the EEG signals. Specific con- technique developed for robot control field[127] . Actually,
straint or regularization (e.g., independence, maximal cor- the BCI technique not only brings benefits to disabled pa-
relation, highest discriminability, and sparsity, etc.) could tients, but also assists normal persons by providing another
be imposed on the corresponding dimension of the multi- “imaginary hand”. Such novel function could fully take
way structure to extract features with reliable neurophysi- advantage of human ability and make the brain control sys-
ological meaning, and hence enhance further robustness of tem more useful. For instance, according to special require-
feature classification. On the other hand, the distribution ments under some working environments, multi-objective
of EEG features is not only subject-specific but also varies and concurrent control operations can also be realized by
along with time to some extent during the real-time applica- brain control when both of hands are occupied, in order to
tion. To overcome these problems, the concept of adaptive save the labor and improve the convenience of operation
learning in the cybernetics could be exploited to optimize effectively.
the classification model, so that it has not only reliable self- 6) Theory and methods on human-machine integra-
learning ability for individual but also high generalization tion system. The brain control system is a very special
capability within group. Combination of multiway analy- human-machine integration system. In view of the con-
sis and adaptive learning may improve the performance of trol mechanism, this system obeys the human0 s “mind” and
the trained classifier further. Thus how to design a multi- “thought”. In terms of the system itself, the human is not
way analysis algorithm with excellent adaptation is consid- only the controller but also the object for controlling while
erably important for BCI research. Recently, Jin et al.[126] the machine is not only the executor of control command
proposed an adaptive calibration method using online LDA but also the guide of control generation. In general con-
with a pre-trained classification model. This adaptive cal- trol system, human is independent from system structure
216 Acta Automatica Sinica, 2013, Vol. 39, No. 3

and the factors related to human usually could be ignored. [9] Vaughan T M , Heetderks W J, Trejo L J, Rymer W Z,
However, with regard to the brain control system, there is Weinrich M, Moore M M, Kübler A, Dobkin B H, Bir-
still no in-depth study about how to model, analyze and baumer N, Donchin E, Wolpaw E W, Wolpaw J R. Brain-
evaluate the human factors, which, however, are worth fur- computer interface technology: a review of the second in-
ther investigating. New theory and approach should be ternational meeting. IEEE Transactions on Rehabilitation
Engineering, 2003, 11(2): 94−109
developed to model and optimize the human-machine in-
tegration system since the traditional control theory and [10] Birbaumer N, Ghanayim N, Hinterberger T, Iversen I,
method cannot be directly applied to solve such problems. Kotchoubey B, Kübler A, Perelmouter J, Taub E, Flor H. A
As Wiener, the founder of cybernetics, wrote: “If a new spelling device for the paralysed. Nature, 1999, 398(6725):
scientific subject has real vitality, the center of interest in 297−298
it must and should shift in the course of years · · · . Thus
it behooves the cyberneticist to move on to new fields and [11] Nicolelis M A L. Brain-machine interfaces to restore motor
to transfer a large part of his attention to ideas which have function and probe neural circuits. Nature Reviews Neuro-
arisen in the developments of the last decade”[128] . Brain science, 2003, 4(5): 417−422
control is generated and developed based on multidiscipline
[12] Pfurtscheller G, Müller G R, Pfurtscheller J, Gerner H J,
interaction, and its further development will certainly un-
Rupp R. “Thought”-control of functional electrical stimu-
dergo new progress remarkably. lation to restore hand grasp in a patient with tetraplegia.
Neuroscience Letters, 2003, 351(1): 33−36
Acknowledgement
[13] Brookhart J M, Arduini A, Mancia M, Moruzzi G. Slow
The authors appreciate Academician Lin Huang et al. for cortical potentials induced by thalamic stimulation. Bollet-
their valuable comments and advices. tino Della Societa Italiana di Biologia Sperimentale, 1957,
33(10−11): 1629−1630
References
[14] Birbaumer N, Elbert T, Canavan A G, Rockstroh B. Slow
[1] Wolpaw J R, Birbaumer N, Heetderkd W J, McFarland potentials of the cerebral cortex and behavior. Physiological
D J, Peckham P H, Schalk G, Donchin E, Quatrono L Reviews, 1990, 70(1): 1−41
A, Robinson C J, Vaughan T M. Brain-computer inter-
face technology: a review of the first international meeting. [15] Sutton S, Braren M, Zubin J, John E R. Evoked-
IEEE Transactions on Rehabilitation Engineering, 2000, potential correlates of stimulus uncertainty. Science, 1965,
8(2): 164−173 150(3700): 1187−1188

[2] Wang X Y. Automatic control: virtuality vs. reality. Acta [16] Farwell L A, Donchin E. Talking off the top of your head:
Automatica Sinica, 2002, 28 (Suppl): 77−84 toward a mental prosthesis utilizing event-related brain po-
tentials. Electroencephalography and Clinical Neurophysi-
[3] Vidal J J. Toward direct brain-computer communication. ology, 1988, 70(6): 510−523
Annual review of Biophysics and Bioengineering, 1973, 2:
157−180 [17] Bayliss J D, Ballard D H. Single Trial P300 Recognition in
a Virtual Environment. New York: University of Rochester,
[4] Gazzaniga M S, Ivry R, Mangun G R. Cognitive Neuro- 1998
science. New York: W. W. Norton and Company, Inc., 2002
[18] Miller A R, Rosenfeld J P, Soskins M, Jhee M. P300
[5] Galambos R, Sheatz G C. An electroencephalography study amplitude and topography in an autobiographical odd-
of classical conditioning. American Journal of Physiology, ball paradigm involving simulated amnesia. Journal of Psy-
1962, 203(1): 173−184 chophysiology, 2002, 16(1): 1−11

[6] Walter W G, Cooper R, Aldridge V J, McCallum W C, [19] Renterghem V D, Wyns B, Devlaminck, D. Adding machine
Winter A L. Contingent negative variation: an electric sign vision to enhance P300 BCI speller. Advanced Materials
of sensori-motor association and expectancy in the human Research, 2011, 271: 102−107
brain. Nature, 1964, 203(4943): 380−384
[20] Piccione F, Giorgi F, Tonin P, Priftis K, Giove S, Silvoni
[7] Wolpaw J R, McFarland D J, Neat G W, Forneris C A. S, Palmas G, Beverina F. P300-based brain computer in-
An EEG-based brain-computer interface for cursor con- terface: reliability and performance in healthy and paral-
trol. Electroencephalography and Clinical Neurophysiology, ysed participants. Clinical Neurophysiology, 2006, 117(3):
1991, 78(3): 252−259 531−537

[8] Mason S G, Bashshati A, Fatourechi M, Navarro K F, Birch [21] Citi L, Poli R, Cinel C, Sepulveda F. P300-based BCI mouse
G E. A comprehensive survey of brain interface technol- with genetically-optimized analogue control. IEEE Trans-
ogy designs. Annals of Biomedical Engineering, 2007, 35(2): actions on Neural Systems and Rehabilitation Engineering,
137 −169 2008, 16(1): 51−61
 Xing-Yu WANG et al./ Brain Control: Human-computer Integration Control Based on · · · 217

[22] Rebsamen B, Burdet E, Guan C, Zhang H H, Teo C L, Zeng [34] Müller-Putz G R, Pfurtscheller G. Control of an electrical
Q, Ang M, Laugier C. A brain-controlled wheelchair based prosthesis with an SSVEP-based BCI. IEEE Transactions
on P300 and path guidance. In: Proceedings of the 1st on Biomedical Engineering, 2008, 55(1): 361−364
IEEE/RAS-EMBS International Conference on Biomedical
Robotics and Biomechatronics. Pisa, Tuscany, Italy: IEEE, [35] Birch G E, Mason S G. Brain-computer interface research
2006. 1101−1106 at the neil squire foundation. IEEE Transactions on Reha-
bilitation Engineering, 2000, 8(2): 193−195
[23] Bell C J, Shenoy P, Chalodhorn R, Rao R P N. Control of a
humanoid robot by a noninvasive brain-computer interface [36] Levine S P, Huggins J E, BeMent S L, Kushwaha R K,
in humans. Journal of Neural Engineering, 2008, 5(2): 214− Schuh L A, Rohde M M, Passaro E A, Ross D A, Elise-
220 vich K V, Smith B J. A direct brain interface based on
event-related potentials. IEEE Transactions on Rehabilita-
[24] Hoffmann U, Vesin J M, Ebrahimi T, Diserens K. An effi- tion Engineering, 2000, 8(2): 180−185
cient P300-based brain-computer interface for disabled sub-
[37] Daly J J, Cheng R, Roqers J, Litinas K, Hrovat K, Dohring
jects. Journal of Neuroscience Methods, 2008, 167(1): 115−
M. Feasibility of a new application of noninvasive brain
125
computer interface (BCI): a case study of training for re-
[25] Pfurtscheller G, Aranibar A. Evaluation of event-related covery of volitional motor control after stroke. Journal of
desynchronization (ERD) preceding and following volun- Neurologic Physical Therapy, 2009, 33(4): 203−211
tary self-paced movement. Electroencephalography and
[38] Broetz D, Braun C, Weber C, Soekadar S R, Caria A, Bir-
Clinical Neurophysiology, 1979, 46(2): 138−146
baumer N. Combination of brain-computer interface train-
[26] Pfurtscheller G, Flotzinger D, Kalcher J. Brain-computer ing and goal-directed physical therapy in chronic stroke: a
interface — a new communication device for handicapped case report. Neurorehabilitation and Neural Repair, 2010,
persons. Journal of Microcomputer Applications, 1993, 24(7): 674−679
16(3): 293−299
[39] Silvoni S, Ramos-Murquialday A, Cavinato M, Volpato C,
[27] Pfurtscheller G, Lopes da Silva F H. Event-related EEG/ Cisotto G, Turolla A, Piccione F, Birbaumer N. Brain-
MEG synchronization and desynchronization: basic princi- computer interface in stroke: a review of progress. Clinical
ples. Clinical Neurophysiology, 1999, 110(11): 1842−1857 EEG and Neuroscience, 2011, 42(4): 245−252

[40] Bin G Y, Gao X R, Yan Z, Hong B, Gao S K. An online

[28] Scherer R, Müller G R, Neuper C, Graimann B, Pfurt-
multi-channel SSVEP-based brain-computer interface using
scheller G. An asynchronously controlled EEG-based vir-
a canonical correlation analysis method. Journal of Neural
tual keyboard: improvement of the spelling rate. IEEE
Engineering, 2009, 6(4): 046002
Transactions on Biomedical Engineering, 2004, 51(6):
979−984 [41] Lin Z L, Zhang C S, Wu W, Gao X R. Frequency recognition
based on canonical correlation analysis for SSVEP-based
[29] Leeb R, Pfurtscheller G. Walking through a virtual city by
BCIs. IEEE Transactions on Biomedical Engineering, 2006,
thought. In: Proceedings of the 26th Annual International
53(12): 2610−2614
Conference on IEEE Engineering in Medicine and Biology
Society. San Francisco: IEEE, 2004. 4503−4506 [42] Zhang Y S, Xu P, Liu T J, Hu J, Zhang R, Yao D Z. Mul-
tiple frequencies sequential coding for SSVEP-based brain-
[30] Lalor E C, Kelly S P, Finucane C, Burke R, Smith R, Reilly
computer interface. PLoS One, 2012, 7(3): e29519
R B, McDarby G. Steady-state VEP-based brain-computer
interface control in an immersive 3D gaming environment. [43] Jin J, Zhang Y, Wang X Y. A novel combination of time
EURASIP Journal on Applied Signal Processing, 2005, 19: phase and EEG frequency components for SSVEP-based
3156−3164 BCI. In: Proceedings of the 18th international confer-
ence on Neural Information Processing. Berlin, Heidelberg:
[31] Martinez P, Bakardjian H, Cichocki A. Fully online multi-
Springer, 2011. 273−278
command brain-computer interface with visual neurofeed-
back using SSVEP paradigm. Computational Intelligence [44] Belitski A, Farquhar J, Desain P. P300 audio-visual speller.
and Neuroscience, 2007 2007: 1−9 Journal of Neural Engineering, 2011, 8(2): 025022

[32] Trejo L J, Rosipal R, Matthews B. Brain-computer inter- [45] Muller-Putz, Scherer R, Neuper C, Pfurtscheller G. Steady-
faces for 1-D and 2-D coursor control: designs using voli- state somatosensory evoked potentials: suitable brain sig-
tional control of the EEG sperctrum or steady-state visual nals for brain-computer interfaces? IEEE Transactions
evoked potentials. IEEE Transactions on Neural Systems on Neural Systems and Rehabilitation Engineering, 2006,
and Rehabilitation Engineering, 2006, 14(2): 255−229 14(1): 30−37

[33] Firman O, Lüth T, Volosyak I, Graser A. Spelling with [46] Martens S M M, Hill N J, Farquhar J, Schölkopf B. Overlap
steady-state visual evoked potentials. In: Proceedings of the and refractory effects in a brain-computer interface speller
3rd International IEEE/EMBS Conference on Neural Engi- based on the visual P300 event-related potential. Journal
neering. Kohala Coast, Hawaii, USA: IEEE, 2007. 354−357 of Neural Engineering, 2009, 6(2): 026003
218 Acta Automatica Sinica, 2013, Vol. 39, No. 3

[47] Gonsalvez C L, Polich J. P300 amplitude is determined [60] Bigdely-Shamlo N, Vankov A, Ramirez R R, Makeig S.
by target-to-target interval. Psychophysiology, 2002, 39(3): Brain activity-based image classification from rapid serial
388−396 visual presentation. IEEE Transactions on Neural Systems
and Rehabilitation Engineering, 2008, 16(5): 432−441
[48] Hill N J, Farquhar J, Martens S, Biebetamann F, Schölkopf
B. Effects of stimulus type and of error-correcting code de- [61] Treder M S, Schmidt N M, Blankertz B. Gaze-independent
sign on BCI speller performance. In: Proceedings of the brain-computer interfaces based on covert attention and
2008 Conference of Advances in Neural Information Pro- feature attention. Journal of Neural Engineering, 2011,
cessing Systems 21. Cambridge, MA, USA: MIT Press, 8(6): 066003
2009. 665− 672
[62] McFarland D J, Sarnacki W A, Townsend G, Vaughan
[49] Woldorff M G. Distortion of ERP averages due to overlap
T, Wolpaw J R. The P300-based brain-computer interface
from temporally adjacent ERPs: analysis and correction.
(BCI): effects of stimulus rate. Clinical Neurophysiology,
Psychophysiology, 1993, 30(1): 98−119
2010, 122(4): 731−737
[50] Kranczioch C, Debener S, Maye A, Engel A K. Temporal
dynamics of access to consciousness in the attentional blink. [63] Wang Xing-Yu. Human-computer integration control based
NeuroImage, 2007, 37(3): 947−955 on EEG signal. In: Proceedings of the 23rd Chinese Control
Conference. Shanghai: Press of East China University of
[51] Fazel-Rezai R. Human error in P300 speller paradigm Science and Technology, 2004. 13−14
for brain-computer interface. In: Proceedings of the 29th
Annual International Conference on IEEE Engineering in [64] Jin J, Horki P, Brunner C, Wang X Y, Neuper C,
Medicine and Biology Society. Lyon, France: IEEE, 2007. Pfurtscheller G. A new P300 stimulus presentation pat-
2516−2519 tern for EEG-based spelling systems. Biomedizinische Tech-
nik/Biomedical Engineering, 2010, 55(4): 203−210
[52] Townsend G, LaPallo B K, Boulay C B, Krusienski D J,
Frye G E, Hauser C K, Schwartz N E, Vaughan T M, Wol- [65] Jin J, Allison B Z, Sellers E W, Bruner C, Horki P, Wang
paw J R, Sellers E W. A novel P300-based brain-computer X Y, Neuper C. Optimized stimulus presentation patterns
interface stimulus presentation paradigm: moving beyond for an event-related potential EEG-based brain computer
rows and columns. Clinical Neurophysiology, 2010, 121(7): interface. Medical and Biological Engineering and Comput-
1109−1120 ing, 2011, 49(2): 181−191

[53] Sellers E W, Krusienski D J, McFarland D J, Vaughan

[66] Jin J, Sellers E W, Wang X Y. Targeting an efficient target-
T M, Wolpaw J R. A P300 event-related potential brain-
to-target interval for P300 speller brain-computer inter-
computer interface (BCI): the effects of matrix size and in-
faces. Medical and Biological Engineering and Computing,
ter stimulus interval on performance. Biological Psychology,
2012, 50(3): 289−296
2006, 3(3): 242−252

[54] Shishkin S L, Ganin I P, Kaplan A Y. Event-related po- [67] Jin J, Allison B Z, Brunner C, Wang B, Wang X,
tentials in a moving matrix modification of the P300 brain- Zhang J, Neuper C, Pfurtscheller G. P300 Chinese in-
computer interface paradigm. Neuroscience Letters, 2011, put system based on Bayesian LDA. Biomedizinische Tech-
496(2): 95−99 nik/Biomedical Engineering, 2010, 55(1): 5−18

[55] Jin J, Allison B Z, Sellers E W, Brunner C, Horki P, Wang [68] Kaufmann T, Schulz S M, Grünzinger C, Kübler A. Flash-
X Y, Neuper C. An adaptive P300-based control system. ing characters with famous faces improves ERP-based
Journal of Neural Engineering, 2011, 8(3): 036006 brain-computer interface performance. Journal of Neural
Engineering, 2011, 8(5): 056016
[56] Liu Y, Zhou Z T, Hu D W. Gaze independent brain-
computer speller with covert visual search tasks. Clinical [69] Allison B Z, Brunner C, Kaiser V, Müller-Putz G, Neu-
Neurophysiology, 2010, 112(6): 1127−1136 per C, Pfurtscheller G. Toward a hybrid brain-computer
interface based on imagined movement and visual atten-
[57] Takano K, Komatsu T, Hata N, Nakajima Y, Kansaku K.
tion. Journal of Neural Engineering, 2010, 7(2): 026007
Visual stimuli for the P300 brain-computer interface: a
comparison of white/gray and green/blue flicker matrices.
[70] Brunner C, Allison B Z, Altstätter C, Neuper C. A com-
Clinical Neurophysiology, 2009, 120(8): 1562−1566
parison of three brain-computer interfaces based on event-
[58] Salvaris M, Sepulveda F. Visual modifications on the P300 related desynchronization, steady state visual evoked po-
speller BCI paradigm. Journal of Neural Engineering, 2009, tentials, or a hybrid approach using both signals. Journal
6(4): 046011 of Neural Engineering, 2011, 8(2): 025010

[59] Gerson A D, Parra L C, Sajda P. Cortically Coupled com- [71] Horki P, Solis-Escalante T, Neuper C, Müller-Putz G. Com-
puter vision for rapid image search. IEEE Transactions bined motor imagery and SSVEP based BCI control of a 2
on Neural Systems and Rehabilitation Engineering, 2006, DoF artificial upper limb. Medical and Biological Engineer-
14(2): 174−179 ing and Computing, 2011, 49(5): 567−577
 Xing-Yu WANG et al./ Brain Control: Human-computer Integration Control Based on · · · 219

[72] Allison B Z, Brunner C, Altstätter C, Wagner I C, Griss- [85] Wu Z H, Yao D H. Frequency detection with stability co-
mann S, Neuper C. A hybrid ERD/SSVEP BCI for contin- efficient for steady-state visual evoked potential (SSVEP)-
uous simultaneous two dimensional cursor control. Journal based BCIs. Journal of Neural Engineering, 2008, 5(1): 36−
of Neuroscience Methods, 2012, 209(2): 299−307 43

[73] Guo F, Hong B, Gao X R, Gao S K. A brain-computer in- [86] Pires G, Nunes U, Castelo-Branco M. Statistical spatial fil-
terface using motion-onset visual evoked potential. Journal tering for a P300-based BCI: tests in able-bodied, and pa-
of Neural Engineering, 2008, 5(4): 477−485 tients with cerebral palsy and amyotrophic lateral sclerosis.
Journal of Neuroscience Methods, 2011, 195(2): 270−281
[74] Liu T, Goldberg L, Gao S K, Hong B. An online brain-
computer interface using non-flashing visual evoked poten- [87] Blankertz B, Lemm S, Treder M, Haufe S, Muller K R.
tials. Journal of Neural Engineering, 2010, 7(3): 036003 Single-trial analysis and classification of ERP components
— A tutorial. NeuroImage, 2011, 56(2): 814−825
[75] Hong B, Guo F, Liu T, Gao X R, Gao S K. N200-speller us-
ing motion-onset visual response. Clinical Neurophysiology, [88] Boye A T, Kristiansen U Q, Billinger M, Nascimento O F,
2009, 120(9): 1658−1666 Farina D. Identification of movement-related cortical po-
tentials with optimized spatial filtering and principal com-
[76] Jin J, Allison B Z, Wang X Y, Neuper C. A combined
ponent analysis. Biomedical Signal Processing and Control,
brain computer interface based on P300 potentials and
2008, 3(4): 300−304
motion-onset visual evoked potentials. Journal of Neuro-
science Methods, 2012, 205(2): 265−276 [89] Xu N, Gao X R, Hong B, Miao X B, Gao S K, Yang F S.
BCI competition 2003-data set IIb: enhancing P300 wave
[77] Long J Y, Li Y Q, Wang H T, Yu T Y, Pan J H, Li F. A
detection using ICA-based subspace projections for BCI ap-
hybrid brain computer interface to control the direction and
plications. IEEE Transactions on Biomedical Engineering,
speed of a simulated or real wheelchair. IEEE Transactions
2004, 51(6): 1067−1072
on Neural Systems and Rehabilitation Engineering, 2012,
20(5): 720−729
[90] Wang X H. Optimizing spatial filters for single-trial EEG
classification via a discriminant extension to CSP: the
[78] Zhang Y, Zhao Q B, Jin J, Wang X Y, Cichocki A. A
Fisher criterion. Medical and Biological Engineering and
novel BCI based on ERP components sensitive to configural
Computing, 2011, 49(9): 997−1001
processing of human faces. Journal of Neural Engineering,
2012, 9(2): 026018
[91] Cichocki A, Zdunek R, Phan A H, Amari S. Nonnega-
tive Matrix and Tensor Factorizations: Applications to Ex-
[79] Jin J, Allison B Z, Kaufmann T, Kübler A, Zhang Y, Wang
ploratory Multi-Way Data Analysis and Blind Source Sep-
X Y, Cichocki A. The changing face of P300 BCIs: a com-
aration. Chichester, UK.: John Wiley and Sons, 2009
parison of stimulus changes in a P300 BCI involving faces,
emotion, and movement. PLoS ONE, 7(11): e49688
[92] Cichocki A, Washizawa Y, Rutkowski T, Bakardjian H,
[80] Vedel-Larsen E, FugløJ, Channir F, Thomsen C E, Sorensen Phan A H, Choi S, Lee H, Zhao Q B, Zhang L Q, Li Y
H B D. A comparative study between a simplified Kalman Q. Noninvasive BCIs: multiway signal-processing array de-
filter and sliding window averaging for single trial dynami- compositions. IEEE Computer, 2008, 41(10): 34−42
cal estimation of event-related potentials. Computer Meth-
[93] Kolda T G, Bader B W. Tensor decompositions and appli-
ods and Programs in Biomedicine, 2010, 99(3): 252−260
cations. Siam Review, 2009, 51(3): 455−500
[81] Bostanov V. BCI competition 2003-Data sets Ib and IIb:
feature extraction from event-related brain potentials with [94] Washizawa Y, Higashi H, Rutkowski T, Tanaka T, Cichocki
the continuous wavelet transform and the T-value scalo- A. Tensor based simultaneous feature extraction and sam-
gram. IEEE Transactions on Biomedical Engineering, 2004, ple weighting for EEG classification. In: Proceedings of the
51(6): 1057−1061 17th International Conference on Neural Information Pro-
cessing. Berlin, Heidelberg: Springer, 2010. 25−33
[82] Sellers E W, Donchin E. A P300-based brain-computer in-
terface: initial tests by ALS patients. Clinical Neurophysi- [95] Zhao Q B, Zhang L Q, Cichocki A. Multilinear general-
ology, 2006, 117(3): 538−548 ization of common spatial pattern. In: Proceedings of the
2009 IEEE International Conference on Acoustics, Speech
[83] Gao X R, Xu D F, Cheng M, Gao S K. A BCI-based envi- and Signal Processing. Washington, DC, USA: IEEE, 2009.
ronmental controller for the motion-disabled. IEEE Trans- 525 −528
actions on Neural Systems and Rehabilitation Engineering,
2003, 11(2): 137−140 [96] Zhang Y, Zhou G X, Zhao Q B, Onishi A, Jin J, Wang X Y,
Cichocki A. Multiway canonical correlation analysis for fre-
[84] Schlögl A, Lee F, Bischof H, Pfurtscheller G. Characteriza- quency components recognition in SSVEP-based BCIs. In:
tion of four-class motor imagery EEG data for the BCI- Proceedings of the 18th International Conference on Neural
competition 2005. Journal of Neural Engineering, 2005, Information Processing. Berlin, Heidelberg: Springer, 2011.
2(4): 14−22 287−295
220 Acta Automatica Sinica, 2013, Vol. 39, No. 3

[97] Li J, Zhang L Q, Tao D C, Sun H, Zhao Q B. A prior [110] Jrad N, Congedo M, Phlypo R, Rousseau S, Flamary R,
neurophysiologic knowledge free tensor-based scheme for Yger F, Rakotomamonjy A. sw-SVM: sensor weighting sup-
single trial EEG classification. IEEE Transactions on Neu- port vector machines for EEG-based brain-computer inter-
ral Systems and Rehabilitation Engineering, 2009, 17(2): faces. Journal of Neural Engineering, 2011, 8(5): 056004
107−115
[111] Xu Q, Zhou H, Wang Y J, Huang J. Fuzzy support vec-
[98] Duda R O, Hart P E, Stork D G. Pattern Classification tor machine for classification of EEG signals using wavelet-
(2nd edition). New York: Wiley, 2001 based features. Medical Engineering and Physics, 2009,
31(7): 858−865
[99] Bishop C M. Pattern Recognition and Machine Learning.
New York: Springer, 2006 [112] Lemm S, Schafer C, Curio G. BCI competition 2003-
Data set III: probabilistic modeling of sensorimotor mu
rhythms for classification of imaginary hand movements.
[100] Lotte F, Congedo M, Lécuyer A, Lamarche F, Arnaldi B.
IEEE Transactions on Biomedical Engineering, 2004, 51(6):
A review of classification algorithms for EEG-based brain-
1077−1080
computer interfaces. Journal of Neural Engineering, 2007,
4(2): R1-R13.
[113] Obermaier B, Guger C, Neuper C, Pfurtscheller G. Hid-
den Markov models for online classification of single trial
[101] Pfurtscheller G, Neuper C. Motor imagery and direct
EEG data. Pattern Recognition Letters, 2001, 22(12):
brain-computer communication. Proceedings of the IEEE,
1299−1309
2001, 89(7): 1123−1134
[114] Haselsteiner E, Pfurtscheller G. Using time-dependent neu-
[102] Müller-Putz G R, Scherer R, Brauneis C, Pfurtscheller G. ral networks for EEG classification. IEEE Transactions on
Steady-state visual evoked potential (SSVEP)-based com- Rehabilitation Engineering, 2000, 8(4): 457−463
munication: impact of harmonic frequency components.
Journal of Neural Engineering, 2005, 2(4): 123−130 [115] Palaniappan R. Brain computer interface design using
band powers extracted during mental tasks. In: Proceed-
[103] Zhang Y, Jin J, Qing X Y, Wang B, Wang X Y. LASSO ings of the 2nd International IEEE EMBS Conference on
based stimulus frequency recognition model for SSVEP Neural Engineering. Arlington, VA: IEEE, 2005. 321−324
BCIs. Biomedical Signal Processing and Control, 2012,
7(2): 104−111 [116] Balakrishnan D, Puthusserypady S. Multilayer percep-
trons for the classification of brain computer interface data.
[104] Krusienski D J, Sellers E W, Cabestaing F, Bayoudh S, In: Proceedings of the IEEE 31st Annual Northeast on Bio-
McFarland D J, Vaughan T M, Wolpaw J R. A comparison engineering Conference. Hoboken, New Jersey: IEEE, 2005.
of classification techniques for the P300 speller. Journal of 118−119
Neural Engineering, 2006, 3(4): 299−305
[117] Rakotomamonjy A, Guigue V. BCI competition III:
[105] Krusienski D J, Sellers E W, McFarland D J, Vaughan dataset II-ensemble of SVMs for BCI P300 speller. IEEE
T M, Wolpaw J R. Toward enhanced P300 speller per- Transactions on Biomedical Engineering, 2008, 55(3):
formance. Journal of Neuroscience Methods, 2008, 167(1): 1147−1154
15−21
[118] Lee H, Choi S. PCA + HMM + SVM for EEG pattern
classification. In: Proceedings of the 7th International Sym-
[106] Schäfer J, Strimmer K. A shrinkage approach to large-scale
posium on Signal Processing and Its Applications. Paris,
covariance matrix estimation and implications for func-
France: IEEE, 2003. 541−544
tional genomics. Statistical Applications in Genetics and
Molecular Biology, 2005, 4: Article32
[119] Vidaurre C, Schlogl A, Cabeza R, Scherer R, Pfurtscheller
G. Study of on-line adaptive discriminant analysis for EEG-
[107] Vidaurre C, Kramer N, Blankertz B, Schlogl A. Time
based brain computer interfaces. IEEE Transactions on
domain parameters as a feature for EEG-based brain-
Biomedical Engineering, 2007, 54(3): 550−556
computer interfaces. Neural Networks, 2009, 22(9):
1313−1319 [120] Neurosky: BCl technology grounded in laboratory research
[Online], available: http://company.neurosky.com/univer-
[108] Kaper M, Meinicke P, Grossekathoefer U, Lingner T, Rit- sity/, August 13, 2012
ter H. BCI competition 2003-Data set IIb: support vector
machines for the P300 speller paradigm. IEEE Transactions [121] Emotiv forum thread about the epochs sensors [Online],
on Biomedical Engineering, 2004, 51(6): 1073−1076 available: http: //emotiv.com/ forum/ forum10/ topic697,
August 13, 2012
[109] Thulasidas M, Guan C, Wu J K. Robust classification of
EEG signal for brain-computer interface. IEEE Transac- [122] Pfurtscheller G, Solis-Escalante T. Could the beta rebound
tions on Neural Systems and Rehabilitation Engineering, in the EEG be suitable to realize a “brain-switch”? Clinical
2006, 14(1): 24−29 Neurophysiology, 2009, 120(1): 24−29
 Xing-Yu WANG et al./ Brain Control: Human-computer Integration Control Based on · · · 221

[123] Wei J H, Luo Y J. Principle and Technique of Event- Jing JIN Assistant professor at East
China University of Science and Technol-
Related Brain Potentials. Beijing: Science Press, 2010
ogy. He received his bachelor degree from
School of Information and Electrical Engi-
[124] Zhou J Y, Wang F Y, Zeng D J. Hierarchical dirichlet
neering of Hunan University of Science and
processes and their applications: a survey. Acta Automatica
Technology in 2004, and his Ph. D. degree
Sinica, 2011, 37(4): 389−407 in control theory and control engineering
from East China University of Science and
[125] Wang H Q, Sun F C, Cai Y N, Chen N, Ding L G. On Technology in 2010. His research interest
multiple kernel learning methods. Acta Automatica Sinica, covers brain-computer interface, signal processing, and pattern
2010, 36(8): 1037−1050 recognition.
E-mail: [email protected]
[126] Jin J, Sellers E W, Zhang Y, Daly I, Wang X Y, Cichocki A.
Whether generic model works for rapid ERP-based BCI cal- Yu ZHANG Ph. D. candidate at East
ibration. Journal of Neuroscience Methods, 2013, 212(1): China University of Science and Tech-
94−99 nology. He received his bachelor degree
in electrical engineering and automation
[127] Fu Y F, Wang Y C, Li H Y, Xu B L, Li Y C. Direct from East China University of Science
brain-controlled robot interface technology. Acta Automat- and Technology in 2008, His research in-
ica Sinica, 2012, 38(8): 1229−1246 terest covers machine learning and pat-
tern recognition, multiway signal process-
[128] Wiener N [Author], Hao J R [Translator]. Cybernetics; or, ing, and brain-computer interface. E-mail:
Control and Communication in the Animal and the Ma- [email protected]
chine (2nd edition). Beijing: Science Press, 1963

Bei WANG Associate professor at East

Xing-Yu WANG Professor at East
China University of Science and Technol-
China University of Science and Tech-
ogy. She received her bachelor degree in
nology. He received his bachelor degree
electrical and electronics from East China
in mathematics from Fudan University in
Normal University in 1998, and her Ph. D.
1967, and his Ph. D. degree in industrial
degree in system control from Saga Uni-
automation from East China Chemical In-
versity, Japan in 2009. Her research in-
stitute in 1984. His research interest cov-
terest covers intelligent signal processing
ers brain control and its application, intelli-
and pattern recognition, complex system
gent control, pattern recognition, and con-
and its application in artificial life science. E-mail: bei-
trol theory. Corresponding author of this paper. E-mail: xy-
[email protected]
[email protected]