ISSN 0362-1197, Human Physiology, 2018, Vol. 44, No. 3, pp. 280–288. © Pleiades Publishing, Inc., 2018. Original Russian Text © M.V. Lukoyanov, S.Yu. Gordleeva, A.S. Pimashkin, N.A. Grigor’ev, A.V. Savosenkov, A. Motailo, V.B. Kazantsev, A.Ya. Kaplan, 2018, published in Fiziologiya Cheloveka, 2018, Vol. 44, No. 3, pp. 53–61.
The Efficiency of the Brain-Computer Interfaces Based on Motor Imagery with Tactile and Visual Feedback M. V. Lukoyanova, c, S. Yu. Gordleevaa, *, A. S. Pimashkina, N. A. Grigor’eva, A. V. Savosenkova, A. Motailoa, V. B. Kazantseva, and A. Ya. Kaplana, b aLobachevskii
Nizhny Novgorod State University, Nizhny Novgorod, Russia b Moscow State University, Moscow, Russia c Nizhny Novgorod State Medical Academy, Nizhny Novgorod, Russia *e-mail:
[email protected] Received April 19, 2017
Abstract—In this study we compared tactile and visual feedbacks for the motor imagery-based brain–computer interface (BCI) in five healthy subjects. A vertical green bar from the center of the fixing cross to the edge of the screen was used as visual feedback. Vibration motors that were placed on the forearms of the right and the left hands and on the back of the subject’s neck were used as tactile feedback. A vibration signal was used to confirm the correct classification of the EEG patterns of the motor imagery of right and left hand movements and the rest task. The accuracy of recognition in the classification of the three states (right hand movement, left hand movement, and rest) in the BCI without feedback exceeded the random level (33% for the three states) for all the subjects and was rather high (67.8% ± 13.4% (mean ± standard deviation)). Including the visual and tactile feedback in the BCI did not significantly change the mean accuracy of recognition of mental states for all the subjects (70.5% ± 14.8% for the visual feedback and 65.9% ± 12.4% for the tactile feedback). The analysis of the dynamics of the movement imagery skill in BCI users with the tactile and visual feedback showed no significant differences between these types of feedback. Thus, it has been found that the tactile feedback can be used in the motor imagery-based BCI instead of the commonly used visual feedback, which greatly expands the possibilities of the practical application of the BCI. Keywords: brain–computer interface, EEG, motor imagery, pattern classification, sensorimotor, mu rhythm, stroke, rehabilitation DOI: 10.1134/S0362119718030088
The brain-computer interface (BCI) makes it possible for a person to train to mentally form specific EEG patterns that can be then transformed into commands for external execution devices. Specifically, the person gains the ability to control these devices and possibility of communication with the outside world directly from the brain without using nerves and muscles [1, 2]. At present, neural interface technologies are mostly required in rehabilitation medicine for patients with severe speech and movement impairments to compensate for the loss of communication and to control exoskeletons and service devices with mind [3–5]. In addition, neural interface technologies for motor recovery by ideomotor training, repeated motor imagery, become more widespread [3, 6, 7]. Recent research has demonstrated that the motor imagery is accompanied by an increase in the cortical and spinal excitability [8], which can be the basis for induction of plastic rearrangements and restoration of the mechanisms of the motor control.
The role of neural interfaces in the ideomotor training is detection of EEG patterns that are associated with motor imagery (motor imagery-based brain–computer interface, MI-BCI) and using it to form feedback on the efficiency of the mental imagery [9]. It has been demonstrated that poststroke patients are able to successfully imagine various movements by paralyzed body parts [10]. However, MI-BCI is quite difficult even for a healthy person, let alone poststroke patients, and, therefore, requires an elaborate design of the training protocols and the MI-BCI contours to provide high engagement of the patient during the ideomotor training and formation of the stable motor imagery skill [11]. One of the most important approaches to optimize MI-BCI is development of the best variants of the feedback [12]. Visual feedback is the most widespread and wellstudied feedback in MI-BCI [13]. It creates an additional load on the visual system, which not only can distract the patient from the motor imagery, but also may create additional patterns in the EEG complicating further classification of the states [14]. Thus,
280
THE EFFICIENCY OF THE BRAIN-COMPUTER INTERFACES
the development and investigation of BCI feedback that is not based on the visual modality seems to be promising. To date several studies that compare different feedback modalities have been carried out [15–18]. The use of the audio feedback modality implies the use of sound stimuli that vary in the loudness and duration and their combination [19–23]. Besides, nonvisual stimuli can have some semantic load [24]. The use of the audio modality for BCI became widespread also in the experiments with localization of sound sources in space [25, 26]. Recently, the formation of the feedback contour that is based on tactile (vibration) stimulation in motor imagery-based BCI has been investigated with the research being performed mainly by the group headed by Millan [27–29]. It was demonstrated that replacing the visual feedback by the vibrotactile feedback did not prevent recording EEG patterns in the MI-BCI [27] and did not negatively affect the classification accuracy [27, 28]. On the other hand, it was noted that this replacement reduced the load on the visual motor activity during tracking of multiple objects [29]. It was also demonstrated that a MI-BCI operator with tactile feedback could pay more attention to the problem and not to the feedback [28], and in [30] MI-BCI users with tactile feedback achieved higher results in the accuracy of classification. In [31], it was demonstrated for six healthy volunteers and two poststroke patients that the tactile feedback significantly increased the accuracy of decoding the motor imagery. We should also note an interesting study that is devoted to the creation of an open source system for rehabilitation, NeuRow, which is a MI-BCI with a virtual reality environment and vibrotactile feedback [32]. In some studies, vibration motors were in the neck region [27, 28]. In addition, some of the studies investigated continuous tactile stimulation as feedback [28, 29]. However, given the variety of works that employ different feedback types in BCI, few studies deal with the comparative tests of different feedback modalities. This study was aimed at comparison of the efficiency of the motor imagery-based BCI with tactile and visual feedback and investigation of the dynamics of the formation of the movement imagery skill in MI-BCI users with tactile feedback. METHODS Five healthy volunteers (three women and two men) at an age of 18–23 years participated in the study. None of the subjects had prior BCI operator experience, and all of them were right-handed (the median of 0.7 points according to the manual asymmetry questionnaire [33]). All the subjects gave their written consent to participate in the study. The experimental protocol was approved by the ethics commitHUMAN PHYSIOLOGY
Vol. 44
No. 3
2018
281
tee of the Institute of Biology and Biomedicine of the Lobachevskii Nizhny Novgorod State University. EEG was recorded using a NVX52 encephalograph (Meditsinskie Komp’yuternye Sistemy, Russia) by 30 Cl/Ag electrodes that were arranged according to the International 10–10 System (FT7, FC5, FC3, FC1, FCz, FC2, FC4, FC6, FT8, T7, C5, C3, C1, Cz, C2, C4, C6, T8, TP7, CP5, CP3, CP1, CPz, CP2, CP4, CP6, TP8, P3, P4, POz) with respect to reference electrodes that were placed on mastoids. The ground electrode was placed on the forehead. The contact resistance for all the electrodes did not exceed 10 kΩ. The EEG was filtered within 1–30 Hz along with 50 Hz Notch filter and digitized at a signal sampling rate of 1000 Hz. The structure of the study. Each subject participated in two experimental sessions. At the beginning of each session, the general condition, activity, and mood of the subject were controlled by the Doskin’s test (the CAM test [34]). During the experiment, the participant was in a comfortable armchair with armrests and a footrest. The experimental session included 10– 15 recordings, during which the participant with short pauses continuously performed a series of mental tasks in response to command pictograms, hand images or background pictures, that were demonstrated by a 19-inch TFT monitor at a distance of 2 m from the eyes. The mental tasks for the subjects were the motor imagery of hand movements and the rest task, during which the subject was advised to relax and to focus on breathing. One recording combined the three types of stimuli, a left hand movement imagery, right hand movement imagery, and rest. The purpose of the first session was to familiarize the subject with the motor imagery technique, with the main part focusing on tactile sensations. A subject was offered to choose a hand movement that, in his/her opinion, was comfortable to the imagery. Fingering and wrist rotation in the radiocarpal joint were given as examples. The duration of the demonstration of the command pictograms was 5 s with an interstimulus interval of 3 s. During the recording, ten stimuli were presented for each mental task. The success in mastering the motor imagery was determined by desynchronization of the sensorimotor rhythm (see the Evaluation of desynchronization of the sensorimotor rhythm section). The subject was considered to successfully master the motor imagery when the power of EEG spectrum (desynchronization) during the motor imagery decreased more than by 50% of the initial level. In the case of several unsuccessful attempts, the procedure was repeated upon changing the imagined movement. If a subject failed in mastering the technique for 2 h, the experimental session was stopped and repeated on a different experimental day using the same protocol. Subjects that successfully mastered the motor imagery technique were admitted to the second
282
LUKOYANOV et al.
session of the experiment where visual and tactile feedback types were added to the BCI contour. In the beginning of the session, feedback types that were used in the experiment were explained to the subject. A vertical green bar from the center of the fixing cross to the edge of the screen was used as visual feedback during the recording for each type of the mental task. The tactile feedback was implemented using vibration motors (flat linear resonance actuators, LRAs) without an eccentric, 3 V, 10 mm in diameter that were placed on the forearms of the right and the left hands and on the back of the subject’s neck to signal on the successful recognition of the EEG patterns of the motor imagery of right hand movement and left hand movement and the rest task, respectively. The vibration motors were fixed on the skin using grip tape. To confirm the correct classification of the state, a 500 ms vibration signal was applied. The confirmation was given in the end of the command (4.5 s after the beginning of the demonstration of the pictogram on the screen) only in the case of its correct recognition. The second experimental session included 12 recordings: 4 recordings for each type of the feedback and 4 recordings without feedback (that were used as the control). The order, in which the feedback types were used, was determined randomly. Each recording had 30 stimuli (10 for each mental task) with a duration of 5 s and interstimulus interval of 3 s. The accuracy of recognition of the commands was recorded as a ratio of the commands that were recognized correctly to the total number of the stimuli. Moreover, a degree of desynchronization of the sensorimotor rhythm was calculated for each command. To analyze the EEG during each state (right hand movement imagery, left hand movement imagery, rest), a time interval that corresponded to the duration of the stimulus demonstration was used. To analyze and classify EEG patterns that corresponded to each mental task, the spectral characteristics of the signal during the selected epochs were used. Pattern classification. To select features that are significant for classification of the EEG patterns, the EEG was filtered within 6–26 Hz with further calculation of the individual spatial CSP filter [35]. The classification was carried out by linear discriminant analysis. To decrease the dimensionality of the data, principal component analysis was used, which was reduced to the calculation of the singular value decomposition of the data matrix. This method without calculation of the covariance matrix and its spectrum is more efficient and stable [36]. To implement the mathematical classification algorithms, Python library tools (CSP from the MNE library [37], Linear Discriminant Analysis from the Sklearn library) were used. The accuracy of classification was represented by the probability of the correct classification that was
calculated as a ratio of tasks that were classified correctly to the total number of the tasks. Evaluation of desynchronization of the sensorimotor rhythm. To evaluate the degree of desynchronization of the sensorimotor rhythm during the motor imagery, patterns that corresponded to the rest task were taken as a reference state. The EEG recording that was obtained was spatially filtered using Surface Laplacian for all the channels [38]. Then, for each signal the power spectral density was estimated with a step of 1 Hz, and the desynchronization was calculated as a difference in the signal powers during the motor imagery and the rest signal that was divided by the signal power corresponding to the rest task. For each channel, a frequency with the maximum desynchronization was selected within 7–16 Hz, and these values were plotted on the map of the surface of the cerebral cortex. Thus, the degree of changes in the power spectral density during the motor imagery, when compared to the rest state, was mapped. The data that were obtained were used to assess the success in mastering the skill of the motor imagery at the first stage of the experiment and to evaluate the effect of the different feedback types on the desynchronization at the second and the third stages. Statistical analysis. To assess the individual peculiarities in the accuracy of the classification of EEG patterns when the different feedback types were used, the Fisher test from the ‘stats’ software package for R 3.3.2 was used [39]. To solve the multiple comparison problem, the Bonferroni correction was used. In all the other cases, ANOVA in the ez package was used [40]. RESULTS AND DISCUSSION All the subjects passed the first session of the experiment with the desynchronization being higher 50% during the motor imagery at the C3 and C4 electrodes. Four subjects had the maximum desynchronization at 12 Hz, and one subject had it at 14 Hz. Most subjects chose fingering for the motor imagery. Nobody wanted to change the type of the movement that was imagined during the experiment. In the second session of the experiment, the classification accuracy for the different feedback types was assessed (Fig. 1). The accuracy of recognition in the classification of the three states (right hand movement, left hand movement, rest) in the BCI without feedback exceeded the random level (33% for the three states) for all the subjects and was rather high, 67.8% ± 13.4% (mean ± SD). It can be seen from Fig. 1 that the visual and tactile feedback in the BCI did not significantly change (F(2, 8) = 0.423; p = 0.669) the mean accuracy of recognition of the mental states for all the subjects (70.5% ± 14.8% for the visual feedback and 65.9% ± 12.4% for the tactile feedback). HUMAN PHYSIOLOGY
Vol. 44
No. 3
2018
THE EFFICIENCY OF THE BRAIN-COMPUTER INTERFACES
Probability of correct classification, %
100
80
60
40
20
0
Without feedback
Visual/Tactile feedback
Fig. 1. Probability of the correct classification during simultaneous recognition of the three classes: right hand motor imagery, left hand motor imagery, and rest task. The level of the random recognition is denoted by a dotted line. Mean values for subjects (n = 5) are given.
Table 1 gives the results of evaluation of the classification accuracy for the different feedback types for all the subjects. The classification accuracy is given in percent. It was calculated as a ratio of commands that were classified correctly to the total number of the commands. The differences were considered signifi-
283
cant at p < 0.05 (the Fisher test with the Bonferroni correction). It can be seen from the table that there were no differences in the classification accuracy observed for the different feedback types in three out of five subjects. For one of the subjects (OL), the use of the tactile feedback resulted in a decrease in the classification accuracy. For another subject (DB) the tactile feedback made it possible to increase the percent of correct classifications of the commands, when compared to the control and tactile feedback. When the dynamics of estimates of the probability of the correct classification was studied with respect to the number of the recording using the different feedback types (Fig. 2), no significant differences (F (6,24) = 2.281; p = 0.070) were observed. However, it should be noted that the personal assessments of the subjects indicate strangeness of the tactile feedback during the first recording. Nevertheless, all the subjects were able to adjust to this type of feedback by the second recording, which is demonstrated in the plot in Fig. 2. To analyze differences in the desynchronization between the feedback types that were used, the С3 and С4 electrodes were chosen, since they had the maximum desynchronization observed for all the BCI operators at all the stages of the experiment (Fig. 3). The dynamics of changes in the desynchronization degree depending on the number of the recording was analyzed for the С3 electrode during the motor imagery of the right hand movement and for the С4 electrode during the motor imagery of the left hand movement, and it is given in Fig. 4. As can be seen from Figs. 3 and 4, no significant differences in the study were observed for the dynamics of the desynchronization and between the feedback types. It can be seen in topographic maps of the degree of desynchronization that were averaged for all the subjects (Fig. 5) that the motor imagery of the right hand
Table 1. Classification accuracy using different feedback types Subject
Without feedback (control), %
Visual feedback, %
Tactile feedback, %
OL
87.5 ± 9.2
89.2 ± 9.6
70.8 ± 11.0*#
GCH
55.0 ± 13.5
55.8 ± 12.9
55.8 ± 9.6
KP
75.0 ± 8.8
76.7 ± 7.2
85.0 ± 5.8
DB
58.3 ± 13.7
75.8 ± 6.9*^
55.5 ± 23.7
TI
63.3 ± 6.1
55.0 ± 6.4
62.5 ± 11.0
* Significant difference from the control; # significant difference from the visual feedback; ^ significant difference from the tactile feedback. HUMAN PHYSIOLOGY
Vol. 44
No. 3
2018
284
LUKOYANOV et al.
Probability of correct classification, %
90
80 a b
70
60
c
50
40 1
2 3 Number of recording
4
Fig. 2. Change in the classification accuracy depending on the number of the recording for different feedback types. Data are presented as (Mean ± SD). (a) Without feedback; (b) visual feedback; (c), tactile feedback.
C3 Left hand
C4 Right hand
C3 Left hand
C4 Right hand
Desynchronization degree
0
–0.25
–0.50
–0.75
–1.00 a
b
c
a
b
c
a
b
c
a
b
c
Fig. 3. EEG desynchronization for C3 and C4 electrodes during the motor imagery of left hand and right hand movements for all subjects (n = 5) and different feedback types. Box borders are the first and the third quartiles, the line in the middle of the box is the median, the error bars are halves the interquartile ranges. Points denote outliers. For designations, see Fig. 2. HUMAN PHYSIOLOGY
Vol. 44
No. 3
2018
THE EFFICIENCY OF THE BRAIN-COMPUTER INTERFACES
C3
285
C4
Desynchronization degree
–0.4
–0.6 b
a
c
–0.8
1
2
3
4 1 Number of recording
2
3
4
Fig. 4. Changes in the desynchronization for C3 and C4 electrodes during the motor imagery of left hand and right hand movements, respectively, depending on the number of the recording for different feedback types. Data are presented as (Mean ± SD). For designations, see Fig. 2.
movement induced contralateral desynchronization with the maximum near the C3 electrode independently on the feedback type that was used. The motor imagery of the left hand movement induced both contra- and ipsilateral desynchronization. This is likely to be connected with all the subjects' being righthanded. CONCLUSIONS The results that we obtained correspond to the literature data where vibrotactile feedback is studied in the BCI with the purpose of unloading the visual feedback channel [27]. It has been found that the best results in the BCI are achieved during combined use of the visual and tactile channels when the vibration feedback is congruent to the movement that is imaged; i.e., the vibration stimulus is applied to the hand that corresponds to the imagery. This interaction is known as control-display mapping [41]. For example, when the right hand movement imagery is recognized, the tactile stimulus that is applied to the right part of the body is more effective (i.e., it would positively affect the command execution and experience formation, when compared to a similar signal that is applied to the left part). The HUMAN PHYSIOLOGY
Vol. 44
No. 3
2018
comparison of the effectiveness of the performance of the subjects in the BCI with the visual and tactile feedback has demonstrated that the both feedback types result in a similar character of the task performance in the BCI, which has also been demonstrated in studies by other authors [29]. Tactile feedback, despite its popularity in other BCI types, has not been paid enough attention during the development of motor imagery-based neurointerface. At the same time, the critical advantage of the tactile feedback is, first, the possibility to use this technology in medicine for patients with impaired vision, and second, the fact that tactile feedback completely excludes the possibly negative interference of the visual feedback with the mental motor imagery, which is required by the procedure. Thus, it has been shown that the tactile feedback can be used in the motor imagery-based BCI instead of the commonly used visual feedback, which greatly expands the possibilities of the practical application of the BCI. However, an unexpected result of the study was the fact that after some training of a subject the presence of feedback independently on its type did not affect the classification accuracy in the MI-BCI, though according to the subjects’ statements the presence of
286
LUKOYANOV et al.
Left hand
FT7
FC5
Right hand
FC3
FC1
FCz
FC2
FC4
O3
O1
Cz
O2
O4
FC6
FT8
FT7
FC5
O6
O5
(a)
CP1 CPz CP2 CP3
FCz
FC2
FC4
O3
O1
Cz
O2
O4 T4
CP3 CP6
CP6 TP8
TP7
P4
P3
P4
P3
POz
FT7
FC5
POz
FC3
FC1
FCz
FC2
FC4
O3
O1
Cz
O2
O4
FC6
FT8
FT7
FC5
O6
O5 T3 CP1 CPz CP2 CP3
FC1
FCz
FC2
FC4
O3
O1
Cz
O2
O4 T4
CP3 CP6
CP6 TP8
TP7
P4
P3
P4
P3
POz
FT7
FC5
POz
FC3
FC1
FCz
FC2
FC4
O3
O1
Cz
O2
O4
FC6
FT8
FT7
FC5
O6
O5
CP1 CPz CP2 CP3
POz
FC2
FC4
O3
O1
Cz
O2
O4
FC6
FT8
O6 T4
CP3
CP4 CP6
CP5 TP8
P4
P3
FCz
CP1 CPz CP2 CP6
TP7
FC1
T3
CP4
CP5
FC3
O5 T4
T3
(c)
CP4
CP5 TP8
FT8
O6
CP1 CPz CP2
TP7
FC6
T3
CP4
CP5
FC3
O5 T4
(b)
CP4
CP5 TP8
FT8
O6
CP1 CPz CP2
TP7
FC6
T3
CP4
CP5
FC1
O5 T4
T3
FC3
TP8
TP7 P4
P3
POz
Fig. 5. Topographic maps of EEG patterns that were averaged for all subjects (n = 5) during the motor imagery of right and left hand movements in the BCI with different feedback types. The weighted coefficients of channels are mapped: a dark color denotes the largest values, a light color denotes the smallest values. For designations, see Fig. 2.
HUMAN PHYSIOLOGY
Vol. 44
No. 3
2018
THE EFFICIENCY OF THE BRAIN-COMPUTER INTERFACES
the feedback created comfortable conditions for task performance in the BCI. ACKNOWLEDGMENTS The study was supported by the Russian Science Foundation (project no. 15-19-20053).
13.
14.
REFERENCES 1. Wolpaw, J.R., Birbaumer, N., Mcfarland, D.J., et al., Brain–computer interfaces for communication and control, Clin. Neurophysiol., 2002, vol. 113, no. 6, p. 767. 2. Kaplan, A.Ya., Zhigul’skaya, D.D., and Kir’yanov, D.A., The possible control of individual fingers of the phantom of the human hand in the contour of the brain-computer interface on the P300 wave, Vestn. Ross. Gos. Med. Univ., 2016, no. 2, p. 26. 3. Birbaumer, N. and Cohen, L.G., Brain–computer interfaces: communication and restoration of movement in paralysis, J. Physiol., 2007, vol. 579, no. 3, p. 621. 4. Frolov, A.A., Húsek, D., Biryukova, E.V., et al., Principles of motor recovery in post-stroke patients using hand exoskeleton controlled by the brain-computer interface based on motor imagery, Neural Network World, 2017, vol. 107, p. 137. 5. Kotov, S.V., Turbina, L.G., Bobrov, P.D., et al., Rehabilitation of stroke patients with a bioengineered “brain–computer interface with exoskeleton” system, Neurosci. Behav. Physiol., 2016, vol. 46, no. 5, p. 518. 6. Kaplan, A., Vasilyev, A., Liburkina, S., and Yakovlev, L., Poor BCI performers still could benefit from motor imagery training, Proc. 10th Int. Conf. “AC Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience,” Toronto, ON, Canada, July 17–22, 2016, New York: Springer-Verlag, 2016, part 1, p. 46. 7. Vasilyev, A., Liburkina, S., Yakovlev, L., et al., Assessing motor imagery in brain-computer interface training: psychological and neurophysiological correlates, Neuropsychologia, 2017, vol. 97, p. 56. 8. Mokienko, O.A., Chervyakov, A.V., Kulikova, S.N., et al., Increased motor cortex excitability during motor imagery in brain-computer interface trained subjects, Front. Comput. Neurosci., 2013, vol. 7, p. 168. doi 10.3389/fncom.2013.00168 9. Mulder, T., Motor imagery and action observation: cognitive tools for rehabilitation, J. Neural Transm., 2007, vol. 114, no. 10, p. 1265. 10. De Vries, S., Tepper, M., Feenstra, W., et al., Motor imagery ability in stroke patients: the relationship between implicit and explicit motor imagery measures, Front. Hum. Neurosci., 2013, vol. 7, p. 790. 11. Simmons, L., Sharma, N., Baron, J.C., and Pomeroy, V.M., Motor imagery to enhance recovery after subcortical stroke: who might benefit, daily dose, and potential effects, Neurorehabil. Neural Repair, 2008, vol. 22, no. 5, p. 458. 12. Vuckovic, A. and Osuagwu, B.A., Using a motor imagery questionnaire to estimate the performance of a HUMAN PHYSIOLOGY
Vol. 44
No. 3
2018
15.
16.
17.
18.
19. 20.
21. 22.
23. 24.
25.
26.
27.
287
brain–computer interface based on object oriented motor imagery, Clin. Neurophysiol., 2013, vol. 124, no. 8, p. 1586. Alimardani, M., Nishio, S., and Ishiguro, H., The importance of visual feedback design in BCIs; from embodiment to motor imagery learning, PloS One, 2016, vol. 11, no. 9, p. e0161945. Sollfrank, T., Ramsay, A., Perdikis, S., et al., The effect of multimodal and enriched feedback on SMR-BCI performance, Clin. Neurophysiol., 2016, vol. 127, no. 1, p. 490. Kaufmann, T., Holz, E.M., and Kübler, A., Comparison of tactile, auditory, and visual modality for brain– computer interface use: a case study with a patient in the locked-in state, Front. Neurosci., 2013, vol. 7, p. 129. Kauhanen, L., Palomäki, T., Jylänki, P., et al., Haptic feedback compared with visual feedback for BCI, Proc. 3rd Int. Brain-Computer Interface Workshop & Training Course 2006, Graz, 2006. Angulo-Sherman, I.N. and Gutierrez, D., Effect of different feedback modalities in the performance of braincomputer interfaces, 24th Int. Conf. on Electronics, Communications and Computers (CONIELECOMP 2014), México: Univ. Am. Puebla, 2014, p. 14. Godlove, J.M., Whaite, E.O., and Batista, A.P., Comparing temporal aspects of visual, tactile, and microstimulation feedback for motor control, J. Neural Eng., 2014, vol. 11, no. 4, p. 046025. Halder, S., Rea, M., Andreoni, R., et al., An auditory oddball brain–computer interface for binary choices, Clin. Neurophysiol., 2010, vol. 121, no. 4, p. 516. Schreuder, M., Rost, T., and Tangermann, M., Listen, you are writing! Speeding up online spelling with a dynamic auditory BCI, Front. Neurosci., 2011, vol. 5, p. 112. Hinterberger, T., Neumann, N., Pham, M., et al., A multimodal brain-based feedback and communication system, Exp. Brain Res., 2004, vol. 154, no. 4, p. 521. Pham, M., Hinterberger, T., Neumann, N., et al., An auditory brain-computer interface based on the selfregulation of slow cortical potentials, Neurorehabil. Neural Repair, 2005, vol. 19, no. 3, p. 206. Nijboer, F., Furdea, A., Gunst, I., et al., An auditory brain–computer interface (BCI), J. Neurosci. Methods, 2008, vol. 167, no. 1, p. 43. Furdea, A., Halder, S., Krusienski, D.J., et al., An auditory oddball (P300) spelling system for brain computer interfaces, Psychophysiology, 2009, vol. 46, no. 3, p. 617. Schreuder, M., Blankertz, B., and Tangermann, M., A new auditory multi-class brain-computer interface paradigm: spatial hearing as an informative cue, PloS One, 2010, vol. 5, no. 4, p. e9813. Changa, M., Morib, K., Makinoa, S., and Rutkowsk, T.M., Spatial auditory two-step input Japanese syllabary brain-computer interface speller, Procedia Technol., 2014, vol. 18, p. 25. Leeb, R., Gwak, K., Kim, D.-S., and del Millán, J.R., Freeing the visual channel by exploiting vibrotactile BCI feedback, 35th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC), Piscataway, NJ: IEEE Eng. Med. Biol. Soc., 2013, p. 3093.
288
LUKOYANOV et al.
28. Cincotti, F., Kauhanen, L., Aloise, F., et al., Vibrotactile feedback for brain-computer interface operation, Comput. Intell. Neurosci., 2007, vol. 2007, art. ID 48937. doi 10.1155/2007/48937 29. Gwak, K., Leeb, R., del Millán, J.R., and Kim, D.-S., Quantification and reduction of visual load during BCI operation, IEEE Int. Conf. on Systems, Man, and Cybernetics, San Diego, 2014, p. 2795. 30. Jeunet, C., Vi, C., Spelmezan, D., et al., Continuous tactile feedback for motor-imagery based brain-computer interaction in a multitasking context, Int. Conf. on Human-Computer Interaction, INTERACT 2015, New York: Springer-Verlag, 2015, p. 488. 31. Gomez-Rodriguez, M., Peters, J., Hill, J., et al., Closing the sensorimotor loop: haptic feedback facilitates decoding of motor imagery, J. Neural Eng., 2011, vol. 8, no. 3, p. 036005. 32. Vourvopoulos, A., Ferreira, A., and Badia, S.B., NeuRow: an immersive VR environment for motor-imagery training with the use of brain-computer interfaces and vibrotactile feedback, Proc. 3rd Int. Conf. on Physiological Computing Systems (PhyCS 2016), Lisbon, 2016, p. 43. doi 10.5220/0005939400430053 33. Oldfield, R.C., The assessment and analysis of handedness: the Edinburgh inventory, Neuropsychologia, 1971, vol. 9, no. 1, p. 97.
34. Doskin, V.A., Lavrent’eva, N.A., Miroshnikova, M.P., and Sharai, V.B., Test of differential self-evaluation of a functional state, Vopr. Psikhol., 1973, vol. 6, p. 141. 35. Koles, Z.J., Lazar, M.S., and Zhou, S.Z., Spatial patterns underlying population differences in the background EEG, Brain Topogr., 1990, vol. 2, no. 4, p. 275. 36. Trefethen, L.N. and Bau, III D., Numerical Linear Algebra, Philadelphia: Soc. Ind. Appl. Math., 1997. ISBN 978-0-89871-361-9. 37. http://martinos.org/mne/stable/generated/mne.decoding.CSP.html. 38. Hjorth, B., An on-line transformation of EEG scalp potentials into orthogonal source derivations, Electroencephalogr. Clin. Neurophysiol., 1975, vol. 39, no. 5, p. 526. 39. R.C. Team, R: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, 2016. http://www.R-project.org. 40. Lawrence, M.A., ez: Easy analysis and visualization of factorial experiments, R Package version 4.4-0, 2016. https://CRAN.R-project.org/package=ez. 41. Thurlings, M.E., van Erp, J.B.F., Brouwer, A.M., et al., Control-display mapping in brain–computer interfaces, Ergonomics, 2012, vol. 55, no. 5, p. 564.
Translated by E. Berezhnaya
HUMAN PHYSIOLOGY
Vol. 44
No. 3
2018
