USING BRAIN-COMPUTER INTERFACE TECHNOLOGY AS A CONTROLLER IN VIDEO GAMES

. Nowadays, control in video games is based on the use of a mouse, keyboard and other controllers. A Brain Computer Interface (BCI) is a special interface that allows direct communication between the brain and the appropriate external device. Brain Computer Interface technology can be used for commercial purposes, for example as a replacement for a keyboard, mouse or other controller. This article presents a method of controlling video games using the EMOTIV EPOC + Neuro Headset as a controller.


Introduction
The history of the first controllers is very extensive since it dates back to 1926. The history of the first video games controllers have their roots in completely different fields. The twoaxis electric joystick was first patented in the USA in 1926 and it served as a control handle on airplanes. In 1944, a German invention with a two-axis joystick was used to remotely control rockets. It can therefore be concluded that the roots of gaming controllers are derived from military inventions [31].
Only one button and one potentiometer ( Fig. 1) were enough to play the first video game evermade by William Higinbotham in 1958. Tennis for Two was a sports video game which simulates a game of tennis, in which an oscilloscope was used as the display, and the ballistics of the tennis ball was calculated by a Doner Model 30 analogue computer. To play with another player, a pair of such controllers was necessary. Controllers have changed over the years. The first commercial digital video game consolethe Magnavox Odyssey 100debuted on the market in 1972. The console included two analogue controllers with one dial to control horizontal movement and one for vertical.
The Atari Home Pong Console debuted on the market in 1975. The console contained one of the most powerful computer chips on the market. Players could control the game using controllers which were built into the console.
The Atari 2600 debuted on the market in 1977. The console included the first recognisable console joystick with a number of controller options, and a pair of dial controllers for paddle or driving games.
The Nintendo Entertainment System (NES) debuted on the market in 1983. This console was the most iconic console of all time. The original controller consisted of two round buttons labelled: A and B, a START button, and a SELECT button.
Sega Genesis debuted on the market in 1988. The Sega Genesis controller was ergonomic and consisted of three round buttons labelled: A, B and C, a START button, a SELECT button and a four direction buttons.
The Super Nintendo Entertainment System (SNES) debuted on the market in 1990. A new controller introduced a rounded evolution of controllers and established a new format of having four buttons under the right thumb and shoulder buttons for the fingers.
The Sony PlayStation debuted on the market in 1994. The DualShock controller (Fig. 2) consisted of two analog knobs, 11 buttons: L1, L2, R1, R2, square, triangle, circle, cross, START, SELECT and ANALOG, and a D-pad. Controllers are not only used in the video games industry, but also in other areas, for example: to remotely control underwater vehicles [9] and to control wheelchairs [6]. Other types of controllers such as isometric controllers can be useful for users with spasticity or complex movement disorders [7].
The current development of technology and controllers for controlling games is very fast and progressive. Modern games give the player many control options, for example: using a Gamepad, driving wheel, keyboard, mouse or Virtual Reality Headset. Taking into account this development, the Brain Computer Interface (BCI) may be the next step in controlling and playing video games. A brain computer interface (BCI), also known as a brain machine interface (BMI), is a device for direct communication between the brain and a computer. BCIs can be divided into two categories [19]: invasive brain computer interfaces [5,22] and noninvasive brain computer interfaces [8,21]. This research is based on non-invasive BCIs. In a BCI, signals from the brain are analyzed to determine the user's state of mind or intentions. By detecting features of the brain activity, users can communicate to a computer using electroencephalography (EEG) [15].
Research of BCI technology focuses mainly on disabled people (prosthetic hands) but also on controlling wheelchairs, humanoid robots, and mobile robots [4,20,25].
Electroencephalography (EEG) is one of the basic non-invasive ways to analyze resources and record human brainwaves. We can divide brain waves into several groups as shown in Table 1. Each of these waves has a different function and frequency [14,16].  [1,2].
The process of converting brain wave signals into action recognition starts with signal acquisition by biopotential sensors. After this, the signal is pre-processed (Feature extraction, Classification) by a signal processor. In the next step, the signal is translated and matched to saved thought patterns. After recognition of the signal, the specified action is performed [32].
Machine learning is implemented in BCI systems in order to recognise thought patterns. It uses classifiers (classification algorithms) which are divided into several categories, such as: Linear classifiers, Neural networks and Nearest neighbor classifiers. A support vector machine (SVM) is an example of a Linear classifier algorithm. In order to recognize classes, it finds an optimal hyperplane by using margins. Multilayer Perceptron (MLP) is an example of a non-linear neural network classifier. It is based on three layers of neurons (input, hidden and output). Neurons from different layers are connected with each other in such a way that the output data from one layer is transmitted to the input of the following layer. The k-Nearest Neighbor (kNN) algorithm is one of Nearest neighbor classifiers. It matches a tested sample according to the majority of its k nearest neighbors variable [3,13].
The BCI seems to be a promising and engaging technology. However the current state of its implementation in video games is still insignificant and does not affect the video game industry [10]. There has been much research focused on using BCI technology as a controller in video games, for example in BCI tested in games created for research purposes. A good example is MindBalance, created by Researchers at the University College Dublin and MediaLabEurope. It shows that the control of a game depends on many factors and may differ across users' minds and their level of engagement [12]. Another example is a special adaptation of the BreakOut game for BCI which achieved a control level for different subjects of about 80% [18]. As shown in other research, the controlling effectiveness highly depends on the level of engagement of the user [24]. All of the abovementioned examples show that BCI has been a field of research in video games for many years, but its usefulness in current, complex and popular video games is still nonexistent.
The studies described in this article show the possibility of controlling recent video games via BCI using the example of Dragon Age II. The research involves using an EMOTIV EPOC+ Neuroheadset in order to improve the users' gaming experience and to give them new and more intuitive game controlling, as well as a way to play videogames. It can be done by recording brain signals via EEG and then assigning them to specific actions in the game. Software added to the headset will make this possible. In some games, the player must react quickly and select many shortcuts and keys in the same time. This makes the game harder and more complicated. A BCI can give player a big advantage against other players by using brain signals to activate multiple functions in the game during intense action sequences without thinking about which key or shortcut should be chosen.

Research methodology
The research was carried out in the laboratory during threehour sessions. It was equipped with computers with EMOTIV EPOC+ Neuroheadsets [17]. All computers in laboratory had the EMOTIV software installed. The most used software was EPOC Control Panel 2.0.0.21. The other used software was EMOTIV Xavier TestBench.
The first step of this study was to prepare the device to work. The sensors were hydrated using saline solution which helped to improve the detection of brain signals. Every sensor was checked to ensure that it was properly installed. A Bluetooth receiver was plugged into a USB port for wireless connectivity. The EPOC Control Panel 2.0.0.21 was launched and the Neuroheadset was brought within range of the Bluetooth receiver and turned on in order to connect the two devices: the neuroheadset with computer. After a successful connection, the wireless signal status in the EPOC Control Panel changed to "Good". The Neuroheadset was placed on the user's head and the electro sensors were adjusted in order to ensure good signal quality. The EPOC Control Panel displays the contact quality of all sensors. The most reliable results will be achieved when the color of every sensor is green, while black signifies that there is no signal. In between these two extremes, the three colors are red (very poor signal, not acceptable), orange (poor signal), and yellow (fair signal).
After setting everything up, the experimenting step started. The experiment was based on training different actions performed on a virtual cube in the EPOC Control Panel 2.0.0.21 while monitoring the brain wave activity in the EMOTIV Xavier TestBench (Fig. 3). The goal was to train four levels of controlling the cube (from easy to extreme). Firstly, the neutral state was trained. The user had to be silent and keep their mind calm. Then, the first action to be performed on cube was selected. For example, the selected action was: push the cube (Fig. 4). During the easy training, the user was trying to imagine the cube moving forward. Then, the result of the training was checked by repeating the thoughts from the training and checking if it changed the cube's position. This process was repeated until the end of the research duration but on different levels and with different actions.

Measuring Device
EMOTIV EPOC+ is an EEG Neuroheadset (Fig. 5). It consists of 14 biopotential sensors with gold-plated connectors. The sensors locations are: AF3, AF4, F3, F4, F7, F8, FC5, FC6, T7, T8, P7, P8, O1, and O2 (Fig. 6). This neuroheadset includes a 3-axis accelerometer and a 3-axis magnetometer. The communication between the neuroheadset and a computer is based on wireless connection (Bluetooth 2.4 GHz). It is powered by a rechargeable Lithium Polymer battery which offers twelve hours of work. The EEG is filtered by the EPOC+ hardware with a fifth order digital Sinc filter using bandpass of 0.16-43 Hz and a notch at 50 Hz and 60 Hz [11]. The sampling rate of the headset is 128 Hz [26]. It is supported on Windows, MAC, iOS and Android. It recognizes facial expressions such as blink, wink, surprise and smile. It also measures emotions such as excitement, stress, focus and relaxation.

Measurement result
The experiment was performed on one person. The goal was to train four levels of controlling the cube (from easy to extreme).
At the beginning of the experiment, the training was performed at the easy level. When the neutral state was trained, the EEG signal in the EMOTIV Xavier TestBench did not differ very much within the graphs. The first selected action was to push the cube. As a result of this action, the biggest changes were observed on the F7, F8 and AF4 sensors (Fig. 7). One of the sensors -AF3was not connected, and due to this, the signal was not recorded. On other sensors (O2, FC6 and F4), the changes were weaker. The changes on the rest of the sensors were insignificant. The second selected action was disappearing the cube. As a result of this action, the biggest changes were observed again on the F7, F8 and AF4 sensors (Fig. 8). On other sensors (T8, FC6 and F4), the changes were weaker. The third selected action was lifting the cube. As a result of this action, the biggest changes were observed on the F7 and F8 sensors (Fig. 9). On other sensors (AF4, P8, T8 and FC6), the changes were weaker. The fourth selected action was dropping the cube. As a result of this action, the biggest changes were observed on the F7, F8 and AF4 sensors (Fig. 10). On other sensors (P8, T8 and FC6), the changes were weaker. The next step of the experiment was performed at the medium level. The training results of the next level (medium level) were the same as the easy level but some issues appeared. Controlling the cube was harder and more complex.
The third step of the experiment was performed at the hard and extreme levels. The trained action of the experiment used facial muscle movements. Considering the complexity and the high level of difficulty, the facial muscles were used to improve the intuitiveness of manipulating the cube. Big signal changes were noticed on the signal graphs. Some signal changes were off the scale.
The complexity of controlling the cube increased with the number of trained actions. It can be concluded that controlling the cube using user thoughts is much harder than using facial muscles because the EEG signal from user's brain is much weaker than facial muscle movements.

Implementation concept
The main intention of this concept is to help a player controlling a game with their brain. In order to make this option possible, the player's thoughts must first be record, analysed and recognised by the EMOTIV Application. The recognised thought patterns are assigned to specific shortcutskeys during the training session. The EMOTIV Applications are a connector between the player and the game.
In the EMOTIV Control Panel, the user can display and check all of the necessary information about the BCI system such as the battery power of the device and the signal quality of all sensors. The user can also train in this application. The EMOTIV Control Panel also includes other options such as: Headset Setup, Expressiv suite, Affecitv suite, Cognitiv suite and Mouse emulator. The Headset Setup panel is displayed by default and displays the sensors' contact quality. The Expressiv suite is represented by a 3D avatar which repeats the user's facial movements. The user can assign specific keys to these movements. The Affecitv suite measures and displays a wide range of emotional responses in the form of graphs. The Cognitiv suite includes a virtual 3D cube. This cube represents cognitive detection output in the form of an animation. The Mouse emulator allows the user to activate the neuroheadset gyroscope and link it to the control of their computer mouse cursor.
The Cognitiv suite is the most important program for the user because they train every action here and assign keys to the trained actions. It allows the user to train up to four actions which can be performed simultaneously. These actions will be converted into emulated keys and shortcuts by detecting the corresponding signal during the game.
This concept can be used in almost every game which requires the user to control the game, but in some games, it might be more useful than in others. These games requires, for example, using the keyboard and mouse simultaneously. As a the result, this kind of games requires pressing multiple keys or complicated shortcuts on the keyboard very quickly in addition to moving the mouse and pressing the mouse buttons. The best example of this game genre is real time strategy (RTS). One of the most popular representatives of this game genre is StarCraft 2. Another example of this game genre is massively multiplayer online role-playing games (MMORPGs). One of the most popular representatives of this game genre is World of Warcraft (WOW).

Tests
The EMOTIV tests were performed on the Dragon Age II game using the EMOTIV Control Panel application. Dragon Age II is a Role Playing Game (RPG) that includes a complex combat system. The player uses a lot of skills and spells by pressing specific keys and shortcuts. The four actions trained in the EMOTIV Control Panel application were used as emulated keys. These four keys, as shown in Table 2, were assigned to four trainings actions. These emulated keys were then used by the player to control the character. The first action was moving forward. In order to go forward, the player has to press and hold the W button. When it comes to emulating this key using the brain, some issues appear. The action was stopping at random times, and as a the result, the movement of the player character was not continuous. It required a lot of focus to continue this movement for more than two seconds. After that, the second movement was added (moving backwards). This was similar to the first movement, i.e. the movement was delayed and not continuous. There was no problem in recognising the player's intention and the character was controllable in the way the player wanted. The next step was to include movement to the left. Movement to the left was assigned to the face movement (in this case closing left eye). The attempt at combining the forward/backward movement with movement to the left failed. It can be concluded that connecting the player's brain signals with their face movements in order to control the character in the game is almost impossible. The signal from face muscles was too strong compared to the brain signals. Due to this issue, the emulated keys were changed. Table 3 shows the new key mapping and the new method that the game was controlled. The changes were focused on simple actions such as using character skills and abilities instead of controlling the movement of the character (Fig. 11-14). These actions were based on pressing a key instead of pressing and holding it. In this way, only one key could be emulated at a time and only one character skill could be activated. This change ensured that there was not interference between the four actions.
The second method of controlling the game was more practical and helpful for the player. Controlling the movement of a character with the brain signals and face movements simultaneously is not recommended.

Summary
This experiment demonstrated that using EEG and a BCI to control some actions in a video game can be useful and helpful for the player. An EMOTIV EPOC+ Neuroheadset and included software were used to perform all of the tests. The brain wave signals were registered by the EMOTIV EPOC+ Neuroheadset, processed in the EMOTIV applications and matched to trained actions. The result of this were emulated keys in a game.
In this paper, two types of controlling the game were described. The control of character movement was not useful for the player because controlling the game became problematic and hard. The next type of controlling the game was more intuitive and really helpful for the player as intended at the beginning.
The current state of development of BCI technology already allows video games to be controlled, but in a limited way. It can be used in the form of simple operations and actions as an add-on to control specific game functions and options. Currently, total control of the game using only the brain may be too problematic and even impossible because of the complexity of some games. It all depends on the type of game in which it will be used. There are games in which this technology will work very well because of their mechanics and controlling method.
BCI technology may greatly affect the entire gaming market in the future and the way they are controlled. This technology has great potential and may have a big impact on video games. For example, it can give the player an amazing gaming experience, in combination with Virtual Reality (VR). As an example, by dynamically adjusting the difficulty level of the game in real time [23]. It can also help professional players to improve the precision and speed of performing specific actions in the game.
However, first, some things have to be improved. The main issue to improve is better recognition and analysis of thoughts, as well as eliminating delays. The distortion caused by head movement should also be eliminated.