computers Review Recommendations for Integrating a P300-Based Brain–Computer Interface in Virtual Reality Environments for Gaming: An Update Grégoire Cattan 1,* , Anton Andreev 2 and Etienne Visinoni 3 1 IBM, Cloud and Cognitive Software, Department of SaferPayment, 30-150 Krakow, Poland 2 GIPSA-lab, CNRS, Department of Platforms and Project, 38402 Saint Martin d’Hères, France; [email protected] 3 SputySoft, 75004 Paris, France; [email protected] * Correspondence: [email protected] Received: 19 September 2020; Accepted: 12 November 2020; Published: 14 November 2020 Abstract: The integration of a P300-based brain–computer interface (BCI) into virtual reality (VR) environments is promising for the video games industry. However, it faces several limitations, mainly due to hardware constraints and limitations engendered by the stimulation needed by the BCI. The main restriction is still the low transfer rate that can be achieved by current BCI technology, preventing movement while using VR. The goal of this paper is to review current limitations and to provide application creators with design recommendations to overcome them, thus significantly reducing the development time and making the domain of BCI more accessible to developers. We review the design of video games from the perspective of BCI and VR with the objective of enhancing the user experience. An essential recommendation is to use the BCI only for non-complex and non-critical tasks in the game. Also, the BCI should be used to control actions that are naturally integrated into the virtual world. Finally, adventure and simulation games, especially if cooperative (multi-user), appear to be the best candidates for designing an effective VR game enriched by BCI technology. Keywords: BCI; VR; design; game design; brain–computer interface; virtual reality 1. Introduction A video game can be defined as “a mental contest, played with a computer according to certain rules, for amusement, recreation, or winning a stake” [1]. It has also been defined briefly as “story, art, and software” [1]. Sometimes, for example, in serious games, amusement is not the main goal. However, to date, amusement still plays a major role in the video game industry. Although by different means, virtual reality (VR) and brain–computer interface (BCI) are both excellent candidates for enhancing the possibilities of entertainment and satisfaction in video games. Indeed, both enhance immersion and it is a common belief that this encourages the amusement feeling. The concept of immersion was defined in [2], observing that everybody may enjoy a game with immersion, even if the gaming control seems to play the main role in the user’s enjoyment. According to Zyda [1], this immersion feeling is created by computer graphics, sound, haptics, affective computing and advanced user interfaces that increase the sense of presence. Virtual reality refers to a collection of devices and technologies enabling the end user to interact in 3D [3], e.g., spatialized sounds and haptic gloves (for example Dexmo, Dexta Robotics, Shenzhen, China). Steuer [4] emphasizes the particular type of experience that is created by VR. Such experience is named telepresence, defined as the experience of presence in an environment by means of a communication medium [4], joining the Computers 2020, 9, 92; doi:10.3390/computers9040092 www.mdpi.com/journal/computers Computers 2020, 9, 92 2 of 26 concept of presence [1]. A BCI can also enhance the feeling of presence in the virtual world since it can replace or enhance mechanical inputs. According to Brown and Cairns [2], immersive games are played using three kinds of inputs: visual, auditory and mental; since a BCI may transform ‘mental’ signals intoComputers input commands,2020, 9, 92 such interface may play a unique role in the mentalization2 process of 26 involved in the feeling of immersion. However, considering the limitations of a BCI system (to be analysed later),the concept it is still of presence not clear [1]. toA BCI what can extent also enhance current theBCI feeling technology of presencemay in the improve virtual world immersion. since it can replace or enhance mechanical inputs. According to Brown and Cairns [2], immersive games As pointed out by Brown and Cairns [2], “engagement, and therefore enjoyment through immersion, are played using three kinds of inputs: visual, auditory and mental; since a BCI may transform is impossible‘mental’ if there signals are usability into input and commands, control such problems”. interface may play a unique role in the mentalization An elementprocess of involved amusement in the feeling derives of fromimmersion. the originality However, considering and futuristic the limitations aspect of of BCI a BCI technology system as compared(to to be other analysed traditional later), it inputs,is still not like clear a mouse, to what a extent joystick, current or a BCI keyboard. technology Nonetheless, may improve as it often happensimmersion. in thetechnological As pointed out by industry, Brown and BCI Cairns technology [2], “engagement, risks being and droppedtherefore enjoyment by the public through if the immersion, is impossible if there are usability and control problems”. improvement it brings is not worthwhile compared to the effort needed for its use. Virtual reality has An element of amusement derives from the originality and futuristic aspect of BCI technology already enjoyedas compared the “wow-factor” to other traditional and VRinputs, systems like a mouse, tend to a bejoystick, employed or a keyboard. nowadays Nonetheless, in commercial as it events especiallyoften happens for raising in the technological this effect industry, (Feel Wimbledon BCI technology by risks Jaguar, being Coca droppe Cola’sd by the Santa’s public if Virtual the Reality Sleighimprovement Ride, McDonald’s it brings is not Happy worthwhile Meal compared VR Headset to the effort and needed Ski App, for its Michelleuse. Virtual Obama’s reality has VR Video, XC90already Test Drive enjoyed by the Volvo, “wow-factor” etc.) (http: and// VRmbryonic.com systems tend /tobest-vr be employed/). The nowadays “wow-factor” in commercial is defined in the Cambridgeevents especially dictionary for raising as “a this quality effect (Feel or featureWimbledon of by something Jaguar, Coca that Cola’s makes Santa’s people Virtual feelReality great Sleigh Ride, McDonald’s Happy Meal VR Headset and Ski App, Michelle Obama’s VR Video, XC90 excitement or admiration”, and was previously studied in the domain of marketing or education Test Drive by Volvo, etc.) (http://mbryonic.com/best-vr/). The “wow-factor” is defined in the (e.g., [5,6]).Cambridge The recent dictionary development as “a quality of dedicated or feature VR of headsets, something thatthat is,makes head-mounted-devices people feel great excitement (HMDs, e.g., the Oculus,or admiration”, Facebook, and Menlo was Park,previously CA, studied USA; HTC in the Vive, domain HTC, of marketing Taoyuan, or Taiwan; education Google (e.g., [5,6]). Cardboard, The Google, Mountainrecent development View, CA, USA)of dedicated has paved VR headsets, the way tothat the is, commercialization head-mounted-devices of combined(HMDs, e.g., BCI the+ VR technology.Oculus, Indeed, Facebook, HMDs Menlo provide Park, an CA, already USA; HTC build-in Vive, structure HTC, Taoyuan, that canTaiwan; support Google the Cardboard, embedding of EEG (electroencephalography)Google, Mountain View, CA, electrodes, USA) has paved which the are way needed to the commercialization for the BCI. The of Neurable combined BCI+VR Company technology. Indeed, HMDs provide an already build-in structure that can support the embedding of (Cambridge,EEG MA, (electroencephalography) USA) has recently announced electrodes, awhich product are combiningneeded for the an BCI. HTC The Vive Neurable (Taoyuan, Company Taiwan) with an EEG(Cambridge, cap. The companyMA, USA) claims has recently to develop announce an everydayd a product BCI combining and successfully an HTC Vive raised (Taoyuan, 6 million dollars for thisTaiwan) purpose with an (https: EEG //cap.www.forbes.com The company claims/sites to/solrogers develop an/2019 everyday/12/17 BCI/exclusive-neurable-raises- and successfully raised series-a-to-build-an-everyday-brain-computer-interface6 million dollars for this purpose (https://www.forbes.com/sites/solroge/). The HTC Vive (Taoyuan,rs/2019/12/17/exclusive- Taiwan), as well as other HMDsneurable-raises-series-a-to-build-an-everyday-bra such as the SamsungGear (Samsung,in-computer-interface/). Seoul, Korea) or the The Oculus HTC QuestVive (Taoyuan, (Facebook, Taiwan), as well as other HMDs such as the SamsungGear (Samsung, Seoul, Korea) or the Oculus Menlo Park, CA, USA) use inboard electronics, thus herein we refer to them as active devices. On the Quest (Facebook, Menlo Park, CA, USA) use inboard electronics, thus herein we refer to them as contrary, passiveactive devices. HMDs On consist the contrary of a simple, passive mask HMDs with consist lenses of a insimple which mask a smartphonewith lenses in iswhich inserted a (Figure1). Passivesmartphone HMDs is inserted are particularly (Figure 1). promising Passive HMDs for the are BCI particularly+VR field promising since they for arethe BCI+VR very aff ordablefield and smartphonessince they are nowadaysare very affordable ubiquitous. and Insmartphones addition, passiveare nowadays HMDs, ubiquitous. like SamsungGear In addition, and passive Oculus Quest, are easilyHMDs, transported like SamsungGear since theyand Oculus do not Quest, require are theeasily use transported of additional since materials,they do not suchrequire as the personal use computers orof additional external sensors.materials, such as personal computers or external sensors. (a) (b) (c) (d) Figure 1. TheFigure SamsungGear 1. The SamsungGear HMD ( aHMD) can ( bea) can used be used in passive in passive mode mode (inserting (inserting aa smartphonesmartphone without without plugging itplugging into the it mask into the through mask through the micro-USB the micro-USB port) port) or activeor active mode mode (with(with on-board electronic, electronic, mainly the gyroscope, supplied through the micro-USB port).