Hacking Interfaces: How To Control A Computer Using Your Mind Abstract Cutting-edge technology including brain-computer interfaces, kinetic user interfaces, and augmented reality hardware will radically shift the way we interact with technology. These new methods of communication between humans and computers will dictate the ways in which our society – in fact, how technology itself – will function in the future. This talk focuses on these new developments in interfaces including EEG and fMRI brain-computer interfaces which allow people to interact with computers using their minds, kinetic devices such as LeapMotion and Kinect, and AR and VR devices including The Oculus Rift and the Omni Treadmill. Examples of open frameworks for developing and hacking the software and hardware for these devices will be provided along with a brief history of the way humans and computers have communicated historically and how that process is being transformed. Outline I. Brief history of human/computer interfaces a. Key milestones in the development of primary human/computer interaction i. Punch Card 1. An ancestor of the punch card was invented in the early 18th century for controlling textile manufacturing and started to develop into what we would recognize as punch cards in the 19th century, thanks to Charles Babbage. 2. Until the 1960s, it was the primary method of both input and output communication with computers.1 ii. Keyboard2 1http://homepage.cs.uiowa.edu/~jones/cards/history.html 2http://www.daskeyboard.com/blog/?page_id=1329 1. The teletype machine was invented in the 1940s for use with the telegraph. 2. In 1948, the Binac – the world's first commercially available digital computer – used a modified typewriter to input data on magnetic tape as well as to print the data it returned. 3. When MIT, Bell, and GE created MULTICS, a CRT display combined with a typewriter allowed programmers to see what they were typing on their screen. iii. Mouse 1. Douglas Engelbart and his assistant Bill English unveiled the first mouse prototype in 1963 at Stanford. In 1968, he took the stage to use a mouse during a presentation that became known as “The Mother of All Demos” as an example of intelligence amplification.3 2. When both Bill Gates and Steve Jobs saw it in use at Xerox PARC, they realized that they had to steal this, because people had a new way to talk to computers. Interesting sidenote: despite revolutionizing the way that human- computer interaction, Engelbart never received any royalties for his invention.4 iv. What’s next? 1. We’ve reached a kairotic moment in the development of how we communicate with computers and the way they communicate with us. 2. Here are some of the advances that will act as the impetus to the next wave of the evolution of computers, the way we communicate with them, and the way they communicate with us. II. Brain-computer interfaces (BCI) a. EEG: Electroencephalography i. Disclaimer: IANANS(I Am Not A Neuroscientist) ii. How does it work? 3http://sloan.stanford.edu/mousesite/1968Demo.html 4http://www-sul.stanford.edu/depts/hasrg/histsci/ssvoral/engelbart/engfmst1-ntb.html 1. EEG uses electrodes to record the electrical activity of your brain by measuring the voltage of the billions of neurons in different parts of your brain.5 2. First human EEG recording performed in 1924.6 3. In order to prepare the electrodes, a conductor such as contact-lens saline solution is applied to the sensors. 4. Uses machine learning algorithms to determine the difference between brain states to train the computer to perform specific actions. iii. Emotiv EPOC 1. First commercially-available BCI comparable to medical- grade scanners. 2. Only $500 with SDK, affordable for developers. 3. Open-source Python library called emokit released by Daeken after reverse-engineering the Emotiv protocol.7 iv. OpenEEG: GPLed hardware and software for monitoring brainwaves. 1. Hardware a. ModularEEG. “Made up of two or more EEG amplifiers, and a 6-channel signal capture board that connects to a PC via a standard serial cable. The standard setup has two EEG channels.”8 2. Software a. BioEra: “visual designer useful for analyzing [EEG] signals in real time.”9 b. OpenViBE: C++ LGPL software platform.10 v. Other EEG devices 1. Muse a. Going to be released in December 2013 from InteraXon at $199.11 b. Designed to be worn all day. 5http://en.wikipedia.org/wiki/Electroencephalography 6http://en.wikipedia.org/wiki/Electroencephalography 7https://github.com/daeken/Emokit/ 8http://openeeg.sourceforge.net/doc/modeeg/modeeg.html 9http://www.bioera.net/ 10http://openvibe.inria.fr/ 11http://interaxon.ca 2. Neurosky’sMindWave and MindSet12 a. Detects fewer mental states than the Emotiv, but less expensive. 3. XWave Sonic a. Compatible with iOS and uses Neurosky chips.13 4. MyndPlayBrainBand a. Uses Bluetooth and Neurosky chips.14 b. Less expensive than competing devices. vi. Issues with EEG15 1. Signal-to-noise ratio a. Because of interference, even extensive data needs to be scrubbed. b. Noise issue can be mitigated with invasive EEG, which implants electrodes directly into the brain. 2. Low resolution a. Doesn’t provide information about specific areas of the brain that are active, and there are issues recording activity in the lower layers of the brain. 3. Time a. It usually takes a considerable amount of time to calibrate and orient the EEG machine on the user’s head. b. fMRI: Functional Magnetic Resonance Imaging i. How does it work?16 1. MRI measures brain activity by monitoring changes in blood flow. 2. When neurons are activated, blood rushes to a particular part of the brain, and MRI measures this change. ii. New, expensive research method with much higher accuracy than EEG, ranging from $150,000 to $2 million per machine. iii. Open source libraries: FSL, SPM (Statistical Parametric Mapping) iv. fMRI machines make more accurate BCIs, but development has been slow due to cost and complexity.17 12http://www.neurosky.com/ 13http://www.plxdevices.com/product_info.php?id=XWAVESONIC 14http://myndplay.com/products.php?prod=7 15http://en.wikipedia.org/wiki/Electroencephalography 16http://en.wikipedia.org/wiki/Functional_magnetic_resonance_imaging c. PET: Positron Emission Tomography i. How does it work?18 1. A PET scan uses high-resolution radiation imaging to track a tracer in your blood. This can be done through an injection of the tracer or breathing in the radioactive material. ii. Hailed by Ray Kurzweil among others. iii. HRRT: High Resolution Research Tomographhas a scanning resolution of 2mm.19 iv. Tons of different free software available, including AMIDE, OxiriX, and STIR.20 III. Current and Potential Uses of BCIs a. Current uses i. Robotics software 1. EEG already powers technology like wheelchairs.21 2. ExtremeTech did a demo controlling a robot using the Emotiv EPOC over Skype using their minds.22 3. The new Braindriver application from the Freie University in Berlin even allows you to drive a car using your mind.23 ii. Brain input 1. Transcranial magnetic stimulation uses electromagnetic induction to polarize or depolarize neurons.24 This is currently used for clinical purposes, but could have brain input uses as the field of neuroscience continues to develop. 2. Optogenetics uses a concept called neuromodulation to manipulate neurons. It uses light-activated channels and enzymes to influence the way that neurons operate. 3. tDCS (Transcranial direct-current stimulation) runs a 2 milliamp current through your brain to slightly depolarize 17http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2233807/ 18http://www.nlm.nih.gov/medlineplus/ency/article/007341.htm 19http://corelabs.emory.edu/csi/equipment/hrrt.html 20http://cdn.intechopen.com/pdfs/27812/InTech-Free_software_for_pet_imaging.pdf 21http://www.gizmag.com/toyota-wheelchair-powered-brain-waves/12121/ 22http://www.engadget.com/2010/04/27/rovio-robot-controlled-via-skype-with-emotiv-brain-reading-heads/ 23http://spectrum.ieee.org/automaton/robotics/robotics-software/braindriver-a-mind-controlled-car 24http://en.wikipedia.org/wiki/Transcranial_magnetic_stimulation neurons, increasing learning and performance of test subjects in a study funded by DARPA.25 iii. Polysomnography 1. Colin Petty, a Brooklyn-based musician, uses audio triggers with his Emotiv EPOC during dreaming to make him understand he’s lucid dreaming and to increase his own memory of the dream using Audiokinetic’sWwise software. iv. General computer control 1. The Emotiv EPOC has a built-in gyroscope. In the past,I have experimented with using GlovePIE26 to map my blinking to mouse clicks, which allows me to use my head as a fully functional mouse. v. Neuromarketing 1. As lucrative as it is ethically questionable. 2. Main players: Sands Research, MindLab International and NeuroSense are leaders in the industry, which is constantly growing.27 3. Might be more evil than Project MK Ultra. b. Eventual uses of BCIs i. Exocortices 1. An exocortex is essentially a motherboard that would augment a user’s brain in the same way that AR augments reality. ii. Neurocommunities 1. The ability to network with people based on their brain data, including everything from the way they react to a particular song to the way they react to you reacting to how they react to a particular song. What better way to meet people with similar interests than you than actually see what their brain looks like? This is real neural networking: across different brains across different computers and would create a network of both biological and artificial neurons. iii. Synthetic telepathy28. 25http://www.extremetech.com/extreme/84232-boost-your-brains-power-with-a-9volt-battery-and-some-wet- sponges 26https://sites.google.com/site/carlkenner/glovepie 27http://www.neurosciencemarketing.com/blog/companies 1. DARPA is conducting research on synthetic telepathy which would allow people to communicate silentlyusing their minds.