MindReading: An Ultra-Low-Power Photonic Accelerator for EEG-based Human Intention Recognition Qian Loux∗ Wenyang Liuz Weichen Liuz Feng Guox Lei Jiangx xIndiana University Bloomington, USA zNanyang Technological University, Singapore flouqian, fengguo, [email protected] fwenyang.liu, [email protected] Abstract— A scalp-recording electroencephalography (EEG)- The essential computing effect of the EEG-based intention based brain-computer interface (BCI) system can greatly im- recognition makes mobile CPUs and GPUs [7] hardly meet prove the quality of life for people who suffer from motor disabil- the real-time processing requirement under the power and ities. Deep neural networks consisting of multiple convolutional, temperature constraints. LSTM and fully-connected layers are created to decode EEG Although FPGA [6], ASIC [7], ReRAM [9], and even signals to maximize the human intention recognition accuracy. photonic [10] neural network accelerators are proposed to However, prior FPGA, ASIC, ReRAM and photonic accelerators process neural network inferences in an energy-efficient way, cannot maintain sufficient battery lifetime when processing real- it is still difficult for the BCI system to adopt these solutions, time intention recognition. In this paper, we propose an ultra-low- because of its tight power budget and real-time requirement. power photonic accelerator, MindReading, for human intention The CMOS-based FPGA [6] and ASIC [7] designs cannot recognition by only low bit-width addition and shift operations. maintain reasonable battery lifetime when processing neural Compared to prior neural network accelerators, to maintain network inferences. For instance, the battery of Google Glass the real-time processing throughput, MindReading reduces the using an ASIC accelerator stands for only 45 minutes [11] power consumption by 62.7% and improves the throughput per when tracking consecutive object actions. The power-hungry Watt by 168%. CMOS analog-to-digital converters dominate > 80% of the total power consumption of the ReRAM-based accelerator [9] I. INTRODUCTION and hence becomes the obstacle to this accelerator’s fast adop- Brain-computer interface (BCI) [1] enables the direct com- tion in the wearable BCI systems. Inspired by the low power munications and control using brain intentions alone, and thus photonic network-on-chip [12], a recent work [10] creates offers a practical way to help people suffering from motor a photonic accelerator to significantly improve the inference disabilities. Particularly, scalp-recording electroencephalogra- throughput per Watt of convolutional neutral networks by phy (EEG) [2], [3] is one of the most promising solutions to compact optical micro-disks. But the eDRAM and optical implementing BCIs, due to its low-cost and portable acqui- adders in the photonic accelerator consume 79.1% of its total sition system. When a person is intent on moving different power and prevents it from achieving higher power efficiency. parts of his body, the EEG signals from his scalp fluctuates To process the real-time EEG-based human intention recog- in different modes. In this way, human intentions can be nition more efficiently under tight power and temperature con- recognized by decoding EEG signals. EEG-based BCI has straints, in this paper, we propose an ultra-low-power photonic been widely adopted in controlling wheelchairs, prosthetics accelerator, MindReading, for the wearable BCI system. Our and exoskeletons [4]. contributions can be summarized as follows. However, recognizing human intentions by decoding EEG • We present universal logarithmic quantization to quantize signals is challenging. EEG-based BCI systems suffer from not only weights but also activations of convolutional, inevitable noises [3], due to human physiological activities, LSTM and fully-connected layers into the data represen- e.g., eye blinks and heart beats. Moreover, the correlations [3] tation of power-of-2 with trivial accuracy degradation. In between EEG signals and their corresponding brain intentions arXiv:2001.11954v1 [eess.SP] 30 Jan 2020 this way, expensive floating point matrix-vector multipli- are not straightforward. To denoise EEG signals and detect cations can be replaced by low bit-width addition and human intentions, prior works [5], [6] create neural networks shift operations. consisting of multiple LSTM and convolutional layers that • We build a novel photonic human intention accelerator, obtain high recognition accuracy (e.g., 98.3% [5]). Because MindReading, to process the neural network composed of the 128Hz raw EEG signal sampling rate [5], to recognize of power-of-2 quantized weights and activations by on- intentions in real time, a BCI system processes the inference chip photonic low-bit adders and shifters. Particularly, of a typical EEG neural network [5] under the throughput of we create a photonic activation unit to directly quantize 128 times per second. For 64-channel EEG signals, the BCI the outputs of various activations, i.e., T anh, ReLU and system has to support a ∼100M-FLOPS throughput, which Sigmoid, to power-of-2 representations. is difficult to be delivered by mobile CPUs and GPUs [7] • We evaluated and compared MindReading against the under the tight power constraint and the temperature budget state-of-the-art CPU, GPU, FPGA, ASIC, ReRAM, pho- ◦ of a 2 C increase [8] for most bio-embedding applications. tonic neural network accelerators. Our experimental re- sults show that to maintain the real-time processing ∗Qian Lou and Wenyang Liu contributed equally. This work was supported in part by NSF CCF-1908992 and CCF-1909509. Wenyang Liu throughput, MindReading reduces the power consump- and Weichen Liu were supported by NAP M4082282 and SUG M4082087. tion by 63% and improves the throughput per Watt by 978-1-7281-4123-7/20/$31.00 c 2020 IEEE N-timesteps Ht-1 Xt 0 0 0 0 S22 S23 S24 0 0 0 0 0 0 0 S25 S26 S27 S28 S29 0 0 0 L L s1 0 S30 S31 S32 S33 S34 S35 S36 S37 S38 0 LSTM 0 S39 S1 S2 S3 S4 S5 S6 S7 S40 0 C C C S43 S41 S8 S9 S10 S11 S12 S13 S14 S42 S44 F S It Ft s2 0 S S S S S S S S S 0 S 45 15 16 17 18 19 20 21 46 N × σ σ × ... N N 0 S47 S48 S49 S50 S51 S52 S53 S54 S55 0 C T 0 0 0 S56 S57 S58 S59 S60 0 0 0 T 0 0 0 S S S 0 0 0 N + 0 61 62 63 0 N N W & W Wxf & Whf 0 0 0 0 0 S64 0 0 0 0 0 M M xi hi Ct Ct-1 s64 ... 64Electrodes ... Wxc & Whc ʃ Wxo & Who Electrodes S[0][1] S[1][1] S[N][1] S[0][2] S[1][2] S[N][2] L L Jt Ot S[0][3] S[1][3] S[N][3] × σ σ × ... ... ... ...... H ULTRACORTEX N-timesteps S S t MARK III S[0][64] S[1][64] S[N][64] ... T T LSTM Cell Structure t=0 t=1 t=N S22 S23 S24 0 0 0 0 S22 S23 S24 0 0 0 0 0 0 0 0 S22 S23 S24 0 0 0 0 M ... M S25 S26 S27 S28 S29 0 0 0 S25 S26 S27 S28 S29 0 0 0 0 0 0 S25 S26 S27 S28 S29 0 0 0 Intention Prediction S S38 30 S31 S32 S33 S34 S35 S36 S37 0 S30 S31S32 S33 S34 S35 S36 S37 S38 0 0 S30 S31S32 S33 S34 S35 S36 S37 S38 0 S39 S40 0 S39 S1 S2 S3 S4 S5 S6 S7 S40 0 0 S39 S1 S2 S3 S4 S5 S6 S7 S40 0 0 0 0 0 S22 S23 S24 0 0 0 0 S1 S2 S3 S4 S5 S6 S7 S25 1. Eye closed 0 0 0 S26 S27 S28 S29 0 0 0 S43 S41 S8 S9 S10 S11 S12 S13 S14 S42 S44 S43 S41 S8 S9 S10 S11 S12 S13 S14 S42 S44 L L 0 S30 S31 S32 S33 S34 S35 S36 S37 S38 0 S43 S41 S8 S9 S10 S11 S12 S13 S14 S42 S44 C ... 0 S39 S1 S2 S3 S4 S5 S6 S7 S40 0 C C 0 S45 S15 S16 S17 S18 S19 S20 S21 S46 0 0 S45 S15 S16 S17 S18 S19 S20 S21 S46 0 2. Both feet S43 S41 S8 S9 S10 S11 S12 S13 S14 S42 S44 S15 S16 S17 S18 S19 S20 S21 F S S F Soft S45 S46 0 S47 S48S49 S50 S51 S52 S53 S54 S55 0 0 S47 S48S49 S50 S51 S52 S53 S54 S55 0 0 S45 S15 S16 S17 S18 S19 S20 S21 S46 0 N 3. Both fists S48 S49 S50 S51 S52 S53 S N N S 54 S55 0 S47 S48 S49 S50 S51 S52 S53 S54 S55 0 47 0 0 0 S56 S57 S58 S59 S60 0 0 0 0 0 0 S56 S57 S58 S59 S60 0 0 0 C T T C max S56 S57 S58 S59 S60 0 0 0 S56 S57 S58 S59 S60 0 0 0 4. Left fist 0 0 0 0 S61 S62 S63 0 0 0 0 0 0 0 0 S61 S62 S63 0 0 0 0 N N N 0 0 0 0 S61 S62 S63 0 0 0 0 S61 S62 S63 0 0 0 0 0 S64 0 0 0 0 0 0 0 0 0 0 S64 0 0 0 0 0 0 0 0 0 0 S64 0 0 0 0 0 M M 5. Right fist S64 (a) Collect EEG data (b) Predict movement intention from EEG data Fig. 1. The EEG-based Human Intention Recognition. 170% over a recent photonic accelerator.