Fine-Grained Hand Pose Estimation System Based on Channel State
Total Page:16
File Type:pdf, Size:1020Kb
Fine-Grained Hand Pose Estimation System based on Channel State Information Thesis Presented in Partial Fulfillment of the Requirements for the Degree Master of Science in the Graduate School of The Ohio State University By Weijie Yao Graduate Program in Computer Science and Engineering The Ohio State University 2020 Thesis Committee Dr. Dong Xuan, Advisor Dr. Wei-Lun (Harry) Chao 1 Copyrighted by Weijie Yao 2020 2 Abstract In recent years, WiFi-based human-computer interaction has achieved significant progress in localization, fall detection, activity recognition applications since the innovation of CSI (Channel State Information). But WiFi sensing for fine-grained activity recognition like hand pose estimation is not yet discovered. In this study, we present a WiFi sensing system that only utilizes commercial off-the-shelf WiFi devices to capture human hand pose. To our knowledge, this is the first system that considers the application of hand pose estimation using CSI. We provide configuration details of data collection, data processing for CSI and image that can be reused for any other WiFi-based sensing research. And we propose a deep learning approach that achieves cross-modal learning from CSI to hand pose labels. Our system collects the CSI signals and 2D images in a time-synchronized manner. The 2D images are used to generate hand pose labels. And the CSI signals are collected from 3 x 3 transmitter and receiver antenna pairs and used as the input to our model. Our model includes 3 different learning targets. Experiment results show that CSI measurements have similar structures to digital images and popular network architecture for hand pose estimation in images can be applied to CSI measurements with slight modification. iii Dedication Dedicated to my parents, who give me freedom of decision for my life and keep supporting me as always. iv Acknowledgments I first want to thank my advisor, Dr. Dong Xuan, for giving me free space to explore and resource to conduct my master’s research during this special period. This thesis would have been impossible to make without his support. He was instrumental in pointing out the right direction throughout the process. It’s a great learning experience to work with him. I also have to thank my thesis committee member Dr. Wei-Lun (Harry) Chao for his insightful comments. Next, I would like to thank Cheng Zhang. He has a conscientious work manner and rich research experience. The idea of this research was originated from him. And we had extensive discussions during the process. His suggestions were proved to be correct every time. I wish I could have rethought carefully before retorting and followed all the useful suggestions at first. Special thanks to his patience and effort during our discussions. I wish we can have more chances to cooperate in the future. Finally, I would like to thank the people that helped and encouraged me during my time at Ohio State University. You make my life wonderful and color my world. I will keep the memory of our time as precious treasure in my heart. v Vita 2014 - 2018……………………………………….............B.E. Chemical Engineering, Tianjin University, Tianjin, China August 2018 - present ……………………………………M.S. Electrical and Computer Engineering, Ohio State University, Columbus, USA Fields of Study Major Field: Computer Science and Engineering vi Table of Contents Abstract...............................................................................................................................iii Dedication...........................................................................................................................iv Acknowledgments................................................................................................................v Vita......................................................................................................................................vi Table of Contents...............................................................................................................vii List of Tables...................................................................................................................... ix List of Figures......................................................................................................................x Chapter 1. Introduction........................................................................................................ 1 1.1 Human Activity Recognition................................................................................... 1 1.2 WiFi Sensing............................................................................................................2 1.3 Channel State Information....................................................................................... 4 1.4 Contribution............................................................................................................. 6 1.5 Organization of This Thesis.....................................................................................7 Chapter 2. Related Work......................................................................................................8 2.1 Vision-based pose estimation.................................................................................. 8 2.2 WiFi-based pose estimation.....................................................................................8 Chapter 3. System Framework and Implementation..........................................................10 3.1 Hardware Device................................................................................................... 10 3.1.1 Antenna..........................................................................................................11 3.1.2 Wireless card and Laptop...............................................................................11 3.1.3 Camera........................................................................................................... 11 3.2 CSI extraction........................................................................................................ 12 3.3 Video Recording.................................................................................................... 13 3.4 Data Processing......................................................................................................13 vii 3.4.1 Time Alignment.............................................................................................13 3.4.2 Signal Processing...........................................................................................15 3.4.3 Keypoint Generation......................................................................................17 3.5 Deep Learning........................................................................................................18 3.5.1 Learning Target..............................................................................................18 3.5.2 Network Architecture.....................................................................................20 3.5.3 Loss Function.................................................................................................21 3.6 Implementation Details..........................................................................................21 Chapter 4. Experiments and Discussions...........................................................................23 4.1 Testbed Installation................................................................................................23 4.2 Testbed Setup.........................................................................................................24 4.3 CSI tool configuration............................................................................................26 4.3.1 Mode Selection.............................................................................................. 26 4.3.2 Parameter Configuration................................................................................27 4.4 Dataset....................................................................................................................30 4.5 Evaluation Metrics.................................................................................................30 4.6 Experiment and Discussion....................................................................................31 Chapter 5. Conclusion and Future Work........................................................................... 36 5.1 Conclusion............................................................................................................. 36 5.2 Future Work...........................................................................................................37 Bibliography...................................................................................................................... 40 viii List of Tables Table 1 . All possible parameters of monitor_tx_rate in CSI tool.....................................29 Table 2 . A, B, C, D The performance of the baseline, PAM, PAM + 2 model................32 Table 3 . The comparison between PAM+2 model, Person-in-WiFi and Openpose.........33 ix List of Figures Figure 1 . WiFi signal transmission that observes human movement in the indoor environment[2].....................................................................................................................3 Figure 2 . Subcarrier-level signal strength computed from channel state information for four single-antenna 802.11n links [5].................................................................................