University of Warwick GlaxoSmithKline Department of Computer Science Immersive Intelligent Manufacturing Kinect PPE Gowning Conformation Author Benjamin J. BIGGS Industrial Sponsor Project Supervisor Mr Patrick HYETT Dr Abhir BHALERAO CS344 Final Report Discrete Mathematics, Final Year May 2016 Abstract In order to maintain positive relationships with regulatory authorities, GSK must have confidence that all staff members are properly dressed when located in high-risk facilities (e.g. areas in which employees are exposed to raw product). This project aims to address this need by constructing an automatic verification process to determine whether a lab operator has correctly followed the in-place gowning procedure – i.e. the operator has correctly put on the specified safety clothing/equipment in the appropriate order – before allowing access to the facility. If successful, the developed application will allow GSK, and similar companies, to prove their adherence to EU regulations that ensure both staff and product are kept safe. Keywords Computer Vision; Machine Learning; Kinect; Gowning Procedures; MATLAB; C# Acknowledgements This project would not have been possible without the help and support of the following people: Dr Abhir Bhalerao – Project Supervisor (University of Warwick) Mr Patrick Hyett – Industrial Sponsor (GSK) Dr Theo Damoulas, Lewis Rendell – Technical advice and discussion around machine learning and statistical methods Casey-Elizabeth Hayden – Sketched interpretation of technical diagrams, represented in Figure 5, Figure 10, Figure 11, Figure 13, Figure 14, Table 3, Table 6, Table 7 and Table 8 Philip Shepherd, Mark Allen and the rest of the IIM team GSK data collection volunteers, notably Joana Gomes (key test volunteer) and Eddie Dukes (video camera) Liam Steadman, Nicholas Biggs and Anthony ‘Tony’ Potts – General advice and discussion Jacqueline Biggs, Emma French, and Stephen Attwood i ii Contents List of Figures ......................................................................................................................................... 1 List of Tables .......................................................................................................................................... 3 1 Introduction and Motivation ........................................................................................................... 5 1.1 Motivation ............................................................................................................................... 5 1.1.1 Pharmaceutical Regulations ............................................................................................ 5 1.1.2 GSK Gowning Procedure (SOP-PDK-0012) .................................................................. 6 1.1.3 Beard Snood .................................................................................................................... 8 1.1.4 Alternative Use-Cases ..................................................................................................... 8 1.2 Project Aims ............................................................................................................................ 9 1.3 Stakeholders .......................................................................................................................... 10 2 System Requirements ................................................................................................................... 11 3 Potential Solution Designs ........................................................................................................... 13 3.1 RFID Tagging ....................................................................................................................... 13 3.1.1 Explanation ................................................................................................................... 13 3.1.2 Hardware Options ......................................................................................................... 13 3.1.3 Evaluation ..................................................................................................................... 15 3.2 Computer Vision ................................................................................................................... 16 3.2.1 Explanation ................................................................................................................... 16 3.2.2 Hardware Consideration ............................................................................................... 16 3.2.3 Diagram ......................................................................................................................... 19 3.2.4 Pros/Cons ...................................................................................................................... 20 4 Solution Architecture and Development Strategy ........................................................................ 21 4.1 Training Module ................................................................................................................... 21 4.1.1 PPE Item Classifier Training Scheme ........................................................................... 21 4.1.2 PPE Gowning Procedure Training Scheme .................................................................. 23 4.2 Recognition Module .............................................................................................................. 24 4.3 Diagram................................................................................................................................. 25 5 Evaluation of Chosen Solution, Timeline and Anticipated Issues ............................................... 27 iii 5.1 Existing Solutions ................................................................................................................. 27 5.1.1 StileEye ......................................................................................................................... 27 5.1.2 Real-Time Clothing Recognition in Surveillance Videos ............................................. 27 5.1.3 Betaface API ................................................................................................................. 27 5.2 Anticipated Challenges ......................................................................................................... 28 5.2.1 Anticipated Legal, Social, Ethical and Professional Issues .......................................... 29 5.3 Research and Implementation Strategy ................................................................................. 29 6 Preliminary Kinect SDK Work .................................................................................................... 31 6.1 Extracting Body Components ............................................................................................... 31 6.1.1 Extracting Skeletal Components ................................................................................... 35 6.1.2 Extracting Facial Components ...................................................................................... 37 6.1.3 Extracting Beard Region Components .......................................................................... 39 6.2 Gesture Detection .................................................................................................................. 41 6.2.1 Entrance Gesture ........................................................................................................... 41 6.2.2 Presentation Gesture ..................................................................................................... 42 7 Developing a Gowned/Ungowned Image Classifier .................................................................... 43 7.1 Strategy and Preliminary Data Collection ............................................................................ 43 7.1.1 Preliminary Data Collection .......................................................................................... 43 7.1.2 Evaluating Techniques .................................................................................................. 43 7.2 HSV Colour Thresholding .................................................................................................... 44 7.2.1 Explanation ................................................................................................................... 44 7.2.2 Solution Design ............................................................................................................. 46 7.2.3 Implementation ............................................................................................................. 48 7.2.4 Preliminary Testing ....................................................................................................... 49 7.2.5 Evaluation ..................................................................................................................... 49 7.3 SURF Bag of Visual Words with SVM Classifier (SBS) ..................................................... 50 7.3.1 Explanation ................................................................................................................... 50 7.3.2 Implementation ............................................................................................................. 53 7.3.3 Diagram ......................................................................................................................... 54 iv 7.3.4 Preliminary Testing ......................................................................................................