European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

The Systematic Review On Analysis: Trends And Developments

Nithyakani P1, M Ferni Ukrit2 Assistant Professor, School of Computing, SRMIST. [email protected] [email protected]

ABSTRACT -The research on gait analysis has a drastic growth in recent years for various applications like authentication, video surveillance, health monitoring system and animation. Gait recognition is an emerging intelligent system because of its recognition at a distance and it works well with low resolution videos. This paper provide the various gait features extracted in model based approach and model free approach. This paper describes the survey of gait recognition techniques which include the four major steps such as data acquisition, preprocessing, feature extraction and classification. Descriptions about wearable sensor, Vision base and pathological database is summarized. In recent years, performance of human recognition has been impressively enhanced by deep architectures. This paper offers an up to- date survey of deep architectures on gait recognition, mainly focusing on the performance of convolutional neural network combined with other architectures. Furthermore, general challenges faced in gait recognition and directions for future research are discussed.

1. INTRODUCTION

Human analysis and identification has been an attractive technology in many fields.In the digital imaging, a human can be analyzed by his/her unique feature of face, iris, sweat, hair, periocular region of eyes,Gait, smell, finger and palm[18]. Gait defines the style of walking. Human Gait analysis is a study of human movement to analyze the style of walking and its related features which are applied in the field of biometrics, surveillance, diagnosis of gait disease, therapy and rehabilitation, etc. Gait recognition can be performed through involvement and non-involvement of a person to the recognition system. In human involvement approach, person is directly contacted to the recognition system with the help of different sensors, accelerometer or monitoring devices.Non-involvement approach uses camera at a distance to identify a person identity with or without their knowledge based on its application. Gait analysis is prevalent than other features since thefeatures can be extraction from low resolution and without the cooperation of any individual.

“On the Gait of Animals” is the first book to report the gait analysis written by Aristotle in 350BC[1]. In 1680, Giovanni Alfonso Borelli made the extensive study on the mechanism. In 1890, Christian Wilhelm Braune and Otto Fischer performed the investigation of gait activity over human and published a series of papers. They have extended their research on the center of gravity, moments of inertia of the human body and its segments. In 1880s and Étienne- Jules Marey developed the animated photography to capture the several movement phases in a photographic surface. Muybridge used multiple cameras to record the locomotion and presented through his public lectures and demonstrations. In 1970s, video camera systems were available to study the several human pathological conditions. Human individuals were recognized with their gait style in 1970s. Later gait analysis was impressed by the forensic and animation experts [3-5]. Advancements in technologies of artificial intelligence and computing,Gait recognition has become the most popular data driven and artificial surveillance tool which can recognize the human at a distance. This survey aims to highlight the technologies developed for gait analysis. Section II provides the application of gait recognition in various fields. Section III describes about the human gait features considered for model based and appearance based recognition. Section IV focuses on the role of machine learning algorithms in recognizing human. Section V presents the description and

1636

European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020 comparison of recently implemented deep architecture which gives the high accuracy. Section VI discusses about future directions in the field of human gait analysis andSection VII concludes with the challenges and the future work.

2. APPLICATIONS OF GAIT RECOGNITION

2.1. Application-Based Systems Authentication is necessary to avoid violation in secured protected areas, home, public security,etc,. Authentication processes[3] are carried out in three methodologies such as knowledge based (credentials, PIN, MPIN), object or token based (E.g bank card, identity card etc.,) and biometric based. Knowledge based authentication process faces memorability issues specifically robust passwords. Security breach arises when the password is reused in many services. Object based method usually combined with knowledge based method for accessing the services provided with physical Keys. Biometric authentication is used to identify a person by his/her unique features like fingerprints, face, and periocular region of eye, retina, palm, sweet, body odor and gait. These features are unique in all subjects and also remain stable for a specific period. Behavioural features are difficult to imitate. Gait recognition method includes computer vision techniques, signal analysis, motion estimation, processing the video data, and ubiquitous applications.

2.2. Biometrics is divided into physical and behavioral features. Physical features are the pattern of height, structure of face, retina, eye etc. Physical features has universality, unique features for a long term period. Behavioural features includes the way of walking, speaking, singing, movement etc,.Though behavioral features varies on emotional factor, difficult to forge or imitate the individuality. A set of biometric images are recorded for verifying the similarity of captured image. Due to the recognition accuracy in various capturing modes, gait recognition has the highest popularity among the other biometric approach.

2.3. Life And Medical Sciences Gait recognition helps to identify the quality of walking of an individual with several common gait disease such as parkinson’s disease , multiple sclerosis, cardiopathies, post-stroke Hemiplegic gait, spastic diplegic, neuropathic, myopathic, choreiform, ataxic (cerebellar) and sensory [9-10]. Study of the gait sequences helps for early diagnoses of their musculoskeletal and neurological complications to provide the best treatment[62]. Kinematic and kinetic data are extracted from gait analysis helps the clinicians to interpret, assess, and evaluate the individual gait disorder. Data collection is performed by videotaping the individual before the examination, examining physically to analyze the muscle strength and tone, motion of joint, abnormalities in bone, stubborn muscular contractions, knee and ankle width, distance between right and left sacroiliac spine, etc.

2.4. Animation Animation of human gait is focused by the fields of , signal processing, kinematics, dynamics and robotics[6-7,11]. Interest in human gait animation is growing rapidly in the Computer Animation Community despite of the complex techniques. The first computer animation approach for human was developed in 1970s which depicts the early sequential pictorial representation of series. Later Humanoid animation(H-Anim) standard was approved in 1990s and greatly influenced by the research experts of gaming,movies, virtual avatars, graphics, ergonomics and simulations industry. The challenge in human animation is to simulate the natural motion style into the virtual characteristics. Animation is performed by the motion sequence extraction of skeleton representation of human connected hinges points like shoulder, neck , elbow, waist, knee and ankle . Higher quality is obtained by modelling the skeleton with surfaces. Methods of human gait animation are knowledge based kinematic animation, dynamic animation[61] and interactive synthetic walking motion data.

1637

European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

3. Gait Recognition Features Gait is a synchronized movement of bones, muscles and nerves of hip, knee and ankle joints rotations. The cyclic nature of the walking style provides the periodic leg movement. Gait of a person is generally represented as a walk cycle describes in the Figure 1which has two phases: stance phase and swing phase ; gait cycle has eight stages[2] as (i) initial contact of the foot or toe. (ii) Loading response as placing the entire foot over the floor with the limb segments reaction. (iii) Mid-stance as the intermediate support. (iv) Terminal stance is denoted as heel off. V) Pre-swing is the initialization of knee movement. VI) Initial Swing .Parameters extracted from gait sequence are step length, stride length, cadence, speed, dynamic base, progression line, foot angle, hip angle and squat performance. The measurements extracted were analyzed, interpreted to recognize the individual through the techniques like temporal or spatial, kinematics, Marker less gait capture, pressure analyzer, kinetics and dynamic electromyography.

Fig. 1.Asingle Gait cycle is also termed as stride. Stance phase begins with the heel strike and toe off the same foot. Average seconds of stance phase is 0.64s. Swing phasebegins with another foot heel strike and toe off. Average seconds of swing phase is 0.40s.A stride can also be represented with 8 stages: Initial contact, loading response, mid stance, terminal stance, pre-swing, initial swing, mid swing, terminal swing.

Human gait recognition system depends on the video has two major categories as model free gait recognition and model based gait recognition. Model based approaches extract the structural pattern of model from human body using various methodologies such as pendulum, leg inclination, stride, trajectories, joint angles, 2D and 3D temporal models [19-22] with their length, width and position dimensions. It helps to track the individual with set of mathematical parameters and its quantifiable relationships. Model based approach used in handling self-occlusions. Issues faced in Model based approach were viewpoint invariant, physiological changes such as age, weight, height , walking speed,etc., and external factors like clothes , foot wear, load , etc., and environmental factors.In model based approach kinematics track joints of an individual are head, shoulder, spine, foot, knee, ankle, thumb, neck, hand,etc shown in the Figure 2a.It also defines the dynamics or kinematics of the body movements in the terms of gait cycle time, stride length, Fourier magnitude, footstep , ground reaction force, pressure mat features, etc, Model based gait recognition utilizes spatio temporal volume, linear regression analysis, trignometric-polynomial interpolant functions, biped locomotion feature , Principal component Analysis, canonical analysis, 1NN, KNN, Hidden markov Models. Accuracy of the model based approach depends on the silhouette quality.

1638

European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

Fig. 2.Gait measurements andtypes of wearable sensors a) Measurements of human body used for feature extractionb) wearable sensors used to extract gait features

The other categories of gait recognition approaches were image and video based approach, wearable sensors, floor sensor and radar sensor[26]. Image and video based approach which involves remote capturing of human movement. Image and video processing capture the gait sequence of an individual with the help of several optic sensors like digital camera ,RGB, Low RGB-D Microsoft Kinect, laser range scanners, infrared sensors and time to flight cameras. Table 1 presents the description about various gait datasets collected using camera and wearable sensors. Wearable sensors which are attached to the human body measures movement time, calculate joint or inclination angle, walking speed, step or stride length, segment position , acceleration ,etc. Wearble sensors used for human recognition are accelerometer, gyroscope, force sensors, strain gauges, inclinometers, goniometer, Magnetoresistive sensor, electromagnetic tracking system, fabric sensor, and Electromyography sensor (Figure 2b). Sensors can be combined with the respect to application of gait analysis. Floor sensor based approach measures pressure , step length ,etc. with the help of pressure under floor mat and floor accelerometer .In gait recognition, various factors can modify the gait pattern permanently or transiently. External factors are terrain, footwear, clothing and cargo; Internal factors are sex, weight, height, age, physique, etc.

3.1. Model Based Features Model based approach model[19]generate the structure for human body and the arbitrary 2D views from the 3D model using the techniques like pendulum, cadence , stride, 2d Stick , kinematic features, angular trajectories etc. 3D model uses more than two calibrated cameras to capture the gait sequences and to avoid the problems of variations in surfaces, self-occlusion and fixed viewpoint. Structural method uses geometrical patterns of time variations and motion variations of gait sequence. This method has overcome the problem of variation in light, background, view and segmentation. The features extracted in model based approaches were classified using K- Nearest Neighbor, Back Propagation Neural network, Multilayer Perceptron, Principal Component Analysis, Support Vector machine and Linear Discriminant Analysis.

3.2. Model Free Features Model free approach extract the gait features for classification and analysis from the captured video or image without the person direct involvement with the system. The general process involves person

1639

European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

detection, background subtraction, silhouette extraction, feature extraction and classification. Model free approach is preferred for its low computational cost and better performance in low quality video. Background subtraction is a process of classifying the pixel as foreground and background. Basic background techniques are frame differencing and filtering which uses the concept of min, max, mean, median and histogram filter. Gait image descriptors are body length and height, GEI, gradient histogram energy image , Fourier descriptors, normal vectors, Principal Component Analysis, k- means, HMM, standard deviation and frequency domain features , Gabor filters Multi-linear Discriminant Analysis, Chrono-Gait Image, Linear Discriminant Analysis , Microsoft Kinect distance method.

TABLE 1Datasets available for gait biometric analysis

METHOD DATASET DEVICE SUBJECTS DATA FORMAT 744 tri-axial 389 – Males OU-ISIR Inertial accelerometer 355 – females, Sensor and gyroscope, Age- 2 to 87 smartphone

175 Subjects 22-session 0 Wii Remote - ZJU-gaitacc 153- session 1 5 153- session 2

WEARABLE SENSOR 3 mobile 25 subjects BWRMultiDevice devices with (session 1& 2) accelerometer

Foot pressure CASIA D – FOOT measurement 88 subjects PRINT plate + foot scanner

OU-ISIR 62 528, Biometric camera Single view Database

OU-ISIR VISION 10,307, Multiview 7 cameras BASED Multi view Database

20 person ; CASIA A – each with 12 STANDARD camera sequence in DATASET three directions

1640

European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

CASIA B – Multiview Gait Captured in 11 124 subjects dataset views

Infrared CASIA C – thermal 153 subjects INFRARED camera

RGB Video , Tum Gaid Dataset audio and 305 subjects depth

Camera (front DBMHI-DB2 100 subjects and back)

64 Subjects (Parkinson Disease -15 , Gait in Huntington force-sensitive Neurodegenerative Disease – 20, resistors Disease Database amyotrophic lateral sclerosis- 13, Healthy – 16)

Pathological Dataset

10 Parkinson disease Daphnet Freezing Accelerator Patient(3 of Gait Data Set Sensor measurements- ankle, knee and hip)

1641 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

4. MACHINE LEARNING IN GAIT RECOGNITION Machine learning concepts are required to build the intelligent system to learn, recognize, analyze the gait characteristics with large number of image sequence collected to create the gait dataset. Figure 3gives the general process of gait analysis using machine learning algorithms. Artificial intelligence techniques like data mining, machine learning, deep learning and automation are considered for the innovation in gait analysis fields. In machine learning paradigm, learning models are like supervised learning, unsupervised learning, reinforcement learning used in various design perspective to capture the gait sequence, finding the gait pattern from the characteristic features extracted with respect to the implementing field.

Machine learning approach follows the sequential procedure of data collection, feature extraction, learning algorithm and the evaluation methodology. Feature selection is a process used to identify the unique characteristics and eliminate the unnecessary information to reduce the problem of over fitting and training time thereby increasing the accuracy of the model. Feature selection techniques which are commonly used are univariate selection, feature importance, correlation matrix with heatmap and filter methods. Filter and wrapper feature selection method is specially used for gait sequence images. Wrapper methods is quite expensive since the learning algorithm repeats the random resampling technique usage for obtaining the feature subset accurately. Filter method use heuristic search procedure to obtain the undesirable feature subset. Without the help of learning algorithm, undesirable features were removed in terms of correlation between the examined data. Popular filter method is recursive feature elimination algorithm which removes the low weighted features. Feature subset were fed into evaluation procedure which determines the quality by comparing with the previous best value given by predefined stopping criterion and validated [22].

Model selection is the technique of selecting the appropriate model from the set of models which has high classification or prediction accuracy. Model is selected with the help of statistical technique, exploratory data analysis, scientific techniques and specification of a model. It is recommended that learning approach should be selected along with model selection [23, 24]. Generally, dataset is divide into three portions namely training data, cross-validation data and testing data. Training data is utilized to train the model with the input and known outcome. Cross validation data helps for frequent evaluation to improve the effectiveness of the model. Testing data is used for validating the specific model. Selected model is estimated to perform well with the training data and computed with similar learning model with various hyper parameters, complexities. Validation dataset is opted using the methods: holdout method, k-fold cross validation, leave-one cross validation, random subsampling cross validation and bootstrapping.

1642 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

Fig. 3. Basic machine learning approaches used for Gait classification, prediction and analysis

4.1. Classification Network Classification Network has statistical and derivative phase. Statistical phase extracts the statistical measurements from the non-linear transformed input. Derivative phase improves the accuracy with mathematical model. Classification networks are classified into shallow network and deep neural network. Shallow neural network usessolitary hidden layer whereas deep neural network uses more than one hidden layer. Shallow networks used for gait recognition are K-Neighbor, Hidden Markov Model, Bayesian Network, Support Vector machine and BPNN. Deep neural network increases the accuracy rate when compared to shallow network and has become more interesting approaches in extracting features in recent years. Wang et al[33] used nearest neighbour approach to generate the silhouette signal to identify the person and achieved 77.4% classification rate with CASIA B dataset. Jeevan et al [34] recognizes the gait of an individual with Gait Pal and Pal Entrophy using Principal Component Analysis with Support Vector machines. Implementing with treadmill dataset and CASIA A, B and C datasets, achieved 92.3% accuracy. Xiao et al[35] recognizes the individual by their gait sequence using Zernike moments and Back Propagation Neural Network (BPNN)

4.2. Deep Learning Approaches In the family of machine learning algorithms, deep learning has become the most important methods due to its wide applications. Deep architectures represent the high level abstractions using multiple

1643 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020 stacked learning neural network layers and separate network segment is used to learn separate data representation. The energetic factors of deep neural network are feature learning, hierarchical and distributed representations, computational speed with large dataset varies in the automation process and analysis of depth. Multiple neural network layers are convolutional layer, sampling pooling layers, activation layer and fully connected layer. The multiple arrangement of these layer and dimensions varies corresponding to the application and architecture.

5. DEEP ARCHITECTURES In deep learning approaches, there is no need to create a separate feature set since the discriminative features were found while training the modelDeep architectures which are used widely for gait recognition is convolutional neural network, multiple stacked auto encoder and deep Boltzmann machine and Restricted Boltzmann Machine.

5.1. Deep Boltzmann Machine A Deep Boltzmann Machine uses hidden random variables for multiple layers of symmetrically couple stochastic Markov Random Field[44]. It has visible unit set  {0,1}D and hidden layer series units h(1) {0,1}F1, h(2) {0,1}F2 ,…… h(L) {0,1}FL . Units in adjacent layers have connection between them. For instance, consider three hidden layers for DBM as shown in Figure 4. The probability of a visible vector assigned by Deep Boltzmann Machine is given as (1). 1 푃(푣; 휃) = ∑ 푒푥푝 (∑ 푊(1)푢 ℎ(1) + ∑ 푊(2) ℎ(1)ℎ(2) + ∑ 푊(3)ℎ(2) ℎ(3)) (휃 푖푗 푖 푗 푗푙 푗 푙 푙푚 푙 푚 ℎ 푖푗 푗푙 푙푚 (1) In equation (1) h is a set of hidden units represented as {h(1) , h(2) , h(3)} and the model parameter  = {W(1) , W(2) , W(3)}. This represents symmetric interactions between the visible units and hidden units. Restricted Boltzmann Machine model is recovered by assigning W(2) and W(3) as 0. Initially the weights in hidden units are asymmetric as the weights are twice as the top layer. Later to maintain the symmetry, weights are modified.

Fig. 4.a) Deep Boltzmann machine has undirected connection. b) DBM is pretrained with RBM stack. The weights of first and last layers of RBM is modified as W(1) and W(3) to maintain the symmetric pattern [44].

Yu et al[43] proposed the Convolutional Restricted Boltzmann Machine to extract the gait features to identify the individuals. CRBM was experimented on CASIA B gait dataset with 11 different angles and 3 variations in outfit obtained the accuracy of 96.5%.

1644 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

5.2. Deep Stacked Autoencoder

Fig. 5. a) single autoencoder with same dimensionality of input layer (x1, x2, , x6) and output layer , lower dimensionality in feature extraction layer. b) Stacked autoencoder enables the supervised classification in the final output layer [45].

Deep Stacked Autoencoder has more layers to extract the discriminative and representative features. It is formed by stacking the autoencoder network one above the other and implemented multilayer perceptron. A single autoencoder network has encoding layer for encoding the input and decoder layer for reconstructing the input. In decoding layer, number of neurons and dimension of input are considered to be in the same count. It recovers the input x with high accuracy using the two stage approximation model. The code h used for recovering is

푓푑푒푐 (푓푒푛푐(푥)) = 푓푑푒푐(ℎ) = 푥̂ ≈ 푥

Where fenc is encoding function and fdec is the decoding function computed by their respective coding layer. The main function of stacked autoencoder to train the unsupervised input layer by layer. Figure 5 shows how the single autoencoder turns to stacked autoencoder to perform the classification. The final output layer produces the supervised classification by fine tuning the pre-trained network with Back Propagation[45]. Sun et al [46] proposed the deep network by combining the convolutinal neural network and stacked autoencoder to recognize the human with the gait subtle features. Micro-motion dataset is a radar echo dataset collected using the Radio Frequency device is used to implement the proposed network and achieved 96% as highest average precision.

5.3. Recurrent Neural Network Recurrent Neural Network (RNN) is designed to solve classification problems. RNN deals with sequential data with more interdependencies especially well-suited for clinical data.In RNN, input sequence is termed as X1, X2,…., Xnand output sequence is termed as Y1, Y2,…., Yn. Output Vector Yi depends on the past values of X and it is given by 푌푖 = 푓푖(푋1, … . . 푋푛) Hidden states are utilized to learn parameters from the sequential data given in the previous and current recurrence. The hidden state, hi and recurrent layer output, yi is calculated as ℎ푖 = tanh(푊ℎℎℎ푖−1 + 푊푥ℎ푥푖 + 푏ℎ )

1645 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

Where W denote weight, b denote biases and i denote the frame index.To solve the vanishing gradient problem, Long Short Term Memory (LSTM) and Gated Recurrent units(GRU) is proposed. These networks maintain the long term inter relations and nonlinear dynamics effectively.To control the number of parameters in the network, same weight is determined for all layers. Hidden state is updated with forget gate value fi, input gate value gi and output gate value oi.. Learning variables are given as follows: 푓푖 = 휎(푊푥푓푥푖 + 푊ℎ푓ℎ푖−1 + 푏푓)

푔푖 = 휎(푊푥푖푥푖 + 푊ℎ푖ℎ푖−1 + 푏푖)

퐶푖 = 푓푡 ⃘ 퐶푖−1 + 𝑖푡 ⃘ tanh(푊푥퐶푥푖 + 푊ℎ퐶ℎ푖−1 + 푏퐶)

표푖 = 휎(푊푥푖푥푖 + 푊ℎ푖ℎ푖−1 + 푏푖)

ℎ푖 = 𝑖푡 ⃘ tanh(퐶푖)

Where W denote weights and b denote biases. Prado et al [65] used RNN to classify the cerebral palsy disease in adults and children. Data were collected from 28 adults and 7children with cerebral palsy using the pressure inertial foot sensor. They have proposed RNN with instrumented foot sensor which does not require any preporocessing technique, thereby processing time is reduced to less than 1 second.

5.4. Convoultional Neural Network Convolutional neural network (CNN) is multilayer perceptron in a regularized version effectively implemented on Graphics Processing Unit (GPU) or General Purpose computing on Graphics Processing Unit (GPGPU). GPU which is mainly developed for video game application is used to increase the speed of deep Neural Network processing. Convolutional Neural network has convolution layer, Pooling Layer, Activation function, Loss function, regularization, optimization process. Figure6 demonstrates the general outline of convolutional neural network.First parameter of the CNN is the input data in the form of image or signal. Second parameter is the filter. Fully connected layer gives the output in the form of vector from which feature map is obtained. In the form of multidimensional array, each input and filter are saved independently in the network. Local information of the image is identified by convolutional layer.

Fig. 6. Schematic diagram of convolutional Neural network

Convolution layer is categorized into titled convolution, transposed convolution, dilated convolution, network in network and inception module. Pooling layer is categorized into average pooling, max pooling, Lp pooling , mixed pooling , stochastic pooling , spectral pooling , spatial pyramid pooling and multi-scale order less pooling. Activation function used are Rectified Linear Unit (ReLu), Leaky ReLU, parametric ReLU, Randomized ReLU, Exponential Linear Unit, Maxout and Probout. Methods used for convolution neural network are as follows : LeNet 5 is a 8 Layered architecture

1646 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

used for image recognition. AlexNet uses GPU to train the high resolution large dataset with 1000 different classes. Alexnet does faster processing compared to LeNet5 due to its non saturating neurons. Overfitting problem is reducing by using dropout method. Zeiler and Fergus proposed ZFNet as multi-layered deconvolutional neural network with 8 layers. ZfNet uses GPU to classify large image dataset and dropout to reduce the parameters.Network – In-Network ( NIN) is proposed to implement per-pixel nonlinearity and to reduce the number of parameters. NIN uses 1 x 1 convolutional network with ReLU activation. Global average pooling is used to produce logit vectors.Region based Convolutional Neural Network (R-CNN) detects the object by extracting the features with region proposals. Forward computions is required to choose the thousand number of region proposals from a single image increase the processing time. To overcome this, Fast RCNN is proposed to perform the forward computation as a whole. Entire image is given as an image for processing each region proposals at a time unlike RCNN where region proposals are given as a input. Faster R-CNN uses region proposal network, thereby reducing the number of generated region proposal and maintaining the accuracy of object detection. Mask R-CNN uses fully convolutional layer for locating the objects at pixel level. Comparatively Mask RCNN enhanced object detection accuracy rather than Faster RCNN. GoogLeNet uses 22 convolutional layer for transfer learning. Optimization is increased when image classification is done via GoogLeNet CNN model. Visual Geometry Group (VGG) is widely used in gait recognition when compared to other CNN model since it provides high accuracy. VGG performs training faster since it has reduced number of parameters. Table 3 provides the overview of recent work in gait recognition using deep architectures.

TABLE 3Review of recent gait recognition using deep architectures.

GAIT YEA REFERENC APPROA CLASSIFIE ACCURA FEATUR DATASET PURPOSE R E CH R CY ES

2006 [40] Yu at al Model GEI CASIA B Multilayer Identificatio - free autoencoder n 2015 [28] Chao Yan Model GEI CASIA B CNN Identificatio 96% et al free n

2015 [29] Perlin et al - Cloth and H3D CNN Pattern for - gender dataset, clothes and ViPer gender dataset, HATdb 2016 [48] Hannink et Model acclerator GAITRite CNN with Identificatio - al Based , modifications n gyroscope parameter s 2017 [37] Alotaibi et Model CASIA B CNN, keras Identificatio Mis- al free ,python n classificatio n rate - <0.15% 2017 [41] Sokolova Model Optical TUM- CNN - VGG Identificatio 16 hours , and Konushin free Flow GAID n 97% dataset 2017 [42] Dehzangi Model Gait Sensor Deep CNN Identificatio 97.60% et al Based measurem collected n ents data 2017 [30] Alotaibi et Model GEI CASIA B Deep CNN Identificatio 98.30%

1647 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

al free n 2018 [31] Gadaleta et Model acclerator GaitDataset CNN Identificatio 94% al Based , McGill CS n gyroscope parameter s 2018 [32] batchulunn Model GEI DBMHI- CNN - VGG Authenticati 100% et al free DB1 on Identificatio n 2018 [47] Liu et al Model GEI CASIA B CNN - VGG Identificatio 89.62% free n 2018 [39] Wu et al Model GEI CASIA B, Deep CNN Identificatio 86% free OU-ISIR with location n weight descriptor 2018 [53] Yuan et al Model acclerator GITHUB CNN Authenticati 87% Based , on gyroscope Identificatio parameter n s 2018 [54] Batchuluun Model camera Sensor LSTM CNN Authenticati 97% et al free image collected on data Identificatio DBMHI- n DB1,DBM HI-DB2 2018 [55] Sun et al Model spectogra Self CNN, Stacked Authenticati 96.36% Based m collected Auto encoder on dataset Identificatio from radar n and microwave anechoic chamber 2018 [43] Yu et al Model GEI CASIA B CNN- Identificatio 96.50% free BOLTZMAN n N MACHINE 2018 [57] Liu et al Model GEI OU-ISIR CNN, Identificatio 88 free SIAMESE n NEURAL NETWORK 2018 [49] Tong et al Model GEI CASIA-B, SPATIAL Identificatio 95.67% free OU-ISIR, TEMPORAL n and CMU DEEP NN MoBo 2018 [58] Takemura Model GEI OU-ISIR CNN Identificatio 95% et al free n 2018 [59] Yao et al Model Skeleton CASIA B CNN Identificatio 92.09% free Gait n image 2018 [64] Verlekar et Model GEI DAI , DAI2 CNN – LSTM Clinical gait 95% al free INIT disease

1648 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

classificatio n 2018 [68] Wang et al Model Threadnill Recorded CNN- Gesture 99.98% based sensor with Body AlexNet recognition weight support threadmill 2019 [27] Maryam et Model GEI OULP and CNN Identificatio - al free Casia-B n

2019 [60] Wang et al Model GEI CASIA B , CNN- 2 Authenticati 94% free OU-ISIR BRANCH on

2019 [52] Hawas et al Model GEI CASIA B CNN Identificatio 95% free n 2019 [56] Wang et al Model GEI CASIA A, CNN Identificatio 95% free B, OU- n ISIR 2020 [63] Shao et al Model GEI CASIA B CNN , LeNet5 Identificatio 98.3% free n with (normal) Covariates 89.2 % ( Backpack) 95.8% ( Wearing coat)

Human Gait Recognition: Hannink et al [48] used convolutional neural network with ReLU activation function to extract the gait feature from wearable sensor dataset. Tong et al [49] used spatial-temporal convolutional neural network to recognize the gait in multi-view angle with 92.54% accuracy. Liu et al[47] combined the convolutional neural network and support vector machine to predict the gender of an individual using the CASIA B dataset with 87.94%. The convolution model VGGNet-16 is used to train the network with GEI input while SVM classify the descriptors provided by optimizing the weight parameters. Shao et al [63] proposed improved LeNet5 network combined with Residual Network to accelerate the convergence speed and to improve the recognition rate. CASIA B dataset is trained and tested with the improved LeNet model of CNN. They focused to work on the covariates like Viewing angle, backpack, jacket and speed. The accuracy of gait recognition with backpack, wearing coat and normal are 89.2%, 95.8% and 98.3% respectively. Maryam et al [27] used FCN (Fully Convolutional Neural Network) to reconstruct the incomplete Gait Energy Image(GEI) into a complete GEI. Experiments were carried out in OU-ISIR Biometric – OULP database and CASIA B database to obtain the complete GEI from 0.1 part of a gait cycle. This method helps in matching view-variant incomplete gait image.

Clinical Gait Analysis: Verlekar[64] proposed a fine tuned VGG-19 Deep architecture to classify gait disease type. The pathological dataset with diplegia, Hemiplegia, neuropathy, parkinson and normal individual gait dataset is used to experiment the proposed model. The classification of normal and impaired gait type is performed with high accuracy. Wang et al [64] used Alexnet CNN model to perform the gesture recognition system with 99.98% accuracy. Five different gestures such as moving forward, moving backward, moving left, moving right and stop are recognized with Body Weight Support Treadmill (BWST) Platform. This recognition system helps in lower limb rehabilitation. This system is controlled by a trainer to monitor the patient walking sequence in BWST platform.

1649 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

6. DISCUSSION A comprehensive survey of gait recognition and analysis methods based on wearable sensors and vision based methods, various features and deep architecture techniques were presented. A review of gait databases with different techniques used to capture gait features were given. Figure 7 depicts the recurrently used gait features in various fields and obtained the best accuracy with the recent techniques of deep architectures.

(a) (b)

(c)

Fig. 7. Statistical figures based on the referred papers a) gait features and usage b) Gait analysis applied in the various fields c) Different combinations of Deep architectures used in gait recognition.

Future Research Directions. The scope of future research provides an insight into remote health monitoring, surveillance and robotics. Immense opportunities are open to enhance and innovate in the current technological

1650 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020 environment in the following fields. Application of Multivariate statistical analysis and deep learning techniques is becoming increasing popular in the area of clinical gait analysis, thereby providing new insights into human gait control. According to Global Burden of Disease (GDB), musculoskeletal disorder is estimated to witness the second highest growth rate across the globe. In recent years, number of people with abnormal walking posture is increasing. Research were carried out to differentiate the healthy walking posture and abnormal walking posture. Sports professionals utilized gait analysis to monitor the movement of leg in terms of angle, position etc. The measurement and assessment of the players improved their performance as well as decreased the risk of injury. Model free method is preferred quickly to interpret about the movement of the players. Trajectory movements of model based human gait helps to design the walking nature of robots.Human gait recognition system is becoming widely spread due to its nature of identifying individual without their intervention with the system. Still there exist some limitations such as impact of environmental changes, user location and orientation, sensing multi-user activity and security considerations. To perform the longitudinal study over the wide range of covariates, there is a need for a large multi- modality, multi-covariate standard dataset. Our future work is to carry out the experiments with the recent network architectures and novel approaches with the huge dataset of different view angles, clothing and walking style promoting the better automatic gait recognition and gait analysis system.

7. CONCLUSION The most interesting research in biometrics is automatic gait recognition when compared to other human unique features. Though there are many approaches to overcome the variations in gait recognition, still there are challenges to recognize a person. Challenges are distinctive gait datasets, degree of stability in identifying, sensing modality, covariates and spoofing effects, and exploring new algorithms. From the developmental perspective, human gait recognition has gained maturity in adopting recent methodologies to provide the high accuracy and the same was analyzed and studied in the view of vision based system. In this paper, the basic knowledge about the human gait is identified and explored. The comparative analysis with the existing techniques of gait recognition were discussed. In this work, the survey of recent deep architecture model on human gait identification, authentication and clinical applications were discussed. Future research directions and challenges were identified.

8. REFERENCES

[1] http://classics.mit.edu/Aristotle/gait_anim.html [2] George A. Bekey, Fellow, Ieee, Chi-Wu Chang, Jacqueline Perry, And M. Mark Hoffer ,“Pattern Recognition of Multiple EMG Signals Applied to the Description of Human Gait”, 1977. [3] In-Sheng Cheng, S. H. Koozekanani, Member, IEEE, and M. T. Fatehi, Student Member, IEEE, “A Simple Computer-Television Interface System for Gait Analysis”,1974 [4] Roger Clarke. 1994, “Human identification in information systems”, Management challenges and public policy issues. Info. Technol. People 7, 4 (1994), 6–37. [5] Christopher L. Vaughan, Brian L. Davis, C. O. Jeremy et al. 1999. “Dynamics of human gait”, Human Kinetics Publishers. [6] Armin Bruderlin, Thomas W. Calvert, School of Computing Science Simon Fraser University Burnaby, British Columbia, Canada V5A 1S6, “Goal-Directed, Dynamic Animation of Human Walking”, Computer Graphics, Volume 23, Number 3, July 1989. [7] Michael McKenna and David Zeltzer, “Dynamic Simulation of Autonomous Legged Locomotion” , Computer Graphics and Animation Group The Media Laboratory Massachusetts Institute of Technology Cambridge, MA , Computer Graphics, Volume 24, Number 4, August 1990 [8] Monika Köhle, Dieter Merkl, “Identification of gait patterns with self-organizing maps based on ground reaction force”, Institute of Software Technology, Vienna University of Technology, Proc. of the European Symposium on Artificial Neural Networks (ESANN’96). Bruges. Belgium. April 24-26. 1996.

1651 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

[9] Monika Kohle, Dieter Merkl, “Clinical Gait Analysis by Neural Networks: Issues and Experiences” , Institut fur Softwaretechnik, Technische Universitat Wien, A-1040 Wien, Austria, Tenth IEEE Symposium on Computer-Based Medical Systems,1997 [10] Monika KOhle, Dieter Merkl ,“Analyzing Human Gait Patterns for Malfunction Detection”, Institut for Softwaretechnik Technische UniversitSt Wien Favoritenstral~e 9-11/188 A-1040 Wien, Austria, SAC '00: Proceedings of the 2000 ACM symposium on Applied computing - Volume 1, 2000 [11] Harold C. Sun, Dimitris N. Metaxas, “Automating gait generation”, University of Pennsylvania, ACM SIGGRAPH 2001, 12-17 August 2001, Los Angeles, CA, USA, 2001 [12] Aaron E Bobick, Amos Y Johnson , “Gait Recognition Using Static, Activity-Specific Parameters” , GW Center/College of Computing Georgia Tech Atlanta, Electrical and Computer Engineering Georgia Tech Atlanta, GA 30332, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. [13] Arunachalam Somasundaram, Rick Parent,“3D Reconstruction Of Walking Behaviour Using A Single Camera”, SIGGRAPH Abstracts and Applications 2002 [14] Chiraz BenAbdelkadery, Ross Cutlerz, and Larry Davisy, “Motion-based Recognition of People in EigenGait Space”, University of Maryland, College Park, Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition (FGR02) ,2002 [15] James B. Hayfron-Acquah, Mark S. Nixon, John N. Carter, “ Automatic gait recognition by symmetry analysis”,Image, speech and Intelligent Systems Group, Department of Electronics and computer science, University of Southampton, Pattern Recognition Letters 24(2003) 2175- 2183, 2003 [16] Andrew Crossan, Roderick Murray-Smith, Stephen Brewster, James Kelly, Bojan Musizza, “Gait Phase Effects in Mobile Interaction”, CHI 2005 , Late Breaking Results: Posters, April 2-7 , Portland, Oregon, USA, 2005 [17] Anuradha Annadhorai, Eric Guenterberg, Jaime Barnes, Kruthika Haraga and Roozbeh Jafari Department of Computer Engineering The University of Texas at Dallas, Richardson, TX – 75083, USA , “Human Identification by Gait Analysis”, HealthNet'08, June 17, 2008, Breckenridge, CO Copyright 2008 ACM ISBN 978-1-60558-199-6/08/06,2008. [18] Yang Hu ; Konstantinos Sirlantzis ; Gareth Howells, “Optimal Generation of Iris Codes for Iris Recognition”, IEEE Transactions on Information Forensics and Security , Volume: 12 , Issue: 1 , Jan. 2017 [19] Imed Bouchrika and Mark S. Nixon, “Model-Based Feature Extraction for Gait Analysis and Recognition”, in:proceedings of Computer Vision/Computer Graphics Collaboration Techniques, Springer Berlin Heidelberg,2007, pp.150-160. [20] C. Yam, M. Nixon, J. Carter, Automated person recognition by walking and running via model- based approaches, Pattern Recognition.37(5)(2004) 1057 -1072. [21] A.F. Bobick, A.Johnson, Gait recognition using static , activity-specific parameters,in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1,2001,pp.423-430. [22] C.Yam, M.Nixon, Gait recognition, model-based,Encylopedia of Biometrics,2015,pp,799-805. [23] Dash, Manoranjan, and Huan Liu. 1997. Feature selection for classification. Intelligent Data Analysis 1(3): 131–156. [24] Guyon, Isabelle, Amir Saffari, Dror Gideon, and Gavin Cawley. 2010. Model selection: Beyond the bayesian/frequentist divide. Journal of Machine Learning Research 11: 61–87. [25] Model Selection. 2015. http://bactra.org/notebooks/model-selection.html. [26] Michael Otero. 2005. Application of a continuous wave radar for human gait recognition. Proceedings of SPIE—The International Society for Optical Engineering 5809, 538–548 [27] Maryam Babaee , Linwei Li , Gerhard Rigoll, "Person identification from partial gait cycle using fully convolutional neural networks", Neurocomputing 338 (2019) 116–125 [28] C. Yan , B. Zhang , F. Coenen , Multi-attributes gait identification by convolutional neural networks, in: Proceedings of the Eighth International Congress on Im- age and Signal Processing (CISP), IEEE, 2015, pp. 642–647 [29] Perlin, H. A., & Lopes, H. S. (2015). Extracting human attributes using a convolutional neural network approach. Pattern Recognition Letters, 68, 250–259

1652 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

[30] Alotaibi, M., & Mahmood, A. (2017). Improved gait recognition based on specialized deep convolutional neural network. Computer Vision and Image Understanding, 164, 103–110. [31] Gadaleta, M., & Rossi, M. (2018). IDNet: Smartphone-based gait recognition with convolutional neural networks. Pattern Recognition, 74, 25–37. doi:10.1016/j.patcog.2017.09.005 [32] Batchuluun, G., Naqvi, R. A., Kim, W., & Park, K. R. (2018). Body-movement-based human identification using convolutional neural network. Expert Systems with Applications, 101, 56– 77. doi:10.1016/j.eswa.2018.02.016 [33] Wang L et al (2003) Silhouette analysis-based gait recognitionfor human identification. IEEE Trans Pattern Anal Mach Intell 25(12):1505–1518 [34] Jeevan M, et al (2013) Gait recognition based on gait pal and pal entropy image. in Image Processing (ICIP), 2013 20th IEEE International Conference on. IEEE [35] Degui Xiao and Lei Yang. 2008. Gait recognition using Zernike moments and bp neural network. In IEEE International Conference on Networking, Sensing and Control. 418–423. [36] D. Liu , M. Ye , X. Li , F. Zhang , L. Lin , Memory-based gait recognition, in: Pro- ceedings of the British Machine Vision Conference (BMVC), 2016 . [37] X. Ben , P. Zhang , W. Meng , R. Yan , M. Yang , W. Liu , H. Zhang , On the distance metric learning between cross-domain , Neurocomputing 208 (2016) 153–164 . [38] S. Yu , H. Chen , Q. Wang , L. Shen , Y. Huang , Invariant feature extraction for gait recognition using only one uniform model, Neurocomputing 239 (2017) 81–93 . [39] Wu, H., Weng, J., Chen, X., & Lu, W. (2018). Feedback weight convolutional neural network for gait recognition. Journal of Visual Communication and Image Representation, 55, 424–432. doi:10.1016/j.jvcir.2018.06.019 [40] Shiqi Yu, Daoliang Tan, and Tieniu Tan. 2006. A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition. In International Conference on Pattern Recognition. 441–444. [41] A. Sokolova and A. Konushin. 2017. Gait recognition based on convolutional neural networks. In International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences. 42, 42, 207–212. [42] O. Dehzangi, M. Taherisadr, and R. Changalvala. 2017. IMU-based gait recognition using convolutional neural networks and multi-sensor fusion. Sensors 17, 12, 2735. [43] Yu, T., Liu, X., Mei, Y., Dai, C., & Yan, J. (2018). Identification by Gait Using Convolutional Restricted Boltzmann Machine and Voting Algorithm. 2018 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData). [44] Hinton, Geoffrey; Salakhutdinov, Ruslan (2012). “A better way to pretrain deep Boltzmann machines” NIPS'12: Proceedings of the 25th International Conference on Neural Information Processing Systems. [45] Lukas Vareka and pavel Mautner “Stacked Autoencoders for the P300 Component Detection”, Neuroinformatics Research Group, Department of Computer Science and Engineering, Faculty of Applied Sciences, University of West Bohemia, Pilsen, Czechia, (2017). [46] Sun, L., Yuan, Y.-X., Zhang, Q., & Wu, Y.-C. (2018). Human Gait Classification Using Micro- Motion and Ensemble Learning. IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium. [47] Liu, T., Ye, X., & Sun, B. (2018). Combining Convolutional Neural Network and Support Vector Machine for Gait-based Gender Recognition. 2018 Chinese Automation Congress (CAC). [48] Hannink, J., Kautz, T., Pasluosta, C. F., Gasmann, K.-G., Klucken, J., & Eskofier, B. M. (2017). Sensor-Based Gait Parameter Extraction With Deep Convolutional Neural Networks. IEEE Journal of Biomedical and Health Informatics, 21(1), 85–93. [49] Tong, S., Fu, Y., Yue, X., & Ling, H. (2018). Multi-View Gait Recognition Based on A Spatial-Temporal Deep Neural Network. IEEE Access, 1–1. [50] Mason, James Eric, Issa Traore, and Isaac Woungang. "Gait Biometric Recognition", Machine Learning Techniques for Gait Biometric Recognition, 2016.

1653 European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 06, 2020

[51] Singh, M., Aujla, G. S. S., Singh, A., Kumar, N., & Garg, S. (2020). Deep Learning based Blockchain Framework for Secure Software Defined Industrial Networks. IEEE Transactions on Industrial Informatics, 1–1. [52] Hawas, A. R., El-Khobby, H. A., Abd-Elnaby, M., & Abd El-Samie, F. E. (2019). Gait identification by convolutional neural networks and optical flow. Multimedia Tools and Applications. [53] Wei Yuan and Linxuan Zhang, Gait Classification and Identity Authentication Using CNN. Methods and Applications for Modeling and Simulation of Complex Systems, 2018, Volume 946,pp 119-128. [54] Batchuluun, G., Yoon, H. S., Kang, J. K., & Park, K. R. (2018). Gait-based Human Identification by Combining Shallow Convolutional Neural Network-stacked Long Short-term Memory and Deep Convolutional Neural Network. IEEE Access, 1–1. [55] Sun, L., Yuan, Y.-X., Zhang, Q., & Wu, Y.-C. (2018). Human Gait Classification Using Micro- Motion and Ensemble Learning. IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium. [56] Wang, X., Zhang, J., & Yan, W. Q. (2019). Gait recognition using multichannel convolution neural networks. Neural Computing and Applications. [57] Liu, W., Zhang, C., Ma, H. et al. Learning Efficient Spatial-Temporal Gait Features with Deep Learning for Human Identification. Neuroinform 16, 457–471 (2018). [58] Takemura, N., Makihara, Y., Muramatsu, D. et al. Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition. IPSJ T Comput Vis Appl 10, 4 (2018). [59] L. Yao, W. Kusakunniran, Q. Wu, J. Zhang and Z. Tang, "Robust CNN-based Gait Verification and Identification using Skeleton Gait Energy Image," 2018 Digital Image Computing: Techniques and Applications (DICTA), Canberra, Australia, 2018, pp. 1-7 [60] Wang, X., Zhang, J. Gait feature extraction and gait classification using two-branch CNN. Multimed Tools Appl 79, 2917–2930 (2020). [61] L. Hoyet, C. Spies, P. Plantard, A. Sorel, R. Kulpa and F. Multon, "Influence of Motion Speed on the Perception of Latency in Avatar Control," 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), San Diego, CA, USA, 2019, pp. 286-2863, doi: 10.1109/AIVR46125.2019.00066. [62] Mazilu, S., Hardegger, M., Zhu, Z., Roggen, D., Troester, G., Plotnik, M., & Hausdorff, J. (2012). Online Detection of Freezing of Gait with Smartphones and Machine Learning Techniques. Proceedings of the 6th International Conference on Pervasive Computing Technologies for Healthcare. [63] Shao, X., Nie, X., Zhao, X., Zheng, R., & Guo, D. (2020). Gait Recognition based on Improved LeNet Network. 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). doi:10.1109/itnec48623.2020.9084747 [64] Ver [65] Prado, A., Cao, X., Robert, M. T., Gordon, A. M., & Agrawal, S. K. (2019). Gait Segmentation of Data Collected by Instrumented Shoes Using a Recurrent Neural Network Classifier. Physical Medicine and Rehabilitation Clinics of North America. doi:10.1016/j.pmr.2018.12.007 [66] Hu, K., Wang, Z., Wang, W., Martens, K. A. E., Wang, L., Tan, T., … Feng, D. D. (2019). Graph Sequence Recurrent Neural Network for Vision-based Freezing of Gait Detection. IEEE Transactions on Image Processing, 1–1. doi:10.1109/tip.2019.2946469 [67] Jun, K., Lee, D.-W., Lee, K., Lee, S., & Kim, M. S. (2020). Feature Extraction Using an RNN Autoencoder for Skeleton-Based Abnormal Gait Recognition. IEEE Access, 8, 19196–19207. [68] Wang, P., Zhang, Q., Li, L., Ru, F., Li, D., & Jin, Y. (2018). Deep learning-based gesture recognition for control of mobile body-weight support platform. 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA). doi:10.1109/iciea.2018.8398001

1654