Muscle fatigue detection using : Image segmentation to extract the region of interest from thermograms

A thesis submitted to the Graduate School at the University of Cincinnati in partial fulfillment of the requirements for the degree of

Master of Science

in the Department of Electrical Engineering and Computing Systems, College of Engineering and Applied Science

November 2018 By Dhyanesh Ramamoorthy, Bachelor of Engineering in Computer Science and Engineering, Anna University, India, 2010

Thesis Committee: Dr. Dharma Agrawal, Advisor Dr. Wen Ben Jone Dr. Yizong Cheng

Abstract

A major public health problem which affects people from all walks of life is muscle injury. Muscle fatigue can be defined as a reversible decrease in the contractile strength of the muscle(s) that occurs after long lasting or repetitive muscular activity. Whenever a certain muscle or muscle groups are subjected to repetitive movements and pushed beyond their limits, it leads to muscle fatigue, which results in sub-optimal performance. Muscle fatigue, when not diagnosed and treated in the early stages, leads to muscle injuries which tend to have long-lasting effects. The current technologies which are used to detect and treat muscle fatigue are highly invasive and often require the personnel to visit specialized clinics.

The invasiveness and inaccessibility of muscle fatigue detection techniques have contributed to delayed diagnosis which leads to undesirable effects like permanent muscle damage. A non-invasive, cost-efficient method for detecting muscle fatigue is crucial.

Whenever a muscle(s) is put to use, the corresponding muscle temperature increases.

The human body maintains thermal homeostasis and it is achieved with the help of two processes called vasodilation and vasoconstriction. Vasodilation occurs when the temperature of a certain muscle group increases. The blood flow to this particular region is increased and the heat is absorbed by the blood capillaries and dissipated through the skin.

Infrared thermography is a non-radiating and contact-free technology which measures the surface temperature of objects. As skin surface temperature has been proven to be a good

i

indicator of localized muscle fatigue, Infrared thermography can be an effective non-invasive associative technology which can be used to detect muscle fatigue.

In our study, we have conducted experiments which involve test subjects performing repetitive exercises involving the upper body muscle groups. The thermal images of the test subjects were taken before and after the exercise routine. We have reviewed the widely used image segmentation and image classification algorithms used to extract the regions of interest from an image and implemented two algorithms which are suitable for analyzing low signal-to-noise thermal images. The changes in temperature values of the regions of interest have been tabulated and the results indicate a clear correlation between muscle fatigue and increase in temperature.

ii

iii

This work is dedicated to my dear parents Dr. M. Ramamoorthy and Dr. V. Renugadevi & my beloved wife Mrs. Vithusini Senthil Kumar

iv

Acknowledgements

First and foremost, I am deeply grateful and thank my advisor and committee chair,

Dr. Dharma P Agrawal for his constant encouragement, guidance and support throughout the duration of my master’s degree. I would like to thank Dr. Wen Ben Jone and Dr. Yizong

Cheng for serving on my thesis defense committee.

I am also thankful to my colleagues at the center for distributed and mobile computing. Their valuable feedback and suggestions during our lab meetings were instrumental in completing my thesis research.

Finally, I must express my gratitude, love and affection to my parents and my wife for believing in me and providing me with constant support and encouragement during all these years of my study.

v

TABLE OF CONTENTS

1. Introduction...... 1

1.1 Research Motivation ...... 1 1.2 Muscle Fatigue ...... 1 1.3 Infrared thermal imaging (IRTG) in Medical applications ...... 3 1.4 Advantages of using IRTG for fatigue analysis ...... 4 1.5 Purpose and Scope of this study ...... 6 2. Foundation ...... 9

2.1 Heat regulation in Human body ...... 9 2.2 Hypothermia and Muscle Fatigue ...... 10 2.3 Infrared Thermal Imaging ...... 10 2.4 Thermal Energy and Emissivity...... 13 2.5 Thermographic ...... 15 3. Experimentation ...... 18

3.1 Intended purpose of the experiments ...... 18 3.2 Guidelines ...... 18 3.3 Participant Selection Criteria ...... 20 3.4 Instruments used for the Experiments ...... 21 3.5 Experimental Results ...... 23 4. Analysis ...... 29

4.1 Thermal Image processing ...... 29 4.2 Thermal Image processing Algorithms ...... 30 4.3 Image Segmentation...... 32 4.4 Image Classification...... 44 5. Results & Conclusion ...... 52

5.1 Analysis of the ROI extraction algorithms ...... 52 5.2 Conclusion ...... 61 6. Future Work ...... 63

7. Bibliography ...... 64

vi

LIST OF FIGURES

Figure

2.1 The and the of Infrared waves ...... 11

2.2 A simplified block diagram of the workings of a thermal camera ...... 15

2.3 Process flow diagram involved in capturing a thermal image ...... 16

3.1 A manual analysis of the first testsubject’s thermograms ...... 24

3.2 A manual analysis of the second testsubject’s thermograms ...... 25

3.3 A manual analysis of the third testsubject’s thermograms ...... 26

3.4 A manual analysis of the fourth testsubject’s thermograms ...... 27

3.5 A manual analysis of the fifth testsubject’s thermograms ...... 28

4.1 The original thermal image before segmentation...... 35

4.2 A histogram plot of the intensities of the original thermal image ...... 36

4.3 The Gaussian-smoothed thermal image ...... 37

4.4 The thermal image classified using 2-level Otsu segmentation ...... 38

4.5 The thermal image classified using 3-level Otsu segmentation ...... 39

4.6 Block diagram of the Canny edge detection algorithm ...... 41

4.7 The Canny Edge detector applied to a thermal image ...... 43

4.8 An illustration of integral image calculation ...... 47

4.9 Calculating the sum of the within a rectangle using integral image...... 48

4.10 Block diagram explaining the functioning of a Cascade classifier ...... 49

5.1 ROI extraction in Subject 1 and Average temperature calculation...... 53

5.2 ROI extraction in Subject 2 and Average temperature calculation...... 54

5.3 ROI extraction in Subject 3 and Average temperature calculation...... 55

vii

LIST OF FIGURES(Cont.)

Figure

5.4 ROI extraction in Subject 4 and Average temperature calculation...... 56

5.5 ROI extraction in Subject 5 and Average temperature calculation...... 57

5.6 Average Temperature of ROI from 3-level Otsu segmentation...... 58

5.7 Average Temperature of ROI from 2-level Otsu segmentation...... 59

5.8 Average Temperature of ROI from Haar-cascade classification ...... 59

5.9 Change in average temperature of ROI…………………………………………60

viii

1. INTRODUCTION

1.1 Research Motivation

A major public health problem which affects people from all walks of life is muscle injury. Whenever a certain muscle or muscle groups are subjected to repetitive movements and pushed beyond their limits, it leads to muscle fatigue, which eventually results in muscle injury. Muscle fatigue, when not diagnosed and treated in the early stages, leads to muscle injuries which tend to have long-lasting effects [1]. Sports personnel and military personnel belong to groups that are at a very high risk for muscle fatigue and muscle tear leading to muscle injuries. Diagnosing muscle injuries is not always an easy task, and even more difficult is identifying the exact region of muscle fatigue without the help of a known invasive technology.

1.2 Muscle Fatigue

Muscle fatigue can be defined as a reversible decrease in the contractile strength of the muscle(s) that occurs after long lasting or repetitive muscular activity. Muscle fatigue is a condition of the muscle in which its capacity to produce maximum voluntary action, or to perform a series of repetitive actions, is reduced. A recent research study shows that more than 60 million industrial workers suffer from job- related muscle fatigue and injuries and they go unnoticed for a long period of time [1].

A study in 2011 shows that 10-55% of all muscle injuries occur during sports activities

[2].

1

1.2.1 Muscle fatigue and A research study about muscle fatigue mechanisms [4] proves that whenever a muscle performs repetitive tasks, the localized muscle temperature increases and eventually leads to muscle fatigue. The data yielded from a study by Ludwig et al., in

2014 provides confirmatory proof that the surface skin temperature of a muscle(s) increases when it is fatigued [13]. Muscle fatigue leads to decreased performance and increases risk of injury in athletes and military personnel.

The increase in localized skin temperature directly correlates with the level of muscle fatigue and decreased power output of the corresponding muscle groups.

Muscle fatigue level has an indirect correlation with a person's ability to function efficiently at his maximum potential and also has a direct correlation with his chances of developing an injury. It is vital to monitor the onset and presence of muscle fatigue to maximize performance and output.

1.2.2 Existing technologies to measure muscle fatigue One of the biggest challenges faced by the medical industry and health professionals in treating muscle injuries is identifying them at an earlier stage as muscle injuries tend to aggravate and lead to drastic consequences over the course of time. As of today, different techniques to diagnose and/or measure muscle fatigue are performed by highly invasive technologies like

2

1.Surface Electromyography (EMG),

2.Dynamometer, and

3.Digital Inclinometer.

Medical professionals are constrained by the current technologies which are in use to diagnose muscle injuries in patients. A research survey [14] done in 2017 compares the currently existing methods to diagnose muscle injury and notes that all of them are invasive in nature. As skin surface temperature has been proven to be a good indicator of localized muscle fatigue, Infrared thermography can be considered as an effective non-invasive technology which can be used to detect muscle fatigue and muscle injuries.

1.3 Infrared thermal imaging (IRTG) in Medical applications

Infrared thermal imaging (IRTG) is a contact-free and a non-radiating technology. IRTG can be used to monitor and diagnose muscle fatigue in its early stages as studies have shown a direct correlation between muscle fatigue and increase in skin temperature. Infrared thermal images are captured with the help of infrared .

An infrared camera or a is used to detect radiation in the long- infrared range of the electromagnetic spectrum. Conventional optical cameras operate in the 400-700 nanometer range to capture visible whereas thermal imaging cameras operate in the 14,000 nanometer (14 µm) range to capture the thermal radiations invisible to human eye.

3

The pictures captured by a thermal imaging camera is referred to as thermograms and these thermograms can be analyzed and processed to gather a host of information about the picture it captures. Practical applications range from health-care, construction, material sciences to defense sectors. One of the biggest advantages of using thermal-imaging cameras over conventional cameras is their ability to record information without the presence of any ambient lighting. IRTG is also not affected by changes in climate like fog, rain, and thunderstorms. This enables the continuous monitoring of areas/subjects in any given environment irrespective of lighting, weather and climatic conditions.

1.4 Advantages of using IRTG for fatigue analysis

There are several reasons on why IRTG is a better option as compared to other currently used fatigue monitoring techniques like using wearable. Some of the compelling advantages of IRTG over existing techniques include

1. Contact-free technology,

2. Low cost-of-implementation/ Setup cost,

3. Uncontrolled environment, and

4. Low Image processing cost and complexity.

Contact-free technology

IRTG is a contact-free and non-radiating technology which is very safe as well as leads to unadulterated and reliable test dataset. Recent studies have shown that test subjects using wearable sensors do not always produce the most reliable test data as

4

their behavior changes consciously while using wearable technology. IRTG doesn’t constrain the test subjects in anyway and experimental readings can be taken in the most candid way possible.

Cost-of-implementation/ Setup cost

IRTG monitoring can be established with the help of a single network capable thermal imaging camera whereas the other techniques need a host of other inter- dependent hardware and services to collect and analyses any useful data. This reduces the cost of setup and implementation by multiple factors. For instance, as shown in an

RF study [15] the received signal strength indicator (RSSI) variations can be a useful fatigue analysis technique only when the subject uses a wearable sensor and also is in close proximity to a plane of other beacon nodes to record changes in RSSI values.

Uncontrolled environment

IRTG can be used effectively in an uncontrolled environment to study the test subjects as well as during implementation. Most other techniques need subjects to be in extremely controlled environments to get any meaningful data which in turn makes these techniques ineffective for most practical implementation. Infrared cameras also do not need any ambient light sources to analyze the environment. Testing and implementation can be done in almost any environment as thermal imaging cameras record only changes in temperature.

5

Processing complexity

One of the main constraints while using conventional cameras for monitoring is their processing complexity. The pixel resolution from conventional cameras need to be very high (1080 x 1920) to analyze the frames and get any meaningful data. On the other hand, meaningful data can be obtained from a 160x120 thermal imaging cameras.

This reduces the image processing cost and complexity while using IRTG and it might be possible to do real-time analysis which is highly cost-ineffective while using conventional cameras.

1.5 Purpose and Scope of this study

A lot of people working in jobs that require intense physical work are at risk of developing muscle fatigue in various parts of their body. Athletes, industrial workers, nurses, military personnel are few examples of people working in environments that constantly demand physically intensive labor. Multiple research studies, such as the study by Paul McCauley in 2012 about work-related musculoskeletal injuries and prevention [1], another study by Hongwei Hsiao and Nancy Stout in 2010 about occupational injury protection [3] have been conducted to educate and prevent these types of muscle injuries from occurring. These are examples of few studies which survey and stipulate detailed guidelines to be followed by personnel to prevent muscle injuries. Despite these measures, accidents do occur, and the risk of muscle injury persists mainly due to ineffective screening methods for muscle injury.

6

Infrared thermal imaging has not yet been properly looked at as a possible solution for applications like fatigue monitoring and biometric authentication. Research work by Hildebrandt et al., [5] shows immense potential for infrared imaging in medical imagery and especially in sports medicine. Several research works done in the field of infrared imaging and medical applications point out that IRTG shows immense potential for being a novel, efficient, non-invasive, computationally cost-effective associative approach towards solving these problems [7,8,12]. The purpose of this study is to conduct experiments to induce localized muscle fatigue in volunteers and record the changes using thermal cameras and employ thermal image processing algorithms to identify potential use of IRTG as an associative method in diagnosing early stages of muscle fatigue.

Another goal of the study is to see if the results from the experiment can support the case for using infrared thermography as a clinical diagnostic tool, primarily as an enhancer of the physical examination to detect inflammatory situations.

The study has been divided into 6 chapters and each chapter focuses on an important aspect of the study.

Chapter 2 explains the foundation of muscle fatigue and infrared thermography in detail and also surveys the current literature in analyzing muscle fatigue

Chapter 3 details the experimental part of our study. It explains in detail the guidelines followed during the selection of volunteers for the study, protocols followed

7

during the experiments and detailed information on all systems used for recording information during the process.

Chapter 4 surveys the state-of-the art in thermal image processing techniques.

A detailed explanation on the various image processing algorithms which were considered for our study and explains the reason for choosing cascading classifiers as the method of choice.

Chapter 5 analyzes the processed thermal images, measures the temperature changes in the regions of interest from each thermogram. The results are tabulated, and graphical versions are discussed in detail.

Chapter 6 discusses the pros and cons of the image segmentation techniques we have used and possible future work that can be done to add more features to our work.

8

2. FOUNDATION

2.1 Heat regulation in Human body

In order to understand muscle fatigue and its correlation to skin temperature, it is important to understand the heat regulation mechanism in the human body. Several research works have been done in the domain of analyzing muscle fatigue and its correlation with muscle temperature. Some of the pivotal research in the area of hypothermia and muscle fatigue has been reviewed in this chapter.

The hypothalamus is the part of the human brain which is responsible for heat regulation and maintains the homeostasis in body temperature(98.6F). The hypothalamus works with other parts of the human body's temperature-regulating systems, such as the skin, sweat glands and blood vessels — the vents, condensers and heat ducts of a human body's heating and cooling system. This is achieved with the help of two vital phenomena namely vasodilation and vasoconstriction. A pivotal study conducted by Hooshmand [16] explains the intricate relationship between the sympathetic nervous system and heat regulation within the human body in detail.

Vasodilation refers to the dilating and opening up of the blood vessels near a muscle group to carry more blood to that particular region and vasoconstriction refers to the converse, where constriction of the blood vessels happens in order to restrict blood supply to a particular region in the body.

9

Vasodilation occurs when the body temperature has increased considerably, and vasoconstriction occurs when the body temperature has decreased considerably. When the hypothalamus senses that the body temperature is too high, it sends impulses which cause blood vessels supplying the capillaries in the skin to dilate. This is called vasodilation. The increased blood flow to the surface tissues under the skin means that more heat is lost/radiated.

2.2 Hypothermia and Muscle Fatigue

As shown in a study by Marconnet et al., [4] muscle fatigue, when not detected earlier could lead to full-blown injuries thereby putting the personnel at great risk. One of the recent challenges involve detecting and analyzing muscle fatigue in its earlier stages and preventing any irreparable damages to the personnel. Technologies to detect muscle fatigue in earlier stages are still being discussed and researched and yet no single effective solution has been arrived at [9,14].

2.3 Infrared Thermal Imaging

2.3.1 The Infrared and the Electromagnetic spectrum Sir , an astronomer, discovered infrared in 1800. He observed the presence of an invisible radiation in the EM spectrum which caused temperature changes during his experiments. He also noticed that the energy levels of this invisible radiation were much lower as compared to the visible red light.

10

Figure 2.1 The Electromagnetic spectrum and the wavelengths of Infrared waves

Figure 2.1 represents the electromagnetic spectrum. The EM spectrum is divided into multiple sections or bands based on their wavelengths. The human eye can detect electromagnetic waves in the wavelengths between 390 nm and 700 nm and hence the name visible light. As seen in the figure, visible light constitutes only a very small portion of the entire electromagnetic spectrum.

The IR section of the EM band ranges from the end of the visible red light at

700 nanometers to the portion of the EM spectrum at 1mm. The wavelengths and energy levels of infrared radiation are much lower than visible light.

Similar to other electromagnetic radiation, infrared has inherent radiant energy and is classified as a particle (photon) as well as a wave.

11

2.3.2 Infrared Thermography (IRTG) The infrared portion of the electromagnetic spectrum can further be classified into 5 different sections based on their wavelengths. They are :

• Near infrared (0.75-1.4 µm),

• Short- infrared (1.4-3 µm),

• Mid-wavelength infrared (3-8 µm),

• Long-wavelength infrared (8-15 µm), and

• Far infrared (15-1000 µm) .

The thermal cameras are manufactured to detect the electromagnetic radiation in the long-wavelength infrared (8-15 µm) section of the electromagnetic spectrum.

Thermal radiations are emitted by all objects which are at a temperature above absolute zero (Kelvin). Infrared Thermography or infrared thermal imaging makes use of these thermal IR cameras to capture the images of this radiation and the resulting images are called as thermograms. The thermograms are hence visual displays of the infrared radiation emitted by an object.

12

2.4 Thermal Energy and Emissivity

According to the laws of thermodynamics, heat transfer occurs constantly between two objects of different inherent temperature. It also states that heat transfer occurs through one of the three heat transfer mechanisms namely conduction, convection, and . Radiant heat exchange is the underlying phenomenon that makes infrared thermography possible. A warmer object is always emitting radiant energy as infrared to a nearby cooler object which keeps absorbing it until an equilibrium has been obtained.

Emissivity is the measure of how effectively the surface of a material emits energy. Emissivity is formally defined as the ratio of the energy radiating from a material's surface to that radiating from a blackbody (which is the perfect emitter) at the same wavelength and temperature and also under the exact same viewing parameters.

It should be noted that emissivity is a property which changes from one material to the other. The variation is between 0 and 1. For instance, the emissivity of human skin is 0.98 whereas the emissivity of unoxidized gold is 0.02. Emissivity is one of the most significant properties for non-contact temperature readings with a higher precision.

13

The following table gives a better idea on the range of emissivity for a few common elements.

Material Thermal Emissivity

Gold(unoxidized) 0.02

Silver 0.05

Zinc(polished) 0.10

Cobalt 0.20

Chromium 0.10

Aluminium (polished) 0.05

Aluminium (anodized) 0.77

Brass (oxidized) 0.61

Glass 0.92

Carbon (purified) 0.80

Ice 0.97

Stainless Steel 0.59

Skin (Human) 0.98

Water (distilled) 0.95

14

2.5

The first modern thermographic camera was designed and developed in the year

1920 by a Hungarian physicist and researcher Kálmán Tihanyi. The motivating factor behind the discovery of the thermographic camera was its possible applications during wartime specially to detect enemy’s flights during night time when there is absence of visible light.

Simply put, the thermal camera captures the infrared radiations just like the way optical cameras capture visible light on to a film or a signal processing board. The technology behind the functioning of the thermal camera can be explained in brief detail such as follows.

Figure 2.2 A simplified block diagram of the workings of a thermal camera

1. The focusing lens of an infrared camera is made from or Sapphire instead of glass, as glass tends to block long wave infrared rays from entering.

2. The infrared light is then focused onto an array of infrared detectors (usually ) which produce a highly detailed thermogram based on the temperature pattern.

15

3. The thermogram information is converted to electric impulses.

4. The electric impulses are sent to the signal processing unit of the camera which then converts them into the corresponding image based on the intensity of thermal radiation at each pixel of the thermogram.

A more detailed block-diagram visualization of the inner workings of a thermographic camera is shown below

Figure 2.3 Process flow diagram involved in capturing a thermal image

Infrared thermographic cameras are extremely sensitive to radiant energy and very minute changes in temperature can be recorded. Most of the thermographic cameras available commercially can easily detect changes in the thermal radiations with an accuracy of up to 0.1 °C. Specialized thermal cameras which are used for medical and military purposes can detect changes in temperature with an accuracy of 0.01°C.

16

Thermal imaging cameras are generally more expensive than conventional cameras and higher-end state-of-the art IR thermal cameras are export-restricted due to their applications in military use. A study performed in 2011[6] and another study conducted in 2016 [17] have experimented with using thermal imaging cameras for capturing thermal images of the human body to detect possible muscle injuries and stress. The findings from the study show that there is a correlation between the temperature changes recorded using the infrared cameras and the change in a person’s activity level.

A case study done in 2015 [10] investigates the option of using IRTG as a possible associative screening technology to detect muscle fatigue and provides results which show that infrared imaging is a promising technology in the field of sport medicine to identify fatigue and injury. The aim of this research is to use the technology of infrared thermography (IRTG) and approach the problem of early detection/onset of muscle fatigue and prevent permanent, irreparable muscle injury. The key area of focus in this study will be the autonomous processing of thermal images using image segmentation and image classification algorithms to locate and identify the region of interest from the background/noise of the thermal images to aid in the measurement of temperature changes.

17

3. EXPERIMENTATION

3.1 Intended purpose of the experiments

The primary purpose of the experiments conducted for this study are to perform a feasibility analysis of using low spatial resolution thermal cameras to detect changes in localized skin surface temperature. It can also be argued that another important goal of this study is to analyze and detect the perfect regions of interest to measure the skin temperature before and after physical activities.

3.2 Guidelines

We decided to follow the following guidelines during our experimentation process based on the parameters followed in various infra-red thermal imaging studies.

A study by Moreira et al., has looked into the variables and parameters affecting infrared thermal readings in experiments involving humans and have published them in their study [33]. We have included these guidelines for the purpose of eliminating noise in the data obtained from the experiments and also to avoid interference from any external radiations.

Guidelines followed during our experiments:

1. The relevant individual data of the participants must be provided. Note: These could include age, sex, body mass, height, body mass index, ethnicity and whether they are

18

smokers or not. An indication of physical activity profile (e.g. frequency, duration, intensity) should be reported.

2. Participants should be instructed to avoid alcohol beverages, smoking, caffeine, ointments, cosmetics and showering for four hours before the assessment.

3. Ambient temperature and relative humidity of the location where the assessment took place must be recorded and reported.

4. The experiments and images should be recorded away from sources of infrared radiation (e.g., electronic devices, lightning) or airflow (e.g., under an air conditioning unit).

5. The manufacturer, model, accuracy and specifications of the camera used should be provided.

6. An acclimation period of 15 minutes by the participants in the experiments should be completed.

7. The camera should be positioned completely perpendicular to the regions of interest.

8. The thermal emissivity settings of the camera must be reported. Note: 0.98 of emissivity is suggested for a dry clean human skin surface.

9. The time of the day at which the images were taken should be reported.

10. If the skin is not dried (e.g., remove surface water), the drying method should be clearly described.

19

3.3 Participant Selection Criteria

A set of 5 people were chosen from the volunteers who signed up for the initial study. The volunteers were chosen irrespective of their fitness levels, diet, age, sex or ethnicity in order to get the most diverse resultant data set. All the volunteers were informed of the purpose of the study. No compensation was provided, and a written consent and injury waiver was obtained from all the volunteers before conducting the experiments.

For the initial set of experiments, we analyze the temperature changes in the bicep muscle group (upper arm) and the facial region of the participants which comprise the exposed skin surface in the experiment. The face and the arms of the participants need to be visible for the entirety of this initial experimental study. All participants were provided with a uniform active-wear in black/gray color which doesn’t reflect heat/light. Following a 15-minute acclimatization, they were asked to model for an initial set of photographs using the infra-red thermal camera.

The participants were asked to stand at varied distances from the aperture of the thermal camera for the initial pre-fatigued set of readings. The initial thermal images of the participants were recorded at distances of 1.5, 2.25, 3, and 4.5 meters respectively from the thermal camera. All the participants were asked to do 3 sets of bicep curls using dumbbells of their preferred weight (10 lbs to 25 lbs for each dumbbell).

Participants were asked to perform each set until failure (maximum repetitions) with a

20

rest of 3 minutes in between sets. The reason for 3 sets of maximum repetitions is to induce localized fatigue in their upper arm muscle groups without causing any muscle injury. After the exercise was complete, they were asked to model for a final set of photographs using the thermal camera. The final set of fatigued thermograms were recorded from the same varied distances mentioned earlier. At the end of our initial experiments, we had a data set comprising of thermograms of the rested and fatigued states of all the 5 volunteers.

3.4 Instruments used for the Experiments

3.4.1 FLIR C2 Camera The FLIR C2 is a full-featured, pocket-sized thermal camera designed for a wide range of applications. The versatility of FLIR C2 can be used to detect thermal radiations from varied sources like humans, objects, animals, hidden hot spots, energy waste, structural defects, plumbing clogs, HVAC issues, and other problems. Most importantly the affordability of FLIR C2 adds to the powerful advantage of thermal imaging.

FLIR C2 uses Focal Plane Array (FPA). Focal Plane Array is an image sensing device which consists of an array (generally rectangular) of light-sensing pixels at the focal plane of a lens. FPAs are used most commonly for imaging purposes (e.g. taking pictures or video imagery). We used a FLIR C2 camera using this technology to record the thermograms for our study.

21

Following are the salient features of FLIR C2:

FLIR C2 has a generous 45° field of view (FOV). As the detector is highly sensitive it captures even subtle temperature differences and thermal patterns so that even minute changes in body temperatures of people can be found accurately (up to a certain distance) likewise, one can also find leaks and building deficiencies.

4,800 individual thermal measurements are stored in FLIR C2 JPEG thermal images. This enables us to analyze and edit, using FLIR Tools software. The other unique feature of FLIR C2 is that it has FLIR's MSX technology which adds key details from the onboard visible light camera to the entire infrared image in real time. As a result, we get an all-in-one, undiluted thermal picture with visible light features that lets us identify the problematic heat pattern area in the image.

Specifications of FLIR C2:

Focal Length: 1.54 mm (0.061 in.),

Spectral Range: 7.5–14 µm,

IR Sensor: 80 × 60 (4,800 measurement pixels),

Storage Media: Internal memory stores at least 500 sets of images,

Operating Temperature Range: –10°C to +50°C (14 to 122°F),

Storage Temperature Range: –40°C to +70°C (–40 to 158°F),

Color palettes: Iron, Rainbow, Rainbow HC, Gray,

22

Digital Camera: 640 × 480 pixels,

Digital Camera Focus: Fixed focus,

Field of View: 41° × 31°,

Image Frequency: 9 Hz,

Minimum Focus Distance: Thermal: 0.15 m (0.49 ft.) MSX: 1.0 m (3.3 ft.),

3-inch Display (color): 320 × 240 pixels, and

Object Temperature Range: –10°C to +150°C (14 to 302°F)

The results from the experiment were processed manually using the FLIR Tools software suite. The thermal images of the volunteers in their initial non-fatigued state and their post-workout fatigued state are shown in the next pages.

3.5 Experimental Results

We have chosen three regions of interest where the skin is exposed to calculate any temperature changes. The manual analysis of thermal images is performed to identify any correlation between muscle fatigue and changes in surface skin temperature. The following results indicate a direct correlation between muscle fatigue and increase in skin temperature. Based on the results of our experiment, we were able to establish the relationship between skin temperature and muscle activity and in

Chapter 4, we have reviewed and identified image processing techniques which aid in the autonomous processing and analysis of thermal images.

23

Figure 3.1 A manual analysis of the first testsubject’s thermograms in Rested(top) and fatigued(bottom) states

24

Figure 3.2 A manual analysis of the second testsubject’s thermograms in Rested(top) and fatigued(bottom) states

25

Figure 3.3 A manual analysis of the third testsubject’s thermograms in Rested(top) and fatigued(bottom) states

26

Figure 3.4 A manual analysis of the fourth testsubject’s thermograms in Rested(top) and fatigued(bottom) states

27

Figure 3.5 A manual analysis of the fifth testsubject’s thermograms in Rested(top) and fatigued(bottom) states

28

4. ANALYSIS

4.1 Thermal Image processing

The experimental results and the feasibility analysis of the results clearly show a direct correlation between an increase in localized muscle temperature and muscle fatigue. It should be noted that the results from the feasibility study were processed and analyzed manually which takes considerable time and manpower. Manual image processing is also prone to clerical errors and doesn’t scale well as the number of thermal images tend to increase exponentially for larger systems. Also, real-time information about a possible muscle fatigue occurrence is more valuable and crucial compared to delayed analysis. All these reasons support a strong case for the development of an automated system which analyzes, classifies and processes the regions of interest (ROI) from the thermal images. The goal of this chapter is to analyze the thermal image processing techniques available and choose a method suitable for the purpose of this study.

4.1.1 Automation in Thermal Image Processing Autonomous thermal image processing is a rapidly growing area of research and lot of IR research is being currently explored in the fields of medicine and defense.

Thermal image processing in the sports domain is relatively nascent and not a lot of research work has been done in this area, especially in muscle fatigue analysis. Infrared

29

thermal image processing of athletes and sports personnel before and after an event shows promise in the field of localized muscle fatigue detection and analysis [24].

Developing an automated application which can analyze and detect the regions of interest (ROI) is crucial. Identifying the regions of interest from the thermal images will be explored in detail in the next sections.

Thermal image processing, similar to optical/visual image processing is the process of extracting useful information from a raw thermogram as opposed to an image format file. Although the process sounds very similar to normal image processing, thermal image processing should employ vastly different techniques as compared to visual image processing techniques. The reason for this is mainly attributed to the lower spatial resolutions of thermograms as compared to high resolutions of other image formats [21].

A research survey[26] shows that infrared thermograms need to undergo extra processing compared to optical images in order to extract meaningful data and to extract the regions of interest for further analysis. The following section explains in detail why some of the most widely used image segmentation algorithms fail to extract valuable information from thermal images.

4.2 Thermal Image processing Algorithms

Identifying the region of interest (ROI) is the preliminary step in thermal image analysis. Most commercial applications available identify the region of interest in a thermal image in regular geometric patterns such as rectangles, ellipse, square and so

30

forth. Applications which employ specialized algorithms are being developed which are capable of extracting non-geometric anatomical shapes from a thermal image

[18,23,29]. Very high-resolution thermal images are required to extract exact anatomical shapes from a thermogram. Considering the resolution of the infra-red camera used in our experiments and to achieve useful results, thermal images produced from the experiments will be used to extract the concrete geometric ROIs.

Each test subject’s body within the thermal images is considered as the region of interest in this study. The aim of the study is to extract the test subject’s boundary from within the image thereby separating noise and background data from the thermal images. After extracting the region of interest from the thermal images, the average temperature of the test subject’s body can be calculated by taking a mean of the intensities of each pixel value within the region of interest. This average value can be compared with the before and after pictures to notice any changes in temperature and identifying a possible muscle fatigue when there is a considerable increase in temperature.

A study performed [31] in 2015 elaborates on the hurdles faced while extracting the regions of interest from low resolution, low-SNR thermal images. Conventionally, a region of interest is identified in normal images using two methods namely image segmentation and image classification.

A suitable algorithm had to be chosen from various available algorithms for this research. The following sections explain different image segmentation and image

31

classification algorithms considered for the project and also the reason for choosing the final thermal image classification algorithm.

4.3 Image Segmentation

Image segmentation is one of the widely used image processing algorithms to identify sharp edges in an image. A comparative study done in 2015[19] employs various image segmentation techniques on thermal images and compares and contrasts each technique and explains the suitable parameters needed for each technique to extract data from thermal images.

Image segmentation can be classified into three different types namely,

1.Thresholding segmentation,

2.Edge detection segmentation, and

3.Region-based segmentation

Differences between the three different types of image segmentation algorithms are explained briefly in the following sections. This helps in understanding why image segmentation algorithms are not suitable to extract a region of interest from a given thermogram of low spatial resolution. Among these three segmentation types, thresholding segmentation methods are relatively better as compared to the other two.

4.3.1 Thresholding Segmentation Thresholding segmentation is the primary image segmentation technique.

Explained concisely, it helps in differentiating the foreground and the background in

32

an image. Image thresholding is more effective when the image being processed has high contrast levels. All images need to be in grayscale or converted to one (multiband thresholding for colored images) for thresholding segmentation. The grayscale images are then converted into binary images based on a certain threshold value for each pixel.

Each thresholding algorithm uses a specific threshold value (T) for each image. If the image intensity (Ii,j) of a given pixel falls below the threshold value (Ii,j < T),the pixel is converted to a black pixel and vie-versa to obtain a binary image. In the resulting image, white pixels correspond to the foreground and the black pixels correspond to the background. There are multiple image thresholding algorithms of which the multi-level thresholding and histogram thresholding are widely used.

Thresholding segmentation techniques are usually considered to be preliminary when it comes to processing of high-resolution optical images. This is attributed to the limitation of the technique where image segmentation can be performed only when there is a clear background and foreground in the image. In the case of thermal images where there is only one test subject in the field of view, image segmentation techniques can perform optimally as compared to the other image processing techniques [34]. In this study, we have used an image segmentation technique called Otsu thresholding segmentation along with image preprocessing. A detailed explanation on the Otsu thresholding segmentation and the preprocessing techniques used are explained in the following section.

33

4.3.2 Otsu Thresholding Segmentation One of the image processing techniques selected for this study is the Otsu thresholding image segmentation. The Otsu thresholding segmentation technique was conceptualized by Nobuyuki Otsu in 1979. Otsu’s algorithm performs a clustering- based image thresholding segmentation. Each image is considered to have bi-modal histogram made up of 2 classes of pixels [35]. These can be understood to be the foreground pixels and background pixels.

Otsu’s method functions by searching for a threshold value which minimizes the intra-class variance. Intra-class variance is defined as the weighted sum of the variances of the two classes:

2 2 2 휎푤(푡) = 푤0(푡)휎0 (푡) + 푤1(푡)휎1 (푡) ,

where 푤0(푡) and 푤1(푡) represent the probabilities of the two classes separated

2 2 by the threshold 푡. 휎0 and 휎1 represent the variances of the two classes. The class probability is calculated from the L bins of the histogram

푡−1 푤0(푡) = ∑ 푝(푖) 푖=0

and

퐿−1 푤1(푡) = ∑ 푝(푖) 푖=푡

34

The following section explains and illustrates the preprocessing steps and the application of the Otsu threshold segmentation for an easier understanding. Before, the thermal image can be processed using the Otsu thresholding, the thermal image needs to be converted to a tagged image file format (‘tiff’). The thermal images recorded using the FLIR C2 camera are stored as ‘radiometric thermal jpg’ format image. ExifTool is a platform independent tool used to extract metadata from images. ExifTool can be used to extract the 16-bit RAW (sensor) data from the ‘thermal jpg’ image format into a ‘tiff’ format. After the image has been converted to the ‘tiff’ file format the original image looks as shown in the Figure 4.1. The image is displayed using a ‘hot’ colormap.

Figure 4.1 The original thermal image before segmentation.

35

The histogram of the original image as shown in Figure 4.2 visualizes a clear difference in the pixel intensities of the image.

Figure 4.2 A histogram plot of the pixel intensities of the original thermal image

The original image needs to be smoothed to a certain degree of smoothness before the Otsu thresholding algorithm can be applied. This step helps in removing any extra noise in the image background while retaining the edges and meaningful data.

The degree of smoothness is calculated from the range of the pixel values of the image.

36

In our algorithm we have calculated the range by finding the difference between the highest pixel intensity and the lowest pixel intensity.

Figure 4.3 The Gaussian-smoothed thermal image

After the preprocessing step, the threshold value of the image is obtained using the Otsu algorithm. The threshold value is calculated in such a way that the inter-class variance is the highest for this particular threshold. Since the thermal images used in this study have a clear demarcation in pixel intensity between the test subject and the background, the Otsu algorithm segments the image into 2 levels using the threshold and the figure 4.4 illustrates the label matrix of the separated classes in the original image.

37

Figure 4.4 The thermal image classified using 2-level Otsu segmentation

Once the image has been segmented into two classes as the ‘background’ and the ‘foreground’, we calculated the average temperature of all the pixels belonging to this class.

The next step involves an improvement to the Otsu thresholding. Instead of just segmenting the image into just 2 classes, a multi-threshold can be used to segment the image into n-number of classes. Any segmentation with more than 4 thresholds results in an image with no meaningful data. In our case, it should be noted that all the test subjects are wearing a uniform black-colored t-shirt which doesn’t reflect the temperature same as the human skin. Equipped with this information, we performed

38

3-level Otsu segmentation using 2 threshold values. Figure 4.5 shows the segmented image with 2 thresholds. It can be noted that this technique is successful in extracting the average temperature of the exposed skin regions from the test subjects.

Figure 4.5 The thermal image classified using 3-level Otsu segmentation

The thermal images of all the test subjects were segmented using the Otsu thresholding algorithm and the average temperature of the regions of interest was calculated. The segmented images and the tabulated temperature values are shown in

Chapter 5.

4.3.3 Edge detection segmentation The goal of edge detection algorithms is to identify and locate sharp discontinuities in an image. This method takes advantage of the closed outlines in an image. The edge detection algorithms help in identifying the boundaries and edges

39

within an image by finding the sudden changes in pixel intensities. One of the advantages of the edge detection method is the desired result is achieved without making any permanent changes to the initial image [22].

Some of the widely used edge detection techniques are:

1. Sobel’s operator,

2. Robert’s cross operator,

3. Prewitt’s operator,

4. Laplacian of Gaussian, and

5. Canny edge detection algorithm.

All the above-mentioned methods apply a unique mask over the given image to extract the object boundaries and edges.

Canny edge detector

Among these different edge detection techniques, the Canny edge detector has been found to be optimal for identifying boundaries and edges in thermal images

[18,25]. A block-schematic diagram explaining the functioning of a Canny edge detector is shown in Figure 4.6

40

Figure 4.6 Block diagram of the Canny edge detection algorithm

Canny edge detector shows a lower error rate with high resolution (320 x 240 minimum) thermal images compared to the other edge detection algorithms [18,30].

Also, the edges are well localized and the response to non-edges are very low. A Canny edge detector employs the following methods to extract the edges and boundaries of objects within a given image.

1. Gaussian Filtering

The input images will contain a lot of white Gaussian noise and the first step of the Canny edge detector is to filter and smooth out the noise using Gaussian filtering.

A convolution mask with a 3 x 3 window is usually employed.

2. Calculating the Intensity gradient of the image

Once the image has been smoothed using a Gaussian filter, the next step is to identify the edge strength. The edge strength can be calculated from the gradient of the image. Edges in an image are regions where the corresponding pixels have higher intensity compared to the surrounding pixels. The Canny edge detector then uses one

41

of Robert’s, Sobel’s or Prewitt’s operators to get the two-dimensional image gradient.

The direction of the edge is calculated using the x and y axis gradients.

3. Non-maxima suppression

Non-maxima suppression is one of the most important phases in the Canny edge detector. Non-maximum suppression smooths out the detected edges by filtering out noisy pixels along the object edge.

4. Hysteresis Thresholding

Hysteresis is the technique employed to remove streaking in the detected edges.

Streaking is breaking up of an edge contour caused by the operator output fluctuating above and below the threshold. Whenever, only a single threshold (T) is applied to an image, and an edge has an average strength equal to T, then due to noise, there will be instances where the edge dips above or below the threshold. In order to avoid streaking, hysteresis uses 2 thresholds (say T1 and T2). For a given image, any pixel above the threshold T1 and below the threshold T2 are considered to be edge pixels. Non- maximum suppression and hysteresis thresholding result in low error and thin edge lines.

The drawback of using the Canny edge detector or any other edge detection segmentation algorithms is that in thermal images unlike optical images, the boundary of an object is not clearly defined. The pixel intensities of the edge pixels represent the temperature values rather than the actual edges. Figure 4.7 shows the result of using the

Canny edge detector with one of the thermal images from our experimental results.

42

Also, the edge detectors tend to identify false-positive edges which actually represent a change in the temperature value rather than a concrete edge within an object due to the variation in the pixel intensities. These drawbacks prevent the edge detection segmentation techniques to be poor classifiers of thermal images.

Figure 4.7 The Canny Edge detector applied to a thermal image

4.3.4 Region-based segmentation The region-based image segmentation techniques consider the images to be made up of a finite number of regions and perform local analysis on each region to segment the image. One of the widely used region-based segmentation techniques is called the ‘Watershed transform’ method.

In this technique, the image is considered to be a topographic region and the algorithm segments the image based on geography. Water ‘fills’ topographic regions

43

of more depth (less gradient intensity) and whenever a region of new height (higher gradient intensity), a boundary is drawn. As a result of this all homogenous regions of the images are segmented from edges and boundaries.

4.4 Image Classification

Image classification is a widely used image processing technique to identify separate objects within a given image. Features like real-time face detection are possible because of Image classification [28]. Image Classification can be defined as the process of classifying each different object in an image into a unique class. It is important to understand the difference between image segmentation and image classification.

Image segmentation algorithms divide an input image into groups of connected pixels. Each segment in the resultant partition has a unique digital ID number (e.g.,

45325). All the pixels belonging to that segment will have the same exact digital ID number (e.g., ID#45325), but no other pixels outside that group will have that ID number.

In the case of image classification algorithms their methods assign a class to each element (can be individual pixels or segments). When an image classifier performs a per-pixel classification, the resultant classified image will consist of groups of connected pixels sharing the same class, but also pixels belonging to different unique groups can have the same digital ID number in the classified raster (e.g., ID85432). In

44

other words, an image treated with an image classifier can still be segmented into different segments, but an image classification cannot be derived from the segmentation as the class information is unavailable.

4.4.1 Cascading Image Classifiers There are multiple image classification methods being widely used in image processing. The image classification algorithms for processing results should be decided based on multiple factors like the spatial resolution of the image, purpose of study, and quality vs cost trade-off. Most of the widely used classification algorithms use machine learning and neural networks to train an image classifier. A study [20] has focused on thermal image processing and classification. In the study it has been shown that an image classification technique known as the Haar-Cascade classifier yields better results in detecting human boundaries even with low spatial resolution images.

4.4.2 Haar-cascade Classifier The Haar-cascade classifier also known as the Viola-Jones classifier is a widely used image classifier introduced by Paul Viola and Michael Jones in 2001 [32]. The

Haar-cascade classifier is used for extracting object boundaries within an image with a high precision [27]. A study was performed to evaluate the cost and quality of the Haar- cascade classifier in extracting regions of interest from thermal images. The study finds that Hear-cascade classifier is effective in classifying thermal images [20]. Based on the findings of this study, the Haar-cascade classifier has been chosen as one of the image classifier algorithms for this research to extract ROIs from the thermal images.

45

This section explains the basis of Haar-cascade classifier as this the chosen technique for extracting the ROIs from the thermal images in this study.

The Haar-cascade classifier employs a supervised machine learning algorithm to train the classification model. The region of an image which needs to be classified is called a positive image and any image without a region of interest is called a negative image. The model is trained with a lot of positive images (e.g., faces) and negative images (e.g., images without faces).

The image classification process of the Haar-Cascade classifier can be divided into 3 stages namely:

1. Formation of Integral Image,

2. AdaBoost learning, and

3. Cascade Classification.

Formation of the Integral Image

The first step in the Haar-cascade classifier is computing the integral image from a given input image. The integral image as illustrated in figure 4.7 is also known as a summed area table, which is a commonly used calculation in image processing algorithms. The summed area table is used to calculate the sum of values in a rectangular subset of a pixel grid. The integral image of a given image at any location x,y contains the sum of all the pixel values to the top and left of x,y inclusive.

푖푖(푥, 푦) = ∑ 푖(푥1, 푦′) 푥′≤푥,푦1≤푦

46

where i(x,y) is the pixel value of the initial image at location x,y and 푖푖(푥, 푦) is the respective integral image value.

Figure 4.8 An illustration of integral image calculation

The integral image provides a supremely efficient way for calculating the sum of all the pixels values within a rectangle. For example in the Figure 4.8 , the sum of all the pixels within rectangle ABCD can be calculated using the formula

∑ 푖(푥, 푦) = 푖푖(퐴) + 푖푖(퐷) − 푖푖(퐶) − 푖푖(퐵) (푥,푦)∈퐴퐵퐶퐷

47

Figure 4.9 Calculating the sum of the pixels within a rectangle using the integral image.

For each Haar-like feature, the value is calculated by subtracting the pixel value of the black rectangle from the pixel value of the light rectangle.

AdaBoost Learning

After the integral image has been created, the classification function can be created with the help of many machine learning algorithms. AdaBoost is the machine learning algorithm used in the Viola-James method [32]. AdaBoost is used for selecting a set of features and to train the classifier with the help of positive and negative images.

An AdaBoost classifier is comprised of the weighted sum of several weak classifiers, in which each weak classifier is a threshold on a Haar- like feature.

48

Cascade Classification

The cascade classifier as shown in Figure 4.9 comprises of multiple stages where each stage has a strong classifier from AdaBoost. In each stage, it is determined if a given sub-region of the input image can be classified as an absolute negative

(definitely not the positive image) or a ‘maybe’ positive image.

Figure 4.10 Block diagram explaining the functioning of a Cascade classifier

The cascade classifier moves onto the next stage only when the current stage is not an absolute negative. The probability of a sub-region containing a positive image increases with every stage in a cascade classifier.

4.4.3 OpenCV and Python for Haar-cascade thermal image classification Having selected the Haar-cascade classifier algorithm as an image classifier for extracting the region of interest from the thermal images, OpenCV (Open Source

49

Computer Vision) was used for accessing the machine learning APIs. Python was chosen as the programming language of choice as it provides excellent support for accessing the OpenCV APIs.

OpenCV provides a vast collection of open source APIs which contains modules for image processing, machine learning, object detection, GPU-acceleration etc., The haar-cascade classifier used for this study was trained to extract the upper body boundary of a human from images. Figure 4.11 shows an illustration of the Haar- cascade classifier used in this study. The classifier is trained to identify the human boundary in a thermogram and draw a bounding rectangle around the identified positive image. The training was performed with the help of open source images available for the purpose of training classifiers. The classified images are illustrated in chapter 5, which discusses the result of the image processing algorithms chosen in chapter 4.

Initial thermogram Classified image using Haar-cascade classification Figure 4.11 An illustration of the Haar-cascade classification of a thermogram

50

Based on our literary review, we have chosen three image processing algorithms to segment and classify the thermal images obtained during the experimentation stage.

We have used Haar-cascade classification, 2-level Otsu thresholding and 3-level Otsu thresholding algorithms for the autonomous extraction of region of interest and for the temperature calculation of the ROI. The results of using the 3 different algorithms for classifying the thermograms have been tabulated and analyzed in Chapter 5.

51

5. RESULTS & CONCLUSION

5.1 Analysis of the ROI extraction algorithms

The three image classification algorithms discussed in the previous chapter were used to process all the thermograms collected during the experimentation phase. The primary purpose of the image classification process is to extract the regions of interest from each thermogram. For each test subject, the region of interest was extracted from both the rested and fatigued state thermograms. The average temperature of the ROI was then calculated with the help of the segmented region properties for each test subject’s thermograms. The difference in the average temperature of the ROI gave us the average increase in temperature of the test subject after the exercises were completed.

Each one of the three classification algorithms identified a certain region of interest for each test subject. We have tabulated and listed the region of interest extracted by each method along with the average temperature value of the ROI in the following pages. For each test subject, the left-side column shows the initial rested thermograms and the right-side column shows the final fatigued state thermograms.

52

Subject 1 Rested, Avg ROI temperature =85.95F Subject 1 Fatigued, Avg ROI temperature =86.47F

Subject 1 Rested, Avg ROI temperature =87.83F Subject 1 Fatigued, Avg ROI temperature =88.21F

Subject 1 Rested, Avg ROI temperature =90.19F Subject 1 Fatigued, Avg ROI temperature =91.14F

Figure 5.1 ROI extraction in Subject 1 and Average temperature calculation.

53

Subject 2 Rested, Avg ROI temperature=86.13F Subject 2 Fatigued, Avg ROI temperature=85.72F

Subject 2 Rested, Avg ROI temperature=89.34F Subject 2 Fatigued, Avg ROI temperature=89.58F

Subject 2 Rested, Avg ROI temperature=90.03F Subject 2 Fatigued, Avg ROI temperature=92.34F

Figure 5.2 ROI extraction in Subject 2 and Average temperature calculation.

54

Subject 3 Rested, Avg ROI temperature=84.83F Subject 3 Fatigued, Avg ROI temperature=85.17F

Subject 3 Rested, Avg ROI temperature=85.12 Subject 3 Fatigued, Avg ROI temperature=86.79F

Subject 3 Rested, Avg ROI temperature=89.55F Subject 3 Fatigued, Avg ROI temperature=91.34F Figure 5.3 ROI extraction in Subject 3 and Average temperature calculation.

55

Subject 4 Rested, Avg ROI temperature=85.46F Subject 4 Fatigued, Avg ROI temperature=85.53F

Subject 4 Rested, Avg ROI temperature=87.97F Subject 4 Fatigued, Avg ROI temperature=88.61F

Subject 4 Rested, Avg ROI temperature=90.36F Subject 4 Fatigued, Avg ROI temperature=91.43F Figure 5.4 ROI extraction in Subject 4 and Average temperature calculation.

56

Subject 5 Fatigued, Avg ROI temperature=83.06F Subject 5 Rested, Avg ROI temperature=82.93F

Subject 5 Rested, Avg ROI temperature=85.45F Subject 5 Fatigued, Avg ROI temperature=85.92F

Subject 5 Rested, Avg ROI temperature=88.62F Subject 5 Fatigued, Avg ROI temperature=89.76F Figure 5.5 ROI extraction in Subject 5 and Average temperature calculation.

57

Based on the results of the image classification techniques applied to the thermal images, the segmentation algorithms using 2-level Otsu and 3-level Otsu fare better in extracting the useful ROIs compared to the Cascade-classfier. This is apparent from the

ROI extracted from the images and also the average change in temperature deduced by the image classification techniques. The Otsu segmentation algorithms have performed better in extracting the exact human boundary from the thermal images. The 2-level

Otsu segmentation was 100% successful in extracting the exact human silhouette from the thermal images. The 3-level Otsu segmentation algorithm was successful to a certain degree in extracting the regions were the skin surface was exposed, thereby giving us a more accurate change in average temperature.

For each image segmentation technique used, we have visually represented the average temperature of the extracted region of interest in Figures 5.6-5.8

Figure 5.6 Average Temperature of ROI from 3-level Otsu segmentation

58

Figure 5.7 Average Temperature of ROI from 2-level Otsu segmentation

Figure 5.8 Average Temperature of ROI from Haar-cascade classification

59

Figure 5.9 plots the average change in the ROI temperature of all 5 test subjects that were calculated using the three different image classification techniques. The 2- level Otsu segmentation algorithm has a mean change in temperature of 0.68 F with a standard deviation of 0.51 F. The 3-level Otsu segmentation algorithm has a mean change in temperature of 1.23 F with a standard deviation of 0.32 F. The Cascade classification algorithm doesn’t indicate any correlation in the temperature change between the fatigued and rested states.

Figure 5.9 Change in Average temperature of the ROI

60

5.2 Conclusion

From figures 5.6-5.8, it can be noted that the change in average temperature of the ROI is more pronounced while using the 3-level Otsu segmentation and it indicates a direct correlation between an increase in the ROI temperature and muscle fatigue.

While using the 2-level Otsu segmentation, though the change in average temperature is not as pronounced as in the 3-level segmentation, it still shows a direct correlation between an increase in the ROI temperature and muscle fatigue. While using the cascade classifier, we can observe that there isn’t any meaningful correlation to be made as the patterns seem to be erratic. We attribute this to the classifier, identifying noisy pixels in the background along with the human boundary during classification.

The extra noisy pixels tend to even out and suppress any meaningful data that can be extracted from the thermograms of the fatigued and rested states.

The main limitations of using the Otsu segmentation technique for calculating the change in average temperature of the ROIs are that the test subject should be at the same distance from the observation point and there can’t be any other test subjects present in the frame. From this study, we are able to conclude that Infrared thermal imaging shows immense capacity to be used as an effective associative screening technique in detecting muscle fatigue. It can replace existing invasive and radiating technologies and provide a safe screening methodology. We have also demonstrated that the unsupervised processing of thermal images is possible and can be used to infer temperature changes provided that the image processing algorithms are chosen specifically for each problem at hand. For example, our image segmentation algorithms

61

are effective in analyzing the temperature change of ROIs in test subjects when the images are recorded in controlled environments by controlling the distance of the test subject from the camera aperture. A possible application of our technique can be the screening of players and/or soldiers before and after a game and/or training session to calculate the change in temperature and infer a possible muscle fatigue situation that needs further medical diagnosis and attention.

62

6. FUTURE WORK

In this study we have demonstrated a technique for the non-invasive screening of test subjects to identify a possible muscle fatigue condition. As the use of IRTG in muscle fatigue analysis is a nascent field, there is a lot of scope for more innovation and development. The unsupervised classification of thermal images is a relatively new frontier and better image processing algorithms can be developed.

One of the limitations of the image segmentation technique we have used in this study is that only one subject can be present in the field of view. An algorithm can be developed which can analyze and extract the region of interest for multiple test subjects in the same frame. Thermal images with a higher resolution can aid in the development of better image processing techniques.

Another area of interest can be the real time analysis of thermal images to extract meaningful data and find matching patterns to infer an event or a condition. Real-time muscle fatigue detection and analysis techniques can be extremely useful in sports and military applications where injury to personnel can be monitored and prevented in real- time. As thermal images represent only 2D data, they can be super-imposed with frames from multiple visual cameras from different angles to possibly develop a 3D model, which can aid in further medical image segmentation and crucial analysis.

63

7. BIBLIOGRAPHY

1. L., I., & Mccauley, P. (2012). Work-Related Musculoskeletal Disorders

Assessment and Prevention. Ergonomics - A Systems Approach.

2. Barroso, G. C., & Thiele, E. S. (2011). Muscle Injuries In Athletes. Revista

Brasileira De Ortopedia (English Edition),46(4), 354-358.

3. Hongwei Hsiao and Nancy Stout, “Occupational Injury Prevention Research in

NIOSH”, 2010.

4. Marconnet, P., Komi, P. V., Saltin, B., Sejersted, O. M., & Kirkendall, D. T.

(1992). Muscle Fatigue Mechanisms in Exercise and Training. Medicine &

Science in Sports & Exercise,24(1), 1309.

5. Hildebrandt, C., Raschner, C., & Ammer, K. (2010). An Overview of Recent

Application of Medical Infrared Thermography in Sports Medicine in Austria.

Sensors, 10(5), 4700-4715.

6. Cooke, W. H., Moralez, G., & Cox, P. (2011). Thermographic Imaging During

Simulated Uncontrolled Hemorrhage in Humans. Medicine & Science in Sports

& Exercise,43(Suppl 1), 650.

7. Al-Nakhli, H. H., Petrofsky, J. S., Laymon, M. S., & Berk, L. S. (2012). The

Use of Thermal Infra-Red Imaging to Detect Delayed Onset Muscle

Soreness. Journal of Visualized Experiments,(59).

8. Arfaoui, A., Polidori, G., Taiar, R., & Pop, C. (2012). Infrared Thermography

in Sports Activity. Infrared Thermography.

64

9. Hong, K., Liu, G., & Chen, W. (2017). Human Stress Research Using Facial

Thermal Imaging. DEStech Transactions on Computer Science and

Engineering,(Cimns).

10. Hadžić, V., Širok, B., Malneršič, A., & Čoh, M. (2015). Can infrared

thermography be used to monitor fatigue during exercise? A case study. Journal

of Sport and Health Science.

11. Retrieved from https://serc.carleton.edu/details/images/3786.html

12. Engert, V., Merla, A., Grant, J. A., Cardone, D., Tusche, A., & Singer, T. (2014).

Exploring the Use of Thermal Infrared Imaging in Human Stress

Research. PLoS ONE,9(3).

13. Ludwig, N., Formenti, D., Gargano, M., & Alberti, G. (2014). Skin temperature

evaluation by infrared thermography: Comparison of image analysis

methods. Infrared Physics & Technology,62, 1-6.

14. Marco, G., Alberto, B., & Taian, V. (2017). Surface EMG and muscle fatigue:

Multi-channel approaches to the study of myoelectric manifestations of muscle

fatigue. Physiological Measurement,38(5).

15. Irani, R., Nasrollahi, K., & Moeslund, T. B. (2014). Contactless measurement

of muscles fatigue by tracking facial feature points and RSSI. 2014 IEEE

International Conference on Image Processing (ICIP).

16. Hooshmand, H. (n.d.). The Role of Sympathetic Nervous System in

Temperature Regulation. Chronic Pain,13-26.

17. Formenti, D., Ludwig, N., Trecroci, A., Gargano, M., Michielon, G., Caumo,

A., & Alberti, G. (2016). Dynamics of thermographic skin temperature response 65

during squat exercise at two different speeds. Journal of Thermal Biology,59,

58-63.

18. Tang, Q., Bu, C., Zang, J., & Li, D. (2016). Infrared Image Edge Detection based

on Morphology-Canny Fusion Algorithm.

19. Duarte, A., Carrão, L., Espanha, M., Viana, T., Freitas, D., Bártolo, P., . . .

Almeida, H. (2014). Segmentation Algorithms for Thermal Images. Procedia

Technology, 16, 1560-1569.

20. Setjo, C. H., Achmad, B., & F. (2017). Thermal image human detection using

Haar-cascade classifier. 2017 7th International Annual Engineering Seminar

(InAES).

21. Chakraborty, S., Roy, M., & Hore, S. (n.d.). A Study on Different Edge

Detection Techniques in Digital Image Processing. Advances in Multimedia and

Interactive Technologies Feature Detectors and Motion Detection in Video

Processing,100-122.

22. Ojha, A. V. (2015). Algorithms for Edge Detection and Enhancement for Real

Time Images: A Comparative Study. International Journal of Innovative

Research in Computer and Communication Engineering,03(02), 651-658.

23. Davis, J., & Keck, M. (2005). A Two-Stage Template Approach to Person

Detection in Thermal Imagery. 2005 Seventh IEEE Workshops on Applications

of (WACV/MOTION05) - Volume 1.

24. Wang, W., Zhang, J., & Shen, C. (2010). Improved human detection and

classification in thermal images. 2010 IEEE International Conference on Image

Processing. 66

25. Canny, J. (1987). A Computational Approach to Edge Detection. Readings in

Computer Vision,184-203.

26. Lu, D., & Weng, Q. (2007). A survey of image classification methods and

techniques for improving classification performance. International Journal of

Remote Sensing,28(5), 823-870.

27. Lienhart, R., & Maydt, J. (n.d.). An extended set of Haar-like features for rapid

object detection. Proceedings. International Conference on Image Processing.

28. Efendi, J., Zul, M. I., & Yunanto, W. (2017). Real Time Face Recognition using

Eigenface and Viola-Jones Face Detector. JOIV : International Journal on

Informatics Visualization,1(1), 16.

29. Vardasca, R., Gabriel, J., Jones, C., Plassmann, P., & Ring, E. (2014). A

Template Based Method for Normalizing Thermal Images of the Human

Body. Proceedings of the 2014 International Conference on Quantitative

InfraRed Thermography.

30. Vu Hoang Minh, Pervaiz,U.(2017). Smoothness-based Edge Detection using

Low-SNR Camera for Robot Navigation.

31. Vardasca,R.(2015). Template Based Alignment and Interpolation Methods

Comparison of Region of Interest in Thermal Images.

32. Viola, P., and M. Jones. “Rapid Object Detection Using a Boosted Cascade of

Simple Features.” Proceedings of the 2001 IEEE Computer Society Conference

on Computer Vision and Pattern Recognition. CVPR 2001.

67

33. Danilo Gomes Moreira, Joseph T. Costello, Ciro J. Brito, Manuel Sillero-

Quintana. “A checklist for measuring skin temperature with infrared

thermography in sports and exercise medicine”

34. Rudi Heriansyah., and Syed Ab Rahman Abu Bakar(2007). “Defect Detection

in Thermal Image using Thresholding Technique”

35. Zhou, Shunyong, et al. “Infrared Image Segmentation Based on Otsu and

Genetic Algorithm.” 2011 International Conference on Multimedia

Technology, 2011.

36. Leonardo S Motta, et al. “Automatic Segmentation on Thermograms in Order

to Aid Diagnosis and 2D Modeling”,2011

68