Development of Image Processing and
Vision Systems with Industrial
Applications
Zhang Yi
A THESIS SUBMITTED
FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
DEPARTMENT OF ELECTRICAL AND COMPUTER ENGINEERING
NATIONAL UNIVERSITY OF SINGAPORE
2009 Acknowledgments
I would like to express my sincerest appreciation to all who had helped me during my study in National University of Singapore. First of all, I would like to thank my supervisors Associate Professor Tan Kok Kiong for his inspirational discussions, support and encouragement. His vision and passion for research enlighten my research work and spurred my creativity. I would like to give my gratitude to all my friends in Mechatronics and Automation Lab. I would especially like to thank Dr. Huang Sunan, Dr. Tang Kok Zuea, Mr. Tan Chee
Siong, Dr. Zhao Shao, Dr. Teo Chek Sing, Dr. Andi Sudjana Putra, Mr. Chen Silu and Mr. Yuan Jian for their helpful discussions and advice. I would also wish to thank Ms Lay Geok from Medical Department, NUS for her assistance for my experiment.
Finally, I would like to thank my family for their endless love and support.
II CONTENTS
Acknowledgments...... II
List of Figures ...... IV
List of Tables ...... VI
List of Abbreviations ...... VII
Summary ...... VIII
CHAPTER 1 Introduction...... 1
1.1 Impact of Computer Imaging Technologies ...... 1
1.2 Contributions...... 4
1.2.1 Text Extraction and Translation...... 4
1.2.2 Vision-based Automatic Cell Manipulation System...... 5
1.2.3 Vision-assisted thermal tracking system for CNC machine ...... 6
1.3 Organization of Thesis ...... 7
CHAPTER 2 Text Extraction and Translation from Images Captured via Mobile and Digital Devices...... 9
2.1 Introduction...... 9
2.2 Text Extraction...... 14
2.2.1 Color to Gray Scale Transformation...... 14
2.2.2 Region Segmentation ...... 15
III 2.3 Character Recognition...... 20
2.4 Experimental Results ...... 23
2.5 Conclusions...... 26
CHAPTER 3 Vision-Servo System for Automated Cell Injection...... 27
3.1 Introduction...... 27
3.2 System Setup...... 32
3.3 Cell Detection...... 33
3.4 Pipette Detection ...... 39
3.5 Tip Focalization ...... 41
3.6 Penetration...... 43
3.7 Validation...... 45
3.8 Conclusions...... 47
CHAPTER 4 Vision-based Tracking and Monitoring System for CNC Machine
Surveillance...... 48
4.1 Introduction...... 48
4.2 Background and problem statement...... 50
4.3 Distributed Wireless Sensor Network for CNC Machine Surveillance...... 51
4.4 Decoupled Tracking and Thermal Monitoring of Non-Stationary Targets58
4.4.1 Overall System Configuration ...... 59
4.4.2 Vision and Image Processing System ...... 63
4.4.3 Non-Contact Temperature Measurement System ...... 68
4.4.4 Tracking Control of Linear Motor ...... 69
IV 4.4.5 Practical Issues...... 74
4.4.6 Experimental Results ...... 77
4.4.7 Conclusions...... 85
CHAPTER 5 Conclusions...... 87
5.1 Summary of Contributions...... 87
5.2 Suggestions for future work...... 89
Author’s Publications...... 92
Bibliography...... 94
V List of Figures
Fig. 2.1 Sample images taken by mobile phones...... 11
Fig. 2.2 Flowchart of text extraction algorithm ...... 13
Fig. 2.3 Image after Gray Scale Transformation...... 14
Fig. 2.4 Edge Detection Kernels ...... 16
Fig. 2.5 Background separation ...... 17
Fig. 2.6 Unwanted parts elimination...... 18
Fig. 2.7 Abnormal Object Removal ...... 20
Fig. 2.8 Pictorial Definition ...... 22
Fig. 3.1 Bio-manipulation System ...... 28
Fig. 3.2 Vision-assisted Servo System...... 29
Fig. 3.3 Flowchart of Process...... 30
Fig. 3.4 Two steps in system setup ...... 33
Fig. 3.5 Hough circle detection...... 35
Fig. 3.6 Faster cell detection ...... 37
Fig. 3.7 Pipette Detection...... 40
Fig. 3.8 Y-axis Coordination...... 41
Fig. 3.9 Tip Focalization...... 42
Fig. 3.10 Value of Entropy...... 43
Fig. 3.11 Penetration ...... 44
Fig. 4.1 A CNC Machine and workshop...... 51
Fig. 4.2 Sensor board and antenna board...... 52
Fig. 4.3 DFDS control structure...... 52
Fig. 4.4 Algorithm flow chart ...... 53
IV Fig. 4.5 Fault detection with SS=1200 rpm, fr =300 mm/min, depth of cut=1 mm
...... 58
Fig. 4.6 Overall System Configuration ...... 60
Fig. 4.7 Vision-assisted Servo System...... 61
Fig. 4.8 Mounting of the Infrared Thermometer...... 62
Fig. 4.9 Process Flowchart...... 64
Fig. 4.10 Moving Object Extraction ...... 67
Fig. 4.11 Thermal devices...... 69
Fig. 4.12 Control System Structure...... 70
Fig. 4.13 Maximum speed permissible ...... 75
Fig. 4.14 Calculation of minimum and maximum speed...... 76
Fig. 4.15 Step response with PID control ...... 78
Fig. 4.16 Controller response and tracking error ...... 79
Fig. 4.17 Simulation Scene ...... 80
Fig. 4.18 Temperature measurement during simulation ...... 81
Fig. 4.19 Temperature measurement in real experiment...... 82
Fig. 4.20 Explanation of sudden temperature raise...... 83
Fig. 4.21 Accuracy testing using thermal camera...... 84
V List of Tables
Table 2.1 Recognition Results ...... 24
Table 3.1 Comparison of Experimental Result...... 46
VI List of Abbreviations
CCD Charge-Coupled Device
CNC Computer Numerical Control
CT Computerized Tomography
ECG Electrocardiogram
DFDS Distributed Fault Detection System
HCDA Hough Cell Detection Algorithm
FCDA Fast Cell Detection Algorithm
FD Frame Difference
LQR Linear Quadratic Regulator
MRI Magnetic Resonance Imaging
OCR Optical Character Recognition
RGB Red, Green, Blue
VII Summary
The rapid advancement of the microprocessor, the perpetually declining cost of electronic devices as well as the increasing availability of handheld equipment for digitizing and displaying images have strongly spurred the continued growth for computer imaging technologies. Other impetus for such development stems from a steady flow of new applications, such as commercial, industrial and medical applications. This trend generates ample opportunities for the development of new image and vision based applications. This thesis addresses different sets of challenges present in different applications of image and vision- based systems. It presents the design of three image and vision-based systems which can be used in different and diverse arenas: mobile and digital devices, bio- manipulation systems and CNC machine surveillance. Through investigation in these diverse areas, the different challenges facing image processing & vision systems are better appreciated.
Mobile applications are rampantly available nowadays for a variety of purposes. The small and inexpensive wearable devices facilitate new ways through which users can interact with the physical world. Multimedia functions are fast expending and reshaping the growth of the market for phone developers.
In the first part of the thesis, a human-machine interactive software has been developed which could be embedded in a mobile or digital device to extract the text from scene images and translate into other languages. Text extraction is mainly based on the color and edge information of characters. A fast yet efficient
OCR engine is also designed to translate the extracted text using template
VIII matching techniques. This software will be extremely useful for tourists travelling in foreign countries who do not know foreign language.
Biological injection has been widely applied in transgenic tasks. In spite of the increasing interest in biomanipulation, it is still time-consuming and laborious work replying on the visual information through the microscope. Under such circumstances, a vision-guided control system has been proposed to be incorporated in cell manipulation systems to replace conventional manual operations in the second part of the thesis. The key component of the system is a self-tracking controller guided by an object recognition and tracking algorithm of a vision system. Comparisons are made between the recent works and our proposed methods for such servo applications. The efficiency of our system has been proven through experiments. This system has far-reaching significance in replacing the manual work with an automated strategy.
Fault diagnosis and predictive maintenance addresses economic issues which thereby impels new techniques for machine surveillance. In the third part of the thesis, two CNC machine surveillance schemes are presented and compared. The first is a wireless sensor network (WSN)-based fault detection system, where a
WSN will be implemented on a CNC machine to collect real-time health parameters. An alarm signal will be generated once the collected data is higher than a threshold. The second scheme is a vision-based real-time temperature monitoring system, where an object recognition and tracking algorithm will be applied to guide a thermometer to monitor the temperature of the working tool while it is in motion. An alarm signal will be generated to stop the machining process if the temperature is higher than a threshold. A comparison of the two methods will be presented and discussed.
IX Throughout this thesis, extensive experimental results will be furnished to illustrate the effectiveness of the proposed approaches.
X CHAPTER 1
Introduction
1.1 Impact of Computer Imaging Technologies
Computer imaging is a fascinating and exciting research area nowadays. The advent of the information technology, with its applications via the World Wide
Web, combined with the advances in computer power has brought the world into our daily lives. Visual Information, transmitted in the form of digital images [65], is becoming a major mean of communication in the modern age. Computer imaging can be defined as the acquisition and processing of visual information by computer which can be divided into two primary categories:
• Computer vision
• Image processing
These two categories are not totally separate and distinct [22]. There are no clear- cut boundaries in the continuum from image processing at the one end to computer vision at the other.
Image Processing:
Image processing is a form of computer imaging where the application involves a human being in the visual loop [68]. In other words, the images are to be examined and acted upon by people. Major application fields of image processing include medical imaging [99] and astronomical observation. Medical
1 imaging has grown over the last decade to become an essential component of diagnosis and medical education, which includes Magnetic Resonance Imaging
(MRI), Computerized Tomography (CT), Radiography, Electrocardiogram (ECG) and Electroencephalography (EEG) etc. With the rapid development of computer and image technology and the increasing mature of picture and image technology, this technology has gradually entered medical field and improved the quality of medical images and vision method [95], so that then the level of diagnosis has greatly improved by using the image operation and analysis. Other ongoing research areas include text extraction and recognition from images. Application fields include text extraction from WWW images [42], natural scene images and videos. A powerful image searching engine can be built using text extraction from
WWW images. Vehicle navigation system [82] can be created based on natural scene image recognition. Automatic video caption translation software can be designed using caption extraction and recognition scheme for every frame of a video [17].
Computer Vision:
Computer vision is the other form of computer imaging where the application does not involve a human being in the visual loop. In other words, the images are examined and acted upon by a computer. Although people are involved in the development of the system, the final application requires a computer to use the visual information directly. One of the major topics within the field of computer vision is image analysis.
The field of computer vision may be best understood by considering different types of applications. Many of these applications involve tasks that either are tedious for people to perform, require work in a hostile environment, require a
2 high rate of processing, or require access and use of a large database of information. Computer vision systems are used in many and various types of environments-from manufacturing plants to hospital surgical suites to the surface of Mars. The most important task of computer vision system is automated visual inspection (AVI) [11], which can be used for the purpose of measurements, gauging, integrity checking and qualify control. In the field of measurements, the gauging of small gaps [62], measurement of object dimension, alignment of the components, and analysis of crack formation are common applications. For example, the computer vision system will scan manufactured items for defects and provide control signals to a robotic manipulator to remove defective parts automatically [3]. During the automotive assembly, a vision guided robot identifies and sorts of the different parts. Computer vision systems are also used in many different areas within the medical and pharmacological community, with the only certainty being that the types of applications will continue to grow.
Current examples of medical systems being developed include: systems to diagnose skin tumors automatically [23], systems to aid neurosurgeons during brain surgery, systems to perform clinical tests and systems for automatic cell injection. Computer vision systems that are being used in the surgical suites have already been used to improve the surgeon’s ability to “see” what is happening in the body during the surgery and consequently improve the quality of medical care available [79]. Systems are also currently being used for tissue and cell analysis.
For example, they are being used to automate the applications that require the recognition and counting of certain types of cells. The field of law enforcement and personal identification is another active area for computer vision system development, with applications ranging from automatic identification of
3 fingerprints and vein to facial and retinal recognition. Currently, vision systems are placed on the streets to take pictures of speeders and in the future, computer vision systems may be used to manipulate the whole transportation systems in an automatic and intelligent way.
Another term which has similar meaning as computer vision is machine vision [10]. Machine vision is concerned with the engineering of integrated mechanical-optical-electronic-software systems for examining natural objects and materials. Although it uses similar computational techniques, it does not necessarily involve a device that is regarded as a computer.
1.2 Contributions
This thesis aims at developing image and vision systems for different application areas with different sets of challenges. Text extraction and translation software for mobile and digital devices, vision based control strategies for biomanipulation and industrial surveillance system.
1.2.1 Text Extraction and Translation
Images play a very important role in information storage and delivery. An efficient text extraction and recognition software, which is a heated research area, would provide a powerful human-environment interface. This software, presented in this thesis, is especially useful for travelers who do not recognize foreign languages or visually impaired patients who can use the software to extract the useful information and play back an audio equivalent using handheld devices. The software can be divided into two parts: text extraction and translation. The main
4 challenges for text extraction are the uncertain features of the characters as well as the background, such as different font size, uneven lighting, odd capturing angle and complex background. Text extraction algorithm is mainly based on a color- edge information fusion. Background will be identified and extracted after grey scale transformation. Characters will be isolated based on the background information. Abnormal objects and noise will be eliminated based on a pre- defined criterion. The binary image will be sent to an OCR engine for recognition.
Final translation result will be generated with the help of a database. The effectiveness of the proposed algorithm in meeting the challenges behind the processing of such images will be highlighted with real images.
1.2.2 Vision-based Automatic Cell Manipulation System
Recent advances in biological sciences, such as transgenic techniques, indicate an increasing need for more advanced and complex micromanipulation strategies for cell injection tasks [16], [51]. Conventionally, cell injection was conducted by skilled operators who need long term training but yet the success rate has not been high due to errors and lack of repeatability of human operators as well as contamination [79]. Besides, cell’s tissue or membrane is very fragile and slippery, a tiny improper operation can cause irreversible damage to the tissue of the cell [99]. Under such situations, an automatic and efficient strategy is required to eliminate these drawbacks and achieve a higher success rate. In this thesis, a vision-servo control system has been developed where the injection process is monitored and controlled automatically via integration of a vision system to an injector manipulation system. The cell is located and the pipette is
5 positioned and driven by an algorithm to realize an effective penetration. The algorithm is based on feature detection, tracking and autofocalization. The purpose of this system is to replace the conventional laborious the repeatable manual work with an automated approach to yield a higher success rate. The verification and accuracy of the scheme will be provided along with experimental demonstration under practical situations.
1.2.3 Vision-assisted thermal tracking system for CNC machine
System monitoring and fault diagnosis attract growing attentions in manufacturing lines due to safety and economical reasons [8]. An efficient diagnostic system can maintain tools in good condition and prevent severe failures by detecting and localizing faulty components at an early stage [18].
Conventionally, signal processing as well as the use of adequate process model form the core of fault detection with normal measurable variables [78]. A common feature of these schemes is the assumption that some states are available which inevitably poses restrictions on their applicability in common and practical scenarios. However, in practice, many of the required variables for monitoring and fault detection are not naturally present [54].
In this thesis, two schemes for CNC machine surveillance are designed. First, a wireless sensor network is implemented on a CNC milling machine and a distributed fault detection model is designed to monitor its health condition
(cutting force, vibration and sound) during the machining process. If the collected data exceeds the pre-set threshold the control center will stop the process.
6 In the second scheme, a vision-assisted thermal monitoring surveillance system is presented. First, a calibrated camera will detect and track the milling and transmit the position data to a host computer. Secondly, a laser built-in thermometer will continuously read the temperature of the milling by following the milling based on the position data. Finally, the host computer will generate an alarm signal when the temperature exceeds a pre-set threshold.
Moving object extraction is an active field of computer vision and has wide practical application in industrial monitoring system. Effectively detecting and tracking target object from video sequences are the main task in our surveillance systems. Currently three classical algorithms were used in video surveillance system. Real experimental results are furnished to highlight the key contribution from the thesis.
1.3 Organization of Thesis
The thesis is organized as follows:
Chapter 2 presents the development of a human-machine interactive software application. A review on recent development on mobile application as well as previous work on text extraction is conducted. Detailed description of the algorithm for text extraction is given which include background identification, text extraction and abnormal objects elimination. A fast yet efficient character recognition method is developed to translate the extracted text into English. The effectiveness is exhibited via experiments on real images.
Chapter 3 describes a vision based control system for automatic cell injection.
Motivation of the study has been stated. A review on previous works has been
7 made followed by a complete description of the proposed vision guided control system. Emphasis is placed on the vision based software. The key parts of the software include object detection, tracking as well as auto-tuning algorithms.
Finally, a verification of the accuracy is provided and the efficiency of the vision- servo system in facilitating a fully automated cell injection task are also demonstrated and duly discussed.
Chapter 4 presents the vision based surveillance system in industrial applications. In the first part of the thesis, a review on conventional techniques used in monitoring system is made along with a discussion of their limitations and drawbacks. Special attention is placed on the image processing and predictive control system design. Practical issues have been discussed in terms of maximum and minimum speed permissible and accuracy. Simulation and real experiment on
CNC machine have been conducted with corresponding results.
Finally, conclusions and suggestions for future work are discussed in Chapter
5.
8 CHAPTER 2
Text Extraction and Translation from
Images Captured via Mobile and
Digital Devices
In this chapter, a human-machine interactive software application is developed, which is specifically useful for text extraction and translation from images captured via mobile and digital devices with cameras. The full application comprises of two stages: an extraction stage and a recognition stage. In the extraction stage, a fast yet efficient algorithm will yield the essential information from the raw image. In the recognition stage, the extracted text will be interpreted and translated through an Optical Character Recognition (OCR) engine. The effectiveness of the proposed algorithm in meeting the challenges behind the processing of such images will be highlighted with real images.
2.1 Introduction
Mobile applications are rampantly available nowadays, for a whole variety of purposes. The small and inexpensive wearable devices facilitate new ways through which users can interact with the physical world. Besides the basic communication function for mobile phones, multimedia entertainment functions are fast expanding and reshaping the growth of this promising market for phone
9 developers. Such existing functions including radio, recording, MPEG3 player, camera, map guide, dictionary, language translation and video conferencing.
With functions such as dictionary and language translation fast becoming a standard part of a mobile phone, coupled with the fact that this mobile device is now essentially an item which follows its owner throughout the day, the stage is set for the development of mobile interpretation applications. An example of such an application scenario; a Japanese tourist in Singapore needs to navigate his way to a unit in a hospital through the text on signages available but which he can hardly understand. He will snap an image of the sign using his mobile. The mobile application will preprocess the image and condition it into a form which contains the key text information he needs in a usable form. The processed form of the image can then be used by the Optical Character Recognition (OCR) and language translation engines to yield the exact meaning of the sign in Japanese. The potential of such an application is immensely extensive.
This chapter will focus on the development of such a mobile translation application. Apart from use for interpretation by transnational travelers as highlighted earlier, with such a function embedded in mobile devices, a message, written on a piece of paper, can be efficiently processed, translated and sent to a target recipient via SMS. The application is also amenable to the development of a seamless interface to the external world by propagating to specific website from a
URL captured from an advertisement or a poster on the mobile phone. These set the motivation to develop a complete text extraction algorithm to equip the phone with real-time or near real-time translation function across different languages.
10
(a) (b)
(a) (b)
Fig. 2.1 Sample images taken by mobile phones Fig 2.1 shows some signs taken by mobile phones. There are many challenges with respect to text extraction and recognition from modest images captured via mobile devices.
First, there can be a large variation in both the font size and font type of the text expected in the diverse forms of images captured (see Fig 2.1 (a)). Therefore, the threshold box for segmentation cannot be fixed at a specific size. Secondly, the resolution of such images will be typically modest. Coupled with an uncontrolled environment, uneven illumination and reflection (see Fig 2.1 (c) where there is an obvious reflection in the image captured), and a possibly odd image capturing angle (see images in Fig 2.1 (b)), the target text captured can be blurred, all of these posing difficulties to text extraction. Thirdly, the text extraction and recognition function will inevitably be limited by the nature of the small-screen mobile devices which will restrict the span of the image which can be captured. It may be difficult to capture a sign with just a homogenous background, and the inevitably unwanted part captured, if it differs from the
11 background, may lead to problems during the processing stage (see images in Fig
2.1 (d) where the unwanted parts outside of the sign boundary were also captured).
Finally, images taken under a poor lighting condition may result in low entropy
(see the image in Fig 2.1 (b)). Low entropy may also cause problems in processing. In addition, it should be noted that this chapter will only focus on text extraction from images containing text in a relatively simple background. Far more intensive computation will be necessary when the text is embedded in a complex background [24], [30], [38], [42], [90].
The proposed text extraction algorithm will be based on four assumptions.
First, the font size of the text in the captured image should be sufficiently large, otherwise it may be ignored in the algorithm. Secondly, the background should be uniform or at least near uniform, and it should constitute a major part of the whole image. Thirdly, the color of the top line of the image should be different from the color of the characters. Finally, character regions must be well contained and cannot be allowed to extend to the edges of the image.
Text extraction algorithm comprises several key steps. First, the original color image will be transformed to an adequate gray image to reduce computation cost.
Secondly, the whole image will be segmented into disjoint regions where each region will be grouped into one of N different gray scale values, (N can be manually defined). Thirdly, background and objects will be discriminated based on the area of each region. Finally, characters will be extracted by eliminating the unwanted parts.
This completes the text extraction stage and at this time, only the desired and labeled characters will be left in the image, and they are ready to be sent to an
OCR engine for interpretation. The flow of text extraction algorithm is shown in
12 Fig 2.2. The details behind each of the step will be duly highlighted in the ensuing sections.
Fig. 2.2 Flowchart of text extraction algorithm
(The alphabets f, T etc. represent the original/transformed images at various
stages of the processing, they will be referred to in the ensuing sections)
The rest of the chapter is organized as follows. Section 2 will describe the text extraction algorithm which contains four steps. Section 3 will present character recognition method. Section 4 will show the experiment results when the software is applied to 28 real images captured with a mobile phone. The accuracy of the software will be presented as well. Finally, in Section 6, the chapter will be concluded with suggestions for future works.
13 2.2 Text Extraction
The main objective behind this part of the algorithm is to extract the text from the raw images by discriminating the objects from the background and eliminating the unwanted parts. It involves five key steps as follow:
2.2.1 Color to Gray Scale Transformation
The standard function to transform a color image to the gray-level image [65] is given in (2.1). Let f(x, y) be the color value of a pixel original image at the (x, y) position. (We will refer to f as the original color image). Then f(x, y).R, f(x, y).G and f(x, y).B, denotes the corresponding value of its red (R), green (G) and blue components (B) respectively. T(x, y) represents the gray scale value of that pixel of the transformed image.
Txy( , )=× 0.114 f ( xy , ). R +× 0.587 f ( xyG , ). +× 0.299 f ( xy , ). B (2.1)
(a) (b)
(c) (d)
Fig. 2.3 Image after Gray Scale Transformation
14 2.2.2 Region Segmentation
The main objective of region segmentation is to segregate the gray image into several regions, so as to separate the background from the objects [65]. Edge detection is the most popular and commonly used tool in image segmentation.
Several edge detection algorithms have been developed [19], [37], [39], [50].
Among them, Canny’s algorithm [37] is arguably among the most popular and widely used. However, we will not adopt edge detection in the segmentation stage for two main reasons. First, mobile device based text extraction is a time-critical application. Computation cost for edge detection is higher than the method we will propose later. Secondly, incomplete edge detection may directly lead to recognition failure especially when there is reflection in the image. An object will be ignored if edge detection result is not a closed shape. [89] proposed a coarse- to-fine algorithm to detect text in video, where text occupied small area. [35] and
[17] proposed color component analysis methods which classified one object in either chromatic or achromatic region. However, this method may not hold true in our application. Through experiments, we found that a single character may contain both chromatic and achromatic regions.
Quantization
After the gray scale transformation, we quantize every pixel of the image into
N levels, where number N can be manually defined for different scenarios.
Suppose maxGray and minGray are the maximum and minimum gray scale values of an image. The quantization process can be described as follows:
BEGIN
Calculate maxGray and minGray
15 ()maxGray− minGray step = N if T (, x y )>+× minGray k step and T (, x y ) <++× minGray ( k 1) step S ( x , y )=+×∈− minGray k step k {0,1,... N 1} Area [ S ( x , y )]++ ;
END
N is chosen as 4 in our application. Results are shown in Fig 2.4.
(a) (b)
(c) (d)
Fig. 2.4 Edge Detection Kernels
Background Extraction
Empirically, it is found that the area in Sxy(, )with the largest number of same gray scale value is always the background or part of the background. This area is made white to highlight the background. Results are shown in Fig 2.5.
16
(a) (b)
(c) (d)
Fig. 2.5 Background separation
Unwanted Parts Elimination
It is assumed that the gray scale value which occupies major area of the top line is different from that of the characters (Assumption three). Therefore, areas which have the same gray scale value as the top line will be regarded as non- character regions. The process can be described as:
BEGIN
GAreaSxystAreaSxyw=>×arg max [ ( , )], . .max [ ( , )] 0.4 xw∈(0, ) ifSxy (, )==∈∈ G , Sxy (, ) 255, { x (0,), w y (0,)} h
(w and h are the the width and height of the image respectively )
END
After the above steps, objects and some non-character regions may still remain in the image. Based on the fourth assumption made, the connected regions which reach the boundary of the image must be non-character regions. A connected region labeling method could be applied to identify these regions.
17 Although computation cost of labeling is usually high, the area in the image which needed to be labeled after all the above steps is small (normally 25% of the whole image). Results are shown in Fig 2.6 after eliminating the regions reaching the edge.
(a) (b)
(c) (d)
Fig. 2.6 Unwanted parts elimination
Abnormal Objects Removal
Abnormal objects include boundary, signs, rifts etc. Usually, these objects have such features in common as a wide span and a large length to width ratio or a large width to length ratio [24]. Therefore, we can locate and remove them based on these features. The following six criterions are used to identify and eliminate these abnormal objects.
• Vertical span > 0.8 * width of the image.
• Horizontal span > 0.4 * length of the image.
• Vertical span / Horizontal span > 16.
18 • Horizontal span / Vertical span > 4.
• Area > 5 * average area.
• Area < 0.2 * average area.
Noise Elimination
At this stage, only characters and noise will remain. The remaining task on hand is to remove the noise portion. The second important function of labeling lies in the calculation of area so as to identify the noise. Denote as max_Area and min_Area, the largest and smallest area of the connected components. The component with max_Area should be classified as an object, thus max_Area can be used as a reference value to differentiate between objects and noise.
Considering the area of the characters in one image should typically not vary considerably from each other, e.g., max_ Area < 6 × min _. Area As a result, if the area of the ith component satisfies a certain condition,
max_ Area say Area < , the area can be treated as noise or an unwanted part. In i ς this application, we use an empirical value of ς = 10. The processed results after filling and elimination are shown below in Fig 2.7.
(a) (b)
19
(c) (d)
Fig. 2.7 Abnormal Object Removal
Sometimes, bubbles may exist inside the extracted characters due to scratches, reflection or nails on the sign board. In these situations, a bubble filling algorithm can be used to fill the bubbles. Therefore, the filling algorithm can be used based on both labeling and area calculation. However, the labeling algorithm will pose additional computational burden with no considerable effect on recognition. Thus, this step has not been adopted, in this chapter, in order to reduce the processing time.
2.3 Character Recognition
OCR (Optical Character Recognition) is required in order to translate the extracted character. Most of traditional OCR algorithms focused on recognizing handwritten characters [34], [55], [56], [73]. The key step in recognizing handwritten character is to segment connected letters, extract features (e.g. contour representation in [34]) and correct misspelling [55]. It will be less efficient to apply the above OCR algorithms to translate characters extracted from scene images since those characters are more standard than handwritten characters.
Considering that mobile application is a time-critical application, a simple yet efficient OCR is developed to strike a balance between computational cost and
20 accuracy. It utilizes the two main features of a character: crosses and white distances.
Definitions of crosses and white distances:
Take character ‘A’ which is strictly confined in a square box as shown in Fig
2.8(a) for example. The white pixels are 0 and blue pixels (i.e. body of the text
‘A’) are 1.
(1) A cross (either vertical or horizontal) is defined as the number of zero-one crosses as the character is traversed from one side to another (either vertically or horizontally) as shown pictorially in Fig 2.8(a).
(2) White distances are defined as the distance as it is traversed from one direction (left, right, downwards, upwards) until the first 1 is met, as shown pictorially in Fig 2.8(b).
Using the information of vertical and horizontal crosses, left white distances, right white distances, top white distances and bottom white distances and comparing them against a template of these six traits of all alphabets, characters can be differentiated with very high accuracy.
21
(a) Definition of crosses
(b) Definition of white distances
Fig. 2.8 Pictorial Definition
(a) Pictorial definition of crosses ; (b) Pictorial definition of white distances (shaded)
For example, the six attributes of letter ‘A’ is as follows in the application.
(White distances are obtained by normalizing the object into a 15x10 size object)
Vertical_crosses = {1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2};
Horizontal_crosses = {1, 1, 1, 2, 2, 2, 2, 1, 1, 1};
Left white distances = {3, 3, 3, 3, 2, 2, 2, 2, 1, 1, 1, 0, 0, 0, 0};
22 Right white distances = {4, 3, 3, 3, 2, 2, 2, 2, 1, 1, 1, 0, 0, 0, 0};
Top white distances = {13, 9, 6, 2, 0, 0, 2, 5, 9, 12};
Bottom white distances = {0, 0, 2, 3, 3, 3, 3, 3, 0, 0};
A group of templates of all information about the 52 alphabetical letters (both capital and small letters) is prepared; detected attributes of each letter are compared against the templates using Least Squares Method.
Finally, letters identified from the image and converted to ASCII values are subsequently filled into a linguistic model to be grouped into words and checked against a dictionary which was embedded inside a mobile device. The meanings or translations of the texts will be retrieved immediately. Those parts, which cannot be recognized, such as the ‘dirty’ spot in Fig 2.9(d), will be ignored.
Certainly, there are situations when a letter is misrecognized and a translation cannot be done. For example, capital letter ‘I’ and small letter ‘l’ are very much alike. Error correcting models can be built to tackle these situations.
Translation includes both words and phrases. Common phrases included in the dictionary like “car park”, “fire hose reel”, “keep door closed” and etc can be directly translated, while cases like “slippery when wet” will be translated separately word by word.
2.4 Experimental Results
In this section, we will demonstrate the effectiveness of the application developed.
28 images captured via the mobile phone were used for the experiment, giving a total of 389 characters. Table 1 lists the recognition results. The table shows problems arising when images are captured at an angle. Therefore, users are
23 encouraged to capture sign images with a minimum offset angle for further
processing.
Table 2.1 Recognition Results
Original Image Extracted Recognition Original Image Extracted Recognition Result Result Result Result
EXIT E4A 7 TH STOREY
Kent Vale
N/A Car Park
FIRE HOSEREE WAY
OUT
Color Image Please use Processing and Underpass Applications
Morphologi PEDESTRI
cal Image AN THIS Analysis WAY
EXECUTIV KENT
E VALE
SEMINAR CARPARK
ROOM 3
24 PREMISES
CAMERA
SUVEILLA SLOW
NCE
Court 1 to 4
Counter 18 Watch Out
Room 19 to For Traffic
27
MINISTRY
OF BLOCKS MANPOWE C D
K
Meeting @ BEWARE
MOM OF LOW
CEILIN
FULLY ONE
PAID WAV
E4 Keep out
4TH
STOREY
LOADING
Keep door AND
closed UNLOADI
NG
FIRE NUS
25 2.5 Conclusions
In this chapter, we have presented an approach for text extraction from images captured from sign boards with mobile phones. The approach is also viable as an alternative way to send SMS from images captured and to extract URL text from a complex background and directly link the user to the website via his mobile device. The approach is based on a fusion of color and edge information. The strategy for text extraction is designed to strike a delicate balance between computational efficiency and identification accuracy. The results of experiments on 28 real images were duly presented in the chapter to demonstrate the viability of the proposed approach for this purpose. Using entropy to calculate the information amount, the method to enhance contrast and approaches for color & edge fusion and character recognition are the innovative methods forthcoming from the chapter to effectively differentiate objects and background.
26 CHAPTER 3 Vision-Servo System for Automated Cell Injection
Recent developments in nuclear reprogramming and intracytoplasmic sperm injection reflect an increasing need for more advanced and automatic micromanipulation technologies. In this chapter, an automatic cell injection system is developed, which is capable of visually monitoring the injecting process and controlling the microactuators. Traditionally, cell injection was manually operated, and it was laborious, time consuming, of low accuracy, and prone to contamination due to the handling requirements. An automatic and efficient strategy is required to eliminate these drawbacks. In this chapter, a system is developed where the injection process is monitored and controlled automatically via integration of a vision system to an injector manipulation system. The cell is located, and the pipette is positioned and driven by the algorithm to achieve effective penetration. The precision achieved is physically proven to be within a good tolerance range.
3.1 Introduction Biological injection has been widely applied in transgenic tasks. In spite of the increasing interest in biomanipulation, it is still mainly time-consuming and laborious work performed by skilled operators, relying only on the visual information through the microscope. The skilled operators require professional training, and success rate has not been high. Moreover, an improper operation may cause irreversible damage to the tissue of the cell due to the delicate
27 membrane, which can arise directly from errors and lack of repeatability of human operators. All of these result in low efficiency and low productivity associated with the process. Hence, in order to improve the biological injection process, a software-based automated biomanipulation vision system is desired to more efficiently replicate the repeatable and laborious manual injection process.
Fig. 3.1 Bio-manipulation System
Vision-based robotics and machine have been widely studied [11], [14], [40],
[76], [82]. Works on vision-based object recognition and tracking techniques with application to control systems are reported in [2], [10], [13], [41], and [98]. The main objectives of the vision system are to enable automation by providing real- time position information to the positioning system, increase the injection speed, and achieve repeatable outputs with a satisfactory success rate. Previous works on the development of such mechanical systems are as follows. Kim et al. [16] and
Cho and Shim [79] studied the injection force and designed controllers to carry out the injection. Mattos et al. [51] designed a semiautomated microinjection
28 system. The video is acquired by a charge-coupled device (CCD) camera, whereas the process is teleoperated by an operator via a joystick. Wang et al. [96] presented a cell detection algorithm using Bayesian analysis. However, their work does not include algorithms for pipette detection and tip focalization. Recently,
Sun and Nelson [95] suggested a software-based automatic injection algorithm using a template matching technique. They updated the position of the pipette by minimizing the difference between the image and the predefined template. In addition, they asserted that the pipette is in the focus plane when the error between the image and the template is minimized. Template matching has its limitations, one of which is to obtain a new template once we change the pipette.
In this chapter, we use a configuration of biomanipulation system for biological injection, which includes a microscope, CCD camera, motion manipulators including a piezo manipulator for the final penetration, and the motion controller residing in a host computer (see Fig 3.1).
Fig. 3.2 Vision-assisted Servo System
29
Fig. 3.3 Flowchart of Process
In this setup, the vision system (comprised of the CCD camera interfaced to an image processor within the host computer) is integrated to the pipette manipulation system, as shown in Fig 3.2, where the vision system provides current and target position information to the control system which will drive the manipulators to bring the pipette to the proximity of the cell and execute the penetration sequence. Thus, the whole process, from relative positioning of the cell and pipette to the injection, is automated, and it can be activated by using a push button.
The software on the host computer differentiates the design from previous work in supervising the process and tuning the pipette to inject the cell automatically without human involvement. A reported piezo injection system is used to carry out the final penetration [7]. This chapter will mainly focus on the
30 algorithms for cell detection and pipette focalization and positioning.
Theoretically, a 2-D image can be constructed if only one lens is installed on the microscope. In our system, a 3-D structure can be accommodated by using this software. There are two main challenges with respect to pipette detection and focalization. First, the injection vibration will affect the accuracy of pipette detection and may cause fatal damage in the cell. To minimize vibration, the injection system is rigidly secured to a stabilized table. Second, the deformation
[86] of each cell is not identical due to the varying elasticity. The proposed algorithm will be based on three assumptions. First, since we will not study the injection force [16] and the cell deformation so as to reduce computation, the cell is modeled as a sphere or “dual-core” sphere for convenience. Second, the holder and the cell remain static before the pipette touches them. Finally, we have the estimated size of the cell, which is defined as R. The complete procedure for the entire application is shown in Fig 3.3, which is composed of two stages described as follows:
Stage A) system setup (manual operation)
1) Cell stabilization and placement;
2) Pipette placement.
Stage B) automated focusing and injection
1) Cell detection;
2) Cell focalization (Z-axis coordination);
3) Pipette detection;
4) Pipette tuning (Y -axis coordination);
5) Pipette focalization (Z-axis coordination);
6) Penetration (X-axis coordination).
31 The rest of this chapter is organized as follows. Section 3.2 will briefly describe the system setup stage. Section 3.3 will highlight the cell detection algorithm.
Section 3.4 will describe the pipette detection and tuning method. Section 3.5 will outline the autofocalization algorithm, which is the key step of the whole process.
Section 3.6 will present the penetration step. Section 3.7 will validate the precision of tip focalization algorithm with a physical test. Finally, in Section 3.8, this chapter will be concluded.
3.2 System Setup
The injection system is composed of a microscope for the initial setup, a set of manipulators for moving the pipette to the desired position near the cell before executing the penetration sequence, a CCD camera serving as a sensor for the relative positions of cell and pipette, and a computer hosting the motion controller and image processing algorithms. The main objective of the system setup part is to make preparations for injection, including cell stabilization, tuning, and pipette placement. It involves two steps as follows (Fig 3.4).
A. Cell Stabilization
The cell is held by using a holder. Meanwhile, other cells are moved outside the screen, and the holder is placed on the left-hand side of the screen.
B. Pipette Placement
Although the pipette positioning and penetration into the cell can be automatically done, some rough initial position of the pipette and the cells is expected to be done manually to speed up the process, which is similar to the works reported in
[51], [79], and [95]. The pipette is thus first placed around the higher right corner of the screen. Both the holder and the pipette should be placed initially above the
32 focus in Z-axis coordinate. There is a risk of breaking the pipette if it is placed below the focus.
(a)
(b)
Fig. 3.4 Two steps in system setup
(a) Cell stabilization. (b) Pipette placement.
3.3 Cell Detection
In this section, we will present and compare the performances of two current methods of locating the cell. Hough circle detection [62] is the most common and classical method to find a circular object such as the cell. We can implement
Hough algorithm on the edge image, which can be created by using the same kernel described in [96] under which a pixel will be regarded as part of the edge if
33 the value of its strongest gradient is higher than 35. The resulting image is represented as E(x, y), as shown in Fig 3.5(a). The Hough transformation result for the cell is shown in Fig 3.5(b). Typically, the 2-D cell is not a perfect circle, and the Hough transformation will create an obvious “dual-core” result, which is shown in Fig 3.5(c). The cell configuration is shown in Fig 3.5(d). The average X and Y locations of the “dual core” will be used to point to the center of the cell.
The final result is shown in Fig 3.5(e). This algorithm will be called Hough cell detection algorithm (HCDA). HCDA is a common method for circle detection.
However, the main drawback is the large amount of computation efforts needed.
(a)
(b)
34
(c)
(d)
(e) Fig. 3.5 Hough circle detection
(a) Edge image. (b) Hough transformation. (c) “Dual-core” effect. (d) Positions of
“dual core.” (e) Final detection result.
For example, the time duration to execute HCDA for the images in Fig 3.5 is 22 s.
There are other recently published methods for cell detection. Wang et al. [99] published a complete cell detection algorithm, where the coarse detection eliminates the unwanted parts and the fine detection precisely tracks the wanted
35 cell by using Bayesian estimation. Canals et al. [66] used multiblock method to track moving targets with varying sizes. However, the aforementioned algorithms also require intensive computations which will not be accommodated for real-time applications such as automation of cell injection. Therefore, we will develop a faster and more efficient detection algorithm, particularly for cell injection scenarios based on the estimated cell size R. Since the holder and the cell are static before penetration, we can detect the cell with the feature of the holder.
Obviously, the boundary of the holder makes a strong gradient than other regions nearby. Hence, we can locate the top region of the holder via high-pass filter and
X-axis scanning. First, we use the same kernel to extract the edges of the holder, as described in [96]. Second, we locate the position of a rectangle in the X-axis which has the maximum number of black pixels in it. By supposing that the width and the height of the screen are W and H (in our application, the size of the screen is 704 ∗ 576 pixels), then the width and the height of the rectangle are 2 and 0.5 ∗
H, respectively. The result must be the position where the top of the holder is placed, which is denoted as X holder . Since the cell is on the right-hand side of the top of the holder, the X-axis coordinate of the center of the cell is thus found, which is X holder + R. The next step is to locate the center Y -axis coordinate of the cell.
(a)
36
(b)
(c)
(d)
Fig. 3.6 Faster cell detection
. (a) X-axis scanning. (b) Y -axis scanning. (c) Values of variances along Y -axis.
(d) Location of the cell.
Variance can be applied to measure disorder. It can also be used to locate a special region. Here, we calculate the variance of the pixels in a moving square along Y -axis, which has a side length that is equal to the diameter of the cell, and
37 the center X-axis coordinate of the square is fixed at the X holder + R. Fig 3.6(c) shows the variance values from Y = 100 to Y = 475. The cell will be located when the variance inside the square reaches a maximum value (Y = 273). The center of the square is also approximately the center of the cell. Strictly speaking, we should use a sphere model instead of a square to locate the cell. However, a sphere model requires higher computation effort than the square model; yet, the square model achieves the comparable result.
The pseudocode for the algorithm is given as follows.
BEGIN
1. Edge detection
2. X-axis scanning, range [0,W/2], find X holder
3. Variance calculation along Y -axis, find the center of the cell YCell
4. Store the position of the cell ( X holder + R, YCell )
END
The aforementioned algorithm is used to detect the object in the first frame of the video. We do not need to execute the same detection algorithm iteratively for every frame; instead, we only execute the first frame once. In spite of the assumption we made that the holder and the cell remain static before penetration, there may be tiny drift between consecutive frames. Therefore, we narrow down our scanning range to [( X holder − 2, YCell − 2), ( X holder + 2, YCell + 2)]. Here, we use [(X1, Y1), (X2, Y2)] to define a rectangle, where (X1, Y1) is the coordinate of the upper left point, and (X2, Y2) is the coordinate of the lower right point. The cell has much higher probability to be inside the above region than other regions, which is similar to “Trust-Region,” as described in [84]. The position of the
38 region will be iteratively updated by later frames. Once the cell is located, entropy-based Z-axis tuning algorithm can be applied to focalize the cell. We will discuss the algorithm in detail in pipette focalization section, where we use the same method to focalize the pipette.
3.4 Pipette Detection
Pipette detection algorithm will be initiated once the cell has been located and focalized. The proposed cell detection algorithm works well for a stationary object.
For the pipette, a more efficient method to detect it is based on its movement when the manipulator moves the pipette along the Y -direction at a low speed.
This movement of the pipette along the Y –direction allows the pipette to be easily identified from the background using a frame difference (FD) [15] method, which can be expressed as: Dt+Δt,t(X) = |X(t+Δt) − X(t)|, where X(t+Δt) and X(t) denote consecutive frames.
(a) (b)
39
(c) (d)
Fig. 3.7 Pipette Detection
(a) DF; (b) Noise reduction result; (c) Pipette highlighted; (d) Final detection result
Fig 3.7(a) shows the FD of image sequences, and Fig 3.7(b) shows the noise reduction result. Pipette is highlighted by a rectangle with similar size Fig 3.7(c).
Finally, the tip of the pipette can be regarded as the first black pixel to the left in the highlighted rectangle in Fig 3.7(c), and Fig 3.7(d) shows the final detection result. The task of this step is to tune the pipette to the center of the cell in Y -axis direction, drive the pipette vertically to YCell , and stop when the Y -coordinate of
pipette Yppt satisfies | YCell − Yppt | < 2 pixels (Fig 3.8).
(a)
40
(b)
Fig. 3.8 Y-axis Coordination
(a) Before coordination. (b) After coordination
After the aforementioned steps, the position of the tip ( Xtip , Ytip ) can be obtained.
3.5 Tip Focalization
This is the key step of the whole process, which will directly determine the success rate of the injection. Template matching technique, which was suggested by Sun and Nelson [95] in focalizing the pipette, has its limitation as discussed before. Here, we can focalize the pipette based on the entropy computation.
Simply put, the tip is focused when it reaches its maximum clarity, and clarity can be indicated by the value of entropy of the corresponding regions (Fig 3.9).
(a)
41
(b)
Fig. 3.9 Tip Focalization
(a) Region of Analysis. (b) Focused
When the tip is focused, only a portion of the pipette near the tip will appear clear through the microscope, and the rest part will appear blurry due to the angle between the pipette and the dish. Thus, we can analyze the value of entropy in a rectangle region which covers the clear portion of the pipette to determine whether the tip is focused.
Suppose the position of the tip is ( XYtip, tip ) , we calculate the entropy inside the
⎡⎤ rectangle ⎣⎦()XYtip−−5, tip20 , ( X tip + 120, Y tip +30) as follows:
1 Hppp=−∑∑iii ×log = × log( ), pi where H denotes the entropy of an image, and pi represents the proportion of gray-scale values in the range i ∈ [0, 255] over the entire image. The pipette is initially and manually placed above the focus. The graph in Fig 3.10 shows the entropy value (multiplied by 1000) through the entire tuning and penetration. In the Y –axis coordination stage, the value is roughly around 3600 due to the dithering of the rectangle. Moreover, there are two stages in the Z-axis coordination. In stage A, the pipette is tuned down to the focus and overtuned to
42 be below the focus to diminish the disturbance of dithering. In stage B, pipette will be tuned back to the position where the maximum entropy value was reached.
Fig. 3.10 Value of Entropy
3.6 Penetration
Through the previous steps, the tip can be adjusted to the center of the cell both along the Y - and Z-axes. Now, we are ready to penetrate the cell. The penetration distance between the tip and the center of the cell through calibration is | Xtip
− X holder − R|. Cell deformation should be taken into account due to elasticity.
Actual distance should thus be | Xtip − X holder − R + η|. The penetration of the cell with the pipette entails a precise and tightly controlled motion of the tip of the pipette (Fig 3.11).
43
(a)
(b)
(c)
Fig. 3.11 Penetration
(a) Before Penetration. (b) After Penetration. (c) Validation
An approach to execute fast and precise linear and partial rotation motion using a piezo manipulator has been developed and proposed in [7] to achieve a more efficient penetration without undue deformation of the cell, thus increasing its survival rate. The same computer-controlled system [7] with adaptive controller
44 [94] is used here, which is similar to [12] and [61]. In [7], manual positioning effort is needed to position the pipette tip along the Y - and Z-directions relative to the center of the cell. With the vision system developed here and integrated to the manipulators, the whole process shown in Fig 3.3 can now be automated (see Fig
3.2).
3.7 Validation
In this section, we will demonstrate the precision of the algorithm. Previous work related to focalization [95] did not experimentally verify if the tip is indeed focused; instead, their focalization methods were purely based on experience and visual information. Here, we provide an experimental verification that the tip of the pipette is focused. After the tip penetrates the cell, we continue to penetrate the pipette along the X-axis until its tip enters the hole of the holder. Entering the hole of the holder proves that the tip is focused and is positioned at the center of the cell. The maximum error distance is DError = | Zcell − Ztip | < Radius of the
hole, where Zcell and Ztip denote the Z-axis coordinates of the cell and the tip, respectively. Compared with the size of the cell, the hole is significantly smaller so that penetration through the hole represents a sufficiently high level of accuracy achieved. It should be clarified that this process is only used to validate the precision of the system. Besides the time efficiency of the automated approach, we investigate the success of the whole process by looking at the survival rates of the cells. The experiment fails if the tip slips over the cell during penetration or the membrane is destroyed after penetration. We adopted the piezo-actuation
(PA)-assisted injection scheme [7], which controls the position of needle directly
[25], [71].
45 Table 3.1 Comparison of Experimental Result
Table 3.1 shows the comparison of actual experiment results conducted by the manual operation, the manual PA-assisted operation [7], and the vision-based PA- assisted operation. It should be noted that apart from the penetration process, there are other factors influencing the survival of the cells after penetration, including the chemical medium used, the needle tip profile, etc. It is clear that the vision- based approach outperforms the previous two by achieving a higher survival rate of the cells after penetration, which demonstrates the precision of the proposed algorithms. In addition, it saves much time and labor by eliminating poor repeatability of an operator to manually carry out the operation with or without the piezo actuator. The cost of the automation system is about $6000, but with automation, the demand on the skills of an embryologist is lower, and a larger number (about 30%) of operations can be carried out consistently over the same time.
46 3.8 Conclusions
In this chapter, we have presented a vision-servo system which can be used for automatic cell injection. The algorithm is based on feature detection, tracking, and autofocalization. The purpose of this system is to replace the conventional laborious and repeatable manual work with an automated approach which yields a higher success rate. Finally, we provide a verification of the accuracy of the algorithm. The efficiency and the accuracy of the vision-servo system in facilitating a fully automated cell injection task are also demonstrated experimentally under realistic practical conditions expected. The feature-based object detecting algorithm and the entropy-based pipette focalization algorithm are innovative methods forthcoming from this chapter to realize automatic injection.
47 CHAPTER 4
Vision-based Tracking and
Monitoring System for CNC Machine
Surveillance
Fault diagnosis and predictive maintenance addresses pertinent economic issues relating to manufacturing systems as an efficient technique can continuously monitor key health parameters and trigger alerts when critical changes in these variables are detected, before they lead to system failures and production shutdowns.
In this chapter, both the conventional sensor based monitoring system as well as the design and development of a vision-based thermal monitoring system for
CNC machine are described and compared to address the aforementioned issues will be presented.
4.1 Introduction
System monitoring, fault diagnosis and predictive maintenance are fast becoming key and integral components of modern production systems. An efficient diagnostic system can maintain tools in good condition and prevent severe failures by detecting and localizing faulty components at an early stage. Various methods have been developed to identify tool wear states. The application of statistical
48 algorithms to associate patterns in measurable signals with wear states is given in
[77], Weck [60] and Byrne et al. [21] used the cutting force signal to monitor tool wear. By using inexpensive current sensors, several intelligent tool wear monitoring systems have been developed [93], [91], [92]. Coker and Shin [72] developed a method for in-process monitoring and control of surface roughness during machining processes via ultrasonic sensing. In [6], the acoustic emission sensor and accelerometers were used to monitor progressive stages of flank wear on carbide tool tips. A different approach to tool wear monitoring is to apply system theory ideas to the estimation of wear states during the cutting process. In
[32], a linear model was built to detect the tool wear and breakage in a drilling process. In [46] and [47], a linear model is used to design an adaptive observer for on-line tool wear estimation of a turning process. The main objective of a tool wear detection approach is to sense the loss in the original functions of tools in order to detect an abnormal state. Tool wear diagnosis approaches have focused on the development of signal processing techniques on the measurements, such as cutting force, vibration and spindle motor current, etc. However, tool information from a single measurement may not give a reliable indication, due to the complicated dynamic characteristics of the cutting process and the sensor noise present. To circumvent this problem, a multi-sensor approach has been presented
[67], [5]. This requires a higher-redundancy hardware device to be used for simultaneous yielding data collection and handling of much more information. In addition, there are many CNC machines in workshop and it is necessary to consider a novel monitoring system for fault detection.
A feature common to these schemes is the assumption that some states are available (measurable). This assumption inevitably poses restrictions on their
49 applicability in common and practical scenarios. In practice, many of the variables required for monitoring and fault detection are not required for the basic functions of the system and are thus not naturally present. Even if the measurements are available, the systems can be of a closed-architecture and not amenable directly for the measurements to be tapped for use in the monitoring systems, without having to disrupt production to allow the necessary retrofitting to be done. In addition, the component to be monitored can be non-stationary and may be engaged in contact abrasive operations (e.g., machine tools) so that attaching separate and additional sensors to the part may not be possible.
4.2 Background and problem statement
In the university, there are many milling machines in a workshop. The typical feed system associated with a milling machine control consists of the following basic components: tool, tool post, slide, bearings, ball screw, feedbox, feed motor, and lubrication system, as illustrated in Fig. 4.1(a). Usually, the workpiece moves horizontally driven by the X-axis and Y-axis motors while the tool moves vertically driven by the Z-axis motor.
(a) CNC Machine
50
(b) Multi-machine in workshop
Fig. 4.1 A CNC Machine and workshop
The whole milling machine workshop is called the machine center as shown in
Figure 1 (b). Motivated by the introduction section, the goal of this chapter is to design a distributed system to detect the tool wear in multi-machine units and sending the signal to maintenance personnel.
In section 4.3, a distributed wireless sensor network has is designed and attached to the machine to sample the health parameter of the machine. In section
4.4 a vision-guided thermal monitoring system is proposed to real-time monitor the working machine. The comparison of the two strategies will be made.
4.3 Distributed Wireless Sensor Network for CNC
Machine Surveillance
Wireless sensor network (WSN) has been widely used in machine surveillance. A typical WSN-based monitoring system consists of wireless sensors, base station to receive data and software in base station to further analyze the data.
51
(a) Antenna board (b) Sensor board
Fig. 4.2 Sensor board and antenna board
A typical distributed fault detection system (DFDS) structure can be presented as following:
Fig. 4.3 DFDS control structure
Sensors are attached to the machines and waiting for the command from the base station to collect the parameters of the machines such as vibration and sound. The parameters will be fused and sent back to base station. Decision making will be done based on the strategy of the software in the base station. An alarm signal will be generated once the collected parameter is higher than a pre- defined threshold. Under such circumstance, the operator shall stop the machining
52 process to prevent severe failure. The flow of such monitoring algorithm is shown below:
Fig. 4.4 Algorithm flow chart
Distributed Fault Detection Model
In this section, the method developed for distributed fault detection system, and its constituent components, will be elaborated in detail.
To describe our ideas about the DFDS we introduce the following model.
Milling machine dynamic model: Models of the cutting process have been derived and studied in [54], [44], [85]. For example, Lauderbaugh and Ulsoy have proposed the following model:
2 FFFKf&&++=ζωnn & ω sr, (4.1)
53 where F is the cutting force which is the output for the system, fr is the feed rate,
and the parameter ζ , ωn , Ks depend on the depth τ of cut, spindle speed υ , and
feed rate fr . This equation can be re-written as:
x& = Ax+ bu, (4.2) yCx= T ,
T ⎡⎤ With x ==⎣⎦FF,& , u fr , and
⎡⎤0 1 ⎡0 ⎤ T A = ⎢⎥, b = ⎢ ⎥ , C = [1, 0] . (4.3) ⎣⎦-ωn −ζωn ⎣Ks ⎦
The model is weakly nonlinear since the model parameters are changing slightly depending on the operation condition, and has significant process parameter variations (see [14]). In this work, we consider a linear model with parameter uncertainties
x& = ( AAxbbud+Δ) +( +Δ) + , (4.4) yCx= T ,
Where A, b are the nominal matrices of the system, and
⎡⎤0 1 ⎡ 0 ⎤ Δ=A ⎢⎥, Δ=b ⎢ ⎥ , (4.5) ⎣⎦−Δat12() − Δ at () ⎣Δbt1 ()⎦
2 where ΔΔat12(), at () and Δb1 are the perturbation parameters of ωζnn, ω and Ks , respectively. d is the bounded disturbance. Clearly, this model can include more classes of systems than those used in [54].
Since the sampling time for cutting forces is very fast (sampling time of
0.00001sec), they cannot be transmitted to the top layer by using wireless network.
A micro-computer sensor instrumentation system is used to handle the real-time signals. The cutting force signal processing is based on an observer given by
54 xˆˆ& =++Ax bu K( y − CT x ˆ) (4.6) yCxˆˆ= T
T where x&ˆˆˆ= [xx12, ] denotes the estimate of the state
T ⎡⎤ T x ==⎣⎦FF,,& K[] k12 ,,, kK kn is the observer gain vector. Only the output y is assumed to be measurable. Define the state and output estimate errors as x%%=−xx and yyy%%=−. Thus, the error dynamics is given by
T x%%& = ( AKCx−+Δ+Δ+) Axbud, (4.7)
T y% = Cx, (4.8)
A possible fault detection approach may check if the following conditions hold.
yE% > T , (4.9)
where ET is a threshold value. If its value exceeds the threshold, a signal will be sent to the top layer using the wireless network. The top layer will evaluate this message together with the sound and vibration information provided by the bottom layer and make a decision.
For the sound and vibration dynamics, it is difficult to use a detailed model to describe them. Instead, we use the following model.
zfzt& = ( ,,) (4.10)
where zzzzz= ()x ,,,yzs is the sound and vibration variable. These variables are
defined as: zzx , y and zz represent the vibration along thex , y and z-directions
respectively; zs is the sound signal.
Sensor-actuator model: We define the sensor set:
Ssss= ( 12,,K ,N ) , (4.11)
55 where ssssiiii= ()12,,,K 5 . Here, syszszszsziixiyizis12= , ====, 3, 4, 5, the actuator set:
A = (aa12,,,K aN ) , (4.12)
where ai is the stop command.
Network model: A network NS is a networked collection of sensors and actuators, which are working together to achieve some type of information collection and processing. We define the network as
NS( S,: A) S− ><− S ; A A , (4.13) where S− > S means that the sensor information at the bottom layer are collected and sent to the top layer and A < −A means that the command messages from the top layer are sent to the bottom layer.
Algorithms for computer center: The goal of the algorithms is mainly for data interpretation and use. An inference system at the top layer is used to make decisions and to give the correct command. The decision principle is to satisfy safety criteria and ensure that a fault occurring is detected. Once the objective has been identified, the decision layer results will be transmitted to the lower layer and be sent to operators. The detailed decision routine is shown in Figure 4.3
Fig. 4.5 shows the vibration detection results in X & Y directions and sound.
56
(a)
(b)
57
(c)
(a) Vibration X-axis (b) Vibration Y-axis (c) Sound measurement
Fig. 4.5 Fault detection with SS=1200 rpm, fr =300 mm/min, depth of cut=1
mm
(SS = spindle speed, fr = feed rate).
The obvious shorting coming of the sensor are the power constraint, low sampling rate and prone to disturbances. Under such circumstances, a vision- based thermal monitoring system is designed which is more reliable and less expensive in such task.
4.4 Decoupled Tracking and Thermal Monitoring of Non-Stationary Targets
In this section, a decoupled tracking and thermal monitoring system is developed which can be used on non-stationary targets of closed systems such as machine tools. There are three main contributions from this chapter. First, a vision
58 component is developed to track moving targets under monitor. Image processing techniques are used to resolve the target location to be tracked. Thus, the system is decoupled and applicable to closed-systems without the need for a physical integration. Secondly, an infrared temperature sensor with a built-in laser is deployed for non-contact temperature measurement of the moving target. Thirdly, a predictive motion control system holds the thermal sensor and follows efficiently the moving target to enable continuous temperature measurement and monitoring.
4.4.1 Overall System Configuration
The monitored system in this chapter is a milling machine and the condition of the cutting tool will be monitored by inferring to the temperature of the cutting point.
The temperature measurements are not naturally available and it is not possible to mount temperature sensors on the tool or the workpiece so a non-contact mode of temperature measurement will be necessary. An infrared thermometer with built- in laser will be used for this purpose. In addition, the tool is non-stationary throughout the cutting process. Hence, the thermometer must be moved in tandem with the milling tool to track the tool. The thermometer is thus mounted on a decoupled motion stage. As the milling machine is a closed system, its encoder signals are not readily available for use by the motion stage carrying the thermometer to track the tool. A video camera is used instead to record the motion of the milling tool and image processing algorithms will extract the position information related to the milling tool tip for the motion stage to track.
59
Fig. 4.6 Overall System Configuration
There are thus three major subsystems in the overall system. The vision and image processing system will process real-time images to extract the position of the milling tool. The non-contact infrared thermometer will efficiently provide temperature measurements of a selected spot on the milling tool. The computer control system will move the thermometer in tandem with the movement of the milling tool based on the position feedback from the vision system. Fig 4.6 shows the interaction among these three subsystems.
60
Fig. 4.7 Vision-assisted Servo System
The development of the proposed system will be based on two assumptions.
First, the camera is stationary. Secondly, the illumination condition is consistent and will not experience considerable changes. The initial setup will mainly involve the calibration of the camera and proper mounting of the thermometer on the motion stage.
Camera Calibration
The camera should be able to resolve the actual motion of the milling tool from the images captured. Therefore proper calibration should first be done as follows.
The camera (screen size: 768× 576 ) is mounted on a table located at a fixed distance of 500mm away from the milling tool for safety reasons and it is adjusted so that the maximum travel of the tool is contained within the view of the camera.
Within this span, the tool is manually displaced by a fixed distance DCR and the
61 equivalent displacement of the image captured by the camera is denoted as DCV .
The transformation coefficient k can be computed from DkDCR= × CV . With this
calibration, subsequently, the displacement of the tool DR can be calculated based
DCR on the image displacement DV through DkDR =×VV = × D. DCV
Thermometer Mounting
The thermometer used is OS550A industrial infrared thermometer from Omega
Engineering. It has a distance to target ratio of 609mm: 8.9mm (diameter of the tip of the cutting tool is 12mm). Thus, the thermometer has to be adequately fixed to yield accurate temperature measurements. First, the linear motor (on which to mount the thermometer) is located at a position 609mm away from the milling tool and aligned with the direction of travel of the tool. The thermometer is mounted on the motor such that its laser is directed at the spot of the tool to measure its temperature (Fig 4.7).
Fig. 4.8 Mounting of the Infrared Thermometer
In the ensuing sections, the three main subsystems of the overall monitoring system will be elaborated in details.
62 4.4.2 Vision and Image Processing System
The purpose of the vision and image processing system is to detect and track the milling tool so that the thermometer can be driven by the linear motor to follow the milling tool and continuously monitor its temperature. The milling tool will be detected through first few frames and subsequently tracked by updating its position iteratively.
Moving object extraction is an active field of computer vision and it has wide practical applications in industrial monitoring system [22]. Effectively detecting and tracking target object from video sequences is the main task in the proposed surveillance system. Currently three classical algorithms can be used in video surveillance systems. In depth reviews on these methods can be found in [26],
[57], [97]. Optical flow [9] is the most computationally intensive method, which is not suitable for real-time processing in many practical applications. Background subtraction [1], [4], [27], [59], [45], [99], which is usually used in traffic monitoring, detects moving object by comparing the current frame with a reference. Typically, the object to be tracked only occupies a small portion of the frame. However, for the target application in this chapter, this method is not applicable since the background reference is not available and to obtain one will entail moving away the machine which is cumbersome. Frame difference [64] is the simplest method and easy to implement. It creates a binary image by comparing two consecutive frames, if the difference of a pixel is higher than a threshold the pixel will result in 1 otherwise 0. The main drawback of frame difference is its sensitivity to Gaussian noise. Considering the time-critical requirement of our monitoring system, the tool detection algorithm will be based on the frame difference method. The process flowchart is shown in Fig. 4.
63
Fig. 4.9 Process Flowchart
The basic idea of our proposed algorithm is to highlight the salient moving gradient [26] between frame sequences, which can be expressed by:
dIxyt(, ,)∂∂∂∂∂ Ixyt (, ,) x Ixyt (, ,) y Ixyt (, ,) =++ dt∂∂ x t ∂∂ y t ∂ t
∂∂xy =++GGxy It ∂∂tt
∂x ∂y Gx and Gy represent the moving gradients along the x and y directions ∂t ∂t
respectively. The CCD camera captures 30 frames per second. After acquisition,
we create the edge images Exyt( ,,− Δ) of Ixyt( ,,− Δ) using the same kernel as
reported in [96]. They are computed as follows:
255 if I ( x , y , t ) is not edge pixel Exyt(),, = { 0 if I ( x , y , t ) is edge pixel
64 Meanwhile, we generate two difference images Dxyt( ,,) and Dxyt(),,−Δ by comparing the illumination difference ΔLxyt( ,,) , computed as follows:
L() xyt, ,=× 0.299 I ( xyt , , ). R +× 0.587 I ( xyt , , ). G
+× 0.114IxytB ( , , ).
⎧0 ifLxyt (, ,)− Lxyt (, ,−Δ ) > T Dxyt(),, = ⎨ ⎩1 ifLxyt (, ,)− Lxyt (, ,−Δ ) < T
A proper value of threshold T should be set based on illumination and contrast. In
our application, a suitableT value is found to be 5. Moving edges MExyt(),, can
be computed as:
ME( xyt,,) =× D( xyt ,,) E( xyt ,,)
As discussed before, frame difference method is sensitive to noise and a simple
moving edge operation actually has the sum of edges of both frames [98], which
will contain the ‘residual’ of previous contours. Thus, common edge operation is
added to eliminate this ‘residual’. Common edge CE( x,, y t) is regarded as AND operation of two consecutive moving edges, which can be computed as:
ME( x,, y t)× ME( x ,, y t −Δ) CE() x,, y t = 255
CE() x,, y t is a binary image with black moving edges and white motionless part.
Noise or tiny moving objects are also highlighted in CE( x,, y t) . Therefore, a noise removal step is needed to eliminate them. We use a statistical approach to realize this noise elimination, through the equations below:
⎧255 if NR( x , y , t) = 0 ⎪ ⎪ n=3m=3NR() x++ n,, y m t ⎪ and ∑∑ >3 NR() x,, y t = ⎨ nm=−33 =− 255 ⎪ ⎪ ⎩⎪ NR() x, y , t otherwise
65 An isolated black pixel is regarded as isolated noise and it will be turned white if it has more than three white neighboring pixels.
(a) (b)
(c) (d)
(e) (f)
66
(g)
Fig. 4.10 Moving Object Extraction
(a) Original Image (b) Difference Image Dxyt( ,,) ; (c) Difference Image
Dxyt(),,−Δ ; (d) Edge Image Exyt( ,,− Δ) ; (e) Common Edge CE() x,, y t ; (f)
Noise Removal Result NR() x,, y t ; (g) Localization
Following the above steps as summarized in Fig 4.9, the contour of the milling tool will be obtained. A feature-based localization method is then used to locate the position of the milling tool. In the proposed algorithm, the edges of the milling tool can be located from two 2 × L rectangles, where L is the estimated length of the milling tool. The final position of the tool can be obtained from the mid-point between the two sides. A template matching algorithm will be required to locate an irregularly shaped milling tool. Salient gradient can be found
horizontally within the estimated vertical range of the milling[YY12, ], which can be expressed by the following pseudo code:
Begin : x==0, Max 1 0, Max 2 === 0, Side 1 0, Side 2 0;
Loop : Count the number of edge pixels PiCt within rectangle
[] (x, Y12 ), (x +1, Y )
67 if ( PiCt>== Max 1) { Max 1 PiCt ; Side 1 x ;} else if(2){2;2;} PiCt>== Max Max PiCt Side x Side12+ Side MillPosition = 2 x ++; if( x=− Width of the screen 1) End; else go to Loop;
We can thus locate the milling tool based on the above described algorithm through the first few frames. To reduce the amount of computation necessary, we can narrow down the search to [MillPosition− 5, MillPosition + 5] along the horizontal direction in the following frames.
4.4.3 Non-Contact Temperature Measurement System
Conventional temperature sensors such as thermistors and thermo-couples are not applicable to measure temperature on parts which are moving and/or in contact with other parts. The target application, addressed in this chapter, is one such example. A non-contact temperature measurement system will be necessary in these cases. Thermal cameras (Fig 4.11 (a)) can fill these gaps, and they are finding their ways to a variety of applications. In [20] and [23], the authors proposed the medical applications of thermal images from thermal cameras. In [33] and [49], fault diagnostic schemes based on thermal images were reported. In [53], a security surveillance system using the thermal camera was reported. However, one main drawback of using a thermal camera is the high cost associated with it.
An alternate way to measure the temperature in a non-contact way is to use an infrared thermometer with a built-in laser tracer (Fig. 4.11 Thermal devices (b)) to pinpoint the spot on the part to be measured. However, such a thermometer can
68 only detect temperature of a spot instead of an extended area. The difficulty becomes even more pronounced when the part is non-stationary. The challenge is thus to formulate a moving platform on which the thermometer is mounted to move in tandem with the moving part closely so as to obtain continuous and reliable temperature measurements from it.
(a) Thermal Camera (b) Infrared Thermometer
Fig. 4.11 Thermal devices
4.4.4 Tracking Control of Linear Motor
In the target application, the infrared thermometer is carried on a linear motor (see
Fig 4.7). It can measure the temperature of a moving object as long as the motion of the object is tracked sufficiently by the motor. Since the cutting tool is a moving target here, the tracking control system has to allow the motor to follow the tool based on the image position from the camera.
The 1-DOF motor system under investigation is a mass constrained to move in one dimension with friction and periodic forces present between the mass and the supporting surface. The equation for this model is described as follows:
M&&xDxFu+ & +=l ,
69 where x denotes position; M, D, Fl denote the mechanical parameters: inertia, viscosity constant, and load force respectively; u denote the control torque.
The PID control design is based on the error ex= d − x, where xd is the objective signal given by the image position whose range belongs to
[X_ Position−+5, X_ Position 5] as shown in the last section. Since
d t ∫ edτ = e, dt 0 the standard PID control structure is given by
t uKeKedKe=+pi∫ τ +d & 0
where K p ,Ki ,K d are the PID control parameters. The trial and error method can be used to tune these parameters. Alternatively, advanced optimum control theory can be applied to tune PID control gains. The PID feedback controller designed using the Linear Quadratic Regulator (LQR) technique can be found in [27]
(Please cite the number for this reference). The control structure is shown in Fig
4.11.
Fig. 4.12 Control System Structure
70 Since the image obtained from the camera includes time delay, the information can not be used for the control loop directly. We propose a signal processing technique to solve this problem.
Consider the following model for the position information
xkdd()(1)(1)(1)= xk−+ Tvk −+ wk −
wherexd (k ) and v(k ) are actual moving position and velocity of cutting tool, k is the sampling time of the control system, and w(k) is the noise. Define y(k ) and z(k) as the position and velocity obtained from the camera. From the previous section, it is known that
yk()= x ( k− h ) d zk()=− vk ( h ) where h is the time delay due to the camera handling.
Due to the delay measurement, it is necessary to develop a model to predict the current position information for the purpose of the motion control. To achieve this objective, we establish the following model
xkˆˆdd()= xk (−+ 1) Tvk ( − 1)
Since z(k) is a delay signal, we predict the position information with assuming that the speed is constant during time [k-h, k].
xkhˆˆdd(1)()()−+ = xkh − + Tvkh − xˆˆ(2)(1)()kh−+ = xkh −+ + Tvkh − dd M xkˆˆ()=−+− xk ( 1) Tvkh ( ) d d
Thus, we have
71 xˆˆdd()()k=−+−=+ xkh hTvkh ()()() zkhTvkh −
For a conventional PID feedback control system, the closed-loop system can be guaranteed to be stable for a nominal system. A stability analysis will be provided below.
t
Let the system state variables be assigned as x123= ∫ edτ ,, x== e x e& . 0
T Denote X = [,xxx123 , ], the above equation can then be put into the equivalent state space formulation.
x&&&&=++−Ax Bu B() Mxddl − Dx − F
With
⎛⎞01 0 ⎛⎞0 A = ⎜⎟00 1 , B = ⎜⎟0 ⎜⎟⎜⎟ ⎝⎠00−DM / ⎝⎠−1/M
For PID control uKx= with K = [,KKip , Kd ], the closed-loop system is given by
x&&&&=+()(ABKxBMxDxF +−ddl − − )
The PID controller can be tuned to stabilize the system by appropriate design of the matrix A+BK. It is well-known that the LQR design can guarantee the closed- loop stability. Let us see the dynamics ignoring the disturbance.
x& = Ax + Bu
The control performance is given by
∞ J = ∫ (xT Qx + ru 2 )dt 0 where Q and r are positive. Our LQR problem is to find the optimal control u such that J is minimized. From the reference of [], the LQR solution with the performance index above is u = −r −1BT Px
72 ⎡ p11 p12 p13 ⎤ where P = ⎢ p p p ⎥ is the positive definite solution of the Riccati ⎢ 12 22 23 ⎥ ⎣⎢ p13 p23 p33 ⎦⎥ equation:
AT P + PA − r −1PBBT P + Q = 0
Therefore, the PID parameters are given by
p p p K = r −1 13 , K = r −1 23 , K = r −1 33 . i M p M d M
Based on the designed PID control, the stability of the overall system is analyzed as follows:
Consider the Lyapunov function VXPX= T . Thus, it follows that
VXABKPPABKX& =+TT[( ) ++ ( )]
T +−−− 2[XPBMx&&ddl Dx & F ].
Since xd is bounded image position, it may be constant or smooth function. Thus,
x&d ,&x&d are also bounded, i.e.,
|[PB− Mx&&dd−−≤ Dx & F ]| f B
with constant f B . Since the matrix A+BK is stable, there exists a Q>0 such that the Lyapunov equation
(A+BK)P+P(A+BK)=-Q.
Therefore, we have
2 VQXfX& ≤−λmin ()||||2 + B ||||
2 fB which is guaranteed negative as long as ||||X > , where λmin ()⋅ is the λmin ()Q smallest eigenvalue of matrix. This implies that X is uniformly ultimately bounded. An arbitrary small error of ||X || may be achieved by selecting large PID
73 gains. This implies that the tracking control will drive the actual position x → xd closely. This also implies that the temperature sensor will follow and sense the moving cutting tool.
4.4.5 Practical Issues
There are two main challenges with respect to milling tool tracking for temperature measurement on the fly due to the different dynamics in the three main components of the system; the camera, the thermometer and the servo system. Firstly, the requirement on motion tracking is high. The spot size of the thermometer is the same as the diameter of the tip of the milling tool. If the thermometer misses the target, the temperature will be a combination of the temperature of milling and ambient temperature, which can be expressed as
αβ TT=+milling TAm , α and β are the sizes of the spot on the milling tool. αβ++ αβ
This temperature measurement will not be accurate when the tracking of the tool is not done accurately. Secondly, to add to the first problem, the frequency of position updating from the camera (15 times per sec) is much slower than motor’s dynamics (1000 times per sec) which may lead to a lag effect.
These issues will impose a constraint on the velocity of the milling tool so that adequate tracking and thus temperature measurements can be obtained.
The infrared thermometer has a time constant of Tther = 0.1sec . Tracking time constant of the servo system is Tmot = 0.01sec . The camera’s response time which can also be regarded as image processing time Tcam = 0.08sec . Therefore, the total
response time is Ttol = 0.19sec .
74 First, we will work out the maximum speed permissible. The thermometer can measure the temperature of a spot of 9mm in diameter.
Fig. 4.13 Maximum speed permissible
Referring to Fig 4.13, when the tip of the milling tool moves away from point O, the servo system transporting the thermometer will experience a time delay equal to TTTmot++ cam ther before a the tracking movement is completed and a temperature measurement is obtained. Over this period, the milling tool should not move out of the laser spot (i.e., it should not move beyond Point 1). Thus, we can obtain the maximum speed of the milling tool.
4.5mm υmax '=≈ 23.7mm / sec TTTmot++ cam ther
Secondly, from vision tracking point of view, since our method is based on frame difference, at least 1 pixel must be generated in the difference image Dxyt(),, , which is depicted in Fig 4.14 (a). Suppose its moving speed is v and time interval between two frames is Δt , the moving distance between two consecutive frame is
1 thus vt×Δ . Based on vt×Δ ≥1, minimum speed is v = pixel. Suppose the min Δt
75 frequency is 10Hz, the minimum speed is 10 pixels per second. The actual speed can be obtained using the equation in camera calibration section. In ball-milling machine experiment, the minimum speed of the tool is 1.5mm/sec. There is also a maximum speed for the image processing algorithm to function adequately. The maximum allowable speed is reached when the object in the second frame
W detaches from that in the first frame, so that υ ' = . W is the width of the max Δt object appearing on the screen. In the target application, this maximum speed of the tool is 27mm/sec.
(a) (b)
Fig. 4.14 Calculation of minimum and maximum speed
76 In the milling machine experimented in the chapter, a typical feed speed is
5mm/s. The monitoring system is thus expected to be adequate to track the milling tool and yield temperature measurements.
4.4.6 Experimental Results
To illustrate the effectiveness of the proposed method, real-time experiments were conducted. The complete experiment was divided into three stages. In the first stage, a part-simulated experiment was conducted where we used a soldering iron (7cm in diameter at the tip) to model a milling tool. The temperature of the soldering iron can be manually changed to simulate different milling situations. In the second stage, real experiments were conducted on ball-milling and face- milling machines respectively. In the third stage, a thermal camera was used to test the accuracy of the above experimental data.
In all the experiments, the infrared thermometer is fixed on a linear motor
(Yaskawa SGML-01AF12) which can move at a maximum speed of 3000rpm.
The dSPACE control development and rapid prototyping system, in particular, the
DS1102 board, is used. dSPACE integrates the whole development cycle seamlessly into a single environment. MATLAB/Simulink can be directly used in the development of the dSPACE real-time control system.
We first tune the PID parameters of the servo system as KKpi==0.4, 0.001, Kd = 0.001 , so that the system can respond to the reference quickly. Fig 4.15 shows the control response for a given step, where the dotted line represents the set point, while the solid line represents the actual response. In
77 Fig 4.15, a steady-state error (<0.1mm) is observed, due to a small integral gain used for rapid response.
Fig. 4.15 Step response with PID control
For a series of image position information fed to the control system, the system response is shown in Fig 4.16 (a)) compares the position of the machine (green line) and response of the controller (dotted line). Fig 4.16 (b) shows the tracking error.
(a)
78
(b)
Fig. 4.16 Controller response and tracking error
The predictive model is used to estimate actual position. The delay time is chosen as 0.2sec which is enough to include the delay of the image processing.
Initially, the linear motor is positioning at 86mm and then follows the image position xd at time T=2.63 sec. The tracking error is within 10mm. After the initial phase, all the errors are within 8mm. Since the target area is a circle with diameter 8.9mm, the tracking error is acceptable and the sensor can measure the temperature accurately in this range.
The part-simulated experiment setup is shown in Fig 4.16. We gathered data from two separate experiments. One was done under normal condition, while the other one was done under abnormal condition. The environmental temperature was 22 ºC. Fig 4.17 (a) and (b) show the two sets of temperature measurements obtained respectively. It is observed that in experiment 1 the temperature measurement is within 110 to 145ºC. In experiment 2, there is an obvious temperature rise at time 40. The peak temperature reached is 170 ºC, triggering an alarm as the threshold of 150 ºC is set to detect abnormal situation. The part-
79 simulated experiment proves the efficiency of our monitoring system in real-time tracking and temperature reading.
Fig. 4.17 Simulation Scene
(a)
80
(b)
Fig. 4.18 Temperature measurement during simulation
(a)Temperature measurements under normal condition; (b)Temperature measurements under abnormal condition
In the second part, the monitoring system was implemented on a ball-milling and a face-milling machine. The ball-milling machine generates less heat than face- milling machine. For ball-milling, the temperature measurements are plotted in
Fig 4.18.
(a)
81
(b)
Fig. 4.19 Temperature measurement in real experiment
(a)Temperature measurements for ball-milling machine;(b)Temperature measurements for face-milling machine.
The environmental temperature for this experiment is 20 ºC. In Fig 4.18 (a), the average temperature for a ball-milling machine is around 33 ºC with a slight temperature rise over time, consistent with extended use of the tool, a phenomenon also observed in Fig 4.18 (b). In Fig 4.18 (b), there is a sudden temperature rise at time 5500 due to the cutter penetrating the work piece at this time as explained in Fig 4.19.
In the final stage of the experiments, we used a thermal camera to verify the accuracy of the temperature measurements thus obtained. Results will be shown for the face-milling machine. Four thermal images were recorded during the milling process. Temperature measurements were listed for each tested spot.
82
Fig. 4.20 Explanation of sudden temperature raise
(a)Milling Processing: Phase 1
SP01: 115.0°C; SP02: 37.8°C; SP03:69°C; SP04: 127.6°C
(b)Milling Processing: Phase 2
SP01: 169.4°C; SP02: 70.8°C; SP03:121.0°C; SP04: 58.9°C; SP05: 78.1°C;
83
(c)Milling Processing: Phase 3
SP01: 115.0°C; SP02:106.6°C; SP03:60.3°C; SP04: 39.2°C;
(d)Milling Processing: Phase 4
SP01: 87.2°C; SP02:76.6°C; SP03:54.4°C; SP04: 41.2°C; SP05: 35.2°C;
Fig. 4.21 Accuracy testing using thermal camera
In the second stage of the experiments, when we used thermometer to measure the temperature of the face-milling machine, we aimed and track spot 2 in Fig 4.20 (d). Comparing with the results from the thermal camera, the temperature of the spot was kept within 60 ºC to 76 ºC during milling (the same spot as spot 3 in (a), spot 2 in (b), spot 3 in (c) in Fig 4.20 which were in accordance with the data furnished in Fig 4.18(b).
84 4.4.7 Conclusions
In this work, a decoupled tracking and thermal monitoring system is presented which can be used on non-stationary targets of closed systems such as machine tools.
The most typical machine surveillance scheme was done by using sensor monitoring, data analyzing and modeling. However, the sensor set has obvious shortcomings such low sampling rate or power constraint. Thermal camera was also used in such applications. Apart from its high price, most of the thermal camera can only save the highest temperature values inside the testing scope unless other software provided. In CNC machine surveillance, the highest temperature is usually generated by metal debris, which is not the point of interest, instead of the milling tool. Manual work is needed to find out the temperature of the milling tool through a look up table provided by the manufacturer (as shown in Fig. 15). In view of this fact, thermal camera is not the optimized solution since our objective is to real-time monitor the temperature of the working tool. In contrast, with the help of vision-based tracking algorithm, the thermometer carried by the controller can follow the moving tool and read its temperature value which overcomes the above mentioned inconveniencies. And the temperature values taken by thermometer (fig. 13) are equal to those taken by thermal camera (fig.
15).
There are three main contributions from the work. First a vision component is developed to track moving targets being monitored. Image processing techniques are used to compute the target location to be tracked. Thus, the system is decoupled and applicable to closed-systems without needing to integrate to them electronically. Secondly, an infrared temperature sensor with built-in laser is
85 deployed for non-contact temperature measurement of the moving target. Thirdly, a predictive motion control system holds the thermal sensor and follows efficiently the moving target to enable continuous temperature measurement and monitoring. Real experimental results are furnished to demonstrate and highlight the key contributions from the work.
86 Conclusions
5.1 Summary of Contributions
Computer imaging technology has been growing rapidly in the recent decades, which has come to pervade every aspect of our daily lives. It has not only significantly improved medical diagnosis but also revolutionized the manufacturing inspection process. It has redefined the concept of surveillance and design of human-machine interaction. With the current development of wireless communication and control technologies, more advanced intelligent applications will be developed based on computer imaging technology. This thesis made use of different image processing and vision technology to develop innovative applications in three different arenas: mobile device application, bio-manipulation system and industrial surveillance. Each of these applications has different challenges: challenges for text extraction and translation using mobile and digital devices are the uncertain background, uneven illumination, reflection and different lighting conditions; challenges for vision-based bio-manipulation system are the fragile and slippery nature of the cell as well as the requirement for high operation accuracy; challenges for vision-based CNC machine surveillance are the requirement of precise tracking and noise due to hash working environment. In this thesis, different strategies were used to meet all the challenges.
First, in mobile device application, a color and edge information fusion method has been developed. The whole process consists of gray scale
87 transformation, quantization, background extraction and noise elimination.
Experimental results show that the fusion method is more efficient and robust than those methods which were simply based on color or edge features. Edge-based algorithm has advantage in objects segmentation. However, a small gap in edge detection may lead to an error in region segmentation by combining two regions together. Such problem can be easily solved by edge and color fusion method. In chapter 2, we firstly quantize an image into different gray scale levels.
Background is identified as the region with the largest area. Objects are extracted by comparing their colors with that of the background.
Secondly, in bio-manipulation system, an object recognition and tracking algorithm is developed to realize automatic cell injection task. The proposed software has unique advantage in guiding the pipette to precisely aim at the center of the cell. The speed of penetration can be set by the controller in order not to destroy the cell membrane. Experimental results show that the vision-based bio- manipulation system is more effective and achieves higher success rate than manual operations.
Thirdly, in CNC machine surveillance, a moving object tracking algorithm is developed to guide a thermometer to monitor the temperature of the working tool. Moving edge is generated in the algorithm to reduce noise from harsh environment. A predictive motion control model is developed to estimate the actual position to meet the demand of high precision tracking.
The procedures concerned with image processing and vision system design are presented thoroughly in this thesis. The proposed algorithms for each application are different to cater to different challenges and requirements. From a design point of view, this thesis presents novel design of vision-control integrated
88 systems. Vision-based bio-manipulation system in Chapter 3 and vision-based industrial monitoring system in Chapter 4 are two examples of such design. The motivation of this thesis is to develop commercial applications for different arenas using image and vision technologies. Text extraction and translation software was designed for travelers who do not know foreign language. Vision-based bio- manipulation system was designed to replace manual work and achieve higher success rate. Vision-based industrial surveillance system was designed to realize real-time monitoring and reduce the cost.
5.2 Suggestions for future work
This thesis has presented the research works on the use of image processing and vision system in various fields. Future research topics in this area are suggested as follows:
Application of image processing techniques in gastrointestinal diagnosis
The field of gastrointestinal (GI) diagnosis has been redefined by PillCam technology. PillCam video capsule is a disposable, miniature video camera contained in a capsule that can be swallowed by the patient. The capsule transmits high quality color images of the GI tract that enable physicians to visualize the small intestine and esophagus and evaluate the possible problem in gastrointestinal. Advanced image processing techniques can be developed to get a better view of the stomach or intestine. The images will be taken under poor lighting conditions. The quality of the image may be deteriorated by the succus entericus inside the digestive tract. An image enhancement and recognition
89 algorithm is needed to highlight the details of those images in order to find out ulcers and tumors.
Vision-based intelligent traffic control system
In recent few years, vision-based traffic surveillance systems were implemented in several cities to capture red light runners. The scheme was automatic but not intelligent. A group of pressure sensors were buried beneath the ground. Upon the start of the red light, the pressure sensors are initiated. Whenever a car runs through the red light at this point in time, the sensor would be activated, which would trigger the camera to capture the situation at the junction.
Modern intelligent traffic surveillance systems, which equipped with computer vision and image processing software, can detect, localize and track vehicles in video sequences captured by road cameras with little or no human intervention, as well as further analyze the traffic accidents. This can facilitate daily traffic management and allow an immediate response when abnormal events occur, consequently providing a more advanced and feasible surveillance scheme.
Based on the above strategy, a similar surveillance system can be implemented on the street to monitor the traffic flow. Moving object detection algorithm can be developed to count the number of cars going through the street.
This information may be sent back to an information center. The center may adjust the time duration of the nearby traffic lights in an intelligent way so as to clear the traffic jam.
Vision-based intelligent ocean monitoring system
Conventionally, satellite and helicopter were commonly used in ocean monitoring, which were expensive and cannot obtain the detailed information of the pollution. Vision-based wireless sensor network can be developed and
90 deployed in the coastal region to monitor the environment in a real-time manner.
Such device can be dived into the ocean to monitor the situation of the ocean surface. The key part of this device is the algorithm in the image processor. If waste was dumped on the surface of the ocean (e.g, a bottle), the algorithm shall be able to detect and localize it. If oil was spilled on the ocean, the transparency of the water would be quite different since the oil blocks the sunshine.
91 Author’s Publications
Journal Publications:
Y. Zhang and K. K. Tan, “Text extraction from images captured via mobile and digital devices”, International Journal of Computational Vision and Robotics,
010102.
Y. Zhang, K. K. Tan, S. Huang, "Vision-Servo System for Automated Cell
Injection," IEEE Trans. on Industrial Electronics, vol. 56, no. 1, pp. 231-238, Jan
2009.
K. K. Tan, S.N. Huang, Y. Zhang, “Distributed Fault Detection System Based on
Sensor Wireless Network”, Journal of Computer Standards and Interfaces, Vol.
31, Issue 3, March 2009, Pages 573-578.
Y. Zhang, S. Huang, K. K. Tan, “Decoupled Tracking and Thermal Monitoring of Non-Stationary Targets”, ISA Transactions, accepted, 2009.
Conference Publications:
Y. Zhang and K. K. Tan, “Text Extraction from Images Captured via Mobile and
Digital Devices,” Fifth International Conference on Industrial Automation,
Montreal, TSTI-07, 2007.
Zhang, Y.; Tan, K.K.; Huang, S., “Software Based Vision System for Automated
Cell Injection”, BioMedical Engineering and Informatics, 2008. BMEI 2008.
International Conference on Volume 1, Issue, 27-30 May 2008 Page(s):718 – 722.
Y. Zhang, S.N. Huang and K. K. Tan, “Vision-assisted thermal monitoring
92 system for CNC machine surveillance”, Proceedings of the IEEE International
Conference on Automation and Logistics, ICAL, Qingdao, China, Best Student
Paper Award, September 2008.
93 Bibliography
[1] A. Elgammal, D. Harwood and L. Davis, “Non-Parametric model for
background subtraction”, European Conference on Computer Vision, pages
751-767, 2000.
[2] A. J. Lipton et al., “Moving target classification and tracking from real-time
video,” in Proc. 4th IEEE WACV, 1998, pp. 8–14.
[3] A. Kumar, “Computer-vision-based fabric defect detection: A survey,” IEEE
Trans. Ind. Electron., vol. 55, no. 1, pp. 348–363, Jan. 2008.
[4] Alan and M. McIvor, “Background Subtraction Techniques”, IVCNZ00,
Hamilton, New Zealand, Nov. 2000.
[5] A. Noori-khajavi, R. Komanduri, On multisensor approach to drill wear
monitoring, Ann.CIRP 42 (1993) 71–74.
[6] A. Prateepasen, Y.H.J. Au, B.E. Jones, Acoustic emission and vibration for
tool wear monitoring in single-point machining using belief network, IEEE
Instrumentation and Measurement Technology Conference, Budapest,
Hungary, 2001, pp. 1541–1546.
[7] A. S. Putra et al., “Design, modeling, and control of piezoelectric actuators
for intracytoplasmic sperm injection,” IEEE Trans. Control Syst. Technol.,
vol. 15, no. 5, pp. 879–890, Sep. 2007.
[8] Atlas, L.E.; Bernard, G.D.; Narayanan, S.B., “Applications of time-frequency
analysis to signals from manufacturing and machine monitoring sensors”,
Proceedings of the IEEE Volume 84, Issue 9, Sep 1996 Page(s):1319 – 1329
[9] Barron J, Fleet D and Beauchemin S, “Performance of optical flow
techniques”, International Journal of Computer Vision, 1994, pp.42-77
94 [10] B.-J. You, Y. S. Oh, and Z. Bien, “A vision system for an automatic assembly
machine of electronic components,” IEEE Trans. Ind. Electron., vol. 37, no. 5,
pp. 349–357, Oct. 1990.
[11] C.-S. Cho, B.-M. Chung, and M.-J. Park, “Development of real-time vision-
based fabric inspection system,” IEEE Trans. Ind. Electron., vol. 52, no. 4, pp.
1073–1079, Aug. 2005.
[12] C.-L. Hwang, C. Jan, and Y.-H. Chen, “Piezomechanics using intelligent
variable-structure control,” IEEE Trans. Ind. Electron., vol. 48, no. 1, pp. 47–
59, Feb. 2001.
[13] C.-C. Kau, K. W. Olson, E. A. Ribble, and C. A. Klein, “Design and
implementation of a vision processing system for a walking machine,” IEEE
Trans. Ind. Electron., vol. 36, no. 1, pp. 25–33, Feb. 1989.
[14] C.-H. Ku and W.-H. Tsai, “Obstacle avoidance in person following for
vision-based autonomous land vehicle guidance using vehicle location
estimation and quadratic pattern classifier,” IEEE Trans. Ind. Electron., vol.
48, no. 1, pp. 205–215, Feb. 2001.
[15] C. Stiller and J. Konrad, “Estimating motion in image sequences,” IEEE
Signal Process. Mag., vol. 16, no. 4, pp. 70–91, Jul. 1999.
[16] D.-H. Kim et al., “Cellular force measurement for force reflected
biomanipulation,” in Proc. IEEE Int. Conf. Robot. Autom., New Orleans, LA,
Apr. 2004, pp. 2412–2417.
[17] D. Karatzasa and A. Antonacopoulos, "Colour text segmentation in web
images based on human perception ", Image and Vision Computing. Volume
25, Issue 5, 1 May 2007, Pages 564-577
95 [18] Eduardo Gilabert and Aitor Arnaiz, “Intelligent automation systems for
predictive maintenance: A case study”, Int. J. Robotics Comput. Intergrated
Manuf. 22 (2006) 543-549
[19] E. Peli, “Feature detection algorithm based on visual system models”, in Proc.
IEEE, vol. 90, no. 1, Jan. 2002, pp. 78-93
[20] Frize, M., Herry, C. and Scales, N., “Processing thermal images to detect
breast cancer and assess pain”, Information Technology Applications in
Biomedicine, 2003. 4th International IEEE EMBS Special Topic Conference
on 24-26 April 2003 Page(s):234 – 237
[21] G. Byrne, D. Dornfled, I. Inasaki, G. Ketteler, W. Konig, R. Teti, Tool
condition monitoring (TCM)—the statue of research and industrial
application, Ann. CIRP 44 (1995) 541–567.
[22] H. Golnabi and A. Asadpour, “Design and application of industrial machine
vision system”, Int. J. Robotics Comput. Intergrated Manuf. 23 (2007) 630-
637
[23] Herry, C.L., Frize, M., Goubran, R.A., “Segmentation and Landmark
Identification in Infrared Images of the Human Body”, Engineering in
Medicine and Biology Society, 2006. EMBS '06. 28th Annual International
Conference of the IEEE, Aug. 2006 Page(s):957 – 960
[24] Hiroki Takahashi and Masayuki Nakajima, “Region Graph based Text
Extraction from outdoor images”, ICITA’05
[25] H.-J. Shieh, F.-J. Lin, P.-K. Huang, and L.-T. Teng, “Adaptive displacement
control with hysteresis modeling for piezoactuated positioning mechanism,”
IEEE Trans. Ind. Electron., vol. 53, no. 3, pp. 905–914, Jun. 2006.
96 [26] Hong-Song and Feng Shi, “A Real-Time Algorithm for Moving Objects
Detection in Video Images”, Proceedings of the 5th World Congress on
Intelligent Control and Automation, June 15-19, 2004, Hangzhou, P. R. China.
[27] Hong-Wen Lin, Shao-Qing Yang, Zhi-Jun Xia and Chun-Yu Kang, “A
Moving Objects Detection Approach for Smart Sensor”, Proceedings of the
Fifth International Conference on Machine Learning and Cybernetics, Dalian,
13-16 August 2006
[28] H. Qi, S.S. Iyengarb, K. Chakrabarty, Distributed sensor networks—a review
of recent research, J. Franklin Inst. 338 (2001) 655–668.
[29] I.F. Akyildiz, W. Su, Y. Sankarasubramaniam, and E. Cayirci, A survey on
sensor networks, IEEE Communications Magazine, vol.40, no.8, pp102–114,
2002.
[30] Ismail Haritaoglu, “Scene Text Extraction and Translation for Handheld
Devices”, Computer Vision and Pattern Recognition, 2001. CVPR 2001.
Proceedings of the 2001 IEEE Computer Society Conference on
Volume 2, 2001 Page(s):II-408 - II-413 vol.2
[31] Isermann, R. “Process fault detection based on modeling and estimation
methods-a survey”, Automatica, vol.20, pp387-404, 1984
[32] Isermann, R., M. Ayoubi, H. Konrad, T. Rei, β, Model based detection of tool
wear and breakage for machine tools, International Conference on Systems,
Man Cybernetics, Le Touquet 3 (1993) 72–77.
[33] Ishino R., “Detection of a faulty power distribution apparatus by using
thermal images”, Power Engineering Society Winter Meeting, 2002. IEEE,
Volume 2, 27-31 Jan. 2002 Page(s):1332 - 1337 vol.2
97 [34] Jaehwa Park and Venu Govindaraju, "OCR in a Hierarchical Feature Space",
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE
INTELLIGENCE, VOL. 22, NO. 4, APRIL 2000
[35] Jaeguyn Lim, Jonghyun Park and Gérard G. Medioni, "Text segmentation in
color images using tensor voting", Image and Vision Computing. Volume 25 ,
Issue 5 (May 2007) table of contents, Pages 671-685, 2007, ISSN:0262-8856
[36] Jardim-Goncalves, R. Martins-Barata, M.; Alvaro Assis-Lopes, J.; Steiger-
Garcao, A., “Application of stochastic modelling to support predictive
maintenance for industrial environments”, Systems, Man, and Cybernetics,
1996., IEEE International Conference on Volume 1, Issue , 14-17 Oct 1996
Page(s):117 - 122 vol.1
[37] J. Canny, “A Computational approach to edge detection”, IEEE Trans. on
Pattern Recognition vol. 17, no. 12, 1995
[38] JiSoo Kim, SangCheol Park and SooHyung Kim, “Text Location from
Natural Scene Images Using Image Intensities”, Proceedings of the Third
International Conference on Information Technology and Applications
(ICDAR’05)
[39] J. M. S. Prewitt. Object enhancement and extraction. In A. Rosenfeld and B.
S. Lipkin, editors, Picture Processing and Psychophysics, pages 75-149.
Academic Press, New York,1970.
[40] J. Kofman, X. Wu, T. J. Luu, and S. Verma, “Teleoperation of a robot
manipulator using a vision-based human–robot interface,” IEEE Trans. Ind.
Electron., vol. 52, no. 5, pp. 1206–1219, Oct. 2005.
98 [41] J. Ping, H.-T. Chen, Y.-J. Wang, and J. Lin, “A decomposed control scheme
for vision-guided manipulators curve tracking,” IEEE Trans. Ind. Electron.,
vol. 46, no. 3, pp. 667–669, Jun. 1999.
[42] Jun Sun, Hao Yu and Yukata Katsuyanma, “Effective text Extraction and
Recognition for WWW Images”, Fujitsu R&D Center, 2003
[43] J. Wunnenberg and P. M. Frank, “Dynamic model based incipient fault
detection concept for robotics”, in Proc. IFAC 11th Triennial World Congr.,
Tallinn, Estonia, pp61-66, 1990
[44] J. Park, A. Ulsoy, On-line tool wear estimation using force measurement and
a nonlinear observer, ASME J. Dynamic Sys., Meas., Control 14 (1992) 666–
672.
[45] Kantip Kiratirataqnapruk, Premnath Dubey and Supakorn Siddhichai, “A
gradient-based foreground detection technique for object tracking in a traffic
monitoring system”, Advanced Video and Signal Based Surveillance, 2005.
IEEE Conference on Volume, Issue, 15-16 Sept. 2005 Page(s):377-381
[46] K. Danai, A.G. Ulsoy, An adaptive observer for on-line tool wear estimation
in turning, Part I: Theory, Mec. Syst. Signal Proc. 1 (1987) 211–225.
[47] K. Danai, A.G. Ulsoy, An adaptive observer for on-line tool wear estimation
in turning, Part II: Results, Mec. Syst. Signal Proc 1 (1987) 227–240.
[48] K.L. Moore, Y.Q. Chen, Model-based approach to characterization of
diffusion processes via distributed control of actuated sensor networks, The
1st IFAC Symposium Applications in Automation and Robotics, Finland,
2004.
[49] Kuo-Chao Lin, Wen-Liang Chen, Shing-Chia Chen and Fu-Sung Wang,
“Diagnostic technique for classifying the quality of circuit boards using
99 infrared thermal image”, IFSA World Congress and 20th NAFIPS
International Conference, 2001. Joint 9th, Volume 1, 25-28 July 2001
Page(s):464 - 469 vol.1
[50] L. G. Roberts. Machine perception of three-dimensional solids. In J. T. Tippet
et al., editor, Optical and Electro-Optical Information Processing, pages 159-
197. MIT Press, Cambridge, Massachusetts, 1965.
[51] L. Mattos et al., “New developments towards automated blastocyst
microinjection,” in Proc. IEEE Int. Conf. Robot. Autom., Roma, Italy, Apr.
10–14, 2007, pp. 1924–1929.
[52] Larry P. Heck, “Signal Processing Research in Automatic Tool Wear
Monitoring”, Acoustic, Speech and Signal Processing, 1993. ICASSP-93.,
1993 IEEE International Conference on Vol. 1, Issue, 27-30 Apr 1993
Page(s):55-58 vol. 1
[53] Leykin, A. and Hammoud, R., “Robust Multi-Pedestrian Tracking in
Thermal-Visible Surveillance Videos”, Computer Vision and Pattern
Recognition Workshop, 2006 Conference on 17-22 June 2006 Page(s):136 –
136
[54] L.K. Lauderbaugh, A.G. Ulsoy, Model reference adaptive force control in
milling, ASME J. Eng. Industry 3 (1989) 13–21.
[55] Marc-Peter Schambach, "A New View of the Output from Word
Recognition", Proceedings of the 9th Int’l Workshop on Frontiers in
Handwriting Recognition (IWFHR-9 2004)
[56] Michael D. Garris, Charles L. Wilson,, "Neural Network-Based Systems for
Handprint OCR Applications", IEEE TRANSACTIONS ON IMAGE
PROCESSING, VOL. 7, NO. 8, AUGUST 1998
100 [57] Mingying Zhao, Jun Zhao, Shuguang Zhao and Yuan Wang, “A Novel
Method for Moving Object Detection in Intelligent Video Surveillance
Systems”, International Conference on Computational Intelligence and
Security, 2006. Volume: 2, On page(s): 1797-1800, Guangzhou
[58] M. L. Visinsky, J. R. Cavallaro, and I. D. Walker, ``Expert system framework
for fault detection and fault tolerance in robotics," Computers and Electr.
Eng., vol.20,pp421-435,1994
[59] M. Piccardi, “Background subtraction techniques: a review”, Proc. of IEEE
SMC 2004 International Conference on Systems, Man and Cybernetics, 2004
[60] M.Weck, Machine diagnostics in automated production, J. Manufact. Syst. 2
(1983) 101–106.
[61] N. Bonnail, D. Tonneau, F. Jandard, G.-A. Capolina, and H. Dallaporta,
“Variable structure control of a piezoelectric actuator for a scanning tunneling
microscope,” IEEE Trans. Ind. Electron., vol. 51, no. 2, pp. 354–363, Apr.
2004.
[62] P. V. C. Hough, “Machine analysis of bubble chamber pictures,” in Proc. Int.
Conf. High Energy Accelerators Instrum., 1959, pp. 554–556.
[63] P. Renton, P. Bender, S. VelDhuis, D. Renton and M. A. Elbestawi, “Internet-
Based Manufacturing Process Optimization and Monitoring System”, Proc. of
the 2002 IEEE Int. Conf. on Robotics & Automation Washington DC. May
2002
[64] P. Rosin and T. Ellis, “Image difference threshold strategies and shadow
detection”, Proc. British Machine Vision Conference, page 347-356, 1995.
[65] Rafael C Gonzalez & Woods. Digital Image Processing, 2nd ed. Englewood
Cliffs, NJ:Prentice-Hall, 2001. Chapter 6, Chapter 10
101 [66] R. Canals et al., “A biprocessor-oriented vision-based target tracking system,”
IEEE Trans. Ind. Electron., vol. 49, no. 2, pp. 500–506, Apr. 2002.
[67] R.C. Luo, M.G. Kay, Multi-sensor integration and fusion in intelligent
systems, IEEE Trans. Systems Man Cybernet. 19 (1989) 901–931.
[68] Reinhard Klette and Piero Zamperoni, “HANDBOOK OF IMAGE
PROCESSING OPERATORS”, JOHN WILEY & SONS, 1996, Page 38
[69] R.E.Kalman, “A new approach to linear filtering and prediction problems”,
ASME Journal of Basic Engineering, Series 82D, pp.35-45, 1960
[70] Rishi R. Rakesh, Probal Chaudhuri, and C.A. Murthy, “Thresholding in Edge
Detection: A statistical Approach”, IEEE TRANSACTIONS ON IMAGE
PROCESSING, VOL. 13, NO.7, JULY 2004
[71] R.-J.Wai and C.-H. Tu, “Adaptive grey control for hybrid resonant driving
linear piezoelectric ceramic motor,” IEEE Trans. Ind. Electron., vol. 53, no. 2,
pp. 640–656, Apr. 2006.
[72] S.A. Coker, Y.C. Shin, In-process control of surface roughness with tool wear
via ultrasonic sensing, Proc. of American Control Conference, Seattle, 1995.
[73] Salim Djeziri, Fathallah Nouboud, and R´ejean Plamondon, "Extraction of
Signatures from Check Background Based on a Filiformity Criterion", IEEE
TRANSACTIONS ON IMAGE PROCESSING, VOL. 7, NO. 10,
OCTOBER 1998
[74] Shunji Mori, Hirobumi Nishida and Hiromitsu Yamada, Optical Character
Recognition, John Wiley & Sons, NIC.1999, Page(s) 2-39
[75] S. N. Huang, K. K. Tan, Y. S. Wong, C. W. de Silva, H. L. Goh and W. W.
Tan, “Tool wear detection and fault diagnosis based on cutting force
102 monitoring”, International Journal of Machine Tools and Manufacture, vol.47,
pp444-451, 2007
[76] S. Tsugawa, “Vision-based vehicles in Japan: Machine vision systems and
driving control systems,” IEEE Trans. Ind. Electron., vol. 41, no. 4, pp. 398–
405, Aug. 1995.
[77] S. Rangwala, S. Liang, D. Dornfeld, Pattern recognition of acoustic emission
signals during punch stretching, Mech. Syst. Signal Process. 1 (1987) 321–
332.
[78] Stegemann, D.; Reimche, W.; Sudmersen, U.; Pietsch, O.; Liu, Y.,
“Monitoring and vibrational diagnostic of rotating machinery in power
plants”, Power Station Maintenance - Profitability Through Reliability, 1998.
First IEE/IMechE Int. Conf. on (Conf. Publ. No. 452) Volume , Issue , 30
Mar-1 Apr1998 Page(s):39 – 44
[79] S.-Y. Cho and J.-H. Shim, “A new micro biological cell injection system,” in
Proc. IEEE/RSJ Int. Conf. Intel. Robots Syst., Sendai, Japan, Sep. 28–Oct. 2,
2004, pp. 1642–1647.
[80] T. Abdelzaher, J. Stankovic, S. Son, B. Blum, T. He, A.Wood, C. Lu, A
communication architecture and programming abstractions for real-time
embedded sensor networks, Proceedings of the 23rd International Conference
on Distributed Computing Systems Workshops, 2003, pp. 220–225.
[81] Tan, KK.,Lee, T.H. and Huang,S.N. Precision motion control, 2nd Edition,
Springer, 2008.
[82] T. Bucher et al., “Image processing and behavior planning for intelligent
vehicles,” IEEE Trans. Ind. Electron., vol. 50, no. 1, pp. 62–75, Feb. 2003.
103 [83] T. Horprasert, D. Harwood and L. S. Davis, “A Statistical Approach for
Real-Time Robust Background Subtraction and Shadow Detection”, Proc.
IEEE ICCV’99 FRAME-RATE Workshop, Kerkyra, Greece, Sept. 1999
[84] T.-L. Liu and H.-T. Chen, “Real-time tracking using trust-region methods,”
IEEE Trans. Pattern Anal.Mach. Intell., vol. 26, no. 3, pp. 397– 402, Mar.
2004.
[85] T.Y. Kim, J. Kim, Adaptive cutting force control for a machining center by
using indirect cutting force measurements, Int. J. Mach. Tools Manufact. 36
(1996) 925–937.
[86] T. Zhang and D. Freedman, “Tracking objects using density matching and
shape priors,” in Proc. 9th IEEE ICCV, 2003, vol. 2, pp. 1056–1062.
[87] Ui-Pil Chong, Sung-Sang Lee and Chang-Ho Sohn, “Fault Diagnosis of the
Machines in Power Plants Using LPC”, Science and Technology, 2004.
KORUS 2004. Proceedings. The 8th Russian-Korean Int. Symposium on
Publication Date: 26 June-3 July 2004 Volume: 1, On page(s): 170- 174 vol.
1
[88] V.A. Shapiro and P.K. Veleva, “An Adaptive Method for Image
Thresholding”, IEEE Comput. Soc. Conf. on Pattern Recognition and Image
Process, 1992
[89] Xiangyun Ye, Cheriet and Chinh Y.Suen, “Stroke-Model-Based Character
Extraction from Gray-Level Document Images”, IEEE Trans. on Image
Processing vol. 10, no. 8 AUGUST 2001
[90] Xiaoheng Yang, Hiroki Takahashi and Masayuki Nakajima, “Investigation of
Robust Color Model for Edge Detection on Text Extraction from Scenery
104 Images”, 2004 IEEE Region 10 Conference Vol. B, 21-24 Nov. 2004
Page(s):85 - 88 Vol. 2
[91] X. Li, A. Djordjevich, P.K. Venuvinod, Current-sensor-based feed cutting
force intelligent estimation and tool wear condition monitoring, IEEE Trans.
Industrial Electron. 47 (2000) 697–702.
[92] X. Li, H.X. Li, X.P. Guan, R. Du, Fuzzy estimation of feed-cutting force from
current measurement—a case study on intelligent tool wear condition
monitoring, IEEE Trans. SMC (part C) 34 (2004) 506–512.
[93] Y. Altintas, Prediction of cutting forces and tool breakage in milling from
feed drive current measurements, ASME J. Eng. Industry 114 (1992) 386–
392.
[94] Y.-F. Peng, R.-J.Wai, and C.-M. Lin, “Implementation of LLCC-resonant
driving circuit and adaptive CMAC neural network control for linear
piezoelectric ceramic motor,” IEEE Trans. Ind. Electron., vol. 51, no. 1, pp.
35–48, Feb. 2004.
[95] Y. Sun and B. J. Nelson, “Biological cell injection using an autonomous
microrobotic system,” Int. J. Robot. Res., vol. 21, no. 10/11, pp. 861–868,
Oct./Nov. 2002.
[96] Y. Zhang and K. K. Tan, “Text extraction from images captured via mobile
and digital devices,” in Proc. 5th Int. Conf. Ind. Autom., Montreal, QC,
Canada, Jun. 11–13, 2007. TITS-07-P03.
[97] ZHAN Chaohui, DUAN Xiaoli, XU Shuoyu, SONG Zheng and LUO Min,
“An Improved Moving Object Detection Algorithm Based on Frame
Difference and Edge Detection”, Fouth International Conference on Image
and Graphics, 2007
105 [98] Z.-F. Yang and W.-H. Tsai, “Viewing corridors as right parallelepipeds for
vision-based vehicle localization,” IEEE Trans. Ind. Electron., vol. 46, no. 3,
pp. 653–661, Jun. 1999.
[99] Z.Wang, B.-G. Hu, L. C. Liang, and Q. Ji, “Cell detection and tracking for
micromanipulation vision system of cell-operation robot,” in Proc. IEEE Int.
Conf. Syst., Man, Cybern., Nashville, TN, Oct. 2000, pp. 1592–1597.
106