Image manipulation Techniques
You ask what is the use of classification, arrangement, systematization. I answer you; order and simplification are the first steps toward the mastery of a subject the actual enemy is the unknown
Allan Sandage Astronomer Intro
• Image processing is essential to mine information from data.
• When dealing with large data sets it is important to have preprocessing algorithms that can reduce the dimensionality of the data space by projecting the data onto adequate subspaces. When we operate on images, this corresponds to the steps of image enhancement and feature extraction.
• Image analysis is quite image specific. Different images have different properties, sizes, resolutions etc and require different methods to extract the optimum amount of data.
• The problem with higher level operations is noise. If we were to look for an object of a particular shape in the image, the object may or may not be a perfectly formed shape. How perfect the shape is has a huge impact on the kind of methods used and its accuracy.
• Another topic will be to deal with missing information. This is because objects in the image may be only partial, fragmented or overlapped with other objects. Image manipulation Techniques
•Image Analysis Concepts ➢Punctual Operators ➢Filters ➢Transforms ➢Morphology •Feature Detection ➢Topology Image Analysis Intensity Histogram - the image intensity distribution
Classification - mapping from pixel values to feature or object classes
Connected Components Labeling - grouping pixels with the same class label into regions Intensity Histogram
Histograms have many uses. One of the more common is to decide what value of threshold to use when converting a grayscale image to a binary one by thresholding. If the image is suitable for thresholding then the histogram will be bi-modal --- i.e. the pixel intensities will be clustered around two well- separated values. A suitable threshold for separating these two groups will be found somewhere in between the two peaks in the histogram. If the distribution is not like this then it is unlikely that a good segmentation can be produced by thresholding. Histogram Method
Broad band average image. Vertical velocity average image.. Average image in Mg wing
In order to identify active regions in MgIb1 images, we adopted the following procedure. Firstly we studied the features of the distribution function of the intensities. The distribution has a bimodal character that can be easily obtained as a superposition of two gaussian distributions. This is an evidence of two distinct populations that can be associated with quiet and active regions.
On the basis of this bimodal character we define a optimum threshold Ith as the intensity value corresponding to the intersection of the two gaussian curves: 2 μ1+μ2 σ p2 T= + ln 2 μ − μ ( p ) 1 2 1 Distribution of MgIb1 intensities. Image of selected active regions.
Vertical velocity distributions associated to the selected active regions. Negative values are up-flows, while positives are down-flows. Classification
Image classification analyzes the numerical properties of various image features and organizes data into categories. Classification algorithms typically employ two phases of processing: training and testing. In the initial training phase, characteristic properties of typical image features are isolated and, We can assign any given based on these, a unique description pattern to the class of its of each classification category, i.e. training class, is created. closest prototype. In the subsequent testing phase, If Euclidean distance is these feature-space partitions are our measure of proximity, used to classify image features. then the distance to the prototype j is given by Distinguish two different types of objects based upon a set of two attribute classes (e.g. length along the major axis and diameter) Connected Components Labeling
Connected components labeling scans an image and groups its pixels into components based on pixel connectivity, i.e. all pixels in a connected component share similar pixel intensity values and are in some way connected with each other.
The labeling of p occurs as follows:
• If all four neighbors are 0, assign a new label to p, else
• if only one neighbor has V={1}, assign its label to p, else
• if one or more of the neighbors have V={1}, assign one of the labels to p and make a note of the equivalences.. Thresholding - select pixels with Punctual Operator given values to produce binary image
Adaptive Thresholding - like Thresholding except choose values locally
Contrast Stretching - spreading out graylevel distribution
Histogram Equalization - general method of modifying intensity distribution
Logarithm Operator - reducing contrast of brighter regions
Exponential/`Raise to Power' Operator - enhancing contrast of brighter regions • To create a TWO-LEVEL version of the original image: – Pixel (intensity) • Single Threshold
• Adaptive Threshold
1if Pxy ≥ T (x,y,Sxy ) Pxy {0 if P xy ≤ T (x,y,Sxy )}
T (x,y,Sxy )= ⟨ I (x,y) ⟩ +ξ⋅ rms(x,y) Contrast Stretching
The simplest sort of normalization then scans the image to find the lowest and highest pixel values currently present in the image. Call these c and d. Then each pixel P is scaled using the following function:
Values below 0 are set to 0 and values about 255 are set to 255. Contrast Stretching
The simplest sort of normalization then scans the image to find the lowest and highest pixel values currently present in the image. Call these c and d. Then each pixel P is scaled using the following function:
Values below 0 are set to 0 and values about 255 are set to 255.
We can enhance the picture by ignoring all pixel values outside the 1% and 99% percentiles, and only applying contrast stretching to those pixels in between. The outliers are simply forced to either 0 or 255 depending upon which side of the range they lie on. Histogram Equalization
Histogram modeling techniques provide a sophisticated method for modifying the dynamic range and contrast of an image by altering that image such that its intensity histogram has a desired shape.
Unlike contrast stretching, histogram modeling operators may employ non-linear and non- monotonic transfer functions to map between pixel intensity values in the input and output images.
This technique is used in image comparison processes (because it is effective in detail enhancement) and in the correction of non-linear effects introduced by a digitizer or display system. Logarithm Operator
The dynamic range of an image can be compressed by replacing each pixel value with its logarithm. This has the effect that low intensity pixel values are enhanced. Applying a pixel logarithm operator to an image can be useful in applications where the dynamic range may too large to be displayed on a screen
Logarithmic mapping functions at two different scales. Exponential/`Raise to Power' Operator
The exponential and `raise to power' operators are two anamorphosis operators which can be applied to grayscale images. Like the logarithmic transform, they are used to change the dynamic range of an image. However, in contrast to the logarithmic operator, they enhance high intensity pixel values.
Contrast in the high pixel values may be increased at the cost of the contrast in the low pixel values
Mapping function of `raise to power' operator for r=0.5 (dashed line), r = 2 (solid line) and r=6 (dotted line) Mean Filter - noise reduction NR Filters using mean of neighborhood
Median Filter - NR using median of neighborhood
Gaussian Smoothing - NR using convolution with a Gaussian smoothing kernel
Conservative Smoothing - NR using maximum and minimum of neighborhood
Crimmins Speckle Removal - more complex NR operator
Frequency Filters - high and low pass image filters.
Laplacian/Laplacian of Gaussian Filter - edge detection filter
Unsharp Filter - edge enhancement filter Conservative Smoothing
This is accomplished by a procedure which first finds the minimum and maximum intensity values of all the pixels Conservative smoothing within a windowed region around the pixel in question. is a noise reduction If the intensity of the central pixel lies within the intensity technique that derives its range spread of its neighbors, it is passed on to the output name from the fact that it image unchanged. employs a simple, fast filtering algorithm that However, if the central pixel intensity is greater than the sacrifices noise maximum value, it is set equal to the maximum value; if suppression power in the central pixel intensity is less than the minimum value, it order to preserve the high is set equal to the minimum value. spatial frequency detail (e.g. sharp edges) in an median image. It is explicitly designed to remove noise spikes --- i.e. isolated pixels of exceptionally low or high pixel intensity (e.g. salt and pepper noise) and is, therefore, less effective at removing additive noise Conservatively smoothing a local pixel neighborhood. The central pixel (e.g. Gaussian noise) from an image. of this figure contains an intensity spike (intensity value 150). In this case, conservative smoothing replaces it with the maximum intensity value (127) selected amongst those of its 8 nearest neighbors. Crimmins Speckle Removal
Crimmins Speckle Removal reduces speckle from an image The operator has a simple form. Suppose that at any time the three using the Crimmins consecutive pixels (e.g. on a N-S column) that are being examined are complementary hulling a, b, c. algorithm. The algorithm is: For each iteration The algorithm has been specifically designed to reduce 1. Do dark pixel adjustment: the intensity of salt and pepper For each of the 4 directions noise in an image. 1. Process whole image with: if a≥ b+2then b=b+1 if a>band b≤ c then b=b+1 Increased iterations of the 2. Process whole image with: if c>band b≤ a then b=b+ 1 algorithm yield increased levels 3. Process whole image with: if c≥ b+2then b=b+1 of noise removal, but also 4. Process whole image with: introduce a significant amount of blurring of high frequency 1. Do light pixel adjustment: details. For each of the 4 directions if a≤ b− 2then b=b −1 It involves a series of pairwise 1. Process whole image with: if a The Laplacian of an image highlights regions of rapid intensity change and is therefore often used for edge detection (see zero crossing edge detectors). The Laplacian is often applied to an image that has first been smoothed with something approximating a Gaussian smoothing filter in order to reduce its sensitivity to noise, and hence the two variants will be described together here. The operator normally takes a single graylevel image as input and produces another graylevel image as output. If a portion of the filtered, or gradient, image is added to the original image, then the result will be to make any edges in the original image much sharper and give them more contrast. This is commonly used as an enhancement technique in remote sensing applications. Unsharp Filter The unsharp filter is a simple sharpening operator which derives its name from the fact that it enhances edges (and other high frequency components in an image) via a procedure which subtracts an unsharp, or smoothed, version of an image from the original image. where k is a scaling constant. Reasonable values for k vary between 0.2 and 0.7, with the larger values providing increasing amounts of sharpening. Distance Transform The distance transform is an operator normally only applied to binary images. The result of the transform is a graylevel image that looks similar to the input image, except that the graylevel intensities of points inside foreground regions are changed to show the distance to the closest boundary from each point. One way to think about the distance transform is to first imagine that foreground regions in the input binary image are made of some uniform slow burning inflammable material. Then consider simultaneously starting a fire at all points on the boundary of a foreground region and letting the fire burn its way into the interior. If we then label each point in the interior with the amount of time that the fire took to first reach that point, then we have effectively computed the distance transform of that region. Watershed Transform This operator segments images into watershed regions and their boundaries. Considering the gray scale image as a surface, each local minimum can be thought of as the point to which water falling on the surrounding region drains. The boundaries of the watersheds lie on the tops of the ridges Hough Transform Points that lie on the same line in the image will produce sinusoids that all cross at a single point in the Hough transform. For the inverse transform, or backprojection, each point in the Hough domain is transformed into a straight line in the image. Usually, the Hough function is used with binary images, in which case H(θ,ρ) gives the total number of sinusoids that cross at point (θ,ρ), and hence, the total number of points making up the line in the original image. By choosing a threshold T for H(θ,ρ), and using the inverse Hough function, you can filter the original image to keep only lines that contain at least T points. Radon Transform The Radon transform is used to detect features within an image. Given a function A(x, y), the Radon transform is defined as: This equation describes the integral along a line s through the image, where ρ is the distance of the line from the origin and θ is the angle from the horizontal. Morphology Dilation - grow image regions Erosion - shrink image regions Opening - structured removal of image region boundary pixels Closing - structured filling in of image region boundary pixels Hit and Miss Transform - image pattern matching and marking Thinning - structured erosion using image pattern matching Thickening - structured dilation using image pattern matching Skeletonization/Medial Axis Transform - finding skeletons of binary regions Erode and Dilate Binary dilation of set α by set β (called the structuring element), denoted αβ, is the union (logical OR) of all translations of set α by elements of set β. The erosion of set α by set β , denoted αӨβ, is the set intersection of all negative translates of set α by elements of set β. Dilation and erosion are often applied to an image in concatenation. Dilation followed by erosion is called a Close operation. The closing of an image with a compact structuring element without holes, smoothes the contours of objects and eliminates small holes in objects. The Open operator, which is the morphological dual of Close, consists of erosion followed by dilation. Opening an image smooths the contours of objects, eliminates small objects, and breaks narrow strokes. Hit and Miss The hit-and-miss transform is a general binary morphological operation that can be used to look for particular patterns of foreground and background pixels in an image. It is actually the basic operation of binary morphology since almost all the other binary morphological operators can be derived from it. As with other binary morphological operators it takes as input a binary image and a structuring element, and produces another binary image as output. The structuring element used with erosion and dilation is often depicted containing both ones and zeros as well, but in that case the zeros really stand for `don't care's', and are just used to fill out the structuring element to a convenient shaped kernel, usually a square. The Hit-and-Miss transform take zeros in consideration Structuring elements for skeletonization by morphological Thinning thinning. At each iteration, the image is first thinned by the left hand Thinning is a morphological operation structuring element, and then by the right hand one, and that is used to remove selected then with the remaining six 90° rotations of the two foreground pixels from binary images, elements. somewhat like erosion or opening. The process is repeated in cyclic fashion until none of the It can be used for several applications, thinnings produces any further change. but is particularly useful for skeletonization. In this mode it is commonly used to tidy up the output of edge detectors by reducing all lines to single pixel thickness. Thinning is normally only applied to binary images, and produces another binary image as output. The thinning operation is related to the hit-and-miss transform and can be expressed quite simply in terms of it. The thinning of an image I by a structuring element J is: where the subtraction is a logical subtraction defined by Thickening Thickening is a morphological operation that is used to grow selected regions of foreground pixels in binary images, somewhat like dilation or closing. It has several applications, including determining the approximate convex hull of a shape, and determining the skeleton by zone of influence. Thickening is normally only applied to binary images, and it produces another binary image as output. The thickening operation is related to the hit-and-miss transform and can be expressed quite simply in terms of it. The thickening of an image I by a structuring element J is: Thus the thickened image consists of the original image plus any additional foreground pixels switched on by the hit- and-miss transform. Skeleton Transform The skeleton of an object in an image is a set of lines that reflect the shape of the object. The set of skeletal pixels can be considered to be the medial axis of the object. The Skeleton by Thinning is the infinite application of the Thinning operator using a sequence of structuring elements A and B generated by successive rotations of a pattern. By changing the pattern and the step used the its rotation, it is possible to generate a large number of thinning operators. Depending of the pattern and step used, the operator can be homotopic (preserve the topology of the original image) or not. Skeletonization/Medial Axis Transform Another way to think about the skeleton is as the loci of centers of bi-tangent circles that fit entirely within the foreground region being considered. The terms medial axis transform (MAT) and skeletonization are often used interchangeably but we will distinguish between them slightly: • The skeleton is simply a binary image showing the simple skeleton. • The MAT on the other hand is a graylevel image where each point on the skeleton has an intensity which represents its distance to a boundary in the original object. The skeleton can be produced in two main ways. The first is to use some kind of morphological thinning that successively erodes away pixels from the boundary (while preserving the end points of line Skeleton MAT segments) until no more thinning is possible. The alternative method is to first calculate the distance transform of the image. The skeleton then lies along the singularities (i.e. creases or curvature discontinuities) in the distance transform. Application of i-MAT procedure to Solar Photosphere Images Roberts Cross Edge Detector - Feature Detection 2×2 gradient edge detector Sobel Edge Detector - 3×3 gradient edge detector Canny Edge Detector - non- maximal suppression of local gradient magnitude Compass Edge Detector - 3×3 gradient edge detectors Zero Crossing Detector - edge detector using the Laplacian of Gaussian operator Line Detector - line (as distinct from edge) feature detector Roberts Cross Edge Detector The Roberts Cross operator performs a simple, quick to compute, 2-D spatial gradient measurement on an image. It thus highlights regions of high spatial frequency which often correspond to edges. In its most common usage, the input to the operator is a grayscale image, as is the output. Pixel values at each point in the output represent the estimated absolute magnitude of the spatial gradient of the input image at that point. In theory, the operator consists of a pair of 2×2 convolution kernels as shown in Figure 1. One kernel is simply the other rotated by 90°. This is very similar to the Sobel operator. The absolute magnitude is the only output the user sees --- the two components of the gradient are conveniently computed and added in a single pass over the input image Sobel Edge Detector The Sobel operator performs a 2-D spatial gradient measurement on an image and so emphasizes regions of high spatial frequency that correspond to edges. Typically it is used to find the approximate absolute gradient magnitude at each point in an input grayscale image. The operator consists of a pair of 3×3 convolution kernels as shown in Figure 1. One kernel is simply the other rotated by 90°. Again, the absolute magnitude is the only output the user sees --- the two components of the gradient are conveniently computed and added in a single pass over the input image Canny Edge Detector The Canny operator was designed to be an optimal edge detector. It takes as input a gray scale image, and produces as output an image showing the positions of tracked intensity discontinuities. The Canny operator works in a multi-stage process. • First of all the image is smoothed by Gaussian convolution. • Then a simple 2-D first derivative operator (somewhat like the Roberts Cross) is applied to the smoothed image to highlight regions of the image with high first spatial derivatives. • Edges give rise to ridges in the gradient magnitude image. • The algorithm then tracks along the top of these ridges and sets to zero all pixels that are not actually on the ridge top so as to give a thin line in the output, a process known as non- maximal suppression. • The tracking process exhibits hysteresis controlled by two thresholds: T1 and T2, with T1 > T2. • Tracking can only begin at a point on a ridge higher than T1. • Tracking then continues in both directions out from that point until the height of the ridge falls below T2. This hysteresis helps to ensure that noisy edges are not broken up into multiple edge fragments. Compass Edge Detector Compass Edge Detection is an alternative approach to the differential gradient edge detection. The operation usually outputs two images, one estimating the local edge gradient magnitude and one estimating the edge orientation of the input image. When using compass edge detection the image is convolved with a set of (in general 8) convolution kernels, each of which is sensitive to edges in a different orientation. For each pixel the local edge gradient magnitude is estimated with the maximum response of all 8 kernels at this pixel location. Zero Crossing Detector The zero crossing detector looks The behavior of the LoG zero crossing edge detector for points where the Laplacian of is largely governed by the standard deviation of the the image changes sign. Gaussian used in the LoG filter. The higher this value is set, the more smaller features Such points often occur at `edges' will be smoothed out of existence, and hence fewer in images --- i.e. points where the zero crossings will be produced. intensity of the image changes rapidly, but they also occur at Hence, this parameter can be set to remove unwanted places that are not as easy to detail or noise as desired. associate with edges. The idea that at different smoothing levels different It is best to think of the zero sized features become prominent is referred to as crossing detector as some sort of `scale'. feature detector rather than as a specific edge detector. Zero crossings always lie on closed contours, and so the output from the zero crossing detector is usually a binary image with single pixel thickness lines showing the positions of the zero crossing points. Zero-crossing (2) Method • Definition of Region of interest (= Compact Structure) – Set of connected pixels included within a boundary defined by the superposition of zero-crossing curves. – For granulation images: min = smallest spatial scale (1 pixel 0.1 Mm) • max = 1/3 characteristic scale of structure (1/3 1.5 Mm 0.5 Mm) Biological Analogy • Image acquisition by visual system: i. Intensity variations recognition ii. Raw primary scene formation • Intensity variation: – Variable spatial scale – Zero-crossing (2) • Isotropic operator with variable extension: x2 y 2 2 2 1 2 2G x, y e 2 2 2 2 x y 2 Application of 2G at diffrfnt valufs: Line Detector A convolution based technique which produces an image description of the thin lines in an input image. Note that the Hough transform can be used to detect lines; however, in that case, the output is a parametric description of the lines in an image. The line detection operator consists of a convolution kernel tuned to detect the presence of lines of a particular width and at a particular orientation. Pair Correlation function: g2(r) • g2(r) = probability to have a structure in the volume element dr at distance r IF at r = 0 we have a structure. • Used in matter physics to characterize amorphous materials or in cosmology for CMB structures. V N N g2(r)= 2 ∑∑ δ (r⃗ − r⃗ ij ) N ⟨ i=1 j ≠ 1 ⟩ Homogeneous and disordered distribution g2(r) =1 To get an idea… 35 − x / λ 30 y=Ae cos(x/ω)+K 25 20 ) r ( 2 g 15 • ω is an estimator of order at 10 short range. 5 0 0 50 100 150 200 250 • λ is an estimator of Square Lattice. Distance (Mm) decorrelation. 1.8 Model: DAMP_COSINE -x/ 1.6 y = e cos(x/P))) + K 1.4 A= 0.42 ± 0.02 1.2 ) r ( 1.0 2 g λ= 340 ± 60 0.8 0.6 ω= 41.7 ± 0.1 0.42 ± 0.02 340 ± 60 Mm 0.4 P 41.7 ± 0.1 Mm K 1.017 ± 0.006 0.2 K= 1.017 ± 0.006 0 50 100 150 200 250 Distance (Mm) Square Lattice + thermal displacement 1.8 Model: DAMP_COSINE -x/ 1.6 y = e cos(x/P))) + K 1.4 A= 1.0 ± 0.6 1.2 ) r ( 1.0 2 g λ= 20 ± 5 0.8 0.6 1.0 ± 0.6 ω= 25 ± 2 20 ± 5 Mm 0.4 P 25 ± 2 Mm K 1.044 ± 0.004 0.2 K= 1.044 ± 0.004 0 50 100 150 200 250 Distance (Mm) Random distribution with minimum distance. Minkowsky functionals The set of MFs in 2D. The MFs form a complete set of measures that can be used as a statistical tool for quantifying geometry and topology of a "convex structure". By definition, a convex structure is a pattern where a line drawn in between any two points on that structure remains inside that structure. Properties of MFs • In D-dimension, there exists a hierarchy of the MFs. For example, in 2D there are functionals of rank 0, rank 1 and rank 2; these are known as, respectively, scalar, vector, tensor functionals. The scalar functionals are coordinate independent but the vectors and tensors depend on coordinate. • In D-dimension, the scalar functionals obey a set of covariance properties such as motion invariance (includes translation and rotation), additivity, and continuity. • In D-dimension, the number of scalar and vector functionals is D+1 where as the number of tensor functionals is 3D+1. • Tensor functionals are symmetric. • The set of the MFs in D-dimension completely characterize the geometry (i.e., shape) and topology of a "convex structure". • The MFs are additive measures. This property implys that the functionals can be used to measure the local as well as global properties of a system. It allows one to calculate the functionals for a larger system by summing up the local contributions. • The parameters constructed from the MFs are generic and robust. Minkowsky functionals M F a sy na mm ly The set of MFs in 2D.e sis tr is The MFs formTh a complete sety oof measures s that can be used as a statisticalis tool for quantifyingf en geometry (g m a p si and topologyal aof a "convexak structure". a tiv By definition, a xconvex structuree is a patterntte wheree ies it s rn to a line drawn in between, C any twou ipoints on that . the structure remains inside thatM structure.ta t B ble op ) f ol or og Properties of MFs a n y, • In D-dimension, there exists a hierarchy of the MFs.um For example, in 2D therege are functionals of rank 0, rank 1 and rank 2; these are known as, respectively, scalar,b vector, tensor functionals.om The scalar er et functionals are coordinate independent but the vectors and tensorsof depend on coordinate.ry • In D-dimension, the scalar functionals obey a set of covariance apropertiesp such as motiona invariancen (includes translation and rotation), additivity, and continuity. pl d ica • In D-dimension, the number of scalar and vector functionals is D+1 where tasio the number of tensor functionals is 3D+1. n • Tensor functionals are symmetric. • The set of the MFs in D-dimension completely characterize the geometry (i.e., shape) and topology of a "convex structure". • The MFs are additive measures. This property implys that the functionals can be used to measure the local as well as global properties of a system. It allows one to calculate the functionals for a larger system by summing up the local contributions. • The parameters constructed from the MFs are generic and robust. 2D Minkowsky functionals For a disk all vectors (normalized curvature centroids) coincide with the symmetry centre and all tensors are We treat an image as a set of contour lines isotropic. For an ellipse the tensors become anisotropic indicating the corresponding to a set of surface brightness ellipticity and the orientation of the shape, but because of levels. the point symmetry the curvature centroids coincide. A contour is constructed by linear interpolation For an asymmetric pattern they begin to fan out along the at a given level. For every contour, the first step symmetry axis. of the functional analysis provides the following In the generic case the curvature centroids span a triangle: parameters. its size can be used to measure the anisotropy of the arrangement. Minkowsky functional can unfold the geometry of patterns made of several components: every group is weighted according to its morphological properties. Three scalars: NOTE: the SCALAR MF simply add up for the various subgroups and fail to capture their relation. Three vectors or centroids: Three symmetric tensors: