Ring-shaped barcodes Bachelor Thesis August 2015
Robin Brügger University of Applied Sciences and Arts Northwestern Switzerland FHNW [email protected]
Supervisor: Prof. Dr. Christoph Stamm University of Applied Sciences and Arts Northwestern Switzerland FHNW [email protected]
Customer: Paul Glendenning NANO 4 U, 6060 Sarnen [email protected]
Abstract: Previous work has shown that a specialized, ring-shaped barcode format is preferable for certain applications. This thesis develops a new ring-shaped, two-dimensional barcode format. The encoder generates an image of a barcode containing the payload data. The decoder recognizes the barcode and reads the payload data. The decoder is capable of dealing with perspective distortion by correcting it prior to reading the payload. Barcodes with small defective areas can still be read thanks to ECC (Error-Correction-Codes). Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
Table of contents
1 Introduction ...... 4 1.1 Project goals and requirements ...... 4 1.2 Integration and software ...... 4 1.3 Existing circular barcodes ...... 4 1.4 Schedule ...... 5 2 Ring-shaped barcode format ...... 8 2.1 Usable payload and limitations ...... 9 3 Encoder (Barcode generator) ...... 10 3.1 Arguments ...... 10 3.2 Implementation ...... 10 4 Recognition and decoding ...... 11 4.1 Preparations ...... 11 4.2 Decoder pipeline ...... 11 4.3 Barcode recognition ...... 12 4.3.1 Observations ...... 12 4.3.2 Blob filter ...... 13 4.3.3 Improvements to the OpenCV SimpleBlobDetector ...... 13 4.4 Find the perspective correction markers and the startmarker ...... 14 4.4.1 Observations ...... 14 4.4.2 Algorithm ...... 14 4.4.3 Common problems and their solutions ...... 15 4.4.4 Increasing accuracy ...... 15 4.5 Rough perspective correction ...... 17 4.5.1 Transformation ...... 17 4.5.2 Reason for the limited accuracy ...... 18 4.6 Read radial timing information ...... 18 4.6.1 Algorithm ...... 19 4.6.2 Verification ...... 19 4.6.3 Selection of a square ...... 20 4.7 Accurate perspective correction ...... 20 4.8 Read vertical timing information ...... 20 4.8.1 Algorithm ...... 21 4.9 Evaluation of the payload modules ...... 21 4.9.1 Grid ...... 21
2 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
4.9.2 Reading a module ...... 22 5 Protocol ...... 23 5.1 Error correction codes ...... 23 6 Tests...... 24 6.1 Tests using a webcam ...... 24 6.2 Batch testing robustness ...... 24 6.2.1 Added defects ...... 24 6.2.2 Tests ...... 25 6.2.3 Interpretation ...... 27 6.3 PSNR (peak signal-to-noise ratio) ...... 28 6.4 Performance ...... 28 7 Further work and improvements ...... 30 7.1 Scanning of actual coded products ...... 30 7.2 ECC license ...... 30 7.3 Parametrizing the decoder ...... 30 7.4 Avoid data looking like a startmarker ...... 30 7.5 Error detection ...... 31 7.6 Error correction in length field ...... 31 8 API and project structure ...... 32 8.1 Project structure ...... 32 8.2 API...... 32 8.2.1 EncoderLib ...... 32 8.2.2 DecoderLib ...... 32 9 Personal reflection ...... 33 9.1 Acknowledgments ...... 33 10 Sources ...... 34 11 Academic honesty statement ...... 35 12 Appendix ...... 36 12.1 Glossary ...... 36 12.1.1 Module ...... 36 12.2 Software used ...... 36 12.2.1 Libraries ...... 36 12.2.2 Others ...... 36
3 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
1 Introduction NANO 4 U develops counterfeit prevention solutions for products. For certain products and applica- tions, a ring shaped 2D barcode is considered to be more efficient than a standard square 2D bar- code.
1.1 Project goals and requirements The goal of this project is to develop a ring-shaped barcode format. For the specific applications of the barcode, NANO 4 U has the following requirements: Payload: The usable payload of the barcode should be at least 25 characters, resulting in 25 bytes. Size: Smallest module size of at least 0.2mm or 5 times the lateral resolution of the scanner. The maximal outer diameter of the barcode is 8mm. Form: Ring-shaped. The project asks for the following software components: Encoder: Generates a new barcode as an image. Takes the payload, desired module size, in- ner and outer radius as arguments und creates a barcode accordingly. The encoder is de- scribed in the chapter Encoder (Barcode generator). Decoder: Detects a barcode in an image and is able to recover the original payload data. The decoder is covered in detail in the chapter Recognition and decoding. Tests: It is not possible within the scope of this project to evaluate the results of the ring- shaped barcodes on real products. The robustness of the algorithm shall therefore be tested using software to distort and add noise to the barcodes.
1.2 Integration and software Because it allows for a platform-independent solution and easy integration into other software systems, C++ was chosen as the programming language for this project. I selected OpenCV as computer vision library because of its excellent documentation, vast functionality and my good experiences with it.
1.3 Existing circular barcodes ShotCode is currently the only existing circular 2D barcode1. An example is shown in Illustration 2. It was developed to share links and can be read with low resolution mobile phone cameras or webcams. Today, it has been replaced by QR-Codes.
Illustration 1: Shotcode4
1 Barcoding Connected (2009): SHOTCODE BARCODE: THE CIRCULAR 2D BARCODE FOR MOBILE TAGGING. http://blog.barcoding.com/2009/02/shotcode-barcode-the-circular-2d-barcode-for-mobile-tagging/ (Accessed August 5, 2015)
4 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
ShotCode is not suitable for this project for two reasons: Limited payload Depending on the source, a ShotCode is limited to 492 or 403 bits (5 to 6 bytes) of data. How- ever, NANO 4 U requires about 25 bytes. Centre is not empty A ShotCode uses the area in the centre of the circle. This area is not available for use on the end-user products and has to be left blank.
1.4 Schedule The project started on 24 February 20154with the deadline on 15 August 2015. Due to the unfore- seeable challenges of decoding a barcode, it was not possible to break the decoding process down into individual steps and schedule them separately. Illustration 2 shows the planned and effective project schedule.
2 Wikipedia (2013): ShotCode. https://de.wikipedia.org/wiki/ShotCode (Accessed June 7, 2015) 3 Wikipedia (2014): Shotcode. https://en.wikipedia.org/wiki/ShotCode (Accessed June 7, 2015) 4 Mudie, Stuart: Shotcode. Licensed under CC BY-SA 2.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Shotcode.png#/media/File:Shotcode.png
5 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
Illustration 2: Planned and effective project schedule
6 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
The workload for the bachelor’s thesis should be approximately 360 hours, spread over the whole project duration. Because of lectures, exams and special occasions such as the project week, the scheduled work hours varied from week to week. Illustration 3 shows the planned and effective work hours per week. During the course of the project, slightly fewer hours were invested than planned. However, this was made up in the last two weeks of the project, which had been allocated as re- serve.
Illustration 3: Planned and effective work hours
7 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
2 Ring-shaped barcode format In this chapter the developed barcode format is explained. Illustration 5 is an original barcode as it is generated by the encoder. Illustration 6 is the same barcode, but areas for recognition, per- spective correction and timing are high- lighted. The coloured areas serve the following purposes: Yellow: Solid black ring for lo- cating the barcode in an image. Orange: Markers for a rough perspective correction. They are arranged in a perfect square. Green: The startmarker defines the beginning in the otherwise continuous circle. It is also used to determine the outer radius of the barcode. Red: The vertical zebra pattern Illustration 4: Ring-shaped barcode generated by the encoder determines the vertical timing of the rings (how far apart the rings are). Violet: The radial zebra pattern determines the angles between the modules. It is also used to achieve an accurate perspective correction, therefore the radial module count is always divida- ble by 8. Thus, it is possible to select four modules of the zebra pattern which lay in a perfect square, facilitating the perspec- tive correction calculation. Blue: Area used for the payload. The data area can be interpret- ed as a curved table, addressing modules via their x and y index. Illustration 7 shows how the ad- dressing of the payload area is Illustration 5: Ring-shaped barcode with special areas highlighted organised. To make use of the bigger modules toward the outside of the barcode, these modules are filled with payload first.
8 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
Illustration 6: Addressing of the payload modules 2.1 Usable payload and limitations The barcode used in Illustration 5 and Illustration 6 was generated following the size requirements described in the chapter Project goals and requirements and is therefore considered realistic in the application at hand. It has a maximum payload of 56 bytes. However, some of that is consumed by the protocol described in the chapter Protocol. The maximum payload the ring-shaped barcode supports is 234 bytes. This is due to constraints of the payload protocol. The barcode design itself poses no upper bound for the payload size.
9 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
3 Encoder (Barcode generator) The encoder generates barcodes as an image. It takes the payload and three sizing arguments as input.
3.1 Arguments The size and shape of the barcode is determined by three basic parameters, measured in pixels. Illustra- tion 8 visualizes the meaning of the parameters. Minimal module size (red): The minimal size of a single module. This is also the thickness of the individual rings. Technical name: moduleSizePixels. Minimal radius (blue): Distance from the centre of the barcode to the inner perspec- tive markers. The perspective markers are the smallest module-sized structure in the barcode. Technical name: minRadiusPixels. Maximum radius (orange): Distance from the centre of the barcode to its outermost point. Technical name: maxRadiusPixels. Illustration 7: Basic barcode encoder parameters The barcode used in Illustration 8 was generated using moduleSizePixels=8, minRadiusPixels=80, maxRadiusPixels=160.
3.2 Implementation
val minRadiusPixels, maxRadiusPixels, moduleSizePixels var currentRadius = maxRadius
while(currentRadius >= minRadius) for(angle : allAngles) if(black) drawCircleSegment(angle, currentRadius, black) drawCircle(currentRadius - moduleSizePixels, white) curentRadius -= moduleSizePixels
Illustration 8: Pseudocode of the barcode encoder
Implementing the encoder was a straightforward process. It uses the OpenCV drawing functions, particularly the circle() function which is able to draw circle sectors. However, OpenCV does not pro- vide a function to draw ring segments. Therefore, the barcode is rendered from the outside to the inside. Firstly, the modules in the outermost circle are drawn as circle sectors. Upon completion of that ring a slightly smaller white circle is drawn, transforming the already drawn circle sectors to ring sectors. This process is repeated until the barcode is completely rendered. Illustration 9 provides pseudocode for this process.
10 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
4 Recognition and decoding The decoder recognizes a barcode in an image and is able to read its payload. The payload itself is subject to a simple protocol described in the chapter Protocol.
4.1 Preparations The input for the decoder can be any image. It could be a greyscale image or even an image with RGB colour channels. Because the barcode only features black and white modules, the input image must be binarized. Selecting fixed threshold values or an adaptive threshold has not yielded good results in correctly binarizing black modules as black and white modules as white. It is particularly challenging that the scanned image not only contains the barcode, but also its irrelevant surroundings. The shape and colour of these surrounding areas can vary widely and cannot be predicted. Satisfactory results were achieved with Otsu’s binarization method, which is explained on “The Lab Book Pages”5 and is readily available in OpenCV. Using this method, the input image is binarized be- fore entering the decoder pipeline. In order to cope with noisy images, a Gaussian blur with a 5x5 kernel is applied prior to binarization.
4.2 Decoder pipeline Intuitively, the process of reading a barcode would be structured as following:
Perspective Timing Evaluation of Recognition correction information the modules
Illustration 9: Basic process pipeline of a barcode reader
Recognition: Finding the barcode in the image provided Perspective correction: Perspective errors are caused by imperfect alignment of the barcode and the camera or scanner. They need to be corrected before the payload can be read. Timing information: Getting the size of the modules Evaluation of the modules: The actual data reading process
5 The Lab Book Pages: Otsu Threshholding. http://www.labbookpages.co.uk/software/imgProc/otsuThreshold.html (Accessed August 1, 2015)
11 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
However, the decoder for the ring-shaped barcode cannot fully follow this clear process because, as described in the chapter Ring-shaped barcode format, the radial zebra pattern contains the timing
Rough Find the perspective correction Recognition perspective markers and the startmarker correction
Read radial Accurate Read vertical Evaluation of timing perspective timing the modules information correction information
Illustration 10: Process pipeline of the decoder information of the angles and is also used to achieve an accurate perspective correction. Therefore, the decoder processes “perspective correction” and “reading timing information” are not strictly sequential. Furthermore, a twostep approach is used for the perspective correction, resulting in the process pipeline displayed in Illustration 11. The following chapters will discuss the decoder pipeline in detail. Each stage of the pipeline works on a best effort basis. If for example the recognition stage fails because it does not find a barcode, the input image is discarded and the rest of the processing pipeline is not activated.
4.3 Barcode recognition The goal of this processing step is to locate a ring-shaped barcode in an image and to estimate its centre. Input: Binarized image which potentially contains a barcode. Output: Approximate centre, major and minor axis length of the central ellipse. Barcodes are recognized using blob analysis. In Illustration 12 the area which is used to recognize a barcode is highlighted in yellow. OpenCV provides the class SimpleBlobDetector. It al- lows filtering the blobs by area, circularity, inertia, convexity and colour6.
4.3.1 Observations The following observations can be made on the input images and lead to the filter settings of the blob detector: The centre of the barcode is always a perfect circle. However, due to perspective errors it can appear as an ellipse on the input image. Henceforth, the central cir- Illustration 11: The yellow area is used for cular blob will therefore be referred to as ellipse. the barcode recognition The centre of the barcode is white. The barcode’s size lies between certain boundaries. If it is bigger than the field of view of the camera, it cannot be decoded because some information is not visible and therefore missing. On the other hand, it can be expected that the barcode is not too small either.
6 Mallick, Satya (2015). Blob Detection Using OpenCV ( Python, C++ ). http://www.learnopencv.com/blob-detection-using-opencv-python- c/ (Accessed July 13, 2015)
12 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
4.3.2 Blob filter Based on these observations, filters for the blob detector can be determined to find the barcode in an image with high accuracy: Circularity between 0.8 and 1.0, filtering out all objects that are not elliptic. Colour must be white Assuming the blob is perfectly circular, its radius must lay between 1/16 and 1/4 of the aver- age image dimensions:
퐼푚푎𝑔푒푊𝑖푡ℎ + 퐼푚푎𝑔푒퐻푒𝑖𝑔ℎ푡 퐴푣𝑔퐷𝑖푚푒푛푠𝑖표푛 = 2
퐴푣𝑔퐷𝑖푚푒푛푠𝑖표푛 푟 = 푚푖푛 16
퐴푣𝑔퐷𝑖푚푒푛푠𝑖표푛 푟 = 푚푎푥 4
2 2 푟푚푎푥 휋 > 퐴퐵푙표푏 ≥ 푟푚푖푛 휋
4.3.3 Improvements to the OpenCV SimpleBlobDetector The SimpleBlobDetector of OpenCV is able to reliably detect ring-shaped barcodes. However, it has some shortcomings which are relevant for the decoder: It is quite slow because SimpleBlobDetector binarizes the image before running the detec- tion, which itself is based on an edge filter. Binarization is not done once, but around 20 times using different fixed values as threshold. For every threshold the complete blob detec- tion algorithm is run. However, the binarization is superfluous because we already binarize the image using Otsu’s method, as described in the chapter Preparations. The algorithm returns a KeyPoint-Object which contains only basic information such as size and location of the blob. However, it was desirable for the encoder to be able to access the blob’s moments to determine the length of both the ellipse’s major and minor axis. This in- formation is useful for the next step in the pipeline and makes it possible to mark the recog- nized ellipse in the input image for debugging purposes. I therefore adapted the SimpleBlobDetec- tor class by adding the mentioned features, basing my work on Joel Temply’s improve- ments of the SimpleBlobDetector class7. Illustration 12 shows debug output of the detector. Thanks to access to the blob’s moments, the detected ellipse can be drawn quite accurately. The green dot marks the estimated centre of the barcode.
Illustration 12: Visualized output of the recognition step
7 Teply, Joel (2014): http://stackoverflow.com/a/25152785/2037769 (Accessed June 15, 2015)
13 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
4.4 Find the perspective correction markers and the startmarker The goal of this step is to find the four perspec- tive markers and the startmarker of an image. In Illustration 14 the perspective markers are high- lighted in orange and the startmarker in green. In an unmodified barcode the startmarker is black and the perspective correction markers are white. In a barcode without perspective distortion the four perspective markers lay in a perfect square. Input: Binarized image with barcode Estimated barcode centre Major and minor axis length of the cen- tral ellipse Output: Illustration 13: Perspective markers (orange) and startmarker Start and end angle of the startmarker (green) relative to a horizontal line through the barcode centre Location of the perspective correction markers
4.4.1 Observations This step is based on the following observations: Imagining a straight line from the barcode centre to infinity in any direction and counting the black pixels in the first continuous stretch, the startmarker is where there is the longest continuous stretch of black pixels. The perspective markers are where there is the shortest one.
4.4.2 Algorithm Based on these observations, the algo- rithm uses a radial scanline revolving around the estimated centre of the barcode. The scanline starts at 70% of the minor axis of the central ellipse. In Illustration 15 this is shown as a green circle. The end of the scanline is theo- retically at infinity. However, because the centre is enclosed by a circle of black pixels, it ends quite soon in prac- tical terms. Along the scanline the length of the first continuous stretch of black pixels is evaluated. Illustration 15 shows a possible plot of the first con- tinuous black distance at different an- gles. The four smallest values signify the location of the perspective mark- Illustration 14: Radial scanline revolving around the estimated ers, the maximum is where the start- centre of the barcode
14 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger marker is located.
Illustration 15: Plot of the continuous amount of black pixels at various angles
4.4.3 Common problems and their solutions Because of the small discreet angular steps the scanline uses to revolve around the cen- tre, it is possible that two of the four lowest points belong to the same perspective mark- er. Errors are avoided by enforcing a mini- mum angular distance from one perspective marker to the next. Sometimes the longest continuous black dis- tance is not the startmarker but an edge be- tween modules. Illustration 17 shows such a situation. Under the red line lies a continuous stretch of black pixels, making its length equal to that of the startmarker. In order to avoid these errors, the found startmarker is verified by checking that it is surrounded by continu- ous white pixels on both sides.
4.4.4 Increasing accuracy Illustration 16: Possible maximum continuous black distance Only having found any direction in which the at a location other than the startmarker startmarker lies is not accurate enough in later stages of the decoding process. Neither is just knowing any positon within each perspective marker. The positions need to be determined as accurately as possible. Increasing the accuracy is done in the same way for both the startmarker and the perspective markers. The first continuous black distance is measured at angles in close proximity to the recognized features. If the measured
15 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger black distance is less than 10% different from the original feature, it is considered as still being part of that feature. To find the beginning and the end of the feature quickly, a binary search is used, yield- ing a O(log n) complexity search for the feature edges. Illustration 17 shows the process of the search for the startmarker edge. Starting from the originally detected feature (line 0), the continuous black distance at the lines 1 to 4 is evaluated, resulting in the discovery of the accurate edge of the start- marker. The binary search is halted when the step distance is smaller than an arbitrary defined ε.
Illustration 17: Binary search for the edge of the startmarker
Illustration 18 visualises the recognized startmarker and the perspective markers in the original im- age the camera captured.
Illustration 18: Debug output showing the located startmarker and the perspective markers
16 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
4.5 Rough perspective correction This step performs a rough perspective correction, making the subsequent steps of the decoding pipeline easier. Input: Input image containing a barcode (In) Location of the four perspective markers in the input image Output: Perspective corrected image (Out)
4.5.1 Transformation This step makes use of the information that the four perspective markers lie in a perfect square in any generated barcode. Due to the perspective distortion induced by the scanner, the detected per- spective markers in the input image may not form a square, but any quad. It is possible to compute a transformation matrix M between a square and the quad detected in the input image. Illustration 19 shows how this transformation matrix works. It can be computed using the OpenCV function getPer- spectiveTransform().
Illustration 19: Transformation matrix between the quad in the perspective distorted input image and a square8.
The inverse matrix of M is then applied to every pixel in the input image: In: Input image Out: Output image
(Inx, Iny): Pixel in the input matrix M: Transformation matrix
−1 ⋀(퐼푛푥, 퐼푛푦) ∶ (푂푢푡푥, 푂푢푡푦) = 푀 ∗ (퐼푛푥, 퐼푛푦)
The OpenCV function warpPerspective()9 provides this functionality. Illustration 20 shows an example input and output image:
8 Stamm, Christoph (2014): Projektive Abbildung. Slides for the module magb, file 09_Bildmatrix.pdf 9 OpenCV (2015): Geometric Image Transformations. http://docs.opencv.org/modules/imgproc/doc/geometric_transformations.html#warpperspective (Accessed August 10, 2015)
17 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
Illustration 20: Input image on the left and output image on the right. The output image is perspective corrected.
4.5.2 Reason for the limited accuracy The limited accuracy of this perspective correction is mainly due to the perspective markers being located well within the barcode, on the inside of the section carrying the payload data. Any inaccura- cy in the location of the markers is amplified in outer areas of the barcode. Therefore a more accu- rate perspective correction is applied later in the decoding pipeline.
4.6 Read radial timing information This step reads the radial timing information of the barcode do determine the angle between the data modules. It also sets the foundation for a more accurate perspective correction. The radial timing information is stored in the zebra pattern around the outer edge of the barcode. In Illustra- tion 22 this area is highlighted in violet. Input: Roughly perspective corrected input im- age Estimated centre of the barcode Location of the startmarker Output: Radial module count Location of four black modules belonging Illustration 21: The radial timing information is stored in the to the zebra pattern. They form a square zebra pattern around the outside of the barcode. It is high- in undistorted barcodes. lighted in violet.
18 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
4.6.1 Algorithm To read the zebra pattern, a scanline revolving around the estimated centre of the barcode is used. The scanline starts just outside the barcode and scans towards the centre of the barcode trying to find the black modules. The distance to the occurrence of the first black pixel is evaluated. If this distance is within certain bounds around B, we have found a black module. Illustra- tion 23 visualises the path of the scanline. An initial value of B is determined by the distance between the blue circle in Illustration 23 and the outside of the startmarker. Although the input image has been roughly perspective corrected prior to this processing step, the correction might not be perfect. Illustration 24 shows such a scenario. B must therefore be con- tinuously updated to reflect the local situation. This is done according to the following formula as the scanline Illustration 22: Scanline starting outside the barcode and revolving around its centre. progresses: S: Constant to regulate the weight of newer measured distances against older ones x: Measured distance between the blue ring and the black module in the zebra pattern 퐵 ∗푆+푥 퐵 = 푛 푛+1 푆+1
4.6.2 Verification Even with measures for dealing with an imperfect perspective correction, reading the radial zebra pattern occasionally fails. To prevent the decoder from progressing with wrong settings, the recog- nized black modules in the zebra pattern are sub- ject to verification. The angle between two black modules relative to the centre of the barcode can easily be estimated: 360 푎푛𝑔푙푒퐸푥푝푒푐푡푒푑 = 푏푙푎푐푘푀표푑푢푙푒푠퐼푛푍푒푏푟푎 The zebra pattern has been successfully read if any two neighbouring black modules fulfil the follow- ing inequality: Illustration 23: Example of an imperfect perspective correc- tion 푎푛𝑔푙푒퐸푥푝푒푐푡푒푑 |푎푛𝑔푙푒 − 푎푛𝑔푙푒 | ≤= 1 2 2
19 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
4.6.3 Selection of a square Since the number of modules in the outer zebra pattern is dividable by 8, the number of black mod- ules is dividable by four. Given an array with the position of every black module in the outer zebra ring, selecting four of them lying in a square is trivial.
4.7 Accurate perspective correction This step performs a second perspective correction to achieve the accuracy needed for reading the modules. Input: Input image containing a barcode (In) Location of four black modules of the outer zebra ring forming a square
Rotation matrix used in the rough perspective correction (M1) Output: Accurately perspective corrected image (Out) The basic principle is the same as described in the chapter Rough perspective correction. A transfor- mation matrix M2 is computed. This matrix M2 performs a mapping between the roughly perspective corrected image and the accurately perspective corrected image. Since the aim is to avoid precision loss by doing two transformations on an image (first a rough correction, than the accurate one), the two perspective correction matrices M1 and M2 can be multiplied to allow transformation from the input image to the accurately perspective corrected image in one step: In: Input image Out: Output image
(Inx, Iny): Pixel in the input matrix
푀퐶표푚푝푙푒푡푒 = 푀2 ∗ 푀1
⋀(퐼푛푥, 퐼푛푦) ∶ (푂푢푡푥, 푂푢푡푦) = 푀퐶표푚푝푙푒푡푒 ∗ (퐼푛푥, 퐼푛푦)
4.8 Read vertical timing information The vertical zebra pattern defines how many rings a barcode has and how far apart they are spaced. In Illustration 25 the vertical zebra pattern is highlighted in red. Input Accurately perspective corrected image Radial module count Angle of the startmarker Output Number of rings Radius of the innermost ring Radius of the outermost ring Direction of reading the barcode: clockwise or Illustration 24: The area of the vertical zebra pattern anti-clockwise is highlighted in red
20 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
4.8.1 Algorithm The vertical zebra pattern is always situated next to the startmarker. Because the barcode has possibly been mirrored, it could lie either to the right or to the left of the startmarker. Illustration 9 depicts how two lines are shot from the barcode centre through the potential locations of the vertical zebra pattern. To evaluate if the line goes through a zebra pattern the alternating continuous stretches of black and white pixels are measured. Because all rings are equal in thickness, a perfect zebra patterns would have X continuous black pixels followed by X contin- uous white pixels followed by X continuous black pixels again. If no continuous stretch of equally col- oured of pixels significantly deviates in length from the average, the vertical zebra pattern has been Illustration 25: Two test lines are shot form the centre of the barcode found. through the potential locations of the vertical zebra pattern The vertical zebra pattern also determines the read- ing direction of the barcode payload. If it is to the right of the startmarker, the reading direction is clockwise.
4.9 Evaluation of the payload modules This is the last step of the decoder pipeline. It reads the actual payload of the barcode. Input Perspective corrected input image Centre of the barcode Angle of the startmarker Number of radial modules Number of rings carrying payload data Radius of the innermost and outermost ring Reading direction (clockwise or anti- clockwise) Output Payload data of the barcode In Illustration 27, the area of the barcode’s pay- load is highlighted.
4.9.1 Grid With the input, it is possible to draw a grid over the barcode in the input image. Illustration 28 shows debug output where a grid was drawn using the input parameters of this processing stage. To keep the grid visualisation readable a line was drawn only through every second radial Illustration 26: The payload area is highlighted in blue
21 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger module. At every intersection of the grid a module shall be evaluated as either black or white.
Illustration 27: Grid matching the barcode modules
4.9.2 Reading a module A total of five pixels are sampled for each module in a cross-shaped pattern as shown in Illustration 28.
Illustration 28 Cross-shaped sampling pattern
Each pixel votes for the module to be either black or white according to its colour. Because the sam- pling is performed on a binarized image and an uneven number of pixels are sampled, a module’s colour is never ambiguous, it can always be classified as either black or white.
22 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
5 Protocol The payload of the barcode is structured according to a simple protocol as shown in Illustration 29:
Illustration 29: Payload protocol
Field Size in bytes Description Length 1 Contains the length of the body including the space used for the error correction codes. Because this field has a fixed length of one byte, the maximum size of the body of a barcode is restricted to 255 bytes. Data variable The data encoded in the barcode ECC 20 20 bytes are always reserved for error correction
5.1 Error correction codes The open source library Rscode10 was used in this project. It implements Reed-Solomon error correc- tion. Rscode performs error correction on 8-bit blocks. The amount of error correction codes has been set to 20 bytes because Rscode needs this as compile-time argument. 20 bytes of error correc- tion codes allows to fix up to 10 faulty blocks of 8 bit each.
10 Minsky, Henry: Welcome to http://rscode.sourceforge.net/. http://rscode.sourceforge.net/ (Accessed August 5, 2015)
23 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
6 Tests
6.1 Tests using a webcam During the development phase, most tests were done using a laptop webcam as scanner to read a barcode shown on a mobile phone display. The scanning process is fast and reliable enough to cause no inconvenience to the user, even under not ideal conditions such as a considerable amount of per- spective distortion. Because the results are dependent on the actions of the person holding the barcode, it is impossible to reliably quantify the success rate and overall quality with this test method.
6.2 Batch testing robustness To measure the quality and capabilities of the decoder in a reproducible way, a batch testing tool has been developed. It generates a barcode, adds some defects to it and then checks if it can be correctly decoded. To obtain reproducible results, each test is run several times.
6.2.1 Added defects The test tool is capable of adding three types of defects to a barcode: Perspective distortion The perspective distortion can be controlled in percent. 0% means no distortion, with 100% it is pos- sible that all four corners of the barcode image are mapped to a single point in the distorted image. For examples, refer to Illustration 30.
Illustration 30: Perspective distortion added by the test tool. From left to right: 0% distortion to 50% distortion in 10% incre- ments.
Salt and pepper noise This defect adds random black and white pixels to the image and is controlled by a percentage value. At 100%, the image is fully covered by noise. For examples, see Illustration 31.
Illustration 31: Salt and pepper noise added by the test tool. From left to right: 0% to 20% noise in 10% increments.
24 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
Area destruction This defect adds random ellipses to the barcode, obscuring some areas. It is controlled by a percent- age value. At 100%, the cumulative area of all ellipses is equal to the image size. For examples, refer to Illustration 32.
Illustration 32: Random ellipses obscuring parts of the image added by the test tool. From left to right: 0% coverage to 2% coverage in 1% steps.
6.2.2 Tests All tests were conducted with three different barcode sizes: Name Encoder parameters 300x300 moduleSizePixels=6; minRadiusPixels=60; maxRadiusPixels=120 400x400 moduleSizePixels=8; minRadiusPixels=80; maxRadiusPixels=160 600x600 moduleSizePixels=12; minRadiusPixels=120; maxRadiusPixels=240
Illustration 33: Successfully read barcodes at different perspective distortion levels. Sample size: 500
25 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
As shown in Illustration 33, the two bigger barcode sizes perform quite well with up to 30% distor- tion. The smaller 300x300 barcode is not well readable when perspective distortion is applied.
Illustration 34: Successfully read barcodes at different noise levels. Sample size: 500
The noise test shown in Illustration 34 shows a picture that is similar to the distortion test. The two bigger formats perform well with up to 10% noise, while the smaller 300x300 barcode struggles as soon as noise is added.
Illustration 35: Successfully read barcodes at different levels of obscured areas. Sample size: 500
As shown in Illustration 35, the decoder cannot cope well when some areas of the barcode are ob- scured. The barcode size does not seem to matter at all.
26 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
Illustration 36: Successfully read barcodes at different perspective distortion and noise levels. Sample size: 500
Illustration 36 shows that applying per- spective distortion and noise at the same time does not significantly wors- en the decoding rate over only adding noise. Illustration 38 shows a barcode that was correctly read after perspec- tive distortion as well as noise was ap- plied.
6.2.3 Interpretation As described in the chapter Recognition and decoding the decoder works on a best effort basis. This means that if there is a problem in any step of the decoding pipeline, the input image is discarded. This naturally leads to de- coding rates lower than 100% as soon as defects are added. The decoding rate needed varies from application to application. If there is a Illustration 37: Example of a barcode that was recognized and correctly read video feed such as from a webcam, a decoding rate of 50% or even lower is acceptable, since if one frame is discarded, the next one is available almost immediately. However, in an application where only a single image of the barcode is available, a decoding rate lower than 100% naturally is a problem. The barcode scanner performs fine for perspective and noise defects, but does not fare well when some areas of the barcode are missing or destroyed. This is because even if only a small percentage of the overall barcode area is destroyed, it is quite likely that at least one detection and recognition
27 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger feature is affected. Illustration 40 shows such a barcode. The vertical zebra pattern is partly de- stroyed, rendering the barcode unreadable. However, thanks to ECC the barcode in Illustration 40 is perfectly readable, despite some of the data area is destroyed.
Illustration 39: Unreadable barcode Illustration 39: Readable barcode despite defects
6.3 PSNR (peak signal-to-noise ratio) “Peak signal-to-noise ratio, often abbreviated PSNR, is an engineering term for the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation” - Wikipedia11 If only a 100% decoding rate considered acceptable, the PSNR is infinity because as can be seen in Illustration 34, the decoding rate drops under 100% as soon as any noise is added to the image. However, if the aim is to achieve a 50% decoding rate, calculating the PSNR is possible. As can be seen in Illustration 34, up to 15% noise can be added to a 400x400 barcode before the decoding rate drops below 50%. The PSNR is defined as11: 푀퐴푋2 푃푆푁푅 = 10 ∗ 푙표𝑔 ∗ ( 퐼 ) 10 푀푆퐸 MSE: Mean squared error
MAXI: Maximum intensity Because the decoder operates on a binary image the maximum intensity is 1 and the mean squared error is half the noise (Because when adding a salt and pepper noise pixel to a binary image there is a 50% chance that the pixel already has the new color). The lower bound of the PSNR is therefore 11.2 decibel if a decoding rate of 50% is desired.
6.4 Performance Performance tests were conducted on the following computer:
11 Wikipedia (2015): Peak signal-to-noise ratio. https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio (Accessed August 5, 2015)
28 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
Component Specification Processor i7-3520M dual core, 2.9 Ghz RAM 8 Gigabyte Operating System Windows 7 64bit Illustration 40 shows the encoding and decoding times using multiple barcode sizes. The encoding process is about 10 times faster than decoding process. Not much time was spent during this project on optimisations as performance was deemed fast enough. It is expected that there are some quite low hanging fruits when it comes to performance optimisations.
Illustration 40: Encoding and decoding performance of multiple barcode sizes
29 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
7 Further work and improvements
7.1 Scanning of actual coded products Because of the long lead time on producing real products with the barcode on them, it was clear from the start of the project that tests with scans of real products is out of scope for this project. Furthermore, it is not clear what kind of defects a barcode on a product would possibly have and which areas are most susceptible to damage. In agreement with the client it has been assumed that the smaller modules on the inside of the barcode are more likely to be unreadable. Data is therefore preferentially placed in the bigger modules towards the outside of the barcode. It is possible that real world experiences with the barcode on products call for changes to the barcode format.
7.2 ECC license The ECC library Rscode12 that was used in this project is licensed under GPL (General Public Li- cense)13. The GPL has a strong copyleft, requiring the whole project code to be licensed under GPL as well. Because this is possibly not desirable for NANO 4 U, two versions of the project are delivered: Version with ECC, licensed under GPL Version without ECC, not specifically licensed The reason a GPL-licensed library was used is was not possible to source a C++ ECC library licensed under a liberal license. For the future, there are multiple options: Obtain a commercial ECC library Port the ECC library used in ZXing from Java to C++. ZXing is licensed under the quite liberal Apache 2.0 license14.
7.3 Parametrizing the decoder The only argument the decoder takes is the input image. All information it requires to decode the barcode is extracted from the barcode itself. This includes the number of rings and the number of modules in each ring. It is conceivable that supplying information about the basic shape of the barcode to the decoder could improve robustness, especially if parts of the barcode are unreadable. Possible arguments are primarily the already mentioned number of rings and number of modules in a ring. Attempts to make use of such arguments have been made during the course of this project. However, it proved to be more complex than expected because of the complex decoder pipeline and dual-use barcode fea- tures such as the radial zebra pattern which is used for perspective correction and also to determine the number of modules in each ring.
7.4 Avoid data looking like a startmarker There is a chance that the payload data in a barcode forms a pattern identical to the startmarker. If two starmarker patterns are present, it is possible that the decoder selects the wrong one and there- fore the payload cannot be read. The chance of a pattern equal to the startmarker occurring in a barcode is dependent on the number of rings and the number of modules per ring. Due to the zebra
12 Minsky, Henry: Welcome to http://rscode.sourceforge.net/. http://rscode.sourceforge.net/ (Accessed August 5, 2015) 13 Wikipedia (2015): GNU General Public License. https://de.wikipedia.org/wiki/GNU_General_Public_License (Accessed August 5, 2015) 14 ZXing (2014): License Questions. https://github.com/zxing/zxing/wiki/License-Questions (Accessed August 5, 2015)
30 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger pattern around the outside of the barcode, a startmarker-lookalike can only occur at every second angular module. The chance of data looking identical to the startmarker for each angle is: 1 푝 = 푎푛푔푙푒 23∗푟푖푛푔퐶표푢푛푡 It is likelier that a startmarker- lookalike appears next to the real startmarker because the white space to the side of the startmarker is already there: 1 푃 = 푔푙표푏푎푙 22∗푟푖푛푔퐶표푢푛푡 For a barcode that has 8 rings, the chance of a startmarker-lookalike is quite small (~1:50000). How- ever, if the number of rings is reduced this quickly becomes a big problem. A possible solution is to check the generated barcode for startmarker-lookalikes. If such a startmark- er-lookalike is detected, a random bitmask is selected. To get rid of the startmarker, an XOR- Operation is performed on the payload body and the bitmask. To be able to decode the barcode again the used bitmask has to be encoded into the barcode payload as meta information.
7.5 Error detection Although ECC has been implemented, its possibilities are limited. If too many modules have been incorrectly sampled, ECC can no longer correct all errors and the payload read by the decoder is in- correct. Currently, this goes undetected. To fix this, a checksum over the barcode’s payload could be used.
7.6 Error correction in length field Currently, the length field of the payload protocol is not protected by any error correction mecha- nism because the ECC decoder itself needs to know the length of the ECC-protected payload. Due to the lack of error correction, any reading error in the length field will lead to an unreadable barcode. A possible solution to this problem would be transmitting the length field three times redundantly to compensate for reading errors.
31 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
8 API and project structure
8.1 Project structure The project workspace is structured in six subprojects: Project Description DecoderLib Library for decoding a barcode EncoderLib Library for generating a barcode DecoderVideoDemo Sample application using the decoder library. The webcam captures con- tinuous frames which are handed to the DecoderLib. Encoder Sample application using the encoder library. BatchTestTool Command line tool used to test the decoder. Uses the EncoderLib to gener- ate barcodes which are deformed and damaged in multiple ways before trying to decode them using the DecoderLib. Rscode Third party ECC library (http://rscode.sourceforge.net/)
8.2 API The EncoderLib and DecoderLib each provide a simple API.
8.2.1 EncoderLib The EncoderLib exposes the method encode(). It creates a new barcode as an image: Mat BarcodeEncoder::encode(BarcodeEncoder::EncoderConfiguration, StringDataProvider) If the data provided is longer than the capacity, an exception is thrown. The EncoderConfiguration determines the size and shape of the barcode by specifying: Inner radius Outer radius Module size It also provides a method to determine how many bytes of payload can be placed in the barcode with that specific barcode configuration: int BarcodeEncoder::EncoderConfiguration::capacityBytes()
8.2.2 DecoderLib The Deocoderlib exposes the method decode(): vector
32 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
9 Personal reflection It has been a very interesting but also challenging project. A ring-shaped barcode is without prece- dent, which makes this project quite innovative. Developing the first ring-shaped barcode format also meant going new ways in detecting the barcode. Although I attended a C++ module during my studies, this project was the first time I used C++ for a productive project. Working with C++ provides tremendous opportunities, but frankly has often been frustrating for a newbie like me. Compiler errors are hard to interpret and I had my fair share of linker problems while transitioning to a multi-project workspace. However, this is made up by C++’s versatility, execution speed and the possibility to integrate it with basically any software on any op- erating system. Overall, I am happy with the outcome of the project, although time was tight towards the end. There are still some small things that could be done to improve the decoding process. The decoder works well when decoding a frame captured with the webcam, but it is difficult to anticipate the specific challenges when the barcode is scanned from a real product. I reckon it is likely that some adjust- ments are needed, but I feel confident that the overall concept will prove effective. A special challenge of this project was that the documentation as well as the communication with the customer was in English. I consider my command of English quite good, but it has proved that it is much harder to express complex technical relationships in a language other than one’s mother tongue.
9.1 Acknowledgments I would like to express my gratitude to my supervisor, Prof. Dr. Christoph Stamm, who has guided me through this project. We had a lot of interesting discussions which often sparked new and creative ideas. Furthermore, I would like to thank NANO 4 U and Paul Glendenning for providing me with this inter- esting and challenging project. Finally, thanks to Dr. Lorenz Frey, with whom I had an interesting debate about error correction codes and their placement within the barcode.
33 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
10 Sources Barcoding Connected (2009): SHOTCODE BARCODE: THE CIRCULAR 2D BARCODE FOR MOBILE TAGGING. http://blog.barcoding.com/2009/02/shotcode-barcode-the-circular-2d-barcode-for- mobile-tagging/ (Accessed August 5, 2015) Mallick, Satya (2015). Blob Detection Using OpenCV ( Python, C++ ). http://www.learnopencv.com/blob-detection-using-opencv-python-c/ (Accessed July 13, 2015) Minsky, Henry: Welcome to http://rscode.sourceforge.net/. http://rscode.sourceforge.net/ (Ac- cessed August 5, 2015) Mudie, Stuart: Shotcode. Licensed under CC BY-SA 2.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Shotcode.png#/media/File:Shotcode.png NANO 4 U : http://www.nano4u.net/ (accessed August 15, 2015) OpenCV (2015): Geometric Image Transformations. http://docs.opencv.org/modules/imgproc/doc/geometric_transformations.html#warpperspective (Accessed August 10, 2015) Stamm, Christoph (2014): Projektive Abbildung. Slides for the module magb, file 09_Bildmatrix.pdf Teply, Joel (2014): http://stackoverflow.com/a/25152785/2037769 (Accessed June 15, 2015) The Lab Book Pages: Otsu Threshholding. http://www.labbookpages.co.uk/software/imgProc/otsuThreshold.html (Accessed August 1, 2015) Wikipedia (2015): GNU General Public License. https://de.wikipedia.org/wiki/GNU_General_Public_License (Accessed August 5, 2015) Wikipedia (2015): Peak signal-to-noise ratio. https://en.wikipedia.org/wiki/Peak_signal-to- noise_ratio (Accessed August 5, 2015) Wikipedia (2013): ShotCode. https://de.wikipedia.org/wiki/ShotCode (Accessed June 7, 2015) Wikipedia (2014): Shotcode. https://en.wikipedia.org/wiki/ShotCode (Accessed June 7, 2015) ZXing (2014): License Questions. https://github.com/zxing/zxing/wiki/License-Questions (Accessed August 5, 2015)
34 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
11 Academic honesty statement Hiermit bestätigt der Autor, diese Arbeit ohne fremde Hilfe und unter Einhaltung der gebotenen Re- geln erstellt zu haben.
Robin Brügger
______Ort, Datum Unterschrift
35 / 36 Bachelor Thesis (IP6) Ring-shaped barcodes Robin Brügger
12 Appendix
12.1 Glossary
12.1.1 Module A module of the barcode is the smallest chunk of information a bar- code stores. Depending on the barcode resolution, it can consist of multiple pixels, all of which are either white or black. Illustration 42 shows a barcode in which a single module has been coloured red in- stead of black. Illustration 41: A single module is 12.2 Software used coloured red
12.2.1 Libraries Library Version OpenCV 2.4.10 Rscode (http://rscode.sourceforge.net) 1.3
12.2.2 Others Software Version IDE Visual Studio 2012 Ultimate Operating system Windows 7 64bit
36 / 36