Recognition of Perspective Distorted QR Codes with a Partially Damaged Finder Pattern in Real Scene Images

applied sciences

Article Recognition of Perspective Distorted QR Codes with a Partially Damaged Finder Pattern in Real Scene Images

Ladislav Karrach 1 , Elena Pivarˇciová 1,* and Pavol Bozek 2

1 Department of Manufacturing and Automation Technology, Faculty of Technology, Technical University in Zvolen, Masarykova 24, 960 01 Zvolen, Slovakia; [email protected] 2 Institute of Production Technologies, Faculty of Materials Science and Technology, Slovak University of Technology in Bratislava, Vazovova 5, 811 07 Bratislava, Slovakia; [email protected] * Correspondence: [email protected]

Received: 21 September 2020; Accepted: 30 October 2020; Published: 4 November 2020

Featured Application: Mobile robot navigation, automatic object identiﬁcation and tracking.

Abstract: QR (Quick Response) codes are one of the most famous types of two-dimensional (2D) matrix barcodes, which are the descendants of well-known 1D barcodes. The mobile robots which move in certain operational space can use information and landmarks from environment for navigation and such information may be provided by QR Codes. We have proposed algorithm, which localizes a QR Code in an image in a few sequential steps. We start with image binarization, then we continue with QR Code localization, where we utilize characteristic Finder Patterns, which are located in three corners of a QR Code, and ﬁnally we identify perspective distortion. The presented algorithm is able to deal with a damaged Finder Pattern, works well for low-resolution images and is computationally eﬃcient.

Keywords: QR code detection; adaptive thresholding; ﬁnder pattern; perspective transformation

1. Introduction Some of the requirements that are placed on autonomous mobile robotic systems include real-world environments navigation and recognition and identification of objects of interest with which the robotic system have to interact. Computer vision allows machines to obtain a large amount of information from the environment that has a major impact on their behavior. In the surrounding environment there are often numerous different static objects (walls, columns, doors, and production machines) but also moving objects (people, cars, and handling trucks). Objects that are used to refine navigation, landmarks, can be artificial (usually added by a human) and natural (which naturally occur in the environment) [1,2]. Robotic systems have to work in a real environment and must be able to recognize, for example, people [3], cars [4], product parameters for the purpose of quality control, or objects which are to be handled. The use of QR (Quick Response) codes (two-dimensional matrix codes) in interaction with robots can be seen in the following areas:

in the field of navigation as artificial landmarks-analogies of traffic signs that control the movement • of the robot in a given area (no entry, driving direction, alternate route, and permitted hours of operation) or as an information board providing context specific information or instructions (such as identification of floor, room, pallets, and working place)

Appl. Sci. 2020, 10, 7814; doi:10.3390/app10217814 www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, 7814 2 of 16 Appl. Sci. 2020, 10, x FOR PEER REVIEW 2 of 17

 in thethe areaarea ofof object object identiﬁcation, identification, 2D 2D codes codes are are often often used used to markto mark products products and and goods goods and thusand • thustheir recognitiontheir recognition will provide will provide information information about the about type ofthe goods type (warehouses), of goods (warehouses), the destination the destinationof the shipment of the (sorting shipment lines) (sorting or control lines) and or control tracking and during tracking track-and-trace. during track-and-trace.

QR Codes QRQR Codes are classified among 2D matrix codes (similar to Data Matrix Matrix codes). codes). QR Codes (Model 1 1 and and Model Model 2) 2)are are squared-shaped squared-shaped 2D matrices 2D matrices of dark of and dark light and squares—so light squares called—so modules. called modules.Each module Each represents module represents the binary t 1he or binary 0. Each 1 QRor 0. Code Each has QR ﬁxed Code parts has (suchfixed asparts Finder (such Patterns as Finder and PatternsTiming Patterns) and Timing that are Patterns) common that to each are common QR Code to and each variable QR Code parts that and di variableﬀer according parts that to the differ data accordingthat is encoded to the bydata a QR that Code. is encoded Finder by Patterns, a QR C whichode. Finder are located Patterns, in the which three are corners located of in a QRthe Code,three cornersare important of a QR for Code, determining are important the position for determining and rotation the ofposition the QR and Code. rotation The size of the of theQR QRCode code. The is sizedetermined of the QR by code the number is determined of modules by the and number can vary of frommodules 21 and21 modulescan vary (Version from 21 1)× to21 177modules177 × × (Version 40).1) to Each 177 × higher 177 (Version version 40). number Each compriseshigher version four number additional comprises modules four per additional side. In the modules Figure1 perwe canside. see In a the sample Figure of 1 version we can 1 see QR a Code.sample of version 1 QR Code.

Figure 1. Version 1: 21 21 QR Code. Figure 1. Version 1: 21 × 21 QR Code. QR Code has error correction capability to restore data if the code is partially damaged. Four error QR Code has error correction capability to restore data if the code is partially damaged. Four correction levels are available (L–Low, M–Medium, Q–Quartile, and H–High). The error correction error correction levels are available (L–Low, M–Medium, Q–Quartile, and H–High). The error level determines how much of the QR Code can be corrupted to keep the data still recoverable (L–7%, correction level determines how much of the QR Code can be corrupted to keep the data still M–15%, Q–25%, and H–30%) [5]. The QR Code error correction feature is implemented by adding recoverable (L–7%, M–15%, Q–25%, and H–30%) [5]. The QR Code error correction feature is a Reed–Solomon Code to the original data. The higher the error correction level is the less storage implemented by adding a Reed–Solomon Code to the original data. The higher the error correction capacity is available for data. level is the less storage capacity is available for data. Each QR Code symbol version has the maximum data capacity according to the amount of data, Each QR Code symbol version has the maximum data capacity according to the amount of data, character type and the error correction level. The data capacity ranges from 10 alphanumeric character type and the error correction level. The data capacity ranges from 10 alphanumeric (or 17 (or 17 numeric) characters for the smallest QR Code up to 1852 alphanumeric (or 3057 numeric) numeric) characters for the smallest QR Code up to 1852 alphanumeric (or 3057 numeric) characters characters for the largest QR Code at highest error correction level [5]. QR Codes support four encoding for the largest QR Code at highest error correction level [5]. QR Codes support four encoding modes—numeric, alphanumeric, Kanji and binary—to store data eﬃciently. modes—numeric, alphanumeric, Kanji and binary—to store data efficiently. QR Code was designed in 1994, in Japan for automotive industry, but currently has a much wider use. QR Code was designed in 1994, in Japan for automotive industry, but currently has a much QR codes are used to mark a variety of objects (goods, posters, monuments, locations, and business wider use. QR codes are used to mark a variety of objects (goods, posters, monuments, locations, cards) and allow to attach additional information to them, often in the form of a URL to a web page. and business cards) and allow to attach additional information to them, often in the form of a URL to QR Code is an ISO standard (ISO/IEC 18004:2015) and is freely available without license fees. a web page. QR Code is an ISO standard (ISO/IEC 18004:2015) and is freely available without license In addition to a traditional QR Code Model 1 and 2, there are also variants such as a Micro QR fees. Code (a smaller version of the QR Code standard for applications where a symbol size is limited) or an In addition to a traditional QR Code Model 1 and 2, there are also variants such as a Micro QR iQR Code (which can hold a greater amount of information than a traditional QR Code and it supports Code (a smaller version of the QR Code standard for applications where a symbol size is limited) or also rectangular shapes) (Figure2). an iQR Code (which can hold a greater amount of information than a traditional QR Code and it supports also rectangular shapes) (Figure 2).

Appl. Sci. 2020, 10, x FOR PEER REVIEW 3 of 17

Appl. Sci. 2020, 10, 7814 3 of 16 Appl. Sci. 2020, 10, x FOR PEER REVIEW 3 of 17

Figure 2. Other types of a QR Code: (a) Micro QR code, (b) iQR code.

2. Related Work Figure 2. Other types of a QR Code: ( a)) Micro Micro QR code, ( b) iQR code code.. 2. RelatedPrior Workpublished approaches for recognizing QR Codes in images can be divided into Finder 2.Pattern Related based Work locatio n methods [6–12] and QR Code region based location methods [13–18]. The first Prior published approaches for recognizing QR Codes in images can be divided into Finder groupPrior locates published a QR Code approaches based on for the recognizing location of QR its Codestypical inFinder images Patterns can be that divided are present into Finder in its Pattern based location methods [6–12] and QR Code region based location methods [13–18]. The ﬁrst Patternthree corners. based locatioThe secondn methods group [6 –locates12] and the QR area Code of region a QR codebased in loca thetion image methods based [1 on3– 1its8]. irregularThe first group locates a QR Code based on the location of its typical Finder Patterns that are present in its groupcheckerboard locates -alike QR structure Code based (a QRon the Code location consists of its of typical many Finder small lightPatterns and that dark are squares present whichin its three corners. The second group locates the area of a QR code in the image based on its irregular threealternate corners. irregularly The second and are group relatively locates close the to area each of other a QR). code in the image based on its irregular checkerboard-like structure (a QR Code consists of many small light and dark squares which alternate checkerboardA shape -oflike the structure Finder Pattern (a QR ( Figure Code 3) consists was deliberately of many chosen small light by the and authors dark osquaresf the QR whichCode, irregularly and are relatively close to each other). alternatebecause “it irregularly was the patternand are leastrelatively likely close to appear to each on other various). business forms and the like” [6]. They A shape of the Finder Pattern (Figure3) was deliberately chosen by the authors of the QR Code, foundA shapeout that of the black Finder and Pattern white areas(Figure that 3) was alternate deliberately in a 1:1:3:1:1 chosen ratioby the are authors the least of the common QR Code, on because “it was the pattern least likely to appear on various business forms and the like” [6]. They found becauseprinted materials.“it was the pattern least likely to appear on various business forms and the like” [6]. They out that black and white areas that alternate in a 1:1:3:1:1 ratio are the least common on printed materials. found out that black and white areas that alternate in a 1:1:3:1:1 ratio are the least common on printed materials.

Figure 3. Finder Pattern. Figure 3. Finder Pattern. In [7] (Lin and Fuh) all points matching the 1:1:3:1:1 ratio, horizontally and vertically, are collected. CollectedIn [7] points (Lin and belonging Fuh) all to one points Finder matching Pattern the are 1:1:3:1:1 merged. ratio Inappropriate, horizontally points and are vertically filtered, outare Figure 3. Finder Pattern. accordingcollected. to Collected the angle points between belonging three of to them. one Finder Pattern are merged. Inappropriate points are filteredIn [[7]out8] (Li (Linaccording et and al.) Fuh) to minimal the all angle points containing between matching three region the of them is 1:1:3:1:1 established. ratio, analyzinghorizontally five and runs vertically in labeled, are connectedcollectedIn [8]. Collected components, (Li et al.) points minimal which belonging are containing compacted to one region Finder using is Pattern run-length established are merged. coding.analyzing Inappropriate Second, five runs coordinates pointsin labeled are of centralfilteredconnected Finderout componentsaccording Pattern to in, the which a QR angle Codeare between comp are calculatedacted three ofusing them by usingrun. -length run-length coding. coding Second, utilizing coordinates modified of Knuth–Morris–PrattcentralIn Finder [8] (Li Pattern et al.) algorithm. inminimal a QR Code containing are calculated region isby established using run- lengthanalyzing codin fiveg utiliz runsing in modified labeled connectedKnuthIn– [Morris9] (Belussi components–Pratt and algorithm, Hirata) which. two-stageare comp detectionacted using approach run-length is proposed. coding. InSecond, the first coordinates stage Finder of PatterncentralIn (locatedFinder[9] (Belussi Pattern at three and in cornersHirata) a QR ofCodetwo a QR-stage are Code) calculated detection is detected byapproach using using run ais cascaded proposed.-length classifiercodin In theg utiliz trainedfirsting stage accordingmodified Finder toKnuthPattern the– rapid Morris (located object–Pratt at detection threealgorithm corners method. of a (Viola–Jones QR Code) is framework). detected using In the a cascaded second stage classifier geometrical trained restrictionsaccordingIn [9] (Belussi to among the rapidand detected Hirata) object components two detection-stage aredetection method verified (Violaapproach to– decideJones is proposed. whetherframework subsets In). the In thefirst of three secondstage of Finder them stage correspondPatterngeometrical (located torestrictions a QR at Code three among or corners not. detected of a QRcomponents Code) is are detected verified using to decide a cascaded whether classifier subsets of trained three accordingof themIn [ 10correspond ] to (Bodn the ár rapid andto a NyQR objectú Codel) Finder detection or not. Pattern method candidate (Viola localization–Jones framework is based on). the In cascade the second of boosted stage weakgeometricalIn classifiers [10] restrictions (Bodnár using Haar-likeand among Nyúl) features,detected Finder whilecomponents Pattern the candidate decision are verified on localization a Finder to decide Pattern is basedwhether candidate on subsets the to cascade be of kept three or of droppedofboosted them correspondweak is decided classifiers byto a geometricalQRusing Code Haar or- constraintlikenot. features, on distanceswhile the anddecision angles on with a Finder respect Pattern to other candidate probable to Finderbe keptIn Patterns. [10]or dropped (Bodnár In isadditionand decided Nyúl) to by Haar-likeFinder a geometrical Patte features,rn candidateconstraint local binary localizationon distances patterns isand based(LBP), angles on and with the histogram cascade respect ofto orientedboostedother probable weak gradients classifiers Finder (HOG) Patterns using based Haar. In classifiers -additionlike features, are to usedHaar while and-like thetrained features, decision to local Finderon a Finderbinary Patterns Patternpatterns and candidate whole (LBP), code and to areasbehistogram kept as or well. ofdropped oriented is gradientsdecided by (HOG a geometrical) based classifiers constraint are onused distances and trained and anglesto Finder with Patterns respect and to otwholeherIn probable code [11] (Tribakareas Finder as and well Patterns Zaz). successive. In addition horizontal to Haar and-like vertical features, scans local are launched binary patterns to obtain (LBP segments), and whosehistogramIn structure [11] of (Tribakoriented complies andgradients withZaz) the(successHOG ratio) basedive 1:1:3:1:1. horizontal classifiers The intersection andare used vertical and between scanstrained are horizontal to Finder launched Patterns and to vertical obtain and wholesegments code whose areas structureas well. complies with the ratio 1:1:3:1:1. The intersection between horizontal and In [11] (Tribak and Zaz) successive horizontal and vertical scans are launched to obtain segments whose structure complies with the ratio 1:1:3:1:1. The intersection between horizontal and Appl. Sci. 2020, 10, 7814 4 of 16

Appl. Sci. 2020, 10, x FOR PEER REVIEW 4 of 17 segments presents the central pixel of the extracted pattern. All the extracted patterns are transmitted tovertical a filtering segments process presents based on the principal central pixel components of the extracted analysis, pattern. which isAll used the asextracted a pattern patterns feature. are transmittedIn [12] (Tribak to a filtering and Zaz) process seven based Hu invariant on principal moments components are applied analysis to the, which Finder is used Pattern as a candidates pattern feature. obtained by initial scanning of an image and using Euclidean metrics they are compared with Hu In [12] (Tribak and Zaz) seven Hu invariant moments are applied to the Finder Pattern moments of the samples. If the similarity is less than experimentally determined threshold, then the candidates obtained by initial scanning of an image and using Euclidean metrics they are compared candidate is accepted. with Hu moments of the samples. If the similarity is less than experimentally determined threshold, In [13] (Sun et al.) authors introduce algorithm, which aims to locate a QR Code area by four then the candidate is accepted. cornersIn detection [13] (Sun ofet 2Dal.) barcode.authors introduce They combine algorithm, the which Canny aims edge to detector locate a QR with Code external area by contours four findingcorners algorithm. detection of 2D barcode. They combine the Canny edge detector with external contours findingIn [14 algorithm.] (Ci ˛a˙zyńskiand Fabijańska)they use histogram correlation between a reference image of a QR code and an input image divided into a block of size 30 30. Then candidate blocks are joined In [14] (Ciążyński and Fabijańska) they use histogram correlation× between a reference image of intoa QR regions code andand morphologicalan input image erosiondivided andinto dilationa block of is size applied 30 × to30. removeThen candidate small regions. blocks are joined intoIn regions [15] (Gaur and andmorphological Tiwari) they erosion propose and approachdilation is whichapplied uses to remove Canny small edge regions. detection followed by morphologicalIn [15] (Gaur dilation and Tiwari) and erosion they propose to connect approach broken which edges uses in aCanny QR Code edge intodetection a bigger followed connected by component.morphological They dilation expect and that erosion the QR to Code connect is the broken biggest edges connected in a QR component Code into ina bigger the image. connected component.In [16,17] They (Szentandr expect thatási etthe al.) QR Code they exploitis the biggest the property connected of component 2D barcodes in the of image. having regular distributionIn [16,17] of edge (Szentandrási gradients. et They al.) they split exploit a high-resolution the property image of 2D into barcodes tiles and of forhaving each regular tile they constructdistribution HOG of (histogram edge gradients. of oriented They gradients)split a high from-resolution the orientations image into of edgetiles and points. for Theneach tile they they select construct HOG (histogram of oriented gradients) from the orientations of edge points. Then they two dominant peeks in the histogram, which are apart roughly 90◦. For each tile, a feature vector is computed,select two which dominant contains peeks a normalizedin the histogram, histogram, which angles are apart of two roughly main gradient90°. For each directions, tile, a afeature number ofvector edge pixels is computed, and an estimationwhich contains of probability a normalized score histogram, of a chessboard-like angles of two structure. main gradient directions, a number of edge pixels and an estimation of probability score of a chessboard-like structure. In [18] (Sörös and Flörkemeier) areas with high concentration of edge structures as well as areas In [18] (Sörös and Flörkemeier) areas with high concentration of edge structures as well as areas with high concentration of corner structures are combined to get QR Code regions. with high concentration of corner structures are combined to get QR Code regions. 3. The Proposed Method 3. The Proposed Method Our method is primarily based on searching for Finder Patterns and utilizes their characteristic Our method is primarily based on searching for Finder Patterns and utilizes their characteristic feature—1:1:3:1:1 black and white point ratio in any scanning direction. The basic steps are indicated feature—1:1:3:1:1 black and white point ratio in any scanning direction. The basic steps are indicated in the flowchart in Figure4. in the flowchart in Figure 4.

FigureFigure 4. 4. TheThe ﬂowflow chart of proposed proposed algorithm algorithm.. Appl. Sci. 2020, 10, 7814 5 of 16

Before searching for a QR Code, the original image (maybe colored) is converted to a gray scale image using Equation (1), because the color information does not bear any signiﬁcant additional information that might help in QR Code recognition.

77R + 151G + 28B I = (1) 256 where I stands for gray level and R, G, B for red, green, and blue color intensities of individual pixels in the RGB model, respetively. This RGB to gray scale conversion is integer approximation of widely used luminance calculation as deﬁned in recommendation ITU-R BT.601-7:

I = 0.299R + 0.587G + 0.114B (2)

Next, the gray scaled image is converted to a binary image using modified adaptive thresholding (with the size of window 35—the window size we choose to be at least five times the size of expected size of QRC module) [19]. We expect that black points, which belong to QRC, will become foreground points. We use the modification of the well-known adaptive thresholding technique (Equation (3)), which calculates individual threshold for every point in the image. This threshold is calculated using average intensity of points under a sliding window. To speed up the thresholding we pre-calculate the integral sum image and we also use the global threshold value (points with intensity above 180 we always consider as background points). Adaptive thresholding can successfully threshold also uneven illuminated images.   0 I(x, y) > 180  ← B(x, y) =  0 I(x, y) >= T(x, y) (3)  ←  1 I(x, y) < T(x, y) ← I(x, y) T(x, y) = m(x, y) 10 − 10 − 17 17 1 X X m(x, y) = I(x + i, y + j) 35 35 i= 17 j= 17 × − − where I is gray scale (input) image, B is binary (output) image, T is threshold value (individual for each pixel at coordinates x, y), and m is average of pixel intensities under sliding window of the size 35 35 pixels. × In order to improve adaptive thresholding results, some of the image pre-processing techniques, such as histogram equalization, contrast stretching or deblurring, are worth to consider.

3.1. Searching for Finder Patterns First, the binary image is scanned from top to bottom and from left to right, and we look for successive sequences of black and white points in a row matching the ratios 1:1:3:1:1 (W1:W2:W3:W4:W5 where W1, W3, W5 indicates the number of consecutive black points which are alternated by W2, W4 white points) with small tolerance (tolerance is necessary due to imperfect thresholding and noise in the Finder Pattern area, black and white points in a line do not alternate in ideal ratios 1:1:3:1:1):

W , W , W , W w 1.5, w + 2.0 1 2 4 5 ∈ h − i W 3w 2, 3w + 2 , where 3 ∈ h − i W max(W + W , W + W ) 3 ≥ 1 2 4 5 (4) W + W + W + W + W W w = 1 2 3 4 5 = 7 7 Appl.Appl. Sci. Sci. 20 2020, 10, 10, x, FORx FOR PEER PEER REVIEW REVIEW 6 6of of 17 17

W W W W W Appl. Sci. 2020, 10, 7814 W1 1 W2 2 W3 3 W4 4 W5 5 WW 6 of 16 ww  77 77

1 2 3 4 5 ForForFor each eacheach match matchmatch in inin a a rowa row row coordinates coordinates coordinates of of of Centroid Centroid Centroid ( C ()C and ) and Width Width ((WW (W = W= W1++1 W+W W+22 + W+ W+3 W++ WW+4 W+4 +W) W5of) 5of ) theofthe sequence the sequence sequence (of (of black (of black black and and andwhite white white poi points)nts) points) are are stored are stored stored in in ain lista list a listof of Finderof Finder Finder Pattern Pattern Pattern candidates candidates candidates (Figure (Figure (Figure 5). 5). 5).

Figure 5. A Finder Pattern candidate matching 1:1:3:1:1 horizontally. FigureFigure 5. 5.A AFinder Finder Pattern Pattern candidate candidate matching matching 1:1:3:1:1 1:1:3:1:1 horizontally horizontally. . Then, Finder Pattern candidates (from the list of candidates) that satisfy the following criteria ThenThen, Finder, Finder Pattern Pattern candidates candidates (from (from the the list list of of candidates candidates) that) that satisfy satisfy the the following following criteria criteria are grouped: areare grouped grouped: : their centroids C are at most 3/7W points vertically and at most 3 points horizontally away from  •• theirtheir centroids centroids C Care are at at most most 3/7 3/7WW points points vertically vertically and and at at most most 3 points3 points horizontally horizontally away away from from each other, eacheach other other, , their widths W does not diﬀer by more than 2 points.  •• theirtheir widths widths W W does does not not differ differ by by more more than than 2 points2 points. . WeWeWe expect, expect,expect, that thatthat the thethe Finder FinderFinder Pattern PatternPattern candidates candidatescandidates in inin one oneone group groupgroup belong belongbelong to toto the thethe same samesame Finder FinderFinder Pattern PatternPattern andandand th thereforethereforeerefore we wewe set setset the thethe new newnew centroid centroidcentroid CC Candand and width widthwidth W WW of ofof the thethe group groupgroup as asas average averageaverage of ofof x, x xy,, ycoordinatesy coordinatescoordinates and andand widthswidthswidths of ofof the thethe nearby nearbynearby Finder FinderFinder Pattern PatternPattern candidates candidatescandidates (Figure(Figure (Figure 6a)6 a).6a). .

FigureFigureFigure 6. 6. 6.(a ()(a Group)) Group Group of of of 8 8 8Finder Finder Finder Pattern Pattern Pattern candidates candidates candidates matching matching matching 1:1:3:1:1 1:1:3:1:1 1:1:3:1:1 in in in rows rows, rows, (,b (()bb )Finder) FinderFinder Pattern Pattern Pattern cacandidatecandidatendidate matching matchingmatching 1:1:3:1:1 1:1:3:1:11:1:3:1:1 vertically vertically.vertically. .

AfterAfterAfter grouping groupinggrouping the thethe Finder FinderFinder Pattern Patterns,Patterns, sit, it itmust mustmust be bebe verif veriﬁedverifiedied whether whetherwhether there therethere are areare sequences sequencessequences of ofof black blackblack and andand whitewhitewhite points points points also also inin thein the the vertical vertical vertical direction, direction, direction, which which which alternate alternate alternate in the in ratio in the the 1:1:3:1:1 ratio ratio 1:1:3:1:1 (Figure 1:1:3:1:1 6(b). Figure(Figure A bounding 6b) 6b). A. A boundingboxbounding around box box the around around Finder the Patternthe Finder Finder candidate, Pattern Pattern candidate in candidate which vertical, in, in which which sequences vertical vertical aresequences sequences looked for,are areis looked deﬁnedlooked for asfor, is, is defineddefined as as D E x Cx 1.3/7W , y Cy 5.5/7W (5) ∈ h ± i ∈ ± , y  C  5.5 7W x x CCx x 1.13.37W7W , y  Cy y  5.5 7W (5()5 ) where C (Cx, Cy) is a centroid and W is width of the Finder Pattern candidate. We work with a slightly larger bounding box in case the Finder Pattern is stretched vertically. Candidates, where no vertical match is found or where the ratio H/W < 0.7, are rejected. For candidates, where a vertical match Appl. Sci. 2020, 10, x FOR PEER REVIEW 7 of 17

where C (Cx, Cy) is a centroid and W is width of the Finder Pattern candidate. We work with a Appl.slightly Sci. 2020 larger, 10, 7814bounding box in case the Finder Pattern is stretched vertically. Candidates, where7 of 16no vertical match is found or where the ratio H/W < 0.7, are rejected. For candidates, where a vertical match is found, the y coordinate of centroid C (Cy) is updated as an average of y coordinates of is found, the y coordinate of centroid C (C ) is updated as an average of y coordinates of centers of the centers of the vertical sequences. y vertical sequences. 3.2. Verification of Finder Patterns 3.2. Veriﬁcation of Finder Patterns Each Finder Pattern consists of a central black square with the side of 3 units (R1), surrounded Each Finder Pattern consists of a central black square with the side of 3 units (R1), surrounded by by a white frame with the width of 1 unit (R2), surrounded by a black frame with the width of 1 unit a white frame with the width of 1 unit (R2), surrounded by a black frame with the width of 1 unit (R3). (R3). In Figure 7 there are colored regions R1, R2 and R3 in red, blue, and green, respectively. For each In Figure7 there are colored regions R1, R2 and R3 in red, blue, and green, respectively. For each Finder Pattern candidate Flood Fill algorithm is applied, starting from the centroid C (which lies in Finder Pattern candidate Flood Fill algorithm is applied, starting from the centroid C (which lies in the the region R1) and continuing through white frame (region R2) to black frame (region R3). As region R1) and continuing through white frame (region R2) to black frame (region R3). As continuous continuous black and white regions are filled, following region descriptors are incrementally black and white regions are ﬁlled, following region descriptors are incrementally computed: computed: area (A = M00) • area (A = M00) centroid (Cx = M10/M00, Cy = M01/M00), where M00, M10, M00 are raw image moments • centroid (Cx = M10/M00, Cy = M01/M00), where M00, M10, M00 are raw image moments bounding box (Top, Left, Right, Bottom) • bounding box (Top, Left, Right, Bottom)

Figure 7. Regions of the Finder Pattern candidate. Figure 7. Regions of the Finder Pattern candidate. The Finder Pattern candidate, which does not meet all of the following conditions, is rejected. The Finder Pattern candidate, which does not meet all of the following conditions, is rejected.

Area(R1) < Area(R2) < Area(R3) and • Area(R1) < Area(R2) < Area(R3) and 1.1 < Area(R )/Area(R ) < 3.4 and 1.1 < Area(R22)/Area(R11) < 3.4 and 1.8 < Area(R )/Area(R ) < 3.9 1.8 < Area(R33)/Area(R11) < 3.9 0.7 < AspectRatio(R ) < 1.5 and • 0.7 < AspectRatio(R22) < 1.5 and 0.7< AspectRatio(R ) < 1.5 0.7 < AspectRatio(R3 3) < 1.5  |Centroid(R1),1 Centroid(R2)2| < 3.7 and • |Centroid(R ), Centroid(R )| < 3.7 and |Centroid(|Centroid(R1R),1), Centroid( Centroid(R3R)3|) <| <4.2 4.2 Note:Note: the the criteria criteria were were set to set be to invariant be invariant to the rotation to the rotation of the Finder of the Pattern, Finder and Pattern, the acceptance and the rangesacceptance were determinedranges were experimentally. determined experimentally. In an ideal undistorted In an ideal Finder undistorted Pattern, theFinder criteria Pattern are met, the ascriteria follows: are met as follows:

 Area(Area(RR)2/)/Area(Area(RR)1)= = 1616/9/9 == 1.8 and Area(R 3)/Area()/Area(R1)) == 24/924/9 == 2.72.7 • 2 1 3 1  AspectRatio(AspectRatioR(R)1)= = 11 andand AspectRatio(AspectRatio(RR2)) == 1 and AspectRatio AspectRatio((R3)) == 11 • 1 2 3  Centroid(Centroid(RR)1)= = Centroid(Centroid(RR2)) == Centroid(R3)) • 1 2 3 In real environments there can be damaged Finder Patterns. The inner black region R1 can be In real environments there can be damaged Finder Patterns. The inner black region R1 can be joined with the outer black region R3 (Figure 8a) or the outer black region can be interrupted or joined with the outer black region R3 (Figure8a) or the outer black region can be interrupted or incomplete (Figure8b). In the ﬁrst case the bounding box of the region R2 is completely contained by bounding box of the region R1 and in the second case is bounding box of the region R3 contained Appl. Sci. 2020, 10, x FOR PEER REVIEW 8 of 17 Appl. Sci. 2020, 10, 7814 8 of 16 incomplete (Figure 8b). In the first case the bounding box of the region R2 is completely contained by bounding box of the region R1 and in the second case is bounding box of the region R3 contained in in boundingbounding box of of the the region region R2.R These2. These cases cases are handle are handledd individually. individually. If the first If case the is ﬁrst detected case is, then detected, thenthe the region region R1R is1 isproportionally proportionally divided divided into into R1 andR1 andR3 andR3 ifand the if second the second case is case detected is detected then the then the region R2 is instantiated using the region R1 and R3. region R2 is instantiated using the region R1 and R3.

Figure 8. Various damages of Finder Patterns: (a) merged inner and outer black regions; (b) interrupted Figure 8. Various damages of Finder Patterns: (a) merged inner and outer black regions; (b) outerinterrupted black region. outer black region

TheT Centroidhe Centroid (C )(C and) and Module Module Width Width ( MW(MW)) of of thethe FinderFinder Pattern candidate candidate are are updated updated using using the regionthe descriptorsregion descriptors as follows: as follows:

M10(R11)) +MM1010((RR22)) +MM1010(R(R3 )3) MM0101((RR11)) +MM0101(R(R2 2)) +MM0101(R(R3 )3) C = ( ( ,, ) ) (6) (6) M ((R )) +M ((R )) +M ((R )) M ((R )) +M (R( )) +M (R( ) ) M00 R11 M0000 R22 M0000 R33 M0000 R11 M0000 R2 2 M0000 R3 3 q MW  M 00(R1 )  M 00(R2 )  M 00(R3 ) 7 MW = M00(R1) + M00(R2) + M00(R3)/7

3.3.3.3. Grouping Grouping of Finderof Finder Patterns Patterns

3.3.1.3.3.1. Grouping Grouping Triplets Triplets of of Finder FinderPatterns Patterns In theIn the previous previous steps, steps, Finder Finder Patterns Patterns in in the the image image werewere identified identified and and now now such such triplets triplets (from (from the list ofthe all list Finder of all Finder Patterns) Patterns) must must be selected, be selected, which which can can represent represent 33 corners of of a aQR QR Code. Code. Matrix Matrix of of the distances between the centroid of all Finder Patterns is build and all 3-element combinations the distances between the centroid of all Finder Patterns is build and all 3-element combinations of of all Finder Patterns are examined. For each triplet it is checked whether it is possible to construct a all Finder Patterns are examined. For each triplet it is checked whether it is possible to construct right-angled triangle from it so that the following conditions are met: a right-angled triangle from it so that the following conditions are met:  the size of each triangle side must be in predefined range the the size difference of each triangle in sizes of side two must legs must be in be predefined less than 21 range • the the diff differenceerence in in sizes size of of twothe real legs and must theoretical be less thanhypotenuse 21 must be less than 12 • theIn di ffthiserence way, in a sizelist of of QR the Code real candidates and theoretical (defined hypotenuse by a triplet must FP1, FP be2, less FP3) than is buil 12t. However, • such a candidate for a QR Code is selected only on the basis of the mutual position of the 3 FP In this way, a list of QR Code candidates (defined by a triplet FP1, FP2, FP3) is built. However, candidates. As is shown in Figure 9 not all QR Code candidates are valid (dotted red FP3′-FP3″-FP2″ is suchfalse a candidate positive). These for a false QR positive Code is QR selected Code candidates only on will the be basis eliminated of the in mutual the next position steps. of the 3 FP candidates. As is shown in Figure9 not all QR Code candidates are valid (dotted red FP -FP -FP is Appl. Sci. 2020, 10, x FOR PEER REVIEW 30 3” 9 2”of 17 false positive). These false positive QR Code candidates will be eliminated in the next steps.

Figure 9. QR Code candidates. Figure 9. QR Code candidates.

Finally, Bottom-Left and Top-Right Finder Pattern from the triplet (FP1, FP2, FP3) is determined by using formula: If (FP3.x − FP2.x)(FP1.y − FP2.y) − (FP3.y − FP2.y)(FP1.x − FP2.x) < 0 then Bottom-Left is FP3 and Top-Right is FP1 else vice versa.

3.3.2. Grouping Pairs of Finder Patterns If any QR Code has one of the 3 Finder Patterns significantly damaged, then this Finder Pattern might not be identified and there remains two Finder Patterns in the Finder Patterns list, that were not selected (as the vertices of a right-angled triangle) in the previous step (Figure 10a). The goal of this step is to identify these pairs and determine the position of the third missing Finder Pattern. A square shape of the QR Code is assumed. All two element combinations of remaining Finder Patterns, whose distance is in a predefined interval, are evaluated. A pair of Finder Patterns can represent Finder Patterns that are adjacent corners of a QR Code square (Figure 10a) or are in the opposite corners. If they are adjacent corners, then there are two possible positions of the QR Code (in Figure 10b depicted as a red square and green square). If they are opposite corners, then there are other two possible positions of the QR Code.

Figure 10. QR Code with one damaged Finder Pattern: (a) two known Finder Patterns; (b) two possible regions of QR Code; (c) two possible positions of 3rd Finder Pattern

All four possible positions of the potential QR Code are evaluated against the following criteria:  Is there a Quiet Zone around the bounding square at least 1 MW wide?  Is there a density of white points inside bounding square in interval (0.4; 0.65)?  Is there a density of edge points inside bounding square in interval (0.4; 0.6)? Density of edge points is computed as the ratio of the number of edge points to area*2/MW. Region of a QR Code is expected to have relative balanced density of white and black points and relative balanced ratio of edges to area. A square region that meets all the above conditions is considered a candidate for a QR Code. There are two possible corners in the QR Code bounding square where 3rd Finder Pattern can be located (Figure 10c). For both possible corners Finder Pattern match score is computed and one with better score is selected (in other words question, “In which corner is the structure that more closely resembles the ideal Finder Pattern?” must be answered). Match score is computed as Appl. Sci. 2020, 10, x FOR PEER REVIEW 9 of 17

Figure 9. QR Code candidates.

IFinally,f any QR Bottom-Left Code has one and of Top-Right the 3 Finder Finder Patterns Pattern significantly from the triplet damaged (FP1,, then FP2, this FP3 )Finder is determined Pattern mightby using not formula: be identified and there remains two Finder Patterns in the Finder Patterns list, that were not selectedIf (FP .x (as FPthe .x)(FPvertices.y of FPa right.y) -angled(FP .y triangle)FP .y)(FP in the.x previFP .x)ous< step0 then (Figure Bottom-Left 10a). The is FPgoaland of 3 − 2 1 − 2 − 3 − 2 1 − 2 3 thisTop-Right step is isto FP identify1 else vice these versa. pairs and determine the position of the third missing Finder Pattern. A square shape of the QR Code is assumed. 3.3.2.All Grouping two element Pairs combinations of Finder Patterns of remaining Finder Patterns, whose distance is in a predefined interval,If any are QR evaluated. Code has A one pair of the of Finder 3 Finder Patterns Patterns can signiﬁcantly represent damaged,Finder Patterns then this that Finder are adjacent Pattern cornersmight not of bea QR identiﬁed Code square and there (Figure remains 10a) twoor are Finder in the Patterns opposite in corners. the Finder If they Patterns are adjacent list, that werecorners not, thenselected there (as are the tw verticeso possible of a positions right-angled of the triangle) QR Code in the (in previous Figure 10b step depicted (Figure 10asa). a red The square goal of and this greenstep is square to identify). If they these are pairs opposite and determine corners, the then position there ofare the other third two missing possible Finder positions Pattern. of A the square QR Code.shape of the QR Code is assumed.

Figure 10. QR Code with one damaged Finder Pattern: (a) two known Finder Patterns; (b) two possible Figure 10. QR Code with one damaged Finder Pattern: (a) two known Finder Patterns; (b) two regions of QR Code; (c) two possible positions of 3rd Finder Pattern. possible regions of QR Code; (c) two possible positions of 3rd Finder Pattern All two element combinations of remaining Finder Patterns, whose distance is in a predeﬁned All four possible positions of the potential QR Code are evaluated against the following criteria: interval, are evaluated. A pair of Finder Patterns can represent Finder Patterns that are adjacent corners of a QRIs there Code a squareQuiet Zone (Figure around 10a) or the are bounding in the opposite square corners. at least If1 MW they wide? are adjacent corners, then there are twoIs there possible a density positions of white of the points QR Code inside (in bounding Figure 10 squareb depicted in interval as a red (0.4; square 0.65) and? green square). If theyIs there are opposite a density corners, of edge then points there inside are other bounding two possible square in positions interval of (0.4 the; 0.6 QR)? Code. DensityAll four of possible edge points positions is computed of the potential as the ratio QR Code of the are number evaluated of edge against points the to following area*2/MW criteria:. Region of a QR Code is expected to have relative balanced density of white and black points Is there a Quiet Zone around the bounding square at least 1 MW wide? • and relative balanced ratio of edges to area. Is there a density of white points inside bounding square in interval (0.4; 0.65)? • AIs theresquare a densityregion that of edge meets points all the inside above bounding conditions square is considered in interval (0.4;a candidate 0.6)? for a QR Code. There• are two possible corners in the QR Code bounding square where 3rd Finder Pattern can be locatedDensity (Figure of 10c). edge For points both is possible computed corners as the Finder ratio of Pattern the number match of score edge is points computed to area*2 and/ MWone .with betterRegion score is of selected a QR Code (in isother expected words to question have relative, “In which balanced corner density is the of structure white and that black more points closely and resemblesrelative balanced the ideal ratio Finder of edges Pattern? to area.” must be answered). Match score is computed as A square region that meets all the above conditions is considered a candidate for a QR Code. There are two possible corners in the QR Code bounding square where 3rd Finder Pattern can be located (Figure 10c). For both possible corners Finder Pattern match score is computed and one with better score is selected (in other words question, “In which corner is the structure that more closely resembles the ideal Finder Pattern?” must be answered). Match score is computed as

MS = argmin(OS + min(BS, WS)) (7) where MS is match score (lower is better), OS is overall pattern match score, BS is black module match score and WS is white module match score. BS stores matches only for expected black points and WS stores matches only for expected white points between the mask and the image. BS and WS was introduced to handle situations when over the area of Finder Pattern is placed black or white spot, which would cause a low match score if only a simple pattern matching technique were used. The match score is computed for several Finder Pattern mask positions by moving the mask in a spiral from its initial position up to a radius of MW with a step of MW/2 (for the case of small geometric deformations of the QR Code). Appl. Sci. 2020, 10, x FOR PEER REVIEW 10 of 17

MS  argmin OS  min( BS ,WS)   (7) where MS is match score (lower is better), OS is overall pattern match score, BS is black module match score and WS is white module match score. BS stores matches only for expected black points and WS stores matches only for expected white points between the mask and the image. BS and WS was introduced to handle situations when over the area of Finder Pattern is placed black or white spot, which would cause a low match score if only a simple pattern matching technique were used. The match score is computed for several Finder Pattern mask positions by moving the mask in a Appl. Sci.spiral2020 from, 10, 7814 its initial position up to a radius of MW with a step of MW/2 (for the case of small10 of 16 geometric deformations of the QR Code).

3.4. Verification3.4. Verification of Quiet of Quiet Zone Zone AccordingAccording to ISOto ISO standard standard a a Quiet Quiet Zone Zone isis defineddefined as as “a “a region region 4X 4X wide wide which which shall shall be free be of free of all otherall other markings, markings, surrounding surrounding thethe symbol on on all all four four sides”. sides”. So, it So,must it be must checked be checked if there are if only there are onlywhite points points in in the the image image in in the the rectangular rectangular area areass wide wide 1MW 1MW whichwhich is parallel is parallel to line to linesegments segments defined by FP1–FP2 and FP2–FP3 (Figure 11). For fast scanning of the rectangle points Bresenham’s defined by FP1–FP2 and FP2–FP3 (Figure 11). For fast scanning of the rectangle points Bresenham’s line algorithm is utilized [20]. line algorithm is utilized [20].

Figure 11. Quiet Zones. Figure 11. Quiet Zones. QR Code candidates which do not have quiet zones around are rejected. Rejected are also QR Code candidates which do not have quiet zones around are rejected. Rejected are also QR QR Code candidates whose outer bounding box (larger) contains outer bounding box (smaller) of Code candidates whose outer bounding box (larger) contains outer bounding box (smaller) of anotheranother QR CodeQR Code candidate. candidate.

3.5. QR3.5. CodeQR Code Bounding Bounding Box Box CentroidsCentroids of the of 3the Finder 3 Finder Patterns Patterns (which (which represent represent the the QR QR CodeCode candidate)candidate) are are the the v verticesertices of of the trianglethe triangle FP1–FP FP2–FP1–FP3.2– ThisFP3. T innerhis inner triangle triangle must must be be expanded expanded to outer outer triangle triangle P1– PP12–P–P23,–P where3, where the the armsarms of the of triangle the trian passgle pass through through boundary boundary modules modules of of the the QR QR Code Code (Figure (Figure 12 12).). ForFor instance,instance, the shift shift of FP3 to P3 may be expressed as of FP3 to P3 may be expressed as

FP31 FP P FPFP3 FP1MW 18 (8) P333= FP3 + − MW √18 (8) FPFP31 ,FP, FP | 3 1| wherewhereMWAppl. MWSci.is module20 is20 , module10, x FOR width PEERwidth (EquationREVIEW (Equation (6)) (6)) 11 of 17

Figure 12. Figure 12. Bounding Box Box..

3.6. Perspective3.6. Persp Distortionective Distortion For perspectiveFor perspective undistorted undistorted QR QR Codes Codes (only(only shifted, shifted, scaled, scaled, rotated rotated,, or sheared) or sheared) it is sufficient it is su ﬃcient to haveto only have 3 only points 3 points to set-up to set-up the the aﬃ affinene transformation transformation from from a square a square to destination to destination parallelogram. parallelogram. However,However, for perspective for perspective (projective) (projective) distorted distorted QRQR Codes 4 4points points are arerequired required to set to-up set-up perspective perspective transformationtransformation from from a square a square to destinationto destination quadrilateral quadrilateral [21 [21]. ]. Some authors (for example [7,22]) search for Alignment Pattern to obtain 4th point. However, Some authors (for example [7,22]) search for Alignment Pattern to obtain 4th point. However, version 1 version 1 QR Codes does not have Alignment Pattern, so we have decided not to rely on Alignment QR CodesPatterns. does not have Alignment Pattern, so we have decided not to rely on Alignment Patterns. Instead of that we use an iterative approach to find the opposite sides to P2–P1 and P2–P3, which aligns to QR Code borders.

1. We start from initial estimate of P4 as an intersection of the line L1 and L3, where L1 is parallel to P2–P3 and L3 is parallel to P2–P1 (Figure 13a). 2. We count the number of pixels which are common to the line L3 and the QR Code for each third of the line L3 (Figure 13a). 3. We shift the line L3 by width of one module away from the QR Code and again we count the number of pixels which are common to the shifted line L3′ and the QR Code. The module width is estimated as MW (from Equation (6)) (Figure 13b). 4. We compare overlaps of the line L3 from the step 2 and 3, and

a. If L3 was whole in the QR Code and the shifted L3′ is out of the QR Code, then initial estimation of P4 is good and we end. b. If L3 was whole in the QR Code and 3rd third of the shifted L3′ is again in the QR Code, then we continue by step 5. c. If 3rd third of L3 was in quiet zone and 2nd and 3rd third of the shifted L3′ is in the quiet zone or if 2nd third of L3 was in the quiet zone and 1st and 2nd third of the shifted L3′ is in the quiet zone then we continue by step 6.

5. We start to move P4 end of line segment P3–P4 away from the QR Code until 3rd third of L3 touches the quiet zone (Figure 13c). 6. We start to move P4 end of line segment P3–P4 towards the QR Code until 3rd third of L3 touches the QR Code. 7. We apply the same procedure also to the line L1 like for the line L3. 8. The intersection of the shifted lines L1 and L3 is a new P4 position. Appl. Sci. 2020, 10, 7814 11 of 16

Instead of that we use an iterative approach to ﬁnd the opposite sides to P2–P1 and P2–P3, which aligns to QR Code borders.

number of pixels which are common to the shifted line L30 and the QR Code. The module width is estimated as MW (from Equation (6)) (Figure 13b).

4. We compare overlaps of the line L3 from the step 2 and 3, and

a. If L3 was whole in the QR Code and the shifted L30 is out of the QR Code, then initial estimation of P4 is good and we end.

b. If L3 was whole in the QR Code and 3rd third of the shifted L30 is again in the QR Code, then we continue by step 5.

c. If 3rd third of L3 was in quiet zone and 2nd and 3rd third of the shifted L30 is in the quiet

zone or if 2nd third of L3 was in the quiet zone and 1st and 2nd third of the shifted L30 is in the quiet zone then we continue by step 6.

5. We start to move P4 end of line segment P3–P4 away from the QR Code until 3rd third of L3 touches the quiet zone (Figure 13c).

6. We start to move P4 end of line segment P3–P4 towards the QR Code until 3rd third of L3 touches the QR Code.

7. We apply the same procedure also to the line L1 like for the line L3. 8. The intersection of the shifted lines L1 and L3 is a new P4 position. Appl. Sci. 2020, 10, x FOR PEER REVIEW 12 of 17

Figure 13.Figure Perspective 13. Perspective distortion distortion:: (a) (a initial) initial estimateestimate of of the the point point P4 and P lines4 and L1 lines,L3;(b L) ﬁrst1, L3 shift; (b) of first the shift of line L ;(c) second shift of the line L . the line L3; (c)3 second shift of the line L3 3

Once the position of 4th point, P4, is obtained, perspective transformation from the source square Oncerepresenting the position the ideal of QR4th Code point, to destination P4, is obtained, quadrilateral perspective representing thetransformation real QR Code in from the image the source square representingcan be set-up (Figure the ideal 14). QR Code to destination quadrilateral representing the real QR Code in the image can be set-up (Figure 14).

Figure 14. Perspective transformation.

Using equations [21]: ax  by  c dx  ey  f u  , v  , where the transformation coefficients can be calculated gx  hy 1 gx  hy 1 from the coordinates of the points P1(u1, v1), P2(u2, v2), P3(u3, v3), P4(u4, v4) as

a  (u3  u2 ) A  gu3 , b  (u1  u2 ) A  hu1 , c  u2

d  (v3  v2 ) A  gv3 , e  (v1  v2 ) A  hv1 , f  v2

du3 du2 du1 du3

dv3 dv2 dv1 dv3 g  , h  du1 du2 du1 du2

dv1 dv2 dv1 dv2

du1  (u3  u2 )A , du2  (u1  u4 )A, du3  u2  u3  u4  u1

dv1  (v3  v2 )A , dv2  (v1  v4 )A , dv3  v2  v3  v4  v1

It sometimes happens, that the estimate of P4 position is not quite accurate so we move P4 in spiral from its initial position (obtained in previous step) and we calculate match score of bottom-right Alignment Pattern (Alignment Pattern exists only in QR codes version 2 and above). For each shift we recalculate coefficients for the perspective transformation, and we recalculate also match score. For the given version of the QR Code we know the expected position and size of Appl. Sci. 2020, 10, x FOR PEER REVIEW 12 of 17

Figure 13. Perspective distortion: (a) initial estimate of the point P4 and lines L1, L3; (b) first shift of

the line L3; (c) second shift of the line L3

Once the position of 4th point, P4, is obtained, perspective transformation from the source Appl.square Sci. representing2020, 10, 7814 the ideal QR Code to destination quadrilateral representing the real QR Code12 of in 16 the image can be set-up (Figure 14).

Figure 14. Perspective transformation.transformation.

Using equationsequations [[2121]:]: = axax+byby+c  c= dx+eydx+ f  ey  f uu  gx+hy+1 , v , vgx+hy+1 , where the, where transformation the transformation coeﬃcients coefficients can be calculated can be calculated from the gx  hy 1 gx  hy 1 coordinates of the points P1(u1, v1), P2(u2, v2), P3(u3, v3), P4(u4, v4) as from the coordinates of the points P1(u1, v1), P2(u2, v2), P3(u3, v3), P4(u4, v4) as a = (u u )/A + gu , b = (u u )/A + hu , c = u 3 − 2 3 1 − 2 1 2 a  (u3  u2 ) A  gu3 , b  (u1  u2 ) A  hu1 , c  u2 d = (v3 v2)/A + gv3, e = (v1 v2)/A + hv1, f = v2 − − d  (v  v ) A  gv , edu3 (vdu2 v ) A  duhv1 , duf 3 v 3 2 3 1 2 1 2 dv3 dv2 dv1 dv3 g = , h = du3 du2 du du1 1 dudu32 du1 du2

dv1 dv2 dv1 dv2 dv3 dv2 dv1 dv3 Appl. Sci. 2020, 10, x FOR PEER REVIEW 13 of 17 g  , hdu1 = (u3 u2)A , du2 = (u1 u4)A, du3 = u2 u3 + u4 u1 du1 du2 du1 −du2 − − − dv = (v v )A, dv = (v v )A, dv = v v + v v bottom-right Alignment1 Pattern3 − so2 we can2 calculate1 − 4 match 3between2 − expected3 4 − and1 real state. The final dv1 dv2 dv1 dv2 position P4 is the one with the highest match score. It sometimes happens, that the estimate of P4 position is not quite accurate so we move P4 in Another alternative method how to handle perspective distorted QR Codes and how to spiraldu1  from(u3  itsu initial2 )A , positiondu2  (u (obtained1  u4 )A in, previousdu3  u2 step) u3 and u we4  calculateu1 match score of bottom-right Alignmentdetermine position Pattern (Alignment of the P4 poin Patternt is based exists on only edge in directions QR codes versionand edge 2 andprojections above). analysis For each [2 shift3]. we recalculatedv1  (v3  coev2ﬃ)Acients, dv for2  the(v1 perspective v4 )A , dv transformation,3  v2  v3  andv4  wev1 recalculate also match score. For the 3.7. Decoding of a QR Code given version of the QR Code we know the expected position and size of bottom-right Alignment It sometimes happens, that the estimate of P4 position is not quite accurate so we move P4 in PatternA QR so weCode can is calculate2D square match matrix between in which expected the dark and and real light state. squares The ﬁnal(modules) position represent P is the bits one 1 spiral from its initial position (obtained in previous step) and we calculate match4 score of withand 0. the In highestfact, each match such score. a module on the pixel level is usually made up of cluster of adjacent pixels. bottom-right Alignment Pattern (Alignment Pattern exists only in QR codes version 2 and above). In a QRAnother Code alternativedecoding process, method we how have to handle to build perspective a 2D matrix distorted that has QR elements Codes and with how a value to determine of 1 or 0 For each shift we recalculate coefficients for the perspective transformation, and we recalculate also position(Figure 1 of5). the P point is based on edge directions and edge projections analysis [23]. match score. For4 the given version of the QR Code we know the expected position and size of The pixels, where the brightness is lower than the threshold, are declared as 1 and the others are 3.7.declared Decoding as 0. of When a QR Code analyzing such a cluster of pixels (module), the central pixel of the module playsA a QR decisive Code is role. 2D square If the matrixcalculated in which position the darkof the and central light squares pixel does (modules) not align represent to integer bits 1coordinates and 0. In fact, in the each image, such the a module brightness on the is determined pixel level is by usually bilinear made inte uprpolation. of cluster of adjacent pixels. In a QROnceCode the decodingbinary matrix process, of 1 we and have 0 is to created, build a the 2D matrixopen-source that has ZBar elements library with [24] a can value be ofused 1 or to 0 (Figurestart the 15 final). decoding of the binary matrix and to receive the original text encoded by the QR Code.

Figure 15. TransformationTransformation of the QR Code into the binary grid.grid.

4. Results We used a test dataset of 595 QR Code samples to verify the method described in this paper. The testing dataset contained 25 artificial QR codes of different sizes and rotations, 90 QR Codes from Internet images, and 480 QR Codes from a specific industrial process. Several examples of the testing samples are in Figure 16.

Figure 16. Testing samples: (a) artificial, (b) Internet, (c) industrial. Appl. Sci. 2020, 10, x FOR PEER REVIEW 13 of 17

bottom-right Alignment Pattern so we can calculate match between expected and real state. The final position P4 is the one with the highest match score. Another alternative method how to handle perspective distorted QR Codes and how to determine position of the P4 point is based on edge directions and edge projections analysis [23].

3.7. Decoding of a QR Code A QR Code is 2D square matrix in which the dark and light squares (modules) represent bits 1 and 0. In fact, each such a module on the pixel level is usually made up of cluster of adjacent pixels. In a QR Code decoding process, we have to build a 2D matrix that has elements with a value of 1 or 0 (Figure 15). The pixels, where the brightness is lower than the threshold, are declared as 1 and the others are declared as 0. When analyzing such a cluster of pixels (module), the central pixel of the module plays a decisive role. If the calculated position of the central pixel does not align to integer coordinates in the image, the brightness is determined by bilinear interpolation. Once the binary matrix of 1 and 0 is created, the open-source ZBar library [24] can be used to start the final decoding of the binary matrix and to receive the original text encoded by the QR Code. Appl. Sci. 2020, 10, 7814 13 of 16

The pixels, where the brightness is lower than the threshold, are declared as 1 and the others are declared as 0. When analyzing such a cluster of pixels (module), the central pixel of the module plays a decisive role. If the calculated position of the central pixel does not align to integer coordinates in the image, the brightness is determined by bilinear interpolation. Once the binary matrix of 1 and 0 is created, the open-source ZBar library [24] can be used to start the final decoding of the binary matrix and to receive the original text encoded by the QR Code. Figure 15. Transformation of the QR Code into the binary grid. 4. Results 4. Results We used a test dataset of 595 QR Code samples to verify the method described in this paper. We used a test dataset of 595 QR Code samples to verify the method described in this paper. The testingThe dataset testing containeddataset contained 25 artificial 25 artificial QR QR codes codes of of di differentfferent sizessizes and and rotations, rotations, 90 QR 90 QRCodes Codes from Internet images,from Internet and 480images QR, and Codes 480 QR from Cod aes specific from a specific industrial industrial process. process Several. Several examples of ofthe the testing samples aretesting in Figure samples 16 are. in Figure 16.

FigureFigure 16. Testing 16. Testing samples: samples: ( a(a)) artiﬁcial,artificial, (b () bInternet,) Internet, (c) industrial (c) industrial..

Appl. Sci. 2020, 10, x FOR PEER REVIEW 14 of 17 In Table1 and Figure 17 there are our results compared to competing QR Code decoding solutions (commercial and alsoIn Table open-source). 1 and Figure 1 In7 there the are table our are results the compared numbers to competing of correctly QR Code decoded decoding QR Codes from solutions (commercial and also open-source). In the table are the numbers of correctly decoded QR the total numberCodes of 595from the QR total Codes. number of 595 QR Codes.

Table 1. ComparisonTable 1. Comparison of competing of competing solutions with with our proposed our proposed method. method.

Artificial Internet Industrial SolutionSolution Artiﬁcial Samples Internet Samples Industrial Samples Samples Samples Samples ZXing (open-source)ZXing [25] (open-source) [25] 22 72 72 48 48 Quirc (open-source) [Quirc22] (open-source) [22] 1212 69 69 45 45 LEADTOOLS QR CodeLEADTOOLS SDK [26] QR Code SDK [2 96] 9 72 72 147 147 ZBar (open-source) [ZBar24] (open-source) [24] 2323 84 84 398 398 Inlite Barcode ReaderInlite SDK Barcode [27] Reader SDK [2 257] 25 80 80 421 421 DataSymbol BarcodeDataSymbol Reader SDK Barcode [28] Reader SDK25 [28] 25 8989 471 471 Dynamsoft Barcode ReaderDynamsoft SDK Barcode [29] Reader SDK 25 [29] 25 88 88 478 478 Our solution Our solution 2525 8787 480 480

Figure 17. ComparisonFigure 17. Comparison of the of recognition the recognition rate rate of of our our method method against againstcompeting competingsolutions. solutions. Our method successfully detected all QR Codes but failed to decode three samples. Two samples were QR Codes placed on a bottle, where perspective and cylindrical distortions were combined. How to deal with this type of combined distortion is a challenge for future research. As the commercial solutions have closed source code, we performed the testing using the black-box method. We have compiled our own QR Code test dataset (published together with this article) to evaluate and compare our method, as a standardized universal dataset is not publicly available. In Table 2 the computational complexity of our algorithm is compared to competing open-source solutions (commercial solutions were tested online). Our algorithm was implemented in Free Pascal and tests were run on an i5-4590 3.3GHz CPU (Intel Corporation, Santa Clara, CA, USA).

Table 2. Dependence of computational complexity on image resolution and number of QR codes in an image.

1296 × 960 2592 × 1920 Solution 1 Code 10 Codes 1 Code 10 Codes 50 Codes ZBar (open-source) 85 ms 185 ms 347 ms 372 ms 1744 ms Quirc (open-source) 13 ms 26 ms 45 ms 130 ms 493 ms Our solution 18 ms 21 ms 73 ms 77 ms 107 ms Appl. Sci. 2020, 10, 7814 14 of 16

Our method successfully detected all QR Codes but failed to decode three samples. Two samples were QR Codes placed on a bottle, where perspective and cylindrical distortions were combined. How to deal with this type of combined distortion is a challenge for future research. As the commercial solutions have closed source code, we performed the testing using the black-box method. We have compiled our own QR Code test dataset (published together with this article under “Supplementary Material”) to evaluate and compare our method, as a standardized universal dataset is not publicly available. In Table2 the computational complexity of our algorithm is compared to competing open-source solutions (commercial solutions were tested online). Our algorithm was implemented in Free Pascal and tests were run on an i5-4590 3.3GHz CPU (Intel Corporation, Santa Clara, CA, USA).

Table 2. Dependence of computational complexity on image resolution and number of QR codes in an image.

1296 960 2592 1920 Solution × × 1 Code 10 Codes 1 Code 10 Codes 50 Codes ZBar (open-source) 85 ms 185 ms 347 ms 372 ms 1744 ms Quirc (open-source) 13 ms 26 ms 45 ms 130 ms 493 ms Our solution 18 ms 21 ms 73 ms 77 ms 107 ms

We see the main contribution of our method in the way the broken Finder Pattern is dealt with (Section 3.3.2) and how the QR Code bounding box is determined (Section 3.6), especially for perspective distorted QR Codes. The presented method can still locate a QR Code if one of the three Finder Patterns is signiﬁcantly damaged. Consecutive tests in the real manufacturing process showed that this situation occurs much more often than a situation where two or three opposite Finder Patterns are damaged. In order to detect a QR Code with multiple damage Finder Patterns, it will be necessary to combine Finder Pattern based localization with the region based localization.

5. Conclusions We have designed and tested a computationally efficient method for precise location of 2D QR Codes in arbitrary images under various illumination conditions. The proposed method is suitable for low-resolution images as well as for real time processing. The designed Finder Pattern based localization method uses three typical patterns of QR Codes to identify three corners of QR Codes in an image. If one of the three Finder Patterns is so destroyed that it cannot be localized, we have suggested a way to deal with it. The input image is binarized and scanned horizontally to localize the Finder Pattern candidates, which are subsequently verified in order to localize the raw QR Code region. For distorted QR Codes, the perspective transformation is set-up by gradually approaching the boundary of the QR Code. This method was validated on the testing dataset consisting of a wide variety of samples (synthetic, real world, and specific industrial samples) and it was compared to competing software. The experimental results show that our method has a high detection rate. The application of QR Codes and their optical recognition has wide use in identification, tracing or monitoring of items in production [30], storing and distribution processes, to aid visually impaired and blind people, to let autonomous robots [31] to acquire context-relevant information, to support authorization during log-in process, to support electronic payments, to increase industrial production surety factor [32], etc. In cases where is required to place the 2D matrix codes on a very small area it may be preferable to use Data Matrix codes [33].

Supplementary Materials: The following are available online at http://www.mdpi.com/2076-3417/10/21/7814/s1, one ZIP ﬁle contains images of QR Codes testing dataset used to evaluate the competing solutions. Appl. Sci. 2020, 10, 7814 15 of 16

Author Contributions: Conceptualization, methodology, software, writing—original draft preparation, L.K.; validation, writing—review and editing, visualization, E.P.; supervision, project administration, funding acquisition, P.B.; All authors have read and agreed to the published version of the manuscript. Funding: The article is funded of the research project KEGA 013TUKE-4/2019 “Modern educational tools and methods for shaping creativity and increasing the practical skills and habits of graduates of technical departments of universities”. Acknowledgments: The contribution is sponsored by the project. Conﬂicts of Interest: The authors declare no conﬂict of interest.

References

1. Zhang, X.; Zhu, S.; Wang, Z.; Li, Y.Hybrid visual natural landmark–based localization for indoor mobile robots. Int. J. Adv. Robot. Syst. 2018, 15.[CrossRef] 2. Bozek, P.; Pokorný, P.; Svetlik, J.; Lozhkin, A.; Arkhipov, I. The calculations of Jordan curves trajectory of the robot movement. Int. J. Adv. Robot. Syst. 2016, 13, 1–7. [CrossRef] 3. Wang, K.; Zhou, W. Pedestrian and cyclist detection based on deep neural network fast R-CNN. Int. J. Adv. Robot. Syst. 2018, 16.[CrossRef] 4. Zhang, X.; Gao, H.; Xue, C.; Zhao, J.; Liu, Y. Real-time vehicle detection and tracking using improved histogram of gradient features and Kalman filters. Int. J. Adv. Robot. Syst. 2018, 15.[CrossRef] 5. Denso Wave Incorporated. What is a QR Code? 2018. Available online: http://www.qrcode.com/en/about/ (accessed on 6 September 2018). 6. Denso Wave Incorporated. History of QR Code. 2018. Available online: http://www.qrcode.com/en/history/ (accessed on 6 September 2018). 7. Lin, J.-A.; Fuh, C.-S. 2D Barcode Image Decoding. Math. Probl. Eng. 2013, 2013, 848276. [CrossRef] 8. Li, S.; Shang, J.; Duan, Z.; Huang, J. Fast detection method of quick response code based on run-length coding. IET Image Process. 2018, 12, 546–551. [CrossRef] 9. Belussi, L.F.F.; Hirata, N.S. Fast Component-Based QR Code Detection in Arbitrarily Acquired Images. J. Math. Imaging Vis. 2012, 45, 277–292. [CrossRef] 10. Bodnár, P.; Nyúl, L.G. Improved QR Code Localization Using Boosted Cascade of Weak Classifiers. Acta Cybern. 2015, 22, 21–33. [CrossRef] 11. Tribak, H.; Zaz, Y. QR Code Recognition based on Principal Components Analysis Method. Int. J. Adv. Comput. Sci. Appl. 2017, 8, 241–248. [CrossRef] 12. Tribak, H.; Zaz, Y. QR Code Patterns Localization based on Hu Invariant Moments. Int. J. Adv. Comput. Sci. Appl. 2017, 8, 162–172. [CrossRef] 13. Sun, A.; Sun, Y.; Liu, C. The QR-code reorganization in illegible snapshots taken by mobile phones. In Proceedings of the 2007 International Conference on Computational Science and its Applications (ICCSA 2007), Kuala Lumpur, Malaysia, 26–29 August 2007; pp. 532–538. 14. Ci ˛a˙zyński,K.; Fabijańska,A. Detection of QR-Codes in Digital Images Based on Histogram Similarity. Image Process. Commun. 2015, 20, 41–48. [CrossRef] 15. Gaur, P.; Tiwari, S. Recognition of 2D Barcode Images Using Edge Detection and Morphological Operation. Int. J. Comput. Sci. Mob. Comput. IJCSMC 2014, 3, 1277–1282. 16. Szentandrási, I.; Herout, A.; Dubská, M. Fast detection and recognition of QR codes in high-resolution images. In Proceedings of the 28th Spring Conference on Computer Graphics, Budmerice, Slovakia, 2–4 May 2012; ACM: New York, NY, USA, 2012. 17. Dubská, M.; Herout, A.; Havel, J. Real-time precise detection of regular grids and matrix codes. J. Real Time Image Process. 2016, 11, 193–200. [CrossRef] 18. Sörös, G.; Flörkemeier, C. Blur-resistant joint 1D and 2D barcode localization for smartphones. In Proceedings of the 12th International Conference on Mobile and Ubiquitous Multimedia, MUM, Lulea, Sweden, 2–5 December 2013; pp. 1–8. 19. Bradley, D.; Roth, G. Adaptive Thresholding using the Integral Image. J. Graph. Tools 2007, 12, 13–21. [CrossRef] 20. Bresenham, J.E. Algorithm for computer control of a digital plotter. IBM Syst. J. 1965, 4, 25–30. [CrossRef] 21. Heckbert, P.Fundamentals of Texture Mapping and Image Warping. Master’s Thesis, University of California, Berkeley, CA, USA, June 1989. Appl. Sci. 2020, 10, 7814 16 of 16

22. Beer, D. Quirc—QR Decoder Library. 2018. Available online: https://github.com/dlbeer/quirc (accessed on 22 September 2018). 23. Karrach, L.; Pivarˇciová, E.; Božek, P. Identification of QR Code Perspective Distortion Based on Edge Directions and Edge Projections Analysis. J. Imaging 2020, 6, 67. [CrossRef] 24. Terriberry, T.B. ZBar Barcode Reader. 2018. Available online: http://zbar.sourceforge.net (accessed on 22 September 2018). 25. Google. ZXing (Zebra Crossing) Barcode Scanning Library for Java, Android. 2018. Available online: https://github.com/zxing (accessed on 22 September 2018). 26. Leadtools. QR Code SDK Technology. 2018. Available online: http://demo.leadtools.com/JavaScript/Barcode/ index.html (accessed on 22 September 2018). 27. Inlite Research Inc. Barcode Reader SDK. 2018. Available online: https://online-barcode-reader.inliteresearch. com (accessed on 22 September 2018). 28. DataSymbol. Barcode Reader SDK. 2018. Available online: http://www.datasymbol.com/barcode-reader- sdk/barcode-reader-sdk-for-windows/online-barcode-decoder.html (accessed on 22 September 2018). 29. Dynamsoft. Barcode Reader SDK. 2018. Available online: https://www.dynamsoft.com/Products/Dynamic- Barcode-Reader.aspx (accessed on 22 September 2018). 30. Bako, B.; Božek, P. Trends in Simulation and Planning of Manufacturing Companies. Procedia Eng. 2016, 149, 571–575. [CrossRef] 31. Frankovsky, P.; Pastor, M.; Dominik, L.; Kicko, M.; Trebuna, P.; Hroncova, D.; Kelemen, M. Wheeled mobile robot in structured environment. In Proceedings of the 12th International Conference ELEKTRO 2018, Mikulov, Czech Republic, 21–23 May 2018; pp. 1–5. 32. Pivarˇciová, E.; Božek, P. Industrial production surety factor increasing by a system of fingerprint verification. In Proceedings of the 2014 International Conference on Information Science, Electronics and Electrical Engineering (ISEEE 2014), Sapporo, Japan, 26–28 April 2014; pp. 493–497. 33. Karrach, L.; Pivarˇciová, E.; Nikitin, Y.R. Comparing the impact of different cameras and image resolution to recognize the data matrix codes. J. Electr. Eng. 2018, 69, 286–292. [CrossRef]

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional aﬃliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).