Estimating Recycling Return of Integrated Circuits Using Computer Vision on Printed Circuit Boards

applied sciences

Article Estimating Recycling Return of Integrated Circuits Using Computer Vision on Printed Circuit Boards

Leandro H. de S. Silva 1,2,* , Agostinho A. F. Júnior 1 , George O. A. Azevedo 1 , Sergio C. Oliveira 1 and Bruno J. T. Fernandes 1

1 Escola Politécnica de Pernambuco, University of Pernambuco, Recife 50720-001, Brazil; [email protected] (A.A.F.J.); [email protected] (G.O.A.A.); [email protected] (S.C.O.); [email protected] (B.J.T.F.) 2 Unidade Acadêmica de Área de Indústria, Federal Institute of Paraíba, Cajazeiras 58900-000, Brazil * Correspondence: [email protected]

Abstract: The technological growth of the last decades has brought many improvements in daily life, but also concerns on how to deal with electronic waste. Electrical and electronic equipment waste is the fastest-growing rate in the industrialized world. One of the elements of electronic equipment is the printed circuit board (PCB) and almost every electronic equipment has a PCB inside it. While waste PCB (WPCB) recycling may result in the recovery of potentially precious materials and the reuse of some components, it is a challenging task because its composition diversity requires a cautious pre-processing stage to achieve optimal recycling outcomes. Our research focused on proposing a method to evaluate the economic feasibility of recycling integrated circuits (ICs) from WPCB. The proposed method can help decide whether to dismantle a separate WPCB before the physical or mechanical recycling process and consists of estimating the IC area from a WPCB, Citation: Silva, L.H.d.S.; Júnior, calculating the IC’s weight using surface density, and estimating how much metal can be recovered A.A.F.; Azevedo, G.O.A.; Oliveira, by recycling those ICs. To estimate the IC area in a WPCB, we used a state-of-the-art object detection S.C.; Fernandes, B.J.T. Estimating Recycling Return of Integrated deep learning model (YOLO) and the PCB DSLR image dataset to detect the WPCB’s ICs. Regarding Circuits Using Computer Vision on IC detection, the best result was obtained with the partitioned analysis of each image through a Printed Circuit Boards. Appl. Sci. sliding window, thus creating new images of smaller dimensions, reaching 86.77% mAP. As a final 2021, 11, 2808. https://doi.org/ result, we estimate that the Deep PCB Dataset has a total of 1079.18 g of ICs, from which it would be 10.3390/app11062808 possible to recover at least 909.94 g of metals and silicon elements from all WPCBs’ ICs. Since there is a high variability in the compositions of WPCBs, it is possible to calculate the gross income for each Academic Editor: Pablo Padilla de la WPCB and use it as a decision criterion for the type of pre-processing. Torre Keywords: waste PCB recycling; integrated circuit detection; YOLO Received: 31 December 2020 Accepted: 29 January 2021 Published: 22 March 2021 1. Introduction Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in Waste electrical and electronic equipment (WEEE) management is a concern in many published maps and institutional affil- countries [1] since electrical and electronic equipment production has increased while the iations. average lifetime of these products has decreased [2]. Consequently, WEEE has the fastest growth rate in the industrialized world [3]. This concern has led to the development of legislation and directives to manage this sort of waste, for example, the European Directive for WEEE and the Brazilian Policy of Solid Waste (BPSW) [4]. The European Commission recently adopted Implementing Regulation (EU) 2019/290, Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. which focuses on restricting the use of hazardous substances in electrical and electronic This article is an open access article equipment in the WEEE Directive of 2003. The evolution of laws and regulations demon- distributed under the terms and strates an advancement regarding the impacts that the improper disposal of electronic conditions of the Creative Commons materials can have on the environment, thus the development of projects in this line of Attribution (CC BY) license (https:// research positively reflects the industry’s current sustainability issues. creativecommons.org/licenses/by/ Printed circuit boards (PCB) are an essential element present in almost all electrical 4.0/). and electronic equipment [2]. The PCB is the platform that connects the discrete and

Appl. Sci. 2021, 11, 2808. https://doi.org/10.3390/app11062808 https://www.mdpi.com/journal/applsci Appl. Sci. 2021, 11, 2808 2 of 11

integrated electronic components. Recycling waste printed circuit boards (WPCBs) can recover valuable materials (metallic and nonmetallic), besides the possibility of the reuse of some components [5]. WPCBs are the major source of metals in electronic waste [6]. Even given the high-value materials, WPCB recycling is difficult because of the diversity in their compositions [7,8]. Dismantling and separating of the WPCB components is very important to achieve the maximum recycling outcomes. However, completely automating this pre-processing is a high-cost process, and it is performed manually in primitive methods in some countries [9]. In order to optimize the recycling WPCB process, vision systems can be used to input data to desoldering systems and identify reusable and hazardous components [7,10]. Aiming to promote computer-vision-based strategies for PCB recycling, the PCB DSLR public dataset was published [10]. DSLR stands for Digital Single-Lens Reflex camera, and it is a reference of the camera type used to acquire the PCB images. The PCB DSLR Dataset contains high-resolution WPCB images with bounding box annotations for integrated circuits (ICs). Since some ICs may contain valuable materials, it is essential to detect such components in WPCBs. Computer vision systems are a feasible way to perform WPCB evaluation to guide automatic dismantling and recycling [11]. The computer vision task performed is named object detection. Since sufficiently large fully-annotated datasets for the PCB domain are expensive and unavailable [12], the use of state-of-the-art object detectors (based on deep learning) can be impaired because of the dataset size [13]. This sort of problem is known as few-shot learning [14]. The main goal is to learn proper feature embedding that allows the model to be trained with fewer images than standard models. Since ICs are a significant source of highly valuable metals, we propose the WPCB economic feasibility assessment (WPCB-EFA) method for IC recycling from a WPCB by estimating recoverable metals and gross earnings using computer vision to detect ICs in WPCBs. As WPCBs have a high composition diversity, they can be grouped according to their value. For instance, mobile phones and mainframe PCBs have a high value (8 to 25 euros per kg) while calculators, audio scraps, and power supplies are low-value PCBs (less than 1 euro per kg) [15]. The proposed WPCB-EFA can be used to evaluate the composition of a WPCB and select the appropriated recycling process focused on recovering precious metals for high-value PCBs or the nonmetal fraction for low-value PCBs. To evaluate the proposed WPCB-EFA, we used transfer learning of the state-of-the-art YOLOv3 model [16] in a few-shot scenario for IC detection on the PCB DSLR Dataset. Then, we proposed an estimation for the IC surface density to calculate the PCB’s IC weight and estimate the financial return of recycling a WPCB’s integrated circuits. The rest of this article is structured as follows. Section2 provides an overall context about WPCB, IC recycling, and object detection. In Section3, there is a detailed description of the proposed method for the economic evaluation of IC recycling from WPCBs. Section4 presents the results of the proposed method applied to the PCB DSLR Dataset. Finally, Section5 contains the final discussion and our conclusions.

2. Background 2.1. Printed Circuit Board (PCB) and Integrated Circuit (IC) Recycling The PCB composition is complex, so the recycling process is arduous [7,9]. A PCB comprises about 30% metallic components. Some of these are high-value metals like gold, silver, and copper. Even though these high-value materials are less than 1% of the PBC weight, it can represent 80% to 88% of the PCB recycling value [9,15]. Li et al. [7] estimated that 1 ton of WPCBs contained approximately 284 g of gold. WPCB recycling processes are mainly interested in the recovery of those valuable metals. However, depending on the recycling process used, there are environmental problems with residues and hazardous substances resulting from the recycling process [17,18]. Usually, the WPCB recycling process involves three steps: dismantling, physical crushing or mechanical process, and some chemical leaching processes [9]. The dismantling Appl. Sci. 2021, 11, 2808 3 of 11

process aims to eliminate hazardous materials, gather components for reuse, and optimize the whole recycling process [18]. Eco-friendly recycling usually involves some WPCB dismantling processes [9]. WPCBs dismantling can reduce metal loss, leading to maximum precious metal recovery. The WPCB dismantling process can be done manually, semi-automatically, or automatically [18]. Some PCB recycling facilities perform crushing without dismantling the WPCB. This method reduces the human cost and protects the worker’s health [18]. However, automatic dismantling can lead to cost-effective and sustainable WPCB recycling, especially if WPCBs are from different equipment types [19]. The current challenge of WPCB recycling is to achieve economic, environmental, and marketability viability [15]. For this purpose, a combination of several recycling techniques and a proper way to choose the recycling process may lead to an outcome that satisﬁes the market standard. For instance, even automatic disassembly adds a cost to the recycling process. Consequently, the outcomes of WPCB recycling can be improved by some methods to choose the recycling process such as when to dismantle a WPCB before shredding [15,18]. Precious metals such as Au, Ag, and Pd are used as contact materials or as plating layers in ICs [18]. Therefore, recycling ICs can lead to a better recovery of these highly valuable metals, while the non-metallic parts can be recycled in another process [1]. Barnwal and Dhawan [20] describe a physical process to recycle ICs and recovery metallic values. This study gathered 500 kg of WPCBs from computers and laptops, from which 15 kg of ICs were removed mechanically or by desoldering. The authors proposed a physical separation recycling method comprised of grinding, screening, water ﬂuidization, magnetic separation, and density separation. The recycling outcomes are reported in Table1.

Table 1. Physical processing of the discarded integrated circuit (IC) outcomes [20].

Recovered Elements for 1 kg of ICs Fe 68.88 g Ferrous metal 120 g Ni 31.2 g Metallic 390 g Cu 311.5 g Non-metallic 490 g Si 411.6 g

Aside from physical separation, leaching is another approach for integrated circuit recycling [21]. The leaching process is applied after physical separation. Lee at. al. reported 100% recovery of gold (Au), silver (Ag), and copper (Cu) with thiourea leaching [21]. Considering recycling the entire WPCB, precious metals (Au, Pd, Pt, and Ag) are about 88% of the intrinsic value of the WPCB [15]. Most studies consider the recovery of these precious metals from printed circuit boards as a whole [15]. Although studies indicate the concentration of the precious metal in integrated circuits, the percentage of the integrated circuit mass corresponding to precious metals is challenging [21]. This measurement difﬁculty occurs because the concentration of precious metals is small, and there is a considerable variation in concentration depending on the sample of the ICs.

2.2. Object Detection Object detection is a computer vision task to determine an object’s location by out- putting a bounding box and, at the same time, classifying the object [22]. When multiple objects are detected on a particular image, this is referred to as multi-class object detection. Deep learning strategies are state-of-the-art for object detection problems [22]. These strategies can be split into two main classes: two-stage and one-stage methods. The two- stage methods, the R-CNN (region with convolutional neural network features) family, ﬁrst generate region proposals and then classify each region. The one-stage method (YOLO model) executes the object detection task in a single forward through the neural network. YOLO stands for You Only Look Once, and it is a real-time object detection system. The general idea is that a single neural network divides the image into regions and then Appl. Sci. 2021, 11, x FOR PEER REVIEW 4 of 11

2.2. Object Detection Object detection is a computer vision task to determine an object’s location by out‐ putting a bounding box and, at the same time, classifying the object [22]. When multiple objects are detected on a particular image, this is referred to as multi‐class object detection. Deep learning strategies are state‐of‐the‐art for object detection problems [22]. These strategies can be split into two main classes: two‐stage and one‐stage methods. The two‐ stage methods, the R‐CNN (region with convolutional neural network features) family, first generate region proposals and then classify each region. The one‐stage method (YOLO model) executes the object detection task in a single forward through the neural network. YOLO stands for You Only Look Once, and it is a real‐time object detection system. The general idea is that a single neural network divides the image into regions and then predicts several bounding boxes for each region. Its probability selects those bounding boxes as a final result [16]. Appl. Sci. 2021, 11, 2808 This work used the third version of YOLO (YOLOv3), which4 of 11 has improved perfor‐ mance compared to earlier versions. Since the first version, YOLO’s backbone for feature

predictsextraction several and bounding classification boxes for each is region.a deep Its convolutional probability selects network. those bounding In YOLOv2, the backbone boxeswas as Darknet a final result‐19, [16 while]. in YOLOv3, it was replaced by Darknet‐53, a deeper network. Both DarknetThis work architectures used the third versionare convolutional of YOLO (YOLOv3), neural which networks. has improved YOLOv3 perfor- also replaced the final mance compared to earlier versions. Since the first version, YOLO’s backbone for feature extractionsoftmax and layer classification used in is aYOLOv2 deep convolutional for independent network. In YOLOv2, logistic the classifiers backbone (with binary cross‐en‐ wastropy Darknet-19, loss). This while change in YOLOv3, made it was the replaced model by become Darknet-53, a multilabel a deeper network. object detector. Both Darknet architectures are convolutional neural networks. YOLOv3 also replaced the final softmax layer used in YOLOv2 for independent logistic classifiers (with binary cross-entropy3. Waste Printed loss). This Circuit change made Boards the model Economic become a multilabelFeasibility object Assessment detector. (WPCB‐EFA)

3. WasteFigure Printed Circuit1 shows Boards the Economic outline Feasibility of the proposed Assessment (WPCB-EFA)WPCB economic feasibility assessment (WPCBFigure‐1EFA). shows theThe outline first ofstep the proposedis to detect WPCB the economic ICs and feasibility then assessmentcalculate the IC area and weight. (WPCB-EFA).With the IC The weight, ﬁrst step isit tois detect possible the ICs to and use then the calculate proposed the IC IC area recycling and weight. methods to estimate the Withrecoverable the IC weight, metals it is possible and tothe use gross the proposed earnings IC recycling [20,21]. methods to estimate the recoverable metals and the gross earnings [20,21].

FigureFigure 1. Diagram 1. Diagram of the proposed of the proposed method evaluation. method evaluation. 3.1. Detect IC Regions 3.1.To Detect detect theIC ICsRegions in the WPCB images, we evaluated the transfer learning strategy from the pre-trained YOLOv3 model. In order to deal with the few-shot learning problems, one approachTo detect is to use the prior ICs knowledge in the andWPCB perform images, a data augmentation we evaluated procedure the [14 transfer]. learning strategy Therefore,from the we pre conducted‐trained data augmentationYOLOv3 model. in the PCB In DSLR order dataset to deal to generate with smaller the few‐shot learning prob‐ sub-images.lems, one A approach sliding window is to of use 416 ×prior416 pixels knowledge generated and sub-images perform by passing a data augmentation proce‐ through the original image. The training process used only sub-images containing at least adure fraction [14]. of some Therefore, IC. After datawe augmentation,conducted data the new augmentation dataset had 5347 in RGB the images PCB DSLR dataset to gener‐ containingate smaller ICs, each sub one‐images. with a size A of sliding 416 × 416 window pixels. of 416 × 416 pixels generated sub‐images by passingSince the through IC detector the works original with chunks image. of full-resolution The training WPCB process images, used the reverse only sub‐images containing process is necessary to obtain the entire WPCB image’s IC bounding boxes. Given that the WPCB will be in a conveyor belt, we deﬁned the region of interest as the conveyor belt area. Figure2 illustrates this region of interest. In summary, we removed the region outside the conveyor belt. Since all of the PCB DSLR images were from the same perspective, the region of interest’s coordinates were deﬁned to remove the region outside the conveyor belt. Appl. Sci. 2021, 11, x FOR PEER REVIEW 5 of 11 Appl. Sci. 2021, 11, x FOR PEER REVIEW 5 of 11

at least ata fraction least a offraction some IC. of After some data IC. augmentation, After data augmentation,the new dataset hadthe 5347new RGBdataset had 5347 RGB images containingimages containing ICs, each one ICs, with each a size one of with 416 × a416 size pixels. of 416 × 416 pixels. Since theSince IC detector the IC works detector with chunksworks ofwith full ‐chunksresolution of WPCB full‐resolution images, the WPCB reverse images, the reverse process processis necessary is necessary to obtain the to entire obtain WPCB the entireimage’s WPCB IC bounding image’s boxes. IC Givenbounding that the boxes. Given that the WPCB willWPCB be in will a conveyor be in a belt, conveyor we defined belt, the we region defined of interest the region as the ofconveyor interest belt as the conveyor belt area. Figure 2 illustrates this region of interest. In summary, we removed the region out‐ side the area.conveyor Figure belt. 2Since illustrates all of the this PCB region DSLR images of interest. were from In summary, the same perspective, we removed the region out‐ the regionside of theinterest’s conveyor coordinates belt. Since were alldefined of the to PCB remove DSLR the imagesregion outside were fromthe con the‐ same perspective,

Appl. Sci. 2021, 11, 2808 veyor belt.the region of interest’s coordinates were defined to remove5 of 11 the region outside the con‐ veyor belt.

(a) (b)

Figure 2. An image sample from the Printed circuit boards digital single‐lens reflex (PCB DSLR) Dataset (a) and the region( aof) interest hatchured (b). (b)

Figure 2. An image sample from the Printed circuit boards digital single-lens reﬂex (PCB DSLR) Therefore,DatasetFigure (a) and2.we theAn applied region image of interesta slidingsample hatchured window from (b). the approach Printed in circuit the region boards of interest,digital singlesimilarly‐lens reflex (PCB DSLR) to the processDataset of (buildinga) and the the region training of set.interest The YOLOhatchured model (b processes). every image chunk gathered byTherefore, the sliding we applied window. a sliding Then, window the approach YOLO inprediction the region of for interest, each similarly image chunk is com‐ to the process of building the training set. The YOLO model processes every image bined overchunk the gatheredTherefore, original by the size sliding we image applied window. to Then, obtain a sliding the YOLO the bounding predictionwindow for boxes eachapproach image over chunk the in theentire region WPCB of interest, similarly is combined over the original size image to obtain the bounding boxes over the entire image. Figureto the 3process shows the of predictionbuilding theresult training for a full set.‐size The image YOLO from model the PCB processes DSLR Da‐ every image chunk taset. WPCB image. Figure3 shows the prediction result for a full-size image from the PCB DSLRgathered Dataset. by the sliding window. Then, the YOLO prediction for each image chunk is com‐ bined over the original size image to obtain the bounding boxes over the entire WPCB image. Figure 3 shows the prediction result for a full‐size image from the PCB DSLR Da‐ taset.

(a) (b) (c) Figure 3. ExampleFigure of 3. ICExample (integrated of IC (integrated circuit) detection circuit) detection for an for entire an entire image: image: ( (a) YOLOYOLO (object (object detection detection model) model) raw IC raw IC detec‐ tion for the entiredetection WPCB for (waste the entire printed WPCB (waste circuit printed board)—a circuit board)—a blue bounding blue bounding box box is drawn is drawn on on eacheach detected detected IC; (IC;b) (b) Predicted Predicted IC mask area—all the bounding boxes are ﬁled and combined over a black background; (c) Ground truth IC IC mask area—allarea—the the maskbounding built on boxes the ground are truth filed bounding and combined boxes. over a black background; (c) Ground truth IC area—the mask built on the ground truth bounding boxes. The sliding window stride allows multiple bounding boxes for the same IC. As we (werea) concerned with the IC area, standard non-maximum(b) suppression (NMS) algorithms (c) Thedid sliding not apply window to this problem. stride Therefore, allows allmultiple the predicted bounding bounding boxes boxes withfor athe score same IC. As we Figure 3. Example ofwere IC (integrated concernedhigher than withcircuit) the threshold the detection IC (0.8) area, were standard for used an to produceentire non‐maximum animage: image mask.(a) suppression YOLO Figure3 (objectshows (NMS) the detection algorithms model) raw IC detec‐ did not ICapply image to masks this for problem. the YOLO predictions Therefore, and groundall the truth predicted bounding boxes.bounding Based onboxes the with a score tion for the entire WPCB (wasteIC image printed mask, the circuit IC pixel areaboard)—a was calculated. blue bounding box is drawn on each detected IC; (b) Predicted IC mask area—all thehigher bounding than the boxes threshold are filed (0.8) and were combined used to produce over a blackan image background; mask. Figure (c) 3Ground shows thetruth IC area—the 3.2. Calculating IC Weight mask built on the groundIC image truth masks bounding for the boxes. YOLO predictions and ground truth bounding boxes. Based on the IC imageWith mask, a WPCB the image, IC pixel it is possible area to was estimate calculated. the surface area of each integrated circuit. However, to calculate the IC’s weight mass, we needed to estimate the average surface density.The Texas Instrumentssliding window (TI) has published stride an Application allows Reportmultiple with information bounding on boxes for the same IC. As we 3.2. Calculatingthe IC weight IC andWeight footprint area of ﬁfteen IC packages [23]. Figure4 shows the calculated surfacewere densityconcerned for each with IC package the and IC the area, average standard IC surface non density‐maximum based on the suppression (NMS) algorithms WithTI application a WPCB report. image, Since it mostis possible of the IC weightto estimate can be assigned the surface to the package,area of we each integrated circuit. calculatedHowever,did not the apply averageto calculate surfaceto this densitythe problem. IC’s for anweight IC of Therefore, 2358 mass, mg/mm we2 . neededall the topredicted estimate thebounding average boxes with a score surface higherdensity. thanTexas the Instruments threshold (TI) (0.8) has werepublished used an to Application produce anReport image with mask. infor‐ Figure 3 shows the mation onIC theimage IC weight masks and for footprint the YOLO area of predictions fifteen IC packages and ground [23]. Figure truth 4 shows bounding the boxes. Based on calculatedthe surface IC image density mask, for eachthe ICIC packagepixel area and was the averagecalculated. IC surface density based

3.2. Calculating IC Weight With a WPCB image, it is possible to estimate the surface area of each integrated circuit. However, to calculate the IC’s weight mass, we needed to estimate the average surface density. Texas Instruments (TI) has published an Application Report with infor‐ mation on the IC weight and footprint area of fifteen IC packages [23]. Figure 4 shows the calculated surface density for each IC package and the average IC surface density based Appl. Sci. 2021, 11, x FOR PEER REVIEW 6 of 11

Appl. Sci. 2021, 11, 2808 6 of 11 on the TI application report. Since most of the IC weight can be assigned to the package, we calculated the average surface density for an IC of 2358 mg/mm2.

Figure 4. Calculated IC surface density for each package and average. Figure 4. Calculated IC surface density for each package and average.

3.3.3.3. Recycling Recycling Economic Economic Evaluation Evaluation TheThe general general economic economic evaluationevaluation ofof WPCBWPCB recyclingrecycling waswas proposedproposed byby NiuNiu etet al.al. [[2].2]. TheThe total total income income for for metal metal powder powder is is given given by by

N 𝐺𝑃G = ∑ P𝐶i × C𝜂𝑀,i × η − M , (1)(1) i=1

wherewhereG 𝐺is is the the net net income income per per kilogram kilogram metal metal powder; powder;Ci is𝐶 theis the commodity commodity cost cost of the of itheth metal𝑖 metal element; element;Pi is 𝑃 the is composition the composition of that of particular that particular metal; metal;η denotes 𝜂 denotes the total the weight total number;weight number; and M is and the recycling𝑀 is the cost.recycling The recycling cost. The cost recycling is highly cost dependent is highly on dependent the speciﬁc on productionthe specific line, production so it must line, be so explicitly it must be calculated explicitly for calculated one industrial for one plant. industrial A complete plant. A discussioncomplete discussion regarding regarding the recycling the recycling cost is beyond cost is thisbeyond paper’s this paper’s scope since scope it since involves it in‐ analyzingvolves analyzing static and static dynamic and dynamic costs. costs. Therefore,Therefore, we we can can use use Equation Equation (1) (1) to to calculate calculate the the gross gross proﬁt profit of of IC IC recycling. recycling. Since Since wewe are are conducting conducting economic economic evaluation evaluation based based on on visual visual analysis, analysis, the the IC weightIC weight needs needs to be to estimatedbe estimated from from the ICthe area. IC area. So, theSo, the IC weight IC weight is given is given by by

𝑊W =𝐴A𝑆𝜌× S × ρ (2)(2)

wherewhere W𝑊 isis the the IC IC weight; weight;A 𝐴is theis the IC IC surface surface area area in pixels; in pixels;S is the𝑆 is scale the factorscale factor from pixel from topixelmm to2; andmmρ; isand the 𝜌 IC is surface the IC density surface in densityg/mm in2 With g/mm the ICWith weight, the IC the weight, gross recyclingthe gross proﬁtrecycling can beprofit calculated can be calculated as as N R = ∑ Pi × W × Ci, (3) 𝑅𝑃i=1 𝑊𝐶, (3) where Pi is the composition of ith metal (obtained from Table1); W is the total ICs weight; andwhereCi is𝑃 theis the metal composition market value. of 𝑖metal (obtained from Table 1); 𝑊 is the total ICs weight; and 𝐶 is the metal market value. 4. Experiments and Results 4.1.4. Experiments Experimental and Setup Results 4.1. ExperimentalFor the development Setup of the IC object detector, we used the PCB DSLR Dataset [10]. × Initially,For this the dataset development has 748 of RBG the images IC object of WPCBs,detector, with we aused resolution the PCB of 4928DSLR Dataset3280 pixels, [10]. from 165 different boards (three to ﬁve images per board). Furthermore, the integrated Initially, this dataset has 748 RBG images of WPCBs, with a resolution of 4928 × 3280 pix‐ circuits present on the boards are all labeled, totaling 9313 components. els, from 165 different boards (three to five images per board). Furthermore, the integrated The WPCBs were extracted from a recycling facility, and some WPCBs had broken circuits present on the boards are all labeled, totaling 9313 components. parts and dust all over them. The WPCBs had different sizes and shapes, as can be seen The WPCBs were extracted from a recycling facility, and some WPCBs had broken in Figure5. This makes the dataset very close to what is encountered in a real-world parts and dust all over them. The WPCBs had different sizes and shapes, as can be seen in scenario. For each PCB, the dataset has images in different positions. This is important Figure 5. This makes the dataset very close to what is encountered in a real‐world scenario. because usually, WPCBs are placed on conveyor belts with arbitrary orientation, and a computer-vision system has to perform visual inspection despite the PCB orientation. Appl. Sci. 2021, 11, x FOR PEER REVIEW 7 of 11

For each PCB, the dataset has images in different positions. This is important because usu‐ ally, WPCBs are placed on conveyor belts with arbitrary orientation, and a computer‐vi‐ sion system has to perform visual inspection despite the PCB orientation. Appl. Sci. 2021, 11, x FOR PEER REVIEW 7 of 11 In order to obtain the IC surface area, we needed to evaluate the dataset image scale. The silkscreen printing easily identified some IC part numbers on the package. With the measurement of the IC dimensions present in the datasheet, it is possible to calculate the Appl.For each Sci. 2021 PCB,, 11 ,the 2808 dataset hasreal images object in size different scale. positions.It is important This tois importantnote that the because dataset usu authors‐ do not provide infor7 of‐ 11 ally, WPCBs are placed on conveyormation about belts the with camera arbitrary calibration. orientation, Table and 2 shows a computer the pixel‐vi area‐ estimate for a particular sion system has to perform ICvisual (part inspection number RTL810L). despite the PCB orientation. In order to obtain the IC surface area, we needed to evaluate the dataset image scale. The silkscreen printing easily identified some IC part numbers on the package. With the measurement of the IC dimensions present in the datasheet, it is possible to calculate the real object size scale. It is important to note that the dataset authors do not provide infor‐ mation about the camera calibration. Table 2 shows the pixel area estimate for a particular IC (part number RTL810L).

FigureFigure 5. Some 5. Some instances instances of ofthe the PCB PCB DSLR DSLR Dataset. Dataset.

Table 2.In Pixel order area to estimate. obtain the IC surface area, we needed to evaluate the dataset image scale. The silkscreen printing easily identified some IC part numbers on the package. With the Dimension in Pix‐ Datasheet Di‐ Pixel Area measurementImage ofIC the Part IC Number dimensions present in the datasheet, it is possible to calculate els [px] mensions [mm] [px/mm2] the real object size scale. It is important to note that the dataset authors do not provide Figure 5. Some instances of the PCBinformation DSLR Dataset. about the camera calibration. Table2 shows the pixel area estimate for a particular IC (part number RTL810L). RTL810L 116 × 116 14.00 × 14.00 68.653 Table 2. Pixel area estimate. Table 2. Pixel area estimate. Dimension in Pix‐ Datasheet Di‐ Pixel Area Image IC Part Number 2 Image IC Part Number Dimensionels [px] in Pixelsmensions [px] Datasheet[mm] [px/mm Dimensions2] [mm] Pixel Area [px/mm ] For all experiments, we used the pre‐trained YOLOv3 provided by Darknet [16]. The fine‐tune strategy was to restore the weights from all layers except the last three convolu‐ RTL810LRTL810Ltional layers. 116Additionally, × 116 × 116 the14.00 YOLOv3 × 14.00 head 14.00 was68.653× fine14.00 ‐tuned with a momentum 68.653 opti‐ mizer and piecewise learning rate starting at 1 × 10−6, decaying at every five epochs (decay −6 factor of 0.96) until it reached 1 × 10 . With the augmented images dataset, 30 original WPCB images were reserved for the For all experiments, wetest used dataset, the pre and‐trained the other YOLOv3 images provided were split by intoDarknet training [16]. (80%) The and validation (20%). The For all experiments, we used the pre-trained YOLOv3 provided by Darknet [16]. fine‐tune strategy was to restoremetric the used weights to evaluate from all the layers IC detection except the was last the three all‐point convolu mean‐ average precision (mAP) The fine-tune strategy was to restore the weights from all layers except the last three tional layers. Additionally,[24]. the YOLOv3Since there head was was a single fine ‐class,tuned the with mAP a momentum was the same opti as‐ the AP (average precision) convolutional layers. Additionally, the YOLOv3 head was fine-tuned with a momentum mizer and piecewise learningfor rate the startingIC class, at defined 1 × 10−6 ,as decaying at every five epochs (decay optimizer and piecewise learning rate starting at 1 × 10−6, decaying at every five epochs factor of 0.96) until it reached 1 × 10−6. −6 (decay factor of 0.96) until𝐴𝑃 it reached𝑅 1𝑅× 10𝑃. 𝑅 , (4) With the augmented images dataset, 30 original WPCB images were reserved for the With the augmented images dataset, 30 original WPCB images were reserved for test dataset, and the other imageswherethe test wereR is dataset, a split recall into andlevel training the and other (80%) images and werevalidation split into (20%). training The (80%) and validation (20%). metric used to evaluate the ICThe detection metric was used the to all evaluate‐point mean the IC average detection precision was the (mAP) all-point mean average precision [24]. Since there was a single class, the mAP was the same𝑃 as the𝑅 AP max(average𝑃𝑅 precision). (mAP) [24]. Since there was a single class,: the mAP was the same as the AP (average(5) for the IC class, defined as precision) for the IC class, defined as 𝐴𝑃4.2. IC Object𝑅 𝑅 Detecion𝑃 𝑅 , (4) AP = (R + − Rn)P (R + ) , (4) ∑ n 1 interp n 1 n where R is a recall level and where R is a recall level and 𝑃𝑅 max𝑃𝑅. : (5) Pinterp(Rn+1) = max P Re . (5) Re:Re≥R 4.2. IC Object Detecion 4.2. IC Object Detecion The achieved results for IC object detection are shown in Table3. The precision was 0.917, and the mAP was 86.77%. Figure6 shows a qualitative analysis of the results. It is Appl. Sci. 2021, 11, x FOR PEER REVIEW 8 of 11

Appl. Sci. 2021, 11, 2808 8 of 11 The achieved results for IC object detection are shown in Table 3. The precision was 0.917, and the mAP was 86.77%. Figure 6 shows a qualitative analysis of the results. It is Appl. Sci. 2021, 11, x FOR PEER REVIEW 8 of 11 possiblepossible to to notice notice small small deviations deviations related related to to bounding bounding boxes, boxes, in general, in general, size, size, centralization, centraliza‐ andtion, rotation. and rotation. However, However, even with even these with problems, these problems, a proper a generalizationproper generalization of the trained of the modeltrained was model noticed. was Fornoticed. example, For example, in some labels, in some speciﬁc labels, components specific components were not considered were not integratedconsideredThe achieved circuits, integrated results but circuits, the for model IC object but was the detection able model to detectwas are ableshown them. to detectin Table them. 3. The precision was 0.917, and the mAP was 86.77%. Figure 6 shows a qualitative analysis of the results. It is possibleTable 3. toPrecision, notice small Recall, deviations and mAP for related the IC to detector bounding model. boxes, in general, size, centraliza‐ tion, and rotation. However, even with these problems, a proper generalization of the Model Recall Precision mAP trained model was noticed. For example, in some labels, specific components were not consideredFine-tuned integrated YOLOv3 circuits, but 0.894 the model was able to 0.917 detect them. 86.77%

Figure 6. Some partitioned instances of the PCB DSLR Dataset, the left image pair contains the expected results and the right one the obtained results.

Table 3. Precision, Recall, and mAP for the IC detector model.

Model Recall Precision mAP Figure 6. Some partitioned instances of the PCB DSLR Dataset, the left image pair contains the FigureFine 6. Some‐tuned partitioned YOLOv3 instances of0.894 the PCB DSLR Dataset, the0.917 left image pair contains86.77% the expectedexpected results results and and the the right right one one the the obtained obtained results. results.

Table4.3.4.3. 3. IC Precision, Area ResultsResults Recall, and mAP for the IC detector model. Using the pixel area scale, Figure 7 shows the predicted and ground truth IC area in UsingModel the pixel area scale, FigureRecall7 shows the predictedPrecision and ground truthmAP IC area in mm𝑚𝑚22 forfor each each WPCB WPCB image. image. ItIt isis possiblepossible toto notenote thatthat thethe PCBPCB DSLRDSLR DatasetDataset hadhad somesome Fine‐tuned YOLOv3 0.894 0.917 86.77% imagesimages withwith highhigh ICIC densitydensity (high(high ICIC area)area) andand othersothers withwith veryvery lowlow ICIC densitydensity (low(low ICIC area)area) oror eveneven WPCBsWPCBs withoutwithout anyany ICIC (IC(IC areaarea equaled equaled zero). zero). 4.3. IC Area Results Using the pixel area scale, Figure 7 shows the predicted and ground truth IC area in 𝑚𝑚2 for each WPCB image. It is possible to note that the PCB DSLR Dataset had some images with high IC density (high IC area) and others with very low IC density (low IC area) or even WPCBs without any IC (IC area equaled zero).

FigureFigure 7.7. ComparisonComparison ofof ICIC areaarea (mm(𝑚𝑚22)) calculatedcalculated andand realreal for for each each PCB. PCB.

FigureFigure8 a8a shows shows the the graph graph of the of rootthe meanroot mean squared squared error (RMSE) error (RMSE) between between the ground the truthground IC truth area andIC area the YOLOand the predicted YOLO predicted IC area. IC The area. highest The RMSEhighest value RMSE was value 6192.38. was The6192.38. RMSE The value RMSE had value a high had dispersion a high dispersion (average of (average 477.12 mm of 477.122 and amm standard2 and a deviation standard 2 Figureofdeviation 698.61 7. Comparison mmof 698.61). However, of mmIC area2). However,(𝑚𝑚2 most) calculated WPCBs most had WPCBsand areal percentage forhad each a percentage PCB. error below error 2%, below as seen 2%, inas Figureseen in8 b.Figure 8b. Figure 8a shows the graph of the root mean squared error (RMSE) between the ground truth IC area and the YOLO predicted IC area. The highest RMSE value was 6192.38. The RMSE value had a high dispersion (average of 477.12 mm2 and a standard deviation of 698.61 mm2). However, most WPCBs had a percentage error below 2%, as seen in Figure 8b. Appl. Sci. 2021, 11, 2808 9 of 11 Appl. Sci. 2021, 11, x FOR PEER REVIEW 9 of 11

(a) (b)

FigureFigure 8. 8. RootRoot mean mean squared squared error error (RMSE) (RMSE) for for every every WPCB WPCB in in the the dataset dataset ( (aa)) and and IC IC area area percentual percentual error error for for every every WPCB WPCB inin the the dataset ( (b).).

4.4.4.4. Estimated Estimated IC IC Weight Weight UsingUsing the the calculated calculated IC IC surface surface density, density, the total IC weight was calculated based on thethe total total IC IC area. area. Table Table 44 showsshows thethe totaltotal areaarea andand weightweight forfor thethe groundground truthtruth andand YOLOYOLO predictions.predictions. The The error error between between the the IC IC weighed weighed calculated calculated from from the the ground ground truth, truth, and and the oneone calculated calculated from from YOLO YOLO predictions predictions was 160.72 g.

TableTable 4. WPCB’s total IC area and weight.

MetricMetric PredictedPredicted GroundGround Truth Truth ErrorError TotalTotal IC Area IC Area (mm) (mm) 525,828.2525,828.2 457,667.37457,667.37 14.89%14.89% TotalTotal IC Weight IC Weight (g) (g) 1239.91239.9 1079.181079.18 14.89%14.89% Average RMSE (mm) 477.12 (698.61) Average RMSE (mm) 477.12 (698.61)

4.5. Calculated Recycling Metal Weight 4.5. Calculated Recycling Metal Weight Metals are commodities whose values fluctuate according to market factors. In order Metals are commodities whose values ﬂuctuate according to market factors. In order to calculate the financial return of each PCB, the average value of metal ore was collected to calculate the ﬁnancial return of each PCB, the average value of metal ore was collected from the stock market. Table 5 shows the estimated return of recycling of all the PCB’s from the stock market. Table5 shows the estimated return of recycling of all the PCB’s ICs, and the right axis of Figure 7 shows the return of IC recycling for each PCB. The ICs, and the right axis of Figure7 shows the return of IC recycling for each PCB. The maximum return was USD0.59 (USD0.69 with the IC detector model estimative), and the maximum return was USD0.59 (USD0.69 with the IC detector model estimative), and the average return was USD0.05 (standard deviation of USD0.06). It is important to state that average return was USD0.05 (standard deviation of USD0.06). It is important to state that these incomes do not consider precious metals as gold and silver, so it is a fraction of the these incomes do not consider precious metals as gold and silver, so it is a fraction of the recyclingrecycling outcomes. outcomes.

Table 5. Total IC recycling estimation outcomes. Table 5. Total IC recycling estimation outcomes. Recycling Rate IC Detection Model Ground Truth Metal Value IC Detection Model Ground Truth Metal Metal Value Recycling(g of Metal Rate/Kg of Metal (USD/Ton) g of Metal USD Return g of Metal USD Return (USD/Ton) (g of Metal/KgIC) of IC) g of Metal USD Return g of Metal USD Return FeFe 100100 68.8868.88 85.40 85.40 0.00854 0.00854 74.33 74.33 0.00743 0.00743 NiNi 13,88713,887 31.231.2 38.68 38.68 0.53722 0.53722 33.67 33.67 0.46758 0.46758 CuCu 64406440 331.5331.5 411.03 411.03 2.64701 2.64701 357.74 2.30390 2.30390 SiSi 10,50010,500 411.6411.6 510.34 510.34 5.35860 5.35860 444.19 444.19 4.66400 4.66400 TotalTotal ‐ ‐ - -1045.46 1045.46 8.55 909.94 7.44 7.44

5.5. Discussion Discussion TheThe proposed proposed economic economic evaluation evaluation output output was was a a total total of of USD7.44 USD7.44 (USD8.55 with IC detectordetector model) with USD0.59 asas thethe highesthighest value value for for a a PCB PCB and and (USD0.69 (USD0.69 with with IC IC detector detec‐ tormodel model estimative), estimative), and theand average the average return return was USD0.05 was USD0.05 (standard (standard deviation deviation of USD0.06). of USD0.06).However, However, precious metals precious (Au, metals Pd, Pt, (Au, and Pd, Ag) Pt, are and a smallAg) are fraction a small of fraction the ICs’ of mass the ICs’ and mass and were not considered in this small sample. As precious metals are responsible Appl. Sci. 2021, 11, 2808 10 of 11

were not considered in this small sample. As precious metals are responsible for most of the ﬁnancial return of WPCB recycling, an increase in the gross income of IC recycling is expected when also recovering precious metals like gold and silver. The complete estimation of WPCB compositions depends on more data regarding the composition of each electronic component. Research in these directions could reduce the uncertainty in the evaluation of a WPCB composition. A ﬁrst application of the proposed method is to cluster PCBs with similar IC densities. Those similar WPCBs could suffer the same recycling process because they probably have similar compositions. Another critical component is the market value of the metal. The market value is positively affected by the purity of the metal. Therefore, depending on the used recycling technique, the gross outcome may be different.

6. Conclusions Recycling waste printed circuit boards (WPCBs) is very important for recovering high-value metals and providing an environmentally safe destination for these electronic residues. Integrated circuit detection and other components can improve the PCB recycling process since it improves automatic disassembly systems and reuses some components. In this sense, the PCB DSLR public dataset was published to support the research and development of computer-vision strategies for PCB recycling. This paper proposed a WPCB economic feasibility assessment (WPCB-EFA) method for recycling ICs from PCBs. As WPCBs have a high composition diversity, the main goal is to provide information regarding the WPCB composition to guide the recycling process such as the need to dismantle a particular PCB before the physical or mechanical recycling process. The ﬁrst step of the WPCB-EFA is to detect ICs in a PCB. Deep learning object detection techniques require large-scale and fully-annotated datasets. Since the PCB DSLR dataset contains only 748 images from 165 different PCBs, transfer learning from the YOLOv3 pre-trained model was evaluated. In order to achieve better results in IC detection, the PCB DSLR Dataset images were split into squares of 416 × 416 pixels, resulting in a total of 5347 images containing integrated circuits. With this image size, YOLOv3 ﬁne-tuning resulted in 0.965 mAP. Qualitative analysis suggests that the model’s errors occur when part of the IC is in the image. Considering the need for calculating the IC weight, we provide an IC surface density estimate based on the IC manufacturing data. Together with ICs recycling outcomes reported in the literature, we calculated the expected recovery mass for some elements. We noted that some PCBs have much more value than others and may go through different recycling processes. For instance, some PCBs may go to crushing, while others need dismantling before directly crushing. The proposed method may help to optimize the recycling process by such analysis. The economic evaluation of PCB recycling can be one motivational factor for recycling, aside from the urgent need to deal with electronic waste. In future works, it is necessary to expand the recycling economic evaluation to other PCB components aside from integrated circuits.

Author Contributions: Conceptualization, L.H.d.S.S.; Methodology, L.H.d.S.S. and B.J.T.F.; Software, A.A.F.J. and L.H.d.S.S.; Investigation, L.H.d.S.S.; Resources, S.C.O. and B.J.T.F.; Writing—original draft preparation, L.H.d.S.S. and G.O.A.A.; Writing—review and editing, L.H.d.S.S., G.O.A.A., and B.J.T.F.; Supervision, B.J.T.F. and S.C.O. All authors have read and agreed to the published version of the manuscript. Funding: This study was ﬁnanced in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior-Brasil (CAPES)-Finance Code 001, and the Brazilian agencies FACEPE and CNPq. Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Appl. Sci. 2021, 11, 2808 11 of 11

Data Availability Statement: The PCB DSRL dataset is a public dataset available in [10]. Source code of WPCB-EFA is available in https://github.com/ldhonorato/WPCB-EFA (accessed on 1 January 2021). Conﬂicts of Interest: The authors declare no conﬂict of interest.

References 1. Marques, A.C.; Marrero, J.-M.; de Fraga Malfatti, C. A review of the recycling of non-metallic fractions of printed circuit boards. Springerplus 2013, 2, 1–11. [CrossRef][PubMed] 2. Niu, Q.; Xiang, N.; Liu, X.; Duan, G.; Shi, C. The recycle model of printed circuit board and its economy evaluation. In Proceedings of the 2007 IEEE International Symposium on Electronics and the Environment, Orlando, FL, USA, 7–10 May 2007; pp. 106–111. 3. Regel-Rosocka, M. Electronic wastes. Phys. Sci. Rev. 2018, 3, 1–30. 4. Dieste, M.; Viagi, A.F.; Panizzolo, R.; Dos Santos, R.F.; Marins, F.A.S. Reverse logistics models for the collection of Waste Electrical and Electronic Equipment: The Brazilian case. In Proceedings of the 2018 International Conference on Production and Operations Management Society (POMS), Peradeniya, Sri Lanka, 14–16 December 2018; pp. 1–8. 5. Sohaili, J.; Muniyandi, S.K.; Mohamad, S.S. A Review on Printed Circuit Boards Waste Recycling Technologies and Reuse of Recovered Nonmetallic Materials. Int. J. Sci. Eng. Res. 2012, 3, 1–7. 6. Marra, A.; Cesaro, A.; Belgiorno, V. The recovery of metals from WEEE: State of the art and future perspectives. Glob. NEST J. 2018, 20, 679–694. 7. Gao, Z.; Li, J.; Zhang, H.C. Printed circuit board recycling: A state-of-art survey. IEEE Trans. Electron. Packag. Manuf. 2003, 27, 33–42. 8. Lee, C.H.; Chang, C.T.; Fan, K.S.; Chang, T.C. An overview of recycling and treatment of scrap computers. J. Hazard. Mater. 2004, 114, 93–100. [CrossRef][PubMed] 9. Awasthi, A.K.; Zlamparet, G.I.; Zeng, X.; Li, J. Evaluating waste printed circuit boards recycling: Opportunities and challenges, a mini review. Waste Manag. Res. 2017, 35, 346–356. [CrossRef][PubMed] 10. Pramerdorfer, C.; Kampel, M. A dataset for computer-vision-based PCB analysis. In Proceedings of the 2015 14th IAPR International Conference on Machine Vision Applications (MVA), Tokyo, Japan, 18–22 May 2015; pp. 378–381. 11. Knoth, R.; Hoffmann, M.; Kopacek, B.; Kopacek, P. Intelligent disassembly of electr(on)ic equipment. In Proceedings of the 2nd International Symposium on Environmentally Conscious Design and Inverse Manufacturing, Tokyo, Japan, 11–15 December 2001; pp. 557–561. 12. Kuo, C.-W.; Ashmore, J.D.; Huggins, D.; Kira, Z. Data-Efﬁcient Graph Embedding Learning for PCB Component Detection. In Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa Village, HI, USA, 7–11 January 2019; pp. 551–560. 13. Chen, H.; Wang, Y.; Wang, G.; Bai, X.; Qiao, Y. Progressive Object Transfer Detection. IEEE Trans. Image Process. 2020, 29, 986–1000. [CrossRef][PubMed] 14. Wang, Y.; Yao, Q.; Kwok, J.T.; Ni, L.M. Generalizing from a Few Examples. ACM Comput. Surv. 2020, 53, 1–34. [CrossRef] 15. Kaya, M. Electronic Waste and Printed Circuit Board Recycling Technologies; Springer: Cham, Switzerland, 2019. 16. Redmon, J.; Farhadi, A. YOLOv3: An Incremental Improvement. arXiv 2018, arXiv:1804.02767. 17. Wong, M.H.; Wu, S.; Deng, W.; Yu, X.; Luo, Q.; Leung, A.; Wong, C.; Luksemburg, W.; Wong, A. Export of toxic chemicals—A review of the case of uncontrolled electronic-waste recycling. Environ. Pollut. 2007, 149, 131–140. [CrossRef][PubMed] 18. Duan, H.; Hou, K.; Li, J.; Zhu, X. Examining the technology acceptance for dismantling of waste printed circuit boards in light of recycling and environmental concerns. J. Environ. Manag. 2011, 92, 392–399. [CrossRef][PubMed] 19. Huisman, J. QWERTY and Eco-Efficiency Analysis on Cellular Phone Treatment in Sweden; TU Delft: Delft, The Netherlands, 2004; pp. 1–33. 20. Barnwal, A.; Dhawan, N. Physical Processing of Discarded Integrated Circuits for Recovery of Metallic Values. JOM 2020, 72, 2730–2738. [CrossRef] 21. Lee, C.-H.; Tang, L.-W.; Popuri, S.R. A study on the recycling of scrap integrated circuits by leaching. Waste Manag. Res. 2010, 29, 677–685. [CrossRef][PubMed] 22. Pathak, A.R.; Pandey, M.; Rautaray, S. Application of Deep Learning for Object Detection. Procedia Comput. Sci. 2018, 132, 1706–1717. [CrossRef] 23. Mortan, F.; Wright, L. Quad Flatpack No-Lead Logic Packages; Application Report February 2004; Texas Instruments: Dallas, TX, USA, 2004; pp. 1–47. 24. Padilla, R.; Netto, S.L.; Da Silva, E.A.B. A Survey on Performance Metrics for Object-Detection Algorithms. In Proceedings of the 2020 International Conference on Systems, Signals and Image Processing (IWSSIP), Niterói, Brazil, 3–5 June 2020; pp. 237–242.