Harnessing Multiscale Nonimaging Optics for Automotive Flash LiDAR and
Heterogenous Semiconductor Integration
by
Todd Houghton
A Dissertation Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy
Approved July 2020 by the Graduate Supervisory Committee:
Hongbin Yu, Chair Hanqing Jiang Suren Jayasuriya Liang Zhang
ARIZONA STATE UNIVERSITY
August 2020 ABSTRACT
Though a single mode of energy transfer, optical radiation meaningfully interacts with its surrounding environment at over a wide range of physical length scales. For this reason, its reconstruction and measurement are of great importance in remote sensing, as these multi-scale interactions encode a great deal of information about distant objects, surfaces, and physical phenomena. For some remote sensing applications, obtaining a desired quantity of interest does not necessitate the explicit mapping of each point in object space to an image space with lenses or mirrors. Instead, only edge rays or physical boundaries of the sensing instrument are considered, while the spatial intensity distribution of optical energy received from a distant object informs its position, optical characteristics, or physical/chemical state.
Admittedly specialized, the principals and consequences of non-imaging optics are nevertheless applicable to heterogeneous semiconductor integration and automotive light detection and ranging (LiDAR), two important emerging technologies. Indeed, a review of relevant engineering literature finds two under-addressed remote sensing challenges. The semiconductor industry lacks an optical strain metrology with displacement resolution smaller than 100 nanometers capable of measuring strain fields between high-density interconnect lines. Meanwhile, little attention is paid to the per- meter sensing characteristics of scene-illuminating flash LiDAR in the context of automotive applications, despite the technology’s much lower cost. It is here that non- imaging optics offers intriguing instrument design and explanations of observed sensor performance at vastly different length scales.
i In this thesis, an effective non-contact technique for mapping nanoscale mechanical strain fields and out-of-plane surface warping via laser diffraction is demonstrated, with application as a novel metrology for next-generation semiconductor packages. Additionally, object detection distance of low-cost automotive flash LiDAR, on the order of tens of meters, is understood though principals of optical energy transfer from the surface of a remote object to an extended multi-segment detector. Such information is of consequence when designing an automotive perception system to recognize various roadway objects in low-light scenarios.
ii DEDICATION
This work is dedicated to my mom, dad, brother, friends, and colleagues, who have
supported and believed in me throughout my time at the university.
iii ACKNOWLEDGMENTS
I would like to gratefully acknowledge the following individuals. Thanks to them, I have achieved more in my academic career than I ever thought possible:
Professor Hongbin Yu for his invaluable guidance, knowledge, mentorship, and generous support throughout my graduate career.
My committee members: Dr. Hanqing Jiang, Dr. Suren Jayasuriya, and Dr. Liang Zhang for their helpful suggestions and evaluation of my work.
Kenneth Mossman of the John M. Cowley Center for High Resolution Electron
Microscopy for providing support and training on equipment invaluable to my research.
My fellow graduate students and colleagues, Jignesh Vanjaria, John Kevin Cava, Michael
Saxon, Haokai Yang, and Mayukh Nandy for their thought-provoking discussions, hard work, and comradery.
iv TABLE OF CONTENTS
Page
LIST OF TABLES ...... vii
LIST OF FIGURES ...... viii
CHAPTER
1 INTRODUCTION ...... 1
2 ADVANCED SEMICONDUCTOR PACKAGING: SCALING,
TECHNOLOGIES, AND FUTURE CHALLENGES ...... 4
2.1 Semiconductor Scaling and Contemporary Microelectronics in 2020 ...... 4
2.2 Advanced Packaging Technologies and Dense Interconnects ...... 6
2.3 Thermal Considerations in Advanced Packaging ...... 14
2.4 Advanced Packaging and the Automotive Market ...... 20
3 NANOSCALE STRAIN SENSING VIA LASER DIFFRACTION ...... 28
3.1 Measuring Strain in Microelectronics Packages ...... 28
3.2 Strain Mapping of Package Microstructures via Laser Diffraction ...... 42
3.3 Theoretical Background ...... 45
3.4 Experimental Methods ...... 62
3.5 Full-Field Pitch Mapping using Computer Automation ...... 76
3.6 Results and Discussion ...... 84
3.7 Conclusion ...... 91
4 EMERGING TECHNOLOGIES FOR VEHICLE PERCEPTION ...... 93
4.1 Introduction ...... 93
4.2 Cameras and Computer Vision ...... 94
v CHAPTER Page
4.3 Light Detection and Ranging (LiDAR) ...... 104
4.4 Millimeter Wave Radar ...... 115
4.5 Sonar ...... 117
4.6 Sensor Fusion ...... 120
4.7 Cost ...... 125
5 ROADWAY OBJECT DETECTION VIA LOW-COST FLASH LIDAR AND
CAMERA ...... 128
5.1 Introduction ...... 128
5.2 Methods ...... 130
5.3 Results and Discussion ...... 141
5.4 Conclusion ...... 147
6 CONCLUSION ...... 149
REFERENCES ...... 124
vi LIST OF TABLES
Table Page
1. Changes in Grating Pitch due to Thermal Expansion ...... 48
2. Influence of Temperature-Dependent nair on Calculated Grating Pitch ...... 52
3. Expected Values of Δθ1 Computed Using Equation 3.32 ...... 60
4. Absolute Change in Pixel Position, |Δp|, of The Diffracted Beam’s Center ...... 62
5. Itemized Cost of the Flash LiDAR/Camera Sensor Platform ...... 132
vii LIST OF FIGURES
Figure Page
1. Trends in Semiconductor Performance...... 5
2. Fan-Out Wafer Level Packaging ...... 8
3. Interconnect Structures Used in 2.5D Packaging ...... 10
4. Silicon Die Stacking Methods ...... 12
5. Concept Design of a Monolithic 3D Integrated Circuit and 3D NAND ...... 14
6. Typical Die/Package Cooling Schemes and Trends in Thermal Density ...... 16
7. Embedded Liquid Cooling Schemes...... 19
8. Differences Between Automotive and Consumer Test Requirements ...... 21
9. Cost of Electronic Components as a Percentage of Automotive Costs ...... 22
10. Advanced Packaging Schemes for Image Sensors, Antennas, and LiDAR ...... 25
11. Early Digital Image Correlation (DIC) Setups ...... 30
12. Hardware Configuration for 2D DIC, Some Sample Speckle Patterns ...... 31
13. Decomposition of a Matched Pixel Subset’s Displacement and Shape vector .... 33
14. Displacement Fields of a Solder Ball and Crack Board Measured via DIC ...... 37
15. Generating a Moiré Pattern, Configuration of a Shadow Moiré tool ...... 38
16. Sample Preparation, Working Principal, and Results of Moiré Interferometry ... 40
17. Illustrated Concept of Nano-Scale Strain Detection via Laser Diffraction ...... 44
18. Theory of Mechanical Strain in Diffraction Gratings ...... 47
19. Diffractive Orders of Transmission and Reflection Gratings ...... 51
20. Interference Properties of Huygens Wavelets ...... 53
21. Predicted Fraunhofer Intensity Profile of Diffracted Light ...... 56
viii Figure Page
22. Horizontal Translation of a Diffractive Order due to Changes in Pitch ...... 58
23. Deposition of a 50nm Gold (Au) Thin Film on a Sample’s Cross Section ...... 63
24. Illustration of a Dual-Beam FIB-SEM Instrument ...... 64
25. SEM Micrograph of Fabricated Gratings ...... 66
26. Experimental Setup for Full-Field Areal Pitch Mapping ...... 68
27. Image of Sample Surface, Gratings and Laser Beam Spot are Visible ...... 69
28. 2D and 1D Intensity Distributions of Light Diffracted into the m=1 Order ...... 70
29. Pixel-to-Angle Calibration Procedure and Resultant Calibration Curve ...... 72
30. Step and Scan Procedure for High-Resolution 2D Pitch Mapping ...... 74
31. LabVIEW Architecture Automating Image Collection and Areal Scanning ...... 77
32. LabVIEW architecture Converting 2D Images to 1D Intensity Profiles ...... 79
33. Python 3 Architecture Fitting 1D Intensity Profiles to a Gaussian Function ...... 81
34. Python 3 Programming Architecture for Constructing Areal Pitch Maps ...... 83
35. Measured Pitch vs. Laser Spot Position for Two Calibration Gratings ...... 85
36. Single-Line Pitch Scans of a Grating ROI Containing a Though-Silicon Via ..... 86
37. High-Resolution Grating Pitch Maps of a TSV at Multiple Temperatures ...... 88
38. Measured Strain in Cu and Si Sub-Regions, Surface Angle Mapping ...... 90
39. The Role of CMOS Image Sensors For Autonomous Perception Tasks ...... 94
40. Backside Illuminated CMOS for High Dynamic Range (HDR) Imaging ...... 96
41. Impact of Camera View Field on Perceived Object Size ...... 97
42. Epipolar Geometry of Stereovision and Typical Programmatic Taxonomy ...... 99
43. Comparing Disparity Maps of Driving Scenes ...... 102
ix Figure Page
44. Monocular Depth Estimation of a Driving Scene Using a Neural Network ...... 103
45. Airborne LiDAR for High Resolution Mapping of Geologic Features ...... 105
46. Self-Driving Vehicles Participating in the 2005 DARPA Grand Challenge ...... 106
47. Basic Configuration of a LiDAR System for Object Rangefinding ...... 107
48. An InGaAs Quantum Well Laser Diode, Laser Power Classification ...... 108
49. Working Principal of Silicon Avalanche Photodiodes (APDs) ...... 110
50. Role of Lenses and Filters in a Functioning LiDAR System ...... 112
51. Depiction of Emitter-Detector Pair Geometry and Supporting Optics ...... 113
52. Spinning-Head and Solid-State Automotive LiDAR Products ...... 114
53. Advantages and Disadvantages of 77GHz Radar vs LiDAR ...... 116
54. Working Principals and Range of Sonar-Based Sensors ...... 118
55. Automotive Sensor Fusion Combining Multiple Sensing Modalities ...... 121
56. Various Sensor Fusion Schemes for Neural Networks ...... 124
57. Photographs of the Low-Cost LiDAR/Camera Sensor Platform ...... 131
58. Multi-Segment LiDAR Returns Overlaid on Video Data ...... 134
59. Transforming LiDAR Returns to Image Coordinates ...... 137
60. Visible/NIR Reflectivity Data of Sample Fabrics ...... 140
x CHAPTER 1
INTRODUCTION
Though a singular means of energy transfer, the range of physical lengths over which optical radiation interacts meaningfully with its surrounding environment is remarkably vast. For this reason, it is often the sensing modality of choice for remotely observing objects, surfaces, and a variety of physical phenomenon. Astronomical instruments, such as telescopes or spectrometers produce detailed images and spectral compositions of light originating from celestial bodies millions of kilometers from Earth, while optical microscopes can resolve objects less than a micrometer in size. In some instances, the wavelike nature of light may even be used to sense physical phenomena occurring at length scales much smaller than the interacting radiation, allowing measurements at the nanometer scale. Many important detection and sensing applications lie between these extremes.
While optical radiation may interact with its surrounding environment over a vast range of length scales, it’s utility as a remote sensing modality would be limited if interactions observed at one length scale could not be easily transformed to another through magnification or reduction. Fortunately, light propagation follows a geometric ray model in many practical instances, enabling construction of optical systems made from lenses and/or mirrors. Such systems collect and focus light from distant objects, mapping points in object space to points on a photosensitive detector, forming an image.
This mapping may confer a great degree of magnification or reduction, permitting high- resolution non-contact sensing of distant objects large and small.
1 For some remote sensing applications, measurement of optical radiation does not necessitate a lens which explicitly maps each point in object space to a point in image space. Instead, relevant optical metrologies are based on principals of energy transfer, in which only edge rays or physical boundaries of the sensing instrument are considered.
Here, the spatial intensity distribution of optical energy received from a distant object contains information regarding the object’s position, reflection characteristics, or physical/chemical state. Multi-scale magnification is less straightforward for non- imaging optical systems. However, lenses which condense incoming light onto a small extended detector surface find use in non-interferometric instruments such as automotive
LiDAR while some characteristic of intensity distributions generated from optical interference may be geometrically amplified through instrument design, although such steps are often unnecessary for interferometers.
In practical terms, the utility of non-imaging optical systems may be best appreciated in the context of two specific measurement scenarios: (1) Where the optical energy received from a distant object or surface is of low intensity, requiring its distribution over a high-gain and/or multi-segmented areal detector by a condenser lens or (2) when the received intensity distribution can be used to directly measure a quantity of interest, as is often the case when interferometric or diffraction-based metrologies are involved. Admittedly specialized, the principals and consequences of non-imaging optics are nevertheless highly applicable when two important emerging technologies are considered. Namely, heterogonous semiconductor integration and automotive LiDAR.
Within these two realms, non-imaging optics offers intriguing instrumentation design and explanations of observed sensor performance at vastly different length scales.
2 Here, an effective non-contact technique for sensing nanoscale mechanical strain fields and out-of-plane surface warping via laser diffraction is demonstrated, with application as a novel metrology for next-generation semiconductor packages. Additionally, object detection distance of an automotive flash LiDAR module, on the order of tens of meters, is understood though principals of optical energy transfer from the surface of a remote object to an extended multi-segment detector. Such information is of consequence when designing an automotive perception system to recognize various roadway objects, especially in low light scenarios where cameras may be of limited use.
In this thesis, the case for developing a full field nanoscale strain metrology for advanced microelectronics packages is made in chapter two. Chapter three describes this novel metrology in detail, reporting tests and qualification results. Chapter four serves as a primer to automotive perception technologies currently deployed on prototype autonomous roadway vehicles, while chapter five reports on the construction and analysis of a low-cost camera/LiDAR sensing platform for automotive perception and highlights the importance of non-imaging optical principals in this application. Finally, chapter six offers a brief summary of all research results in the context of emerging automotive perception technologies and advanced semiconductor packaging.
3 CHAPTER 2
ADVANCED SEMICONDUCTOR PACKAGING: SCALING, TECHNOLOGIES,
AND FUTURE CHALLENGES
2.1 Semiconductor Scaling and Contemporary Microelectronics in 2020
Over the past fifty years, the semiconductor industry has relied extensively on transistor scaling to drive growth, enabling a vast number of new technologies, industries, and innovations across every sector of the global economy. Indeed, scaling has resulted in contemporary silicon chips containing nearly 50 billion transistors, making programmable computers a general-purpose technology. This extraordinary large number of devices is, in part, a product of the industries’ commitment to double the number of transistors present on a silicon die every two years, in accordance with a prediction made by Gordon Moore in 1965.
While Moore’s law has largely been upheld over the past several decades, adherence has become increasingly difficult and expensive in recent years as semiconductor node sizes fall below ten nanometers. Extreme Ultraviolet Lithography
(Miyazaki & Yen, 2019) is the fabrication process of choice for such small features but requires complex equipment and state-of-the-art facilities (Haga, 2018). It is estimated that the cost of a new fabrication plant capable of producing the smallest transistors in the year 2022 will be greater than 16 billion USD. As only a few manufacturers, namely
Intel, TSMC, Samsung, and Global Foundries etc. are likely to continue pursuit of 5nm and 3nm process nodes in the next ten years, alterative scaling methods are needed for the industry as whole.
4 In this regard, improvements to microelectronic packages which provide signal interfaces, power, mechanical rigidity, and thermal dissipation to active silicon offer a path by which scaling continues via multi-die packages and high-density interconnects
(Kwak & Hubing, 2008). Novel packaging techniques have arrived at an important juncture in history of the semiconductor manufacturing; as the pace of transistor scaling decelerates, the number of new computing applications continues to rise. Emerging markets include the internet-of-things, edge computing, smart infrastructure, autonomous vehicles, 5G communication, and artificial intelligence. Indeed, advanced packaging is well-suited to these markets, as dies with different functionalities can be integrated
Figure 1 Trends in semiconductor performance as described in chapter 8 of the 2019 Heterogeneous Integration Roadmap. Tabulated data since the early 1970’s shows that the number of transistors fabricated on a given chip continues to increase. However, other key performance metrics have nearly plateaued. Importantly, the number of transistors which can be bought per dollar has fallen, due to the enormous costs required to run state-of-the-art manufacturing processes. 5 together, resulting in powerful, multi-function, package-level modules (Westling, 2020).
Additionally, multi-die packages may provide expected performance improvements in computing systems between multi-year development cycles of new silicon process nodes, as a form of cooperative scaling.
Importantly, recent collaboration between industry and academia though the
Electronic Packaging Society (EPS) has produced a new Heterogeneous Integration
Roadmap (HIR) identifying future challenges in advanced packaging and possible solutions (W. Chen & Bottoms, 2017). This roadmap, released in 2019, is the largest to date, reflecting the interest of key players in the technology’s long-term potential. While engineering challenges remain, the inherent flexibility and potential scaling capacity of future packaging technologies is compelling at a time when Moore’s law continues to slow.
2.2 Advanced Packaging Technologies and Dense Interconnects
At the package level, scaling is accomplished by increasing the number of silicon dies connected in proximity. Such scaling is advantageous, as each die can be fabricated using mature processes, while dies from both larger and smaller process nodes can be integrated in the same package. (Li et al., 2020) This allows engineers greater flexibility to choose the best node for a die, depending on its function within the system.
Additionally, some connection schemes have shorter interconnect length between different components such as CPU’s and memory modules than monolithic system-on- chip designs, potentially reducing circuit delay (Schaper et al., 2005). While there are many ideas in advanced packaging to be explored, some common paradigms have emerged:
6 2.2.1 Wafer-Level Packaging
In wafer-level packaging (WLP), metallic pads located on a silicon die are bonded to small interconnect lines imbedded in polymer substrate. This interconnect-polymer layer is located beneath the die and is used to reroute electrical connections to various locations near the die. Typically, the layer is referred to as a redistribution layer (RDL).
Wafer-level packaging products are divided into three sub-categories depending on RDL topography. In the simplest configuration, referred to as chip-scale packaging, the RDL is located entirely below the areal footprint of a die, resulting in a completed package not much larger than the die itself (Garrou, 2000). In a second topography referred to as fan- out, the RDL expands into a region surrounding the die (Braun et al., 2019), allowing small dies with numerous pads to be connected to larger components (Lau, 2019). This is important, as small dies are less difficult fabricate and exhibit higher yields than large dies (Jeong et al, 2011). In the third topography, referred to as Fan-In, the RDL bridges an array of solder balls or copper pillars, bonded to an active silicon die, with metallic pads around the die perimeter (Campos et al. 2015).
Unlike ball grid array (BGA) packages, where each die is attached to an individual printed circuit board using small solder balls, fabrication of the RDL occurs at the wafer level. This means that all dies on a given wafer undergo the RDL processing steps simultaneously. A typical WLP process, as outlined in chapter 23 of Heterogeneous
Integration Roadmap (EPS, 2019) involves several steps. First, a silicon wafer is diced into individual dies. Next, the dies are spaced apart on a carrier and the gaps between each filled with polymer molding compound, creating a reconstituted wafer. The RDL is
7 then fabricated on the exposed die face containing copper pads, and completed packages diced into individual units.
While early fan-out package products contained a single die with relatively few electrical connections, the technology advanced greatly in 2016, when the RDL fabrication process was utilized to link multiple dies together with high-density interconnects (C-F. Tseng et al., 2016). Indeed, as of 2016, stacked package-on-package
A.
B. C.
D.
E.
Figure 2: (A.) Illustration of chip-first fan out packaging process flow. (B.) Illustration of RDL functionality, connecting to a large solder ball to small metal pads on a Si wafer. (C.) Illustration of solder bumps intended for connection to a silicon die redistributed to small metal pads using the RDL (D.) SEM image of high-density metal interconnect lines in a fan-out RDL with a 2μm pitch. (E.) Illustration of RDL connecting two silicon dies wafers and various passive circuit elements in a single package. All figures from chapter 23 of the 2019 HIR roadmap 8 and planar multi-die packages making use of fan-out are available commercially. More recently, interest in the technology is prompting development of higher density interconnects in the redistribution layer. Line widths and spacing, also called pitch, are typically on the order of 5-10μm, but spacings below 2μm have been demonstrated (EPS,
2019). In such high-density environments, accurate measurements of sub-micron strain fields will likely be required to understand interconnect reliability and prevent failures.
2.2.2 2.5D Packaging
In 2.5D packaging, dies containing active circuitry are placed side by side, atop a silicon interposer which acts a bridge between dies and the attached circuit board, permitting die-to-die and die-to-board communication (X. Zhang et al., 2015). Typically, active dies are connected to the interposer though a BGA or with small copper pillars.
Lateral die-to-die connections are formed using an RDL located near the interposer’s top surface, while through-silicon vias (TSVs) facilitate vertical die-to-board connections
(Lau et al., 2014). Formation of TSVs involves etching small holes in the silicon interposer, which are then filled with copper (Shen & K. Chen, 2017). Similar to wafer- level packaging, the RDL can be fabricated with copper lines sandwiched between polymer passivation layers. Alternatively, a copper damascene process, which imbeds copper lines in a layer of SiO2 grown on the interposer surface is used.
The primary advantage of 2.5D packaging, compared to WLP, and BGAs is interconnect density. Using copper damascene, lateral line widths and spacings of
≤0.5μm have been reported in chapter 22 of the HIR (EPS, 2019). This results in significantly greater density than what is currently achievable with WLP and BGA, which have pitches of 5-10μm and 40-55μm respectively (EPS, 2019). In the vertical
9 direction, TSV pitch is similar to BGAs, with typical via diameters of 5-50μm (EPS,
2019). While the interposer hosts numerous electrical connections, much of its internal structure remains bulk silicon. This additional bulk has a low coefficient of thermal expansion (CTE) and provides firm mechanical support. As such, 2.5D packaging can accommodate high-performance dies with large areal footprints.
While 2.5D packaging is suitable for performance products, its chief disadvantage is cost. To achieve optimal interconnect density and fabricate TSVs, silicon processing techniques such as oxide formation, photolithography, metal deposition, and reactive ion etching are utilized (X. Zhang et al., 2015), (Lau et al., 2014). These processes are sequential and must often be performed multiple times to form a desired RDL topography, increasing cost and fabrication time. Additionally, cost of silicon processing techniques scale with increasing interposer area. This is problematic, as the interposer must be larger than the footprints of all active dies attached to its surface. Large multi-die topologies compound this issue, as the total interposer area can exceed the lithographic reticle size, requiring multiple exposures and stitching (Premachandran et al., 2019), an
A. B.
Figure 3: (A.) Illustration of two silicon chips connected laterally through the RDL region of a silicon interposer, with vertical connections to underlying package substrate facilitated by through-silicon vias (TSVs). (B.) SEM image of high-density RDL layers fabricated on a silicon interposer using a copper damascene process (Lau et al., 2014). 10 expensive and difficult procedure with contemporary processing equipment. Though silicon interposers continue to be the most widely utilized 2.5D packaging scheme, associated cost challenges have prompted research into alternative designs. Some novel ideas include using small silicon bridge dies for lateral die-to-die interconnect (Mahajan et al., 2016), and a proposal to replace bulk silicon with organic substrate (Oi, et al.,
2014).
2.2.3 3D Integrated Circuits
A three-dimensional integrated circuit is achieved by stacking multiple silicon dies, with die-to-die communication facilitated by vertical interconnections. In such a configuration, areal footprint of the package is minimized while theoretically expanding the multi-die functionality associated with existing 2.5D and WLP paradigms (Z. Wang,
2019). 3D IC packages make extensive use of though-silicon vias, multi-level wire bonding, or both to achieve acceptable though-silicon interconnect densities in the vertical direction as shown in chapter 8 of the HIR (EPS, 2019). Dies are stacked together using copper pillars or micro-bumps which serve as additional die-to-die interconnections while empty space between the bumps or pillars is filled with polymer material (Shen et al., 2018).
Contemporary die-stacking makes use of micro-bumps or copper pillars attached to each die level using thermal compression bonding (TCB). However, the process is not anticipated to be scalable below pitches of 10-20μm, restricting improvements in die-to- die interconnect density (EPS, 2019). While use of multi-level wire bonding can provide additional connections in some instances, the finest wire-pitch realized in volume manufacturing is 35-40μm (EPS, 2019). Additionally, TCB is a slow process, with
11 limited use in high-throughput packaging workflows. Overcoming this technical challenge likely requires adoption of copper hybrid bonding.
Copper hybrid bonding utilizes a copper damascene process to fabricate an extremely dense array of copper interconnect pads on the surface of two silicon dies or wafers (Theil et al., 2019), (Shih et al. 2014). During the process, a layer of silicon oxide is also created on any surface region which does not contain a copper pad. Next, the pads
A.
B. C.
D.
Figure 4: (A.) Illustration of 3D silicon die stacking using wirebonds and vertical through-silicon via (TSV) interconnects. (B.) SEM image of DRAM die stacking showing vertical TSV interconnects (IC insights). (C.) Illustration and SEM image of a copper micro-pillar used in high-density interconnect applications (X. Zhang et al., 2015). (D.) Illustration of copper hybrid bonding between two active silicon die (Shih et al., 2014). SEM image two die bonded together using this method (Theil et al., 2019). 12 on both pieces of silicon are brought in contact at high temperature, forming a metal-to- metal bond in the pad regions and an oxide-to-oxide bond everywhere else. For fine pitch interconnects this process remains challenging, thousands of pads must be the same height to ensure physical contact, with alignment tolerances on the order of a few microns or less to avoid failed connections. Thermal budget is also a consideration, as temperatures involved in the bonding process may damage circuits within the silicon
(Rajendran et al., 2007). Despite these issues, successful fabrication of vertical interconnects with a 1.6μm pitch have been demonstrated in commercial processes by
Theil et al. in 2018.
While copper hybrid bonding provides a potentially scalable solution for die-to- die communication, interconnect density between upper and lower die surfaces must also increase accordingly. Such scaling is likely to be facilitated by incremental improvements in the fabrication of though-silicon vias with small diameters and high aspect ratios. At the time of this report, typical pitch of TSV interconnects is greater than 40μm (EPS,
2019). However, TSV arrays with pitches of 10μm (Temple et al., 2015), 4μm (Civale et al., 2012), and 2μm (De Vos et al., 2018), have been reported in research literature.
Not a packaging technology per se, monolithic 3D integrated circuits present an intriguing scaling opportunity. Forgoing the use of 3D die stacking, monolithic 3D ICs are instead fabricated with front-end silicon processing tools. Starting with a base wafer, transistors, passives, crystalized silicon, and metals are added layer-by-layer to form all system components. Compared to die stacking, monolithic 3D ICs have several advantages. Namely, the potential to achieve 100nm interconnect pitch with 1μm of vertical length between active silicon layers, one to two orders of magnitude less than die
13 A. B. Figure 5: (A.) Illustration of monolithic 3D integrated circuit with several layers of transistor logic (Z. Wang, 2019). (B.) 3D model of a P-BiCS 3D NAND flash array showing multiple levels of memory cells and associated connection lines (Micheloni et al., 2017). stacking schemes using TSVs (Gopireddy & Torrellas, 2019). Additionally, thin silicon layers may improve thermal dissipation in some architectures. However, only lower temperature processes can be used to fabricate transistors in the top layers to avoid damaging layers near the bottom of the stack (Rajendran et al., 2007). This restricts scaling, as the fastest transistors require high temperatures to fabricate.
Monolithic ICs are especially attractive for the fabrication vertical NAND flash memory, as relatively few materials and simplified lithography steps are required per memory layer (Yoon et al., 2019). Indeed, while monolithic CPUs and systems-on-chip remain in development, VNAND products from various manufactures are available for commercial and consumer markets (Micheloni et al., 2017) containing 96 layers per monolithic die, with 128 layer stacks expected within the year.
2.3 Thermal Considerations in Advanced Packaging
Heat generated by a silicon die during operation is dissipated though the package to a cold reservoir, preventing damage to the die and surrounding package structures.
Ideally, the reservoir is maintained at a lower temperature than the package/die system 14 and undergoes only minor temperature change as more energy, in the form of heat, is absorbed. Three physical mechanisms exist which facilitate dissipation of heat; conduction, convection, and radiation. Heat energy transferred though solid material by physical contact is known as conduction. Heat transferred to though the movement of a fluid such as liquids or gasses is known as convection, and heat transferred via emission and absorption of photons is known as thermal radiation. In terrestrial electronics, conduction and convection are the primary mechanisms by which heat escapes from a hot die, while thermal radiation remains secondary (Yeh, 1995). This is because nearly all die/package systems make direct or indirect contact with ambient air in the earth’s atmosphere, which acts as a cold reservoir and fluid.
As described in chapter 20 of the Heterogeneous Integration Roadmap, microelectronics are traditionally cooled though the use of a metal lid, attached to the backside of a silicon die using a thermally conductive material such as indium solder.
This lid acts as a heat spreader, homogenizing temperatures across the die and reducing the thermal energy density of hotspots. A heat sink or metal cold plate is then placed atop the lid and attached though the use of thermal interface material (TIM). The heat sink’s array of vertical metal fins increases areal contact with the surrounding air, enhancing thermal dissipation, while use of a cold plate allows the heat from the package to escape via indirect contact with pumped cooling liquids such as water or ethylene glycol. A heat sink’s thermal dissipation rate can be further elevated using a fan or air jet to improve convection (EPS, 2019).
For low-power devices, the metal lid is often replaced by polymer materials, as the amount of thermal energy dissipated through the ambient air and printed circuit board
15 is sufficient for temperature regulation under most circumstances. In medium or high- power devices, a two-phase thermal transfer solution such as a vapor chamber (S. Han et al., 2019) or heat pipes (Xianping Chen et al., 2016) may be employed to increase heat conduction between the package lid and heat sink. Both vapor chambers and heat pipes employ similar physical principals for thermal transfer. Inside a partially evacuated air- tight metal chamber, heat from the package lid is transferred though the chamber wall to a working fluid. As temperature increases, the fluid undergoes a phase transition to a vapor state, absorbing additional heat during the transition. Vaporized working fluid diffuses towards colder regions of the chamber, due to a difference in pressure.
Vaporized fluid then condenses on the cold chamber wall, which is in contact with a heat sink or cold plate and in its liquid state, transferred back to the hot side of the chamber
A. B.
C. D.
Figure 6: (A.) Illustration of the standard microelectronics cooling scheme. (B.) Illustration of heat transfer modes in the standard semiconductor cooling scheme. (C.) Illustration of vapor chamber and/or heat pipe working principals (S. Han et al., 2019). (D.) Logarithmic plot of package power densities since the beginning of the semiconductor industry shows considerable increase in the past twenty years (S. Wang et al. 2018). 16 via capillary action using a porous wick. This cycle requires no active cooling components such as pumps or fans, simplifying design of a high-performance cooling system.
Adoption of advanced packaging paradigms presents new and unique challenges in thermal management which are not typically encountered in traditional packaging schemes. Silicon dies within the package are located in close proximity to ensure short interconnect lengths, as the topology helps minimize RC delay, reducing communication latency between components (Brain, 2016). However, placement of multiple heat sources in close proximity increases thermal energy density, increasing material temperatures beyond acceptable limits (Moore & Shi, 2014). This issue is especially prevalent in 2.5D and 3D IC packages, as multiple layers of silicon, oxide and polymers greatly increase thermal resistance in the heat conduction pathway to the ambient environment (EPS,
2019). Additionally, some multilayer structures place smaller dies atop larger ones, restricting the surface area over which heat can escape (EPS, 2019). To ensure adequate heat removal from a stack of two or more silicon dies in medium to high power applications, numerous ideas have been proposed to improve each portion of the thermal conduction pathway.
2.3.1 Thermal Interface Materials
Thermal conductivity between two surfaces in contact is degraded by microscopic roughness and small deviations in surface angle. These microscopic gaps are filled using thermal interface material (TIM). Candidate materials or composites must be mechanically compliant to fill all spaces between the two surfaces and provide a low resistance thermal pathway (Vyas et al., 2015). Additionally, such materials must retain
17 their properties over the package’s expected temperature range and thousands of operating hours. This set of desirable physical properties is somewhat difficult to obtain.
Materials with high thermal conductivity are often mechanically rigid, while soft and pliable materials are usually poor thermal conductors (EPS, 2019). In contemporary packaging schemes, this issue can be addressed by utilizing solder or polymer composites. Solder can be liquified at elevated temperatures, filling in microscopic gaps between two metal surfaces before cooling to a rigid state. As metal alloys, the thermal conductivity of solder-based TIM is higher than that of polymer composites. Contrasting a solder-based approach, polymer composites utilize thermally conductive particles such metals or ceramics imbedded in a soft polymer matrix to fill interstitial space.
The next generation of thermal interface materials will likely follow advances in the field of nanotechnology. Indeed, thermal interface materials which provide mechanical compliance and metal-like thermal conductivity are a topic of active research within the scientific community (Balasubramanian & Ramesh, 2018). Polymers containing nanoparticle fillers such as vertically aligned graphene (Y-F. Zhang et al.,
2018) or metal nanowires (Barako et al., 2017) present one possible path to metal-like thermal conductivity. Fabrication of an intrinsically conducive material, such as carbon or copper into a mechanically compliant morphology like vertical carbon nanotubes
(Won et al., 2012) or copper nanosprings (EPS, 2019) is another possibility.
2.3.2 Embedded Liquid Cooling
In any microelectronic cooling solution, the most effective way to reduce thermal resistance between hot silicon dies and a cool ambient environment is to bring coolant as close to the heat source as possible. Placing coolant adjacent to active components
18 A. B.
C.
Figure 7: (A.) Illustration of liquid-carrying microchannels etched into the backside of active silicon dies for an embedded cooling application (Sridhar et al., 2014). (B.) Illustration of an embedded microchannel cooling scheme which accounts for TSV interconnects and multiple layers of silicon, cooling each layer independently (S. Wang et al., 2018). (C.) SEM image of microfluidic channels fabricated on the backside of a silicon die (Purdue Univ.). requires silicon or copper microchannels though which liquid flows, driven by an external pump (Sridhar et al., 2014). In research setups, microchannels are often created by etching into the backside of a silicon wafer and sealing the intestinal space with an additional wafer though the use of thermal compression bonding (Mapelli, 2020). This process can be used to place cooling channels on the backside of an active silicon die, or other silicon structures, such as an interposer. While such a scheme can dissipate heat fluxes up to 790W/cm2 the primary drawback to water-based microchannel cooling is significant pressure losses caused by small channel diameters (Iradukunda et. Al., 2020).
Such losses result in poor temperature uniformity and can require a high-pressure pumping solution, increasing system complexity. While pressure loss can be somewhat mitigated though the use innovative fluid manifold topologies (S. Wang et al., 2018),
(Drummond et al., 2018), the issue’s persistence merits further investigation into two- phase cooling methods (Mandel et al., 2018).
19 2.3.3 Direct Contact Liquid Cooling
While contemporary liquid-cooling does not make direct contact with active circuit layers due to its use of water-based fluids, there exists dielectric liquids which can remain in contact electrical components without causing short circuits. Using such liquids, the active region of a silicon die may be placed inside a cooling channel, maximizing thermal dissipation (Cova et al., 2017). Alternatively, spray and jet impingement cooling using dielectric liquids have demonstrated thermal dissipation rates greater than 1000W/cm2, (EPS, 2019).
2.3.4 Advanced Thermal Materials
All cooling solutions for electronic packages can benefit from parallel development of new thermal conduction materials. These include cubic crystal thin films,
2D layered materials such as graphene, thermally conductive nanostructures like carbon nanotubes and metal nanowires, nanoparticle-polymer composites, and aligned polymer fibers (EPS, 2019). While such materials remain in the research and development phase, continual advances in the fields of materials and nanotechnology present numerous opportunities for a breakthrough.
2.4 Advanced Packaging and the Automotive Market
In automotive device packaging, manufacturers face a set of challenges not typically encountered in the consumer electronics market or high-performance computing
(HPC) space (Islam et al., 2017). Indeed, numerous operational and safety-critical functions are governed by silicon microprocessors in contemporary vehicles, necessitating extensive reliability testing and qualification (Chowdhury, 2019). For consumer electronics, this process is completed in 3-6 months. In contrast, semiconductor
20 Figure 8: Key differences between automotive and consumer semiconductor assembly/test requirements (Chowdhury, 2019. products intended for the automotive market undergo a 12-24 month qualification period before the start of production. Ramping to full-scale may take longer, as documentation related to all aspects of the product’s design and qualification must be written before high-volume manufacturing begins. For some products, 48 months are required to move from an initial design to a revenue-generating production run. Additionally, when full- scale production of a semiconductor product is achieved, automotive companies may require the manufacturer to uphold a 15-20 year supply guarantee, with all production- related data retained over the same time period. Changes in the manufacturing process and/or product design are not usually allowed, with a product change approval process which can take up to 30 months.
Throughout the design, qualification, and production phases, there is a continual drive towards a ‘zero defect’ process. Because of this, automotive semiconductor manufacturers have traditionally favored well-proven packaging technologies for their product lines. Today, these are based on lead frame designs, with side-solderable quad- flat no-lead (QFN) packaging solutions popular due to their small footprint, reliability, and lower cost (Dhond, 2019). However, considerable changes in the automotive market 21 are expected within a decade, driven by advances in vehicle electrification, autonomous operation, and connectivity (Adler et al., 2019). As each of these trends rely on placing more high-performance silicon in a vehicle, new package technologies will play a role in realizing their full potential.
Since 2014, electric roadway vehicles (Un-Noor et al., 2017), have increased in popularity, a trend sustained by several key factors. Lithium ion battery technologies and manufacturing processes continue to improve (Kim et al., 2019), increasing the range and reliability of electric vehicles (EVs) while reducing the cost of battery packs (Wentker et al., 2019). Globally, the transportation sector remains a large contributor to greenhouse gas emissions and air pollution. Vehicle electrification offers a potential means to lower these emissions, especially if electricity is provided though renewable resources. Because of this, many governments actively support the adoption of electric vehicles, with some providing financial or tax incentives to do so. From the consumer perspective, the price of electricity is an order of magnitude lower than that of fossil fuels, making the fuel cost
A. B.
C.
Figure 9: (A.) Projected increase in the total cost of electronic components as a percentage of total automobile manufacturing costs, projected to the year 2030. (B.) Illustration of lead frame packaging scheme presently used for automotive applications. (C.) 3D render of the automotive Quad Flat Pack No-Lead (QFN) package. 22 saved on each trip significant. Additionally, a wider acceptance of EVs and green transportation in general has prompted consumers and businesses to consider purchasing new EVs (Pevec et al., 2019).
Ultimately, electric roadway vehicles use energy stored in battery packs to perform useful tasks such as movement, cabin heating/cooling, lighting, computation, and wireless commination, all of which require power delivery electronics. Such applications often require silicon and packages capable of high-temperature, high-current operation, which has spurred the development of advanced lead frame designs in recent years.
Today, exposed pad QFNs and copper clip QFNs are the most commonly utilized advanced lead frame solutions for automotive power electronics (Lwin et al., 2016). In the future, silver (Ag) sintering is expected to replace traditional solder joints on lead frame packages (Siow, 2014), as sintered connections can potentially accommodate higher current and operating temperatures with minimal risk of failure (Chew & Schmitt,
2019).
While lead frame packaging is used extensively in engine control modules, small dashboard applications, and sensors on today’s vehicles, the technology is somewhat limited in terms interconnect density due to its low I/O pin count (Islam et al., 2017).
This is especially problematic for future vehicles offering some degree of autonomous operation or advanced driving assistance (ADAS) capabilities, as performance computing modules with chip-to-chip interconnections capable of processing 2GB/s will be required for such functions (Liu et al., 2019). Similarly, vehicles connected to high-bandwidth communications networks such as 5G or Wi-Fi will also require active silicon with more
23 interconnect density to transmit, receive, and process information on desirable time scales.
The need for greater interconnect densities, compute performance, and sensing modalities in the automotive space is potentially addressed by several packaging schemes under development. While initially deployed in consumer electronics, fan-out wafer level packaging (FOWLP) may greatly advance vehicle-based sensing and communications with novel Antenna-in-Package (AiP) designs (Y. Zhang & Mao, 2019). Indeed,
FOWLP-based AiP schemes for 77 GHz automotive radar modules are under development (Ho, et al., 2019), with the technology expanding to 5G communications
(Braun et al., 2018). Fan-out WLP may also enable greater integration of radio-based devices with other electrical components, such as processors In addition to WLP, high- performance flip-chip designs which utilize small copper pillars in place of the large solder balls found on BGAs have demonstrated good reliability and electrical performance during automotive qualification tests (Tang et al., 2019).
For autonomous or semi-autonomous operation, next generation roadway vehicles will monitor both their internal and external environment with advanced sensing systems, some of which are unavailable on consumer vehicles today. A more detailed overview of these advanced perception technologies can be found in chapter 4, but succinctly, high- resolution CMOS cameras, light detection and ranging (LiDAR), millimeter wave radar, and high-precision GPS will likely be integrated into production vehicles within two decades. Furthermore, advanced packaging technologies are well suited to address the unique engineering challenges such sensors will inevitably pose.
24 For CMOS imaging sensors (CIS), contemporary packaging solutions use a variety of wire bonding or flip chip BGA (FCBGA) schemes set on ceramic or laminate substrate (Kang et al., 2019), but novel products are rapidly approaching maturity due to cameras being considered a critical perception technology. New CIS packaging schemes using fan-out WLP (Xiao et al., 2017) and/or TSV interconnects are especially intriguing, as coprocessors can be placed adjacent to sensing elements (Coudrain et al., 2013). Such an arrangement is well-suited to real-time image processing scenarios. Each camera module conducts its own image collection and computation, so latency between the sensor and processor is reduced while simplifying thermal management. LiDAR packaging is differentiated by the need for both active light emission elements such as lasers or LEDs and passive light collection elements in the form of avalanche photodiodes (APDs). Notably, APD arrays follow a similar manufacturing path as CMOS image sensors, so improvements in CIS packaging are likely to improve the
A. B.
C.
Figure 10: (A.) SEM image of a vertically stacked CMOS image sensor (CIS) and underlying DRAM die connected using through-silicon vias (TSVs) manufactured by Sony. (B.) Illustration of an antenna-in-package (AiP) concept using fan-out wafer level packaging. (C.) 3D render of a GaAs vertical cavity surface emitting laser (VCSEL) array on a ceramic sub-mount package for automotive LiDAR (Warren et al., 2018). 25 performance and reliability of APDs modules as well (Huikari et al., 2018). Beyond
APDs, a more nascent problem lies in the development of semiconductor packages suitable for single-chip emitters which contain numerous diode lasers in close proximity.
A LiDAR’s spatial resolution is highly dependent on the number of lasers which can be projected into the vehicle’s surrounding environment (Piatek & Li, 2017), and it follows that increasing this number remains a top priority in the emerging automotive LiDAR industry. A second priority is cost, which increases for each laser packaged separately using lead frame designs. Addressing these competing priorities requires further development of semiconductor chips which contain arrays of diode lasers, but research has shown that such chips are likely to require high current, numerous interconnects, and significant cooling (Warren & Dacha, 2016). Notably, flip chip bonding of multiple
GaAs dies containing pulsed laser diode arrays has been successfully demonstrated on ceramic sub-mounts (Warren et al., 2018). However, the long-term reliability of such a packaging scheme in the context automotive use has only recently been investigated
(Fanning et al., 2020).
Similar to millimeter wave radar, it is anticipated that the cost, size, and reliability of GPS receivers will benefit from advances in WLP. Notably, fan-out and fan- in WLP technology can be used to construct high-performance antenna-in-package designs with low signal loss and small areal footprints (C. Jin et al., 2013). Dies with different functionalities may also be integrated into a single package, moving computing blocks closer the RF receiver to reduce latency (Teng et al., 2019).
Vehicles on the road today and in the future must monitor their internal functions using numerous sensors. Often these sensors are uniquely packaged based on their
26 location, working principal, and environment. While some physical sensors based on micro-electromechanical systems (MEMS) such as accelerometers and gyroscopes use standard ceramic or laminate packaging schemes (Chowdhury, 2019), other sensor types such as oil-level and pressure gauges are highly specialized. While many of the topics in this sub-field of electronic transducers and packaging are intriguing (Fleming, 2001)., they are somewhat beyond the intended scope of this report
Electronics for the automotive market must operate reliably over a -40°C to
150°C temperature range (Chowdhury, 2019), increasing the degree of strain caused by mismatched coefficients of thermal expansion (CTE). This strain must be carefully accounted for in advanced automotive packages with high interconnect density, as excessive strains on small interconnect structures will likely result in premature device failure. In this regard, the distinction between next-generation automotive and general computing packages is becoming less clear, with interconnect downscaling expected to remain a key metric in industry roadmaps (EPS, 2019). Regardless of packaging application or scheme, well-understood qualification metrologies are ultimately required to assess the degree of strain present on interconnect microstructures. Fortunately, current methods are suitable for contemporary packaging paradigms in both the consumer and automotive space. However, higher resolution metrologies are likely needed in a future where transportation and computing become indistinguishable.
27 CHAPTER 3
NANOSCALE STRAIN SENSING VIA LASER DIFFRACTION
3.1 Measuring Strain in Microelectronics Packages
The future of wafer level processing and 2.5/3D die stacking involves manufacture and qualification of high-density interconnects in a complex microstructure environment with numerous materials in direct contact. For example, the upper and lower termination points of a copper though-silicon-via (TSV) imbedded in bulk silicon could be connected to small solder balls and RDL pads residing in a polymer or oxide layer
(Premachandran et al., 2019). During normal use, a package/die system undergoes changes in temperature due to Joule heating (Pop et al., 2006) and interaction with the ambient environment. Because each material possesses a different coefficient of thermal expansion (CTE), the appearance of mechanical strain on the system is inevitable.
Perhaps unsurprisingly, this thermally induced strain is a leading cause of device failure (L. Yang & Bernstein, 2009). As a result, advanced packaging must adopt both analytical and experimental methods to predict and measure strain fields. While analytical tools such as finite element analysis can be used to predict strain in various microstructures (Su et al., 2017), real-world stress/strain data (Tian et al., 2016) provides valuable insight into the nature of electrical/mechanical failure modes. Today, the two most commonly used strain mapping metrologies for package inspection are Moiré interferometry and Digital Image Correlation. Fundamentally, both are optical techniques based on the principals of interference and imaging respectively, enabling non-contact, full-field strain measurements of package structures in-situ.
28 3.1.1 In-Plane Strain Mapping using Digital Image Correlation
Digital Image Correlation or DIC, is a statistical image processing technique which correlates a two-dimensional configuration of pixel values present in one image to those in another. While such a technique initially seems limited given the large number of possible pixel microstates present in two random images, its utility becomes much more apparent when correlating high-contrast speckle features in near-identical images. In such scenarios, the pixel displacement between the reference and deformed image can be accurately measured, often with sub-pixel resolution (Pan et al., 2009).
While DIC has gained considerable traction in the research and engineering communities over the past twenty years due to the advances of electronic imaging technology and computing performance (Pan, 2018) the idea of statically correlating features present in digital images was first implemented in 1975 by Keating, Wolf, and
Scarpace. In 1982, Peters and Ranson applied statistical correlation to digital images of laser speckle patterns reflected off a block of aluminum. As the aluminum was translated in the x,y plane, the speckle pattern detected by a TV camera changed. By tracking the location of speckle pattern features using a correlation function and computer, the translation of the aluminum block was discerned from the images alone. Over the next decade, Peters, Ranson, and Chu continued to develop digital image correlation techniques and apply them to digitized video of solid objects undergoing deformation, rotation, and translation, while work using white light illimitation and improved correlation algorithms was carried out by Sutton and his colleagues (Pan, 2018). During this time, it became apparent that many types of object deformations could be tracked
29 from one camera frame to the next, provided that correlation-suitable features were present.
The first use of digital image correlation to obtain a planar strain field was demonstrated in 1987. Using a high-resolution television camera and statistical algorithms similar to those developed by Ranson and Peters, Metwalli et al. measured the stress-strain behavior of aluminum and copper bars undergoing plastic deformation.
Three years later, James et al. (1990) mapped elastic-region strain fields of micro scale structures using images obtained from a scanning electron microscope (SEM). In 1993,
Stereo-DIC for measuring out-of-plane surface deformation was reported by P. Luo et al., further enhancing the capability of DIC in experimental mechanics research. In the late
1990’s the principals of 2D-DIC were extended to the third dimension. Using high resolution x-ray computed tomography (CT) data of bone specimens, digital volume correlation was demonstrated (Bay et al., 1999). In 1998, the often-used Newton-
Raphson pixel correlation method was simplified by the work of Vendroux and Knauss, leading to a more computationally efficient and robust implementation.
A. B.
Figure 11: (A.) Photograph of an early digital image correlation (DIC) setup developed by Sutton et al. in the mid 1980’s for measuring deformation in metal test specimens subjected to mechanical tension. (B.) Connection diagram of an experimental DIC setup used by D. Chen et al. in 1993 during the development of novel correlation algorithms. 30 Starting in the early 2000’s, DIC techniques began to attract the attention and interest of researchers in numerous disciplines, a trend which has continued into the late
2010’s. Indeed, research addressing nearly all aspects of DIC has seen tremendous growth in the past twenty years, affording numerous insights, while the number of new publications this year remains high. For a rigorous overview of the field’s advances in the past two decades, the reader is directed to Pan’s 2009, and 2018 review articles.
Operationally, 2D digital image correlation consists of three steps, (1) preparation of the sample and imaging setup, (2) recording images before and after thermal or mechanical loading, (3) processing images using software algorithms to determine displacement information. In a typical measurement setup (Khoo et al., 2016), a sample is placed at the working distance of a high-resolution optical camera, microscope, or SEM such that the flat sample surface is perpendicular to the imaging system’s optical axis and can remain so throughout the measurement procedure. Out-of-plane motion is to be avoided or negligible. Importantly, a high-quality speckle pattern must be present on the
A. B.
C.
Figure 12: (A.) Illustration of typical hardware components used in 2D DIC (Pan, et al., 2009). (B.) Illustration of ideal speckle density for high-resolution DIC. (C.) Example images of practical DIC speckle patterns. 31 surface of the sample (Crammond et.al, 2013), as pattern features convey displacement information when two images are correlated. An ideal pattern should contain a random distribution of high-contrast speckle. Periodic patterns and structures are avoided, as such patterns do not produce a unique configuration of pixels at any chosen 2D subregion, generating spurious correlations when near-identical pixel configurations appear at multiple locations. Additionally, a speckle pattern with features which are too small, large, sparse, dense, or low-contrast will reduce measurement precision and can even cause the correlation algorithm to fail. While the surfaces of some materials and specimens have acceptable speckle patterns and require minimal preparation (M. Luo et al., 2006), an experimenter must often apply artificial patterns, a task which can be non- trivial for small samples (Hoefnagels et al., 2019).
Next, images of the sample are taken before mechanical or thermal loading is applied, and again after loading occurs. In this step, proper configuration of the imaging system is crucial to impart a linear correspondence between physical points on the specimen and their images. The speckle pattern must be uniformly lit, while geometric distortion caused by the camera lens and/or sample angle are minimized or corrected
(Pan, 2018), (M. Luo et al., 2006). Once suitable images have been obtained, a calculation area within the image, called the region of interest or ROI, is defined by the experimenter, or in some cases, programmatically.
After a suitable ROI has been selected, the region is divided by virtual grids into subregions. Next, a point is chosen at the center of each subregion in the reference image, whose position within the subregion will be re-determined in the deformed image. Pixel greyscale values in the reference image are then correlated against the pixel values in the
32 deformed image using a cross-correlation (CC) or sum squares difference (SSD) criterion
((Khoo et al., 2016). Subregion features are matched by scanning though all computed values of the correlation coefficient for the entire ROI, such that a maximum can be
Reference Image Deformed Image correlation coefficient (CC) map CC
A.
B.
Figure 13: (A.) Illustration depicting pixel subset matching between a reference and deformed image. Here, a correlation criterion function is used to build a correlation coefficient map. The map’s peak value approximates the pixel subset’s location in the deformed image. (B.) Decomposition of the subset’s displacement (u,ν) and shape vector (∂ν/∂y*Δy, ∂u/∂y*Δy, ∂ν/∂x*Δx, ∂u/∂x*Δx) using matched points P and Q in the pixel subset. 33 found. Once a maximum correlation value is located on the deformed image, a displacement vector is generated between the subregion center point on the reference image to the center of the same subregion on the deformed image. This vector represents the displacement of the entire subregion.
While a subregion displacement vector accounts for rigid body translation, a second vector is required to fully describe speckle features which undergo changes in shape due to normal and/or shear strain. If a vector is constructed between a pixel feature and the center of a subregion in the reference image, a similar vector can also be drawn in the deformed image by deducing the feature’s new location via correlation. The difference between of these two vectors computes the shape vector, which is added to the already-determined displacement vector (J. Tong, 2018). The use of pixels, which discretize spatial coordinates, presents a resolution challenge for DIC. Namely, displacements less than one pixel cannot be measured. To address this issue, various sub- pixel interpolation schemes exist which approximate greyscale levels between adjacent pixel centroids. Commonly, cubic B-spline interpolation or quintic B-spline interpolation is applied before subregion matching (Schreier et al., 2000) (Luu et al., 2011).
Like all measurement technologies, ultimate resolution of a practical DIC system is limited by some key factors, the first of which depends on the imaging system’s megapixel count. With additional pixels representing a sample’s area, smaller displacement steps are utilized, leading to an overall increase in spatial precision (Reu et al., 2015). Quality of the experimental setup is a second factor, with ideal configurations featuring uniform lighting, minimal out-of-plane displacement (Haddadi & Belhabib,
2007), and camera optics which suppress geometric distortions (Pan, 2018).
34 In optical microscopy DIC, displacement resolution is fundamentally limited by light diffraction to a few hundred nanometers (Xiadong Chen et al., 2011), while lens distortions can reduce measurement accuracy (M. Luo et al., 2006). Applying DIC to images acquired through Scanning Electron Microscopy (SEM DIC), displacements on the order of nanometers can be resolved. However, SEM DIC measurements are often difficult, as non-trivial errors caused by image noise (Arikawa et al., 2011), spatial distortions (Maraghechi et al., 2019), and various spatial/temporal drifts (Mello et al.,
2017) (P. Jin & Li, 2015), must be accounted for.
Fabrication of quality speckle is often a challenge in SEM DIC (Kammers &
Daly, 2012), illustrating how these patterns effect displacement resolution in all DIC methods. Ultimately, the goal of speckle pattern fabrication is to ensure that each image subregion contains a unique set of pixel features which cannot be found in any other subregion within the ROI. Fulfillment of this condition results in well-defined correlation maxima when a similar feature set is found within the deformed image, minimizing displacement error (Crammond et.al., 2013). In service of this, speckle patterns should exhibit high contrast, contain a variety of feature sizes and shapes, and exhibit no obvious pattern directionality. While somewhat subjective, the distribution of speckle of sizes and shapes should allow for numerous subregion features, with circular or ellipse-like shapes preferred. Structured or periodic patterns such as regular spaced gridlines or dots are to be avoided, as they do not guarantee subregion feature uniqueness.
A fourth factor which influences displacement resolution of a DIC tool is the experimenter’s choice of correlation algorithm and sub-pixel interpolation scheme (W.
Tong, 2005). Today, there are three categories of commonly used algorithms, classified
35 by their underlying mathematical definitions: cross-correlation (CC), sum squared difference (SSD), and parametric sum squared difference (PSSD). More specifically, zero-mean CC, zero-mean SSD, and PSSD implementations with two additional unknown parameters are recommended for practical applications, as they are insensitive to small changes in pixel intensity commonly encountered during deformation measurements (Pan, 2018).
Accurate approximation of the underlying displacement field from a subregions’ rigid body translation and shape vectors requires construction of a shape function, which describes the displacement gradient as a Taylor series expansion. In this formulation, translation is the zeroth order term, linear deformations are described by the first order term, and quadric deformation described by the second order term. Ideally, shape function order must be chosen to accommodate expected displacement contributions
(Pan, 2018). For example, if both translation and linear deformation are expected, a first- order Tayler series may provide the best approximation of the displacement gradient. If a zero-order series is used instead, the shape function will be underfitted, increasing systematic error. In contrast, a second-order Taylor expansion will overfit the measurement, leading to an increase in random error. For practical measurement scenarios, selection an appropriate Taylor series order can be difficult, as actual deformations will not be known in advance of the experiment. However, the second order shape function is recommended for most scenarios, as random error can be somewhat mitigated though use of larger image subregions.
Sub-pixel displacement resolution is achievable in DIC with pixel interpolation schemes, which approximate a series of greyscale values between the centers of adjacent
36 pixels. At the time of report, a set of recursive interpolation methods based around B- spline representation and popularized by Luu et al. in 2011. are considered state-of-the- art. However, while these methods help to reduce measurement-based errors, DIC tools are ultimately limited by errors generated during the interpolation process itself.
Although, some interpolation-based errors can be reduced by image prefiltering (Khoo et al., 2016).
For DIC measurements, a long and complex error chain ultimately reduces to an overall displacement resolution of approximately 0.1 pixels (Caporossi et al., 2018), with resolutions as high as 0.01 pixels in certain experiments (Banks-Sills et al., 2011). While additional resolution is possible, problems related to speckle fabrication, imaging system calibration, shape function selection, optimal subregion size, and interpolation error must be addressed (Pan, 2018). Nevertheless, digital image correlation has mostly fulfilled its promise as a full-field, non-contact, high-resolution displacement metrology, thanks to continual advancements in digital camera technology, computational performance,
A. B.
Figure 14: (A.) y-axis displacement field (ν) of solder ball obtained using planer DIC by Sun and Pang in 2008, following temperature cycling between 25°C and 100°C. (B.) y-axis strain field (eyy) of a crack board subjected to mechanical bending, measured using DIC by Gao et al. in 2015. 37 algorithms and applications. Indeed, interest in novel and expanded use cases for DIC remains at an all-time high, more than thirty years after its initial demonstration.
3.1.2 In-Plane Strain Mapping using Moiré Interferometry
Moiré Interferometry is an optical technique capable of measuring in-plane strain fields or out-of-plane deformation, with semiconductor industry applications related to part inspection and quality control. The Moiré effect can be demonstrated as follows. If a pattern of alternating opaque and transparent lines printed on two plastic transparencies are superimposed on a white background, slight differences in periodicity, position, or angle create secondary fringes observed in the region of overlap. These fringes appear as bands of light and dark which do not overlap.
Industrial Moiré-based metrologies are categorized by their method of fringe generation. True to its name, Shadow Moiré makes use a periodic shadow mask, placed above the sample surface and illuminated from the top (Zhu et al., 2018). A camera, also located above shadow mask, acquires images of the Moiré fringe pattern created when
A. B.
Figure 15: (A.) Example of Moiré pattern generated from two sets of overlapping periodic lines. (B.) Typical configuration of an industrial shadow Moiré tool for height profiling and out-of-plane deformation measurements (W. Zhang et al., 2019.). 38 lines on the shadow mask overlap the periodic shadows projected by the mask onto the sample surface. While such a configuration cannot be used to resolve in-plane strain fields, out-of-plane displacements on the order of 10μm are readily discernable from the imaged Moiré pattern (W. Zhang et al., 2019). This metrology is often used for in-line printed circuit board (PCB), and wafer inspection; where its wide view field, simple optics, fast acquisition times, and contactless operation are valued.
For Moiré interferometry, superimposable patterns are generated by projecting two overlapping beams of coherent light onto a thin polymer diffraction grating, adhered to the surface of a sample. In such a configuration, electric field vectors of the two light beams have identical polarization, producing an interference pattern in the region of overlap (Kadooka et al., 2003). The interference pattern manifests as a periodic series of bright identical lines with unilluminated regions of equal width between them. In this configuration, the beams’ angle of intersection sets the fringe periodicity while interfering photons coupled into the grating’s first diffractive order are subsequently imaged by a camera. Spatial periodicities of the projected interference pattern and grating are not defined arbitrarily. Rather, they are chosen such that light coupled into the first diffractive order exits the grating region along the sample’s surface normal before stain is applied. This practice ensures that the outgoing fringe pattern remains unchanged as it enters the camera (Czarnek, 1990).
With a baseline image of the interference pattern acquired, the sample is then subjected to thermal or machinal loading, changing local periodicity of the diffraction grating as the sample surface is strained. This change in periodicity causes a small difference in the propagation angle of light waves coupled into the first diffractive order,
39 manifesting as warped fringes when viewed by the camera (Czarnek, 1990). Along a given fringe, the sample’s in-plane displacement remains constant, similar to lines on a topographic map (Ifju & B. Han, 2010). For a given sample region, denser fringe spacing corresponds to higher levels of mechanical strain, while lower fringe densities correspond to smaller strain.
The use of Moiré patterns to measure deformation was first proposed in 1945 by
D. Tollenar and experimentally verified by Wellard and Sheppard in 1948. The apparatus
A. C.
B. D.
Figure 16: (A.) Bonding an epoxy diffraction grating to the surface cross section of a microelectronics package in preparation for Moiré interferometry (Ifju & B. Han, 2010). (B.) Illustration of a typical two-beam Moiré interferometry setup. Light from interfering laser beams is projected onto a diffraction grating bonded to the specimen’s surface. Some of this interfering light is coupled into the grating’s first diffractive order and directed towards a digital camera. Images of the diffracted light encode a contour map of the phase difference between the two beams at the specimen’s surface (Czarnek, 1990). (C.) Illustration of a high-resolution Moiré setup used by Wu and Han in 2019 to study mechanical strain and deformation in fan-out wafer-level packages. Horizonal (u) and vertical (ν) displacement contours of fan-out packages and solder balls, as measured by Wu and B. Han during thermal loading experiments. 40 devised by Weller and Shepard consisted of line array bonded to the surface of a specimen. The specimen was then observed, before and after deformation, by a second line array that served as a reference. Light transmitted though both gratings formed a
Moiré pattern which changed as the sample was deformed. In the 1960’s the spatial resolution of geometric Moiré was significantly improved by Holister 1967, Post 1968, and Sciammarella 1969 using fringe multiplication techniques. A decade later,
McDonach et al. and Post replaced the reference grating with an optical interference pattern created by two light coherent light sources, marking the beginning of high- resolution Moiré interferometry.
Today, Moiré interferometry can be used to inspect in-plane strain fields present on the cross-sectional surface of a semiconductor package subjected to thermal loading
(Ifju & B. Han, 2010). Theoretically, the smallest displacement which can be measured by the technique is u=λ/2, where u is the displacement of a sample surface feature and λ is the wavelength of the interfering light beams (Weissman & Post, 1982). However, displacement resolution may be improved by up to a factor of 10 using fringe multiplication (B. Han, 1993). Here, the sample is translated incrementally to induce a shift in fringe location. Images acquired at the two sample positions are then digitally subtracted to reveal additional fringe orders. Recently, a prototype Moiré interferometer intended for semiconductor package inspection has demonstrated displacement resolution on the order of 0.1-0.5μm using this technique (Wu & B. Han, 2018).
While such resolution meets or exceeds that of digital image correlation, Moiré interferometry may be unable to examine regions of electronic packages which exhibit large out-of-plane deformations. This is because additional surface angle causes the
41 diffracted interfering beam pair to miss the collection lens. Alas, out-of-plane deformations are common near material interfaces in microelectronic packages, potentially limiting Moiré interferometry to homogenous regions where its exceptional resolution is unnecessary.
3.2 Strain Mapping of Package Microstructures via Laser Diffraction
Contemporary Digital Image Correlation and Moiré Interferometry techniques resolve in-plane displacements on the order of micrometers in a properly designed experiment. This degree of resolution is sufficient for today’s in-package interconnect pitch, which is greater than 10μm. However, for advanced packaging to address computational scaling challenges in the next decade, interconnect density is expected to increase by a factor of ten or more. As pitch densities approach one micron, currently deployed metrologies based on optical DIC and Moiré Interferometry risk under-defining the complex strain environment which likely exists between interconnects. With limited tools, intricacies of mechanical failure modes may be poorly understood, hampering efforts to prevent premature failures and increase yields. This could prove especially problematic for multi-die packages and smart vehicle technologies, where a single point of mechanical failure in an interconnect layer may render a packaged system inoperable, even if costly components within the package remain undamaged. Additionally, manufacturers seeking to supply the emerging smart-vehicle market must develop multi- die packages with a near-zero failure rate over a wide range of temperatures and moisture conditions. Given such stringent requirements, new metrologies capable of resolving displacements of less than 0.1μm are likely needed in the next ten years.
42 Measurement of sub-micron and nano-scale displacement presents several challenges for in-plane strain metrologies. A calibrated DIC tool is capable of measuring displacements of approximately 0.1 pixels. However, obtaining sub-micron resolution requires high magnification optics, limiting the view field to a small subsection of a package. Furthermore, because micron-scale DIC relies on microscopy hardware, displacement resolution and accuracy are limited by both light diffraction and lens distortion. In practice, obtaining accurate feature displacements less than 0.5μm is difficult. Applying DIC methods to scanning electron microcopy images (SEM DIC), displacement resolution can be improved by more than order of magnitude. However, experimenters must address a series of non-trivial measurement errors caused by temporal drift, spatial distortions, and image noise. Additionally, decoration of the sample surface with high-quality speckle is often challenging for ROIs a few hundred microns in size, increasing sample preparation time if sub-micron resolution is desired.
Moiré Interferometry provides quantized displacement resolution on the order of
λ/2, where λ is the wavelength of laser light used to generate the reference grating pattern projected onto the surface of the sample. For practical measurement scenarios, λ is on the order of a few hundred nanometers, affording displacement resolution similar to DIC.
While resolution can be improved using a fringe multiplication to estimate additional fringe orders, such strategies often perform poorly in regions of high out-of-plane displacement, typically found near material interfaces. Such a limitation may be especially problematic for advanced microelectronics packages, which are likely to contain imbedded interconnect materials only a few microns apart.
43 A. Au Grating B. Laser
θm
Package Surface Grating
C. Grating (no strain) Grating (strained)
d d?
Figure 17: (A.) 3D model of a thin diffraction grating bonded to a flat surface. (B.) Depiction of a grating’s diffraction angle (θm) relative to an incoming collimated laser source. (C.) Illustration of localized changes in grating pitch (d) due to mechanical strain. Due to limitations imposed by both techniques with regard to sub-micron displacement and strain, an alternative metrology is proposed which does not make use imaging lenses, speckle patterns, or fringe projection. Instead, laser diffraction is used to directly measure pitch of thin-film Au diffraction gratings bonded to the surface of exposed package structures such as solder balls and TSVs at various temperatures. As the package undergoes thermal loading, materials beneath the grating expand or contract, creating small changes in grating pitch which can be detected by careful measurement of the first-order diffraction angle using a wide-area CMOS camera sensor (Fig. 17). By scanning a focused laser over the entire grating area and measuring the diffraction angle at each point, a 2-D map of the grating pitch can be constructed. Comparison of pitch
44 maps taken at varying sample temperatures can then be used to simultaneously extract the in-plane displacement field, strain, and out-of-plane warping information.
3.3 Theoretical Background
3.3.1 Relationship Between Grating Pitch and Strain
Strain is a measure of mechanical deformation, representing the displacement between two points located on a solid material relative to a reference displacement. In a
Cartesian coordinate system, the general strain equations are defined as
휕푢 휀 = (3.1) 푥푥 휕푥
휕푣 휀 = (3.2) 푦푦 휕푦
휕푤 휀 = (3.3) 푧푧 휕푧
1 휕푤 휕푣 휀 = ∗ ( + ) = 휀 (3.4) 푦푧 2 휕푦 휕푧 푧푦
1 휕푢 휕푤 휀 = ∗ ( + ) = 휀 (3.5) 푧푥 2 휕푧 휕푥 푧푥
1 휕푣 휕푢 휀 = ∗ ( + ) = 휀 . (3.6) 푥푦 2 휕푥 휕푦 푦푥
Where u, v, and w are displacements in the x, y, and z directions respectively. The strain is represented by ε while its direction in the cartesian basis (x,y,z) is indexed by two subscripts. To describe strain on a flat surface in the x,y plane, any displacement gradients in the z direction can be ignored, simplifying the general set of equations to
45 휕푢 휀 = (3.7) 푥푥 휕푥
휕푣 휀 = (3.8) 푦푦 휕푦
1 휕푣 휕푢 휀 = ∗ ( + ) = 휀 . (3.9) 푥푦 2 휕푥 휕푦 푦푥
A one-dimensional (1D) optical diffraction grating can be constructed from a series of identical narrow trenches, etched into a planar reflective surface parallel to the y-axis. Along the x-direction spacing is periodic, meaning that the distance between centers of any two adjacent reflective lines or non-reflective trenches is equal. The numerical value associated with this equidistant spacing between reflective lines is referred to as the grating pitch.
If a 1D grating is strained in the x-direction, its pitch will change relative to its unstrained reference value. An infinitesimal change in pitch is thus equal to ∂u in equation 3.7 with ∂x as the reference displacement (Fig 18B.). However, the displacement gradient in equation 3.7 is continuous, while experimental strain measurements are discrete. To resolve this issue, the one-dimensional displacement gradient is reformulated into discretized form using the definition
푢(푥 = 푙) = 퐿 − 푙. (3.10)
Where L is the length between two points in a deformed sample and l is the reference length between the same two points (Fig 18A.). Now, εxx can be described by the global
1D strain equations
46 퐿 − 푙 휀 = 퐸푛푔𝑖푛푒푒푟𝑖푛푔 푆푡푟푎𝑖푛 (3.11) 푙
1 퐿2 − 푙2 휀 = ∗ 퐶푎푢푐ℎ푦 푆푡푟푎𝑖푛 (3.12) 2 퐿2
퐿 휀 = 푙푛 ( ) 퐿표푔푎푟𝑖푡ℎ푚𝑖푐 푆푡푟푎𝑖푛 (3.13) 푙
In the present study, a 1D diffraction grating was fabricated by trenching lines into a metal thin film deposited on the surface of a package cross-section. The grating is placed under mechanical strain by heating the package. Under such conditions, solid materials beneath the grating undergo thermal expansion with the relationship between L and l defined as
A. B. ∂x
a b l u a* b* L ∂x + ∂u
C. +ΔT
d 0 d0+Δd Figure 18: (A.) Discreate strain element of a solid material in one dimension. Sample reference length is denoted by l, deformed sample length is denoted by L, while displacement of the material’s edge is denoted by u. (B.) Continuous strain element in a solid material. An infinitesimal reference length between points a and b is defined by ∂x, while length between the same two points á* and b́* in the deformed sample is ∂x+∂u. (C.) Discrete change in diffraction grating pitch (Δd) due to thermal expansion versus a reference pitch value d0 measured at room temperature.
47 퐿 = 푙 ∗ (1 + 훼 ∗ (푇푓 − 푇푖) ). (3.14)
Here, α represents the sample material’s coefficient of thermal expansion (CTE), while Tf and Ti represent initial and final temperatures of the package.
Semiconductor packages often contain silicon substrates, oxide layers, epoxy-based polymers, solder alloys, and various copper structures. Given this list of materials, it is possible to calculate the expected change in pitch (L-l)=Δd for a thin diffraction grating deposited on the surface of each of each material using CTE data (Fig 18C.). Assuming a temperature delta (Tf -Ti) of 100°C, and setting l to a pitch value of 700nm, Δd and ε can be computed using each material’s CTE value. Results of these computations are shown in
Table 1.
Table 1: Changes in Grating Pitch due to Thermal Expansion
Material ΔT (°C) α (10-6m/mK) l (nm) L (nm) Δd (nm) ε (%) Silicon 100 ~5.00 700 700.35 0.35 0.050
Oxide (SiO2) 100 0.75 700 700.05 0.05 0.008 Epoxy 100 ~55.00 700 703.85 3.85 0.55 Solder 100 25.00 700 701.75 1.75 0.25 Copper 100 16.40 700 701.15 1.15 0.16
When the value of L is approximately equal to l, as is the case for heated semiconductor package materials, it can be shown that equations 3.12 and 3.13 are nearly equal to equation 3.11. For equation 3.12, which is known as the Cauchy stain, the dual
퐿−푙 condition of ≪ 1 and 푙 + 퐿 ≈ 2푙 permits the equation to be written as 푙
퐿 − 푙 퐿 + 푙 퐿 − 푙 2푙 퐿 − 푙 휀 = ∗ ≅ ∗ ≅ . (3.15) 푙 2푙 푙 2퐿 푙
48 Taylor expanding the logarithmic expression in equation 3.13, around 퐿 − 푙 ≅ 0 yields
퐿 퐿 − 푙 1 퐿 − 푙 2 퐿 − 푙 푙푛 ( ) ≅ − ∗ ( ) + ⋯ ≈ . (3.16) 푙 푙 2 푙 푙 thus, the linear strain behavior of equation 3.11 is expected for small deformations caused by changes in temperature. This conclusion is further evidenced by the near-constant values of α (CTE) reported experimentally for most solid materials. It follows that a small change in grating pitch (Δd) can be used to compute strain in the x-direction by equating
Δd to L-l and l to the undeformed reference pitch d0, reformulating equation 3.11 as
퐿 − 푙 훥푑 푑 − 푑 휀 = = = 0 . (3.17) 푙 푑0 푑0
Where d is the pitch of the deformed grating and d0 is the undeformed reference pitch
(Fig 18C). Finally, the relationship between d and d0 for a given change in temperature is
푑 = 푑0 ∗ (1 + 훼 ∗ (푇푓 − 푇푖)). (3.18)
3.3.2 The Diffraction Grating Equation
If a reflective layer with near-zero thickness is deposited onto the surface of a sample, the newly bonded layer will conform to planar strains in the underlying material during heating. Furthermore, dividing the layer into a series of narrow, identical, and periodically spaced reflective lines, separated by adjacent non-reflecting trenches a few microns or less apart forms an optical diffraction grating. As the underlying material expands, the horizontal distance between centers of adjacent lines, called the pitch, changes in accordance with equation 3.18. Notably, pitch of an unknown grating can be
49 accurately measured without the use of an optical microscope or SEM. Instead, angles between light rays incident on a grating’s surface and rays coupled into the grating’s diffractive orders’ are used (Fig. 19). Pitch is computed via the diffraction grating equation
푚휆 푠𝑖푛(휃 ) + 푠𝑖푛(휃 ) = ( ) 푚 = 0, ±1, ±2, ±3. (3.19) 푚 푖 푛푑
Here, d is equal to the grating pitch, m is the diffraction order, λ is the wavelength of incident light, n is the refractive index of the media though which diffracted light will propagate, θm is the diffraction angle and θi is the angle of incident light relative the grating’s surface normal vector (Harvey & Pfisterer, 2019). Rearraigning this equation to solve for d yields,
푚휆 푑 = 푚 = 0, ±1, ±2, ±3 (3.20). 푛(푠𝑖푛(휃푚) + 푠𝑖푛(휃푖))
For practical pitch measurements, incident light is provided by a He-Ne laser tube exhibiting a monochromatic wavelength (λ) of 632.8nm while the value of n is that of air.
Light coupled into the first diffractive order (m=±1 ) produces a bright, well-defined laser beam spot when projected onto a screen set perpendicular to the incoming rays, making accurate determination of diffraction angle (θm), and by extension d, possible on a laboratory optical table. In this instance, the angle associated with the m=1 diffractive order (θ1) was selected for measurement as a matter of experimental convenience.
However, it should be noted that measuring the angle associated with the m=-1 order (θ-1) would yield the same value of d, as both sets of diffracted light rays exhibit geometric symmetry across the angle of incidence (θi) 50 A. B.
Figure 19: Illustration (Harvey & Pfisterer, 2019) of light impinging on the surface of a transmission grating (A.) and a reflection grating (B.), generating a series of diffractive orders.
While the refractive index of air at standard temperature and pressure is approximately equal to 1.0, air temperatures near a sample’s surface may vary between
20°C and 125°C during a thermal loading experiment. Because optical diffraction occurs very close to this surface, changes in the value of nair with respect to temperature merit closer examination. The refractive index of air at a given temperature can be calculated using the Edlén Equation, first published in 1966 by Bengt Edlén. The equation has been revised several times, accounting for updated air density and water vapor reflectivity data. A contemporary version of this equation introduced by Birch and Downs in 1993 is widely used today. Indeed, several online calculator tools making use of tabulated atmospheric data are available for use. Table 2 shows the approximate influence of nair on the calculated value of d at various package temperatures for a grating with an actual
51 pitch of 700nm, assuming d would be measured using a He-Ne laser beam directed towards the grating at normal incidence (θi = 0).
As evidenced by Table 2, accounting for the variability of nair at different air temperatures results in a computed grating pitch (dcalc) value approximately 0.18nm smaller than the actual pitch at 25°C. At an air temperature of 125°C, the difference between actual pitch and measured pitch would be approximately 0.12nm, resulting in a
Δd of 0.06nm. For reference, the change in d caused by thermal expansion over this
Table 2: Influence of Temperature-Dependent nair on Calculated Grating Pitch
Tair (ºC) nair m λ (nm) θm (deg) dactual (nm) dcalc (nm) 20 1.000271 1 632.80 63.689 700 699.82 25 1.000267 1 632.80 63.689 700 699.83 45 1.000249 1 632.80 63.689 700 699.84 75 1.000225 1 632.80 63.689 700 699.85 95 1.000209 1 632.80 63.689 700 699.86 125 1.000184 1 632.80 63.689 700 699.88 temperature range is approximately 0.35nm for silicon and 1.15nm for copper, as shown in Table 1. From this information, it can be concluded that the value of nair may need to be considered when measuring changes in grating pitch. However, the influence of nair on the value of Δd is an order of magnitude smaller than that of thermal expansion and is understood well enough to be corrected for. It is thus argued that temperature dependence of nair is not detrimental to the proposed strain metrology provided that its value can be accurately estimated. Finally, replacing constants m, λ, and n in equation 3.20 with their experiment-specific values yields
632.8 푑 = . (3.21) 푛푎푖푟(푠𝑖푛(휃1) + 푠𝑖푛(휃푖)) 52 3.3.3 The Fraunhofer Diffraction Equation
Monochromatic light normally incident on the surface of a 1D reflection-type grating with vertical trenches produces a unique intensity profile, where much of the reflected light is funneled into a narrow, well-defined, set of reflection angles near the reflection hemisphere’s equator. This phenomenon occurs because cylindrical Huygens wavelets (Fig. 20), produced when incoming light is scattered off each reflective grating line, will only exhibit constructive interference at angles where the optical path length difference between two adjacent slits is an integral number of wavelengths (Harvey &
Pfisterer, 2019). Each narrow cluster of angles over which constructive interference occurs is referred to as a ‘diffraction order’. Furthermore, the angular width of these
A. B.
Figure 20: (A.) For diffraction gratings, constructive wavelet interference occurs at angles (θ) where the path length difference between parallel rays emitted from adjacent slits is an integral number of wavelengths. (B.) Illustration of wave interference created by overlapping cylindrical Huygens wavelets emanating from two coherent sources (Pedrotti, 2008). 53 orders is dependent on the number of grating lines illuminated by an axially symmetric light source of finite aperture size. The more lines which are illuminated, the narrower this width becomes, while small secondary maxima appear between each primary maximum.
For reflection hemisphere radii much greater than the grating pitch, occurrence of discreate orders is best described using principals of Fraunhofer diffraction. Here, the one-dimensional (azimuthal) intensity profile of diffracted light is approximated by
푠𝑖푛2(훽/2) 푠𝑖푛2(푁휙/2) 퐼 = 퐼 ∗ , (3.22) 0 (훽/2)2 푠𝑖푛2(휙/2) where the terms variables β and ϕ are defined as
2휋 훽 = 푎 ∗ 푠𝑖푛(휃) (3.23) 휆
2휋 휙 = 푑 ∗ sin(휃). (3.24) 휆
For these equations I0 represents the normalized intensity of light coupled into the m=0 diffractive order, taking on a value of I0=1.0 because the m=0 order is brightest, while I represents optical intensity at a given azimuthal angle θ relative to I0. Moreover, d is the grating pitch, a is the width of a single reflective grating line, λ is the laser source’s wavelength, and N is the number of grating lines illuminated (Noh, 2019). To compute relative intensity (I/I0) for a given value of θ, the constants a,d,λ and N must be known.
In this study, the grating fabrication procedure, described in section 3.4.1, results in line widths (a) of approximately d/2, where d is equal to 700nm at 25°C. As evidenced
54 by Table 1, the value of d varies by less than one percent for thin film gratings bonded to the surface of common semiconductor package materials over a temperature delta of
100°C. Collimated light from the He-Ne laser source described in section 3.4.3 is monochromatic, with a wavelength (λ) of 632.8nm and a circular beam diameter (D) of approximately 0.7mm. However, because this diameter is an order of magnitude larger than the typical length or width of a grating region (0.07mm), the laser beam is passed through a 50X microscope objective lens to reduce its size.
Laser spot diameter at the objective lens’s working distance is greatly reduced, and can be calculated using
4휆푓 2푤 = . (3.25) 0 휋퐷
For this equation, w0 is radius of the condensed light spot, sometimes referred to as waist size for circular laser beams with a gaussian-like intensity cross section, while 2w0 is the spot’s diameter (Fig. 21C,D). Constants D and λ are laser-specific, representing diameter and wavelength of the input beam. An additional constant, f, refers to the microscope objective’s focal length (Mitutoyo Inc., 2017). Setting these constants to their study- relevant values of λ=632.8nm, D=0.7mm, and f=4mm yields a condensed beam spot diameter (2w0) of 4604nm. This number is then divided by grating pitch (d=700nm), resulting in a value of 6.58. From this information, it can be deduced that six grating lines are fully illuminated by the condensed beam spot, making N equal to six (Fig. 21D).
55 With practical values of a,d,λ and N now determined, the Fraunhofer intensity profile can be computed by inserting these constants into equations 3.22, 3.23 and 3.24 and plotting resultant values of relative intensity (I/I0) with respect to azimuth angle θ
(Fig. 21). As expected, inspection of the resulting 1D profile plot shows that nearly all light exiting the grating’s surface is concentrated into a small subset of azimuth angles.
Furthermore, restricting the plot’s range of angles to those around the rightmost
A. B.
m=0 θ=θ1=63.009º
m=-1 m=1
C. D. d α
2w0
Figure 21: (A.) 1D azimuthal intensity profile (I/I0) generated when six reflective grating lines are illuminated by a focused He-Ne laser beam spot, computed using equation 3.22. In this calculation, grating pitch (d) is 700nm while the width of each reflective line (α) is 350nm. Diffractive orders correspond to each intensity peak (B.) Intensity peak shift of the m=1 diffractive order for various values of grating pitch (d), as computed from equation 3.22. Notably, each peak’s azimuthal position (θ) is equal to the diffraction angle (θ1). (C.) Edge ray geometry of collimated laser light condensed by a microscope objective to a minimum spot dimeter (2w0), where w0 is the beam waist size. (D.) Illustration of a condensed laser beam spot at objective’s working distance, illuminating six reflective grating lines. 56 maximum reveals a gaussian-like intensity profile with a single peak value, whose angular position is approximately equal to that of the first-order diffraction angle θ1 (Fig.
21A,B). Given this information, it can be concluded that the values of θm allowed by equation 3.19 (Table 1) represent peak positions in the Fraunhofer intensity profile.
3.3.4 Measuring Diffraction Angle and Angular Sensitivity
As both equations 3.20 and 3.22 demonstrate, the angle between light rays impinging on a grating’s surface and each diffractive order depends on the grating’s pitch. Furthermore, this diffraction angle (θm) is invariant with respect to ray propagation distance in the far field for isotropic media such as air. Because of this, θm can be measured over a large range of reflection hemisphere radii. In practice, determining changes in grating pitch due to strain requires measuring changes in diffraction angle of a given order, as evidenced in equation 3.21. This can be accomplished by first projecting diffracted laser light onto a planar matte screen set perpendicular to the incoming rays, resulting in the appearance of one or more laser beam spots. In accordance with equation
3.22, each beam spot exhibits a gaussian-like intensity profile whose maximum value occurs at the diffraction angle θm.
Assuming a small fixed value of laser incidence (θi), changes in diffraction angle can be related to changes in the beam spot’s horizontal position along the screen using the chord length formula
훥휃 훥푥 = 2푟 ∗ 푠𝑖푛 ( 1). (3.26) 푠푐푟푒푒푛 2
57 Here, Δxscreen is the change in horizontal position of the beam spot’s center along the screen, r is the radial distance between the center of the laser spot projected onto the grating’s surface and the center of the diffracted beam spot present on the screen, while
Δθ1 represents change in the first-order diffraction angle (Fig. 22). The presence of r as a multiplicative factor provides a highly practical means of determining Δθ1 indirectly by measuring Δxscreen. Indeed, the magnitude of Δxscreen be can made larger by simply moving the screen further from the sample in the radial direction if more angular sensitivity is desired. Because Δxscreen is the quantity used to determine Δθ1, it is appropriate to rearrange equation 3.26, solving for Δθ1 in terms of Δxscreen, which yields
훥푥 훥휃 = 2 ∗ 푎푟푐푠𝑖푛 ( 푠푐푟푒푒푛). (3.27) 1 2푟
Laser θ1,xscreen m=0
Δxscreen θ1+Δθ1 m=1
θi θ θ0 1
r
Δθ1
d=d0+Δd
Figure 22: Geometric depiction of laser light diffracted off a grating’s surface and projected onto a matte screen. Changes in grating pitch (Δd) result in changes in first order diffraction angle (Δθ1), translating the projected beam spot along the screen’s surface (Δxscreen). 58 Given the presence of the trigonometric functions sine in equation 3.26 and arcsin in equation 3.27, the small angle approximation sin(θ)≈θ may be warranted, provided the magnitude of Δθ1 is less than 20 degrees (0.349 rad). To confirm this, Δθ1 must be computed for values of d and d0 representative of semiconductor packaging materials undergoing changes in operating temperature (Fig. 22). An equation relating d, d0, and
Δθ1 can be derived as follows. First, equation 3.19 is rearranged to solve for θm, yielding
푚휆 휃 = 푎푟푐푠𝑖푛 ( ) − 휃 . (3.28) 푚 푛푑 푖
Next, the value of θm is computed for d and d0, producing
푚휆 휃 (푑) = 푎푟푐푠𝑖푛 ( ) − 휃 (3.29) 푚 푛푑 푖
푚휆 휃푚(푑0) = 푎푟푐푠𝑖푛 ( ) − 휃푖. (3.30) 푛푑0
Finally, θm (d) is subtracted from θm (d0) to compute Δθm
푚휆 푚휆 훥휃푚 = (푎푟푐푠𝑖푛 ( ) − 휃푖) − (푎푟푐푠𝑖푛 ( ) − 휃푖). (3.31) 푛푑 푛푑0
Setting constants m=1, λ=632.8nm and n=nair, as described in section 3.3.2 and multiplying the subtraction operator through the equation’s second term yields
632.8 632.8 훥휃1 = 푎푟푐푠𝑖푛 ( ) − 푎푟푐푠𝑖푛 ( ). (3.32) 푛푎푖푟푑 푛푎푖푟푑0
Expected values of Δθ1 can now be computed for diffraction gratings bonded to the surface of semiconductor packaging materials subjected to strain. Table 3 shows calculated values of Δθ1, with of d and d0 obtained from Table 1. These calculations 59 Table 3: Expected Values of Δθ1 Computed Using Equation 3.32
nair nair d0 (nm) d (nm) Material Δd (nm) Δθ1 (deg) (25ºC) (125ºC) (25ºC) (125ºC) Silicon 1.000267 1.000184 700.00 700.35 0.35 -0.0504
Oxide (SiO2) 1.000267 1.000184 700.00 700.05 0.05 0.0014 Epoxy 1.000267 1.000184 700.00 703.85 3.85 -0.6441 Solder 1.000267 1.000184 700.00 701.75 1.75 -0.2901 Copper 1.000267 1.000184 700.00 701.15 1.15 -0.1877 assume strain is caused by a change in package temperature from 25°C to 125°C. As evidenced from Table 3’s last column, the small angle approximation is valid when measuring grating pitch within the standard operating temperature range of a semiconductor package. From this information, equation 3.26 may be simplified to
훥휃 훥푥 = 2푟 ∗ 푠𝑖푛 ( 1) ≈ 푟 ∗ 훥휃 . (3.33) 푠푐푟푒푒푛 2 1
The utility of equation 3.33 is especially evident if a photosensitive detector of finite horizontal size, such as a CMOS imaging sensor, is used in place of a matte screen.
Most CMOS image sensors for machine vision are not larger than 1.28 centimeters horizontally, restricting the domain of Δxscreen to Δxcmos. Because of this, the value of Δθ1 does not exceed 20 degrees (0.349 rad) in most practical measurement scenarios, enforcing the small angle approximation. Under this condition, equation 3.33 can be rewritten as
훥휃 1 1 = . (3.34) 훥푥푐푚표푠 푟
In the above expression, r is constant and equal to the slope Δθ1/Δxcmos. Thus, the equation relating θ1 and xcmos must be linear, taking the form 60 1 휃 = ∗ 푥 + 푏. (3.35) 1 푟 푐푚표푠
Here, θ1 is the computed diffraction angle in radians, r radial distance of the CMOS sensor from the laser beam spot projected on the grating’s surface, xcmos is the position of the diffracted beam’s intensity peak along the horizontal axis of the CMOS sensor, and b is a constant with the same units as θ1, in this case radians. If obtaining values of θ1 in degrees is preferred, equation 3.35 becomes
180 휃 = ∗ 푥 + 푏. (3.36) 1 휋푟 푐푚표푠
For most machine vision cameras, the CMOS sensor is located behind a delicate glass filter element and often buried within the camera housing, making the value of r difficult to measure directly. A solution to this issue, as described in section 3.4.4, involves experimental determination of the constants 180/πr and b by plotting known values of θ1 in units of degrees against known values of xcmos in units of pixels and performing a linear regression. In this scenario, all constants in equation 3.36 may be expressed in units of degrees alone by assigning 180/πr to a new variable called m and replacing xcoms with a unitless pixel number called p. These changes result in the equation
휃1 = 푚 ∗ 푝 + 푏. (3.37)
Using values of m=0.00487 and b=53.576, obtained from regressing several (p,θ1) data pairs, the experimental setup’s angular sensitivity (S) can determined from S=1/m, yielding a value of ~205.34pixels/deg. With the relation between pixels and diffraction angle obtained, the magnitude of Δp is thus computed by multiplying Δθ1 by S and taking the absolute value. In Table 4, values of |Δp| rounded to the nearest pixel are listed for 61 700nm gratings bonded to common semiconductor package materials undergoing a change in temperature of 100ºC.
Table 4: Absolute Change in Pixel Position, |Δp|, of The Diffracted Beam’s Center
d0 (nm) d (nm) Material Δd (nm) Δθ (deg) |Δp| (pixels) (25ºC) (125ºC) 1 Silicon 700.00 700.35 0.35 -0.0504 10
Oxide (SiO2) 700.00 700.05 0.05 0.0014 0 Epoxy 700.00 703.85 3.85 -0.6441 132 Solder 700.00 701.75 1.75 -0.2901 60 Copper 700.00 701.15 1.15 -0.1877 38
3.4 Experimental Methods
3.4.1 Preparation of Stacked-Die Package
A stacked-die package, bisected and polished to reveal its cross-section, was provided by a commercial partner for use in this study. The package contained five vertical layers of active silicon die, with electrical connections between layers facilitated by through-silicon vias (TSVs). In a typical structure, the top cap of a TSV pillar tunneling through a silicon layer was bonded to the bottom cap of a TSV reaching into the next layer using a solder ball. In all instances, bonding of the upper a lower cap occurred in a region containing polymer material, located between the two silicon layers.
Fabrication of diffraction gratings on the polished cross-sectional surface of the sample package required deposition of a metal thin film (Fig. 23). This initial step was carried out inside Arizona State University’s NanoFab cleanroom using an electron beam evaporation tool manufactured by the Kurt J. Lesker company. Before deposition, all unpolished sample surfaces were covered with Kapton tape which could be removed after metal deposition was completed. The sample was then loaded into the tool’s vacuum 62 50nm Semiconductor e-Beam Au layer Package Cut/Polish Evaporation
Au
Figure 23: Depositing 50nm of gold (Au) on the exposed cross section of a diced and planarized semiconductor package via electron beam evaporation. chamber, such that evaporated metal could deposit easily on the exposed surface. Next, air in the chamber was evacuated, metal deposition could be performed when chamber pressure was less than 5.00x10-5 Torr. To promote adhesion, a 5nm layer of titanium (Ti) metal was deposited on the sample’s exposed surface. A layer of gold (Au) was deposited next, forming a well-adhered metal film. During a typical deposition, thickness of the Au layer was chosen to be 50nm. These films exhibited a mirror-like reflectance surface well suited for diffraction measurements while minimizing the layer’s influence on mechanical deformation of underlying materials.
Following deposition, the sample was removed from the electron beam evaporation tool. After peeling away any Kapton tape, the package was mounted to a 90º sample holder, exposing its gold-covered surface. Next a small piece of carbon tape was affixed to the sample’s exposed surface and the aluminum sample holder, ensuring both materials shared the same electrical potential. Due to the use of this tape, good adhesion between the gold layer and underlying package materials is a necessity, as the deposited gold layer must stay firmly bonded to the sample when the tape is removed. The mounted sample was then transferred to a FEI Nova 200 focused ion beam (FIB), attached to the multi-axis motion stage inside its vacuum chamber, and pumped down.
63 The FEI Nova 200 Focused Ion Beam is a multi-purpose tool, containing both a scanning electron microscope (SEM) and focused ion beam emitter (Fig. 24). While both components function by focusing a narrow cone of changed particles onto a sample surface, the Ga+ ions emitted by the ion source are approximately 1.30*105 (A. Tseng,
2004) times heavier than elections used by the SEM. In a FIB instrument, images are acquired by scanning the electron and/or ion beam across the sample, measuring the number of secondary electrons escaping from each scanned point on the surface. High spatial resolution is achievable due to narrow beam widths, on the order of 1nm for elections and 7nm for ions at the instrument’s working distance (Ishitani & Tsuboi,
1997).
During a typical FIB session, a video feed from the SEM was first used to identify and inspect regions of interest (ROIs) on the sample’s Au-coated surface using low
Figure 24: Illustration of a dual-beam FIB-SEM instrument depicting orientations of both beam columns. Expanded view shows sample surface interaction with elections and Ga+ ions (Volkert & Minor, 2007). 64 magnification. These regions consisted of one or more packaging microstructures such as
TSV caps, TSV pillars, solder balls, and polymer underfill. Effort was also made to identify flat regions with underlying bulk silicon, as the areas often made ideal locations for milling high-resolution calibration gratings. Once a suitable area was chosen, a high- resolution SEM micrograph was acquired and recorded as a greyscale image. Next, the multi-axis stage was rotated, bringing the sample’s Au-coated surface perpendicular to the ion emitter’s angle of incidence. Following rotation, the micrographed region was re- centered in the SEM’s view field. However, due to the sample’s tilt, surface features within the SEM’s view field were resolved from an isometric perspective.
Diffraction gratings fabricated on a sample’s surface should consist of evenly spaced trenches. Furthermore, the depth of each trench should not exceed the total thickness of the Ti/Au layer, preserving the deformation mechanics of any underlying materials. In a FIB tool, precision fabrication of such trenches is accomplished via ion milling, which utilizes a high-current beam of Ga+ ions focused on the sample to sputter surface material (Volkert & Minor, 2007). However, ion beam current and dwell time must be adjusted by the operator to achieve acceptable trench depth before large gratings can be written.
Experimental determination of these two parameters was accomplished by ion milling a series of ten vertical lines 28μm in length. Each line was 350nm wide, while the distance between centers of adjacent lines was 700nm. The first series of lines was milled at relatively low beam current and dwell time, producing a low contrast pattern when observed by the SEM. Next, beam current and dwell time was increased and a new series of lines written adjacent to the previously milled set. Contrast of the new calibration
65 pattern was then checked against the previous pattern. This procedure was continued until dark, well-defined lines were first observed, indicating that the ion-milled trenches had reached the underlying material. To aid in the determination of trench depth, all pattern sets were milled in flat regions containing an underlying layer of bulk silicon.
After setting ion beam current and dwell times to achieve an optimal milling depth of 55nm, five 28μm x 28μm calibration gratings with uniform pitch values of
700nm and 730nm were fabricated atop the flat layer of bulk silicon (Fig. 25A.). The purpose of these gratings was twofold. Laser light normally incident on the grating is coupled into the first diffractive order at a fixed angle, assisting the experimenter during placement of a CMOS camera sensor which measured the intensity distribution of outgoing light. Additionally, the difference in pitch between two calibration gratings measured via laser diffraction serves as a validation metric, as the actual pitch delta is
A. B.
Figure 25: (A.) SEM micrograph of two 28μm x 28μm calibration gratings trenched into a 50nm layer of gold, deposited on a region of bulk silicon, using a FEI Nova 200 focused ion beam instrument. Upper and lower grating have pitch values of 730nm and 700nm respectively. Due to the electron beam’s 52° offset, spatial compression is observed in the micrograph’s vertical direction. (B.) Select set of 70μm x 77μm gratings with a 700nm pitch fabricated atop though silicon vias (TSVs), defining regions of interest (ROIs)
66 well known. Following construction of calibration gratings, nearby regions of interest
(ROIs) containing package microstructures were inspected with the SEM. ROIs containing several underlying materials, such a copper, solder, and polymers in proximity were given preference during this step. Next, gratings with an areal footprint of 70μm x
77μm and uniform pitch of 700nm were ion milled atop the chosen ROIs (Fig. 25B).
Finally, micrograph images of each ion-milled region and calibration grating were acquired using the SEM.
3.4.2 Sample Package Mounting and Alignment
Concluding fabrication of various calibration gratings, lines, and ROIs, the sample package was removed from the FIB and taken to an optical metrology laboratory containing the diffraction-based strain measurement setup (Fig. 26). In this lab, the package was placed between two copper blocks, and secured with Kapton tape. Both the upper and lower blocks contained ports for cartridge heaters connected to a temperature control module (EuroTherm 2100), allowing heating and temperature regulation of the sample if desired. Before placing the top block, a small thermocouple wire, connected to the temperature controller, was affixed to the top surface of the package using Kapton tape. This provided a means to regulate sample temperature using the controller’s feedback loop feature.
The copper block sandwich containing the sample was then placed in front of a
50X long working distance microscope objective using an insulating sample holder attached to a motorized XYZ stage. A CCD camera directed towards the rear of the objective though a 50:50 beam splitter cube could be used to view the sample surface at the objective’s working distance, while a halogen lamp provided oblique illumination.
67 A mounted sample surface lacking parallelism with the front face of the microscope objective induces systematic errors when measuring diffraction angle, as the condition θ1>>θi does not hold. To verify parallelism, the sample was translated horizontally while observing the clarity of in- focus surface features. Due to the objective’s small depth of field, a misaligned sample rapidly de-focuses during translation. If clear features were still observed after 0.7mm of translation, the sample
A. CMOS Sensor B.
+ΔT He-Ne Y
Laser BS θ1 X wd Sample lens C. CCD camera Lamp
D. E.
Figure 26: (A.) Illustration of experimental setup for measuring local diffraction angle (θ1) at various sample positions in the X,Y plane. (B.) Photograph of a thermocouple, used to monitor/control temperature, attached to the sample package before mounting. (C.) 50nm gold layer on the cross section of a sample package, visible after sandwiching between two copper blocks containing ports for cartridge heaters. (D.) Photograph showing the optical setup in its entirety, with the He-Ne tube laser, 50:50 beam splitter, and CCD imaging camera on the back side of the microscope objective. (E.) Photograph of experimental setup in its typical configuration, showing the copper block sandwich placed on the sample holder with cartridge heaters inserted; XYZ stage, microscope objective and CMOS camera sensor. 68 was deemed sufficiently parallel. If the sample failed to meet this benchmark, it was removed from the copper blocks and remounted.
3.4.3 Grating Pitch Determination via Laser Diffraction
Accurate determination of grating pitch on any ROI required a diffraction angle measurement carried out in two steps. First, light from a He-Ne laser tube was directed though the 50X microscope objective lens to a lamp-illuminated sample. Viewed using the CCD camera, a focused laser spot approximately 4.6µm in diameter was readily visible on the sample surface (Fig. 27) while translation of the sample in the X,Y plane could be used to reposition the spot. Under such viewing conditions, the laser spot was moved to overlap the grating under test. It should be noted that during this step, optical power from the He-Ne laser tube was reduced though the use of neutral density (ND) filters to avoid saturating the CCD camera.
Next, the angle θm, created between normally incident laser light landing on a grating surface and reflected light coupled into the first diffractive (m=1) order was
28μm 680nm
700nm 770nm
Beam Spot 700nm 730nm
Figure 27: Image of several calibration gratings with uniform pitches between 680nm and 770nm as viewed though the microscope objective by the CCD camera. The bright circle at the center of the image is the laser beam spot projected onto the sample surface though the 50X objective lens. This beam spot can be moved to a desired position or scanned across a grating by translating the manual/motorized X,Y stage.
69 measured. While an approximate value of θ1 could be determined without use of electronic measurement aids by projecting diffracted light onto a flat paper screen, detecting small changes in θ1 during a typical 25°C to 125°C thermal loading experiment was impractical on a laboratory optics bench. Materials utilized in contemporary semiconductor packages exhibit low coefficients of thermal expansion as illustrated in
Table 1 and it follows that diffraction gratings bonded to surface of these materials exhibit changes in diffraction angle of less than 1º. To detect such small changes, a five megapixel, one-inch format CMOS camera sensor (JAI GO-5000C) with 5µm pixel spacing was employed in place of a screen (Fig. 26). The sensor was placed directly in the path of light propagating away from the sample at a distance of 5.86 centimeters, with its active area perpendicular to impinging rays. During measurement, all non-laser light sources were turned off and output power of the He-Ne laser adjusted using the ND filters to provide optimal light intensity to the CMOS sensor.
A. B.
Σ
0 1280 2560 Position (pixels)
Figure 28: (A.) Typical intensity distribution of light diffracted into the m=1 diffraction order, as collected by the CMOS camera sensor. (B.) 1D intensity profile generated by summing the CMOS sensor’s pixel columns and its subsequent Gaussian fit. Horizontal pixel position of the Gaussian peak is denoted by “p”. 70 Diffracted laser light intercepted by the sensor exhibited a two-dimensional
Gaussian intensity distribution with a central maximum at the diffraction angle in (θ1), as described by the Fraunhofer diffraction equation in section 3.3.3. A captured image of this distribution (Fig. 28A) was then converted to its respective 1D (azimuthal) intensity profile via column summation using LabVIEW (Fig. 28B). The resulting 1D array was then fitted to a single-peak Gaussian function using Python3’s SciPy library, obtaining an approximate horizontal position (p) of the central peak in pixel coordinates (Fig. 28B). A linear pixel-to-angle calibration function, described in section 3.4.4, was then used to compute the diffraction angle, with grating pitch determined from equation 3.21.
3.4.4 Pixel-to-Angle Calibration of a CMOS Camera Sensor
While the camera’s 5μm pixel spacing allowed precise localization of Gaussian intensity peaks in pixel coordinates, peak locations required conversion to diffraction angle before any experimental determination of grating pitch was performed. To obtain a high-quality pixel-to-angle calibration curve, the sample holder described in section 3.4.2 was replaced with a rotatable, angle-calibrated mirror mount (Fig. 29A). Next, light from the He-Ne laser was passed through the 50X objective lens onto the surface of the rotatable mirror, which directed light towards to the left side of the CMOS camera sensor.
Images of laser light collected by the sensor revealed a Gaussian intensity distribution with a well-defined central peak whose horizontal pixel position was calculated using the LabVIEW/SciPy framework and associated with its respective reflection angle (θr). Next, the mirror surface was rotated by a fixed amount, increasing the reflection angle (θr) by twice the mirror’s change in angle (Fig. 29A) and shifting the intensity peak towards the right side of the CMOS sensor. A new peak position (p) was
71 A. B. End Start End
θr Mirror Start
θr/2
Figure 29: (A.) Illustration of pixel-to-angle calibration procedure for the CMOS camera sensor. Positioning the reflected laser spot on the sensor’s left side (Start), an initial value of θr is associated with the beam spot’s center position in units of pixels. Next, the mirror is rotated, shifting θr to the right by twice the change in rotation angle. The new value of θr is then associated with the beam spot’s new position. This procedure is repeated until the beam spot reaches the left side of the sensor (End), yielding a set of pixel-to-angle calibration points. Notably, value of θr equals the diffraction angle (θ1) if the mirror surface remains close to the microscope objective’s working distance. (B.) Pixel position (p) vs diffraction angle (θ1) curve extrapolated from six calibration points using linear regression. then determined and associated with the new reflection angle. Repetition this procedure across the active sensing area revealed a linear relationship between horizontal pixel position (p) and reflection angle (Fig. 29B). Finally, an approximate mathematical equation relating p to θr was computed via linear regression.
At microscope objective’s working distance, mirror reflection angles are equivalent to diffraction angles generated by a grating parallel to the objective’s front face (θr = θ1). To ensure that the reflection surface remained near the working distance of the objective lens during calibration, the beam spot landing on the mirror was monitored using the 50:50 beam splitter and CCD camera. If the laser spot was sufficiently defocused after rotation, the mirror was re-positioned using the XYZ stage to restore focus.
72 3.4.5 Verification of Known Pitch Deltas by Line Scanning
Validation of the localized pitch detection technique described in sections 3.4.2 through 3.4.4 was performed by measuring a difference in pitch between two calibration gratings fabricated atop a large, flat region of silicon substrate (Fig. 25A). Chosen calibration gratings were 28µm x 28µm in size, containing uniform pitch values of
700nm and 730nm respectively. In preparation, the laser beam spot with a 4.6μm diameter was first moored off the left-central side of the 700nm grating, using the CCD camera to view the illuminated sample surface.
After disabling the halogen lamp, the LabVIEW program described in section
3.5.1 was executed, sweeping the beam spot horizontally (Fig. 27) to the right at a velocity of 10 µm/s while simultaneously acquiring images of light coupled into the first diffractive order at 30 frames/s. Motion and image acquisition were halted when the x- axis stage reached a target position of value of 36µm. Grating pitch was then calculated for all images acquired along the horizontal scan path using additional LabVIEW and
Python3 programing scripts described in sections 3.5.2 and 3.5.3. The scan process described above was then repeated on the 730nm calibration grating.
3.4.6 2D Pitch Mapping of a TSV-Solder Ball Microstructure
While a single horizontal scan is sufficient to measure the uniform pitch of a calibration grating located atop flat regions of homogenous silicon, the area beneath gratings fabricated on regions of interest (ROIs) often contain sub-regions composed of different materials. In such a case, a single horizontal scan is insufficient to describe changes in the grating’s local pitch over its entire area. Instead, horizontal scans taken at a series of sequential vertical steps are required.
73 In a typical full area 2D pitch mapping procedure, the laser beam spot, ~4.6μm in diameter, was first positioned off the lower left corner of a grating ROI using the CCD camera to view the illuminated sample surface. After removing all extraneous light sources, the LabVIEW program described in section 3.5.1 was executed, sweeping the beam horizontally across the sample surface at 10µm/sec while images of light coupled into the first diffraction order were captured at 30 frames/sec by the CMOS camera sensor. Upon reaching its target position along the x-direction, the spot was backtracked to its starting position. Next, the beam spot was translated upward by a vertical step of
1.0μm and a new line scan acquired (Fig. 30). This automatic step-and-scan process was repeated until the number of vertical steps specified by the experimenter was reached, halting the program. Grating pitch was then calculated for all images acquired along each
A. B. nm 48 800 42 780 36 30 760 24 740 3. Vertical Step 18 720 12 2. Zero Stage 700
6 680 1. Horizontal Scan 0 μm 0 10 20 30 40 50 60 70 80 80μm Figure 30: (A.) Illustration of a step and scan procedure for high-resolution 2D pitch mapping of an Au thin film diffraction grating fabricated on a region of interest in this case, a TSV. After manual positioning the laser beam spot off the lower left corner of the ROI, a LabVIEW program sweeps the laser spot across the grating area horizontally while acquiring images of light coupled into the first diffractive order at 30fps. Next, image acquisition is suppressed, and the stage is translated in reverse. Upon reaching its zero position on the left side of the grating, the beam spot is translated by one vertical step. The two-part horizontal scan procedure is then repeated for all vertical steps. (B.) Interpolated grating pitch map of an ROI containing a TSV, black dots show where pitch measurements were made. 74 horizontal scan path using additional LabVIEW and Python3 programing scripts described in sections 3.5.2 and 3.5.3. From the calculated pitch values obtained at each horizontal scan, high-resolution pitch maps of the 70µm x 77µm grating ROI with an underlying TSV-solder ball structure were constructed using the SciPy interpolation routine described in section 3.5.4.
Because local changes in grating pitch are caused by thermal loading, areal scans of the grating-topped ROI were obtained at sequential package temperatures of 25°C,
45°C, 71°C, 93°C and 123°C. During this series of measurements, the ROI was first measured without heating, at a temperature of 25°C. Next, temperature was increased to
45°C using cartridge heaters located inside the copper blocks sandwiching the sample
(Fig. 26). Each cartage heater was connected to the thermal control unit, which automatically regulated sample temperature to the 45°C setpoint using a thermocouple- based feedback loop.
Following an increase in temperature, the experimental setup was left untouched for 15 minutes, allowing the sample to reach thermal equilibrium. Upon thermalization, illumination was switched on and the sample surface inspected using the CCD camera.
Next, the beam spot was refocused as necessary and repositioned off the lower left corner of the grating. After disabling illumination, a new full-area scan was acquired and saved.
This procedure was carried out at each temperature step, generating a series of high- resolution pitch maps at various package temperatures.
75 3.5 Full-Field Pitch Mapping using Computer Automation
3.5.1 Automated Capture of Diffractive Intensity Distributions Over a Grating Area
Focusing collimated light from a He-Ne tube laser though a 50X microscope objective to the sample’s surface results in a beam spot approximately 4.7μm in diameter at the objective’s working distance. Grating pitch measured by this method is highly localized, as the spot’s areal size is significantly less than that of a calibration grating or
ROI. Because of this, full-field pitch mapping requires the beam spot to be systematically swept over a grating’s entire area, with measurements taken at numerous positions. Use of a CMOS camera permits high-speed sequential acquisition of intensity distribution images formed by laser light coupled into the first diffractive order, from which diffraction angle and grating pitch are calculated. If a series of such images is acquired at various beam spot positions, a full-field pitch map can be created and visualized using computer software.
In service of this goal, a computer program for systemically sweeping the small beam spot over a rectangular grating region of arbitrary size while acquiring images from the CMOS camera was implemented in LabVIEW, a graphical programming platform well-suited to hardware automation. The program’s operation can be summarized as follows. At program start, two motorized linear stages (CONEX-CC, Newport Inc.) capable of moving a sample in the horizontal and vertical directions respectively are initialized and set to their home positions using a LabVIEW virtual instrument (VI) block. This VI allowed LabVIEW to send motion commands to the stages using the
CONEX communication library. Following stage homing, a five-megapixel CMOS machine vision camera (GO-5000C, JAI) was initialized for later use via LabVIEW’s
76 OpenCamera VI. Upon completion of motorized stage and camera initialization tasks, the program entered a multi-block nested loop governing spot position and image acquisition.
Within the loop structure, three sequential program blocks were run. The first block moved the sample downward using the vertical stage (Y-axis), shifting the beam spot upward by one step position in the vertical direction. Both vertical step size and number of sequential steps was supplied by the user via the LabVIEW front panel before runtime. The next program block consisted of three parallel while loops, the first of which translated the sample leftward to a user-specified target position using the motorized horizontal (X-axis) stage at a velocity 10μm/s. Such motion could be used to sweep the beam spot from left to right across a grating under test. In the second loop, images of diffracted intensity distributions were grabbed from the CMOS camera sensor and placed in an image buffer. Within the third loop, sequential images placed in the
Initialize Hardware Set Parameters Motion/Acquisition Loop
Conex-cc USB Driver X stop, Xmax Translate max Horizontal Zero stage X Stage + velo Vertical LabVIEW Ymax Horizontal and Step Grab Image Vertical Stage QMH Ystep
Create JAI USB Camera Driver IMAQ Folder Buffer Save Image + Folder to Folder Dir
Figure 31: LabVIEW program architecture for automated capture of diffractive intensity distributions over a grating area. This program sweeps the laser beam spot across the grating’s surface laterally while acquiring images. After each horizontal scan, the laser beam spot is shifted by one step in the vertical direction, allowing full-field areal measurement of grating ROIs 77 image buffer were assigned file names, systematically saved to the computer’s hard drive in a new folder, and deleted from the buffer, freeing memory space for incoming images.
Using a queued message handler, all three while loops ceased operation when the horizontal stage position reported by the first loop reached its target position. Following leftward translation, the horizontal stage was backtracked to its zero position in the third program block. Reaching zero, the loop structure was run again, following a 1000ms delay.
Upon completing a user-specified number of vertical steps, the multi-block loop structure was exited, moving the program onward to its final phase. In this last step, all
LabVIEW-created image buffers and message queues were closed, while control of the motorized stages and camera were returned to the computer’s operating system before program exit.
3.5.2 Post-Acquisition Binning of Intensity Distribution Images
Light from a He-Ne laser focused though a microscope objective and coupled into a grating’s first diffractive order exhibited an oblong two-dimensional (2D) gaussian intensity distribution when captured as an image by the CMOS camera. Such distributions may be represented as a 1D intensity profile following the principals of
Fraunhofer diffraction, as outlined in section 3.3.3. This alterative representation may be used to great effect, reducing the amount of hard disk space required to store areal scan data and improving the speed at which fitting computations are performed, as only a 1D data array a few thousand entries in length is needed. On further inspection, a given distribution’s height was maximum near columns of pixels associated with the theoretical diffraction angle of the grating (Fig. 28A). Horizontally, the distribution’s brightest pixels
78 were also near coordinates associated with the grating’s diffraction angle (Fig. 28A).
Given these observations, it follows that the horizontal profile of a 2D gaussian intensity distribution can be represented one-dimensionally if each column of pixels is summed.
Using LabVIEW, a program for summing the pixel columns of each image and saving the resulting array onto a computer hard disk was written. The program consisted of a nested loop structure, containing several blocks executed in sequence. At program start, the path to a folder containing all images acquired at given vertical step was constructed as a text string. Next, the string was passed into a while loop containing four sequential code blocks. In the first block, the string was appended with the name of a single image matching the while loop’s numeric count. This practice ensured that every image in the folder was loaded by the program and processed only once.
Entering the program’s second block, the appended string containing a full file path to a single image was passed into an image loading VI provided by LabVIEW and loaded into memory. In the third program block, a summation routine looped over each
Set Input Path Load Image Column Summation Save Array
Append Append Path with Path with Folder # Image # npx
Create
Folder Σ
… … … …
Inner
Image Output
Loop … n Count Folder px Folder Set Loop Output Count Path
Figure 32: LabVIEW program architecture for constructing 1D Fraunhofer intensity profiles from 2D Gaussian distributions captured as images at numerous scan locations over a grating area and saving them as 1D arrays. 79 pixel column in the image, creating a 1D array of integers whose length was equal to that of the original image’s width in pixels. The value of each element was then plotted against its respective index and displayed in graphical form on the LabVIEW front panel, allowing the experimenter to observe each gaussian intensity profile.
In the fourth program block, the resulting array was saved as a comma separated value file (CSV) in a user-specified directory which mimicked the folder path structure and file naming scheme of the original image data. When all images in the first vertical position folder were processed and their respective arrays saved in a new directory, the program then looped to the next vertical position folder and the image summation process repeated. The program exited when images in all vertical position folders present in the input directory were processed, and their respective profiles saved to the output directory.
3.5.3 Determination of Intensity Profile Maxima by Gaussian Fitting
1D intensity profiles created from captured images were fitted to a single-peak gaussian function, allowing the horizontal pixel position (p) of each intensity peak to be extracted from the fitted constants. This process was automated using the Python 3 programming language, as many high-quality data analysis methods are available through the SciPy data science libraries. Indeed, the chosen fit routine was able to adequately suppress high-frequency periodic noise present in the profiles, revealing to their intensity envelope.
The fitting script consisted of four program blocks. On program start, the first block loaded the necessary SciPy libraries. In the second block, a path to a folder containing saved intensity profiles acquired at a given vertical position was constructed as a text string. This string was then passed to a loop containing the third code block.
80 Set Load Array Gaussian Peak Fitting Append Input Append and Save Path Path with Append Folder # Path with File #
Loop Count
Pixel-to-Angle Conversion … Create Inner Loop θ = 0.00487p +53.576 Output n 1 Set Folder Count px Compute Grating Pitch (d) Output Path
Figure 33: Python 3 programming architecture for fitting 1D intensity profiles to a single-peak Gaussian and computing the grating pitch for each profile using the pixel-to-angle calibration curve. This script generates two- column arrays which associate each pitch measurement with its horizontal location along the grating area. While each horizontal scan line is indexed by its vertical step position.
Here, the text string was appended with the name of a single intensity profile and the named file loaded into memory as a NumPy array. After loading, the intensity array was fitted to a gaussian function by setting the element index, representing horizontal pixel location, as the independent variable and the element value as the dependent variable.
Next, the horizontal location of the peak was extracted from the fitted gaussian function and its value passed into the linear pixel-to-angle calibration equation described in section 3.3.4. The resulting diffraction angle (θ1) was then converted to grating pitch (d) using equation 3.21 and saved as the first element of a new array. The loop was then repeated, loading the next intensity profile into memory, fitting, and placing the computed pitch value (d) into the first unoccupied element of the pitch array.
After all intensity profiles in a folder labeled by its vertical position were fitted.
The completed pitch array was passed to the fourth program block and saved as a CSV file in a new user-specified directory with a file name based on the scan’s vertical 81 position. The program then shifted to the next vertical position folder, repeating all four programming blocks. The exit condition for the program was met when intensity profiles present in every folder were fitted and their computed grating pitch values saved to the computer’s hard disk.
3.5.4 Construction of an Areal Pitch Map
From horizontal pitch scans obtained at each vertical step, a full-field pitch map of the grating region was constructed using a multi-block program written in Python 3.
On program start, all necessary libraries were imported into Python 3. Next, the program entered a loop structure containing three sub-blocks. In the first sub-block, a path to the
CSV file containing pitches associated with the first vertical scan position was passed into a loader function as a string, loading the CSV file into memory as a 1D NumPy array. In the second sub-block, a row containing the horizontal position of the laser beam spot in units of micrometers was appended to the array, associating each pitch measurement with a horizontal location. Next, a new row containing the loaded scan line’s vertical position at every element was appended to the two-row array. The newly constructed three-row array was then copied into a large pre-allocated array at its first unoccupied column in the third sub-block. This three -part loop was then repeated for each CSV file containing pitch scan data, appending the pre-allocated array on every iteration. When all available CSV files containing pitch measurements were copied to the pre-allocated array, the program exited the loop while completed array was held in memory for later use.
In a full-area pitch measurement routine, the laser beam spot was translated approximately 0.333μm per measurement during a horizontal scan, with each scan line
82 spaced 1.0μm apart in the vertical direction. While this distribution of measurement points could be used to construct a rudimentary 2D pitch map, the raw data is noticeably sparse, as regions between each scan line remain unaccounted for. Fortunately, vertical distance between adjacent scan lines was small compared to the 77μm grating height. In such a scenario, pitch values located between adjacent scan lines can be accurately estimated using SciPy’s implementation of the Clough-Tocher 2D Interpolation algorithm.
For interpolation, measured pitch values were placed on a virtual 81μm x 81μm grid space by passing each column of the recently completed pitch measurement array into a grid space placement function. Next, the pitch value at every intersection point within the grid space was computed via interpolation. Typically, grid intersection points were spaced 0.1μm apart, increasing the map’s point density by a factor of three in the horizontal direction and a factor of ten in the vertical direction. In the final code block,
Set Load Array Construct Sparse Interpolate and Input Append Pitch Map Save Pitch Map Path Path with Horizontal Beam File #
Spot Position
… …
…
Vertical Step Vertical …
… …
Set Loop Output Count N d Path
Figure 34: Python 3 programming architecture for constructing areal pitch maps of a grating ROI by concatenating horizontal scans at each vertical step into a rectangular array. The pitch values between known measurement points in this array were then interpolated to obtain a high-resolution map. 83 the interpolated pitch map was saved to the computer’s hard disk in a user-specified directory as a CSV file.
3.6 Results and Discussion
3.6.1 Verification of Calibration Grating Deltas
Plotting measured pitch against horizontal beam spot position shows a distinctive five-phase pattern for both the 700nm and 730nm calibration gratings. During the first and last measurement phase, no portion of the beam spot overlaps the grating. Because of this, pitch readings remained fixed at the highest available pitch value. This behavior was expected, as acquired intensity images contain only pixel dark noise, producing non- gaussian intensity profiles when each pixel column is summed. Because no Gaussian distribution could be fitted to the such profiles, the chosen fitting algorithm simply located the column of highest total intensity. Typically, this column was positioned on the left edge of the CMOS sensor, corresponding to the highest available pitch value.
The second measurement phase shows a marked drop in the measured pitch as more of the beam spot’s total area overlaps the grating near its left edge. This drop is likely due to the coupling of a semicircular intensity distribution into the first diffractive order. Because the intensity profile of light landing on the grating surface is preserved during diffraction, a partially overlapped Gaussian beam distribution will be brighter on its left side than its right, skewing the diffracted intensity distribution towards the left side of the CMOS sensor.
The third measurement phase is reached when the beam spot completely overlaps the calibration grating. During this phase, no skewing artifacts are present, allowing the
84 1.
5.
2. 3. 4.
Figure 35: Plot depicting measured pitch vs. laser spot position on the sample surface for two calibration gratings with uniform pitch values of 700nm and 730nm at 25°C. The measurements exhibit five distinct phases. In phases 1 and 5 the laser beam spot does not overlap the sample grating. A distinct drop in pitch marks phases 2 and 4 as the laser spot partially overlaps the grating surface, skewing the diffracted intensity peak captured by the CMOS camera. In phase 3, the laser spot completely overlaps the grating. Here, difference in grating pitch is easily discernable. pitch to be accurately measured. Here, the difference in pitch between the 700nm and
730nm grating is easily discernable with minimal measurement noise.
Similar to the second phase, the fourth phase exhibits roll-off, with measured pitch decreasing towards a minimum. As the spot exits the grating area on the far-right edge, the brightest portion of the spot remaining on the grating area shifts further and further to the right, skewing the diffracted intensity distribution towards the right side of the CMOS sensor.
3.6.2 Multi-Temperature Line Scan Analysis of a TSV-Solder Ball Microstructure
Calculated pitch values from scan lines at fixed vertical positions on a 70µm x
77µm ROI containing a copper Through-Silicon Via (TSV) bonded to a solder ball were visualized as line plots. Unlike the calibration gratings, measured pitch varied significantly with horizontal position in regions inside the ROI’s border, despite a uniform grating pitch of 700nm and a constant sample temperature of 45°C. Because
85 grating pitch, diffractive order, and the laser’s optical wavelength remain fixed during a full-area scan, such dramatic shifts in measured pitch indicate that the condition θi << θ1 no longer holds in equation 3.21. This strongly implies that the ROI’s surface is not flat, with the measured value of θ1, and by extension pitch (d), skewed by the laser’s angle of incidence (θi).
Intriguingly, variation of the ROI’s surface normal along a given horizontal scan line leads to many distinct graphical features on the generated plots. Furthermore, these features can used to visually confirm reproducibility of the line scans at various package
III. II.
I. I. 80μm
II. III.
Figure 36: Pitch measurement data from a 71µm x 76µm grating ROI containing a copper Through-Silicon Via (TSV) structure at three different vertical positions. Large deviations of measured pitch from the known reference pitch value of 700nm are due to local changes in surface angle (θi). Scans taken at 25°C and 125°C show good matching of key features such as peaks, valleys, and curves. Under such circumstances, differences in grating pitch (Δd) caused by thermal loading can be measured accurately in some areas.
86 temperatures. Indeed, scans acquired at the same vertical positions show good matching of key features like peaks, valleys, and curves at 45°C and 123°C. Because of this, differences in grating pitch caused by thermal loading can be measured in sub-regions with similar slope profiles.
3.6.3 Strain Measurement of Si and Cu Using High-Resolution 2D Pitch Maps
Interpolated grids generated from 2D scans taken at various package temperatures were then converted to maps of pitch delta via element-wise subtraction. Typically, grating pitch maps acquired at a lower temperature were subtracted from maps acquired at higher temperature (Fig. 37). Upon examination of the maps, measured pitch deltas near the interface region of two materials, such as copper and silicon are often on the order of tens of nanometers or more. However, Δd’s predicted from known material
CTEs are on the order of one nanometer or less, as shown in Table 1. This disparity is likely due to large surface angles (θi) near material interfaces, biasing measured pitch values in accordance with equation 3.21. A map of computed surface angles (Fig. 38C) shows rapid variation near interfaces, with values ranging from 0-3 degrees on one side, and values of -6 degrees or more on the other. These regions correlate closely with spurious measurements of pitch. An additional source of measurement error for interfaces containing copper and solder is the metals’ capacity to produce specular reflections. In the presence of large localized surface angles, light from the incident laser beam may reflect off metal underneath the grating, changing the intensity distribution of light coupled into the first diffraction order, and thus the location of a fitted Gaussian peak.
Surface roughness may also contribute to pitch measurement errors in metal regions, but warrants further investigation using commercial profilometry tools.
87 In regions of the pitch map located away from noisy interfaces, surface angles (θi) were less extreme with measured pitch deltas on the order of +/- 5 nm or less (Fig. 37).
Using the Matplotlib visualization package, sub-regions containing low pitch variance
Polymer Cu Polymer
Solder m
67μm 48μ Cu Si Si
45ºC 95ºC 123ºC
45ºC 45ºC 25ºC
Δd Δd Δd
Figure 37: High resolution grating pitch (d) maps of a though-silicon via (TSV) and its adjoining solder ball at various package temperatures. Lower row shows the difference in pitch (Δd) computed when a pitch maps acquired at higher temperature is subtracted from a map acquired at lower temperature. As temperature difference increases, changes near material interfaces are signifgiantly larger than the values predicted in Table 1, while regions located away from the interfaces are minimally impacted, exhibiting changes in pitch similar to the predicted values. 88 were identified for copper and silicon and the mean value of pitch deltas (Δd) in each sub-region was then computed at several package temperatures (Fig. 38). These means were then divided by the reference pitch of 700nm in accordance with equation 3.17 and multiplied by 100, obtaining the strain, ε as a percentage.
Plotting ε against package temperature shows linear relationships for silicon and copper regions with slopes of -0.0010 and -0.0018, corresponding to CTE values (α) of -
10.0ppm and .-18.0ppm respectably (Fig. 38B) Magnitudes of these experimentally determined CTE’s show reasonable agreement with the actual α values of 5.0ppm and
16.7ppm for silicon and copper, shown in Table 1; although the experimental CTE value for silicon was incorrect by a factor of two. Intriguingly, the presence of a negative sign in the measured CTE values for copper and silicon implies thermally induced compression of the grating and by extension, underlying materials in the chosen sub- regions. This behavior may be due to compression of package materials and structures located below the sample’s surface or is possibly induced by changes to the local surface angle as package temperature increases.
3.6.4 Computation of Surface Angle From 2D Pitch Maps
Preliminary line scans of the grating ROI described in section 3.6.2 imply deviation in local surface angle (θi) from horizontal (θi=0) on the order of a few degrees.
This observation was quantified using a surface angle map (Fig. 38C), generated by rearranging equation 3.20 to solve for θi, and using all entries on the pitch map acquired at 25°C to compute θi element-wise.
푚휆 휽 = 푎푟푐푠𝑖푛 ( − 푠𝑖푛(휃 )) (3.38) 풊 풅 푚
89 In equation 3.38, θi is the computed surface angle map, m=1, λ=632.8nm, d is the pitch map array acquired at 25°C, and θm is the diffraction angle of the m=1 diffraction order.
In this calculation, sin(θm)=632.80/dR where dR is equal to the reference pitch value of
A. B.
II Cu polymer polymer I solder
II
Cu Si I Si
C. D. II II
Surface I I Angle (θi) Δd
Figure 38: (A.) SEM image of the grating ROI that underwent high-resolution 2D pitch mapping. Regions of different materials such as the silicon substrate, copper pillar and solder ball are visible (B.) Plot of the mean strain values in the x-direction computed for each material sub-region at various package temperatures. While magnitudes of the slopes are similar to the actual CTE values for copper and silicon, the measured strain carries a negative sign. This suggests that the materials undergo compression during package heating but other causes must also be investigated. (C.) Visualization of a high-resolution surface angle (θi) map generated from equation 3.38 for a package temperature of 25°C. Material interface regions often exhibit large variances in surface angle, while regions located away from interfaces present smaller, more gradual changes in angle. (D.) Visualization of a pitch delta (Δd) map constructed by subtracting the 123°C pitch map by the 25°C pitch map element-wise. Spurious pitch deltas are often recorded near material interfaces due to rapid changes in the local surface angle (θi). Because of this, sub-regions of silicon (I) and copper (II) located away from any interfaces were used to compute strain at various package temperatures. 90 700nm. Examining θi, it can be seen that the sample contains local surface angles between 3.4 degrees and -6.4 degrees from horizontal, covering the entirety of the CMOS camera sensor’s measurable range (Fig. 38C). As such, it is likely that angles recorded at the extremes are larger in magnitude than what the sensor can measure, given its limited size. Perhaps unsurprisingly, the largest changes in surface angle are often observed across material interfaces. However, such variations also make accurate measurements of grating pitch deltas in these regions challenging. In areas located away from material interfaces, angle magnitudes are smaller, approaching zero (horizontal) in some regions, with gradual changes in angle of a few degrees. Two regions of this type, containing silicon and copper respectively were used to compute stain at various package temperatures (Fig. 38B).
3.7 Conclusion
In this chapter, a method of mapping nanoscale strain and surface angle using thin
Au diffraction gratings fabricated on the surface of a microelectronics package was presented in detail. Changes in pitch on the order of nanometers were observed by illuminating the grating surface with light from a focused He-Ne laser while monitoring the intensity distribution of light coupled into the first diffractive order. This was accomplished by projecting outgoing light onto a wide-area, angle-calibrated CMOS image sensor located in the diffraction pattern’s far-field (Fraunhofer) region. Knowing the horizontal position of the intensity distribution’s peak value, diffraction angle and grating pitch could be accurately calculated. Indeed, changes in sample dimensions on the order of nanometers were magnified by a factor of 105 in the Fraunhofer regime.
Furthermore, if grating pitch was already known, surface angle could also be computed.
91 Full-field pitch mapping of a grating bonded to a TSV-solder ball microstructure at various package temperatures was carried out by sweeping the focused laser beam spot across grating surface using a motorized two-axis translation stage. At each sample position, an optical intensity distribution was recorded by the CMOS image sensor. Later, these intensity distributions would be converted to a pitch value and placed on a virtual
2D grid at their respective points of acquisition, generating a pitch map. Maps of obtained at two different package temperatures were then subtracted from each other to find the value of Δd at each point.
Away from material interfaces, strain values measured in subregions containing silicon and copper were in good agreement with theoretical calculations. However, the measured CTE values were negative, suggesting that the regions undergo compression rather than expansion during thermal loading. A surface angle map of the grating region was also computed by assuming that any deviations in diffraction angle are due to non- zero surface angles at 25ºC. From this map, it was found that surface angle varies greatly across material interface regions, eliciting spurious pitch values. In homogenous regions, these deviations are much smaller.
92 CHAPTER 4
EMERGING TECHNOLOGIES FOR VEHICLE PERCEPTION
4.1 Introduction
Within the past few years, a confluence of advances in computer performance, algorithms, sensors, mapping, and data storage have enabled the development of prototype autonomous roadway vehicles (AVs). Ultimately, the goal of such pilot projects is the production of vehicles capable of driving on public roadways in all scenarios with little to no human assistance (Hussain & Zeadally, 2019). Perhaps due to the numerous economic, environmental, and social implications of such a technology
(Fagnant and Koclelman, 2015), the concept of AVs has captured interest of the automotive industry, academic researchers, entrepreneurs, and public around the world
(D. Yang et al., 2018).
In any self-driving or advanced driver assistance (ADAS) application, accurate, real-time perception of objects within a vehicle’s driving environment is of paramount importance (Gryuer et al., 2017). Today, high-resolution digital cameras, radar, LiDAR and sonar comprise the majority of sensors utilized for this task, as these technologies are principally designed to collect, measure, and image energy received from distant surfaces
(Rosique et al., 2019). Owing to the wide variety of surface characteristics found on roadway objects, each sensor type has unique advantages and limitations making system integration challenging, and a considerable opportunity. The reminder of this chapter provides a brief overview of these remote sensing technologies in the context of autonomous vehicles and serves to motivate original research described in chapter 5.
93
4.2 Cameras and Computer Vision
Perhaps unsurprisingly, high definition (HD) digital cameras play a central role in contemporary AV and ADAS sensor stacks (Ginger et al., 2013). Indeed, the devices provide detail-rich images and video of driving scenes at angular resolutions greatly exceeding state-of-the-art LiDAR or radar systems for a fraction of the cost (Rosique et al., 2019). This is partially due to the ubiquity of camera hardware, as the high- performance technology has been nearly commoditized by the global economy over the past twenty years. Owing to advances in both packaging and CMOS fabrication, automotive machine vison cameras are also robust (Kang et al., 2019), with expected service lifetimes similar to a vehicle’s mechanical components.
A. B. C.
D.
E.
Figure 39: (A.) Demonstration of image-based lane detection demonstrated by Oliveira et al. in 2016 using a convolutional neural network (CNN). (B.) Image-based object detection using the YOLO 3 neural network (Redmon et al., 2019). (C.) Image-based semantic segmentation of a driving scene using a multi-scale attention neural network (Tao et al., 2020). (D.) The market for CMOS image sensors continues to grow worldwide (IC Insights, 2019). (E.) Front and backside of a typical high-performance CMOS image sensor packaged on PCB substrate (ON Semiconductor). 94 Like camera hardware itself, the utility of high-resolution images has greatly improved over the past two decades. Today, important perception tasks such as object detection (Redmon et al., 2019), driving scene segmentation (Tao et al., 2020), and lane tracking (Narote et al., 2018) are best accomplished by applying principals of computer vision and deep learning to images and video sequences. Furthermore, research interest in camera technologies, computer vision, and deep learning remains at an all-time high, with many principals and applications waiting to be explored. This community momentum will likely lead to further advancements in image-based automotive perception, such as robust occlusion-resistant object tracking in the coming years (Wojke et al., 2017).
4.2.1 Image Sensor Requirements
From a hardware standpoint, many machine vision cameras utilize complementary metal oxide (CMOS) imaging sensors. An older semiconductor imaging technology, charge-coupled device (CCD) sensors are also available, with the choice of sensor type often decided by application (Kozacek et al., 2018). Roadway vehicles frequently encounter driving scenes with high levels of illumination variance; caused for example by bright sunlight and deep shadows. Sensors with insufficient dynamic range will be unable to discern image details contained within such regions, leading to potentially dangerous sensing errors (Siyeong Lee et al., 2018). Image quality factors such as noise, blur, and contrast must also be considered, as all three may reduce the effectiveness of computer vision algorithms upon which optical perception depends
(Dodge & Karam, 2016). These algorithms could be similarly affected by motion blur
95 (Noroozi et al., 2017) and other shutter-induced artifacts such as elongation and warping
(Kozacek et al., 2018).
With such scenarios in mind, ideal image sensors for automotive applications must provide excellent dynamic range and low image noise, while reducing motion blur and shutter artifacts. Additionally cost, power, and heat generated should also be taken into consideration. Backside illuminated CMOS image sensors, which are fabricated such that the RDL region is located beneath the photosensitive elements, meet these requirements provided use of a global shutter (Miyauchi et al., 2019). However, all visible light sensor technologies can still generate images in which automatic feature extraction and object identification is difficult. In this regard, low-light and nighttime driving scenarios are particularly notorious, producing reams of poor-quality images with numerous artifacts, even on sensor platforms with high-end hardware (Neumann et al.,
2019).
A. B.
C.
Figure 40: (A.) Practical demonstration of high dynamic range (HDR) imaging. The upper image was captured from an HDR CMOS sensor and shows greater detail in the image’s darkest and brightest regions (Velichko et al., 2017). For comparison, the same scene was also captured using a conventional CMOS sensor, as shown in the lower image. (B.) Illustration of a backside illuminated CMOS sensor’s pixel cross section (Miyauchi et al., 2020) (C.) Nighttime images exhibiting numerous artifacts including sensor noise/grain, motion blur, and poor coloration (Neumann et al., 2018). 96 4.2.2 Depth Estimation
While some remote sensing technologies such as LiDAR and automotive radar can measure the size and distance of surrounding objects directly, these same properties are ambiguous from image data alone. This is because the angular view field of a camera lens remains constant, regardless of how far away an imaged object may be. It follows that the apparent size of an object in an image is simply a function of angular view field occupancy rather than distance from camera lens. In other words, objects which occupy a greater portion of the camera’s angular view field appear larger, while objects which occupy a lesser portion of the angular view field appear smaller, irrespective of actual distance or size. However, it should be noted that the further an object of fixed size is from a camera’s lens, the smaller it appears. This is because the size ratio between the object and physical dimensions of an imaged scene decreases with distance (Joglekar et al., 2011).
A. B.
Figure 41: (A.) A longer focal length (mm) lens reduces a camera’s view field angle but increases the perceived size of distant objects, which now occupy a larger portion of the camera’s small view field (Nikon, 2020). (B.) For a camera with a fixed view field, an object of fixed size appears smaller with increasing distance, as the object occupies less of the camera’s total view field angle (Nikon, 2020). 97 Despite the fundamental ambiguities associated with object size and distance in imaging systems, the cost, performance, angular resolution, and general utility of digital cameras makes the idea of per-pixel depth estimation difficult to ignore. Indeed, development of such methods remains an active area of research within the fields of computer vision, optics, and robotics. Initial attempts to estimate depth from digital images relied on unique solutions for image sets containing specific features such as objects of known size, shading, textures, or vanishing points (Saxena et al., 2008).
Additionally, techniques using alternate image representations such as Fourier transforms
(Torralba & Pami, 2002) or data from sequential images such as optical flow (Basha et al., 2012) (Y. Wang et al., 2019), structure-from-motion (Ranftl et al., 2016) and depth of defocus (Zhuo & Sim, 2011) (El Helou et al., 2019) have also been explored.
Stereovision Today, stereovision is one of the most utilized range finding techniques involving automotive camera hardware. Primarily, this is because depth estimation from a pair of stereo images is a classical problem in computer vision, and among the best studied over the past twenty years (K. Zhou et al., 2020). Automotive stereovision requires two images of a driving scene to be acquired concurrently from identical cameras separated by a fixed distance in one dimension. Upon acquisition, similar configurations of pixels common in the left and right images are found algorithmically along their corresponding epipolar line. Such configuration pairs are assumed as belonging to the same point in real space. Next, the difference in location between each pair along their corresponding epipolar line is calculated in units of pixels, also called the disparity (Hamzah & Ibrahim, 2016). Finally, spatial depth of the real- world point is calculated using the properties of similar triangles.
98 While the described depth estimation framework is conceptually simple, practical implementations of stereo matching are surprisingly complex. Algorithms are typically composed of four sequential blocks, sometimes referred to as the stereovision taxonomy
(Hamzah & Ibrahim, 2016). The first two blocks, which contain matching-cost computations and cost aggregation are responsible for both discovering configurations of pixels with a high degree of similarity in both frames and computing several possible disparity values. A number of matching-cost methods have been proposed and evaluated for use in the first block, however no consensus has been reached regarding their overall performance in automotive perception applications. In contrast, cost aggregation methods based on the adaptive support weights (ASW) are a likely candidate for self-driving and
ADAS owing to their overall efficiency and ability to preserve object boundaries in the output disparity map.
Input A. B. Images Step 1 Cost Matching
Step 2 Cost Aggregation
Step 3 Disparity Selection
Step 4 Disparity Refinement
Disparity Map
Figure 42: (A.) Location of a distant target point (P) in a pair of left (pl) and right (pr) stereovision images. Both points fall along the same Epipolar line (Hamzah & Ibrahim, 2016). (B.) Overview of the programmatic stereovision taxonomy used to construct per-pixel disparity maps from pairs of images.
99 A disparity map created from each pair of matched pixels is calculated in the third block of the stereovision algorithm (Hamzah & Ibrahim, 2016). Here, a final disparity value for each pair of matched pixels is chosen from a set of values generated by the previous cost aggregation step. Typically, the disparity value associated with lowest aggregate cost is used. There are two ways in which the minimum aggregate cost can be evaluated, local and global. In the local method, aggregate cost is evaluated based on a small window of adjacent pixels around a matched pixel. However, this often produces disparity map errors, as only a small number of pixels are used to assign an appropriate disparity value. In contrast, the global method makes additional assumptions about the imaged scene. Commonly, objects are assumed to be smooth except at their boundaries, implying that neighboring pixels on a given object should have very similar disparities.
The objective is then to find an optimal disparity assignment function. Although this process is computationally intensive, fewer local errors in the disparity map will be made using this method.
In the fourth and final block, the previously generated disparity map undergoes postprocessing steps which reduce noise and make improvements to overall quality
(Hamzah & Ibrahim, 2016). Occluded regions are filled with disparity values equal to those of the background or textureless areas, while interpolation schemes similar to those described in 3.5.4 are used to fill regions with no assigned disparity values. In a final step, the entire disparity map is smoothed by applying a gaussian and/or median filter to reduce noise. Other filtering techniques may also be applied at this time. If required, the completed disparity map can be transformed into a depth map using the geometric theorem of similar triangles.
100 For many situations, stereovision provides an accurate-high resolution depth map of the imaged scene. However, use of an algorithmic method raises the potential for measurement errors. In practice, errors generated in the first and second block of the stereovision taxonomy are most common (Hamzah & Ibrahim, 2016). This is because the same real-world point represented by a given configuration of pixels in one image may be dissimilar in the other, resulting in a failed match. Such failures can be caused by image noise, a difference in exposure levels between the left and right camera, illumination changes, and by highly reflective or transparent surfaces like mirrors or windshields.
Matching errors aside, additional issues can occur in algorithms which make use of global disparity selection, as the assumption that neighboring pixels on an object surface will have similar disparity is violated for extended objects such as buildings, large trucks, or busses. In addition to algorithmic sources, stereovision is somewhat poor at estimating the depth of objects at long range (Pinggera et al., 2014.). This is because each pixel represents a quantized length which increases with scene depth. Indeed, due to the geometry of camera view fields, distance dependent measurement errors increase quadratically for stereovision systems.
Recently, new developments in convolutional neural networks have spurred considerable interest in stereovision as deep learning problem. Here, one or more components of the traditional algorithmic approach may be substituted with a variety of neural network architectures, or a new approach based entirely on machine learning principals employed (K. Zhou et al., 2020). Often, disparity maps of driving scenes generated in this manner show significant improvements compared to the traditional pipeline, furthering interest in learning-based stereovision (Kendall et. al, 2017).
101 A. B.
Figure 43: (A.) Disparity maps of a driving scene (top) generated by Smolyanskiy et al. in 2018 using a neural network which incorporates both a classical stereovision algorithm and object detection/segmentation. Upper disparity map shows the results of this network when trained on LiDAR ground truth data from the KITTI 2015 dataset. In the middle image, the network was trained using output of the incorporated stereovision algorithm exclusively. In the lower image, both LiDAR and stereovision were used during network training. (B.) Image of a driving scene (top), disparity map generated by Sehyung Lee et al. using an adaptive random walk stereovision algorithm in 2015 (middle), and LiDAR ground truth data from the KITTI 2012 dataset. However, at the time of this report, all top-performing methods require a high degree of computing power while generating only a few disparity maps per second, too slow for practical use in autonomous driving or ADAS applications (K. Zhou et al., 2020).
Training neural networks also requires vast quantities of ground truth depth data from driving scenes. Typically, ground truths are acquired using LiDAR, but only few datasets of this type exist due to equipment costs (G. Yang et al., 2019). Nevertheless, it is believed that learning-based stereovision represents the future of camera-based depth estimation, as performance is improved, and more datasets become available.
Monocular Depth Estimation While learning-based stereovision pipelines continue to attract the attention of researchers and industry alike, depth estimation from a monocular camera via deep-learning is perhaps the more intriguing long-term problem 102 (Eigen et al., 2014). Object depth is indeed ambiguous from a single image, but the requirement of only a single camera has considerable benefits, as a successful monocular depth estimation (MDE) method can greatly simplify sensor configurations, alignment procedures, and the overall data pipeline. Furthermore, computationally expensive and error-prone epipolar pixel matching of right and left images is avoided with a single camera, which may reduce necessary computing resources. This is especially important in automotive applications due a limited power and/or thermal budget.
Within the past few years, a number of MDE methods have been proposed and evaluated using automotive datasets such as KITTI and Cityscapes. Published literature contains wide variety of neural network architectures, loss functions, and processing pipelines, as depth reconstruction from single image is an ill-posed problem with many potential solutions to explore (Kahn et al., 2020). Currently, several well-known learning frameworks such as long short-term memory, variational autoencoders, and generative adversarial networks have been shown capable of depth estimation tasks, but many others
Figure 44: Monocular depth estimation of a driving scene using the state-of-the-art Monodepth 2 neural network (Godard et al., 2020). 103 lack sufficient evaluation in the context of MDE. Intriguingly, MDE has also been attempted using a handful of unsupervised learning frameworks (Godard et al., 2019).
These do not rely on ground truth data during training, but current implementations exhibit poor depth accuracies compared to supervised, and semi-supervised network types. It also be noted that little research exists which examines the underlying mechanisms leading to successful learning-based MDE, such as what the network has learned, or which depth cues are used during estimation. Indeed, numerous challenges remain in the study of MDE, but new discoveries and innovations may also bring digital cameras a step closer to general propose technology.
4.3 Light Detection and Ranging (LiDAR)
4.3.1 Historical Perspective
LiDAR, an acronym for light detection and ranging, is a remote sensing technique which measures object distance from a light source. Distance is determined by analyzing the change in phase or flight time of the source-emitted light when it returns to a detector, after reflecting off an object. Historically, LiDAR technologies have followed the invention of the laser by T.H. Maiman in 1960. LiDAR benefits greatly from a laser’s ability to produce a narrow beam of high-intensity, monochromatic, collimated light, although none of these properties is strictly necessary for the construction of a LiDAR system. Among the earliest uses of LiDAR was the determination of cloud layer altitudes from ground-based measuring stations (Clemesha et al., 1966). Indeed, over the past several decades, LiDAR has found considerable use in the earth science disciplines such as atmospheric science (Ansmann et al., 1990), geology (Glennle et al., 2013), forestry
(Andersen et al., 2004), and oceanography (Popescu et al., 2004).
104 Figure 45: Using airborne LiDAR for high resolution mapping of geologic features, such as a section of the Paso Superior fault (Glennle et al., 2013). The first widespread use of LiDAR for an automotive application was the adoption of handheld velocity sensors by police precincts for traffic speed enforcement in the early 2000’s. While traffic police routinely used handheld radar-based velocity sensors for several decades, the small beam divergence of LiDAR-based light sources enabled unambiguous velocity measurement of individual vehicles on streets with high traffic flow and congestion (Fisher & Pyhtila, 2000).
In the mid 2000’s, a series of self-driving competitions sponsored by the Defense
Advanced Research Projects Agency (DARPA) required full-size vehicles to drive an off-road desert course without the aid of roadway markings. While no vehicle completed the course during the competition’s 2004 inaugural year, five teams were successful in
2005 (Van Brummelen et al., 2018). Thereafter, interest in automotive perception technologies among research scientists, vehicle manufactures, entrepreneurs, and robotics enthusiasts has grown considerably. Indeed, since 2005 automotive LiDAR sensors have become increasingly sophisticated. Early units often made use of a single fixed laser- detector pair, swept around in a circle by an encoded motor which recorded the azimuthal
105 A. B.
C.
Figure 46: (A.) Self-driving vehicles from various teams participating in the 2nd DARPA Grand Challenge in 2005. (B.) Image of Stanley, a Volkswagen Touareg R5 with a custom roof rack housing five horizontal-plane industrial LiDAR scanners and winner of the 2005 DARPA grand challenge (Thurn, et al., 2006). (C.) Industrial singe-plane LiDAR unit, circa 2005. angle of each distance measurement. While such a scheme can obtain many distance measurements per angle of rotation, it can only do so for a single horizontal plane.
Because most objects around a vehicle cannot be represented accurately using a single horizontal “slice” such LiDAR sensors had limited use for autonomous vehicle applications. In 2007, LiDAR manufacturer Velodyne pioneered it first high-performance automotive LiDAR system designed for perception and street mapping. The while bulky, the system utilized a vertical stack of laser/detector pairs at fixed vertical angles, placed on a rotating turret to simultaneously measure distances from 64 planar slices with a 360º azimuthal view field. This multi-laser, fixed-angle configuration greatly improved the resolution of automotive LiDAR and is used in many modules today.
4.3.2 Components of Automotive LiDAR
Regardless of exact measurement technique, all automotive LiDAR uses the same basic optical configuration. Light from a collimated laser source is projected towards an 106 D
Optics
Emitter and Detector Objects
Figure 47: Basic configuration of a LiDAR system for object rangefinding object some distance away from the laser emitter position. Light from the laser source travels to the object and is reflected in multiple directions. A light detector located in the same plane as the laser source receives a fraction of source light that was reflected from the object. The light incident on the detector is converted to an electrical signal, which undergoes further processing to determine the object distance and/or velocity. It should be noted that in automotive LiDAR applications, time-of-flight (TOF) measurements are favored over phase-shift detection. The InGaAs diode lasers commonly used as emitters have short coherence lengths, making phase change measurements difficult at long ranges
(Rasshofer et al., 2011).
Laser Sources Contemporary automotive LiDAR uses one or more semiconductor laser sources operating between 850nm and 1100nm. This wavelength range covers a portion of the near-infrared (NIR) optical spectrum, which is defined as electromagnetic radiation between 780nm-2526nm (Reich, 2003). NIR light is invisible to the human eye, but wavelengths between 780nm and 1100nm can be easily detected by silicon and InGaAS photodiodes. Use of invisible laser sources is especially important in automotive applications, as any moving visible light source could potentially distract 107 vehicle operators. Despite invisibility, light in the 850nm-1100nm wavelength range is still focused onto the retina of the human eye. Because of this, the total optical power produced by an automotive LiDAR laser source needs to be limited, such that eye damage does not result from exposure. Practically, this means that all automotive LiDAR sources must fulfill the safety requirements for a Class 1 laser emitter using test methods defined by the ANSI Z136 and/or IEC 60825 standards.
Laser diodes operating in the 850-1100nm wavelength range use InGaAs for photon generation (J. Major et al., 1991). This solid material has a tunable direct bandgap which decreases in energy based on the percent of indium which has replaced gallium in the crystal lattice, following Vegard’s law (Nahory et al., 1978). For generating photons with wavelengths between 900nm and 1000nm, indium atoms need to replace between 10-20 percent of the gallium in the lattice structure. Contemporary InGaAs laser diodes are fabricated commercially using a combination of Molecular Beam Epitaxy to grow InGaAs thin films on InP or GaAs substrate and use standard microfabrication processes such as chemical vapor deposition and photoetching (Qiu et al., 2019).
A. B.
Figure 48: (A.) Chart depicting the maximum continuous optical power allowed for each laser class over the UVA, Visible, and NIR wavelength range. (B.) Layered material structure of an InGaAs quantum well laser diode (H. Yang et al., 2009). 108 Detectors NIR optical detectors perform the essential function of converting a fraction of source-generated photons returning from remote objects into meaningful electrical signals. For automotive applications, object distances are typically between 0.5 meters and 200 meters and have a wide variety of reflectivity characteristics (Kashani et al., 2015). Indeed, one of the primary challenges in detector design is the tendency of illuminated objects to exhibit diffuse reflection, in which reflected light rays are projected into hemispherical radiation pattern with the reflective object located the at the focus of the hemisphere (Tan & X. Cheng, 2016). Such a pattern intrinsically grows larger in area as the distance from the reflector increases, while the magnitude of reflected optical power remains constant. As a result, the amount of optical power per unit of area, also called intensity, decreases the further away a detector is placed from the object. In practice, the intensity of light received by a typical detector is small compared to the intensity of the source. Received power falls by a factor of 1/D4, necessitating the use of high sensitivity detection elements (Tan & X. Cheng, 2016).
Silicon-based avalanche photodiodes (APDs) are the detector element of choice in contemporary LiDAR systems. APDs can provide extremely high photocurrent gain and with low noise, essential for detecting low-intensity source light reflected from distant objects (Laforce, 2009). APDs function using a unique method of photoelectron multiplication, such that a single absorbed photon can generate many photoelectrons.
An APD structure consists of two parts (Laforce, 2009), an absorption region and a gain region. Typically, the absorption region is located above the gain region and consists of two layers, a thin, heavy p-doped contact layer followed by a thick layer of lightly P- doped high-quality single crystal silicon. Below the absorption region lies the gain region,
109 which consists of a p-doped silicon top layer and a n-doped lower layer connected a metal bottom contact. A bias voltage is applied across the entirety of the device, with the top contact held at negative voltage, while the bottom contact is placed at positive voltage.
Incident photons passing through the thin upper contact and are absorbed in the thick layer of single-crystal silicon, generating an electron-hole pair. The positively charged hole drifts towards the upper contact, while the negatively charged electron drifts towards the gain region. Electrons entering the gain region are accelerated due to a strong electric field, obtaining significant kinetic energy. Next, accelerating photoelectrons collide with electrons in the valence band of the gain region, imparting enough energy to excite these electrons to the conduction band. This process, called impact ionization, allows one photoelectron in the conduction band to generate many additional conduction band electrons, resulting in significant photocurrent gain when electrons reach the lower metal contact (Laforce, 2009). Silicon APDs are a readily available commercial product
A. B. A. B. Light in E-field P+
- A
e bsorption
h+
P Gain N
Figure 49: (A.) Silicon-based avalanche photodiodes (APD) principal of operation (Laforce, 2009). Light enters though the thin, heavily P-doped top layer and is absorbed by thick layer of lightly p-doped single crystal silicon, generating electron-hole pairs. Electrons drift towards the bottom of the structure due to a positive voltage applied to the metal contact at the bottom of the structure. Electrons passing through the PN sandwich layer experience a strong electric field, accelerating them to high enough energies to cause impact ionization events, releasing more electrons which are collected by the bottom metal contact. (B.) Photodetection response of a typical NIR ADP (Hamamatsu Photonics, 2017.) 110 fabricated using CMOS processes. Many device variants exist, but most provide a combination of high photocurrent gain, low noise, and response-recovery times on the order of .07ns and 30ns respectively.
Filters and Lenses While InGaAs laser sources and silicon APD detectors provide the basic components of automotive LIDAR, lenses and optical wavelength filters are essential for a functioning system. Lenses provide laser diode beam shaping and ADP light collection, while filters reject ambient light which does not originate from the laser diode source. The ideal laser emission pattern is that of narrow beam with a small divergence cone, circular cross-section, and diameter of 1-100mm. However, all InGaAs laser diodes have large emission divergence and highly elliptical illumination cross sections. To correct this, one or more aspherical lens elements are placed in front of the laser diode which reshape the emission profile. Low-loss aspheric lenses for NIR wavelengths are available as commercial products, while design of single or multi-element lens systems can be done on computer software.
The detection of low intensity source light reflections can be enhanced through use of a collection lens, placed in front of the APD such that the focal point of the lens is aligned with the APD’s sensor window. Photons are collected over the entirety of the lens cross section, and directed to the ADP’s small active area, providing optical intensity gain with no additional electronic noise.
Optical filter elements provide ambient light rejection for silicon APDs, which generate photocurrent over a wide range of NIR and visible light wavelengths. Indeed, unfiltered ADP detectors often reach their maximum photocurrent generation rate, even if no light from a LiDAR-specific emission source is present. This is because photons from
111 A. B.
C.
Figure 50: (A.) Optical transmission of NIR light though various thin film bandpass filters containing up to five cavity layers (Johansen et al., 2017). (B.) Divergent light emission cone of a laser diode collimated using an aspheric lens. (C.) Incoming light directed though a thin film bandpass filter and focused onto the surface of an avalanche photodiode (APD) for passive optical gain. other light sources such as the sun or car headlamps are absorbed by the ADP. NIR bandpass filters, which transmit light over a given range of wavelengths and absorb or reflect light outside of the transmission range, are used reject ambient light, while passing photons emitted by the laser source to the ADP (Johansen et al., 2017). Bandpass filters are manufactured by depositing thin layers of dielectric material with different indices of refraction on top of glass or polymer substrate. By changing the layering order and material thickness, properties of thin film interference can be used to create a layer stack that only allows light within a narrow wavelength range to pass through the layers. Such filters are available commercially (Johansen et al., 2017).
Time-of-Flight Measurement Automotive LiDAR makes extensive use of time- of-flight measurements for distance determination. In a typical measurement (Piatek & J.
Lee, 2017), a light pulse with duration on the order of 4ns is emitted from the laser source and directed towards a distant object at a given time, t0. Photons then travel from the light 112 D
Light Source Optics θ
Detector/ Object APD Filter Light Collection Optics
Figure 51: Depiction of a LiDAR system’s emitter-detector pair geometry and supporting optics source towards the object located at distance D at the speed of light which is approximately
3.3m/ns. Some photons reflect off the object at distance D and return to the detector, which is in the same plane as the laser emitter but at a different position, Pd. The time at which the photon pulse returns to the detector is then recorded at t1. It should be noted that the distance between the object and detector is slightly larger than the actual value of D, as the returning photons must travel at an angle of θ with respect to the source’s optical axis to reach the detector. However, in many LiDAR configurations the in-plane offset between the source and detector elements is many times smaller than the value of D, resulting in small values of θ.
In such instances, the distance of the object D can be determined using
푐 ∗ (푡 − 푡 ) 퐷 = 1 0 . (4.1) 2
If necessary, correction terms can be applied to this expression to account for LiDAR geometries in which θ is not negligibly small (Piatek & J. Lee, 2017).
LiDAR Products From 2007 to 2017, LiDAR manufacturer Velodyne remained the only company incorporating high-performance modules intended for automotive
113 perception into its standard product portfolio. While Velodyne introduced several new variants during this ten-year period, all modules were similar in both design and operation to the company’s first product. Velodyne LiDAR systems are built around a series of discrete emitter-detector pairs soldered to a printed circuit board (PCB) using surface mount packages. Here, each pair emits and receives light at a fixed vertical angle.
Depending on the model, a vertical angle range of approximately 40° is covered by 16,
32,64, or 128 emitter-detector pairs, referred to as channels. The PCB is mounted into an
A. B.
C.
D.
Figure 52: Illustration of spinning-head, fixed-angle, multi-beam LiDAR geometry commonly used for automotive perception. (B.) Visualization of point cloud data captured using high-performance 128-channel LiDAR units manufactured by Velodyne (upper image) and Ouster (lower image). (C.) Promotional images of high-performance automotive LiDAR units from Velodyne and Ouster. (D.) Forward-facing solid-state LiDAR unit manufactured by Luminar, integrated into the body of a Volvo prototype vehicle. 114 automotive-grade housing and attached to an angle-encoded 360° motor allowing the vertical beam stack to be swept in complete circle. Today, this design paradigm is referred to as spinning-head LiDAR.
Since 2017, a number of new LiDAR manufacturers have made their way into the automotive perception market, due to a recent surge in autonomous vehicle and ADAS research initiatives by leading automotive manufacturers. The company Ouster, which makes its own spinning-head LiDAR systems, is poised to become Velodyne’s main competitor in North America while the number of new market participants worldwide, both large and small, continues to rise. Cost reduction is of interest to emerging manufacturers, as the monetary value of a single 360° LiDAR module is roughly equal to the price of a consumer automobile. In this regard, LiDAR design paradigms which do not use an angle-encoded 360° motor or large quantities of individually packaged emitter- detector pairs are needed (Warren et al., 2018). Replacement of narrow-line laser emitters with less-expensive light emitting diodes (LEDs) has also been proposed (Griffiths et al.,
2019).
4.4 Millimeter Wave Radar
Radar, an acronym for radio detection and ranging, uses the radio band of the electromagnetic spectrum rather than optical NIR radiation to detect the position and velocity of distant objects. Like LiDAR, radar makes use of an emitter/detector configuration, however both the emitter and detector are radio antennas. Contemporary high-resolution automotive radar operates in the 77GHz or 79GHz frequency band, which affords good angular resolution for a given sensor size (Hasch et al., 2012).
115 For radar, object detection and range finding are carried out using either pulse- doppler measurements or frequency modulated continuous wave (FMCW) measurements
(Kissinger, 2012). Pulse-doppler radar functions using the same principle as time-of-flight
LiDAR. A radar pulse is emitted at t0, reflects off an object, and returns to the detector at
A. B.
C. D.
Figure 53: (A.) Recorded signal strength of LiDAR returns for various reflection targets plotted against measured distance, as reported by Kutila et al. in 2018. Signal data collected using a Velodyne 16-channel LiDAR module deployed in clear weather (upper plot) and dense fog (lower plot), shows significant attenuation. (B.) Top-down view of a LiDAR point cloud acquired using an Ouster 64-channel LiDAR operating on the roof of a vehicle during turbulent snowfall, as reported by Jokela et al. in 2019. This point cloud exhibits many spurious LiDAR returns, large blind spots caused by snow accumulation on the sensor window, and attenuation. (C.) Radar signal (lower right) of a driving scene with corresponding camera image (upper) and LiDAR data (lower left) as reported by B. Major et al. in 2019. Here, the radar signal is visualized in polar coordinates with the vertical axis being range and the horizontal axis set to azimuthal angle, aptly demonstrating the limited angular resolution of 77Ghz FMCW radar. (D.) Radar returns from a 77Ghz radar module pointed towards the backside of a commuter car at a 30º angle, as reported by Blöecher et al. in 2012. White outline is ground truth position of the vehicle. 116 t1. Object distance is then calculated using equation 4.1 from the previous section. In contrast, FMCW radar forgoes the use of discrete radio energy pulses. Instead, the emitter is operated continually with a modulated frequency. Radio energy returned to the detector is analyzed to determine propagation delay and doppler shift through the use of a fast
Fourier transform (FFT) algorithm, from which the object distance and velocity are extracted (Hakobyan & B. Yang, 2019).
Radar offers several advantages over LiDAR. First, both range and spatial resolution of radar modules are minimally degraded by the presence of heavy rain, snow, or fog (Bertoldo et al., 2017) (Kutila et al., 2018). LiDAR fairs worse in these conditions, as light in the NIR wavelength range can be absorbed or scattered by small water droplets
(Kutila et al., 2018) and snow crystals (Jokela et al., 2019). A second advantage is monetary cost, as currently available systems are an order of magnitude less expensive than high- performance LiDAR modules. Unlike rotating LiDAR, automotive radar utilizes monolithic solid-state chip designs and wafer level packaging containing no moving parts
(Jia et al., 2016). This serves to reduce fabrication costs while improving overall system reliability. The primary disadvantage of radar is its limited ability to discern the exact shape and geometry of an object or set of objects in proximity (Blöecher et al., 2012) (Roos et al., 2016) (B. Major et al., 2019). For example, a pedestrian walking onto the street from behind a parked car.
4.5 Sonar
Unlike LiDAR and radar, sonar rangefinders utilize sound waves rather than electromagnetic radiation to determine the distance of objects from the emitter plane.
These sensors feature an emitter/detector configuration of ultrasonic transducers to
117 measure object distance using time-of-flight techniques. There are two primary types of
ultrasound transducers, piezoelectric and capacitive.
Piezoelectric transducers (Akasheh et al., 2004) are constructed using a thin
layer of piezoelectric material such as lead zirconate titanate (PZT) with metal contact
layers deposited on the top and bottom of the of the active material. If a voltage is placed
between the two metal contacts, the PZT will expand or contract, depending on the
direction of the voltage-induced electric field. This mechanical change in the active layer
can be used to generate air pressure waves, which travel away from the transducer at the
speed of sound.
A. B.
C. D.
Figure 54: (A.) capacitive ultrasonic transducer structure (Wygant et al., 2009). (B.) two types of piezoelectric ultrasonic transducers (Akasheh et al., 2004). (C.) Detection pattern and range of an automotive ultrasonic rangefinder (MaxBotix Inc, 2015). (D.) Sound intensity lobe from a typical ultrasonic transducer (Airmar corp.). 118 Capacitive transducers (Wygant et al., 2009) generate air pressure waves by suspending a layer of conductive material, such as metal or doped silicon over a cavity.
At the bottom of the cavity lies an electrode. The upper conducting layer and lower electrode are placed at different voltages, generating an electrostatic force between the two. Due to this force, the upper conducting layer bends downwards. If the voltage is removed, the upper layer returns to its unstrained position due to the restoring force provided by the crystal lattice of the layer. Applying an AC voltage difference to the top layer and lower electrode causes the top layer to vibrate, producing air pressure waves.
There are two reasons for using ultrasonic frequencies for automotive sonar. First, ultrasonic frequencies above 25kHz are inaudible to the human ear. This property is an important to automotive safety, as it prevents pedestrians and cyclists from distraction.
Secondly, the acoustic energy emission profile is easy to direct, shape, and concentrate using simple mechanical parts, such as direction plates, cones, and horns.
The primary advantages of sonar are low sensor cost when compared to
LiDAR/radar and small total module size. However, sound waves at ultrasonic frequencies attenuate rapidly in air, restricting detection ranges to less than 10 meters for most products (Cui et al., 2016). Additionally, sonar-based sensors systems are incapable of discerning detailed object shapes or geometry, limiting their use in many-object environments such as a traffic circle or intersections. Despite these issues, Ultrasonic sensors find use in parking assist systems, available on some consumer and fleet vehicles
(Jamil, 2009).
119 4.6 Sensor Fusion
During operation, both autonomous vehicles and ADAS will encounter a diverse set of driving conditions and scenarios, many of which cannot be fully understood using data from a single sensor. Rather, autonomy requires synthesizing data from multiple sensor types to construct a persistent model of the surrounding environment and the vehicle’s place within it. Use of multi-modal data in this manner is commonly referred to as sensor fusion (Kocić et al., 2018).
There are two broad categories of sensors used in autonomous driving, those intended for localization and those intended for perception (Jahromi et al., 2019).
Localization sensors measure position and orientation of a vehicle, while perception sensors measure the surrounding environment. Sensors used for localization include global positioning system (GPS) receivers, inertial measurement units (IMUs) and vehicle odometry sensors such a speedometer and mileage counter. GPS receivers locate the vehicle on Earth’s surface, IMUs determine spatial orientation, and odometry sensors measure speed and/or distance driven. Perception sensors include digital cameras,
LiDAR, radar, and sonar, as detailed in sections 3.2-3.5.
The driving task, whether manned or autonomous, requires understanding an object’s orientation and spatial position within a scene in each instant of time. Indeed, every sensor relevant to perception and localization supplies spatial information regarding the vehicle and/or its environment (Yurtsever et al., 2020). Classically, data from such inputs can be represented in a two-dimensional or three-dimensional virtual space with the vehicle centered near the origin of a coordinate system. Often, the virtual space is discretized into small 2D or 3D subregions sometimes referred to as an
120 occupancy grid, with exact spatial positions obtained from a given sensor module placed at the nearest subregion’s coordinates (Kocić et al., 2018). This is done to reduce computational complexity when the position data is required for further operations.
While all localization and perception sensors can supply spatial points, many provide additional data about the vehicle and its surroundings. IMUs for example provide an orientation vector, cameras track lanes and identify objects, while radar can measure relative velocities (Feng et al., 2020). This additional data requires specialized software
A. B.
C.
Figure 55: (A.) Autonomous vehicle sensor data visualized using a 3D occupancy grid (Baidu Apollo, 2017). (B.) Sensing modalities with overlapping view fields allow for many fusion schemes (Michaelis et al., 2017). (C.) High- level overview of an autonomous vehicle’s hardware/software architecture (Jahromi et al., 2019). 121 frameworks which either extend the functionality of a classic occupancy grid or utilize a new data representation altogether (Floudas, 2007). A vast number of advanced sensor fusion frameworks have been reported in academic literature, and while no formal classification system has been settled on (Castanedo, 2013), some rudimentary categorization is possible using generalized criteria. Broadly speaking, sensor fusion frameworks may be classified into three types, methods based on Gaussian filtering such as Kalman or particle filters, probabilistic inference methods, and deep learning-based methods (Rosique et al., 2019).
Within the past few years, Gaussian filtering has demonstrated the utility of fusing monocular camera images with point cloud data generated from LiDAR or radar to identify and track roadway objects in 3D space. While both Kalman and particle filters are suited to this task due to their ability to represent disparate data types in a single state and make short-range future predictions based on current-state information (García, et al.,
2017) their value is perhaps best appreciated in the context of localization. Indeed, centimeter-level positional accuracy has been reported for practical gaussian filter frameworks which fuse LiDAR, GPS, and IMU data (Wan et al., 2018).
Fusion methods based on probabilistic inference can be viewed as the direct extension of a basic occupancy grid (Meyer et al., 2019). Here, subpopulations of spatial data are used to infer the properties of the entire population. Generally, each subpopulation is supplied by one sensor type, while global properties of the overall population are inferred using a gaussian mixture model. In addition to estimating an object’s most likely location within a scene, this method can also drastically reduce the overall level of sensor noise, as evidenced by real-time fusion of stereovision and LiDAR
122 data (Maddern & Newman 2016). Furthermore, if sufficient population densities exist over multiple data frames, statistical prediction of future frames is possible using a
Gaussian mixture probability hypothesis density filter (Vo & Ma, 2006) (Michaelis et al.,
2017). Indeed, both object tracking and trajectory prediction have been demonstrated by fusing concurrent radar and camera data in this manner (X. Wang et al., 2016).
Deep learning represents a somewhat novel approach to sensor fusion, as neural networks are primarily designed to accomplish higher-level tasks by implicitly learning relationships and patterns present in low-level or processed sensor data. There exists a large variety of neural networks which can make use of data from multiple sensor systems, typically classified by location of the fusion step within the architecture (Feng et al., 2020).
In networks designed for early fusion, raw or pre-processed sensor data from multiple sensor inputs is passed into the network directly for fusion with minimal alterations. Here, the network must be designed to accept the data representation provided by each sensing module, even if the formats are vastly different. The primary advantage of early fusion is that the network learns joint features from the input data at an early stage. Additionally, this type of fusion requires less computation and memory than other methods. The main disadvantage of early fusion is its inflexibility, once the network is trained on one hardware configuration, it must be completely retrained if components are to be added or changed. Additionally, early fusion networks are somewhat sensitive to sensor calibration errors.
In late fusion, raw sensor data is operated on by sensor-specific neural networks first. Next, the output of each individual network is combined using a fusion network.
123 This fusion scheme is more flexible than early fusion, as only the network responsible for a particular sensor type will require retaining if the sensing hardware is updated or changed. However, the computational penalty for late fusion is significant, as the scheme requires more networks overall. It should also be noted that fusion networks used in this scheme only observe abstracted and/or processed data provided by the front-end networks. Because of this, intermediate feature information available in each sensor- specific network, some of which may benefit the fusion task, is lost.
Middle fusion represents a compromise between late and early fusion schemes, combining feature representations found in the intermediate layers of each sensor-specific network. This allows the fusion network to learn relationships between features present in each sensor-specific network which would be lost using late fusion. While this scheme is inherently flexible, finding an optimal method which can fuse intermediate layers of a given architecture remains difficult.
Figure 56: Various sensor fusion schemes for neural networks, classified by location of the fusion step within the network and/or programming architecture (Feng et al., 2020). 124 While sensor fusion based on deep learning frameworks shows great promise, many challenges involving training data preparation, fusion methodologies, and evaluation metrics remain unaddressed. In many instances, fusion architectures presently studied involve only two types of data, chiefly camera and LiDAR, as the addition of more unique data sources greatly increases overall system’s complexity (Feng et al.,
2020). However, it should be noted that fusion of radar, camera, and/or LiDAR data using neural networks, while less prevalent, is still a topic of active research within the machine learning community.
4.7 Cost
Financial cost of a given automotive perception stack depends primarily on chosen sensing modalities. Indeed, it may be claimed that each technology effectively occupies a unique per-unit cost tier, unshared or minimally shared with other perception technologies. Sonar sensors occupy the lowest cost tier, with prices less than 30.00 USD for automotive-grade units. While such sensors have detection ranges of only a few meters and very limited angular resolution, their ability to accurately measure distance coupled with a low per-unit price has prompted use in parking assist systems. Low- performance CMOS image sensors with integrated optics may also be available in this price tier, however most imaging sensors suitable for automotive use have higher per-unit costs.
CMOS cameras with various capabilities comprise the second price tier, which ranges between 100.00 and 2,000.00 USD. Low-cost modules are typically utilized for backup or blind-spot assistance, while lane departure warning systems or other ADAS technologies are the purview of mid-range units. High-end units may utilize stereovision
125 and/or incorporate advanced machine vision features. Notably, CMOS cameras have the highest angular resolution of any contemporary automotive perception technology.
Radar modules operating in the 77-79 GHz frequency range occupy the third cost tier, which spans from 2,000.00 USD to approximately 3,600.00 USD. Such sensors have less angular resolution than both CMOS cameras and LiDAR modules, which comprise the second and fourth cost tiers respectively. Nevertheless, interest in millimeter wave technologies remains at an all-time high, as the sensing modality offers several distinct advantages. Radar sensors can have exceptionally long range, on the order of 250 meters for some units and are minimally affected by poor weather or ambient lighting conditions, unlike cameras and LiDAR. Moreover, relative velocity of detected objects may be directly measured using radar.
High-performance, multi-beam, spinning-head LiDAR units comprise the fourth and highest-cost tier, which ranges between 3500.00USD to greater than 100K USD.
Here, cost is primarily a function of vertical resolution, which is dictated by the number of fixed angle laser emitter/detector pairs (called channels) covering a vertical angle range of approximately 40 degrees. Lower-resolution units feature 16 vertical laser channels costing between 3,500.00 and 8,000.00 USD. Mid-tier units supporting 32 or 64 channels cost between 30,000.00 and 70,000.00 USD, with premium units supporting 128 vertical channels priced near 100,000.00 USD. Given such prices, use of LiDAR modules requires careful consideration, as the cost of these units may exceed the total financial value of the vehicle they are placed on.
Despite their high cost, LiDAR modules have been integrated into a number of high-profile autonomous vehicle pilot projects, as the technology’s ability to produce
126 accurate, real-time, high-resolution 360º spatial maps of driving scenes remains unparalleled. As such, cost reduction remains a primary goal in the emerging automotive
LiDAR industry, which has proposed a number of alternatives to the fixed-angle spinning head designs currently seen in the market today. Examples of these new ideas include micromirror beam steering, LED-based full scene illumination (also called flash LiDAR) and the fabrication of silicon die containing arrays of vertical cavity surface emitting lasers (VCSELs).
127 CHAPTER 5
ROADWAY OBJECT DETECTION VIA LOW-COST FLASH LiDAR AND
CAMERA
5.1 Introduction
Development of a capable and cost-effective, sensing platform for autonomous driving or ADAS applications remains a formidable engineering challenge (Van
Brummelen et al., 2018). Contemporary digital cameras produce detail-rich images of the driving environment, but overall image quality is dependent on ambient lighting conditions (Neumann et al., 2018). Additionally, cameras provide no intrinsic means of determining an external object’s size or distance from the lens. Instead, object distance is computed using stereovision (G. Yang et al., 2019) or monocular depth estimation algorithms (Godard et al., 2019) which can produce a variety estimation and/or detection errors. Light Detection and Ranging, also known as LiDAR is an emerging perception technology with a distinct set of advantages compared to cameras. LiDAR is insensitive to changes in ambient lighting and provides a direct means of measuring object size and distance (Piatek and Li, 2017). Many units are also capable of 360° object detection.
However, LiDAR lags behind cameras significantly in terms of angular resolution, frame rate, and cost. Indeed, LiDAR with performance suitable for use on a self-driving vehicle is often of greater financial value than the vehicle itself.
One proposed method of overcoming the technological and financial gap between
LiDAR and optical cameras is solid-state flash LiDAR, which replaces banks of infrared laser diodes emitting narrow beams of collimated light with less expensive LEDs which illuminate all objects in a driving scene. Areal illumination eliminates the need for a
128 rotating sensor head, farther reducing cost and increasing the achievable frame rate
(Piatek and Li, 2017).
While the integration and characteristics of high-cost, high-performance optical sensors, such as cameras and LiDAR (Park et al., 2014), on autonomous roadway vehicles (Chavez-Garcia and Aycard, 2016) has been the subject of intensive study among academic researchers and industry (Huang et al., 2018) comparatively few reports examine the use of low-cost LiDAR/Camera perception systems (Lin and Lin, 2016),
(Fremont et al., 2016). Furthermore, there appear to be no public reports detailing the use of low-cost flash LiDAR/camera sensing platforms for automotive applications. To better understand the practical merit and challenges associated with low-cost flash LiDAR technology, a windshield-mounted sensing platform was constructed using an inexpensive 16-segment flash LiDAR module, camera, and 3D printed housing.
Synchronous camera/LiDAR data was then collected while driving on public roadways.
With segmented flash LiDAR returns overlaid on concurrent camera images using a coordinate system transform, it was observed that common classes of roadway objects often exhibited a distinct distribution of maximum detection distances. While such marked variations in sensor performance could be seen as detrimental to the adoption of flash LiDAR technologies in the automotive space, this observation elicits a unique non- imaging method of roadway object and surface classification. Here, the LiDAR’s maximal detection distance is best understood through macroscopic reflection and absorption characteristics of materials illuminated by diffuse infrared light. Such a classification method may benefit future work on cooperative optical sensing and perception, especially during nighttime driving.
129 5.2 Methods
5.2.1 Sensor Platform Hardware
Automatic capture of optical images and segmented flash LiDAR depth data of the same scene though the windshield of a moving automobile required the construction of a custom sensor platform. High definition (HD) images and video were acquired using a Basler puA1280-54uc USB 3.0 CMOS machine vision camera (Fig. 35). This camera was chosen for its low cost of 200.00 USD, high frame rate of 55 fps, and automatic gain control features. When attached to a 4 mm fixed focal length lens, the camera featured a horizontal view field of 63.9° and a vertical view field of 49.1°.
Depth data associated with every captured image was collected using a M16 Solid
State LiDAR module manufactured by Leddartech Inc. (Fig. 57A). The module provides simultaneous data from 16 independent adjacent segments arranged horizontally. Each segment features a 3° horizontal and 7.5° vertical field of view. The total view field of the unit was 48° in the horizontal direction and 7.5° in the vertical direction. This LiDAR sensor was chosen for its low cost of $780.00 USD, detection range of 50m, fast default refresh rate of 50 Hz, and small 10.4 x 6.6 x 4.8 cm size.
Mechanically robust alignment of the camera and LiDAR module, affording both sensors an overlapping field of view, was accomplished using a 3D printed plastic housing and brackets (Fig. 57A,B). All 3D printed parts were designed using Autodesk Inventor CAD software and fabricated at the Arizona State University 3D printing lab. The camera and
LiDAR module were mounted into the housing using machine screws and locked in place using 3D printed removable brackets. Once secured within the housing, the optical axes of both the LiDAR and camera are aligned horizontally. In the vertical direction, the
130 A. B. C.
Figure 57: (A.) Photograph of LiDAR/camera sensor platform components. (B.) Photograph of the camera and LiDAR module mounted inside the 3D printed plastic housing, affording overlapped optical axes on the horizontal plane and a well-defined vertical offset. (C.) Sensor platform affixed to the windshield of an automobile during data collection using a suction cup mount. Exposed cables were connected to a laptop and battery pack located in the front passenger seat. optical axis offset is known from the Autodesk CAD file, but can also be measured using a digital caliper (Fig. 57A,B).
While both the camera and LiDAR module are rugged devices which can operate in industrial environments, neither are designed for use on the exterior of a moving vehicle. The completed sensor module was therefore temporarily mounted on the interior side of an automobile windshield using a PanaVise 13166 double suction cup mount (Fig.
57C) This mount was attached to the top of the 3D printed housing using screws (Fig.
57A,B,C.). The complete sensor platform required the use of three external electrical connections for operation. USB 3.0 and USB 2.0 ports, corresponding to the camera and
LiDAR module respectively (Fig. 57C), were connected to a laptop running C++ scripts on Ubuntu Linux. Additional power for the LiDAR was provided by a battery pack, equipped with an internal 120V AC inverter circuit.
131 5.2.2 Sensor Platform Cost Assessment
At the time of this report, the cost of all components and fabrication, excluding the price of a 3D printer tool was approximately 1500.00 USD, an itemized list is shown in Table 5.
Table 5: Itemized Cost of The Flash LiDAR/Camera Sensor Platform
Item Description Vender/Manufacturer Price in USD HD Machine Vision Camera Basler 200.00 Camera Lens B&H, Computar 45.00 16 Segment LiDAR Module LeddarTech 780.00 Suction Cup Mont PanaVise 30.00 3D Printer PLA Filament University Provided 30.00 Laptop Computer Dell 400.00
5.2.3 Computer Code for Automatic Data Collection
External objects passing though the view field of the LiDAR module trigger detections from some or all the segments. At the time of detection, each segment reports the distance of the external object it has detected. Thus, a real-time synchronous distance/image data recording program was implemented in C++ based on the
LeddarTech ‘LeddarCDemo’ example. At program start a timer object is initiated, returning a timestamp delta since the start time in milliseconds. A data callback function, automatically called following each full detection, is specified. Upon each detection the timestamp delta is saved in a csv file row along with the 16-dimensional detection distance data vector. Using the Basler Pylon SDK, an image is collected and converted into an OpenCV Mat object, which gets saved in a .jpg file with the timestamp delta in the filename.
132
5.2.4 Displaying flash LiDAR Detections on Video Data
A rendering Python script was written to convert the images and output csv file into a real-time video showing the detection state of each segment, their reported detection distances, and their corresponding view fields relative to the camera’s FOV. To render a frame, a line of output csv data is matched to the appropriate image based on timestamp. Then the OpenCV drawing functions are used to superimpose the detection rectangles on the image in the appropriate locations as described in section 5.2.5 The corresponding distance measurement was also superimposed in a small black text box above the top right corner of each rectangle. The frames were then composited into a .avi file using the OpenCV VideoWriter class.
5.2.5 Dynamic Display of LiDAR Segments
The angular view field of a LiDAR segment was represented by an unfilled rectangle overlaid on each video frame (Fig 58A). In most instances, detected external objects did not occupy an entire segment’s view field, however, because the LiDAR module only provides one distance measurement per segment, the entire segment view field was used when displaying the location of detected objects on video (Fig. 58A). The horizontal and vertical dimensions of each rectangular LiDAR segment in pixels was computed using the equations SH = (α/FH)*pH, and SV = (β/FV)*pV respectively. In these equations, α and β represent the angular view field of a single LiDAR segment while FH and FV represent the total angular view field of the camera in the horizontal and vertical directions respectively (Fig 58 B,C,D). The number of pixels along the camera’s horizontal and vertical axes are represented by pH and pV respectively (Fig. 58B). Along
133 the horizontal direction, α = ϕ/16, where ϕ is the total angular view field of the sixteen- segment LiDAR module (Fig. 58B,D). Notably, the horizontal and vertical dimensions of an overlaid segment are invariant with respect to detected object distance. This is a consequence of using each pixel to represent a portion of the angular view field covered by both the camera and LiDAR. The view field ratios, α/FH and β/FV along their
A. B.
C. D.
Figure 58: (A.) Example of LiDAR segment rectangles overlaid on a frame of video data collected while driving on public roadways in Tempe, AZ. The overlaid segment was generated using a Python3 script and the OpenCV image processing library. (B.) Diagram depicting the view fields of the camera, LiDAR, and a LiDAR segment overlaid on a video frame collected by the camera. The width and height of a LiDAR segment in pixels is represented by SH and SV. (C.) Side view of the LiDAR/camera sensor platform depicting the vertical view field angles of both sensors. (D.) Top- down view of the sensor platform depicting view field angles of the camera, LiDAR, and a LiDAR segment. 134 respective axes are always the same, regardless of the size or distance of an external object detected within the frame.
When mounted inside a 3D printed housing, the optical axes of both the LiDAR and camera lens overlap along the housing’s x-axis but are separated by a few centimeters along the y-axis (Fig 57B) (Fig 58C) (Fig. 59A). Consequently, overlaid rectangular LiDAR segments undergo vertical translation whose magnitude depends on the external object distance reported by each segment. Horizontal segment translation does not occur due to the sensors’ overlapping optical axes (Fig. 58D).
An equation describing the vertical translation of a LiDAR segment can be derived as follows. If the optical axes of a camera and LiDAR are separated by a fixed distance in vertical dimension (Fig. 59A), the angle between the center of the camera lens and an external object located on the LiDAR’s optical axis is θ =Arctan(L/D). In this equation, D represents object distance and L is the fixed distance between the optical axes of camera and LiDAR module. In the camera’s pixeled view field, the angular relationship between the video frame’s horizontal midline (Fig. 59A,B), called the horizon, and the center of the overlaid LiDAR segment can be defined as
퐹푉 푦푁 ∗ 푇푎푛 ( ) 푇푎푛(휃) = 2 . (5.1) 푝푉
Here, the angle between the horizon line and the center of a rectangular segment N is represented by θ, while the vertical location of the segment’s midpoint is represented by yN. FV is the angular view field of the camera, and pV is the number vertical of pixels in the video frame (Fig. 58B). Substituting the expression Arctan(L/D) in place of θ into
(5.1) yields
135 퐹푉 퐿 2푦푁 ∗ 푇푎푛 ( ) 푇푎푛 (퐴푟푐푡푎푛 ( )) = 2 . (5.2) 퐷 푝푉
Equation (5.2) reduces to
퐹푉 퐿 2푦푁 ∗ 푇푎푛 ( ) = 2 . (5.3) 퐷 푝푣
Solving for yN results in the equation
퐿 ∗ 푝푉 푦푁 = , (5.4) 퐹푉 2퐷 ∗ 푇푎푛 ( 2 ) where yN is the location of a LiDAR segment’s midpoint, in units of pixels, on the video frame.
The behavior of equation 5.4 is that of a reciprocal function with yN=0 defined as the camera frame horizon. Because of this, the midpoint of a rectangular LiDAR segment will be displayed along the horizon line when D>>L. The sign of L defines a translation direction of each segment as the value of D decreases. Choosing a negative value of L results in segment translation towards the bottom of the camera frame, while a positive L results in segment motion towards the top. More practically, the sign of L is chosen based on the location of the LiDAR with respect to the camera’s optical axis. If the LiDAR is located above the camera, L will be positive. If the LiDAR is located below the camera, L will take on a negative sign (Fig. 59B,C)).
Along the horizontal axis of the video frame, all sixteen rectangular LiDAR segments retain fixed positions from the vertical centerline (Fig. 59B,C). If the assumption is made that segments to the right of the centerline have positive angles, and
136 segments to the left of the centerline have negative angles, the center point of each segment is described by
휙 푝퐻 훼 푝퐻 푝퐻 푥푁 = ( − 푁 ∗ 훼) ∗ − ( ∗ ) − . (5.5) 2 퐹퐻 2 퐹퐻 2
A. B.
C.
Figure 59: (A.) Angle geometry of the LiDAR and camera detecting an external object at a distance D. L is the separation distance between the two sensors in the vertical direction and y represents the vertical pixel displacement between the external object’s midpoint and the camera’s optical axis. Using equations 5.1 to 5.4 the value of θ is used to relate D and y. (B.) Diagram depicting the horizontal and vertical pixel displacements, xN and yN, of several LiDAR segment midpoints overlaid on a video frame. While values of yN depend on D due to the separation of the camera and LiDAR optical axes in the vertical direction, xN values remain fixed in accordance with equation 5.5, as the axes overlap in the horizontal direction. Segment midpoints are used in place of object midpoints because the LiDAR module only provides one distance measurement per segment. In most cases the detected object did not occupy the entire segment’s view field. (C.) Example of LiDAR segment rectangles overlaid on a frame of video collected at Arizona State University’s Tempe campus with Palm Walk visible in the background. Detected foreground objects such as the large metal pole are closer to the sensor platform and thus possess a smaller value of D. As a result, the two rightmost segments exhibit noticeable vertical displacement in accordance with equation 5.4. Segments associated with objects further away are minimally displaced. 137 In this equation, xN is the horizontal pixel coordinate at the center of LiDAR segment N,
ϕ is the view field of the LiDAR module, α is the view field of a single LiDAR segment, pH is the number of horizontal pixels in the video frame, and FH is the angular horizontal view field of the camera. The first term defines the maximum view angle of LiDAR segment N and converts this angle to a pixel location. The second term moves the pixel location to the center of the LiDAR segment, while the third term sets the video frame centerline to a value of zero.
5.2.6 Fine Tuning of Displayed LiDAR Segments
Without correction terms, the values of xN and yN computed using equations 5.4 and 5.5 respectively do not account for angular misalignment of the LiDAR and camera optical axes. After several test runs, it was found that a slight angle misalignment in the horizontal direction caused segments to be displayed approximately twenty pixels to the left of a matte reflection target moving across the video frame at twelve meters. To address this issue, the center location of each segment was shifted twenty pixels left by subtracting a value of twenty pixels from xN. With this correction, detected objects at distances greater than three meters were sufficiently aligned with each segment. The cause of this angular misalignment is likely imperfect mounting of the optical components placed inside the 3D printed housing.
5.2.7 Automatic Data Collection, Driving on Roadways
In a typical data collection run, the LiDAR/Camera sensor platform was first mounted to the interior side of an automobile windshield before being connected to a laptop and Li-ion battery pack located in the front passenger seat (Fig 57C). A researcher sitting in the front passenger seat ran the automated data collection script on the laptop,
138 while another researcher operated the automobile. Time synchronized LiDAR and camera data were collected for a few minutes at a time on numerous streets surrounding
Arizona State University’s campus located in Tempe, Arizona. Lengthy data collection runs were avoided to conserve hard disk space on the laptop computer.
5.2.8 Reflectivity Measurement of Sample Fabrics
The LeddarTech M16 LiDAR module exhibits a maximum range of approximately 50 meters when light emitted by the module’s pair of infrared LEDs strikes a distant, highly reflective object such as roadway retroreflector. However, maximum detection range decreases significantly when the module is used to detect pedestrians. To gain a better understanding of this marked drop in distance, an optical reflectometer setup capable of measuring the reflectivity of common clothing fabrics in the visible and near infrared wavelength range was constructed on an optical table (Fig. 60F).
Light impinging on a fabric sample was first generated via a tungsten-halogen lamp with a ground-glass diffuser placed in front of the lamp box exit hole. Light passing though the diffuser was directed though a Mitutoyo long working distance objective with a magnification of 1X. After exiting the objective, light was projected on a fabric sample sandwiched between two optical cage plates. Use of the cage plates allowed the experimenter to place the fabric under a slight amount of tension, preventing the appearance of wrinkles. Due to the low magnification and long working distance of the objective, light rays directed at the sample exhibited a minimal degree of convergence.
Additionally, the diffuse nature of the light source resulted in a circular beam with a near- uniform intensity profile upon exiting the objective lens.
139 To measure the Visible/IR reflectivity characteristics of a fabric sample, the holder was rotated, forming an angle of twenty degrees between the sample surface
A. B.
C. D.
E. F.
Figure 60:(A.-D.) Visible/Infrared reflectivity data of colored Nylon and Felt fabric samples measured using the reflectometer setup described in section 5.2.8. (E.) Photograph of Nylon and Felt samples tested using the reflectometer setup. 95% and 18% Lambertian-like reflectivity cards are shown for reference. (F.) Photograph of reflectometer setup described in section 5.2.8. 140 normal and the microscope objective’s optical axis. This allowed reflected light to be gathered by a plano-convex collector lens with a focal length of 40mm. An optical fiber connected to a StellarNet EPP-2000 single grating spectrometer was then placed at the focal point of the collector lens, significantly increasing the amount of reflected light accepted by the fiber’s small entrance port (Fig. 60F).
In a typical reflectivity measurement, a baseline reflectivity value of 100% was first established using a photography whitecard (Fig. 60E) placed in the sample holder and illuminated by the tungsten-halogen light source. Next, the whitecard was removed and a baseline reflectivity of 0% was established. After the baseline values had been entered, the spectrometer software was switched to reflectivity measurement mode.
Nylon and Felt fabric samples of various colors were then placed in the sample holder one at a time, and a reflectivity spectrum was recorded for each sample (Fig. 60E,F). It should be noted that during these reflectivity tests, all extraneous light sources such florescent room lights and computer monitors were switched off.
5.3 Results and Discussion
5.3.1 LiDAR Module Detection Range
Detection and ranging of relevant objects approaching a vehicle must occur at distances that allow a driver/vehicle sufficient time to evaluate the sensors’ data and take appropriate actions. Because of this, detection range is considered a primary performance metric for automotive LiDAR and typically depends on four external object properties; distance from the sensor module, areal cross-section, surface angle, and the external objects’ near-infrared reflection, absorption, transmission characteristics (Kashani et al.,
2015). While it is feasible to quantify all four factors in a laboratory environment, such
141 detailed object characterization is impractical while driving on a public roadway. Indeed, only detected object distance is measured during data collection. Due to this limitation, an alternative method for characterizing the average detection range of various roadway objects encountered by the M16 LiDAR module was devised.
Video files compiled after each data collection run using the methods described in sections 5.2.4 and 5.2.5, were reviewed by a member of the author’s laboratory group.
The protocol for reviewing video is described as follows. Upon the appearance of a rectangular segment containing a distant object, video file playback was paused, and the region bounded by the segment rectangle examined. During examination, the object deemed most likely to have caused the detection was identified and the segment’s reported detection distance recorded. In most cases, objects triggered a single LiDAR segment, and were associated with a singular maximum distance. However, if an object triggered a detection in more than one segment, the average range was used instead.
Additionally, if the object which caused the detection could not be clearly identified, the segment was deemed spurious and skipped. After associating the detected object with the distance reported by the LiDAR segment or segments, video playback was resumed. This process was repeated for each new segment which appeared.
After reviewing videos of multiple data runs, it was found that four object classes were frequently detected by the multi-segment LiDAR module. Namely, motor vehicles, pedestrians, street signs, and traffic cones. Because each object was associated with only one maximum distance, athematic mean and standard deviation could be calculated for each object class (Table 6).
142 Table 6: Low-cost LiDAR detection distances of encountered roadway objects
Mean Detection Object Class Std. Dev (m) Num. of Objects Range (m) Motor Vehicles, Front/Rear* 25.80 5.31 31 Motor Vehicles, Side* 25.02 5.63 18
Street Signs, Small † 40.92 6.53 26 Street Signs, Medium † 41.35 5.95 32 Street Signs, Large † 44.5 N/A 4
Pedestrians 10.21 2.04 14
Traffic Cones 41.52 3.41 27
* Vehicles classification includes cars, trucks, and buses.
† Street sign size classifications are based on the approximate pixel area of a hexagonal stop sign in a video frame at a distance of 12 meters. Signs with at least 50% greater pixel area at 12 meters were considered large, while signs with at least 50% less area at 12 meters were considered small. Signs with pixel areas between these two limits are considered medium sized.
5.3.2 Detection Range of Street Signs and Traffic Cones
Street signs and traffic cones were often detected at a range of approximately of
40 meters (Table 6). This was likely due to the presence of integrated retroreflective materials, which afford nighttime visibility when illuminated by vehicle headlamps. Such visible-light reflectivity characteristics extend into the near-infrared (NIR) wavelength range (Schultz, et al., 2012) generating high-quality reflection returns from the LiDAR module’s pair of NIR LEDs.
Because street signs can vary greatly in both size and shape, detection data was separated into three sub-classes (Table 6) based on the pixel area occupied by the sign in the video frame at a detection range of 12 meters. The approximate pixel area of the 143 frequently encountered hexagonal stop sign was chosen as a reference for a medium sized street sign. Signs with at least 50% greater pixel area were considered large, while signs with at least 50% less area were considered small.
Intriguingly, the mean detection distance and standard deviation for both small and medium sized signs are similar, despite differences in optical cross-section (Table 6).
For large signs, a slight increase in distance was observed, however only four such signs were detected. Additionally, small signs triggered a detection before occupying an entire segment’s view field. These observations suggest that for small and medium sized signs, the magnitude of optical power returning to the LiDAR depends more on retroreflective properties than changes in cross-sectional area. This may also explain why traffic cones, which feature similarly sized retroreflectors, are detected at approximately 40 meters. It should also be noted that while the mean detection range of a traffic cone is similar to a street sign, the standard deviation is smaller by a factor of two (Table 6). This is likely due to the fact that encountered traffic cones were nearly identical to each other, while the width and height of street signs varied significantly.
5.3.3 Detection Range of Motor Vehicles
The mean detection range of a motor vehicle, such as a car or truck is approximately 25 meters from the front of the LiDAR module (Table 6). This value is longer than the mean pedestrian detection range of 12 meters but significantly less than the 40-meter detection range associated with retroreflective street signs and traffic cones.
Given the large areal cross-section of most vehicles, a decrease in maximum detection distance is likely due to the NIR transmission and reflection characteristics of automotive windows and body panels.
144 Retroreflective surface materials reflect approximately 40-70% of the 900-
1000nm NIR light generated by the LiDAR module (Schultz, et al., 2012). By comparison, laminated automotive glass, used for windshields, transmits approximately
70% of all incoming optical radiation in the 900-1000nm range (Aguilar et al., 2004) and reflects only 10% (Aguilar, et al., 2012). Tempered glass, which is commonly used for side windows, transmits 80% of NIR light in the 900-1000nm range and reflects approximately 10% (Aguilar, et al., 2012). Such numbers suggest that the modest reflectivity of automotive glass likely contributes to the observed decrease in motor vehicle detection distance compared to street signs and traffic cones.
Vehicle body panels and doors are coated with paint containing pigment particles
0.01μm-80.0μm in diameter and metal flakes 1-80μm in size dispersed in a transparent polymer binder (Seubert et al., 2015). Aluminum flakes are often used as the primary metallic component. The surface of this paint results in a semi-specular reflection which directs impinging optical radiation back towards the LiDAR module. However, experimental data has shown that the intensity of returning light is often one to five orders of magnitude less (Rump et al., 2008) than a surface coated with retroreflective materials (Marchant et al., 2010) over the same range of incident angles. This results in a shorter detection range compared to street signs and traffic cones.
Despite a larger optical cross-section, the sides of a motor vehicle offered no advantage in detection distance compared to the front or rear. This is likely due to the segmented nature of the LiDAR’s view field. A detected vehicle presenting its side may occupy 2 to 5 segments at a range of 25 meters (Fig. 58A), however each segment has a limited view field and only receives a portion of the total optical radiation reflecting off
145 the vehicle’s body panels, windows, and doors. Additionally, at the maximum detection range of 25 meters, the front or rear of a detected vehicle occupies or slightly exceeds the view field of a single segment in the horizontal direction. From this information, it can be concluded that a vehicle must nearly fill the horizontal view field of a LiDAR segment to trigger a detection. Because each segment cannot be more than completely filled, the mean detection distance reported by each segment is approximately 25 meters regardless of how many segments the vehicle occupies.
5.3.4 Detection Range of Pedestrians
Pedestrians were first detected by the LiDAR module at a range of 8 to 12 meters, with a mean of approximately 10 meters, considerably shorter than other commonly encountered roadway objects (Table 6). This observation can likely be attributed to the optical properties of skin and clothing.
Human skin reflects more than 65% (Kanzawa, et al., 2011) of incident NIR light in the 900-1000nm wavelength range used by the LiDAR module (Vaudelle, et al,, 2017).
However, skin also exhibits a Lambertian-like reflection profile, which distributes impinging optical radiation over most of the reflection angle hemisphere (Sohaib, et al.,
2017). This means that only a small portion of the total incoming NIR light is reflected back to the LiDAR, resulting in a short detection range. In contrast, street signs, traffic cones and motor vehicles exhibit semi-specular reflection, directing more optical radiation into a subset of angles, many of which point towards the LiDAR.
The mean reflectivity value of common clothing items is approximately 58% with a standard deviation of 16% in the 900-1000nm range wavelength range used by the
LiDAR module (Haran and Chien, 2016). This range of values is consistent with
146 reflectivity data obtained for various Nylon and Felt fabrics using the reflectometer setup described in section 5.2.8. Over a wavelength range of 900-950nm, Nylon fabrics tested by the authors of this report exhibited reflectivity values between 28% and 75% while
Felt fabrics exhibited reflectivity values between 59% and 79%. Similar to human skin, common clothing materials exhibit a Lambertian-like reflection profile (Choe,
Narasimhan, and Kweon, 2016) which limits the total number of photons directed back to the LiDAR module, significantly reducing detection range.
5.4 Conclusion
This chapter describes the construction and evaluation of an inexpensive flash
LiDAR/camera sensor platform, installed inside the cabin of an automobile using a removeable 3D printed suction-cup mount. Optical images and segmented depth data of the same scene were captured while driving on public roadways near Arizona State
University’s Tempe campus. LiDAR returns were then dynamically overlaid on each frame of video using a coordinate transform and the OpenCV programming library.
Reviewing video containing overlaid LiDAR data, it was observed that some commonly encountered object classes were first detected by the LiDAR module at a well- defined average range. Street signs and traffic cones exhibited a mean detection distance of approximately 40 meters while motor vehicles and pedestrians were detected at mean distances of 25 meters and 10 meters respectively. This observation was attributed in detail to the NIR reflection and absorption characteristics of each object class.
Additionally, it was found that NIR reflectivity values of various Nylon and Felt fabric samples under full-scene illumination were similar to those reported by other researchers using narrow-beam laser illumination at the same range of wavelengths. Future work
147 will utilize these observations to explore cooperative object classification, with camera and flash-LiDAR detection data used as training input to a neural network capable of fusing both sensing modalities. The classifier performance of such a network will then be evaluated in the context of low-light and nighttime driving scenarios.
148 CHAPTER 6
CONCLUSION
In proceeding chapters, a case for the applicability of non-imaging optical systems in the context of heterogonous semiconductor integration and automotive LiDAR, was presented in detail. Chapters two and four contain a thorough review of current academic and industry literature, revealing two under-addressed remote sensing challenges pertaining to these emerging technologies at vastly different length scales. The semiconductor industry lacks a high-precision wide-field strain metrology with displacement resolution smaller than 100 nanometers capable of measuring strain fields between interconnect wires or vias only a few micrometers apart. Meanwhile, little attention is paid to the per-meter sensing characteristics of scene-illuminating flash
LiDAR in the context of automotive applications, despite the technology’s potential to lower module manufacturing costs by two orders of magnitude. Critically, these challenges could be confronted and understood through the principals and consequences of non-imagining optics, prompting the development of a non-contact nanoscale strain sensing metrology and low-cost flash LiDAR/camera sensor platform for automotive perception, described in chapters three and five respectively.
To overcome the resolution limits and practical challenges associated with digital image correlation (DIC) and Moiré interferometry, two popular micro-strain mapping techniques for semiconductor package inspection, a method of mapping strain and surface angle using thin Au diffraction gratings fabricated on the surface of a microelectronics package was utilized. Changes in pitch on the order of nanometers were observed by illuminating the grating surface with a focused He-Ne laser while monitoring
149 the intensity distribution of light coupled into the first diffractive order. This was accomplished by projecting outgoing light onto a wide-area, angle-calibrated CMOS image sensor located in the diffraction pattern’s far-field (Fraunhofer) region. Diffraction angle and grating pitch were then computed from the horizontal pixel position of the intensity distribution’s peak value. In this instance, pitch deltas on the order of 1nm resulted in peak shifts greater than 100μm, obviating need of an imaging lens.
Full-field pitch mapping of a grating bonded to a TSV-solder ball microstructure at various package temperatures was carried out by sweeping the focused laser beam spot across grating surface using a motorized two-axis translation stage. At each sample position, an optical intensity distribution was recorded by the CMOS image sensor. Later, these intensity distributions would be converted to a pitch value and placed on a virtual
2D grid at their respective points of acquisition, generating a pitch map. Maps of obtained at two different package temperatures were then subtracted from each other to find the change in pitch at each measured point.
Away from material interfaces, strain values measured in subregions containing silicon and copper were in good agreement with theoretical calculations. However, the measured CTE values were negative, suggesting that the regions undergo compression rather than expansion during thermal loading. A surface angle map of the grating region was also computed by assuming that any deviations in diffraction angle are due to non- zero surface angles at 25ºC. From this map, it was found that local surface angles deviate from horizontal by only a few degrees but change rapidly across material interface regions, eliciting spurious pitch values. Homogenous regions are much flatter, exhibiting only small changes in surface angle.
150 To better understand the practical merit and challenges associated with low-cost flash LiDAR technology, a windshield-mounted sensing platform capable of acquiring both images depth data was constructed using an inexpensive 16-segment flash LiDAR module, USB-powered machine vision camera, and 3D printed housing. Importantly, this sensing platform had a total cost of 1500.00 USD, one order of magnitude less than a typical high-performance automotive LiDAR unit. Optical images and segmented depth data of the same scene were captured while driving on public roadways near Arizona
State University’s Tempe campus. LiDAR returns were then dynamically overlaid on each frame of video using a derived coordinate transform and the OpenCV programming library.
Reviewing video containing overlaid LiDAR data, it was observed that some commonly encountered object classes were first detected by the LiDAR module at a well- defined average range. Street signs and traffic cones exhibited a mean detection distance of approximately 40 meters while motor vehicles and pedestrians were detected at mean distances of 25 meters and 10 meters respectively. For flash LiDAR, such variances in detection distance can be explained by the macroscopic reflection and absorption characteristics of materials illuminated by diffuse infrared light. For example, common clothing items and fabrics have moderate reflectivity, but scatter incident optical radiation over the entirety of their reflection hemisphere. Such reflection characteristics greatly limit the amount of energy returned to the LiDAR’s detector array, shortening the maximum distance at which a pedestrian is detected. Additionally, it was found that infrared reflectivity values of various Nylon and Felt fabric samples under full-scene
151 illumination were similar to those reported by other researchers using narrow-beam laser illumination over the same range of infrared wavelengths.
While such marked variations in sensor performance due to material characteristics could be seen as detrimental to the adoption of flash LiDAR technologies in the automotive space, this observation elicits a unique non-imaging method of roadway object and surface classification. It not difficult to imagine how such a method may benefit cooperative optical sensing in driving scenarios where image-based object classification is insufficient. Indeed, nighttime driving provides ample opportunities to develop and evaluate these kinds of multi-modal sensing schemes.
In summary, two remote sensing applications harnessing the principals and consequences of non-imaging optics have been demonstrated at the length scales relevant to advanced semiconductor packaging and automotive LiDAR. Admittedly specialized, such applications may be appreciated in these emerging and consequential fields which present a unique set of remote sensing challenges. Indeed, given the immediately practical nature of the sensing applications developed in this thesis, it is anticipated that non-imaging optics will be of importance to future technology stacks involving automotive electronics and LiDAR-based perception.
152 REFERENCES
Adler, M. W., Peer, S., & Sinozic, T. (2019). Autonomous, connected, electric shared vehicles (Aces) and public finance: An explorative analysis. Transportation Research Interdisciplinary Perspectives, 2, 100038. https://doi.org/10.1016/j.trip.2019.100038
]Aguilar, J. O., Gomez-Daza, O., Brito L., A., Nair, M. T. S., & Nair, P. K. (2005). Optical and mechanical characteristics of clear and solar control laminated glass using zinc sulphide and copper sulphide thin films. Surface and Coatings Technology, 200(7), 2557–2565. https://doi.org/10.1016/j.surfcoat.2004.10.140
Akasheh, F., Myers, T., Fraser, J. D., Bose, S., & Bandyopadhyay, A. (2004). Development of piezoelectric micromachined ultrasonic transducers. Sensors and Actuators A: Physical, 111(2–3), 275–287. https://doi.org/10.1016/j.sna.2003.11.022
Andersen, H.-E., McGaughey, R. J., & Reutebuch, S. E. (2005). Estimating forest canopy fuel parameters using LIDAR data. Remote Sensing of Environment, 94(4), 441– 449. https://doi.org/10.1016/j.rse.2004.10.013
Ansmann, A., Riebesell, M., & Weitkamp, C. (1990). Measurement of atmospheric aerosol extinction profiles with a Raman lidar. Optics Letters, 15(13), 746. https://doi.org/10.1364/OL.15.000746
Arikawa, T., Murasawa, G., & Yoneyama, S. (2011). Accurate displacement field measurement with scanning electron microscopy imaging. The Journal of Strain Analysis for Engineering Design, 46(5), 337–346. https://doi.org/10.1177/0309324711405751
Banks-Sills, L., Shklovsky, J., Krylov, S., Bruck, H. A., Fourman, V., Eliasi, R., & Ashkenazi, D. (2011). A Methodology for Accurately Measuring Mechanical Properties on the Micro-Scale. Strain, 47(3), 288–300. https://doi.org/10.1111/j.1475-1305.2009.00692.x
Barako, M. T., Isaacson, S. G., Lian, F., Pop, E., Dauskardt, R. H., Goodson, K. E., & Tice, J. (2017). Dense Vertically Aligned Copper Nanowire Composites as High Performance Thermal Interface Materials. ACS Applied Materials & Interfaces, 9(48), 42067–42074. https://doi.org/10.1021/acsami.7b12313
Basha, T., Moses, Y., & Kiryati, N. (2013). Multi-view Scene Flow Estimation: A View Centered Variational Approach. International Journal of Computer Vision, 101(1), 6–21. https://doi.org/10.1007/s11263-012-0542-7
153 Bay, B. K., Smith, T. S., Fyhrie, D. P., & Saad, M. (1999). Digital volume correlation: Three-dimensional strain mapping using X-ray tomography. Experimental Mechanics, 39(3), 217–226. https://doi.org/10.1007/BF02323555
Bertoldo, S., Lucianaz, C., Allegretti, M., & Perona, G. (2017). Preliminary analysis to design a 77 GHz weather radar. 2017 18th International Radar Symposium (IRS), 1–8. https://doi.org/10.23919/IRS.2017.8008172
Birch, K. P., & Downs, M. J. (1993). An Updated Edlén Equation for the Refractive Index of Air. Metrologia, 30(3), 155–162. https://doi.org/10.1088/0026- 1394/30/3/004
Blöecher, H.-L., Andres, M., Fischer, C., Sailer, A., Goppelt, M., & Dickmann, J. (2012). Impact of system parameter selection on radar sensor performance in automotive applications. Advances in Radio Science, 10, 33–37. https://doi.org/10.5194/ars- 10-33-2012
Brain, R. (2016). Interconnect scaling: Challenges and opportunities. 2016 IEEE International Electron Devices Meeting (IEDM), 9.3.1-9.3.4. https://doi.org/10.1109/IEDM.2016.7838381
Braun, T., Becker, K.-F., Hoelck, O., Kahle, R., Woehrmann, M., Toepper, M., Ndip, I., Maass, U., Tschoban, C., Aschenbrenner, R., Voges, S., & Lang, K.-D. (2018). Fan-out Wafer Level Packaging for 5G and mm-Wave Applications. 2018 International Conference on Electronics Packaging and IMAPS All Asia Conference (ICEP-IAAC), 247–251. https://doi.org/10.23919/ICEP.2018.8374297
Braun, T., Becker, K.-F., Hoelck, O., Voges, S., Kahle, R., Dreissigacker, M., & Schneider-Ramelow, M. (2019). Fan-out Wafer and Panel Level Packaging as Packaging Platform for Heterogeneous Integration. Micromachines, 10(5), 342. https://doi.org/10.3390/mi10050342
Bruck, H. A., McNeill, S. R., Sutton, M. A., & Peters, W. H. (1989). Digital image correlation using Newton-Raphson method of partial differential correction. Experimental Mechanics, 29(3), 261–267. https://doi.org/10.1007/BF02321405
Campos, J., O’Toole, E., Pinto, R., Cardoso, A., Toussaint, P.-L., & Vernhes, P. (2015). Development of Fan-In WLP with Mechanical Protection and Dynamic Warpage Characterization of eWLB. 2015 IEEE 65th Electronic Components and Technology Conference (ECTC), 1689–1694. https://doi.org/10.1109/ECTC.2015.7159824
Caporossi, P., Mazzanti, P., & Bozzano, F. (2018). Digital Image Correlation (DIC) Analysis of the 3 December 2013 Montescaglioso Landslide (Basilicata, Southern
154 Italy): Results from a Multi-Dataset Investigation. ISPRS International Journal of Geo-Information, 7(9), 372. https://doi.org/10.3390/ijgi7090372
Carson, R. F., Warren, M. E., Dacha, P., Wilcox, T., Maynard, J. G., Abell, D. J., Otis, K. J., & Lott, J. A. (2016). Progress in high-power high-speed VCSEL arrays (K. D. Choquette & J. K. Guenter, Eds.; p. 97660B). https://doi.org/10.1117/12.2215009
Castanedo, F. (2013). A Review of Data Fusion Techniques. The Scientific World Journal, 2013, 1–19. https://doi.org/10.1155/2013/704504
Chavez-Garcia, R. O., & Aycard, O. (2016). Multiple Sensor Fusion and Classification for Moving Object Detection and Tracking. IEEE Transactions on Intelligent Transportation Systems, 17(2), 525–534. https://doi.org/10.1109/TITS.2015.2479925
Chen, D. J., Chiang, F. P., Tan, Y. S., & Don, H. S. (1993). Digital speckle-displacement measurement using a complex spectrum method. Applied Optics, 32(11), 1839. https://doi.org/10.1364/AO.32.001839
Chen, W., & Bottoms, B. (2019). Heterogeneous integration roadmap: Driving Force and Enabling Technology for Systems of the Future. 2019 Symposium on VLSI Technology, T50–T51. https://doi.org/10.23919/VLSIT.2019.8776484
Chen, X., Ye, H., Fan, X., Ren, T., & Zhang, G. (2016). A review of small heat pipes for electronics. Applied Thermal Engineering, 96, 1–17. https://doi.org/10.1016/j.applthermaleng.2015.11.048
Chen, X., Zheng, B., & Liu, H. (2011). Optical and digital microscopic imaging techniques and applications in pathology. Analytical Cellular Pathology, 34(1–2), 5–18. https://doi.org/10.1155/2011/150563
Chew, L. M., & Schmitt, W. (2019). High reliable silver sintered joint on copper lead frame by pressure sintering process. 2019 IEEE International Workshop on Integrated Power Packaging (IWIPP), 44–47. https://doi.org/10.1109/IWIPP.2019.8799094
Choe, G., Narasimhan, S. G., & Kweon, I. S. (2016). Simultaneous Estimation of Near IR BRDF and Fine-Scale Surface Geometry. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2452–2460. https://doi.org/10.1109/CVPR.2016.269
Chowdhury A. R. (May-June). OSAT perspective: Automotive semiconductor market & manufacturing challenges. Chip Scale Review, 23(3), 22–29.
155 Chu, T. C., Ranson, W. F., & Sutton, M. A. (1985). Applications of digital-image- correlation techniques to experimental mechanics. Experimental Mechanics, 25(3), 232–244. https://doi.org/10.1007/BF02325092
Civale, Y., et al., (2012). Enhanced barrier seed metallization for integration of high- density high aspect-ratio copper-filled 3D through-silicon via interconnects. 2012 IEEE 62nd Electronic Components and Technology Conference (ECTC), 822– 826. https://doi.org/10.1109/ECTC.2012.6248928
Clemesha B. R. , G. S. Kent, & Wright R.W.H. (1967). A Laser Radar for Atmospheric Studies. Journal of Applied Meteorology, 6(2), 386–395. https://doi.org/10.1175/1520-0450(1967)006<0386:ALRFAS>2.0.CO;2
Coudrain, P., et al., (2012). 3D Integration of CMOS Image Sensor with Coprocessor Using TSV last and Micro-Bumps Technologies. 2013 IEEE 63rd Electronic Components and Technology Conference (ECTC), 674–682. https://doi.org/10.1109/ECTC.2013.6575646
Cova, P., Aliyu, A. M., Castellazzi, A., Chiozzi, D., Delmonte, N., Lasserre, P., & Pignoloni, N. (2017). Thermal design and characterization of a modular integrated liquid cooled 1200 V-35 A SiC MOSFET bi-directional switch. Microelectronics Reliability, 76–77, 277–281. https://doi.org/10.1016/j.microrel.2017.06.062
Crammond, G., Boyd, S. W., & Dulieu-Barton, J. M. (2013). Speckle pattern quality assessment for digital image correlation. Optics and Lasers in Engineering, 51(12), 1368–1378. https://doi.org/10.1016/j.optlaseng.2013.03.014
Cui, H., Li, Y., & Liu, J. (2016). An Obstacle Detection Algorithm Based on Ultrasonic Sensors for Autonomous Land Vehicle. Proceedings of the 2016 International Conference on Mechatronics, Control and Automation Engineering. 2016 International Conference on Mechatronics, Control and Automation Engineering, Bangkok, Thailand. https://doi.org/10.2991/mcae-16.2016.35
Czarnek, R. (1990). Super High Sensitivity Moiré Interferometry with Optical Multiplication. Optics and Lasers in Engineering, 13(2), 87–98. https://doi.org/10.1016/0143-8166(90)90028-8
De Vos, J., Van Huylenbroeck, S., Jourdain, A., Heylen, N., Peng, L., Jamieson, G., Tutunjyan, N., Sardo, S., Miller, A., & Beyne, E. (2018). “Hole-in-one TSV”, a new via last concept for high density 3D-SOC interconnects. 2018 IEEE 68th Electronic Components and Technology Conference (ECTC), 1499–1504. https://doi.org/10.1109/ECTC.2018.00228
156 Dodge, S., & Karam, L. (2016). Understanding How Image Quality Affects Deep Neural Networks. 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX), 1–6. https://doi.org/10.1109/QoMEX.2016.7498955
Drummond, K. P., Back, D., Sinanis, M. D., Janes, D. B., Peroulis, D., Weibel, J. A., & Garimella, S. V. (2018). A Hierarchical Manifold Microchannel Heat Sink Array for High-Heat-Flux Two-Phase Cooling of Electronics. International Journal of Heat and Mass Transfer, 117, 319–330. https://doi.org/10.1016/j.ijheatmasstransfer.2017.10.015
Eigen, D., Puhrsch, C., & Fergus, R. (2014). Depth Map Prediction from a Single Image using a Multi-Scale Deep Network. ArXiv:1406.2283 [Cs]. http://arxiv.org/abs/1406.2283
El Helou, M., Shahpaski, M., & Süsstrunk, S. (2019). Solving the depth ambiguity in single-perspective images. OSA Continuum, 2(10), 2901. https://doi.org/10.1364/OSAC.2.002901
Fagnant, D. J., & Kockelman, K. (2015). Preparing a nation for autonomous vehicles: Opportunities, barriers and policy recommendations. Transportation Research Part A: Policy and Practice, 77, 167–181. https://doi.org/10.1016/j.tra.2015.04.003
Fanning, T. R., Maynard, J., Helms, C. J., Yang, L., Podva, D., Earls, J., Lopez, J. U., Chung, M., Foresi, J., Warren, M. E., & Bacchin, G. (2020). Performance, manufacturability, and qualification advances of high-power VCSEL arrays at TriLumina Corporation. In C. Lei & L. A. Graham (Eds.), Vertical-Cavity Surface-Emitting Lasers XXIV (p. 1). SPIE. https://doi.org/10.1117/12.2546689
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2020). Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges. IEEE Transactions on Intelligent Transportation Systems, 1–20. https://doi.org/10.1109/TITS.2020.2972974
Fisher, P. D., & Pyhtila, J. W. (2000). Timing Quantization Error in Lidar Speed- Measurement Devices. IEEE Transactions on Vehicular Technology, 49(1), 276– 280. https://doi.org/10.1109/25.820720
Fleming, W. J. (2001). Overview of Automotive Sensors. IEEE Sensors Journal, 1(4), 296–308. https://doi.org/10.1109/7361.983469
Floudas, N., Polychronopoulos, A., Aycard, O., Burlet, J., & Ahrholdt, M. (2007). High Level Sensor Data Fusion Approaches for Object Recognition in Road
157 Environment. 2007 IEEE Intelligent Vehicles Symposium, 136–141. https://doi.org/ 10.1109/IVS.2007.4290104
Fremont, V., Bui, M., Boukerroui, D., & Letort, P. (2016). Vision-Based People Detection System for Heavy Machine Applications. Sensors, 16(1), 128. https://doi.org/10.3390/s16010128
Gao, Y., Cheng, T., Su, Y., Xu, X., Zhang, Y., & Zhang, Q. (2015). High-efficiency and high-accuracy digital image correlation for three-dimensional measurement. Optics and Lasers in Engineering, 65, 73–80. https://doi.org/10.1016/j.optlaseng.2014.05.013
Garcia, F., Martin, D., de la Escalera, A., & Armingol, J. M. (2017). Sensor Fusion Methodology for Vehicle Detection. IEEE Intelligent Transportation Systems Magazine, 9(1), 123–133. https://doi.org/10.1109/MITS.2016.2620398
Garrou, P. (2000). Wafer Level Chip Scale Packaging (WL-CSP): An Overview. IEEE Transactions on Advanced Packaging, 23(2), 198–205. https://doi.org/10.1109/6040.846634
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013). Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, 32(11), 1231–1237. https://doi.org/10.1177/0278364913491297
Glennie, C. L., Carter, W. E., Shrestha, R. L., & Dietrich, W. E. (2013). Geodetic imaging with airborne LiDAR: The Earth’s surface revealed. Reports on Progress in Physics, 76(8), 086801. https://doi.org/10.1088/0034-4885/76/8/086801
Godard, C., Aodha, O. M., Firman, M., & Brostow, G. (2019b). Digging Into Self- Supervised Monocular Depth Estimation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 3827–3837. https://doi.org/10.1109/ICCV.2019.00393
Gopireddy, B., & Torrellas, J. (2019). Designing Vertical Processors in Monolithic 3D. Proceedings of the 46th International Symposium on Computer Architecture, 643–656. https://doi.org/10.1145/3307650.3322233
Griffiths, A. D., Chen, H., Li, D. D.-U., Henderson, R. K., Herrnsdorf, J., Dawson, M. D., & Strain, M. J. (2019). Multispectral time-of-flight imaging using light-emitting diodes. Optics Express, 27(24), 35485. https://doi.org/10.1364/OE.27.035485
Gruyer, D., Magnier, V., Hamdi, K., Claussmann, L., Orfila, O., & Rakotonirainy, A. (2017). Perception, information processing and modeling: Critical stages for autonomous driving applications. Annual Reviews in Control, 44, 323–341. https://doi.org/10.1016/j.arcontrol.2017.09.012
158 Haddadi, H., & Belhabib, S. (2008). Use of rigid-body motion for the investigation and estimation of the measurement errors related to digital image correlation technique. Optics and Lasers in Engineering, 46(2), 185–196. https://doi.org/10.1016/j.optlaseng.2007.05.008
Haga, T. (2018). The Early Days of R&D on EUV Lithography and Future Expectations. Journal of Photopolymer Science and Technology, 31(2), 193–200. https://doi.org/10.2494/photopolymer.31.193
Hakobyan, G., & Yang, B. (2019). High-Performance Automotive Radar: A review of signal processing algorithms and modulation schemes. IEEE Signal Processing Magazine, 36(5), 32–44. https://doi.org/10.1109/MSP.2019.2911722
Hamzah, R. A., & Ibrahim, H. (2016). Literature Survey on Stereo Vision Disparity Map Algorithms. Journal of Sensors, 2016, 1–23. https://doi.org/10.1155/2016/8742920
Han, B. (1993). Interferometric methods with enhanced sensitivity by optical/digital fringe multiplication. Applied Optics, 32(25), 4713. https://doi.org/10.1364/AO.32.004713
Han, S., Yang, L., Tian, Z., Yuan, X., & Lu, H. (2019). Research on a Simplified Model of an Aluminum Vapor Chamber in a Heat Dissipation System. Entropy, 22(1), 35. https://doi.org/10.3390/e22010035
Haran, T., & Chien, S. (2016). Infrared Reflectivity of Pedestrian Mannequin for Autonomous Emergency Braking testing. 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), 2230–2235. https://doi.org/10.1109/ITSC.2016.7795916
Harvey, J. E., & Pfisterer, R. N. (2019). Understanding diffraction grating behavior: Including conical diffraction and Rayleigh anomalies from transmission gratings. Optical Engineering, 58(08), 1. https://doi.org/10.1117/1.OE.58.8.087105
Hasch, J., Topak, E., Schnabel, R., Zwick, T., Weigel, R., & Waldschmidt, C. (2012). Millimeter-Wave Technology for Automotive Radar Sensors in The 77 GHz Frequency Band. IEEE Transactions on Microwave Theory and Techniques, 60(3), 845–860. https://doi.org/10.1109/TMTT.2011.2178427 Heterogeneous Integration Roadmap 2019 Edition. (2019). IEEE Electronics Packaging Society (EPS). http://eps.ieee.org/hir
Ho, C.-Y., Hsieh, S.-C., Jhong, M.-F., Kuo, H.-C., Ting, C.-Y., & Wang, C.-C. (2019). A Low-Cost Antenna-in-Package Solution for 77GHz Automotive Radar Applications. 2019 International Conference on Electronics Packaging (ICEP), 110–114. https://doi.org/10.23919/ICEP.2019.8733518
159 Hoefnagels, J. P. M., van Maris, M. P. F. H. L., & Vermeij, T. (2019). One-step deposition of nano-to-micron-scalable, high-quality digital image correlation patterns for high-strain in-situ multi-microscopy testing. Strain, 55(6), e12330. https://doi.org/10.1111/str.12330
Holister G. S. (1967). Experimental Stress Analysis: Principles and Methods (1st ed.). Cambridge University Press.
Huang, X., Cheng, X., Geng, Q., Cao, B., Zhou, D., Wang, P., Lin, Y., & Yang, R. (2018). The ApolloScape Dataset for Autonomous Driving. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1067–10676. https://doi.org/10.1109/CVPRW.2018.00141
Huikari, J., Jahromi, S., Jansson, J.-P., & Kostamovaara, J. (2018). Compact laser radar based on a subnanosecond laser diode transmitter and a two-dimensional CMOS single-photon receiver. Optical Engineering, 57(02), 1. https://doi.org/10.1117/1.OE.57.2.024104
Hussain, R., & Zeadally, S. (2019). Autonomous Cars: Research Results, Issues, and Future Challenges. IEEE Communications Surveys & Tutorials, 21(2), 1275– 1313. https://doi.org/10.1109/COMST.2018.2869360
Ifju, P. G., & Han, B. (2010). Recent Applications of Moiré Interferometry. Experimental Mechanics, 50(8), 1129–1147. https://doi.org/10.1007/s11340-010-9404-9
Iradukunda, A.-C., Huitink, D. R., & Luo, F. (2020). A Review of Advanced Thermal Management Solutions and the Implications for Integration in High-Voltage Packages. IEEE Journal of Emerging and Selected Topics in Power Electronics, 8(1), 256–271. https://doi.org/10.1109/JESTPE.2019.2953102
Ishitani, T., & Tsuboi, H. (1996). Objective Comparison of Scanning Ion and Scanning Electron Microscope Images. Scanning, 19(7), 489–497. https://doi.org/10.1002/sca.4950190707
Islam, N., Hsieh, M.-C., & KeonTaek, K. (2017). Advanced Packaging Need for Automotive In-Cabin Application. 2017 IEEE 67th Electronic Components and Technology Conference (ECTC), 1468–1472. https://doi.org/10.1109/ECTC.2017.159
James, M. R., Morris, W. L., & Cox, B. N. (1990). A high accuracy automated strain- field mapper. Experimental Mechanics, 30(1), 60–67. https://doi.org/10.1007/BF02322704
Jamil, T. (2009). Design and Implementation of a Parking Helper. Proceedings of the World Congress on Engineering and Computer Science, 1, 1–6.
160
Jeong, K., Kahng, A. B., Progler, C. J., (2011). Cost-driven mask strategies considering parametric yield, defectivity, and production volume. Journal of Micro/Nanolithography, MEMS, and MOEMS, 10(3), 033021. https://doi.org/10.1117/1.3633246
Jin, C., Sekhar, V. N., Xiaoyue Bao, Bangtao Chen, Boyu Zheng, & Rui Li. (2013). Antenna-in-Package Design Based on Wafer-Level Packaging With Through Silicon Via Technology. IEEE Transactions on Components, Packaging and Manufacturing Technology, 3(9), 1498–1505. https://doi.org/10.1109/TCPMT.2013.2261855
Jin, P., & Li, X. (2015). Correction of image drift and distortion in a scanning electron microscopy. Journal of Microscopy, 260(3), 268–280. https://doi.org/10.1111/jmi.12293
Joglekar, A., Joshi, D., Khemani, R., Nair, S., & Sahare, S. (2011). Depth Estimation Using Monocular Camera. International Journal of Computer Science and Information Technologies, 2(4), 1758–1763.
Johansen, A., Czajkowski, A., Scobey, M., Egerton, P., & Rance, F. (2017). Thin-Film Interference Filters for LIDAR (pp. 1–13) [White Paper]. Alluxa Corporation. https://www.alluxa.com/learning-center/white-papers/thin-film-interference- filters-for-lidar/
Jokela, M., Kutila, M., & Pyykönen, P. (2019). Testing and Validation of Automotive Point-Cloud Sensors in Adverse Weather Conditions. Applied Sciences, 9(11), 2341. https://doi.org/10.3390/app9112341
Kadooka, K., Kunoo, K., Uda, N., Ono, K., & Nagayasu, T. (2003). Strain Analysis for Moiré Interferometry Using the Two-Dimensional Continuous Wavelet Transform. Experimental Mechanics, 43(1), 45–51. https://doi.org/10.1007/BF02410483
Kammers, A. D., & Daly, S. (2013). Self-Assembled Nanoparticle Surface Patterning for Improved Digital Image Correlation in a Scanning Electron Microscope. Experimental Mechanics, 53(8), 1333–1341. https://doi.org/10.1007/s11340-013- 9734-5
Kang, T., Attard, A., & Abela, J. (2019). CMOS IMAGE SENSOR PACKAGING TECHNOLOGY FOR AUTOMOTIVE APPLICATIONS.
Kanzawa, Y., Kimura, Y., & Naito T. (2011). Human Skin Detection by Visible and Near-Infrared Imaging. IAPR Conference on Machine Vision Applications, 503– 507.
161
Kashani, A., Olsen, M., Parrish, C., & Wilson, N. (2015). A Review of Lidar Radiometric Processing: From Ad Hoc Intensity Correction to Rigorous Radiometric Calibration. Sensors, 15(11), 28099–28128. https://doi.org/10.3390/s151128099
Kashani, A., Olsen, M., Parrish, C., & Wilson, N. (2015). A Review of LIDAR Radiometric Processing: From Ad Hoc Intensity Correction to Rigorous Radiometric Calibration. Sensors, 15(11), 28099–28128. https://doi.org/10.3390/s151128099
Keating T. J., Wolf P. R., & Scarpace F. L. (1975). An Improved Method of Digital Image Correlation. Photogrammetric Engineering & Remote Sensing, 41(8), 993– 1001.
Kendall, A., Martirosyan, H., Dasgupta, S., Henry, P., Kennedy, R., Bachrach, A., & Bry, A. (2017). End-to-End Learning of Geometry and Context for Deep Stereo Regression. 2017 IEEE International Conference on Computer Vision (ICCV), 66–75. https://doi.org/10.1109/ICCV.2017.17
Khan, F., Salahuddin, S., & Javidnia, H. (2020). Deep Learning-Based Monocular Depth Estimation Methods—A State-of-the-Art Review. Sensors, 20(8), 2272. https://doi.org/10.3390/s20082272
Khoo, S.-W., Karuppanan, S., & Tan, C.-S. (2016). A Review of Surface Deformation and Strain Measurement Using Two-Dimensional Digital Image Correlation. Metrology and Measurement Systems, 23(3), 461–480. https://doi.org/10.1515/mms-2016-0028
Kim, T., Song, W., Son, D.-Y., Ono, L. K., & Qi, Y. (2019). Lithium-ion batteries: Outlook on present, future, and hybridized technologies. Journal of Materials Chemistry A, 7(7), 2942–2964. https://doi.org/10.1039/C8TA10513H
Kissinger, D. (2012). Millimeter-Wave Receiver Concepts for 77 GHz Automotive Radar in Silicon-Germanium Technology. Springer US. https://doi.org/10.1007/978-1- 4614-2290-7
Kocic, J., Jovicic, N., & Drndarevic, V. (2018). Sensors and Sensor Fusion in Autonomous Vehicles. 2018 26th Telecommunications Forum (TELFOR), 420– 425. https://doi.org/10.1109/TELFOR.2018.8612054
Kozacek, B., Grauzel, J., & Frivaldsky, M. (2018). The Main Capabilities and Solutions for Different Types of the Image Sensors. 2018 ELEKTRO, 1–5. https://doi.org/10.1109/ELEKTRO.2018.8398278
162 Kutila, M., Pyykonen, P., Holzhuter, H., Colomb, M., & Duthon, P. (2018). Automotive LiDAR performance verification in fog and rain. 2018 21st International Conference on Intelligent Transportation Systems (ITSC), 1695–1701. https://doi.org/10.1109/ITSC.2018.8569624
Kwak, H., & Hubing, T. (2007). An Overview of Advanced Electronic Packaging Technology (Technical Report CVEL-07-001; pp. 1–53). Clemson University . https://www.researchgate.net/publication/260798384
Laforce, F. (2009). Low noise optical receiver using Si APD (S. Jiang, M. J. F. Digonnet, J. W. Glesener, & J. C. Dries, Eds.; p. 721210). https://doi.org/10.1117/12.809071
Lau, J. H. (2019). Recent Advances and Trends in Fan-Out Wafer/Panel-Level Packaging. Journal of Electronic Packaging, 141(4), 040801. https://doi.org/10.1115/1.4043341
Lau, J., Tzeng, P., Lee, C., Zhan, C., Li, M., Cline, J., Saito, K., Hsin, Y., Chang, P., Chang, Y., Chen, J., Chen, S., Wu, C., Chang, H., Chien, C., Lin, C., Ku, T., Lo, R., & Kao, M. (2014). Redistribution Layers (Rdls) for 2.5D/3D IC Integration. Journal of Microelectronics and Electronic Packaging, 11(1), 16–24. https://doi.org/10.4071/imaps.406
Lee, Sehyung, Lee, J. H., Lim, J., & Suh, I. H. (2015). Robust stereo matching using adaptive random walk with restart algorithm. Image and Vision Computing, 37, 1–11. https://doi.org/10.1016/j.imavis.2015.01.003
Lee, Siyeong, An, G. H., & Kang, S.-J. (2018). Deep Chain HDRI: Reconstructing a High Dynamic Range Image From a Single Low Dynamic Range Image. IEEE Access, 6, 49913–49924. https://doi.org/10.1109/ACCESS.2018.2868246
Li, T., Hou, J., Yan, J., Liu, R., Yang, H., & Sun, Z. (2020). Chiplet Heterogeneous Integration Technology—Status and Challenges. Electronics, 9(4), 670. https://doi.org/10.3390/electronics9040670
Lin, B.-Z., & Lin, C.-C. (2016). Pedestrian Detection by Fusing 3D Points and Color Images. International Journal of Networked and Distributed Computing, 4(4), 252. https://doi.org/10.2991/ijndc.2016.4.4.6
Lineback, R. (2019). CMOS Image Sensors Stay on Stairway to Record Revenues (pp. 1– 3) [O-S-D Market Analysis and Forecast]. IC Insights . www.icinsights.com
Liu, S., Liu, L., Tang, J., Yu, B., Wang, Y., & Shi, W. (2019). Edge Computing for Autonomous Driving: Opportunities and Challenges. Proceedings of the IEEE, 107(8), 1697–1716. https://doi.org/10.1109/JPROC.2019.2915983
163 Luo, M. (2006). Displacement/strain measurements using an optical microscope and digital image correlation. Optical Engineering, 45(3), 033605. https://doi.org/10.1117/1.2182108
Luo, P. F., Chao, Y. J., Sutton, M. A., & Peters, W. H. (1993). Accurate measurement of three-dimensional deformations in deformable and rigid bodies using computer vision. Experimental Mechanics, 33(2), 123–132. https://doi.org/10.1007/BF02322488
Luu, L., Wang, Z., Vo, M., Hoang, T., & Ma, J. (2011). Accuracy enhancement of digital image correlation with B-spline interpolation. Optics Letters, 36(16), 3070. https://doi.org/10.1364/OL.36.003070
Lwin, K. K., Tubillo, C. E., Dimaano Panumard, T. J., Suthiwongsunthorn, N., & Sirinorakul, S. (2016). Copper Clip Package for high performance MOSFETs and its optimization. 2016 IEEE 18th Electronics Packaging Technology Conference (EPTC), 123–128. https://doi.org/10.1109/EPTC.2016.7861457
Maddern, W., & Newman, P. (2016). Real-time probabilistic fusion of sparse 3D LIDAR and dense stereo. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2181–2188. https://doi.org/10.1109/IROS.2016.7759342
Mahajan, R., Sankman, R., Patel, N., Kim, D.-W., Aygun, K., Qian, Z., Mekonnen, Y., Salama, I., Sharan, S., Iyengar, D., & Mallik, D. (2016). Embedded Multi-Die Interconnect Bridge (EMIB)—A High Density, High Bandwidth Packaging Interconnect. 2016 IEEE 66th Electronic Components and Technology Conference (ECTC), 557–565. https://doi.org/10.1109/ECTC.2016.201
Maiman, T. H. (1960). Stimulated optical radiation in ruby. Nature, 187(4736), 493–494. https://doi.org/10.1038/187493a0
Major J.S., Plano W.E., Welch D.F., & Scifres D. (1991). SINGLE-MODE InGaAs- GaAs LASER DIODES OPERATING AT 980nm. Electronics Letters, 27(6), 539–540.
Major, B., Fontijne, D., Ansari, A., Sukhavasi, R. T., Gowaikar, R., Hamilton, M., Lee, S., Grzechnik, S., & Subramanian, S. (2019). Vehicle Detection with Automotive Radar Using Deep Learning on Range-Azimuth-Doppler Tensors. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 924–932. https://doi.org/10.1109/ICCVW.2019.00121
Mandel, R. K., Bae, D. G., & Ohadi, M. M. (2018). Embedded Two-Phase Cooling of High Flux Electronics Via Press-Fit and Bonded Feeds Coolers. Journal of Electronic Packaging, 140(3), 031003. https://doi.org/10.1115/1.4039264
164 Mapelli, A. (2020). Microfabricated silicon substrates for pixel detectors assembly and thermal management a.k.a. Silicon Microchannel Cooling Plates. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 958, 162142. https://doi.org/10.1016/j.nima.2019.04.096
Maraghechi, S., Hoefnagels, J. P. M., Peerlings, R. H. J., Rokoš, O., & Geers, M. G. D. (2019). Correction of Scanning eElectron Microscope Imaging Artifacts in a Novel Digital Image Correlation Framework. Experimental Mechanics, 59(4), 489–516. https://doi.org/10.1007/s11340-018-00469-w
Marchant, A. B. (2010). Conspicuity tape for enhanced laser range finding. Optical Engineering, 49(4), 046401. https://doi.org/10.1117/1.3378104
McDonach, A., McKelvie, J., & Walker, C. A. (1980). Stress analysis of fibrous composites using moiré interferometry. Optics and Lasers in Engineering, 1(2), 85–105. https://doi.org/10.1016/0143-8166(80)90002-0
Mello, A. W., Book, T. A., Nicolas, A., Otto, S. E., Gilpin, C. J., & Sangid, M. D. (2017). Distortion Correction Protocol for Digital Image Correlation After Scanning Electron Microscopy: Emphasis on Long Duration and Ex-Situ Experiments. Experimental Mechanics, 57(9), 1395–1409. https://doi.org/10.1007/s11340-017- 0303-1
Metwalli, S. M., Ragab, A. R., Kamel, A. H., & Saheb, A. A. (1987). Determination of plastic stress-strain behavior by digital-image-processing techniques. Experimental Mechanics, 27(4), 414–422. https://doi.org/10.1007/BF02330316
Meyer, G. P., Laddha, A., Kee, E., Vallespi-Gonzalez, C., & Wellington, C. K. (2019). Lasernet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12669–12678. https://doi.org/10.1109/CVPR.2019.01296
Michaelis, M., Berthold, P., Meissner, D., & Wuensche, H.-J. (2017). Heterogeneous Multi-Sensor Fusion for Extended Objects in Automotive Scenarios Using Gaussian Processes and a GMPHD-filter. 2017 Sensor Data Fusion: Trends, Solutions, Applications (SDF), 1–6. https://doi.org/10.1109/SDF.2017.8126351
Micheloni, R., Crippa, L., Zambelli, C., & Olivo, P. (2017). Architectural and Integration Options for 3D NAND Flash Memories. Computers, 6(3), 27. https://doi.org/10.3390/computers6030027 Microscope Units and Objectives No. E14020. (2017). Mitutoyo Corporation.
Miyauchi, K., Mori, K., Otaka, T., Isozaki, T., Yasuda, N., Tsai, A., Sawai, Y., Owada, H., Takayanagi, I., & Nakamura, J. (2020). A Stacked Back Side-Illuminated
165 Voltage Domain Global Shutter CMOS Image Sensor with a 4. 0 μm Multiple Gain Readout Pixel. Sensors, 20(2), 486. https://doi.org/10.3390/s20020486
Miyazaki, J., & Yen, A. (2019). EUV Lithography Technology for High-Volume Production of Semiconductor Devices. Journal of Photopolymer Science and Technology, 32(2), 195–201. https://doi.org/10.2494/photopolymer.32.195
Moore, A. L., & Shi, L. (2014). Emerging challenges and materials for thermal management of electronics. Materials Today, 17(4), 163–174. https://doi.org/10.1016/j.mattod.2014.04.003
Nahory, R. E., Pollack, M. A., Johnston, W. D., & Barns, R. L. (1978). Band gap versus composition and demonstration of Vegard’s law for In 1− x Ga x As y P 1− y lattice matched to InP. Applied Physics Letters, 33(7), 659–661. https://doi.org/10.1063/1.90455
Narote, S. P., Bhujbal, P. N., Narote, A. S., & Dhane, D. M. (2018). A review of recent advances in lane detection and departure warning system. Pattern Recognition, 73, 216–234. https://doi.org/10.1016/j.patcog.2017.08.014
Neumann, L., Karg, M., Zhang, S., Scharfenberger, C., Piegert, E., Mistr, S., Prokofyeva, O., Thiel, R., Vedaldi, A., Zisserman, A., & Schiele, B. (2019). Nightowls: A Pedestrians at Night Dataset. In C. V. Jawahar, H. Li, G. Mori, & K. Schindler (Eds.), Computer Vision – ACCV 2018 (Vol. 11361, pp. 691–705). Springer International Publishing. https://doi.org/10.1007/978-3-030-20887-5_43
Nilagiri Balasubramanian, K. B., & Ramesh, T. (2018). Role, effect, and influences of micro and nano-fillers on various properties of polymer matrix composites for microelectronics: A review. Polymers for Advanced Technologies, 29(6), 1568– 1585. https://doi.org/10.1002/pat.4280
Noh, H.-R. (2019). Analytical solutions for the subsidiary maxima in multiple slit interference. Optical Review, 26(6), 561–567. https://doi.org/10.1007/s10043- 019-00526-2
Noroozi, M., Chandramouli, P., & Favaro, P. (2017). Motion Deblurring in The Wild. In V. Roth & T. Vetter (Eds.), Pattern Recognition (Vol. 10496, pp. 65–77). Springer International Publishing. https://doi.org/10.1007/978-3-319-66709-6_6
Oi, K., Otake, S., Shimizu, N., Watanabe, S., Kunimoto, Y., Kurihara, T., Koyama, T., Tanaka, M., Aryasomayajula, L., & Kutlu, Z. (2014). Development of new 2.5D package with novel integrated organic interposer substrate with ultra-fine wiring and high density bumps. 2014 IEEE 64th Electronic Components and Technology Conference (ECTC), 348–353. https://doi.org/10.1109/ECTC.2014.6897310
166 Oliveira, G. L., Burgard, W., & Brox, T. (2016). Efficient Deep Models for Monocular Road Segmentation. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 4885–4891. https://doi.org/10.1109/IROS.2016.7759717
Pan, B. (2018). Digital image correlation for surface deformation measurement: Historical developments, recent advances and future goals. Measurement Science and Technology, 29(8), 082001. https://doi.org/10.1088/1361-6501/aac55b
Pan, B., Qian, K., Xie, H., & Asundi, A. (2009). Two-dimensional digital image correlation for in-plane displacement and strain measurement: A review. Measurement Science and Technology, 20(6), 062001. https://doi.org/10.1088/0957-0233/20/6/062001
Park, Y., Yun, S., Won, C., Cho, K., Um, K., & Sim, S. (2014). Calibration between Color Camera and 3D LIDAR Instruments with a Polygonal Planar Board. Sensors, 14(3), 5333–5353. https://doi.org/10.3390/s140305333
Pedrotti L. S. (2008). Basic Physical Optics. In FUNDAMENTALS OF PHOTONICS (Vol. TT79, pp. 117–166). International Society for Optics and Photonics (SPIE).
Peters, W. H., & Ranson, W. F. (1982). Digital imaging techniques in experimental stress analysis. Optical Engineering, 21(3). https://doi.org/10.1117/12.7972925
Peters, W. H., Ranson, W. F., Sutton, M. A., Chu, T. C., & Anderson, J. (1983). Application of digital correlation methods to rigid body mechanics. Optical Engineering, 22(6). https://doi.org/10.1117/12.7973231
Pevec, D., Babic, J., & Podobnik, V. (2019). Electric Vehicles: A Data Science Perspective Review. Electronics, 8(10), 1190. https://doi.org/10.3390/electronics8101190
Piatek, S., & Li, J. (2017, November 9). A photonics guide to the autonomous vehicle market. Laser Focus World, 53(11), 28–31.
Pinggera, P., Pfeiffer, D., Franke, U., & Mester, R. (2014). Know Your Limits: Accuracy of Long Range Stereoscopic Object Measurements in Practice. In D. Fleet, T. Pajdla, B. Schiele, & T. Tuytelaars (Eds.), Computer Vision – ECCV 2014 (Vol. 8690, pp. 96–111). Springer International Publishing. https://doi.org/10.1007/978- 3-319-10605-2_7
Pop, E., Sinha, S., & Goodson, K. E. (2006). Heat Generation and Transport in Nanometer-Scale Transistors. Proceedings of the IEEE, 94(8), 1587–1601. https://doi.org/10.1109/JPROC.2006.879794
167 Popescu, A., Schorstein, K., & Walther, T. (2004). A novel approach to a Brillouin– LIDAR for remote sensing of the ocean temperature. Applied Physics B, 79(8), 955–961. https://doi.org/10.1007/s00340-004-1666-4
Post, D. (1968). New optical methods of moiré fringe multiplication. Experimental Mechanics, 8(2), 63–68. https://doi.org/10.1007/BF02326128
Post, D. (1979). Moiré interferometry with white light. Applied Optics, 18(24), 4163. https://doi.org/10.1364/AO.18.004163
Prasad, D. (May-June). Automotive OSAT challenges. Chip Scale Review, 23(3), 18–19.
Premachandran, C., Tran-Quinn, T., Burrell, L., & Justison, P. (2019). A Comprehensive Wafer Level Reliability Study on 65nm Silicon Interposer. 2019 IEEE International Reliability Physics Symposium (IRPS), 1–8. https://doi.org/10.1109/IRPS.2019.8720515
Qiu, Y., Xie, Y., Wang, W., Liu, W., Kuang, L., Bai, X., Hu, M., & Ho, J. (2019). Ultra- High-Power and High-Efficiency 905nm Pulsed Laser for LiDAR. 2019 IEEE 4th Optoelectronics Global Conference (OGC), 32–35. https://doi.org/10.1109/OGC.2019.8925087
Rajendran, B., Shenoy, R. S., Witte, D. J., Chokshi, N. S., DeLeon, R. L., Tompa, G. S., & Pease, R. F. W. (2007). Low Thermal Budget Processing for Sequential 3-D IC Fabrication. IEEE Transactions on Electron Devices, 54(4), 707–714. https://doi.org/10.1109/TED.2007.891300
Ranftl, R., Vineet, V., Chen, Q., & Koltun, V. (2016). Dense Monocular Depth Estimation in Complex Dynamic Scenes. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4058–4066. https://doi.org/10.1109/CVPR.2016.440
Rasshofer, R. H., Spies, M., & Spies, H. (2011). Influences of weather phenomena on automotive laser radar systems. Advances in Radio Science, 9, 49–60. https://doi.org/10.5194/ars-9-49-2011
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You Only Look Once: Unified, Real-Time Object Detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 779–788. https://doi.org/10.1109/CVPR.2016.91
Reich, G. (2005). Near-infrared spectroscopy and imaging: Basic principles and pharmaceutical applications. Advanced Drug Delivery Reviews, 57(8), 1109– 1143. https://doi.org/10.1016/j.addr.2005.01.020
168 Reu, P. L., Sweatt, W., Miller, T., & Fleming, D. (2015). Camera System Resolution and its Influence on Digital Image Correlation. Experimental Mechanics, 55(1), 9–25. https://doi.org/10.1007/s11340-014-9886-y
Roos, F., Kellner, D., Dickmann, J., & Waldschmidt, C. (2016). Reliable Orientation Estimation of Vehicles in High-Resolution Radar Images. IEEE Transactions on Microwave Theory and Techniques, 64(9), 2986–2993. https://doi.org/10.1109/TMTT.2016.2586476
Rosique, F., Navarro, P. J., Fernández, C., & Padilla, A. (2019). A Systematic Review of Perception System and Simulators for Autonomous Vehicles Research. Sensors, 19(3), 648. https://doi.org/10.3390/s19030648
Rump, M., Müller, G., Sarlette, R., Koch, D., & Klein, R. (2008). Photo-realistic Rendering of Metallic Car Paint from Image-Based Measurements. Computer Graphics Forum, 27(2), 527–536. https://doi.org/10.1111/j.1467- 8659.2008.01150.x
Saxena, A., Chung, S. H., & Ng, A. Y. (2007). 3-D Depth Reconstruction From a Single Still Image. International Journal of Computer Vision, 76(1), 53–69. https://doi.org/10.1007/s11263-007-0071-y
Schaper, L. W., Burkett, S. L., Spiesshoefer, S., Vangara, G. V., Rahman, Z., & Polamreddy, S. (2005). Architectural implications and process development of 3- D VLSI Z-axis interconnects using through silicon vias. IEEE Transactions on Advanced Packaging, 28(3), 356–366. https://doi.org/10.1109/TADVP.2005.853271 Schreier, H. W. (2000). Systematic errors in digital image correlation caused by intensity interpolation. Optical Engineering, 39(11), 2915. https://doi.org/10.1117/1.1314593
Schultz, P., Cumby, B., & Heikenfeld, J. (2012). Investigation of five types of switchable retroreflector films for enhanced visible and infrared conspicuity applications. Applied Optics, 51(17), 3744. https://doi.org/10.1364/AO.51.003744
Sciammarella, C. A. (1969). Moiré-fringe multiplication by means of filtering and a wave-front reconstruction process. Experimental Mechanics, 9(4), 179–185. https://doi.org/10.1007/BF02326567
Seubert, C. M., Nichols, M. E., Frey, J., Shtein, M., & Thouless, M. D. (2016). The characterization and effects of microstructure on the appearance of platelet– polymer composite coatings. Journal of Materials Science, 51(5), 2259–2273. https://doi.org/10.1007/s10853-015-9528-5
169 Shahian Jahromi, B., Tulabandhula, T., & Cetin, S. (2019b). Real-Time Hybrid Multi- Sensor Fusion Framework for Perception in Autonomous Vehicles. Sensors, 19(20), 4357. https://doi.org/10.3390/s19204357
Shen, W.-W., & Chen, K. N. (2017). Three-Dimensional Integrated Circuit (3D IC) Key Technology: Through-Silicon Via (TSV). Nanoscale Research Letters, 12(1), 56. https://doi.org/10.1186/s11671-017-1831-4
Shen, W.-W., Kao, M.-J., Chen, K.-N., Lin, Y.-M., Chen, S.-C., Chang, H.-H., Chang, T.-C., Lo, W.-C., Lin, C.-C., Chou, Y.-F., & Kwai, D.-M. (2018). 3-D Stacked Technology of DRAM-Logic Controller Using Through-Silicon Via (TSV). IEEE Journal of the Electron Devices Society, 6, 396–402. https://doi.org/10.1109/JEDS.2018.2815344
Shih, J.-Y., Huang, W.-C., Ko, C.-T., Yang, Z., Hu, S.-H., Leu, J., Chou, K. C., & Chen, K.-N. (2014). Adhesion Investigation Between Metal and Benzocyclobutene (BCB) Polymer Dielectric Materials in 3-D Integration Applications. IEEE Transactions on Device and Materials Reliability, 14(3), 914–920. https://doi.org/10.1109/TDMR.2014.2343793
Siow, K. S. (2014). Are Sintered Silver Joints Ready for Use as Interconnect Material in Microelectronic Packaging? Journal of Electronic Materials, 43(4), 947–961. https://doi.org/10.1007/s11664-013-2967-3
Smolyanskiy, N., Kamenev, A., & Birchfield, S. (2018). On The Importance of Stereo for Accurate Depth Estimation: An Efficient Semi-Supervised Deep Neural Network Approach. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1120–11208. https://doi.org/10.1109/CVPRW.2018.00147
Sohaib, A., Broadbent, L., Farooq, A. R., Smith, L. N., & Smith, M. L. (2017). BRDF of human skin in the visible spectrum. Sensor Review, 37(4), 390–395. https://doi.org/10.1108/SR-11-2016-0258
Sridhar, A., Sabry, M. M., & Atienza, D. (2014). A Semi-Analytical Thermal Modeling Framework for Liquid-Cooled ICs. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 33(8), 1145–1158. https://doi.org/10.1109/TCAD.2014.2323194
Su, Y.-F., Chiang, K.-N., & Liang, S. Y. (2017). Design and Reliability Assessment of Novel 3D-IC Packaging. Journal of Mechanics, 33(2), 193–203. https://doi.org/10.1017/jmech.2016.82
170 Sun, Y., & Pang, J. H. L. (2008). Digital image correlation for solder joint fatigue reliability in microelectronics packages. Microelectronics Reliability, 48(2), 310– 318. https://doi.org/10.1016/j.microrel.2007.03.007
Sutton, M., Mingqi, C., Peters, W., Chao, Y., & McNeill, S. (1986). Application of an optimized digital correlation method to planar deformation analysis. Image and Vision Computing, 4(3), 143–150. https://doi.org/10.1016/0262-8856(86)90057-0
Sutton, M., Wolters, W., Peters, W., Ranson, W., & McNeill, S. (1983). Determination of displacements using an improved digital correlation method. Image and Vision Computing, 1(3), 133–139. https://doi.org/10.1016/0262-8856(83)90064-1
Tan, K., & Cheng, X. (2016). Correction of Incidence Angle and Distance Effects on TLS Intensity Data Based on Reference Targets. Remote Sensing, 8(3), 251. https://doi.org/10.3390/rs8030251
Tang, T., Fang, B.-S., Ho, D., Ma, B. H., Tsai, J., & Wang, Y.-P. (2019b). Innovative Flip Chip Package Solutions for Automotive Applications. 2019 IEEE 69th Electronic Components and Technology Conference (ECTC), 1432–1436. https://doi.org/10.1109/ECTC.2019.00220
Tao, A., Sapra, K., & Catanzaro, B. (2020). Hierarchical Multi-Scale Attention for Semantic Segmentation. ArXiv:2005.10821 [Cs]. http://arxiv.org/abs/2005.10821
Temple, D. S., Lueck, M. R., Malta, D., & Vick, E. P. (2015). Scaling of three- dimensional interconnect technology incorporating low temperature bonds to pitches of 10 µm for infrared focal plane array applications. Japanese Journal of Applied Physics, 54(3), 030202. https://doi.org/10.7567/JJAP.54.030202
Theil, J. A., Mirkarimi, L., Fountain, G., Gao, G., & Katkar, R. (2019). Recent Developments in Fine Pitch Wafer-To-Wafer Hybrid Bonding with Copper Interconnect. 2019 International Wafer Level Packaging Conference (IWLPC), 1– 6. https://doi.org/10.23919/IWLPC.2019.8913862
Thrun, S., Montemerlo, M., Dahlkamp, H., Stavens, D., Aron, A., Diebel, J., Fong, P., Gale, J., Halpenny, M., Hoffmann, G., Lau, K., Oakley, C., Palatucci, M., Pratt, V., Stang, P., Strohband, S., Dupont, C., Jendrossek, L.-E., Koelen, C., … Mahoney, P. (2006). Stanley: The Robot That Won the DARPA Grand Challenge. Journal of Field Robotics, 23(9), 661–692. https://doi.org/10.1002/rob.20147
Tian, T., Morusupalli, R., Shin, H., Son, H.-Y., Byun, K.-Y., Joo, Y.-C., Caramto, R., Smith, L., Shen, Y.-L., Kunz, M., Tamura, N., & Budiman, A. S. (2016). On the Mechanical Stresses of Cu Through-Silicon Via (TSV) Samples Fabricated by SK Hynix vs. Sematech – Enabling Robust and Reliable 3-D Interconnect/Integrated
171 Circuit (IC) Technology. Procedia Engineering, 139, 101–111. https://doi.org/10.1016/j.proeng.2015.09.242
Tollenar D. (1945). Moiré interferentievershijnselen pij rasterdruk. Technick, 4.
Tong, J. (2018). Full-field characterization of crack tip deformation and fatigue crack growth using digital image correlation-a review. Fatigue & Fracture of Engineering Materials & Structures, 41(9), 1855–1869. https://doi.org/10.1111/ffe.12859
Tong, W. (2005). An Evaluation of Digital Image Correlation Criteria for Strain Mapping Applications. Strain, 41(4), 167–175. https://doi.org/10.1111/j.1475- 1305.2005.00227.x
Torralba, A., & Oliva, A. (2002). Depth Estimation from Image Structure. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9), 1226–1238. https://doi.org/10.1109/TPAMI.2002.1033214
Tseng, A. A. (2004). Recent developments in micromilling using focused ion beam technology. Journal of Micromechanics and Microengineering, 14(4), R15–R34. https://doi.org/10.1088/0960-1317/14/4/R01
Tseng, C.-F., Liu, C.-S., Wu, C.-H., & Yu, D. (2016). INFO (Wafer Level Integrated Fan-Out) technology. 2016 IEEE 66th Electronic Components and Technology Conference (ECTC), 1–6. https://doi.org/10.1109/ECTC.2016.65
Un-Noor, F., Padmanaban, S., Mihet-Popa, L., Mollah, M., & Hossain, E. (2017). A Comprehensive Study of Key Electric Vehicle (EV) Components, Technologies, Challenges, Impacts, and Future Direction of Development. Energies, 10(8), 1217. https://doi.org/10.3390/en10081217
Van Brummelen, J., O’Brien, M., Gruyer, D., & Najjaran, H. (2018). Autonomous vehicle perception: The technology of today and tomorrow. Transportation Research Part C: Emerging Technologies, 89, 384–406. https://doi.org/10.1016/j.trc.2018.02.012
Vaudelle, F., L’Huillier, J.-P., & Askoura, M. L. (2017). Light source distribution and scattering phase function influence light transport in diffuse multi-layered media. Optics Communications, 392, 268–281. https://doi.org/10.1016/j.optcom.2017.02.001
Velichko, S., Johnson, S., Pates, D., Silsby, C., Hoekstra, C., Mentzer, R., & Beck, J. (2017). 140 dB Dynamic Range Sub-electron Noise Floor Image Sensor (Technical Report No. R34; pp. 294–297). ON Semiconductor.
172 Vendroux, G., & Knauss, W. G. (1998). Submicron Deformation Field Measurements: Part 2. Improved Digital Image Correlation. Experimental Mechanics, 38(2), 86– 92. https://doi.org/10.1007/BF02321649
Vo, B.-N., & Ma, W.-K. (2006). The Gaussian Mixture Probability Hypothesis Density Filter. IEEE Transactions on Signal Processing, 54(11), 4091–4104. https://doi.org/10.1109/TSP.2006.881190
Volkert, C. A., & Minor, A. M. (2007). Focused Ion Beam Microscopy and Micromachining. MRS Bulletin, 32(5), 389–399. https://doi.org/10.1557/mrs2007.62
Vyas B.G., Sheth N.R., & Keshwani M.P. (2015). Review: Thermal Contact Resistance of Metallic Composite Pair. International Journal of Application or Innovation in Engineering & Management, 4(2), 193–200.
Wan, G., Yang, X., Cai, R., Li, H., Zhou, Y., Wang, H., & Song, S. (2018). Robust and Precise Vehicle Localization Based on Multi-Sensor Fusion in Diverse City Scenes. 2018 IEEE International Conference on Robotics and Automation (ICRA), 4670–4677. https://doi.org/10.1109/ICRA.2018.8461224
Wang, S., Yin, Y., Hu, C., & Rezai, P. (2018). 3D Integrated Circuit Cooling with Microfluidics. Micromachines, 9(6), 287. https://doi.org/10.3390/mi9060287
Wang, X., Xu, L., Sun, H., Xin, J., & Zheng, N. (2016). On-Road Vehicle Detection and Tracking Using MMW Radar and Monovision Fusion. IEEE Transactions on Intelligent Transportation Systems, 17(7), 2075–2084. https://doi.org/10.1109/TITS.2016.2533542
Wang, Y., Wang, P., Yang, Z., Luo, C., Yang, Y., & Xu, W. (2019). Unos: Unified Unsupervised Optical-Flow and Stereo-Depth Estimation by Watching Videos. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8063–8073. https://doi.org/10.1109/CVPR.2019.00826
Wang, Z. (2019). Microsystems using three-dimensional integration and TSV technologies: Fundamentals and applications. Microelectronic Engineering, 210, 35–64. https://doi.org/10.1016/j.mee.2019.03.009
Warren, M. E., Block, M. K., Dacha, P., Carsonn, R. F., Podva, D., Helms, C. J., & Maynard, J. (2018). Low-divergence high-power VCSEL arrays for lidar application. In K. D. Choquette & C. Lei (Eds.), Vertical-Cavity Surface-Emitting Lasers XXII (p. 14). SPIE. https://doi.org/10.1117/12.2290937
Weissman, E. M., & Post, D. (1982). Moire interferometry near the theoretical limit. Applied Optics, 21(9), 1621. https://doi.org/10.1364/AO.21.001621
173 Wellard R. & Sheppard B.M. (1948). Displacement measurement by mechanical interferometry. Proceedings SESA II, 1.
Wentker, M., Greenwood, M., & Leker, J. (2019). A Bottom-Up Approach to Lithium- Ion Battery Cost Modeling With a Focus on Cathode Active Materials. Energies, 12(3), 504. https://doi.org/10.3390/en12030504
Wesling, P. (2020). The Heterogeneous Integration Roadmap: Enabling Technology for Systems of the Future. 2020 Pan Pacific Microelectronics Symposium (Pan Pacific), 1–4. https://doi.org/10.23919/PanPacific48324.2020.9059347
Wojke, N., Bewley, A., & Paulus, D. (2017). Simple Online and Realtime Tracking with a Deep Association Metric. 2017 IEEE International Conference on Image Processing (ICIP), 3645–3649. https://doi.org/10.1109/ICIP.2017.8296962
Won, Y., Gao, Y., Panzer, M. A., Dogbe, S., Pan, L., Kenny, T. W., & Goodson, K. E. (2012). Mechanical characterization of aligned multi-walled carbon nanotube films using microfabricated resonators. Carbon, 50(2), 347–355. https://doi.org/10.1016/j.carbon.2011.08.009
Wu, B., & Han, B. (2018). Advanced Mechanical/Optical Configuration of Real-Time Moiré Interferometry For Thermal Deformation Analysis of Fan-Out Wafer Level Package. IEEE Transactions on Components, Packaging and Manufacturing Technology, 8(5), 764–772. https://doi.org/10.1109/TCPMT.2018.2805873
Wygant, I. O., Kupnik, M., Windsor, J. C., Wright, W. M., Wochner, M. S., Yaralioglu, G. G., Hamilton, M. F., & Khuri-Yakub, B. T. (2009). 50 kHz Capacitive Micromachined Ultrasonic Transducers for Generation of Highly Directional Sound with Parametric Arrays. IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control, 56(1), 193–203. https://doi.org/10.1109/TUFFC.2009.1019
Xiao, Z., Yao, M., Yu, D., Zhou, M., Dai, F., Fan, J., Xiang, M., & Zhang, W. (2017). Development and reliability study of 3D WLCSP for CMOS image sensor using vertical TSVs with 3:1 aspect ratio. Microsystem Technologies, 23(10), 4879– 4889. https://doi.org/10.1007/s00542-016-3267-7
Yang, D., Jiang, K., Zhao, D., Yu, C., Cao, Z., Xie, S., Xiao, Z., Jiao, X., Wang, S., & Zhang, K. (2018). Intelligent and connected vehicles: Current status and future perspectives. Science China Technological Sciences, 61(10), 1446–1471. https://doi.org/10.1007/s11431-017-9338-1
Yang, G., Song, X., Huang, C., Deng, Z., Shi, J., & Zhou, B. (2019). Drivingstereo: A Large-Scale Dataset For Stereo Matching in Autonomous Driving Scenarios.
174 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 899–908. https://doi.org/10.1109/CVPR.2019.00099
Yang, H. D., Shih, C., Yang, S., Lee, T. (2009). Broad-area InGaNAs/GaAs quantum- well lasers in the 1200 nm range. 59th Electronic Components and Technology Conference (ECTC), 820–826. https://doi.org/10.1109/ECTC.2009.5074106
Yang, L., & Bernstein, J. B. (2009). Failure rate estimation of known failure mechanisms of electronic packages. Microelectronics Reliability, 49(12), 1563–1572. https://doi.org/10.1016/j.microrel.2009.06.005
Yeh, L. T. (1995). Review of Heat Transfer Technologies in Electronic Equipment. Journal of Electronic Packaging, 117(4), 333–339 https://doi.org/10.1115/1.2792113
Yoon, K. J., Kim, Y., & Hwang, C. S. (2019). What will come after V‐NAND—Vertical Resistive Switching Memory? Advanced Electronic Materials, 5(9), 1800914. https://doi.org/10.1002/aelm.201800914
Yurtsever, E., Lambert, J., Carballo, A., & Takeda, K. (2020). A Survey of Autonomous Driving: Common Practices and Emerging Technologies. IEEE Access, 8, 58443– 58469. https://doi.org/10.1109/ACCESS.2020.2983149
Zhang, W., Zhu, F., Wang, S., Lu, P., & Wu, X. (2019). An accurate method to calibrate shadow moiré measurement sensitivity. Measurement Science and Technology, 30(12), 125021. https://doi.org/10.1088/1361-6501/ab1e2d
Zhang, X., Lin, J. K., Wickramanayaka, S., Zhang, S., Weerasekera, R., Dutta, R., Chang, K. F., Chui, K.-J., Li, H. Y., Wee Ho, D. S., Ding, L., Katti, G., Bhattacharya, S., & Kwong, D.-L. (2015). Heterogeneous 2.5D integration on through silicon interposer. Applied Physics Reviews, 2(2), 021308. https://doi.org/10.1063/1.4921463
Zhang, Y., & Mao, J. (2019). An Overview of the Development of Antenna-in-Package Technology for Highly Integrated Wireless Devices. Proceedings of the IEEE, 107(11), 2265–2280. https://doi.org/10.1109/JPROC.2019.2933267
Zhang, Y.-F., Ren, Y.-J., & Bai, S.-L. (2018). Vertically aligned graphene film/epoxy composites as heat dissipating materials. International Journal of Heat and Mass Transfer, 118, 510–517. https://doi.org/10.1016/j.ijheatmasstransfer.2017.11.014
Zhou, K., Meng, X., & Cheng, B. (2020). Review of Stereo Matching Algorithms Based on Deep Learning. Computational Intelligence and Neuroscience, 2020, 1–12. https://doi.org/10.1155/2020/8562323
175 Zhu, F., Lin, X., Zhang, W., Fan, J., & Liu, S. (2018). Morphology Evaluation of Microelectronic Packaging Substrates Using Shadow Moiré Technique. IEEE Access, 6, 33099–33110. https://doi.org/10.1109/ACCESS.2018.2842199
Zhuo, S., & Sim, T. (2011). Defocus map estimation from a single image. Pattern Recognition, 44(9), 1852–1858. https://doi.org/10.1016/j.patcog.2011.03.009
176