Optical Camera Communications: Principles, Modulations, Potential and Challenges

electronics

Review Optical Camera Communications: Principles, Modulations, Potential and Challenges

Willy Anugrah Cahyadi 1 , Yeon Ho Chung 2,* , Zabih Ghassemlooy 3 and Navid Bani Hassan 3

1 School of Electrical Engineering (SEE) and The Center for Advanced Intelligent Communications (AICOMS), Telkom University, Bandung 40257, Indonesia; [email protected] 2 Department of Information and Communications Engineering, Pukyong National University, Busan 48513, Korea 3 Mathematics, Physics and Electrical Engineering Department, Northumbria University, Newcastle upon Tyne NE1 8ST, UK; [email protected] (Z.G.); [email protected] (N.B.H.) * Correspondence: [email protected]

Received: 20 July 2020; Accepted: 17 August 2020; Published: 19 August 2020

Abstract: Optical wireless communications (OWC) are emerging as cost-eﬀective and practical solutions to the congested radio frequency-based wireless technologies. As part of OWC, optical camera communications (OCC) have become very attractive, considering recent developments in cameras and the use of ﬁtted cameras in smart devices. OCC together with visible light communications (VLC) is considered within the framework of the IEEE 802.15.7m standardization. OCCs based on both organic and inorganic light sources as well as cameras are being considered for low-rate transmissions and localization in indoor as well as outdoor short-range applications and within the framework of the IEEE 802.15.7m standardization together with VLC. This paper introduces the underlying principles of OCC and gives a comprehensive overview of this emerging technology with recent standardization activities in OCC. It also outlines the key technical issues such as mobility, coverage, interference, performance enhancement, etc. Future research directions and open issues are also presented.

Keywords: optical camera communications; visible light communications; camera receiver; wireless communications; LED-camera

1. Introduction The rapid increase in the data being generated and the growing demand for capacity have pushed existing congested radio frequency (RF)-based wireless communications to their limits. In addition, we are witnessing incommensurate spectral efficiency as well as bandwidth with increasing traffic demands and requirements in the RF domain [1]. From a futuristic perspective, it is foreseen that high-rate and high-capacity RF communications coexist with more affordable, offloading, and unlimited bandwidth wireless communications. As an attractive complementary solution, optical wireless communications (OWC) have been considered for easing the bandwidth usage pressure on RF wireless systems in a harmonious manner. OWC covers the entire optical spectrum of ultraviolet (UV), visible and infrared (IR) [1,2], see Figure1, thus o ffering a massive theoretical bandwidth of 20 THz (IR), 320 THz (visible) and 30 PHz (UV) compared with the bandwidth of 300 GHz in the RF technology [1,3,4]. Within the OWC umbrella, there are a number of transmission schemes for both indoor and outdoor environments including infrared communications (IRC), free space optics (FSO), visible light communications (VLC) and ultraviolet communications (UVC) [1,3–6], as shown in

Electronics 2020, 9, 1339; doi:10.3390/electronics9091339 www.mdpi.com/journal/electronics Electronics 2020, 9, 1339 2 of 44 Electronics 2020, 9, x FOR PEER REVIEW 2 of 45

Figure2. The VLC and IRC are generally considered for short-range indoor applications, while FSO rangeElectronics outdoor 2020, 9 ,environments. x FOR PEER REVIEW Note that FSO can also use the visible band for short to medium 2range of 45 transmissions.(using the IR band) and UVC (using the C band) are predominantly employed for medium-long rangerange outdooroutdoor environments. environments. Note Note that that FSO FSO can canalso also use the use visible the visible band bandfor short for to short medium to medium range rangetransmissions. transmissions.

Figure 1. The electromagnetic spectrum. Figure 1. The electromagnetic spectrum. Figure 1. The electromagnetic spectrum.

Figure 2. Overview of optical wireless communications (OWC) transmission schemes. Figure 2. Overview of optical wireless communications (OWC) transmission schemes. The VLCFigure technology, 2. Overview a subsetof optical of wireless OWC ascommunications defined in the (OWC) IEEE transmission 802.15.7, has schemes. specifically been envisionedThe VLC for technology, high capacity a subset mobile of data OWC networks as defined since in 2000, the toIEEE complement 802.15.7, has the RF-basedspecifically mobile been envisionedcommunicationsThe VLC for hightechnology, [1, 7capacity–10]. The a mobilesubset popularity ofdata OWC networks of VLC as defined development since in2000, the toIEEE is drivencomplement 802.15.7, by its has etheffi ciency specificallyRF-based in utilizing mobile been communicationsexistingenvisioned light-emitting for high [1,7–10]. capacity diodes The mobile popularity (LEDs)-based data ofnetworks VLC lighting development since infrastructure 2000, tois drivencomplement as theby its transmitter theefficiency RF-based (Tx) in utilizing mobile to offer existingsignificantlycommunications light-emitting higher [1,7–10]. luminous diodes The e popularity(LEDfficacy,s)-based data of communications VLC lighting development infrastructure and is indoordriven as the localizationby itstransmitter efficiency [1,9 (Tx)]. in In utilizing contrastto offer significantlytoexisting other lighting light-emitting higher sources, luminous diodes LEDs efficacy, can(LED bes)-based switched data communications lighting at a very infrastructure high speed,and indoor thusas the localization facilitating transmitter data [1,9]. (Tx) transmission In tocontrast offer tothatsignificantly other can belighting picked higher sources, up luminous using LEDs photodiodes efficacy, can databe (PDs) switchedcommunications or image at a sensors very and indoorhigh (ISs). speed, localization thus [1,9].facilitating In contrast data transmissionto otherA VLC lighting system that can sources, employing be picked LEDs an up IS, canusing i.e., be aphotodiodes camera,switched is knownat (PDs) a very asor opticalimagehigh speed,sensors camera thus (ISs). communication facilitating (OCC)data accordingtransmissionA VLC to systemthe that standards, can employing be picked which an up IS, requiresusing i.e., a photodiodes camera, no hardware is known (PDs) modifications as or optical image camerasensors [5,11, 12communication(ISs).]. The IS in a camera, (OCC) accordingwhichA is VLC a to massive systemthe standards, matrixemploying ofwhichµ an-scale IS, requires i.e., PDs, a camera, togetherno hard isware withknown modifications a lens,as optical can becamera [5,11,12]. used communication to provideThe IS in an a imaging camera,(OCC) according to the standards, which requires no hardware modifications [5,11,12]. The IS in a camera, whichmassive is a multiple-input massive matrix multiple-output of μ-scale PDs,(MIMO) together system,with a lens, i.e., enablingcan be used spatial to provide separation an imaging of light which is a massive matrix of μ-scale PDs, together with a lens, can be used to provide an imaging massivesources onmultiple-input the IS. In addition, multiple-output the camera (IS)-based(MIMO) sy receiverstem, i.e., (RX) enabling is capable spatial of capturing separation information of light sourcesmassive on multiple-input the IS. In addition, multiple-output the camera (IS)-based(MIMO) sy receiverstem, i.e., (RX) enabling is capable spatial of capturing separation information of light notsources only onfrom the multiple IS. In addition, TXs (i.e., the LEDs) camera but (IS)-based also from receiver digital (RX) displays is capable such of as capturing liquid crystal information display (LCD)not only and from organic multiple LED (OLED)-basedTXs (i.e., LEDs) screens but also [5,12]. from digital displays such as liquid crystal display (LCD) and organic LED (OLED)-based screens [5,12].

Electronics 2020, 9, 1339 3 of 44 not only from multiple TXs (i.e., LEDs) but also from digital displays such as liquid crystal display (LCD) and organic LED (OLED)-based screens [5,12]. Furthermore, an OCC image capturing increases the communication range in outdoor environments up to a kilometer, due to the use of image processing, long sampling duration and lenses [5]. Nevertheless, most OCC schemes reported in the literature are intended for indoor applications because of smart-devices availability and a lower intensity of ambient light in indoor environment [5]. With regard to the OCC, it is important to note that OCC-based systems are mainly targeted for a low-rate transmission, due to the reception-sampling rate, which is determined by the camera frame rate FR, which is typically in the range of 30 to 60 frames per second (fps) at a common resolution of 1920 1080 pixels. However, with current advances made in cameras, for the same resolution, F of × R new smartphone cameras can reach up to 960 fps [13]. Different from VLC that uses a PD to process the received optical intensity, OCC is based on capturing the source using the IS, i.e., a two-dimension (2D) RX containing millions of pixels, and carrying out a significant amount of image processing to acquire the information. Therefore, the data transmission rate of OCC is mostly limited to either kilobits per second (Kbps) or megabits per second (Mbps) in contrast to multi-gigabit per second (Gbps) transmission rates in PD-based VLC systems [1,5,11,14]. The IEEE 802.15.7 standard was initially formulated in 2011 for VLC without considering the IS-based RX [1,11]. In 2014, however, OCC was included in the revised IEEE 802.15.7r1 standard [11,15], which was managed by the TG7r1 task group and covered wider light spectra such as near infrared (NIR) and near UV [5,11]. OCC is currently organized by the updated task group TG7m that finalized the initial draft in 2018 [14,16]. The task group is a joint consortium of academic researchers from multiple countries including United States, South Korea, Turkey, Taiwan and China [16]. Due to the inclusion of nearly the entire optical spectrum in the standardization, there are a wide range of potential applications for OCC within the context of Internet-of-things (IoT) such as augmented reality, color code, device-to-device communications, digital signage, exhibition/store service, in-flight service, indoor positioning, vehicular communications and underwater/seaside communications [5,17,18]. At a global level, a number of companies including Fraunhofer Heinrich Hertz Institute (Germany), Intel (United States), Panasonic (Japan), Casio (Japan), Huawei Technologies (China), Electronics and Telecommunications Research Institute (Korea), LG Electronics (Korea) and China Telecom (China) have been working on the application of OCC [5]. In recent years, the interest in OCC has been growing, judging from four surveys published in the past four years, mostly driven by the early standardization activity (i.e., IEEE 802.15.7r1) [5,11,17,18]. The first survey focused on key technologies used in OCC [11], while the second survey investigated various implementation challenges associated with OCC with no comprehensive structures [17]. The third survey covered concise performance analysis of OCC intended for inclusion in the fifth generation (5G) wireless networks [5]. Finally, the fourth survey, which was the most recently published work, is a broad survey of OCC covering localization, navigation, motion capture, and some insights into future research direction [18,19]. This survey, however, only introduced several challenges and future research directions in the limited sense. This paper, on the other hand, presents a comprehensive survey on OCC in such a way that it focuses heavily on the principles and modulations of OCC to date, together with potential applications and future research directions in mind. It also outlines key technologies and standardization as well as modulation schemes that have been proposed and investigated. The paper aims to enhance practical implementations of OCC, while disseminating promising OCC technologies and future challenges. The rest of the paper, as illustrated in Figure3, is organized as follows. Section2 describes OCC in detail and Section3 reviews existing modulations of OCC. Section4 presents various performance enhancements in terms of mobility, coverage, and interference. Section5 is dedicated to recent OCC schemes for both indoor and outdoor applications. Section6 discusses future potential developments, Electronics 2020, 9, 1339 4 of 44 enhancements, and challenges in OCC. Finally, Section7 closes the paper with a brief summary. ElectronicsIn addition, 2020, Appendix 9, x FOR PEERA shows REVIEW a list of both abbreviations and acronyms used throughout the paper.4 of 45

Figure 3. The paper organization. 2. Optical Camera Communication 2. Optical Camera Communication 2.1. Principles 2.1. Principles OCC employs either a single or multiple-camera setup as the RX, which can now be found in mostOCC mobile employs devices either such a as single mobile or phones, multiple-camera smart watches, setup laptops,as the RX, tablets, which as can well now as vehiclesbe found and in mostbuildings mobile [2]. devices Therefore, such theas mobile existing phones, LED lighting smart wa fixturestches, andlaptops, widely tablets, used as cameras, well as vehicles i.e., cheaper and buildingsand off-the-shelf [2]. Therefore, devices, the can existing be employed LED lighting in OCC. fixtures While and the majoritywidely used of VLC cameras, systems i.e., are cheaper limited and to off-the-shelfutilizing the visibledevices, light can spectrum, be employed OCC employingin OCC. Wh diilefferent the typesmajority of camera of VLC can systems utilize theare entirelimited light to utilizingspectrum the from visible visible light light, spectrum, NIR to UV-C OCC band.employing In OCC, different the camera types captures of camera images can /utilizevideo streamsthe entire of lightthe intensity spectrum modulated from visible light light, sources, NIR i.e.,to UV-C a single band. LED, In multiple OCC, the LEDs camera and digitalcaptures display images/video screens, streamsand extracts of the the intensity information modulated by means light ofsources, image i. processinge., a single LED, [17]. multiple Thus, camera-based LEDs and digital OCC display can be screens,used in bothand indoorextracts and the outdoorinformation environments by means forof image a range processing of applications [17]. Thus, [11]. Thecamera-based development OCC of cancomplementary be used in metal-oxide both indoor semiconductor and outdoor (CMOS) environments sensor technologies for a range has of created applications a new generation[11]. The developmentof high-pixel-resolution of complementary and high-speed metal-oxide built-in semicond camerasuctor [11 (CMOS),17]. Although sensor technologies most cameras has capturecreated a new generation of high-pixel-resolution and high-speed built-in cameras [11,17]. Although most images at a FR of 30 to 60 fps, the OCC standardization covers high-speed cameras, which can support cameras capture images at a FR of 30 to 60 fps, the OCC standardization covers high-speed cameras, a FR of up to thousands of fps [5,11]. whichSimilar can support to VLC, a F OCCR of up predominantly to thousands reliesof fps on[5,11]. the directional line-of-sight (LOS) transmission mode.Similar In addition, to VLC, OCC OCC conveniently predominantly offers relies a MIMO on the capability directional by extracting line-of-sight the (LOS) data from transmission captured mode. In addition, OCC conveniently offers a MIMO capability by extracting the data from captured multiple light sources. Therefore, for these reasons, OCC can be considered as a subset of VLC. It is interesting to note that the imaging MIMO capability of the camera can also be exploited for other applications, such as range estimation, shape detection, scene detection, etc.

Electronics 2020, 9, 1339 5 of 44 multiple light sources. Therefore, for these reasons, OCC can be considered as a subset of VLC. It is interesting to note that the imaging MIMO capability of the camera can also be exploited for other applications,Electronics 2020, 9 such, x FOR as PEER range REVIEW estimation, shape detection, scene detection, etc. 5 of 45

2.2. Standardization OCC has been in the process of standardization since 2014 by the IEEE 802.15.7r1 task group TG7r1, which is is a a revised revised version version of of IEEE IEEE 802.15.7 802.15.7 standard standard for for VLC VLC [11,20,21]. [11,20,21 ].OCC OCC is considered is considered as asan anextension extension of the of theVLC VLC standard standard in IEEE in IEEE 802.15.7 802.15.7 that was that initially was initially defined defined as a specific as a specific task group task groupto formulate to formulate the standard the standard revision revision for VLC, for VLC,which which is called is called the IEEE the IEEE802.15.7-2011 802.15.7-2011 standard standard [1]. The [1]. Thetask taskgroup group for OCC for OCC was was renamed renamed to IEEE to IEEE 802.15.7m 802.15.7m OWC OWC Task Task Group Group TG7m TG7m in in order order to to develop technical specificationsspecifications for the OCC standardstandard [[14].14]. TG7m covers NIR (700–1000 nm wavelengths), UV (100–400 nm wavelengths) and visible lightlight (400–700(400–700 nmnm wavelengths)wavelengths) bands.bands. The TG7m task group has contributed to various OWC solutions including OCC, LED IdentificationIdentification (LED-ID) and LiFi [[14].14]. OCC OCC is is alternatively known known as IS communications (ISC) or camera-based VLC, since it utilizes an IS as the RX, RX, which which provides provides a supplem supplementaryentary functionality of communications in addition to video streaming andand imageimage capturingcapturing [[22–24].22–24]. Figure 44 illustrates illustrates didifferentfferent examplesexamples ofof OCCOCC consideredconsidered inin TG7m,TG7m, highlightinghighlighting vehicular communications (vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I)) and indoor applications [[14].14]. Additional standardized applications were proposed in late 2018 by TG7m, such as 2D color code, code, TX/RX TX/RX relay, relay, digital digital signage, signage, exhibiti exhibitionon/store/store service, service, in-flight in-flight service, service, underwater underwater (or (orseaside/coastal) seaside/coastal) communications, communications, drone-to-drone, drone-to-drone, smart smart living, living, street-lamp street-lamp access access point, point, smart smart city, city,smart smart factory, factory, and smart and smart house house [14,25,26]. [14,25 These,26]. Theseapplications applications are classified are classified into flicker into and flicker flicker- and flicker-freefree physical physical (PHY) (PHY)layer modes, layer modes, depending depending on the on switching the switching rate rateof the of theTX TXbeing being perceptible perceptible or orimperceptible imperceptible to tohuman human eyes, eyes, respectively. respectively.

Figure 4. Example of promising serv servicesices in IEEE 802.15.7m.

Regarding modulations,modulations, five five additional additional schemes schemes are standardized are standardized to deliver optimumto deliver performance optimum inperformance OCC. That is,in accordingOCC. That to theis, according finalized document to the finalized in 2017, document the TG7m Committeein 2017, the approved TG7m Committee additional setsapproved of OWC additional PHY operating sets of OWC modes, PHY i.e., operating PHY IV to modes PHY, VIII, i.e., PHY as shown IV to inPHY Table VIII,1. PHYas shown IV, V, in and Table VI are1. PHY directly IV, V, related and VI to are OCC directly developments, related to O whileCC developments, PHY VII and while VIII are PHY intended VII and for VIII LiFi. are intended In direct detectionfor LiFi. In VLC direct systems, detection the VLC simple systems, on-o theff keying simple(OOK) on-off keying and variable (OOK) pulseand variable position pulse modulation position (VPPM)modulation schemes (VPPM) were schemes adopted were in the adopted initial version in the in ofitial the version IEEE 802.15.7-2011 of the IEEE standard.802.15.7-2011 Table standard.2 details theTable OCC 2 details modulation the OCC schemes modulation that were schemes introduced that inwe there introduced IEEE 802.15.7m in the standard IEEE 802.15.7m for PHY IV, standard PHY V, andfor PHY PHY IV, VI. PHY The nextV, and revised PHY draftVI. The of thenext standard, revised draft known of asthe IEEE standard, 802.15.7-2018, known as was IEEE finalized 802.15.7- in 20182018, [was27,28 finalized]. More detailed in 2018 OCC[27,28]. technologies More detailed are presentedOCC technologies in Sections are3 andpresented4. in Sections 3 and 4.

Electronics 2020, 9, 1339 6 of 44

Table 1. IEEE 802.15.7m initial operating modes.

PHY Description Modulation Scheme

OOK • I Existing IEEE 802.15.7-2011 VPPM •

OOK • II Existing IEEE 802.15.7-2011 VPPM •

CSK III Existing IEEE 802.15.7-2011 •

UFSOOK • Twinkle VPPM • Image sensor communication S2-PSK IV • modes (added to TG7m) HS-PSK • Oﬀset-VPWM •

PWM/PPM • CM-FSK Image sensor communication • V C-OOK modes (added to TG7m) • RS-FSK •

Table 2. IEEE 802.15.7m: Characteristics and performance estimation of OCC technologies.

Flicker-Free Modulation Flicker Modulation Parameter High Frame Rate RoI Signaling Rolling Shutter Screen/Projector Processing Technique Technique Distance <100 m Hundreds of meters Several meters Several meters

Tens of bps for RoI • signaling stream Tens of kbps for the Data rate Tens of Kbps • <10 Kbps Kbps to Mbps high-data rate stream

Traﬃc light Car lights/traﬃc LED panel/arrays • • • Screen TX LEDs-array TX lights/LED of LED-based TX • Typical Intended system High-speed signage TX Rolling shutter • • • cameras RX camera RX RoI-camera RX camera RX •

The integration of a The resolution of • • RoI signaling the LED panel on The LED array • stream and a the captured based on the TX MIMO beneﬁts high-speed data image limits the • allowing higher from the spatial stream is amount of data data transmission resolution of implemented in a per image per frame both TX and RX hybrid system The trade-oﬀ Characteristics The detection of • Well suited for • The RoI signaling between distance • light source based • short-range stream allows faster, and data rate on image OWC system reliable detection makes this the processing is utilizing screen and tracking of most suitable the drawback multiple light option for sources simultaneously indoor application Electronics 2020, 9, 1339 7 of 44

2.3. Transceivers The operation and performance of IS-based OCC systems are affected by both the IS characteristics and camera’s image processing [17]. In digital cameras, the IS is typically based on the CMOS architecture, which is a mature, highly developed and less costly technology than charge-coupled device (CCD)-based cameras [29]. Note that the size of CMOS-based ISs is increasingly smaller, while still offering higher resolution images with high pixel density. Smaller sized CMOS ISs are made possible using the advanced electronics fabrication technology with a 7 nm manufacturing scale [30]. As shown in Figure5, a typical camera structure is composed of an imaging lens, a color filter array (CFA), an image sensor (IS) and an image processing unit (IPU) [29]. The lens projects the light illuminated or reflected from a light source or an object on the IS, respectively. The diagonal field of view (FOV) of the camera is determined by the focal length of the camera lens and the sensor size as Electronics 2020, 9, x FOR PEER REVIEW 7 of 45 illustrated in Figure6. In addition, the working distance defines the distance for the camera to capture Electronics 2020, 9, x FOR PEER REVIEW 7 of 45 the image in focus.

Figure 5. A brief schematic of a digital camera. Figure 5. A brief schematic of a digital camera. Figure 5. A brief schematic of a digital camera.

FigureFigure 6.6. AA diagonaldiagonal fieldfield ofof viewview (FOV)(FOV) ofof aa camera.camera. Figure 6. A diagonal field of view (FOV) of a camera. AA CFACFA typicallytypically withwith BayerBayer patternpattern recordsrecords thethe colorcolor informationinformation ofof thethe illuminatedilluminated lightlight inin aa spatial configuration [31]. Bayer pattern is structured based on 2 2 blocks of filters (i.e., one blue, spatialA CFAconfiguration typically [31].with BayerBayer patternpattern isrecords structured the color based information on 2 ×× 2 blocks of the of illuminated filters (i.e., lightone blue, in a one red, and two green), which can have different arrangements of BGGR, RGBG, GRGB or RGGB. spatialone red, configuration and two green), [31]. which Bayer canpattern have is differen structuredt arrangements based on 2 of× 2 BGGR, blocks RGBG,of filters GRGB (i.e., orone RGGB. blue, The reason for more green filters is the human eye, which is more sensitive to the green light; hence, oneThe red,reason and for two more green), green which filters can is thehave human differen eye,t arrangements which is more of sensitive BGGR, RGBG,to the green GRGB light; or RGGB. hence, more information about the green color is required [32]. The image is then processed by an image Themore reason information for more about green the filters green is thecolor human is required eye, which [32]. isThe more image sensitive is then to processed the green bylight; an hence,image processor utilizing a demosaicing algorithm, also known as deBayering, to reconstruct the image colors moreprocessor information utilizing about a demosaicing the green coloralgorithm, is required also known [32]. The as imagedeBayering, is then to processed reconstruct by thean imageimage processorcolors similar utilizing to the a demosaicingreal scene, as algorithm, illustrated also in Figureknown 7as [31,32]. deBayering, The output to reconstruct of the IPU the isimage then colorsprocessed similar to extract to the the real information scene, as illustratedin the image. in Figure 7 [31,32]. The output of the IPU is then processed to extract the information in the image.

Electronics 2020, 9, 1339 8 of 44 similar to the real scene, as illustrated in Figure7[ 31,32]. The output of the IPU is then processed to Electronicsextract the 2020 information, 9, x FOR PEER in REVIEW the image. 8 of 45

Figure 7. An example of the demosaicing process. Figure 7. An example of the demosaicing process. The internal shutter mechanism of cameras determines the exposure of pixels in the IS. Two main typesThe of ISsinternal in terms shutter of the mechanism architecture of are cameras CCD and deter CMOSmines [33 the]. Aexposure CCD sensor of pixels has a largerin the physicalIS. Two maindimension types dueof ISs to in the terms utilization of the ofarchitecture a relatively are larger CCD analog-to-digital and CMOS [33]. converter A CCD sensor (ADC), has compared a larger withphysical a CMOS dimension IS [29 ,34due]. Therefore,to the utilization CCD is of not a generallyrelatively implementedlarger analog-toin mobiledigital devices,converter especially (ADC), comparedsmartphones. with In a comparisonCMOS IS [29,34]. with CCD,Therefore, CMOS CCD cameras is not provide generally lower implemented power consumption, in mobile devices, smaller especiallyIS, faster readout, smartphones. lower costIn andcomparison more programmability. with CCD, CMOS In addition, cameras since provide pixel scanning lower ispower based consumption,on the X–Y address smaller scheme IS, faster in thereadout, CMOS lower architecture, cost and directmore programmability. access to a desired In part addition, of the since IS is pixel scanning is based on the X–Y address scheme in the CMOS architecture, direct access to a possible, which can increase the FR at the cost of reduced FOV [35,36]. Cameras can have two different desiredexposure part methods, of the IS i.e., is possible, global shutter which (GS)can increase and rolling the shutterFR at the (RS), cost of as reduced illustrated FOV in Figure[35,36].8 a.Cameras In the canglobal-shutter-based have two different ISs, exposure the entire methods, sensor is i.e., exposed global to shutter the light (GS) at the and same rolling time, shutter whereas (RS), in theas illustratedrolling-shutter-based in Figure 8a. cameras, In the global-shutter-based each row is exposed to ISs, the the light entire at a time,sensor thereby is exposed considerably to the light reducing at the samethe power time, consumption.whereas in the CCD rolling-shutter-based sensors inherently came captureras, each images row using is exposed the GS, to while the light the at exposure a time, therebymethod inconsiderably most CMOS reducing ISs is based the onpower RS [29 consum,33]. ption. CCD sensors inherently capture images usingIt the should GS, while be noted the thatexposure in the method global-shutter-based in most CMOS CCD ISs is ISs, based considerably on RS [29,33]. longer frame capture timeIt is should needed be to noted convert that the in the charge global-shutter-ba into a voltagesed signal CCD [29 ISs,]. considerably Figure8b shows longer the frame structures capture of timeCCD is and needed CMOS to convert sensors the [29 charge]. Due into to the a voltage fabrication signal of [29]. much Figure more 8b compact shows the ADC structures for each of pixel,CCD andthe CMOSCMOS sensorsensors is [29]. preferable Due to nowadays the fabrication compared of much with more relatively compact larger ADC CCD for sensors.each pixel, CMOS the CMOS is also sensor is preferable nowadays compared with relatively larger CCD sensors. CMOS is also employed employed for high-speed cameras with FR > 1000 fps, which cannot be accomplished using the CCD forsensors high-speed due to slowercameras ADCs. with F NoteR > 1000 that fps, the CMOSwhich cannot sensor inbe smartphoneaccomplished cameras using isthe capable CCD sensors of reaching due to slower ADCs. Note that the CMOS sensor in smartphone cameras is capable of reaching an FR of an FR of 1000 fps [13,37]. The shutter methods are implemented within the sensor itself; however, 1000the information fps [13,37]. can The be shutter treated methods differently are as implem part ofented image within processing the sensor following itself; image however, capturing, the whichinformation is related can be to thetreated OCC differently modulation as part schemes. of image The processing detailed description following image of modulation capturing, schemes which isbased related on RSto the in OCCOCC ismodulation providedin schemes. Section 3The. detailed description of modulation schemes based on RS inAs OCC mentioned is provided previously, in Section OCC 3. only utilizes CMOS sensors, since it is widely employed in the majorityAs mentioned of mobile devicespreviously, including OCC only smartphones utilizes CMOS [11,17 ].sensors, With regards since it to is image widely capturing employed in in OCC, the majorityCMOS sensors of mobile can devices have either including GS or RSsmartphones mechanism, [11,17]. depending With regards on the image to image processing capturing algorithms in OCC, CMOSand the sensors TX switching can have rate either [5,17 GS,22 or,38 RS,39 mechanis]. For example,m, depending a camera on with the image a full processing high definition algorithms (FHD) andresolution the TX (1920 switching1080 rate pixels) [5,17,22,38,39]. and a frame For rate example, of 60 fps a employingcamera with the a GSfull mechanism high definition can sample(FHD) × resolution60 complete (1920 frames × 1080 every pixels) second. and On a theframe other rate hand, of 60 the fps identical employing camera the with GS mechanism RS processes can individual sample 60rows complete on each frames frame asevery a sample, second. thus On yielding the other a samplinghand, the rate identical of 64,800 camera (60 with1080) RS samples processes per × individualsecond (sps). rows on each frame as a sample, thus yielding a sampling rate of 64,800 (60 × 1080) samples per second (sps).

Electronics 2020, 9, 1339 9 of 44 Electronics 2020, 9, x FOR PEER REVIEW 9 of 45

Figure 8.8. (a) AA comparisoncomparison betweenbetween the GSGS andand RS;RS; ((bb)) aa structuralstructural comparisoncomparison betweenbetween CCDCCD andand CMOS sensors.sensors. The complete frame capture period comprises the exposure and frame-saving periods as shown The complete frame capture period comprises the exposure and frame-saving periods as shown in Figure9a. The exposure time, which can be set by the camera controller, is a period when the IS in Figure 9a. The exposure time, which can be set by the camera controller, is a period when the IS is is exposed to the light. In the RS cameras, the rows of the IS are exposed sequentially. Therefore, exposed to the light. In the RS cameras, the rows of the IS are exposed sequentially. Therefore, each each pixel in a row of frame is treated as an individual sample, which increases the sampling rate by pixel in a row of frame is treated as an individual sample, which increases the sampling rate by thousand times, thus oﬀering data rates much higher than FR. Figure 10 demonstrates the LED status thousand times, thus offering data rates much higher than FR. Figure 10 demonstrates the LED status and the intensity in each row of the IS against the time. It is important to note that the switching rate of and the intensity in each row of the IS against the time. It is important to note that the switching rate the transmitting LED should be set to be lower than the sampling rate of the RS mechanism. of the transmitting LED should be set to be lower than the sampling rate of the RS mechanism. Following the exposure time Texp, the camera’s shutter is closed, followed by the image processing Following the exposure time Texp, the camera’s shutter is closed, followed by the image (including image compression) as well as saving the frame. Customarily, the image processing and processing (including image compression) as well as saving the frame. Customarily, the image frame saving are called the frame saving period. Obviously, the frame saving period is determined processing and frame saving are called the frame saving period. Obviously, the frame saving period by the speed of the IPU and the storage device. In the RS, the frame-saving period is compressed by is determined by the speed of the IPU and the storage device. In the RS, the frame-saving period is pipelining the readout time sequentially on each row as illustrated in Figure9b. compressed by pipelining the readout time sequentially on each row as illustrated in Figure 9b. The RS mechanism in CMOS, however, has an undesirable skew in the captured frame (i.e., RS distortion) during object capturing. This is due to the fact that the object moves at a high speed during the exposure period as illustrated in Figure9b [ 29,39]. In order to avoid this RS distortion for a 10-percent blur threshold, the exposure time must meet the following condition [39]:

∆x Texp (1) ≤ 10v where ∆x and v denote the object’s displacement and velocity, respectively. However, the image distortion is largely discounted as the image-capture functionality of the camera is ignored. The primary concerns of the RS-based OCC are image skew, link range and the limited camera resolution. Meanwhile, in the GS-based OCC with an N M pixels IS with each pixel producing G-level × grayscale signals and FR, a theoretical maximum achievable data rate limit is given by [5,40,41]:

1 r = NM log G F (2) 8 2 R The constant value of 1/8 is a rate reduction factor for three-dimensional signals, which is a downsamplingFigure 9. ( factora) Frame for capture the data period; rate in(b a) RS camera distortion to avoid causing spatial a skewed aliasing, object since in the the capture pixels frame. in an image sensor are densely organized [41]. The rate reduction factor can only be omitted from the equation when each pixel in the image sensor is placed with adequate gap [41]. For example, a camera with a

Electronics 2020, 9, x FOR PEER REVIEW 9 of 45

Figure 8. (a) A comparison between the GS and RS; (b) a structural comparison between CCD and CMOS sensors.

The complete frame capture period comprises the exposure and frame-saving periods as shown in Figure 9a. The exposure time, which can be set by the camera controller, is a period when the IS is exposed to the light. In the RS cameras, the rows of the IS are exposed sequentially. Therefore, each pixel in a row of frame is treated as an individual sample, which increases the sampling rate by thousand times, thus offering data rates much higher than FR. Figure 10 demonstrates the LED status Electronicsand the intensity2020, 9, 1339 in each row of the IS against the time. It is important to note that the switching10 ofrate 44 of the transmitting LED should be set to be lower than the sampling rate of the RS mechanism. Following the exposure time Texp, the camera’s shutter is closed, followed by the image full high deﬁnition (FHD) resolution (1920 1080 pixels), 256 levels of grayscale in the IS with a FR processing (including image compression)× as well as saving the frame. Customarily, the image ofprocessing 60 fps yields and aframe theoretical saving maximum are called achievablethe frame sa dataving rate period. of 124.42 Obviously, Mbps [the41]. frame However, saving this period data rateis determined can only beby obtainedthe speed when of the each IPU IS and pixel the is stor illuminatedage device. by In an the independent RS, the frame-saving TX. Unfortunately, period is thiscompressed individual by illuminationpipelining the is readout impractical time due sequentially to the transmission on each row distance as illustrated between in theFigure camera 9b. and the TX.

Electronics 2020, 9, x FOR PEER REVIEW 10 of 45 Figure 9. (a) Frame capture period; (b) RS distortion causing a skewed object in the capture frame. Figure 9. (a) Frame capture period; (b) RS distortion causing a skewed object in the capture frame.

Figure 10. RS mechanism on image sensor for data capture.

ToThe carry RS mechanism out a more in accurate CMOS, measurement however, has of an the undesirable captured image,skew in a minimumthe captured of twoframe pixels (i.e., perRS thedistortion) smallest during feature objectSF is capturing. required. This The is smallest due to the feature fact that indicates the object the sizemoves of aat smallest a high speed identifiable during propertythe exposure in an period image, as i.e., illustrated a clear shapein Figure and 9b color. [29,39]. The In minimum order to ISavoid (camera) this RS resolution distortionS Rforis a then 10- twopercent times blur the threshold, size of FOV the (exposureFC), either time horizontal must meet or vertical the following FOV, divided condition by [39]: the smallest feature SF. Therefore, to capture a clear image, the minimum resolutionΔ of the camera is given as [42]: (1) 10FC SR = 2 (3) where Δ and denote the object’s displacement andsF velocity, respectively. However, the image distortion is largely discounted as the image-capture functionality of the camera is ignored. The For example, if the FOV covers 100◦ and the smallest feature is 1 mm, the required minimum primary concerns of the RS-based OCC are image skew, link range and the limited camera resolution. resolution is 200 pixels. Thus, a camera with a resolution of 640 480 pixels would be adequate. Meanwhile, in the GS-based OCC with an N × M pixels IS with× each pixel producing G-level This is because 200 pixels are less than 480 pixels, which is the smallest dimension. In other words, grayscale signals and FR, a theoretical maximum achievable data rate limit is given by [5,40,41]: the number of pixels (in IS) define the maximum distance of capturing a clear image, related to the FOV in Equation (3) and its working distance1 illustrated in Figure6. = ( ) (2) In a camera-based OCC, a range of TXs8 can be adopted, e.g., digital display modules and LEDsThe [5,17 constant,22,36,38 value]. Color of 1/8 filter is arrays,a rate reduction wavelength factor division for three-dimensional multiplexing (WDM) signals, and which color shiftis a keyingdownsampling (CSK) based factor on RGBfor the LEDs data or rate displays in a havecamera been to employedavoid spatial in OCC aliasing, [5]. Interestingly, since the pixels the typical in an refreshimage sensor rate of are a digital densely display organized is ~60 [41]. Hz, whichThe rate is inreduction the same factor range can of F onlyR [38 ].be The omitted type offrom digital the displayequation used when as each the TX pixel is either in the LED,image LCD sensor screen, is pl OLED,aced with or e-paperadequate [5 gap,17,38 [41].,43]. For In example, display-to-camera a camera communications,with a full high definition multiple (FHD) colors resolution with multiple (1920 intensity × 1080 pixels), levels are 256 deployed levels of forgrayscale each pixel, in the since IS with the a of 60 fps yields a theoretical maximum achievable data rate of 124.42 Mbps [41]. However, this data rate can only be obtained when each IS pixel is illuminated by an independent TX. Unfortunately, this individual illumination is impractical due to the transmission distance between the camera and the TX. To carry out a more accurate measurement of the captured image, a minimum of two pixels per the smallest feature is required. The smallest feature indicates the size of a smallest identifiable property in an image, i.e., a clear shape and color. The minimum IS (camera) resolution is then two times the size of FOV (), either horizontal or vertical FOV, divided by the smallest feature . Therefore, to capture a clear image, the minimum resolution of the camera is given as [42]:

=2 (3) For example, if the FOV covers 100° and the smallest feature is 1 mm, the required minimum resolution is 200 pixels. Thus, a camera with a resolution of 640 × 480 pixels would be adequate. This is because 200 pixels are less than 480 pixels, which is the smallest dimension. In other words, the number of pixels (in IS) define the maximum distance of capturing a clear image, related to the FOV in Equation (3) and its working distance illustrated in Figure 6. In a camera-based OCC, a range of TXs can be adopted, e.g., digital display modules and LEDs [5,17,22,36,38]. Color filter arrays, wavelength division multiplexing (WDM) and color shift keying (CSK) based on RGB LEDs or displays have been employed in OCC [5]. Interestingly, the typical refresh rate of a digital display is ~60 Hz, which is in the same range of FR [38]. The type of digital display used as the TX is either LED, LCD screen, OLED, or e-paper [5,17,38,43]. In display-to-camera

ElectronicsElectronics2020 2020,,9 9,, 1339 x FOR PEER REVIEW 1111 ofof 4445

communications, multiple colors with multiple intensity levels are deployed for each pixel, since the illuminationillumination footprint footprint is is mapped mapped on on the the spatially spatially separated separated PDs ofPDs the of IS. the Since IS. current Since current display devicesdisplay supportdevices FHDsupport resolution, FHD resolution, higher data higher rates data can rates be achieved. can be achieved. 2.4. Channel Characteristics 2.4. Channel Characteristics Although the OCC channel is largely considered to be free space, other mediums such as vacuum, Although the OCC channel is largely considered to be free space, other mediums such as gas and water can also be used [5,14]. In OCC, the transmission mode can be LOS [1–3], non-LOS vacuum, gas and water can also be used [5,14]. In OCC, the transmission mode can be LOS [1–3], (NLOS) [4,5] or a combination of both. The channel impulse response for the LOS link can be non-LOS (NLOS) [4,5] or a combination of both. The channel impulse response for the LOS link can represented as a matrix H = [h ] , where U and V are the numbers of rows and columns of ch =ℎch,u,v U V be represented as a matrix ,,× ×, where and are the numbers of rows and columns the image sensor, respectively and hch,u,v is the channel impulse response of the (u, v)-th pixel. Using a of the image sensor, respectively and ℎ,, is the channel impulse response of the (, )-th pixel. single LED and a Gaussian mixture model, hch,u,v is estimated as [6]: Using a single LED and a Gaussian mixture model, ℎ,, is estimated as [6]: a a   v 2 Z ax+ Z ay+ 2 2 X κ ck 2 2  x y  hℎ,,==A × exp−exp − dydx,, (4) (4) ch,u,v 2 2  22 22  2πσ,σ,κ × a a −2σ, − 2σ,  k=1 x,k y,k 0 ax 2 ay 2 x,k y,k − − where =1/ , is the pixel length, and are the magnification factors of the camera at a where A = 1/a2, a is the pixel length, κ and κ are the magniﬁcation factors of the camera at a reference reference link span , and at a distance0 of , respectively. , = +, and , = 2 2 2 κ 2 2 2 κ 2 2 link span L and at a distance of L , respectively. σ = σ + σ and σ = σ + σ with s,0 s x,k κ0 i0 b,x y,k κ0 i0 b,y +, with denoting the parameter of the model and , and , are the standard σi0 denoting the parameter of the model and σb,x and σb,y are the standard deviations in the x and y deviations in the x and y directions on the image plane, respectively. In (4), = ( − )( + ) − directions on the image plane, respectively. In (4), ax = (u i)(a + g) ξx and ay = (v j)(a + g) ξy, − − − − and = ( − )( + ) − , where (,) and (, ) are the coordinates of the center of the where ξx, ξy and (i, j) are the coordinates of the center of the image and nearest pixel to this center, respectivelyimage and nearest and g is pixel the gap to this between center, two respectively pixels. Figure and 11 illustrates is the gap the between coordinates two pixels. system Figure visually. 11 illustrates the coordinates system visually.

FigureFigure 11.11. ImageImage planeplane coordinatescoordinates system.system. MostMost OCCOCC systemssystems reportedreported areare basedbased onon thethe LOSLOS configuration.configuration. However,However, inin scenariosscenarios wherewhere TXsTXs areare notnot within FOV of of the the LOS, LOS, such such as as two two ve vehicleshicles approaching approaching crossroads, crossroads, the the channel channel will will be bevia via NLOS NLOS paths. paths. In InNLOS NLOS links, links, reflected reflected beam beamss normally normally have have large large off-axis off-axis projectedprojected opticaloptical illuminationillumination footprintsfootprints thatthat cancan increaseincrease linklink tolerancetolerance to to the the camera camera movements. movements. ConsideringConsidering thethe NLOSNLOS OCCOCC link,link, thethe camera’scamera’s( i(,,j) -th)-th pixel pixel is is given given by by [ 44[44]:]:

() Z p Z q(y) ,(, )(, ) i j AaperR l(x, y)Rr(x, y) ℎ,(, ) = t, × cos((, )) dd (5) h (i, j) = (, )(, ) ρ cos(ψc(x, y))dxdy (5) ch,l () 2, 2 pi 1 qj 1(y) d (x, y)dc (x, y) × − − t,l where and denote the area of the aperture and the reflection coefficient, respectively. where Aaper and ρ denote the area of the aperture and the reflection coefficient, respectively. R (x, y) ,(, ) and (, ) are radiation patterns of the LED and the reflection element, respectively.t,l and Rr(x, y) are radiation patterns of the LED and the reflection element, respectively. p and q and are the boundaries of the area covered by pixels in and dimensions, respectively.i j are the boundaries of the area covered by pixels in y and x dimensions, respectively. dt,l(x, y) = q (, ) =2 − 2 +− + ( , , ) , , 2 , is the distance between the TX , , and the x xt,l + y yt,l + Ht is the distance between the TX xt,l, yt,l, Ht and the reflecting element − − reflecting element (,,0)q , and (, ) = (−) + (−) + is the distance between the 2 2 2 (x, y, 0), and dc(x, y) = (x xc) + (y yc) + H is the distance between the camera (xc, yc, Hc) and camera (,,) and the −reflecting element.− Forc a matte surface, reflections follow the 1st order theLambertian reflecting pattern. element. For a matte surface, reflections follow the 1st order Lambertian pattern. For the camera, the incident angle is given by:

Electronics 2020, 9, 1339 12 of 44

For the camera, the incident angle is given by:

1 ψc(x, y) = cos− (→n l.→n c), (6) where →n = (x xc, y yc, Hc) and →n c = ( cos(φ ) cos(θ ), sin(φ ) cos(θ ), sin(θ )) are l − − − − tilt tilt tilt tilt − tilt the unitized incident beam and the camera plain normal vector, respectively, and the dot is the inner product. For the CMOS active pixel, the frequency response as a function of the ampliﬁcation gain G per pixel and the angular frequency ω is given by [45]: Texp 2 sin 0.5ωTexp jω hcp(ω) = Ge− 2 (7) × ω

In (7), the 2nd term represents the delay of 0.5Texp between the center of the exposure and when the sample is actually read, while the 3rd term is the frequency response of integrating x(t) over Texp. One advantage of OWC against RF communications is the channel modeling, which involves considerably higher frequencies of the optical signal, compared with the optical clock signal [17]. Thus, the multipath fading and the Doppler eﬀect can be ignored. For OCC, the signal-to-noise ratio (SNR) can be obtained as [17]: 2 2 4 S (Rhch,lPt) αPt d− SNR = = (8) N qRPnAW qRPnAW where N and S denote the noise and received signal powers, respectively. hch,l is the channel impulse response (the channel DC gain), R denotes the camera’s responsivity, Pt,l denotes transmit power of l-th light source, q denotes the electron charge, Pn represents ambient light per unit area, A is the area R2h2 = ch,l of the PD (area of the IS) and α d4 . Depending on the focusing distance of the camera lens, the received power can be projected on a single PD or multiple PDs, which are known as focused and defocused modes, respectively. Figure 12 shows diﬀerent focusing scenarios based on the TX’s location [17], in which l and c indicate the height of the object and its projected height on the sensor, respectively. Moreover, d shows the distance between the lens and the object, whereas f is the distance from the lens to the IS. When the camera is defocused, the image of a point source will illuminate an area of IS with the same shape of the lens aperture, which is typically a circle. This area is known as a circle of confusion (CoC). The shape of the defocused image can be obtained as [46,47]:

I(x, y) = Io(x, y) I (x, y), (9) ⊗ ⊗ CoC p 2 2 p 2 2 DCoC where, ICoC(x, y) = U x + y U x + y is the CoC disc function, U(.) is Heaviside step − − 2 p 2 2 p 2 2 function, Io(x, y) = h U x + y U x + y d /2 is the intensity function of the image when I − − I the TX is in focus and is the 2D convolution operation. D is the diameter of the CoC and is ⊗⊗ CoC given by: Dlens DCoC = di ds , (10) 2di | − | f where D = l , f , and f are the lens aperture diameter, camera lens focal length and f-stop of lens fstop l stop the aperture, respectively. While SNR calculates signal-to-noise intensity on a sensor as a single unit, a more speciﬁc approach for the energy to noise ratio (Eb/N0) calculation on the IS of a camera is provided to estimate the amount of light reaching each pixel of the sensor, compared with the light interference [5]. In this regard, the Eb/N0 value per pixel on the IS, which is directly related to the amount of illuminance received, which is given by: h 2i E E s a2t b = (11) 2 N0 E[n ] ≈ αat + β Electronics 2020, 9, 1339 13 of 44

where Eb denotes the energy per bit, N0 denotes the noise spectral density, s is the pixel value, a is the amplitude of the signal (light), t is the camera exposure duration as a ratio of the signal cycle, n is noise value and both α and β are noise model fit parameters. In addition to the signal quality evaluation for OCC reception, the peak signal-to-noise ratio (PSNR) approach is often considered for the image quality evaluation in OCC with the GS and reflects the quality of OCC channel in terms of 2D image degradation [48]. The PSNR is formulated in OCC as a ratio of the original image quality (transmitted) to the captured image/frame quality, which is degraded by the channel impairments, and is given by [37,48]: MR PSNR = 10 log10 (12) MN where MR denotes a matrix containing the mean of the maximum pixel value of RGB color channels in the originally transmitted image. MN is the average value of mean squared error (MSE) from all RGB color channels. Calculating a noise-free monochrome original image (I) with a resolution of m n × pixels and the noisy captured frame (K), the MSE for each color channel (either R, G, or B) is defined as:

m 1 n 1 1 X− X− MSE = [I(i, j) K(i, j)]2 (13) mn − i=0 j=0 Electronics 2020, 9, x FOR PEER REVIEW 13 of 45

FigureFigure 12. 12.Lens Lens focuses focuses incoming incoming light light on on the the IS: IS: (a) ( ina) focus;in focus; (b )(b near) near focus; focus; (c) ( farc) far focus focus [17 [17].]. 3. Modulations While SNR calculates signal-to-noise intensity on a sensor as a single unit, a more specific

approachModulations for the and energy coding to noise schemes ratio are(𝐸/𝑁 the) key calculation drivers inon thethe performanceIS of a camera of is aprovided communications to estimate system.the amount In OCC of where light reaching a camera iseach employed pixel of as the the sensor, RX to capturecompared the 2Dwith image the light of the interference TX, the information [5]. In this dataregard, can be the used 𝐸/𝑁 to modulate value per color, pixel intensity, on the IS, spatial which coordinate is directly (deﬁnedrelated to by the camera amount resolution) of illuminance and received, which is given by: 𝐸 𝐸𝑠 𝑎 𝑡 = (11) 𝑁 𝐸𝑛 𝛼𝑎𝑡 +𝛽

where 𝐸 denotes the energy per bit, 𝑁 denotes the noise spectral density, 𝑠 is the pixel value, 𝑎 is the amplitude of the signal (light), 𝑡 is the camera exposure duration as a ratio of the signal cycle, 𝑛 is noise value and both 𝛼 and 𝛽are noise model fit parameters. In addition to the signal quality evaluation for OCC reception, the peak signal-to-noise ratio (PSNR) approach is often considered for the image quality evaluation in OCC with the GS and reflects the quality of OCC channel in terms of 2D image degradation [48]. The PSNR is formulated in OCC as a ratio of the original image quality (transmitted) to the captured image/frame quality, which is degraded by the channel impairments, and is given by [37,48]:

𝑀 PSNR = 10 log (12) 𝑀

where 𝑀 denotes a matrix containing the mean of the maximum pixel value of RGB color channels in the originally transmitted image. 𝑀 is the average value of mean squared error (MSE) from all RGB color channels. Calculating a noise-free monochrome original image (𝐼) with a resolution of 𝑚× 𝑛 pixels and the noisy captured frame (𝐾), the MSE for each color channel (either R, G, or B) is defined as: Electronics 2020, 9, x FOR PEER REVIEW 14 of 45

1 MSE = (,)−(,) (13)

3. Modulations Modulations and coding schemes are the key drivers in the performance of a communications system.Electronics In2020 OCC, 9, 1339 where a camera is employed as the RX to capture the 2D image of the TX,14 ofthe 44 information data can be used to modulate color, intensity, spatial coordinate (defined by camera resolution) and the form of shape of the source [5,14,17]. It is worth noting that these modulation the form of shape of the source [5,14,17]. It is worth noting that these modulation options are limited, options are limited, due mainly to the number of cameras; therefore, the TX is manipulated in order due mainly to the number of cameras; therefore, the TX is manipulated in order to compensate for to compensate for the number of the RX that could be employed. In OCC, there are several factors the number of the RX that could be employed. In OCC, there are several factors that determine the that determine the modulation schemes, such as IS type, exposure time, ISO, image processing, type modulation schemes, such as IS type, exposure time, ISO, image processing, type of TXs and human of TXs and human eye perception. Broadly speaking, the OCC modulation can be classified into four eye perception. Broadly speaking, the OCC modulation can be classified into four major categories: major categories: Nyquist sampling modulation (NSM), region-of-interest signaling (RoI-S), hybrid Nyquist sampling modulation (NSM), region-of-interest signaling (RoI-S), hybrid camera-PD and camera-PD and RS-based modulation. RS-based modulation. 3.1. Nyquist Sampling Modulation 3.1. Nyquist Sampling Modulation The principle of NSM is based on limiting a switching/refresh rate Rs of the TX to be less than or The principle of NSM is based on limiting a switching/refresh rate Rs of the TX to be less than or equal to 0.5FR in order to comply with Nyquist sampling theorem as shown in Figure 13 [5]. Since FR equal to 0.5FR in order to comply with Nyquist sampling theorem as shown in Figure 13[ 5]. Since FR is mostly limited to 60 fps, the TX’s Rs is generally set to 30 Hz for NSM. Furthermore, due to a low is mostly limited to 60 fps, the TX’s Rs is generally set to 30 Hz for NSM. Furthermore, due to a low Rs Rs of the TX that causes perceptible flicker to the human eyes, NSM generally utilizes a digital display of the TX that causes perceptible flicker to the human eyes, NSM generally utilizes a digital display as as the TX instead of LEDs, since display flickering is less discomforting than LED [5,17,49]. Moreover, the TX instead of LEDs, since display flickering is less discomforting than LED [5,17,49]. Moreover, a single LED TX is not used in NSM since the effective data rate is limited to be lower than 60 bps. a single LED TX is not used in NSM since the effective data rate is limited to be lower than 60 bps.

Figure 13. Nyquist sampling modulation. Figure 13. Nyquist sampling modulation.

In NSM, the data transmission throughput is limited by FR. To increase the data rate, multi-dimensionIn NSM, the modulationdata transmission schemes throughput based on is colors, limited intensity by FR. levels,To increase and shapesthe data spread rate, multi- across dimensionspace (spatial modulation coordinate) schemes are employed based [on5,11 colors,,17]. Initial intensity studies levels, related and to shapes NSM OCCspread based across on digitalspace (spatialdisplay arecoordinate) PixNet and are COBRA employed codes [5,11,17]. [50,51]. Initial PixNet studies encodes related the bits to in NSM the frequency OCC based domain on digital using display2D orthogonal are PixNet frequency and COBRA division codes multiplexing [50,51]. PixNet (OFDM) encodes modulation, the bits in thus the frequency offering throughputs domain using in 2Dthe orthogonal order of several frequency Mbps division with the multiplexing processing time (OFDM) limited modulation, by the clock thus rate offering (i.e., capping throughputs the TX in to the5 Hz) order [50 ].of Inseveral [51], anMbps enhanced with the 2D processing color code time known limited as COBRA by the clock codes rate (i.e., (i.e., in thecapping form the of coloredTX to 5 Hz)QR codes)[50]. In was [51], reported an enhanced in order 2D tocolor optimize code known real-time as streamingCOBRA codes using (i.e., five in colors the form and of three colored optional QR codes)colors withwas areported higher transmissionin order to optimize rate of 10 real-tim Hz. It introducede streaming a conceptusing five of embeddingcolors and anchors,three optional at the colorscorners with(CT) a andhigherborders transmission(TRB) of rate the of transmission 10 Hz. It introduced frame to a synchronize concept of embedding between adjacent anchors, frames at the cornersduring reception(CT) and [borders51]. The (TRB) synchronization of the transmission between cameraframe to frame synchronize rates and between the switching adjacent rate frames of TXs using LightSync was investigated in [52]. The 2D sequential coding scheme was the first used in the NSM following COBRA, which employed modulation entities of spatial coordinate (cell) and color with the support of anchors on the corners [53]. This scheme functions as a dynamic colored QR code and performs synchronization. In addition to the 2D-sequential coding, an invisible watermarking was proposed to reduce discomfort from perceptible flickers by varying the color chromaticity slightly as part of modulation [12,54]. The transmission frame was divided into m n embedded cells as shown in Figure 14 with 4 2 embedded cells. The change × × in chromaticity within each cell, i.e., the mean value of blue, is imperceptible by human eyes since they Electronics 2020, 9, x FOR PEER REVIEW 15 of 45 during reception [51]. The synchronization between camera frame rates and the switching rate of TXs using LightSync was investigated in [52]. The 2D sequential coding scheme was the first used in the NSM following COBRA, which employed modulation entities of spatial coordinate (cell) and color with the support of anchors on the corners [53]. This scheme functions as a dynamic colored QR code and performs synchronization. In addition to the 2D-sequential coding, an invisible watermarking was proposed to reduce discomfort from perceptible flickers by varying the color chromaticity slightly as part of modulation Electronics[12,54]. The2020 transmission, 9, 1339 frame was divided into m × n embedded cells as shown in Figure 14 with15 of 444 × 2 embedded cells. The change in chromaticity within each cell, i.e., the mean value of blue, is imperceptibleare less sensitive by tohuman the blue; eyes however, since they these are changes less sensitive in chromaticity to the blue; can behowever, perceived these by changes the camera in chromaticityand thus decoded can be as perceived invisible watermarkby the camera codes. and thus decoded as invisible watermark codes.

Figure 14. Invisible watermarking OCC. Figure 14. Invisible watermarking OCC. Several studies employed NSM with an array of LED TXs and provided a higher light intensity, comparedSeveral with studies digital employed display, NSM thereby with achieving an array of a LED longer TXs transmission and provided range a higher [22,55 light–57 ].intensity, In [22], comparedsome additional with digital LEDs display, were employed thereby forachieving illumination, a longer camera transmission focusing range and light[22,55–57]. metering In [22], that someare required additional due LEDs to thewere flickering employed LEDs. for illumination On the other, camera hand, focusing thecolor-intensity and light metering modulation that are requiredmulti-input-multi-output due to the flickering (CIM-MIMO) LEDs. On the scheme other washand, proposed the color-intensity using an modulation accelerated multi-input- smartphone multi-output (CIM-MIMO) scheme was proposed using an accelerated smartphone camera with the camera with the image processing carried out in FPGA to achieve a higher FR of 330 fps [55]. It achieved imagea data rateprocessing of 126.72 carried Kbps overout in a longerFPGA to transmission achieve a higher range ofFR 1.4of 330 m. Anfps alternate[55]. It achieved approach, a data termed rate the of 126.72 Kbps over a longer transmission range of 1.4 m. An alternate approach, termed the split- split-shutter, was presented using a pair of camera setup to increase the FR by splitting the captured shutter,image between was presented two cameras; using a thus, pair itof yields camera a datasetup rate to increase of 11.52 Kbpsthe FR over by splitting a link span the captured of 144 cm image using betweenoff-the-shelf two smartphone cameras; thus, cameras it yields [43 ].a data rate of 11.52 Kbps over a link span of 144 cm using off- the-shelfIn the smartphone IEEE 802.15.7m cameras standard, [43]. an interesting modulation scheme that may increase the OCC data rate dramaticallyIn the IEEE 802.15.7m was proposed standard, [14,16 ].an It interesting is a shape modulation modulation called scheme variable that may transparent increase amplitude the OCC datashape rate code dramatically (VTASC) as illustratedwas proposed in Figure [14,16]. 15. VTASCIt is a shape was experimentally modulation called evaluated variable in OCC transparent in a form amplitudeof high-density shape modulation code (VTASC) (HDM) as illustrated with a neural in Fi networkgure 15. (NN)VTASC assisted was experimentally demodulation evaluated scheme [37 in]. TheOCC HDM, in a formingform of the high-density most complex modulation NSM, modulates (HDM) color, with intensity a neural and shapenetwork simultaneously (NN) assisted in a demodulationcell (a spatial coordinate). scheme [37]. Thus, The yieldingHDM, forming 13-bit modulated the most datacomplex on each NSM, cell. modulates The NN wascolor, adopted intensity for andclassifying shape thesimultaneously shapes efficiently in a cell on the(a spatial receiver coordinate). side [58,59]. Thus, The HDM yielding achieved 13-bit anmodulated effective datadata rate on eachof 2.66 cell. Mbps The overNN was a short adopted transmission for classifying distance the of shap 20 cm,es efficiently which can on be the considered receiver side appropriate [58,59]. The for HDMdevice-to-device achieved an (D2D) effective communications. data rate of 2.66 Mbps over a short transmission distance of 20 cm, which can beOn considered the other hand,appropriate a high-rate for dev NSMice-to-device shown in Figure(D2D) 16communications. is similar to NSM except for the use of a high-speed camera with the FR of 1–10,000 fps [5,60,61], thus increasing the Rs of the TX (i.e., LEDs) to be higher than 100 Hz. However, a digital display can also be used with Rs still limited to 144 Hz for FHD resolution. Note that a high-rate NSM offers an advantage when utilizing LEDs with Rs higher than the flicker fusion threshold, i.e., perceptible flicker by human eyes, thus preventing discomfort to the users [43,62]. Electronics 2020, 9, x FOR PEER REVIEW 16 of 45

Electronics 2020, 9, 1339 16 of 44 Electronics 2020, 9, x FOR PEER REVIEW 16 of 45

Figure 15. (a) A high-density modulation scheme and (b) example of an HDM-generated TX frame.

On the other hand, a high-rate NSM shown in Figure 16 is similar to NSM except for the use of a high-speed camera with the FR of 1–10000 fps [5,60,61], thus increasing the Rs of the TX (i.e., LEDs) to be higher than 100 Hz. However, a digital display can also be used with Rs still limited to 144 Hz for FHD resolution. Note that a high-rate NSM offers an advantage when utilizing LEDs with Rs higher than the flicker fusion threshold, i.e., perceptible flicker by human eyes, thus preventing discomfort to the users [43,62]. Figure 15. (a) A high-density modulation scheme and (b) example of an HDM-generated TX frame. Figure 15. (a) A high-density modulation scheme and (b) example of an HDM-generated TX frame.

FigureFigure 16. High-rateHigh-rate Nyquist Nyquist sampling sampling modulation modulation (NSM). (NSM).

TheThe high-rate NSM is, however, notnot readilyreadily applicableapplicable to to smartphones. smartphones. Furthermore, Furthermore, the the use use of of a ahigh-speed high-speed camera camera is envisionedis envisioned for V2Vfor V2V and V2Iand outdoor V2I outdoor applications applications only, since only, vehicles since arevehicles traveling are travelingat high speeds, at high thus speeds, requiring thus flicker-freerequiring flicker-free transmission transmission due to traffi cdue regulations to traffic [ 5regulations,60,63]. Even [5,60,63]. though Eventhese though schemes these are provenschemes to are be reliable,proven to they be are,reliable, in general, they are, relatively in general, unattractive relatively for unattractive OCC due tofor a OCCconsiderable due to a costconsiderable of both high-speed cost of both cameras high-sp andeed high-performancecameras and high-performance image processing image hardware. processing hardware. 3.2. Region-of-Interest SignalingFigure 16. High-rate Nyquist sampling modulation (NSM). Region-of-interest signaling (RoI-S) is an improvement of NSM that utilizes the region of interest The high-rate NSM is, however, not readily applicable to smartphones. Furthermore, the use of (RoI) in image post-processing in order to reduce the portion of the image capture [5,64,65]. RoI reduces a high-speed camera is envisioned for V2V and V2I outdoor applications only, since vehicles are the resolution of the image being processed at the RX and thus the proposed RoI-S method offers a traveling at high speeds, thus requiring flicker-free transmission due to traffic regulations [5,60,63]. more efficient and faster processing time than the NSM-based OCC. The implementation of RoI-S is Even though these schemes are proven to be reliable, they are, in general, relatively unattractive for basedOCC ondue the to combinationa considerable of cost NSM of andboth high-rate high-speed NSM cameras schemes. and high-performance image processing hardware.

Electronics 2020, 9, x FOR PEER REVIEW 17 of 45

3.2. Region-of-Interest Signaling Region-of-interest signaling (RoI-S) is an improvement of NSM that utilizes the region of interest (RoI) in image post-processing in order to reduce the portion of the image capture [5,64,65]. RoI reduces the resolution of the image being processed at the RX and thus the proposed RoI-S method Electronics 2020, 9, 1339 17 of 44 offers a more efficient and faster processing time than the NSM-based OCC. The implementation of RoI-S is based on the combination of NSM and high-rate NSM schemes. ROI-S utilizes a low-rate camera to fix fix the ROI first, first, then carries out the actual reception using a high-rate camera employing the previously setset ROI. It is necessary for ROI-S to use two cameras since the original ROI method is post-processing (after the capture has been completed) and the rate cannot be changed anymore afterward. It is indeed possible to use a single camera with a variable-sized capture frame (different (different from post-processing ROI) that eeffectivelyffectively reduces the resolutionresolution and increases the capture rate during the frame capture, which is called Selective Capture as proposed in [[33].33]. Selective Capture is an alternate form of the RoI-S that was proposed as part of pre-processingpre-processing (i.e., during the capture) in order to shorten the frame capture period, thus resultingresulting inin anan overalloverall higherhigher FFRR [36].[36]. Figure 1717 showsshows thatthat the the RoI-S RoI-S carries carries two two reception reception streams streams with with diff differenterent frame frame rates, rates, i.e., ai.e., low a ratelow streamrate stream for determining for determining the RoI the (i.e.,RoI RoI(i.e., signaling) RoI signaling) and aand high a ratehigh stream rate stream for the for high-rate the high-rate OCC (i.e.,OCC high-rate (i.e., high-rate MIMO MIMO data stream) data stream) [17,64 ].[17,64]. Note thatNote the that low the and low high and rate high streams rate streams are used are to used define to thedefine specific the specific RoI and RoI carry and the carry actual the data,actual respectively. data, respectively.

Figure 17. Region-of-interest signaling.

3.2.1. RoI Signaling Since the switching rate of the LED is defineddefined to be flickerflicker free (i.e.,(i.e., >>100100 Hz), the undersampled frequency shiftshift on-o on-offff keying keying (UFSOOK) (UFSOOK) scheme scheme was proposedwas proposed for RoI for signaling RoI signaling [66,67]. The[66,67]. principle The F R ofprinciple undersampling of undersampling is based on is havingbased on the havi camera’sng the samplingcamera’s sampling rate (i.e., rateR) to (i.e., be lowerFR) to be than lowers of than the TXRs of [5 the,68]. TX At [5,68]. the RX At end, the RX the end, differentiation the differentiation between between bits ‘1’ andbits ‘1’ ‘0’ and is based ‘0’ is onbased the on diff theerence difference of the frequencyof the frequency of the transmittedof the transmitted light pulse light as pulse in FSK. as Thein FSK. encoding The encoding is illustrated is illustrated in Figure 18in .Figure The decoding 18. The isdecoding performed is performed based on a based series on of a several series adjacentof several samples. adjacent UFSOOK samples. allows UFSOOK the useallows of both the use RS andof both GS mechanisms.RS and GS mechanisms. This compatibility This compatibility of two shutter of mechanisms two shutter is themechanisms key for practical is the UFSOOK.key for practical UFSOOK.An enhanced version of the UFSOOK modulation was reported with the encoding performed on the phase shift rather than the frequency. It is known as undersampled phase shift on-off keying (UPSOOK) [68,69]. UPSOOK offers improved transmission efficiency of 1 bit/frame/LED, compared with only 0.5 bit/frame/LED for UFSOOK [68]. In [66], it was reported that the UPSOOK with dual LEDs with a designated mapping and framing method offers a data rate of 150 b/s over a transmission link span of longer than 10 m. Alternatively, undersampling for the low-rate data stream can also be performed spatially using a spatial modulation scheme, i.e., spatial 2-phase shift keying (S2-PSK), which was submitted to TG7m [14,64]. The capability of S2-PSK to demodulate with random sampling allows it to be used with time-variant frame rate cameras. Electronics 2020, 9, 1339 18 of 44 Electronics 2020, 9, x FOR PEER REVIEW 18 of 45

Figure 18. UFSOOK encoding of a logic “0 1” bit pattern. Figure 18. UFSOOK encoding of a logic “0 1” bit pattern. 3.2.2. High Rate MIMO Data Stream An enhanced version of the UFSOOK modulation was reported with the encoding performed on theThere phase are shift two rather proposed than schemesthe frequency. for a high-rate It is known MIMO as undersampled in RoI-S, i.e., thephase twinkle shift variableon-off keying pulse (UPSOOK)position modulation [68,69]. UPSOOK (VPPM) andoffers hybrid improved spatial transmi phasession shift keyingefficiency (HS-PSK) of 1 bit/frame/LED, [5,67,70]. The compared former is a withcombination only 0.5 ofbit/frame/LED UFSOOK and for VPPM, UFSOOK wherein [68]. VPPMIn [66], is it achieved was reported by changing that the the UPSOOK duty cycle with of dual high LEDsrate pulses. with a Thedesignated low-frequency mapping UFSOOK and framing signal method is the amplitude offers a data envelope rate of 150 of the b/s high-rate over a transmission data signal, linkwhich span is usedof longer for intensity than 10 m. modulation Alternatively, of the under LED.sampling The term for “twinkle” the low-rate in the data twinkle stream VPPM can also stems be performedfrom the LED spatially lights beingusing perceiveda spatial modulation as ﬂashing rapidlyscheme, on i.e., the spatial RX [67 ].2-phase On the shift other keying hand, HS-PSK(S2-PSK), is whicha combination was submitted of both S2-PSKto TG7m and [14,64]. dimmable The spatialcapability multiple-PSK of S2-PSK (DSM-PSK). to demodulate Comparable with random to the samplingtwinkle VPPM, allows theit to amplitude be used with modulated time-variant signal frame overlaps rate cameras. the S2-PSK signals with that of generated by DSM-PSK (i.e., multiple VPPM signals with delays). DSM-PSK is also considered for the NIR at a 3.2.2.wavelength High Rate of 850 MIMO nm [Data67]. Stream

3.2.3.There Compatible are two Encodingproposed Schemeschemes in for the a Timehigh-rate Domain MIMO in RoI-S, i.e., the twinkle variable pulse position modulation (VPPM) and hybrid spatial phase shift keying (HS-PSK) [5,67,70]. The former is A study adopted a multiple frame rate encoding scheme for OCC that enables data transmission a combination of UFSOOK and VPPM, wherein VPPM is achieved by changing the duty cycle of high through different shutter speeds, sampling rates, as well as resolutions and distances [71]. This scheme rate pulses. The low-frequency UFSOOK signal is the amplitude envelope of the high-rate data signal, encodes a special data packet on the TX that transmits both data and clock information. In case of a which is used for intensity modulation of the LED. The term “twinkle” in the twinkle VPPM stems frame rate drop on the receiver camera, the error can later be corrected using external codes [71]. from the LED lights being perceived as flashing rapidly on the RX [67]. On the other hand, HS-PSK is3.3. a combination Hybrid Camera-PD of both Modulation S2-PSK and dimmable spatial multiple-PSK (DSM-PSK). Comparable to the twinkle VPPM, the amplitude modulated signal overlaps the S2-PSK signals with that of generated by DSM-PSKA hybrid (i.e., camera-PD multiple optical VPPM communication signals with delays). known DSM-PSK as optical communicationis also considered image for the sensor NIR (OCI) at a wavelengthhas been proposed. of 850 nm The [67]. OCI employs an LED, camera and PD to provide a substantially improved response to light intensity variation compared with conventional imaging pixels [5,35,72,73]. In OCI, 3.2.3.the detection Compatible of the Encoding light source Scheme from thein the image Time cells Domain is performed prior to establishing a communication link and activating the PD cells. In addition, the detection of RoI is required; however, there is no A study adopted a multiple frame rate encoding scheme for OCC that enables data transmission RoI signaling and therefore a computer vision-based RoI algorithm is required. Figure 19 illustrates through different shutter speeds, sampling rates, as well as resolutions and distances [71]. This the operation, where a transmission range of 100 m was experimentally demonstrated. This range scheme encodes a special data packet on the TX that transmits both data and clock information. In is considered exceptionally large for OCC and could provide a huge bandwidth of up to 10 MHz case of a frame rate drop on the receiver camera, the error can later be corrected using external codes due to the use of PD cells [5,72]. Unfortunately, the hybrid camera-PD concept is not applicable to [71]. mobile devices in general, since OCI is still at the prototype stage at the time of writing and is not widely available. The advantage of OCI is the provision of a significantly higher SNR for real-time 3.3. Hybrid Camera-PD Modulation communications in both V2V and V2I outdoor OCC applications [72]. OCI is also envisioned for future indoorA hybrid LiFi since camera-PD it offers the optical benefit co ofmmunication spatial separation known between as optical OCC communication and VLC, while image satisfying sensor the (OCI)5G data has rate been requirement proposed. [ 5The,14]. OCI employs an LED, camera and PD to provide a substantially improved response to light intensity variation compared with conventional imaging pixels [5,35,72,73]. In OCI, the detection of the light source from the image cells is performed prior to establishing a communication link and activating the PD cells. In addition, the detection of RoI is required; however, there is no RoI signaling and therefore a computer vision-based RoI algorithm is required. Figure 19 illustrates the operation, where a transmission range of 100 m was experimentally demonstrated. This range is considered exceptionally large for OCC and could provide a huge bandwidth of up to 10 MHz due to the use of PD cells [5,72]. Unfortunately, the hybrid camera-PD

Electronics 2020, 9, x FOR PEER REVIEW 19 of 45 Electronics 2020, 9, x FOR PEER REVIEW 19 of 45 concept is not applicable to mobile devices in general, since OCI is still at the prototype stage at the concept is not applicable to mobile devices in general, since OCI is still at the prototype stage at the time of writing and is not widely available. The advantage of OCI is the provision of a significantly time of writing and is not widely available. The advantage of OCI is the provision of a significantly higher SNR for real-time communications in both V2V and V2I outdoor OCC applications [72]. OCI higher SNR for real-time communications in both V2V and V2I outdoor OCC applications [72]. OCI is also envisioned for future indoor LiFi since it offers the benefit of spatial separation between OCC is also envisioned for future indoor LiFi since it offers the benefit of spatial separation between OCC Electronicsand VLC,2020 while, 9, 1339 satisfying the 5G data rate requirement [5,14]. 19 of 44 and VLC, while satisfying the 5G data rate requirement [5,14].

Figure 19. Basic operation of IS-OWC system. Figure 19. Basic operation of IS-OWC system. 3.4. Rolling-Shutter-Based OCC 3.4. Rolling-Shutter-Based OCC In order order to to increase increase the the data data throughput, throughput, a dedica a dedicatedted IS ISwith with a built-in a built-in PD PD(i.e., (i.e., PIN) PIN) array array was In order to increase the data throughput, a dedicated IS with a built-in PD (i.e., PIN) array was wasproposed proposed in [59]. in [59It ].is Ittoo is complex too complex to fabricate to fabricate and andnot notcommercially commercially available available yet. yet. However, However, by proposed in [59]. It is too complex to fabricate and not commercially available yet. However, by byadopting adopting the the RS RSeffect effect of ofa CMOS-based a CMOS-based IS, IS, it itis ispossible possible to to increase increase the the data data rate rate of of OCC OCC [5,24,74,75]. [5,24,74,75]. adopting the RS effect of a CMOS-based IS, it is possible to increase the data rate of OCC [5,24,74,75]. The RS-based OCC is the second commonly used schemescheme following NSM, since the implementation The RS-based OCC is the second commonly used scheme following NSM, since the implementation is practicalpractical onon anyany CMOS-basedCMOS-based camerascameras widelywidely available.available. Moreover, Moreover, the the RS RS is also termed a RSRS is practical on any CMOS-based cameras widely available. Moreover, the RS is also termed a RS patterning demodulation due to the focus on the demodulationdemodulation [[76].76]. Figure 20 shows the RS process patterning demodulation due to the focus on the demodulation [76]. Figure 20 shows the RS process where thethe datadata isis sampledsampled sequentially sequentially by by the the pixel pixe rows.l rows. In In RS, RS, the the sampling sampling rate rate is theis the product product of FofR where the data is sampled sequentially by the pixel rows. In RS, the sampling rate is the product of andFR and the the number number of pixelof pixel rows rows (i.e., (i.e., vertical vertical resolution). resolution). For For example, example, with with a 30-fpsa 30-fps camera camera having having a FR and the number of pixel rows (i.e., vertical resolution). For example, with a 30-fps camera having resolutiona resolution of FHDof FHD (1920 (19201080 × 1080 pixels), pixels), the RS the samples RS samples each imageeach image row, thus row, yielding thus yielding 30 1080 30 samples × 1080 a resolution of FHD (1920× × 1080 pixels), the RS samples each image row, thus yielding× 30 × 1080 onsamples each frame. on each This frame. leads This to an leads effective to an rate effective of 32.4 rate Ksps. of Therefore,32.4 Ksps. itTherefore, is obvious it that is obvious in RS, the that eff ectivein RS, samples on each frame. This leads to an effective rate of 32.4 Ksps. Therefore, it is obvious that in RS, datathe effective rate depends data onrate both depends the number on both of imagethe numb rowser (i.e., of theimage camera’s rows vertical(i.e., the resolution) camera’s vertical and FR. the effective data rate depends on both the number of image rows (i.e., the camera’s vertical Noteresolution) that in and RS, F theR. Note sampling that in rate RS, of the the sampling LED TXs rate could of bethe 1 LED KHz TXs or higher. could be 1 KHz or higher. resolution) and FR. Note that in RS, the sampling rate of the LED TXs could be 1 KHz or higher.

Figure 20. RS-based demodulation process. Figure 20. RS-based demodulation process.

The width of diﬀerent light intensity bands in the capture frame is proportional to the symbol rate of the LED TXs and the exposure period of the camera [17]. Adjusting the TX frequency aﬀects the width of the captured light intensity bands as illustrated in Figure 21. Electronics 2020, 9, x FOR PEER REVIEW 20 of 45

The width of different light intensity bands in the capture frame is proportional to the symbol rate of the LED TXs and the exposure period of the camera [17]. Adjusting the TX frequency affects Electronicsthe width2020 of, the9, 1339 captured light intensity bands as illustrated in Figure 21. 20 of 44

Figure 21. Captured frames of OOK modulation with different frequencies. Figure 21. Captured frames of OOK modulation with different frequencies. With respect to the operation of the CMOS-based camera, the number of samples (i.e., pixel rows) acquiredWith from respect the capturedto the operation image of of the the light CMOS-based source in RS camera, is given the as number [5]: of samples (i.e., pixel rows) acquired from the captured image of the light source in RS is given as [5]: L/ 2 tan FC 2 Nrow(d) = w/ 2tan (14) d 2 (14) () = where w denotes the image width (i.e., horizontal resolution) and L is the normalized length (i.e.,where the diameter) denotes theof the image light width source (i.e., along horizontal the width resolution) of the image and sensor. is the dnormalizedindicates the length distance (i.e., betweenthe diameter) the light of the source light (TX) source and thealong camera the width (RX). Theof the number image of sensor. samples acquired indicates from the andistance image Nbetweenrow(d) determines the light source the amount (TX) and of the information camera (RX) that. anThe image number can of capture samples in RS.acquired from an image (In) RS, determines however, the we amount have the of following information issues: that ( ani) the image blooming can capture effect, in which RS. is the difference of lightIn intensity RS, however, between we the have area the around following the light issues: source (i) the and blooming the remaining effect, frame which area is [the23, 74difference,77,78] and of (lightii) demodulation intensity between of the the bright area and around dark fringes,the light which source is challenging,and the remaining due to frame its high area data [23,74,77,78] fluctuation and a(ii large) demodulation variation in of the the extinction bright and ratio dark (ER), fringes, i.e., awhich significant is challenging, blooming due effect, to seeits Figurehigh data 22. Tofluctuation mitigate theand blooming a large variation effect, a suitable in the extinction threshold schemeratio (ER), is required i.e., a significant to level out blooming the intensity effect, across see theFigure captured 22. To framesmitigate so the that blooming an accurate effect, binary a suitab thresholdle threshold can be appliedscheme inis required order to demodulateto level out the capturedintensity bitacross stream the [ 74captured,79]. There frames are aso number that an of accu imagerate processing-based binary threshold threshold can be applied schemes in including order to 3rddemodulate order polynomial the captured curve bit fitting, stream the [74,79]. iterative There method are a number [80], and of quick image adaptive processing-based method [81 threshold]. schemesThe including quick adaptive 3rd order thresholding polynomial utilizes curve afitting, moving the average iterative calculation method [80], of grayscaleand quick valuesadaptive of pixelsmethod across [81]. the image rows [79,81]. The binary threshold is then set at the center row of the moving average results. The iterative method, on the other hand, divides the image rows into 24 sections and then applies iterative operation within each section. The iterative operation determines an initial grayscale value per each section and then divides the section into two subsets prior to averaging using addition and division by 2. Thus, it yields the ultimate grayscale values that are employed in the binary thresholding scheme [79]. These binary thresholding schemes were investigated for the RS in [79]. Note that they were implemented at the post-processing phase following image capturing. It was shown that the iterative and quick adaptive threshold schemes were able to reduce the processing time by 97% and 6%, respectively, when compared with [75].

Electronics 2020, 9, 1339 21 of 44 Electronics 2020, 9, x FOR PEER REVIEW 21 of 45

Figure 22. BloomingBlooming effect eﬀect on on RS-OCC. RS-OCC.

TheIn [79 quick], it was adaptive shown thresholding that the adaptive utilizes scheme a moving offered average the lowest calculation bit error rateof grayscale (BER) performance values of pixelscompared across with the theimage other rows two [79,81]. but atThe the binary cost ofthre increasedshold is then processing set at the time center (i.e., row 7 ms).of the Themoving 3rd averageorder polynomial results. The curve iterative fitting method, offers on the the least other processing hand, divides time the (i.e., image 0.95 rows ms) forinto the 24 sections same image and thenprocessing applies hardware iterative [operation79]. The variationswithin each of sectn-thion. order The polynomial iterative operation curve fitting determines and the an iterative initial grayscaleschemes arevalue preferable per each in section most RS-OCCsand then divides due to theirthe section lower into processing two subsets times prior and lessto averaging complex usingimplementation addition and [23 ,division74,76,77 ,by82]. 2. Thus, it yields the ultimate grayscale values that are employed in the binaryIt should thresholding be noted that scheme the RS-based [79]. OCC has more limited transmission distance compared with the RoI-S,These due binary to the thresholding dependence schemes on the requirement were investigat thated the for TX the needs RS toin be[79]. captured Note that by whole they were pixel implementedrows [83]. The at transmissionthe post-processing data rate phase can following be increased image using capturing. a 2D LED It was array shown (i.e., that spatial the iterative division andmultiplexing) quick adaptive [56]. However, threshold decoding schemes the were captured able to 2D reduce RS pattern the usingprocessing a CMOS time IS-based by 97% camera and 6%, is a respectively,challenging task when due compared to the merger with [75]. of the RS pattern produced by different LEDs in the array. Hence, thereIn is a[79], need it forwas advanced shown signalthat the processing adaptive in scheme order to offered separate the these lowest patterns. bit error It is important rate (BER) to performancenote that decoding compared of the with 2D RS the pattern other withtwo but reduced at the o ffcosetst toleranceof increased and processing flexibility requires time (i.e., capturing 7 ms). Thelights 3rd from order all LEDspolynomial using thecurve IS [fitting82]. offers the least processing time (i.e., 0.95 ms) for the same image processing hardware [79]. The variations of n-th order polynomial curve fitting and the 3.5. Performance Comparison of Modulation Schemes iterative schemes are preferable in most RS-OCCs due to their lower processing times and less complexNSM-based implementation OCC schemes [23,74,76,77,82]. are mostly intended for short-range communications, i.e., several metersIt should with a databe noted rate of that Kbps the to RS-based Mbps. The OCC RoI-S has provides more limited improved transmissi efficiencyon indistance image processingcompared withand thusthe RoI-S, yields due higher to the data dependence rates of at on least the tens requirement of Kbps for that use the in TX outdoor needs applications to be captured such by aswhole V2V pixeland V2I rows communications. [83]. The transmission In addition, data rate an can OCI be sensor increased provides using an a 2D enhanced LED array version (i.e., spatial of RoI-S division with a multiplexing)novel IS, which [56]. is a However, hybrid between decoding IS and the PD, captured thus potentially 2D RS pattern increasing using the a CMOS data rate IS-based up to Gbps camera for isfuture a challenging applications task similar due to to the VLC. merger Finally, of RS-basedthe RS pattern schemes produced utilize theby inherentdifferent RSLEDs properties in the array. of the Hence,IS to multiply there is the a samplingneed for rateadvanced based onsignal the numberprocessing of pixelin order rows. to Thus, separate RS is these intended patterns. for indoor It is importantapplications to andnote can that also decoding comply withof the LED-based 2D RS pattern lighting with as reduced TXs. In this offset regard, tolerance RS yields and aflexibility data rate requiresof up to tenscapturing of Kbps. lights Table from3 lists all aLEDs concise using performance the IS [82]. comparison of representative OCC schemes. OCI exhibits the highest data rate of 55 Mbps due to its higher sampling rate by using PD-based 3.5.RXs Performance (communication Comparison cells) [ 35of ].Modulation It is interesting Schemes to note that both RoI-S and high-rate NSM offer a longer transmissionNSM-based distance OCC of schemes up to 30 m,are which mostly is evidentlyintended suitablefor short-range for long-range communications, V2V applications i.e., several [60,64]. metersCeilingTalk with providesa data rate the of longestKbps to RS-OCC Mbps. The transmission RoI-S provides distance improved of 5 m efficiency at the cost in of image a lower processing data rate andof 1 thus Kbps yields due to higher the implementation data rates of at of least systematic tens of Kbps raptor for coding use in in outdoor multiple applications LED-based such TXs [as84 ].V2V and V2I communications. In addition, an OCI sensor provides an enhanced version of RoI-S with a novel IS, which is a hybrid between IS and PD, thus potentially increasing the data rate up to Gbps for future applications similar to VLC. Finally, RS-based schemes utilize the inherent RS properties of the IS to multiply the sampling rate based on the number of pixel rows. Thus, RS is intended for indoor applications and can also comply with LED-based lighting as TXs. In this regard, RS yields a

Electronics 2020, 9, 1339 22 of 44

Table 3. Performance comparison of representative OCC scheme.

Rate of TX Rate of RX (Camera) Typical Distance Eﬀective Data Rate Scheme/Name (Hz) (fps) (m) (bps) NSM/PixNet [50] 3 60 (DSLR) 0–10 6.4 K–12 M NSM/Cobra [51] 15 30 (phone) 0–0.15 80–225 K NSM/2D Seq. code [53] 15 30 (webcam) <0.1 6.09–18.27 K NSM/Invisible Watermarking [12] 10 20 (webcam) 0.5–0.7 60–140 NSM/CIM-MIMO [55] 82.5 330 (FPGA-cam) 1.0 126.72 K NSM/IAC [22] 20 30 (phone) 0.3 1.28 K NSM/Split shutter [43] 60 120 (phone) 2 11.52 K NSM/HDM [37] 60 120 (phone) 0.2–0.5 326.82 K–2.66 M High-rate NSM/V2V-VLC [60] 500 1000 (Fastcam) 30–50 10 M RoI-S/Spatial 2-PSK [64] 125–200 30 and 1000 Up to 30 Up to 10 M RoI-S/Selective Capture [36] 217 435 (Raspicam) 1.75 3.46 K RoI-S/UPSOOK [68] 200–250 50 (DSLR) 0.25–0.8 50–150 Hybrid Camera-PD/OCI [35] 14.5 M 35 MHz (OCI) 1.5 55 M RS/EVA [77] 200 60 (phone) <10 cm 7.68 K RS/Beacon Jointed Packet [82] 1.92–14.04 K 30 (phone) 0.1–0.25 5.76–10.32 K RS/Ceiling-Talk [84] 80 K 30 (phone) 5 1 K

4. Mobility, Coverage, and Interference As discussed in previous sections, the low data rate of OCC (i.e., due to the limitations of the camera) is the main issue in applications where there are demands for higher transmission rates. However, OCC can be conceived as an affordable and viable short-range transmission technology in the emerging field of IoT, where a range of information from different sources can be transmitted and captured in a cost-effective manner. In other words, OCC is best suited to ubiquitous, low-rate tolerable and license-free data communications rather than conventional high-speed data transmission systems. For OCC to evolve further, several important issues need to be resolved including mobility, coverage and interference. Recently, extensive studies have been focused on addressing these issues. Some mitigation techniques are elaborated in this section.

4.1. Mobility and Coverage Both mobility and coverage are important in OCC since mobile devices are rarely static in indoor and outdoor environments. Seemingly, a high-speed camera RX is a prerequisite to solve these issues [35,60,72]. The mobility issue is more pronounced in OCC-based V2V and V2I communications. A number of studies have shown that the performance of OCC with no high-speed camera can be enhanced [17,35,78,85–87]. To support user mobility, tracking the RX’s position is essential in mobile environments. In [78], a tracking scheme based on the region-grow algorithm for the fixed column matrix selection technique was reported. However, there are still a number of issues in this scheme: (i) captured images may not have the TXs with the user mobility; (ii) high computational complexity; (iii) system performance being affected by the link span between the TX and the mobile RX due to light intensity fluctuation; (iv) inter-symbol interference (ISI), i.e., each row of pixels suffers more from the neighboring pixels induced interference, compared with stationary OCCs. Although mobility can involve the movement of both the TX and the RX, the mobility in OCC studies is largely evaluated from the camera perspective [17,78,86]. In this approach, the movement is calculated from the captured frames as well as the orientation and acceleration data provision from additional sensors in smartphone-based OCC systems. A straightforward approach was proposed through an asynchronous unidirectional scheme by employing an oversampling technique to sample more frames at the RX [85]. It is termed asynchronous since there is no uplink channel for synchronization. Note that a data-shower method without an uplink channel is based on this approach. An enhanced RS-based OCC is proposed to mitigate signal degradation caused by translational and rotational motions [88]. The solutions are Row Grayscale Redistribution (RGR) scheme and Differential Grayscale Packet Selection (DGPS) scheme. These schemes work by redistributing the Electronics 2020, 9, x FOR PEER REVIEW 23 of 45

Although mobility can involve the movement of both the TX and the RX, the mobility in OCC studies is largely evaluated from the camera perspective [17,78,86]. In this approach, the movement is calculated from the captured frames as well as the orientation and acceleration data provision from additional sensors in smartphone-based OCC systems. A straightforward approach was proposed through an asynchronous unidirectional scheme by employing an oversampling technique to sample more frames at the RX [85]. It is termed asynchronous since there is no uplink channel for synchronization. Note that a data-shower method without an uplink channel is based on this approach. An enhanced RS-based OCC is proposed to mitigate signal degradation caused by translational and rotational motions [88]. The solutions are Row Grayscale Redistribution (RGR) scheme and Electronics 2020, 9, 1339 23 of 44 Differential Grayscale Packet Selection (DGPS) scheme. These schemes work by redistributing the varying intensity (in grayscale values) between neighboring pixel rows to improve the demodulation ofvarying the binary intensity thresholding (in grayscale employ values)ed in between the RS-based neighboring OCC [88]. pixel The rows study to improveproved that the these demodulation schemes canof the correctly binary thresholdingidentify the data employed packet in captured the RS-based at 100 OCC cm [at88 a]. data The studyrate of proved 1.51 Kbit/s, that these while schemes being undercan correctly the edge-to-edge identify the translational data packet capturedmotion and at 100random cm at rotational a data rate motion. of 1.51 Kbit/s, while being under the edge-to-edgeThe oversampling translational method motion has an andoptimum random number rotational of oversampling motion. defined by Equation (15) [85], whichThe oversampling provides redundant method hasframes an optimum during data number capturing of oversampling as illustrated deﬁned in Figure by Equation 23. With (15) the [RX85], havingwhich providesredundant redundant frames for frames the same during bit, data it select capturings the best as illustrated frame with in less Figure degradation 23. With the in RXthe havingimage quality.redundant frames for the same bit, it selects the best frame with less degradation in the image quality. pps −1 2 pps k 1 = 2 (15) fps − = 3 (15) fps ≤ k 3 where pps denotes the number of pulses per second (i.e., TX switching rate) and k is a constant where pps denotes the number of pulses per second (i.e., TX switching rate) and k is a constant indicating an extra frame captured beyond the number of transmitted frames. (k − 1) is the number indicating an extra frame captured beyond the number of transmitted frames. (k 1) is the number of of transmitted pulses, which is also equal to the captured frame. For example, in −Equation (15), if pps transmitted pulses, which is also equal to the captured frame. For example, in Equation (15), if pps is is set to 667, then the minimum fps of the camera should be at least 1000. set to 667, then the minimum fps of the camera should be at least 1000.

Figure 23. Selecting two good frames among three adjacent captured frames. Figure 23. Selecting two good frames among three adjacent captured frames. Most OCC schemes are based on the LOS transmission mode due to the inherent nature of photography.Most OCC However, schemes are reflected based lights on the (i.e., LOS non-LOS transmission (NLOS) mode link) due can to also the be inherent captured nature using of a photography.camera, although However, it is not reflected possible lights for RS-based (i.e., non-LOS OCC requiring(NLOS) link) only can the also image be rowscaptured [44]. using In [76 a], camera,a unique although NLOS OCCit is not system possible for anfor indoor RS-based environment OCC requiring was investigatedonly the image in [rows76]. The[44]. bloomingIn [76], a uniqueeffect [75 NLOS] was OCC mitigated system using for aan background indoor environment compensation was algorithm investigated combined in [76]. with The Bradleyblooming adaptive effect [75]thresholding. was mitigated Note thatusing the a bloomingbackground mitigation compensa schemetion algorithm can only work combined well when with theBradley TX is locatedadaptive at thresholding.the center of capturedNote that images the blooming at the RX. mitigation The proposed scheme NLOS can only OCC work link achievedwell when a totalthe TX transmission is located atdistance the center of 1.5 of m. captured Assuming images that the at transmission the RX. The of proposed each independent NLOS OCC frame link instead achieved of a complete a total transmissionvideo stream distance with a seriesof 1.5 m. of imagesAssuming was that performed, the transmission the data of rates each (theindependent net data frame per frame) instead were of aapproximately complete video 96 stream and 172 with bits /aframe series over of images the link was spans performed, of 1.5 and the 0.40 data m, rates respectively. (the net data In [89 per], to frame) avoid werethe blooming approximately effect, a96 dynamic and 172 columnbits/frame matrix over selection the link spans algorithm of 1.5 matrix and 0.40 with m, a respectively. high extinction-ratio In [89], was proposed for OCC with mobility, achieving a data throughput of 1.08 Kbps. The experimental results showed that the link performance degraded when increasing both the transmission span and the user’s speed. Another approach to address both the OCC coverage and mobility was reported, which used a wide RX orientation (WRO) for LOS and NLOS links simultaneously, as shown in Figure 24[ 86]. WRO was proposed along with the design of the TX in order to distribute the illumination equally over a 120◦ irradiance angle, thus providing a wider coverage compared with existing OCC systems, albeit with a tradeoff in distance. Mobility is supported by utilizing a short exposure time, which successfully mitigated the need for the static RX. An adaptive blooming mitigation algorithm was employed to provide robust blooming mitigation for both LOS and NLOS based OCC systems [86]. Electronics 2020, 9, x FOR PEER REVIEW 24 of 45 to avoid the blooming effect, a dynamic column matrix selection algorithm matrix with a high extinction-ratio was proposed for OCC with mobility, achieving a data throughput of 1.08 Kbps. The experimental results showed that the link performance degraded when increasing both the transmission span and the user’s speed. Another approach to address both the OCC coverage and mobility was reported, which used a wide RX orientation (WRO) for LOS and NLOS links simultaneously, as shown in Figure 24 [86]. WRO was proposed along with the design of the TX in order to distribute the illumination equally over a 120° irradiance angle, thus providing a wider coverage compared with existing OCC systems, albeit with a tradeoff in distance. Mobility is supported by utilizing a short exposure time, which successfully mitigated the need for the static RX. An adaptive blooming mitigation algorithm was Electronicsemployed2020 to, 9provide, 1339 robust blooming mitigation for both LOS and NLOS based OCC systems24 [86]. of 44

Figure 24. Wide RX orientation scheme: (a) providing both LOS and NLOS communication links; (Figureb) wide 24. RX Wide orientation. RX orientation scheme: (a) providing both LOS and NLOS communication links; (b) wide RX orientation. The WRO scheme provides a free orientation of 90◦, confirmed on a small-scale indoor testbed. It providesThe WRO maximum scheme transmission provides a free distances orientation of up of to 90 50o, and confirmed 141 cm foron LOSa small- andscale NLOS, indoor respectively testbed. andIt provides a data ratemaximum of 6.72 kbps.transmission The scheme distances wasomnidirectional, of up to 50 and 141 i.e., cm reflections for LOS fromand NLOS, all walls respectively within the room,and a data and therate mobility of 6.72 kbps. was achieved The scheme by means was omnidirectional, of a handheld RX, i.e., which reflections was not from strictly all walls static within even underthe room, a hand and movement the mobility of upwas to achieved 0.30 m/s. Similarly,by means higherof a handheld mobility RX, was which further was evaluated not strictly in the static LOS RS-basedeven under OCC a hand link movement for up to 0.80 of mup/s to [89 0.30]. m/s. Similarly, higher mobility was further evaluated in the LOS RS-based OCC link for up to 0.80 m/s [89]. 4.2. Interference Mitigation 4.2. InterferenceAnother performance Mitigation enhancement feature of OCC is the interference mitigation capability due to theAnother versatility performance of the camera enhancement for capturing feature light of overOCC ais wide the interference range spectrum. mitigation In OCC, capability one of due the importantto the versatility interferences of the observedcamera for in thecapturing captured ligh imaget over is a adjacent wide range TXs induced spectrum. inter-pixel In OCC, interference one of the (IPI)important [17,22 ].interferences IPI could also observed be due to improperin the captured focusing whenimage capturing is adjacent flickering TXs lightsinduced [22]. inter-pixel In general, cameras’interference autofocusing (IPI) [17,22]. feature IPI could is used also priorbe due to to capturing improper the focusing image [when29]. The capturing proposed flickering IAC scheme lights includes[22]. In general, dedicated cameras’ illumination autofocusing LEDs (i.e., feature TX) tois mitigateused prior the to interference capturing the by providingimage [29]. fixed The referencesproposed forIAC focusing scheme for includes the camera dedicated [5,22]. illumination Most cameras LEDs have (i.e automatic., TX) to focusmitigate and the light interference metering functionalities. by providing Sincefixed references OCC utilizes for flickeringfocusing for LEDs, the thesecamera flickers [5,22]. might Most confusecameras the have camera’s automatic automatic focus and focusing light andmetering light-metering functionalities. function. Since The OCC illumination utilizes LEDsflickering are thus LEDs, kept these always flickers on (no might flickering) confuse to serve the ascamera’s fixed references automatic for focusing auto focusing and light-metering and light metering function. in theThe camera. illumination As illustrated LEDs are in Figurethus kept 25, IACalways successfully on (no flickering) minimizes to the serve corona as /bloomingfixed references (inter-pixel for auto interferences focusing betweenand light the metering adjacent in LEDs) the Electronicsaroundcamera. LEDs 2020As ,illustrated 9 with, x FOR higher PEER in REVIEW accurateFigure 25, color IAC demodulation. successfully minimizes the corona/blooming (inter-pixel25 of 45 interferences between the adjacent LEDs) around LEDs with higher accurate color demodulation.

Figure 25. Camera capture frame of IAC: (a) corona/blooming causing inter-pixel interference; (b) correct Figure 25. Camera capture frame of IAC: (a) corona/blooming causing inter-pixel interference; focus and accurate color interpretation [22]. (b) correct focus and accurate color interpretation [22].

Alternately, the IPI can be mitigated by using visual MIMO, specifically utilizing an image processing technique to extract the specific RoI prior to demodulation as illustrated in Figure 26 [17]. RoI is extracted by detecting the difference in intensity between the TXs and the background lights. It is true that visual MIMO is a throughput enhancement scheme in OCC. However, since the camera utilizes 2D spatial information, the visual MIMO can also be employed to classify and separate the transmitter lights (LEDs) and the interfering lights spatially through the captured frames. Therefore, providing an interference mitigation. Focus is indeed one of the major causes for interferences in a camera-based receiver, the solution of which is provided by IAC [22]. In addition, a tracking functionality is employed in visual MIMO to detect the movement of the TX, which is observable between frames.

Figure 26. A visual MIMO scheme.

Since OCC operation is generally bounded by both the LOS path and the limitations of cameras, there are several issues that need addressing: directionality, focusing, alignment and distance. In [90], a pixelated MIMO scheme with a TX (an array of LEDs) fitted with a collimating lens was experimentally investigated for transmission in the angular domain rather than in the space domain. A series of time-varying coded (i.e., Raptor code) images were captured and decoded using a camera, which was focused at infinity, thus achieving a throughput of 30 bit/frame at a link span of 1 m. In addition, this scheme is robust against misalignment, camera focus and orientation. It is evident that additional optical filters can be used to reduce the interference, which resides outside the modulated color and intensity of the light [17,56]. However, including additional filters will make the camera bulky, therefore making the OCC system impractical. An alternative interference mitigation scheme would be to shorten the camera’s exposure time, thus capturing a reduced amount of light from surroundings, while focusing mainly on the light from the TXs [86,91]. In [91], as illustrated in Figure 27, an interference mitigation scheme was proposed for IR-based OCC (IOCC) operating at the NIR spectrum band, more specifically at a wavelength of 940 nm. Although all cameras can capture NIR lights, the IR-blocking filter attached to the IS reduces the sensitivity at the NIR light. In IOCC, it is customary that the IR blocking filter is replaced with a diffuser in order to increase its robustness for capturing from IR LEDs and thus reducing the interference from other sources [91].

Electronics 2020, 9, x FOR PEER REVIEW 25 of 45

Figure 25. Camera capture frame of IAC: (a) corona/blooming causing inter-pixel interference; (b) correct focus and accurate color interpretation [22].

Electronics 9 Alternately,2020, , 1339 the IPI can be mitigated by using visual MIMO, specifically utilizing an 25image of 44 processing technique to extract the specific RoI prior to demodulation as illustrated in Figure 26 [17]. RoI isAlternately, extracted by the detecting IPI can the be mitigateddifference byin in usingtensity visual between MIMO, the TXs specifically and the utilizingbackground an imagelights. processingIt is true that technique visual MIMO to extract is a throughput the specific enha RoI priorncement to demodulation scheme in OCC. as illustratedHowever, insince Figure the camera26[ 17]. RoIutilizes is extracted 2D spatial by information, detecting the the diff visualerence MIMO in intensity can also between be employed the TXs to and classify the background and separate lights. the Ittransmitter is true that lights visual (LEDs) MIMO and is athe throughput interfering enhancement lights spatially scheme through in OCC. the captured However, frames. since the Therefore, camera utilizesproviding 2D an spatial interference information, mitigation. the visual Focus MIMO is indeed can alsoone beof employedthe major causes to classify for interferences and separate in the a transmittercamera-based lights receiver, (LEDs) the and solution the interfering of which lightsis provided spatially by throughIAC [22]. the captured frames. Therefore, providingIn addition, an interference a tracking mitigation. functionality Focus is employed is indeed in one visual of the MIMO major to causes detect forthe interferencesmovement ofin the a camera-basedTX, which is observable receiver, the between solution frames. of which is provided by IAC [22].

FigureFigure 26.26. A visual MIMO scheme.

InSince addition, OCC operation a tracking is functionality generally bounded is employed by both in the visual LOS MIMO path and to detect the limitations the movement of cameras, of the TX,there which are several is observable issues that between need addressing: frames. directionality, focusing, alignment and distance. In [90], a pixelatedSince OCC MIMO operation scheme is generallywith a TX bounded (an array by both of theLEDs) LOS fitted path andwith the a limitationscollimating of lens cameras, was thereexperimentally are several investigated issues that need for transmission addressing: directionality,in the angular focusing,domain rather alignment than andin the distance. space domain. In [90], aA pixelated series of MIMOtime-varying scheme coded with a(i.e., TX (anRaptor array code) of LEDs) images fitted were with captured a collimating and decoded lens was using experimentally a camera, investigatedwhich was focused for transmission at infinity, inthus the achieving angular domain a throughput rather of than 30 bit/frame in the space at a domain. link span A of series 1 m. ofIn time-varyingaddition, this codedscheme (i.e., is robust Raptor against code) images misalignment, were captured camera and focus decoded and orientation. using a camera, which was focusedIt is at evident infinity, that thus additional achieving optical a throughput filters can of be 30 used bit/frame to reduce at a the link interference, span of 1 m. which In addition, resides thisoutside scheme the modulated is robust against color misalignment,and intensity of camera the light focus [17,56]. and orientation.However, including additional filters will Itmake is evident the camera that additional bulky, opticaltherefore filters making can be the used OCC to reduce system the impractical. interference, An which alternative resides outsideinterference the modulated mitigation colorscheme and would intensity be to of theshor lightten the [17 ,camera’s56]. However, exposure including time, additionalthus capturing filters a willreduced make amount the camera of light bulky, from therefore surroundings, making while the OCC focusing system mainly impractical. on the Anlight alternative from the interferenceTXs [86,91]. mitigationIn [91], scheme as illustrated would bein Figure to shorten 27, an the interference camera’s exposure mitigation time, scheme thus capturing was proposed a reduced for IR-based amount ofOCC light (IOCC) from surroundings, operating at the while NIR focusing spectrum mainly band, on more the light specifically from the TXsat a [wavelength86,91]. of 940 nm. AlthoughIn [91 ],all as cameras illustrated can in capture Figure 27NIR, an lights, interference the IR-blocking mitigation filter scheme attached was proposed to the IS forreduces IR-based the OCCsensitivity (IOCC) at the operating NIR light. at the In NIRIOCC, spectrum it is customary band, more that specificallythe IR blocking at a filter wavelength is replaced of 940 with nm. a Althoughdiffuser in all order cameras to increase can capture its robustness NIR lights, for the capturing IR-blocking from filter IR attachedLEDs and to thethus IS reducing reduces thethe sensitivityinterference at from the NIR other light. sources In IOCC, [91]. it is customary that the IR blocking filter is replaced with a diffuser in order to increase its robustness for capturing from IR LEDs and thus reducing the interference from other sources [91]. In OCC, IOCC was used as an uplink to provide a data rate of 6.72 Kbps over a link distance of 100 cm using IR LEDs with a total rated transmit power of 600 mW. Note that the achieved transmission distance is relatively longer compared with other RS-OCC schemes, due mainly to the effectiveness of mitigating the visible light induced interference. Note that in daytime outdoor applications, sunlight is considered the main interference for OCC schemes [35]. The use of the RoI technique (Visual MIMO) is mostly utilized in outdoor OCC schemes. The 2D captured images from the camera are effective to classify and separate the TXs spatially from the sunlight, thus improving the SNR [17,35]. The employment of additional polarizer would further improve the signal intensity in outdoor OCC applications, especially during daytime. Alternatively, outdoor OCC applications at night can be provided using an NIR light TX such as IOCC. It is applicable at night since NIR is unobtrusive to the human eyes and there is no sunlight that emits strong IR interference. Electronics 2020, 9, 1339 26 of 44 Electronics 2020, 9, x FOR PEER REVIEW 26 of 45

Figure 27. IOCC: (a) experiment setup; (b) block diagram [91]. Figure 27. IOCC: (a) experiment setup; (b) block diagram [91]. In addition, moving transmitters, generally in vehicular outdoor OCCs, present a unique interferenceIn OCC, of IOCC overlapping was used LEDs. as an The uplink solution to provide was proposed a data rate by of tracking 6.72 Kbps the movementover a link ofdistance LEDs onof 100each cm successive using IR frame LEDs [92 with]. To separatea total rated the overlapping transmit power LEDs, of the 600 proposed mW. Note method that uses the aachieved series of transmissionconsecutive frames distance as theis relatively input to tracklonger the comp movementared with of each other LED RS-OCC based schemes, on their centroids due mainly (center to the of Electronics 2020, 9, x FOR PEER REVIEW 27 of 45 effectivenesscircle), as illustrated of mitigating in Figure the 28 .visible light induced interference. Note that in daytime outdoor applications, sunlight is considered the main interference for OCC schemes [35]. The use of the RoI technique (Visual MIMO) is mostly utilized in outdoor OCC schemes. The 2D captured images from the camera are effective to classify and separate the TXs spatially from the sunlight, thus improving the SNR [17,35]. The employment of additional polarizer would further improve the signal intensity in outdoor OCC applications, especially during daytime. Alternatively, outdoor OCC applications at night can be provided using an NIR light TX such as IOCC. It is applicable at night since NIR is unobtrusive to the human eyes and there is no sunlight that emits strong IR interference. In addition, moving transmitters, generally in vehicular outdoor OCCs, present a unique interference of overlapping LEDs. The solution was proposed by tracking the movement of LEDs on each successive frame [92]. To separate the overlapping LEDs, the proposed method uses a series of consecutive frames as the input to track the movement of each LED based on their centroids (center of circle), as illustrated in Figure 28. Figure 28. Detection algorithm for overlapping LEDs [92]. Figure 28. Detection algorithm for overlapping LEDs [92]. 5. Applications 5. Applications OCC is deemed pragmatic due to the use of existing camera-based RXs that are widely available in smartOCC devices. is deemed This pragmatic versatility due of to the the camera-based use of existing RX camera-based allows OCC forRXs a that range are of widely applications available in indoorin smart and devices. outdoor This environments versatility of [the25]. camera-based As outlined inRX the allows IEEE OCC 802.15.7m for a range services, of applications various OCC in applicationsindoor and outdoor are supported environments by PHY IV,[25]. V, andAs outl VI [ined14] for in RoI-S,the IEEE RS OCC,802.15.7m and NSM services, OCC, various respectively. OCC applications are supported by PHY IV, V, and VI [14] for RoI-S, RS OCC, and NSM OCC, respectively.

5.1. Indoor Schemes Indoor OCC schemes are the most prominent, as the majority of the intended applications among the use cases listed in the standardization are for indoor environments [25]. The use cases for indoor schemes include 2D LED color code [43,51,52,55], D2D as well as relay applications [37,38], digital signage [12,52,93], exhibition or store services [23,49,84,94,95] and in-flight services [49,52]. A combined version of these use cases is proposed for a more complex system such as smart factories, smart offices, and smart homes as part of the 4th Industrial Revolution solutions [25]. Several applications of OCC are described in this section to highlight the versatility of indoor OCC systems. An OCC combined with motion detection was proposed in the framework of motion over camera. Figure 29 shows its concept [96]. The main principle behind this concept is based on the nature of image capturing at the RX, where the differences between adjacent frames can be used to detect the object motion, such as a pointing finger. The motion detection works in conjunction with the OCC through RoI selection, which can further be enhanced using a highly accurate NN-based motion detection scheme [97]. Note that the motion detection scheme is envisioned as an additional control function in OCC for use in smart homes and offices.

Figure 29. A motion over camera scheme.

Electronics 2020, 9, x FOR PEER REVIEW 27 of 45

Figure 28. Detection algorithm for overlapping LEDs [92].

5. Applications OCC is deemed pragmatic due to the use of existing camera-based RXs that are widely available in smart devices. This versatility of the camera-based RX allows OCC for a range of applications in indoor and outdoor environments [25]. As outlined in the IEEE 802.15.7m services, various OCC Electronicsapplications2020, 9are, 1339 supported by PHY IV, V, and VI [14] for RoI-S, RS OCC, and NSM OCC, respectively.27 of 44

5.1. Indoor Schemes 5.1. Indoor Schemes Indoor OCC schemes are the most prominent, as the majority of the intended applications Indoor OCC schemes are the most prominent, as the majority of the intended applications among among the use cases listed in the standardization are for indoor environments [25]. The use cases for the use cases listed in the standardization are for indoor environments [25]. The use cases for indoor indoor schemes include 2D LED color code [43,51,52,55], D2D as well as relay applications [37,38], schemes include 2D LED color code [43,51,52,55], D2D as well as relay applications [37,38], digital digital signage [12,52,93], exhibition or store services [23,49,84,94,95] and in-flight services [49,52]. A signage [12,52,93], exhibition or store services [23,49,84,94,95] and in-flight services [49,52]. A combined combined version of these use cases is proposed for a more complex system such as smart factories, version of these use cases is proposed for a more complex system such as smart factories, smart offices, smart offices, and smart homes as part of the 4th Industrial Revolution solutions [25]. Several and smart homes as part of the 4th Industrial Revolution solutions [25]. Several applications of OCC applications of OCC are described in this section to highlight the versatility of indoor OCC systems. are described in this section to highlight the versatility of indoor OCC systems. An OCC combined with motion detection was proposed in the framework of motion over An OCC combined with motion detection was proposed in the framework of motion over camera. camera. Figure 29 shows its concept [96]. The main principle behind this concept is based on the Figure 29 shows its concept [96]. The main principle behind this concept is based on the nature of image nature of image capturing at the RX, where the differences between adjacent frames can be used to capturing at the RX, where the differences between adjacent frames can be used to detect the object detect the object motion, such as a pointing finger. The motion detection works in conjunction with motion, such as a pointing finger. The motion detection works in conjunction with the OCC through the OCC through RoI selection, which can further be enhanced using a highly accurate NN-based RoI selection, which can further be enhanced using a highly accurate NN-based motion detection motion detection scheme [97]. Note that the motion detection scheme is envisioned as an additional scheme [97]. Note that the motion detection scheme is envisioned as an additional control function in control function in OCC for use in smart homes and offices. OCC for use in smart homes and offices.

Figure 29. A motion over camera scheme. Figure 29. A motion over camera scheme. While most OCC studies have focused on unidirectional links, a bidirectional OCC utilizing RS was proposed with the camera-based RX for the uplink and the solar-cell-based RX for the downlink for IoT applications [98]. Note that the solar cell was used for a low-light requirement (i.e., as low as 100 lux of illuminance) for the downlink. One of the most attractive OCC applications can be in D2D communications employing NSM [37,38,43]. In [38], illustrated in Figure 30, a smartphone VLC (SVLC) link employing a camera-based RX for short-range D2D communications was investigate. In this scheme, the RX captured the data from the screen of the TX and used a robust algorithm for detection. It applied projective transformation to accurately eliminate perspective distortions caused by the displacement of the devices with a 98% success rate. Similar to a dynamic 2D barcode, SVLC offers the possibility of transferring a significant amount of data by simply displaying it on the screen of a smart device. A data rate of 112.5 Kbps was achieved with only single-color modulation. SVLC has the potential of delivering a higher data rate of ~300 Kbps using RGB-based modulation. Despite its limited transmission distance of 15 cm, SVLC is preferable for D2D based financial transactions. An interesting smartphone-based scheme for D2D using NSM was proposed in [37], which used shape modulation to increase the data being modulated on each frame. Due to the high-density modulation on each frame, the achieved data rate was 2.66 Mbps at a transmission distance of 20 cm. Note that there is a tradeoff between the distance and the data rate, i.e., increasing the distance to 200 cm results in reduced data rate of 15.38 bps [93]. The scheme was designed to support the uplink Electronics 2020, 9, x FOR PEER REVIEW 28 of 45

While most OCC studies have focused on unidirectional links, a bidirectional OCC utilizing RS was proposed with the camera-based RX for the uplink and the solar-cell-based RX for the downlink for IoT applications [98]. Note that the solar cell was used for a low-light requirement (i.e., as low as 100 lux of illuminance) for the downlink. One of the most attractive OCC applications can be in D2D communications employing NSM [37,38,43]. In [38], illustrated in Figure 30, a smartphone VLC (SVLC) link employing a camera-based RX for short-range D2D communications was investigate. In this scheme, the RX captured the data from the screen of the TX and used a robust algorithm for detection. It applied projective transformation to accurately eliminate perspective distortions caused by the displacement of the devices with a 98% success rate. Similar to a dynamic 2D barcode, SVLC offers the possibility of transferringElectronics 2020 ,a9 ,significant 1339 amount of data by simply displaying it on the screen of a smart device.28 of A 44 data rate of 112.5 Kbps was achieved with only single-color modulation. SVLC has the potential of delivering a higher data rate of ~300 Kbps using RGB-based modulation. Despite its limited transmission of a more complex OCC system, such as exhibition service or digital signage. The system transmission distance of 15 cm, SVLC is preferable for D2D based financial transactions. is robust with an uplink failure rate of 10% [93,94].

Figure 30. A screen-based TX for SVLC [38]. Figure 30. A screen-based TX for SVLC [38]. In recent years, we have seen a growing level of interest in indoor positioning systems (IPSs). Currently,An interesting there are smartphone-based a number of IPSs suchscheme as radiofor D2D frequency using NSM identification was proposed (RFID), in [37], ultra-wideband which used shape(UWB), modulation and Wi-Fi [99to –increase101]. However, the data these being schemes modu sulatedffer on from each RF electromagneticframe. Due to the interference high-density and modulationare costly. Bothon each camera frame, and the PD achieved can be data used rate as thewas RX 2.66 to Mbps detect at intensity a transmission modulated distance light of signals20 cm. Notefrom that LEDs. there OCC-based is a tradeoff IPSs between have extensively the distance been an investigatedd the data rate, for i.e., IoT increasing applications. the The distance OCC-based to 200 cmIPS results for 2D in and reduced 3D positioning data rate oofffers 15.38 a number bps [93]. of The advantages scheme was including designed high to positioning support the accuracy uplink transmission(i.e., less than of centimeters), a more complex license-free, OCC system, no RFsuch electromagnetic as exhibition service interference or digital and signage. low cost, The sincesystem it isuses robust the with 2D information an uplink failure from rate the camera-basedof 10% [93,94]. RX [68,95,102–109]. The OCC based positioning schemeIn recent is also years, deemed we more have practical seen a growing since it canlevel be of implemented interest in indoor directly positioning on the smartphones’ systems (IPSs). front Currently,cameras [105 there,107 are]. a number of IPSs such as radio frequency identification (RFID), ultra-wideband (UWB),In theand camera-based Wi-Fi [99–101]. IPS, However, the RX’s positionthese schemes is determined suffer from based RF on electromagnetic the coordinates interference of LEDs and and the arecaptured costly. image Both camera [14–16]. and There PD are can various be used algorithms as the RX for to implementingdetect intensity IPS modulated in OCC, which light signals are classified from LEDs.into two OCC-based categories IPSs based have on extensively the reference been for inve positioning,stigated for see IoT Figure applications. 31. The The illumination OCC-based LEDs IPS forbroadcast 2D and the 3D references positioning in theoffers form a number of fixed of coordinates advantages relative including to the high dimension positioning of the accuracy room/indoor (i.e., lessspace than [105 centimeters),]. In the first license-free, IPS scheme, no the RF smartphoneelectromagnetic camera interference acting as and the low user cost, determines since it uses its own the 2Dposition information based onfrom the the references camera-based set by theRX illumination[68,95,102–109]. LEDs. The In OCC the secondbased positioning type of IPS scheme scheme, is a alsocamera deemed is used more as the practical reference, since which it can actively be implemen looks forted moving directly users on the to performsmartphones’ localization front cameras for each [105,107].userElectronics [91,104 2020]., 9, x FOR PEER REVIEW 29 of 45 In the camera-based IPS, the RX’s position is determined based on the coordinates of LEDs and the captured image [14–16]. There are various algorithms for implementing IPS in OCC, which are classified into two categories based on the reference for positioning, see Figure 31. The illumination LEDs broadcast the references in the form of fixed coordinates relative to the dimension of the room/indoor space [105]. In the first IPS scheme, the smartphone camera acting as the user determines its own position based on the references set by the illumination LEDs. In the second type of IPS scheme, a camera is used as the reference, which actively looks for moving users to perform localization for each user [91,104].

Figure 31. Indoor positioning schemes for OCC. Figure 31. Indoor positioning schemes for OCC. The most popular localization algorithm adopted in IPS is the angle-of-arrival (AOA), The most popular localization algorithm adopted in IPS is the angle-of-arrival (AOA), which which calculates the user’s position based on the projected lights from three or more LED calculates the user’s position based on the projected lights from three or more LED TXs [11,102,105]. The AOA is a highly regarded IPS scheme in OCC, since it calculates both 3D position and the orientation of the camera. The optical AOA is commonly employed in OCC with a minimum of three TXs [105]. Among some IPS schemes, a dual-camera implementation was performed for improved accuracy, including the NIR medium that is considered safer to human eyes [91,104,106,110]. Machine learning has also been widely used to solve a range of problems in various applications such as data mining, pattern recognition, medical imaging, artificial intelligence, etc. In addition, NN is utilized to increase the accuracy of single-camera positioning by training the variants of pitch, roll, and yaw at the coordinate of the camera [95]. In [111], an OCC based IPS using NN was reported, where LEDs were grouped into blocks and the block coordinate was encoded on a single LED per block. A camera was used to decode the block coordinate to estimate the RX’s position using a trained back propagation NN. The experimental results showed that the mean positioning error, which is defined as the distance difference between the reference and estimated points, was 1.4 cm. Philips Lighting launched an OCC-based IPS, which works in conjunction with Bluetooth low energy concept [112]. It showed the practicality of OCC as a complementary wireless IPS to be adopted in IoT and more specifically in indoor environments for commercial purposes. It should be noted that the OCC solution [113] cannot be implemented for long period usage similar to wireless sensor network since the average power consumption for OCC capable devices is approximately 3.5 W [113,114].

5.2. Outdoor Schemes The main outdoor OCC applications are in smart cities for V2V, V2I, and underwater and seaside communications [5,14,25,115]. In V2V and V2I communications, spatial separation is the key advantage of an IS-based RX, as shown in Figure 32 [35,60,64,72,87,116]. Since either the TX or the RX are moving in outdoor environments, most popular modulation schemes for OCC are high-rate NSM, RoI-S, and hybrid camera-PD utilizing OCI sensors [35]. RoI image post-processing is predominantly utilized in outdoor OCC schemes to separate the TXs from severe light interference and noise, such as sunlight, thus improving the signal to noise and interference ratio.

Electronics 2020, 9, 1339 29 of 44

TXs [11,102,105]. The AOA is a highly regarded IPS scheme in OCC, since it calculates both 3D position and the orientation of the camera. The optical AOA is commonly employed in OCC with a minimum of three TXs [105]. Among some IPS schemes, a dual-camera implementation was performed for improved accuracy, including the NIR medium that is considered safer to human eyes [91,104,106,110]. Machine learning has also been widely used to solve a range of problems in various applications such as data mining, pattern recognition, medical imaging, artificial intelligence, etc. In addition, NN is utilized to increase the accuracy of single-camera positioning by training the variants of pitch, roll, and yaw at the coordinate of the camera [95]. In [111], an OCC based IPS using NN was reported, where LEDs were grouped into blocks and the block coordinate was encoded on a single LED per block. A camera was used to decode the block coordinate to estimate the RX’s position using a trained back propagation NN. The experimental results showed that the mean positioning error, which is defined as the distance difference between the reference and estimated points, was 1.4 cm. Philips Lighting launched an OCC-based IPS, which works in conjunction with Bluetooth low energy concept [112]. It showed the practicality of OCC as a complementary wireless IPS to be adopted in IoT and more specifically in indoor environments for commercial purposes. It should be noted that the OCC solution [113] cannot be implemented for long period usage similar to wireless sensor network since the average power consumption for OCC capable devices is approximately 3.5 W [113,114].

5.2. Outdoor Schemes The main outdoor OCC applications are in smart cities for V2V, V2I, and underwater and seaside communications [5,14,25,115]. In V2V and V2I communications, spatial separation is the key advantage of an IS-based RX, as shown in Figure 32[ 35,60,64,72,87,116]. Since either the TX or the RX are moving in outdoor environments, most popular modulation schemes for OCC are high-rate NSM, RoI-S, and hybrid camera-PD utilizing OCI sensors [35]. RoI image post-processing is predominantly utilized in outdoor OCC schemes to separate the TXs from severe light interference and noise, such as sunlight, thusElectronics improving 2020, 9, x the FOR signal PEER REVIEW to noise and interference ratio. 30 of 45

FigureFigure 32.32. SpatialSpatial separationseparation ofof multiplemultiple sourcessources usingusing anan imageimage sensor.sensor.

TheThe flickeringflickering ofof thethe LED-basedLED-based TXsTXs inin bothboth V2VV2V andand V2IV2I communicationscommunications isis regulatedregulated toto bebe imperceptible to human eyes. Therefore, the camera’s FR must be higher than 100 fps [60]. However, imperceptible to human eyes. Therefore, the camera’s FR must be higher than 100 fps [60]. However, sincesince high-speedhigh-speed andand high-costhigh-cost camerascameras areare notnot usedused inin vehicles,vehicles, anan OCIOCI sensorsensor isis anan importantimportant breakthroughbreakthrough inin OCCOCC forfor vehicular applications as mentioned mentioned previously previously in in Section Section III.C. 3.3. OCIOCI ooffersffers thethe dualdual benefitsbenefits ofof spatialspatial separationseparation ofof thethe cameracamera andand thethe higherhigher bandwidthbandwidth ofof thethe PD-basedPD-based RX,RX, thusthus oofferingffering aa practicalpractical datadata raterate ofof 10–5510–55 MbpsMbps inin vehicularvehicular applicationsapplications [[3355,60].,60]. OnOn thethe other hand, relay-basedrelay-based communicationscommunications areare alsoalso investigatedinvestigated forfor outdooroutdoor OCCOCC V2VV2V applications.applications. TheThe relayrelay isis especiallyespecially importantimportant betweenbetween vehiclesvehicles becausebecause itit worksworks asas aa complementarycomplementary positioningpositioning systemsystem sincesince the global positioning system (GPS) is inaccessible when vehicles are passing through tunnels [60,117]. It is worth mentioning that the OCC-based V2V and V2I systems require a robust tracking algorithm to identify RoI for the moving TX, while minimizing the amount of light interference in outdoor environments [118,119]. It is worth noting that under abrupt movements in outdoor vehicular applications, LED blurriness occurs due to lens misfocus that affects OCC transmission quality. A robust vehicular OCC system was proposed to address the issues [87]. It was evaluated in simulations, employing two novel algorithms, i.e., average greyscale ratio (AGR) and gradient radial inwardness (GRI), to decode the OCC transmission even when the captured LED transmitters appear blurry. The simulation results exhibited a robust transmission for decoding the OCC transmission at a BER rate of lower than 10−3, up to a distance of 60 m [87]. Underwater and seaside communications are also envisioned within the OCC standardization activity; however, further studies need to be conducted to investigate their full potentials [25,115,120]. In [115], an initial experimental investigation of an OCC link with a camera-based RX compared with the PD-based VLC was reported. In underwater environments with high degrees of absorption and scattering, the visibility of LEDs is not significantly affected. Thus, the camera can capture the transmitting light sources over longer distances compared with the PD-based RXs. A transmission distance of 1 m was achieved at a data rate of ~100 bps with a considerable limitation of the data rate due to a low camera frame rate of 30 fps [115].

6. Potential and Challenges

6.1. Potential Developments of Optical Camera Communication Based on what has been reported in the literature on OCC, it is clear that the use of camera-based RXs makes communication much more versatile as a complementary wireless technology to existing RF-based wireless solutions. This section of the paper elaborates on several potential developments that would bring about further advancements of OCC within the OWC community.

Electronics 2020, 9, 1339 30 of 44 the global positioning system (GPS) is inaccessible when vehicles are passing through tunnels [60,117]. It is worth mentioning that the OCC-based V2V and V2I systems require a robust tracking algorithm to identify RoI for the moving TX, while minimizing the amount of light interference in outdoor environments [118,119]. It is worth noting that under abrupt movements in outdoor vehicular applications, LED blurriness occurs due to lens misfocus that affects OCC transmission quality. A robust vehicular OCC system was proposed to address the issues [87]. It was evaluated in simulations, employing two novel algorithms, i.e., average greyscale ratio (AGR) and gradient radial inwardness (GRI), to decode the OCC transmission even when the captured LED transmitters appear blurry. The simulation results 3 exhibited a robust transmission for decoding the OCC transmission at a BER rate of lower than 10− , up to a distance of 60 m [87]. Underwater and seaside communications are also envisioned within the OCC standardization activity; however, further studies need to be conducted to investigate their full potentials [25,115,120]. In [115], an initial experimental investigation of an OCC link with a camera-based RX compared with the PD-based VLC was reported. In underwater environments with high degrees of absorption and scattering, the visibility of LEDs is not significantly affected. Thus, the camera can capture the transmitting light sources over longer distances compared with the PD-based RXs. A transmission distance of 1 m was achieved at a data rate of ~100 bps with a considerable limitation of the data rate due to a low camera frame rate of 30 fps [115].

6. Potential and Challenges

6.1.1. Camera Hardware Advancements Camera frame rates are projected to increase in the near future, owing to recent advancements in the IS nanotechnology [8,11,14,121]. Additionally, the camera resolution is also progressively increasing, according to demands on the market and the development of ISs in the smart devices industry. Moreover, the image quality of the miniaturized camera has become a top priority in most recent smartphone developments. These advancements will further drive the next generation OCC systems with a significantly faster and highly accurate camera-based RX. Currently, research focus is placed on increasing the camera frame rate by allocating a higher storage speed within the camera sensors, employing a faster image processor and miniaturizing the size of a high-speed IS for use in mobile devices [13,61,122–124]. In the next decade, cameras with frame rates of thousands of fps in mobile devices are expected to be available in the market. The existing OCI hybrid sensor is also an excellent potential candidate for the RX in OCC-based V2V and V2I communications, offering a data rate of Gbps with a transmission distance of up to hundreds of meters in the near future [35,60]. On the other hand, the NSM-based OCC has an unresolved drawback of perceptible flickering of the TX, which can be mitigated by using higher speed cameras. The perceptible flickering is caused by a switching/refresh rate that is less than 100 Hz, due to the limited camera frame rate of 200 fps or lower [17,49,125]. The perceptible flicker below the flicker-fusion threshold of human eyes (i.e., 100 Hz) introduces fatigue and is therefore considered unhealthy [62]. The future research directions of OCC are to increase the data transmission rate and transmission distance. It can be made possible with a higher frame rate camera (faster image processor) beyond Electronics 2020, 9, 1339 31 of 44

200 fps and higher image resolution (larger image sensor) beyond 3840 2160 pixels in mobile devices, × especially smartphones.

6.1.2. Neural Network Assistance The developments of cameras for use in smartphones and other mobile devices can also be in augmented reality (AR) and mixed reality (MR). In addition, the implementation of intelligent image processing using artificial intelligence (AI) (i.e., NN) has become a prominent method in smartphones, which will be supported by online services such as Google [126–128]. The NN performance has also been accelerated with the implementation of dedicated hardware within smartphones in the form of a neural processing unit (NPU) as was first implemented in Apple A11 bionic chip and then Huawei HiSilicon Kirin 970 [121,129,130]. The use of NPUs accelerates the processing speed for NN-based processing including AI-based image processing in smartphones. The NN implementation has been shown to be beneficial in image processing and in data communications; furthermore, OCC relies on extensive image processing techniques [37,95,97,117]. NN and its complex form of deep/machine learning are mostly employed for pattern recognition [59]. It is worth noting that there are always synchronized patterns between the TX and the RX in a communications scheme, which can be identified by NN in a straightforward manner [37,117,131,132]. The future developments of OCC will strongly benefit from NN advancements in order to offer increased data rates, robustness in reception and improved accuracy of RoI definition. In addition, an invisible watermarking through OCC as introduced in [12] will also be improved by using NN at the RX for higher data rates and longer transmission spans.

6.1.3. Multiple Cameras Development The availability of multiple cameras in recent smartphones over the last five years has also provided additional opportunities and benefits for OCC including several FOVs, different light sensitivities and different resolutions for the RX. These features will result in wider coverage area, increased range of sensitivity for various light spectra, higher image details, increased robustness against the noise and higher data rates in OCC systems. A salient advantage of multiple-camera-based OCC is the image redundancy, i.e., multiple copies of the same captured image of the TX, which can be exploited to increase the data rate by timing the shutter of each camera. An exemplary concept is illustrated in Figure 33, where four cameras in a device with different FOVs are employed to capture the same TX. Let us assume that each camera has a frame rate of 60 fps, i.e., a total capture period of approximately 16.5 ms. It is elaborated visually through Figure 34. Provided that cameras have enough storage for capturing four frames in less than 16.5 ms, then the frame rate of the combined multiple cameras can be increased to up to 240 fps by applying a shutter delay of 4 ms between the cameras. Note that the image processor should also be capable of processing at 240 fps. Multiple cameras also provide a potential benefit in the OCC research by elevating the 2D information to 3D. This is achieved by introducing distance/depth information through the stereoscopic vision of at least two cameras or more. Figure 35 shows that multiple copies of the same object are captured by the multiple cameras, thus providing different perspectives of the same captured object. Parallax is formed due to the difference between dml and dmr for the magenta LED relative to the lateral displacement between two cameras (dl). Similarly, parallax can also be formed from dyl and dyr. A distance estimation between camera and the LED can be produced by calibrating the parallax with the physical distance measurement. Future OCC can benefit from the availability of multiple camera and stereoscopic vision to calculate distances between the multiple TXs and multiple RXs (cameras). It is a huge potential to overcome several limitations in OCC, such as data rate and mobility. Electronics 2020, 9, x FOR PEER REVIEW 32 of 45

timing the shutter of each camera. An exemplary concept is illustrated in Figure 33, where four camerasElectronics in 2020 a ,device 9, x FOR with PEER different REVIEW FOVs are employed to capture the same TX. Let us assume32 that of 45 each camera has a frame rate of 60 fps, i.e., a total capture period of approximately 16.5 ms. It is elaboratedtiming the visually shutter through of each Figure camera 34.. An Provided exemplary that camerasconcept haveis illust enoughrated storage in Figure for capturing33, where four four framescameras in inless a thandevice 16.5 with ms, different then the FOVs frame are rate employ of the edcombined to capture multiple the same cameras TX. Let can us be assume increased that toeach up tocamera 240 fps has by a frameapplying rate a ofshutter 60 fps, delay i.e., aof to 4tal ms capture between period the cameras.of approximately Note that 16.5 the ms. image It is processorelaborated should visually also through be capable Figure of processing34. Provided at th240at fps.cameras have enough storage for capturing four frames in less than 16.5 ms, then the frame rate of the combined multiple cameras can be increased Electronicsto up2020 to, 9240, 1339 fps by applying a shutter delay of 4 ms between the cameras. Note that the image32 of 44 processor should also be capable of processing at 240 fps.

Figure 33. Multiple camera OCC enhancement: (a) perspective of each camera; (b) multiple camera concept. Figure 33. Multiple camera OCC enhancement: (a) perspective of each camera; (b) multiple camera concept. Figure 33. Multiple camera OCC enhancement: (a) perspective of each camera; (b) multiple camera concept.

Electronics 2020, 9, x FOR PEER REVIEW 33 of 45

captured object. Parallax is formed due to the difference between dml and dmr for the magenta LED

relative to the lateral displacement between two cameras (). Similarly, parallax can also be formed from dyl and dyr. A distance estimation between camera and the LED can be produced by calibrating the parallax with the physicFigureal 34. distanceTimed shuttermeasurement. for multiple-camera frame rates. Figure 34. Timed shutter for multiple-camera frame rates.

Multiple cameras also provide a potential benefit in the OCC research by elevating the 2D

information to 3D. ThisFigure is achieved 34. Timed byshutter intr foroducing multiple-camera distance/depth frame rates. information through the stereoscopic vision of at least two cameras or more. Figure 35 shows that multiple copies of the same object Multipleare captured cameras by thealso multiple provide cameras,a potential thus benefit providing in the differentOCC research perspectives by elevating of the thesame 2D information to 3D. This is achieved by introducing distance/depth information through the stereoscopic vision of at least two cameras or more. Figure 35 shows that multiple copies of the same object are captured by the multiple cameras, thus providing different perspectives of the same

FigureFigure 35. 35.Stereoscopic Stereoscopic visionvision for distance distance measurement. measurement.

Future OCC can benefit from the availability of multiple camera and stereoscopic vision to calculate distances between the multiple TXs and multiple RXs (cameras). It is a huge potential to overcome several limitations in OCC, such as data rate and mobility.

6.1.4. Advanced Modulation and Communication Medium Complex modulation schemes (including multi-level and multi-carrier) as well as WDM technologies can be used to increase the data throughput in OCC systems [90,133]. In addition, cameras with increased resolution and the development of more powerful image processing algorithms will enhance the efficiency and robustness of OCC [17,37]. Another potential of OCC lies in the broader communication media of IR and UV, since there are specific types of cameras that can operate in either IR or UV light [3]. IR has the advantage of being invisible to human eyes, therefore allowing higher transmission power with much reduced health hazard, compared with the visible light spectrum. It is ideal for nighttime operations, particularly for V2V and V2I applications [3]. On the other hand, the UV light has been used for long- distance outdoor communications (mainly for military applications), since it takes advantage of the scattering properties of the atmosphere, specifically the UV-C band, which is not present at the terrestrial zone below the atmosphere [134]. These specially designed OCCs utilizing either IR or UV as a medium have yet to be studied and are going to open a new horizon for co-existing with or complementing other wireless communication schemes.

6.1.5. Automated Indoor Navigation Most of OCC studies have been focused on static cameras and an LOS communication link. However, the studies described in Section IV and V have shown several possible implementations of OCC schemes with a dynamic camera and an NLOS link. It is worth noting that there is potential development for OCC implementation in which the camera cannot look directly toward the light source while the RX also moves around continuously, namely automated indoor navigation. A high demand for automated guided vehicles (AGV), a robot for transporting goods, in industries has evolved a camera for the automatic navigation (line

Electronics 2020, 9, 1339 33 of 44

6.1.4. Advanced Modulation and Communication Medium Complex modulation schemes (including multi-level and multi-carrier) as well as WDM technologies can be used to increase the data throughput in OCC systems [90,133]. In addition, cameras with increased resolution and the development of more powerful image processing algorithms will enhance the efficiency and robustness of OCC [17,37]. Another potential of OCC lies in the broader communication media of IR and UV, since there are specific types of cameras that can operate in either IR or UV light [3]. IR has the advantage of being invisible to human eyes, therefore allowing higher transmission power with much reduced health hazard, compared with the visible light spectrum. It is ideal for nighttime operations, particularly for V2V and V2I applications [3]. On the other hand, the UV light has been used for long-distance outdoor communications (mainly for military applications), since it takes advantage of the scattering properties of the atmosphere, specifically the UV-C band, which is not present at the terrestrial zone below the atmosphere [134]. These specially designed OCCs utilizing either IR or UV as a medium have yet to be studied and are going to open a new horizon for co-existing with or complementing other wireless communication schemes.

6.1.5. Automated Indoor Navigation Most of OCC studies have been focused on static cameras and an LOS communication link. However, the studies described in Sections4 and5 have shown several possible implementations of OCC schemes with a dynamic camera and an NLOS link. It is worth noting that there is potential development for OCC implementation in which the cameraElectronics cannot2020, 9, x look FOR PEER directly REVIEW toward the light source while the RX also moves around continuously,34 of 45 namely automated indoor navigation. A high demand for automated guided vehicles (AGV), a robot forfollowing) transporting to transport goods, instorage industries items has indoors. evolved Unfort a cameraunately, for thethe automaticmaintenance navigation of the lines (line on following) the floor tois not transport as convenient storage as items maintaining indoors. Unfortunately,constant illumination the maintenance on the ceiling. of the lines on the floor is not as convenientMoreover, as maintaining AGV demands constant a illuminationmore robust on indoor the ceiling. positioning scheme which is currently dominatedMoreover, by RF-based AGV demands solutions, a more such robust as WiFi indoor and positioning Bluetooth. scheme A light-based which is positioning currently dominated solution bycan RF-based be provided solutions, by using such either as WiFi a andphotodiode Bluetooth. or Athe light-based existing camera positioning embedded solution on can the be AGV. provided The bylatter using solution either can a photodiode be more efficient or the as existing it is provided camera by embedded OCC, as onillustrated the AGV. in The Figure latter 36. solution It is, however, can be morenot practical efficient for as the it is camera provided on AGV by OCC, to face as illustratedupward, even in Figure though 36 the. It most is, however, common not light practical source for theTX would camera be on the AGV illumination to face upward, on the even ceiling. though Therefore, the most the common NLOS link light benefits source in for OCC TX wouldby receiving be the illuminationthe signal from on the diffused ceiling. Therefore, reflection theon the NLOS walls link or benefits other objects. in OCC by receiving the signal from the diffused reflection on the walls or other objects.

Figure 36. AGV potential indoor positioning scheme. Figure 36. AGV potential indoor positioning scheme.

6.1.6. Beyond 5G There is a certain massive development of wireless technologies beyond the current 5G wireless networks [19]. OCC, utilizing both the visible light and NIR light, becomes a part of future communication system envisioned in 5G and 6G standards. It is due to the fact that frequencies from 100 GHz to 3 THz, including visible light and NIR, are considered promising communication bands for the next gen wireless communication system [19]. In the wake of 5G implementation, 6G conception was formulated further to emphasize on increasing its massive bandwidth and massive interconnectivity among wireless devices, acting similar to massive relays. In order for this 6G vision to be realized, it is important to ensure various options and availability of wireless communication. OCC has a unique role in being a part of OWC for future communications beyond 5G and holds a potential for pragmatic wireless schemes since the users hold cameras in smartphone. On the other hand, unmanned aerial vehicles (UAVs) are also becoming an important part of 5G and IoT, and expected to play a crucial role in enabling more functional diversity for wireless communications [135]. It is worth noting that UAVs often have one or more cameras attached in order to see, record, evaluate, and communicate efficiently [135,136]. OCC applications within UAV therefore have potential for significant benefits beyond 5G.

6.2. Challenges While witnessing great potential and future applications for OCC, there are a number of challenges that are not fully addressed or only partially studied [5,11,17], which need further investigations in order to make OCC a more practical wireless technology for emerging future applications within the context of smart societies.

Electronics 2020, 9, 1339 34 of 44

6.1.6. Beyond 5G There is a certain massive development of wireless technologies beyond the current 5G wireless networks [19]. OCC, utilizing both the visible light and NIR light, becomes a part of future communication system envisioned in 5G and 6G standards. It is due to the fact that frequencies from 100 GHz to 3 THz, including visible light and NIR, are considered promising communication bands for the next gen wireless communication system [19]. In the wake of 5G implementation, 6G conception was formulated further to emphasize on increasing its massive bandwidth and massive interconnectivity among wireless devices, acting similar to massive relays. In order for this 6G vision to be realized, it is important to ensure various options and availability of wireless communication. OCC has a unique role in being a part of OWC for future communications beyond 5G and holds a potential for pragmatic wireless schemes since the users hold cameras in smartphone. On the other hand, unmanned aerial vehicles (UAVs) are also becoming an important part of 5G and IoT, and expected to play a crucial role in enabling more functional diversity for wireless communications [135]. It is worth noting that UAVs often have one or more cameras attached in order to see, record, evaluate, and communicate efficiently [135,136]. OCC applications within UAV therefore have potential for significant benefits beyond 5G.

6.2.1. Synchronization The intrinsic principle of OCC operation is based mainly on a broadcasting topology with no feedback information being provided during transmission [16,17]. Synchronization is, therefore, a challenging issue that needs addressing for two main reasons: (i) sampling is carried out randomly in a camera, i.e., it can take place during any symbol of the transmitted signal, thus leading to an increased symbol error rate and (ii) diﬀerent frame sampling periods (i.e., frame saving periods) for cameras. Table4 shows a few examples of the frame rates for several cameras for the same FHD resolution and a frame rate of 30 fps.

Table 4. Camera frame rate on several devices.

Device Average Frame Rate (fps) Minimum (fps) Maximum (fps) Samsung Smartphone (SHV-N916L) 29.656 15.77 31.44 LG Smartphone (F700L) 29.97 22.85 30.13 Sony Smartphone (G8141) 30.00 29.654 30.36 Thinkpad webcam (Sunplus IT) 30.00 20.819 31.94 Microsoft webcam (HD-3000) 29.87 23.09 30.80

Figure 37 illustrates the implication of inconsistent frame sampling interval that is caused by different frame saving periods. The operation of these synchronization protocols depends on the IS characteristics. In addition, the synchronization complexity increases when the inconsistent refresh/switching rate is considered at the TX. Note that in addition to synchronization, color calibration is also mandatory on color modulation used in OCC since both cameras and the TX (LED/digital display) do not share the same calibrated color profile. Several OCC schemes have been proposed including keyframes and anchors, in order to alleviate the synchronization issue by employing fixed references for the color and frame boundaries [22,37,55]. It is worth noting, however, that further studies are required since channel modeling for OCC has not been extensively investigated. Electronics 2020, 9, x FOR PEER REVIEW 35 of 45

6.2.1. Synchronization The intrinsic principle of OCC operation is based mainly on a broadcasting topology with no feedback information being provided during transmission [16,17]. Synchronization is, therefore, a challenging issue that needs addressing for two main reasons: (i) sampling is carried out randomly in a camera, i.e., it can take place during any symbol of the transmitted signal, thus leading to an increased symbol error rate and (ii) different frame sampling periods (i.e., frame saving periods) for cameras. Table 4 shows a few examples of the frame rates for several cameras for the same FHD resolution and a frame rate of 30 fps.

Table 4. Camera frame rate on several devices.

Minimum Maximum Device Average Frame Rate (fps) (fps) (fps) Samsung Smartphone (SHV-N916L) 29.656 15.77 31.44 LG Smartphone (F700L) 29.97 22.85 30.13 Sony Smartphone (G8141) 30.00 29.654 30.36 Thinkpad webcam (Sunplus IT) 30.00 20.819 31.94 Microsoft webcam (HD-3000) 29.87 23.09 30.80

Figure 37 illustrates the implication of inconsistent frame sampling interval that is caused by different frame saving periods. The operation of these synchronization protocols depends on the IS characteristics. In addition, the synchronization complexity increases when the inconsistent refresh/switching rate is considered at the TX. Note that in addition to synchronization, color calibration is also mandatory on color modulation used in OCC since both cameras and the TX (LED/digital display) do not share the same calibrated color profile. Several OCC schemes have been proposed including keyframes and anchors, in order to alleviate the synchronization issue by employing fixed references for the color and frame boundaries [22,37,55]. It is worth noting, however, that further studies are required since channel modeling for OCC has not been extensively Electronics 2020, 9, 1339 35 of 44 investigated.

Figure 37. Effects of frame rate inconsistency. Figure 37. Effects of frame rate inconsistency. On the other hand, the RS OCC encounters interframe synchronization due to a considerable delayOn between the other the hand, consecutive the RS OCC frames encounters captured, inte dependingrframe synchronization on the frame-saving due to period a considerable for each delayframe between [82]. It is anthe inconsistentconsecutive period frames across captured, different depending camera hardware,on the frame-saving due to image period processing for each and framestorage [82]. speed. It is an inconsistent period across different camera hardware, due to image processing and storage speed. 6.2.2. Data Rate Efficiency 6.2.2. Data Rate Efficiency ElectronicsAs mentioned2020, 9, x FOR previously, PEER REVIEW the data rate can also be increased further. There are some noticeable36 of 45 solutions,As mentioned e.g., a high-speed previously, camera the data with rate a frame can ratealso ofbe beyond increased 1000 further. fps and There a reduced are some resolution noticeable for a for a higher frame rate [36,60]. However, the commonly employed solution to increase the camera solutions,higher frame e.g., rate a high-speed [36,60]. However, camera the with commonly a frame employedrate of beyond solution 1000 to fps increase and a the reduced camera resolution sampling sampling rate is to utilize a RS mechanism in CMOS-based sensors, since it requires no hardware rate is to utilize a RS mechanism in CMOS-based sensors, since it requires no hardware alteration. alteration. In [133], an NSM using shape modulation and 2D OFDM was proposed to increase the bit In [133], an NSM using shape modulation and 2D OFDM was proposed to increase the bit density density per frame. per frame. Image processing is part of OCC challenges relative to the nature of camera functionality itself Image processing is part of OCC challenges relative to the nature of camera functionality itself for for capturing pictures/photos. The image processing load increases linearly with the camera capturing pictures/photos. The image processing load increases linearly with the camera resolution, resolution, which is crucial to resolve details in the captured image [29]. In addition, the theoretical which is crucial to resolve details in the captured image [29]. In addition, the theoretical data rate data rate of the OCC cannot be achieved, since not all pixels of the IS are being exposed to of the OCC cannot be achieved, since not all pixels of the IS are being exposed to independent light independent light sources. For example, for OCC with 64-RGB LEDs modulated with OOK at 30 bps sources. For example, for OCC with 64-RGB LEDs modulated with OOK at 30 bps and a camera and a camera with a resolution of 1920 × 1080 pixels and a frame rate of 60 fps at a short link span of with a resolution of 1920 1080 pixels and a frame rate of 60 fps at a short link span of 30 cm, a data 30 cm, a data rate of 5760 ×bit/s can be achieved. However, the theoretical limit of the data rate for this rate of 5760 bit/s can be achieved. However, the theoretical limit of the data rate for this system is system is 1920 × 1080 × 8 × 60 × 3 bits for an 8-bit quantization. When the camera is employed solely 1920 1080 8 60 3 bits for an 8-bit quantization. When the camera is employed solely for OCC, for OCC,× the× efficiency× × of processing images requires further improvement. Several solutions were the efficiency of processing images requires further improvement. Several solutions were proposed proposed either by reducing the resolution of the image to be captured [17,36] or utilizing the only either by reducing the resolution of the image to be captured [17,36] or utilizing the only parts of the parts of the image [38,74]. image [38,74].

6.2.3. Camera Lens Distortion The useuse ofof a a lens lens in in a cameraa camera with with a focal a focal length length not matchingnot matching to the to size the of size the IS,of the introduces IS, introduces optical distortionsoptical distortions in the captured in the captured image [137 image–139 [137–139].]. These distortions These distortion can be barrel,s can be pincushion, barrel, pincushion, and mustache and distortions,mustache distortions, as illustrated as illustrated in Figure 38 in. Figure 38.

Figure 38. Distortions caused by various lenses [140]. Figure 38. Distortions caused by various lenses [140].

Barrel distortion occurs when a significantly wide FOV of the lens is compressed to fit on the image sensor, which is common in a shorter focal length (wide lenses), thus causing curved lines similar to a barrel. Pincushion distortion occurs when an FOV of the lens is much narrower than the size of the image sensor, which is common in a longer focal length (telescopic lenses). Mustache distortion is a combination of both barrel and pincushion distortion due to the lower quality of the lens build. These distortions are present in many cameras to a varying degree and require correction prior to the use in OCC [139,141]. Unfortunately, this distortion issue has not been investigated in existing OCC studies. It was largely circumvented via image cropping or idealized TX images positioned in the sensor area with no lens induced distortions [5,11,17]. Clearly, for the deployment of practical OCC schemes, it is required to address these distortions in the future. A multiple-camera setup can be used to mitigate the image distortion as it provides multiple perspectives of the images [138,139,142].

6.2.4. Flicker-Free Modulation Formats Some modulation schemes, such as a low-rate NRZ OOK can result in flickering. For the flicker- free modulation, a number of schemes have been reported in the literature. RS-based OCC modulation schemes such as UPSOOK may be seen as dark and bright ribbons in large footprints. To address this problem, differential signaling should be adopted. That is, following frame subtraction, only dark or bright regions will remain under the assumption that the TX and the camera are synchronized in the frequency domain.

Electronics 2020, 9, 1339 36 of 44

Barrel distortion occurs when a signiﬁcantly wide FOV of the lens is compressed to ﬁt on the image sensor, which is common in a shorter focal length (wide lenses), thus causing curved lines similar to a barrel. Pincushion distortion occurs when an FOV of the lens is much narrower than the size of the image sensor, which is common in a longer focal length (telescopic lenses). Mustache distortion is a combination of both barrel and pincushion distortion due to the lower quality of the lens build. These distortions are present in many cameras to a varying degree and require correction prior to the use in OCC [139,141]. Unfortunately, this distortion issue has not been investigated in existing OCC studies. It was largely circumvented via image cropping or idealized TX images positioned in the sensor area with no lens induced distortions [5,11,17]. Clearly, for the deployment of practical OCC schemes, it is required to address these distortions in the future. A multiple-camera setup can be used to mitigate the image distortion as it provides multiple perspectives of the images [138,139,142].

6.2.4. Flicker-Free Modulation Formats Some modulation schemes, such as a low-rate NRZ OOK can result in flickering. For the flicker-free modulation, a number of schemes have been reported in the literature. RS-based OCC modulation schemes such as UPSOOK may be seen as dark and bright ribbons in large footprints. To address this problem, differential signaling should be adopted. That is, following frame subtraction, only dark or bright regions will remain under the assumption that the TX and the camera are synchronized in the frequency domain.

6.2.5. Higher Order Modulation Formats Since vehicles move at a relatively high-speed in V2V communications, it is necessary that the transmitted packets are short and captured instantly at the RX, prior to the scene being changed signiﬁcantly. Therefore, given that the data rate is ﬁxed, high-order and multi-carrier modulation schemes can be employed to keep the packet size as short as possible, thus ensuring the received packets with a low probability of error.

7. Summary Optical camera communications have the potential to provide a complementary solution to OWC by employing a highly versatile camera-based RX. OCC gains significant practical benefit over VLC due to the availability of camera in daily used smartphones without further hardware modifications on the RX side. This paper has presented a comprehensive survey on OCC, covering its current status as well as potential developments and challenges facing the OCC system. The state-of-the-art technologies for advanced cameras and image processing algorithms have been introduced rapidly for the past decade, which will surely expedite the full-fledged commercialization of high-rate OCCs. Even with the current relatively low-rate OCCs, a range of attractive applications in the era of IoT are envisioned as a practical means of data communications in certain indoor and outdoor environments. Moreover, therefore, the potential of reliable, practical and cost-effective OCC schemes is yet to be unlocked in future research. This paper has provided a discourse on the current OCC status and future research directions in a timely manner, together with recent IEEE standardization activities.

Author Contributions: Conceptualization, W.A.C. and Y.H.C.; methodology, Y.H.C.; software, W.A.C.; validation, W.A.C., Y.H.C. and Z.G.; formal analysis, Y.H.C. and Z.G.; investigation, W.A.C. and N.B.H.; resources, W.A.C. and N.B.H.; data curation, W.A.C. and N.B.H.; writing—original draft preparation, W.A.C.; writing—review and editing, W.A.C., Y.H.C., and Z.G.; visualization, W.A.C.; supervision, Y.H.C. and Z.G.; project administration, Y.H.C.; funding acquisition, Y.H.C and W.A.C. All authors have read and agreed to the published version of the manuscript. Funding: This research was funded by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2018R1D1A3B07049858) and The APC was funded by Telkom University. Acknowledgments: This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2018R1D1A3B07049858). Electronics 2020, 9, 1339 37 of 44

Conﬂicts of Interest: The authors declare no conﬂict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Appendix A This appendix is intended to list all abbreviations and acronyms used throughout the paper.

Table A1. List of abbreviations and acronyms.

Acronym Full Title 2D Two-dimension 5G Fifth generation wireless networks A-QL Asynchronous quick link ACO-OFDM Asymmetrically clipped optical OFDM ADC Analog-to-digital Converter Adv Advanced AGC Automatic gain Control AI Artificial intelligence AR Augmented reality bps Bits per second C-OOK Camera OOK CFA Color filter array CM-FSK Camera multiple frequency shift keying CMOS Complementary metal-oxide semiconductor CoC Circle of confusion Comm Communication CSK Color shift keying DC-biased Direct current-biased DCO-OFDM Direct current-biased optical OFDM Dev Development DMT Discrete multitone eU-OFDM Enhanced unipolar OFDM ER Extinction ratio fps Frames per second FR Frame rate FHD Full high definition FOV Field of view FSO Free space optics Gbps Gigabits per second GPS Global positioning system HA-QL Hidden asynchronous quick link HDM High density modulation HS-PSK Hybrid spatial phase shift keying IAC Illumination augmented OCC IOCC Infrared-based OCC IoT Internet-of-things IPS Indoor positioning system IPU image processing unit IR Infrared IRC Infrared communications IS Image sensor ISC Image sensor Communications Kbps Kilobits per second LCD Liquid crystal display LED Light-emitting diode LED-ID LED identification LOS Line of sight Electronics 2020, 9, 1339 38 of 44

Table A1. Cont.

Acronym Full Title Mbps Megabits per second MIMO Multiple-input-multiple-output MSE Mean squared error MR Mixed reality NIR Near infrared NLOS Non line of sight NN Neural network NSM Nyquist sampling modulation OCC Optical camera communications OFDM Orthogonal frequency division multiplexing OLED Organic light-emitting diode OOK On-off Keying OWC Optical wireless communications PD Photodiode/photodetector PHY Physical PPM Pulse position modulation PWM Pulse width modulation RGB Red, green, and blue RF Radio frequency RFID RF identification RoI Region of Interest RoI-S RoI signaling RPO-OFDM Reverse polarity optical OFDM RS Rolling shutter RS-OCC Rolling-shutter-based OCC RS-FSK Rolling shutter frequency shift keying RX Receiver S2-PSK Spatial 2-phase shift keying SNR Signal-to-noise ratio sps Samples per second SVLC Smartphone VLC TX Transmitter UAV Unmanned Aerial Vehicle UFSOOK Undersampled frequency shift OOK UPSOOK Undersampled phase shift OOK UV Ultraviolet UVC UV communications UWB Ultra-wide band V2I Vehicle-to-infrastructure communications V2V Vehicle-to-vehicle communications VLC Visible Light communications VPPM Variable pulse position modulation VTASC Variable transparent amplitude shape code Wi-Fi Wireless fidelity WRO Wide receiver orientation

References

1. Paathak, P.H.; Feng, X.; Hu, P.; Mohapatra, P. Visible Light Communication, Networking, and Sensing: A Survey, Potential and Challenges. IEEE Commun. Surv. Tutor. 2015, 17, 2047–2077. [CrossRef] 2. Ghassemlooy, Z.; Popoola, W.; Rajbhandari, S. Optical Wireless Communications; CRC Press: Boca Raton, FL, USA, 2013. 3. Koonen, T. Indoor Optical Wireless Systems: Technology, Trends, and Applications. J. Lightwave Technol. 2018, 36, 1459–1467. [CrossRef] 4. O’Brien, D. Optical wireless communications: Current status and future prospects. In Proceedings of the 2016 IEEE Photonics Society Summer Topical Meeting Series, Newport Beach, CA, USA, 11–13 July 2016. 5. Nguyen, T.; Islam, A.; Hossan, M.T.; Jang, Y.M. Current Status and Performance Analysis of Optical Camera Communication Technologies for 5G Networks. IEEE Access 2017, 5, 4574–4594. [CrossRef] Electronics 2020, 9, 1339 39 of 44

6. Arya, S.; Chung, Y.H. Non-line-of-sight Ultraviolet Communication with Receiver Diversity in Atmospheric Turbulence. IEEE Photonics Technol. Lett. 2018, 30, 895–898. [CrossRef] 7. Ayyash, M.; Elgala, H.; Khreishah, A.; Little, T.; Shao, S.; Rahaim, M.; Schulz, D.; Hilt, J.; Freund, R. Coexistence of WiFi and LiFi toward 5G: Concepts, Opportunities, and Challenges. IEEE Commun. Mag. 2016, 54, 64–71. [CrossRef] 8. Elgala, H.; Mesleh, R.; Haas, H. Indoor Optical Wireless Communication: Potential and State-of-the-art. IEEE Commun. Mag. 2011, 49, 56–62. [CrossRef] 9. Ghassemlooy, Z.; Alves, L.N.; Zvanovec, S.; Khalighi, M.-A. Visible Light Communications: Theory and Applications; CRC Press: Boca Raton, FL, USA, 2017. 10. Ghassemlooy, Z.; Popoola, W.O.; Tang, X.; Zvanovec, S.; Rajbhandar, S. Visible Light Communications—A Review. IEEE COMSOC MMTC Commun. Front. 2017, 12, 6–13. 11. Saha, N.; Ifthekhar, M.; Shareef, N.; Le, T.; Jang, Y.M. Survey on optical camera communications: Challenges and opportunities. IET Optoelectron. 2015, 9, 172–183. [CrossRef] 12. Le, N.T. Invisible watermarking optical camera communication and compatibility issues of IEEE 802.15.7r1 specification. Opt. Commun. 2017, 390, 144–155. [CrossRef] 13. Haruta, T.; Nakajima, T.; Hashizume, J.; Umebayashi, T.; Takahashi, H.; Taniguchi, K.; Kuroda, M.; Sumihiro, H.; Enoki, K.; Yamasaki, T.; et al. 4.6 A 1/2.3inch 20Mpixel 3-layer stacked CMOS. In Proceedings of the IEEE International Solid-State Circuits Conference, San Fransisco, CA, USA, 5–9 February 2017. 14. Nguyen, T.; Islam, A.; Yamazato, T.; Jang, Y.M. Technical Issues on IEEE 802.15.7m Image Sensor Communication Standardization. IEEE Commun. Mag. 2018, 56, 213–218. [CrossRef] 15. IEEE. IEEE 802.15.7-2011—IEEE Standard for Local and Metropolitan Area Networks—Part 15.7: Short-Range Wireless Optical Communication Using Visible Light. IEEE Standards Association. 6 September 2011. Available online: https://standards.ieee.org/standard/802_15_7-2011.html (accessed on 23 October 2018). 16. IEEE. IEEE 802.15 WPANTM 15.7 Maintenance: Short-Range Optical Wireless Communications Task Group (TG 7m). 23 October 2018. Available online: http://www.ieee802.org/15/pub/IEEE%20802_15%20WPAN% 2015_7%20Revision1%20Task%20Group.htm (accessed on 23 October 2018). 17. Le, N.T.; Hossain, M.A.; Jang, Y.M. A Survey of Design and Implementation for Optical Camera Communication. Signal Process. Image Commun. 2017, 53, 95–109. [CrossRef] 18. Saeed, N.; Guo, S.; Park, K.-H.; Al-Naffouri, T.Y.; Alouini, M.-S. Optical camera communications: Survey, use cases, challenges, and future trends. Phys. Commun. 2019, 37, 100900. [CrossRef] 19. Rappaport, T.S.; Xing, Y.; Kanhere, O.; Ju, S.; Madanayake, A.; Mandal, S.; Alkhateeb, A.; Trichopoulos, G.C. Wireless Communications and Applications Above 100 GHz: Opportunities and Challenges for 6G and Beyond. IEEE Access 2019, 7, 78729–78757. [CrossRef] 20. Wikipedia. IEEE 802.15. Wikimedia Foundation, 19 September 2018. Available online: https://en.wikipedia. org/wiki/IEEE_802.15 (accessed on 20 October 2018). 21. IEEE. IEEE 802.15 WPAN™ 15.7 Amendment—Optical Camera Communications Study Group (SG 7a). 23 October 2018. Available online: http://www.ieee802.org/15/pub/SG7a.html (accessed on 23 October 2018). 22. Cahyadi, W.A.; Kim, Y.H.; Chung, Y.H.; Ahn, C.-J. Mobile Phone Camera-Based Indoor Visible Light Communications with Rotation Compensation. IEEE Photonics J. 2016, 8, 1–8. [CrossRef] 23. Chow, C.W.; Chen, C.Y.; Chen, S.H. Visible light communication using mobile-phone camera with data rate higher than frame rate. Opt. Express 2015, 23, 26080–26085. [CrossRef][PubMed] 24. Danakis, C.; Afgani, M.; Povey, G.; Underwood, I.; Haas, H. Using a CMOS camera sensor for visible light communication. In Proceedings of the 2012 IEEE Globecom Workshops, Anaheim, CA, USA, 3–7 December 2012. 25. Cha, J. IEEE 802 Optical Wireless Communication (IEEE 802 OWC Technology Trends and Use Cases). In Proceedings of the IEEE Blockchain Summit Korea, Seoul, Korea, 7–8 June 2018. 26. Serafimovski, N.; Jungnickel, V.; Jang, Y.; Qiang, J. An Overview on High Speed Optical Wireless/Light Communication. 3 July 2017. Available online: https://mentor.ieee.org/802.11/dcn/17/11-17-0962-01-00lc-an- overview-on-high-speed-optical-wireless-light-communications.pdf (accessed on 14 March 2019). 27. Jang, Y.M. IEEE 802.15 Documents: 15.7m Closing Report. 14 November 2018. Available online: https://mentor.ieee.org/802.15/dcn/18/15-18-0596-00-007a-15-7m-closing-report-nov-2018.pptx (accessed on 8 February 2019). Electronics 2020, 9, 1339 40 of 44

28. IEEE. IEEE 802.15.7-2018—IEEE Draft Standard for Local and Metropolitan Area Networks—Part 15.7: Short-Range Optical Wireless Communications. 5 December 2018. Available online: https://standards.ieee. org/standard/802_15_7-2018.html (accessed on 12 February 2019). 29. Gamal, A.E.; Eltoukhy, H. CMOS Image Sensors. IEEE Circuits Devices Mag. 2005, 21, 6–20. [CrossRef] 30. Merritt, R. TSMC Goes Photon to Cloud; EE Times: San Jose, CA, USA, 2018. 31. Celebi, M.E.; Lecca, M.; Smolka, B. Image Demosaicing. In Color Image and Video Enhancement; Springer International Publishing: Cham, Switzerland, 2015; pp. 13–54. 32. Tanaka, M. Give Me Four: Residual Interpolation for Image Demosaicing and Upsampling; Tokyo Institute of Technology: Tokyo, Japan, 2018; Available online: http://www.ok.sc.e.titech.ac.jp/~{}mtanaka/research.html (accessed on 8 November 2018). 33. Meroli, S. Stefano Meroli Lecture: CMOS vs CCD Pixel Sensor. 17 August 2005. Available online: Meroli.web.cern.ch/lecture_cmos_vs_ccd_pixel_sensor.html (accessed on 22 October 2018). 34. Chen, Z.; Wang, X.; Pacheco, S.; Liang, R. Impact of CCD camera SNR on polarimetric accuracy. Appl. Opt. 2014, 53, 7649–7656. [CrossRef] 35. Goto, Y.; Takai, I.; Yamazato, T.; Okada, H.; Fujii, T. A New Automotive VLC System Using Optical Communication Image Sensor. IEEE Photonics J. 2016, 8, 1–17. [CrossRef] 36. Teli, S.; Chung, Y.H. Selective Capture based High-speed Optical Vehicular Signaling System. Signal Process. Image Commun. 2018, 68, 241–248. [CrossRef] 37. Cahyadi, W.A.; Chung, Y.H. Smartphone Camera based Device-to-device Communication Using Neural Network Assisted High-density Modulation. Opt. Eng. 2018, 57, 096102. [CrossRef] 38. Boubezari, R.; Minh, H.; Ghassemlooy, Z.; Bouridane, A. Smartphone Camera Based Visible Light Communication. J. Lightwave Technol. 2016, 34, 4121–4127. [CrossRef] 39. QImaging. Rolling Shutter vs. Global Shutter: CCD and CMOS Sensor Architecture and Readout Modes. 2014. Available online: https://www.selectscience.net/application-notes/Rolling-Shutter-vs-Global-Shutter- CCD-and-CMOS-Sensor-Architecture-and-Readout-Modes/?artID=33351 (accessed on 21 March 2018). 40. Do, T.-H.; Yoo, M. Performance Analysis of Visible Light Communication Using CMOS Sensors. Sensors 2016, 16, 309. [CrossRef][PubMed] 41. Shlomi Arnon, B.G. Visible Light Communication; Cambridge University Press: Cambridge, UK, 2015. 42. National Instruments, Calculating Camera Sensor Resolution and Lens Focal Length. 15 February 2018. Available online: http://www.ni.com/product-documentation/54616/en/ (accessed on 6 November 2018). 43. Cahyadi, W.A.; Kim, Y.H.; Chung, Y.H. Dual Camera based Split Shutter for High-rate and Long-distance Optical Camera Communications. Opt. Eng. 2016, 55, 110504. [CrossRef] 44. Hassan, N.; Ghassemlooy, Z.; Zvanovec, S.; Luo, P.; Le-Minh, H. Non-line-of-sight 2 N indoor optical × camera communications. Appl. Opt. 2018, 57, 144–149. [CrossRef][PubMed] 45. Chau, J.C.; Little, T.D. Analysis of CMOS active pixel sensors as linear shift-invariant receivers. In Proceedings of the 2015 IEEE International Conference on Communication Workshop, London, UK, 8–12 June 2015. 46. Hassan, N.B. Impact of Camera Lens Aperture and the Light Source Size on Optical Camera Communications. In Proceedings of the 11th International Symposium on Communication Systems, Networks & Digital Signal Processing (CSNDSP), Budapest, Hungary, 12–18 July 2018. 47. Hossain, M.A.; Hong, C.H.; Ifthekhar, M.S.; Jang, Y.M. Mathematical modeling for calibrating illuminated image in image sensor communication. In Proceedings of the 2015 International Conference on Information and Communication Technology Convergence (ICTC), Jeju, Korea, 28–30 October 2015. 48. Huynh-Thu, Q.; Ghanbari, M. Scope of validity of PSNR in image/video quality assessment. Electron. Lett. 2008, 44, 800–801. [CrossRef] 49. Iizuka, N. CASIO Response to 15.7r1 CFA. March 2015. Available online: https://mentor.ieee.org/802.15/dcn/ 15/15-15-0173-01-007a-casioresponse-to-15-7r1-cfa.pdf (accessed on 24 January 2019). 50. Perli, S.D.; Ahmed, N.; Katabi, D. PixNet: Interference-free wireless links using LCD-camera pairs. In Proceedings of the MobiCom ACM, New York, NY, USA, 20–24 September 2010. 51. Hao, T.; Zhou, R.; Xing, G. COBRA: Color barcode streaming for smartphone system. In Proceedings of the MobiSys ACM, Ambleside, UK, 25–29 June 2012. 52. Hu, W.; Gu, H.; Pu, Q. LightSync: Unsynchronized visual communication over screen-camera links. In Proceedings of the 19th Annual International Conference on Mobile Computing & Networking, New York, NY, USA, 30 September–4 October 2013. Electronics 2020, 9, 1339 41 of 44

53. Nguyen, T.; Jang, Y.M. Novel 2D-sequential color code system employing Image Sensor Communications for Optical Wireless Communications. ICT Express 2016, 2, 57–62. [CrossRef] 54. Hossain, M.A.; Lee, Y.T.; Lee, H.; Lyu, W.; Hong, C.H.; Nguyen, T.; Le, N.T.; Jang, Y.M. A Symbiotic Digital Signage system based on display to display communication. In Proceedings of the 2015 Seventh International Conference on Ubiquitous and Future Networks, Sapporo, Japan, 7–10 July 2015. 55. Huang, W.; Tian, P.; Xu, Z. Design and implementation of a real-time CIMMIMO optical camera communication system. Opt. Express 2016, 24, 24567–24579. [CrossRef] 56. Chen, S.H.; Chow, C. Color-filter-free WDM MIMO RGB-LED visible light communication system using mobile-phone camera. In Proceedings of the 2014 13th International Conference on Optical Communications and Networks (ICOCN), Suzhou, China, 9–10 November 2014. 57. Chen, S.; Chow, C.W. Color-Shift Keying and Code-Division Multiple-Access Transmission for RGB-LED Visible Light Communications Using Mobile Phone Camera. IEEE Photonics J. 2014, 6, 1–6. [CrossRef] 58. Basu, J.K.; Bhattacharyya, D.; Kim, T. Use of Artifical Neural Network in Pattern Recognition. Int. J. Softw. Eng. Appl. 2010, 4, 23–34. 59. Yu, P.;Anastassopoulos, V.;Venetsanopoulos,A.N. Pattern classification and recognition based on morphology and neural networks. Can. J. Electr. Comput. Eng. 1992, 17, 58–64. [CrossRef] 60. Yamazato, T.; Takai, I.; Okada, H.; Fujii, T.; Yendo, T.; Arai, S. Image-Sensor-Based Visible Light Communication for Automotive Applications. IEEE Commun. Mag. 2014, 52, 88–97. [CrossRef] 61. Photron. Photron Fastcam SA-Z. 2018. Available online: https://photron.com/fastcam-sa-z/ (accessed on 8 November 2018). 62. Davis, J. Humans perceive flicker artifacts at 500 Hz. Sci. Rep. 2015, 5, 1–4. [CrossRef][PubMed] 63. Ebihara, K.; Kamakura, K.; Yamazato, T. Layered transmission of space-time coded signals for image-sensor-based visible light communications. J. Lightwave Technol. 2015, 33, 4193–4206. [CrossRef] 64. Nguyen, T.; Islam, A.; Jang, Y.M. Region-of-interest signaling vehicular system using optical camera communications. IEEE Photonics J. 2017, 9, 1–20. [CrossRef] 65. Yuan, W.; Dana, K.; Varga, M.; Ashok, A.; Gruteser, M.; Mandayam, N. Computer vision methods for visual MIMO optical system. In Proceedings of the 2011 Computer Vision and Pattern Recognition Workshops (CVPRW), Colorado Springs, CO, USA, 20–25 June 2011. 66. Roberts, R.D. A MIMO protocol for camera communications (CamCom) using undersampled frequency shift ON-OFF keying (UFSOOK). In Proceedings of the IEEE Globecom Workshops, Atlanta, GA, USA, 9–13 December 2013. 67. Roberts, R.D. Intel Proposal in IEEE 802.15.7r1. 28 March 2017. Available online: https://mentor.ieee.org/802. 15/dcn/16/15-16-0006-01-007a-intel-occ-proposal.pdf (accessed on 28 January 2019). 68. Luo, P.; Zhang, M.; Ghassemlooy, Z.; Zvanovec, S.; Feng, S.; Zhang, P. Undersampled-Based Modulation Schemes for Optical Camera Communications. IEEE Commun. Mag. 2018, 56, 204–212. [CrossRef] 69. Luo, P.; Zhang, M.; Ghassemlooy, Z. Experimental Demonstration of RGB LED-Based Optical Camera Communications. IEEE Photonics J. 2015, 7, 1–12. [CrossRef] 70. Jang, Y.M.; Nguyen, T. Kookmin University PHY Sub-Proposal for ISC Using Dimmable Spatial M-PSK (DSM-PSK). January 2016. Available online: https://mentor.ieee.org/802.15/dcn/16/15-16-0015-02-007a- kookmin-university-phy-sub-proposal-for-isc-using-dimmable-spatial-m-psk-dsm-psk.pptx (accessed on 29 January 2019). 71. Nguyen, T.; Hossain, M.A.; Jang, Y.M. Design and Implementation of a Novel Compatible Encoding Scheme in the Time Domain for Image Sensor Communication. Sensors 2016, 16, 736. [CrossRef] 72. Takai, I.; Ito, S.; Yasutomi, K.; Kagawa, K.; Andoh, M.; Kawahito, S. LED and CMOS image sensor based optical wireless communication system for automotive applications. IEEE Photonics J. 2013, 5, 6801418. [CrossRef] 73. Itoh, S.; Takai, I.; Sarker, M.Z.; Hamai, M.; Yasutomi, K.; Andoh, M.; Kawahito, S. BA CMOS image sensor for 10 Mb/s 70 m-range LED-based spatial optical communication. In Proceedings of the IEEE International Solid-State Circuits Conference, San Fransisco, CA, USA, 7–11 February 2010. 74. Liu, Y. Decoding mobile-phone image sensor rolling shutter effect for visible light communications. Opt. Eng. 2016, 55, 016103. [CrossRef] 75. Chow, C.; Chen, C.; Chen, S. Enhancement of signal performance in LED visible light communications using mobile phone camera. IEEE Photonics J. 2015, 7, 7903607. [CrossRef] Electronics 2020, 9, 1339 42 of 44

76. Wang, W.C.; Chow, C.W.; Wei, L.; Liu, Y.; Yeh, C.H. Long distance non-line-of-sight (NLOS) visible light signal detection based on rolling-shutter-patterning of mobile-phone camera. Opt. Express 2017, 25, 10103–10108. [CrossRef][PubMed] 77. Chen, C.; Chow, C.; Liu, Y.; Yeh, C. Efficient demodulation scheme for rollingshutter-patterning of CMOS image sensor based visible light communications. Opt. Express 2017, 25, 24362–24367. [CrossRef][PubMed] 78. Liang, K.; Chow, C.; Liu, Y. Mobile-phone based visible light communication using region-grow light source tracking for unstable light source. Opt. Express 2016, 24, 17505–17510. [CrossRef][PubMed] 79. Liu, Y.; Chow, C.; Liang, K.; Chen, H.; Shu, C.; Chen, C.; Chen, S. Comparison of thresholding schemes for visible light communication using mobile-phone image sensor. Opt. Express 2016, 24, 1973–1978. [CrossRef] [PubMed] 80. Gonzales, R.; Woods, R. Digital Image Processing; Pearson: Tennessee, TN, USA, 2002. 81. Wellner, P. Adaptive thresholding for the digital desk. Xerox 1993, EPC-1993-110, 1–19. 82. Wang, W.; Chow, C.; Chen, C.; Hsieh, H.; Chen, Y. Beacon Jointed Packet Reconstruction Scheme for Mobile-Phone Based Visible Light Communications Using Rolling Shutter. IEEE Photonics J. 2017, 9, 7907606. [CrossRef] 83. Chow, C.; Shiu, R.; Liu, Y.; Yeh, C. Non-flickering 100 m RGB visible light communication transmission based on a CMOS image sensor. Opt. Express 2018, 26, 7079–7084. [CrossRef] 84. Yang, Y.; Hao, J.; Luo, J. CeilingTalk: Lightweight indoor broadcast through LED-camera communication. IEEE Trans. Mob. Comput. 2017, 16, 3308–3319. [CrossRef] 85. Nguyen, T.; Le, N.; Jang, Y. Asynchronous Scheme for Optical Camera Communication-Based Infrastructure-to-Vehicle Communication. Int. J. Distrib. Sens. Netw. 2015, 11, 908139. [CrossRef] 86. Cahyadi, W.A.; Chung, Y.H. Wide Receiver Orientation Using Diffuse Reflection in Camera-based Indoor Visible Light Communication. Opt. Commun. 2018, 431, 19–28. [CrossRef] 87. Do, T.-H.; Yoo, M. A Multi-Feature LED Bit Detection Algorithm in Vehicular Optical Camera Communication. IEEE Access 2019, 7, 95797. [CrossRef] 88. Chow, C.-W.; Li, Z.-Q.; Chuang, Y.-C.; Liao, X.-L.; Lin, K.-H.; Chen, Y.-Y. Decoding CMOS Rolling Shutter Pattern in Translational or Rotational Motions for VLC. IEEE Photonics J. 2019, 11, 7902305. [CrossRef] 89. Shi, J.; He, J.; Jiang, Z.; Zhou, Y.; Xiao, Y. Enabling user mobility for optical camera communication using mobile phone. Opt. Express 2018, 26, 21762–21767. [CrossRef][PubMed] 90. Han, B.; Hranilovic, S. A Fixed-Scale Pixelated MIMO Visible Light Communication. IEEE J. Sel. Areas Commun. 2018, 36, 203–211. [CrossRef] 91. Cahyadi, W.A.; Chung, Y.H. Experimental Demonstration of Indoor Uplink Near-infrared LED Camera Communication. Opt. Express 2018, 26, 19657–19664. [CrossRef] 92. Huynh, T.-H.; Pham, T.-A.; Yoo, M. Detection Algorithm for Overlapping LEDs in Vehicular Visible Light Communication System. IEEE Access 2019, 7, 109945. [CrossRef] 93. Chow, C.; Shiu, R.; Liu, Y.; Liao, X.; Lin, K. Using advertisement light-panel and CMOS image sensor with frequency-shift-keying for visible light communication. Opt. Express 2018, 26, 12530–12535. [CrossRef] 94. Kuraki, K.; Kato, K.; Tanaka, R. ID-Embedded LED Lighting Technology for Providing Object-Related Information. January 2016. Available online: http://www.fujitsu.com/global/documents/about/resources/ publications/fstj/archives/vol52-1/paper12.pdf (accessed on 6 February 2019). 95. Ifthekhar, M.; Saha, N.; Jang, Y.M. Neural network based indoor positioning technique in optical camera communication system. In Proceedings of the 2014 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Busan, Korea, 27–30 October 2014. 96. Teli, S.; Cahyadi, W.; Chung, Y. Optical Camera Communication: Motion Over Camera. IEEE Commun. Mag. 2017, 55, 156–162. [CrossRef] 97. Teli, S.; Cahyadi, W.A.; Chung, Y.H. Trained neurons-based motion detection in optical camera communications. Opt. Eng. 2018, 57, 040501. [CrossRef] 98. Liu, Y.; Chen, H.; Liang, K.; Hsu, C.; Chow, C.; Yeh, C. Visible Light Communication Using Receivers of Camera Image Sensor and Solar Cell. IEEE Photonics J. 2016, 8, 1–7. [CrossRef] 99. Rida, M.E.; Liu, F.; Jadi, Y. Indoor Location Position Based on Bluetooth Signal Strength. In Proceedings of the International Conference on Information Science and Control Engineering IEEE, Shanghai, China, 24–26 April 2015. 100. Yang, C.; Shao, H. WiFi-based indoor positioning. IEEE Commun. Mag. 2015, 53, 150–157. [CrossRef] Electronics 2020, 9, 1339 43 of 44

101. Huang, C. Real-Time RFID Indoor Positioning System Based on Kalman-Filter Drift Removal and Heron-Bilateration Location Estimation. IEEE Trans. Instrum. Meas. 2015, 64, 728–739. [CrossRef] 102. Li, Y.; Ghassemlooy, Z.; Tang, X.; Lin, B.; Zhang, Y. A VLC Smartphone Camera Based Indoor Positioning System. IEEE Photonics Technol. Lett. 2018, 30, 1171. [CrossRef] 103. Zhang, R.; Zhong, W.; Qian, K.; Wu, D. Image Sensor Based Visible Light Positioning System with Improved Positioning Algorithm. IEEE Access 2017, 5, 6087–6094. [CrossRef] 104. Gomez, A.; Shi, K.; Quintana, C.; Faulkner, G.; Thomsen, B.C.; O’Brien, D. A 50 Gb/s transparent indoor optical wireless communications link with an integrated localization and tracking system. J. Lightwave Technol. 2016, 34, 2517. [CrossRef] 105. Kuo, Y.; Pannuto, P.; Hsiao, K.; Dutta, P. Luxapose: Indoor Positioning with Mobile Phones and Visible Light. In Proceedings of the 20th Annual International Conference on Mobile Computing and Networking (MobiCom), Maui, HI, USA, 7–11 September 2014. 106. Wang, A.; Wang, L.; Chi, X.; Liu, S.; Shi, W.; Deng, J. The research of indoor positioning based on visible light communication. China Commun. 2015, 12, 85–92. [CrossRef] 107. Brena, R.F.; García-Vázquez, J.P.; Galván-Tejada, C.E.; Muñoz-Rodriguez, D.; Vargas-Rosales, C.; Fangmeyer, J.J. Evolution of indoor positioning technologies: A survey. J. Sens. 2017, 2017.[CrossRef] 108. Kim, H. Three-Dimensional Visible Light Indoor Localization Using AOA and RSS With Multiple Optical Receivers. Lightwave Technol. J. 2014, 32, 2480–2485. 109. Hassan, N.; Naeem, A.; Pasha, M.; Jadoon, T. Indoor Positioning Using Visible LED Lights: A Survey. ACM Comput. Surv. 2015, 48, 1–32. [CrossRef] 110. Renesas. Eye Safety for Proximity Sensing Using Infrared Light-Emitting Diodes. 28 April 2016. Available online: https://www.renesas.com/doc/application-note/an1737.pdf (accessed on 29 October 2018). 111. Khan, F.N. Joint OSNR monitoring and modulation format identiﬁcation in digital coherent receivers using deep neural networks. Opt. Express 2017, 25, 17767. [CrossRef] 112. Philips Lighting (Signify), Enabling the New World of Retail: A White Paper on Indoor Positioning. 2017. Available online: http://www.lighting.philips.com/main/inspiration/smart-retail/latest-thinking/indoor- positioning-retail (accessed on 10 February 2019). 113. Carroll, A.; Heiser, G. An analysis of power consumption in a smartphone. In Proceedings of the 2010 USENIX Conference on USENIX Annual Technical Conference, Boston, MA, USA, 22–25 June 2010. 114. Bekaroo, G.; Santokhee, A. Power consumption of the Raspberry Pi: A comparative analysis. In Proceedings of the 2016 IEEE International Conference on Emerging Technologies and Innovative Business Practices for the Transformation of Societies (EmergiTech), Balaclava, Mauritius, 3–6 August 2016. 115. Akram, M.; Aravinda, L.; Munaweera, M.; Godaliyadda, G.; Ekanayake, M. Camera based visible light communication system for underwater applications. In Proceedings of the IEEE International Conference on Industrial and Information Systems (ICIIS), Peradeniya, Sri Lanka, 15–16 December 2017. 116. Takai, I.; Andoh, M.; Yasutomi, K.; Kagawa, K.; Kawahito, S. Optical Vehicle-to-Vehicle Communication System Using LED Transmitter and Camera Receiver. IEEE Photonics J. 2014, 6, 1–14. [CrossRef] 117. Ifthekhar, M.; Saha, N.; Jang, Y. Stereo-vision-based cooperative vehicle positioning using OCC and neural networks. Opt. Commun. 2015, 352, 166–180. [CrossRef] 118. Yamazato, T.; Kinoshita, M.; Arai, S.; Souke, E.; Yendo, T.; Fujii, T.; Kamakura, K.; Okada, H. Vehicle Motion and Pixel Illumination Modeling for Image Sensor Based Visible Light Communication. IEEE J. Sel. Areas Commun. 2015, 33, 1793–1805. [CrossRef] 119. Kawai, Y.; Yamazato, T.; Okada, H.; Fujii, T.; Yendo, T.; Arai, S. Tracking of LED headlights considering NLOS for an image sensor based V2I-VLC. In Proceedings of the International Conference and Exhibition on Visible Light Communications, Yokohama, Japan, 25–26 October 2015. 120. Akram, M.; Godaliyadda, R.; Ekanayake, P. Design and analysis of an optical camera communication system for underwater applications. IET Optoelectron. 2019, 14, 10–21. [CrossRef] 121. Wikichip. Wikichip Semiconductor & Computer Engineering. Wikimedia Foundation. 12 January 2019. Available online: https://en.wikichip.org/wiki/hisilicon/kirin/970 (accessed on 21 January 2019). 122. Kleinfelder, S.; Chen, Y.; Kwiatkowski, K.; Shah, A. High-Speed CMOS Image Sensor Circuits with In Situ Frame Storage. IEEE Trans. Nucl. Sci. 2004, 51, 1648–1656. [CrossRef] 123. Millet, L. A 5500-frames/s 85-GOPS/W 3-D Stacked BSI Vision Chip Based on Parallel In-Focal-Plane Acquisition and Processing. IEEE J. Solid State Circuits 2019, 54, 1096–1105. [CrossRef] Electronics 2020, 9, 1339 44 of 44

124. Gu, J.; Hitomi, Y.; Mitsunaga, T.; Nayar, S. Coded rolling shutter photography: Flexible space-time sampling. In Proceedings of the IEEE International Conference on Computational Photography (ICCP), Cambridge, MA, USA, 29–30 March 2010. 125. Wilkins, A.; Veitch, J.; Lehman, B. LED lighting flicker and potential health concerns: IEEE standard PAR1789 update. In Proceedings of the 2010 IEEE Energy Conversion Congress and Exposition, Atlanta, GA, USA, 12–16 September 2010. 126. Alphabet Deepmind, AlphaGo. Alphabet (Google). 2018. Available online: https://deepmind.com/research/ alphago/ (accessed on 31 October 2018). 127. iClarified. Google Uses Machine Learning to Improve Photos App. 13 October 2016. Available online: http://www.iclarified.com/57301/google-uses-machine-learning-to-improve-photos-app (accessed on 8 February 2019). 128. Carneiro, T.; Da Nóbrega, R.M.; Nepomuceno, T.; Bian, G.; De Albuquerque, V.; Filho, P. Performance Analysis of Google Colaboratory as a Tool for Accelerating Deep Learning Applications. IEEE Access 2018, 6, 61677–61685. [CrossRef] 129. Banerjee, A. A Microarchitectural Study on Apple’s A11 Bionic Processor; Arkansas State University: Jonesboro, AR, USA, 2018. 130. Sims, G. What is the Kirin 970’s NPU?—Gary Explains. Android Authority. 22 December 2017. Available online: https://www.androidauthority.com/what-is-the-kirin-970s-npu-gary-explains-824423/ (accessed on 21 January 2019). 131. Haigh, P.; Ghassemlooy, Z.; Rajbhandari, S.; Papakonstantinou, I.; Popoola, W. Visible Light Communications: 170 Mb/s Using an Artificial Neural Network Equalizer in a Low Bandwidth White Light Configuration. J. Lightwave Technol. 2014, 32, 1807–1813. [CrossRef] 132. Liu, J.; Kong, X.; Xia, F.; Wang, L.; Qing, Q.; Lee, I. Artificial Intelligence in the 21st Century. IEEE Access 2018, 6, 34403–34421. [CrossRef] 133. Nguyen, T.; Thieu, M.; Jang, Y. 2D-OFDM for Optical Camera Communication: Principle and Implementation. IEEE Access 2019, 7, 29405–29424. [CrossRef] 134. Xu, Z.; Sadler, B.M. Ultraviolet Communications: Potential and State-Of-The-Art. IEEE Commun. Mag. 2008, 46, 67–73. 135. Huo, Y.; Dong, X.; Lu, T.; Xu, W.; Yuen, M. Distributed and Multilayer UAV Networks for Next-Generation Wireless Communication and Power Transfer: A Feasibility Study. IEEE Internet Things J. 2019, 6, 7103–7115. [CrossRef] 136. Huo, Y.; Dong, X.; Xu, W.; Yuen, M. Enabling Multi-Functional 5G and Beyond User Equipment: A Survey and Tutorial. IEEE Access 2019, 7, 116975–117008. [CrossRef] 137. Ahmed, M.; Farag, A. Nonmetric calibration of camera lens distortion: Differential methods and robust estimation. IEEE Trans. Image Process. 2005, 14, 1215–1230. [CrossRef][PubMed] 138. Tian, H.; Srikanthan, T.; Asari, K.; Lam, S. Study on the effect of object to camera distance on polynomial expansion coefficients in barrel distortion correction. In Proceedings of the Fifth IEEE Southwest Symposium on Image Analysis and Interpretation, Santa Fe, NM, USA, 7–9 April 2002. 139. Tang, Z.; Von Gioi, R.G.; Monasse, P.; Morel, J. A Precision Analysis of Camera Distortion Models. IEEE Trans. Image Process. 2017, 26, 2694–2704. [CrossRef][PubMed] 140. Drew Gray Photography. Distortion 101—Lens vs. Perspective. Drew Gray Photography. 8 November 2014. Available online: http://www.drewgrayphoto.com/learn/distortion101 (accessed on 22 March 2019). 141. Brauers, J.; Aach, T. Geometric Calibration of Lens and Filter Distortions for Multispectral Filter-Wheel Cameras. IEEE Trans. Image Process. 2010, 20, 496–505. [CrossRef] 142. Xu, Y.; Zhou, Q.; Gong, L.; Zhu, M.; Ding, X.; Teng, R.K.F. High-Speed Simultaneous Image Distortion Correction Transformations for a Multicamera Cylindrical Panorama Real-time Video System Using FPGA. IEEE Trans. Circuits Syst. Video Technol. 2014, 24, 1061–1069. [CrossRef]

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).