electronics
Article Smartphone Camera-Based Optical Wireless Communication System: Requirements and Implementation Challenges
Md. Shahjalal , Moh. Khalid Hasan , Mostafa Zaman Chowdhury and Yeong Min Jang *
Department of Electronics Engineering, Kookmin University, Seoul 02707, Korea * Correspondence: [email protected]; Tel.: +82-2-910-5068
Received: 22 July 2019; Accepted: 15 August 2019; Published: 19 August 2019
Abstract: Visible light and infrared bands of the optical spectrum used for optical camera communication (OCC) are becoming a promising technology nowadays. Researchers are proposing new OCC-based architectures and applications in both indoor and outdoor systems using the embedded cameras on smartphones, with a view to making them user-friendly. Smartphones have useful features for developing applications using the complementary metal-oxide-semiconductor cameras, which can receive data from optical transmitters. However, several challenges have arisen in increasing the capacity and communication range, owing to the limitations of current cameras and implementation complexities. In this paper, we provide a comprehensive analysis of the OCC technology requirements and opportunities using smartphone cameras from an implementation point of view. Furthermore, we demonstrate an OCC system using a low frame rate smartphone camera to particularly analyze the requirements and critical implementation challenges. Also, some possible solutions are provided with a view to improving the overall system capacity, communication distance, and stability.
Keywords: visible light communication (VLC); image sensor; rolling shutter camera; open computer vision (CV)
1. Introduction Fifth-generation (5G) technology has not only been characterized with a 10-Gbps data rate but also requires massive user connectivity, internet-of-things (IoT) paradigm, ultra-low power consumptions, and ultra-high security [1]. However, existing RF resources are not enough to support massive user access and IoT paradigm due to limited band and inter channel or inter-symbol interferences. To support the enormous demand for 5G, researchers had to rely on other suitable bands along with the current RF spectrum. The optical spectrum could be a possible solution to the bandwidth problem because it provides a wide range of frequencies (0.3–3000 THz). Optical wireless communication (OWC) technologies, such as visible light communication (VLC), light fidelity, optical camera communication (OCC), and free-space optical communication, can be utilized as ultra-dense small cell networks and high-capacity backhaul support for 5G and beyond. Additionally, OWC can support high data rates [2] and communication distances within a very short to high range [1,3–5]. Although OCC does not support high capacity link connectivity due to the camera limitations, it can be employed in numerous low data rate applications, such as indoor positioning, mobile robot navigation, small identification information, and advertisements. As light-emitting diodes (LEDs) are used as transmitters for OCC technology, they can further be used for illumination purposes. These LED lights are fabricated from semiconductor materials, which have fast switching capabilities [6]. Also, cameras are commonly used devices in recent times and they are incorporated into every
Electronics 2019, 8, 913; doi:10.3390/electronics8080913 www.mdpi.com/journal/electronics Electronics 2019, 8, 913 2 of 17 smartphone. Thus, they are an ideal wireless communication platform, which uses an unlicensed spectrum that provides a very low installation cost and consumes very little energy. In order to support 5G massive connectivity and low power consumption, a large number of users can be connected with the OCC network and access the same information transmitted through light; research for the improvement of this capacity is ongoing. However, due to the existence of smartphones with various operating systems (Android, Windows, or iOS), the same OCC application cannot be used in all types of devices. Among these operating systems, Android has the highest number of users worldwide; thus it has a great prospect for creating applications in this particular area. Moreover, OCC is becoming a promising technology in both indoor and outdoor applications such as localization, smart lighting, patient monitoring, autonomous vehicles, transportation, and augmented reality [1]. However, the implementation of OCC using smartphone cameras is more complicated compared with conventional cameras because current smartphones do not have features that support high data rates and high-speed communication. The receiver camera can be of two types: global shutter and rolling shutter. In global shutter cameras, the entire frame is exposed at one time; therefore, it captures either the ON or OFF state of an LED. In rolling shutter cameras, the image is captured using row-by-row exposure [7]. For a fixed frame rate, the camera exposure time needs to adjust to cope with the LED’s ON-OFF status and capture a black and white striped image. However, the widths of these strips depend on the frequency of modulation. Most phone cameras use rolling shutter mechanisms and complementary metal-oxide-semiconductor (CMOS) image sensors, owing to the greatly reduced power consumption. In the process of developing smartphone-based OCC technology, several challenges can be faced owing to a limitation of camera frame rate and exposure control of current devices and a lack of suitable modulation schemes. In this study, we analyze the key requirements and challenges in the implementation of an Android smartphone-based OWC system. We provide the necessary experimental results with an Android smartphone to explain why it displayed lower throughput and limited communication distance. We also suggest possible mitigation techniques for these particular challenges. Moreover, we focus on the essential requirements, such as an enhanced data rate and communication distance, flickering mitigation, and multilateral communication with smartphones. The main contributions of this paper can be summarized as follows:
We provide the OCC architecture in brief and the requirements for the transmitter and receiver • design for the system. A discussion on the requirements for designing a flicker-free transmitter and necessary modulations and encoding schemes is also included in this paper. Regarding the receiver, the key parameters, limitations, and necessary programming platforms • (OpenCV) for modern smartphones are explained in detail. We demonstrated an Android application and LED transmitter system to visualize the • implementation challenges in real-time communication. The key OCC system parameter challenges are analyzed, with possible solutions for issues • such as multilateral communication, data rate, communication distance, visual flickering, and user mobility. We reviewed the existing deep learning-based real-time object detection techniques-based on • their inference time and simplicity. Also, we analyzed opportunities to support user mobility by integrating TensorFlow Lite framework for learning-based LED detection in the OCC application.
The rest of the paper is organized as follows. In Section2, we provide a brief overview of the OCC system. The transmitter design and requirements are characterized in Section3. In Section4, a review of the receiver camera specifications and required processing platforms is provided. In Section5, the implementation challenges are explained in detail with possible mitigation techniques by applying the real-time OCC system on a smartphone. Section6 describes the OCC system demonstrated in this paper. Finally, we provide a conclusion to the paper in Section7. Electronics 2019, 8, 913 3 of 17
Electronics 2019, 8, x FOR PEER REVIEW 3 of 17 2. OCC Overview A detailed and comprehensive study on the introduction to OCC technology is provided in Ref. [8]. InA detailedthis section, and comprehensive we present studya brief on over the introductionview of the to OCC technologysystem before is provided discussing in Ref. its [8]. implementationIn this section, we in presentdetail. The a brief OCC overview system ofcomp therises OCC LEDs system and before image discussing sensors its(cameras) implementation which are in useddetail. as Thetransmitters OCC system and receivers, comprises respectively. LEDs and image Moreover, sensors LEDs (cameras) are used which in almost are used every as transmitterslight-based application,and receivers, such respectively. as indoor illumination, Moreover, LEDs street are sign usedals, in and almost car tail every lights light-based and headlights application, [9]. They such quicklyas indoor switch illumination, between streetthe ON signals, and OFF and states. car tail By lights employing and headlights this property, [9]. They LED quickly light can switch be modulatedbetween the using ON andhigh OFF frequencies. states. By employingFigure 1 sh thisows property, the basic LED block light diagram can be modulated of an OCC using system. high Initially,frequencies. optical Figure channel1 shows encoding the basic is blockperformed diagram to detect of an OCCand system.correct the Initially, error opticalelement channel while transmittingencoding is performedthrough the to detectchannel and with correct noise the and error interference element while induced transmitting by ambient through light the sources. channel Manchesterwith noise andcoding interference (described induced later) helps by ambient to elim lightinate sources. the light-flickering Manchester effect. coding However, (described visual later) flickeringhelps to eliminate may occur the if light-flickeringthe change in the eff ect.state However, of an LED visual is observed flickering by the may human occur ifeye the (above change 200 in Hz the isstate considered of an LED safe is observed [10]). Further, by the humanencoded eye data (above is utilized 200 Hz isto considered modulate safe the [10LED]). Further,light at encodedhigher frequenciesdata is utilized than toa camera modulate can the detect. LED Current light at highercamera frequenciesimage sensors than usually a camera show can clear detect. detection Current of frequencycamera image responses sensors up usually to 6 kHz. show All clear of detectionthe tasks of are frequency performed responses by an upLED to 6driver kHz. circuit, All of the which tasks consistsare performed of a high-switching by an LED driver metal-oxide-semiconduc circuit, which consiststor of field-effect a high-switching transistor metal-oxide-semiconductor device, Arduino, and a powerfield-e ffsupply.ect transistor Finally, device, LED Arduino,light transmits and a power the mo supply.dulated Finally, bits through LED light the transmits optical thechannel. modulated The receiverbits through extracts the opticalthe information channel. The by receiver capturing extracts and theprocessing information the by frames capturing with and the processing help of thean integratedframes with OpenCV the help library of an integratedas an image-processing OpenCV library platform. as an image-processing In a rolling shutter platform. camera, In the a rolling LED imagesshutter are camera, captured the LED as bundles images areof capturedblack and as wh bundlesite strips. of black Demodulation and white strips.is performed Demodulation on these is imagesperformed by measuring on these images the widths by measuring and thresholding the widths andof strips. thresholding After the of strips.demodulation After the and demodulation decoding tasksand decodingare completed, tasks arethe completed,output data the is shown output on data a display. is shown on a display.
LED driver
Input Channel Manchester Modulation LED data coding coding
Optical channel
Output Frame Decoding Demodulation data processing Camera Figure 1. The block diagram of an optical camera communication (OCC) transmitter and receiver system. Figure 1. The block diagram of an optical camera communication (OCC) transmitter and receiver system.Smartphone-based real-time processing is a challenging issue as it has camera limitations and lower processing strength. However, in the near future, this technology could provide a large contribution to Smartphone-based real-time processing is a challenging issue as it has camera limitations and the upcoming IoT paradigms and the requirements of massive user connectivity. Recently, numerous lower processing strength. However, in the near future, this technology could provide a large experimental works have been published [11–18] for OCC with their individual implementation goals. contribution to the upcoming IoT paradigms and the requirements of massive user connectivity. According to the application, every work meets a certain level of results which can satisfy the particular Recently, numerous experimental works have been published [11–18] for OCC with their individual goals. Almost all the experiments are conducted using digital single-lens reflex (DSLR) or smartphone implementation goals. According to the application, every work meets a certain level of results which cameras and use MATLAB processing to analyze. Table1 shows the existing OCC implementation can satisfy the particular goals. Almost all the experiments are conducted using digital single-lens results using smartphone and other DSLR cameras with their challenges and limitations. An RGB reflex (DSLR) or smartphone cameras and use MATLAB processing to analyze. Table 1 shows the LED-based OCC system was proposed by P. Luo et al. [11], where a combined modulation scheme of existing OCC implementation results using smartphone and other DSLR cameras with their undersampled phase-shift ON-OFF keying (UPSOOK) and wavelength-division multiplexing (WDM) challenges and limitations. An RGB LED-based OCC system was proposed by P. Luo et al. [11], where was used. Although the authors achieved reasonable communication distance, the data rate is limited a combined modulation scheme of undersampled phase-shift ON-OFF keying (UPSOOK) and to bit per second scale only. According to another work of P. Luo et al. [12], a data rate of 500 bps is wavelength-division multiplexing (WDM) was used. Although the authors achieved reasonable achieved at a distance of 1.5 m. This work also has to perform off line simulations as well. Multi-level communication distance, the data rate is limited to bit per second scale only. According to another intensity modulation (MI) and color-shift keying (CSK) can provide a comparatively higher data rate work of P. Luo et al. [12], a data rate of 500 bps is achieved at a distance of 1.5 m. This work also has (up to 10 kbps) but are limited with the communication distance [14,16,17]. to perform off line simulations as well. Multi-level intensity modulation (MI) and color-shift keying (CSK) can provide a comparatively higher data rate (up to 10 kbps) but are limited with the communication distance [14,16,17].
Electronics 2019, 8, 913 4 of 17
Table 1. Survey of the existing optical camera communication (OCC) implementations with their limitations and challenges.
Article Implementation Direction Achievement Limitations/Challenges
- Performed off-line RGB LED-based Optical camera - Data rate 150 bps. processing in MATLAB [11] (DSLR) communication using - Communication distance 60 m. - Uses single RGB LED UPSOOK and WDM - Lower capacity
- Lower Dual LED-based optical camera - Data rate 500 bps. communication distance [12] (Canon 700D) communication - Communication distance 1.5 m. - Performed off-line using 1024 – QAM. - Uses 2 LED processing in MATLAB - Lower capacity
- Don’t provide Indoor positioning using optical real-time processing - 5 to 6.6 cm positioning error [13] camera (Canon EOS 7D) - Not mentioned about the communication data rate
- Does not provide Optical camera (smartphone) - Data rate 10 kbps real-time processing [14] communication using four-level - Communication distance 2 m - Uses single LED intensity modulation. - Lower distance
- Not processed by - 100 cm reflective back with Smartphone (Galaxy s6) based smartphone in real-time [15] 112 lux illuminance NLOS OCC. - Very short distance
- Does not support multiple - Data rate 8.64 kbps OCC using color-shift keying LED communications. [16] - At 4 cm distance modulation - Too short distance
- 98% data recovery at - Communication distance Screen to smartphone camera [17] 10–15 cm distance is too short in case communication using VLC - Achieved 112.5 kbps rate. of smartphone.
- Does not support - Achieved kbps range data rate Indoor VLC using mobile essential mobility [18] at a few cm distance phone camera - Too short distance
3. OCC Transmitter Characteristics In OCC systems, the transmitter has low power consumption and is cost-effective because it uses existing lighting equipment such as LEDs. Also, it consists of simple circuitry design and the only requirements are a signal modulator and a power supply. We must select the proper encoding and modulation techniques to ensure a flicker-free light transmission system. The following subsections discuss the types of LEDs used in OCC technology and the existing modulation and encoding techniques, along with their pros and cons.
3.1. Types of LED Transmitters Either an infrared (IR) or a visible light-based LED transmitter can be used in OCC depending on the application. IR-LED is used for such OCC applications that require non-luminous communication. For example, IR-LED was used for uplink connectivity in the VLC system [19,20]. An advantage of using IR light as a transmitter is that it eliminates the problem of flickering. However, visible LEDs are Electronics 2019, 8, 913 5 of 17
Electronics 2019, 8, x FOR PEER REVIEW 5 of 17 widely used as OCC transmitters where illumination and communication are both important. We can useare both existing important. LEDs as We indoor can use light existing for indoor LEDs localization as indoor light and for other indoor OCC-based localization applications, and other OCC- street lightsbased andapplications, car back lightsstreet forlights OCC-based and car back smart lights vehicular for OCC-based applications. smart In thisvehicular case, flickering applications. can beIn mitigatedthis case, flickering using proper can changesbe mitigated in the using modulation proper changes schemes. in the modulation schemes.
3.2. Modulations and Encoding Modulation isis performed performed in in a wirelessa wireless communication communication system system to transmit to transmit data through data through a defined a frequencydefined frequency band. For band. an OCC For system, an OCC researchers system, haveresearchers developed have many developed modulation many techniques modulation to overcometechniques several to overcome limitations several or to limitations improve system or to performance, improve system depending performance, on the receiver depending parameters. on the Generally,receiver parameters. these modulation Generally, techniques these aremodulation divided into techniques three types: are screen-modulation, divided into three under-sampled types: screen- modulation, andunder-sampled oversampled modula modulationtion, and [21]. oversampled However, screen-based modulation modulation [21]. However, techniques screen-based are not preferablemodulation because techniques of their are short not preferable range (10–110 because cm) communicationof their short range system (10–110 [17]. cm) When communication the sampling frequencysystem [17]. is When two times the sampling higher than frequency the highest is two frequencytimes higher component than the highest of the signal,frequency it is component termed as oversampling.of the signal, it is Binary termed color-shift as oversampling. keying (BCSK) Binary andcolor-shift rolling keying shutter-based (BCSK) modulationsand rolling shutter-based are types of oversamplingmodulations are techniques types of oversampling in which the scanning techniques rate in is which very high the scanning to achieve rate improved is very high resolution to achieve and dataimproved rates. resolution However, and these data techniques rates. However, are also thes restrictede techniques by communication are also restricted distance by communication and mobility supportdistance [ 22and,23 mobility]. Current support commercial [22,23]. cameras Current are commercial built with low cameras sampling are built rates with which low significantly sampling rates limit thewhich overall significantly data rate limit for OCC. the overall To deal data with rate this for problem, OCC. To undersampling-based deal with this problem, modulation undersampling- schemes havebased been modulation introduced. schemes Undersampling have been introduced. is done with Undersampling necessary changes is done in OCC with tonecessary avoid flickering changes byin adoptingOCC to avoid some flickering subcarrier by intensity adopting modulation some subcar techniques.rier intensity Low modulation frame rate (LFR) techniques. cameras Low can frame easily integraterate (LFR) this cameras modulation can easily technique. integrate The this frequency modula shifttion on-otechnique.ff keying The (FSOOK) frequency modulation shift on-off technique keying is(FSOOK) the simplest modulation among all technique the techniques is the that simplest employ theamong undersampling all the techniques mechanism. that In employ this scheme, the twoundersampling frequencies mechanism. are exploited In tothis represent scheme, 1two and freque 0, calledncies “mark”are exploited and “space” to represent [8,21 ],1 and respectively. 0, called One“mark” of theand frequencies “space” [8,21], is represented respectively. as One the of integer the frequencies multiple ofis represented the camera as frame the integer rate. Figure multiple2a showsof the camera an example frame in rate. which Figure the mark2a shows and an space example frequencies in which are the 105 mark and 120and Hz, space respectively, frequencies with are a105 camera and 120 sampling Hz, respectively, rate of 30 fps.with Thea camera frame sampling header is rate used of to 30 indicate fps. The the frame start header of a data is used packet. to Thisindicate modulation the start of scheme a data opacket.ffers a This communication modulation distance,scheme offers of up a communication to 5 m, although distance, the capacity of up to is very5 m, limitedalthough [23 the]. Phase-shiftcapacity is very on-o fflimitedkeying [23]. (PSOOK) Phase-shift was proposedon-off keying in Refs. (PSOOK) [24,25 ],was employing proposed the in undersamplingRefs. [24,25], employing technique. the This undersampling technique can technique. be utilized This to increase technique the can sampling be utilized efficiency. to increase Figure the2b showssampling an exampleefficiency. of Figure this scheme 2b shows in which an example mark and of spacethis scheme frequencies in which are themark same and (120 space Hz). frequencies However, aare phase the same difference (120 wasHz). utilizedHowever, to logicallya phase representdifference 1 was and utilized 0. Luo et to al. logically [11] achieved represent a communication 1 and 0. Luo distanceet al. [11] of achieved over 60 a m communication using this scheme. distance However, of over the 60 achieved m using datathis scheme. rate was However, significantly the low.achieved data rate was significantly low. Signal amplitude LED ON
Bit 0 Bit 1 LED OFF Time Sampling interval (a)
LED ON Bit 1 Bit 0 Bit 1 Bit 0
LED OFF Sampling Time interval (b) Figure 2. An example of undersampling-based modulation schemes for OCC systems [[8].8]. ( a) FSOOK, FSOOK, ((bb)) PSOOK.PSOOK.
Additionally, an undersampled pulse-position modulation technique was proposed by L. Shangguan et al. [26] to develop cm-level accurate indoor positioning system using OCC. A combined
Electronics 2019, 8, 913 6 of 17
Additionally, an undersampled pulse-position modulation technique was proposed by L. Shangguan et al. [26] to develop cm-level accurate indoor positioning system using OCC. A combined red-green-blue (RGB) color intensity modulation technique was proposed by P. Tian et al. [27], which was able to achieve a data rate of up to 95 kbps over a 1.2 m communication range. According to V.P. Rachim et al. [14], a data rate up of to 10 kbps was developed with a 2 m range using a four-intensity modulation technique. Hence, research is being carried out to develop such modulation schemes for OCC, which can offer both improved capacities and higher communication distances. Moreover, encoding in communication is a process of changing the original bit sequences to another format for efficient transmission. For OCC technology, the encoding technique can be of two types. The transmission channel can be encoded to detect and eliminate any bit errors through the optical channel. Another technique that is usually used for OCC is a type of line coding called Manchester coding. This encoding allows a transition between the high and low for equal amounts of time within the time period for each bit. Thus, the flickering problem can be eliminated even if a continuous 1 or 0 is transmitted for a long period of time.
4. Smartphone as OCC Receiver Image sensors cannot have identical characteristics in nature, their imaging quality and behavior vary for different smartphones. Also, a camera’s sampling rate can vary from very low to very high. Moreover, OCC performance such as the data rate and communication distance depends highly on these camera parameters. Cameras with higher sampling rates can offer much higher throughput but these types of cameras are much costlier than those with low sampling rates. The following subsections explain these parameters in detail.
4.1. Frame Rate and Exposure Control The frame rate is defined as the number of individual frames captured within a single second. Sometimes, there is confusion between frame rate and shutter speed, but shutter speed is referred to as the amount of time for which an individual frame is exposed to light. Exposure time and shutter speed are particularly the same thing, expressed in different ways. Based on the frame rate, cameras rated at higher than 200 fps are considered as high frame rate (HFR) cameras, whereas those rated below 60 fps are classified as LFR cameras. Although HFR cameras promise higher data rates, most of today’s smartphone cameras are of the LFR type and have varying frame rates (24–30 fps) when recoding video. In an OCC system, a light source such as an LED transmits data according to how it is modulated. The receiver camera receives the data by processing its individual image frames. Each frame contains the data in the form of captured LED light. To avoid unexpected interference from the surroundings, each frame should be exposed to light for a very short time which is controlled by the shutter speed of the camera. Therefore, depending on the LED light intensity, an LED image is formed on the image sensor of the camera keeping the low illuminating surroundings dark. This is done for avoiding the complexity of the data recovery and error minimizing. To implement OCC using a smartphone camera, the frame rate should be fixed at a certain value. Two factors cause frame rate variation: the exposure duration and the focus adjustment. To have a fixed frame rate, the exposure duration should be fixed and the autofocus should be locked. Android versions up to 4.4 (camera application programming interfaces (APIs)) do not support exposure control. Only the camera2 API has additional features, such as manually-controlled exposure, focus, raw capture etc. Also, cameras should open on a secondary handler thread and the camera2 API makes thread management easier.
4.2. Rolling Shutter Effect Currently, most consumer cameras contain CMOS sensors, which include rolling shutter mechanisms. The sequential readout technique is a key feature of rolling shutter cameras in which frames are not captured at exactly the same instant [7]. The scene is captured by rapid scanning, either Electronics 2019, 8, 913 7 of 17 horizontally or vertically, unlike a global shutter camera, in which the complete scene is captured at the same time. Because a global shutter camera allows its sensor to be exposed to light once, it can holdElectronics either 2019 the, 8, x ON FOR or PEER OFF REVIEW state of an LED in a single frame. In a rolling shutter camera, each row7 of of17 pixels is exposed simultaneously at the exposure time. The readout time protects the rows of pixels frompixels overlapping. is exposed simultaneously In a single captured at the exposure image, the time. rolling The shutterreadout allows time protects multiple the exposures. rows of pixels This enablesfrom overlapping. multiple LED In statesa single to becaptured obtained image, at the the same rolling time shutter in a single allows frame multiple as each exposures. row is exposed This onceenables to light.multiple Therefore, LED states for to an be LED, obtained that flickers at the sa onme and time off inaccording a single frame to the as modulated each row is binary exposed bit stream,once to thelight. captured Therefore, image for contains an LED, a bunchthat flickers of black on and and white off according strips. The to strip the modulated widths depend binary on thebit modulationstream, the captured frequencies, image and contains the number a bunch of strips of black depends and white on the strips. distance The between strip widths the camera depend and on LED.the modulation Figure3 shows frequencies, the image and formation the number technique of strips in a depends rolling shutter on the camera. distance Here between the LED the status camera is definedand LED. using Figure the 3 black shows and the white image blocks formation and the technique captured in LED a rolling image isshutter shown camera. after it Here is rotated the LED 270º clockwise.status is defined For example, using the the black region and is marked white blocks in the imageand the to captured show what LED happens image during is shown the after exposure it is timerotated for 270º row clockwise. n and row For n+ 1.example, In the second the region case, is wemarked get a in transition the image region to show for thewhat LED happens between during ON the exposure time for row n and row n+1. In the second case, we get a transition region for the LED and OFF, as it changes its state during the exposure time. between ON and OFF, as it changes its state during the exposure time.
LED state Bright Dark Bright Dark Bright Dark Bright Dark
Trow−exp osure TRead− out Row n Row n+1 Row n+2 :.
:.
Row x Captured Image
Figure 3. LED image capturing mechanisms for a rolling shutter camera [7]. Figure 3. LED image capturing mechanisms for a rolling shutter camera [7]. 4.3. Image Processing Platform (Android with OpenCV) 4.3. Image Processing Platform (Android with OpenCV) For real-time OCC applications, smartphone devices should perform fast processing of the received imageFor information. real-time OCC Because applications, the frame smartphone rate needs todevi beces fixed should for aperform smartphone fast camera,processing real-time of the received image information. Because the frame rate needs to be fixed for a smartphone camera, real- processing should be completed within the individual frame duration. If R f is the frame rate of the camera,time processing then 1/R shouldf is the be time completed limit for wi eachthin frame the individual in real-time. frame All duration. smartphone If Rf camerasis the frame are rollingrate of shutter type and captured images are scanned row by row. Generally, for an Android smartphone-based the camera, then1/ Rf is the time limit for each frame in real-time. All smartphone cameras are rolling OCC system, an application needs to be developed with the help of OpenCV to process the captured shutter type and captured images are scanned row by row. Generally, for an Android smartphone- image and extract data. OpenCV is a library of programming functions which mainly handles real-time based OCC system, an application needs to be developed with the help of OpenCV to process the computer vision and it can be easily integrated with Android studio. To develop such a real-time captured image and extract data. OpenCV is a library of programming functions which mainly application with Android Studio and OpenCV, we have to follow the step by step procedure that is handles real-time computer vision and it can be easily integrated with Android studio. To develop described in Figure4. such a real-time application with Android Studio and OpenCV, we have to follow the step by step procedure that is described in Figure 4. Initially, a camera permission setup is needed to build any camera-based application. Also the API should be higher than level 21 to use camera2 packages. The camera setup is required to define the window size and preview management. Because camera2 packages support dynamic exposure and frame rate control, we need to set the CaptureRequest such as camera focus, frame rate, exposure duration, sensitivity, and image format. The device orientation needs to be fixed using the DeviceManager function. The LED image can be processed to get information using the OpenCV library, which helps to obtain an image and convert it to required formats. An image detection function should be developed to perform LED stripe width measurements. Moreover, the OpenCV library is integrated with this platform to provide pixel reading, thresholding, and object detecting
Electronics 2019, 8, x FOR PEER REVIEW 8 of 17
functionalities. Some of the Android features described in this study are used to create the camera device and process the requests. One request acts to capture a single frame and output a set of image buffers for the request. A CameraManager class is required when multiple requests are in the queue, to maintain the full frame rate. The image data is encapsulated in Image objects and can be directly accessed through a class named ImageReader using a YUV_420_888 format. Initially, the camera shutter speed is controlled to focus only on the white LEDs and to detect only the region of interest (RoI). To extract the transmitted bit stream, the first thing that needs to be found is the header, followed by the original payload. As the widths of the strips vary with frequencies, we can determine Electronics 2019, 8, 913 8 of 17 the frequencies that the received signal contains by measuring the stripe width. Finally, after completing the image detection and data extraction, the received information can be displayed.
Import OpenCV Setup CaptureRequest Setup camera -get Image -focus Setup DeviceManager -convert to required Check -frame rate Output string as -(height, width) formate CameraAvaibality -exposure duration received data -preview -orientation -call detection -sensitivity -rotation function -image format -measure strips width
Figure 4. Procedure for getting and processing images in Android Studio using OpenCV. Figure 4. Procedure for getting and processing images in Android Studio using OpenCV. Initially, a camera permission setup is needed to build any camera-based application. Also the 5. Implementation Challenges API should be higher than level 21 to use camera2 packages. The camera setup is required to defineThe the windowprevious section size and describes preview the management. overall requirements Because of a camera2 smartphone-based packages supportOCC system. dynamic In exposurethis section, and framethe key rate challenges control, weandneed limitations to set theof thisCaptureRequest system are addressedsuch as camera by demonstrating focus, frame an rate, exposureandroid duration, smartphone-based sensitivity, OCC and system. image The format. detail Theed demonstration device orientation methodology needs to beand fixed test bed using are the DeviceManagerexplained infunction. Section 6. The These LED challenges image can includ be processede low data to get rates, information short distance using the communication, OpenCV library, multilateral communication, visual flickering, and user mobility support. These challenges arise which helps to obtain an image and convert it to required formats. An image detection function should owing to the limitations of sampling rate, exposure control, and processing speed of current be developed to perform LED stripe width measurements. Moreover, the OpenCV library is integrated smartphone cameras. We discuss the major challenges and techniques for their possible remedies in with this platform to provide pixel reading, thresholding, and object detecting functionalities. Some the following subsections. of the Android features described in this study are used to create the camera device and process the requests.5.1. Visual One Flickering request acts to capture a single frame and output a set of image buffers for the request. A CameraManager class is required when multiple requests are in the queue, to maintain the full frame OCC technology can suffer from visual flickering. Visual flickering occurs when a change in rate. The image data is encapsulated in Image objects and can be directly accessed through a class LED brightness (depending on modulation frequency) is perceived by the human eye. In general, named ImageReader using a YUV_420_888 format. Initially, the camera shutter speed is controlled to frequencies greater than 200 Hz are considered safe [10]. Lower carrier frequencies allow LEDs to focus only on the white LEDs and to detect only the region of interest (RoI). To extract the transmitted turn on and off for longer periods of time, whereas higher carrier frequencies allow it for a very short bit stream, the first thing that needs to be found is the header, followed by the original payload. period of time. The flickering of an LED also depends on its circuital properties. In an OCC Asdemonstration, the widths of thethe stripsLED driver vary withand its frequencies, circuitry have we to can support determine the required the frequencies fast switching that the between received signalthe ON contains and OFF by measuringstates. Visual the flic stripekering width. may cause Finally, some after biological completing problems the image or even detection health issues. and data extraction,Neurological the receivedproblems, information headaches, canfatigue be displayed. blurred vision, eyestrain, and reductions in visual task performance can occur from continuous visual flickering. For that reason, when implementing any 5. Implementation Challenges OCC system, the flickering issue should be kept in mind. Figure 5 illustrates the variation in the ON andThe OFF previous times of section an LED describes when captured the overall using requirements a rolling shutter of a smartphone-based camera with a variation OCC system. in carrier In this section,frequency. the key The challenges widths of the and strips limitations decrease of as this th systeme carrier are frequency addressed increases. by demonstrating We used an anLED android of 9 smartphone-basedcm diameter and OCCtransmitted system. digital The detailed bits with demonstration carrier frequencies methodology from 200 andHz to test 6000 bed Hz. are Also, explained we inset Section the frame6. These rate challenges and exposure include duration low dataof the rates, camera short to distancethe values communication, of 20 fps and multilateral1/8000 sec communication,respectively. The visual camera flickering, could successfully and user mobilitydecode the support. received Theseinformation challenges to a maximum arise owing of 6000 to the limitationsHz because of of sampling a limitation rate, in exposure the number control, of pixels and on processing the image sensor. speed of Differentiating current smartphone between cameras. 1 and We0 discussbits becomes the major difficult challenges in this situation, and techniques becaus fore the their width possible of the remedies individual in On-OFF the following state comprises subsections. less than 1 pixel. However, at lower frequencies of around 200 Hz, width detection for the purpose 5.1.of Visualdecoding Flickering becomes very easy but increases the risk of causing visual flickering. In this case, the width becomes larger and the human eye can differentiate the state change between ON and OFF. AfterOCC considering technology both can these suffer limitations, from visual research flickering.ers are Visual interested flickering to use occurs modulation when frequency a change in LEDbetween brightness (1 to 5) (depending kHz [23]. on modulation frequency) is perceived by the human eye. In general, frequencies greater than 200 Hz are considered safe [10]. Lower carrier frequencies allow LEDs to turn on and off for longer periods of time, whereas higher carrier frequencies allow it for a very short period of time. The flickering of an LED also depends on its circuital properties. In an OCC demonstration, the LED driver and its circuitry have to support the required fast switching between the ON and OFF states. Visual flickering may cause some biological problems or even health issues. Neurological problems, headaches, fatigue blurred vision, eyestrain, and reductions in visual task performance can occur from continuous visual flickering. For that reason, when implementing any OCC system, the flickering issue should be kept in mind. Figure5 illustrates the variation in the ON and OFF times of an LED when captured using a rolling shutter camera with a variation in carrier frequency. The widths of the strips decrease as the carrier frequency increases. We used an LED of 9 cm diameter and transmitted digital bits with carrier frequencies from 200 Hz to 6000 Hz. Also, we set the frame rate Electronics 2019, 8, 913 9 of 17 and exposure duration of the camera to the values of 20 fps and 1/8000 sec respectively. The camera could successfully decode the received information to a maximum of 6000 Hz because of a limitation in the number of pixels on the image sensor. Differentiating between 1 and 0 bits becomes difficult in this situation, because the width of the individual On-OFF state comprises less than 1 pixel. However, at lower frequencies of around 200 Hz, width detection for the purpose of decoding becomes very easy but increases the risk of causing visual flickering. In this case, the width becomes larger and the human eye can differentiate the state change between ON and OFF. After considering both these limitations, Electronicsresearchers 2019 are, 8, x interested FOR PEER REVIEW to use modulation frequency between (1 to 5) kHz [23]. 9 of 17
200 Hz 500 Hz
1000 Hz 3000Hz
5000 Hz 6000 Hz Figure 5. CapturedCaptured LED images for various transmitted frequencies. 5.2. Multilateral Communication 5.2. Multilateral Communication OCC technology is expanding its application toward both indoor and outdoor systems, such as OCC technology is expanding its application toward both indoor and outdoor systems, such as mobile robot localization and navigation, and V2X communications. In those systems, the camera mobile robot localization and navigation, and V2X communications. In those systems, the camera needs to communicate with multiple LEDs. However, challenges such as low frame rates, function needs to communicate with multiple LEDs. However, challenges such as low frame rates, function overloading, and slow processing arise when smartphones try to establish multiple communication overloading, and slow processing arise when smartphones try to establish multiple communication links between the LEDs and camera. To achieve the multilateral communication, the first requirement links between the LEDs and camera. To achieve the multilateral communication, the first requirement is to detect and track each LED. Also, the positions of the LEDs on the image sensor can change is to detect and track each LED. Also, the positions of the LEDs on the image sensor can change because the user can be in a mobile condition. Therefore, the Android application needs to record because the user can be in a mobile condition. Therefore, the Android application needs to record the the pixel values and compare between previous and present pixel values to track the detected LEDs. pixel values and compare between previous and present pixel values to track the detected LEDs. The The second requirement is to measure the stripe widths to extract the transmitted frequencies and second requirement is to measure the stripe widths to extract the transmitted frequencies and demodulate the received payload. Sometimes, function overloading occurs when the input to the demodulate the received payload. Sometimes, function overloading occurs when the input to the processing unit overloads the system capacity. In Figure6, the limitations behind the multiple object processing unit overloads the system capacity. In Figure 6, the limitations behind the multiple object detections and processing in real-time are explained. The main objective is getting the stripe widths, detections and processing in real-time are explained. The main objective is getting the stripe widths, which can be detected individually but require higher processing time (Figure6a). However, detection which can be detected individually but require higher processing time (Figure 6a). However, of all the strips of an LED can be processed using comparatively less time (Figure6b). In the latter detection of all the strips of an LED can be processed using comparatively less time (Figure 6b). In case, a rectangle is drawn that meets the edge region of the LED, and a straight line is considered as a the latter case, a rectangle is drawn that meets the edge region of the LED, and a straight line is reference to obtain the widths of the strips. considered as a reference to obtain the widths of the strips. Therefore, the total time to detect and track each LED and the time to demodulate the received signal should be within the frame processing time.
(a)
(b) (c)
Figure 6. Multiple object detection. (a) Individual stripe detection of a single LED (requires much processing time). (b) A single LED detection using a rectangle (requires less processing time). (c) Multiple object detection using a rectangle (optimum).
Electronics 2019, 8, x FOR PEER REVIEW 9 of 17
200 Hz 500 Hz
1000 Hz 3000Hz
5000 Hz 6000 Hz Figure 5. Captured LED images for various transmitted frequencies.
5.2. Multilateral Communication OCC technology is expanding its application toward both indoor and outdoor systems, such as mobile robot localization and navigation, and V2X communications. In those systems, the camera needs to communicate with multiple LEDs. However, challenges such as low frame rates, function overloading, and slow processing arise when smartphones try to establish multiple communication links between the LEDs and camera. To achieve the multilateral communication, the first requirement is to detect and track each LED. Also, the positions of the LEDs on the image sensor can change because the user can be in a mobile condition. Therefore, the Android application needs to record the pixel values and compare between previous and present pixel values to track the detected LEDs. The second requirement is to measure the stripe widths to extract the transmitted frequencies and demodulate the received payload. Sometimes, function overloading occurs when the input to the processing unit overloads the system capacity. In Figure 6, the limitations behind the multiple object detections and processing in real-time are explained. The main objective is getting the stripe widths, which can be detected individually but require higher processing time (Figure 6a). However, detection of all the strips of an LED can be processed using comparatively less time (Figure 6b). In
Electronicsthe latter2019 case,, 8, 913a rectangle is drawn that meets the edge region of the LED, and a straight line10 of 17is considered as a reference to obtain the widths of the strips.
(a)
(b) (c) Figure 6. Multiple object detection. (a) Individual stripe detection of a single LED (requires much Figure 6. Multiple object detection. (a) Individual stripe detection of a single LED (requires much processing time). (b) A single LED detection using a rectangle (requires less processing time). (c) Multiple processing time). (b) A single LED detection using a rectangle (requires less processing time). (c) object detection using a rectangle (optimum). Multiple object detection using a rectangle (optimum). 5.3. Data Rate OCC is considered as a promising technology for low data rate applications, such as vehicular communications and indoor positioning and navigation. There are a number of factors that limit the data rate of this system. Specially, the data rate depends on technical factors such as modulation frequencies, frame rate, shutter speed, signal-to-noise ratio (SNR), and spatial distortion. Because almost every smartphone camera is rated with LFR ranges between 20 and 60 fps, it is required to allocate the transmitted symbol rate within at least half the region of the frame rate to decode the received data [28]. Therefore, a higher frame rate (i.e., within the 1000 fps range) enables us to capture a higher number of transmitted bits, and thus, increases the data rate. However, this is not currently available from a regular user point of view. By controlling the exposure time of the camera, we can develop a noise-free receiver. To ensure a healthier data rate, researchers are concentrating on developing new modulation techniques as mentioned in Section3. However, the data rate cannot be achieved at the desired level when implemented on a smartphone camera platform because of the existing limitations on frame rate and processing speed. As a solution to overcoming this data rate shortage, data can be received from multiple LEDs simultaneously, an operation known as MIMO. According to the work of L. Zeng et al. [3], the authors have verified the data rate up to the Gbps range through simulation. They considered the optical MIMO operation using an imaging diversity receiver. Some RGB LED-based intensity modulation techniques have been developed to increase the data rate up to the several kbps range while implementing with the smartphone, but it exhibits limited communication distance [29]. Using the color intensity modulation technique, rates of up to 95 kbps have been achieved by a 330 fps camera, which also requires a PC for processing [27]. However, this OCC technique limits the communication range to 1.5 m, which is not very effective in real-time OCC-based applications. Current smartphone versions cannot support this high frame rate for real-time data receptions. Still, it remains challenging to develop MIMO using smartphones to overcome the processing time limitations because real-time detection and tracking are needed to support mobility.
5.4. Communication Range The distance between the transmitter and receiver is considered as an important parameter for evaluating the performance of a camera-based optical communication system. Recently, researchers have been trying to find solutions to increase the communication link range [28–31]. They have tried to apply various light modulation techniques, but the distance also depends on the LED size and pixel size in some cases. Figure7 describes the captured LED images at various distant LED positions. Electronics 2019, 8, 913 11 of 17
We can see that for the nearest LED, the covered pixel area on the image sensor is larger than the farthest one, although the actual sizes of the LEDs are the same, which is described in Chen, H.W et al. [16] as well. However, the strip’s widths do not change with the variation in distance between transmitter and receiver because the light ON-OFF time remains the same. Therefore, the LED situated at the greater distance occupies a limited number of black and white strips. We use UFSOOK modulation to experimentally observe the distance variation between the LED and the camera, and the variation in the number of strips. We observe 26, 18, and 9 strips at LED distances of 0.5, 2.0, and 4.0 m, respectively (2 and 4 kHz are used for the ON and OFF conditions of an LED, respectively), as shown in Figure7. Earlier in Section3, we describe that when Manchester line coding is used and a digital 1 or 0 is transmitted, the LED state is changed twice per bit. As a result, the captured image on the image sensor consists of two strips per bit. Therefore, we can say that at least three strips are needed on the image sensor to extract the received data. This problem limits the communication distance for a smartphone-based OCC system. We experimentally found that an LED of 9 cm diameter provides the minimal required number of strips at a maximal distance of 5 m. Therefore, the communication distance can be increased if a larger LED is used for this system. Another solution to increase the communicationElectronics 2019, 8, x FOR range PEER is increasingREVIEW the modulation frequency. However, synchronization with11 of the 17 sampling rate is also needed. We experimentally evaluated the variation in the number of strips with synchronization with the sampling rate is also needed. We experimentally evaluated the variation in the change in distance between the camera and LED, as shown in Figure8. Here, we take three LEDs the number of strips with the change in distance between the camera and LED, as shown in Figure 8. with surface areas corresponding to diameters of 6, 9.5, and 14 cm. It is usual for a larger LED to have Here, we take three LEDs with surface areas corresponding to diameters of 6, 9.5, and 14 cm. It is a higher number of strips and provide a longer communication distance for the same transmitted bit usual for a larger LED to have a higher number of strips and provide a longer communication frequency. In Figure8a,b, we considered two di fferent carrier frequencies of 3 and 5 kHz, respectively. distance for the same transmitted bit frequency. In Figure 8a,b, we considered two different carrier Because a higher frequency causes reduced stripe widths, it can provide a higher number of LED frequencies of 3 and 5 kHz, respectively. Because a higher frequency causes reduced stripe widths, it ON-OFF strips. If frequency synchronization with the camera sampling rate can be confirmed, then can provide a higher number of LED ON-OFF strips. If frequency synchronization with the camera this higher carrier frequency should also provide a higher communication distance. Furthermore, we sampling rate can be confirmed, then this higher carrier frequency should also provide a higher provide a comparison between the maximal communication ranges for these different-sized LEDs with communication distance. Furthermore, we provide a comparison between the maximal respect to the two individual frequencies (3 and 5 kHz) in Figure9. This figure depicts that the 14 cm communication ranges for these different-sized LEDs with respect to the two individual frequencies diameter LED can provide maximal range that is twice as long for 5 kHz as it is for 3 kHz. However, (3 and 5 kHz) in Figure 9. This figure depicts that the 14 cm diameter LED can provide maximal range existing cameras can only differentiate the black and white strips of frequency that are not higher than that is twice as long for 5 kHz as it is for 3 kHz. However, existing cameras can only differentiate the 6 kHz because of the limitation of unit pixel size. Beyond that frequency, the LED will switch very black and white strips of frequency that are not higher than 6 kHz because of the limitation of unit quickly between the ON and OFF states, which will result in greatly reduced strips widths that are pixel size. Beyond that frequency, the LED will switch very quickly between the ON and OFF states, even smaller than the unit pixel width. which will result in greatly reduced strips widths that are even smaller than the unit pixel width.
4 m 9 strips
18 strips 2 m
26 strips
0.5 m
(a) (b) Figure 7.7. Distance problem visualization for a smartphone-basedsmartphone-based OCC system. ( a) LEDs at variousvarious distances, (b)) imageimage ofof LEDsLEDs afterafter changingchanging thethe exposureexposure durationduration toto 11/8000/8000 s.s.
45 120 LED diameter of 6 cm LED diameter of 6 cm 40 LED diameter of 9.5 cm LED diameter of 9.5 cm 100 LED diameter of 14 cm 35 LED diameter of 14 cm
30 80
25 60 20
Number of strips 15 Number of Strips 40
10 20 5
0 0 12345678 2 4 6 8 1012141618 Communication distance (m) Communication distance (m) (a) (b)
Figure 8. Variation in the number of strips with communication range for a CMOS camera at a transmitted bit frequency of (a) 3 kHz, and (b) 5 kHz.
Electronics 2019, 8, x FOR PEER REVIEW 11 of 17
synchronization with the sampling rate is also needed. We experimentally evaluated the variation in the number of strips with the change in distance between the camera and LED, as shown in Figure 8. Here, we take three LEDs with surface areas corresponding to diameters of 6, 9.5, and 14 cm. It is usual for a larger LED to have a higher number of strips and provide a longer communication distance for the same transmitted bit frequency. In Figure 8a,b, we considered two different carrier frequencies of 3 and 5 kHz, respectively. Because a higher frequency causes reduced stripe widths, it can provide a higher number of LED ON-OFF strips. If frequency synchronization with the camera sampling rate can be confirmed, then this higher carrier frequency should also provide a higher communication distance. Furthermore, we provide a comparison between the maximal communication ranges for these different-sized LEDs with respect to the two individual frequencies (3 and 5 kHz) in Figure 9. This figure depicts that the 14 cm diameter LED can provide maximal range that is twice as long for 5 kHz as it is for 3 kHz. However, existing cameras can only differentiate the black and white strips of frequency that are not higher than 6 kHz because of the limitation of unit pixel size. Beyond that frequency, the LED will switch very quickly between the ON and OFF states, which will result in greatly reduced strips widths that are even smaller than the unit pixel width.
4 m 9 strips
18 strips 2 m
26 strips
0.5 m
(a) (b) Figure 7. Distance problem visualization for a smartphone-based OCC system. (a) LEDs at various Electronics 2019, 8, 913 12 of 17 distances, (b) image of LEDs after changing the exposure duration to 1/8000 s.
45 120 LED diameter of 6 cm LED diameter of 6 cm 40 LED diameter of 9.5 cm LED diameter of 9.5 cm 100 LED diameter of 14 cm 35 LED diameter of 14 cm
30 80
25 60 20
Number of strips 15 Number of Strips 40
10 20 5
0 0 12345678 2 4 6 8 1012141618 Communication distance (m) Communication distance (m) (a) (b) Figure 8. Variation in the number of strips with communication range for a CMOS camera at a ElectronicsFigure 2019 8., 8 , Variationx FOR PEER in REVIEW the number of strips with communication range for a CMOS camera at12 a of 17 transmitted bit frequency of (a) 3 kHz, and (b) 5 kHz. transmitted bit frequency of (a) 3 kHz, and (b) 5 kHz. 18 LED diameter of 6 cm 16 LED diameter of 9.5 cm LED diameter of 14 cm 14
12
10
8
6
Communication range [m] 4
2
0 3 kHz 5 kHz Figure 9. Variation of communication range with the variation of modulation frequency and change of Figure 9. Variation of communication range with the variation of modulation frequency and change LED size. of LED size. 5.5. User Mobility 5.5. User Mobility It is crucial that the smartphone-based OCC system should support user mobility. In the static condition,It is crucial a user that can the get smartphone-based the data from LED OCC regardless system of should the processing support time user dependency. mobility. In However,the static condition,when a user a user is in can the get mobile the data state, from the user’sLED regard maximumless of speed the processing depends on time the dependency. total processing However, time to whenget the a informationuser is in the from mobile the state, LED. the Figure user’s 10 maximumillustrates thespeed concept depends practically. on the total Suppose processing the mobile time touser get of the Google information pixel 2 from is at the initialLED. Figure position 10 andillustrates moves the to itsconcept final positionpractically. at the Suppose user of the Galaxy mobile s7 useredge. of Meanwhile, Google pixel the 2 useris at coversthe initial the position distance and x. The moves LED to image its final on theposition image at sensor the user also of moves Galaxy from s7 edge.left to Meanwhile, right as shown the inuser the covers figure. the Hence, distance to cover x. The the distanceLED image x, the on user’s the image velocity sensor can bealso increased moves fromif effi cientleft to processing right as shown is ensured in the to figure. minimize Hence, the to total cover processing the distance time. x, Besides, the user’s LED velocity detection can and be increasedtracking isif neededefficient to processing accept mobility, is ensured and this to minimize requires extra the total processing proces timesing astime. well Besides, [23,32]. LED It is detectionrealized experimentally, and tracking is thatneeded for ato 20 accept fps camera, mobility, each and frame this requires requires 20 extra ms additionalprocessing time time to as detect well [23,32].and track It is LEDs realized inside experimentally, the camera field-of-view that for a (FoV).20 fps Anothercamera, each dependency frame requires of mobility 20 ms support additional arises timewhen to the detect user and moves track from LEDs one inside LED the field camera to another. field-of-view In this case, (FoV). a new Another LED entersdependency to the cameraof mobility FoV supportand the arises user will when receive the user new moves data without from one aff LEDecting fiel thed to OCC another. performance. In this case, However, a new LED the enters previous to theLED camera information FoV and can the be user removed will receive if there new is nodata data-storing without affecting system the associated OCC performance. with the application. However, theSince previous LED detection LED information and tracking can be in real-timeremoved areif ther considerede is no data-storing as a major system challenge associated in OCC with mobility the application.support, we Since investigate LED detection conventional and tracking object detection in real-time methods are considered and their as relative a major processing challenge in time OCC in mobility support, we investigate conventional object detection methods and their relative processing time in the following subsections. Later, we discuss new opportunities to build an Android learning- based detection platform with the existing OCC application which is TensorFlow Lite.
Google pixel2 Galaxy s7 edge
LED transmitter
− x tan1 ( ) 2h h
Initial x Final position position
Figure 10. Demonstrations of the OCC using Google pixel2 and Samsung Galaxy s7 edge smartphone.
Electronics 2019, 8, x FOR PEER REVIEW 12 of 17
18 LED diameter of 6 cm 16 LED diameter of 9.5 cm LED diameter of 14 cm 14
12
10
8
6
Communication range [m] 4
2
0 3 kHz 5 kHz
Figure 9. Variation of communication range with the variation of modulation frequency and change of LED size.
5.5. User Mobility It is crucial that the smartphone-based OCC system should support user mobility. In the static condition, a user can get the data from LED regardless of the processing time dependency. However, when a user is in the mobile state, the user’s maximum speed depends on the total processing time to get the information from the LED. Figure 10 illustrates the concept practically. Suppose the mobile user of Google pixel 2 is at the initial position and moves to its final position at the user of Galaxy s7 edge. Meanwhile, the user covers the distance x. The LED image on the image sensor also moves from left to right as shown in the figure. Hence, to cover the distance x, the user’s velocity can be increased if efficient processing is ensured to minimize the total processing time. Besides, LED detection and tracking is needed to accept mobility, and this requires extra processing time as well [23,32]. It is realized experimentally, that for a 20 fps camera, each frame requires 20 ms additional time to detect and track LEDs inside the camera field-of-view (FoV). Another dependency of mobility support arises when the user moves from one LED field to another. In this case, a new LED enters to the camera FoV and the user will receive new data without affecting the OCC performance. However, Electronicsthe previous2019, 8LED, 913 information can be removed if there is no data-storing system associated with13 ofthe 17 application. Since LED detection and tracking in real-time are considered as a major challenge in OCC mobility support, we investigate conventional object detection methods and their relative processing the following subsections. Later, we discuss new opportunities to build an Android learning-based time in the following subsections. Later, we discuss new opportunities to build an Android learning- detection platform with the existing OCC application which is TensorFlow Lite. based detection platform with the existing OCC application which is TensorFlow Lite.
Google pixel2 Galaxy s7 edge
LED transmitter
− x tan1 ( ) 2h h
Initial x Final position position
Figure 10. DemonstrationsDemonstrations of the OCC using Google pixe pixel2l2 and Samsung Galaxy s7 edge smartphone.
5.5.1. Object Detection Methods Android-based object recognition requires fast learning from a pre-trained model, classification, and detection from a real-time video. There are existing deep neural network-based detection techniques that can be used for real-time video processing that are tabulated in Table2. Faster region-convolutional neural network (R-CNN) and mask R-CNN are conceptually similar. The only difference is that in mask R-CNN, a segmentation mask is added on each RoI along with the bounding boxes which facilitates the extensive usage. Spatial Pyramid Pooling network (SPP-net) can generate a fixed-length representation regardless of image size or scale. In this system, the feature maps of a convolution layer are fed into an SPP-layer, which then represents fixed-length outputs to fully-connected layers.
Table 2. Real-time object detection methods with their inference time.
Methods Detection Procedures Frameworks Inference Time Language Faster R-CNN [33] RPN Caffe 0.2 s Python/ MATLAB Mask R-CNN [34] RPN TensorFlow/ Keras 0.35–2 s Python SPP-net [35] Edge Boxes Caffe 0.4 s MATLAB R-FCN [36] RPN Caffe 0.1 s MATLAB FPN [37] RPN TensorFlow 0.2 s Python SSD [38] - Caffe 0.15 s C++ YOLOv3 [39] - Darknet 0.05 s C
You only look once v3 (YOLOv3) is an entirely different method of object detection where it passes the whole image only once, the same as the single-shot multi-box detector (SSD). In this process, the image is divided into a grid of cells which depend on the size of the input image. Each cell is responsible for predicting the number of boxes in the image. Then, the confidence of prediction is performed for each of the boxes. Finally, boxes of lower values are eliminated using non-maximum suppression technique. The SSD is a simple, faster and comparably more accurate method. It eliminates both proposal generation and resampling stages and encapsulates a single detector which makes it simple for training and testing.
5.5.2. Tensorflow Lite for Android Mobile processing units are linked with a large computational overhead; consequently, it requires a software which is computationally simple, uses minimum load, and reduced execution time. TensorFlow Electronics 2019, 8, 913 14 of 17
Lite is a lightweight framework of TensorFlow for mobile and embedded devices developed by Google for fast mobile inference on small devices. Android 8.1 or API level 27 as well as their higher versions are linked with Android neural network API for hardware acceleration and are supported by TensorFlow Lite. TensorFlow Lite can utilize those pre-trained models on MobileNets to perform several selective tasks such as object detection, face attributes detection, fine-grained classification, and landmark recognition. To use TensorFlow Lite in Android Studio, one has to include the library in build.gradle dependencies as “compile‘org.tensorflow:tensorflow-lite: + ”. It enables user to import the TensorFlow Lite interpreter into the OCC application to detect particular LEDs while in a mobile situation.
6. Demonstration of OCC We demonstrate an OCC system using Android smartphone to illustrate the implementation challenges more visibly. The camera and transmitter configurations are listed in Table3. On the transmitter side, we use a single, circular LED of 15 W power with a surface diameter of 14 cm as the OCC transmitter. Arduino uno was programmed to switch the LED at the transmitting frequency. FSOOK modulation was used and we defined 2 and 4 kHz to transmit bits 1 and 0. Multiple users can be connected to the same LED as shown in Figure 10. The Android application was developed in Android Studio for the API 26 which supports exposure and frame control. To facilitate LED detection and width measurement, we had to control the camera exposure to a very low value and in our case we fixed it to 1/8000 s. Two different types of Android phones were used to receive data from LED. The image view is given with a height of 800 px and width of 600px. We selected a fixed RoI for the detected LED and processed the images using OpenCV320 library.
Table 3. Experimental parameters.
Parameters Values LED 15 W, 14 cm Diameter Modulation frequency 2 & 4 kHz (FSOOK) Smartphone model Google pixel2, Galaxy s7 edge API Level 26 Frame rate, exposure time 20 fixed, 1/8000 sec ImageView 600/800 pixel, 270 degree clockwise rotation
Initially, we detected the width of the strips and measured the frequencies that are allocated for the individual bits 1 and 0. The extracted digital bits were then converted to the received text format that is “OCC Demonstration !!!”. The total time required to extract this information was 45 milliseconds per frame, and we were able to communicate within the maximal range of 7.5 m. However, this system provides low data rate due to the limitation of frame rate and increased processing time for each frame. The demonstration shows that the OCC application is running on two different smartphones. However, it still needs some changes on threshold value due to the non-identical characteristics of image sensors of the different smartphones. The current OCC system does not support the required level of mobility support, which is one of the key requirements. Although the requirements and implementation challenges are described in Section5 in detail, we can summarize the limitations with the possible solution issues as follows.
To mitigate visual flickering in the transmitter, modulation frequencies should be used within the • range of 1~5 kHz. In our demonstration, we used 2 & 4 kHz for FSOOK modulation. Achieving data from multiple LEDs requires fast processing to measure strips width. Readers can • look into our previous work [23] regarding achieving multilateral data reception from multiple fixed receiving windows. Here the challenging issue is achieving mobility support, which can be eliminated by developing both LED detection and data reception in smartphones. We can use TensorFlow Lite for smartphone for this purpose in future which is currently under development by Google. Electronics 2019, 8, 913 15 of 17
The data rate can be increased by adopting several hardware and technical considerations. • We explained that in detail in Section 5.3. We hope in future we can develop smartphones with a higher frame rate and greater image sensor resolution to support the required data rate. Communication range of an OCC system mostly depends on the data transmission frequency, • LED and pixel size. Experimentally, we showed the communication range variation with the change of LED size and frequency.
7. Conclusions Implementation of an OCC system using a smartphone is quite a challenging task due to the existing limitations of the smartphone camera specifications, which limit the overall system capacity and communication distance. The main objective of this paper is to address the real-time implementation challenges of such systems and also carry out comprehensive analysis as part of this work. Furthermore, we compare the existing modulation schemes with respect to their particular advantages and limitations. We also demonstrate an Android camera-based application and a transmitter system using FSOOK to analyze the requirements and difficulties in implementing an OCC system for smartphone cameras. This system had a maximal communication range of 7.5 m. Specifically, we analyzed the major challenges such as flickering, multilateral communication, communication distance, data rate, and user mobility support. Moreover, we provided some possible solutions to increase system performance.
Author Contributions: All authors have significant contribution in this paper. Specifically, Conceptualization and methodology, M.S.; software, M.K.H. and M.Z.C.; validation, M.S., M.K.H. and M.Z.C.; formal analysis, M.Z.C.; investigation, M.Z.C., Y.M.J.; resources, M.S.; data curation, M.S., M.K.H.; writing—original draft preparation, M.S.; writing—review and editing, all authors; visualization, M.S., M.Z.C.; supervision, M.Z.C. and Y.M.J.; project administration, Y.M.J.; funding acquisition, Y.M.J. Funding: This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2018-0-01396) supervised by the IITP (Institute for Information & communications Technology Promotion). Conflicts of Interest: The authors declare no conflict of interest.
References
1. Chowdhury, M.Z.; Hossan, M.T.; Islam, A.; Jang, Y.M. A comparative survey of optical wireless technologies: Architectures and applications. IEEE Access 2018, 6, 9819–9840. [CrossRef] 2. Hossan, M.T.; Chowdhury, M.Z.; Islam, A.; Jang, Y.M. A novel indoor mobile localization system based on optical camera communication. Wirel. Commun. Mob. Comput. 2018, 2018, 9353428. [CrossRef] 3. Zeng, L.; O’Brien, D.C.; Le Minh, H.; Faulkner, G.E.; Lee, K.; Jung, D.; Oh, Y.; Won, E.T. High data rate multiple input multiple output (MIMO) optical wireless communications using white LED lighting. IEEE J. Sel. Areas Commun. 2009, 27, 1654–1662. [CrossRef] 4. Hossan, M.; Chowdhury, M.Z.; Hasan, M.; Shahjalal, M.; Nguyen, T.; Le, N.T.; Jang, Y.M. A new vehicle localization scheme based on combined optical camera communication and photogrammetry. Mob. Inf. Syst. 2018, 2018, 8501898. [CrossRef] 5. Hasan, M.K.; Chowdhury, M.Z.; Shahjalal, M.; Jang, Y.M. Fuzzy based network assignment and link-switching analysis in hybrid OCC/LiFi system. Wirel. Commun. Mob. Comput. 2018, 2018, 2870518. [CrossRef] 6. Nakazawa, Y.; Makino, H.; Nishimori, K.; Wakatsuki, D.; Komagata, H. Indoor positioning using a high-speed, fish-eyelens-equipped camera in visible light communication. In Proceedings of the Fourth International Conference on Indoor Positioning Indoor Navigation, Montbeliard-Belfort, France, 24 May 2013. 7. Lee, H.Y.; Lin, H.M.; Wei, Y.L.; Wu, H.I.; Tsai, H.M.; Lin, K.C. RollingLight: Enabling line-of-sight light-to-camera communications. In Proceedings of the Thirteenth Annual International Conference on Mobile Systems (ACM MobiSys), Florence, Italy, 18–22 May 2015. 8. Ghassemlooy, Z.; Luo, P.; Zvanovec, S. Optical camera communications. In Optical Wireless Communications; Uysal, M., Capsoni, C., Ghassemlooy, Z., Boucouvalas, A., Udvary, E., Eds.; Springer: Cham, Switzerland, 2016; pp. 547–568. Electronics 2019, 8, 913 16 of 17
9. Saha, N.; Ifthekhar, M.S.; Le, N.T.; Jang, Y.M. Survey on optical camera communications: Challenges and opportunities. IET Optoelectron. 2015, 9, 172–183. [CrossRef] 10. Berman, S.M.; Greenhouse, D.S.; Bailey, I.L.; Clear, R.D.; Raasch, T.W. Human electro retinogram responses to video displays, fluorescent lighting, and other high frequency sources. Optom. Vis. Sci. 1991, 68, 645–662. [CrossRef][PubMed] 11. Luo, P.; Zhang, M.; Ghassemlooy, Z.; Le Minh, H.; Tsai, H.M.; Tang, X.; Png, L.C.; Han, D. Experimental demonstration of RGB LED-based optical camera communications. IEEE Photonics J. 2015, 7, 1–12. [CrossRef] 12. Luo, P.; Zhang, M.; Ghassemlooy, Z.; Le Minh, H.; Tsai, H.M.; Tang, X.; Han, D. Experimental demonstration of a 1024-QAM optical camera communication system. IEEE Photonics Technol. Lett. 2016, 28, 139–142. [CrossRef] 13. Lin, B.; Ghassemlooy, Z.; Lin, C.; Tang, X.; Li, Y.; Zhang, S. An indoor visible light positioning system based on optical camera communications. IEEE Photonics Technol. Lett. 2017, 29, 579–582. [CrossRef] 14. Rachim, V.P.; Chung, W. Multilevel intensity-modulation for rolling shutter-based optical camera communication. IEEE Photonics Technol. Lett. 2018, 30, 903–906. [CrossRef] 15. Lain, J.; Jhan, F.; Yang, Z. Non-line-of-sight optical camera communication in a heterogeneous reflective background. IEEE Photonics J. 2019, 11, 1–8. [CrossRef] 16. Chen, H.W.; Wen, S.S.; Wang, X.L.; Liang, M.Z.; Li, M.Y.; Li, Q.C.; Liu, Y. Color-shift keying for optical camera communication using a rolling shutter mode. IEEE Photonics J. 2019, 11, 1–8. [CrossRef] 17. Boubezari, R.; Le Minh, H.; Ghassemlooy, Z.; Bouridane, A. Smartphone camera based visible light communication. J. Lightwave Technol. 2016, 34, 4121–4127. [CrossRef] 18. Cahyadi, W.A.; Kim, Y.H.; Chung, Y.H.; Ahn, C. Mobile phone camera-based indoor visible light communications with rotation compensation. IEEE Photonics J. 2016, 8, 1–8. [CrossRef] 19. Karunatilaka, D.; Zafar, F.; Kalavally, V.; Parthiban, R. LED based indoor visible light communications: State of the art. IEEE Commun. Surv. Tutor. 2015, 17, 1649–1678. [CrossRef] 20. Bui, T.; Kiravittaya, S. Demonstration of using camera communication based infrared LED for uplink in indoor visible light communication. In Proceedings of the Sixth International Conference on Communications and Electronics (ICCE), Ha Long, Vietnam, 27–29 July 2016. 21. Luo, P.; Zhang, M.; Ghassemlooy, Z.; Zvanovec, S.; Feng, S.; Zhang, P. Undersampled-based modulation schemes for optical camera communications. IEEE Commun. Mag. 2018, 56, 204–212. [CrossRef] 22. Roberts, R.D. Undersampled frequency shift ON-OFF keying (UFSOOK) for camera communications (CamCom). In Proceedings of the 22nd Wireless and Optical Communication Conference, Chongqing, China, 16–18 May 2013. 23. Shahjalal, M.; Hossan, M.; Hasan, M.; Chowdhury, M.Z.; Le, N.T.; Jang, Y.M. An implementation approach and performance analysis of image sensor based multilateral indoor localization and navigation system. Wirel. Commun. Mob. Comput. 2018, 2018, 7680780. [CrossRef] 24. Luo, P.; Ghassemlooy, Z.; Le Minh, H.; Tang, X.; Tsai, H. Undersampled phase shift ON-OFF keying for camera communication. In Proceedings of the Sixth International Conference on Wireless Communications and Signal Processing (WCSP), Hefei, China, 23–25 October 2014. 25. Kim, B.W.; Jung, S. Novel flicker-free optical camera communications based on compressed sensing. IEEE Commun. Lett. 2016, 20, 1104–1107. [CrossRef] 26. Shangguan, L.; Zhou, Z.; Yang, Z.; Liu, K.; Li, Z.; Zhao, X.; Liu, Y. Towards accurate object localization with smartphones. IEEE Trans. Parallel Distrib. Syst. 2013, 25, 2731–2742. [CrossRef] 27. Tian, P.; Huang, W.; Xu, Z. Design and experimental demonstration of a real-time 95kbps optical camera communication system. In Proceedings of the 10th International Symposium on Communication Systems, Networks and Digital Signal Processing (CSNDSP), Prague, Czech Republic, 20–22 July 2016. 28. Luo, P.; Jiang, T.; Haigh, P.A.; Ghassemlooy, Z.; Zvanovec, S. Undersampled pulse width modulation for optical camera communications. In Proceedings of the IEEE International Conference on Communications Workshops (ICC Workshops), Kansas City, MO, USA, 20–24 May 2018. 29. Chen, S.; Chow, C. Color-shift keying and code-division multiple-access transmission for RGB-LED visible light communications using mobile phone camera. IEEE Photonics J. 2014, 6, 1–6. [CrossRef] 30. Li, Y.; Ghassemlooy, Z.; Tang, X.; Lin, B.; Zhang, Y. A VLC smartphone camera based indoor positioning system. IEEE Photonics Technol. Lett. 2018, 30, 1171–1174. [CrossRef] Electronics 2019, 8, 913 17 of 17
31. Yang, Y.; Hao, J.; Luo, J. CeilingTalk: Lightweight indoor broadcast through LED-camera communication. IEEE Trans. Mob. Comput. 2017, 16, 3308–3319. [CrossRef] 32. Jadhav, S.P.; Tomy, S.; Jayswal, S.S.; Dhaware, H.D.; Vijapure, A.R. Object detection in android smartphone for visually impaired users. Int. J. Adv. Res. Comput. Commun. Eng. 2016, 5, 332–334. 33. Ren, S.; He, K.; Girshick, R.; Sun, J. Faster RCNN: Towards real-time object detection with region proposal networks. In Proceedings of the Twenty-eighth International Conference on Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2015. 34. He, K.; Gkioxari, G.; Dollar, P.; Girshick, R. Mask R-CNN. arXiv 2017, arXiv:1703.06870. 35. He, K.; Zhang, X.; Ren, S.; Sun, J. Spatial pyramid pooling in deep convolutional networks for visual recognition. In Proceeding of the Thirteenth European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014. 36. Dai, J.; Li, Y.; He, K.; Sun, J. R-FCN: Object Detection via Region-Based Fully Convolutional Networks. arXiv 2016, arXiv:1605.06409. 37. Lin, T.-Y.; Dollar, P.; Girshick, R.; He, K.; Hariharan, B.; Belongie, S. Feature pyramid networks for object detection. In Proceedings of the Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017. 38. Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S. SSD: Single Shot Multibox Detector. arXiv 2015, arXiv:1512.02325. 39. Redmon, J.; Farhadi, A. Yolov3: An Incremental Improvement. arXiv 2018, arXiv:1804.02767.
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).