electronics

Article Data Augmentation Schemes for Deep Learning in an Indoor Positioning Application

Rashmi Sharan Sinha 1, Sang-Moon Lee 2, Minjoong Rim 1 and Seung-Hoon Hwang 1,*

1 Division of Electronics and Electrical Engineering, Dongguk University-Seoul, Seoul 04620, Korea; [email protected] (R.S.S.); [email protected] (M.R.) 2 JMP Systems Co., Ltd, Gyeonggi-do 12930, Korea; [email protected] * Correspondence: [email protected]; Tel.: +82-2-2260-3994



Received: 25 April 2019; Accepted: 13 May 2019; Published: 17 May 2019


Abstract: In this paper, we propose two data augmentation schemes for deep learning architecture that can be used to directly estimate user location in an indoor environment using mobile phone tracking and electronic fingerprints based on reference points and access points. Using a pretrained model, the deep learning approach can significantly reduce data collection time, while the runtime is also significantly reduced. Numerical results indicate that an augmented training database containing seven days’ worth of measurements is sufficient to generate acceptable performance using a pretrained model. Experimental results find that the proposed augmentation schemes can achieve a test accuracy of 89.73% and an average location error that is as low as 2.54 m. Therefore, the proposed schemes demonstrate the feasibility of data augmentation using a deep neural network (DNN)-based indoor localization system that lowers the complexity required for use on mobile devices.

Keywords: augmentation; deep learning; CNN; indoor positioning; fingerprint

1. Introduction Identifying the location of a mobile user is an important challenge in pervasive computing because their location provides a lot of information about the user with which adaptive computer systems can be created. The challenge of accurately estimating position both indoors and outdoors has thus received significant attention within both industry and academia. In particular, the successful application of Global Positioning System (GPS) has enabled travelers to move around the world more freely. However, GPS is sensitive to occlusion and does not work in indoor environments. A large number of methods has been proposed to overcome this limitation, including estimating indoor location using mobile devices such as smartphones. Measuring the intensity of a received signal makes interior localization using wireless signals such as Wi-Fi possible. Measurement-based wireless positioning systems infer position based on the time of arrival (TOA) or time difference of arrival (TDOA) of the signals. However, general wireless signal receivers are not capable of measuring round-trip times or angles. Additional devices are thus required, which makes this type of system impractical for many applications. An alternative option is the use of fingerprint-based approaches, which do not need any special devices and are thus more feasible. The fingerprint method proposed in the present study consists of two stages (Figure1). In the o ffline stage, the received signal strength indications (RSSIs) from all access points (APs) are collected from known positions, referred to as reference points (RPs), to build a fingerprint database for the environment. Therefore, each RP has a fingerprint characterized by its position and the captured RSSIs from all APs at that location. At the positioning stage, the currently captured RSSIs are matched with those of the RPs and position is determined using the positions of several of the best-fitting RPs. However, a major issue for accurate fingerprint-based

Electronics 2019, 8, 554; doi:10.3390/electronics8050554 www.mdpi.com/journal/electronics Electronics 2019, 8, 554 2 of 19

Electronics 2019, 7, x FOR PEER REVIEW 2 of 20 localization is the variation in RSSIs due to the fluctuating nature of wireless signals, caused by multipathnature of fading wireless and signals, attenuation caused by by static multipath or dynamic fading objectsand attenuation such as walls by static or moving or dynamic people. objects such as walls or moving people.

FigureFigure 1. 1.The The fingerprint fingerprint method method (For (For offline offl inedeep deep neur neuralal network network (DNN) (DNN) training; training; a four-layer a four-layer DNN- DNN-basedbased localizer localizer is trained is trained to extract to extract reliable reliable features features from from massive massive noisy noisy received received signal signal strength strength indicationindication (RSSI) (RSSI) samples samples from from a pre-built a pre-built fingerprint fingerprint data set. data For onlineset. For positioning, online positioning, the preprocessed the RSSIpreprocessed readings are RSSI fed readings into the are localizer fed into to the estimate localizer the to final estimate position). the final position).

ItIt is is also also necessary necessary toto collect more more RPs RPs to toallow allow more more accurate accurate positioning, positioning, especially especially when whenthe thetarget target environment environment has has a alarge large area, area, which which lead leadss to toan an extremely extremely large large fingerprint fingerprint database. database. Consequently,Consequently, the the challenge challenge in in wireless wireless positioning positioning is is how how to to extract extract reliable reliable features features and and optimize optimize the mappingthe mapping function function using ausing large a collectionlarge collection of RPs of with RPs widely with widely fluctuating fluctuating RSSI signals. RSSI signals. The methods The representmethods a formrepresent of shallow a form learning of shallow architecture learning that hasarchitecture limited modeling that has and limited representational modeling powerand whenrepresentational dealing with power large and when noisy dealing volumes with of large data. and To extractnoisy volumes complex of structures data. To andextract build complex accurate structures and build accurate internal representations from rich data sources, human information internal representations from rich data sources, human information processing mechanisms (e.g., vision processing mechanisms (e.g., vision and speech) suggest the need for deep learning architecture and speech) suggest the need for deep learning architecture consisting of multiple layers of nonlinear consisting of multiple layers of nonlinear processing [1]. Deep learning simulates the hierarchical processing [1]. Deep learning simulates the hierarchical structure of the human brain, processing data structure of the human brain, processing data from a lower to a higher level, gradually producing from a lower to a higher level, gradually producing more semantic concepts. Deep neural networks more semantic concepts. Deep neural networks (DNNs) have been employed to address these types (DNNs) have been employed to address these types of issue with notable success, outperforming of issue with notable success, outperforming state-of-the-art techniques in certain areas such as vision state-of-the-art[2], audio [3], and techniques robotics in[4,5]. certain However, areas the such use as of vision DNNs [in2], Wi-Fi audio localization [3], and robotics has largely [4,5 ].remained However, theuninvestigated. use of DNNs With in Wi-Fi this in localization mind, we haspresent largely a novel remained Wi-Fi positioning uninvestigated. method With based this on in deep mind, welearning. present aIn novel this Wi-Fipaper, positioning we propose method data augmen based ontation deep schemes learning. to Inestimate this paper, user we position propose and data augmentationdemonstrate schemeshow our tomethods estimate can user be employed position and in convolutional demonstrate neural how our network methods (CNN) can besettings. employed in convolutionalA deep learning neural structure network is (CNN) built to settings.extract features from widely fluctuating massive Wi-Fi data, oneA which deep learning automatically structure performs is built probabilistic to extract features position from estimation. widely fluctuating In addition, massive a CNN-based Wi-Fi data, onelocalizer which automaticallyis introduced performs to formulate probabilistic the relation positionship estimation. between adjacent In addition, position a CNN-based states and localizer to is introducedintuitively reduce to formulate variation the in relationship the estimates. between In this adjacent work, a position Wi-Fi-based states indoor and to RSSI intuitively fingerprint reduce variationpositioning in the system estimates. is proposed In this work,and implement a Wi-Fi-baseded using indoor data RSSI augmentation, fingerprint which positioning is a common system is proposedtechnique and that implemented has been proven using to data benefit augmentation, the training whichof machine is a commonlearning models technique in general that has and been provendeep toarchitecture benefit the in training particular. of machine It either speeds learning up models convergence in general or acts and as deepa regularizer, architecture thus in avoiding particular. It eitheroverfitting speeds and upincreasing convergence generalization or acts aspower a regularizer, [6,7]. Data augmentation thus avoiding typically overfitting involves and applying increasing generalizationa set of transformations power [6,7]. to Data either augmentation the data spac typicallye or the involvesfeature space, applying or both. a set The of transformations most common to eitheraugmentations the data space are orperformed the feature in space, the ordata both. sp Theace, most generating common new augmentations samples by are applying performed transformations to the already existing data. There are many transformations that can be applied, in the data space, generating new samples by applying transformations to the already existing data. such as translation, rotation, warping, scaling, color space shifts, and cropping. The goal of these There are many transformations that can be applied, such as translation, rotation, warping, scaling, transformations is to generate more samples to create a larger data set, to prevent overfitting, to color space shifts, and cropping. The goal of these transformations is to generate more samples to create presumably regularize the model, to balance the classes within the database, and to even synthetically a larger data set, to prevent overfitting, to presumably regularize the model, to balance the classes produce new samples that are more representative for the use case or the task at hand. Augmentation withinis especially the database, useful for and small to even data syntheticallysets and has been produce proven new successful samples in thatmany are previous morerepresentative reports. For

Electronics 2019, 8, 554 3 of 19 for the use case or the task at hand. Augmentation is especially useful for small data sets and has been proven successful in many previous reports. For instance, in Shan et al. [8], a data set of 1500 portrait images was augmented by applying four new scales (0.6, 0.8, 1.2, and 1.5), four new rotations ( 45, 22, 22, and 45), and four gamma variations (0.5, 0.8, 1.2, and 1.5) to generate a new data set of − − 19,000 training images. This process increased the accuracy of their system for portrait segmentation from an Intersection over Union (IoU) score of 73.09 to one of 94.20 when the augmented data set was included for fine-tuning. Although this augmentation example was implemented for a small data set, the synthesizing process with rotation increases the complexity of the algorithms. In our augmentation technique, a new data set is generated using AP number variation and uniform random numbers. In terms of complexity, our augmentation technique is thus more convenient to implement. The remainder of the paper is structured as follows. First, Section2 describes the augmentation schemes to conduct the experiment. The augmented database is input into the classifier for training in Section3. In Section4, the simulation results for the prediction of position using the CNN classifier is compared with results using the unaugmented data set. Finally, the positioning success rate of the proposed system is verified in a real-time indoor positioning experiment.

2. Data Augmentation Schemes Data augmentation is a common method in deep learning used to reduce the effect of overfitting. The idea is to expand an existing data set using only the available data so that the learning algorithm can more effectively extract those features essential to the task. To train deep learning models, typically big data sets are required, usually from manual data collection or from already existing databases. However, in some cases only a limited data set is available. Therefore, to expand the size of the data set, data augmentation can be employed. The complex indoor environment and APs may cause problems [9,10] because of the limited coverage of Wi-Fi APs and faulty RSSI measurements. The purpose of data augmentation in this case is to detect and remove faulty measurement data or to remove invalid data, thus improving the accuracy and efficiency of the entire positioning system by creating a database representation that is more suitable for downstream deep learning classifiers. Data augmentation adds value to base data by adding information derived from internal and external sources within the database. It can also reduce the manual intervention required to develop meaningful information and gain insight from business data, as well as significantly enhancing data quality. In this way, we can produce multiple copies of available data with slight variations. Some common techniques used in data augmentation include extrapolation [11], in which the relevant fields are updated or assigned values based on heuristics; tagging [12,13], in which common records are tagged to a group, making it easier for the group to be understood and differentiated; aggregation [14], in which values are estimated for relevant fields if needed using mathematical averages and means; and probability [15], in which values are populated based on the probability of events based on heuristics and analytical statistics. The data set collected during the measurement is in text format. However, the deep learning code is designed for a comma-separated values (CSV) input file, so the text files are converted into a CSV file for training and testing. The training and testing CSV files contain 257 columns and 74 RPs in the 257th column as labels (Figure2). The data conversion of the text dataset from the experiment is conducted using Python. Folders containing text files are used as input to file converter code designed in Python. The APs visible from one RP each have a unique media access control (MAC) address. After defining the number of visible APs (N), the individual MAC addresses and their RSSI values are extracted. Each MAC address has an individual column. If the current MAC address is already present in the header list, the corresponding RSSI is placed below that MAC address; if not, a new column is allocated to this MAC address. For the other MAC addresses that are not present in the current text file for the current row, the RSSI values are replaced with zero. The process continues until all of the text files from all of the folders are processed. If the total number of columns in final CSV file is lower than 256, then the remaining columns are filled with zeros. If the total number of columns is greater than Electronics 2019, 8, 554 4 of 19

256,Electronics then the 2019 extra, 7, x FOR columns PEER REVIEW are deleted on the basis of minimum RSSI entries for an individual4 of MAC 20 address. If the total number of columns in the final CSV file is equal to 256, then the reference data set then the reference data set is complete. Figure 2 shows the CSV format for input file where NN is complete. Figure2 shows the CSV format for input file where NN represent the RSSI values. represent the RSSI values.

Figure 2. The comma-separated values (CSV) file format used as the input file for data augmentation. Figure 2. The comma-separated values (CSV) file format used as the input file for data augmentation. 2.1. Scheme 1: Augmentation with the Total Number of APs 2.1. Scheme 1: Augmentation with the Total Number of APs In a fingerprinting algorithm, the fingerprint of each training point is a sequence of an equal In a fingerprinting algorithm, the fingerprint of each training point is a sequence of an equal number of measurements from different APs. Therefore for M measurements per RP with A APs, number of measurements from different APs. Therefore for M measurements per RP with A APs, the the fingerprint at each RP has the dimensions A M. The aim of the algorithm is to classify a test fingerprint at each RP has the dimensions A×M. The× aim of the algorithm is to classify a test point as point as one of those partitions using CNN classifiers. However, if the problem contains only a few one of those partitions using CNN classifiers. However, if the problem contains only a few observations for a given class, the CNN will not be able to gather enough information for a particular observations for a given class, the CNN will not be able to gather enough information for a particular location.location. Therefore, Therefore, multiple multiple observations observations areare requiredrequired for for each each class. class. To To increase increase the the training training data data setset to anto an appropriate appropriate size, size, augmentation augmentation isis employed.employed. In the the first first augmentation augmentation scheme, scheme, the the data data is is augmentedaugmented based based on on the the number number of of APs APs inin eacheach reading.reading. In this this augmentation augmentation scheme, scheme, the the number number of APsof APs visible visible at at one one RP RP is is used used to to augment augment thethe data.data. A global declaration declaration for for the the total total number number of of MACMAC addresses, addresses, i.e., i.e., the the header header and and a a constant constant value,value, is made in in this this scheme, scheme, with with the the constant constant value value usedused in in this this paper paper set set at at 5. 5. This This constant constant isis usedused throughoutthroughout Scheme Scheme 11 toto introduce introduce diversity diversity to to the the augmentedaugmented numbers. numbers. The The diversity diversity of theof datathe data is controlled is controlled by repeating by repeating the same the RSSIssame andRSSIs changing and onlychanging one RSSI only by one subtracting RSSI by subtracting the constant. the The constant total. number The total of number APs with of APs a nonzero with a RSSInonzero is calculated RSSI is forcalculated each RP. These for each nonzero RP. These values nonzero will be values the number will be ofthe repetitions number of for repetitions the current for RSSI.the current The RSSIs RSSI. for eachThe RP RSSIs are collectivelyfor each RP arrangedare collectively in the arranged form of rowsin the containingform of rows 256 containing numbers 256 in thenumbers present in study.the Thepresent zero RSSI study. value The is zero repeated RSSI samevalue asis nonzerorepeated values.same as Most nonzero of the values. RSSIs areMost 0 becauseof the RSSIs a maximum are 0 of 26because APs isa visiblemaximum from of one26 APs RP. Therefore,is visible from approximately one RP. Therefore, 230 RSSIs approximately are 0. In the 230 example RSSIs are shown 0. In in Figurethe example3, a reading shown containing in Figure nine 3, a APsreading can becontaining used to producenine APs ninecan be copies used to to obtain produce 10 nine readings copies at to the referenceobtain 10 point readings under at consideration. the reference point For every under copy, consideration. 5 is subtracted For every from copy, the RSSI 5 is valuesubtracted of one from AP at the RSSI value of one AP at a time. In this way, 10 copies of the original reading can be made with a time. In this way, 10 copies of the original reading can be made with only a single difference in one only a single difference in one RSSI value for each. Scheme 1 focuses on less detailed data and keeps RSSI value for each. Scheme1 focuses on less detailed data and keeps the augmentation simple with the augmentation simple with respect to the RSSIs. From a small input data size of 3 to 7 kilobytes, respect to the RSSIs. From a small input data size of 3 to 7 kilobytes, this data augmentation technique this data augmentation technique increases the size to ~30–50 megabytes. The pseudocode for Scheme increases the size to ~30–50 megabytes. The pseudocode for Scheme1 outlines the technique in more 1 outlines the technique in more detail. A detailed pseudocode for this augmentation scheme is detail.Electronics A detailed 2019, 7, x FOR pseudocode PEER REVIEW for this augmentation scheme is presented in Scheme1. 6 of 20 presented in Scheme 1. 1. Define Header 2. Define Constant number ‘C’ 3. for input csv file 4. for total number of reference points (RP) 5. for each RSSI value at reference point ‘RP’ 6. Calculate total number of RSSI ≠ 0 ‘N’ on that RP 7. Repeat each RSSI == N 8. Subtract C from each RSSI for (N − 1) 9. end for 10. end for 11. end for

SchemeScheme 1. 1.1. Pseudocode PseudocodePseudocode for for for data data data augmentation augmentation augmentation with with with total total number number of AP. of of AP. AP.

2.2. Scheme 2: Augmentation with the Mean and Uniform Random Numbers This data augmentation technique uses mean value and uniform random numbers to add information into the reference data set. The total number of RSSIs in the reference data set is increased in a fashion such that overall RSSI remains in range of minimum and mean with original RSSI. To identify the potential range for the uniform random number generator, the calculation of the mean RSSI value for each RP is conducted. The global repetition number N is declared and the input data set established in the form of CSV files. For each RP, the mean is calculated for the visible APs, excluding those with RSSI = 0. After calculating the mean, the range is determined for the generation of uniform random numbers. This range is calculated by subtracting the current RSSI value from the mean. The total number of random numbers will be equal to N. For example, let the global N be 60, the current RSSI element be 33, and the RP mean be 45. In this case, the potential range for the uniform random numbers will be 12 (= 45 − 33), which means the uniform random number will generate any of the 13 numbers (including 33) within the 33–45 range 60 times. If the range is equal to zero, the current RSSI will be generated 60 times. In Figure 4, there are only nine visible APs at one RP. Figure 4a shows the input data for augmentation, while Figure 4b shows the output after augmentation. The RSSI values 53 and 28 in Figure 4a have zero difference from the mean, thus there is no change in their augmented RSSI values shown in Figure 4b. Each RSSI element forms an image of 16 x 16 pixels. The addition of a random number to the augmented data set increases the information for each image by keeping the random number within the range set by the mean and the current RSSI. Compared to Scheme 1, the information for every image in Scheme 2 increases by the constant number N (N = 60 in our paper). Therefore, the information in each image for machine learning is equal. However, in Scheme 1, the size of the augmented data set is equal to the number of visible APs, which is not constant. Hence, the information available for machine learning is limited by the number of visible APs. The input file size for Scheme 2 is 3 to 7 kilobytes. The output augmented data size is approximately 300 to 700 megabytes. This larger data set helps the CNN classifier to extract more information for each location. The mean of the data set for the individual APs in each column is calculated with respect to individual RPs. Therefore, there are 74 mean values for each AP. The augmentation further generates N uniform random numbers to achieve the desired augmentation. The pseudocode explains the coding details for Scheme 2.

Electronics 2019, 8, 554 5 of 19 Electronics 2019, 7, x FOR PEER REVIEW 5 of 20

FigureFigure 3.3. Scheme1 1 augmentation: augmentation: ((a) RSSI RSSI values values present present at one one refere referencence point with nine visible APs. ((bb)) TheThe augmentedaugmented datadata setset withwith ninenine copiescopies andand changeschanges inin onlyonly oneone RSSIRSSI valuevalue byby 5.5. TheThe solidsolid boxesboxes representrepresent RSSIsRSSIs withoutwithout anyany change,change, whilewhile thethe dotteddotted redred boxbox representrepresent RSSIs RSSIs with with a a di differencefference of of 5. 5.

Electronics 2019, 8, 554 6 of 19

2.2. Scheme 2: Augmentation with the Mean and Uniform Random Numbers This data augmentation technique uses mean value and uniform random numbers to add information into the reference data set. The total number of RSSIs in the reference data set is increased in a fashion such that overall RSSI remains in range of minimum and mean with original RSSI. To identify the potential range for the uniform random number generator, the calculation of the mean RSSI value for each RP is conducted. The global repetition number N is declared and the input data set established in the form of CSV files. For each RP, the mean is calculated for the visible APs, excluding those with RSSI = 0. After calculating the mean, the range is determined for the generation of uniform random numbers. This range is calculated by subtracting the current RSSI value from the mean. The total number of random numbers will be equal to N. For example, let the global N be 60, the current RSSI element be 33, and the RP mean be 45. In this case, the potential range for the uniform random numbers will be 12 (= 45 33), which means the uniform random number will generate any of the 13 − numbers (including 33) within the 33–45 range 60 times. If the range is equal to zero, the current RSSI will be generated 60 times. In Figure4, there are only nine visible APs at one RP. Figure4a shows the input data for augmentation, while Figure4b shows the output after augmentation. The RSSI values 53 and 28 in Figure4a have zero di fference from the mean, thus there is no change in their augmented RSSI values shown in Figure4b. Each RSSI element forms an image of 16 x 16 pixels. The addition of a random number to the augmented data set increases the information for each image by keeping the random number within the range set by the mean and the current RSSI. Compared to Scheme1, the information for every image in Scheme2 increases by the constant number N (N = 60 in our paper). Therefore, the information in each image for machine learning is equal. However, in Scheme1, the size of the augmented data set is equal to the number of visible APs, which is not constant. Hence, the information available for machine learning is limited by the number of visible APs. The input file size for Scheme2 is 3 to 7 kilobytes. The output augmented data size is approximately 300 to 700 megabytes. This larger data set helps the CNN classifier to extract more information for each location. The mean of the data set for the individual APs in each column is calculated with respect to individual RPs. Therefore, there are 74 mean values for each AP. The augmentation further generates N uniform random numbers to achieve the desired augmentation. The pseudocode explains the coding details for Scheme2. Electronics 2019, 8, 554 7 of 19 Electronics 2019, 7, x FOR PEER REVIEW 7 of 20

Figure 4. Scheme2 augmentation. ( a) RSSI data set without augmentation. (b) For N = 60, there are aFigure total of 4. 60 Scheme augmented 2 augmentation. RSSI data sets (a) withRSSI meandata set and without uniform augmentation. random numbers. (b) For The N solid= 60, there black are boxes a representtotal of 60 the augmented original RSSI RSSI valuedata sets and with the dottedmean and red uniform boxes represent random augmentednumbers. The RSSI solid values. black boxes represent the original RSSI value and the dotted red boxes represent augmented RSSI values.

Electronics 2019, 8, 554 8 of 19 Electronics 2019, 7, x FOR PEER REVIEW 8 of 20

1. Define ‘N’ 2. Define Header 3. for input csv file 4. for total number of reference points (RP) 5. for individual AP address points RP 6. if current RSSI == 0 7. Repeat 0 for N times 8. for current RSSI ≠ 0, do summation ‘S’ 9. for each S calculate mean ‘M’ 10. if |M – RSSI|≠ 0 11. Calculate range ‘R’ for difference between |M – RSSI| 12. Generate N random numbers for corresponding R 13. else if |M – RSSI|== Mean 14. Write current RSSI for N times 15. Repeat for all the RSSI in current RP 16. else |M – RSSI|== 0 17. Write current RSSI for N times 18. Repeat for all the RSSI in current RP 19. end else 20. end else if 21. end else 22. end for 23. end for 24. end if 25. end for 26. end for 27. end for

SchemeScheme 2. 2.PseudocodePseudocode for for data data augmentation augmentation with with mean mean and and uniform uniform random random numbers. numbers. 3. System Model and Experiment Setup 3. System Model and Experiment Setup 3.1. System Model 3.1. System Model In our CNN model, a 5-layer network is designed to predict 74 classes. The input image is generatedIn our CNN from themodel, RSSI a values 5-layer received network during is designed the experiment to predict by 74 the classes. 74 RPs. The At eachinput RP, image the RSSI is generatedvalue is recorded from the for RSSI 256 values APs, though received only during a small the subset experiment of these by APs the are 74 visible RPs. At at each RP.RP, Thesethe RSSI RSSI valuevalues is recorded from different for 256 APs APs, create though a 16 xonly 16 image. a small As subset shown of in these the exampleAPs are invisible Figure at5 ,each there RP. are These a total RSSIof nine values visible from APs different out of APs 256 withcreate RSSI a 16 values x 16 imag betweene. As 25shown to 70, in with the example the other in APs Figure having 5, there a value are of a 0.total The of RSSIs nine visible from di APsfferent out APs of 256 are convertedwith RSSI intovalues a grayscale between image.25 to 70, The with image the hasother di APsfferent having levels a of valuebrightness of 0. The depending RSSIs from on different the recorded APs RSSIare conver values,ted with into higher a grayscale RSSI image. values beingThe image brighter. has different As shown levelsin Figure of brightness5a, the highest depending RSSI valueon the isrecorded 70, which RSSI produces values, thewith brightest higher RSSI spot values in the grayscalebeing brighter. image Asshown shown in in Figure Figure5b, 5a, while the highest the lowest RSSI value value isis 25,70, whichwhich isproduces represented the brightest by the darkest spot in nonblackthe grayscale spot. imageRSSI shown values in of Figure 0 produce 5b, while no brightness, the lowest thus value the is remaining 25, which is 247 represented spots are black. by theSimilarly, darkest nonblack the input spot.RSSI RSSI files values for the of other 0 produce 73 RPs no will brightness, produce di thusfferent the images remaining for input 247 spots into the are deep black. learning Similarly, network. the inputThis RSSI leads files to a for total the of other 9602 input73 RPs images will produce for the 74different RPs without images augmentation. for input into Withthe deep augmentation, learning network.there are This total leads 122,760 to inputa total training of 9602 images input usingimages Scheme for the1 augmentation 74 RPs without and 585,722augmentation. input training With augmentation,images using Schemethere are2 augmentation. total 122,760 Theinput total trai numberning images of test using images Scheme for the lab1 augmentation simulations is and 1479. 585,722 input training images using Scheme 2 augmentation. The total number of test images for the lab simulations is 1479.

Electronics 2019, 8, 554 9 of 19 Electronics 2019, 7, x FOR PEER REVIEW 9 of 20

(a) (b)

Figure 5.5. Deep learning input filefile conversion from a CSVCSV filefile toto anan image.image. (a) InputInput CSVCSV readingsreadings ofof thethe ninenine visiblevisible RSSIsRSSIs fromfrom aa totaltotal ofof 256256 APs.APs. ((bb)) ConvertedConverted grayscalegrayscale imageimage withwith ninenine brightbright spotsspots representingrepresenting APsAPs visiblevisible atat thethe RP.RP.

A deep learning learning architecture architecture is is useful useful for for audio, audio, video video and and image image data data types. types. Especially, Especially, feed- feed-forwardforward neural neural networks networks (FNNs) (FNNs) and andartificial artificial neural neural networks networks (ANNs) (ANNs) spur spur the theidea idea for for CNNs CNNs in inwhich which the the output output from from one one layer layer is isthe the input input for for the the subsequent subsequent layer. layer. In InFNNs, FNNs, each each neuron neuron is isinterconnected interconnected and and it has it has weight weight associated associated to each to each neuron neuron in the in thefollowing following layer. layer. As a As result, a result, the thenumber number of required of required connect connectionsions in inthis this type type of of network network rapidly rapidly grows grows as as the the input input size size increases to an unmanageable level.level. For example, inin [[16]16] if the input to an FNN network is through a VGA camera (640(640 × 480 × 33 pixels), pixels), there there would would be be aa weigh weigh difference difference of of 921,600 921,600 between between an an input input neuron neuron and and a × × asingle single hidden hidden neuron. neuron. In In addition addition first first hidden hidden laye layerr needs needs to to comprise comprise of of thousands of neurons to manage the dimensionality of of the the input, input, leading leading to to a amodel model with with billions billions of of weight weight and and all all needed needed to tobe belearned. learned. It Itis isvery very difficult difficult to to work work with with this this many many weights weights since since it increases the computationcomputation complexity as wellwell asas thethe memorymemory requirements.requirements. A CNN is a variantvariant of thethe standardstandard multilayermultilayer perceptronperceptron (MLP).(MLP). AA significantsignificant advantageadvantage of this method compared with conventional approaches, especially for pattern recognition, is its ability to reduce thethe dimensionsdimensions of thethe data,data, extractextract featuresfeatures sequentially, andand classifyclassify thethe imageimage atat thethe outputoutput ofof CNNCNN networknetwork [[17].17]. Our CNN network comprises fivefive layers,layers, withwith thethe firstfirst layerlayer havinghaving inputinput grayscalegrayscale imagesimages ofof sizesize 1616 × 1616 × 1, R Rectifiedectified Linear Unit × × (ReLU),(ReLU), andand dropout.dropout. Due to the small size of the inputinput data set, max pooling is not used in the firstfirst layer.layer. The second layer consists of a 1616 × 16 convolutionconvolution with ReLU and then an 8 x 8 maxmax poolingpooling × layerlayer withwith aa total total of of 18,496 18,496 parameters. parameters. This This produces produces output output for for the the third third 8 x 88 convolutionx 8 convolution layer layer with ReLUwith ReLU and a and 4 a4 max4 × 4 poolingmax pooling layer. layer. This outputThis output is fed is directly fed directly to a fully to a connectedfully connected (FC) layer(FC) layer with × 3072with nodes,3072 nodes, which which leads toleads the nextto the hidden next FChidden layer FC with layer 1024 with nodes. 1024 Finally, nodes. the Finally, output the iscalculated output is usingcalculated a softmax using layer a softmax with 74 layer nodes, with which 74 nodes, is the totalwhic numberh is the total of RPs number in our set-up.of RPs Thein our inner set-up. width The is 1024,inner andwidth the is first 1024, four and layers the first use afour dropout layers of use 0.5. a Thedropout learning of 0.5. rate The of learning our model rate is 0.001.of our Themodel total is number0.001. The of parameterstotal number in ourof parameters model is 2,266,698. in our model Figure 6is provides 2,266,698. a complete Figure 6 summary provides ofa thecomplete CNN modelsummary used of inthe the CNN present model study. used in the present study.

ElectronicsElectronics 20192019,, 87,, 554x FOR PEER REVIEW 1010 ofof 1920

Figure 6. CNN architecture for the indoor Wi-Fi positioning system proposed in this study. Figure 6. CNN architecture for the indoor Wi-Fi positioning system proposed in this study. 3.2. Experiment Setup 3.2. Experiment Setup RSSI fingerprint data collection and the final experiment were both performed on the 7th floor of theRSSI new fingerprint engineering data building collection at Dongguk and the final University, experiment Seoul, were South both Korea. performed As shown on the in Figure7th floor7, theof the 52 new32 mengineeringtarget area building is divided at into Dongguk 74 target Universi RPs spacedty, Seoul, at intervals South ofKorea. 2 m. As The shown positioning in Figure server 7, × usedthe 52 in × this 32 m study target is aarea Dell is Alienware divided into Model 74 target P31E (Alienware,RPs spaced hardwareat intervals subsidiary of 2 m. The of Dell, positioning Miami, FL,server USA), used while in this a Samsung study is SHV-E310Ka Dell Alienware smartphone Model(Yongin-si, P31E (Alienware, Gyeonggi-do, hardware Korea) subsidiary is used forof Dell, data collection.Miami, FL, TheUSA), fingerprint while a Samsung database SHV-E310K construction, smartphone classification (Yongin-si, (i.e. position Gyeonggi-do, prediction), Korea) and is online used experimentalfor data collection. setup The aredeveloped fingerprint using database Python. construc The experimentaltion, classification system (i.e. setup, position in which prediction), an Android and deviceonline listensexperimental to the RSSIssetup fromare developed surrounding using APs, Python. is depicted The experimental in Figure8. system setup, in which an AndroidThe datadevice read listens by theto the Android RSSIs from device su isrrounding stored in APs, a bu isff er.depicted If there in isFigure an error 8. in the recorded data,The an errordata read message by the is Android displayed device on the is stored serially in a connected buffer. If there console. is an Otherwise,error in the recorded the RSSI data, data isan stored error message in the bu isff displayeder and, after on athe complete serially connec scan, isted transferred console. Otherwise, by the Android the RSSI console data is connected stored in bythe an buffer interface and, after cable a to complete the server scan, through is transferred a Wi-Fi by AP. the The Android server determinesconsole connected the Android by an interface device’s locationcable to bythe comparing server through the measured a Wi-Fi RSSI AP. valuesThe server with referencedetermines data. the The Android Android device’s device location used in our by experimentcomparing isthe shown measured in Figure RSSI8. It isvalues serially with connected reference to the data. Android The consoleAndroid and device processes used the in RSSIs our fromexperiment surrounding is shown APs in with Figure its CPU8. It is unit. serially The operatingconnected frequency to the Android of the deviceconsole is and 2.412–2.480 processes GHz the forRSSIs the from 802.11bgn surrounding wireless APs standard. with its TheCPU input unit. /Theoutput operating sensitivity frequency is 15–93 of dBm.the device The referenceis 2.412–2.480 file, representedGHz for the by802.11bgn REF in textwireless file format, standard. is saved The input/output as a REF000000XX sensitivity file, whereis 15–93 XX dBm. is the The RP reference number. Thefile, sizerepresented of each file by is REF 1 kilobyte in text (Figure file format,9). is saved as a REF000000XX file, where XX is the RP number.In collecting The size theof each data, file the is wireless 1 kilobyte mobile (Figure phone 9). first measures the RSSIs from the available APs at each RP. The measurements are combined with the MAC addresses of the APs and the RP label as a single fingerprint data point and transmitted to the positioning server. Considering the multipath effects that exist in indoor environments, the reference data (Figure9) are collected five times at each RP. Data is collected 4 times a day and total for 7 days. In total, 10,360 RSSI fingerprint data points are acquired using this process. All of these data points are saved as one CSV file on the positioning server. The data set is then used as reference data set to train the deep learning classifier on the server. The proposed indoor positioning system depends on the installed APs and the observation of their RSSI values using a user’s mobile device. RSSI values are highly vulnerable to interference from walls, human body movement, and the direction in which the user is facing. Therefore, the overall accuracy of the indoor positioning system can be weakened. To overcome this problem, we build our data set using data collected from four orientations, i.e., forward, backward, left, and right, at each individual reference point. The flexibility of the system is higher if the reference data set includes all four orientations. The location of a user can thus be identified without them having to face in one direction.

Electronics 2019, 7, x FOR PEER REVIEW 11 of 20

Electronics 2019, 8, 554 11 of 19 Electronics 2019, 7, x FOR PEER REVIEW 11 of 20

Figure 7. Indoor environment (radio map) for data collection divided into 74 reference points (2 m × Figure 7. Indoor environment (radio map) for data collection divided into 74 reference points 2 mFigure each). 7. Indoor environment (radio map) for data collection divided into 74 reference points (2 m × (2 m 2 m each). ×2 m each).

Electronics 2019Figure, 7, x 8.FORIllustration PEER REVIEW of the proposed fingerprint-based Wi-Fi positioning system. 12 of 20 FigureFigure 8. Illustration8. Illustration of of the the prop proposedosed fingerprint-based fingerprint-based Wi-Fi positioningpositioning system system. .

FigureFigure 9. Reference 9. Reference data data files files for for RP RP 1 1 andand RPRP 2 collected during during the the experiment. experiment.

In collecting the data, the wireless mobile phone first measures the RSSIs from the available APs at each RP. The measurements are combined with the MAC addresses of the APs and the RP label as a single fingerprint data point and transmitted to the positioning server. Considering the multipath effects that exist in indoor environments, the reference data (Figure 9) are collected five times at each RP. Data is collected 4 times a day and total for 7 days. In total, 10,360 RSSI fingerprint data points are acquired using this process. All of these data points are saved as one CSV file on the positioning server. The data set is then used as reference data set to train the deep learning classifier on the server. The proposed indoor positioning system depends on the installed APs and the observation of their RSSI values using a user’s mobile device. RSSI values are highly vulnerable to interference from walls, human body movement, and the direction in which the user is facing. Therefore, the overall accuracy of the indoor positioning system can be weakened. To overcome this problem, we build our data set using data collected from four orientations, i.e., forward, backward, left, and right, at each individual reference point. The flexibility of the system is higher if the reference data set includes all four orientations. The location of a user can thus be identified without them having to face in one direction.

4. Numerical Results

4.1. Simulation Results In order to evaluate the validity of our approach, we create several data sets over a period of four weeks. These data sets are then used to assess which CNN layer is best used to transfer knowledge from classification to indoor positioning and to determine the optimal classification algorithm. We show that a relatively simple classification model fits the data well, producing generalization results over a one-week period of 91% in the lab-based simulations and 89% in real- time testing with Scheme 2. The long-term introduction of new APs and drift in the existing APs need to be trained and learned. To generate the data set, the data is gathered at the 74 RPs over 7 days in four directions. The data set in then divided into four sets (Set 1: 7 days of data; Set 2: 5 days of data; Set 3: 3 days of data; and Set 4: 2 days of data), and each set is divided further into separate cases based on the ratio of reference to trial data. For example, Set 1 (7 days of data) is divided into the three cases 6-1, 5-2, and 4-3. Table 1 summarizes the data sets and cases. The dataset with 7 days of data has the maximum number of input files, thus its overall test accuracy is higher than that of the other sets (Table 2). Set 1/Case 1, which has six days of data set for training and 1 day of data for testing, produced a highest test accuracy of 91%, with a minimum loss of 0.3. Similarly, Set 2/Case 1 with a 4:1 data set reference to testing data ratio has a highest accuracy of 88%, with a loss of 0.2. Set 3 exhibits a similar accuracy to that of Set 2. Set 4 produces the lowest accuracy with 84%. Therefore, it can be concluded that training with only 1 day of data is not sufficient for the CNN classifier. From these lab simulations, it is clear that a large data set is very important for a classifier. The accuracy of our CNN classifier drops with a decrease in the size of the training data set. Thus, with an augmented data set, the accuracy of the complete system can be increased in both lab simulations and in real- time positioning.

Electronics 2019, 8, 554 12 of 19

4. Numerical Results

4.1. Simulation Results In order to evaluate the validity of our approach, we create several data sets over a period of four weeks. These data sets are then used to assess which CNN layer is best used to transfer knowledge from classification to indoor positioning and to determine the optimal classification algorithm. We show that a relatively simple classification model fits the data well, producing generalization results over a one-week period of 91% in the lab-based simulations and 89% in real-time testing with Scheme2. The long-term introduction of new APs and drift in the existing APs need to be trained and learned. To generate the data set, the data is gathered at the 74 RPs over 7 days in four directions. The data set in then divided into four sets (Set 1: 7 days of data; Set 2: 5 days of data; Set 3: 3 days of data; and Set 4: 2 days of data), and each set is divided further into separate cases based on the ratio of reference to trial data. For example, Set 1 (7 days of data) is divided into the three cases 6-1, 5-2, and 4-3. Table1 summarizes the data sets and cases. The dataset with 7 days of data has the maximum number of input files, thus its overall test accuracy is higher than that of the other sets (Table2). Set 1 /Case 1, which has six days of data set for training and 1 day of data for testing, produced a highest test accuracy of 91%, with a minimum loss of 0.3. Similarly, Set 2/Case 1 with a 4:1 data set reference to testing data ratio has a highest accuracy of 88%, with a loss of 0.2. Set 3 exhibits a similar accuracy to that of Set 2. Set 4 produces the lowest accuracy with 84%. Therefore, it can be concluded that training with only 1 day of data is not sufficient for the CNN classifier. From these lab simulations, it is clear that a large data set is very important for a classifier. The accuracy of our CNN classifier drops with a decrease in the size of the training data set. Thus, with an augmented data set, the accuracy of the complete system can be increased in both lab simulations and in real-time positioning.

Table 1. Data sets and cases.

Case 1 Case 2 Case 3 DATA SET Reference-Test Set 1: 7 Days 6-1 5-2 4-3 Set 2: 5 Days 4-1 3-2 - Set 3: 3 Days 2-1 - - Set 4: 2 Days 1-1 - -

Table 2. Lab simulation results for accuracy and loss.

Case 1 Case 2 Case 3 Sets Approaches Loss Accuracy (%) Loss Accuracy (%) Loss Accuracy (%) Scheme 1 1.4 90.46 1.3 89.85 1.7 88.88 Set 1 Scheme 2 1.0 91.32 1.2 90.09 1.1 90.45 Without Augmentation 1.5 89.45 1.5 89.55 1.4 88.83 Scheme 1 1.3 86.73 0.9 87.10 - - Set 2 Scheme 2 1.0 87.34 1.2 88.08 - - Without Augmentation 1.3 86.87 1.3 87.33 - - Scheme 1 1.4 88.65 - - - - Set 3 Scheme 2 1.0 88.89 - - - - Without Augmentation 1.2 87.67 - - - - Scheme 1 0.8 84.11 - - - - Set 4 Scheme 2 1.0 84.79 - - - - Without Augmentation 0.9 54.27 - - - -

Figure 10a,b shows the loss and test accuracy for 6-1 Reference-Test Set 1, Case 1. The loss value for Schemes1 and2 starts at 2.8 and 1.7, which reduced to 0.4 and 0.3 after 200 epochs, respectively, Electronics 2019, 8, 554 13 of 19

The unaugmented data set have initial loss value at 4.7 which comes down to 1.0, after 200 epochs. The loss decreases after 200 epochs which shows robustness of the CNN model however, the accuracy of the model also decreasing which is undesirable for indoor positioning application. The highest accuracy achieved during lab simulations for each method is used as optimum value to generate the metafile [7] for real time testing for each method. Therefore the highest accuracies for Scheme1, Scheme2, and the unaugmented data set are chosen in this work. The highest accuracy for 6-1 Reference-Test Set 1 is 90.46% (after five epochs, loss = 1.4), 91.32% (after three epochs, loss = 1), and 89.45% (after 43 epochs, loss = 1.5) for Scheme1, Scheme2, and unaugmented data, respectively. The accuracy of all three methods fall to 84.30% for Scheme1, 85.50% Scheme2, and 88.43% unaugmented data set after 200 epochs. Figure 10c,d shows the loss and test accuracy for 5-2 Reference-Test Set 1, Case 2. The initial and final loss values after 200 epochs are 2.8, 1.8, and 4.7, and 0.4, 0.3, and 1 for Scheme1, Scheme2, and the unaugmented data set, respectively. The highest lab accuracy achieved for this data set is 89.85% (after 7 epochs, loss = 1.3) for Scheme1, 90.09% (after 2 epochs, loss = 1.2) for Scheme2, and 89.55% (after 65 epochs, loss = 1.5) for unaugmented data set. After 200 epochs the accuracies are 82.91%, 81.50%, and 88.01% for each method, respectively. Figure 10e,f represents the loss and test accuracy for 4-3 Reference-Test Set 1, Case 3. The four days of reference data set is used as training data set for CNN classifier in this case. The initial loss values for Scheme1, Scheme2 and unaugmented data are 3.2, 2.0, and 4.9, while the final loss values after 200 epochs are 0.3, 0.2, and 1.0. The highest accuracies for this case are 88.88% (after seven epochs, loss = 1.7), 90.45% (after three epochs, loss = 1.1) and 88.83% (after 86 epochs, loss = 1.4). After 200 epochs the accuracies are 82.91%, 81.50%, and 88.01% for each method, respectively. Figure 11a,b shows the loss and test accuracy for 4-1 Reference-Test Set 2, Case 1. Total four days of reference data set is used as training data set for CNN classifier. The initial loss value for this case is 3.0, 1.9, and 4.8 for Scheme1, Scheme2, and the unaugmented data set, respectively. The highest accuracies achieved during lab simulation for this case are 86.73% (after six epochs, loss = 1.3) for Scheme1, 87.34% (after three epochs, loss = 1.0) for Scheme2, and 86.87% (after 77 epochs, loss = 1.3) for unaugmented data set. The accuracies after 200 epochs are 77.93%, 74.80%, and 84.63% for each method, respectively. Figure 11c,d shows the loss and test accuracy for 3-2 Reference-Test Set 2, Case 2. The total three days of the reference data set is used as training data set for CNN classifier. The initial loss values for this case are 3.1, 2.0, and 4.9, while the final loss values are 0.2, 0.1, and 0.8 for Scheme1, Scheme2, and unaugmented data, respectively. The highest accuracies achieved in this case are 87.10% (after 15 epochs, loss = 0.9) for Scheme1, 88.08% (after two epochs, loss = 1.2) for Scheme2, and 87.10% (after 75 epochs, loss = 1.3) for unaugmented data. The accuracies after 200 epochs are 80.90%, 78.50%, and 85.57% for each method, respectively. Figure 12a,b shows the loss and test accuracy for 2-1 Reference-Test Set 3, Case 1. In this case only two days of reference data set is used as training data set for CNN classifier. The initial loss values are 3.7, 2.3, 5.1, and final loss values are 0.2, 0.1, and 0.8 for Scheme1, Scheme2, and unaugmented data set. The highest accuracies achieved in this case are 84.11% (after six epochs, loss = 1.4) for Scheme1, 84.79% (after three epochs, loss = 1.0) for Scheme2, and 54.27 (after 109 epochs, loss = 0.2) for unaugmented data set. The accuracies decrease after 200 epochs to 76.07%, 72.23%, and 54.27% for each method, respectively. Figure 13a,b shows the loss and test accuracy for 1-1 Reference-Test Set 4, Case 1. In this case, only one day of the reference data set is used as training data set for CNN classifier. The initial and final loss values for Scheme1, Scheme2 and unaugmented data sets are 4.2, 2.7, 5.2, and 0.1, 0.0, and 0.9 for each method, respectively. The highest accuracies achieved during lab simulation for this case are 84.11% (after 16 epochs, loss = 0.8) for Scheme1, 84.79% (after two epochs, loss = 0.8) for Scheme2, and 54.27% (after 200 epochs, loss = 0.9). The accuracies after 200 epochs for this case are 76.07%, 72.23%, and 54.27% for each method, respectively. Electronics 2019, 7, x FOR PEER REVIEW 14 of 20 loss and test accuracy for 4-1 Reference-Test Set 2, Case 1. Total four days of reference data set is used as training data set for CNN classifier. The initial loss value for this case is 3.0, 1.9, and 4.8 for Scheme 1, Scheme 2, and the unaugmented data set, respectively. The highest accuracies achieved during lab simulation for this case are 86.73% (after six epochs, loss = 1.3) for Scheme 1, 87.34% (after three epochs, loss = 1.0) for Scheme 2, and 86.87% (after 77 epochs, loss = 1.3) for unaugmented data set. The accuracies after 200 epochs are 77.93%, 74.80%, and 84.63% for each method, respectively. Figure 11c,d shows the loss and test accuracy for 3-2 Reference-Test Set 2, Case 2. The total three days of the reference data set is used as training data set for CNN classifier. The initial loss values for this case are 3.1, 2.0, and 4.9, while the final loss values are 0.2, 0.1, and 0.8 for Scheme 1, Scheme 2, and unaugmented data, respectively. The highest accuracies achieved in this case are 87.10% (after 15 epochs, loss = 0.9) for Scheme 1, 88.08% (after two epochs, loss = 1.2) for Scheme 2, and 87.10% (after 75 epochs, loss = 1.3) for unaugmented data. The accuracies after 200 epochs are 80.90%, 78.50%, and 85.57% for each method, respectively. Figure 12a,b shows the loss and test accuracy for 2-1 Reference- Test Set 3, Case 1. In this case only two days of reference data set is used as training data set for CNN classifier. The initial loss values are 3.7, 2.3, 5.1, and final loss values are 0.2, 0.1, and 0.8 for Scheme 1, Scheme 2, and unaugmented data set. The highest accuracies achieved in this case are 84.11% (after six epochs, loss = 1.4) for Scheme 1, 84.79% (after three epochs, loss = 1.0) for Scheme 2, and 54.27 (after 109 epochs, loss = 0.2) for unaugmented data set. The accuracies decrease after 200 epochs to 76.07%, 72.23%, and 54.27% for each method, respectively. Figure 13a,b shows the loss and test accuracy for 1-1 Reference-Test Set 4, Case 1. In this case, only one day of the reference data set is used as training data set for CNN classifier. The initial and final loss values for Scheme 1, Scheme 2 and unaugmented data sets are 4.2, 2.7, 5.2, and 0.1, 0.0, and 0.9 for each method, respectively. The highest accuracies achieved during lab simulation for this case are 84.11% (after 16 epochs, loss = 0.8) for Scheme 1, 84.79% (after two epochs, loss = 0.8) for Scheme 2, and 54.27% (after 200 epochs, loss = 0.9).Electronics The2019 accuracies, 8, 554 after 200 epochs for this case are 76.07%, 72.23%, and 54.27% for each method,14 of 19 respectively.

Loss (6-1) Test Accuracy (6-1) 6 95 90 85 4 80 75 %

Loss 70 2 65 60 55 0 50 1 36 71 106 141 176 1 36 71 106 141 176 Epoch Epoch Electronics 2019, 7, x FOR PEER REVIEW 15 of 20 (a) (b)

Loss (5-2) 95 Test Accuracy (5-2) 6 90 85 80 4 75 % 70 Loss 2 65 60 55 0 1 36 71 106 141 176 50 1 36 71 106 141 176 Epoch Epoch

(c) (d) Loss (4-3) Test Accuracy (4-3) 6 95 90 85 4 80 75 % Loss 70 2 65 60 55 0 50 1 36 71 106 141 176 1 36 71 106 141 176 Epoch Epoch

(e) (f)

Figure 10. Lab simulation results. ( a,,c,e) Set 1 loss for the three cases. ( (b,,d,,f) Set 1 test test accuracy for the three cases.

Loss (4-1) Test Accuracy (4-1)

6 95 90 85 4 80 75 %

Loss 70 2 65 60 55 0 50 1 36 71 106 141 176 1 36 71 106 141 176 Epoch Epoch

(a) (b)

Electronics 2019, 7, x FOR PEER REVIEW 15 of 20

Loss (5-2) 95 Test Accuracy (5-2) 6 90 85 80 4 75 % 70 Loss 2 65 60 55 0 1 36 71 106 141 176 50 1 36 71 106 141 176 Epoch Epoch

(c) (d) Loss (4-3) Test Accuracy (4-3) 6 95 90 85 4 80 75 % Loss 70 2 65 60 55 0 50 1 36 71 106 141 176 1 36 71 106 141 176 Epoch Epoch

(e) (f)

Electronics 2019Figure, 8, 554 10. Lab simulation results. (a,c,e) Set 1 loss for the three cases. (b,d,f) Set 1 test accuracy for the 15 of 19 three cases.

Loss (4-1) Test Accuracy (4-1)

6 95 90 85 4 80 75 %

Loss 70 2 65 60 55 0 50 1 36 71 106 141 176 1 36 71 106 141 176 Epoch Epoch ElectronicsElectronics 2019 2019, 7, ,x 7 FOR, x FOR PEER PEER REVIEW REVIEW 16 of16 20 of 20 (a) (b)

LossLoss (3-2) (3-2) TestTest Accuracy Accuracy (3-2) (3-2) 6 6 95 95 90 90 85 85 4 4 80 80 75 75 Loss Loss % % 70 70 2 2 65 65 60 60 55 55 0 0 11 36 36 71 71 106 106 141 141 176 176 50 50 11 36 36 71 71 106 106 141 141 176 176 EpochEpoch Epoch Epoch (c) (c) (d)(d)

FigureFigure 11.Figure Lab11. 11.Lab simulation Lab simulation simulation results. results. results. (a(a,c,c )()a,c SetSet) Set 22 loss 2 loss for for forthe the the two two two cases. cases. cases. (b , (d(b) ,dSet) Set 2 lab 2 2 lab labsimulation simulation simulation test test test accuracyaccuracyaccuracy for thefor for twothe the two cases. two cases. cases.

LossLoss (2-1) (2-1) TestTest Accuracy Accuracy (2-1) (2-1)

6 6 95 95 90 90 85 85 4 4 80 80 75 75 % % Loss Loss 70 70 2 2 65 65 60 60 55 55 0 0 50 50 11 36 36 71 71 106 106 141 141 176 176 11 36 36 71 71 106 106 141 141 176 176 EpochEpoch EpochEpoch (a) (a) (b)( b)

FigureFigureFigure 12. 12.Lab 12.Lab simulation Lab simulation simulation results. results. results. ((aa)) Set(Seta) Set 3 loss loss 3 loss and and and (b (b) (Set)b Set) Set 3 lab 3 3 lab lab simulation simulationsimulation test test testaccuracy. accuracy. accuracy.

Electronics 2019, 8, 554 16 of 19 Electronics 2019, 7, x FOR PEER REVIEW 17 of 20

Loss (1-1) Test Accuracy (1-1) 6 95 90 85

4 80 75 %

Cost 70

2 65 60 55 50 0 136711014 17 1 36711014 17 6 1 6 6 1 6 Epoch Epoch (a) (b)

FigureFigure 13. 13.Lab Lab simulation simulation results. results. (a(a)) Set Set 4 4 loss loss and and ( b(b)) Set Set 4 4 lab lab simulation simulation test test accuracy. accuracy.

4.2. Experiment Results 4.2. Experiment Results There are three approaches for the proposed CNN positioning system, as mentioned in Section 4.1. There are three approaches for the proposed CNN positioning system, as mentioned in Section The highest accuracy of each scheme is: Scheme1 shows 90.46%, Scheme2 shows 91.32%, and Without 4.1. The highest accuracy of each scheme is: Scheme 1 shows 90.46%, Scheme 2 shows 91.32%, and Aug.Without shows Aug. 89.45%. shows Each89.45%. approach Each approach is tested is in tested an online in an real-time online real-time experiment experiment to verify to the verify actual the performanceactual performance of the proposed of the proposed system. In system. the experiment, In the experiment, a mobile phone a mobile carried phone by a usercarried captures by a user the real-timecaptures RSSIsthe real-time of visible RSSIs APs of and visible sends APs them and with sends the them associated with the MAC associated addresses MAC to theaddresses positioning to the server.positioning The trainedserver. The classifier trained model classifier is loaded model and is lo usedaded toand predict used to the predict user’s the position user’s position based on based the receivedon the received data. In data. our onlineIn our experiment,online experiment, one user one carrying user carrying a mobile a mobile phone movedphone moved from RP from 1 to RP RP 1 74to (i.e.,RP 74 in (i.e., a forward in a forward direction) direction) and thenand then moved moved from from RP RP 74 to74 RPto RP 1 (i.e.,1 (i.e., in in a a backward backward direction). direction). AtAt each each RP, RP, RSSI RSSI measurements measurements were were taken taken and and the the user’s user’s position position predictedpredicted fivefive timestimes inin sequence.sequence. InIn the the simulation, simulation, the the test test data data set set is is constructed constructed from from data data collected collected from from four four directions. directions. AsAs aa result,result, thethe simulationsimulation resultsresults shouldshould bebe comparedcompared withwith thethe averagedaveraged resultsresults forfor bothboth thethe forwardforward andand backwardbackward experiment. experiment. AsAs shown shown in in Table Table3, 3, the the accuracy accuracy achieved achieved during during real-time real-time positioning positioning is 89.73% is 89.73% with with Scheme Scheme2, which2, which was was slightly slightly lower lower than than the the lab lab simulation simulation results results presented presented in in Table Table2 in 2 in terms terms of of the the highest highest accuracyaccuracy (91.32%). (91.32%). SchemeScheme1 1achieves achieves an an accuracy accuracy of of 88.11%, 88.11%, 2.35% 2.35% lower lower than than the the lab lab simulations. simulations. TableTable4 compares4 compares the the average average location location error error for for the th threee three augmentation augmentation schemes. schemes. The The augmentation augmentation schemesschemes were were tested tested for for a a total total of of 6 6 days days to to produce produce 30 30 test test runsruns forfor eacheach RP.RP. It It can can be be seen seen that that the the averageaverage mean mean location location error error for for Scheme Scheme2 is 2 2.54is 2.54 m. m. In In comparison, comparison, the the mean mean location location error error for for Scheme Scheme1 is1 2.70is 2.70 m. m. The The eff effectivenessectiveness of of the the augmentation augmentation schemes schemes is is also also evaluated evaluated in in terms terms of of positioning positioning accuracy,accuracy, which which is is defined defined as as the the cumulative cumulative percentage percentage of of location location error error within within a a specified specified distance distance (Figure(Figure 14 14).). Overall, Overall, the the augmentation augmentation schemes schemes are are more more e effective,ffective, outperforming outperforming the the unaugmented unaugmented datadata over over the the entire entire range range of of the the graph. graph. Schemes Scheme1 and 1 and2 do Scheme not di ff2er do greatly not differ in terms greatly of positioning in terms of accuracy,positioning e.g., accuracy, cases where e.g., the cases error distancewhere the is withinerror 5distance m. Both is schemes within have5 m. probabilities Both schemes of above have 90%probabilities of being withinof above an error90% distanceof being ofwithin 5 m. However,an error distance for cumulative of 5 m. distribution However, functionsfor cumulative over 90%,distribution the positioning functions accuracy over 90%, of Scheme the positioning1 falls behind accuracy Scheme of2 Scheme. Under 90%,1 falls the behind error distanceScheme 2. for Under the augmentation90%, the error schemes distance is for about the 2.5 augmentation m, with Scheme scheme2 ~0.15s is mabout more 2.5 accurate m, with compared Scheme with2 ~0.15 Scheme m more1. Theaccurate gap between compared the with two Scheme increases 1. graduallyThe gap between and eventual the two the increases error distance gradually rises and to eventual nearly 8 m.the Aserror shown distance in Table rises3, to the nearly accuracy 8 m. without As shown augmentation in Table 3, the is 74.68%, accuracy while without the accuracy augmentation with Scheme is 74.68%,1 augmentationwhile the accuracy is 88.11% with and Scheme Scheme 1 augmentation2 augmentation is 88.11% is 89.73%. and Scheme The overall 2 augmentation accuracy is improvedis 89.73%. byThe 13.43%overall for accuracy Scheme is1 improvedand 15.05% by for 13.43% Scheme for2 .Scheme 1 and 15.05% for Scheme 2.

Electronics 2019, 8, 554 17 of 19 Electronics 2019, 7, x FOR PEER REVIEW 18 of 20

Figure 14. Cumulative distribution function curves comparing position accuracy using data Figure 14. Cumulative distribution function curves comparing position accuracy using data augmentation and unaugmented data. augmentation and unaugmented data. Table 3. Experimental results for real-time positioning. Table 3. Experimental results for real-time positioning. Accuracy (%) Accuracy (%) Accuracy (%) Accuracy(0 RP Margin) (%) (1Accuracy RP Margin) (%) (2 RPAccuracy Margin) (%)

Scheme 1 (0 RP Margin)33.78 (1 RP 71.76 Margin) (2 88.11 RP Margin) SchemeScheme 1 2 33.7835.81 74.0571.76 89.7388.11 WithoutScheme Augmentation 2 35.8125.67 60.5474.05 74.6889.73 Without Augmentation 25.67 60.54 74.68 Table 4. Average positioning error for the augmentation schemes. Table 4. Average positioning error for the augmentation schemes. Mean (m) Standard Deviation (m) Scheme 1 Mean2.70 (m) Standard Deviation 3.43 (m) SchemeScheme 2 1 2.542.70 3.43 3.25 Without AugmentationScheme 2 5.622.54 3.25 8.68 Without Augmentation 5.62 8.68 In Figure 15, the average test accuracy for both augmentation schemes are compared for a real environment.In Figure 15, The the environment average test conditions accuracy for are both the sameaugmentation for all six schemes days of are testing. compared Day 3for has a real the highestenvironment. accuracy The (89.73% environment for Scheme conditions2 and 88.11%are the forsame Scheme for all1), six while days Day of 5testing. and Day Day 2 have3 has the the lowesthighest test accuracy accuracies (89.73% (86.76% for Scheme for Scheme 2 and2 and 88.11% 84.05% for forScheme Scheme 1),1 while, respectively). Day 5 and Figure Day 215 have shows the thelowest localization test accuracies results (86.76% for the for augmented Scheme 2 and and 84.0 unaugmented5% for Scheme data: 1, respectively). augmentation Figure using 15 the shows total numberthe localization of APs inresults Scheme for1 ,the augmentation augmented usingand unaugmented only the mean data: and uniformaugmentation random using numbers the total in Schemenumber2 of, and APs the in raw Scheme data. 1, Overall, augmentation Scheme 2usin performsg only slightlythe mean better and thanuniform the others. random For numbers example, in Scheme 22, has and a the mean raw error data. of Overall, 2.54 m Scheme when using 2 performs a training slightly data better set with than 7 daysthe others. of data For to example, assist in localization,Scheme 2 has while a mean Scheme error1 hasof 2.54 a mean m when error using of 2.70 a m.training In other data words, set with Scheme 7 days2 performs of data to 6% assist better in thanlocalization, Scheme while1. The Scheme performance 1 has a mean of Scheme error2 of in 2. terms70 m. In of other accuracy words, is also Sche veryme 2 closeperforms to the 6% lower better errorthan Scheme bound, for1. The which performance the median of errorScheme isalso 2 in 2.54terms m. of The accuracy unaugmented is also very data close performs to the lower poorly error due tobound, insuffi forcient which training the median data. It error is interesting is also 2.54 to m. observe The unaugmented that the raw datadata setperforms performs poorly better due than to theinsufficient complex training augmented data. data It is setsinteresting in the lab to simulations.observe that Thethe raw reason data is set that performs the real environmentbetter than the is extremelycomplex augmented complex, withdata asets number in the of lab diff erentsimulations. types of The wall, reason resulting is that in morethe real localization environment errors. is Figureextremely 14 illustrates complex, thewith improvement a number of in different terms of type accuracys of wall, with resulting the augmented in more data localization set as compared errors. toFigure raw 14 data illustrates set. Overall, the improvement the proposed in localizationterms of accuracy techniques with the can augmented provide a higherdata set accuracy as compared (i.e., ato smaller raw data error). set. WeOverall, observe thethat proposed a DNN localization could exploit techniques the additional can provide measurements a higher very accuracy well, making (i.e., a itsmaller a promising error). We technique observe for that environments a DNN could with exploi at high the additional density of measurements APs. In addition very to well, the making greater performanceit a promising using technique Scheme for2 augmentation,environments with extant a fingerprintinghigh density of approaches APs. In addition demand to athe laborious greater performance using Scheme 2 augmentation, extant fingerprinting approaches demand a laborious offline calibration phase. For example, our proposed method requires 140 observations per direction

Electronics 2019, 8, 554 18 of 19

offlineElectronics calibration 2019, 7, phase.x FOR PEER For REVIEW example, our proposed method requires 140 observations per19 directionof 20 and 560 observations per RP. The data needs to be measured in four directions. Thus, when applying the sameand 560 calibration observations process per RP. to ourThe experimentdata needs to space be measured with 60 in test four runs directions. for each Thus, RP, when at least applying 1 hour and the same calibration process to our experiment space with 60 test runs for each RP, at least 1 hour 16 min (1 s per observation) is required. and 16 min (1 s per observation) is required.

Average Test Accuracy 100 90 80 70 60 50 40 Accuracy % Accuracy 30 20 10 0 Scheme 1 Scheme 2 Without Aug

Day-1 Day-2 Day-3 Day-4 Day-5 Day-6

FigureFigure 15. 15.Average Average test test accuracy accuracy for six days days with with 30 30 test test runs runs at each at each RP. RP.

5. Conclusions5. Conclusions ThisThis research research presents presents a a novelnovel approach to to indoor indoor localization localization that has that been has proven been provento be fast to be fast enoughenough to to run run in in real-time real-time and and robust robust to changes to changes in environmental in environmental conditions. conditions. In this paper, In this we paper, we developeddeveloped augmentation augmentation techniques for for a adeep deep learni learningng scheme scheme for forWi-Fi Wi-Fi based based localization. localization. In deep In deep learning,learning, in the in otheffl ineoffline stage, stage, a four-layera four-layer CNN CNN structurestructure is is trained trained to toextract extract features features from from fluctuating fluctuating Wi-FiWi-Fi signals signals and and to build to build fingerprints. fingerprints. In In the the onlineonline positioning stage, stage, the the proposed proposed augmentation augmentation technique with a CNN-based localizer estimates the position of the target. Three approaches to data technique with a CNN-based localizer estimates the position of the target. Three approaches to augmentation for the CNN positioning system are proposed: Scheme 1, Scheme 2, and without data augmentationaugmentation. Each for thescheme CNN produces positioning a highest system simu arelation proposed: success rate Scheme of 90%1, Schemeor higher.2, andThe real- without augmentation.time testing Each of Scheme scheme 2 producesaugmentation a highest produces simulation an accuracy success of 89.73% rate of with 90% a or 2-RP higher. margin The using real-time testingseven of Schemedays of data.2 augmentation This means that produces Scheme an2 augm accuracyentation of with 89.73% a CNN with is more a 2-RP capable margin of handling using seven daysthe of data.instability This and means variability that Scheme of RSSIs2 foraugmentation Wi-Fi signals within a complex a CNN isindoor more environment, capable of handlingand thus the instabilityis more and powerful variability for use of in RSSIs classification for Wi-Fi tasks signals in fingerprint in a complex indoor indoor positioning. environment, Future research and thus will is more powerfulseek to for expand use in the classification algorithm to tasks work inseamlessly fingerprint thro indoorughout positioning.an entire building. Future The research first step willin this seek to expandis testing the algorithm and adapting to work theseamlessly algorithm to throughout work inside an rooms entire rather building. than Theonly first hallways. step in Next, this isthe testing individual maps of an entire building would need to be learned, and some way to identify which and adapting the algorithm to work inside rooms rather than only hallways. Next, the individual room or hallway the user is traveling through would need to be developed. This could be in the form maps of an entire building would need to be learned, and some way to identify which room or hallway of a global classification model that is trained to predict which local area of the building the user is the userlocated is travelingin. Then, from through there, woulda local map need and to classifier be developed. could take This over. could be in the form of a global classification model that is trained to predict which local area of the building the user is located in. Then,Author from Contributions: there, a local R.S.S. map and and S.-H.H. classifier contributed could to take the main over. idea of this research work. R.S.S. and S.-M.L. wrote computation codes, performed the simulations, experiments, and database collection. M.R. confirmed Authornumerical Contributions: results of R.S.S.the work. and The S.-H.H. research contributed activity was to planned the main and idea executed of this under research the supervision work. R.S.S. of S.-H.H., and S.-M.L. wroteR.S.S. computation and S.-H.H. codes, contributed performed to the thewriting simulations, of this article. experiments, and database collection. M.R. confirmed numerical results of the work. The research activity was planned and executed under the supervision of S.-H.H., R.S.S.Funding: and S.-H.H. This contributedwork was supported to the writing by the Technology of this article. Innovation Program (or Industrial Strategic Technology Development Program-Industrial site core technology Early development project) (10084909, Development of Funding:ImprovedThis workRTLS engine was supported and IoT tag by using the Technologymachine learning Innovation technique Program reflecting (or real Industrial environments) Strategic funded Technology by Developmentthe Ministry Program-Industrial of Trade, Industry, and site Energy core technology (MOTIE, Korea). Early This development work was supported project) by (10084909, the National Development Research of ImprovedFoundation RTLS engineof Korea and (NRF) IoT grant tag using funded machine by the Korea learning government technique (MSIT) reflecting (No. 2016R1A2B1008953). real environments) funded by the Ministry of Trade, Industry, and Energy (MOTIE, Korea). This work was supported by the National Research FoundationConflicts of of Korea Interest: (NRF) The grant authors funded declare by theno conflict Korea of government interest regarding (MSIT) the (No. publication 2016R1A2B1008953). of this article.

Conflicts of Interest: The authors declare no conflict of interest regarding the publication of this article. Electronics 2019, 8, 554 19 of 19

References

1. Schmidhuber, J. Deep learning in neural networks: An overview. Neural Netw. 2015, 61, 85–117. [CrossRef] [PubMed] 2. Jia, Y.; Shelhamer, E.; Donahue, J.; Karayev, S.; Long, J.; Girshick, R.; Guadarrama, S.; Darrell, T. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, USA, 3–7 November 2014; pp. 675–678. 3. LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444. [CrossRef][PubMed] 4. Szegedy, C.; Liu, W.; Jia, Y.Q.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going Deeper with Convolutions. In Proceedings of the IEEE Computer Vision and Pattern Recognition, Boston, MA, USA, 8–10 June 2015; pp. 1–9. 5. Donahue, J.; Jia, Y.; Vinyals, O.; Hoffman, J.; Zhang, N.; Tzeng, E.; Darrell, T. Decaf: A deep convolutional activation feature for generic visual recognition. In Proceedings of the International Conference on Machine Learning, Beijing, China, 16–21 June 2014; pp. 647–655. 6. Graves, A.; Mohamed, A.R.; Hinton, G. Speech recognition with deep recurrent neural networks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vancouver, BC, Canada, 26–31 May 2013; pp. 6645–6649. 7. Haider, A.; Wei, Y.; Liu, S.; Hwang, S.-H. Pre- and Post-Processing Algorithms with Deep Learning Classifier for Wi-Fi Fingerprint-Based Indoor Positioning. Electronics 2019, 8, 195. [CrossRef] 8. Shen, X.; Hertzmann, A.; Jia, J.; Paris, S.; Price, B.; Shechtman, E.; Sachs, I. Automatic portrait segmentation for image stylization. Comput. Graph. Forum 2016, 35, 93–102. [CrossRef] 9. Murata, M.; Ahmetovic, D.; Sato, D.; Takagi, H.; Kitani, K.M.; Asakawa, C. Smartphone-based Indoor Localization for Blind Navigation across Building Complexes. In Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom), Athens, Greece, 19–23 March 2018; pp. 1–10. 10. Singh, V.; Gorish, A.; Ujwal, B.V.S. Ensemble based real-time indoor localization using stray WiFi signal. In Proceedings of the 2018 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA, 12–14 January 2018; pp. 1–5. 11. DeVries, T.; Taylor, G.W. Dataset Augmentation in Feature Space. arXiv 2017, arXiv:1702.05538. 12. Wang, Y.; Patel, A.; Jin, H. A New Concept of Deep Reinforcement Learning based Augmented General Tagging System. In Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, NM, USA, 20–25 August 2018; pp. 1683–1693. 13. Iqbal, T.; Kong, Q.; Plumbley, M.D.; Wang, W. General-purpose audio tagging from noisy labels using convolutional neural networks. In Proceedings of the DCASE2018 Workshop on Detection and Classification of Acoustic Scenes and Events, Surrey, UK, 19–20 November 2018; pp. 212–216. 14. Tang, Q.; Kurnianggoro, L.; Jo, K.-H. Traffic sign classification with dataset augmentation and convolutional neural network. In Proceedings of the Ninth International Conference on Graphic and Image Processing (ICGIP 2017), Qingdao, China, 14–16 October 2017; p. 106152W. 15. Zhou, B.; Buyya, R. Augmentation Techniques for Mobile Cloud Computing: A Taxonomy, Survey, and Future Directions. ACM Comput. Surv. (CSUR) 2018, 51, 13. [CrossRef] 16. Lauzon, J.F. Sensor Fusion and Deep Learning for Indoor Agent Localization. Master’s Thesis, Rochester Institute of Technology, Rochester, NY, USA, June 2017. 17. Liu, W.; Wang, Z.; Liu, X.; Zeng, N.; Liu, Y.; Alsaadi, F.E. A survey of deep neural network architectures and their applications. Neurocomputing 2017, 234, 11–26. [CrossRef]

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).