Remote Sensing

remote sensing

Article 3D-TTA: A Software Tool for Analyzing 3D Temporal Thermal Models of Buildings

Juan García 1, Blanca Quintana 2,* , Antonio Adán 1,Víctor Pérez 3 and Francisco J. Castilla 3

1 3D Visual Computing and Robotics Lab, Universidad de Castilla-La Mancha, 13071 Ciudad Real, Spain; [email protected] (J.G.); [email protected] (A.A.) 2 Department of Electrical, Electronic, Control, Telematics and Chemistry Applied to Engineering, Universidad Nacional de Educación a Distancia, 28040 Madrid, Spain 3 Modelización y Análisis Energético y Estructural en Ediﬁcación y Obra Civil Group, Universidad de Castilla-La Mancha, 16002 Cuenca, Spain; [email protected] (V.P.); [email protected] (F.J.C.) * Correspondence: [email protected]

Received: 10 June 2020; Accepted: 9 July 2020; Published: 14 July 2020

Abstract: Many software packages are designed to process 3D geometric data, although very few are designed to deal with 3D thermal models of buildings over time. The software 3D Temporal Thermal Analysis (3D-TTA) has been created in order to visualize, explore and analyze these 3D thermal models. 3D-TTA is composed of three modules. In the first module, the temperature of any part of the building can be explored in a 3D visual framework. The user can also conduct separate analyses of structural elements, such as walls, ceilings and floors. The second module evaluates the thermal evolution of the building over time. A multi-temporal 3D thermal model, composed of a set of thermal models taken at different times, is handled here. The third module incorporates several assessment tools, such as the identification of representative thermal regions on structural elements and the comparison between real and simulated (i.e., obtained from energy simulation tools) thermal models. The potential scope of this software and its applications within the field of energy efficiency are presented in various case studies at the end of the paper.

Keywords: 3D thermal models; 3D data analysis; 3D visualization; image processing

1. Introduction Nowadays, the 3D digitization of indoors/outdoors building is extensively used in the fields of Architecture, Engineering and Construction (AEC). This has led to the development of multiple robotic platforms for the digitization of buildings in recent years [1]. Nevertheless, the point cloud models that are generated are almost exclusively used to evaluate the geometric features of a building [2–5]. Although the geometric data is of great importance to the assessment of the state of the building and its conservation, other sorts of data and evaluations regarding buildings are also required. Among other applications, this information can be used to detect the thermal features of the building, with which its energy efficiency and zones with special thermal features (e.g., thermal bridges and energy losses) can be assessed [6]. Currently, thermal imaging cameras can be used to perform a thermal assessment of a building, by recording the temperatures of a limited area within their field of view. Thermal imaging camera manufacturers market their own tools for the analysis of these images [7–9]. Likewise, 3D laser scanner manufacturers—such as [10–12]—also offer their own software tools for point-cloud visualization and evaluation. There are also applications that are not strictly focused on the assessment of buildings (e.g., [13,14]), which are also supplied with a wide range of tools. However, these applications may be of little use to AEC professionals because their functionalities are neither adapted to architectural nor

Remote Sens. 2020, 12, 2250; doi:10.3390/rs12142250 www.mdpi.com/journal/remotesensing Remote Sens. 2020, 12, 2250 2 of 21 to civil engineering applications. Recently, new tools for the analysis of point clouds in an AEC context have been placed on the market ([15–17]), and some of the common software tools used by architects and civil engineers now include new functionalities that handle thermal information [18–20]. Due to the interest in obtaining and analyzing the thermal information of buildings in a 3D context, different methods ([21–23]), systems ([24–26]) and even autonomous thermal scanning robots ([27,28]), which obtain 3D thermal models of buildings, have been developed in the last years. As discussed above, several tools that are marketed in the field of AEC explore 2D thermal images and process point clouds, but there are no specific tools that deal with 3D thermal models of buildings. The user can import point-cloud coordinates with certain tools, such as [14,15], together with associated values. These values are not the real temperature, but relative measures of the temperature, which are visualized according to a specific color palette. Recently, new developments have come online that address the evaluation of thermal information inserted in 3D data. Nüchter et al. developed a tool called The 3D Toolkit [29] that provides various 3D thermal point-cloud functionalities. The visualizer can show the reflectance, color or temperature of the point cloud. Thus, the user can choose among different point cloud representations and localize energy leaks. Moghadam et al. presented Spectra [30], which contains a set of tools for multispectral fusion and visualization. Both the thermal and visual spectrum can be simultaneously modeled on a 3D mesh with this software. However, rather than buildings, Spectra is designed for smaller-scale scenarios, such as air-conditioning and compressor units. The visualization options of the tool can highlight regions with extreme temperatures, among other options. In [31], the authors presented a software for the ex situ inspection of building interiors using 3D data, previously acquired with the IMAS (Indoor Multi-sensor Acquisition System) digitalization platform. The scene can be inspected by carrying out an immersive virtual navigation, in which the user can see: the point cloud and the RGB (Red, Green, Blue) images, the thermal images, the luminance, the temperature and the humidity while navigating within IMAS. Natephra et al. [32] proposed a system for the eventual evaluation of the thermal comfort conditions at different locations within the building. The authors used commercial tools (i.e., Rhinoceros [33], together with the plug-in Grasshopper [34]) to visualize the evolution of thermal information over time, in the context of Building Information Modeling (BIM). In [35], a method was presented for the automatic analysis and visualization of the deviations between the energy status of a building and the expected 3D spatial-thermal simulated model. The energetic model of the building was obtained by using a set of thermal images, and the simulated model was calculated by means of a Computation Fluid Dynamics (CFD) analysis. Finally, a 3D virtual environment was used to superimpose both models, highlighting the areas identified as potentially problematic. Table1 shows a comparison of the aforementioned software tools. The columns of this table refer to: the data sources, dealing with temporal thermal data, 3D thermal data processing algorithms, functionalities in a thermal context, the kind of software (commercial/research) and applications in the AEC thermal context. In this article, we present 3D-TTA, a tool for the visualization and exploration of 3D thermal models of buildings, which offers a set of functionalities designed to facilitate the assessment of these models. Compared with the above-mentioned tools, we can highlight several major contributions:

1. 3D-TTA deals with temporal thermal models in 3D environments, which is totally novel in the state of the art. To date, variable time has neither been introduced into 3D thermal models, nor has it been considered in the analysis phase. 2. 3D-TTA includes original 3D thermal data processing algorithms—some based on temporal thermal data—that can eﬃciently identify and delimit zones with diﬀerent thermal behaviors over time. 3. 3D-TTA can compare real and simulated thermal models of buildings, providing a useful calibration tool for architects and engineers. Remote Sens. 2020, 12, 2250 3 of 21

This paper will be organized into four sections. In Section2, the current version of 3D-TTA will be presented; in Section3, the utility of 3D-TTA will be demonstrated in a case study; and, in Section4, the conclusions will be set out, together with future improvements.

Table 1. Software technologies for 3D point clouds and 3D thermal data. (Abbreviations. 3D-T = 3D thermal data, 2D-T = 2D thermal data, C = Commercial, R = Research).

Dealing with 3D Thermal Functionalities in Data Type AEC-T Software Temporal Data Processing a Thermal Sources C/R Context Thermal Models Algorithms Context [10–12] 3D data - - - C - [13] 3D data - - - C - [14] 3D data - - - C - [15–17] 3D data - - - C X Visualization, [18–20] 3D-T - - CX Inspection [29] 3D-T - - Visualization R X Multispectral [30] 3D-T - - fusion and R- visualization [31] 3D-T - - Ex situ inspection R X Visualization, [32] 2D-T X - Inspection, RX Thermal comfort Comparison of [35] 2D-T - - real and simulated RX thermal models Data Comparison of segmentation real and simulated Detection of 3D-TTA 3D-T X thermal models of RX thermal zones structural Evolution of the elements temperature

2. 3D-TTA Software The main objective of the 3D-TTA tool is to monitor 3D thermal models of building interiors. It provides Architecture, Engineering and Construction professionals with the possibility of visualizing and exploring thermal behavior at diﬀerent times. It could therefore be a valuable tool with which to detect regions that have special thermal features with a high degree of precision, such as moisture and thermal bridges, and to position them in a 3D model of the building. The 3D-TTA inputs are 3D thermal point clouds, which are collected by a 3D thermal acquisition system. Figure1a shows the 3D thermal acquisition system used for this study, comprising a 3D laser scanner Riegl Vz-400, a Nikon D90 RGB camera and a FLIR Ax5 thermal camera [24]. A view of the 3D thermal model of a building obtained from this system is shown in Figure1b. Remote Sens. 2020, 12, x FOR PEER REVIEW 4 of 21

Remote Sens. 2020, 12, 2250 4 of 21 Remote Sens. 2020, 12, x FOR PEER REVIEW 4 of 21

(a) (b)

Figure 1. (a)(a) 3D thermal acquisition system composed of a 3D scanner, (b) an RGB camera and a thermal FigureFigurecamera. 1. (((ba)) 3DExample 3D thermal thermal of acquisition thermal acquisition point system system cloud composed composed of a building. of ofa 3D a 3D scanner, scanner, an RGB an RGB camera camera and anda thermal a thermal camera.camera. ( b) Example Example of of thermal thermal point point cloud cloud of of a building. a building. 3D-TTA is divided into three different modules. The first module deals with the visualization and 3D-TTAexploration3D-TTA is divided of the 3Dinto into building three three different di thermalﬀerent modules. modules.model, whThe Theich first ﬁrstcan module later module be deals segmented deals with with the into thevisualization visualizationthe structural andandelements explorationexploration (SEs) of that the constitute3D 3D building building the thermal thermal building. model, model, The wh whichoutputsich can can laterof later this be besegmented module segmented are into intothe the inputs thestructural structural for the elementselementsremaining (SEs)(SEs) modules. that that constitute constitute With the the thesecond building. building. module, The The outputs theoutputs user of this ofcan this module analyze module are the theare thermal inputs the inputs for evolution the for remaining the of the modules.remainingbuilding Withover modules. thetime, second With and the module,the secondthird the modulemodule, user can isthe analyzeused user tocan theimplement analyze thermal the evolutionnew thermal thermal-data ofevolution the building ofprocessing the over time,buildingalgorithms. and over the third time, module and the is usedthird tomodule implement is used new to thermal-data implement new processing thermal-data algorithms. processing algorithms.ModulesModules 1 1 and and 2 were2 were developed developed in Cin++ C++,, based based on on the the Qt frameworkQt framework [36], [36], using using the PCL the PCL [37] and[37] OpenCVand ModulesOpenCV [38] libraries,1 [38]and 2libraries, were and developed Module and Module 3 wasin C++, implemented 3based was onimplemented the using Qt framework Matlab using and [36],Matlab subsequently using and the PCLsubsequently integrated [37] intoandintegrated theOpenCV whole into [38] tool. the libraries,whole The diagram tool. and The Module in diagram Figure 32 wasinillustrates Figure implemented 2 the illustrates connection using the Matlab connection between and the betweensubsequently three modules. the three integrated into the whole tool. The diagram in Figure 2 illustrates the connection between the three Themodules. following The sectionsfollowing will sections provide will a moreprovide in-depth a more explanation in-depth explanation of each module. of each module. modules. The following sections will provide a more in-depth explanation of each module.

3D Thermal 3D ThermalModel Model

3D-TTA3D-TTA

t=i Module 1 t=i Module 1 SEs 3D3D thermal thermal exploration exploration SEs

t=n t=n t=2 t=2t=3 t=3 ModuleModule 2 2 t=1 t=1t=2 t=2 TemporalTemporal evolution evolution of 3Dof 3D thermal models thermal models SEs SEs

ModuleModule 3 3 Helping tools for thermal Helping tools for thermal CFD CFD assessment (.csv) assessment (.csv)

Figure 2. Main modules of 3D-TTA.

2.1. Module 1: 3D Thermal ExplorationFigure 2. Main modules of 3D-TTA. Figure 2. Main modules of 3D-TTA. 2.1. ModuleModule 1: 1 comprises3D Thermal severalExploration tools with which to explore 3D thermal models of buildings at any one 2.1. Module 1: 3D Thermal Exploration particular time. The exploration ranges from the generation of the building blueprint to the extraction

Remote Sens. 2020, 12, x FOR PEER REVIEW 5 of 21

Module 1 comprises several tools with which to explore 3D thermal models of buildings at any Remoteone particular Sens. 2020, 12 time., 2250 The exploration ranges from the generation of the building blueprint 5to of the 21 extraction of thermal orthoimages of structural elements. Figure 3 shows a diagram with the different oftools thermal and orthoimagesfunctionalities of of structural Module elements. 1. A brief Figure step-by-step3 shows description a diagram withof the the module di ﬀerent is toolspresented and functionalitiesbelow. of Module 1. A brief step-by-step description of the module is presented below.

Figure 3. Diagram of the tools and functionalities of Module 1. Figure 3. Diagram of the tools and functionalities of Module 1. 2.1.1. 3D Exploration: Building Level 2.1.1 3D Exploration: Building Level 3D-TTA imports coarse unorganized 3D thermal data, i.e., point clouds containing 3D coordinates and temperatures.3D-TTA imports This iscoarse a raw textunorganized file organized 3D intothermal four columnsdata, i.e., that point is selected clouds at containing the beginning 3D ofcoordinates the session. and temperatures. This is a raw text file organized into four columns that is selected at the beginningAfter opening of the the session. data file, the total thermal point cloud of the building can be seen as a largeAfter unorganized opening point the data cloud. file, Nevertheless,the total thermal this poin softwaret cloud has of the been building created can to explorebe seen as di ffaerent large partsunorganized of the building point cloud. in greater Nevertheless, detail, whichthis software have been has been organized created into to explore rooms and,different afterwards, parts of intothe Structuralbuilding in Elements. greater detail, which have been organized into rooms and, afterwards, into Structural Elements.The original 3D point cloud is therefore first processed with the objective of generating a 2D buildingThe blueprint. original 3D The point process cloud consists is therefore of extracting, first pr fromocessed the initial with pointthe objective cloud, the of pointsgenerating contained a 2D withinbuilding a horizontal blueprint. 40-cm The wideprocess slice, consists positioned of extracting, midway betweenfrom the the initial maximum point andcloud, the minimumthe points Zcontained values. These within points a horizontal are then 40-cm projected wide ontoslice, the positioned XY plane, midway thus creating between the the 2D maximum blueprint and of the the building.minimum This Z values. representation These points quickly are then displays projected a sharply onto definedthe XY plane, overview thus of creating the building the 2D structure, blueprint withof the no building. need for theThis use representation of building layouts. quickly The displays user can a thensharply move defined around overview the blueprint of the and building select the room that is to be explored.

Remote Sens. 2020, 12, x FOR PEER REVIEW 6 of 21

structure, with no need for the use of building layouts. The user can then move around the blueprint and select the room that is to be explored.

Remote2.1.2 Sens. 3D2020 Exploration:, 12, 2250 Room Level 6 of 21 Virtual navigation is implemented in this part of the module. The observer can move and 2.1.2.orientate 3D Exploration: the camera Room in order Level to view the thermal point cloud from all possible perspectives and positions. The user can also orbit the whole scene around a fixed point and zoom into specific regions. Virtual navigation is implemented in this part of the module. The observer can move and The temperature scale and the color palette can be customized in the visualizer. Apart from the visual orientate the camera in order to view the thermal point cloud from all possible perspectives and exploration and qualitative analysis via the use of a color code, the tool also shows the temperature positions. The user can also orbit the whole scene around a fixed point and zoom into specific regions. of the regions that are selected by the user, providing mean, maximum and minimum temperatures. The temperature scale and the color palette can be customized in the visualizer. Apart from the visual exploration and qualitative analysis via the use of a color code, the tool also shows the temperature of 2.1.3 3D Exploration: Structural Element Level the regions that are selected by the user, providing mean, maximum and minimum temperatures. The point cloud of any one room that is selected can subsequently be segmented into its 2.1.3.structural 3D Exploration: elements Structural(i.e., walls, Elementceiling and Level floor). This is an iterative process, in which plane regions areThe detected point cloudby applying of any onethe roomRANSAC that isalgorithm selected can[39] subsequently. An array of beparameters, segmented such into itsas the structural distance elementsthreshold (i.e.,, the walls, stopping ceiling threshold and floor). and This the is number an iterative of iterations, process, can in which be pre-set. plane The regions user arecan detected first check bythe applying results the and RANSAC then modify algorithm the parameters [39]. An array in or ofder parameters, to obtain sucha precise as the segmentation, distance threshold, as each thevisualization stopping threshold depends and to a the great number extent of on iterations, the parameter can be values. pre-set. Figure The 4 user shows can a firstflowchart check of the the resultswhole and process. then modify the parameters in order to obtain a precise segmentation, as each visualization dependsOnce to a the great room extent has on been the parametersegmented, values. the visualiz Figureer4 showscan show a flowchart (or hide), of in the different whole process. colors, the resultingOnce the segments room hasof the been room. segmented, In this way, the potentia visualizerl errors can can show be detected, (or hide), and, in diif ffneeded,erent colors, the user thecan resulting modify segments the input of parameters the room. In and, this once way, again, potential repeat errors the can segmentation be detected, process. and, if needed, The temperature the user canof modify the structural the input elements parameters can likewise and, once be again,separately repeat explored, the segmentation by using the process. same local The temperaturefunctionalities of thethat structural are found elements in Module can 1. likewise be separately explored, by using the same local functionalities that are found in Module 1.

Figure 4. Steps of the 3D segmentation process. A point cloud of the room is split into diﬀerent point Figure 4. Steps of the 3D segmentation process. A point cloud of the room is split into different point clouds, each representing a structural element. clouds, each representing a structural element. 2.1.4. 2D Exploration: Structural Element Level

Thermal inspections of buildings are usually carried out with a thermal camera. Since the field of view of these cameras is limited, the operator must take a large number of photos in order to cover the complete scene. The off-site evaluation, registration and segmentation of these large data volumes is a tedious process. Apart from that, it is difficult to know precisely the position of the image within the Remote Sens. 2020, 12, x FOR PEER REVIEW 7 of 21

2.1.4 2D exploration: Structural Element Level Thermal inspections of buildings are usually carried out with a thermal camera. Since the field of view of these cameras is limited, the operator must take a large number of photos in order to cover the complete scene. The off-site evaluation, registration and segmentation of these large data volumes Remote Sens. 2020, 12, 2250 7 of 21 is a tedious process. Apart from that, it is difficult to know precisely the position of the image within the 3D model of the building. As a solution to this problem, 3D-TTA offers the possibility of extracting 3Da single model thermal of the building.image (i.e., As thermal a solution orthoimage) to this problem, per each 3D-TTA structural offers theelement possibility of the of building, extracting by a singletaking thermaladvantage image of the (i.e., aforementioned thermal orthoimage) segmentation per each tool. structural element of the building, by taking advantageThe generation of the aforementioned of the thermal segmentation orthoimage tool.of a structural element is illustrated in Figure 5. The first Thestep generationconsists in of obtaining the thermal a coarse orthoimage image of fr aom structural the point element cloud iscorresponding illustrated in Figure to the5 .structural The first stepelement. consists Let inSE(i) obtaining be the i a-th coarse structural image element from the of point a cert cloudain room. corresponding The segment to theof points structural P3D(i) element. is first LetprojectedSE(i) be onto the ithe-th structuralplane calculated element by of aRANSAC, certain room. in order The segmentto generate of points the correspondingP3D(i) is first projected thermal ontoorthoimage, the plane I(i) calculated, thus obtaining by RANSAC, the set P in2D order(i). P2D to(i) generateis then discretized the corresponding in a gridthermal with a resolution orthoimage, of I1cm/pixel,(i), thus obtaining thereby generating the set P2D the(i). thermalP2D(i) is orthoimage, then discretized I0(i). in a grid with a resolution of 1cm/pixel, therebyOwing generating to the limited the thermal resolution orthoimage, of the Iscanner0(i). and the potential occlusion on the plane of the structuralOwing element to the limited(SE), I0(i) resolution might contain of the gaps scanner and regions and the without potential data. occlusion It is therefore on the planenecessary of the to structuralimplement element a hole-filling (SE), I 0algorithm(i) might containthat restores gaps andI0(i).regions This is the without second data. step It of is thereforethe process. necessary to implementAs mentioned a hole-filling above, algorithm the gaps thatwithin restores the orthoimaI0(i). Thisge iscan the be second produced step by of thedifferent process. factors, such as occlusions,As mentioned an absence above, of the data, gaps openings within the on orthoimage the wall and can irregularities be produced in by the diff structuralerent factors, element such asgeometry occlusions, [40]. anIn absenceview of the of data,wide openingsdiversity onof size thes wall and andshapes irregularities of these holes, in the the structural user selects element some geometryfilling-process [40]. parameters In view of the from wide a menu diversity and then of sizes checks and the shapes results. of these The procedure holes, the useris as selectsfollows. some filling-processWe first generate parameters an initial from abinary menu mask, and then M0(i), checks of the the same results. size Theas I0 procedure(i), in which is asa pixel follows. is set to 1 whenWe it first contains generate data an and initial is otherwise binary mask, 0. MM0(i)0(i ),is ofmodified the same by size applying as I0(i), morphological in which a pixel operations is set to 1 whenwith a it kernel contains whose data size and is isset otherwise by the user. 0. M This0(i) process is modified is repeated by applying until the morphological white pixels operationsof the new withversions a kernel of M whose0(i) roughly size is cover set by the the area user. of Thisthe SE process, excluding is repeated openings. until The the user white can pixels then of manually the new versionsrefine the of newM0 (versioni) roughly of the cover mask the M area0(i) in of order the SE to, excludingobtain the openings.final mask,The M(i) user. can then manually refineThe the initial new version thermal of orthoimage, the mask M I00((i),i) in and order the tomask, obtain M(i), the are final used mask, in orderM(i). to generate the filled thermalThe orthoimage, initial thermal I(i), orthoimage, by applyingI 0the(i), Navier–Stokes and the mask, Malgorithm(i), are used [41]. in order to generate the filled thermal orthoimage, I(i), by applying the Navier–Stokes algorithm [41].

Figure 5. Flowchart showing the generation of the thermal orthoimages of the structural elements.

(Up)Figure Generating 5. Flowchart the coarseshowing orthoimage the generationI0(i); (Down) of the ther Imagemal filling orthoimages algorithm. of the structural elements. (Up) Generating the coarse orthoimage I0(i); (Down) Image filling algorithm. 2.2. Module 2. Temporal Evolution of 3D Thermal Models The second module focuses on analyzing the thermal evolution of the building over time. The input of this module is the set of thermal orthoimages corresponding to each structural element at different times. Note that the interval of time between orthoimages could either be minutes, hours or days. Figure6 shows a diagram with di fferent tools and functionalities of Module 2 divided into three blocks: SE Temporal Exploration, Thermal Characterization by regions and Graphic Analysis. Remote Sens. 2020, 12, x FOR PEER REVIEW 8 of 21

2.2. Module 2. Temporal Evolution of 3D Thermal Models The second module focuses on analyzing the thermal evolution of the building over time. The input of this module is the set of thermal orthoimages corresponding to each structural element at different times. Note that the interval of time between orthoimages could either be minutes, hours or days. Figure 6 shows a diagram with different tools and functionalities of Module 2 divided into three blocks: SE Temporal Exploration, Thermal Characterization by regions and Graphic Analysis.

Remote Sens. 2020, 12, 2250 8 of 21

Figure 6. Diagram of tools and functionalities of Module 2. Note: (RoI=Region of Interest). Figure 6. Diagram of tools and functionalities of Module 2. Note: (RoI=Region of Interest). 2.2.1. Temporal Exploration of a Structural Element (SE) 2.2.1. Temporal Exploration of a Structural Element (SE) Once the segmented models have been uploaded in the tool, the user can conduct an exploration and visualOnce the comparison segmented of models the thermal have been evolution uploaded over in the time. tool, This the evaluationuser can conduct is carried an exploration out by using bothand thevisual point comparison cloud segments of the thermal and the evolution SE thermal over orthoimages time. This atevaluation different is times. carried For out each by structuralusing element,both the point a list ofcloud the segments different and times the at SE which thermal the or modelsthoimages were at different acquired times. is displayed. For each structural The user can element, a list of the different times at which the models were acquired is displayed. The user can then select several points in time from the list, and the corresponding orthoimages will be shown then select several points in time from the list, and the corresponding orthoimages will be shown in in a multi-window environment. Moreover, a different visualization option is offered to the users, a multi-window environment. Moreover, a different visualization option is offered to the users, which shows a temporal animation of any specific SE and its thermal variations. An operation that which shows a temporal animation of any specific SE and its thermal variations. An operation that is is performed by clicking a slider bar allows the user to visualize a sequence of thermal images at performed by clicking a slider bar allows the user to visualize a sequence of thermal images at didifferentfferent times. times. 2.2.2. Thermal Characterization by Regions 2.2.2. Thermal Characterization by Regions InIn thisthis block,block, the user can can select select different different region regionss of of a structural a structural element element that that may may be of be interest of interest byby clicking clicking andand draggingdragging on on the the corresponding corresponding orth orthoimage.oimage. The The thermal thermal evolution evolution over over time time is then is then calculatedcalculated forfor eacheach labeled region, region, evaluating evaluating a aset set of ofbasic basic statistical statistical indicators, indicators, such such as the as mean, the mean, minimumminimum and and maximum maximum temperatures.temperatures. TheThe resultsresults ofof thisthis analysisanalysis cancan bebe exportedexported to aa ..csvcsv file.file. In this way,this theway, evaluation the evaluation results results can be can easily be easily compared compared with with local local data data collected collected with with other other sensors sensors or with dataor with that data are estimatedthat are estimated with energy with energy simulation simulation software. software.

2.2.3.2.2.3. GraphicGraphic Analysis TheThe previouslypreviously calculatedcalculated statistical indicators can can be be graphically graphically visualized visualized in in the the third third block block of thisof this module. module. It It provides provides thermal thermal comparisonscomparisons between between regions regions composed composed of ofdifferent diﬀerent materials materials or or betweenbetween didifferentﬀerent areasareas belonging to to the the same same SE. SE.

2.3. Module 3. Useful Tools for Thermal Assessment The third 3D-TTA module comprises two sub-modules that implement innovative research methods for the automatic extraction of useful information from 3D thermal models: Real vs. Simulated Thermal Data, and Automatic Region Segmentation. A diagram illustrating both sub-modules is presented in Figure7, together with the di ﬀerent inputs and outputs. Remote Sens. 2020, 12, x FOR PEER REVIEW 9 of 21

2.3. Module 3. Useful Tools for Thermal Assessment The third 3D-TTA module comprises two sub-modules that implement innovative research methods for the automatic extraction of useful information from 3D thermal models: Real vs. Simulated Thermal Data, and Automatic Region Segmentation. A diagram illustrating both sub- modulesRemote Sens. is2020 presented, 12, 2250 in Figure 7, together with the different inputs and outputs. 9 of 21

Figure 7. Diagram of the tools and functionalities of Module 3. Figure 7. Diagram of the tools and functionalities of Module 3. 2.3.1. Comparing Real and Simulated Thermal Models 2.3.1.The Comparing first sub-module Real and Simulated addresses Thermal the issue Models of comparing real thermal data (obtained from the firstThe module) first sub-module and simulated addresses thermal the data issue (usually of compar obtaineding real from thermal Computational data (obtained Fluid from Dynamics the first module)tools (CFD)). and Thesimulated comparison thermal is implemented data (usually by obtained means offrom a di ffComputationalerence thermal Fluid image, Dynamics which yields tools a (CFD)).quantitative The evaluationcomparison of is the implemented differences betweenby means both of datasets.a difference thermal image, which yields a quantitativeThis sub-module evaluation has of the two differences inputs. The between first input both is datasets. a thermal orthoimage at any one time, t, of an SEThis of the sub-module building. has The two second inputs. input The is first obtained inputfrom is a thermal the energy orthoimage analysis at software any oneDesignBuilder time, t, of an. SEAmong of the its building. other functionalities, The second thisinput software is obtained conducts from energy the energy simulations analysis of buildings software by DesignBuilder using Energy. AmongPlus as aits calculation other functionalities, engine. A CFD this simulationsoftware conducts of the geometric energy simulations model of the of building buildings at theby using same Energytime, t, Plus generates as a calculation the second engine. input of A the CFD sub-module, simulation eventually of the geometric obtaining model a simulated of the building thermal at map the sameof the time, SE that t, generates is selected. the The second thermal input map of the isa sub-module, non-uniform eventually grid composed obtaining of a seta simulated of different thermal sized mapzones, of whichthe SE isthat then is selected. exported The to athermal.csv file. map In is the a non-uniform last stage, after grid a composed scaling algorithm of a set of procedure, different sizedthe sub-module zones, which carries is then out exported a comparison to a .csv between file. In boththe last thermal stage, images. after a scaling algorithm procedure, the sub-module carries out a comparison between both thermal images. 2.3.2. Segmenting SEs into Smaller Regions through a Temporal Analysis of Thermal Maps 2.3.2.The Segmenting second sub-module SEs into Smaller is focused Regions on through the automatic a Temporal segmentation Analysis of structuralThermal Maps elements into representativeThe second regions sub-module from a thermal-evolutionis focused on the pointautoma oftic view. segmentation The input toof thisstructural sub-module elements is the into set representativeof thermal orthoimages regions from of a structurala thermal-evolution element at poin differentt of view. times. The input to this sub-module is the set ofThe thermal process orthoimages is divided of into a structural two stages. element In the firstat different stage, the times. set of temporal thermal orthoimages is segmentedThe process into is divided regions into after two using stages. three In the di firsfferentt stage, clustering the set of methods: temporal thermal k-means, orthoimages Gaussian isMixture segmented Distribution into regions and after Hierarchical using three Clustering different algorithms.clustering methods: The structural k-means, element Gaussian is Mixture defined as a three-dimensional thermal data structure D (w h t) in order to perform the temporal Distribution and Hierarchical Clustering algorithms. The× structural× element is defined as a three- segmentation, where w h are references to the size of the orthoimage and t is the time. Each algorithm dimensional thermal data× structure D (w x h x t) in order to perform the temporal segmentation, whereyields aw set x h of are segments, references each to characterizedthe size of the byorthoimage a temporal and temperature-prototype-vector. t is the time. Each algorithm A yields consensus a set ofalgorithm segments, is each then characterized conducted. To by do a temporal so, two distance temperature-prototype-vector. metrics have been defined A consensus between thealgorithm earlier ismethods, then conducted. which evaluate, To do so, on two one hand,distanced1 ,metrics which ishave the overlapbeen defined between between the segments the earlier in themethods, image and, on the other hand, d2, which is the Euclidean distance between the respective prototype thermal which evaluate, on one hand, d1, which is the overlap between the segments in the image and, on the vectors. For a specific region, its associated prototype thermal vector stores the average temperature of the region over time. After the consensus, several regions of the structural element are characterized by a consensus temperature-prototype-vector. A flowchart illustrating the whole process is presented in Figure8. More information on this algorithm can also be found in [42]. Remote Sens. 2020, 12, x FOR PEER REVIEW 10 of 21 other hand, d2, which is the Euclidean distance between the respective prototype thermal vectors. For a specific region, its associated prototype thermal vector stores the average temperature of the region over time. After the consensus, several regions of the structural element are characterized by a consensus temperature-prototype-vector. A flowchart illustrating the whole process is presented in Figure 8. More information on this Remotealgorithm Sens. 2020can, also12, 2250 be found in [42]. 10 of 21

Figure 8. Flowchart of the automatic region segmentation through a temporal analysis of the Figure 8. Flowchart of the automatic region segmentation through a temporal analysis of the thermal thermal maps. maps. 3. Case Studies 3. Case Studies In this section, representative case studies of the application of the 3D-TTA software are presented. TwoIn 3D this thermal section, models representative of two different case scenarios studies areof the presented application in this of section. the 3D-TTA The first software 3D thermal are presented.model is composed Two 3D ofthermal 16 rooms models and 122of two million different points. scenarios The model are waspresented obtained in this after section. digitizing The a partfirst 3Dof the thermal Building model Science is composed Institute atof the16 rooms University and of122 Castilla-La million points. Mancha The (UCLM) model inwas Spain. obtained Figure after9a digitizingshows the a blueprint part of the of Building the part ofScience the building Institute that at isthe digitized University and of some Castilla-La pictures Mancha of different (UCLM) rooms. in TheSpain. second Figure 3D 9a thermalshows the model blueprint corresponds of the part to theof the 3D building Visual Computing that is digitized & Robotics and some Lab pictures (UCLM), of differentcomposed rooms. of one The room second and 2.5 3D million thermal of model points corresponds (see Figure9 b).to the 3D Visual Computing & Robotics Lab (UCLM),In order tocomposed analyze theof one thermal room evolution and 2.5 million over time of points of a particular (see Figure wall, 9b). a scanning session was conductedIn order in to both analyze scenarios. the thermal To do this, evolution the scanner over ti wasme placedof a particular in a fixed wall, position, a scanning and thesession wall was conducteddigitalized in over both a period scenarios. of 24 To h, do with this, a time-lapse the scanner of was 30 min. placed Thus, in a a fixed total ofposition, 48 3D thermal and the models wall was of digitalizedthe wall was over obtained. a period of 24 hours, with a time-lapse of 30 minutes. Thus, a total of 48 3D thermal models of the wall was obtained.

RemoteRemote Sens. Sens. 20202020,, 1212,, x 2250 FOR PEER REVIEW 1111 of of 21 21

(a)

(b)

FigureFigure 9. 9. (a(a)) First First scenario. scenario. Blueprint Blueprint and and pictures pictures of of some rooms. (b) Second scenario.scenario. 3.1. Module 1. Results

After importing the 3D thermal model of the building in Module 1 of 3D-TTA, a coarse 2D blueprint of the building is generated, as illustrated in Figure 10a. 3.1. ModuleFigure 1. 10 Resultsb illustrates the “Room Exploration” mode. The viewer displays a 3D view of room #13 from scenario 1, previously selected from the blueprint. As can be seen, the warm and the cold areas After importing the 3D thermal model of the building in Module 1 of 3D-TTA, a coarse 2D are clearly distinguished in the thermal point cloud. Moreover, the options to customize the scale and blueprint of the building is generated, as illustrated in Figure 10a. the color palette make it easier to visualize thermal variations in the model. Figure 10b illustrates the “Room Exploration” mode. The viewer displays a 3D view of room #13 from scenario 1, previously selected from the blueprint. As can be seen, the warm and the cold areas are clearly distinguished in the thermal point cloud. Moreover, the options to customize the scale and the color palette make it easier to visualize thermal variations in the model.

Remote Sens. 2020, 12, x 2250 FOR PEER REVIEW 1212 of of 21

(a)

(b)

Figure 10. 3D-TTA Module 1. (a) 2D building blueprint of scenario 1. (b) Room exploration mode. Figure 10. 3D-TTA Module 1. (a) 2D building blueprint of scenario 1. (b) Room exploration mode.

Figure 11a shows the diﬀerent colored segments that correspond to the structural elements of the room, whereas Figure 11b presents a view of the multi-window mode. In this visualization mode, eachFigure window 11a contains shows the the different thermal colored point cloud segments of a selected that correspond structural to element. the structural elements of the room, whereas Figure 11b presents a view of the multi-window mode. In this visualization mode, each window contains the thermal point cloud of a selected structural element.

RemoteRemote Sens. Sens. 20202020, 12, 12, x, 2250FOR PEER REVIEW 1313 of of21 21

(a)

(b)

Figure 11. 3D-TTA Module 1: (a) Room segmented into structural elements in different colors. Figure(b) Multi-window 11. 3D-TTA Module mode showing 1: (a) Room the 4-th segmented SE of room into #3 structural and 4-th SE elements of room in #0 different (Scenario colors. 1). (b) Multi-window mode showing the 4-th SE of room #3 and 4-th SE of room #0 (Scenario 1). Figure 12 presents two orthoimages corresponding to the point clouds shown in Figure 11b. NoteFigure that the12 orthoimagepresents two on orthoimages the left has yet correspond to be filled,ing whereas to the thepoint orthoimage clouds shown on the in right Figure was 11b. filled Noteusing that the the hole-filling orthoimage algorithm, on the left described has yet into Sectionbe filled, 2.1 whereas. It is clear the that orthoimage the visualization on the right of the was last filledimage using is greatly the hole-filling enhanced algorithm, after applying described the hole-filling in Section algorithm. 2.1. It is clear The that user the can visualization then select di offf erentthe lastregions image of is the greatly orthoimage enhanced and obtainafter applying the corresponding the hole-filling mean, maximumalgorithm. and The minimum user can temperatures.then select differentAs was mentionedregions of the in Section orthoi mage2.2, these and imagesobtain canthe becorrespondi exportedng to severalmean, maximum formats (TXT, and PNG, minimum XML). temperatures. As was mentioned in Section 2.2, these images can be exported to several formats (TXT, PNG, XML).

Remote Sens. 2020, 12, x FOR PEER REVIEW 14 of 21

Remote Sens. 2020, 12, 2250 14 of 21

Figure 12. 3D-TTA Module 1: Visualization of two SE orthoimages of the 4-th SE of room #3 and 4-th SEFigure of room 12. 3D-TTA #0 (Scenario 1). Note that the filling-hole process has not been applied to the first image. Figure 12. 3D-TTA Module 1: Visualization Visualization of two SE orthoima orthoimagesges of the 4-th SE of room #3 and 4-th SE of room #0 (Scenario 1). Note that the filling-hole process has not been applied to the first image. SE of room #0 (Scenario 1). Note that the filling-hole process has not been applied to the first image. 3.2. Module 2. Results 3.2. Module 2. Results 3.2. ModuleThe thermal 2. Results evolution of a wall from Scenario 2 (see Figure 13a) is shown in this module. To do The thermal evolution of a wall from Scenario 2 (see Figure 13a) is shown in this module. To do so, so, a Theset ofthermal 48 3D evolutionthermal models of a wall from from Scenario Scenario 2 was 2 (see obtained Figure 13a over) is a shown 24-hour in time-period,this module. withTo do a timea set lapse of 48 3Dof 30-minutes. thermal models With fromthis dataset, Scenario the 2 wasuser obtainedcan perform over an a 24-houranalysis time-period,of the thermal with evolution a time so,lapse a set of 30-min.of 48 3D Withthermal this models dataset, from the userScenario can perform2 was obtained an analysis over of a the24-hour thermal time-period, evolution with of the a timeof the lapse room of over 30-minutes. time. The With data this input dataset, for Module the user 2 can is obtained perform anby analysissegmenting of the each thermal of the evolution 48 room modelsroom over into time. their The respective data input structur for Moduleal segments 2 is obtained by using by segmentingModule 1. eachFigure of 13 theb 48shows room a modelsvisual ofinto the their room respective over time. structural The data segments input for by Module using Module 2 is obtained 1. Figure by 13 segmentingb shows a visualeach of comparison the 48 room in comparison in a multi-window environment of the temperature of the chosenFigure wall 13 at different times. modelsa multi-window into their environment respective structur of the temperatureal segments of by the using chosen Module wall at di1. fferent times.b shows a visual comparison in a multi-window environment of the temperature of the chosen wall at different times.

(a) Figure(a) 13. Cont.

Remote Sens. 2020, 12, x FOR PEER REVIEW 15 of 21

Remote Sens. 2020, 12, 2250x FOR PEER REVIEW 15 of 21

(b)

(b) Figure 13. (a) Wall used to illustrate the thermal evolution of the Module 2 results. (b) Thermal Figureorthoimages 13. (a) Wall for different used to illustratetimes. the thermal evolution of the Module 2 results. (b) Thermal Figureorthoimages 13. (a for) Wall diﬀ erentused times.to illustrate the thermal evolution of the Module 2 results. (b) Thermal orthoimagesThe thermal for different characterization times. per region is carried out by selecting several regions of interest (FigureThe thermal 14a). For characterization each Region of perInterest region (RoI), is carried some basic out bystatistical selecting indicators several (mean, regions maximum of interest and (FigureminimumThe 14 thermala). For temperature) eachcharacterization Region are of then Interest per computed, region (RoI), is some carriedwhich basic ma out statisticaly byalso selecting be indicators graphically several (mean, analyzed,regions maximum of as interest shown and in (minimumFigureFigure 14 a).14 temperature) b.For each Region are thenof Interest computed, (RoI), whichsome basic may alsostatistical be graphically indicators analyzed,(mean, maximum as shown and in minimumFigure 14b. temperature) are then computed, which may also be graphically analyzed, as shown in Figure 14b.

(a)

Figure 14. Cont. (a)

Remote Sens. 2020, 12, x FOR PEER REVIEW 16 of 21 Remote Sens. 2020, 12, 2250 16 of 21 Remote Sens. 2020, 12, x FOR PEER REVIEW 16 of 21

(b)

(b) Figure 14. 3D-TTA Module 2: (a) Example of the selection and labeling of three regions. (b) Figure 14. 3D-TTA Module 2: (a) Example of the selection and labeling of three regions. (b) Graphics Graphics of the temperature evolution over time and basic statistical indicators for each region. of the temperatureFigure 14. evolution3D-TTA over Module time 2: and (a basic) Example statistical of indicators the selection for each and region.labeling of three regions. (b) Graphics of the temperature evolution over time and basic statistical indicators for each region. 3.3.3.3. Module Module 3.1. 3.1. Results Results AA set set3.3. of of Moduleorthoimages orthoimages 3.1. Results of of a awall wall from from Scenario Scenario 1 1was was generated, generated, to to compare compare the the real real thermal thermal model model withwith the the simulated simulatedA set ofthermal thermal orthoimages model. model. of Both Botha wall walls walls from were wereScenario digitized digitized 1 was with withgenerated, our our 3D 3D tothermal thermal compare scanning scanning the real system systemthermal model duringduring 24 24with hours, h, atthe 30-min atsimulated 30-minute intervals. thermal intervals. model. Both walls were digitized with our 3D thermal scanning system AsAs explainedduring explained 24 hours, in SectionSection at 30-minute 2.3 2.3,, the the ﬁrst firstintervals. sub-module sub-module (i.e., (i.e., 3D-TTA 3D-TTA Module Module 3_1) is3_1) employed is employed to perform to performa comparison a comparisonAs between explained between the realin Section data,the real obtained 2.3, data, the obtain infirst our sub-moduleed 3D in thermal our 3D (i.e., scanningthermal 3D-TTA scanning system, Module and system, the3_1) simulated and is employed the to simulateddata, obtainedperform data, fromobtained a comparison the energyfrom betweenth simulatione energy the simulationreal software data, DesignBuilderobtain softwareed in DesignBuilder our. Figure 3D thermal 15 .shows Figure scanning a 15 ﬂowchart shows system, a of and the flowchartthis process.simulated of this process. data, obtained from the energy simulation software DesignBuilder. Figure 15 shows a flowchart of this process.

Figure 15. Flowchart of the thermography-simulation comparison process. Figure 15. Flowchart of the thermography-simulation comparison process. In this case study, we chose the thermal orthoimage at 2 pm of a wall from Scenario 1. Figure 16b Figure 15. Flowchart of the thermography-simulation comparison process. showsIn this the case user study, interface we ofchose this the sub-module. thermal orthoima The outputsge at 2 of pm this of comparison a wall from were Scenario two di1. ﬀFigureerence 2 16bthermal-images shows theIn user this of diinterface caseﬀerent study, resolutionsof this we sub-module. chose (pixel the/cm thermal Th).e One outputs orthoima of them of thisge was at compar adapted 2 pm isonof to a thewallwere original fromtwo differenceScenario resolution 1. Figure 2 thermal-imagesof the thermal16b shows orthoimage of thedifferent user (Figure interface resolutions 16b, of left), this (pixel/cm andsub-module. the other). One wasTh eof adaptedoutputs them toofwas thethis adapted resolution compar isonto of theDesignBuilder were original two difference resolution(Figure thermal-images16 ofb, right).the thermal oforthoimage different (Figureresolutions 16b, le(pixel/cmft), and the2). Oneother of was them adapted was toadapted the resolution to the original of DesignBuilderresolution (Figure of the 16b, thermal right). orthoimage (Figure 16b, left), and the other was adapted to the resolution of DesignBuilder (Figure 16b, right).

RemoteRemote Sens. Sens.2020 2020, 12, 12, 2250, x FOR PEER REVIEW 17 of17 21 of 21

(a)

(b)

Figure 16. 3D-TTA Module 3_1: (a) Wall used to illustrate the results of Module 3.1. (b) User interface showingFigure the16. di3D-TTAfference Module thermal-images 3_1: (a) Wall with diusedfferent to illustrate resolutions: the results original of resolution Module 3.1. of the(b) thermalUser orthoimageinterface showing (left) and the resolution difference of thermal-images DesignBuilder with (right). different resolutions: original resolution of the thermal orthoimage (left) and resolution of DesignBuilder (right). After observing the results, it can be stated that the differences are below 1 degree Celsius in the wall areaAfter (in observing blue). It canthe results, also be it seen can thatbe stated the temperaturethat the differences difference are below is slightly 1 degree greater Celsius around in the the lowerwall partarea and(in blue). sides It of can the also wall be than seen elsewhere. that the temperature With regard difference to the window is slightly frame greater area around (aluminum the lower part and sides of the wall than elsewhere. With regard to the window frame area (aluminum carpentry), the temperature differences were between 1.5 and 3 degrees Celsius, differences that carpentry), the temperature differences were between 1.5 and 3 degrees Celsius, differences that increase whenever the point was closer to the center. increase whenever the point was closer to the center. 3.4. Module 3.2. Results 3.4. Module 3.2. Results With regard to the second sub-module, we present the segmentation results for two structural With regard to the second sub-module, we present the segmentation results for two structural elements of Scenario 2. We first obtained 48 thermal orthoimages of the walls at different times and elements of Scenario 2. We first obtained 48 thermal orthoimages of the walls at different times and intervals, and the algorithm then yielded the regions together with their temporal evolution over time. intervals, and the algorithm then yielded the regions together with their temporal evolution over Figurestime. Figures17 and 1817 showand 18 the show results the of results this sub-module. of this sub-module. In both scenes,In both the scenes, wall hasthe wall been has segmented been intosegmented four regions. into four The regions. test walls The and test the walls visualization and the visualization of the segmented of the segmented regions inregions different in different colors are presentedcolors are in presented Figures 17 ina Figures and 18 a.17a The and black 18a. The color black signifies color nosignifies data. no Figures data. 17Figuresb and 17b 18b and show 18b the prototypeshow the vectors prototype of the vectors segmented of the segmented regions. regions. InIn the the first first case, case, the the cyan cyan segment segment is the is warmestthe warmest part ofpart the of wall, the wherewall, where the temperature the temperature decrease is slowerdecrease than is slower in the than other in regions.the other Curiously regions. Curiou enough,sly enough, the red regionthe red covers region acovers vertical a vertical zone between zone the windows, which might suggest that part of the wall was constructed from a different material. The upper horizontal part in red corresponds to the roll-up blinds, which are raised. The blue and green Remote Sens. 2020, 12, x FOR PEER REVIEW 18 of 21

Remote Sens. 2020, 12, x FOR PEER REVIEW 18 of 21 Remotebetween Sens. the2020 windows,, 12, 2250 which might suggest that part of the wall was constructed from a different18 of 21 material. The upper horizontal part in red corresponds to the roll-up blinds, which are raised. The between the windows, which might suggest that part of the wall was constructed from a different blue and green regions are clearly located within the window frames. As expected, the temperature material. The upper horizontal part in red corresponds to the roll-up blinds, which are raised. The regionsof the (aluminum) are clearly located window within follows, the with window a degree frames. of thermal As expected, inertia, the the temperature outdoor temperature of the (aluminum) and blue and green regions are clearly located within the window frames. As expected, the temperature windowdecreases follows, faster withthan athe degree other of segments. thermal inertia,The difference the outdoor between temperature the blue andand decreasesthe red zones faster is than of the (aluminum) window follows, with a degree of thermal inertia, the outdoor temperature and theprobably other segments.explained by The a higher diﬀerence or lower between thermal the loss, blue depending and the on red whether zones is the probably locking mechanisms explained by a decreases faster than the other segments. The difference between the blue and the red zones is higherare within or lower the segment thermal and loss, on depending the enclosure on whetherof the window. the locking mechanisms are within the segment probably explained by a higher or lower thermal loss, depending on whether the locking mechanisms and onIn the the enclosure second case, of the a large window. (green) region covers most of the wall, which signifies a uniform are within the segment and on the enclosure of the window. temperatureIn the second explained case, by a the large thermal (green) inertia region of the covers wall. mostThe left, of theright wall, and lower which borderlines signiﬁes a of uniform the In the second case, a large (green) region covers most of the wall, which signifies a uniform temperaturewall are clearly explained the coolest by thezones thermal with the inertia highest of the thermal wall. inertia. The left, Finally, right andthe blue lower spot borderlines corresponds of the temperature explained by the thermal inertia of the wall. The left, right and lower borderlines of the wallto an are alarm clearly light, the the coolest warmest zones segment with the point highest in the thermal image. inertia. Finally, the blue spot corresponds to wall are clearly the coolest zones with the highest thermal inertia. Finally, the blue spot corresponds anto alarm an alarm light, light, the the warmest warmest segment segment point point in in the the image. image.

(a)

(b)

(b) Figure 17. 3D-TTA Module 3.2: Segmentation into regions through a temporal analysis of thermal Figuremaps. 17. (a) 3D-TTA The test Module wall and 3.2: the Segmentation visualization into of regions the segmented through a regions temporal in analysis different of colors. thermal (b maps.) Figure 17. 3D-TTA Module 3.2: Segmentation into regions through a temporal analysis of thermal (aCorresponding) The test wall prototype and the visualization vectors are in of the the same segmented colors. regions in diﬀerent colors. (b) Corresponding prototypemaps. (a vectors) The test are wallin the and same the colors. visualization of the segmented regions in different colors. (b) Corresponding prototype vectors are in the same colors.

(a)

Figure(a) 18. Cont.

RemoteRemote Sens. Sens.2020 2020, 12, ,12 2250, x FOR PEER REVIEW 19 of19 21 of 21

(b)

Figure 18. 3D-TTA Module 3.2: Segmentation into regions through a temporal analysis of thermal maps. (a)Figure The test 18. wall 3D-TTA and visualizationModule 3.2: Segmentation of the segmented into regions regions through in different a temporal colors. analysis (b) Corresponding of thermal prototypemaps. ( vectorsa) The are test in wall the same and visualizationcolors. of the segmented regions in different colors. (b) Corresponding prototype vectors are in the same colors. 4. Conclusions and Future Developments 4. Conclusions and Future Developments Many of the 3D-data acquisition devices in use today are used to generate geometric models of buildingsMany and of facilities, the 3D-data which acquisition is very useful devices in thein use field today of AEC. are used Additionally, to generate a large geometric number models of related of softwarebuildings tools, and with facilities, a broad which range is very of geometric useful in functionalities, the field of AEC. have Additionally, been developed. a large Nevertheless,number of untilrelated now, software there have tools, been with no specific a broad applications range of geometric focusing onfunctionalities, the exploration have of 3Dbeen thermal developed. models of buildingsNevertheless, and until their now, analysis. there have been no specific applications focusing on the exploration of 3D thermalThe e ffmodelsective exportof buildings of new and technologies their analysis. and formats for the analysis of the thermal efficiency of buildingsThe can effective be the export next goal of new for technologies this kind of software.and formats The for analysis the analysis can goof fromthe thermal a study efficiency with local andof staticbuildings data can in wallsbe the tonext a methodology goal for this kind based of software. on the exploration The analysis and can identification go from a study of problemswith local in and static data in walls to a methodology based on the exploration and identification of problems in three-dimensional thermal models. Additionally, the data processing of 3D thermal models of entire three-dimensional thermal models. Additionally, the data processing of 3D thermal models of entire buildings can have an appreciable impact on the world of engineering/architecture. buildings can have an appreciable impact on the world of engineering/architecture. In this paper, the first steps have been taken in the exploitation of these technologies for the study In this paper, the first steps have been taken in the exploitation of these technologies for the of energy efficiency. 3D-TTA has been presented as a tool that visualizes, explores and processes 3D study of energy efficiency. 3D-TTA has been presented as a tool that visualizes, explores and thermalprocesses models 3D thermal of buildings. models Among of buil itsdings. functionalities, Among its functionalities,3D-TTA can be 3D-TTA used for: can the be visual used for: detection the ofvisual regions detection with special of regions thermal with featuresspecial thermal (e.g., thermalfeatures (e.g., bridges thermal and bridges moistures) and inmoistures) a 3D framework; in a 3D theframework; generation the of generation thermal images of thermal of structuralimages of stru elements;ctural elements; the visualization the visualization of the of evolution the evolution of the temperatureof the temperature on different on structuraldifferent structural elements andelem surfacesents and over surfaces time; andover thetime; automatic and the segmentation automatic of structuralsegmentation elements of structural by their elements thermal behavior.by their th Moreover,ermal behavior. the tool Moreover, includes an the additional tool includes module an for comparisonsadditional module between for simulated comparisons and betw realeen thermal simulated data. and real thermal data. 3D-TTA3D-TTA is isa first a first version version that that will will be be improved improved and and extended extended to toother other functionalities functionalities in infuture future versions.versions. A greatA great challenge challenge consists consists in translating in translating the 3Dthe thermal 3D thermal model, model, as well as aswell the as results the results obtained fromobtained the di fffromerent the steps different in the steps thermal in the analysis thermal process, analysis to theprocess, BIM world.to the BIM This world. entails This adapting entails the 3D-TTAadaptingoutputs the 3D-TTA so as to outputs introduce so as them to introduce into gbxml them and intoIDF gbxml file and formats. IDF file More formats. effective More algorithmseffective thatalgorithms process 3Dthat thermal process point 3D thermal clouds point must clouds be introduced. must be introduced. For example, For the example, automatic the detectionautomatic of thermaldetection failures of thermal at higher failures levels at than higher those levels used than in thethose current used in3D-TTA the currentversion 3D-TTA (e.g., SEversion level) (e.g., would SE be a verylevel) useful would attribute be a very in useful thermal attribute imaging in thermal software, imaging offering software, a global offering characterization a global characterization of the building withof the building option of with thermal the option evaluations of thermal of a evaluati single room,ons of a a set single of rooms room, ora set a complete of rooms floor.or a complete floor.

AuthorAuthor Contributions: Contributions:Conceptualization, Conceptualization, J.G., J.G., A.A., A.A., V.P. V.P. and F.J.C.;and F.C.; Data Data curation, curation, J.G. andJ.G. A.A.;and FormalA.A.; Formal analysis, J.G.; Funding acquisition, A.A.; Investigation, A.A.; Methodology, J.G.; Project administration, A.A.; Resources, J.G., analysis, J.G.; Funding acquisition, A.A.; Investigation, A.A.; Methodology, J.G.; Project administration, A.A.; B.Q. and A.A.; Software, J.G.; Supervision, B.Q., V.P. and F.J.C.; Validation, J.G., A.A., V.P. and F.J.C.; Visualization, J.G.,Resources, B.Q. and J.G., A.A.; B.Q. Writing—original and A.A.; Software, draft, J.G.; B.Q.; Supervision, Writing—review B.Q., V.P. &and editing, F.C.; Validation, J.G., B.Q., J.G., A.A., A.A., V.P. V.P. and and F.J.C. All authors have read and agreed to the published version of the manuscript.

Funding: This work has been ﬁnanced by the European Regional Development Fund [SBPLY/19/180501/000094 project] and the Ministry of Science and Innovation [PID2019-108271RB-C31]. Remote Sens. 2020, 12, 2250 20 of 21

Acknowledgments: Thanks to the European Regional Development Fund [SBPLY/19/180501/000094 project] and to the Ministry of Science and Innovation [PID2019-108271RB-C31]. Conﬂicts of Interest: The authors declare no conﬂict of interest.

References

1. Adán, A.; Quintana, B.; Prieto, S.A. Autonomous mobile scanning systems for the digitization of buildings: A review. Remote Sens. 2019, 11, 306. [CrossRef] 2. Valero, E.; Adan, A.; Huber, D.; Cerrada, C. Detection, Modeling, and Classification of Moldings for Automated Reverse Engineering of Buildings from 3D Data. In Proceedings of the 28th International Symposium on Automation and Robotics in Construction (ISARC), Seoul, Korea, 29 June–2 July 2011. [CrossRef] 3. Quintana, B.; Prieto, S.A.; Adán, A.; Vázquez, A.S. Semantic Scan Planning for Indoor Structural Elements of Buildings. Adv. Eng. Inform. 2016, 30, 643–659. [CrossRef] 4. Macher, H.; Landes, T.; Grussenmeyer, P. From point clouds to building information models: 3D semi-automatic reconstruction of indoors of existing buildings. Appl. Sci. 2017, 7, 1030. [CrossRef] 5. Wang, Q.; Kim, M.K. Applications of 3D point cloud data in the construction industry: A fifteen-year review from 2004 to 2018. Adv. Eng. Inform. 2019, 39, 306–319. [CrossRef] 6. Brooke, C. Thermal imaging for the archaeological investigation of historic buildings. Remote Sens. 2018, 10, 1401. [CrossRef] 7. Fluke SmartView. Available online: https://www.fluke.com/es-es/producto/accesorios/software/fluke- smartview-ir-mobile (accessed on 8 April 2019). 8. Thermal Analysis and Reporting (Desktop). FLIR Tools. Available online: https://www.flir.com/products/flir- tools/ (accessed on 8 April 2019). 9. Advanced Thermal Analysis and Reporting Software. FLIR Tools+. Available online: https://www.flir.com/ products/flir-tools-plus/ (accessed on 8 April 2019). 10. RiSCAN PRO 2.0. Available online: http://www.riegl.com/products/software-packages/riscan-pro/ (accessed on 8 April 2019). 11. FARO SCENE. Available online: https://www.faro.com/en-gb/products/construction-bim-cim/scene- webshare-cloud/software-for-the-faro-laser-scanner-focus-series/ (accessed on 8 April 2019). 12. Leica Cyclone. Available online: https://leica-geosystems.com/products/laser-scanners/software/leica- cyclone (accessed on 8 April 2019). 13. Cignoni, P.; Callieri, M.; Corsini, M.; Dellepiane, M.; Ganovelli, F.; Ranzuglia, G. MeshLab: An Open-Source Mesh Processing Tool. In Proceedings of the Eurographics Italian Chapter Conference; The Eurographics Association: Aire-la-Ville, Switzerland, 2008. [CrossRef] 14. CloudCompare (version 2.X) [GPL software]. Available online: http://www.cloudcompare.org/ (accessed on 8 April 2019). 15. ReCap Pro. Available online: https://www.autodesk.com/products/recap/overview (accessed on 8 April 2019). 16. PointCab. Available online: https://www.pointcab-software.com/en/ (accessed on 8 April 2019). 17. NUBIGON. Available online: https://www.nubigon.com/ (accessed on 8 April 2019). 18. Revit. Available online: https://www.autodesk.co.uk/products/revit/overview (accessed on 8 April 2019). 19. Autocad. Available online: https://www.autodesk.co.uk/products/autocad/overview (accessed on 8 April 2019). 20. IFC Builder. Available online: http://ifc-builder.en.cype.com/ (accessed on 8 April 2019). 21. Alba, M.I.; Barazzetti, L.; Scaioni, M.; Rosina, E.; Previtali, M. Mapping Infrared Data on Terrestrial Laser Scanning 3D Models of Buildings. Remote Sens. 2011, 3, 1847–1870. [CrossRef] 22. López-Fernández, L.; Lagüela, S.; González-Aguilera, D.; Lorenzo, H. Thermographic and mobile indoor mapping for the computation of energy losses in buildings. Indoor Built Environ. 2017, 26, 771–784. [CrossRef] 23. Lagüela, S.; Martínez, J.; Armesto, J.; Arias, P. Energy efficiency studies through 3D laser scanning and thermographic technologies. Energy Build. 2011, 43, 1216–1221. [CrossRef] 24. Adan, A.; Prado, T.; Prieto, S.A.; Quintana, B. Fusion of thermal imagery and LiDAR data for generating TBIM models. In Proceedings of the 2017 IEEE Sensors, Glasgow, UK, 29 October–1 November 2017; pp. 1–3. [CrossRef] Remote Sens. 2020, 12, 2250 21 of 21

25. Lin, D.; Bannehr, L.; Ulrich, C.; Maas, H.G. Evaluating thermal attribute mapping strategies for oblique airborne photogrammetric system AOS-Tx8. Remote Sens. 2020, 12, 112. [CrossRef] 26. Ham, Y.; Golparvar-Fard, M. An automated vision-based method for rapid 3D energy performance modeling of existing buildings using thermal and digital imagery. Adv. Eng. Inform. 2013, 27, 395–409. [CrossRef] 27. Borrmann, D.; Nüchter, A.; Dakulović,M.; Maurović,I.; Petrović,I.; Osmanković,D.; Velagić,J. A mobile robot based system for fully automated thermal 3D mapping. Adv. Eng. Inform. 2014, 28, 425–440. [CrossRef] 28. Adan, A.; Prieto, S.A.; Quintana, B.; Prado, T.; García, J. An autonomous thermal scanning system with which to obtain 3D thermal models of buildings. In Proceedings of the 35th CIB W78 2018 Conference: IT in Design, Construction, and Management, Chicago, IL, USA, 1–3 October 2018. 29. Nuchter, A. 3DTK—The 3D Toolkit. 2011. Available online: http://slam6d.sourceforge.net/ (accessed on 8 April 2019). 30. Moghadam, P.; Vidas, S.; Lam, O.; Systems, A. Spectra: 3D Multispectral Fusion and Visualization Toolkit. In Proceedings of the Australasian Conference on Robotics and Automation, Melbourne, Australia, 2–4 December 2014; pp. 2–4. 31. Armesto, J.; Sánchez-Villanueva, C.; Patiño-Cambeiro, F.; Patiño-Barbeito, F. Indoor multi-sensor acquisition system for projects on energy renovation of buildings. Sensors 2016, 16, 785. [CrossRef][PubMed] 32. Natephra, W.; Motamedi, A.; Yabuki, N.; Fukuda, T. Integrating 4D thermal information with BIM for building envelope thermal performance analysis and thermal comfort evaluation in naturally ventilated environments. Build. Environ. 2017, 124, 194–208. [CrossRef] 33. Rhinoceros. Available online: https://www.rhino3d.com/ (accessed on 8 April 2019). 34. Grasshopper. Available online: https://www.grasshopper3d.com/ (accessed on 8 April 2019). 35. Golparvar-Fard, M.; Ham, Y. Automated Diagnostics and Visualization of Potential Energy Performance Problems in Existing Buildings Using Energy Performance Augmented Reality Models. J. Comput. Civ. Eng. 2014, 28, 17–29. [CrossRef] 36. Qt. Available online: https://www.qt.io/ (accessed on 8 April 2019). 37. Rusu, R.B.; Cousins, S. 3D is here: Point Cloud Library (PCL). In Proceedings of the 2011 IEEE International Conference on Robotics and Automation, Shanghai, China, 9–13 May 2011. [CrossRef] 38. Bradski, G. The OpenCV Library. Dr. Dobb’s J. Softw. Tools 2000, 120, 122–125. 39. Fischler, M.A.; Bolles, R.C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 1981, 24, 381–395. [CrossRef] 40. Pérez, E.; Salamanca, S.; Merchán, P.; Adán, A. A comparison of hole-filling methods in 3D. Int. J. Appl. Math. Comput. Sci. 2016, 26, 885–903. [CrossRef] 41. Bertalmio, M.; Bertozzi, A.L.; Sapiro, G. Navier-stokes, fluid dynamics, and image and video inpainting. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), Kauai, HI, USA, 8–14 December 2001; Volume 1. [CrossRef] 42. Adán, A.; García, J.; Quintana, B.; Castilla, F.J.; Pérez, V. Temporal-Clustering Based Technique for Identifying Thermal Regions in Buildings. In International Conference on Advanced Concepts for Intelligent Vision Systems (ACIVS 2020); Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2020; Volume 12002, pp. 290–301. [CrossRef]

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).