TOWARDS A DIGITAL TWIN FOR HERITAGE INTERPRETATION

From HBIM to AR visualization

ELOISA DEZEN-KEMPTER1, DAVI LOPES MEZENCIO2, ERICA DE MATOS MIRANDA3, DANILO PICO DE SÁ4 and ULISSES DIAS5 1,2,3,4,5School of Technology, UNICAMP [email protected] 2,3,4{d169759|e155254| d214848}@dac.unicamp.br [email protected]

Abstract. Data-driven Building Information Modelling (BIM) technology has brought new tools to efficiently deal with the tension between the real and the virtual environments in the field of Architecture, Engineering, Construction, and Operation (AECO). For historic assets, BIM represents a paradigm shift, enabling better decision-making about preventive maintenance, heritage management, and interpretation. The potential application of the Historic-BIM is creating a digital twin of the asset. This paper deals with the concept of a virtual environment for the consolidation and dissemination of heritage information. Here we show the process of creating interactive virtual environments for the Pampulha Modern Ensemble designed by in the 1940s, and the workflow to their dissemination in an AR visualization APP. Our results demonstrate the APP feasibility to the Pampulha’s building interpretation.

Keywords. Augmented Reality (AR); Historic Building Information Modelling (HBIM); Heritage Interpretation; Modern Architecture.

1. Introduction The Digital Twin (DT) can be understood as a probabilistic, multiscale, multiphysics integrated simulation of a system that uses the best physical models, sensors, and history to mirror the life cycle of its corresponding twin. The DT can also predict the system’s response to security-critical events and uncover previously unknown issues before they become critical by comparing current and predicted responses. Systems involving DT are capable of mitigating damage or degradation by activating self-healing mechanisms or recommending changes to the mission profile, thereby increasing the life and probability of success (Glaessgen and Stargel 2012). The DT consists of three components: physical product in a real monitored space, data and information connections, and the corresponding virtual product in virtual space (Grieves and Vickers 2017). At the AECO industry, the

RE: Anthropocene, Proceedings of the 25th International Conference of the Association for Computer-Aided Architectural Design Research in Asia (CAADRIA) 2020, Volume 2, 183-191. © 2020 and published by the Association for Computer-Aided Architectural Design Research in Asia (CAADRIA), Hong Kong. 184 E. DEZEN-KEMPTER ET AL. virtual product in the virtual space corresponds to the BIM model, which in the case of the built heritage, is the HBIM. Real-time monitoring is performed by sensors, generating data in quantity. Data is communicated, stored, processed, and associated with the virtual product allowing awareness of physical space performance and simulation in virtual space for decision making on how to act in real space. The potential application of the DT for Heritage is its realistic representation in the form of an intelligent and semantically enriched 3D model (HBIM), becoming a tool capable of managing information collected and modeled, improving its availability and accessibility. According to Nagakura et al. (2015), having a digital model of historical heritage is a cheaper tool to allow building investigations because, unlike other areas of study, there is no way to take buildings to laboratories or store them in museums galleries like other historical artifacts. The use of digital scanning technologies to survey the current state of historic buildings, such as photogrammetry and laser scanning, expedites the process of generating a digital model. Photogrammetry is a low-cost method and captures building textures and materials by creating a 3D model from photographs (Nagakura and Sung 2014). To assist in the understanding of 3D projects and to promote social interaction between architects, historians, and visitors to historical heritage sites, Augmented Reality (AR) is a tool that allows users to interact with the digital model (Nagakura and Sung 2017). AR supports the real experience of historical heritage sites without having to go there.

2. Research Aim and Roadmap This paper presents ongoing research aiming to develop a AR APP from the digital twin of two buildings, the Ballroom and the St. Francis of Assisi Church, part of the Pampulha Modern Ensemble (PME) designed by the world-famous architect Oscar Niemeyer in the 1940s. The UNESCO’s World Heritage Convention listed the Pampulha Modern Ensemble, in 2016, due to its Outstanding Universal Value as a cultural landscape. This research explores how users, professional and non-professional public, can interact with AR to visualize, filter, and retrieve any information enclosed within the Digital Twin of historic buildings. AR is a growing area in the AEC industry. It offers a new tool for visualization and interaction in the field of heritage preservation, operation, and maintenance, as well as for tourist purposes. Regarding the visualization and interaction purposes, the following research questions are drawn up: Is it possible to filter and access only specific components and information from the Digital Twin into an AR environment? How can the modeler prepare the BIM Model for the AR environment to retrieve the knowledge-based information? The digital twins’ creation employed terrestrial laser scanning (TSL) and a low cost unmanned aerial vehicle (UAV). The process was based on three fundamental steps: (1) collection of spatial and documentary data, (2) data processing and dense surface model (DSM) creation, and (3) HBIM modeling. The framework includes TOWARDS A DIGITAL TWIN FOR HERITAGE 185 INTERPRETATION creating the HBIM model in Autodesk’s REVIT authoring. For the AR process, the model organization is crucial. The heritage elements were organized in the HBIM model using the Dynamo visual programming tool. Dynamo allowed the creation of building components groups to make the HBIM model easy to import and interpret into the AR model-authoring platform. The Diagram below illustrates the Roadmap of the research project and the distinct computational tools, which integrate each step. Step 1 addresses the HBIM creation of both buildings, which were created in previous research, in which the details of the Scan2BIM process were already published (Cogima et al. 2018, Cogima et al. 2019). The focus of the present work is to present the results from the AR APP creation for the first building, the Ballroom.

Figure 1. Roadmap of the AR creation and testing from the Digital Twin of Niemeyer’s Pampulha Buildings.

3. Case Study: The Ballroom and St. Francis of Assisi Church in the Pampulha Modern Ensemble The PME was the center of a garden city project created at , the capital of State. Built between 1942 and 1943, the PME was developed by architect Oscar Niemeyer and by landscaper Burle Marx, in collaboration with great artists and professionals, among them, the painter Candido Portinari. The PME is composed of four buildings: St. Francis of Assisi Church, the Cassino (current Pampulha Art Museum), the Ballroom (current Centre of Reference in Urbanism, Architecture, and Design), and the Yacht Golf Club. Although each of the four building offers a slightly different architectural solution to the challenge of adapting the common formal Modernist vocabulary to the climate and environment of the new city, overall what emerged was a fluid and plastic architecture that embraces views and joins to the picturesque qualities of the lake and mountain landscape. The PME and the selected buildings are relevant to 186 E. DEZEN-KEMPTER ET AL. the application of AR, as they are heritage sites of high significance to the country and receive thousands of annual visitors.

3.1. ST. FRANCIS OF ASSISI CHURCH The church is constructed of five adjacent ellipsoid concrete shell structures of different heights. Although industrial buildings have used concrete shells previously, their use here marked the first occasion for a religious structure. The most massive shell faces the lake where a slender freestanding ‘tower’ in the shape of an inverted pyramid is linked to the church by the flat roof of the porch. Paulo Werneck decorated part of the outer surface of the shell with tiles, while blue and white murals by Cândido Portinari cover the facades of the lower shells facing the street. The unusual and innovative design of the church generated a significant disagreement within the Catholic Church, which caused the postponement of its consecration and, consequently, the opening to the public until 1959.

3.2. BALLROOM - CENTRE OF REFERENCE IN URBANISM, ARCHITECTURE, AND DESIGN The more modest building from PME, the Ballroom, sits on a small island surrounding mostly paved minimalist gardens and linked to the shore by a bridge. The circular flat-roof ends in a curved walkway which connects the main building with the bathrooms facing the circular stage. The gallery is supported by expressive columns, which also contour the entire circular volume ending. Decorative blue and white tiles designed by Candido Portinari coated all surfaces. The Ballroom worked as urban equipment for being a place of dances, shows, and dinner, to provide the public with the experience and the allure of the new region of Belo Horizonte. The Ballroom has been twice restored in the last twenty years and now houses the Centre of Reference in Urbanism, Architecture, and Design.

4. Methodology The digital twins’ creation employed terrestrial laser scanning (TSL) and a low cost unmanned aerial vehicle (UAV). The process was based on three fundamental steps: (1) collection of spatial and documentary data, (2) data processing and dense surface model (DSM) creation, and (3) HBIM modeling, as detailed in Cogima et al. (2018).The framework includes creating the HBIM model in Autodesk’s REVIT authoring. For the AR process, the model organization is crucial. The heritage elements were organized in the HBIM model using the Dynamo visual programming tool. Dynamo allowed the creation of building components groups to make the HBIM model easy to import and interpret into the AR model authoring platform.

4.1. AR VISUALIZATION Augmented reality (AR) overlays digital content on real-world objects that a computer sees employing a regular camera, which enriches our view of our surroundings. AR needs a device with at least three components : (i) a camera to TOWARDS A DIGITAL TWIN FOR HERITAGE 187 INTERPRETATION provide input from the real world to the computer, (ii) a screen or glasses so that the user can see the real world enhanced by computer-generated digital information, and (iii) enough processing power for the device to retrieve features from the real world, using computer vision, and overlay digital information. Smartphones and tablets are ubiquitous devices that already support these three components. They have been used in many AR applications, and at least while AR glasses are not widespread, they will continue as the primary resource for AR experiences. For successful augmented reality experience, some elements are desirable: (i) the augmented experience should bring new information to augmented objects, (ii) the digital models should allow real-time interaction, and (iii) they should convey accurate information. We kept these goals in mind when modeling the augmented experience of Niemeyer buildings aiming to keep the same visionary vision he had in mind when designing the buildings. In our work, we would like to allow people to enjoy the buildings and the architectural details in compelling new ways. We are talking about four buildings that were constructed very close to each other. Any visitor traveling through Belo Horizonte would probably visit them on the same trip. Our goal is to create an environment where any user would do the same using the digital world. The user should have a clear notion about where the buildings are around Pampulha’s lake and the proportions of each building. Therefore, we implemented an augmented map experience where the user handles a conventional map (like those we receive at touristic information posts) and installs the AR APP on the tablet or smartphone. The intention is not to replace the experience of seeing these buildings in the site but bringing some aspects of them, which include the accurate 3D models, the surrounds, and some modern architectural supplementary information. This view is much more complete than just traditional means like videos and photos. Other than the augmented experience where overlaid digital information is additive to the real-world objects (map), we created a first-person view of the four buildings. The visitors can walk through the buildings, enter into the rooms, and look through the windows as if they were in the model site. This paper explains each step of building the AR application, as well as the material and tools used, aiming replicability. The 3D architecture and source codes are publicly available at the GitHub home page Davilopesm/PampulhaModernEnsembleAR. Visitors have the choice to see the architectural matters in detail, which they are readily available to be inquired by the audience by clicking on icons designed to represent common themes in Niemeyer’s design. Each icon is reduced to its minimal form, expressing characteristics of the buildings. The best way to introduce our augmented reality application is by showing how it behaves as an application. We exercise the freedom to experiment during the development, so each detail is linked to a concept we want to convey, trying to bridge the gap between the real buildings and the technology. Figure 2a shows the city map that triggers the augmented reality environment and 2b depicts all components of the Ballroom (above) and the St. Francis of Assisi Church (below) Digital Twins in an exploded view. The buildings are around Pampulha’s lake, and tourists may visit them in one single day. The fact that we are using a map 188 E. DEZEN-KEMPTER ET AL. highlights the spatial distribution.

Figure 2. (a) Pampulha’s lake map with St. Francis of Assisi Church [1], and the Ballroom [2]; (b) Exploded Views of the Ballroom (above) and The St. Francis of Assisi Church (below) . .

Figure 3 gives an overview of the proposed APP with the augmented reality triggered in front of the map and the first scene of the application. Some natural gestures like pinching with two fingers to zoom in/out or moving two fingers around each other to rotate are available. However, the most relevant features in this environment are those reached through the icons.

Figure 3. overview of the APP.

The Icons represent Le Corbusier’s Five Points of the Modern Architecture, which could be identified in Niemeyer’s proposal for the Ballroom. We can see in Figure 4 the five main buttons to engage with the augmented reality. The first icon shown in Figure 4 is called “The Pilotis”. It has vertical bars and red arrows to recollect the fact that there is a grid of concrete columns supporting the roof structure to make the soil freely usable. The second icon is called “The Free Facade”. It is reminding of open and closed sections on the facade that enable the separation and connection of the exterior design from the building TOWARDS A DIGITAL TWIN FOR HERITAGE 189 INTERPRETATION structure. The third button, which is called “Horizontal Windows”, is reminding the windows along the facades that provide the interior lightness and offers views of the surroundings. The fourth button, the free ground plan, reminds the user of the absence of load-bearing walls, which allows flexible use of the space. The fifth icon, called “The flat roof,” is inspired in the building itself.

Figure 4. AR from the Ballroom (A) and the 5 buttons that represent the five points of Architecture, plus the First person View (B).

Figure 5 (left) shows the effect of selecting each icon. These views are helpful to students since they complement explanations found in their textbooks. By showing particular design patterns of modern architecture and allowing viewers to play with them, the APP provides valuable insights that could not be reached by merely reading about those buildings on books or looking at pictures of it. The Digital Twin enables an experience of first-person. The last picture right in Figure 5 shows a first-person user entering the building, getting close to the front door. The textures built are based on the real ones, allowing no specialists to enjoy the view and specialists to study it.

Figure 5. The six buttons of APP in action. 190 E. DEZEN-KEMPTER ET AL.

5. Results To create the Augmented Reality application, the Unity Engine was used. Its main application is to create video games, but due to its integration with smartphones and its ability to create complex scenarios, it becomes a useful tool to create and deploy applications that uses 3D models. The implementation of the application was entirely done within Unity. However, to get the 3D model from Autodesk REVIT to Unity, we first passed it through the Autodesk 3DS MAX, which allowed us to convert the RVT file format into the FBX. The native file format of REVIT cannot be imported directly in Unity; furthermore, all texture from the Revit model should be re-applied in the components using the 3DS MAX. Figure 6 shows the workflow to implement the application.

Figure 6. Workflow of Software for AR Application .

First, a new Unity Engine project was created to receive the 3D model. In the new project, the “Vuforia Augmented Reality” option should be selected for the appropriate downloads to be made. Six game scenes were created inside this project. These six scenes being one for the main app scene, one for the Pampulha’s Modern ensemble explanation, and two for each model: The Ballroom and St. Francis Assisi Church. The augmented reality was created using a plugin called Vuforia AR Camera that searches for Image Targets in the environment. When the search finds a specific image target, the proper 3D model is overlaid. In each scene, when the marker for the whole Pampulha map is tracked, meaning when it is visible on the smartphone screen, it triggers the AR visualization for the models onto the map. In the main application scene, four buttons were created; each one of these buttons was given a C Sharp script that allows opening the corresponding scene. In the scene for the quick explanation of the Pampulha Modern Ensemble, two elements were created. One of them uses the same image used as a logo for the application, and the other provides textual explanations. The other two scenes, concerning the Ballroom and St. Francis of Assisi Church, contain six buttons. Five of these buttons allow users to make specific elements visible or invisible at runtime, so, for example, the user can see inside the buildings after removing the roof. Sometimes, the user may want to visualize only one element while making all the others invisible. That is also possible by playing with the buttons. The sixth button transports the user to the first-person view scene. In this scene, it is possible to move around with a UI element that is used as a joystick. TOWARDS A DIGITAL TWIN FOR HERITAGE 191 INTERPRETATION

6. Conclusions and Future Works In this paper, we proposed and detailed a methodological framework to implement AR visualisation and interpretation of two masterpieces of Modern Architecture in , through the HBIM model, developed by the authors in another research. This work aimed to use AR to promote the diffusion of historical content differently and innovative with the use of immersive environments, fully synthetic or mixed, with Augmented and Virtual Reality mediation, enabling to offer an interpretation and understanding of architectural objects of high historical and cultural value. We assume that using AR solutions enhances visitors’ engagement in Heritage data visualizations, but this remains to be formally studied. Preliminary tests with the members of the research team attested that they experienced a more significant interaction with the building, made possible by the AR application. Users highlighted the ease of accessing with one click the history of the building, the biography of the architect, and explore the components of the building, in an interactive way. The validation tests demonstrate that the proposed framework is consistent and is also able to support the intended goals for the interpretation of heritage. Future validation will include feedback from the users, including the usefulness, historical content provided, and ease of interaction with the interface. Additionally, future work will include knowledge-based information to be used for planning and decision-making by the building maintenance staff.

References COGIMA, C.K., PAIVA, P.V.V., DEZEN-KEMPTER, E. and CARVALHO, M.M.G.: 2018, Digital scanning and BIM modeling for modern architecture preservation: the Oscar Niemeyer’s Church of Saint Francis of Assisi, Proceedings of the 22nd SIGRADI Conference, São Carlos. COGIMA, C.K., PAIVA, P.V.V., DEZEN-KEMPTER, E., CARVALHO, M.A.G. and SOIBELMAN, L. 2019, The role of knowledge-based information on BIM for built heritage, in I. MUTIS and T. HARTMANN (eds.), Advances in Informatics and Computing in Civil and Construction Engineering, Springer, Cham, 27-34. GLAESSGEN, E. and STARGEL, D.: 2012, The digital twin paradigm for future NASA and US Air Force vehicles, Proceedings of 53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference, 20th AIAA/ASME/AHS Adaptive Structures Conference and 14th AIAA. GRIEVES, M. and VICKERS, J. 2017, Digital twin: Mitigating unpredictable, undesirable emergent behavior in complex systems, in F.-J. KAHLEN, S. FLUMERFELT and A. ALVES (eds.), Transdisciplinary perspectives on complex systems, Springer, Cham, 85-113. NAGAKURA, T. and SUNG, W.: 2014, Ramalytique: Augmented Reality in Architectural Exhibitions, Proceedinns of the 19th International Conference on Cultural Heritage and New Technologies, Viena, 1-20. NAGAKURA, T. and SUNG, W.: 2017, AR Mail from Harbin, Proceedinns of Siggraph 2017, Los Angeles. NAGAKURA, T., TSAI, D. and PINOCHET, D.: 2015, Digital Heritage Visualizations of the Impossible: Photogrammetric Models of Villa Foscari and Villa Pisani at Bagnolo, Proceedings of the 20th International Conference on Cultural Heritage and New Technologies, VIENA.