Away with apps? Interactive web 3D driven by relational databases Jelle Vermandere

Away with apps? Interactive web 3D driven by relational databases.

Jelle Vermandere Student number: 01306207

Supervisors: Willem Bekers, Prof. ir.-arch. Ruben Verstraeten Counsellor: Nino Heirbaut

Master's dissertation submitted in order to obtain the academic degree of Master of Science in de ingenieurswetenschappen: architectuur

Academic year 2019-2020 The author gives permission to make this master dissertation available for consultation and to copy parts of this master dissertation for personal use.

In all cases of other use, the copyright terms have to be respected, in particular with regard to the obligation to state explicitly the source when quoting results from this master dissertation.

13/01/2020 Foreword

This thesis aims to make the process of adding 3D models to relational databases easier while providing more functionality. Before I explain this, I would like to thank everyone who made my life easier.

First of all, my supervisors Willem Bekers and Ruben Verstraeten for their valuable guidance sessions, their inspiring ideas and their support in my journey to the completion of this thesis.

I would also express my gratitude towards my favourite spelling checkers Lotte and Tessa for their extreme focus on letters and punctuation marks and Jonathan for the more rigorous language corrections.

Special thanks goes to my sister Floor for being my lay-out support and showing me the wonderful world of web development.

Thanks to the numerous people who created forum posts and online tutorials who helped me solve many problems. I have learned many new skills along the way, many of which I did not even know existed.

I also want to thank my friend Sytse for keeping me physically fit in between the long and sometimes frustrating coding sessions by being my climbing buddy.

Lastly, an enormous thank you to my parents for always being there. They have given me everything I need in my journey to adulthood and allowed me to pursue my passions and dreams. Without them, I would have never been where I am today. Table of Contents

Defining the challenge: Introduction 15

Architectural Databases 15 Database websites 15 App based 3D model viewing 16

Mixed reality 17 Augmented reality examples 17

Providing a solution: The website 19

Objective: no more apps 19

Workflow 20 Collecting information 20 Creating the data 20 Filling the database 20 Linking the data 20 Displaying on the website 20 Modeller vs Curator vs User 21

Organisation of the thesis 21 3D model creation 21 Database creation 21 Website creation 21 Case Study 22 Le Corbusier 22 Villa Savoye 22 The Philips pavilion 23 The Chapel Notre Dame du Haut 23 Software used 24 3D model creation and export 24 Database creation and management 24 Website creation and hosting 24 Creating 3D models for the web 25

Modelling: Creating 3D geometry 26 Different types of 3D modelling 26 Modelling for real time rendering 27 Modelling for the section plane 27 Texturing: Creating a material 28 UV layout 28 PBR Materials 31 Texture baking 33 Lighting: defining the lights 34 Real time lights 34 Image-based lighting 34 Rendering: calculating the final image 35 Real time rendering 35 Post-processing 36 Exporting: choosing a file type 38

Optimizations 39 3D model 39 Textures 40

Constructing a database for the web 41

What is a (relational) database? 41 Structured query language SQL 41 Not only SQL 44 Choosing a Database Framework 44 MySQL 44 NodeGoat 44 Integration in the website 44 Database structure 45

Collecting and storing information 46 Keeping it up to date 46 Building the website 47

Dynamic websites 48

Responsive websites 48

Choosing the coding languages 49 HTML 49 PHP 49 Bootstrap 50 Creating a working environment 51 Setting up a local server 51 Setting up an online server through Heroku 51 Database integration 52 Establishing the connection 52 Interacting with the database: CRUD 52 File renaming system 53 Alternative option: Nodegoat 54 3D model integration 55 Testing different Technologies 55 Programming the 3D viewport 57 Programming the section plane 59 The viewport 60 Augmented reality integration 61 Augmented reality 61 Testing different technologies 62 Programming The AR experience 64 Programming the QR code generator 65 The AR experience 66 Website Structure 67 Different pages 67 The navigation bar 68 The home page 68

The search page 69

The person page 69

The building page 70

The AR page 70

The submit page 71 Conclusion 72

The finished product 72

Workflow system 73

Applications 74

Future expandability 74

Appendix 75

References 75 Websites 75 Books/ Papers 76 Frameworks 77 Images 77 Source code 77

Example catalogue 78 AWAY WITH APPS? INTERACTIVE WEB 3D DRIVEN BY RELATIONAL DATABASES

Jelle Vermandere

Supervisors: Willem Bekers, Ruben Verstraeten

Counsellor: Nino Heirbaut

Abstract- Architectural databases mostly consist of text and 2D plans. Adding 3D models can create a better understanding of the buildings. This thesis documents the creation of a website, in which interactive 3D models are linked to relational databases. By providing guidelines and best practices about the creation of these 3D models, this thesis allows modellers to create 3D models that are compatible with all the provided features. By keeping in mind the fact that this website can be used with very large databases, an effort has been made to automate as much of the process as possible. The resulting website can be used for exhibitions, archiving and much more.

Keywords- Relational database, 3D model, Real time rendering, Website, Augmented reality

DEFINING THE CHALLENGE: Everything that is produced for this thesis and the website is made with open source or free software. INTRODUCTION This ensures the future expandability of the website is not limited by expensive programs. The ever increasing amount of architectural data needs to be stored in online databases in order to be accessible and searchable. The data mostly consists of text and architectural plans, because storing and conserving one of the most useful documents in architectural presentations, the scale model, is very inefficient.

This is where virtual 3D models can help fill the need for a better understanding of the buildings. The website is a combination of different Furthermore, placing these virtual models in the technologies. Because of the rapid development real world with the aid of a mobile device, can of these kinds of technologies, most parts are provide a very similar experience to a physical interchangeable in case a better alternative scale model, with the added benefits of accurate becomes available. material representation, animations and more. This technology is called Augmented reality. CREATING 3D MODELS FOR THE WEB

Creating 3D models that are fully compatible with all the features of the website will require some specific needs. Displaying a 3D model on a screen requires three main parts: a 3D form, textures and a PROVIDING A SOLUTION: renderer. All these parts have different parameters THE WEBSITE and functions that need to be optimised in order to provide a smooth experience on the website.

This thesis aims to lower the barrier of interacting with 3D models as much as possible by providing a full web based solution. This means that there is no need to download any extra apps or programs on the user’s device. The website links a database and 3D models that are connected to specific buildings 3D form Textures Renderer of that database. For storing the final 3D models, the popular and efficient GL Transmission Format (glTF) file format was chosen. This format is built for fast rendering and is widely adopted on most platforms. 10 CONSTRUCTING A Furthermore, every type of data has a dedicated page. That page will also show all the connected DATABASE FOR THE WEB people and buildings. This way, the user can explore the data and find new relations. This includes The database is an integral part of the website buildings, the centre piece of the website. The structure, because all the content of the website is connection with the 3D models is a integral part of stored in it. It also holds the references to the 3D this thesis. models in order to make sure the correct models The website has a page for viewing the 3D models are linked to the buildings. The database is built with in a exhibition context. The page is linked to the the open standard “Serial Query Language”. This camera of the mobile device of the visitor. With the allows multiple databases to be linked together. By aid of special bar codes (QR codes), the user will linking the datasets together, relations can be set be able to view the 3D models in augmented reality. up.

Building ID Name Architect ID Built Date 1 Philips Pavilion 1 1958 2 Villa Savoye 1 1931 3 ......

Architect ID Name Birthday Nationality 1 Le Corbusier 06/10/1887 French 2 Jelle 26/06/1995 Belgian 3 ......

These relations form the basis for large scale discoveries of new connections between buildings, architects, locations, time periods, and anything The website also has a submit page. Here, the 3D that is added to the database. models and their information can be added to the database. The whole process of linking the 3D models to the database, setting up the different data pages and generating the markers for the augmented reality experience is automated. This BUILDING THE WEBSITE allows the website to easily scale towards a large amount of different buildings.

The website allows the end users to explore all the data, present in the database, in a visual way. The 3D models will provide the extra information simple text and images can not provide. CONCLUSION

The website provides a working product that can be used for exhibitions and online archiving The website uses three types of underlying of architectural information. It is a great tool for technologies to create the functioning website: the visualizing large amounts of data and because of its database integration, the 3D model integration and search functionality, it can help researchers in their the augmented reality integration. work. Much of the process of linking the 3D models to the database has been automated, ensuring The website has a search page that is able to search scalability to very large databases. Because the the entire database for matches and connections. website is made with open source frameworks, it This feature allows curators, researchers and others is encouraged to expand this website with more to make a selection of works for an exhibition or functionality and future technology improvements. research.

11 WEG MET APPS? INTERACTIEVE WEB 3D GEBASEERD OP RELATIONELE DATABANKEN.

Jelle Vermandere Promotors: Willem Bekers, Ruben Verstraeten Begeleider: Nino Heirbaut Abstract-Databanken met betrekking tot architectuur bevatten voornamelijk teksten en schetsen terwijl geen ruimte voorzien wordt om maquettes op te slaan. De creatie van een website, waar interactieve 3D modellen worden gekoppeld aan relationele databanken bieden hierop een antwoord. Door richtlijnen en tips te geven in verband met de creatie van de 3D modellen, zorgt deze thesis ervoor dat makers 3D modellen kunnen maken die compatibel zijn met alle functies van de website. Doordat de website ook inzetbaar blijft voor grote hoeveelheden van gebouwen, is er moeite gedaan om zoveel mogelijk te automatiseren. Dit resulteert in een website die toepasbaar is voor tentoonstellingen, archivering en zoveel meer.

Kernwoorden- Relationele databanken, 3D model, Real time rendering, Website, Augmented reality DE UITDAGING: Alles dat gemaakt is voor deze thesis en de website, is gemaakt aan de hand van open source INTRODUCTIE programma’s. Dit zorgt ervoor dat de potentiële toekomstige uitbreidingen niet gelimiteerd zijn door De steeds stijgende hoeveelheid van architecturale dure software. informatie moet opgeslagen worden in databanken om eenvoudig toegankelijk en doorzoekbaar te zijn. De data bestaat meestal uit teksten en plannen, omdat het stockeren van maquettes erg inefficiënt is.

De website is een combinatie van verschillende bestaande technologieën die elk kunnen vervangen worden indien er in de toekomst betere alternatieven zijn. Hier kunnen 3D modellen de nood voor een betere verstaanbaarheid van de gebouwen invullen. Verder kunnen deze 3D modellen in de echte wereld worden geplaatst aan de hand van een mobiel 3D MODELLEN MAKEN apparaat. Deze ervaring kan een meerwaarde VOOR HET WEB creëren ten opzichte van een echte maquette aan de hand van accurate materiaal representatie, animaties en interactieve snedes. Deze technologie 3D modellen maken die volledig compatibel zijn heet augmented reality. met alle functies van de website, moeten specifieke eigenschappen bevatten. Het weergeven van 3D modellen vereist drie dingen: een 3D vorm, texturen en een renderer. Deze onderdelen hebben EEN OPLOSSING: verschillende parameters die elk zoveel mogelijk geoptimaliseerd zien te worden. Op die manier DE WEBSITE zullen de 3D modellen een vlotte ervaring bieden.

Deze thesis tracht de interactie met 3D modellen zo vlot mogelijk te laten verlopen. Dit kan aan de hand van een website. Dit betekent dat er geen externe apps moeten gedownload worden op het apparaat van de gebruiker. De website linkt een database aan 3D vorm Texturen Renderer 3D modellen van gebouwen in die database. De 3D modellen worden geëxporteerd naar het GL Transmission Format (glTF) formaat, dit is geoptimaliseerd voor een snelle weergave op het web. 12 EEN DATABASE MAKEN Ook heeft elk type data van de database een eigen webpagina. Die pagina toont ook alle VOOR HET WEB geconnecteerde gebouwen en architecten. Op die manier kunnen gebruikers de verschillende relatie De database is een essentieel onderdeel van de verkennen. Hierbij zitten ook de 3D modellen, die website, omdat alle data hierin is opgeslagen. een essentieel onderdeel zijn van de informatie De database bevat ook de referenties naar de 3D De website heeft ook een pagina om de 3D modellen modellen en afbeelding van de gebouwen om ervoor te zien in augmented reality op bijvoorbeeld te zorgen dat de juiste data bij de 3D modellen tentoonstellingen. De pagina is verbonden met komt. De database is gebouwd met de open de camera van het mobiele apparaat, waarbij de source standaard “Serial Query Language”. Deze bezoeker aan de hand van speciale markers de 3D standaard laat toe om verschillende databases met modellen in de echte wereld zullen zien verschijnen. elkaar te verbinden. Hiermee kunnen relaties tussen verschillende gebouwen en architecten ontdekt worden.

Building ID Name Architect ID Built Date 1 Philips Pavilion 1 1958 2 Villa Savoye 1 1931 3 ......

Architect ID Name Birthday Nationality 1 Le Corbusier 06/10/1887 French 2 Jelle 26/06/1995 Belgian 3 ......

Deze relaties vormen de basis voor grootschalige onderzoeken naar nieuwe relaties tussen gebouwen, Daarnaast bevat de website ook een pagina waarin architecten, tijdsperiodes, plaatsen en alle andere makers van de 3D modellen hun creaties kunnen soort data dat in de database verwerkt is. uploaden, samen met de bijhorende informatie. Het hele proces van het toevoegen van de modellen is geautomatiseerd, net zoals het genereren van de markers voor de augmented reality pagina. Dit laat de website toe inzetbaar te blijven voor grote DE WEBSITE MAKEN hoeveelheden van informatie.

De website maakt de exploratie van de informatie mogelijk in een visuele manier aan de hand van de 3D modellen. CONCLUSIE

De website is een werkend eindproduct dat diverse mogelijkheden biedt rond onderzoek en visualisaties De website gebruikt drie onderliggende van grote hoeveelheden van informatie. Dankzij technologieën om tot een werkend geheel te komen: de zoekfunctie kan de website hulp bieden aan de database integratie, de 3D model integratie en de onderzoekers. Een groot deel van het toevoegen van augmented reality integratie. 3D modellen is geautomatiseerd, zodat de website grote hoeveelheden van informatie kan verwerken. De website heeft een zoekfunctie waarbij de hele database kan doorzocht worden naar bepaalde De website is gebouwd met open source termen. Dit laat onderzoekers en curators toe frameworks en het wordt aangemoedigd om hier om een selectie te maken op basis van bepaalde verder op te bouwen met meer functionaliteiten en parameters. toekomstige technologische vernieuwingen.

13

Defining the challenge: Introduction

Architectural Databases

The amount of data about buildings and architects that is collected and stored will continue to increase over time. In the digital age, this data is stored in large collections also known as databases. Having all the data accessible means it can be read and interpreted by computers. This way, data can be linked to each other. By connecting all the buildings by the same architect or by their location, new relations can be discovered that were previously too large to be researched by hand.

Databases that mainly contain textual information can be difficult to visualize because of the lack of visual information. Simple text doesn’t provide the same experience as a visual medium. That’s why these datasets should be accompanied by pictures and models to increase the readability.

This thesis aims to build a framework of a system which allows a visual, textual and interactive exploration of architectural data with the aid of 3D models.

Database websites

Providing the information of the database to a potential viewer can be done in different ways. One of the clearest ways to navigate this is through a website. Websites can be easily found through search engines and accessed everywhere.

These websites are made up of different webpages which can serve a specific subset of the data.

Building a unique webpage for every piece of data is not only a copious amount of work, but also limits future expandability. That is where using an online database can solve both these hurdles. The database allows for large amounts of data to be stored and interpreted by the website and can be kept up to date at all times. This way, a blueprint can be made where all the info of the database can be inserted. When a piece of data from the database is altered, it will also change on the website without having to rewrite the website itself.

It also allows for relations to be discovered between the different datasets. Works can be sorted by built-date or architect, improving the searchability. The website then adapts to the contents of the database.

15 App based 3D model viewing

When confronted with a new building or structure, it is not always easy to grasp the full 3D form, especially when only plans or 2D images are available. That’s why scale models are used. With the help of a scale model, users can explore the full composition and get a feeling for the relative scale of the different elements. However, these models don’t allow for much flexibility after they are built. Therefore the use of digital 3D models can enhance this experience. By providing a clear view of all angles, accurate material representation and adding interaction such as animations, 3D sections or walkthroughs, this can be a good alternative to physical models. Such physical models also take up actual storage space, which can become very valuable in the future as the amount of models gets bigger as time goes on.

Unlike physical models, which are by themselves self-sufficient, 3D models do require some amount of technical infrastructure to view and interact with - at the very minimum, a screen, and some software that can interpret and render the model. The technical choices can determine how many people can access the model data, and enable or limit certain modes of interaction. This thesis is interested in building a system which will enable many people to individually explore 3D models interactively and simultaneously, with as little technical overhead as possible on the user side.

From the perspective of technical reliability, cost and maintenance, the option of a large touch table or screen seems like a good option to show these models. However, these kinds of devices are more suited to interactions involving multiple people sharing the same device. In the case of 3D models, this is not ideal; in order to explore the model, users should individually be able to control and explore it without negotiating with other users. Therefore, a system based on individual devices is preferable.

Building an app for personal devices would allow for the individual control, if the system catered only to a known collection of devices maintained by the system owner. The app can be made for specific hardware, limiting maintenance and compatibility issues. However, the necessity to acquire hardware means this is not the most cost-efficient way to distribute access to a database. Downloadable apps would allow more users to access the dataset from their own personal devices. Native apps, requiring users to install applications on their phones or tablets, have the downside that they must deal with a large variety of devices. On the user side, they may not want to download the app due to device storage concerns or slow download times, leading to an overall low adoption of the application.

To address these concerns, this thesis chooses to build a web application. By focusing on an browser-based application, the system can avoid compatibility issues with different operating systems and instead use the power provided by browsers, a standard feature on most personal devices. Being browser-based also makes it easier for users to find and explore the content, without the need to download anything: all that would be needed is a simple URL, or even a scannable QR code.

16 Mixed reality

The rise of computers and the increase in their graphical capabilities enabled the creation of virtual 3D spaces. Which, at first, users could explore through pointing and clicking with a mouse, but increasingly can also be explored by smart usage of sensors. Nowadays, users can manipulate mobile devices in the physical world, and their movements and rotations can be mapped into the virtual space, making navigation in these virtual spaces increasingly intuitive.

The spaces presented on a digital screen can range from the real world, where everything is real (e.g. looking at the camera of your phone) to fully virtual (e.g. tracking a camera through a video game). Everything in between can be considered as mixed reality. When we occlude our view to present a fully digital world, we call it virtual reality. When we look at the real world and add virtual objects in that world, we call it augmented reality.

Reality Augmented Reality Virtual reality

Making these virtual worlds come to life requires large amounts of computing power and complex technologies. That is why mixed reality has only recently come to the consumer.

Nowadays, augmented reality is sometimes used in exhibitions as an extra layer of information, enhancing the real-life experience. But due to its low adoption rate, it is usually limited to a supplementary role, leaving essential aspects of an exhibition in the real world. Nonetheless, augmented reality brings a different perspective to the user and can cause a higher level of engagement.

This thesis aims to use augmented reality to enhance the traditional 3D model interaction on a webpage.

Augmented reality examples

Some applications and exhibitions already use augmented reality to enhance the user experience. Some use it to show 3D objects in the room, others to enhance 2D images, like paintings on a wall, that can come to life due to augmented reality.

17 Philips museum design tour:

The Philips museum in Eindhoven in The Netherlands gives the user a tablet to walk around the space. This is a way to ensure everything works as intended and technical problems of device incompatibility are avoided. Throughout the exhibition there are images on the ground that, when seen by the tablet’s camera, show a room in augmented reality. The user then is able to walk through and experience the room as if it were real.

Google Maps AR:

Augmented reality can be very useful in wayfinding. Not everyone feels at home navigating with a flat image on a screen. Having signs and directions that point in the right direction are much easier to read than following a point on a map. That’s why Google is starting to implement augmented reality in Google maps. Using the device’s camera, the user can look where he has to go. This makes it significantly easier to get started in the right direction.

Ikea Place:

Famous furniture seller Ikea has an app where products from their catalogue can be placed in a room in augmented reality. This enables the users to make decisions in the comfort of their own home. The fact that the furniture is always to scale and the lighting in the room is matched as close as possible to the real conditions also shows if the furniture will fit with the rest of the room.

18 Providing a solution: The website

Objective: no more apps

This thesis tries to lower the barrier of viewing 3D models linked to large databases as much as possible.

Web-based applications form one approach to lowering this barrier. Present-day smart devices are equipped with a browser to view web pages, making web applications device agnostic. Imagine the user is just one tap away from interacting with the model. By scanning a QR-code with the built-in camera app, the user is directed to the correct page on a website which can access the device’s camera. By simply pointing the device, users can view the 3D model in augmented reality. In order to achieve this, a website has to be built where these 3D models can be displayed based on the data from a connected database. This thesis will go through all the steps that are necessary to complete this website.

Another part of lowering the barrier is the way this website is produced. By only using open source and free to use applications and frameworks, everyone can freely expand and improve this initial set-up. However, for the sake of completion and benchmarking purposes, other ways of achieving augmented reality experiences were also researched, not necessarily with free or open source software. These other solutions have some interesting advantages and could serve as a benchmark against the web-based solution of this thesis.

This thesis will also examine the creation of content for this website. Just like the website itself, the content can be made with free or open source programs. This thesis emphasizes the correct production of 3D models that need to comply to some rules in order to be fully compatible with all the features of the website.

To manage all the data that such a website should contain, a database is needed. Choosing the right type of database is important to ensure optimal performance and stability. There must also be a way to manage the data directly from the website. This way, there is no coding knowledge required to add or delete data from the database. The website should enable searching through the database, and allow users to visually explore connections between buildings, architects, and dates.

All these different components must work together to form a functional product which is usable today, but which can also be expanded upon in the future as technologies change or user requirements adapt. This thesis stresses the modularity of the different components, allowing developers to replace individual components without having to rebuild the entire website.

19 Workflow

The process of adding 3D models to a website through a database will require a specific workflow. Collecting information

The first part of the process is collecting all the necessary information. The plans and drawings for the 3D model, and the background information about the building or architect.

Creating the data

After all the information is collected, the 3D model has to be created according to some standards. The background information needs to conform to the database structure.

Filling the database

The 3D model and information will be uploaded to the database.

Linking the data

All the buildings, their architects and other contributors will be indexed in the database and the connections will be made.

Displaying on the website

After all the data is inserted in the database, the website will be able to display the buildings with their 3D models and extra information. All the connections that are set up in the database will be visible on the site. Users will be able to explore them in a visual way.

20 Modeller vs Curator vs User

This thesis will make a distinction between 3 main actors of the website. All these use cases have different needs so the website needs to accommodate all these people the right way. There will be a distinction between the public website where the buildings are shown, and the private site where the modellers can manage and add the information to the database. The modeller The person who will create the information and the 3D model that will be displayed on the website. For the modeller, it is important that there is a clear and easy way to add the data. It is also very important that any 3D model that will be shown on the website is made appropriately so it can be displayed correctly and efficiently. The curator The person who will decide what data is visible at a given time or place. This could be an exhibition manager that only wants to select buildings from a certain place or year. So it is important there is a way to find and select the data that will fit the exhibition. The end user The person who will use the website once it is built and contains information, either at an exhibition or at home. Their main concern is the user experience and whether the website is clear and easy to navigate. The website must cater to this user by making information easily searchable and providing an overall pleasant user experience.

Organisation of the thesis

This thesis consists of three parts: 3D model creation, database construction and website building. 3D model creation

The first part looks into how 3D models can and should be created; explain basic terminology and provide best practices for optimized display on websites. This is aimed at the modellers who intend to make these models for use on the website.

Database creation

The second part will go deeper in the choice and set up of a correct database, how to connect it to a website and how to create, read, update and delete data of that database.

Website creation

The third and last part looks into how to build a dynamic and responsive website which connects the models and the database to the website, how augmented reality can be implemented and some user experience best practices will be discussed. 21 Case Study

Everything will be illustrated by a selection of works by Le Corbusier. The data consists of text, images and 3D models. The 3D models are created for this thesis according to the best practices that will be explained in the next chapter. All the necessary information was gathered for the database integration.

Le Corbusier

Charles-Édouard Jeanneret, known as Le Corbusier, was a Swiss-French architect and one of the pioneers of modern architecture. Over his long career he has designed buildings around the world, many of those buildings have become world heritage sites.

• Nationality: Swiss, French • Birth date: 1887/10/06 • Death date: 1965/08/27

Villa Savoye

An icon of the modern architecture movement, Villa Savoye was built in Poissy, France in 1931. This elegant square building is lifted from the ground by a series of columns, providing a magnificent view in all directions. The design embodies the five points of his architectural philosophy. The building is raised on pilotis, allowing the ground floor to be used as an extra external space. The roof is filled with nature and terraces, giving the area back to nature. The house has a free floor plan, meaning the internal walls can be repositioned and the space can be ordered differently. The windows are horizontal to provide • Built date: 1931 a better view to the outside and giving • Architect: Le Corbusier enough surface for ventilation. Because • Client: Pierre Savoye the building was supported by columns, the facades could be freely designed without • Location: Poissy, Île-de-France the constraints of needing to transfer the loads.

22 The Philips pavilion

The Philips Electronic Company commissioned Le Corbusier’s office to design a unique experience for the thousands of visitors of the 1958 world exhibit in Brussels,. Together with his employee Iannis Xenakis, Le Corbusier created a multimedia event with projections, lights and music. The structure itself is a three-pronged tent made from precast concrete elements, held together by tensile strength from steel cables that are connected to reinforced concrete pillars. • Built date: 1958 • Architect: Le Corbusier, Iannis Xenakis • Client: Philips • Location: Heizel, Brussels, BE

The Chapel Notre Dame du Haut

This chapel, located on a hill in Ronchamp, was built in 1955, in Le Corbusier’s later life. The design is very organic and sculptural. The Chapel is made from large concrete walls. Embedded in these walls are large columns that support the roof. The walls have distinct openings that allows light to pass through in various ways.

• Built date: 1955 • Architect: Le Corbusier • Client: Association Oeuvre Notre-Dame du Haut • Location: Ronchamps, Haute-Saône, FR

23 Software used

Every component of the designed framework is created with open source or free software. This means that anyone wanting to improve or alter this study can do so without the need to purchase expensive software.

3D model creation and export Blender 3D

https://www.blender.org All 3D models are made and exported to the correct format in the software “Blender 3D”. This is the biggest open source 3D authoring software package available. Other software could do the same thing but these are generally very expensive. In the chapter about 3D modelling, most of the necessary information is written program agnostic to ensure it is applicable to most 3D creation programs.

Database creation and management phpMyAdmin

https://www.phpmyadmin.net This software is used to manage databases locally. It can be installed on the local server of the computer, which means that it can be accessed during the offline stages of the website.

MySQL

https://www.mysql.com This software is used to manage the database when it is hosted on the internet.

Website creation and hosting Apache

https://httpd.apache.org Apache is a local web server program that allows the website to be hosted on the local computer. This is very useful in the development stage as it allows for faster iteration and avoids online hosting limitations.

Heroku

https://www.heroku.com Heroku is a free online web hosting service that allows the website to be hosted online, making it accessible anywhere and on any device. This was very important since the website is mainly designed to be used on mobile devices.

Github

https://www.github.io Github is an online repository for code and programs. After the completion of the thesis, all the code and data can be accessed on the GitHub page set up for this thesis. 24 Creating 3D models for the web

The main way of interacting with the building on the website is through a 3D model. Users can rotate the model and look around the virtual space on the screen and are able to see the inside of the model. Translating the 3D information to the screen in real time requires three things: 3D geometry Materials Renderer

This is the polygonal This gives the 3D model This calculates the lights representation of the colour and texture. It also and shadows in the 3D real world object. Built determines the way the scene. This can be done out of virtual connected object will react to light. in a number of different triangles. ways.

Final Image

25 Modelling: Creating 3D geometry

Different types of 3D modelling

Making a 3D model involves manipulating surfaces and volumes in a virtual space. These models can be made in many different ways and software packages. We can divide these ways into 3 distinct modelling styles.

Solid modelling

Solid modelling makes use of basic 3D primitives such as cubes and spheres which can be combined in many different ways to create more complex shapes. Boolean operators (AND, OR, NOT, AND NOT) use logic to determine the combination of these shapes. These operators translate to difference, union and intersection to form the basis of this modelling technique.

Shapes Difference Union Intersection Because of its parametric nature, meaning parameters like size and scale of these primitives can be changed at all time during the modelling process, this method is mainly used in mechanical design and construction detailing where precision and flexibility are crucial. This way of modelling is not optimized for real time rendering, because the shapes need to be calculated by the Central Processing Unit (CPU) before the Graphics Processing Unit (GPU) can render the image.

Software that uses this kind of modelling: Solidworks, AutoCad,...

Surface modelling

Surface modelling is a way to manipulate different surfaces into 3D space. By pushing and pulling surfaces along straight lines or curves, more organic shapes can be made more easily. This way of modelling also allows for a large amount of precision and flexibility. It suffers from the same drawbacks as solid modelling but at a less severe scale. Objects are generally less parametric than solid models, but the CPU still needs to calculate the geometry before it can be displayed.

Software that uses this kind of modelling: SketchUp, AutoCad, Rhino,...

Mesh modelling

In Mesh modelling, the 3D model is a collection of vertices (points in 3D space), edges (connections between vertices) and faces (polygons that are created by a loop of connected vertices). Here each vertex can be manipulated individually to create complex objects very efficiently. The collection of the faces that make up the model is called the topology. Modellers should always strive to make the topology as simple as possible, while still retaining enough detail, as rendering speeds decrease with the number of polygons.

3D models can be created with vertices and polygons right from the start, however it is also possible to first make the object in a different way. At a later stage it is possible to overlay that model with polygons to replicate the shape. This process is called retopology. This gives the modeller freedom to create the model in the method they prefer. A well done retopology can make a model that was previously inefficient to render far more render-friendly.

Software that uses this kind of modelling: 3Ds Max, Maya, Blender,..

26 Modelling for real time rendering

Real time rendering uses a process called rasterization, this will be explained in the rendering chapter. This process is optimized for polygons because it calculates pixels based on the visible polygons. With this method, the number of polygons in a given scene are directly linked to the speed at which the scene can be rendered. The fewer polygons there are, the faster the scene will render. In other words, the 3D models should be meshes.

This process uses the most basic form of polygons, triangles. This does not mean modellers can only use triangles in their 3D model. Triangulation occurs at the exporting stage of model creation. So, modellers can also model in quads (four sided polygons) and N-gons (more sided polygons). Quads are generally easier to work with , especially in buildings where most of the surfaces consist of rectangles. After the 3D model is created, the quads get divided into triangles by the model exporter.

For the purpose of the web application, all models that will be used should be mesh based. Modellers can use other methods such as solid or surface modelling because of the higher precision that can be achieved, however this method leaves less control over the final topology. So it is advised to retopologize, or at least optimize the existing topology, after it is created. This will be explained in a later part about optimizing the full 3D model.

Modelling for the section plane

One of the features the website has, is the ability to activate a 3D section plane. This plane acts like a virtual knife that will cut through the model. Everything in front of that plane will be invisible. This allows the user to view the inside of the 3D model.

This means the 3D models can have some interior decoration which will be visible once the section plane is activated.

An important side note to this, is the fact that real time rendering has an optimization feature called backface culling. Backface culling means only the front of a polygon is rendered. In order to know what side of the polygon is the front, every polygon has a property called the surface normal. The normal indicates the direction and is a vector that is perpendicular to the polygon. When creating a 3D model, the surface normals should always face outwards.

The normal faces the camera, The normal does not face the camera, so the plane is visible so the plane is not rendered. In the case of this website however, backface culling has been disabled to prevent the inside of the model being invisible when the section plane is active.

27 Texturing: Creating a material

Texturing is the process of adding colour and material properties to the 3D model. Multiple textures are combined to create a material. Texturing can be done in a few different ways: procedurally or UV mapped. When a model is textured procedurally, all the textures are math- based. Rather than manipulating individual pixels like a 2D image, the texture is equipped with some rules and equations to form a resolution-independent image. Using random noise patterns, grids, and gradients, rich and complex textures can be generated. This process is fully parametric, so all the parameters can be changed to suit the modeller’s need.

The 3D modelling software Blender and other more dedicated texturing software such as Substance Designer use a node-based texturing system. Nodes are individual mathematical expressions that can be linked together. This is a visual way of designing the material, similar to the popular 3D plug-in for Rhino, Grasshopper.

If the modeller wants to control every pixel, the textures can also be created by painting or drawing on a flat surface which is then projected on the model. The location of the texture on the model is determined by the so-called UV map. This map is a representation of every polygon of the model layer flat on a single image. (see example) The term UV originates in the fact that rather than a x-y coordinate system, this uses an alternate u-v coordinate system, which is functionally identical while using a different namespace to minimize confusion.

This method also allows the modeller to apply textures that are not created by himself. Using predefined textures for standard materials, such as concrete or bricks, can save a considerable amount of time in the process of creating the model.

UV layout

As stated above, the UV layout is the 2D representation of every polygon of the model. The 3D model is unfolded to a flat plane by a process called UV unwrapping. Several ways to map these polygons exist, but most 3D programs have built in tools to translate the 3D model to a 2D plane. For this thesis Blender 3D was used, which has intelligent unwrapping algorithms to automate the process.

The UV layout should be preferable square and have a resolution in a power of two. Most used texture sizes are: 256x256, 512x512, 1024x1024 and 2048x2048 pixels. Graphics cards Use binary language to calculate pixels, and have many optimization features that are specifically build around dividing the textures by a power of two for faraway objects in a virtual scene.

28 Seams

The most important thing to keep in mind when making a UV layout is the correct representation of the polygon on a 2D plane. For most shapes, it is impossible to flatten the geometry without obtaining deformations of the polygons. This is why UV layouts have separations between the polygons. These separations are called seams and can be chosen freely by the modeller. For simple forms like a cube, this can be done easily. However, more complex shapes require more elaborate cuts, as shown below. Seams should be chosen in a manner that minimises the deformations in the polygons as much as possible.

It is also important to maximize the area of the polygons on the UV map. Using lower texture sizes will result in faster rendering times. As a consequence, the resolution should be as low as possible, while still retaining enough visual fidelity.

For continuous textures, the UV mapping becomes crucial to allow the texture to continue over the model. Seams will break this flow and result in visible discontinuities in the texture. Locating the seams is therefore important. Some tips are:

• Make seams follow hard edges. Here discontinuities are less noticeable. • Make seams where they are obscured by other parts of the model.

Checking whether the model is UV unwrapped well can be done with a generic chequerboard texture. This will accentuate the deformations and discontinuities in the texture and help the creator to optimize the UV map.

29 Overlapping UVs

Overlapping UVs are two or more polygons that are overlapping each other on the UV map. Generally speaking, overlapping UVs should be avoided, to keep the texture varied and allow every part of the polygon to be unique. This is especially important when the model will be used in a real time application, when some of the lighting calculations can be done beforehand and saved on the model as a texture. Overlapping UVs will result in lighting artefacts, whereby parts of the model will look like they belong somewhere else on the model.

However, there are instances where overlapping UVs come in very helpful. Parts that are repeated throughout the model, like a brick wall, can be mapped in such a way that every polygon of the wall is mapped side by side on the UV map. This will result in a much lower resolution per brick compared to the same couple of bricks repeated over the wall. This can be achieved by overlapping the UVs on purpose.

Overlapping UVs also work in another way than of placing the different polygons over each other on the UV map. In some cases, it can be useful for a repeated element in the scene, like a railing, to share a single UV map. This way the same texture will be used for all elements. In the example below, all the columns share the same UVs, meaning there is only one unwrapped column on the UV map.

UV Channels

Having overlapping UVs can save texture space and will result in a higher final resolution per polygon. However, this means that every polygon will no longer have a unique place on the UV map. Luckily, 3D models can have multiple UV maps, so different textures can be mapped in different ways. This will be explored further in the chapter on optimization of the 3D model creation.

30 PBR Materials

Beyond colour, real-world objects all have different properties which add up to rendering a realistic material. A brick and a soda can may both be orange, even if made from different materials. A number of other properties define how the material will respond to light. The unique combination of these properties will determine the degree of realism of any rendered material.

The standard workflow for materials is optimized for Physically Based Rendering (PBR). This means that all properties are different variations on how the incoming light gets reflected.

All these light reaction properties can be translated to textures. Most of them are expressed in a range from zero to one, which can be mapped to a grayscale image. Some properties like colour are full Red-Green-Blue (RGB) textures. This allows the creator to define these properties in a visual way and create them in image processing software like Gimp or Photoshop.

The most common properties are:

Metalness

This defines whether a material is metallic or not. This map mostly exists of black (not metal) or white (metal) values, nothing in between. As a result, the texture can be saved in a very small file size. It is important to remember that only the topmost layer of the material defines this property. A painted car for example may consist of metal, but because of the layer of paint on top, the material no longer behaves like a metal. It should therefore have a metalness value of zero.

Base Colour

This property is also known as diffuse or albedo. This defines the colour of the diffuse light that gets reflected. The base colour corresponds with what we perceive as the colour of an object. For metallic surfaces, since they don’t have a diffuse reflection, this map is used to define the colour of the reflection. Roughness

Roughness defines the microsurface of the material, since the amount of incoming light on an object should always be the same as the light being reflected back; the only varying parameter being the roughness of the reflection. When a surface is 100% rough, light gets reflected in all directions, resulting in a diffuse look. A perfectly smooth material (mirror) reflects the environment perfectly where the highlights are more focused. Most materials classify somewhere in between and have variations of roughness across their surface. This map is a grayscale image.

31 Bump

A bump map is used to give a surface more detail without having to model these details. Small bumps and holes can be simulated with a grayscale texture. This means the model can have fewer polygons, making it faster to render. The value for a given pixel of the bump map determines the height in which the pixel moves up or down.

Normal

This map too is used to give a surface more detail without having to model these details. This means the model can have fewer polygons, making it easier to render. Contrary to the bump map, a normal map can move the surface in three dimensions instead of just up and down. The texture is expressed in red (x) green (y) and blue (z), where every pixel moves according to the resulting colour. This allows for more possibilities, such as overhangs, as the movement of the pixels is not restricted to up and down but can also be sideways. Normal maps are preferred over bump maps, since they result in much more accurate displacements. However, the result is a larger file because it has to store three values per pixel instead of only one.

Ambient Occlusion

This map simulates the lack of light rays that are scattered in little cracks or corners of the model. Ambient occlusion increases the model depth as details and intersections are perceived easier. This result can also be achieved as a post-processing effect as well, which will be discussed in the next segment. When the renderer supports screen space ambient occlusion, the map does not have to be made. Emission

Some objects or parts of objects can emit light. This can be achieved by assigning the object an emission value. The emission texture attributes colour and brightness to the light object. Since it acts as a light source, emission overrides all other material properties. Objects that are emissive therefore don’t need definitions for other properties.

32 Not every material needs all described properties to look realistic and appealing. The modeller can choose which property to add to the overall material. If a material has a uniform value of a certain property, for example the material has no metal parts, then the material can just have a single float value for that property. This saves on texture space and increases the performance for display.

Below are some examples of the textures that were created for the 3D model of the Philips Pavilion. In this case, five properties were defined by textures; the others were simple float values. In general, the base colour, the normal map and the roughness have the most impact on the overall look of the material.

UV map Metalness Base Colour

Roughness Normal Ambient Occlusion

Texture baking

An important part of texturing is the baking of a texture. Some materials can be defined procedurally or can be a combination of overlaying different materials. When such models are exported, these special materials will most likely not look like as they were intended in the original software, because of the difference in texturing workflows between programs. That is why textures can be baked: a process whereby each pixel’s final appearance is calculated and then represented in a UV map. This can be done individually for each property. For the case studies, all materials were created procedurally and baked in the end. Most 3D modelling packages are equipped with built-in baking tools to facilitate this process. 33 Lighting: defining the lights

A vital part of the visualization of the 3D model is setting up the lights in a scene. Different real time lighting methods exist, each disposing of unique properties.

Real time lights

Real time lights are special scene objects that constitute sources of light. Some notable types are: Point light:

This emits light in all directions from a single point in space and affects faraway objects less (also known as falloff).

Directional light:

This emits light in a single direction without falloff, acting as a sun.

Spot light:

This emits light in a cone from a single point, meaning the directions the light is emitting is limited the inside of that cone.

Area light:

This emits light from a plane in space, this gives more realistic lighting as the light does not come from a single point in space, giving the light and shadows a softer look. Ambient light:

This emits a light equally on everything in the scene. This gives lights to parts that would otherwise be occluded from other light sources.

Image-based lighting

Instead of placing every light manually in a scene, a spherical image can be used to simulate a real environment. These images are 360° photographs that are taken with a very high dynamic range, meaning both very bright and very dark parts of the image are exposed properly. These images are also called HDRI’s (High Dynamic Range Imaging). This method results in realistic lighting with a minimum effort and also produces realistic reflections in the objects.

The resulting reflection from the left HDRI

34 Rendering: calculating the final image

When creating pre-rendered images such as animations or visualizations, the lighting calculation can be done in different ways. Precisely because time-consuming rendering methods are allowed, a very precise and realistic way of rendering called Path-tracing can be used. In contrast, real time applications don’t have that luxury.

Real time rendering

Because the models have to be intractable and react to the user input, they don’t have the luxury of carefully rendering each ray and need to be rendered in real time. This means the final image has to be rendered over the duration of one frame. Typically, computers refresh the screen 60 times per second. This implies that the image must be rendered in 1/60th of a second. If, in the context of this thesis, 3D models are to be rendered in real time, they should be optimized for this way of displaying. Modern computers can use a few different application programming interfaces (API’s) to display real time graphics. Common examples are: OpenGL, Vulcan, DirectX. Modern internet browsers have built-in support for viewing 3D models on any device. For that purpose an open source API called WebGL is applied. WebGL is a binding for OpenGL, which means it is built on top of it. These methods all use the GPU (Graphical Processing Unit) for rendering. The GPU is highly optimized for calculating images at a rapid rate and can thus produce these images at the required 60 frames per second.

At the moment of writing, real time rendering uses a process called rasterization to calculate the 3D viewport. Rasterization is a very efficient way to draw the image on the screen. It does so in two steps. First it projects all the triangles in the scene onto a 2D plane that is parallel to the screen. Then, it checks what triangle each pixel is located in and then gives the pixel the base colour of the triangle.

The polygons A pixel raster is created The pixels take the colour of the largest polygon

Further rendering can be done after the base colour is determined. This is called shading. The present-day standard for shading realistic scenes is to use a PBR workflow (Physically Based Rendering). as seen in the chapter about texturing. This rendering method is based on the physical principle of energy conservation. It implies that the amount of light that hits an object can never be greater than the amount of light that is reflected. All a material does, is determine the way that light gets reflected.

Currently, a lot of development is happening around real time raytracing. Instead of projecting the triangles on the 2D plane, the camera casts a ray for every pixel into the 3D space. It then calculates when a ray hits a triangle and at what angle it reflects a light source. This creates a more realistic image since it is based on the way photons interact in the real world. However, since this method is at present underdeveloped for mobile devices, we will not dive deeper into this topic.

35 Post-processing

After the shading of the model is done, there are a number of effects that can be applied to the 3D image to make it look more realistic and appealing.

No post processing

This is the output before any post possessing effects are applied.

Screen space ambient occlusion (SSAO)

This effect simulates darker areas in crevices and holes where realistic light rays would be unlikely to hit the surface. Because the rendering is done with rasterization instead of path tracing, this needs to be faked. SSAO gives the model a sense of depth and accentuates smaller details that would otherwise get lost. As the name suggest, SSAO only takes into account the parts of the object that are visible to the camera. If a large piece of geometry that would normally give some occlusion in its environment is just of screen, then it will not have any effect on the scene.

Bloom

This effect simulates a glow of very bright colours. By drawing a halo around the very bright object, the effect gives the illusion of the presence of an atmosphere that captures the light in a cloud around the object.

36 Screen space reflections (SSR)

This effect simulates realistic reflections from other objects in the scene. When rasterizing, reflections are based on the environment, meaning the other objects of the scene are ignored. This environment can be a spherical image or HDRI’s, which gives the objects more to reflect than a single sky colour. When screen space reflections are turned on, objects can reflect other objects that are visible to the camera. Like SSAO, this only affects objects that are visible at that moment.

Depth of field (DOF)

This effect simulates real depth of field by a camera. Every camera has a certain focal length and aperture that determines the amount of blur that occurs for objects that are out of focus. Lower apertures result in a tighter range of focus and thus a more pronounced blurring off out-of-focus objects.

Colour grading

This effect applies a certain colour filter over the rendered image. This is also called a colour grade, something which is usually used in movies to enhance the look of the image and create a distinct visual style.

37 Exporting: choosing a file type

When the model is ready to be exported for use on the website, a decision has to be made about what kind of file the model will be exported to. It is essential to use a universal file type that every device can read and display. Depending on what information has to be included, certain file types are more suitable than others.

Below is a table of the most viable file formats, each with their (dis-) advantages.

File Format File Extension Mesh Support Textures Animations Lights Alembic .abc Y Y Y Y Autodesk FBX .fbx Y Y Y N Collada .dea Y Y Y Y glTF .gltf .glb Y Y Y Y Obj .obj Y Y with .mtl N N Stl .stl Y N N N Universal scene .usd Y Y Y Y descrition Step .stp Y N N N

After comparing these file types, the GL Transmission Form (glTF) file format was chosen. It supports all the necessary features and was specifically created for real time rendering through the integration of many optimizations.

glTF files store textures in a very efficient way. The base colour, normal map and emission textures are separate files, because they require full RGB coverage. All three other grayscale textures are stored in a single RGB texture, where every colour corresponds to another property.

• Red: ambient occlusion • Green: roughness • Blue: metallic

Using this method, only four textures have to be saved instead of six. This decreases the rendering times because the renderer makes a draw call for every texture. Having fewer different textures means fewer draw calls.

There are three different types of glTF formats: .gltf, .gltf(embedded) and .glb. These are all variations on the same principles but come with some differences.

• .gltf: This is the basic form of the 3D model, which comes with separate texture files linked by a .bin file. The .gltf file is written in a JavaScript Object Notation (JSON) format, a coding language that can be easily understood and edited. This feature, in combination with the separate texture files, makes the .gltf file easy to alter after the file is created. It has a significant drawback in that it consists of several different files. That’s why, for this thesis, this format is not used. • .gltf (embedded): This is similar to the regular .gltf format in that it is written in JSON and can be edited after creation. It has an advantage over the other as it embeds all the textures in the .gltf, meaning only one file is used to display the model. This makes sharing and storing the model much easier. • .glb: This is also an embedded file format where the textures are stored in the same file. However, this file is written in binary, the most basic computer language. It is therefore less human-readable, so editing is very challenging. On the other hand, because of the binary language, it is much more efficient to store, so file sizes will generally be smaller. The .glb is also read faster by the computer, so it will render faster. 38 The choice of extension is up to the creator. The .glb format is used for all models on the website because of its efficiency. The embedded .gltf may also be used if the creator may need to alter the file after its creation.

Blender has a built-in exporter for glTF formats, which works with the standard texture set up and automatically converts all the textures to the corresponding glTF set up.

Optimizations

There are a number of small optimizations that can be applied to the 3D model in order to lower rendering times and to provide a faster and smoother experience. Especially on older devices where the computing power is not necessarily as high, every little thing can improve the performance.

3D model

Modellers should always try to use as few polygons as possible. There is no need to separate flat planes for small details like bricks or wooden planks. These details can be added with a normal map. The only part where the number of polygons is important is in the silhouette of the model. Because of the polygonal structure, curves will always be a collection of small flat planes. More polygons will result in a smoother curve. Modellers should therefore find a balance between the smoothness of the curve and the number of polygons. This is much more noticeable for large surfaces so smaller curves will need less polygons.

To optimize the topology, the model can be retopologized, by overlaying the existing complex model with new, much less, polygons. As a general rule of thumb, it can be stated that polygons should only be used to define the silhouette of the model. All surface details are then added with a normal map which simulates these small displacements by bending the normal vectors on that part of the polygon. In the example below, the windows are not modelled. They are created with a normal map to simulate the depth difference between the window and the wall.

The main focus should be the visible parts of the model. Adding small details like screws will not only be unnoticeable, they will also slow down the performance of the model.

39 Textures

The UV map can increase efficiency considerably. To maximize performance as few different textures as possible should be used and these textures should have the lowest acceptable resolution. This is because computer graphics calculate the final image for each texture individually, which it later combines into one. Instead of using a different texture for the different parts of the model, modellers should combine these textures onto a single texture atlas, which is a combination of all different textures in one image. The UV map should then link all the polygons to their corresponding texture on the atlas. This can be easily done by combining all the different parts of the model on one UV map and then bake all the different textures on the atlas. Duplicate items and surfaces with the same material can use the same texture space, resulting in overlapping UV’s, which can be a problem for properties such as ambient occlusion where duplicate objects don’t necessarily have the same amount of light hitting them. However, screen space ambient occlusion (SSAO) solves this problem: instead of using a texture, the shader calculates the ambient occlusion based on what is visible on the screen.

Generally, all polygons of the model should be scaled according to their 3D counterpart on the UV map, this will result in a consistent resolution throughout the model. However, there are always parts of the model that require more detail than others. Instead of increasing the resolution of the texture as a whole. Certain polygons can be made bigger on the UV map, so they take up a larger area on the map. This can be done by altering the scale of these objects in the scene. The polygons first have to be scaled up or down by a factor in which the UV size should scale. Then the scale can be normalized in the scene, meaning it has a scale factor of one, while still being smaller or bigger than it originally was. Lastly, rescale the polygons back to their original scale without normalizing the scale afterwards. This will result in that specific polygon to have a higher workable resolution than the other parts. For polygons that are hidden or less relevant, this lowering of resolution will not be noticeable.

The original generated UV map The optimized UV map where the ground cylinder is much smaller, resulting in more space for the walls.

Texture compression can be applied as much as necessary without compromising the quality. Different image file types are optimized for different kinds of images: Portable Network Graphics (PNG) images are used for textures with large flat colours and Joint Photographic Expert Group (JPEG) for images with more realistic textures.

40 Constructing a database for the web

When confronted with large amounts of data, managing all available information becomes a challenging task. This is why databases are used: they make it easy to store and collect data and it keep them organized. This segment will explore different kinds of existing databases and will look at what type of database is best suited for the proposed web application. This section also handles the structure of the database and what kind of data needs to be collected by the modellers.

What is a (relational) database?

A database is a structured collection of information that can link different types of data together. For example, when information about a certain group of people is collected, all the different pieces of data (such as, for instance, names or addresses) can be stored in a table. This table can be expanded by adding more people over time, or by adding additional properties for individuals.

All databases share a similar structure: they contain the type of data that can be entered (e.g. building, architect) and the input for that specific type (e.g. Philips pavilion, Le Corbusier). There are however differences in how these data types and inputs are stored. In general, two different structures of databases can be distinguished: SQL (structured query language) and NoSQL (not only structured query language).

Structured query language SQL

SQL databases imply a predefined structure that is very clear but difficult to alter after data has been entered into the database. This structure is called the database schema. The SQL database is very similar to an excel spreadsheet and consists of 3 basic elements: tables, columns and rows. The database schema is the whole of tables and columns, whereas the rows are the data itself.

Architect ID Name Birthday Nationality 1 Le Corbusier 06/10/1887 French 2 Jelle 26/06/1995 Belgian Row 3 ...... Column Table 41 Tables

Tables are collections of data, mostly limited to a single subject, for example people, buildings or places. Databases can have multiple tables that can be referenced by one another.

Columns

Columns are a single type of data that can be stored in a table. For example the name, birthday or age of a person.

Rows

Rows are the input data itself that can be entered at any time. For example, in a “people” table, every column could be filled in with the correct data for that specific person. For example, if there are three columns named: “Name”, “Birthday”, “Age”, the inputs should be: “Jelle”, “26/06”, “24”.

Relations

Tables can be more than just a collection of user input. They can also reference each other by linking whole rows of data in a single input of another table. For example, a table can contain information on buildings in which the architect is a column. Instead of just entering the name of the architect in the column, a reference to the row of that specific architect from a table of architects can be inserted. That way, the building can use all the information from the architect when needed. This also prevents the need to insert duplicate data when another building by the same architect has to be entered. This type of database is also referred to as a relational database.

Building ID Name Architect ID Built Date 1 Philips Pavilion 1 1958 2 Villa Savoye 1 1931 3 ......

Architect ID Name Birthday Nationality 1 Le Corbusier 06/10/1887 French 2 Jelle 26/06/1995 Belgian 3 ......

42 Three different types of relations exist in such database: one to one relations, one to many relations, and many to many relations.

One to one

A row can only reference one piece of data while that data itself can also only be referenced once. One example of this relation is when a website user has to be assigned a unique user-ID that will be stored in another table. In this case, it would be very undesirable multiple users to have the same ID or have multiple IDs linked to the same user.

One to many

In this type of relation, multiple rows can link to the same piece of data located in another table. This comes in handy in the case of buildings, where multiple buildings can have the same architect.

Many to many

Here, multiple rows can link to the same piece of data, while the row itself can also have multiple pieces of that type of data. For example, a building can have more than one architect, while architects can have designed multiple buildings in their careers.

Advantages

• Structured Query Language (SQL) databases are mature technologies with a very large user base. It is the most popular form of databases. • Since tables can establish relations, there is no duplicate data in the database. A piece of data can be altered without breaking the relation. • Because of its clear schema, this type of database is easily vertically scaled, meaning new input data can be simply added and deleted. • SQL Relational databases are also ACID-compliant (standing for Atomicity, Consistency, Isolation and Durability. This ensures that data remains consistent throughout software and hardware in case of a crash or data loss.

Disadvantages

• Because of the rigid schema, the database does not efficiently allow for the adding or removing of extra data columns. As a result, SQL databases do not scale well as data requirements change. • They are generally slower because of the focus on compatibility and data safety. • It can be difficult to transfer the data to another database since the schema needs to be exactly the same in order to be compatible.

43 Not only SQL

Contrary to SQL, NoSQL has no predefined structure. This gives the database a certain flexibility and makes it more lightweight. The reason for this is because empty fields don’t have to be made, whereas in a structured database all data columns have to be filled out (this can also be an empty value). This also means that after a certain point in time, if more information on other projects has to be entered, one can simply add an extra column without having to go back to the older data and add the new information. This added flexibility means the NoSQL database can be more difficult to manage.

Advantages

• NoSQL databases are generally faster than SQL databases. • Since they have no predefined schema, they scale horizontally very well. • Because of their flexibility, they are more suited for very complex databases.

Disadvantages

• It is much harder to set up relations between tables, since NoSQL databases don’t work with rows and columns, but rather with datatypes and input. • They have no built in ACID compliance. • They were more recently developed and are less widely adopted, so less documentation is available.

Choosing a Database Framework

MySQL

The choice for a particular database structure for the proposed application was based on the ease of use and availability of documentation. Because of the importance of the relations between different data types, a SQL database was chosen. More specifically, MySQL was chosen. MySQL is the second most popular database structure and is as such used by many large companies all over the world. MySQL is open source and very well-documented. It disposes of a graphical user interface which facilitates creation and management of the database tables. This proved very helpful during the development process, as the structure of the database was still evolving. The database was used during both the offline and the online stage.

NodeGoat

Another option that was researched is the Nodegoat platform. This is a web application that was developed to be easily managed by multiple people. It has a user interface through which multiple tables can be managed and data can be added in a number of different ways.

Integration in the website

In the following chapter about the website, both options will be explained. For the final product, MySQL was chosen as database framework; mainly because it provides an easy management structure that integrates well with the website. NodeGoat was not used in the website. However, a proof-of-concept connection has been established between the database and the website in case future projects prefer this type of database structure. 44 Database structure

In the context of a web application centred on building collections, the database has to contain all the necessary information concerning buildings and architects: the building name, the architect and the 3D model, built date…

Buildings

The database should have a clear structure. The most obvious initial table should be buildings.

• ID • BuildingName • Architect (Link to architect table) • Client (Link to client table) • BuiltDate • Description • BuildingLocation • ModelName (The file name of the 3D model) • ImageName (The file name of the building image)

Architects

For the architect, only basic biographical information is considered sufficient in the context of this thesis: important data such as country of residence, birth year and year of death can be used to connect different architects to each other. This table can be expanded in the future with other attributes to create more nuanced and less obvious connections.

• ID • ArchitectName • Nationality • BirthDate • DeathDate • Description • ImageName (The file name of the architect profile image)

Clients

In the future, the database can be expanded with additional tables that contain other relevant actors in the lifecycle of the building (such as contractors, structural and technical engineers,…). As proof of concept , a table for the client was added.

• ID • ClientName • Nationality • BirthDate • DeathDate • Description • ImageName (The file name of the client profile image)

45 Collecting and storing information

When all necessary information has been gathered, it needs to be stored in the database. To keep the database small in size, the full 3D files and images were not uploaded directly into the database. This makes the database smaller and easier to manage. Larger files can be uploaded directly to the server where the website is hosted, and the database only saves the reference to that file.

During development stages, the maximum upload size of file to the server was 2 MB, so all 3D models had to be optimized extensively. However, this proved a good practice to maintain when this limitation no longer exists, as it makes the experience much smoother and more reactive.

As explained in the next chapter, a submit page was created for this purpose. Instead of having to add all data directly into the database and set up the connections between the architects, buildings and 3D models, a form was made to streamline this process and prevent errors during the input stage.

Keeping it up to date

To facilitate future management of the database, a mechanism could be added to the website. This would allow to update existing items the database, or to remove items when they are no longer required or relevant. Such management features can be added in the future, but is at this time not included in the web application.

46 Building the website

The end product of this thesis is a website in which the case studies are used as an example. This section will explain the back-end structure and all the individual pages that make up the website.

This website is fully functional and available to the public as a proof of concept. The appendix contains different QR codes that link to the website itself. The reader in encouraged to follow along the working website while reading this chapter.

All the source code is also available in a public repository on Github.

Link to the website Link to the repository

https://thesisjelle.herokuapp.com https://github.com/Jellevermandere/Thesis

First, this chapter will go into the technical set-up for the website. After that the integration of the database, the 3D models and the augmented reality are explained. These parts can be rather technical. Lastly, the resulting website and the individual pages are laid out.

47 Dynamic websites

Most basic websites are created page by page, in a static way, whereby all information on the pages is inserted by hand. If something needs to be changed, this has to be done manually and repeated for every single page. Other websites are more complex and have more recurring parts. Youtube, for example, contains over 7 billion videos. It would be impossible to code every video page manually as a unique web page. To solve this problem, websites such as Youtube use standardized pages that are updated, based on content that is stored elsewhere in a database. The same principle will be applied to this website.

As stated in the previous chapter, the website will be linked to a database. Instead of writing every entry from the database statically in the website, one template page is provided for every type of data. This template will be filled based on what building or architect the site is currently referring to. This allows the website to be infinitely expandable in the future without having to modify the underlying code.

Responsive websites

Because the website aims mostly for viewing on mobile devices, it needs to work on and be readable for a large number of screen sizes . The user interaction with a website on a mobile device also differs substantially from a computer. Instead of having fine mouse controls, touch input is less precise, so larger buttons are required. The screen ratio is also opposite: instead of horizontal based navigation, a mobile device is vertically focused and tends to have singular elements stacked on top of each other.

Building separate websites for these different platforms would be repetitive and redundant work. This is why in recent years, with the surge of mobile platforms, responsive websites have become the norm. Responsive websites are set up to work on all sizes and shapes by setting rules for different elements. Content can be displayed horizontally on a larger screen while stacking them vertically on a much smaller screen. This means the website only has to be designed once to be compatible with all devices.

48 Choosing the coding languages

The website needs to be both dynamic and responsive. Because of this, The website has to be coded in multiple coding languages.

HTML

All basic websites use HTML as their coding language. Every page has the same basic layout:

..the page text..

..

Html uses nesting to set up and connect different elements: the head section is used for the title of the website and other invisible elements that make the different parts of the page work. The body part is where all content of the webpage goes. It uses a markup system of headings and body text to define a hierarchy.

This website uses HTML for the static parts of the pages.

PHP

For the dynamic aspect of the website (the part based on a database) we cannot just use basic HTML. A language like PHP or JavaScript will help us to create dynamic websites and allows to use extra frameworks in order to display the 3D model and enable user interaction on the page.

Interacting with a database requires a CRUD (Create, Read, Update and Delete). In this website, PHP was used, as it is a well-documented language that is easy to use and implement. PHP works very well with HTML. Both languages can be used in the same file, meaning that the static parts of the website can remain in simple HTML. When using PHP code in a HTML page, the code needs to be enclosed by special PHP markers, as shown below.

PHP has a set of logical operators and loops that make it easy to locate and display data from a database. Basic functions like “IF” and “ELSE” together with “=, <, >, !=“ allow for data to be displayed if it matches certain parameters.

PHP also offers the ability to change the content of the database directly from the site itself by sending commands to the database. This comes in very handy for adding new information without having to access the database itself. For this purpose, a dedicated submit page was made.

49 Bootstrap

HTML was created to display content on a page. However, it has only limited layout and graphic possibilities to customise the look of the page.

Most websites use CSS (Cascading Style Sheets) to alter the design of the page. CSS is a ruleset that lays out the website based on those rules. In order for our website to be responsive, extra rules need to be added that instruct the page how to lay out the website for a given resolution. Instead of coding these responsive rules ourselves, the open source framework Bootstrap is used. Bootstrap stands for a “mobile-first” approach, ensuring the website will be readable even on the smallest devices. It uses a grid-based system, where all elements are structured in columns and rows.

The creator of the website can define rows that can hold any type of content. Rows are automatically placed underneath the previous row and a website can have an unlimited number of rows.

These rows are divided in 12 virtual segments. Any number of segments can be combined into a column. These columns are displayed next to each other, until their combined width reaches 12 segments, in which case the next one will be displayed underneath.

Row 1 12 width column Row 2 6 width column 6 width column Row 3 2 width column 4 width column 3 width column 3 width column

The amount of segments of a given column can be changed based on the size of the webpage. This is where the responsive functionality of Bootstrap becomes clear. A column can be given 6 segments for a wider page, where it will take up half of the width. That same column can then be given the full 12 segments when the page needs to be displayed smaller. This way, the column will take on the full width off the page.

This thesis uses Bootstrap to ensure that the right content is prominently visible for any resolution. Smaller screens will focus on the 3D model, and all subsequent data will be placed below in a single column structure. This works well for most mobile devices because of their vertical nature and intuitive scrolling.

Larger screens, such as laptop and desktop computers will see the entire content on the same screen. Because of the larger screen area and more horizontal aspect ratio, the 3D model and the associated data can be displayed side by side without the constituent parts getting too small.

The same website, but with different screen sizes.

50 Creating a working environment

While under development, the website needs to be displayed somewhere in order to be tested. This was done in two stages: an offline stage during the early development phase and an online stage for the later iterations and final product.

Setting up a local server

Both the database and the website need to be hosted somewhere in order to work. This allows a device to interpret the code and convert it into a visual website.

To minimize the complexity of hosting and storage, the database and website were hosted on a local server during development. The local server was hosted on the computer itself. Any device that has access to the IP address of the server can then access the website.

For this thesis, an Apache server with phpMyAdmin was installed to interface with the database. Apache is a local server client that was designed for hosting websites. PhpMyAdmin is an additional program that collaborates with Apache to host the database. Both were selected because they offer support for PHP, something most other servers don’t have.

Setting up an online server through Heroku

https://www.heroku.com With the local development largely completed, the website was transferred to an online location. Since most web servers don’t support PHP encoding on their servers, not just any hosting service proved suitable. Ghent University offers free web space to all students, however their hosting service does not support server-side encoding, which is indispensable if the website uses PHP. This is why, in order to set up our website, we need a special server that can translate PHP code into code that can be viewed by the end user.

Heroku (see “Software used” or the official documentation for more information) is a platform designed for intelligent web hosting and supports PHP encoding, which makes it a good choice for our application. Files can be uploaded with the aid of the command line interface (CLI) as shown below.

$ cd thesisjelle

$ git add . $ git commit -am “update message” $ git push heroku master

Furthermore, MySQL can be enabled with a free add-on through Heroku called ClearDB MySQL.

https://www.cleardb.com/ It offers the same functionality as the local Database management tools and it is hosted online. Hosting the database through the same service as the website itself makes the overall application more manageable and less dependent on other services.

51 Database integration

The most important part of the website is the content from the database. This content is laid out in modular containers and adapts to the specific data from the database. These modular parts can be used throughout the website to prevent duplicate code. The container’s information varies based on what page is currently displayed. For example, only buildings for which the architect matches the current architect page will be displayed.

Establishing the connection

Before the website can interact with the database, it needs to establish a connection. This requires a PHP command which is included on every page that requires a connection to the database. The PHP command varies, depending on the location of the database. When proceeding from an offline to an online database, this link had to be changed to match the corresponding online location.

$dbServername = “localhost or url to database server”; $dbUsername = “username”; $dbPassword = “password”; $dbName = “databaseName”;

$conn = mysqli_ connect($dbServername, $dbUsername, $dbPassword, $dbName); ?>

Instead of writing this for every page, PHP has a function called:

This function allows files to include other files as code into pages. This way, the code only needs to be written once and can be referenced wherever needed. This also comes in very handy for recurring pieces of code throughout the site and has been used many times in the code of the website.

Interacting with the database: CRUD

To interact with the database through PHP, four basic functions are needed: Create, Read, Update and Delete (CRUD).

Create

Adding new data is done with an insert statement

$sql = “INSERT INTO a database ( database columns ) VALUES ( column values );”; mysqli_query($conn, $sql);

52 Read

The reading of certain data is done with a select statement:

$sql = “SELECT * FROM a database WHERE a row = ‘ a value’”; mysqli_query($conn, $sql);

The query statement sends the command to the connected database as shown above.

Certain columns of the selected row can then be referenced with:

.row[‘column name’].

Update and Delete

Updating and deleting can be done with similar commands; they are however not used in the website at present.

File renaming system

Before the files get uploaded to the website, it is good practice to rename them. This will prevent different files having the same name and creating conflicts in the database. First the page collects the formation from a form where the name of the building is entered. Then it renames the files to match the building name and moves the file to the correct directory.

$fileDestination = ‘../gltf/’.$fileNameNew; move_uploaded_file($fileTmpName, $fileDestination); echo “ file upload success”;

$imageDestination = ‘../images/buildings/’.$imageNameNew; move_uploaded_file($imageTmpName, $imageDestination); echo “ image upload success”; .. ?>

53 Alternative option: Nodegoat

During testing, a Nodegoat connection was also set up. This method works differently from the SQL counterpart. Instead of directly sending commands to the server, the Nodegoat integration requires a connection with the online application which sends a JSON file as a response. This file will remain unchanged as long as the site remains active by the user. The returned JSON file can then be read and filtered by the PHP code.

The JSON file can be referenced as stated below:

$url = ‘https://api.nodegoat.ugent.be/project/xxx/data/type/xxxx/ object/’; $ch = curl_init($url);

curl_setopt($ch, CURLOPT_HTTPHEADER, array(‘Content-Type: application/ json’, ‘Authorization: Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxx’)); curl_setopt($ch, CURLOPT_TIMEOUT, 5); curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_HEADER, 0);

$JSON = curl_exec($ch); $data = json_decode($JSON); curl_close($ch);

NodeGoat has the advantage of being an established platform that is used throughout the academic world. Using this connection can therefore be useful for future expandability, as the datasets get increasingly large and they become difficult to manage by a single person. NodeGoat allows multiple users to manage the database at the same time.

Because the connection with the database is done through a static JSON file, it becomes very difficult to manage the database through the website. All the data, including the 3D models, would have to be added through the NodeGoat application. The website would then serve as a pure visualization tool, instead of the management tool it is now.

54 3D model integration

The 3D models that are linked to the database need to be displayed on the website. A correct and smooth experience is essential ,so a lot of thought and experimentation has gone into developing the 3D viewport that will display the models.

Testing different Technologies

Many different 3D model display technologies, each with their respective pros and cons are available. In the context of adding AR functionality as well, some options are better suited than others. Different options were tested to see which one provides the required functionality for this use case. All options tested are based on WebGL, the web standard for 3D.

Sketchfab

Sketchfab is a 3D model repository, developed to share and view 3D models. It is widely used across the world and disposes of features specifically designed for making 3D models looks as good as possible. Many of these features involve post-processing, as explained in the chapter about creating the 3D model. There is also full control of model lighting, through the possibility to use a spherical image or HDRI. This will, as explained before, provide more realistic lighting compared to virtual lights. Creators can also use custom a light set up with different types of lights, such as point lights or infinite lights. Such virtual lights offer the advantage of being able to generate hard shadows, which is not possible with spherical images in real time rendering. Luckily, both can be used together to get the benefits of both systems.

A screenshot of the editor in Sketchfab where different properties can be altered.

Sketchfab offers a number of means to create custom interactions, such as annotations and built-in animations of the model. Options also exist for the user to view all the textures and UV layout, as explained in previous chapters. Even if this is useful for creators wanting to download and edit models, this doesn’t provide any real added value for the typical viewer of architectural buildings.

55 Sketchfab has a dedicated app where the models can be viewed in augmented reality. However since this feature only works within the additional locally installed app, it is not usable for this thesis.

In the end, Sketchfab has its focus on making the 3D models look as appealing as possible. It does have great visual enhancement features, however it lacks the freedom that is needed to implement the AR and interaction features. Most importantly, it has no way of displaying 3D models directly, without uploading them first, from a 3D file stored on a server.

Unity 3D

A screenshot of a application, where the user can walk through the pavilion in first person and explore the multimedia exhibition of the Philips Pavilion. This is made with Unity. Unity is a full game engine which offers the creator of a game full control over the final product. Anything from the 3D model to the renderer itself can be programmed to suit the needs of the creator. Unity can provide a viewport with high visual fidelity and features such as live sectioning

However, building a Unity app for the web through WebGL makes the app difficult to link to databases. Unity WebGL is, to this date, also not officially supported on mobile devices. The unity WebGL app also doesn’t load automatically meaning an interaction is required before the scene can start loading.

JavaScript Frameworks

JavaScript is a coding language developed for web interaction, making it an ideal choice for an interactive website. Over the years, multiple open source frameworks have been developed using JavaScript to enable 3D on the web. Notable examples are:

• Babylon.js • Cecium.js • Three.js

All three are all very similar codebases, making it rather difficult to choosing between them. In the end, Three.js was used to test, more specifically because of the existence of A-Frame, a framework built on top of Three.js. Three.js has a very intuitive interface to create 3D scenes from simple HTML objects, making it easy to use for anyone with basic HTML coding knowledge

Another reason to choose for Three.js is the built-in integration of VR. With a simple switch, the view can be transformed into a stereo view to use with VR goggles. A-Frame has a large user base and a well fleshed out documentation section. This made experimenting very easy and allowed to combine existing features into a useful whole in the context of this case.

56 Another advantage of Three.js are the multiple extra frameworks that are available. One of them is AR.js, which offers the possibility for models to be viewed in Augmented Reality by using the same structure as A-Frame. This will be explained in the camera page section. This will be explained in the “Integrating augmented reality” section.

Coding in JavaScript offers as much freedom as Unity, since games can also be created. The downside to JavaScript is that it is less user friendly, as it has no visual user interface and requires a more thorough coding and development knowledge to get started. For more complex features, such as a live sectioning, we can rely on the underlying technology of Three.js. This has useful clipping features that can be used to selectively render certain parts of the model.

Programming the 3D viewport

In the end, the A-Frame framework was used because of its ease of use and multiple available frameworks. A-Frame works object oriented, meaning objects are placed in a virtual scene and properties are defined within the object.

Basic scene

The basic set up for an A-Frame scene is:

..

Here, all entities should be placed like

All these entities can be assigned components that in turn affect various parameters. Things like location, rotation and scale are some of these parameters.

The basic scene consists of a glTF model (this is the model that will be visible), a directional light (this lights the scene and provides shadows) and a camera (this allows to see what is in the scene). the glTF model is placed in the scene by first fetching it from the database link as shown below:

..

57 Beyond this basic set up, two existing extra features were implemented. These are both existing frameworks, found on various platforms. The full source code can be found on the thesis Github repository.

Orbit controls

https://github.com/supermedium/superframe#readme Orbit controls is an existing JavaScript framework that allows the user to interact with the 3D viewport with the help of a mouse or touch input. The object remains in the centre of the screen, while the camera orbits the object based on the user input. The component is linked to the camera as shown below, it requires a number of parameters to be defined.

Resize

https://stackoverflow.com/questions/51380941/aframe-size-of-model Because the buildings are made with real world dimensions, they can vary widely in scale. Therefore, the resize component was used to standardize the size of the models. It works with two parameters, the “axis”, and the “value”. The “axis” defines what axis (x, y or z) to base the scaling on. The “value” defines the size of the model based on the axis. This method allows all buildings to be roughly the same virtual size, meaning the camera can remain in place throughout the different models. The component is added to the model as shown below.

58 Programming the section plane

Creating a 3D section of a virtual building can be done with a rendering technique called clipping. Clipping means not rendering certain parts of the scene. It is mostly used by the camera to increase performance by not rendering very far objects. However, clipping is not limited to the camera. It is possible to define what parts of the model are visible (not clipped) based on its relative position to a plane. In this case the clipping plane. Everything that is behind the plane will be clipped and become invisible.

This plane is available in the underlying framework of A-Frame, Three.js. However, in order to be able to use this clipping plane in the A-Frame scene, a component has to be written to translate the Three.js code into something A-Frame can understand. This is done as shown below:

AFRAME.registerComponent(‘clippingcontrol’, { schema:{ normal: {type: ‘vec3’, default: {x: 1, y: 0, z: 0}}, constant: {type: ‘number’, default: 0} },

update: function() { .. var renderer = this.el.sceneEl.renderer; var globalPlane = new THREE.Plane( data.normal, data.constant ); .. renderer.clippingPlanes =[globalPlane]; .. } });

Here, we can see a “globalPlane” is created and added to the renderer’s list of clipping planes. The component is then used as shown below where “normal” indicates the vector direction of the plane’s normal vector and the “constant” determines the offset of the plane from the centre of the scene. This constant can be altered by a slider to move the position of the plane.

59 The viewport

The end result is a simple scene, where the correct 3D model is displayed, based on the current page. There is a slider which controls the position of the section plane. There is also a “VR” button in the bottom right corner. This button activates the first person VR experience, which can be viewed with some basic cardboard VR goggles.

The 3D viewport on the building webpage.

The 3D viewport where the section plane is used.

60 Augmented reality integration

Augmented reality

Different technologies exist that can provide an AR experience. In this research, the following well-known frameworks were tested and checked for usability and compatibility.

3D object Coding Free to Application OS Markerless? Marker? marker? env. use? Vuforia Multiplatform Y Y Y Unity3D Y IOS, A9 chip or Unity3D / AR kit Y Y Y Y newer Swift Android 7.0 & IOS AR Core Y Y Y Unity3D Y 11 or newer Amazon In browser Y N N In browser N Sumerian AR.js In browser N Y N JavaScript Y 8th wall In browser Y Y N JavaScript N

These frameworks can be grouped in two different categories: Marker-based and markerless AR. As the names suggest, marker-based requires an image (marker) to be tracked by the device in order to calculate its position in the world, while markerless uses the whole camera feed and constructs a virtual 3D point cloud in which the device can orient itself.

Marker-based AR

Marker-based means that the device’s camera looks for a very specific image. Typically, this image contains very high contrast and is rotationally asymmetric, so it looks different from different angle rotations. Because the app knows what the image looks like head on, it can calculate the rotation and scale of the marker, based on the perspective deformation. When the app knows how the marker’s rotation, it can place a virtual object on the screen with the same transformation as the marker. This technology is less complex and easier to calculate than markerless AR, because it has a clear understanding of the position of the marker. It is therefore available both on more and less powerful devices. As it does not require data from the device’s accelerometer, it can also be used on computers and laptops with their integrated webcams. This comes in very handy during development as these apps are built on computers.

The most important part of this technology is the quality of the marker. Different frameworks use various ways to calculate and validate the markers. This will be explained for each framework individually.

Markerless AR

Using 3D point cloud data, the same concept of a marker can be applied to the real world itself. When a mobile device moves around, it creates a cloud of points in the real world. This allows for the creation of a complex approximation of the room. The room itself then acts as some sort of marker; this comes with some inherent advantages. Because the room is mapped in 3D, the scale of the environment can be calculated. This allows virtual objects to be scaled according to their real-world dimensions. Taking in the full camera feed also allows the device to create a virtual environment texture. This texture can be used to create more accurate reflections that mimic the real world. At the same time some environment light estimations can be executed. This is an important feat, since lighting realism constitutes a huge part of the added realism of markerless AR. 61 Despite looking very promising, this method is computationally more expensive and at the moment of writing not yet available as open source web framework. On the other hand, this technology however provides an even less technical overhead, as no marker are required. Google Chrome’s support for AR Core is currently in development and an AR kit is already available in MacOS Safari with “IOS-Quicklook”. We can therefore expect this technology to become more accessible through web browsers in the near future.

This should be taken into account in further development of this website, as markerless AR clearly offer advantages over marker based AR.

Testing different technologies

During this research, many of these frameworks have been tested, the most notable were: Vuforia, ARCore, Arkit and AR.js. All of these frameworks have their advantages and disadvantages.

Vuforia/ Unity3D

Vuforia is a built-in Augmented Reality feature in the Unity3D gaming engine. Its functionality is mainly based on markers, meaning the content will be locked into the image in 3D space. Vuforia markers are generated based on images that the user can provide himself, for example a floor plan or picture of a building. There are a number of best practices to ensure that markers perform well and are easily recognized by the device. Markers are then processed and converted to patterns on the website of Vuforia. The generated markers can then be downloaded and imported into Unity3D.

During testing, this procedure provided a very well-trackable and credible image. It also allowed the use of all the other features in the Unity3D game engine. The same features that were tested in the regular viewport of the website could be implemented here. A live section demo was developed as a proof of concept, which is also implemented in the final website.

Proof of concept of a working Vuforia application where multiple section planes can interact with the 3D model, each manipulating a different layer of the model.

Unfortunately, Vuforia is currently not supported by the WebGL builder of Unity3D. This means it cannot be used for the purpose of this thesis. It did provide a useful first step towards the section plane, because the same principle could be applied to the website renderer.

62 ARCore

ARCore is the augmented reality framework made by google. It is currently used by the more recent AR apps available on the Google Play Store. Sketchfab has a dedicated app where the 3D models that are available on its site can be viewed in AR by devices that have support for ARCore. Being a recent development by Google, only a limited amount of devices currently support ARCore.

As of yet, ARCore is not yet supported on the web. Google is currently actively working on implementing this as a standard feature in its Chrome browser. For the meantime, this is not compatible with the website. But this could be an interesting improvement in the future.

Arkit

Arkit is the Apple variant of Google’s ARCore. Contrary to ARCore, this feature is already supported on the web, but only on Apple devices.

In collaboration with the animation studio Pixar, Apple has developed a new file format that was created to display and share augmented reality models. The .usdz file format is similar to the .gltf file format adopted for this thesis’ website, as it was also developed for fast display on the web. Apple developed an automatic script that converts .gltf file into .usdz files. This means that, theoretically, .usdz files can be implemented into the website as a future feature. The .usdz file can be shared between devices and do not require the use of any apps to view the 3D model in AR.

It can display the 3D models in the real world with very high fidelity. As it stands, it is one of the best AR experiences currently on the market.

The main drawback of this file format is that, again, it is only available on Apple devices. So, until that changes, .usdz files are not supported by the website.

AR.js

As stated in the 3D viewport chapter, A-Frame is used for the basic viewport. This was partly chosen because of the existence of an open source framework called AR.js. As the name suggest, AR.js is a framework that allows augmented reality through A-Frame. It is based on the open source framework ARToolkit. ARToolkit was developed 1999 and has been kept up-to-date since then. It provides very efficient marker tracking so even old and slow devices are supported.

AR.js uses special markers that are created for the ARToolKit web library. These markers have large black borders which provide a high contrast image to track. The image inside the black box is then identified to determine the specific marker. It is possible to create custom markers, however it is more limited than the Vuforia markers. To the left is an example of the default “Hiro” marker.

63 Programming The AR experience

Because AR.js is built on top of A-Frame, the same syntax is used to code the scene. AR functionality can be added to the A-Frame scene through the “arjs” component as shown below. The 3D model is then placed inside a “a-marker” entity. This enables the scene to follow the position of the marker and place the 3D model in the real world.

Adding the QR code reader

https://github.com/cozmo/jsQR Because all the buildings use the same basic page, where the correct building is shown based on the database filter, they all use the same basic marker. This makes the progress of creating markers much more easy. To differentiate between the different buildings, each marker has a unique QR code, which links to the correct page of that model.

This QR code can be read by the camera app of the user, however once the user scans a QR code, he is taken to the website. If the user then goes to another marker, the wrong building would show up. To combat this, a QR code reader was also added to the web page. This way, if the device sees a marker with a QR code that does not match the current page, it will go to the correct page.

The QR code reader was added through the open source framework jsQR.js. Some alterations were made to better suit the need of the website. As shown below, the website url is collected with some PHP code.

After the url is collected, a JavaScript script checks if the url matches the url that is detected by the QR code reader. If it is different, the website will go to that page.

64 Programming the QR code generator

To automate the process of creating the marker for the AR experience, a JavaScript script was used. This script is called AR-Code. It combines a automatically generated QR code with a AR.js marker as shown below. For this website, the default “Hiro” marker was altered to a custom marker. The resulting QR code is generated, based on the active page.

The default AR-Code, based on the “Hiro” The AR-Code generated by the website,. preset. Philips Pavilion The AR-Code is generated in three steps. First, The QR code is generated based on the building name.

function generateQrCodeImage(text, onLoaded){ var qrcode = new QRCode(container, { text: text, width: 256, height: 256, }); var qrCodeImage = container.querySelector(‘img’) qrCodeImage.addEventListener(‘load’, function(){ onLoaded(qrCodeImage) }) }

Second, the QR code is placed in the marker preset.

function generateArCodeCanvas(canvas, text, onLoad){ generateQrCodeImage(text, function onLoaded(qrCodeImage){ context.drawImage(qrCodeImage,canvas.width*0.50,canvas height*0.30,canvas.width*0.20, canvas.height*0.20); }) }

Third, the AR-Code is displayed on the website.

function updateARCode(){ var urlQrCode = `website_url/buildingAR.php?title=${buildingname}` generateArCodeCanvas(canvas, urlQrCode, function onReady()) }

65 The AR experience

The AR page has a full screen camera view, that view checks for markers and QR codes. If the page detects a marker, it will display the current 3D model on the marker. The marker also contains a QR code. If that QR code does not match the current page, the website will change to the correct page.

Villa Savoye

66 Website Structure

After the integration of the database, 3D viewport and Augmented reality. This part focusses on the lay-out and usability of the website.

The website is designed around two main use cases. The first use case is a visitor of an exhibition who scans a QR-code and is directed to the corresponding AR building page. Here the camera is the main interface and the 3D model will be viewable in AR. Additional information will be displayed to complement the 3D model. The camera page will be an integral part of the website. It is therefore important to always have a direct link back to the camera when the user starts browsing the rest of the website for more information.

The second use case is when the website is visited elsewhere when no QR codes are available to link directly to buildings. Here, a clear navigational system should be in place, allowing the user to explore the website without any problems.

Different pages Home page

The first page every user who stumbles upon this website, through a search engine or url, will see is the home page. The page should explain the context of the website and make the discoverability of the buildings its main feature. This can be done by highlighting the search functionality and showing the collection of buildings.

Search page

All data pages need to be easily discoverable. Therefore it is important to have a fully functional search page. On this page, the user will be able to filter and search all buildings and people with the aid of different parameters.

Data pages

Another page is dedicated to every type of data from the database. This includes a person page, where a person is shown with all necessary information and all buildings this person is involved with, and a separate building page, which displays the building itself and all persons who have worked on it.

AR page

When the user enters the site through a scanned QR code, the main focus is the AR experience. In this case the camera remains the main focus of the page. Additional information about the building or architect is still available, but only secondary. It should also be mentioned that, if the user accesses the site away from the potential exhibition, the camera page serves no function unless a QR code is acquired in another way (by downloading the QR-code from the building page, or through a publication). This is because the AR functionality relies of the QR code to work.

Submit page

New data into the database can be added through this page. It is hidden from the public to prevent false inputs.

67 The navigation bar

The navigation bar should always remain visible to the user. It is therefore prominently placed on the top of the page at all times. The navigator holds a link back to the home page, a direct search bar and a link to all buildings, architects and more.

There is also a link to the camera page, if users have access to one of the valid QR codes.

The navigation bar, like the website, is responsive. This means that smaller devices will see a collapsed navigation bar with a “hamburger” menu that expands into the full menu when clicked.

The home page

The home page is the first page users will come across. It explains the intent and functioning of the website. The search button is located at the centre of the page, similar to the Google homepage.

Below the search button, all available buildings and architects are displayed; this allows users to quickly explore the data.

68 The search page

The search page serves as a filter page for all available data. It can be reached either by entering a search term on the home page, or directly from the navigation bar. The most basic form of filtering is searching for a certain value. In this case, the search function checks if a word is found anywhere in the database and, if so, returns the corresponding row of data. For example, if one were to search for the name of an architect, that architect and all his works show up.

Another way of filtering would be by the use of ranges, i.e. looking for buildings that were built in a certain period in history or within a specific distance from a given point of reference. Such functionality can be added in a future expansion.

The person page

The person or, in the given example, the architect is the main focus of this page. Essential biographical data about age, nationality and background should therfore be clearly visible.

Other than this basic information, all buildings connected to the architect appear below. These are all linked to their corresponding building page.

Other people who have worked together with the central person on the page will also be shown. This allows the user to further explore and continue to browse other architects and buildings.

69 The building page

A 3D view of the building is the most important part of the building page. It is therefore prominently placed on the top left of the screen. The building information is displayed aside. The architect also needs to be prominently featured as the designer provides a way to explore connections with other buildings on the site. Extra information, such as the location of the page, is displayed below, in case the user wants to know more about a building.

On the bottom of the page, the marker for the AR experience is displayed. This allows users to use another device to scan the QR code and activate the AR. It is also possible to download and print the marker for use in exhibitions for example.

On a mobile device, all different information containers are stacked below each other. This makes all parts more readable and better suits the vertical display of a mobile device.

The AR page

The main page for the mobile user is the camera page. This page connects to the QR-codes and thus directs the user directly to that page. When the device is pointed at the marker, the corresponding 3D model will show up at the marker location. The model will then be tracked to the marker, offering the perception of it “really being there”.

Further information about the building appears and the user can explore connections and other facts about the building or architect by clicking through.

70 The submit page

On this page, the creator can submit all the needed for a given building or architect. It is therefore important that everything is clear and is protected against invalid data.

Two different forms are available, the person and the building form. Since the building requires information about the architect and client, these should be added to the database before the building data can be entered.

The person submit form enables the creator to insert all necessary information about that person. After the form is submitted, all information is converted to the corresponding datatypes of the database and is uploaded to the online database. With this process completed, the building can be added. The creator must select the correct architect from the ones available in the database. If the architect is added correctly, the new building should be available in the menu.

After the architect and building are correctly added to the database, they will both become visible in the website.

This page was active during the development phase of this Thesis. However, after the completion, this page was taken down to prevent misuse in the future. The source code is still available on the Github repository.

71 Conclusion

The finished product

With everything put together a fully fledged website is obtained. The website has achieved the main challenge of adding 3D models to a relational database. Many different frameworks had to be brought toghether. The end result is a seamless experience where viewing 3D models “just works”.

The demo website can be visited in two main ways.

The first being through a QR code that is located somewhere at an exhibition or demo environment. This makes viewing these models as easy as possible. The markers generated for the three case studies are available in the appendix.

The second way is to find the site through Google or by directly entering this url:

https://thesisjelle.herokuapp.com Link to the website

72 Workflow system

The process of adding 3D models has been streamlined to ensure anyone can add information and 3D models to the site. The only technical part the modellers and other creators have to interact with is the submit system. Everything else is automated in the back-end of the website.

• The buildings, architects,... and their information are added to the database. • The 3D models and pictures are uploaded to the server and renamed appropriately. • New web pages are generated based on the data from the database. • The data automatically scales and is laid out to fit all screen sizes. • The marker and QR code for the AR web page are created and can be downloaded instantly.

Export to .gltf

Blender3D Creating the 3D model Upload through the submit page 3D model Submit .gltf file Export to blender

Buildings stored Other 3D software Information on server Creating the 3D model Data added to database

Existing Database Thesis Database Server For example Nodegoat Made for this website

Other databases can be linked instead

3D models and information is displayed on the website

Users can enjoy the AR experience Access through QR codes 73 Applications

The website can be used in a number of different environments.

Exhibitions are the main application for this website, since it was first concieved with this application in mind. The user friendly interface and low barrier of entry provice an ideal product to provide to a visitor. The augmented reality intergration also gives the website some appeal to the mass audience that otherwise could have little interest in the buildings.

Another important use case is the archiving and research possibilities of the connected database. Places like government facilities, architecture offices, EBP companies, facility managers, heritage care,... can use this website to store the building information and to communicate more visually. The added value of interactive 3D models could allow for a more complete collection of information and a futureproofing of the buildings in question. There will be no more need to store large physical scale models. The website makes it possible to share this information very easily and without the need of extra software.

Future expandability

This thesis encourages future expansion of this website. Because everything is modular and open source, parts can be switched without much coding work.

The technology that supports this website will inevitable become outdated and should be improved in the future. The advancements in AR technology will allow the 3D models to place in the real world without the need for static markers. This will expand the AR section of this website out of the exhibition context and into the living room.

The database will over time be filled with data. Not all information has the same value or correctness when uploaded to the site. That is why a complete database management system should be created instead of a simple upload feature.

74 Appendix

References

Websites

A-Frame. (n.d.). A-Frame – Make WebVR. Retrieved January 10, 2020, from https://aframe.io

Ayusharma0698. (2018, October 26). Difference between SQL and NoSQL. Retrieved January 10, 2020, from https://www.geeksforgeeks.org/difference-between-sql-and-nosql/

Caulfield, B. (2019, April 11). Whats the Difference Between Ray Tracing, Rasterization? | NVIDIA Blog. Retrieved January 10, 2020, from https://blogs.nvidia.com/blog/2018/03/19/whats- difference-between-ray-tracing-rasterization/

Dj, W. (2018, March 19). How I built a content management system for WebVR using A-Frame. Retrieved January 10, 2020, from https://medium.com/mozilla-tech/how-i-built-a-content- management-system-for-webvr-using-A-Frame-cd3b5a128c6b

Eden Networks Ltd. (2017). Marker generator. Retrieved January 10, 2020, from https:// au.gmented.com/app/marker/marker.php

Engadget. (2019, February 11). Google Maps AR First Look: Helping you navigate the city [YouTube]. Retrieved January 10, 2020, from https://www.youtube.com/ watch?v=XWbY5jdJnHg&feature=youtu.be

I. (2017, October 9). What does ACID mean in Database Systems? Retrieved January 10, 2020, from https://database.guide/what-is-acid-in-databases/

IBM. (2019, December 16). Relational Databases. Retrieved January 10, 2020, from https://www. ibm.com/cloud/learn/relational-databases

JSConf. (2018, February 8). Jerome Etienne: How To Do Augmented Reality On The Web - The AR.js OS Project - JSConf.Asia 2018 [Youtube]. Retrieved January 10, 2020, from https://www. youtube.com/watch?v=NBrM7YVznzk&feature=youtu.be

Mapbox. (2018, January 17). Multi-user AR experience. Retrieved January 10, 2020, from https:// blog.mapbox.com/multi-user-ar-experience-1a586f40b2ce

75 Marmoset. (2018, November 1). Physically-Based Rendering, And You Can Too! Retrieved January 10, 2020, from https://marmoset.co/posts/physically-based-rendering-and-you- can-too/

Paracuellos, A. (2019, June 17). Unboring.net | Workflow: How to make a PBR 3D model for the web. Retrieved January 10, 2020, from https://unboring.net/workflows/how-to-make-pbr- web/

Porterie Notre-Dame du Haut . (n.d.). Colline Notre-Dame du Haut, Ronchamp - site officiel / A nonprofit private property. Retrieved January 10, 2020, from http://www. collinenotredameduhaut.com/discover/a_nonprofit_private_property.1594.html

Schmelyun, A. (2019, March 23). So you want to get started with AR.js. Retrieved January 10, 2020, from https://medium.com/@aschmelyun/so-you-want-to-get-started-with-ar-js- 41dd4fba5f81

Scratchapixel. (n.d.). rasterization practical implementation. Retrieved January 10, 2020, from https://www.scratchapixel.com/lessons/3D-basic-rendering/rasterization-practical- implementation?url=3D-basic-rendering/rasterization-practical-implementation

Three.js. (n.d.). three.js docs. Retrieved January 10, 2020, from https://threejs.org/docs/index. html#api/en/renderers/WebGLRenderer

Ugochukwu, H. (2017, August 30). deploying a php and mysql web app with heroku. Retrieved January 10, 2020, from https://scotch.io/@phalconVee/deploying-a-php-and-mysql-web- app-with-heroku

Books/ Papers

Curtis, W. J. R. (1996). Modern Architecture Since 1900 (3rd ed.). London: Phaidon Press.

Duckett, J. (2011). HTML and CSS: Design and Build Websites (1st ed.). Indianapolis: John Wiley & Sons.

Duckett, J. (2014). JavaScript and JQuery: Interactive Front-End Web Development (1st ed.). Indianapolis: Wiley.

Pharr, M., Jakob, W., & Humphreys, G. (2016). Physically Based Rendering: From Theory to Implementation (3rd ed.). Burlington: Morgan Kaufmann.

Wever, P. (2015). Inside Le Corbusier’s Philips Pavilion: A Multimedial Space at the 1958 Brussels World’s Fair. Rotterdam: nai010 publishers.

Kato, h, & Billinghurst, m. (1999). Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System. Retrieved from http://www.hitl.washington.edu/ artoolkit/Papers/IWAR99.kato.pdf

76 Frameworks

Cozmo. (n.d.). cozmo/jsQR. Retrieved 20 November 2019, from https://github.com/cozmo/jsQR

davidshimjs. (2015, November 25). davidshimjs/qrcodejs. Retrieved January 10, 2020, from https://github.com/davidshimjs/qrcodejs

Ethienne, J. (n.d.). AR.js Marker Training. Retrieved January 10, 2020, from https://jeromeetienne. github.io/AR.js/three.js/examples/marker-training/examples/generator.html

jeromeetienne. (2020). jeromeetienne/AR.js. Retrieved January 10, 2020, from https://github. com/jeromeetienne/AR.js/blob/master/README.md

RmR. (2018, July 17). Aframe: size of model [Forum post]. Retrieved January 10, 2020, from https://stackoverflow.com/questions/51380941/aframe-size-of-model

NGOKevin. (2018, June 17). npm: aframe-orbit-controls. Retrieved 11 January 2020, from https:// www.npmjs.com/package/aframe-orbit-controls

Images

All images are created by the author except:

(p.18.2) Google. (2019). Google test nieuwe Maps-functie met augmented reality [Photograph]. Retrieved from https://www.bright.nl/nieuws/artikel/4605501/google-maps-augmented- reality-ingewikkelde-situaties

(p.18.3) Ikea. (n.d.). IKEA Apps [Photograph]. Retrieved from https://www.ikea.com/be/nl/ customer-service/shopping-at-ikea/ikea-apps-pub1fb256d1

(p.22.1) Wikipedia-contributor. (2019). Le Corbusier [Photograph]. Retrieved from https:// nl.wikipedia.org/wiki/Le_Corbusier#/media/Bestand:Le_Corbusier_(1964).jpg

(p.34.6) Town Square 2. (n.d.). [Photograph]. Retrieved from https://hdrihaven.com/hdris/ category/?c=all

Source code

Link to the repository All the source code is available on the github repository. Everything, except the mentioned frameworks throughout the thesis, is written by the author.

https://github.com/Jellevermandere/Thesis

77 Example catalogue

These are the generated AR-Code markers that can be used for the augmented reality page.

Villa Savoye Notre Dame du Haut

Philips Pavilion

78