Thesis Submitted to Florida Institute of Technology in Partial Fulﬁllment of the Requirements for the Degree Of

A Mixed-Reality Approach for Cyber-situation Awareness

by Tapas Dipakkumar Joshi

A thesis submitted to Florida Institute of Technology in partial fulﬁllment of the requirements for the degree of

Master of Science in Computer Science

Melbourne, Florida December 2017 We, the undersigned committee, hereby approve the attached thesis.

A Mixed-Reality Approach for Cyber-situation Awareness

by Tapas Dipakkumar Joshi

Marco Carvalho, Ph.D. Major Advisor Dean, College of Engineering and Computing Executive Director, Harris Institute for Assured Information

William Allen, Ph.D. Committee Member Associate Professor Harris Institute for Assured Information

Muzaﬀar Shaikh, Ph.D. Committee Member Department Head Engineering Systems

Philip Bernhard, Ph.D. Department Head Associate Professor Computer Sciences and Cybersecurity Abstract

Title: A Mixed-Reality Approach for Cyber-situation Awareness Author: Tapas Dipakkumar Joshi Committee Chair: Marco Carvalho, Ph.D.

As the proliferation and adoption of smart devices increase, there is an unprece- dented amount of data being released into the network every second. Computer networks are the carriers for the movement of this ever generating amounts of data, but are dumb in a way that they do not distinguish between suspicious data traffic and valid data traffic. Also, for however secure a computer network be, suspicious network traffic activities are bound to happen. These suspicious traffic activities have to be detected, analyzed and stopped before it compromises the entire network and leaves the information and data security of an organization in mayhem. There are several tools and visualizations for cybersecurity awareness at the disposal of security analysts and network operators. These helpful tools often lack in having the property of being intuitive and cognitive. Using the amalgamation of Mixed Reality, localization and mapping techniques, this thesis researches and implements a way of augmenting network traffic information right on the top of every computer system. The availability of network data rightly where it needs to be tagged and right when it is needed will help security analysts to better understand network activities and also locate the infected system and take quick actions if need be. In this thesis, we design and build a prototype that demonstrates how Mixed Reality provides better situation awareness to cyber operators managing a physical space.

iii Contents

1 Introduction 1

2 Background 4

2.1 Virtuality Continuum ...... 4

2.1.1 Virtual Reality ...... 5

2.1.2 Augmented Reality ...... 8

2.1.3 Mixed Reality ...... 10

2.2 General Applications ...... 11

2.3 Applications to Security ...... 13

2.4 Types of Mixed Reality Interfaces ...... 14

3 Literature Review 20

3.1 Traditional Visualization Techniques ...... 20

3.2 Information Filtering ...... 25

3.3 Situation Awareness ...... 26

4 Proposed Approach 30

iv 5 Implementation 35

5.1 Initial Setup ...... 35

5.2 Augmenting Information ...... 42

5.3 Data Retrieval ...... 44

5.4 Integrating Components ...... 46

5.5 Designing Prototype ...... 51

6 Case Study 54

6.1 Scenario Setup ...... 54

6.2 Scenario Observations ...... 58

7 Conclusion 60

v List of Figures

2.1 Visualizing data using virtual reality [3] ...... 6

2.2 First virtual reality system: The Sensorama which was release in 1950s [6] ...... 7

2.3 Virtuality Continuum ...... 10

2.4 Mediated Reality ...... 11

2.5 Microsoft Hololens (view from left-side) ...... 15

2.6 META 2 (view from right-side) ...... 16

2.7 Moverio Pro BT-2200 (Front view) ...... 17

2.8 Asus’ Windows Mixed Reality Headset ...... 18

3.1 Visalert Visualization [16] ...... 22

3.2 Gephi Visualization ...... 23

3.3 Deep Dive feature from DAEDALUS-VIZ [20] ...... 24

3.4 Courtesy of Naval Research Lab Information filtering: Unfiltered view (left) and filtered view (right) [7] ...... 25

3.5 Augmented Reality Situation Awareness in U.S. Military ...... 26

3.6 ”The Super Cockpit” by U.S. Air-Force [17] ...... 27

vi 3.7 WEAR step-by-step assistance ...... 28

3.8 Model based cyber defense situational awareness visualization . . . 29

4.1 Model of our proposed approach ...... 33

4.2 Proposed Approach Scenario indicating network operator wearing Mixed Reality Headset along with using traditional tools for network data monitoring ...... 34

5.1 Unity Editor Windows [42] ...... 36

5.2 Unity Script Lifecycle and Execution order [41] ...... 40

5.3 Orthographic vs Perspective Projection [4] ...... 41

5.4 Flowchart representation for initializing main camera and enabling auto-focus ...... 41

5.5 QR code as Image Target Behavior ...... 42

5.6 How data is collected and then processed in logstash pipeline through various ﬁlters ...... 45

5.7 Kibana displaying data for a dummy query ...... 46

5.8 Empty UI elements aka UI Brackets ...... 48

5.9 Mapping Longitude and Latitude on the globe using mathematical conversions from Latitude/Longitude to Cartesian coordinates . . . 49

5.10 Coroutine: MoveOverSeconds() ...... 50

5.11 Screenshot of development build of Step 1 transition ...... 51

5.12 Live Prototype for Step-1 Transition ...... 52

5.13 Live Prototype for Step-2 Transition ...... 52

vii 6.1 Flowchart showing the execution of bash script for getting IP addresses from MAC addresses ...... 56

6.2 Lab setup with QR codes stuck to the monitors of each machine . . 57

6.3 Visual augmentation of information of suspicious machine (View from far distance) ...... 58

viii Acknowledgements

I would like to thank my advisor and mentor Dr. Marco Carvalho whose wisdom, expertise and optimism were a gift throughout these two years.

I express my profound gratitude to my parents, my sister Mansi Shukla and my brother Siddharth Joshi for providing me with unfailing support and encourage- ment throughout these two years. This achievement would not have been possible without them.

Finally, I would like to thank my friend and colleague Evan Stoner for helping me with the network data retrieval; Nima Aghli, who helped me with several localization techniques for this project; and Ravi Pandhi who helped me evaluate and correct the thesis document.

ix Chapter 1

Introduction

With the advent of ’Internet of Things’, the number of devices that are being connected to the internet is increasing more than ever. These devices present in the computer networks record and transfer data about even the minutest changes in the surrounding environment. The massive quantities of data generated as a result provide the organization with ample insights in order to make better decisions. Also, with an increase in the number of devices, a number of potential target points for network intrusion also increases. An organization, when developing a product, should always strive to ensure that all the updated security measures related to the development and distribution of a product are followed. But we cannot solely rely on default configurations, antivirus software or firewalls as self-awareness and alertness are the keys to mitigating any virtual threat. Of course, precaution is better than cure, but it is always better to prepare for the remedies beforehand just to be safe in case network gets infected. A security analyst should not go around searching for the best network tools and visualizations in times when the network is already flooded with suspicious activities. Also, because advances in technology don’t always bring about the corresponding security advances that are needed to protect the advanced systems, it is logical to re-analyze the network and its vulnerabilities that can be exploited. Cybersecurity is designed to protect device hardware, software from any unauthorized access and computer networks from any

1 threats caused by suspicious network traffic. Organizations with becoming more ’connected’, should also ensure that they do not lag behind in the adoption of modern cybersecurity technologies. There will always be a possibility of network breach incidents even if the latest security practices are followed. These network breach incidents have to be stopped before the network infection becomes rampant and spreads through the entire network. Network administrators and security analysts use Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) to discover new network threats and automatically prevent them. IDS will discover abnormal traffic/packets as it is continuously monitoring the network while an IPS will prevent any known attacks from occurring. However, due to the great com- plexity of larger networks and thousands of alerts generated by IDS, security analyst needs to evaluate all alerts and therefore, the need to smartly visualize the network information escalates. In the past, security analysts used command line tools to monitor data and mitigate threats but as time passed, visualization tools that provide network information in a visual form started to take birth. Later on, intrusion detection/prevention systems were built along with antivirus tools but again they visualized data in forms of graphs and charts and finding a single machine that is infected and taking necessary actions before the threat intensifies was almost impossible. Several pieces of research also showed a mapping of cyber operations to individual machines using network maps and inventory databases which were more beneficial than traditional visualizations.

We propose an approach which uses SLAM (Simultaneous Localization and Map- ping) and extended tracking methods to improve above-mentioned tactics to map individual machines to their location and smartly augment meaningful ﬁltered information based on the needs of a network operator.

The rest of the thesis is organised as follows: Chapter 2 contains the background of all the terms and technologies used throughout this thesis. Chapter 3 contains the literature review of all the work done in the past for cyber-situation awareness and mapping the digital space to physical space. Chapter 4 describes our proposed

2 approach with a comprehensive overview and our detailed model. Chapter 5 contains the implementation of our prototype along with our detailed architecture and ﬂow diagrams. Chapter 6 describes the case study performed by us and Chapter 7 concludes our entire thesis.

3 Chapter 2

Background

In this chapter, we go through the history, general applications and cyber-security applications for all components of Virtuality Continuum. We also compare diﬀerent mixed reality interfaces choosing the most eﬃcient interface for our study.

2.1 Virtuality Continuum

Reality is the condition of things as they exist, as opposed to as they may show up or may be envisioned. Reality incorporates everything that is and has been, regardless of whether it is detectable or fathomable. A still more extensive deﬁni- tion incorporates what has existed, exists, or will exist. There are several theories and hypothesis which questions the existence of reality like the Simulated Reality hypothesis which claims that we all might be in a computer simulation or virtual reality which was either made by humans (from future) or higher power. Plato in “Allegory of the Cave” argues that reality (one perceived through our senses) is just an opinion and real knowledge is gained using philosophical reasoning [14]. However, for the simplicity of this thesis, we would consider reality as everything that exists, that is and has been whether its observable or not. Let’s look at some types of virtual realities that exists.

4 2.1.1 Virtual Reality

Virtual reality (VR) is a technology that makes use of virtual reality headsets and it is sometimes combined with physical spaces or multi-projected environments in order to create real images, sounds and other sensations that help stimulate the user’s physical presence in a virtual or imaginary environment [39]. Some VR systems are incorporated with transmissions of vibrations and other sensations or other devices known as Haptic systems. The current development of the virtual reality is very advanced, but the development has really slowed in recent years. This is more than likely due to public disappointment, further due to the high expectation created by media sensation. The development in the virtual world combine the power of 3D graphics and the Internet, giving users the ability to create new versions of themselves literally within a virtual world. Virtual redefines the idea of communication, expanding it from people you know in real life to anyone connected to the internet. This virtual means that relationship will begin to be formed more based off interest than general location. In the virtual, many different fields can use it to train people in a way that would usually put them at risk. This is used most notably in the fields of medicine, law enforcement, architecture and aviation. Those who do not have access or cannot afford this technology will be left out and could further separate social classes in information age based societies. Mostly the virtual reality makes tourism and visiting sites easier giving a detailed view of the original place like virtual tour. Virtual reality provides a chance to experience things that are impossible in real life as if you were there like fighting zombies in a virtual reality game where the environment is so real that sometimes a user cannot tell what is real or virtual. Figure 2.1 shows a man wearing a virtual reality headset and information is visualized through advanced techniques.

History of Virtual Reality

Early attempts were made at developing VR with the panoramic paintings or 360- degree murals in the 19th century. These paintings were designed to ﬁll the viewer’s mind with the image created and make them fell present in a historical event or

5 Figure 2.1: Visualizing data using virtual reality [3] scene. Charles Wheatstone’s in 1838 proved that the brain processes the two- dimensional image from each eye into a single image of three dimensions [10]. When the two stereoscope images or photos are viewed side by side or through a stereoscope it gives the user a feel of depth and immersion. The “Link Trainer” was developed by Edward Link in 1929 and it is regarded as the first commercial flight simulator which was completely electromechanical [22]. The motors controlled the rudder and steering column to change the pitch and roll. Morton Heilig in the 1950s created the Sensorama (Figure 2.2) which was designed as an arcade-style theatre cabinet that will excite all the senses [29]. It also had stereo speakers, a stereoscopic 3D display, fans, smell generator and a vibrating chair. All these were incorporated to make the user feel completely immersed in the film. In 1960 he

6 developed the Telesphere Mask which was the first head-mounted display (HMD) [11]. The headset had a wide vision with stereo sound and a stereoscopic 3D. Ivan Sutherland and Bob Sproull in 1968 created the first VR head mounted display also known as the Sword of Damocles and it was connected to a computer. The VR was bulky and terrifying for any user to happily wear it hence it was hung from a ceiling. Myron Krugere in 1969 developed a series of experiences which he called ”artificial reality” in which he created a computer-generated scene that responded to people in it [29]. The projects were named Glow flow, Metaplay and Psychic space which resulted in the creation of VideoPlace technology. In 1980 the Eye Tap - Miniaturization was created by the Steve Mann. It was wearable, and it was a heavy backpack computer that was connected to a helmet cam and view finder [28]. It coupled a beam splitter to transfer a scene to the computer and the connected camera which allows the overlay of real-time data.

Figure 2.2: First virtual reality system: The Sensorama which was release in 1950s [6]

Jaron Lanier in 1984 created the ﬁrst commercial VR system which included the DataGlove [44] which allowed the users ﬂexibility to turn virtual objects that showed

7 in the EyePhone heads up display. The term Virtual Reality was ﬁrst used in the mid-1987 by Jaron Lanier the originator of the VPL research as he began to develop the gear which included the googles and Data gloves which was important in order to experience VA.

In 1995 the CAVE was created by students at the University of Illinois and it used stereoscopic LCD shutter glasses and wall projections to create a three-walled room that the user could walk through. A lightweight VR vision was developed, and this was groundbreaking in that it allowed multiple users enjoy the same experience. NASA has been leading the VR dream alive for the past four decades and they made advancements in combining LEDs, liquid crystals displays and wide angle optics. The early part of the 21st century has seen remarkable improvements in the development of VR. The increase in smartphones with high density displays and 3D graphics has helped in the creation of lighter and more practical VR devices. The video game is one of the largest consumers of VR and these has increased the demand for consumer VR. Large corporations such as Google have launched interim products such as Google Cardboard, Samsung have taken it a step further by creating the Galaxy Gear.

2.1.2 Augmented Reality

Ivan Sutherland created the ﬁrst head-mounted display in 1968 but it was heavy, so it had to be suspended from the ceiling hence the name ”Sword of Damocles” [30]. During the 1980s and 1990s improvement in computing performances where required in order for AR to be established. During the 1970s and 1980s Scott Fisher, Dan Sandin, Myron Kruger amongst others experimented with diﬀerent concepts of human interaction with computer generated overlays on video for interactive art experiences [33].

8 In 1992 the term ”Augmented reality” was discovered. It was ﬁrst showed in the work of Caudell and Mizell at Boeing which was aimed at assisting workers in an airplane factory by showing wire rolls assembly schematics in a see-through HMD [13]. In 1993, Fitzmaurice developed the ﬁrst handheld spatially ware display which was a predecessor to the handheld AR [15]. It consisted of a handheld LCD screen. The LCD showed the video output of the SGI graphics workstations and was spatially tracked using a magnetic tracking device.

A fascinating medical AR was tabled at the Chapel Hill of the University of North Carolina in 1994, in which a physician can carefully examine a fetus directly within a pregnant woman [21]. Even though they are challenges when viewing such a de- formed object, this shows that considerable effort is being put on the power of AR in the field of medicine and other areas. Rekimoto and Nagao in 1995, developed the first handheld AR display [36]. The NaviCam [35] used was connected to a workstation and it was fitted with a front facing camera and from the video feed colour coded markers could be seen in the camera image and a display information on video transparent view. Schmalstieg et al. in 1996 created the Studierstube which was the first collaborative AR system [19]. In this system multiple users could have the feeling of virtual objects the same environment. Every user was tracked with the HMD and each person’s view point was different through the stereoscopic images. In contrast to multi-user VR, regular communication signs such as voice, body posture and gestures where not in any way hindered in Studierstube. There was a smooth flow because the virtual content was added to a predictable combined situation in the best minimal way.

The 21st century has brought us to an era where the development of cellular devices and mobile computing is huge. Wagner and Schmalstieg in 2003 presented the very ﬁrst handheld AR system that runs separately on a personal digital assistant. It was not until several years later that the ﬁrst usable natural feature tracking system for smartphones was developed.

9 2.1.3 Mixed Reality

Mixed reality is the convolution of virtual and real worlds which creates a whole new environment where the real and virtual objects coincide. It is real-time environment where all physical and digital objects are interactable. Mixed reality is a subclass of Virtual reality where a completely artiﬁcial world is present to interact with. The idea of Mixed Reality can be deeply understood by Virtuality Continuum [31] which assorts several classes of objects in real and virtual worlds. The very ends of Virtu- ality Continuum are the Real and Virtual environments which consists of real and virtual objects respectively. The very left side of the Continuum is Real environment which consists of all the physical objects and very right side of the Continuum is Virtual Environment which strictly consists of all digital objects. Augmented Reality and Augmented Virtuality also resides between the real and virtual environments. Therefore, Mixed Reality is a combination of all the four variables in Virtuality Con- tinuum making it the hybrid reality. Virtual objects are augmented in the physical world and users can fully interact with virtual objects without losing the sight of real environment and accompanying users. This allows users to interact with real world as well as digital objects, enabling real-time communication with users as well as virtual objects/information [9].

Figure 2.3: Virtuality Continuum

On the other hand, Mediated Reality is an older concept where one’s perception of reality can be altered by adding or removing information in real environment using smartphone or wearable computer initially proposed in 1994 by Mann [27]. Me- diated Reality had uses like ”Digital Eye Glass” and ”EyeTap” [27] which provided

10 users an aid to ﬁlter several information in their environment. Not much research was conducted on the long-term uses of Mediated Reality and therefore, long-term exposure to this ”perceived” reality is an area of ongoing research. But we can say for sure that Mediated Reality is a superset of virtual reality, augmented reality, mixed reality and diminished reality [26] as well.

Figure 2.4: Mediated Reality

2.2 General Applications

VR, AR and MR are frequently used in business operations as part of the overall digital transformation strategy. There has been a surge in the adoption of these realities among different enterprises. They offer business a chance to host con- vincing presentations in an immersive real way that is cheap way of developing a product or service. It has been assumed that if UK’s retail markets make use of these realities their businesses could increase as much as a billion pounds because it affords the customers the opportunity to virtually place products in their homes before buying a product. AR, VR and MR give business an opportunity to build, sell and examine their products virtually instead of physically. The use of VR and

11 AR gives unhindered access to the flow of information. Non-analyst cannot have the access, navigate and understand 3D visualization. It enhances productivity and adaptability. This technology allows the user to be fully immense in data without the need to see a single data which saves a lot of time. It is very persuasive as users can interact better with a 3D image which appears real instead of black and white digits on paper. Large corporations such as Samsung, Sony, Microsoft, HTC, and Google have discovered the potential it carries and they have been making signifi- cant headway. The advantages of VR, AR and MR are endless to enterprise broader digital transformation. For the transition to new technologies for businesses high bandwidth that produces immense connectivity is going to be very important to the success of the business. They can be classified as next generation networks which are packet based networks which are mobile and they can both be used for tele- phony and data. Various other applications include Architecture, Medicine, Sports, Arts and Entertainment.

When an experiment is being carried out and it is found out to be too dangerous, expensive or impractical, VR is used to simulate such circumstances. Whether it is the trainee fighter pilot or trainee medical surgeons, VR allows us to have an insight into the real world experience. Although there are different types of VR systems they all have the same characteristics as they allow the user to view three- dimensional images. They also change as the user moves around the scene which matches the change in their field of vision. There are various types of virtual reality systems but they all have the same purposes such as the ability to allow the user to view three-dimensional images. The users have a near real life experience of these images. The images changes as the person moves around which changes with their field of vision. The purpose is for a seamless change between the person’s imagination and eye movements and the right response. This makes it is both realistic and pleasurable. A virtual environment should give the right response- in real time- as the person discovers their environment. Problems arise when there is a delay between the user’s actions and system response which then causes a disruption of the process. The user then becomes aware of the artificial environment and ad-

12 justs accordingly which results in an unreal form of reaction. The purpose is for a natural, form of interaction which results in a pleasurable yet meaningful experience.

2.3 Applications to Security

In today’s era, it is almost impossible to be entirely secure. As we consume more Internet and virtual information, risk of getting hacked and losing personal information also increases. Due to the introduction of Dark (deep) web, catching a hacker is getting harder and diﬃcult every day. Therefore, we need security for our digital data the same way we need security for our precious physical items. Cybersecurity deals with the protection of digital machines for it’s hardware, software as well as its data [18]. As the world is shifting towards cloud computing and digital trends, it is a compulsion that we should secure our system with the best security architecture. Implementing security as a key factor while developing any software or a program is one of the most eﬃcient ways for protection. However, as time goes by, new methodologies and techniques are discovered every day and therefore external security measures are essential in order to keep the systems updated. Even after following all the measures and architecture, it is not guaranteed that our systems are fully secure. New vulnerabilities are discovered every day and therefore the initial step to remove vulnerabilities is to discover them.

Cybersecurity is like an insurance policy, the more you pay the better you get. How- ever, we also cannot afford to spend an infinite amount on Cybersecurity. Not too much and not too less, therefore a balance needs to be maintained. A sound business practice would be to keep balance between the cost of recovering an attack and cost of measures to moderate a risk [12]. It is also predicted that cyber-attacks are likely to increase with time as digital trend surges. Therefore, Cybersecurity needs to improve constantly in order to put up a fight to the new yet challenging

13 cyber-attacks.

To enhance security in cyberspace, network operators and security analysts should have a better understanding of traffic and data flowing at that present time. How- ever, the text based monitoring tools are more frequently used as they are easier to set up and usually not expensive. It is harder to monitor a large network with millions of packets flowing through text even if the data is filtered. Network operators are transitioning towards more graphical tools and visualizations in which they can map the textual data to a more visual approach. Visualizations can represent immense amount of data through patterns and vivid colors which can be far more helpful to network operators than the textual data. Visualizations, either in three or two dimensions represent art which we see every day around us in nature and we, as human beings are more prone towards visual patterns as we absorb those patterns subconsciously every day. Even though visualizations are far more helpful than textual data, it still needs to be viewed through a screen. We wanted to take one step further, where the network operator would be fully immersed into that cyberspace s/he’s gazing at with amazing patterns and graphical three-dimensional visualizations taking place right in front of his/her eyes while still being aware of his/her surroundings.

2.4 Types of Mixed Reality Interfaces

Microsoft made a lot of noise around its holographic augmented reality system - HoloLens [27]. The headset wraps around the head comfortably, as the visor-like band ensures its weight is distributed evenly along the crown of your head, avoiding putting pressure on your eyes and nose. You can use it for hours without the least bit of unease as a vent ensures the heat doesn’t accumulate inside. This is a stan- dalone device, so no wires or phones are needed. The holographic high-deﬁnition lenses utilize an advanced projection system to generate multi-dimensional full-color

14 images with low latency. Yet another captivating feature is the sophisticated sensors that are able to understand your actions and what environment you are in, using ambient light. All of this information is then processed by the custom HPU (holographic processing unit), mapping everything out in real time. You can record mixed reality videos or take pictures using the 2-megapixel HD camera with audio capture that has four microphones. Since it has spatial audio, so you can hear the holograms regardless of your position in the room. Based on Windows 10, the holograms can be blended seamlessly into the real environment, after the room is mapped out by the HoloLens.

Figure 2.5: Microsoft Hololens (view from left-side)

META-2’s prototype hardware last year definitely left a solid impression as it showed great promise for the Meta 2 augmented reality headset. User comfort is always of prime importance to developers, which is why memory foam is wrapped around the headset for a cushioning effect, so you can wear it for hours on end [28]. There is some support on top of the head, so weight distribution is even. Meta 2 has a custom operating system, and a new language of gestures for augmented reality, which have been well-received. The overall AR experience is ”wholesome”, as described by several users. It features a wide 90-degree field of view, so it fills up more of your vision. A slight drawback is that Meta 2 needs to be connected to a computer as it isn’t completely wireless. Due to excellent resolution the images are clear, while movements are precisely recorded as well. It has four speakers, but

15 the sound quality needs to be improved, and the headset has trouble mapping the area at times.

Figure 2.6: META 2 (view from right-side)

The Moverio Pro BT-2200 smart headset is extremely durable and sturdy, which is why it is best suited for enterprise applications [29]. It has safety-glass compliance, and is incorporated with dust and water-resistant characteristics, so you don’t have to worry about it getting dirty or scratched, nipped, or cracked, anytime soon. It can be worn with most front-brim safety helmets. It is a hands-free operational device - the wearable display has built-in voice commands, while head tracking and gesture control is supported too. The high-resolution front-facing camera and remote visual support makes it easy to help with complex tasks, while hot swappable batteries ensure all-day usage without interruption. You can easily adjust display brightness using the ambient light sensor, which has a low margin of error.

Windows Mixed Reality headsets work based on Windows 10. It doesn’t follow the constraints of mobile VR, where you are static, or tracked VR, in which the movements are monitored by cameras but limited by the range of the play area [30]. Windows Mixed Reality headsets ensure six degrees of tracking from the headset in

16 Figure 2.7: Moverio Pro BT-2200 (Front view) a Windows 10 environment. These devices are tethered, connected via HDMI and USB, again like existing VR systems. The unique aspect is they feature ”inside-out tracking”, which makes them diﬀerent from other systems on the market. They use front-mounted cameras to detect the room or certain objects in the surrounding environment, which are then added to the virtual scene viewed through the headset - this idea is what forms the mixed reality. There are Mixed Reality controllers which can be tracked by the headset too, so the sensors aren’t required.

While the competition is tough, as each headset stands out in its own way, Microsoft Hololens stands out here with its advanced features and excellent functionality. The reason this particular headset is extremely popular is it goes full AR with virtual 3D models of objects that can either appear as part of your real surroundings or combine with reality to make up entirely new ones. You can lock the hologram in place and then move it around and check it out from other angles, thus making it seem as it is a real part of your environment.

17 Figure 2.8: Asus’ Windows Mixed Reality Headset

The HoloLens is controlled by gestures, voice, and a small Clicker controller to make things a little more varied. The only physical controls on the device are the power switch, a volume button and contrast controls for the holographic lenses. When it comes to voice controls, the voice assistant Cortana has undergone massive improvements, and can effectively take you through the HoloLens experience for better comprehension. Gestures don’t always make sense when navigating menus or opening apps, so the HoloLens microphones are able to capture your voice commands. For your convenience, it is compatible with universal Windows 10 apps including OneDrive, Maps, Remote Desktop, Groove Music and Microsoft Office apps. Applications of HoloLens Offering mixed reality isn’t the sole objective of this headset as it has come with fabulous experiences:

• HoloGuide - It provides assistance as you make way through low visibility areas.

18 • HoloHear - It is a huge help for disabled people, as it can instantly translate speech into sign language.

• Teomirn - It can overlay prompts and instructions on a real piano.

• HoloStudio - You can create holograms and turn them into 3D objects using a 3D printer.

• Actiongram beta uses holograms to tell stories.

Although Pokemon Go has become the rage in the world of AR, Microsoft feels it is Minecraft that will ride the crest. Just picture playing holographic Minecraft from the privacy of your home, collaborating and building things - sounds amazing, right? For avid gamers, there is also going to be support for Xbox Live, which means that games, online multiplayer, and extras like friends’ lists and achievements are coming to HoloLens as well. Fragments is another game to watch out for - those who like solving mysteries will love it as you get to be in the midst of a crime drama, interacting with life size characters and getting to the bottom of the mystery using clues. Roboraid is a mixed reality ﬁrst person shooter that uses gestures and gaze to ﬁre at the enemy.

Apart from these exciting applications and gaming, NASA and Microsoft are putting in a lot of eﬀort for Project Sidekick that will help astronauts use HoloLens as a virtual aid when they are up in space. Microsoft also has an early app that lets an engineer give you instructions over Skype showing you how to ﬁx a light switch safely, sketching circles and arrows as holograms onto what’s in front of you. That’s why we decided to move further with HoloLens as HoloLens provides SLAM (Simultaneous Localization and Mapping) and amalgamation of several sensors which we will be using throughout our approach.

19 Chapter 3

Literature Review

On average, an attack was performed every 39 seconds [1] by the hacker and one in three Americans were hacked in 2016 [24]. The need to secure our devices is inevitable as 2016 was marked as an extraordinary year for cyber-attacks. Ran- somware, multi-million-dollar online bank heist and Distributed Denial of Service (DDoS) through Internet of Things (IoT) devices were just the beginning [40]. The estimated annual damage due to cybercrime is approximately 100 billion dol- lars across the globe [5]. The idea of the above statistics is not to scare people yet opposite, to aid awareness and acknowledge people about cyber-threats and importance of security.

3.1 Traditional Visualization Techniques

In 1995, A. Wise proposed an approach for Information Visualization which involved transforming text content to visual spatial representation for enhanced browsing and analysis [43]. This enabled analyst to avoid language processing and reduce mental load work as visualization utilizes cognitive and visual processes enabling spatial interactions with natural world. Designing advanced graphical user interfaces (GUI) and efficient visualization based on different data-types with different tasks were

20 just the beginning [38]. Since then, numerous techniques have been imagined, visualized and created. In 1996, Ben Shneiderman proposed the famous mantra adapted by a lot of advanced visualization interfaces which suggested: ”Overview first, zoom and filter, then details on demand” [38]. Digital and cyber-space were no exception as thousands of different techniques to visualize networks and data (over those networks) were used.

In 2003 JNUG (Java Universal Network/Graph Framework) was introduced which is used to visualize data as graphs or network. It is also used for modelling as well as analysis of data. It provides the user a way to browse and select vertices, filter the graph, and run analysis despite visualizing the network which can be very crucial to analyze large networks [32]. JUNG currently provides multiple tools and elements that are commonly required to write software that can manipulate, analyze, and visualize network data sets for visualization and further studies. Thus, these features would significantly expand the set of available tools and drastically enhance user ca- pabilities to write robust codes. Java universal network/graphs filtering mechanisms allows the network analyst to select any subgraph to be analyzed or visualized. The available filters involve degree filters as well as neighborhood filters. Thus, user can create composite filters that consist of a series of filters that needs to be applied in sequence. It also allows to create subgraphs of any loaded or derived network and analyst to load several networks, with subject to the restriction imposed by amount of memory available. The analyst uses a tree-based interface to navigate from these networks. Various JUNG’s algorithms are provided with different interfaces for social network analysis including ranking algorithms such as PageRank. Yet this tool was simple, it was not capable of abstraction with different layers of detail.

In 2005, Visalert was introduced which was a ﬂexible visualization tool for IDS alerts and system logs monitoring [16]. It featured correlations based on What, Where and When attributes for the data. Integration between alert time (When), network topology (Where) and type (What) is all fused into this one single two-dimensional display in a sophisticated and novel way. It provides a unique way to integrate multiple data sources within one cohesive display. In Figure 3.1, the outer ring represents all the IDS alerts infusing the inner ring with network topologies.

21 Figure 3.1: Visalert Visualization [16]

In 2009, Gephi was introduced which gained popularity as an open-source tool that was (and still is) used to analyze graphs and networks. Gephi does not require any programming knowledge. Its main strength being that it can produce high quality visualization handling relatively large graphs; actual size depending upon the infras- tructure [8]. It does have the abilities to calculate a few of the more common metrics such as degree, centrality, etc. but it is recognized as a stronger tool for visualization than analysis. Gephi is equipped with sophisticated data structure for object handling along with lightning fast rendering engine and, thus it becomes one of the most effective tools for network visualizations. It offers increasingly appealing visualizations easily rendering network up to 300,000 nodes and 1,000,000 edges. In comparison to other tools, it comes with very efficient multithreading schemes, enabling users to perform multiple analysis simultaneously without suf- fering from panel ”freezing” issues [8]. In a bigger-scale network analysis, quick layout becomes bottleneck as more and more sophisticated layout algorithms become CPU and memory-consuming by requiring more running time for completion. In this case, Gephi offers variety of layout algorithms, OpenOrd and Yifan-Hu are mostly recommended to obtain best throughput for large-scale network visualization [34]. OpenOrd, has ability to scale up to over a million nodes within half an

22 hour while Yifan-Hu is a recommended option to apply after the OpenOrd layouts are made. Notably, as compared to conservative and time-consuming Fruchter- man and Reingold algorithms, Yifan-Hu layout give aesthetically comparable view. Gephi offers other algorithms such as circular, contraction, dual circle, Isometric, GraphViz, random, MDS, Geo, and Force atlas layout. Most of these algorithms run in an affordable time but the combination of OpenOrd and Yifan-Hu seems to give the most appealing visualization results of all. OpenOrd layout algorithm also offers Descent visualization when a user stops the process when 50 to 60% of the progress is completed. If you parameterize any of these layout algorithms it can drastically affect both the running time and the visual results to greater extent [34]. Regardless of its very limited documentation, Gephi is a great and generic two-dimensional network visualization tool. It mainly emphasizes fast and smooth rendering, fast layouting, efficient filtering, and interactive data exploration and that it remains one of the best options for generic large-scale network visualization. However, Gephi lacks the skill to collectively visualize information for a single machine.

Figure 3.2: Gephi Visualization

In 2012, DAEDALUS-VIZ was introduced whose purpose was real-time dark-net monitoring [20]. It enables network operators to visualize real-time alerts and cir-

23 cumstances with great interactivity. It includes several features and new ways to visualize in 2D and 3D space among which Deep

Figure 3.3: Deep Dive feature from DAEDALUS-VIZ [20]

Dive is one feature where a detailed view of several dark-net packet is visualized. Deep Dive feature from DAEDALUS-VIZ is shown in Figure 3.3 where a detailed information from one of the dark-net packets is seen which includes timestamp, source and destination IP address and port numbers.

We live in a three-dimensional world, yet we are mainly surrounded by two dimensional visualizations. Even visualizations that are rendered in 3D are still displayed on two dimensional screens. The time to bridge the gap between physical and digital world has arrived and the need to smartly visualize in our own space has aroused. We live in an era where augmented and virtual realities are emerging. We ﬁnd several applications that are gaming-related or situational-awareness related. Augmented reality has changed the way we perceive things as they are rendered by a computer ﬁrst, retrieve information from knowledge database and then the output is overlaid on the top of surrounding. Let’s look at a few advances and applications for augmented and mixed realities.

24 3.2 Information Filtering

In 2000, S. Julier proposed region-based information filtering algorithm for Aug- mented Reality where the state of user and the state of objects perceived by the camera on which information is to be presented is considered and analyzed [23]. The time, intent and the location of the objects and the information that is to be augmented is prioritized by the algorithm and only relevant information that is needed at that specific time is displayed. To achieve this, the state of the user, the state of physical object, focus of the camera and nimbus (virtual information displayed) are all calculated initially. Once all the information is calculated, the nimbus focus, information filtering, user interaction and display cues are taken into account and therefore the final result is a well-baked and focused augmentation of the scenario [23]. We can see the unfiltered view on the left of Figure 3.4 and filtered view after algorithm implementation on the right.

Figure 3.4: Courtesy of Naval Research Lab Information filtering: Unfiltered view (left) and filtered view (right) [7]

If we augment all the information on the surface, the visualization would look messy and cluttered. Augmented and Mixed Reality applications should display information without changing the originality of physical reality. Intelligent environment sensing, low latency for virtual objects, auto calibration, synced focusing with real objects, ﬁltered interface and information, error estimation and advanced rendering makes one robust augmented and mixed reality application [7].

25 Figure 3.5: Augmented Reality Situation Awareness in U.S. Military

3.3 Situation Awareness

Augmented and mixed realities are also used in the military for situation awareness which keeps track of past, future and present military operations. Information Overload from a lot of sources is the main reason why an efficient situation awareness application is needed for military. Some effective projects that helped military were ”The Super Cockpit” which served as an AR-based application whose goal was to refer landmarks on the terrain when the visibility was limited and during weapon delivery and target acquisition [17]. The sensors mounted on aircrafts were used to enhance visibility during low light conditions and also when the view was blocked due to complex aircraft structure. The AR application was fused to the glass right in front of pilot as seen in Figure 3.6 and filtered data streams were provided according to different situation; hence situation awareness. Wearable Augmented Reality (WEAR) was used by military as an AR application in space. It had head-mounted display to render 3D graphics and information in the viewer’s field of view. As the main purpose was to be accessible in space, it was controlled by voice. WEAR included object identification onboard location to present astronauts with accurate and precise information about where the exact equipment is located and where the viewer is looking at. It also provided with step by step instructions to assist astronauts with lengthy and complex tasks as seen in Figure 3.7 and as voice was the primary input; it was a complete hands-free operation.

26 Figure 3.6: ”The Super Cockpit” by U.S. Air-Force [17]

Currently, network administrators and security analyst use inventory databases to log all the information about their network. To map a threat to a device, firewalls, intrusion detection/prevention and anti-virus scanners are used widely. Network analyst also uses Unified Threat Management (UTM) and Security Incident Event Message (SIEM) systems which allows central monitoring of event messages and then the results are displayed at security console. Manually processing all these data and taking a quick action against detected threat almost seems impossible due to abundant data. To mitigate this problem, in 2010 G. Klein proposed a model-based approach for cyber situational awareness where they built visualization of background information and network structure with security data to improve network operator’s situational awareness [25]. They created several visualizations based on use-cases such as different users like System administrator or Manager which had different tasks and abstraction. One of their unique technique was to create visualization based on Geolocation with respect to system location in respective building as seem in Figure 3.8.

In 2000, S. Savage proposed a technique for tracing DOS (Denial of Service) packets and flooding using network mapping [37]. Their approach identifies the network paths traversed by attack traffic without the support of ISPs (Internet Service Providers). These were a few examples of how data present in digital and textual forms could be filled with life, bridging the gap between the physical and virtual yet merging it with the real so that the authenticity of the ”real” doesn’t

27 Figure 3.7: WEAR step-by-step assistance fade out. Many applications have been developed for situation awareness merged with AR and MR, yet a lot of work needs to be done to fuse AR and MR with cyber-situation awareness. Two dimensional tools are better to understand a comprehensive system but when it comes to individual machines, even their detailed views are ﬂat. Those with three dimensional functionalities are including one more axes in a two-dimensional world. Therefore, in the era of Virtual and Mixed reality, we need to propose a system that identiﬁes compromised machine and guide us there as well as augments network information along with suspicious data on the machine similar to the ”The Super Cockpit” project.

28 Figure 3.8: Model based cyber defense situational awareness visualization

29 Chapter 4

Proposed Approach

We live in a three-dimensional world, yet we are mainly surrounded by two dimensional visualizations. Even visualizations that are rendered in 3D are still displayed on two dimensional screens. The time to bridge the gap between physical and digital world has arrived and the need to smartly visualize in our own space has aroused. We live in an era where augmented and virtual realities are emerging. In the era of Virtual and Mixed reality, we need to propose a system that identiﬁes a compromised machine and guide us there as well as augments network information along with suspicious data on the machine.

Figure 4.1 shows canonical workflow for using Mixed Reality for Cyber-Situational awareness. The initial step in the creation of this model is to have live data which is stored and indexed in real-time. Subsequently, threats are to be identified by Intrusion Detection Systems (IDS), Security Incident Event Message (SIEM) and Unified Threat Management (UTM) through central monitoring of the data from security components. Once the threats are identified, next step would be to map the environment using Simultaneous Localization and Mapping (SLAM) algorithm and unique tags containing MAC address of individual machines.

They are registered by marking Anchor elements in the spatial mapping system which allows us to map and store individual MAC address to its Anchor element in

30 world space (environment) along with their coordinates. Registering machines/tags and scanning the environment is an initial setup and is never required again. If the threat is detected, we trace the MAC address of the machine to its Anchor elements which eventually gives us the physical location in our environment. As soon as the operator looks around, Mixed-reality headset will ﬁnd the machine through spatial coordinate system and information/visualization will popup right above that machine.

We propose a mixed reality framework to be deployed in Microsoft HoloLens that would identify suspicious activities and exactly map where the activity is happening. Whether it would be a machine, wireless device or a server we would know where the suspicious activity is taking place and our framework would guide us there. It will detect the machine using unique markers which are placed on each machine embedded with their MAC address. Once detected, it would match the MAC address with the IP address currently assigned to that machine and pull out necessary information which will then be translated into Unity game engine and would be processed there and assigned to several GUI brackets. The information would be smartly and efficiently augmented on the device using surface detection will all the data into it. We propose a two-step transition system for our augmentation where the first step will have transparent GUI brackets showing the volume of the data flowing; Upload and Download rates and inbound query information. The second step will have a 3D globe which will be able to pull geolocation data from the servers in Latitude and Longitude form and then would be converted to Cartesian coordinates and mapped to the 3D globe to have the exact position where the query arrived. We will have transitioning functionalities with animations where user can use gestures as input as HoloLens support multi-gesture functionality.

Localization of the machine will be accomplished using SLAM (Simultaneous Lo- calization and Mapping) where our device will constantly construct or update map of an unknown environment while keeping track of its own location. We will use extended tracking mechanism where even if the unique marker is out of view, we transform the position of the target into spatial coordinate system of HoloLens.

31 This allows us to keep anchor markers on already detected objects. We will assign anchor component to all our UI brackets (Unity Gameobjects) allowing them to be locked at the same place in real world from the spatial mapping map. The spatial mapping would augment even if the marker is not visible. This allows us to have two mapping systems where if one fails, another one will always augment. Therefore, the chances of failure reduce signiﬁcantly.

The scenario of our proposed approach can be visualized in Figure 4.2 where all the devices are connected to a data collecting and ﬁltering server as a middle man between machines and the network. The network operator wearing Mixed Reality Headset uses our proposed implementation and can see network data and meaningful information augmented on individual machines which are in his/her range. On the contrary, another network operator uses traditional tools and techniques to visualize data on the monitor as seen in 4.2. To try out our approach in real-time, we implemented a prototype (found in Chapter 5) which uses most of the techniques proposed in this thesis. Chapter 5 also describes the implementation of the prototype with detailed description of every components constructed.

32 Figure 4.1: Model of our proposed approach

33 Figure 4.2: Proposed Approach Scenario indicating network operator wearing Mixed Reality Headset along with using traditional tools for network data monitoring

34 Chapter 5

Implementation

5.1 Initial Setup

For our approach, we use Unity Game Engine to create a mixed reality environment that can be deployed into any platform. For the sake of this thesis, we will be deploying the implementation on Android. Implementation for our prototype was done in Windows 10 operating system along with deployment in an Android tablet. Initially, Unity provides us with two default objects -Main Camera and Di- rectional Light providing us with an empty 3D canvas where ideas and approaches are limited to one’s imagination. We also need to choose whether our project is two-dimensional or three-dimensional. As we are creating a Mixed Reality application, we choose 3D project. As we are developing this for Android, we need to ﬁrst switch our platform from Windows (Default) to Android. In order to do that, we need to have Android SDK’s latest version installed as well as Unity Android Mod- ule -”Unity Support for Android” package. Then we have to switch our deploying platform from Windows to Android. Once we were past that, we started developing the application. Unity development interface can be a bit complicated with lots of windows. Let’s start by understanding the diﬀerent sections of Unity development interface -Unity Editor.

35 Figure 5.1: Unity Editor Windows [42]

By default, Unity Editor is divided into ﬁve sections (Figure 5.1):

• Toolbar

• Hierarchy Window

• Scene View

• Inspector Window

• Project Window

Starting from the top, the toolbar is a static panel which cannot be moved or rearranged. It provides with control options in the middle where a scene can be tested using Play, Pause and Step options. On the left-most, there are buttons to toggle diﬀerent scene views and manipulate objects. On the right-most, there are options to connect to cloud and also Layer option where one can see or hide layers when developing. The Hierarchy Window consists of hierarchical representation of all the objects present in the scene. They also show parent and child objects in hieratical form. The scene view is the visual representation of all the objects in the scene. It provides a free view camera where one can navigate to any object and

36 can manipulate it however they want. The inspector window on the right allows to edit and see all the properties of a selected object. New scripts and properties can be added or removed from this window. Lastly, we have the project window where all the assets from library are displayed and can use used in the project.

All objects in Unity can be attached to a script and behaviors can be set for them how to interact conditionally. There are two languages Unity supports for scripting. One is C# and another is JavaScript (Unity Script). Unity has modified both these languages to work with Unity by default. For our implementation, we used C# as our main language due to its Object-oriented model. Unity has its own preset rules of execution order for several in-build functions known as event functions. Whenever a scene loads, Awake, OnEnable and OnLevelWasLoaded gets called. Once the prefabs (objects) are instantiated, Awake is called and therefore is executed. Before the first frame updates, Start function is called. These all functions are called before the application is started and used to initialize or create prefabs. However, during the running of an Unity application, Update orders are performed. Update function is the main workhorse function which runs on every frame update and used for animations, camera positions, interactions and much more. Coroutines run after normal Update function returns. It is used mainly to run delayed execution once the given instruction finishes. We have used ”yield WWW” function to query the ELK database for data. Also, we have used ”yield WaitForFixedUpdate” for several animations and GUI rendering. There are more than 30 functions that are executed by Unity but as we are not creating a full-fledge game, we do not require Physics and other Gizmo rendering functions. However, all functions with their execution order and description can be seen in Figure 5.2. It is really important to keep these execution order in mind before designing the application. That’s why we followed the script lifecycle and execution orders in every phase of our implementation.

Initially, we had to make the camera compatible to Augmented Reality application. As our ﬁnal goal is to make a Mixed Reality application for Microsoft Hololens, for the sake of this thesis, the prototype was created in Android as an Augmented Reality application. For an AR app, as the video stream from the camera is our background, we need to change ”Clear Flags” option in main camera from ”Skybox”

37 to ”Solid Color”. Clear Flags property of a camera determines what parts of screen would be cleared when dealing with buffers. Skyboxes wrap around the entire scene to show what 3D world looks like and they are mainly a set of six images -Up, Down, Left, Right, Front and Back. All these images render in a full 360-degree view to give user an impression of complex scenery at the horizon. By default, the camera replaces color buffer with the skybox and completely clears depth buffer too on each frame. However, as we are not dealing with Skybox, we must use Solid Color as the color buffers becomes one solid color - Black. Unity also offers two types of projection in camera; Perspective and Orthographic. Perspective projection renders 3D objects with the sense of depth just like real world. On the other hand, orthographic projection removes the sense of perspective and objects are rendered on screen without depth and perspective distortion as seen in Figure 5.3. As we were creating an AR app that looks realistic and uses depth, we used Perspective projection for our main camera. We also used several components for the AR application from Vuforia1 SDK. One important children of Camera component provided by Vuforia1 is BackgroundPlaneBehaviour which creates a mesh at the distant end of camera frustum and video provided from camera feed is rendered over it. It uses its custom video material in the mesh renderer of that object. While the VideoBackgroundBehaviour handles native rendering of video as background, HideExcessAreaBehaviour prevents the augmented prefab to go off the limit of video background due to scaling. These steps were needed to complete the setup needed for the Camera component in order to be fully compatible with an AR application. Once we were done setting up the camera and subtle scripting behaviors, it was time to move forward with image targets and GUI design.

As we were using several behaviors from Vuforia1, we found out that while testing the camera on an Android device, the device camera is not able to Auto-Focus. Investigating through the SDK documentation, we found out that Auto-Focus is not enabled by default and therefore we needed to enable Auto-Focus using script.

38 Therefore, we created Manager script which handled all the small tweaks. Figure 5.4 shows the flowchart of the script we created where we had to enable Auto- Focus. The SDK provided five different focus modes as listed below:

• FOCUS_MODE_NORMAL -Default mode by device’s camera driver

• FOCUS_MODE_TRIGGERAUTO -Triggers single autofocus operation (Touch to focus)

• FOCUS_MODE_CONTINIOUSAUTO -Enables driver-level continuous autofocus

• FOCUS_MODE_INFINITY -Sets device’s camera focus to inﬁnity

• FOCUS_MODE_MACRO -Enables macro-level focus for closeups

Of-course we used FOCUS_MODE_CONTINIOUSAUTO at ﬁrst but we were not getting any results. Thus, we found out that it was taking at least 1-2 seconds for the video-feed and behaviors to load in the application. Therefore, we decided to create a Unity Co-routine which enabled us to wait 3 seconds before executing the ”InitializeARCam” function. We set the focus mode to CONTINUOUSAUTO and it worked perfectly. To prove that our script is robust and would work globally, we tested it on three other devices and unfortunately, it did not work in one of them. After some investigating, we ﬁgured out that some devices, when forced to Auto- Focus, were not able to enable it. Therefore, we added another condition at the end to check that even after forcing Auto-Focus, if the device is not automatically focus, we would set the focus mode to FOCUS_MODE_NORMAL which would give control to default camera driver. After testing again on all the three devices, it worked like a charm.

39 Figure 5.2: Unity Script Lifecycle and Execution order [41]

40 Figure 5.3: Orthographic vs Perspective Projection [4]

Figure 5.4: Flowchart representation for initializing main camera and enabling auto-focus

41 5.2 Augmenting Information

For our approach to uniquely identify every machine, we decided to go with QR (Quick Response) codes for their simplicity yet highly effectivity. Every machine has a unique identifier, also known as MAC (Media Access Control) address and we encode MAC address into QR codes. The reason we encode MAC address is that MAC address is static unlike IP (Internet Protocol) address which can be changed if not static or if a machine is configured to get IP address from a DHCP (Dynamic Host Configuration Protocol) server. To recognize and decode QR code’s information, we had to set Image Targets in Unity for them to be recognized. The ARCam we set above should be able to detect those unique QR codes and decode MAC addresses (data) from them. In order to recognize and decode this information, we used the default SDK’s Image Target Behaviors. For our prototype, we selected three random MAC addresses and encoded them into QR codes. Once done, we had to upload them to SDK’s Target Manager on their website. The SDK would extract unique features and patterns from the QR codes and therefore it would make them unique. Once they have been uploaded and processed, we could download the Image Target Database and import it into our project in Unity. Once the database had been uploaded, we had to set up the database in Image Target Behavior along with the Width and Height of the target image.

Figure 5.5: QR code as Image Target Behavior

42 Once the image was detected, the SDK provides an option to augment a 3D object (in our case, Globe) which we tested and can be seen in Figure 5.5. Augmenting a 3D object is easy, however as we were making the prototype with 3D and 2D objects, we had to make Canvas compatible with this application as the default SDK does not provide an option to augment 2D objects.

In Unity, a Canvas is where all UI elements are rendered as children elements. It uses EventSystem which is a way to send messages/events to objects based on user and custom inputs. Canvas renders in either screen space or world space. Screen space is used when the GUI is static and even if the screen is resized or changes resolution, the Canvas will render itself automatically to match the size. Another way the Canvas is rendered is World Space where it will behave as any other object in the scene. For our implementation, we had to set Canvas to be rendered in World Space as the Canvas elements would be of same size and no resizing is needed. We also had to set Canvas Event Camera in World Space to be our ARCam as ARCam will be used to process the GUI events. But as we know, default SDK does not provide an option to render GUI elements. Therefore, we needed to make it compatible by getting Camera Position, Trackable events and Event Handler. Determining camera position and rotation in Unity is simple as we can get the Transform values from the function. However, the SDK’s camera position and rotation are delivered in 3x4 row-major matrix. Matrix represents information in two forms; the initial 3x3 sub-matrix on left-side column is a pure rotation matrix which is also Ortho-normal (orthogonal and normalized both). The rotation matrix represents how the target is rotated with respect to the camera plane. It also says where the target is with respect to the camera. The right-most column, also known as Translation Matrix tells where the target is as seen from the camera, therefore the distance can be calculated. For example, if the value of <0,0,0> is returned, it means that the camera and target are at the same position while if a value of <0,0,5> is returned, it means that the target is 5 units away from the camera. We wanted to estimate the camera with respect to the target, therefore we had to invert the pose matrix. Let’s assume that R is the rotation matrix and t is the translation vector and ’ is the matrix inverse. Therefore:

43 [R|t]0 = [R0| − R0t] (5.1)

Once we were done getting the position of camera with respect to target, we had to get values from TrackableEventHandler for the current detection and tracking. We set the condition that when the TrackableBehaviour detects the target or already tracking, we enable a flag -”ShowGUI”. This flag is constantly monitored by update function and as soon as this flag is turned on, it activates the Canvas elements and therefore Canvas elements would be seen when the target is detected or tracked. Once that’s done, if the target is either not tracked or detected, we had to deactivate the Canvas elements which we accomplished using status values from TrackableBehaviour.

5.3 Data Retrieval

In order to have a working prototype, we needed data that was live. Harris Institute of Assured Information were able to provide us access to Logstash database where the log of all devices connected to Florida Institute of Technology was found. Of course, there were anonymization filters that filtered out sensitive information like user’s email address or passwords, but we were able to log messages capturing timestamp, HTTP response code, DNS queries, geo-location including latitude and longitude and much more. The Logstash pipeline collected data from several sources including External DNS, Internal DNS, Apache servers and spam filters. Those data were captured as TCP packets and were passed through several filters and then passed to elastic search and Kibana was able to pull data using elastic search queries and display them as visualization. We can see the flow of data in Figure 5.6 as flowchart.

Figure 5.7 shows us how Kibana displays the query along with visualization, but it is mainly text, and we do not need everything to be displayed as ﬁltering is one essential part for Augmented Reality application. To see the visualization

44 Figure 5.6: How data is collected and then processed in logstash pipeline through various ﬁlters of data was very easy but to retrieve it in Unity required us to make an HTTP elastic search query that returns data in JSON. Therefore, we had to dig into ElasticSearch and found out that it is amazingly powerful, yet extremely complex query system. Let’s look into some terminologies for queries in ElasticSearch. The default port to access logstash is 9200. Once the DNS or IP address of the server is known, we start with endpoint which are tables/index types in ElasticSearch. In our case, the host/server is ”elasticsearch01.nsoc.hiai.net” followed with port 9200. The endpoint is the database name which is ”logstash-nfdump-*”. Basic query is followed by ”_search?” and only string parameters. We wanted personalized queries for speciﬁc IP address (mapped to MAC address) and therefore we had to perform Full Query API. These queries are written as a JSON structure and then sent to search endpoints where it is processed and returns data as JSON objects. Thus, we created the query where all data belonging destination IP address should

45 be displayed. The query also included the time -From and To, for getting data from speciﬁc time range including live. We also managed to get geolocation data including country name as well as latitude and longitude of the source. Full query can be in the appendix.

Figure 5.7: Kibana displaying data for a dummy query

5.4 Integrating Components

Once we had a working query, we had to develop APIs to integrate Unity with ElasticSearch query and retrieve all data. In order to perform a HTTP GET request through Unity, we had to use coroutines. Unity coroutines are special functions that allows pausing and resuming its own execution. A typical C# function can return any type while coroutines must return IEnumerator. In Unity, a coroutine can be used to handle asynchronous code or for code that takes time to compute over several frames. It basically breaks work into multiple frames giving us full control on demand. We use coroutine function as we need to query ElasticSearch and get JSON data. If we performed this function on Update, it would never be completed as HTTP request takes a few milliseconds to complete its execution. Once the

46 coroutine was designed and programmed, we had to call that coroutine from Start function so that the HTTP request was queried before the start of ﬁrst frame of our application. We call coroutine by ”StartCoroutine(WaitForRequest(www))” where ”WaitForRequest” is the name of coroutine and ”www” is a WWW object provided by Unity as a small utility module to retrieve contents of URLs. We also created a String variable in our script which was used to input the destination IP address. Similarly, we created variable named ”elk_from” for inputting the minutes which were used to have data from those minutes. So, if ”elk_from” would be ”now-60m” that means that show data from now to the last hour. We created these variables so that they could easily be modiﬁed without modifying the ElasticSearch query and injected these variables inside the query. Therefore, we used coroutine to query the ElasticSearch database and downloaded data as JSON object. We stored data in ”www” variable and then next step was to process JSON data.

The next step after retrieving data as a www object was to parse the data from JSON to normal text. For this purpose, we used SimpleJSON API which is a powerful JSON parser and builder. We parsed JSON data using JSON.parse() function build in SimpleJSON API and converted all the data to string and ﬂoat values as needed. Below is the list of data we retrieved along with their ﬁnal type.

• Total hits: Total number of queries retrieved

• Total geolocation hits: Total number of queries with geolocation data

• Latitude: Geolocation children speciﬁcally targeting value for Latitude

• Longitude: Geolocation children speciﬁcally targeting value for Longitude

• Source IP: IP address from where the query was generated

• Source Port: Port number from where the query was generated

All the above-mentioned data was stored as JSON objects which can be retrieved whenever needed. The next step was to create Unity Canvas UI elements. We created two UI prefabs and used transparent background in PNG format to make

47 Figure 5.8: Empty UI elements aka UI Brackets them look modern. These elements were needed to display data onto them, that’s why we call them UI Brackets (Figure 5.8). Once we are done creating UI brackets, we would start arranging the geolocation data. First of all, we count the Total geolocation hits and store it in an integer variable named ”total_req”. Then we run the geolocation parsed data through a for loop and data is written sequentially as String into elk_data.json file. The file currently stores data in the following way: ”Data”: [ Lat1, Lon1, Lat2, Lon2, Lat3, Lon3, . . .] For the simplicity purposes, we have kept data to be in this format. Once the file’s two values are filled, the coroutine triggers and it displays value on the globe. The globe is handled using two main scripts: DataVisualizer and DataLoader. DataLoader consists of only one Unity Start function which makes sure that the data is loaded as a Text Asset in Unity and imported as JSON file. Then it is converted to text and it creates meshes of all data. Once the meshes are created, it instantiates a Gameobject named PointPrefab while calculating mesh vertices and triangles. Once that’s accomplished, every mesh data creates its own object (PointPrefab) consisting values of mesh vertices, indices and colors based on the magnitude. All the latitude/longitude data must be first converted to radians. We used an inbuild constant in Mathf library called Deg2Rad which converts Degrees to Radians and has a constant value of:

48 Figure 5.9: Mapping Longitude and Latitude on the globe using mathematical conversions from Latitude/Longitude to Cartesian coordinates

Therefore, we used Deg2Rad to convert all the latitude/longitude data to radians. Once that’s accomplished, we use the following formula to convert latitude/longitude to Cartesian coordinates (x,y,z).

x = Cos[(Longitude ∗ P i/180) ∗ Cos(Latitude ∗ P i/180)] (5.2)

y = Sin(Latitude ∗ pi/180) (5.3)

z = Sin(Longitude ∗ pi/180) ∗ Cos(Latitude ∗ pi/180) (5.4)

As we get x, y and z coordinates, we can map them to the Globe prefab making Globe its parent and all point vertices to be Globe’s children. The result is displayed in Figure 5.9 after correctly mapping all the Latitude and Longitude data. Data used in the demonstration of Figure 5.9 was to verify whether the conversion and mathematical equations are properly implemented, and the dataset was World’s population for the year 1990 [2].

After ﬁnishing with the Globe and ﬁlling data in UI brackets, next step was transitioning. We categorized four UI brackets into two steps, each providing us with a two-step transition. We allocated the upload/download data bracket and query text information bracket into step 1 and the Globe and Upload/Download graph

49 to step 2. To enable transitioning between step 1 and step 2, we had to create Touch events on the two-equal half of the screen. The upper half of the screen was designed to change to step 1 and if the user is already inside step 1, nothing should happen. The lower half of the screen was designed to change to step 2 and the same validation was created if the user was already inside step 2.

Figure 5.10: Coroutine: MoveOverSeconds()

We then create a coroutine MoveOverSeconds() in Manager script which moves our target from position A to position B in given number of seconds. We use coroutines as they can delay the transition so that it would look more realistic. If we would be using the same code in Update function, the end result would be the same, but the animation would not be visible. This coroutine takes three parameters. First one is objectToMove which we used our UI brackets. Second one is end which denotes the end position of the object (UI brackets in our case) and the third one is seconds which takes float as an input and transitions the position from its current position to end position in that number of seconds. Figure 5.10 shows the flow chart of our implementation. Then we created two functions: changestep1() and changestep2() in our Manager script. We also created two Boolean flags to keep track whether the steps have changed or not to prevent changing the same steps twice. Then we call the MoveOverSeconds() coroutine sequentially four times for

50 four UI brackets in both steps while updating our Boolean ﬂags to have robust transition. During the transition process, we took the screenshot shown in Figure 5.11 showing what step 1 looked like.

Figure 5.11: Screenshot of development build of Step 1 transition

5.5 Designing Prototype

After the implementation was over, it was time to test this application in real-time. We exported this application as Android Package Kit (APK) ﬁle and we were able to accomplish the following goals:

• We built an interactive visualization technique in Unity Game Engine to visualize and augment machine-speciﬁc information eﬀectively

• Our application detects marker tags (QR codes) on machines with MAC addresses uniquely assigned to individual tags and augment network information on the top of it.

• Our application performs elastic search query to the database for getting data linked to that MAC address, parses it and smartly and beautifully assigns it to pre-deﬁned brackets made beforehand in Unity.

51 • Our application visualizes following information:

– IP address for that machine – Inbound and Outbound queries from all the machines within an hour – Volume of traﬃc being exchanged for that machine – Geolocation data visualized in a Globe

Figure 5.12 and 5.13 shows the prototype detecting a tag on the machine and showing information on Step 1 and Step 2 respectively.

Figure 5.12: Live Prototype for Step-1 Transition

Figure 5.13: Live Prototype for Step-2 Transition

In summary, the Implementation start by conducting an initial setup where we put together required components in order to start building our application. We then move towards creating User Interface (UI) for augmenting our information on the

52 top of machine. We also retrieve real-time data from the server, parse it and clean it to make it compatible with our User Interface and ﬁnally integrate everything to result into a working prototype.

53 Chapter 6

Case Study

6.1 Scenario Setup

Let us assume a scenario where an institution has a cybersecurity laboratory with four machines in it and a network operator is responsible for all these machines. The network operator is situated in another room than the cybersecurity lab. Network operator also plays the role of security administrator who monitors all machines for any suspicious activities. The data is collected from various sources like Internal and External DNS (Domain Name Servers), Apache logs and Spam Filters. This data is collected in logstash pipeline as TCP input and processed there through various ﬁlters. The network operator has already implemented our implementation as mentioned in Chapter 5 and is ready to use our application for Cyber-situation awareness. The network operator receives a threat in his monitoring tool signify- ing that one of the machine infused with our technology has been compromised. S/he uses our application to identify the compromised machine to see an intuitive visualization of all network data including the suspicious queries made, augmented right above the machine. To put the above scenario to play, we tested a similar scenario to test the integrity and robustness of our application. We picked the Cyber-security lab in our department of Florida Institute of Technology to setup and track the entire scenario.

54 Initially, we encoded four QR codes with the MAC addresses of four machines in our lab. The embedded QR codes with MAC addresses now had to be linked to their IP addresses. Therefore, we ran a Linux bash script to get all the IP addresses from the arp (Address Resolution Protocol) cache to link to their specific MAC address. Address Resolution Protocol (ARP) is used to map an IP address to the physical address of the machine in a local network. We accomplished that by populating the arp cache by pinging the entire network and grepping the exact string which contained MAC address to get the IP address. Initially, we stored the MAC addresses of our machines to an array. Then, to ping the entire network, we used nmap utility tool with [-sP] flags which are flags that sends ICMP ECHO REQUEST (ping request) to all the machines in the range that we specified. This request receives ICMP ECHO REPLY to ensure that the system is running and up and ICMP packets are not blocked. This would refresh the arp cache to get all the updated MAC addresses and their respective IP addresses. Finally, we stored the MAC address -> IP address correlation to our database. Figure 6.1 explains the workflow of the bash script we implemented to get IP addresses from MAC addresses. We run this script on loop to get the updated IP addresses as most machines automatically gets IP address from DHCP (Dynamic Host Configuration Protocol) server.

As the network operators (we) were continuously monitoring our four machines, we had all information about all queries made to all four machines in text form. Next thing was to create an alert notifying us that one of the machines was receiving suspicious queries. We created an alert notifying us that one of our machine was receiving apprehensive queries and as we were located in diﬀerent room, we loaded our application on our tablet and started to head towards the lab where one of the machine was compromised. We already had QR codes stuck on the top of monitors in the lab as seen in Figure 6.2.

We started to scan every machine and the ﬁrst machine showed no signs of suspicious activity. As we moved to the second machine, we found out from our all red GUI that this machine was receiving suspicious queries. We could see that in Figure 6.3 that this machine was marked as ”Suspicious” by our algorithms and the globe

55 Figure 6.1: Flowchart showing the execution of bash script for getting IP addresses from MAC addresses on the left-hand side of the screen showed through color representation (Red) which queries were suspicious. The globe also graphically represented the source location (Start-Point) of the query where it was originated and its path using animated Line Rendering till our machine (End-Point) in Melbourne, Florida. The globe was fully interactive with gestures like Rotate using touch events and pinch to zoom to get a better idea of the ﬂow of data. The right side of the augmentation showed a UI bracket with continuously updated information such as Source IP, Source Port, Destination Port and Country of origin (of query). Below this information, it also showed the total volume of data which too was continuously updated.

From the initial setup of our case study, we anticipate that our prototype:

• Visualizes real-time data in an intuitive and in a more natural way

• Displays output of real-time queries with a delay of as trivial as 100ms

56 Figure 6.2: Lab setup with QR codes stuck to the monitors of each machine

• Detects marker tags (QR codes) on machines with MAC addresses uniquely assigned to individual tags and augment network information on the top of it.

• Automatically sets the color of User Interface according to the severity of the threat.

57 Figure 6.3: Visual augmentation of information of suspicious machine (View from far distance)

6.2 Scenario Observations

In a traditional way without using our approach, the network operator localizes the suspicious machine in a large network by dividing the Internet Protocol (IP) addresses into several subdomains according to the region and access points. While asking where the speciﬁc machine was, the network operator was only able to provide us with the approximate location of a speciﬁc area where that particular machine might be. In our case, there were seven machines from which we had to guess the suspicious machine, however in the worst-case scenario (a library), there can be hundreds of machines and this is when the traditional approach starts showing its limitations. By using our approach, the network operator saves time by not waiting for the user if absent at that time. The network operator also eliminates the need to ask users information about their machines.

Once the scenario was setup and ready to be tested, we oﬀered the prototype to our several colleagues and observed the outcomes. Whether the user was present on the machine or not and even if the machine automatically connects to a ﬁshy server without the user knowing about it, we could just scan the tag and informa-

58 tion would augment on the top of the machine. After localizing the compromised machine through our prototype, it appeared that our colleagues were able to har- vest information faster than the traditional way. Therefore, it seems that we have achieved promising results through our case study even though the prototype does not fully use our proposed approach. We are optimistic that once we implement this prototype in a mixed reality headset, we can construct several other localization algorithms to enhance the stability and robustness of our approach to get better results and provide network operators a novel way to visualize replacing traditional tools and techniques.

59 Chapter 7

Conclusion

In this thesis, we conducted a literature review which helped us study approaches and visualization techniques similar to our solution. Based on the findings of literature review, we proposed our approach to convolute the digital with the physical; hence proposing a mixed reality approach for network operators and security analysts to have better understanding of the flow of data in their surroundings. While most of the approaches and visualizations focus on visualizing large networks in one place, we conduct the opposite; we map the devices in our surrounding and augment information specific to that device, on that device. To test our approach, we created a prototype in Android system using Unity Game Engine which augments real-time data on individual machines marked with tags. For the sake of prototype, we had to use larger tags but we tend to avoid that in our final implementation. We intend to increase the efficiency of our image recognition algorithm along with Simultaneous Localization and Mapping techniques which will eliminate the need for larger tags and machines will be tagged with smaller tags. Therefore, we intend to use tags only for the initial setup to set world anchors on our system which will make our final implementation much robust where network operator does not need to go near the machines to scan the tag and augmentation will be displayed as soon as the operator gazes at the machine.

60 From this thesis, we speculate that our approach is intuitive and cognitive; yet fully functional without compromising useful attributes from other visualization techniques. Our approach proposes a way that, eliminates the need to manually local- ize each machine, find suspicious machine which can be time consuming, and take quicker actions to mitigate the threat. Our case study also shows the efficiency of our visualization which is adaptive and cognitive through various color embedded User Interfaces (mapped according to the severity of the threats) along with animated three-dimensional objects such as Globe with various gesture controls. Augmented Reality and Mixed Reality are the future from which we are not far, and we used these new and efficient techniques to improve and secure our situation awareness in cyber-security.

61 Bibliography

[1] Study: Hackers attack every 39 seconds. http://eng.umd.edu/news/ story/study-hackers-attack-every-39-seconds, 2007. Online; accessed 27 September 2017.

[2] Webgl globe - chrome experiments. https://experiments.withgoogle. com/chrome/globe, 2009. Online; accessed 7 October 2017.

[3] Can virtual reality transform the data visualization market?, Dec 2015.

[4] Orthographiccamera class. https://msdn.microsoft.com/en-us/ library/system.windows.media.media3d.orthographiccamera(v=vs. 110).aspx, 2015. Online; accessed 25 October 2017.

[5] 10 alarming cyber security facts that threaten your data. https://goo.gl/ rVBKoN, Jun 2017. Online; accessed 5 October 2017.

[6] Alloftech. How does virtual reality vr work? what are the most important supported glasses?

[7] Ronald Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner, Simon Julier, and Blair MacIntyre. Recent advances in augmented reality. IEEE computer graphics and applications, 21(6):34–47, 2001.

[8] Mathieu Bastian, Sebastien Heymann, Mathieu Jacomy, et al. Gephi: an open source software for exploring and manipulating networks. Icwsm, 8:361–362, 2009.

62 [9] Mark Billinghurst and Hirokazu Kato. Collaborative mixed reality. In Proceed- ings of the First International Symposium on Mixed Reality, pages 261–284, 1999.

[10] David Brewster. The Stereoscope; Its History, Theory and Construction, with Its Application to the Fine and Useful Arts and to Education, Etc. John Murray, 1856.

[11] Holly Brockwell. Forgotten genius: the man who made a working vr machine in 1957, Apr 2016.

[12] Eric Byres and Justin Lowe. The myths and facts behind cyber security risks for industrial control systems. In Proceedings of the VDE Kongress, volume 116, pages 213–218, 2004.

[13] Thomas P Caudell and David W Mizell. Augmented reality: An application of heads-up display technology to manual manufacturing processes. In System Sciences, 1992. Proceedings of the Twenty-Fifth Hawaii International Confer- ence on, volume 2, pages 659–669. IEEE, 1992.

[14] Eduardo Duarte. Plato’s "allegory of the cave". In Being and Learning, pages 69–106. Springer, 2012.

[15] George W Fitzmaurice. Situated information spaces and spatially aware palm- top computers. Communications of the ACM, 36(7):39–49, 1993.

[16] Stefano Foresti and James Agutter. Visalert: From idea to product. VizSEC 2007, pages 159–174, 2008.

[17] Thomas A Furness III. The super cockpit and its human factors challenges. In Proceedings of the Human Factors Society Annual Meeting, volume 30, pages 48–52. SAGE Publications Sage CA: Los Angeles, CA, 1986.

[18] Morrie Gasser. Building a secure computer system. Van Nostrand Reinhold New York, 1988.

63 [19] Michael Gervautz, Dieter Schmalstieg, Z Szalavri, Konrad Karner, Franz Madritsch, and Axel Pinz. Studierstube-a multi-user augmented reality environment for visualization and education. Technical report TR-186-2-96-10, 1996.

[20] Daisuke Inoue, Masashi Eto, Koei Suzuki, Mio Suzuki, and Koji Nakao. Daedalus-viz: novel real-time 3d visualization for darknet monitoring-based alert system. In Proceedings of the ninth international symposium on visualization for cyber security, pages 72–79. ACM, 2012.

[21] J Charles Jennette, Ronald J Falk, Konrad Andrassy, Paul A Bacon, Jacob Churg, Wolfgang L Gross, E Christiaan Hagen, Gary S Hoﬀman, Gene G Hunder, Cees GM Kallenberg, et al. Nomenclature of systemic vasculitides. Arthritis & Rheumatology, 37(2):187–192, 1994.

[22] Chihyung Jeon. The link trainer, ﬂight simulation, and pilot identity. ASME International, 56(1):28–53, Jun 2000.

[23] Simon Julier, Marco Lanzagorta, Yohan Baillot, Lawrence Rosenblum, Steven Feiner, Tobias Hollerer, and Sabrina Sestito. Information ﬁltering for mobile augmented reality. In Augmented Reality, 2000.(ISAR 2000). Proceedings. IEEE and ACM International Symposium on, pages 3–11. IEEE, 2000.

[24] Richard Kirk. One in three americans hacked in past year (via passle). http://www.cybersecurityinsights.net/post/102eij3/ one-in-three-americans-hacked-in-past-year, Oct 2017. Online; accessed 25 September 2017.

[25] Gabriel Klein, Christoph Ruckert, Michael Kleiber, Marko Jahnke, and Jens Toelle. Towards a model-based cyber defense situational awareness visualization environment. In Proceedings of the RTO Workshop "Visualising Net- works: Coping with Chance and Uncertainty". Rome, NY, USA, 2010.

[26] Steve Mann and James Fung. Videoorbits on eye tap devices for deliberately diminished reality or altering the visual perception of rigid planar patches of a real world scene. EYE, 3:P3, 2001.

64 [27] Steve Mann and Steve Mann Nnlf. Mediated reality. 1994.

[28] W Stephen G Mann. Eye-tap for electronic newsgathering, documentary video, photojournalism, and personal safety, Sep 2003. US Patent 6,614,408.

[29] Tomasz Mazuryk and Michael Gervautz. Virtual reality-history, applications, technology and future. 1996.

[30] Hilary McLellan. Virtual realities. Handbook of research for educational communications and technology, pages 457–487, 1996.

[31] Paul Milgram and Fumio Kishino. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, 77(12):1321– 1329, 1994.

[32] Joshua O’Madadhain, Danyel Fisher, Padhraic Smyth, Scott White, and Yan- Biao Boey. Analysis and visualization of network data using jung. Journal of Statistical Software, 10(2):1–35, 2005.

[33] Randall Packer and Ken Jordan. Multimedia: from Wagner to virtual reality. WW Norton & Company, 2002.

[34] Georgios A Pavlopoulos, David Paez-Espino, Nikos C Kyrpides, and Ioannis Iliopoulos. Empirical comparison of visualization tools for larger-scale network analysis. Advances in bioinformatics, 2017, 2017.

[35] Jun Rekimoto. Navicam: A magnifying glass approach to augmented reality. Presence: Teleoperators and Virtual Environments, 6(4):399–412, 1997.

[36] Jun Rekimoto and Katashi Nagao. The world through the computer: Com- puter augmented interaction with real world environments. In Proceedings of the 8th annual ACM symposium on User interface and software technology, pages 29–36. ACM, 1995.

[37] Stefan Savage, David Wetherall, Anna Karlin, and Tom Anderson. Network support for ip traceback. IEEE/ACM transactions on networking, 9(3):226– 237, 2001.

65 [38] Ben Shneiderman. The eyes have it: A task by data type taxonomy for information visualizations. In Visual Languages, 1996. Proceedings., IEEE Symposium on, pages 336–343. IEEE, 1996.

[39] Jonathan Steuer. Deﬁning virtual reality: Dimensions determining telepres- ence. Journal of communication, 42(4):73–93, 1992.

[40] Symantec Corporation. 2017 internet security threat report. https://www. symantec.com/security-center/threat-report, 2017. Online; accessed 20 September 2017.

[41] Unity Technologies. Execution order of event functions. https://docs. unity3d.com/Manual/ExecutionOrder.html, 2016. Online; accessed 15 October 2017.

[42] Unity Technologies. Learning the interface. https://docs.unity3d.com/ Manual/LearningtheInterface.html, 2016. Online; accessed 11 October 2017.

[43] James A Wise, James J Thomas, Kelly Pennock, David Lantrip, Marc Pottier, Anne Schur, and Vern Crow. Visualizing the non-visual: Spatial analysis and interaction with information from text documents. In Information Visualiza- tion, 1995. Proceedings., pages 51–58. IEEE, 1995.

[44] Thomas G Zimmerman, Jaron Lanier, Chuck Blanchard, Steve Bryson, and Young Harvill. A hand gesture interface device. In ACM SIGCHI Bulletin, volume 18, pages 189–192. ACM, 1987.