Challenges and opportunities in ARToolKit development Contents Introduction................................................................................................................. 3 How does an ARToolKit function?...................................................................... 4 Marker and Non-marker based techniques for selecting an ARToolKit.................................................................................. 5 Software toolkits available for building AR applications................................................................................. 7 Considerations for building an ARToolKit........................................................ 9 Challenges with AR Development.................................................................. 10 How does the future look for ARToolKits?...................................................... 11 hedgehoglab.com 2 Introduction Recent developments in the sphere of HCI A review of scientific literature indicates (human-computer interaction) have opened new that AR is currently being used across many frontiers in immersive arenas including Virtual fields - extensively when it comes to teaching Reality (VR) and Augmented Reality (AR). Both geometry, spatial relationships and molecular technologies offer users a great interface and structures. And although the majority of people have been successfully used in areas ranging remain unfamiliar with it, we have no doubt it will from gaming and entertainment to education spread into the mainstream as more and more and even health. Our previous white paper AR apps make their way on to the market. on the subject focused predominantly on the driver’s, growth and challenges for VR. Now, by In saying this, developers need the best contrast, we would like to look at the growth possible toolkits to build good quality apps, and of AR, and the ARToolKits that are taking the these would combine a number of elements, software development world by storm. Indeed, such as voice, gesture, animation and persistent such tools have been instrumental in building data to create a mesmerising impact on users. some of the most powerful AR apps on the This technical white paper aims to improve market, across various platforms, including iOS understanding of the ARToolKit, explore the and Android. techniques for selecting toolkits, look at the popular toolkits available and consider what In terms of investment, all looks promising. In needs to be taken into account to build a high 2016, the AR market was valued at US$2.39bn. quality AR experience. By 2023, that figure is expected to reach US$61.39bn, growing at a CAGR of 55.71%. And, of course, advancements in computer graphics, Apple ARKit on iPhone X hardware, vision and UI (user interface) will all add impetus to this expansion. AR, as we currently know it, represents a merger 2023 of several technologies, which aim to enhance EXPECTED VALUATION the user’s experience of the ‘real world’. Far $61.39bn To understand from being a novel development, it has existed for more than 35 years, since Ivan Sutherland AR, it is thus first introduced the Virtual Reality application. essential to To understand AR, it is thus essential to distinguish it from distinguish it from VR. The former overlays CAGR of virtual computer graphic images on top of the 55.71% VR. The former real world, supplementing reality as we know it, rather than replacing it. The latter, meanwhile, overlays virtual confines the user to a digitally constructed environment, with which he or she can interact. 2016 AR computer graphic A sound AR system would therefore be capable MARKET VALUE of merging computer-generated graphics with images on top the real world so seamlessly that it may even be $2.39bn hard to spot the difference between the two. An of the real world, ideal AR platform should comprise the following features: supplementing • The ability to combine real and virtual reality as we environments • The potential for real time interaction with know it, rather the user • Potentially registering real and virtual than replacing it. objects with one another hedgehoglab.com 3 How does an ARToolKit function? ARToolKits are mainly coded software libraries The answers to all of the above lie in five that enable developers to build Augmented simple steps. Reality apps using languages like C and C++. Such libraries, which were initially available • First, an image is captured through a for free, were developed for research and camera and transmitted to the app non-commercial purposes. However, with • The image is then fragmented into black the rise of powerful cellular devices, other and white squares digital gadgets and improved applications, • Computer vision algorithms calculate the newer commercial ARToolKits today assist position of the camera, relative to the black developers in creating and constructing mind- squares bending AR apps. • A process of ‘homography’ transforms the image inside the black squares and So the pertinent questions are: how does compares the template with the patterns in toolkit function in an AR app? What role the database. This process identifies the does the hardware play? What happens to marker the image when captured? How are markers • Once the marker is detected and the identified and what roles do markers play in position of the camera is known, a an ARToolKit? computer graphics model is drawn from the top of the video. The next section throws light on two common techniques used for selecting an ARToolKit. hedgehoglab.com 4 Marker and Non-marker based techniques for selecting an ARToolKit Camera-based ARToolKits can be selected The camera-based techniques employed in AR using two techniques, marker-based and non- have opened up new frontiers in commercial marker-based. and entertainment spaces. We envisage that the following areas will be greatly impacted by AR Most AR apps with image recognition are marker- tracking: based. Digital images are identified with the help of optical square markers, usually comprising black Entertainment and Gaming squares within white boxes of predefined sizes. Markerless technology has the potential to take The square markers are identified with the support the entertainment sector to previously unimagined of a camera or a digital device using an algorithm. places. Imagine walking into a room and being The algorithm uses the image of the camera, able to weave your own gaming world... just as well as the decoded ID markers, to augment by scanning the information available to you. the virtual object onto the physical world model. Hardware such as the Microsoft HoloLens can Well-known examples of marker-based AR apps create incredible immersive experiences in the include: ScanLife, Popcode and, of course, Google gaming sphere. The beauty of AR lies in the fact Goggles. that it does not require a television, console or standard method of control. Markerless AR can With the advent of powerful HMDs (head mounted convert any plane into a gaming field and we are devices) and wearables accompanied by the certain it can change the face of the gaming world. best built-in cameras, AR apps are now capable of running markerless tracking. This medium allows the use of all small parts of the environment as a base, replacing the superimposed virtual objects. Markerless AR is thus assisted by one’s surroundings, rather than the static identified markers. Non-marker-based AR is capable of extracting and storing information about the surroundings for future use. The technology is preferred over marker-based technology, as it is non-invasive, light and can store larger images ARKit-Powered Game ‘The Machines’ on iPhone X from the environment for future use. Google’s Tango, for example, uses camera vision to detect Product showcasing and visualisation and scan the environment in real time and creates Markerless AR has the potential to impact our virtual images to superimpose on it, without using buying habits, especially when it comes to assets external signals or the GPS. requiring long-term investments. For instance, imagine you are considering buying an electric vehicle from the likes of Tesla. However, you’re unsure which colour it might look best in, or what it might feel like to drive it in the city. How about taking a prototype experience? Jelmer Verhool released a recent prototype of such an interactive experience from Oslo, using an Apple ARKit to showcase the unreleased Tesla 3. On a similar note, imagine you want to get a new appliance fitted in your kitchen, but are unsure how the kitchen will look after this alteration. Markerless AR can help since it lets you use your mobile to scan the kitchen environment, and potentially place a virtual version of the product within it, taking customer visualisation to the next level. Such tools also allow customers to browse and shop in their Jelmer Verhoog - Tesla Model3 in Apple ARKit own time, free from pushy salespeople. hedgehoglab.com 5 Marker and Non-marker based techniques for selecting an ARToolKit Healthcare education and surgery Medical tutoring apps are capable of making healthcare education more immersive and interactive. This is especially so with apps made with the recent ARToolkits, as compared to those developed without ARKit features. So, how would ARKit framework help improve learning in medicine? • Plane detections make it possible for medical students to study virtual 3D models directly Limpid Armor AR system for armoured vehicles on their desks. The learning experience is thereby rendered richer and more