Challenges and opportunities in ARToolKit development Contents
Introduction...... 3
How does an ARToolKit function?...... 4
Marker and Non-marker based techniques for selecting an ARToolKit...... 5
Software toolkits available for building AR applications...... 7
Considerations for building an ARToolKit...... 9
Challenges with AR Development...... 10
How does the future look for ARToolKits?...... 11
hedgehoglab.com 2 Introduction
Recent developments in the sphere of HCI A review of scientific literature indicates (human-computer interaction) have opened new that AR is currently being used across many frontiers in immersive arenas including Virtual fields - extensively when it comes to teaching Reality (VR) and Augmented Reality (AR). Both geometry, spatial relationships and molecular technologies offer users a great interface and structures. And although the majority of people have been successfully used in areas ranging remain unfamiliar with it, we have no doubt it will from gaming and entertainment to education spread into the mainstream as more and more and even health. Our previous white paper AR apps make their way on to the market. on the subject focused predominantly on the driver’s, growth and challenges for VR. Now, by In saying this, developers need the best contrast, we would like to look at the growth possible toolkits to build good quality apps, and of AR, and the ARToolKits that are taking the these would combine a number of elements, software development world by storm. Indeed, such as voice, gesture, animation and persistent such tools have been instrumental in building data to create a mesmerising impact on users. some of the most powerful AR apps on the This technical white paper aims to improve market, across various platforms, including iOS understanding of the ARToolKit, explore the and Android. techniques for selecting toolkits, look at the popular toolkits available and consider what In terms of investment, all looks promising. In needs to be taken into account to build a high 2016, the AR market was valued at US$2.39bn. quality AR experience. By 2023, that figure is expected to reach US$61.39bn, growing at a CAGR of 55.71%. And, of course, advancements in computer graphics, Apple ARKit on iPhone X hardware, vision and UI (user interface) will all add impetus to this expansion.
AR, as we currently know it, represents a merger 2023 of several technologies, which aim to enhance EXPECTED VALUATION the user’s experience of the ‘real world’. Far $61.39bn To understand from being a novel development, it has existed for more than 35 years, since Ivan Sutherland AR, it is thus first introduced the Virtual Reality application. essential to To understand AR, it is thus essential to distinguish it from distinguish it from VR. The former overlays CAGR of virtual computer graphic images on top of the 55.71% VR. The former real world, supplementing reality as we know it, rather than replacing it. The latter, meanwhile, overlays virtual confines the user to a digitally constructed environment, with which he or she can interact. 2016 AR computer graphic A sound AR system would therefore be capable MARKET VALUE of merging computer-generated graphics with images on top the real world so seamlessly that it may even be $2.39bn hard to spot the difference between the two. An of the real world, ideal AR platform should comprise the following features: supplementing • The ability to combine real and virtual reality as we environments • The potential for real time interaction with know it, rather the user • Potentially registering real and virtual than replacing it. objects with one another
hedgehoglab.com 3 How does an ARToolKit function?
ARToolKits are mainly coded software libraries The answers to all of the above lie in five that enable developers to build Augmented simple steps. Reality apps using languages like C and C++. Such libraries, which were initially available • First, an image is captured through a for free, were developed for research and camera and transmitted to the app non-commercial purposes. However, with • The image is then fragmented into black the rise of powerful cellular devices, other and white squares digital gadgets and improved applications, • Computer vision algorithms calculate the newer commercial ARToolKits today assist position of the camera, relative to the black developers in creating and constructing mind- squares bending AR apps. • A process of ‘homography’ transforms the image inside the black squares and So the pertinent questions are: how does compares the template with the patterns in toolkit function in an AR app? What role the database. This process identifies the does the hardware play? What happens to marker the image when captured? How are markers • Once the marker is detected and the identified and what roles do markers play in position of the camera is known, a an ARToolKit? computer graphics model is drawn from the top of the video.
The next section throws light on two common techniques used for selecting an ARToolKit.
hedgehoglab.com 4 Marker and Non-marker based techniques for selecting an ARToolKit
Camera-based ARToolKits can be selected The camera-based techniques employed in AR using two techniques, marker-based and non- have opened up new frontiers in commercial marker-based. and entertainment spaces. We envisage that the following areas will be greatly impacted by AR Most AR apps with image recognition are marker- tracking: based. Digital images are identified with the help of optical square markers, usually comprising black Entertainment and Gaming squares within white boxes of predefined sizes. Markerless technology has the potential to take The square markers are identified with the support the entertainment sector to previously unimagined of a camera or a digital device using an algorithm. places. Imagine walking into a room and being The algorithm uses the image of the camera, able to weave your own gaming world... just as well as the decoded ID markers, to augment by scanning the information available to you. the virtual object onto the physical world model. Hardware such as the Microsoft HoloLens can Well-known examples of marker-based AR apps create incredible immersive experiences in the include: ScanLife, Popcode and, of course, Google gaming sphere. The beauty of AR lies in the fact Goggles. that it does not require a television, console or standard method of control. Markerless AR can With the advent of powerful HMDs (head mounted convert any plane into a gaming field and we are devices) and wearables accompanied by the certain it can change the face of the gaming world. best built-in cameras, AR apps are now capable of running markerless tracking. This medium allows the use of all small parts of the environment as a base, replacing the superimposed virtual objects. Markerless AR is thus assisted by one’s surroundings, rather than the static identified markers. Non-marker-based AR is capable of extracting and storing information about the surroundings for future use. The technology is preferred over marker-based technology, as it is non-invasive, light and can store larger images ARKit-Powered Game ‘The Machines’ on iPhone X from the environment for future use. Google’s Tango, for example, uses camera vision to detect Product showcasing and visualisation and scan the environment in real time and creates Markerless AR has the potential to impact our virtual images to superimpose on it, without using buying habits, especially when it comes to assets external signals or the GPS. requiring long-term investments. For instance, imagine you are considering buying an electric vehicle from the likes of Tesla. However, you’re unsure which colour it might look best in, or what it might feel like to drive it in the city. How about taking a prototype experience? Jelmer Verhool released a recent prototype of such an interactive experience from Oslo, using an Apple ARKit to showcase the unreleased Tesla 3.
On a similar note, imagine you want to get a new appliance fitted in your kitchen, but are unsure how the kitchen will look after this alteration. Markerless AR can help since it lets you use your mobile to scan the kitchen environment, and potentially place a virtual version of the product within it, taking customer visualisation to the next level. Such tools also allow customers to browse and shop in their Jelmer Verhoog - Tesla Model3 in Apple ARKit own time, free from pushy salespeople.
hedgehoglab.com 5 Marker and Non-marker based techniques for selecting an ARToolKit
Healthcare education and surgery Medical tutoring apps are capable of making healthcare education more immersive and interactive. This is especially so with apps made with the recent ARToolkits, as compared to those developed without ARKit features. So, how would ARKit framework help improve learning in medicine?
• Plane detections make it possible for medical students to study virtual 3D models directly Limpid Armor AR system for armoured vehicles on their desks. The learning experience is thereby rendered richer and more interesting, compared to learning through photographs AR apps in Navigation and videos. Enhanced GPS combined with AR would allow • Visual Inertial Odometry supports training easy movement from one location to another. medical interns. 3D models mean the Using a smartphone’s camera in combination anatomy, including organs, can be viewed with the GPS, users could view selected from any angle and distance. routes overlaid on the live view in front of their vehicles. ARToolkits can even make it easier, safer and faster to carry out surgery, improving a number of processes, from tumour identification and ARToolkits can even removal to visualising blood flows. make it easier, safer Military application In the army, HMDs with AR apps can provide and faster to carry out real time assistance to soldiers in battle zones. The devices can transmit critical information, surgery, improving a including the location of enemies, battle tanks or any other element that poses a threat. Similarly, number of processes, fighter jets can be equipped with ‘heads up’ AR apps featuring transparent displays to relay from tumour critical information such as altitude, airspeed, the horizon line and key data relevant to war zones. identification and removal to visualising blood flows.
Repairs and maintenance in manufacturing Using HMDs, technicians can see superimposed images of machines and engines with detailed information in the actual line of sight. Detailed procedures for fixing the machines could also be displayed in the corner of the devices, with live 3D images showing exactly what has to done to repair broken parts. AR apps can label essential components and help simplify the complex steps involved in major repairs. Similarly, simulations can be used to train technicians, significantly reducing training expenses. HoloLens Mixed Reality Surgery
hedgehoglab.com 6 Software toolkits available for building AR applications
AR applications are built using different software toolkits. In this section, we will compare them to help developers make wise choices.
Daqri-ARToolKit Initially introduced as an open source code, this toolkit was then acquired by Daqri, which now provides the commercialised version. The original toolkit still exists, though, and hence free libraries are available to all. The toolkit includes work instructions, data visualisation Limpid Armor AR system for armoured vehicles templates, and distinct methods and best practices for creating AR content. It supports the Vuforia following functions: One of the most popular platforms for AR development, Vuforia - which is supported on • Tracking black square and planar images Android, iOS, UWP and Unity Editor - boasts • 2D image recognition, camera calibration the following features: and optical stereo calibration Daqri Smart Glasses • Single camera orientation tracking • recognises the different types of visual and Helmet • Optical HMD display support objects (a box, cylinder, planes), as well as • Mapping additional elements via OpenGL text and environments (graphics library) • facilitates scanning of targeted objects using • Plugins for Unity and OpenSceneGraph Vuforia Object Scanner • Supports optical head-mounted display • integrates powerful plugins at various levels • Fast enough for real time AR applications • incorporates virtual buttons and maps additional elements via OpenGL Supported platforms: Android, iOS, Linux, • extends tracking to deliver a continuous Windows and Mac OS. visual experience, even when the target is out of sight The open source comes with disadvantages of its own — most significantly, the SDK’s Vuforia’s framework is incomplete, which can documentation is quite weak. And although the complicate matters for developers working with libraries contain prototypes for building test apps, the SDK for the first time. Commercial versions the illustrations are fairly mediocre and offer no of the software are free from watermark, while information on the plans for a framework update. the free versions are not particularly flexible to work with and less suited for cloud platforms.
Wikitude Wikitude, another powerful library, allows developers to create AR apps with next level functions including: reconstructing places on virtual maps; facilitating AR gaming; and offering real time searching of events, articles and social media posts. The kit can be used in both Android and iOS, and supports Unity. Moreover, the documentation of the library is well structured and elaborate. Developers can avail of the free versions initially. However, premium versions are chargeable on a regular basis.
Features of Wikitude include: image recognition and tracking, 2D and 3D tracking technology, GEO data, cloud recognition and Wikitude SDK animated 3D models HTML augmentation.
hedgehoglab.com 7 Software toolkits available for building AR applications
Kudan ARKit Kudan boasts one of the fastest frameworks Apple’s ARToolKit, known as the ARKit, has of all software toolkits. The SDK assists in been doing the rounds for quite some time, and mapping multi-polygonal models and facilitates was officially acknowledged at the WWDC 2017. easy import of 3D models from the modelling It was introduced to help developers integrate software packages. The libraries require minimal AR experiences into apps and games. Working memory space to store images on devices. with the iOS and using the phone’s camera, However, the toolkit library framework is short processors and motion sensors, the toolkit aims and developers would require more detailed to enhance immersive experiences. information to build AR apps. The library also suffers from limited built-in functionality as it has ARKit uses Visual Inertial Odometry (VIO) no direct access to OpenGL. Kudan offers the to track the space around Apple’s devices, following capabilities : including iPads or iPhones. VIO enables the devices to sense empty spaces while in motion. • Easy recognition of 2D and 3D images Further, the kit is capable of analysing room • Compatibility with Unity and the ability to layouts and detecting horizontal planes like generate databases tables and floors. This means virtual objects can • Allows markerless tracking (instead of fiducial be placed on those surfaces. Take, for instance, marks, it relies on the use of natural features IKEA’s new app, developed using ARKit. This like edges, corners or textures) enables customers to visualise the placement • Maps additional elements through separate and positioning of furniture on planes and component over OpenGL empty spaces before making purchases.
ARCore Supported platforms: Android and iOS. ARCore ARCore, Google’s toolkit, is available on Android Apart from the above, two new kits, introduced 7.0 Nougat and all more recent operating Apple and Google have made their foray to the systems. Developers can use it for building world of app development. It’s certainly worth apps for the Pixel, Galaxy S8 or any other OEM looking into these toolkits offered by the giants, devices. It works with an OpenGL, Unity and and how they are capable of possibly changing Unreal, and boasts three distinct features: the immersive landscape? • Motion tracking - Using the phone’s camera to detect an individual’s position in an open space • Environmental understanding - capable of detecting plane and horizontal surfaces in the surroundings • Light estimation - allows lighting and shadow of virtual objects to match the surroundings
ARCore supports other 3D tools from Google like Blocks and Tilt Brush, as well as gaming projects from the likes of Epic Games, Niantic (the builders of Pokemon Go) and Wayfair. Given Android’s huge market base - millions of devices are powered by the platform - the toolkit has the potential to become the biggest in the world.
Apple ARKit
hedgehoglab.com 8 Considerations for building an ARToolKit
As discussed previously, one of the Generation of user feedback fundamental problems involved in developing Tracking assists in analysing a real life image AR applications is that of tracking the user’s when a device is in motion. ARToolKit helps viewpoint. ARToolKit uses computer vision users understand the scene better, even if the algorithms to solve this issue by calculating the device moves subtly. However, vigorous motion real camera position and orientation relative to can result in a blurred image. Good quality AR physical markers in real time. It’s not an exact cameras provide tracking state information, science, but is impressively accurate, resulting which programmers can use for developing in realistic AR experiences. So the key question UI and to resolve issues related to low-quality is: how does one build a robust, high-quality tracking scenarios. AR experience? What factors need to be considered? Plane detection Plane detection is an important phenomenon Lighting conditions for AR, as it is capable of producing sharp, Tracking becomes challenging and tedious unequivocal images. ARToolKits refine the when lighting conditions are poor, since the position and time extent once a plane is process involves image analysis. Indeed, detected. tracking quality becomes abysmal when a camera is projected on a blank wall or used in an area that is too dark. Good quality lighting conditions are therefore advisable for generating amazing AR experiences.
ARCore plane detection
hedgehoglab.com 9 Challenges with AR Development
Building an AR application isn’t easy and GPS faults developers face challenges from two ends - Current digital devices can detect readings firstly on the hardware front and secondly when up to six metres away, quite enough for digital it comes to software. maps and navigation systems. However, GPS tracking can impact the AR software Hardware related challenges performance if the AR markers installed in the Sensor readings are crucial for AR apps. phones are not big enough. However, accelerometer and compass mechanisms in digital devices are not capable Integration and security issues of filtering electric inference from vastly Current web application designs restrict AR populated surfaces. The problem can possibly browsers to integrate their functions into be resolved through an exponential smoothing social media platforms. AR solutions are still technique, used for smoothing time series data. susceptible to data leaks and software is incapable of preventing hackers from invading Weak pattern recognitions the personal information of users. One possible Cameras in current hardware are well equipped solution, however, would be to use strong AR to take high quality 2D images. However, they frameworks in the first place. are still quite weak when it comes to identifying 3D images. One possible solution could be to Lack of multi-user experience use 2D QR and barcode as Augmented Reality AR solutions developed for digital devices are markers. incapable of providing a multi-user experience, with apps being limited to single users. Therefore, ARToolkits need to develop libraries that allow multi-user software to be built.
Google AR Experiments - Hidden World. Reveal an animated world at your feet. By Rachel Park Goto & Jane Friedhoff
hedgehoglab.com 10 How does the future look for ARToolKits?
With the current US$ 2bn market expected to The different ARToolKits discussed in this white grow around 30 times in 5-6 years the future paper offer various features - and a wide range certainly looks bright. Hardware improvements of possibilities. As toolkits become better, they in digital devices combined with rapid strides will provide an even wider range of options for in computer graphic development and UI developers looking to create world-class apps. design will also boost the growth of the sector. A quantum leap in AR technology is certainly on Development of non-invasive markerless the anvil, as the ARToolKit libraries present great technology is certainly set to disrupt the opportunities for the prominent platforms like field, taking over marker-based technology iOS and Android. and opening new avenues in areas like entertainment and commerce.
hedgehog lab AR projects Northumbria Water - read more. Road Respect - read more.
Microsoft and Zero Carbon Futures - read more
hedgehoglab.com 11 October 2017
References http://eprints.utm.my/6013/1/147-nihra.pdf https://developer.apple.com/documentation/arkit/understanding_ augmented_reality#see-also http://citeseerx.ist.psu.edu/viewdoc/ download?doi=10.1.1.64.5966&rep=rep1&type=pdf http://www.tinmith.net/lca2004/ARToolkit/ARToolKit2.33doc.pdf https://developer.apple.com/documentation/arkit/understanding_ augmented_reality https://www.ideals.illinois.edu/bitstream/handle/2142/27688/AR_Smart_ Phone_Note_rev3.pdf
Blog references http://cases.azoft.com/top-5-tools-creating-augmented-reality-apps/ https://thinkmobiles.com/blog/best-ar-sdk-review/ http://r-stylelab.com/company/blog/it-trends/the-challenges-of-augmented- reality-development https://thinkmobiles.com/blog/best-ar-sdk-review/ https://appreal-vr.com/blog/apple-arkit-healthcare-applications/
About us hedgehog lab is a global software consultancy that specialises in multi- platform software and connected device innovation. We architect, design, develop, and market apps, augmented reality, virtual reality, mixed reality & other digital solutions for phones, tablets, TV, kiosks & other post-PC devices.
We help brands & enterprises prepare for a future dominated by post-PC technologies. We specialise in working with our clients on Innovation & R&D on mobile & emerging platforms.
Founded as a pure play post-PC consultancy in 2010, and with offices in London, Newcastle Upon Tyne, Boston, Austin, India & Copenhagen, hedgehog lab has a team that is growing rapidly by focusing on delivering great post-PC experiences and campaigns to clients across the UK & abroad. hedgehog lab has architected and developed mobile solutions and software for clients like News International, Channel 4, Financial Times, Booths, Societe Bic, East Coast Trains, AkzoNobel, B&M, Mitsubishi, CBRE, England Rugby, NESN, Northumbrian Water, Camping and Caravanning Club, HULT, Thinking Digital and Grainger plc.
With smartphone sales accelerating at a rapid pace and tablet technologies being spear-headed by the iPad, we have bet our future on a completely post-PC world where mobile devices, smart connected devices and touch screen technologies out-pace fixed desktop PC devices.
We are one of the leading smart phone & tablet application developers in our region, working with cutting-edge mobile platforms like the iOS, tvOS, WatchOS, Android, Augmented Reality and Virtual Reality.
w: hedgehoglab.com e: [email protected] @hedgehoglab
Newcastle, UK London, UK Boston, USA Austin, USA Copenhagen, Denmark India Generator Studios, 18-21 Corsham Street, 15th Floor, 600 Congress Avenue, Hejrevej 30, Unit 301 Maple Celestia, Trafalgar Street, London, 50 Milk Street, 14th Floor, 2400 Nørrebro, Plot 49 Jayabheri Enclave, Newcastle upon Tyne, N1 6DR Boston, MA, Austin, TX, Copenhagen, Gachibowli, Hyderabad, NE1 2LA 02109 78701 Telangana 500032 t: 0191 249 8039 t: 020 3198 2470 t: 617-806-6434 t: 512-348-6566 t: +45 78755970 t: +9140 65888039