Markerless Augmented Reality Android App for Interior Decoration

International Journal of Engineering Research & Technology (IJERT) ISSN: 2278-0181 Vol. 2 Issue 4, April - 2013

Markerless Augmented Reality Android App For Interior Decoration Prasad Renukdas#1, Rohit Ghundiyal#2, Harshavardhan Gadgil#3, Vishal Pathare#4 #Department of Computer Engineering, MES College of Engineering, Pune, India

Abstract—We propose a markerlessapplication for pieces. A notable example of this is the interior decoration purposes, in which any novice user Google Glass Project, currently under can easily decorate his/her home. We include our development and expected to release observations and reminisce on potential future commercially around 2014. improvements. This paper also discusses mobile application of 2) Virtual Retinal Display: Augmented Reality (AR) on the Android platform. We A virtual retinal display (VRD) is a personal discuss existing SDKs for developing AR applications on display device under development at the mobile platforms and elaborate their functionalities and University of Washington's Human Interface limitations. Technology Laboratory. With this technology, a display is scanned directly onto the retina of Keywords— Augmented Reality, AR, interior decoration, a viewer's eye. The viewer sees what appears Metaio, Artoolkit. to be a conventional display floating in space I. Overview in front of them.

Augmented reality (AR) is a live, direct or 3) Handheld: indirect, view of a physical, real-world environment Handheld displays employ a small display that whose elements are augmented by computer-generated fits in a user's hand. All handheld AR sensory input such as sound, video, graphics or GPS solutions to date opt for video see-through. data. Initially handheld AR employed fiduciary This technology is still in its infant stage, markers, and current and future approaches particularly in mobile computing, but is quickly include GPS and markerless tracking through maturing with numerous innovative applications scene recognition. Handheld display AR already developed for archaeology, art, advertising, promises to be the first commercial success education, entertainment, medicine, industrial design,IJERT for AR technologies. The two main architecture, tourism and more. IJERTadvantages of handheld AR is the portable Earlier, backpack portable computers and nature of handheld devices and ubiquitous head mounted displays (HMDs) were used for nature of camera phones. augmented reality. The advancements in mobile hardware and software platforms brought AR into the B. SDKs currently available for Android development: hands of the common man. 1) ARToolkit: The hardware used for this paper was an It is an open source marker based Alcatel OT995 Ultra smartphone. It has 512 MB of AR library developed at the University of RAM, a 5 mega-pixel auto focus camera and a Washington. Qualcomm Snapdragon 1.4 GHz processor based on Some of the features of ARToolkit include: the ARM A9 (Cortex) architecture. The Eclipse IDE  Single camera position/orientation tracking. (ver Juno) was used to develop the applicationand 3D modeling was done in Blender.  Tracking code that uses simple black squares.  The ability to use any square marker patterns.  Easy camera calibration code. II. INTRODUCTION TO AUGMENTED REALITY  Fast enough for real time AR applications. A. Types of Displays Technologies.  SGI IRIX, Linux, MacOS and Windows OS distributions. 1) Eye Glasses:  Distributed with complete source code. AR displays can be rendered on devices It is designed for personal computers resembling eyeglasses. Versions include eye and not for embedded devices. Hence porting wear that employ cameras to intercept the real it directly to mobile platforms is difficult and world view and re-display its augmented view impractical because it uses a lot of FPU through the eye piecesand devices in which calculations. the AR imagery is projected through or reflected off the surfaces of the eye wear lens 2) ARToolkitPlus:

www.ijert.org 1367 International Journal of Engineering Research & Technology (IJERT) ISSN: 2278-0181 Vol. 2 Issue 4, April - 2013

This was an enhanced version in the The latest release, Metaio SDK 4.1 ARToolkit. The library was ported from C to includes AREL (Augmented Reality C++. It is available for download under Experience Language) which is a javascript general public license. binding of the metaio SDK's API in It does not read camera images or combination with a static XML content render geometry of any kind nor is it an out- definition. AREL allows scripting of powerful, of-the box solution. highly interactive Augmented Reality Some of the features of experiences based on common web ARToolKitPlus include: technologies such as XML, HTML5 and  Class-based API (compile-time JavaScript. AREL allows creation of platform parametrization using templates) independent applications instead of using  Up to 4096 id-based markers (no speed platform specific programming languages - penalty due to large number of markers) Java for the Android SDK, Objective C for  New camera pixel-formats (RGB565, Gray) iOS and C++ for Windows 7 and up.  Many speed-ups for low-end devices (fixed- III. RESTRICTIONS ON MOBILE DEVICES point arithmetic, look-up tables, etc.) Smartphones with RAM as low as 64 MB won‟t  New Pose estimation algorithm gives more support this app. Android system will force close the stable tracking (less jitter) app if it won‟t respond within stipulated time (3 to 5  Loads MATLAB camera calibration toolbox sec). The CPUs on the embedded processors mostly files don‟t allow parallel processing. Even then AR can  Automatic thresholding make use of multithreading which speeds up certain  Tools: pattern-file to TGA converter, pattern marking and detection algorithms. Many of the phones mirroring, off-line camera calibration, id- don‟t have a separate FPU. Hence the app which runs marker generator. on the phone will work 3 to 4 times slower than that of the average PC and will consume the battery more than 3) Vuforia: other apps due to more computations required. Another Vuforia is an Augmented Reality hurdle is the OpenGL. This app requires OpenGL ES SDK provided by Qualcomm, AG. It supports version 2.0 to run and will not run on lower versions. a variety of 2D and 3D target types including Image Targets, 3D Multi-Target IV. CHOICE OF SDK configurations, and a form of addressable Each SDK has its own merits and limitations. Fiduciary Marker known as a Frame Marker. We chose Metaio mobile SDK 4.1 to develop our Additional features of the SDK includeIJERT application for the following reasons: localized Occlusion Detection using „VirtualIJERT Buttons‟, runtime image target selection, and 1) High level of abstraction: the ability to create and reconfigure target sets Metaio implements complex computer vision programmatically at runtime. algorithms in native C++ libraries and ports The latest release of the SDK, them to Android. This abstraction enables a Vuforia 2.0, also supports cloud recognition, developer to assume low level where a user points his smartphone at a book, implementations and design and develop apps packaging or publication and information with greater ease, efficiency and enhances about the item like the price, reviews, etc are productivity. shown. 2) Advanced Tracking: 4) Metaio Mobile SDK: Metaio‟s tracking algorithms are highly The Metaio SDK includes a advanced, allowing us to develop apps free of powerful 3-D rendering engine in addition to visual artifacts. The algorithms are also plug-ins for Unity. It has advanced tracking greatly efficient, and can be executed in features, such as Markerless 2D and 3D reasonable time on sufficiently low cost Tracking, client-based visual search and devices. SLAM. Some of its features are:  Native App development for iOS, Android 3) Support for a wide range of formats: and PC (Windows) Modeling in 3D is an arduous and time  Includes powerful 3D Rendering engine. consuming task, and Metaio‟s support for obj,  Highly advanced tracking and minimal fbx and md2 model formats gave us greater artefacting. flexibility in our choice of 3D modeling tool  FREE commercial usage (with watermark) and made switching between and comparing  Huge developer support community. the benefits of different formats easier.

www.ijert.org 1368 International Journal of Engineering Research & Technology (IJERT) ISSN: 2278-0181 Vol. 2 Issue 4, April - 2013

4) Direct loading of 3D models with abstraction 푎 푖 = 0.1 × 푎 푖 + 0.9 × 푎 푖 − 1 of OpenGL calls: Unlike Vuforia and other SDKs which need 푇 Where 푎 = [ 푎 푥 푎 푦 푎 푧 ] and 푎 푖 is theith 3D models in obj format to be converted to orientation vector estimation. C++ header files (.h), Metaio directly reads the respective formats without conversion. This 2) Calculation of rotation matrix: allows faster and more efficient use of

smartphone resources like device memory and The rotation matrix, R, is a 3-by-3 minimizes I/O load times. matrix in the form: V. DESIGN AND IMPLEMENTATION

We have chosen a hybrid tracking system[18] 푅 = 푟 1푟 2푟 3  that uses integrated sensors embedded in the mobile  device like accelerometer and 2D visual tracking to  Where푟 1, 푟 2, 푟 3 are the three 3-by-1 column estimate pose and create an AR environment.The main vectors of R. 푟 1 represents the device‟s objective is to calculate pose which consists of six orientation vector as a unit vector: degrees of freedom. Three of the six degrees of  freedom are represented in terms of a translation 푎 푟 = 푎 =  vector 푡 , which relates the position of an observer 1 푎 relative to a plane and the remaining three in terms of  a rotation matrix R that relates the orientation of an  Where 푎 is the length of 푎 . observer relative to a plane.

Accelerometer measures at least two of the three degrees of freedom, R is determined using samples  푟 2 influences the direction rendered models from the accelerometer and 푡 must be determined will face and shouldbe an arbitrary unit vector on the plane perpendicular to 푎 definedby: using 2-D visual tracking.

A. Pose calculation Steps: 푎 푥 푥 + 푎 푦 푦 + 푎 푧 푧 = 0 1) Accelerometer values:  푎 푦  Choosing x=0 and y=1 so that z=− . We An accelerometer measures the 푎 푧 acceleration of the sensor relative to an object define in free-fall. The accelerometer measures the this new vector, 푏 as: force of gravity along each of its axes x, y IJERTIJERT and z. It is calibrated to return an acceleration 푎 reading of 1g upwards along its z-axis when 푏 = [0 1 − 푦 ]푇 it lies flat on a table since it is accelerating 푎 푧

upwards at 1g relative to an object in free-fall. Rotations are represented as a combination of 푟 2is then the device‟s rotated axes readings. This combination forms the acceleration vector, a: 푏 푟 = 푏 = (6) 2 푏 푇 푎 = [푎 푥 푎 푦 푎 푧 ] (1) Where 푏 is the length of푏 Where 푎 푥  푎 푦  푎 푧 arethereadings from the 푟 3 must be the cross product of 푟 1and 푟 2so accelerometer‟s x-, y- and z-axes respectively. that it is orthogonal:

An accelerometer can only determine two of the 푟 3 = 푟 1 × 푟 2 = 푎 × 푏 three degrees of freedom since rotationaroundthe device‟s z-axis when it‟s lying ona flat Substituting (3), (6) and (7) into (2) gives: surface will not result in a change in the force of gravity along any of its axes. We will “guess” the 푅 = [푎 푏 푎 × 푏 ] (8) third rotation component: direction. An accelerometer inside a moving object The OpenGL Model View matrix, returnsacceleration vectorsinstead of 푀푚표푑푒푙푣푖푒푤 is a 4-by-4 matrix that determines the orientation vectors.We find the rendering system‟s translation, orientationvectors of the device by using a low- rotation and scale. For multiplication with the pass filter on consecutive readings of 푎 : 푖 4-by-4 Model View matrix,푀푚표푑푒푙푣푖푒푤 , the R matrix

www.ijert.org 1369 International Journal of Engineering Research & Technology (IJERT) ISSN: 2278-0181 Vol. 2 Issue 4, April - 2013 must be represented in homogeneous 푄4 coordinates: = 푑푥 1 , 푑 푦 1 , 푑푥 2 , 푑 푦 2 , … 4 4 4 4

푟 푟 푟 0 R = 1 2 3 = 푎 푏 푎 × 푏 0 With Q1 being the top left quadrant of the 0 0 0 1 0 0 0 1 image, Q2 top right, Q3bottom left and Q4

Multiplying R with the model view matrix, bottom right. We then find푡푥and푡푦 for eachquadrant:

푀푚표푑푒푙푣푖푒푤 correctly rotates the OpenGL view to reflect the orientation of the real world in the 푡 푥 ,1 ,푡 푦 ,1 = 푄1 scene. We multiply these two matrixes by first 푡 푥 ,2 ,푡 푦 ,2 = 푄2 initializing the Model View matrix and multiplying it by R: 푡 푥 ,3 ,푡 푦 ,3 = 푄3

glMatrixMode(GL_MODELVIEW); 푡 푥 ,4 ,푡 푦 ,4 = 푄4 glLoadIdentity(); glMultMatrixf(R); where the angle brackets represent averaging

over the arrays Q1,Q2, Q3 and Q4.We then find푡 as 3) Calculating Relative Translation Between 푧 follows: Two Frames:

푡 = 푡 +푡 − 푡 + 푡 We define 푡 (and 푡 ) to be the 푧 푥,1 푥,4 푥,2 푥,3 푥 푦 relative change in translation of acamera Translation and scale is represented betweenthe current frame and the previous frame along the camera‟s xand by the translation vector, 푡 .We use an y-axis in image coordinates. iterative approach to estimate 푡 , updating it When tracking points between accordingto the tracking results between consecutive frames using optical flow, we each two frames. We use a obtainanarrayrepresenting the relative poseestimate buffer to store three movement of all matched points between valuesrelated to translation that arethe total frames. The array is given as a set of (dx,dy) translations along each axis of the camera movement values,one for each tracked point. relative to aninitial starting position: To find 푡 푥and푡 푦 we average all the dxand dy  Total translation along the camera‟s values for two consecutive frames. For n x-axis:푡 푥 ,푡 표 푡 푎 푙. points tracked:  Total translation along the camera‟s IJERTIJERT 푛 y-axis:푡푦,푡표푡푎푙. 1  Scale. Scale is equivalent to 푡 , 푡 = 푑푥 , 푑 푦  푥 푦 푛 푘 푘 translation along the camera‟s z- 푘=1 axis:푡 .  푧,푡표푡푎푙

 ofind the change in scale between two Overall translation, 푡 , is the consecutive frames ( 푡 )is slightly trickier. 푇표푡푎푙 푧 compound effect of all iterative Whenthe camera is translated along its z-axis measurements of change in translation that the movement of points in the top and left between two frames since the start of segments of an image move in the opposite tracking. direction relative to pointsin the bottom and right segments. therefore thearray of (dx, 푇 dy)values is split up into four quadrants: 푡 푡 표 푡 푎 푙 = 푡 푥 ,푡 표 푡 푎 푙푡 푦 , 푡 표 푡 푎 푙 푡 푧 ,푡 표 푡 푎 푙   Every time a new frame is processed 푄1 and a relative change in translationfrom theprevious frame is calculated, ,we = 푑푥1 1 , 푑푦1 1 , 푑푥1 2 , 푑푦1 2 , … 푡푖 update 푡 푇 표 푡 푎 푙 by adding 푡 푖: 푄2 푡 = 푡 + 푡 = 푑푥2 1 , 푑푦2 1 , 푑푥2 2 , 푑푦2 2 , … 푇표푡푎푙,푖 푇표푡푎푙,푖−1 푖

푄3 Where푡푇표푡푎푙,푖 is the ith estimate of 푡 푇 표 푡 푎 푙 and is the relative changein translation = 푑푥3 1 , 푑푦3 1 , 푑푥3 2 , 푑푦3 2 , … 푡푖 between the current frame i and i-1.

www.ijert.org 1370 International Journal of Engineering Research & Technology (IJERT) ISSN: 2278-0181 Vol. 2 Issue 4, April - 2013

Eclipse lets us design powerful and beautiful layout for the GUI, and Android, from 2.3 B. Algorithm Steps[2]: (Gingerbread) onwards, has significantly improved the WebView class to render web Initialization pages faster and make browsing and touch 1. Initialize the video captureand events more responsive. camera parameters and acquire device position information from the The Rendering interface: magnetic compass and As the camera captures the image accelerometer. and renders the chosen item, items similar to Main Loop current are shown in the top left corner. We 2. Freeze video input frame. believe this will improve accessibility and 3. Getdevice position and improve user satisfaction. A few screenshots orientation from the magnetic of the app in action are as follows: compass. 4. Calculate the camera transformation relative to the device position. 5. Draw the virtual objects in the center of the screen. Shutdown 6. Close the video capture down.

C: The App: The User Interface

The user interface navigation screens in the app will be designed asnative Android layouts. The WebViewclass in Android enables loading and resizing of web pages at runtime, which will be used for the start screen. A few preliminary designs of the user interface include: IJERT IJERT

Figure 5:The app in action, rendering 3D objects without Fiduciary markers

D. 3D Modeling: We choose the open source tool, Blender for creating our 3D models because it is free and easy to use, and has extensive documentation. It also runs well on a moderately powered PC, like the ones the authors possessed. The following is a screenshot of a model of a sofa under development:

Figure 3:Item selection screen, level 1

www.ijert.org 1371 International Journal of Engineering Research & Technology (IJERT) ISSN: 2278-0181 Vol. 2 Issue 4, April - 2013

2) This app can be further improved and extended for architecture, civil engineering, advertorial and other purposes.

VIII. CONCLUSION This paper provided a brief introduction to handheld Augmented Reality, discussed the various free SDKs available for developing AR applications and outlined their merits and limitations. We also discussed the hurdles in achieving fast and efficient tracking on mobile devices. In the context of this paper an Android application for interior decoration purposes is being Figure 6: A sofa 3d model under development developed using the Metaio SDK 4.1 described earlier. The model requirements of the Metaio SDK were sufficiently flexible and resistant to tracking artefacts, without being IX. ACKNOWLEDGEMENT too restrictive, unlike those of the Vuforia The authors wish to thank all referees and all SDK from Qualcomm which is another those who contributed to this project in their own small reason we chose Metaio over Vuforia. ways. Special thanks to Dr. Ahmad Raza Khan for 3D modeling was done on a desktop providing inspiration to use the Android platform for PC with an AMD Phenom II X4 Quad Core this project and to Prof. RevatiWahul for providing Processor, 3.5 GB RAM and on a Dell guidance for this paper. Insperon 15R laptop with an Intel i3 dual core processor and 3 GB RAM. REFERENCES

VI. OBSERVATIONS 1] Woohun Lee; Jun Park; , "Augmented foam: a tangible augmented reality for product design," Mixed and Augmented Reality, 2005. Proceedings. Fourth IEEE 1) Application works perfectly on high end and ACM International Symposium on , vol., no., pp. phones but performance degrades on low end 106- 109, 5-8 Oct. 2005 2] Tobias Domhan, “Augmented Reality on Android phones. Smartphones”. 2) Very high quality models give some minimalIJERT 3] Gabriel Takacs, Vijay Chandrasekhar, Bernd Girod, artifacting. IJERTRadekGrzeszczuk, “Feature Tracking for Mobile 3) Models are right now locally stored, resulting Augmented Reality Using Video Coder Motion Vectors”. 4] Kenji Oka and Yoichi Sato Hideki Koike, “An in high size of application and initial loading Augmented Reality Application for Previewing 3D times. Décor Changes”. 4) Application can run in realtime video mode as 5] Agusanto, K.; Li Li; Zhu Chuangui; Ng Wan Sing; , well as static photo mode. "Photorealistic rendering for augmented reality using environment illumination," Mixed and Augmented 5) All models in a particular room are loaded Reality, 2003. Proceedings. The Second IEEE and ACM immediately loaded when activity for that International Symposium on , vol., no., pp. 208- 216, 7- room starts, resulting in high memory usage 10 Oct. 2003 and loading times. 6] Ruobing Yang; , "The study and improvement of Augmented reality based on feature matching," Software 6) Models possess low shadow and reflection Engineering and Service Science (ICSESS), 2011 IEEE effects, resulting in a not-so-realistic feel. In 2nd International Conference on , vol., no., pp.586-589, the future, models can be improved for cutting 15-17 July 2011 edge hardware. 7] MartinKurze, Axel Roselius; ,” Smart Glasses: An Open Environment for AR Apps”, Deutsche Telekom AG, Laboratories &funijiInc 8] Keil, J.; Zollner, M.; Becker, M.; Wientapper, F.; VII. FUTURE SCOPE Engelke, T.; Wuest, H.; , "The House of Olbrich — An Augmented Reality tour through architectural history," Mixed and Augmented Reality - Arts, Media, and 1) The models can be stored on file hosting sites Humanities (ISMAR-AMH), 2011 IEEE International and loaded on demand. Although this will Symposium On , vol., no., pp.15-18, 26-29 Oct. result in increased performance and low 2011Keil, J.; Zollner, M.; Becker, M.; Wientapper, F.; memory consumption and increased reliability, Engelke, T.; Wuest, H.; , "The House of Olbrich — An Augmented Reality tour through architectural history," it will require the user to have a high speed Mixed and Augmented Reality - Arts, Media, and internet connection. Humanities (ISMAR-AMH), 2011 IEEE International Symposium On , vol., no., pp.15-18, 26-29 Oct. 2011

www.ijert.org 1372 International Journal of Engineering Research & Technology (IJERT) ISSN: 2278-0181 Vol. 2 Issue 4, April - 2013

9] BorkoFurht, “Handbook of Augmented Reality”, ISBN- 10: 1461400635 10] http://dev.metaio.com/arel/overview/ 11] http://dev.junaio.com/arel/documentationArelJS/ 12] http://handheldar.icg.tugraz.at/artoolkitplus.php 13] https://developer.vuforia.com/ 14] http://www.hitl.washington.edu/artoolkit/ 15] http://wiki.blender.org/index.php/Doc:2.6/Manual 16] http://www.blender.org/education-help/ 17] https://developer.vuforia.com/resources/sample- apps/frame-markers-sample-app 18] Carel P.J. van Wyk, “Markerless Augmented Reality on Mobile Devices with Integrated Sensors”

IJERTIJERT