The AM-Bench: An Android Multimedia Benchmark Suite Chayong Lee Euna Kim Hyesoon Kim School of Computer Science Georgia Institute of Technology fchayong82, [email protected], [email protected] Abstract—Despite the significant evolution of mobile devices Galaxy Tab 7.0 [10], Galaxy Player YP-GP1 [9] and Nexus and the increased use of mobile devices, not many mobile One architectures [4]. benchmarks have been studied. Even though mobile applications share similar characteristics with traditional desktop oriented applications, different programming environments and user us- II. BACKGROUND age patterns present different characteristics. In this paper, we The Android system includes a three part of software layer; introduce an open source based mobile multimedia benchmark for Android platforms (AM-Bench). The AM-Bench consists of mobile applications, middle-ware, and operating system [18]. several multimedia benchmarks running on Android platforms. Most Android applications are written in Java, and they We explain the characteristics of the AM-Bench and compare are compiled as the .class file format. The Android system performance on four Android-based platforms. converts .class files into a Dalvik Executable (.dex) file and I. INTRODUCTION executes on the Dalvik virtual machine in the middle-ware layer. Each Android application runs on its own process with The popularity of Smart phones shifts the major com- its own instance of the Dalvik virtual machine. puting paradigms. Mobile platforms have different types of applications than those of traditional desktop applications. A. Android Software Stack Nonetheless, not many benchmarks are available to understand the characteristics of the benchmarks and architectures. 1) Application Layer: The application layer is the highest Traditional benchmark suites such as MiBench [19] and layer in the architecture and contains the built-in applications SPEC benchmark suite [25] are not adequate to characterize such as Phone Dialer, Email, Contact, Camera, Gallery and the performance of mobile devices due to the compatibility others. The application framework layer provides the core issue with mobile device platforms such as Android and iOS. platform services such as activity manager, package manager, These new systems all run programs written in new object window manager and others. This layer provides an abstract oriented program languages such as Java and Objective-C, interface to hardware access and also provides a way to which show very different program characteristics than many manage the user interface or application resources. The appli- other desktop applications. This fact provides the motivation cation framework layers enable developers to piece together to propose a new commercially free benchmark suite for the services to create an Android application. mobile devices. We present Android Multimedia Benchmark 2) Middle-ware Layer: The middle-ware layer includes Suite, called AM-Bench. AM-Bench is composed of Android C/C++ core libraries and Android Run Time. C/C++ core open source applications that are drawn from the multime- libraries can be classified into four categories; bionic libc, dia domain. Our survey of Android applications shows that function libraries, native servers, and hardware abstraction multimedia benchmarks are one of the most commonly used libraries. These core libraries support built-in Android-specific application categories. Since mobile applications are heavily services and standard video, audio and other data storage. dependent on underlying operating systems and programming The Android run time comprises core libraries that are platforms such as libraries and we are targeting the develop- required for Java libraries and the Dalvik VM, which is a ment of an open source benchmark suite, an Android operating register based virtual machine. The Dalvik VM is optimized system is chosen. Android instrumentation [17] also provides for low memory requirements and designed to support multiple end users an easy interface for running each benchmark in virtual machine processes per device with optimized bytecode AM-Bench. interpreter using runtime memory efficiently. The Dalvik VM In this paper, we analyze end users’ activities on mobile also supports process isolation, memory management and devices and discuss the representatives of Android-open source threading from the underlying operating system. application in the multimedia domain. We then describe our 3) Operating System: Android relies on the Linux Kernel benchmarks, Android Multimedia Benchmark Suite, called and communicates underlying hardware. This layer includes AM-Bench. Using AM-Bench, we evaluate the performance of Android’s memory management programs, security settings, four different mobile devices in terms of application execution power management software, several drivers, file system ac- time and CPU performance on NVIDIA Tegra 2 [8], Samsung cess, networking and inter-process-communication. B. Memory Limitation as heavy load on animation, mapping tests, user interface One of the key differences between mobile applications simulation and etc. However, JBenchmark is only compatible and desktop applications is the limitation of memory usage. with J2ME enabled devices. GLBenchmark [22] is a 3D test Multimedia applications often deal with memory intensive suite designed for OpenGL ES in various operating systems; items such as images, audio, and video, so the memory Android, iOS, Linux, Symbian and Windows Mobile. This is a limitation is the key challenge of multimedia application on licensed commercial 3D testing tool that does not allow source mobile devices. Also, how to utilize the memory limitation code to be manipulated. Recently, BBench [20] was proposed significantly determines the performance of the application. for modern Smart phones. However, BBench only targets Web Typically, each application on an Android platform is allowed browsers. to use 16MB of memory (24 MB for recently released Android B. User Preferences of Mobile Application devices) [1]. Android devices with 16 GB and 32 GB are common these day; however, that is solid state storage and To analyze user preferences of mobile applications, the not RAM. top 50 Android applications have been investigated [15]. We assume that the most frequently downloaded applications In AM-Bench, we exam how Android devices can handle 2 this memory limitation issue and how they utilize the sec- would run on the mobile devices most frequently. ondary storage to overcome this problem. We will describe this problem in more detail using the FBReader application. C. Java Native Interface (JNI) Some of the benchmarks in AM-Bench examine a com- putational performance using JNI. JNI refers to Java Native Interface [5] and it allows the interaction between Java VM and native code. There are two types of interactions [14], downcalls and upcalls. Downcalls is what Java applications call a native method1 and upcalls is what a native method needs to access Java applications for the application’s re- sources. Although the Android system does not include Java Fig. 1. Top 50 Android Applications VM, it is still possible to use the JNI interface in an Android Figure 1 shows the top 50 free Android applications avail- application using Android NDK [2]. able in the official Android market from mid 2011. Since III. MOTIVATION the Android market categorizes the application into several A. Existing Benchmark Suites categories, we narrow down the categories to ensure the users preferences more clearly. We suggest three new categories; To the best of our knowledge, a benchmark suite specifi- Information/Communication, Multimedia, and Utility. We re- cally for Android mobile applications has not yet proposed. arrange the top 50 Android applications into the new categories There are some benchmarks for mobile devices, however, they as shown in Table I. As the Table I shows the total number typically they target one specific feature on mobile devices. of applications in Multimedia categories is more than twice Also traditional benchmark suites such as MiBench [19] or larger than the total number of applications in the Informa- PARSEC [13] are not adequate to characterize the performance tion/Communication category and three time larger than the of mobile devices due to the compatibility issue with mobile total number of applications in the Utility category. devices’ operating system such as iOS, or Android. SD- If we consider native multimedia Android applications such VBS [26] is the San Diego Vision Benchmark Suite that as Camera, Gallery, Music, and others, the importance of comprises vision applications drawn from the vision domain. multimedia applications is somewhat higher than what we can However, the applications are written in Matlab and C so see from Table I. Based on this, we can ensure that the most there is no known way to run code on an Android plat- frequent and important user activities on Android devices are form. DENBench [24] is a software benchmark that contains related to the multimedia applications. This fact also conforms widely used algorithms in mobile applications such as AES to the necessities of our benchmark suite, which is composed (Advanced Encryption Standard), DES (Digital Encryption of multimedia Android applications. Standard) and others. However, DENBench only evaluates the media library and this is not a collection of representative IV. BENCHMARK SUITE mobile applications. DENBench is not an open source or In this section, we describe the