Moby: a Mobile Benchmark Suite for Architectural Simulators
Total Page：16
File Type：pdf, Size：1020Kb
Moby: A Mobile Benchmark Suite for Architectural Simulators Yongbing Huang∗y, Zhongbin Zha∗y, Mingyu Chen∗, Lixin Zhang∗ ∗State Key Laboratory of Computer Architecture, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China yUniversity of Chinese Academy of Sciences, Beijing, China Email:fhuangyongbing, zhazhongbin, cmy, [email protected] Abstract—Mobile devices such as smartphones and tablets Snapdragon [6] are more prevalent than processors like Intel’s have become the primary consumer computing devices, and Atom [7]. Generally, as the performance of these mobile their rate of adoption continues to grow. The applications that processors improves, their microarchitectures become more run on these mobile platforms vary in how they use hardware complicated. For example, mobile processors with four-cores, resources, and their diversity is increasing. Performance and an out-of-order execution model, and two-level caches have be- power limitations also vary widely across mobile platforms. Thus come the mainstream. Mobile system designers must consider there is a growing need for tools to help computer architects design systems to meet the needs of mobile workloads. Full-system how application and OS diversity affect their design choices simulators are invaluable tools for designing new architectures, in this increasingly complex design space. but we still need appropriate benchmark suites that capture the Benchmarking and architectural simulation are two im- behaviors of emerging mobile applications. Current benchmark portant tools for processor design and computer architecture suites cover only a small range of mobile applications, and many cannot run directly in simulators due to their user interaction research. To be relevant, a benchmark suite for architectural requirements. research must satisfy at least two properties. First, work- loads in the benchmark suite should be diverse enough to In this paper, we introduce and characterize Moby, a bench- exhibit the range of behaviors of the target applications. mark suite designed to make it easier to use full-system architec- Second, all the applications should be portable to architec- tural simulators to evaluate microarchitectures for mobile pro- tural simulators. However, most of current mobile benchmark cessors. Moby contains popular Android applications, including a web browser, a social networking application, an email client, a suites represent only a small subset of mobile application- music player, a video player, a document processing application, s [8], [9], [10], [11], [12], [13], and some cannot be run directly and a map program. To facilitate microarchitectural exploration, in simulators due to user-interaction requirements (e.g., the we port the Moby benchmark suite to the popular gem5 simulator. interactive games and audio player of Gutierrez et al. [14]. We characterize the architecture-independent features of Moby Meanwhile, existing benchmarks such as SPEC CPU2006 applications on the simulator and analyze the architecture- exhibit significantly different behaviors from interactive mobile dependent features on a current-generation mobile platform. Our applications [13], [14]. results show that mobile applications exhibit complex instruction execution behaviors and poor code locality, but current mobile In this paper, we develop Moby, a benchmark suite de- platforms especially instruction-related components cannot meet signed to evaluate microarchitectures of mobile platforms in their requirements. full-system architectural simulators. Generally, there are two design issues that drive our benchmark suite. First, mobile I. INTRODUCTION applications on different operating systems are incompatible. Since Android is the commonly used operating system for Mobile devices, especially smartphones and tablets, have mobile devices, Moby contains only mobile applications that become an important world-wide market. From the application run on the Android OS. Second, most popular mobile ap- point of view, a wide variety of mobile applications are now plications are commercial, and thus their source codes are widely used; these include web browsers, social networks, e- not generally available. We choose only applications that can mail clients, audio and video players, document processing be freely downloaded, in order to avoid licensing issues. In systems, and map programs, to name a few. Different types of total, Moby contains 10 mobile applications spanning nine applications present different requirements for the hardware categories, including a web browser, a social networking components of mobile platforms. From the mobile operating application, email, audio, video, document, map and game. system point of view, Android [1] and iOS [2] have the highest Except the web browser application BBench [14], all the other market occupancy and growth speed. Android adoption has applications are selected from Google Play Store [15]. ramped up quickly, gaining in popularity six times faster than iOS. Android and iOS use different programming languages Since our benchmark suite is intended to drive architec- and execution models, and they differ in their utilization of tural simulators, all applications should be executable without hardware resources. Therefore, the requirements placed on manual user inputs. Although AutoGUI [13] provides a user hardware resources for different mobile applications vary. interface automation tool to record and deterministically replay user actions, we use an alternative method to bypass user As for mobile platforms, ARM [3] based mobile proces- interaction by executing only typical representative operations sors such as Apple’s Ax [4], TI’s OMAP [5], Qualcomm’s for these mobile applications. While Moby can be executed on many simulators that support Android OS, we take the Benchmark suites intended to support architectural design commonly used gem5 simulator [16] as an example to test space exploration and system research should make it pos- and characterize Moby. The gem5 disk image for Moby has sible to instrument, manipulate, and model the constituent already become public [17]. applications in detail. However, most popular mobile appli- cations are commercial, which complicates instrumentation We measure microarchitecture-independent features on the because their source codes are unavailable. Although most gem5 simulator with the ARM ISA and microarchitecture- Moby applications lack source codes, all can be downloaded dependent features on the ARM-based Pandaboard develop- for free. Note that most mobile applications involve user ment board [18]. The microarchitecture-independent features interaction and require a network connection, both of which are include instruction mix, working set size, data and instruc- difficult to implement or model in architectural simulators. The tion locality, and binary execution behaviors. The instruction dependence on networks can easily be removed by buffering features show that the representative operations for all applica- any required remote data in local storage. tions execute several billion instructions and that nearly 70% of branches are conditional. Furthermore, most applications 2) User Interaction: A major difficulty in analyzing inter- spawn about 20 processes and invoke more than 20 libraries. active mobile applications is generating reproducible results On the Pandaboard, measured microarchitecture-dependent without manual user inputs. Moreover, the slow execution features mainly include CPI and the behaviors of the branch speed of full-system simulators makes it impractical to incor- predictor, cache, TLB, and memory components. Experimental porate user-action inputs in experiments. There are two main results demonstrate that all applications present high CPIs, solutions to cope with user interaction in simulators. Tools like which implies that these mobile applications and current ARM- AutoGUI [13] and Xnee [19] provide automation capabilities based mobile platforms are not well matched. In particular, to record and replay user inputs. Using similar tools, we are the instruction-related resources (branch predictor, instruction working to identify representative code pieces that suffer from cache, and TLB) suffer from serious miss rates. large response latencies. However, most current automation tools still suffer from shortcomings like nondeterministic re- In summary, we make three contributions: play. • We present Moby, a new mobile benchmark suite that The other simple solution is to avoid user interaction contains a diverse set of applications and is appropriate in simulators. We find that most main activities of mobile for simulation-based design space exploration. applications can be executed as separate processes, and user • We extract typical representative operations of interactive interactions mainly used specify inputs for these activities. By applications in Moby and automatically execute them on executing these activities by specifying their inputs manually full-system architectural simulators. in command lines, user interactions are no longer required. • We describe both microarchitecture-independent and Compared to using automation tools, this method is both microarchitecture-dependent characteristics of all Moby simple and efficient, and thus we adopt this approach for Moby. applications. The main activities illustrated above are considered to be The rest of this paper is organized as follows. Section representative operations of mobile applications, and they