FLUID: Flexible User Interface Distribution for Ubiquitous Multi-device Interaction Sangeun Oh*, Ahyeon Kim*, Sunjae Lee*, Kilho Lee* Dae R. Jeong*, Steven Y. Koy, Insik Shin* *KAIST, South Korea yUniversity at Buffalo, The State University of New York, USA {ohsang1213,nonnos,sunjae1294,khlee.cs,dae.r.jeong,insik.shin}@kaist.ac.kr [email protected] ABSTRACT CCS CONCEPTS The growing trend of multi-device ownerships creates a need • Human-centered computing → Graphical user in- and an opportunity to use applications across multiple de- terfaces; User interface management systems; Ubiqui- vices. However, in general, the current app development and tous computing; Mobile computing. usage still remain within the single-device paradigm, falling far short of user expectations. For example, it is currently KEYWORDS not possible for a user to dynamically partition an existing Multi-device Mobile Platform; Multi-surface Computing; live streaming app with chatting capabilities across different User Interface Distribution; devices, such that she watches her favorite broadcast on her smart TV while real-time chatting on her smartphone. ACM Reference Format: In this paper, we present FLUID, a new Android-based Sangeun Oh, Ahyeon Kim, Sunjae Lee, Kilho Lee, Dae R. Jeong, multi-device platform that enables innovative ways of using Steven Y. Ko, and Insik Shin. 2019. FLUID: Flexible User Interface multiple devices. FLUID aims to i) allow users to migrate or Distribution for Ubiquitous Multi-device Interaction. In The 25th replicate individual user interfaces (UIs) of a single app on Annual International Conference on Mobile Computing and Network- multiple devices (high flexibility), ii) require no additional ing (MobiCom ’19), Oct. 21–25, 2019, Los Cabos, Mexico. ACM, New development effort to support unmodified, legacy applica- York, NY, USA, 16 pages. https://doi.org/10.1145/3300061.3345443 tions (ease of development), and iii) support a wide range of apps that follow the trend of using custom-made UIs (wide 1 INTRODUCTION applicability). Previous approaches, such as screen mirror- Recent years have seen rapid development of mobile com- ing, app migration, and customized apps utilizing multiple puting and IoT technologies, such as smartphones, tablets, devices, do not satisfy those goals altogether. FLUID, on the smart home appliances, and smart cars. These smart devices other hand, meets the goals by carefully analyzing which are now a natural part of our everyday lives, and many users UI states are necessary to correctly render UI objects, de- own and interact with multiple devices. According to Cisco’s ploying only those states on different devices, supporting Trend Report [9], there will be 8.9 to 13.4 connected devices cross-device function calls transparently, and synchronizing per user in Western Europe and North America by year 2020. the UI states of replicated UI objects across multiple devices. It will be unsurprising to see the trend continue beyond that. Our evaluation with 20 unmodified, real-world Android apps With the ownership of multiple devices, one can envision shows that FLUID can transparently support a wide range many interesting and useful use cases that allow users to of apps and is fast enough for interactive use. interact with an application using the multiple screens of different devices (so-called multi-surface computing). More Permission to make digital or hard copies of all or part of this work for specifically, there are three driving factors that make multi- personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear surfaces a more attractive interaction environment. i) Multi- this notice and the full citation on the first page. Copyrights for components function: a user can distribute different functionalities offered of this work owned by others than ACM must be honored. Abstracting with by a single application across multiple surfaces. For example, credit is permitted. To copy otherwise, or republish, to post on servers or to a user can watch a live-streaming video on a surface while redistribute to lists, requires prior specific permission and/or a fee. Request chatting with other viewers in real-time using an in-app permissions from [email protected]. messenger on another surface, as illustrated in Figure 1. ii) MobiCom ’19, October 21–25, 2019, Los Cabos, Mexico © 2019 Association for Computing Machinery. Multi-device: different tasks can have different device prefer- ACM ISBN 978-1-4503-6169-9/19/10...$15.00 ences due to their distinct form factors. For example, a user https://doi.org/10.1145/3300061.3345443 can watch a movie on the larger screen of a smart TV while a Host device Guest device same screen content on multiple surfaces. It does not allow No name No name any other level of granularity, e.g., displaying different UI Goal!!!!! Goal!!!!! Nice Goal!! User 1 User 1 What a goal What a goal elements on different surfaces. On the other hand, the app Nickname Nickname Wow!!! Wow!!! migration approach allows the app to use only one surface at a time and not multiple surfaces simultaneously. For in- stance, it does not allow an app to display a video clip on a What just happened? smart TV and a video progress bar on a smartphone at the same time. In this paper, we propose a new multi-surface platform, FLUID (FLexible UI Distribution)1, a systems solution that Single Surface Multi Surface overcomes the limitations of previous approaches and aims Figure 1: Live streaming app on single- and multi- to achieve the following design goals. i) Flexibility: the unit surfaces; Multi-surfaces allow users to enjoy live of deployment between different devices should be as fine- streaming in full screen without interruption from grained as possible to allow users to utilize multi-surfaces keyboard. to their maximum flexibility. ii) Ease of development (trans- parency): it should not require any additional complexity for video progress bar on her smartphone, which is easier to in- developing multi-surface applications, as compared to the de- teract than a smart TV remote. iii) Multi-user: some tasks are velopment of single-device applications. It should also be able associated with multiple users. For example, when booking to support existing unmodified applications. iii) Applicabil- flight tickets for a group of colleagues, a user can distribute ity: it should be able to support a wide range of applications, personal information input boxes to other smartphones in including the applications using custom UI components. iv) order that each colleague can provide their personal infor- Responsiveness: it should provide prompt response to user in- mation (i.e., passport number) on his/her own smartphone. teraction across devices. For instance, users expect feedback Previous solutions for supporting multi-surface use cases on their inputs within 50 to 200 ms [5, 37, 46]. typically fall into one of the four categories with various lim- In order to satisfy the above design goals, FLUID allows itations. First, some custom apps are developed with specific users to dynamically select individual UI elements from the multi-surface usages in mind, such as cross-device contin- surface of one device (the host), migrate or replicate them on uous video display (e.g., Netflix [35]) and multi-user collab- the surfaces of different devices (guests), and interact with orative document editing (e.g., Google Docs [19]). Yet, this them on all or some of the surfaces. In order to enable this, approach has limited applicability since it is only supported FLUID addresses the following three technical challenges. i) by custom applications. Second, several studies [53, 56] have UI partitioning and distribution: FLUID carefully analyzes ap- introduced new programming models and/or development plication and platform UIs to find a minimal and complete set tools to extend existing applications for multi-surface opera- of UI objects and their associated graphical states required tions, reducing a significant amount of development effort. to correctly render selected UI elements, and distributes only However, their applicability can be limited to a small set of those objects and states across a host and guest devices. Our applications in practice, since they only maintain the same decision of performing local rendering on each device is bene- look-and-feel for stock Android UIs (user interfaces) but ficial in mobile wireless environments, since it not only saves not for app-specific custom UIs. Using custom UIs inan the network bandwidth but also significantly reduces the application is a common practice for mobile application de- number of network round-trips. ii) Transparent distributed UI velopment. Our analysis of the top 100 apps of Google Play execution: With distributed UIs, correct execution of an app Store, conducted in May 2018, has revealed that all of the top requires cooperation between distributed states (UI objects 100 have their own custom UI components. Third, screen and graphical states) and the rest of the states. FLUID trans- sharing (or mirroring) and app migration support a wider forms local method calls to RPCs (Remote Procedure Calls) range of unmodified applications. The former copies a screen to enable seamless cross-device execution for unmodified from one device to another usually with a larger screen or applications. iii) UI state synchronization: When a user repli- a better resolution (e.g., Vysor [10] and Chromecast [18]), cates a single UI element on multiple surfaces and interacts while the latter moves an app process to an external device with it simultaneously, FLUID synchronizes all the UI states in the middle of execution and allows the app to use the involved to ensure the overall correctness of execution. This surface of the external device [52]. However, they come with enables FLUID to provide greater flexibility with an option limited flexibility such that an app cannot fully exploit the to replicate UI elements.