POD: a Smartphone That Flies
Total Page:16
File Type:pdf, Size:1020Kb
POD: A Smartphone That Flies Guojun Chen Noah Weiner Lin Zhong Yale University Yale University Yale University [email protected] [email protected] [email protected] ABSTRACT We present POD, a smartphone that flies, as a new way to achieve hands-free, eyes-up mobile computing. Unlike exist- ing drone-carried user interfaces, POD features a smartphone- sized display and the computing and sensing power of a modern smartphone. We share our experience in prototyping POD, discuss the technical challenges facing it, and describe early results toward addressing them. Figure 1: POD (top left corner) interacting with our test mannequin mounted on a Wi-Fi-controlled robotic chassis. 1. INTRODUCTION Our driving vision is hands-free, eyes-up mobile comput- human-drone interaction and requires new investigation. ing. That is, mobile users are able to interact with a computer To address these challenges and experiment with POD, without holding a device in hand and looking down at it. The we have implemented a prototype. As shown in Figure 1, key to this vision is a user interface (both input and output) it is based on a mechanical, electrical and computational that does not require a human hand. Wearable devices, such partnership between a micro-controller-based drone and a as Google Glass, are perhaps the most studied implementa- stripped-down smartphone (Google Pixel 4). It has a flight tion of this type of mobile, hands-free user interface. time of four minutes, uses computer vision to track and follow This paper explores an alternative idea for these new user its user, and harnesses the phone’s sensors to improve the interfaces: one carried by a small drone. Compared to wear- usability of its display. able user interfaces, drone-carried UIs do not impose gadgets This paper describes the POD prototype and shares our on the user’s head or eyes. More importantly, they can go experience in building it (§4). It presents our early results (§5) beyond the physical reach of the human body, empowering in addressing the challenges facing POD (§3) and discusses interesting use cases. interesting, novel use cases for it (§2). While there is a growing literature about human-drone in- teraction [16, 31, 13], this work envisions a drone-carried UI as rich as that of modern smartphones, called POD. Unlike 2. THE CASE FOR POD previously studied drone-carried UIs, POD features a small We first describe several compelling applications of POD, (potentially foldable) display and allows bidirectional interac- a smartphone that flies. Because POD carries cameras and tions with mobile users. We imagine that POD would rest on microphones, it can fill roles intended for existing cinemato- the user’s shoulder most of the time and fly out occasionally graphic drones. We ignore these use cases and instead focus arXiv:2105.12610v1 [cs.HC] 26 May 2021 to interact with the user only on demand. on those uniquely enabled by POD’s display. We envision Compared to wearable glasses and other drone-carried UIs, that POD will share much of the hardware and software of POD faces its own unique challenges that have not been smartphones. Our prototype uses a Google Pixel 4, allow- addressed by the human-drone interaction community. First, ing it to tap into the mature ecosystem of Android, which like all flying objects, drones are not steady. This challenges facilitates app development and enables “retrofitting” exist- the usability of an onboard display, especially if the drone ing smartphone apps for POD. This flexibility contrasts with needs to maintain a safe distance of one to two feet from the other drone platforms that develop their own ecosystems, e.g., user’s eyes. Second, drones are noisy. Even the quietest drone DJI and Skydio. operating at one or two feet away from the user produces Hands-free “Smartphone”: POD’s hands-free experience noise at 70 dBA, which is equivalent to being on a busy street opens up a host of new multitasking opportunities for mobile or near a vacuum cleaner [8]. Finally, because we intend users in situations when holding a device in hand is either POD to serve the user on the go, it must decide between inconvenient or impossible. Users could make video calls or following the user or hovering in place when the user moves consume media while cooking, walking around the house, or around. This challenge represents a very specialized case of jogging. 800 Peaks y displacement of static content 700 pixels y displacement of stabilized content 600 500 time/s Environment noise level: -20dB 400 0 2 4 6 8 10 12 14 Figure 2: Y-axis (vertical) movement measured by a camera located Figure 3: Noise spectrum of our POD prototype. It has multiple 20 cm from our POD prototype while the latter was hovering. strong tonal components (peaks) and a relatively wide band. Personable Interaction: There is a growing literature POD moves along the vertical axis during one of its flights, about human-drone interaction with a focus on how users despite its best effort to hover, as captured by a camera. Such command a drone [13], e.g., using gestures [12, 23, 25] and motion poses significant usability challenges for the display, touch controls [9]. Few works, however, examine how a especially for reading small text. Using a drone-carried iPad, drone could communicate with users directly, without using the authors of [29] found, not surprisingly, that a moving an intermediary device like a handheld terminal. For example, display carried by the drone requires a larger text font for Szafir, Mutlu and Fong [30] studied how a drone could use readability, although the study did not isolate the impact of the an LED ring to communicate its flight intention. drone’s inability to stay stationary in the air, as is important POD, with its display, provides a much richer platform for to POD. this type of direct communication. For example, POD can use smiley faces to comfort the user or indicate its intention, which is important because many users are uncomfortable 3.2 Noise Suppression with a drone flying around at such close proximity. POD can Drones are noisy, and drone noise suppression has been also use arrows or other symbols to direct the user’s attention. studied extensively [21]. The noise worsens disproportion- First Response and Public Safety: We also envision POD ately when the user is close to the drone. And POD intends to being used in the context of first response and public safety. serve its user at the intimate smartphone distance, much more Its display can present instructions to first responders as they closely than cinematographic drones, which acerbates the navigate difficult scenarios, like fires and disasters. It could noise problem. Our POD prototype, though quite small, can present interactive maps or other informative graphics to produce noise up to 75 dBA, if measured from two feet away. those trapped in hard-to-reach areas. At the scenes of car Commercial cinematographic drones are about as noisy, or accidents, police could send POD to facilitate a remote video noisier, due to their larger size [3]. Noise this loud is known call with the driver and inspect their license, or to review to be harmful of humans during long periods of exposure [8]. damages and interview witnesses on the scene. Cinematographic drones usually operate far from users, and noise captured by their microphones can largely be removed 3. OPEN CHALLENGES through post-processing. In contrast, POD operates at the smartphone distance from the user and the noise must be While drones for cinematography have been extensively suppressed in real-time. studied and widely used, drones as a personal device have Many have explored using passive methods to reduce drone not. Our experience with the POD prototype has revealed noise: covering the propellers with sound-absorbing materials technical challenges unique to personal drones, especially and metal surfaces [22]; using ducted (or shielded) propellers those related to their interacting with the user at such a close [19]; using specially shaped propellers, e.g., DJI low-noise proximity. propellers [2]; and using propellers with more blades. These methods have only limited success. For example, ducted 3.1 Display Usability propellers [19] only reduce the noise by up to 7 dBA. Drones are dynamic systems. Keeping them stationary The synchrophaser system [17] is an active drone noise in the air is a long-studied and difficult problem. In order cancellation method. It requires that all propellers have ex- to capture high-quality videos and photos, commercial cin- actly the same rotational speed and a fixed phase difference. ematographic drones have gone a long way toward solving It thus only works for fixed-wing aircraft with symmetric this problem. Unlike POD, they benefit from two factors: sets of engines. However, small rotorcraft like drones do not they usually hover far away from the target scene; and the meet the synchrophaser requirements—their rotors constantly impact of instability on the footage they take can be removed change speed to keep the craft stable. Noise cancellation tech- by post-processing. niques widely used in headphones can reduce noise at the ear. POD, on the other hand, must present a stationary, readable These techniques work well with low-frequency (< 2 KHz) display to its user, and at a much closer distance, with the and tonal noise. Unfortunately, as shown in Figure 3, drone screen content often being generated in real time. Figure 2 noise consists of tonal noise of both high and low frequencies, (red, dotted line) presents how the screen of our prototype as well as random wide-band noise higher than 6 KHz.