Interacting with Hidden Content Using Content-Aware Free-Space Transparency Edward W. Ishak Steven K. Feiner Columbia University, Department of Computer Science New York, NY 10027 Email: {ishak, feiner}@cs.columbia.edu ABSTRACT We present content-aware free-space transparency, an ap- proach to viewing and manipulating the otherwise hidden content of obscured windows through unimportant regions of overlapping windows. Traditional approaches to interact- ing with otherwise obscured content in a window system render an entire window uniformly transparent. In contrast, content-aware free-space transparency uses opaque-to- transparent gradients and image-processing filters to mini- mize the interference from overlapping material, based on properties of that material. By increasing the amount of si- Figure 1: Window rendered (left) without free- multaneously visible content and allowing basic interaction space transparency, and (right) with content-aware with otherwise obscured content, without modifying win- free-space transparency, exposing blurred hidden dow geometry, we believe that free-space transparency has content underneath. the potential to improve user productivity. We also present a set of interaction techniques that utilize Categories and Subject Descriptors: H.5.2 [Information the benefits of FST. The pop-though technique allows a user Interfaces and Presentation]: User Interfaces—Graphical to interact with content beneath unimportant regions of an user interfaces (GUI), Interaction styles, Screen design, obscuring window without moving or resizing it. The focus Windowing systems filter allows a user to temporarily transform a filtered por- tion of obscured content to its original unfiltered form, General Terms: Human Factors, Design which clarifies the exposed content beneath an obscuring window. Finally, since any pixel may render information Additional Keywords and Phrases: Transparency, screen from multiple windows, we allow users to determine the space, interaction techniques, content disambiguation, space window with which they will interact by using the mouse- management, pie menu over pie menu. 1. INTRODUCTION 2. RELATED WORK To view or interact with the content of an obscured window To increase the amount of simultaneously visible content, in conventional window managers, users are often forced to some systems have tried rendering the entire obscuring win- resize or move the obscuring window, or bring the obscured dow semi-transparently [1,2,7,13,14]. This traditional use of window to the top. To avoid this problem, users sometimes semi-transparency (accomplished by uniformly alpha blend- resort to using window managers that automatically cascade ing the window with the contents of the frame buffer behind or tile each window without regard to its content [5]. How- it), allows a user to visualize content that is behind an ob- ever, this is undesirable when certain windows need to dis- structing window, but often makes it difficult to determine play more content than others or require particular aspect visually which content belongs to which window. Multi- ratios. Free-space transparency [11] allows a user to make blending [4] addresses this issue when interacting with an efficient use of screen space by rendering unimportant win- application’s palette windows in visual workspaces. Al- dow regions transparent and important window regions though multiblending preserves the visibility of both back- opaque, with a smooth gradient between them. In this paper, ground and foreground windows containing familiar con- we extend this work to introduce content-aware free-space tents, users may have difficulty understanding unfamiliar transparency (FST), which takes into consideration various overlapping contents. Users may also have difficulty with characteristics of the obscured and overlaid content and ap- similar appearing overlapping contents, especially text. FST plies image-processing filters and gradients to further reduce takes a different approach by selectively rendering every content ambiguity [12]. This guarantees that the important important region opaque. By preventing obscured content content of overlaid windows will be readable at all times, from showing through those important regions, FST in- while simultaneously exposing hidden content beneath the creases the legibility of even unfamiliar overlapping infor- unimportant regions. mation. Permission to make digital or hard copies of all or part of this work for In Macintosh OS X [3], a text-only terminal window can personal or classroom use is granted without fee provided that copies are have an opaque text color rendered on a semi-transparent not made or distributed for profit or commercial advantage and that cop- background color. The Macintosh Command-Tab menu uses ies bear this notice and the full citation on the first page. To copy other- the same approach, rendering opaque icons on a transparent wise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. UIST'04, October 24–27, 2004, Santa Fe, New Mexico, USA. Copyright 2004 ACM 1-58113-957-8/04/0010...$5.00. dered opaque and which are to be rendered transparent. When rendering the gradient between these regions, it is de- sirable that the opaque-to-transparent transition be made smooth and visually appealing, since an abrupt boundary could imply a separation of the opaque and transparent sec- tions, leading the user to believe that one window is actually divided into multiple objects. We render no pixel 100% transparent, since it would completely expose the content underneath, potentially misleading the user into believing that this content was associated with the overlaid window. Figure 2: Window rendered (left) with opaque text We have found that a 75% transparency value works well. on transparent background (alpha = 0.5), and (right) We have also found that rendering important regions with a using FST, where background pixels of important 10% transparency (90% opacity) value allows for some regions are rendered opaque, producing more legi- visualization beneath overlaid content, with little risk of ble overlaid content. content ambiguity; this supports earlier evaluations of us- background. Although this approach may produce more able and efficient transparent user interfaces, such as in the legible overlaid content than does uniform semi- Stroop Experiment [9,10]. transparency, content ambiguity can still arise when ob- scured pixels are blended with background pixels that lie in Since users often interact with window decoration (title bar, between the pixels of overlaid content, especially when the menus, border, scroll bars), we consider this important con- obscured pixels are from content of the same type as the tent. Therefore, FST does not affect the pixels that make up overlaid region (e.g., text, in the case of the Macintosh ter- these regions, and consequently, considers only the window minal window). FST generalizes the Macintosh approach by body when classifying important and unimportant regions. rendering the content pixels, as well as those in the back- In the case of opaque windows, keeping the window decora- ground on which the content lies, opaque, making the con- tion opaque allows users to disambiguate window bounda- tent more legible, as shown in Figure 2. ries more easily, as in the Macintosh terminal window men- tioned above. Inherently transparent windows maintain their 3. FST OVERVIEW transparent window decoration, while the gradient between FST allows a window’s application to inform the rendering important and unimportant regions transitions from the win- engine of the unimportant regions, as we have done in the dow's inherent alpha value to a more transparent value. implementation described in Section 4. It evaluates and compares content characteristics, such as type (text, images, 4.1. Important vs. Unimportant Regions icons, or a combination), colors, and spatial frequency of We have considered several approaches to determining im- both the obscuring and hidden window content. Based on portant and unimportant regions in an FST window. In one the combinations of these characteristics, image-processing approach, the rendering engine can classify window regions filters, such as Gaussian blur [4] or desaturation, alone or in containing only a particular background color or texture as combination, are applied to the content exposed through the unimportant. Alternatively, the user could explicitly identify unimportant regions of the obscuring window. Additionally, unimportant regions manually with a mouse or touchpad, or gradients are applied between the opaque and transparent automatically by using an eye tracker to detect window re- regions. gions on which their gaze does not dwell. Regions contain- ing white space (i.e., ones devoid of text, icons, and images) Although many approaches allow visualization of overlaid can also be automatically classified as unimportant, as in our content, they rarely allow interaction with the obscured con- implementation. However, since some applications utilize tent (with the Task Gallery [15] as a notable exception). One white space (e.g., displaying page margins in a document), reason may be that with uniform semi-transparency, disam- we allow the window’s application to notify the rendering biguating obscuring content from hidden content can be engine of these unimportant regions. This may require the hard, making user interaction with hidden content difficult. application