Photofinder: Browsing and Searching for Personal Photos Within a Collection

PhotoFinder: Browsing and Searching for Personal Photos within a Collection

Hyunmo Kang Jin Tong Computer Science Department Computer Science Department University of Maryland at College Park University of Maryland at College Park College Park, MD 20742 USA College Park, MD 20740 USA +1 301 405 2725 +1 301 405 6746 [email protected] [email protected]

ABSTRACT features implemented in PhotoFinder and discuss the Digital photo organization software usually lacks user- advantages and disadvantages of these features. friendly search and browsing functions. We implemented PhotoFinder, a digital photo library organization tool, as a solution to some of these user interface problems. 2. BACKGROUND PhotoFinder provides a set of visual Boolean query 2.1 The Photo Database interfaces, coupled with dynamic query and query preview We used a digital photo database from HCIL as our features. It gives users powerful search capabilities. Using example data for this project. This database contains 5 scatter plot thumbnail display and drag and drop metaphor, tables and the relationships among these tables are shown PhotoFinder is easy to use for searching and browsing in Figure 1. photos. Collection Table: This table represents the collections of photos with important attributes such as KeyWord, StartingYear, Keywords EndingYear, RepresentativePhotoID. Collections are PhotoFinder, dynamic query, query preview, search, mutually exclusive to each other so that one photo can only browsing, Boolean query, digital photo library. be contained in one collection. Linkage Table: 1. INTRODUCTION This is the connection between Collection Table and Photo Digital cameras, digital scanners and personal computers Table. It stores the links between collections and photos. are commonly used in everyday life now. The need to store, Photo Table: organize and browse digital photos is huge. There are This is where photos are stored with important descriptive plenty of personal digital photos organization software attributes such as the date of the photo, Event, KeyWord, available either commercially or non-commercially. A the place where it was taken, Rating, Color, Locale, search with terms "digital photo" and "software download" Contents, PeopleCount, URL. A thumbnail image is also on ZDNet.com or CNet.com will return a lot of hit software available for each picture. entries. These software products vary a lot in terms of functionality, features and quality. We find what most of Appearance Table: these products lack are user-friendly and efficient searching This table stores the appearance information of individuals and browsing capabilities, especially for collections of (who appears in which picture). It serves as the linkage photos. This has been a long-standing problem for digital between Photo Table and Person Table. Attributes include photo library tools. AppearanceID, PersonID, PhotoID, XYCoordinates of people in the pictures. Spatial coordinates of each individual are later used by PhotoFinder to provide direct PhotoFinder is built as an attempt to address some of these annotation by pointing and clicking at people appearing in problems. We implemented some novel widgets that can be the photos. used to do powerful Boolean queries. A variety of information visualization techniques, such as dynamic Person Table: This is the table that stores all the information about the queries, query previews, are employed to provide facilities people who appeared in the photo collections. Attributes to conduct search and browsing easily and effectively. We include names and PersonID. also designed interfaces to show missing data in the database searched. In this paper, we will describe the Figure 1. The Structure of Photo Library Database

The sample database we are using contains a few hundred to simplify the operations for searching and browsing: e.g pictures classified into 28 collections. It is projected that pointing and selecting will be favored against typing. The the database will grow to have tens of thousands of pictures interface should produce immediate, continuous and visual in the near future. feedback of search/query results.

2.2 Relationship between Collection Browser and Visual Information Seeking Strategy PhotoFinder We designed the PhotoFinder to have a 3-level-hierarchy A peer project - Digital Photo Collection Browser - is also window strategy: a Searching window (top level), a being built by another group. The Collection Browser is Thumbnail Browser Window (intermediate level), a Detail closely related to the PhotoFinder we are building. The Viewer Window (bottom level). This is an echo for the Collection Browser will be used to browse collections from tried and tested motto: "Overview first, then zoom and a large set of pictures, whereas PhotoFinder emphasizes filter, and finally details on demand". (North & more on searching and browsing within a collection. These Shneiderman, 1996) 2 projects may be integrated into one organization tool to offer a more complete set of functions. Query Previews Combined with dynamic query techniques, we also wanted 3. DESIGN OUTLINES to use Query Preview to enhance the user’s searching Influenced by some earlier projects at HCIL, our design for experience. PhotoFinder is trying to address the following questions: - How to provide powerful Boolean queries? Multiple Tightly Coupled Views - How to make use of dynamic queries? The 3-level-hierarchy window strategy put tightly coupled - How to make use of the advantages of query preview? views together to form a coherent view to the data the user - How to properly handle missing data for the attributes? is searching and browsing. Centered on these issues, we proposed the following user interface strategies. 3.2 Exploring PhotoFinder Figure 2 shows the overall user interface of PhotoFinder with a dummy collection browser. We assume such a 3.1 User Interface Strategies collection browsing will be used to browse and find a Direct Manipulation and Dynamic Query particular collection that PhotoFinder will be working on. We wanted to use a visual representation for the search There are 3 major parts of PhotoFinder: space, which allows rapid, incremental, and reversible exploratory actions. We wanted to construct visual widgets Figure 2. PhotoFinder with Collection Browser

Searching Tool Window can also just point and click to locate and select an This sub-window contains a series of tabbed panels of individual picture or a set of pictures in the thumbnail dialog boxes for setting query attributes. On each tabbed browser window. panel, an Attribute Distribution Chart is displayed to show the hits and misses of items for those particular attributes. Innovative multiple 2-sided slider bars are used for query Detail Viewer Window range selections. The overview window gives the user 3 This panel is used to display the meta data and annotation filtering mechanisms to display photo thumbnails: show all for the picture selected from Thumbnail Browser and hits for current query (default behavior), show hits for each picture itself in full size. Slide show function is also independent attribute query (if “Show Thumbnail” button is implemented to let the user to display a set of selected clicked), show hits for corresponding histogram bar (if a pictures one after another. Detail Viewer Window also histogram bar is clicked and highlighted). provides facilities for annotating a single picture.

Thumbnails Browser Window 3.3 Implementation Environment There are 3 major display modes for the Thumbnails PhotoFinder is implemented using Microsoft Visual Basic Browser Window: sequential display, scatter plot display, 6.0. The sample digital photo library database was and histogram display. Zooming is provided to better use implemented and maintained as a Microsoft Access the display space and for the ease of navigation. The user database. Through ODBC, PhotoFinder is able to connect queries involving disjunctive relations among attributes if to and query the database. they are willing to. This is achieved by sequentially performing different queries and merging the query results.

4. SYSTEM FEATURES 4.1 Boolean Query and Search Interface How to enable the user to do Boolean queries visually is a big topic in HCI fields. Many approaches have been suggested with different degrees of implementation difficulties and different degrees of success (Chui, 1999; Willie, 1994). To provide full Boolean query power in a visual form is often not feasible. In our particular case, we find that users of digital photo libraries tend to perform Boolean queries for searching pictures with some limited patterns (Garber & Grunes, 1992). Namely, users tend to use Boolean queries as follows:

 make OR queries within a attribute;  make AND queries among different attributes; Figure 3. Multiple 2-Sided Slider Bars

For example, you may want to find pictures "of Ben OR other HCIL people" that (AND) "were taken in 1999", but not pictures "of Ben OR those taken in Las Vegas". Our approach for disjunctive queries within attributes is to use Multiple-Two-Sided (MTS) slider bars for each attribute. The Multiple-Two-Sided (MTS) slider bar is designed specifically to enable disjunctive range selections. Traditional double-sided slider bars give the users the ability to select one range. Using MTS, users can specify any number of disjunctive ranges of one attribute. We find MTS slider bars a tremendous advantage over the traditional slider bars or a list of combo boxes for range selections. The use of MTS is also very simple, just right click the slider will generate a new range slider bar. This slider bar can be dragged around and resized to set new range values. If 2 slider bars overlap, they will be automatically merged as one slider bar. Figure 3 shows a set of MTS slider bars in action where 5 MTS sliders are generated and the overall disjunctive ranges are shown. The MTS slider is implemented as a Microsoft ActiveX control and can be used just as a regular widget in Visual Basic. Figure 4 shows how MTS sliders are used for each of the attribute setting panel. 11 tabbed panels are used to represent query formations for different attributes (10 for each attribute, 1 for “All” attributes). The relationship among these attributes in forming a query is by default conjunctive. The "All Figure 4. Examples of Dynamic Query Preview. Missing Attributes" tabbed panel shows in a compact display all the data are represented as a separate attribute value. In this query parameter settings. Histograms are used to show example, no pictures have missing dates. If there are any query results for each individual attribute. The yellow part pictures with missing dates, they will show up as a separate of each histogram bar indicates the “hits” of query results. histogram bar. The grayed part of the histogram bar indicates the “misses”. Although the relationship among different query attributes defaults to be conjunctive, users can still form complicated Grouping techniques can be used to merge the query result collections into one bigger set. We did not implement an explicit user interface to perform such merging actions. It is large portion of the display space is wasted white space. implicitly available through the thumbnail selection and This is the non-compact mode, which is the default mode. grouping facilities in the Thumbnail Browser window. If "Compact" mode is selected, our heuristics will try its best to re-order the thumbnails so that pictures of the same size and orientations will be placed together to reduce the 4.2 Dynamic Queries wasted white space. Figures 5a-b show the effect of non- Dynamic Query techniques have proved to be very intuitive compact and compact mode of sequential display. The and effective. (North & Shneiderman, 1996) Immediate compact mode is especially useful when the order of the feedback often provides a lot more information than photos are not important to the user so that the display statically displayed query results. In PhotoFinder, Dynamic space can be better utilized. Queries are also used to enhance the searching capabilities. Query results are updated in real time for preview in the search panel. Thumbnail Browser window is not updated in real time for each query setting change. This is largely due to performance issues of continuously executing SQL queries. We may implement dynamic updating in Thumbnail Browser Window later.

Dynamic filtering is made possible by clicking on individual histogram bars to further filter the query results. When this happens, only pictures meeting that particular Figure 5a. Non-Compact mode of Sequential Display histogram bar will be shown. All attribute panels, query result histograms are updated in accordance to reflect the effect of query change.

4.3 Query Previews There are many advantages of Query Previews (Greene, et. al, 1997). In PhotoFinder, we provided a histogram chart for each attribute as a query preview tool. The user can see the distribution of pictures for each attribute in advance and therefore gain a better understanding of the whole photo Figure 5b. Compact Mode of Sequential Display collection. The histogram chart also gives the user immediate feedback of the number of hits for the query they are performing. Query parameters can then be adjusted Scatter Plot Display to form more desired queries. It presumably will prevent Users can choose to have a scatter plot display for the users from performing no-hit queries. thumbnail pictures. Like SpotFire, such a scatter plot display, coupled with the flexibility of choosing different coordinate attributes, often provides the user with quick Missing data are displayed as an additional value in the and insightful understanding of the pictures displayed. By data preview panel. No loss of information in visualizing choosing different coordinate attributes, the distribution of the database is resulted. Figure 4 shows the example of pictures might tell some interesting stories that can never dynamic query preview of searching window. It also shows be found from purely a tabular display of the attribute the way missing data are displayed. values. This way, users browsing capabilities have been expanded. Figure 6 shows an example of the scatter plot display. 4.4 Multiple Displays of Thumbnail Browser The thumbnail browser is designed to hold and display the query result for the user to browse and select for detailed Histogram Display viewing. By the nature of the complexity of the photo We also plan to implement a Histogram Display for attributes, no single way of displaying the thumbnails will thumbnail images. The idea is that thumbnails will be fit all users’ needs. Therefore we designed to have multiple displayed as histograms for different attributes so that the displays of thumbnails in this browser. user can browse and select pictures based on the distribution more easily. Such histograms correspond to the histogram charts in query result preview panel. The Sequential display: Compact and Non-Compact mode difference is that now each histogram bar is a stack of In sequential display of photos, one picture is placed after picture thumbnails that can be selected individually or as a another. Due to different sizes and orientations of photos, a group. Figure 6. Scatter Plot Display of Thumbnail Browser with Zooming Function. The x-coordinate is “number of people in a picture” and the y-coordinate is the rating of the picture. Outliers are easily spotted at the far right corner: Figure 7. Drag and Drop in Detail Viewer Window: one while most pictures in this collection have only a few picture is shown in the Detail Viewer Window, while a people in it, the north east corner one has 7 people. The group selection is performed in the thumbnail browser and distribution of pictures in this scatter plot also shows that the selection is being dragged to the Detail View Window pictures of multiple people tend to have higher ratings.

6. CONCLUSION 4.5 Drag and Drop in Detailed Viewer Window We implemented PhotoFinder to provide better searching From Thumbnail Browser Window, users can point and and browsing capabilities. By using novel attribute click to select a single picture or a group of pictures. After selection query interface, PhotoFinder gives the user near the selection is made, the user can drag and drop the full Boolean query power with ease of use. Information selection into the Detailed Viewer Window for viewing or visualization techniques, such as dynamic query, dynamic slide show. Such an interface is intuitive and easy to use. It query preview, can further strengthen the search also makes browsing and viewing photos a lot more fun. capabilities of PhotoFinder. Multiple forms of thumbnail Figure 7 shows how drag and drop works for detailed display and drag and drop interface make PhotoFinder a viewer window. very fun and powerful photo-browsing tool. We consider PhotoFinder a better solution to the average users’ digital photo library management task than most existing software. 5. FUTURE WORK We feel our design and implementation of PhotoFinder has a lot to offer than most of the digital photo organization ACKNOWLEDGMENT software products currently available out there. Still, to We thank for Dr. Ben Shneiderman for his invaluable make a really good product for home users to organize, suggestions on improving the features and quality of this browse and search their digital photo library is hard. We project. Classmates from CMSC838S have also provided realize PhotoFinder has many limitations. There is still a lot their clear and useful critique and comments on our project of work need to be done in terms of features and interfaces. which resulted in many changes of the application and this Currently, we have not integrated PhotoFinder with a paper itself. proper Collection Browser. To combine PhotoFinder with a Collection Browser will give users a more complete solution to their digital photo library organization problem. REFERENCES We also have plans to implement automatic annotation 1. Allan Kuchinsky, Celine Pering, Michael L. mechanism so that the users can just use a mouse to click, Creech, Dennis Freeze, Bill Serra, Jacek Gwizdka, drag and drop to make annotations instead of typing in "FotoFile: A Consumer Multimedia Organization and endless fields. These techniques can be further combined Retrieval System", Proceedings of CHI99, pp. 496-503, with some proper functionality for annotations of single, May, 1999. group or collection of pictures. Also planned for future 2. C. North, B. Shneiderman, C. Plaisant, "User work is a user study of PhotoFinder or a formal usability Controlled Overviews of an Image Library: A Case Study test. of the Visible Human", Proceedings of ACM Digital Libraries '96 Conference, ACM Press, 1996. 3. Joemon M.Jose, Jonathan Furner, and David 7. Sharon R. Garber, Mitch B. Grunes, "The Art of J.Harper, "Spatial querying for image retrieval: a user- Search: A Study of Art Directors", Proceedings of CHI92 oriented evaluation", Proceedings of the 21st annual Conference on Human Factors in Computing Systems, international ACM SIGIR pp. 157-163, 1992. 4. Ken Fishkin, Maureen C. Stone, "Enhanced 8. Stephan Greene, Gary Marchionini, Catherine Dynamic queries via Movable Filters", Proceedings of Plaisant, and Ben Shneiderman, "Previews and CHI95, pp. 415-420, May, 1995. Overviews in Digital Libraries: Designing Surrogates to 5. Lisa Tweedie, Robert Spence, Huw Dawkes and Support Visual Information Seeking", University of Hua Su, "Externalising Abstract Mathematical Models", Maryland Technical Report CS-TR-3838, 1997. Readings in Information Visualization, pp. 253-259, 9. Sylvia Willie, "Query Context: Would a graphical 1999. interface help?", School of Information Systems, 6. Michael Chui, "Pattern, Procedurality & Pictures: Queensland University of Technology Factors Affecting Boolean Query Interface Design for the http://www.icis.qut.edu.au/~willie/My_Papers/ozchi- Web", Proceedings of SIGIR99 94.html http://www.cs.indiana.edu/~mchui/sigir99/