ATHABASCA UNIVERSITY

SMALL SCREEN; BIG PROBLEMS:

THE CONTRIBUTIONS AND PROMISE (PAST, PRESENT, & FUTURE)

OF HCI IN PROVIDING USABLE MOBILE INTERFACES

BY

STEVEN DEWOLFE

An essay submitted in partial fulfillment

Of the requirements for the degree of

MASTER OF SCIENCE in INFORMATION SYSTEMS

Athabasca, Alberta

March, 2011

© Steven DeWolfe, 2011 DEDICATION

This essay is dedicated to my wife Jennifer whose support, encouragement, patience, and understanding has always been exemplary. Thank you for always believing in me.

I love you.

ii ABSTRACT

Going mobile is the logical step in the information era. In fact, today‟s typical smart phones are more like computers. However, though researchers have long known about the special constraints of small screens, initial interface research was largely focused on information visualization techniques such as Furnas‟ fisheye. The real problems with small screens began to surface when networks became increasingly data-capable, which led early researchers down an initial path to offer the web in a condensed format called the mobile internet.

But unhappy consumers began to demand a better mobile experience which led

Apple in 2007 to adopt and implement cutting edge multi-touch research via the iPhone.

The iPhone unhinged the mobile market and created a second path for manufacturers with its advanced zooming, multi-touch and gesture-based interface – which even today every other competitor is trying to beat.

After a comprehensive study of past, present, and future interface techniques, this essay finds that researchers are working within the overall context of mobile interaction to offer new interaction, interface, visualization, and device improvements.

However, in contrast, this study also finds that today‟s phones (though very powerful tools) are not yet capable of becoming the new computational platform of the world – a monumental milestone the industry is progressing towards. While manufacturers are experimenting with simplistic device improvements, researchers are currently following a third path of future innovation that could offer the world the potential for a new form of personal computing.

iii Replacing the personal computer with a mobile personal computer based on an already massive and saturated worldwide cell phone market has too many benefits to ignore. Continuing to address the small screen visualization issue will change the way we all inter-relate, work, and live; particularly for the estimated 35% of the world‟s workforce that will be mobile by 2013.

iv ACKNOWLEDGMENTS

Thank you to my family and friends for your support and understanding during my time spent pursing my interests. Thank you to my parents. Dad, thank you for your consistent encouragement to do well, your support, and your love. Mom, thank you for every day and every lesson that made me who I am, for your support, and your love.

Thank you to all of the dedicated professors and staff at Athabasca University for providing and facilitating the great learning opportunities in the MScIS program. The underlying message is that the field of information technology is both promising and filled with great opportunities for those willing to accept the challenge.

Speaking of challenging, I would like to thank Dr. Maiga Chang for an illuminating and collaborative course in Mobility that will continue to fascinate me for years to come.

Special thanks to Steve Leung who helped to forge a curiosity about the works and field of Human Computer Interaction. Thank you very much Dr. Xiaokun Zhang for your supervision of this essay where I was able to successfully merge these two interests.

And, thank you to the late Dr. Peter Holt for proposing this fascinating research topic that challenges the computing industry to meet the rapid changes in today‟s information- based world. Knowledge truly is an insatiable commodity that provides more questions than answers.

v TABLE OF CONTENTS

CHAPTER I ...... 1

INTRODUCTION ...... 1

THE PROBLEM WITH SMALL SCREENS ON EVOLVING MOBILE PHONES ..... 1

RESEARCH PROBLEM ...... 6

LIMITATIONS OF THIS STUDY ...... 6

DEFINITION OF TERMS ...... 8

ORGANIZATION OF THE REMAINING CHAPTERS ...... 10

CHAPTER II ...... 11

REVIEW OF HCI RESEARCH IN TECHNIQUES ...... 11

RESEARCH METHOD ...... 11

RELATED WORK ...... 13

CONNECTION TO CURRENT RESEARCH ...... 42

PURPOSE OF THIS STUDY ...... 43

CHAPTER III ...... 44

SURVEY OF THE MOBILE INDUSTRY, HARDWARE, AND UI ...... 44

RESEARCH METHOD ...... 44

SURVEY OF MOBILE INDUSTRY ...... 47

SURVEY OF PRESENT MOBILE HARDWARE INTERFACES ...... 57

SURVEY OF PRESENT MOBILE SOFTWARE INTERFACES ...... 68

vi CHAPTER IV ...... 74

MOBILE INTERFACE ISSUES, CHALLENGES, AND TRENDS ...... 74

TRENDS ...... 74

ISSUES AND CHALLENGES ...... 79

CHAPTER V ...... 83

CONCLUSIONS AND RECOMMENDATIONS ...... 83

SUMMARY ...... 83

RECOMMENDATIONS FOR FURTHER RESEARCH ...... 85

REFERENCES ...... 86

APPENDIX A ...... 104

CONTEXT OF MOBILE INTERACTION RESEARCH MAPPING ...... 104

vii CHAPTER I

INTRODUCTION

THE PROBLEM WITH SMALL SCREENS ON EVOLVING MOBILE PHONES

The cell phone is more like a computer now. And, because of this people expect a "desktop" in their pocket. However, the small screens on mobile devices are one of the biggest barriers to giving people what they want.

The problem of display size is not new – it has been researched extensively since before the advent of the personal computer in the early 1980s. The “new smallness” (Baudisch & Holz, 2009) - or need to address the visualization problem on small screens - does present new and distinct challenges though. In particular, that much of the research that applied to PC user interface techniques is not directly transferrable to fixing the problems with small screens because of their unique constraints. This paper will show that it is the field of Human Computer Interaction

(HCI) - a sub-field within the study of information systems - that has been, is currently, and will be addressing this problem ongoing.

But, can this problem be fixed? Researchers have long recognized that devices must remain small to be truly mobile – and this core belief has led them down many paths. In the research presented, it will be argued that some ideological choices seem destined to fail, and that the changing face of the mobile market combined with

1 advancements in devices and infrastructure have led to a „re-alignment‟ of research directions. These new directions are both exciting and promising as the “new smallness” will forever alter the way we work, inter-relate, and live.

The Large Problems with Small Screens

Displaying information on devices is not a new challenge to the field of human computer interaction. Displaying information on small screen mobile devices is however, which means researchers must revisit an old and complex issue with new challenges. This makes sense when we realize that mobile phones are relatively new and device screens are rapidly evolving (see Chapter 3). Researchers generally agree that mobile devices must remain small to be usable and have formulated a long list of constraints presented by small screens in widely differing mobile devices.

Paelke, Reimann, & Rosenbach (2003) highlighted mostly hardware related constraints. They viewed physical limitations on the screen such as resolution, size, available colors, and processing power, to be very limiting factors that must be overcome. A second set of limiting factors related to input and output involved the lack of standardization between devices, where PC based peripherals such as the full keyboard and mouse are completely absent, compounded by each device having slightly different methods to overcome this limitation. This view reinforces the common belief that past PC based visualization solutions cannot simply be transferred to mobile devices. Chittaro (2006) also discussed most of these limitations but also added that most devices do not use the standard 4:3 width and height aspect ratios. This only emphasizes the constraints created by non-standard displays and device peripherals.

2 Paelke, Reimann, & Rosenbach (2003) also discussed more human related factors that constrain interaction with small screens such as user context. They viewed context related factors such as the physical environment to be important, as a user‟s level of attention to a screen or interface is more limited in a mobile setting. For instance, users may only be able to divide part of their attention, or things like light- levels may change, or a user may only have a single hand to devote to the device.

Baudisch and Holz (2010) added to this statement by drawing attention to limiting human factors like limited eye sight and finger sizes that occlude useful screen selections.

What can HCI do to fix the small screen? How?

The visualization problem likely extends beyond the small screen. By examining the related research and surveying the existing cellular market, findings are anticipated to be both positive and negative. On a positive note, it will be shown that a number of existing interface techniques are “tried and true” and work. As well, there is likely promise in newer techniques to solve existing screen and design constraints.

Conversely, some constraints might not have an ideal solution or a solution at all. This essay hopes to answer questions like:

1. Is there a need to push for new standards for hardware and software for new

techniques to work?

2. Is there a clear direction for addressing the visualization problem yet?

3 Why is fixing the small screen so important?

Fixing, or at least addressing, the small screen problem will cause a surge of advancements in how we work and live. This is important as the mobile work force is growing and the need for a desktop-like experience from a pocket-sized device will further enable this trend. But users have shown that they will not adopt a mobile environment that promises and does not deliver.

World-wide use of small screen devices is growing as shown by the rampant growth of the mobile workforce. Drake, Boggs, & Jaffe (2010) offered some astounding conclusions from the IDC study of mobile growth which projected growth statistics from 2008 to 2013:

 The highest percentage is in the US which will grow its mobile workforce from

72.2% to 75.5%,

 The largest total is in Asia/Pacific with 546.4 million workers and is forecast to

grow by almost 200 million,

 Western Europe currently has about a 50% mobile workforce that will grow each

year by 6% in a compounding manner,

 Almost 75% of Japan‟s workforce will be mobile in 2013, and

 The rest of the world (including Canada) has the lowest mobile rates but the

largest opportunity for growth.

4 In essence, forecasts are that over 1/3rd of the world will need the capacity to work in a mobile manner within the next 5 years, and this realization is best reflected by the IDC statement that:

“As mobility continues to play a key role in enabling companies to achieve greater productivity worldwide, IDC expects the global mobile worker population to increase from 919.4 million in 2008 to more than 1.19 billion in 2013, representing nearly 35% of the worldwide workforce.” (Drake, Boggs, & Jaffe, 2010).

Even if the mobile workforce is ready to expand, users are quite vocal that they will not accept new mobile services like the until problems like the small screen are adequately addressed. Karp (2007) maintains a webpage (five reasons why the mobile web sucks) that openly criticizes the mobile web for not having sites properly formatted for small screens and he also vocalizes the fact that device screens are just too small (even though he admits it is a tougher problem as devices must remain small).

These assessments are confirmed by Gomez (2009) which studied 1001 mobile users with an in-depth survey concerning mobile web usage and acceptance. The findings support Karp‟s assessment as follows:

 60% of users overall had problems accessing the mobile web while 48% found

mobile sites hard to use primarily because of format,

 52% were very or somewhat unlikely to return to a website they had problems

with, and

 64% would not recommend sites they had problems with.

So, why is fixing the small screen issue so important? It is important because,

“more than 80% of mobile users would access websites more often from their phone if the experience was as fast and reliable as it is at home” (Gomez, 2009) but it is 5 currently “like surfing the web through a keyhole…” (Karp, 2007), so, “…the mobile web will continue for some time to be about getting done what can‟t wait until later” (Karp,

2007).

RESEARCH PROBLEM

Therefore, this paper intends to research and identify examples of how small screens on mobile tools put special demands on HCI for the design of mobile interfaces.

More specifically, to look at both current and proposed solutions to managing small screen constraints with a discussion on how new HCI-researched techniques can help solve these issues in the future with an emphasis on the possible benefits to further research of this issue.

LIMITATIONS OF THIS STUDY

This study assumes that the reader is familiar with the field of human computer interaction and its purpose. As well, this study organizes its information around important chronological events but does not provide a detailed chronological history of when interface designs were proposed or implemented.

The past, present, and future user interface techniques that will be reviewed in

Chapter II are only considered to be excellent examples of the direction of mobile techniques to address the limitations of small screens. It would simply be beyond the scope of this essay to try and cover every technique that has, is, or will be created. For instance, Dr. Patrick Baudisch, a prominent researcher in the field of human computer interaction and a specialist in interaction and visualization, has published numerous

6 papers offering solutions to overcoming the problem investigated in this paper

(Baudisch, 2011). Though many of Dr. Baudisch‟s papers are discussed in the essay, the aim is to cover a wide range of researchers who offer their own unique designs and contributions while trying to avoid duplication of concepts (like wearable displays) discussed in much of the research uncovered.

For the interested reader, I would certainly recommend investigating each researcher further. Their work is both fascinating and far reaching in terms of innovation. Dr. Baudisch maintains excellent examples of this on his personal website at http://www.patrickbaudisch.com/publications/index.html, which I would consider to be an excellent place to start furthering your understanding of future mobile computing beyond what this paper introduces.

7 DEFINITION OF TERMS

1G First Generation

2G Second Generation

3G Third Generation

4G Fourth Generation

2D Two Dimensional

3D Three Dimensional

HCI Human Computer Interaction - a sub-field within the study

of information systems.

HTML5 A new and open standard of HTML that “lets web developers create

advanced graphics, typography, animations, and transitions without

relying on third party browser plug-ins

(like Flash) (Apple, 2011e)

Mobile Visualization The visual presentation component of mobile UI‟s

(Paelke et al., 2003)

Mobile Interaction The design of interactive (mobile) products that support people

every day, both personally and professionally, (Huang, 2009) where

culture, environment, activity, user goals, attention level, tasks and

goals are considered as varying levels of interaction. (Savio and

Braiterman, 2007)

8 Mobile User Interface The means in which a user interacts with a device

(Savio and Braiterman, 2007)

Multimodal Interaction Use of more than one interaction technique to gather input, provide

output, or interact with a user.

Mobile Worker People who work from either offices or home-based businesses

using mobile techniques. Categories include: office-based, non-

office-based, and home-based.

(Drake, Boggs, & Jaffe, 2010)

TFT LCD Thin film transistor liquid crystal display is a technology used to

improve quality on computing screens. Relevant types are Twisted

Nematic (TN) and In-plane Switching (IPS) as described in Chapter

III (TFT LCD, 2010)

W3C World Wide Web Consortium – a standards community governing

the protocols and guidelines used in the world wide web.

WAP Wireless Application Protocol - a protocol to provide web-like

material on current and future phone networks.

(Buchanan et al., 2009)

9 ORGANIZATION OF THE REMAINING CHAPTERS

The remaining chapters in this study provide a solid background, analysis, and discussion concerning the introduced research problem - what is being done to mitigate the small screen visualization and interaction problem by HCI researchers. While chapter II studies the overall contributions and state of HCI research, it is chapter III that concretely demonstrates how both past and present solutions have been implemented into the hardware and software of today's cellular-based mobile devices. These chapters portray the background and application of HCI research in order that the observed issues, challenges, and trends can be introduced in Chapter IV. Chapter V concludes the paper by discussing the implications of the main findings in the previous chapter and offers recommendations for future research.

For a better understanding of the used to obtain results in Chapter IV,

Appendix A relates the research to a context of mobile interaction model proposed by researchers that is introduced in Chapter II. This model helps to elegantly organize and highlight the findings.

10 CHAPTER II

REVIEW OF HCI RESEARCH IN USER INTERFACE TECHNIQUES

RESEARCH METHOD

Information presented in Chapter II was found using three distinct, but complementary, strategies. First, two key journal articles were obtained that provided a good overview and history of research into the addressing the visualization problem.

These articles helped to formulate a framework (which I call the 3 paths followed by researchers since 1997 to today) that helps find and categorize research into past, present, and future user interface techniques. Second, keywords were developed that identify research relating to the distinct focus researchers had in each path. These keywords were used to search for peer-reviewed journal articles, that combined, provide a good overview of work that has gone on over the last two decades to address the small screen issue. And finally, I rounded off the search for information by using knowledge obtained in the last year of the MScIS program at Athabasca University.

Collaborative coursework in understanding both HCI and Mobility was reviewed over

Athabasca‟s Moodle e-learning tool to ensure that both breadth and depth of this issue was covered appropriately for the purpose of this essay. The next few sections provide details of the three strategies listed above.

11 Key Articles

 Browsing as the killer app: Explaining the rapid success of Apple‟s iPhone (West

& Mace, 2010)

 My new PC is a (Baudisch & Holz, 2010)

Three Paths Found:

1. Pre-iPhone: Up until 2007, research was geared mostly towards overcoming

small screen constraints through better visualization and interaction techniques.

Research was geared towards migrating existing web to a mobile format

and creating a second version of the internet called the mobile web. Two primary

reasons: low data speeds and stakeholders wanted control over this market to

maximize their profits. (West & Mace, 2010)

2. iPhone: The iPhone in 2007 changed the view of adapting the web with Steve

Job‟s vision of offering “the real web” on a mobile device. Statistics confirmed

that the larger screen and newer interface techniques leveraged by the iPhone

led to dramatic adoption of mobile web access by users expecting and finding a

more desktop like experience. (West & Mace, 2010)

3. Post iPhone: Researchers have been running with this new shift in ideology to

present user interface designs that borrow from past work where applicable or

are innovative in nature.

12 Journal Article Search (Keywords or Ideas)

 Path 1: Mobile Web, Migration, Screen Visualization & Mobile Interaction

Techniques

 Path 2: Small Screens & HCI

 Path 3: Journal articles and websites that show advanced techniques that build

on the new direction set in path 2. By advanced, this means both innovative and

fresh ideas, and a re-application of existing work that fits well into the new

ideology.

RELATED WORK

The goal of this literature review is to review examples of past, present, and future user interface research. Though it would be impractical to introduce all of the researched interface techniques investigated since the beginning of the mobile market, it is prudent to review many of the major discoveries and innovations that have helped shape today‟s mobile products.

There is some key terminology that is particularly fitting to the study of small screen user interface techniques – multi-modal interaction, mobile interaction, user interface, and visualization. It is therefore important to review these four concepts as they appear frequently in the related research presented in this chapter.

Multi-modal Interaction

Karray (2008) highlights and emphasizes the changing focus from Unimodal HCI systems (that perform input and output functions based on visual, audio, and sensor 13 based modalities) to Multimodal HCI systems (where combinations of these interfaces types work together to enhance interaction). Examples highlighted include:

1. Visual-Based  Facial expression analysis, body movement tracking,

gesture-recognition, and gaze detection,

2. Audio-Based  Speech recognition, speaker recognition, auditory emotion

analysis, User-generated detections, and musical interaction, and

3. Sensor-Based  Pen, mouse, keyboard, joystick, haptic, , pressure,

and taste/smell interactions.

Recent innovations that provide users with more interactions that are multi-modal are discussed frequently in the paper and it is therefore important to understand this distinction and movement to how both researchers and manufacturers are combining modalities to improve user experiences with small screen devices. An excellent example of manufacturers combining sensor-based touch screen technology to enhance visual-based interactions is discussed in Chapter III with Apple‟s introduction of AppleTV which permits users to move digital content to larger screen devices like

Projection TVs or to screen-less devices from their and/or .

Elements of Mobile User Interaction Design

Huang (2009) defines interaction design as “designing interactive products to support people in their everyday and working lives”. It could then be argued that interaction design is synonymous with mobile interaction, with the understanding that the definition now concerns mobility. The definition of mobile interaction was explored

14 by Savio and Braiterman (2007) and showed it within various levels of context (figure 1).

According to the authors, designing for mobile interactions therefore involves a combination of factors that must be considered such as: culture, environment, activity, user goals, attention level, tasks and goals. As the interface and device fall within the domain of mobile interaction that is under control of researchers, it becomes clear the importance and context of the interface in the larger picture.

Figure 1: Context of Mobile Interaction (Savio and Braiterman, 2007) 15 Permission to make digital or hard copies of all or part of this work for personal or use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Mobile HCI 2007, September, 2007, Singapore.

Copyright 2004 ACM 1-58113-000-0/00/0004…$5.00.

As well, Huang (2009) stated that HCI is multi-disciplinary and often has specific challenges for mobile interaction design at both the hardware and software level.

Briefly, hardware often has limited input and output features and keeping the device mobile is a challenge. Concerning software, it is challenging to design effective hierarchical structures, creating usable navigation and browsing techniques, and designing images and icons that are optimized for and usable on the small screen.

These observations are useful when understanding and evaluating interface techniques as covered in the remainder of this chapter.

Mobile User Interface

Savio and Braiterman (2007) showed the user interface in Figure1 as the means in which the user interacts with the device. Quite simply, the user interface is the going concern in this paper as the goal is to discover how researchers have risen to the challenge of displaying information on small screens by reviewing the parallel development of user interface techniques alongside the maturation process of the mobile phone/device.

16 Mobile Visualization

So what is mobile visualization? Paelke, Reimann, & Rosenbach (2003) use the definition - it is “the visual presentation component of mobile UI‟s – because graphical display is a critical factor in mobile UI design, that differs significantly from conventional graphical UIs (Brewster, 1999; Rist and Brandmeier, 2001)”. A simpler way to understand mobile visualization is to think of it as the way users see and make sense of information on mobile devices.

Migration and Navigation Techniques

It is also useful to introduce some background knowledge of different migration techniques before beginning to review the various UI techniques – where migration is the adaptation of internet content for use on small screen browsers – as it will help to make sense of strategies leading up to today‟s presentation of the web on mobile devices. MacKay (2003) discussed web page migration and navigation techniques used by researchers as follows:

1. Direct Migration – no change to data and a reliance on scrolling or paging is

used,

2. Data Modification – data is changed (reduced) to work better on the small

screen,

3. Data Suppression – certain data is selected for display, but other levels can be

accessed

4. Data Overview – the whole data can be displayed as a summary and also broken

down.

17 Examples of Past User Interface Techniques

Selected techniques discussed in this section relate to path one – the period of time that researchers devoted to overcoming small screen limitations by creating or improving interaction and visualization techniques to improve the small screen. Major focuses ranged from: dealing with how to optimize the view of items on the small screen, to making navigation easier, to changing content (migration) for better viewing on the mobile web. It is important to note that each idea discussed was refined and extended into other ideas, which demonstrates a stable progression towards techniques in use today and allows an educated forecast into future techniques.

Pre-iPhone: Small Screen Visualization Techniques and the Mobile Internet (WAP)

Werner (2002) identified and discussed many past visualization techniques intended to improve the display of information on small screens. These included: bifocal display, fisheye, graphical fisheye, cone tree, perspective wall, document lens, hyperbolic browser, and Table Lens. Around the mid-to-late 1990s, mobile devices became powerful enough to access online content and technologies like WAP were developed to create a mobile web. However, WAP was not widely liked; in fact many hated it (Buchanan et al., 2001). So, other researchers focused on techniques like

SmartView, The Gateway, and Summary Thumbnail as alternatives to migrate websites through adaptation for use on the mobile web. The remainder of this section provides a description of each technique, along with information on its purpose and benefit, original inventor, dates, and important results or conclusions obtained from the research. The purpose of this section on past interface techniques is to show how early work into 18 addressing small screen limitations started before a clear mobile data market emerged.

Then, as the mobile market matured and devices became more powerful, small but incremental advances propelled the industry towards today‟s robust interfaces.

Werner (2002) cited the bifocal display (created by Spence and Apperley in

1982) as one of the earliest attempts to show detail from large data visualizations on single displays. However, information displayed in detail had a clean separation from the context it appeared in, which made it less effective and more two dimensional than other techniques.

Furnas (1986) described a fisheye technique that could “provide a balance of local detail and global context by trading off a priori importance against distance”. In simpler terms, a user could see the bigger picture (which was usually too small to make sense of) but the focused area of the fisheye lens would magnify the information being presented. Conclusions showed that participants were able to review information more effectively with use of the fisheye technique.

Sarkar & Brown (1992) extended Furnas‟ research by providing a detailed process that could apply the fisheye technique to graphs (called the graphical fisheye).

Their study focused on how to implement this technique and concluded that it was only one of many possible ways to visualize graphical data but could merit more research.

This technique is important as it is deployed in many of today‟s mapping utilities.

Robertson, Mackinlay, & Card (1991) introduced the cone tree technique where large amounts of information related in a hierarchy could be presented in a 3D manner.

19 As related data expanded from the root, the information took the shape of a cone organized in a tree. Their study applied this technique to typical tasks such as file browsing and visualization of document structure though their technique could be applied to other abstract areas like network organization, software and document management. They believed that use of 3D and interactive animation in the cone tree technique could overcome the problem of displaying and navigating large amounts of information which is why Werner (2002) stated that techniques like this would become very important in overcoming the small screen issue.

Mackinlay, Robertson, & Card (1991) also introduced the perspective wall where, again, they employed 3D and animation to display detail and context regions. The detail region was a rectangular area centered on the display while information to the left and right formed a three-dimensional wall also displaying related information meant to provide context. They stated that this organization scheme was very similar to how the human eye organizes vision by focusing elements and were excited that it could visualize complex structures where “highly interactive user interfaces are likely to support large scale cognition” (Mackinlay, Robertson, & Card, 1991).

Robertson and Mackinlay (1993) proposed the document lens which allowed large amounts of text to be read in context by focusing on parts of the text in a magnified manner while related parts were pulled around the lens (resembling a truncated pyramid) to provide context. They were trying to make reading entire documents easier but admitted that contextual information in the pyramid became harder to read at points which somewhat defeated the purpose. However, this initial

20 attempt to promote readability certainly led to innovations seen in today‟s document readers such as Apple‟s iBooks application.

Lamping and Rao (1994) introduced a hyperbolic technique that combined the idea of a tree structure with the fisheye technique. This allowed for central information to be shown within its location in a hierarchy. When focus was shifted to another node, the tree could realign elements around it so that a user could always see both the node‟s context and the parent and children‟s detail in a single view. The benefit to this technique is the ability to display up to 10 times more nodes than traditional browsers.

Werner (2002) concluded the study by testing information browsers that had implemented these techniques such as Inxight‟s StarTree, The Brain, HypViewer, and

Microsoft Internet Explorer. An interesting finding showed that Internet Explorer was both the fastest and most accurate means of navigating large amounts of information.

Though, Inxight (2008) lists their hyperbolic-based browser as the “fastest way to integrate the most powerful navigation tools available today into your software applications” and states that it is in use by many applications and information portals.

Rao & Card (1994) presented The Table Lens which “uses a focus + context

(fisheye) technique that works effectively on tabular information because it allows display of crucial information and multiple distal focus areas”. Again, this was a reapplication of a prior technique to solve a specific visualization problem – in this case display of tabular data.

21 Around the year 2000, mobile phones were just starting to access the internet and a new mobile web was created using a new mobile markup language called WAP

(Wireless Application Protocol). However because the new protocol was so limiting,

Buchanan et al. (2001) performed a usability study intended to solicit the help of HCI to redeem it, as there was a large rejection and backlash from people who saw the potential of accessing the internet over mobile devices but were not happy with “the

Wrong Approach to Portability” (Buchanan et al. (2001). The study recommended mobile sites should “trim”, “reduce”, and “simplify” web sites for better usability and the researchers concluded that, though the market was in transition, WAP could still work.

Milic-Frayling & Sommerer (2002) presented SmartView (a Microsoft initiative) which was a web page document viewer that could be either built into mobile device browsers or used as a service. They stated that in 2002 mobile devices began to outnumber traditional desktops and that many available migration techniques modified the page making it hard to use. SmartView (like the next two migration techniques presented) was a more elegant way to view both overview and detail content as it attempted to keep the of the original web page. This was accomplished by creating a thumbnail image that used the HTML format to break the page into logical sections. When a user clicked a section, they would be zoomed into that content creating a better viewing experience.

MacKay (2003) described the Gateway as a data overview migration and navigation technique applied to standard web pages. This technique displayed the web page the way it was authored and intended to be viewed but also moved to a detailed

22 view of a page element (like navigation) when the user selected it. The researcher thought a primary benefit was to reduce the work a user does to effectively navigate and will help keep context to prevent disorientation.

Lam & Baudisch (2005) presented a similar but improved technique over

SmartView called Summary Thumbnail (also a Microsoft initiative) three years later.

The difference was the inclusion of a semantic zoom feature that modified text based sections to display in a more readable format.

Hakala, Lehikoinen, & Aaltonen (2005) introduced the concept of spatial interactive visualization. The proposed Space Manager, a graphical document management system that uses a tree structure combined with 3D aspects to interact more fully with users via spatial memory. There is a navigation level which shows context and a folder level that graphically shows files in the folder. Users can then use navigation arrows to flip through files in the folder or move between levels of view. This visualization and interaction technique is optimized for small screen navigation of documents which the researchers claim is promising for its use of memory techniques and deserves more investigation.

Examples of Present User Interface Techniques iPhone: Zooming User Interfaces, Multi-touch, Multi-modal Interaction, and Gestures

Selection of interface techniques in this section are based on those present and generally accepted in today‟s mobile devices. These techniques also reflect the apparent change in focus ushered in by the release of the iPhone in 2007. This 23 milestone signified a shift towards providing the “real internet” (path 2), as promised by

Apple and CEO , on mobile devices that were growing in computational power (West & Mace, 2010). This shift also sees WAP and adaptive research like

SmartView, the Gateway, and Summary Thumbnail as being slowly left behind in light of more promising techniques. This is recognized by some mobile technology commentators (Bournique, 2010) who see the decline of the mobile web because of major advances in mobile device browsers. Some of these advancements discussed in this section are: touch and multi-touch screens, gesture-based interaction, zooming interfaces, multi-modal interaction, and one-handed techniques.

Albinsson and Zhai (2003) discuss touch screen interaction as “literally the most

„direct‟ form of HCI, where information displayed and control are but one surface”. This distinction demonstrates that visualization and interaction can work together simultaneously – particularly in mobile situations. The researchers noted that two major drawbacks included the display being obscured by users and that interaction with fingers can be imprecise. Their study reviewed prior selection techniques such as zoom-pointing (making a target bigger by zooming to better select it) and take-off (using a precision pointer that is off set above the finger which selects targets when the finger is removed from the screen). They investigated a number of techniques and decided on evaluating two of the best, cross-keys and precision-handle, against the zoom-pointing and take-off techniques. They concluded that some techniques work better than others depending on the usage and situation. As such, “in practice it is probably more desirable to switch tools according to different needs, just as in the physical world we

24 use pliers, wrenches, screw drivers and other tools selectively” (Albinsson and Zhai,

2003).

Multi-touch surfaces, or MTS, was defined by Westerman (1999) as “a surface with a proximity sensor array underneath capable of unambiguously measuring the positions of multiple finger contacts”. In his PhD Dissertation, this researcher claimed that MTS, though a more advanced technique than the mouse, had not been widely accepted because the technique was not refined enough for seamless integration with today‟s devices. This understanding is important as according to Buxton (2007), multi- touch systems have been around since 1982 and are an important HCI technique because “if you can only manipulate one point, regardless of with a mouse, touch screen, joystick, trackball, etc., you are restricted to the gestural vocabulary of a fruit fly.

We were given multiple limbs for a reason. It is nice to be able to take advantage of them”. Westerman used his research to co-start a company called FingerWorks which was bought by Apple in 2005 (Buxton, 2007). And two years later, the iPhone was released which set a new precedent for mobile computing interface techniques.

Moyle and Cockburn (2003) identified simple web navigation techniques, such as back and forward in a web browser, as an important area of research because simplification of small tasks can lead to large increases in efficiency. The problem with the traditional back is that the user must navigate large distances to click on a small object which is inefficient according to Fitt‟s Law (a basic HCI design principle that relates speed of a task to the distance a user must travel to select it and the size of the selection). These researchers noted that certain browsers had just implemented the

25 technique called flicking which used a back or forward flick gesture on a screen to move between pages. However, as no significant usability analysis had been performed they surveyed a typical user group to perform common browsing tasks using the technique.

They concluded that users had a strong preference for the technique and that it improved the speed of navigation in some situations by 18%.

Johnson (1995) evaluated panning techniques on touch screens, where panning is movement of the visible area on a screen that is too small to display the entire area at once. The study evaluated three techniques (panning by pushing the background, the /view, or the side of a window) by user survey. They concluded that touch screen interface design could benefit from different techniques than design for normal computer screens such as the use of . They found that users preferred pushing the background to pan over other techniques.

Cho, Murray-Smith, & Kim (2007) studied tilting (changing the physical orientation of a device) as a more effective means of interaction with a photo than previous techniques such as button browsers and iPod wheels. Their proposal was that the library could be scrolled based on the amount of tilt. However, three issues became apparent such as scrolling too far, imprecise selection if users had shaking hands, and stopping between photos while actually wanting to view one at a time. The researchers concluded that tilt based browsing has many advantages like not using any space and being straight-forward and acceptable to users. Their method was found to be more precise and faster than previous methods.

26 Hornbaek, Bederson, and Plaisant (2002) wanted to know if zoomable user interfaces (ZUI) were more effective using an overview or not – where the overview showed the overall context for map based searches. They defined ZUIs as techniques that organize information by both zooming and panning to show context and detail.

Results showed that users liked having context through use of an overview mechanism but performance was better for maps without an overview. In conclusion, the researchers stated that more research should be done to improve the overview and its performance as people really liked it.

Fallman, Lund, and Wiberg (2004) identified scrolling or adaptation as accepted methods to presenting lots of information on small mobile screens. To improve scrolling with small devices, they matched circuitry of an optical mouse with the device and used software to convert movements into scrolling actions on a display. They highlighted that evaluation was not the purpose of their study (they just wanted to introduce another possibility for scrolling) but did comment that users seemed to view the technique as intuitive, possibly because it matched scrolling done with regular computing devices.

Di Fabbrizio, Okken, and Wilpon (2009) examined methods of multi-modal interaction to enhance and facilitate better mobile interaction. They proposed that a speech recognition mashup (combination of multiple technologies) could be used to overcome traditional limitations of input to small screen devices. In other words, if devices could be controlled with speech then devices might become more usable. The study specifically looked at combining speech web services with cloud computing to enhance web content and interaction on mobile devices. Though millions of scenarios

27 are possible, the study introduced three case studies where a user could speak a request to the mobile device and be presented with results – such as the ordering of pizza or finding of business listings. This type of research is relevant to overcoming the limitations imposed by small screens as it lessens the interaction needed with the display in order for users to achieve their goals efficiently. Zhang, North, and Koutsofios

(2010) supported these conclusions in a similar study that looked at evaluating a speech mashup against traditional (GUI) techniques. They wanted to see if speech would be more efficient for navigating or inputting information in mobile devices. This study concluded that “speech input is as effective, is more efficient, and is preferred by users for operating a web-based application on the iPhone”

(Zhang, North, & Koutsofios, 2010).

Brewster (2002) studied the use of sound to overcome small screen button sizes as another multi-modal technique. This study hypothesized that the introduction of sound onto button selection would promote better usability. It was found that button sizes could be reduced significantly if users had auditory selection feedback, and that button states could be communicated with the use of more sophisticated sounds. At some point, buttons became too small however which increased workloads (time, effort, frustration, etc.) so “designers in general should avoid very small buttons (or targets) as they will significantly reduce usability, but if they must be used then they should have sounds” (Brewster, 2002).

Researchers have also looked at ways of improving single-handed mobile techniques. For instance, Roudaut, Huot, and Lecolinet (2008) introduced TapTap and

28 MagStick as methods to better select screen elements. TapTap allows a user to tap once to zoom into a target area, and then the second tap provides a better opportunity and view to select the target more accurately. MagStick, on the other hand, permits a user to touch the screen close to an object and then move the thumb slightly to shift the protruding selector to select another nearby target (in a magnetic fashion). The study concluded that both techniques are superior to other selection techniques and have some inherent benefits such as no thumb occlusion and ability to target anywhere on the screen.

Examples of Future User Interface Techniques

Mixed Reality and Advanced Interfaces: Gesture UI, Tactile UI, New Visualization

Techniques, Screen-less & Eyes-Free UI, and Extending the Screen

This section introduces some of the most recent research directions in mobile interface techniques. It is interesting to note that research is both progressing towards and shifting away from the use of the traditional small screen. For example, some researchers are trying to augment the screen to make it better by making it double sided, transparent, or building in tactile feedback mechanisms. While other researchers are leaving the screen behind in favor of imaginary (screen less) interfaces, projection, or screen less device interaction (i.e. foot gestures). A third observation shows the extension of the screen to view on other devices such as whiteboards, surfaces, or wearable displays built into glasses. This section will review research on these and

29 other new techniques that address the small screen, such as: hardware and software augmentation, gesture interfaces, tactile interfaces, new visualization strategies, use of other screens, and wearable solutions.

Augmentation of the Screen (Hardware and Software)

Figure 2: Paul Milgram's Virtuality Continuum (VC)

(Mixed Reality, n.d.) .Reproduced with permission

According to Mixed reality (n.d.), mixed reality constitutes a spectrum of real versus virtual environments (figure 2). Researchers are developing techniques across this spectrum to address the visualization of information on small displays. Schinke,

Henze, and Boll (2010) introduce an excellent example of how point of interest (POI) sites can be displayed on the small screen to deliver information to the tourist using the application. They introduce another HCI innovation, off-screen areas, as a means to only display relevant information based on what is being captured by the mobile phone‟s camera. Their proposal to advance this technique was to use 3D arrows to show other

POI to tourists as a means of improving usability. This is also an excellent example of how HCI is placing priority on Natural User Interfaces (NUIs) on mobile devices.

Natural user interface (n.d.) describes NUIs as “effectively invisible, or becomes invisible with successive learned interactions...(and) relies on a user being able to quickly transition from novice to expert”. 30 Yee (2003) presented peephole displays which used the small display on a mobile device to peer onto a larger virtual canvas. Using movement sensors, a user was able to pan over the virtual canvas by using one hand to move over the display while the other hand used a pen to select and manipulate the screen display. Imagine having a large monitor in front of you that cannot be seen without the use of your mobile phone window. Yee provided an example where two applications are open side-by- side. On the left is a /mail application while on the right is a drawing application. A user could move their phone toward the right display to draw a map for a party and then use the pen to select the finished map. The map could then be dragged over to the left display via movement of the mobile device and then dropped into an email as an attachment. A message could then be typed using the pen and the invitation could be mailed.

Another research area involves changing the screen or other input/output facilities such as see-through or multi-side devices. Wigdor, Forlines, Baudisch,

Barnwell, & Shen (2007) introduced LucidTouch, a transparent multi-sided device that overcomes small screen constraints such as occlusion (fat fingers getting in the way of the display during use). Users were able to touch the front or touch the back of the screen in a touchpad manner while being able to see the entire front view unfettered.

Conclusions stated that users have different preferences based on the task they are performing. For instance, text entry was preferred from the front while manipulating objects was split between front and back.

31 Shen, Tsai, Chu, Hsu, and Chen (2009) extended dual sided research to introduce double side, multi-touch interaction of 3D objects. Their research intended to demonstrate how it may be possible to interact with objects on a small screen, using advanced double sided methods, to perform life-like interactions upon the object. Their research defined simple dual sided gestures such as grabbing, dragging, pushing, flipping, and stretching an object and they concluded that future and more advanced research could be very exciting. For instance, it may be possible to exert force on a virtual object that would behave like real world equivalents - like elastics may stretch back when exerted and released, or objects could be bent, torn, or even broken.

Go and Tsurumi (2010) highlight the problem of finger selection occluding keyboard text selection on very small touch screens as they considered it to be a critical task. They introduced a dynamic split-key software keyboard technique that operates like a full QWERTY keyboard for precise selection using a stylus, but that can also allow fingers on very small screens to make accurate selections through a two-part process.

Basically, the first selection of a finger triggers a pie-shaped area full of the possible keys selected by the user. A second selection is then used to select the actual key.

The study supported their hypothesis that for stylus use this method is as fast and has the same error rate as a full QWERTY keyboard and for finger use it has equal performance to a static split-key keyboard.

Other types of advancements are focusing on improving the output of the mobile display. Researchers at Nokia (2009) have created a Multiview Video Coding (MVC) extension to the widely used H.264/AVC standard seen in 3D televisions to showcase

32 Mobile 3D Video. Some key features cited include better compression of 3D video for use on a mobile device and ability to efficiently use currently deployed data streams with backward compatibility using techniques aimed at threading information for performance and better buffering. The end goal is to create a more virtual environment that can enhance current display capabilities on capable devices or work seamlessly in

2D on less capable devices.

Gesture Interfaces

Stobel and Blessing (2010) were interested in showing that differences can exist between younger and older users in their preference and usage of gestures. Their study looked at how different generations viewed gesture input of 34 common tasks that are performed on mobile devices (like confirm, delete, end call, etc.). The results showed that older users prefer symbolic gestures like double tapping to zoom over pinch and zooming techniques, and were less likely to like multi-finger gestures, but were also more willing to accept more complex interaction techniques. The research highlighted a need to further investigate the differences between old and young in using new computing techniques for small devices, like gestures, to improve the experience for all. It was suggested that multiple gesture profiles may need to be implemented to optimize devices for different ages.

Li (2010) stated that finding information on mobile phones using current text entry techniques is challenging due to the limitations of small devices and therefore proposed

Gesture Search as a gesture based solution. The main idea is that a background program overlays the entire mobile screen and watches for pre-recorded or text-based 33 gestures. When the overlay recognizes a gesture, it turns the gesture yellow to indicate to the user that it is performing a search. Then, a list of results is shown that the user can select from. A difference from similar techniques is Gesture Search‟s ability to learn from past user selections to make frequent access of popular like contacts and applications more intuitive. The study initially obtained positive feedback concerning usefulness and usability and was then made publically available by Google.

Costanza, Inverso, and Allen (2005) proposed the creation of intimate interfaces where body sensors (using EMG, or muscle contraction) could be used to control input to mobile devices in discreet ways that would be acceptable and unobtrusive in public places. They stated that advancements in eyewear based display could be used to create a new generation of mobile devices that are even more personal than today‟s mobile devices. Results showed that input could be distinguished with high precision, even without prior user calibrations, which offers potential solutions to the problems of using mobile device screens while walking, running, or being engaged otherwise.

Tactile interfaces

Stewart, Rohs, Kratz, and Essl (2010) investigated pressure-based input as a method to enhance touch screen interaction. Pressure based input may be important in the future in certain types of applications that must be discreet or involve security. The researchers stated that “typical applications for pressure input are widget control, menu item selection, expressive typing, conveying the urgency of phone calls, and zooming

(Stewart, Rohs, Kratz, and Essl (2010) and is useful for situations where the small screen is a hindrance or using finger based input is not desirable. 34 Bau, Poupyrev, Israr, and Harrison (2010) also aimed to enhance touch screen surfaces with tactile feedback that uses electro-vibration, and their implementation was called TeslaTouch. Electro-vibration is caused by energizing an insulating layer on the touch screen that reacts by vibrating when a finger moves across the surface. As this current does not pass into the finger, this technique is considered to be safe. Test subjects reported different sensations at different frequencies, such as feeling paper or fluid, among others which is promising for all sorts of applications that could benefit from touch sensation. One drawback to this technique is that a finger must be in motion to feel anything, though other similar techniques that use mechanical vibration do not have this limitation. The researchers stated that their implementation is very reliable as there are no moving parts and that the entire surface can provide tactile feedback.

Hall, Hoggan, and Brewster (2008) introduced T-Bars, which is a tactile replacement for traditional GUI buttons and is intended for non-visual use on mobile screens. The researchers were focused on applying tactile research to create better UI designs and succeeded in creating an experience that some testers commented was like flipping through files in a filing cabinet. The premise is that a user slides their finger across a virtual bar that increases its tactile intensity as they near the end of the bar. At the end, a click signals a selection has been made and then the appropriate action is triggered (i.e. a file opening). User testing showed that a number of small adjustments to the original design could be made to improve the experience but overall the potential for non-visual feedback was good and warranted further research.

Advanced Visualization Techniques and Frameworks

35 As mentioned, visualizing large amounts of information on a small screen is challenging. Buring, Gerken, and Reiterer (2007) introduced the use of Scatterplots as a technique to view large datasets such as a book database containing thousands of books. They used two different visualization techniques with the scatter plot to test user preferences. The first scatter plot used a geometric and semantic zoom where pixels represented books in the graph. The user could zoom in and change the size of the pixels (geometric) and when they selected a book the view was altered to show detailed textual information (semantic). In the second scatter plot, a fisheye technique was used that recursively allowed users to zoom into a book on the graph while retaining more control of the context. It surprised the researchers to learn that users preferred the ability to see both context and detail on a small screen over the easier to use zooming method. Both methods had the same performance.

Francone, Bailly, Lecolinet, Mandran, and Nigay (2010) presented Wavelet

Menus which is a space-saving technique for menu selection that involves stacked concentric rings which assists users in making selections in a one-handed and gestural manner. When a user touches a ring, another ring with more sub-menus appears around the initial ring. The technique has a novice and expert mode where experts can use gestures in an „eye-free‟ manner to make selections; they don‟t have to be looking at the display to use the device. The researchers stated that this technique is an extension of a previous technique called the wave menu and is intuitive to use once participants actually began to play with it. An advantage over desktop based menu selection is the ability to use the technique in mobile contexts such as walking or running 36 Sousa, Nisi, and Oakley (2009) introduced the Glaze Visualization Framework which builds on work done by Card in 1999 to create a visualization framework intended to simplify and expand the capabilities of mobile screen visualization techniques. They expanded Card‟s information visualization reference model to more cleanly separate graphical components based on tasks needed to display them. Their framework uses:

1. A core – to manage core items in Card‟s model such as the transition of raw data

to organized data, to be implemented with a chosen visualization structure(s) to

be displayed in an intended view,

2. Input – capturing the data to be used,

3. Visualization – choosing an appropriate visualization technique, and the

4. Front-end – which controls the display and maintenance of the resulting view

They were able to show how an organized framework could include multiple sources of input from all of a mobile device‟s hardware (i.e. the compass, GPS, or accelerometer) to handle complex visualizations elegantly. Examples included using data from a device location to show a user‟s surroundings to help navigate at night or overlaying information on a map using context and detail techniques like Furnas‟ fisheye. This study demonstrates that researchers are trying to organize and simplify the use of techniques that will help overcome small screen display issues.

Screen-less Techniques & Eyes-Free Interaction

Ko, Chan, and Hung (2010) envision a future where Projected User Interfaces

(PUI) become fully integrated into mobile devices and their study is aimed at addressing

37 some of the human factors that may become problematic when this happens.

Currently, companies like Nokia have built projectors into hand-held devices like cameras and other companies are currently selling mobile handheld projector units to work with mobile devices. The researchers covered a broad spectrum of issues such as projecting around others (into personal or intimate space), public viewing of private content, projection onto others, competition for projection space, multi-user interaction with projection, and impact on the environment (light pollution). They offered suggestions for addressing some of these issues such as sensors that enforce no- projection areas, limiting view to, or using alternative light spectrums to hide projections, or shifting projections away from other projections automatically.

Gustafson, Bierwirth, and Baudisch (2010) introduce the concept of Imaginary

Interfaces where short-term memory and imagination can be used with gestures and mobile cameras to create virtual products. Their example shows how a mobile user can activate a camera and use a series of gestures to draw and refine an imaginary stock graph that can be seen by an accountant in front of a computer. They stated that this technique could be combined with other future interfaces like wearable display glasses to offer feedback but emphasized that smaller, screen-less devices could leverage this technique to interact better with users. The focus of the study was to show how spatial interaction can be created with varying combinations of mobile hardware to create a more mobile experience that lets people perform relatively complex tasks while on the go, and without the need of a screen.

38 Scott, Dearman, Yatani, and Truong (2010) were interested in developing hand and eye-free techniques that could monitor foot gestures to gather input for mobile devices. Initially, they looked at all of the possible ways that the human foot could form a foot gesture and concluded that their “system can classify ten different foot gestures at approximately 86% accuracy” for mobile gestures where a phone might be in a pocket or on a belt. They stated this type of interaction would be useful for controlling devices with common hardware such as the accelerometer in a socially acceptable and non-intrusive fashion.

Extension of the Screen

Boring, Baur, and Butz (2010) developed Touch Projector to enable manipulation of content on existing surfaces in an environment (such as a whiteboard or projection desk) using their mobile phone as a control device. Server software within the environment, called an environment manager, keeps track of all documents and devices using the environment and allows devices to interact with the documents through gesture events. Events can include moving documents on the screen or transferring documents to different displays using the mobile phone as a multi-touch control pad.

The researchers stated that some limitations exist such as distance to the projector and shaking of the mobile screen while interacting with the environment. Their work built in and evaluated features such as zooming and interacting with snap-shots to address these concerns. Ongoing, they mentioned the future improvements in hardware will improve this technique. This technique can be used to interact with public displays or displays without any interactive features built-in. During future research, they would like

39 to find out if computer generated content will help alleviate the need for better zooming ability and where the best area is for giving feedback to a user.

Ferscha and Vogl (2010) recognized the need to bring the aging concept of wearable computing (in particular a wearable display) to fruition. Their research assembled publically available computing components into a new breed of innovative mobile device they called Spectacles. The device itself is modular and consists of a display unit built into eyewear and an arm-band mounted power unit. The overall unit contains:

 A 4.8 x 3.6 mm screen,

 A camera capable of gesture recognition,

 A voice/speech processing unit,

 An array of biometric (heart rate, respiration, etc.) and environmental sensors

(light, temperature, etc.)

 GPS and other Wi-Fi/RF capabilities (like ZigBee to control automation

technologies),

 A power unit,

Therefore, the system is modular in design which allows for customization according to need, and all is controlled by a layer based framework that encapsulates the communication and application protocols used by the various components. Some case examples were given to highlight the potential impact of this merging of new techniques with new technology. First, personal health could be monitored for many different scenarios such as mountain biking or running. Second, information could be easily 40 transmitted to shoppers about products and services. And finally, complex activities like medical surgeries, or other could use Spectacles as an assistive technology.

And finally, Doring, Shirazi, and Schmidt (2010) also extended the mobile display to other displays. They were also focused on exploring how mobile devices could interact with public displays but a goal of the study was to examine how to manage information in both a private and public manner. To do this, they implemented a number of scenarios including a poker table on a Microsoft Surface table. While the table displayed the public deck of cards and hands dealt, the mobile phone was used to hide personal information such as the face-down cards. Similar to other studies, the phone was used in a touchpad manner (using multi-touch gestures) to allow people to interact with the public displays. The researchers reported that users liked the new interaction technique and that it was easy to use.

Industry User Interface Guidelines and Other Useful Developments

This section has been included to highlight a possible trend seen in recommendations given to provide mobile content and also to report on an aging repository idea that gets to the heart of this paper – which interface techniques should developers and users adopt ongoing?

The W3C (2010) currently offers guidelines on mobile application development.

One of their stated best practices is to provide a choice of interfaces as device detection is not perfect. The recommendation is to default to the most appropriate interface, but

41 to give users the choice, and then remember the preference for future visits. This is significant as it may signify that interface design has a larger element of preference and that if we as users do not like an interface, we should be able to simply download another one that works better for us. I hope to show in Chapter III that this is a trend seen and supported in the new mobile landscape perforated by “apps” (interface type applications).

As well, Paelke, Reimann, & Rosenbach (2003) proposed a visualization design repository for mobile devices. The purpose was to help developers match visualization techniques to tasks they need to perform as well as to categorize their findings for possible inclusion into future mobile development guidelines. Though it is unclear if this repository was ever started, it was certainly an excellent idea that gets to the heart of this paper. Further research into this area using the defined process created by the authors may someday lead to the creation of a design tool for mobile devices that resembles the finds of other disciplines such as seen at http://www.visual- literacy.org/periodic_table/periodic_table.html.

CONNECTION TO CURRENT RESEARCH

Studying the related research will allow us to match solution-based user interface techniques to the actual devices that are employing them in today‟s mobile market

(Chapter III). This background knowledge, combined with visions for future interfaces, will solidify a base of information that the reader can leverage in their own understanding and selection of mobile products that utilize interface techniques matched to their need.

42

PURPOSE OF THIS STUDY

The purpose of this study was therefore to:

1. Examine how researchers have addressed the challenge presented by small

screens on mobile devices, and have changed their approach for both current

and future UI design.

2. Demonstrate how a shift in approach to the problem is being seen in today‟s

mobile devices (cellular phones, smart phones, and tablets), and

3. Challenge the reader to consider the importance of future and ongoing research

into addressing the limitations created by small screens.

43 CHAPTER III

SURVEY OF THE MOBILE INDUSTRY, HARDWARE, AND SOFTWARE

UI

RESEARCH METHOD

Various research methods were used to achieve an overview of the present direction of hardware and software seen in today‟s mobile devices. This chapter is intended as a factual listing of relevant information that will be used to identify industry issues and challenges, and formulate trends in the next chapter.

As such, the chapter begins with general background information including a historical overview of the cellular mobile market and a specific look at the changing nature of small screens. As most smart phone devices now support touch screen displays, this section provides a brief history of touch screens and a description of the various types being used. As well, a survey of the various operating systems is performed alongside a look at their features and innovative distinctions – one of which is the strong tie between manufacturer, proprietary , and their respective internet browsers. Where possible, interaction techniques discussed in Chapter II are matched to the operating systems that use them.

In the next section, a discussion of manufacturer specifications and features for select leading edge smart phones, tablets, projectors, e-readers, and other mobile devices is undertaken. An effort has been made to identify and discuss mostly the

“latest and greatest” mobile devices – or those being introduced or discussed in current

44 literature as being extremely popular or meant to enhance the mobile screen. For instance, the mobile industry‟s yearly Consumer Electronic Show (CES) in Las Vegas cites 2010 as the year of the smart phone and 2011 as the year of the tablet, which is important to this paper as it identifies a strong trend to making mobile device screens larger (see trends in chapter IV) at the expensive of device size. This section closely examines Apple mobile devices (iPhone 4 and iPad) as they are considered by most sources to be the most innovative and the best consumer devices on the market currently. Paul (2011) identified two large focuses at this trade show to be “iPad Killer

2011” (which devices might supplant the reigning iPad dominance), and for the next section “App-Tastic” (what mobile applications are innovative).

Finally, the chapter concludes this by examining software and user interface directions. The section starts with a broad look at “application stores” – stores owned and operated by manufacturers – as it could be argued that they are becoming a major source of software and interfaces being developed and distributed both openly and in a

“walled garden” approach, which is total control of everything by the manufacturer

(Vendel, 2009).

Research material for the above sections was selected using many different sources that could be considered to be current, factual, and objective in the review of devices, and is assembled from: manufacturer information (from their published web sites), reputable technical product review sites (Engadget and CNET) which includes consumer reviews and complaints, product magazines (Gadget, MacLife, and iPhone), and internet searches. By the end of this chapter the goal is to help answer the

45 following questions about the mobile industry and its‟ recent hardware and software innovations:

1. What is the basic history of the mobile market in North America? Briefly, how

have phones become more like personal computers?

2. How has the small screen evolved in this market?

3. What are the different types of touch screens available today?

4. What are the different mobile operating systems, and how do they use larger

screens with touch capabilities? (i.e. gestures)

5. What are the current flagship products for major mobile manufacturers, and how

are they competing in terms of overcoming the limitations of small screens? (i.e.

retina display)

6. What are newer product lines? (i.e. Tablets)

7. How is the small screen being extended? (i.e. Pico projector, smart boards

integration)

8. What software advancements have been made for use with the smaller mobile

screen?

9. What user interface techniques are being used for each mobile operating

system/product line?

46 SURVEY OF MOBILE INDUSTRY

Evolution of the Cellular Industry in Regards to the Screens of Mobile Devices

1G, , , and Mobile Networks

It is important to understand that mobile devices, though based on older technology like the telephone, are relatively new and are constantly evolving – rapidly.

According to Nubarrón (2011) many key breakthroughs in radio transmission (as early as 1857) have led to four distinct phases of growth in the mobile market. These phases are 1G, 2G, 3G, and recently 4G. Each stage has a distinctive set of characteristics which help to make sense of the initially slow evolution of the mobile device screen.

Nubarrón (2011) states that 1G devices were introduced in the 1970s and built on radio advancements performed around the world in the 1940s and 1950s. This era lasted until the early 1990s and was important as it helped establish a large pool of subscribers worldwide. One important characteristic of 1G was that it was only capable of analog voice transmission and there was no capability for data services.

2G mobile phones switched from insecure analog transmission (anyone could listen to the signal with proper equipment) to a digital based one in the early 1990s.

Characteristic of this era included a push for improving voice quality and expansion of the provider cell tower based networks to support better coverage. Digital services became more prevalent as well including faxing, short messaging, paging, and voicemail (Nubarrón, 2011). These advancements led to an increased need for more digital services including data. A new 2. standard was developed that like computer

47 networks allowed for the transmission of data packets over a standard IP protocol.

However, 2.5G networks were not capable of supporting the volume of rich media enjoyed on today‟s devices which led to 3G.

3G is the current standard and supports high data rates capable of mobile streaming media and voice over IP transmission (VOIP); basically voice services over mobile internet. A key characteristic of 3G was the development of worldwide transmission standards that allow for global with mobile devices, however, the standard still suffers from data speeds incapable of supporting mass usage of high data demand (Nubarrón, 2011).

On the horizon is 4G which promises substantially higher data rates (a ten-fold increase from the current 2Mbps to 20Mbps). Nubarrón (2011) concluded that these data rates will help to enable the new mobile world and its technologies – such as video conferencing in connected and moving vehicles.

In conclusion, advancements in mobile devices are seen to be market driven, and the mobile screen has evolved over time to meet the demands of consumers. It is therefore highly possible that the screen will continue to evolve and offer a desktop like computing experience – if – the world demands more capable devices to meet the demands of a global and highly mobile workforce. The next section demonstrates this evolution of the screen.

48 Mobile Display by Year

Mobile Screens as early as 1983 are visually shown by WebDesignerDepot

(2009) to include simple one or two line dot matrix type displays. Ongoing, screens trended to increase in size, color, and contrast up to around the year 2000 when simple touch screens began to be seen. Most phone models included a simple numeric keypad initially but the integration of PDA (personal digital assistants) capabilities led to alpha numeric keypads that became popularized as QWERTY keyboards for their similar to PC but simplified key layouts. In 2001, color screens began to be seen as devices had more internal memory to support features that needed better screens. This is consistent with the various generations of mobile networks previously discussed.

Pen-based color touch screens were seen in the early 2000s and devices also began to be equipped with more advanced hardware like cameras and dual screens.

In 2003, large manufacturers began to release product lines that used larger screens such as Research in Motion‟s first generation Blackberry. Internet connected devices around this time were seen to have increasingly larger screens and most included a keyboard – whether flip out, sliding, or integrated. However, like many industries, both low-end and high-end product lines were offered as seen by various offerings such as the low-end Nokia 1110 versus the higher-end 6680 model in 2005

(WebDesignerDepot, 2009). Pocket PCs loaded with Windows Mobile operating systems began to form a strong smart phone market. This immensely competitive market pushed all of the major manufacturers to begin competing heavily for market share and a large selling feature became the screen.

49 In 2006, models began being released with advanced hardware seen first in desktop computing such as dual core but the whole industry was turned upside down in

2007 by the release of the iPhone that furthered touch screens to include multi-touch interaction, virtual interface layouts, advanced cameras, and input sensors. Since

Apple‟s domination with the iPhone in 2007, WebDesignerDepot (2009) demonstrated a rush to create the next “iPhone killer” and shows how new operating systems such as

Google‟s Android in 2008 have begun to heavily compete for market share. It is very evident while following this visual blog that screen sizes, colors, and resolution increased dramatically after the introduction of the iPhone.

Touch Screen Natural User Interfaces

According to touch screens (n.d.), there are many types of touch screen technologies available today whose origins start with sensor experimentation in the early 1970s. These ongoing improvements have resulted in the various types seen in use today. Buxton (2007), a prominent researcher of multi-touch systems, provides a detailed chronological breakdown of the major advancements spanning his lengthy career in this area. However, for the purposes of this paper on mobile touch screens,

Joel (2009) provides a succinct description of the three most common types of touch screen technologies present in today‟s mobile devices, namely: resistive, capacitive, and infrared.

Resistive touch screens are the most common type used in mobile devices as they are cheap to make and relatively durable but not scratch resistant. They consist of two conductive layers separated by a spacer and backlit. When an object (any object) 50 presses the conductive layers together, a controller records the position of the touch event. Therefore a key characteristic is the need for pressure to trigger a touch event and two primary weaknesses are the inability to register multi-touch events and scratching will cause reduced sensitivity (Joel, 2009).

According to Joel (2009), capacitive touch screens are more expensive and can either support multi-touch or not. Voltage is applied to the corners of the touch screen and any touch by an object that emits electricity (i.e. a finger) will register to a controller as a voltage drop so that the co-ordinates can be registered. This type of touch screen has some advantages such as scratch resistance, more light transfer, multi-touch, and almost 10 times the average lifespan compared to resistive screens. However, a touch cannot be done with just any object. The iPhone and iPad are good examples of capacitive touch screens.

Infrared touch screens, though expensive, are very responsive as they use either optics or heat to sense touch events (Joel, 2009). The optical type uses a grid of light beams that register a touch when the beam is broken by an object. Because of this, these types suffer from unresponsiveness in high light environments. Heat based infrared touch screens register an increase in temperature as a touch event but therefore suffer in cold environments.

It is important to understand the usage of touch screens in mobile devices as these will be shown to be the preferred interface among recent and advanced smart phones and tablets.

51 Mobile Operating Systems, Manufacturers, and Internet Browsers

WhyTheLuckyMobile (2010) provides an excellent visual depiction of the core mobile operating systems (Figure 3), along with second quarter 2009 market share, compatible internet browser, whether it is open or closed source, whether it is capable of multi-tasking, and what type of interface each o/s supports. Currently, all of the major operating systems use a multi-touch interface and support the new multi-touch UI techniques discussed in Chapter II. The remaining paragraphs will highlight important distinctions for each of these core operating systems in their current release. Note however that other systems do exist, such as Qualcomm BREW, SavaJE, , and

MonteVista, but they will not be discussed due to their small market share.

Figure 3: Quick Visual Overview of Mobile Operating Systems

(whytheluckymobile, 2010) .Reproduced with permission 52 iOS 4

Considered by Apple to be “the world‟s most advanced ”, iOS 4 is built on Apple‟s OSx operating system but has been developed to use “the most natural pointing device ever: your finger” (Apple, 2011a). Apple also states that their interface has advanced interaction techniques employed such as “tap, drag, swipe, pinch, flick, or twist of your fingers”, and in my own usage of both the iPhone 4 and iPad, I have seen applications such as GestureDial that demonstrate industry research is making an impact in this leading smart phone. Apple likes to keep their interfaces simple and a good example of this is the home screen that uses icons to represent applications. Recent upgrades to this operating system have seen it become multi-task capable which is important as the device manages many capabilities for the user like iPod functions, video calling, etc.

In terms of the screen friendly advancements, Apple has invested a lot of time into making the phone accessible to even disabled users using speech recognition features, the ability for the phone to read screen text (multi-modal interaction via

VoiceOver screen-reading technology) and little advancements like built in light sensors that adjust the backlight to promote better viewing in mobile situations. Users can tilt the iOS devices and/or lock the orientation for better reading layout and sensors like

GPS, accelerometers, and cameras are being used as input devices to overcome the barrier of input and output on the smaller mobile screens. For instance, the RedLaser application uses built in cameras to scan bar codes of products and search for the best price and additional product information without the user having to manually enter query

53 information using the screen. As the apps store currently contains over 300,000 applications, users are given many choices for interfaces and features from everything to calculators to web browser and more.

One major drawback to this leading mobile o/s is Apple‟s decision to not support

Flash. As many web sites use Flash to provide a rich, interactive experience, Apple is hoping that the new HTML5 standard will phase out the use of Flash in favour of more stable and energy efficient media presentation (Apple, 2011e). Another notable observation is the close integration of Apple hardware and software to work together and provide the best possible user experience. Myslewski (2011) identifies this trend as dangerous because Apple is giving itself the ability to decide which applications are good enough which circumvents the traditional competition-based “let the best product win” free marketplace. Myslewski (2011) believes this will stifle innovation, influence fairness, and allow manipulation of the market.

Android 3.2 (Gingerbread)

Google offers an open-source operating system that can be used on any device by any manufacturer that wishes to use it. It is also capable of being a multi-touch interface and the current version has many features meant to improve the interface.

Android also has an called Android Market.

Android (2011) has developed assistive features in their latest version as follows:

 Easier to navigate and control,

54  Faster (especially for text input), with an improved predictive / suggestive text

feature,

 Larger type for features like the dictionary,

 Use of Near-Field Communications (NFC) that use an electronic tag read

wirelessly to use the tag data to perform an action like finding related internet

based information,

 Faster 3D graphics drivers, and

 Support for new types of sensors such as rotation, gravity, and barometric.

Windows 7 Phone

Microsoft was a bit late to market with its Windows 7 mobile operating system however it demonstrates some very original techniques. Live

Tiles for instance allows real time interaction from the home screen where customizable tiles display application data in a highly visual and interactive format. Being another multi-touch environment, these “glance and go” features provide users with relevant information (or most used applications) “all wirelessly refreshed via the web” (Microsoft

Corporation, 2011a). The Windows phone application store is called Marketplace.

Symbian 3

Nokia (2010a) released the following broad statement concerning the Symbian operating system:

“More people in the world use a Symbian than any other. The Symbian platform, uniquely designed with in mind, offers a host of experiences that consumers demand today including multiple home screens, gesture interaction and visual multitasking. With the flexibility to scale and

55 extend smartphone features to lower price points, the Symbian platform is well positioned for continued growth.”

Savov (2010) commented on a very thorough review of Symbian 3 that was performed by Eldar Murtazin. He stated that Murtazin found the o/s to: have a “weak spot” web browser, to be not very innovative, and that it does not bring any new desirable features to fruition. However, this multi-touch platform is found on the most consumer devices (as high as 50%) and does offer some interesting abilities such as online widgets that mimic a desktop interface.

MeeGo

MeeGo is one of the latest mobile operating systems. MeeGo (2011) stated that it is a joint effort between Intel and Nokia to create an open source platform based on

Linux that will run on many types of devices. Also a touch screen based system,

MeeGo is able to support applications published to the Ovi App Store run by Nokia but also allows for customized interface creation by operators that adopt it for their devices.

Being a hybrid of Nokia‟s and Intel‟s projects, a key benefit cited is the ability for the Linux-based operating system is tuning for small devices that lets it boot on an Intel powered device in mere seconds (Haddad, 2010).

Palm WebOS 2.0

Palm (2010) announced its latest mobile version as “the most significant update to webOS ever”. They stated that the new OS is unique as it layers content, has superior support for type, supports Flash, and integrates well with common software such as Yahoo! IM, Facebook, Quickoffice, and Skype. Palm‟s use of layering 56 demonstrates that manufacturers are competing heavily using the visual interface as many devices appear as just large glass screens when compared side by side.

BlackBerry 6

The latest Blackberry O/S is an attempt to catch up to the competition since previous versions were not multi-touch. This version supports tabbed internet browsing, has better universal search, a simpler multi-tasking interface, and support for multi- touch screens (Blackberry, 2011). Sharp (2010) reports that Blackberry is well aware that they must catch up to the competition and has even bought a Swedish firm to help improve their screen design and functionality for devices that use Blackberry 6. A new web browser called WebKit is also being used (named in honor of the company).

SURVEY OF PRESENT MOBILE HARDWARE INTERFACES

Mobile / Smart Phones

This section is a survey of six mobile phones of different types observed in the present mobile market. These flagship products include the: Apple iPhone 4, RIM

Blackberry Torch 9800, Nokia N8, Windows Phone HTC 7 Surround, and Google‟s

Motorola Milestone. The LGTU750 is included as an example of a typical non-smart phone running a Symbian operating system with a WAP browser. The focus of this section is to demonstrate how manufacturers are overcoming limitations of the small screen beyond the inclusion of a touch screen natural interface by discussing screen user interface techniques, the physical components of the screen, and usage of input and output methods that support the small screen by minimizing or simplifying user

57 interaction. The discussion begins with the leading iPhone 4 and incremental differences and distinctive features are discussed for brevity.

Apple iPhone4

As previously mentioned, the Apple iPhone started the mass movement towards touch screen interfaces in 2007 and is considered the leader that everyone is trying to compete with. The screen interface on the iPhone 4 uses various techniques such as offering a:

 Tilting widescreen multi-touch format, with

 Tap to zoom, pinch, swipe, scroll, etc. features built into Safari and other apps, a

 Tap to focus camera,

based apps on the home screen that can be organized into folders, and a

 Home button, where double taps allow multi-tasking or application switching

The physical screen is an innovation leader, according to Apple (2011c), that offers a:

 Retina Display where “pixel density is so high that the human eye is unable to

distinguish individual pixels. Which means Text in books, web pages, and email

is crisp at any size” (Apple, 2011b), a

 widescreen multi-touch display, a

 960 x 640 resolution at 326 ppi (pixels per inch), an

 800:1 contrast ratio, a

 500 cd/m2 max brightness, and a

58  Hard glass screen that uses an oleophobic coating for fingerprint resistance.

Input techniques used to help simplify reliance on the screen involve use of sensors, GPS, the camera, and ability to pair blue tooth devices such as a keyboard for those that do not like the built in virtual keyboard. Sensors include a three-axis gyro, accelerometer, proximity sensor, ambient light sensor (adjusts screen brightness based on conditions), assisted GPS with a digital compass, a virtual keyboard (which can be tilted for landscape upsizing), and a 5 mega pixel back camera and a front facing camera for FaceTime (Apple, 2011c).

Output techniques include ability to use external displays via various cables.

Resolutions include 1024 x 768 (with VGA Adapter), 576p and 480p (with Component

AV Cable), and 576i and 480i (with Composite AV Cable). The screen can also display most common file types, languages, keyboard layouts, and dictionary languages (Apple,

2011c).

RIM - Blackberry Torch 9800

As mentioned, Research in Motion (2011a) has just begun to compete with touch screen interfaces and has even bought a Swedish company to improve their screens.

Their flagship product, the Blackberry Torch 9800, is very similar to the iPhone except that it has a slide out physical keyboard and smaller 3.2” HVGA+ 480 x 360 ppi color display (almost half the resolution of the iPhone 4 screen). Beneath the display are menu and back buttons to assist in navigation. A nice feature is the ability for users to select font size. Similar to the iPhone, the home page also uses icon based

59 applications. Concerning the screen interfaces, traditionally complex menus have been simplified with common setup and customization tasks being found on a single page.

One review concludes that the Torch is a major upgrade to other Blackberry devices but that it lags behind significantly in terms of screen resolution, processor, and camera capabilities (Baker, 2010). Overall, Topolsky (2010) supports this claim and views this phone as a great attempt to clean up past hardware and operating system deficiencies but thinks that RIM is falling behind most smart phone manufacturers particularly in its low resolution screen – possibly due to backward compatibility issues with existing

Blackberry applications.

Nokia N8

Nokia (2010b) recently launched the N8 smart phone that globally introduces the

Symbian 3 multi-touch operating system with a 3.5 inch capacitive touch (wide) screen.

It has a rather low 640 x 360 pixel resolution but Nokia has created a multi-media marvel that connects via HDMI to external displays, integrates a 12 megapixel camera for 720p HD-recording quality, and offers Dolby Digital Plus surround sound.

Of note, Symbian 3 competes with Apple with its new touch interface that “has most of the features that iPhone 4 has (I was comparing these two before I purchased

N8): social networking, touch screen, internet connection, apps, business tools” (Queen,

2010). It is important to see how the world‟s most distributed mobile operating system is matching the iPhone in capability and distinguishing itself based on features like offering a mobile “theatre” experience – or more simply the improvement of output to external displays to overcome limitations of the smaller mobile one. 60 Microsoft / Windows 7 Phone – HTC 7 Surround

Like the N8, HTC Corporation (2011) has created a “pop up cinema; a phone with a richer listening and viewing experience” by merging the Window 7 Phone operating system with a 3.8 inch touch screen with a 480 x 800 resolution display.

However, Surur (2010) critiques this phone extensively and points out how manufacturers are rushing new products to market hoping that it is what consumers are looking for. He points out that though the touch Windows 7 Phone o/s is nice - there are inherent design flaws in this phone. First, a kickstand displays the phone in landscape mode yet most navigational tasks are locked in portrait view. Secondly, the bulky speaker that advertises surround sound capability seems muffled and tinny – and when the device speaker is closed the sound is extremely muffled. And finally, using the phone for activities beyond multi-media viewing does not compete with other phones.

Google / Android – Motorola Milestone 2

Motorola Mobility (2011) is currently getting a new improved second version of their Milestone product ready for market. The TFT screen is 3.7 inch with a 480 x 854 resolution display (FWVGA) and is coated to reduce glare. It runs the Android 2.2 O/S with a (ZUI) and the ability to multi-task applications. To compete with the iPhone, it has full Adobe Flash Player 10.1 support optimized to run on a 1 GHz processor. Device improvements include a redesigned sliding keyboard with predictive text SWYPE feature that allows users to trace words without lifting a finger. Though, a virtual keyboard is also available. It has an improved 5 MP camera

61 with a dual LED flash. Expected sensors are present including: proximity, light, and compass.

The interface includes Widgets (customizable sub-screens found on the home page) that are both resizable and filterable. These widgets can be added to seven different home screens. The device has dedicated navigation keys for common tasks and has a voice to text feature and voice recognition to help users input information.

Motorola makes device-to-device communication easy with built-in Bluetooth and

DLNA (Digital Living Network Alliance) support. DLNA devices can intra-connect with one another if they share a local network (Digital Living Network Alliance, 2011) which makes external displays like big screen TVs useful for streaming content to. This smart phone can also service as a mobile hotspot for up to five devices.

LGTU750 (LG Secret)

Manufacturers, like LG Electronics (2011), provide intermediate cell phone solutions as not everyone wants to buy a $500+ smart phone with an expensive monthly data plan. Therefore, traditional WAP-enabled cell phones like the LGTU750 supply only the basics in connectivity. Though it has a 2.4 inch 240 x 320 TFT capacitive touch screen and a sliding numeric keypad, it is suitable for casual mobile web inquiries over

WAP 2.0 and sending SMS text (Short Message Service) and MMS (Multimedia

Messaging Service) picture based messages. According to GSMArena (2011), it is also capable of using Java 2.0 to play both pre-loaded and downloadable java-based games, and has a document viewer capable of reading common Word, Excel, PowerPoint, and

62 PDF formats. Surprisingly, this phone also uses an accelerometer to rotate the display, has a 5 MP camera with face detection that can output shots / movies to a TV, and can use Bluetooth to share documents between other devices.

Tablet Devices

Another prominent trend in overcoming the small screen on mobile devices is to make the screens larger. This is seen on today‟s slate and tablet devices which, as mentioned, are a large part of this year‟s CES show. This section briefly introduces three of these devices including the: Apple iPad, Samsung Galaxy , and Blackberry

Playbook (though many more exist).

Apple iPad

Apple recently launched a tablet manufacturing race with the iPad. Screen highlights according to Apple (2011d) include a:

 9.7” 1024 x 768 resolution LED-backlit multi-touch screen with 132 PPI

(Pixels per inch),

 An uncommon aspect ratio of 4:3 where most others are 16:10 or 16:9,

 A viewing angle of 178 degrees and 8 bit color due to IPS Screen Technology

(In-Plane Switching) where TN (Twisted Nematic) displays are commonly less

and offer 6 bit color (Brandrick, 2010),

 Oleophobic coating (same as iPhone 4),

 Accelerometer (tilting) and light sensor (auto adjusting) -- but no cameras,

 Same text to voice as iPhone 4, 63  TV output at 1024 x 768 pixels (VGA cable), 576p x 480p (Component AV cable),

576i x 480i (Composite AV cable), and

 Up to 720p video output (High Definition) with varying resolutions by file type.

So, why not build a retina display into the iPad? Kendrick (2010) points out that cost and manufacturing problems are likely the reason as a portion of these screens fail during construction which would compound with a larger screen size. Rumors are that the iPad 2 will also not incorporate a retina display but instead use the existing one.

Samsung Galaxy Tab

Samsung (2010a) released a tablet shortly after the iPad that targeted perceived deficiencies in Apple‟s product. Specifically, Samsung chose to use the Android 2.2

O/S (Google) that supports Flash 10.1 – and markets their tablet as “supporting any file type”. Secondly, the Galaxy Tab uses only a 7 inch touch screen that is marketed as being one-handed and more mobile than the iPad. Finally, a rear and front camera allows the tablet to be just as capable as the iPhone for increased user interaction. The screen, in particular, is a 7 inch TFT LCD touch screen display that offers a 1024 x 600 resolution (Samsung, 2010b).

CNET (2011) describes the Galaxy tab as lighter than the iPad and able to fit into most pockets. They comment that the pixels-per-inch is higher due to the smaller screen size even though the resolution is lower than the iPad. It differs with built in navigation buttons and the small size makes thumb typing easier on the virtual keyboard. Of note, the keyboard has a SWYPE feature which allows typing without

64 lifting the finger (similar to a connect-the-dots exercise). The Google app Market is also competitive. In conclusion though, CNET ranks the iPad as a net book alternative and the Galaxy Tab as an alternative for the smart phone or e-Reader.

Blackberry Playbook

Research in Motion (2011b) is also soon to release the Blackberry Playbook, which also has a 7 inch LCD 1024 x 600 resolution capacitive multi-touch display. Also able to support Flash 10.1 with support for HTML5, this tablet distinguishes itself with integration to Blackberry Enterprise server and secure communication with other

Blackberry smart phones. RIM has promised a 4G capable tablet which is emphasized to be secure and business oriented. Video output is listed at 1080p high definition.

Miller (2010) compares the Playbook to the iPad and Galaxy Tab. The higher

170 PPI is noted with the 3MP front and 5MP camera (higher than the Galaxy Tab‟s 1.3 and 3 MP camera respectively). However, all other core features are closely competitive leaving a market ripe to compete on peculiarities.

Projectors

In 2008, Pico introduced miniature projector technology at the Consumer

Electronics Show in Las Vegas. Then, CES 2009 saw Samsung integrate one into a mobile phone (GSMArena, 2009). Now some are hailing 2010 as “the year of cell phone projectors” as more manufacturers such as LG release their versions of this innovation and rumors of possible integration of projectors into the Apple iPhone as early as 2011 Q2 (Darrell, 2010). 65 Certainly this is a trend though as seen in mobile laptop computing. For those unwilling to wait for built in models, McDonough (2010) introduces four of the latest mobile projectors as discussed below. In fact, the only thing needed is a compatible cable, such as miniUSB, DVI, or HDMI, or an iPhone/iPad connection that outputs VGA

(available in ). However, be aware that many mobile manufacturers restrict the output to certain file types. For instance, viewing the projected web on an iPhone 4 would not be possible but viewing pictures, videos, Netflix, and other mobile content would be possible.

 AAXA P2 ($299) - 80” display with 33 lumens of LED light, weighs only 0.6

pounds, supports most popular file types

 Optoma PK301 ($399) - 120” display with 50 lumens of light, 0.5 pounds, can

output 480i, 480p, and 1080i resolution

 Samsung SP-H03 ($299) - 2 hour battery, 1080i and 720p resolution, 0.4 pounds

 Beambox Evolution R-2 ($363) - 1 GB memory, 40 lumens

E-Readers

Bradford (2010) explains that mobile handheld e-readers are popular due to their lower cost, size, and use of e-Ink screens that promote better reading. These screens have no glare outside leaving less eye strain and use power so efficiently that a single charge can last for weeks. And, e-readers are usually in the 5 to 6 inch range making them ultra portable for their purpose. They are an important niche hardware innovation that uses the right tool for the job – a screen that was meant to be portable and mobile for extended reading of text. 66 Micro Displays & Wireless Transmission of Video

And finally, another observation is the extension of mobile information to external screens. Advances in the ability to wirelessly transmit video have started to see products and services enter the market that extend the screen outwards to other devices.

Sony Ericsson‟s LiveView is a good example of this where a small watch sized cube display is able to connect via Blue Tooth to a mobile device to display social media updates, phone information, and other data from the wrist (Sony Ericsson, 2010).

On the other hand, some mobile displays can also be used as external monitors for both mobile computers and desktop setups. For instance, DisplayLink (2010) permits users to download driver software onto their PCs while installing a matching driver application on their iPad. The iPad simply logs into any enabled PC on the same network and suddenly becomes another external monitor for that system. Very innovative!

AppleTV& AirPlay

Apple (2011f) has recently released the second generation of AppleTV which primarily offers the ability to share digitial content organized in the user‟s iTunes library to traditional media consumption devices like large screen TVs and music audio systems. AppleTV also offers a new rental model for high-definition content such as the latest movies and television shows.

67 Apple (2011g) introduces AirPlay as a means to transfer content ubiquitously and in a multi-modal manner between devices. For instance, most digital content that can be viewed or heard on both the iPhone and iPad can now seamlessly be transferred to more capable devices such as the user‟s large screen TV. The small screen device is then capable of acting like a remote control to both control AppleTV directly or to control the playable content such as movies or songs.

SURVEY OF PRESENT MOBILE SOFTWARE INTERFACES

Mobile Application Development Overview

Mobile application stores operate as both open source and restrictive “walled gardens”. Any developer can develop for an app store. Development is limited to the tools and interface techniques available to the operating system being used. The following sections list some observable developments concerning mobile software.

Distribution - Application (App) Stores and UMI

App stores are growing as demonstrated by Apple‟s catch phrase “There‟s an

App for that”. Currently businesses all across the world and in all industries are developing mobile applications for competitive advantage – for instance .

Some businesses are creating their own app stores for platforms that support multiple app stores like Android (Perez, 2008).

Vendel (2010) stated that Universal Mobile Interface (UMI) software is an alternative to app distribution with some inherent advantages. First, multiple versions of apps would not need to be created as a single UMI implementation would cover all 68 mobile devices regardless of device or operating system. As well, UMI websites would not need to be downloaded or installed and therefore could be shared immediately between businesses and consumers.

Productivity Software – Major Software Companies

Larger software manufacturers are releasing mobile versions of major consumer software. For instance, products such as Microsoft Office have mobile versions that integrate with other Microsoft Products such as SharePoint Server. This makes sharing business documents very easy as they can be ported across servers and devices seamlessly (Microsoft Corporation, 2011b). However, some software, like Office

Mobile 2010, only works with their parent operating systems.

Third party software helps to get around these limitations. QuickOffice (2011) provides document viewing and editing for commonly used file types across every platform (except Windows 7 Phone, of course). There is some duplication though.

QuickOffice provides a mobile PDF viewer - but so does Adobe – whose version is likely more feature-rich. For instance, Adobe uses the accelerometer to re-orient the document and document flow when devices are re-oriented (Adobe Systems

Incorporated, 2011). Flash, another of Adobe‟s core browser technologies, has been pushed to prove itself as a viable mobile product by Steve Jobs who refuses to use it in

Apple mobile products due to concerns over “stability, security, resource usage and whatnot” and has therefore released Flash 10.1 to all major device makers (Wauters,

2010).

69 Custom Interfaces

Application store development has also created customizable interfaces. If a

Facebook user does not like the current Facebook webpage, a new interface can be downloaded such as the Flipboard app which puts Facebook content into a magazine format. Devices are also allowing interface customization on their home screens such as Live Tiles in Windows 7 Phone or Widgets in the Android O/S. There are also multiple apps for the same thing. Users that do not like the build in weather app on the iPhone can simply download a better one, with more information, such as WeatherEye

(from the Weather Channel).

WAP Development

Implementation of common software is also available over the mobile web (or

WAP) and includes websites for areas like social networking, electronic media, news, photos, utilities, etc. (Dahiya, 2010).

Development for Multiple or Niche Devices

Other devices are integrating mobile software into them. Digital picture frames are now capable of advanced tasks such as displaying photos or videos off a user‟s

Facebook account or playing internet radio. The downside is some of them are as expensive as an average netbook. However, feature wise these devices can be

70 “lukewarm” and have more potential than what they currently are capable of (Bell,

2010).

Another example would be Kindle‟s development of a reader app for the iPad…instead of solely using their own devices to host their software (Gary, 2010). In this case, it is obviously a push to expand their market share though any means available, including making their software capable of running on competing devices.

Increased Reliance on Internal and External Device Components for Input and Output

Applications are using all available hardware components to minimize reliance on the screen for input and output. For instance, the redLaser application uses the camera to scan barcodes to find product prices while the MobileTag app scans barcodes or tags to get product information. External display over Bluetooth is now a reality using

DisplayLink app-based software, as well as using Bluetooth external keyboards.

External output to existing displays is also becoming more common. Netflix recently released software to allow users with smart phones to stream movies to their mobile devices which can then output them to any TV or monitor with the correct cable.

Observed Usage of Multi-touch techniques

Multi-touch Gestures

The various mobile operating systems are now multi-touch capable and the highly capable smart phone “flag ship” products use gesture based interaction as proposed by researchers. Specifically, it is not uncommon for mobile applications to:

71 Swipe, Flick, Pinch, Tap, Tilt, Shake, Scroll, Pan, or Bump to manipulate the display and exchange of information. Some manufacturers have improved on these interaction techniques in order to be competitive. Apple created and patented „inertial scrolling‟ which is a swipe-to-scroll gesture, considered to be critically important to Apple‟s competitiveness (Aimonetti, 2010).

Navigation

Advances have also been made in navigation which assists the user overcome limitations of the small mobile screen. Most mobile operating systems employ the concept of home screens and icons that launch applications. This is important as this arrangement assists users select items with large fingers. Some phones do continue to build hardware buttons to assist with common navigation tasks however. Voice control is another method that most phones employ which decreases reliance on input and interaction through the screen. And finally, universal search abilities allow users to type what they are looking for to get a listing of available matching software. Again, this simplifies input and selection.

Small Screen Design Considerations

The iPhone calculator app is simple in portrait mode but changes to a scientific format when tilted. This is a good example of design considerations when Nokia (2007) discussed how to overcome the small screen while developing mobile games. Nokia emphasizes the need to use “the judo rule” which is turning weaknesses into strengths.

Their best practices included: making sure key elements had sufficient contrast to

72 overcome outdoor light, being very selective on what to display, using the smallness factor to increase challenge (perhaps using memory for off-screen objects), and, like the calculator example, using landscape orientation where appropriate.

Mobile Browsers

As previously mentioned, mobile browsers are closely tied to their mobile operating system. They can also be implemented quite differently as shown by compatibility testing performed by Quirksmode (n.d.) and can be proprietary variations of the same layout engine (“Grigsby, 2010). Grigsby (2010) also stated that while the volume of operating systems is increasing, modern mobile browsers are using Webkit engines and most are geared towards running HTML5. This is significant as Webkit could be used in as much as 85% of the smart phones sold.

73 CHAPTER IV

MOBILE INTERFACE ISSUES, CHALLENGES, AND TRENDS

TRENDS

A research mapping has been created, in Appendix A, to help identify the issues, challenges, and trends (listed below) using the Context of Mobile Interaction framework as proposed by Savio and Braiterman (2007). This mapping organizes the research presented in this essay in terms of the overall context of mobile interaction, specifically: broad interaction issues, interface-related research, visualization techniques, and physical device improvements architected by manufacturers. Appendix A can therefore be used as a starting point to look at a research area further and highlights the fact that research is occurring at many levels surrounding the issue of overcoming limitations of the small screen.

Interface Trends

The touch screen and zooming interface have been adopted as the preferred interface method in today‟s mobile market. Closely related to this preference for a is the proliferation of gesture-based interaction and multi-touch displays within these touch screen devices.

There is also evidence that researchers have updated their direction away from past “make-do” interface techniques (like the many mobile web migration strategies) toward more powerful and elegant techniques that offer mobile data consumers an

74 experience closer to that of a desktop experience. This has been accomplished by continuing critical (but updated) research into common PC issues like interface navigation and resource searching in ways that simplify a mobile users experience

(such as tilting or flicking).

Newer interface techniques are also important and welcomed however. Today‟s mobile devices are more widely used in the world than the common personal computer but are becoming outlets for different sorts of interaction such as their use in multi-user environments. The market demand for more advanced devices has put pressure on researchers to deliver complex interaction scenarios where input and output methods must be more precise, more readily available, and more advanced to lessen the greater demands brought out by simply being mobile. As such, screens are becoming more advanced in these areas as seen by novel and extremely recent research in areas like multi-modal interaction. Unfortunately, the mobile home screen remains relatively simplistic and icon-based however manufacturers are beginning to compete on improved home screen interfaces with novel features like widgets and LiveTiles.

The distribution of these interfaces is also critical. Even the W3C regulatory body has suggested guidelines to promote consumer choice in interface. Hence the need for distribution methods like abundant application stores and advanced distribution channels like UMI.

Information Visualization Trends

There are also observable trends in the visual presentation of the interface.

Older techniques such as Fisheye are innovating new techniques like the scatter plot,

75 but are also still being used in today‟s mobile interfaces as seen by the direct use of the fisheye in Apple‟s mobile Safari Webkit-based internet browser. This trend is not surprising as mobile interfaces like browsers are starting to become as feature rich as

PC browsers while trying to provide the “real web” to users – though because of this, older WAP innovations are seemingly being abandoned and possibly border on being obsolete except for use in the weakest non-smart phone mobile browsers (as seen in the LG Secret).

Researchers are seen to be pushing the boundaries of the small screen to present information as seen in the introduction of off-screen areas and screen-less interfaces. Some research has been adopted and implemented by manufacturers (such as the multi-touch or zooming interface) however manufacturers are presenting simple home screen UIs that do not resemble or have the capability of the complex personal computer interfaces. Currently, complex mobile visualizations are typically found in mobile apps and users are given the choice to accept standard implementations of common software (i.e. Facebook) or to download variations that cater to preference.

Device Hardware and Software Trends

A major focus of addressing the limitations of the small screen is happening at the device level. As demonstrated in Appendix A, both researchers and manufacturers are investigating methods to overcome the constraints of small mobile screens such as:

1) improving it, 2) increasing its size, 3) decoupling it, 4) creating specialized types, 5) extending it, and 6) decreasing the user‟s reliance, 7) entering into new markets, 8) removing the screen, 9) evolving it, and 10) integrating it into computing environments. 76 Improving the Screen

The research shows that screen improvements are occurring with experimentation in optimal size and aspect ratio, increasing resolution, colors, and processing power. Hardware advancements are also allowing for better viewing angles, are relying on sensors to adjust screen options like tilt angle and backlight levels. Even mainstream advancements like 3D are finding their way onto the small screen.

Increase Screen Size

One to five inch cell phone displays are the norm but manufacturers have also begun to offer seven to ten inch tablet displays. However, tablets bring a new class of mobility product that sacrifices mobility for increased functionality. Size is even a competitive feature as seen in the seven inch Samsung Galaxy Tab that claims to be one-handed.

Specialized Screens

E-ink displays on e-readers are a good example of product specialization as e- ink is more readable in high light environments.

Decouple Screen

Researchers are proposing wearable screens like the LiveView watch or

Spectacles sunglasses that separate the screen from the device in order to increase mobility. Even ubiquitous screens such as Facebook picture frames or Bluetooth

77 enabled tablet monitors are separating the screen from the mobile devices that utilize them.

Extend Screen

The screen is also being extended using projection technology and output to external monitors (i.e. 1080p TV output or using public displays like MS Surface).

Decrease Screen Reliance

Sensors are being used for easier mobile input and output that decreases reliance on the screen. For instance, built-in cameras or RFID sensors are being used by some apps to scan tags to lookup information automatically or take other complex action. Manufacturers are also using multi-modal techniques such as Apple‟s

VoiceOver (text to voice) or speech controlled interfaces with reportedly good user acceptance results. Even decoupled peripherals (such as external keyboards via

Bluetooth) are helping to improve the mobile experience.

Entry into Non-related PC Areas

Manufacturers are experimenting to see if the mobile phone can replace non-PC devices like the media centre. Two good examples are the Nokia N8 theatre-in-a-phone that outputs 1080p and the HTC 7 Surround with a built in multi-media speaker.

78 Remove Screen

Researchers are even creating screen-less interfaces that monitor user activity with foot-gestures, or eyes free (imaginary), or subtle/intimate interfaces based on muscle contraction.

Evolve / Augment Screen

On-going device improvements are also happening such as see-through

(LucidTouch) or two-sided screens, proposals of advanced tactile feedback interfaces

(TeslaTouch, T-Bars), wearable, and pressure sensitive devices. Research into is also realizing concepts like off-screen areas and peep-hole displays.

Integrate Screen

And finally, the small mobile screen is also being integrated into existing computing environments. Examples of this include the TouchProjector and integration with public displays like playing Poker using the mobile for privacy and MS Surface for publically displayed cards.

ISSUES AND CHALLENGES

Designing within the Context of Mobile Interaction

Broadly, researchers are aware of various aspects of mobility while designing for mobile interaction. Mobility is seen to bring special challenges not present in stationary computing such as the home personal computer. Research in Appendix A shows that 79 mobile users are clearly different in their goals, attention level, and performed tasks than a stationary user and face unique challenges brought about by the culture in which they use their devices, their daily activity requirements, and the physical environment that often times they have no control over.

Designing within the context of mobile interaction involves addressing human factors such as poor eye sight, finger occlusion, age, culture, appropriateness, attention-level, and understanding of user goals and tasks. As well, environmental factors like low light, distraction, or privacy issues are important considerations. And finally, being aware of the correlation between device size and mobility is an important consideration that researchers must be aware of.

Developer Support

Developers are critical stakeholders in creating mobile user interfaces and therefore need support in various forms. Over the last decade, researchers have identified a need (and offered guidelines) for a visualization framework. Currently, standards and suggestions offered by the W3C concerning mobile interface design seem vague and non-inclusive. Again, researchers have offered suggestions such as the Universal Mobile Interface (UMI) to simplify designs and get around the need to code multiple versions of things for different devices. Standards like DLNA are shown to offer uniformity across device platforms.

80 Market Challenges

The mobile market is very competitive and relatively new. Visionaries like Steve

Jobs are needed to adopt, or try, the excellent research that aims to address the visualization problem with small screens. Currently there is slow movement towards defining the mobile device as a desktop-replacement, even though consumers have strongly demanded a better experience with the mobile web. Disconnection between research potential and current market adoption is certainly a core challenge.

Device Challenges

The current standard, multi-touch gesture-based mobile computing, has many core challenges. Input and output issues like finger occlusion, camera input, keyboard type and optimization methods combined with the ongoing need for better data visualization output are shown as special challenges for researchers. Navigation is also a core challenge that is inspiring the unique and innovative interaction techniques shown in the research. A high profile challenge lately is defining the optimal size for mobility as different segments such as the tablet and the smart phone are becoming more visible.

Manufacturer Issues

Vendel (2009) states that there are two approaches to providing mobile products and services – namely, some manufactures take a vertical approach to their products where they control absolutely everything. While this creates very stable platforms, critics view this “walled garden” approach as stifling to innovation and market 81 competitiveness. A great example is Apple‟s refusal to use Flash in their mobile platform. Mobile interfaces also relatively simple compared to desktop interfaces which can be seen in the prevalent icon-based home screen. And lastly, expense is a concern as shown by Apple‟s inability to move their retina display from the iPhone to the iPad.

Consumer Perception

A key barrier to overcoming the small screen is the general perception that mobile devices are auxiliary to the PC. Users must tether their mobile devices to their more powerful systems as current mobile capabilities are not yet powerful enough to replace the PC. A change in this perception would alter the direction and pace of manufacturers and researchers even further because the mobile industry is very much market driven.

82 CHAPTER V

CONCLUSIONS AND RECOMMENDATIONS

SUMMARY

Both researchers and manufacturers are overcoming the problem of small screens from many different angles using both new and old interaction, interface, and visualization techniques that work well with ongoing improvements to display and ability of the mobile devices surveyed. However, though the problem of displaying information on small and very small screens is not new, the recent release of capacitive touch screen technology by the iPhone in 2007 has set a new direction for both ongoing research and device improvements that promises to move the massive cell phone user base into an era of personal computing that mimics the capabilities of the traditional desktop computer. Examples of this work are highlighted in this essay and a strong match between ongoing research and current market capabilities is made to the various elements presented in a high-level context of mobile interaction framework.

It has been shown that small screens on mobile tools definitely put special demands on HCI for the design of mobile interfaces as difficult constraints must be addressed to enable a quickly growing mobile workforce. For instance, one to five inch screens are currently considered to be ultra portable as they may be easily slipped into a pocket or purse or worn (i.e. on the wrist) but common complaints include difficulty in viewing, manipulating, inputting, and outputting information. While manufacturers are

83 currently experimenting with simplistic improvements like increasing screen size (like the 3.5” iPhone 4 with the exact same software capabilities as the 9.7” iPad tablet), researchers are proposing advanced interaction, interface and visualization techniques that aim to keep devices ultra portable (such as peephole displays that allow work on large virtual canvases). This mismatch between the capabilities of the current mobile market and the potential shown by cutting edge research could easily be attributed to the fast paced and highly competitive environment that permeates the current mobile market.

The world is on the verge of a new computational platform where success largely depends on solving the problem of computing with small screens. And, we all stand to benefit. So yes, the problem can be fixed as small screen constraints are being addressed. New techniques are shown to be both novel and promising, and are building on past research with great success.

But, are new standards needed? This age old question of Innovation versus uniformity is almost rhetorical as consumers want more, and researchers and manufacturers would like to give it to them. For now, the solution is based on a clear direction - gesture-based touch screen interfaces. Steve Jobs recently commented that we are in a “post PC” world, however, in terms of solving the small screen problem we are in a “post iPhone” world where ongoing research and device improvements are moving toward creating a truly mobile PC replacement computing platform. In conclusion, the observed trends and direction of future research looks very promising… and just in time for an increasingly mobile world that demands it.

84 RECOMMENDATIONS FOR FURTHER RESEARCH

As demonstrated by the March 3, 2011 unveiling of the iPad 2, the currently leading manufacturer and innovator of touch screen mobile interfaces is still competing on simplistic improvements such as thinness and processing power. Like any industry, market leaders must keep one step ahead of the competition to retain their advantageous market position.

The challenge (and recommendation) to move the mobile phone towards its ultimate potential is to convince manufacturers of the inherent competitive advantages seen in the past, present, and future research that aims to solve the limitations of the small screen. Exactly how to do this is a question that is suitable for a closer look.

As well, creating and maintaining a mobile visualization repository would serve many purposes. Manufacturers and developers would benefit from the increased organization of a comprehensive framework and this would result in more capable mobile devices and an increasingly better consumer experience. And, researchers would be better able to identify deficient areas which would lead to even more novel innovations. And finally, true choices – beyond just multi-touch gesture-based interfaces – would exist for consumers allowing for a less generalized and specific match (or recommendation) to their needs.

85 REFERENCES

Adobe Systems Incorporated (2011). Adobe reader x / features. Retrieved from

http://www.adobe.com/products/reader/mobile-pdf-reader.html

Aimonetti, J. (2010, August 31). Apple wins patent for critically important IOS swipe-to-

scroll gesture. Retrieved from http://reviews.cnet.com/8301-19512_7-20015237-

233.html

Albinsson, P., & Zhai, S. (2003). High precision touch screen interaction. Proceedings of

the Sigchi conference on human factors in computing systems (pp. 105-112).

New York: ACM. 10.1145/642611.642631

Android (2011). Android 2.3 platform highlights. Retrieved from

http://developer.android.com/sdk/android-2.3-highlights.html

Apple (2011a). iOS 4. Retrieved from http://www.apple.com/iphone/ios4/

Apple (2011b). Retina Display. Retrieved from

http://www.apple.com/ca/iphone/features/retina-display.html

Apple (2011c). iPhone 4 Technical Specifications. Retrieved from

http://www.apple.com/ca/iphone/specs.html

Apple (2011d). iPad technical specifications. Retrieved from

http://www.apple.com/ca/ipad/specs/

86 Apple (2011e). Thoughts on flash. Retrieved from

http://www.apple.com/hotnews/thoughts-on-flash/

Apple (2011f). AppleTV. Retrieved from

http://www.apple.com/appletv/

Apple (2011g). iPad AirPlay. Retrieved from

http://www.apple.com/ipad/features/airplay.html

Baker, A. (2010). Blackberry torch 9800 review by aaron. Retrieved from http://

www.phonedog.com/2010/08/11/aaron-s-blackberry-torch-9800-review-at-t/

Bau, O., Poupyrev, I., Israr, A., & Harrison, C. (2010). Teslatouch: electrovibration for

touch surfaces. Proceedings of the 23nd annual ACM symposium on user

interface software and technology (pp. 283-292). New York: ACM.

10.1145/1866029.1866074

Baudisch, P. (2011). Cirriculum vitae patrick baudisch. Retrieved from

http://www.patrickbaudisch.com/resume/ resumePatrickBaudisch.pdf

Baudisch, P., & Holz, C. (2010). My new pc is a mobile phone. XRDS: Crossroads, The

ACM Magazine for Students - The Future of Interaction, 16(4), doi:

10.1145/1764848.1764857

Bell, D. (2010, July 13). Hp dreamscreen 130 (13-inch). Retrieved from

http://reviews.cnet.com/digital-photo-frames/hp-dreamscreen-130-13/4505-

13499_7-33796633.html?tag=mncol;lst 87 Blackberry (2011). Blackberry 6. Retrieved from http://us.blackberry.com/apps-

software/blackberry6/

Bournique, D. (2010, January 4). What's ahead for the mobile web in 2010?

[Web log message]. Retrieved from http://blog.wapreview.com/?p=6011

Bradford, K.T. (2010). eReader mania. Gadget Gift Guide, A

Special Edition of Laptop, XXX(13), 39-40.

Brandrick, C. (2010, February 1). iPad: IPS screen technology explained. Retrieved

from http://www.macworld.com/article/146028/2010/02/inplaneswitching.html

Brewster, S. (2002). Overcoming the lack of screen space on mobile computers.

Personal and Ubiquitous Computing, 6(3), doi: 10.1007/s007790200019

Boring, S., Baur, D., Butz, A., Gustafson, S., & Baudisch, P. (2010). Touch projector:

mobile interaction through video. Proceedings of the 28th international

conference on human factors in computing systems (pp. 2287-2296). New York:

ACM. 10.1145/1753326.1753671

Buchanan, G., Farrant, S., Jones, M., Thimbleby, H., Marsden, G. & Pazzani, M. (2001).

Improving mobile internet usability. Proceedings of the 10th international

conference on world wide web (pp. 673-680). New York: ACM.

10.1145/371920.372181

B¨uring, T., Gerken, J., & Reiterer, H. (2006). User interaction with scatterplots on small

screens - a comparative evaluation of geometric-semantic zoom and fisheye 88 distortion. Unpublished manuscript, Retrieved from http://www.inf.uni-

konstanz.de/cgip/bib/files/BuGeRe06b.pdf

Buxton, B. (2007, January 12). Multi-touch systems that i have known and loved [Web

log message]. Retrieved from http://www.billbuxton.com/multitouchOverview.html

Chittaro, L. (2006). Visualizing information on mobile devices. Computer, 39(3), doi:

10.1109/MC.2006.109

Cho, S., Murray-Smith, R., & Kim, Y. (2007). Multi-context photo browsing on mobile

devices based on tilt dynamics. Proceedings of the 9th international conference

on human computer interaction with mobile devices and services (pp. 190-197).

New York: ACM. 10.1145/1377999.1378006

Costanza, E., Inverso, S. A., & Allen, R. (2005). Toward subtle intimate interfaces for

mobile devices using an EMG controller. Proceedings of the Sigchi conference

on human factors in computing systems (pp. 481-489). New York: ACM.

10.1145/1054972.1055039

CNET (2011). Samsung galaxy tab (sprint). Retrieved from http://reviews.cnet.com/

tablets/samsung-galaxy-tab-sprint/4505-3126_7-34194814.html

Dahiya, H. (2010, September 25). List of wap sites for mobile. Retrieved from

http://www.technoloze.com/2010/09/list-of-wap-sites-for-mobile.html

Darrell, R. (2010, May 1). 2010 – the year of cell phone projectors!. Retrieved from

http://www.bitrebels.com/technology/2010-the-year-of-cell-phone-projectors/ 89 Di Fabbrizio, G., Okken, T., & Wilpon, J. G. (2009). A speech mashup framework for

multimodal mobile services. Proceedings of the 2009 international conference on

multimodal interfaces (pp. 71-78). New York: ACM. 10.1145/1647314.1647329

Digital Living Network Alliance (2011). How it works. Retrieved from

http://www.dlna.org/digital_living/how_it_works/

DisplayLink (2010). iPad software downloads. Retrieved from

http://displaylink.s3.amazonaws.com/index.html

Döring , T., Shirazi, A. S., & Schmidt, A. (2010). Exploring gesture-based interaction

techniques in multi-display environments with mobile phones and a multi-touch

table. Proceedings of the International conference on advanced visual interfaces

(pp. 419-419). New York: ACM. 10.1145/1842993.1843097

Drake, S.D., Boggs, R., & Jaffe, J. (2010, July 29). Worldwide mobile worker population

2009-2013 forecast. Retrieved from http://www.workshifting.com/downloads/

2010/07/29/Worldwide%20Mobile%20Worker%20Population%202009-

2013%20Forecast.pdf

Fallman, D., Lund, A., & Wiberg, M. (2004). Scrollpad: tangible scrolling with mobile

devices. Proceedings of the 37th annual hawaii international conference on

system sciences (pp. 6). Umeå: IEEE 10.1109/HICSS.2004.1265676

Ferscha, A., & Vogl, S. (2010). Wearable displays for everyone! Pervasive Computing,

IEEE, 9(1), doi:10.1109/MPRV.2010.13

90 Francone, J., Bailly, G., Lecolinet, E., Mandran, N., & Nigay, L. (2010). Wavelet menus

on handheld devices: stacking metaphor for novice mode and eyes-free selection

for expert mode. Proceedings of the International conference on advanced visual

interfaces (pp. 173-180). New York: ACM. 10.1145/1842993.1843025

Furnas, G. W. (1986). Generalized fisheye views. Proceedings of the SIGCHI

conference on human factors in computing systems (pp. 16-23). New York: ACM.

10.1145/22339.22342

Gary (2010, March 23). Amazon kindle app for ipad previewed. Retrieved from

http://www.ipadincanada.ca/app- store/amazon-kindle-app-for-ipad-previewed/

Go, K., & Tsurumi, L. (2010). Arranging touch screen software keyboard split-keys

based on contact surface. Proceedings of the 28th of the international

conference extended abstracts on human factors in computing systems (pp.

3805-3810). New York: ACM. 10.1145/1753846.1754060

Gomez (2009). Why the Mobile Web is Disappointing End-Users: A study of

consumer‟s mobile web experiences conducted by Equation Research on behalf

of Gomez, Inc. (White Paper). Retrieved from

http://www.championmobilesolutions.com/uploads/4/8/0/6/4806168/gomez_

mobile_web_experience_survey.pdf

Grigsby, J. (2010, May 17). Mobile operating systems and browsers are headed in

opposite directions. Retrieved from http://radar.oreilly.com/2010/05/mobile-

operating-systems-and-b.html 91 GSMArena (2009, January 12). Samsung's first projector phone sees daylight at ces

2009. Retrieved from http://www.gsmarena.com/samsungs_first_

projector_phone_sees_daylight_at_ces_2009-news-732.php

GSMArena (2011). Lg kf750 secret. Retrieved from

http://www.gsmarena.com/lg_kf750_secret-2352.php

Gustafson, S., Bierwirth, D., & Baudisch, P. (2010). Imaginary interfaces: spatial

interaction with empty hands and without visual feedback. Proceedings of the

23nd annual acm symposium on user interface software and technology (pp. 3-

12). New York: ACM. 10.1145/1866029.1866033

Gutwin, C., & Fedak, C. (2004). Interacting with big interfaces on small screens: a

comparison of fisheye, zoom, and panning techniques. Informally published

manuscript, Department of Computer Science, University of Saskatchewan,

Saskatoon, SK. Retrieved from http://hci.usask.ca/publications/2004/bigui-

gi04.pdf

Haddad, I. (2010, December 1). Introduction to the software platform. Retrieved

from http://www.linuxjournal.com/ article/10797

Hakala, T., Lehikoinen, J., & Aaltonen, A. (2005). Spatial interactive visualization on

small screen. Proceedings of the 7th international conference on human

computer interaction with mobile devices & services (pp. 137-144). New York:

ACM. 10.1145/1085777.1085800

92 Hall, M., Hoggan, E., & Brewster, S. (2008). T-bars: towards tactile user interfaces for

mobile . Proceedings of the 10th international conference on human

computer interaction with mobile devices and services (pp. 411-414). New York:

ACM. 10.1145/1409240.1409301

Hornbæk, K., Bederson, B., & Plaisant, C. (2002). Navigation patterns and usability of

zoomable user interfaces with and without an overview. ACM Transactions on

Computer-Human Interaction (TOCHI), 9(4), 362-389.

HTC Corporation (2011). Htc surround - specification. Retrieved from

http://www.htc.com/www/product/ 7surround/specification.html

Huang, K. (2009). Challenges in human-computer interaction design for mobile devices.

Proceedings of the World congress on engineering and computer science 2009

Vol I, http://www.iaeng.org/publication/WCECS2009/WCECS2009_pp236-

241.pdf

Inxight (2008). Inxight star tree. Retrieved from

http://www.inxightfedsys.com/products/sdks/st/default.asp

Joel (2009, January 23). Different types of touch screens in mobile phones. Retrieved

from http://www.bizzntech.com/2009/01/23/different-types-of-touch-screens-in-

mobile-phones

Johnson, J. A. (1995). A comparison of user interfaces for panning on a touch-

controlled display. Proceedings of the Sigchi conference on human factors in

93 computing systems (pp. 218-225). New York: ACM Press/Addison-Wesley

Publishing Co. 10.1145/223904.223932

Karp, S. (2007, September 26). Five reasons why the mobile web sucks. Retrieved from

http://publishing2.com/2007/09/26/five-reasons-why-the-mobile-web-sucks/

Karray, F., Alemzadeh, M., Saleh, J.A., & Arab, M.N. (2008). Human-computer

interaction: overview on state of the art. Proceedings of the International journal

on smart sensing and intelligent systems,

http://www.s2is.org/Issues/v1/n1/papers/paper9.pdf

Kendrick, J. (2010, June 8). Will the iPad get the retina display? Retrieved from

http://gigaom.com/mobile/ipad-retina-display/

Ko, J., Chan, L., & Hung, Y. (2010). Public issues on projected user interface.

Proceedings of the 28th of the international conference extended abstracts on

human factors in computing systems (pp. 2873-2882). New York: ACM.

10.1145/1753846.1753874

Lam, H., & Baudisch, P. (2005). Summary thumbnails: readable overviews for small

screen web browsers. Proceedings of the SIGCHI conference on human factors

in computing systems (pp. 681-690). New York: ACM.

10.1145/1054972.1055066

94 Lamping, J., & Rao, R. (1994). Laying out and visualizing large trees using a hyperbolic

space. Proceedings of the 7th annual ACM symposium on user interface

software and technology (pp. 13-14). New York: ACM. 10.1145/192426.192430

LG Electronics (2011). Secret LGTU750. Retrieved from

http://www.lg.com/ca_en/mobile-phones/all-phones/LG-LGTU750.jsp

Li, Y. (2010). Gesture search: a tool for fast mobile data access. Proceedings of the

23nd annual ACM symposium on user interface software and technology (pp. 87-

96). New York: ACM. 10.1145/1866029.1866044

MacKay, B. (2003). The gateway: a navigation technique for migrating to small screens.

Proceedings of the Chi '03 extended abstracts on human factors in computing

systems (pp. 684-685). New York: ACM. 10.1145/765891.765928

Mackinlay, J. D., Robertson, G. G., & Card, S. K. (1991). The perspective wall: detail

and context smoothly integrated. Proceedings of the SIGCHI conference on

human factors in computing systems: reaching through technology (pp. 173-176).

New York: ACM. 10.1145/108844.108870

McDonough, M. J. (2010). Gadget Gift Guide, A Special Edition of Laptop, XXX(13), 41.

MeeGo (2011). About. Retrieved from http://meego.com/about

Microsoft Corporation (2011a). Windows phone. Retrieved from

http://www.microsoft.com/windowsphone/en-ca/default.aspx?wt.srch=1

95 Microsoft Corporation (2011b). Top 10 reasons to try office mobile 2010. Retrieved from

http://office.microsoft.com/en-ca/mobile/top-10-reasons-to-try-office-mobile-2010-

HA101794278.aspx?CTT=1

Milic-Frayling, N., & Sommerer, R. (2002). SmartView: Enhanced Document Viewer for

Mobile Devices (Technical Report MSR-TR-2002-114). Redmond, WA: Microsoft

Research

Miller, R. (2010, September 28). Blackberry playbook vs. ipad vs. galaxy tab vs. streak:

the tale of the tape. Retrieved from http://www.engadget.com/2010/09/28/

blackberry-playbook-vs-ipad-vs-galaxy-tab-vs-streak-the-tale/

Mixed Reality. (n.d.). In Wikipedia. Retrieved January 14, 2011, from

http://en.wikipedia.org/wiki/Mixed_reality

Motorola Mobility (2011). Motorola milestone 2 with motoblur. Retrieved from

http://www.motorola.com/Consumers/XW-EN/Consumer-Products-and-

Services/Mobile-Phones/ci.Milestone-2-XW -EN.alt

Moyle, M., & Cockburn, A. (2003). The design and evaluation of a flick gesture for „back‟

and „forward‟ in web browsers. Proceedings of the Fourth australasian user

interface conference on user interfaces 2003 - volume 18 (pp. 39-46).

Darlinghurst: Australian Computer Society, Inc.

Myslewski, R. (2011, January). How dangerous is the . MacLife, 5(1), 96

96 Natural User Interface. (n.d.). In Wikipedia. Retrieved January 14, 2011, from

http://en.wikipedia.org/wiki/Natural_User_Interface

Nokia (2007, January 16). graphics – overcoming the small screen

challenge. Retrieved from http://www.symbian-freak.com/downloads/pdf/Mobile_

Game_Graphics_Overcoming_the_Small_Screen_Challenge_v1_0_en.pdf

Nokia (2009, May). 3d video & the mvc standard. Retrieved from

http://research.nokia.com/files/3D_Video.pdf

Nokia (2010a). About symbian. Retrieved from http://symbian.nokia.com/about/

Nokia (2010b). Nokia n8 highlights. Retrieved from

http://events.nokia.com/nokian8/home.html

Nubarrón, J. (2011, January 12). Evolution of mobile technology: a brief history of 1g,

2g, 3g, and 4g mobile phones. Retrieved from http://www.brighthub.com/mobile/

emerging-platforms/articles/30965.aspx

Paelke, V., Reimann, C., & Rosenbach, W. (2003). A visualization design repository for

mobile devices. Proceedings of the 2nd international conference on Computer

graphics, , visualisation and interaction in Africa.

10.1145/602330.602341.

Palm (2010, October 19). Why webOS 2.0 goes to 11. Retrieved from

http://blog.palm.com/palm/2010/10/why-webos-20-goes-to-11.html

97 Paul, I. (2011, January). Tech giants battle for future gadget dominance at CES.

Retrieved from http://tech.ca.msn.com/pcworld-article.aspx?cp-

documentid=27021146

Perez, S. (2008, October 23). Three places to shop for android apps? how confusing!.

Retrieved from http://www.readwriteweb.com/archives/three_places_

to_shop_for_android_apps.php

Queen (2010, December 6). Beautiful phone but.. [Online Forum Comment]. Retrieved

from http://www.rogers.com/web/link/wirelessBuyFlow?forwardTo=PhoneThen

Plan&productType=normal&productId_Detailed=N8-00GRYR&N=52+11

Quickoffice (2011). Quickoffice. mobilize your world. Retrieved from

http://www.quickoffice.com/

Quirksmode (n.d.). Mobile compatibility tables. Retrieved from

http://www.quirksmode.org/m/table.html

Rao, R., & Card, S. K. (1994). The table lens: merging graphical and symbolic

representations in an interactive focus+context visualization for tabular

information. Proceedings of the Conference companion on human factors in

computing systems (pp. 222). New York: ACM. 10.1145/259963.260391

Research in Motion (2011a). Blackberry torch phone specifications. Retrieved from

http://us.blackberry.com/smartphones/blackberrytorch/#!phone-specifications

98 Robertson, G. G., & Mackinlay, J. D. (1993). The document lens. Proceedings of the 6th

annual acm symposium on user interface software and technology (pp. 101-108).

New York: ACM. 10.1145/168642.168652

Robertson, G. G., Mackinlay, J. D., & Card, S. K. (1991). Cone trees: animated 3d

visualizations of hierarchical information. Proceedings of the SIGCHI conference

on human factors in computing systems: reaching through technology (pp. 189-

194). New York: ACM. 10.1145/108844.108883

Roudaut, A., Huot, S., & Lecolinet, E. (2008). Taptap and magstick: improving one-

handed target acquisition on small touch-screens. Proceedings of the working

conference on advanced visual interfaces (pp. 146-153). New York: ACM.

10.1145/1385569.1385594

Samsung (2010a). Samsung galaxy tab (gt-p1000). Retrieved from

http://www.samsung.com/ca/consumer/mobile/mobile- phones/tablets/GT-

P1000ZKMBMC/index.idx?pagetype=prd_detail&returnurl=

Sarkar, M., & Brown, M. H. (1992). Graphical fisheye views of graphs. Proceedings of

the SIGCHI conference on human factors in computing systems (pp. 83-91). New

York: ACM. 10.1145/142750.142763

Savio, N., & Braiterman, J. (2007). Design sketch: the context of mobile interaction.

MMA International Journal of , 2(1), Retrieved from

http://69.89.31.51/~deciphe2/giantant/output/mobile_context_model.pdf

99 Savov, V. (2010, June 24). Symbian^3 reviewed in exquisite and ruthless detail by eldar

murtazin. Retrieved from http://www.engadget.com/2010/06/24/symbian-3-

reviewed-in-exquisite-and-ruthless-detail-by-eldar-mur/

Schinke, T., Henze, N., & Boll, S. (2010). Visualization of off- screen objects in mobile

augmented reality. Proceedings of the 12th international conference on human

computer interaction with mobile devices and services (pp. 313-316). New York:

ACM. 10.1145/1851600.1851655

Scott, J., Dearman, D., Yatani, K., & Truong, K. N. (2010). Sensing foot gestures from

the pocket. Proceedings of the 23nd annual acm symposium on user interface

software and technology (pp. 199-208). New York: ACM.

10.1145/1866029.1866063

Sharp, A. (2010, December 2). Rim buys swedish firm to improve screen design.

Retrieved from http://www.reuters.com/article/idUSTRE6B14RD20101202

Shen, E., Tsai, S. D., Chu, H., Hsu, Y. J., & Chen, C. E. (2009). Double-side multi-touch

input for mobile devices. Proceedings of the 27th international conference

extended abstracts on human factors in computing systems (pp. 4339-4344).

New York: ACM.10.1145/1520340.1520663

Sony Ericsson (2010). Liveview. Retrieved from

http://www.sonyericsson.com/cws/products/accessories/overview/liveview

100 Sousa, R., Nisi, V., & Oakley, I. (2009). Glaze: a visualization framework for mobile

devices. Proceedings of the 12th IFIP TC 13 international conference on human-

computer interaction: part i (pp. 870-873). Heidelberg: Springer-Verlag Berlin.

10.1007/978-3-642-03655-2_95

Stößel , C., & Blessing, L. (2010). Mobile device interaction gestures for older users.

Proceedings of the 6th nordic conference on human-computer interaction:

extending boundaries (pp. 793-796). New York: ACM.

10.1145/1868914.1869031

Stewart, C., Rohs, M., Kratz, S., & Essl, G. (2010). Characteristics of pressure-based

input for mobile devices. Proceedings of the 28th international conference on

human factors in computing systems (pp. 801-810). New York: ACM.

10.1145/1753326.1753444

Surur (2010, October 21). Engadget reviews the htc surround. Retrieved from

http://wmpoweruser.com/engadget-reviews-the-htc-surround/

TFT LCD. (2010, December, 27). In Wikipedia. Retrieved January 19, 2011 from

http://en.wikipedia.org/wiki/TFT_LCD

Topolsky, J. (2010, August 4). Blackberry torch review. Retrieved from

http://www.engadget.com/2010/08/04/blackberry-torch-review/

Touchscreens. (n.d.). In Wikipedia. Retrieved January 14, 2011, from

http://en.wikipedia.org/wiki/Touchscreen

101 Vendel, M. (2009, January 27). White Paper on Universal Mobile Interface, UMI.

Retrieved from http://hosted.squace.com/public_resources/umi/

UMI_white_paper.pdf

Vendel, M. (2010, February 21). Universal Mobile Interface. Does anyone dare to say

no to apps? Retrieved from http://universalmobileinterface.wordpress.com/

2010/02/21/does-anyone-dare-to-say-no-to-apps/

W3C (2010, December 14). Mobile web application best practices. Retrieved from

http://www.w3.org/TR/mwabp/

Wauters, R. (2010, June 22). Adobe starts shipping flash player 10.1 for mobile – let the

real testing begin. Retrieved from http://techcrunch.com/2010/06/22/flash-player-

mobile/

WebdesignerDepot. (2009, May 22). The evolution of cell phone design between 1983-

2009 [Web log message]. Retrieved from http://www.webdesignerdepot.com/

2009/05/the-evolution-of-cell-phone-design-between-1983-2009/

Werner, K. (2002). A comparison of information visualization methods. Proceedings of

the Chi '02 extended abstracts on human factors in computing systems (pp. 872-

873). New York: ACM. 10.1145/506443.506641

West, J., & Mace, M. (2010, February 6). Browsing as the killer app: explaining the rapid

success of apple's iphone. Telecommunications Policy, 34(5-6), doi:

10.1016/j.telpol.2009.12.002

102 Westerman, W.C. (1999). Hand Tracking, Finger Identification, and Chordic

Manipulation on a Multi-Touch Surface. University of Delaware. PhD

Dissertation. Retrieved from http://www.ece.udel.edu/~westerma/main.pdf

Wigdor, D., Forlines, C., Baudisch, P., Barnwell, J., & Shen, C. (2007). Lucid touch: a

see-through mobile device. Proceedings of the 20th annual ACM symposium on

user interface software and technology (pp. 269-278). New York: ACM.

10.1145/1294211.1294259

WhyTheLuckyMobile (2010, February). Mobile operating systems. Retrieved from

http://whytheluckymobile.com/downloads/mobile-os-overview.pdf

Yee, K. (2003). Peephole displays: pen interaction on spatially aware handheld

computers. Proceedings of the SIGCHI conference on human factors in

computing systems (pp. 1-8). New York: ACM. 10.1145/642611.642613

Zhang, R., North, S., & Koutsofios, E. (2010). A comparison of speech and gui input for

navigation in complex visualizations on mobile devices. Proceedings of the 12th

international conference on human computer interaction with mobile devices and

services (pp. 357-360). New York: ACM. 10.1145/1851600.1851665

103 APPENDIX A

CONTEXT OF MOBILE INTERACTION RESEARCH MAPPING

According to Savio and Braiterman (2007), there are key elements involved in designing for mobile interaction (called the Context of Mobile Interaction). The purpose of this appendix is to closely tie the research and manufacturing innovation that address the limitations of small mobile screens to these elements and highlight that the problem of small screen visualization is being addressed using seemingly different approaches that fit together harmoniously into a larger framework. This appendix is both an organizational technique to phrase the findings, as well as a high-level overview of work being performed – though the contents of this appendix are discussed throughout the paper.

The following four diagrams match research papers and material uncovered in this essay to these core elements. First, research that contributes to the greater context of mobile interaction is identified followed by elements that improve the user interface.

Then, methods that improve information visualization are mapped ending with a look at the elements that have improved the physical device.

As discussed by Karray (2008), results can also be mapped to three aspects of user interaction where “the physical aspect determines the mechanics of interaction between human and computer while the cognitive aspect deals with ways that users can understand the system and interact with it”. Affective aspects aim to both increase a user‟s pleasure in the interaction and to encourage continued interaction through

104 favorable changes in the user‟s perception. General mappings (the most direct links) to research material organized in this appendix are made below with the understanding that some research aims to improve man-machine interactions at more than a single level.

Physical

Improvements made at the device level (D) deal directly with the physical mechanics of interaction such as input and output. For instance, improving the physical size or abilities of the screen aims to improve the physical aspects of interaction between user and device through better input and output.

Cognitive

New interface (B) and visualization (C) techniques presented by researchers aim to increase understanding and usability of devices. For instance, new visualization techniques such as the scatter plot allow users to more easily comprehend and use information in ways that overcome limitations of the small screen.

Affective

Higher-level research of mobile interaction issues (A) aims to make the usage and interaction of mobile devices more pleasurable and promote continued use. For instance, research that makes interacting in low light situations, mobile settings, or with user goals in mind addresses affective interaction issues.

105 A. Mobile Interaction

3. User Tasks 4. Culture

2. User A 5. Activity Attention Mobile Interaction

1. User Goals 6. Environment

A. Mobile Interaction

a. Huang (2009) – HCI challenges in mobile interaction design

1. User Goals

a. Information

b. Entertainment

i. Nokia (2007, January 16) – Designing games for small screens

c. Social Interaction

2. User Attention

a. Intermittent/Continuous (full/partial)

i. Ferscha et al. (2010) – Wearable Displays (Spectacles)

ii. Francone et al. (2010) – Wavelet menus / Eyes-free selection

iii. Sony Ericsson (2010) – LiveView (Wearable Watch Screen)

106 3. User Tasks

a. View content (photo/video/web page)

i. Cho et al. (2007) – Tilt dynamics and photo browsing

b. Viewing public versus private content

i. Döring et al. (2010) – Multi-display gesture based interaction

ii. Ko et al. (2010) – Public Issues on Projected UI

4. Culture

a. Etiquette

i. Costanza et al. (2005) – Intimate interfaces using EMG

ii. Ko et al. (2010) – Public Issues on Projected UI

b. Generational Differences

i. Stößel et al. (2010) – Interaction gestures for older users

5. Activity

a. Walking/Running

i. Costanza et al. (2005) – Intimate interfaces using EMG

ii. Scott et al. (2010) – Sensing foot gestures

iii. Sony Ericsson (2010) – LiveView (Wearable Watch Screen)

b. Single-handed

i. Roudaut et al. (2008) – Taptap and magstick

ii. Samsung (2010a) – Galaxy Tab

c. Two-handed

d. No-handed

i. Costanza et al. (2005) – Intimate interfaces using EMG

107 6. Environment

a. Low light

i. Ko et al. (2010) – Public Issues on Projected UI

b. Privacy

i. Döring et al. (2010) – Multi-display gesture based interaction

ii. Ko et al. (2010) – Public Issues on Projected UI

108 B. Interface

4. Multi-user Interaction

3. Multi-modal 5. Gestures Interaction

2. Searching B 6. Multi-touch Interface

1. Navigation 7. Software

8. Application Stores

B. Interface

a. Huang (2009) – HCI challenges in mobile interaction design

1. Navigation

a. Cho et al. (2007) – Tilt dynamics and photo browsing

b. Fallman et al. (2004) – Tangible scrolling

c. Francone et al. (2010) – Wavelet menus / Eyes-free selection

d. Hornbæk et al. (2002) – Navigation patterns and ZUI usability with

overviews

e. Lamping et al. (1994) – Hyperbolic tree navigation (based on Fisheye)

f. MacKay (2003) – The Gateway: Navigation Technique 109 g. Milic-Frayling et al. (2002) – SmartView Document Viewer

h. Moyle et al. (2003) – Flick gesture for browser navigation

i. Zhang et al. (2010) – Speech and GUI input for navigation

2. Searching

a. Hakala et al. (2005) – Spatial interactive visualization

b. Li (2010) – Gesture Search

3. Multi-modal Interaction

a. Bau, et al. (2010) – Teslatouch

b. Brewster (2002) – Effect of sound on button sizes

c. Di Fabbrizio et al. (2010) – Speech mashup framework

d. Hall et al. (2008) – T-Bars / Touchable tactile buttons

e. Zhang et al. (2010) – Speech and GUI input for navigation

4. Multi-user Interaction

a. Döring et al. (2010) – Multi-display gesture based interaction

5. Gestures

a. Döring et al. (2010) – Multi-display gesture based interaction

b. Li (2010) – Gesture Search

c. Moyle et al. (2003) – Flick gesture for browser navigation

d. Scott et al. (2010) – Sensing foot gestures

e. Stößel et al. (2010) – Interaction gestures for older users

6. Multi-touch (Natural User Interfaces)

a. Albinsson et al. (2003) – High precision touch screen interaction

b. Shen et al. (2009) – Double-side multi-touch input

110 c. Westerman (1999) – PhD dissertation directly related to Apple‟s success

7. Software

a. Microsoft Corporation (2011b) – Office Mobile 2010

b. Quickoffice (2011) - Quickoffice

8. Application Stores

a. Perez (2008, October 23) – Android application stores

b. Vendel (2010) – Universal Mobile Interface. Does anyone dare to say no

to apps?

c. Vendel (2011) – White paper on UMI

111 C. Information Visualization

5. Zooming 3. Off-screen 4. Tilting / User Objects / 6. Object Size 7. WAP Panning Interfaces Areas (ZUI)

2. Visualization C 8. Real Web Frameworks Visualization

11. 1. Information 12. Software 10. Manufacturer 9. Screen-less Visualization Development Customizable Specific User Interfaces Methods Practices Interfaces Interface

C. Visualization

a. Huang (2009) – HCI challenges in mobile interaction design

b. Werner (2002) – Comparison of visualization methods

1. Information Visualization Methods

a. B¨uring et al. (2006) – Scatterplots

b. Furnas (1986) – Fisheye

c. Gutwin et al. (2004) – Fisheye, zoom, and panning comparison

d. Hakala et al. (2005) – Spatial interactive visualization

e. Inxight (2008) – Star Tree

f. Lam et al. (2005) – Summary Thumbnail

g. Lamping et al. (1994) – Hyperbolic tree navigation (based on Fisheye)

112 h. MacKay (2003) – The Gateway: Navigation Technique

i. Mackinlay et al. (1991) – Perspective Wall

j. Milic-Frayling et al. (2002) – SmartView Document Viewer

k. Rao et al. (1994) – The Table Lens

l. Robertson et al. (1993) – The Document Lens

m. Robertson et al. (1991) – Cone Trees

n. Sarkar et al. (1992) – Graphical fisheye

2. Visualization Frameworks

a. Paelke et al. (2003) – Mobile device visualization design repository

b. Sousa et al. (2009) – Glaze: visualization framework

3. Off-screen Objects/Areas

a. Schinke et al. (2010) – Visualization of off-screen objects in Mobile AR

4. Tilting / Panning

a. Adobe Systems Incorporated (2011) – Reflow Reader Content with Tilting

b. Cho et al. (2007) – Tilt dynamics and photo browsing

c. Gutwin et al. (2004) – Fisheye, zoom, and panning comparison

d. Johnson (1995) – UI comparison for panning on touch displays

5. Zooming User Interfaces (ZUI)

a. Hornbæk et al. (2002) – Navigation patterns and ZUI usability with

overviews

6. Size related Research (buttons, font-type)

a. Brewster (2002) – Effect of sound on button sizes

113 7. WAP

a. Buchanan et al. (2001) – Improving mobile internet usability

b. Gomez (2009) – Study of WAP likeability factors

c. Lam et al. (2005) – Summary Thumbnail

d. MacKay (2003) – The Gateway: Navigation Technique

e. Milic-Frayling et al. (2002) – SmartView Document Viewer

8. Real Web

a. West et al. (2010, February 6) – Browsing as the killer app

9. Screen-less Interfaces

a. Costanza et al. (2005) – Intimate interfaces using EMG

b. Gustafson et al. (2010) – Imaginary interfaces

c. Scott et al. (2010) – Sensing foot gestures

10. Customizable Interfaces / Migration Techniques

a. MacKay (2003) – The Gateway

11. (Observed) Manufacturer Specific Interface Ideas

a. Palm (2010, October 19) – WebOS Layering

12. Recommended Software Development Practices

a. Nokia (2007, January 16) – Designing games for small screens

b. W3C (2010, December 14) – Mobile web application best practices

114 D. Device

3. Output

2. Input D 4. Connectivity Device

1. Screen 5. Browser

D. Device

a. Huang (2009) – HCI challenges in mobile interaction design

1. Screen

a. Improve

i. Range of Mobile Device Screen Sizes (1 to 5 inches) 

1. Samsung (2010a) – Galaxy Tab

ii. Increasing Resolution

1. Apple (2011b) – Retina Display

iii. Increasing Colors

iv. Increasing Processor Power

v. Different Aspect Ratios

1. Chittaro (2006) – Aspect Ratios

vi. Different Viewing Angles

115 vii. Sensors

viii. 3D

1. Nokia (2009) – 3d video & the mvc standard b. Increase Size

i. Range of Tablet Screen Sizes (7 to 11 inches) – some one-handed c. Decouple

i. Ferscha et al. (2010) – Wearable Displays (Spectacles)

ii. Sony Ericsson (2010) – LiveView (Wearable Watch Screen) d. Specialized

i. Bradford (2010) - E-Ink e-reader screens e. Extend

i. Ko et al. (2010) – Public Issues on Projected UI

ii. Schinke et al. (2010) – Visualization of off-screen objects in Mobile

AR

iii. Yee (2003) – Peephole displays f. Decrease Reliance

i. Bau, et al. (2010) – Teslatouch g. Remove Screen

i. Costanza et al. (2005) – Intimate interfaces using EMG

ii. Gustafson et al. (2010) – Imaginary Interfaces

iii. Scott et al. (2010) – Sensing foot gestures h. Evolve / Augment Screen

i. Bau, et al. (2010) – Teslatouch

116 ii. Hall et al. (2008) – T-Bars / Touchable tactile buttons

iii. Shen et al. (2009) – Double-side multi-touch input

iv. Stewart et al. (2010) – Pressure-based input

v. Wigdor et al. (2007) – Lucid touch (see through)

i. Integrate Screen

i. Boring et al. (2010) – Touch Projector

ii. Döring et al. (2010) – Multi-display gesture based interaction

2. Input

a. Costanza et al. (2005) – Intimate interfaces using EMG

b. Fallman et al. (2004) – Tangible scrolling

c. Go et al. (2010) – Split-key software keyboard, finger occlusion

d. Roudaut et al. (2008) – Taptap and magstick

e. Shen et al. (2009) – Double-side multi-touch input

f. Stewart et al. (2010) – Pressure-based input

g. Zhang et al. (2010) – Speech and GUI input for navigation

3. Output

a. (Same research as device. For example - evolve / augment screen)

4. Connectivity

a. (No interface research related to connectivity)

5. Browser

a. Lam et al. (2005) – Summary Thumbnail

b. MacKay (2003) – The Gateway: Navigation Technique

c. Milic-Frayling et al. (2002) – SmartView Document Viewer

117