Cloud and Mobile Web-Based Graphics and Visualization SIBGRAPI 2012 Full-Day Tutorial 22 August 2012, Ouro Preto – MG, Brazil

Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Full-Day Tutorial 22 August 2012, Ouro Preto – MG, Brazil

Haim Levkowitz Department of Computer Science University of Massachusetts Lowell Lowell, Massachusetts 01854, USA and ICMC University of Sao˜ Paulo, Sao˜ Carlso – SP, Brazil [email protected], http://www.cs.uml.edu/∼haim/

ABSTRACT sor and a Fulbright Scholar to Brazil at ICMC, The University of Sao˜ Paulo, Sao˜ Carlos – SP, Brazil; and an associate Cloud computing is becoming the most prevailing comput- professor of computer science and director of the Human- ing platform. The combination of mobile devices and cloud- Information Interaction Research Group at the University of based computing is changing how users consume and use Massachusetts Lowell, Lowell, MA, USA. He is a world- computing resources. With the introduction and penetration renowned authority on visualization, perception, color, and of HTML5, and, in particular, its visual Canvas element, their application in human-information interaction, data min- high-quality Web-based graphics has become a reality. We- ing, and information retrieval. He is the author of “Color bGL offers capabilities comparable to OpenGL utilizing Web- Theory and Modeling for Computer Graphics, Visualization, based computing resources. It is now feasible to have high- and Multimedia Applications” (Springer 1997) and co-editor performance graphics and visualization “in your palm,” utiliz- of “Perceptual Issues in Visualization” (Springer 1995), as ing a mobile device as the front end interface and the display, well as many papers in these subjects. He has more than 40 performing the graphics “heavy lifting” on a cloud platform years experience teaching and lecturing, and has taught many as needed. We argue that this will become the most common tutorials and short courses, in addition to regular academic platform for computer graphics and visualization in the not- courses. too-distant future. Acknowledgments: The author wishes to acknowledge the The goals of this course are to make students familiar with profound and essential contributions of Curran Kelleher to this the underlying technologies that make this possible, includ- tutorial. ing cloud computing, mobile computing, their combination, HTML5 and the Canvas element, WebGL, other libraries, and CONTENTS general Web-based graphics and visualization. • Introduction, motivation, overview, and focus Who should attend: researcher, practitioners, and students • Visual HTML5 focused on the ﬁelds of cloud computing, mobile computing, • Developing and deploying mobile apps graphics, visualization, Web-based environments and their • Interactive visualization on mobile+cloud applications. Students will gain a deep understanding of these • Data on the Web techniques and technologies, and will become capable of • Concluding remarks, ﬁnal thoughts, future outlook applying their knowledge to develop interactive mobile- and • (Due to a page limitation, this notes-set is not cloud-based graphics and visualization applications. Previous complete. A relevant bibliography,additional materi- knowledge of and experience with interactive computer graph- als, and copy of this document are available at ics and visualization will is recommended. www.cs.uml.edu/∼haim/conferences/sibgrapi2012.) keywords: cloud computing; computer graphics; visual analytics; visual data mining; visualization; Web-based graphics; Web-based visualization; About the instructor: Haim Levkowitz is a visiting profes- SIBGRAPHI 2012 Tutorial: Overview Cloud and mobile Web- based graphics and • Why? visualization • Goals • Who? Haim Levkowitz • Benefits University of Masschusetts Lowell • Topics University of Sao Paulo, Sao Carlos

Why?

• Compute model changing • Cloud computing becoming most • “visual computing” part of that prevailing computing platform • How new wave of computing actually • Mobile + cloud changing how users implemented? • consume & use compute resources • HTML5, visual capabilities • canvas element • ==> … SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 3 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 4

• ==> high-quality Web-based • mobile device = front end interface + graphics --> reality display • WebGL ==> comparable to OpenGL • some / all graphics “heavy lifting” on • client-side + cloud-based cloud • ==> high-performance graphics and visualization “in your palm”

• Most common platform for computer • Major components enabling shift? graphics and visualization • Bring previous/current practices in? • in not-too-distant future • Familiarize with underlying • >= 80% of visual computing migrate technologies that make this possible • only small percentage • including (but not limited to) … • desktop computing

Prevailing technologies

• cloud-based computing • HTML5 • mobile computing • canvas element • cloud + mobile combination • WebGL graphics library • … • Web-based graphics and visualization • Web-based interactive development environments

Compare tools Benefits (to the world)

• Criteria in choosing technologies • Democratization • Recommendations which tools to use • Tools: specialist --> everyone • Vis in apps for all • E.g., no “paper” in “newspaper” • E.g., Wired magazine …

• Paper • Tablet • Paper • Tablet • Static • Interactive • Static • More pictures content • Some pictures • Videos

Resource shift Future: interactive

• Paper --> electronic • E.g., what’s better for economy? • Static --> interactive • Austerity vs. stimulus • ==> Next few years, more • Interactive vis on mobile publications exclusively on mobile • ==> user: “what if?” devices • desktop: afterthought • paper: abandoned SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 15 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 16

Topics: computer Topics: Brief overview of graphics • the Internet; • Visual computing: • history, goals, fundamentals • CG vs. Image Processing vs. vision • the WWW; • objects-to-pictures vs pictures-to- • history, goals, fundamentals pictures vs pictures-to-objects • “e-bboard” --> interactive prog env • background material/presentations: Visual computing

• • Introduction to mobile computing • • cloud computing iOS vs. Android vs. other • • … technical • deployment/logistic • native vs. Web-app • cross platform

• mobile and cloud computing • this is it! • “ping-pong" between local and remote

• Computer graphics Visual computing • Image processing • Vision • Visualization

Computer graphics Image processing

• models/objects --> pictures • pictures --> pictures • “synthesize” • 2D --> 2D • “render” • 2D, 3D --> 2D • viewer, light source(s), object(s) • realistic

Vision Visualization

• pictures --> models/objects • data --> visual • 2D x 2D --> 3D • images • models • animation • large amounts • utilize perception power

What is cloud computing?

• Many definitions; here are some: • Use commodity computers • Access applications hosted on web • distributed throughout internet through web browser (Software as a • to perform Service -- SaaS) • parallel processing, distributed • Pay-as-you-go model for IT storage, indexing and mining of resources accessed over Internet data (Platform as a Service – PaaS)

Common cloud themes

• “ Gartner: … style of computing where § They’re big -– massively scalable massively scalable IT-related § Always there when need them capabilities … provided ‘as a service’ § across the Internet to multiple on-demand, dynamic external customers” § Only use what need § elastic, no upfront commitments, use on short term basis

Accessibility = Any time, any place, any • “location-independent device compuing” (Wikipedia) • Leverage low cost compute cycles • Many definitions, always “location” • assured data storage in cloud • The “cloud” • Communications is pacing factor • But first, a little history …. • Challenge: balance platform agnostic vs. end point device innovations SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 7 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 8

A little history? More history

• Batch (“there”): • Mini computers (still “there”) • paper tapes, punch cards • PDP 11 / Eclispe / Nova • dumm terminals • Super minis (ditto) • More interactive (still “there”) • VAX • still big “IBM/NCR …” in basement • Worstations (“here” and “there”) • big, heavy, expensive • Apollo, Sun, HP

• Client-server (“here” and “there”) • Mobile (“here here”) • Even graphics & interaction • In Web browser (“here” … and “there”) • X-Windows • like client-server • PCs (“here”) • How much “here”? How much “there”? • Apple, IBM, MS • Laptop (very “here”)

• resource delivery and usage model • network of providing resources • hardware + software • hardware resources • via network • infinitely scalable • “on-demand” • can be used whenever • “at scale” • in multi-tenant environment

The cloud

§ Out there on Internet somewhere § accessible, location independent § Transparent -– complexity concealed from users, virtualized, abstracted § Service oriented § easy to use, SLAs, accessible

Trends enabling (& driven by) cloud computing • parallelism Simple Metaphor: Better Metaphor: • virtualization Like Power Company Cooperatively Owned • Semiconductor Fab commodity components • core elements outsourcing

• key technology App App App App App App OS OS OS Operating System Hypervisor Hardware Hardware Traditional Stack Virtualized Stack

Cloud deployment models • Community cloud • cloud infrastructure jointly owned • Internal (private) cloud by several organizations • cloud infrastructure operated within • supports specific community organization • shared concerns / needs, e.g., • mission, security requirements, policy, compliance considerations

• Hybrid cloud • cloud infrastructure = composition • Public cloud of two or more clouds (internal, • cloud infrastructure owned by community, public) organization selling cloud services • remain unique entities but bound to together by standardized / • general public proprietary technology ==> • large industry group enables data and application portability

SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 23 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 24 Business case for cloud computing • • Automation/On-Demand ==> Division of labor • • Better, Faster & Cheaper developers focus on new software • • Move ‘hand crafted’ SW --> Ease of use • • repeatable assembly abstract complexity out of developers’ lives • Reuse interchangeable components • Avoid over & under provisioning • Repeatable processes w/ increased • automation & collaboration CAPEX outlays SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 25 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 26

Data Intensive Computing Tenets • Index & make sense of large data sets • parallelization • “* as a Service” (*aaS), * = • Pre-format data in large repositories • Software (SaaS) for low BW transmissions • Platform (PaaS) • Better access to data w/ large multi- • Infrastructure (IaaS) tenant distributed cloud databases • Commodity HW • Default backup + cost effective • No special purpose (usually) archival of large data sets SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 27 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 28

Technologies (1/2) Technologies (2/2)

• new advances in processors • parallel computing • virtualization • distributed, large-scale server • distributed storage clusters • broadband Internet access • server virtualization software • automated management • fast, inexpensive servers

PaaS SaaS Platform as a Service

Advantages Platform delivery model • Pay per use • Platforms built upon Infrastructure • Instant Scalability • expensive • Security? • Estimating demand difficult • Reliability? • Platform management not fun! • APIs

PaaS PaaS

Popular services Advantages • Storage • Pay per use • Database • Instant Scalability • • Scalability Security? • Reliability? • APIs

SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 35 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 36 IaaS PaaS Infrastructure as a Service Computer infrastructure Examples delivery model • Google App Engine Access to infrastructure stack: • Mosso • Full OS access • AWS: S3 (Amazon Web Services: • Simple Storage Service) Firewalls • Routers • Load balancing SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 37 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 38

IaaS IaaS

Advantages Examples • Pay per use • Flexiscale • Instant Scalability • AWS: EC2 (Amazon Web Services: • Security? Elastic Compute Cloud) • Reliability? • APIs

{S/P/I}aaS

Common Factors Advantages • Pay per use • Lower cost of ownership • Instant Scalability • Reduce infrastructure management • Security? responsibility • Reliability? • Allow for unexpected resource loads • APIs • Faster application rollout

• HTML5 Visual HTML5 • Building apps with HTML5 • Visual HTML5 • Canvas • SVG – Scalable Vector Graphics • WebGL • 2D + 3D libraries SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 1 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 2

HTML5 HTML5 (2)

• HTML5 to Cloud & Mobile = Java to • “revolutionizing the way the Web desktop computing: evolves, works, and is used” • cross-platform application building • -- “HTML5 leads a Web revolution” technology CACM, July 2012 • single specification • whole set of technologies

HTML5 (3) HTML5 (4)

• Associated standards • More broadly, include specific (APIs) • Document Object Model (DOM) • enable new browser-based graphics • access & manipulate HTML docs • geolocation • Cascading Style Sheets (CSS) • local storage • define presentation & appearance • video capabilities of HTML doc • JavaScript SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 5 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 6 HTML5 (5) HTML5 (6)

• Heart of W3C’s Open Web Platform • “The Web over 20 years has • umbrella term developed from a Web of more-or- • changes over time less static documents to, now, a platform for applications.” • markup language & various • technologies that pertain to it HTML = centerpiece • HTML5 = most recent + most capable

Roles Roles (2)

• W3C oversees development • Vendors • HTML5 + graphics capabilitie • implement standards independently • W3C governs HTML5 standards • freedom to innovate • e.g., Apple introduced Canvas • innovations --> W3C standards • ==> …

Building apps with Roles (3) HTML5 • standards implementations • Avoids vendor lock-in continually increase in quality • affords compatibility across most • technology evolution not desktop browsers and mobile devices responsibility of any single entity

SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 11 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 12 Primary W3C graphics Visual HTML5 technologies today • “Visual” = • Canvas • graphics • WebGL • 2D + 3D • Scalable Vector Graphics (SVG) • image • Graphics and visualization libraries • audio-visual built on these • ==> higher levels of abstraction for working with interactive graphics SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 13 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 14

2D HTML5 Canvas

• immediate-mode graphics API for Web • no new concept • well known 2D graphics techniques into HTML and JavaScript • Many similar APIs before Canvas …

Similar APIs before Frameworks Canvas • Cairo • Give developers full access to • Java2D graphical display • .NET graphics libraries • ==> developers to build entire apps • QT ground • • GTK custom designed looks and feels & graphical behavior

HTML5 Canvas API

• Data Driven Documents (D3.js) • notable library • based on use of SVG • for developing interactive visualizations

Canvas API Canvas API (2)

• Developers can insert rectangular • API provides functions for bitmap manipulating bitmap • as element in HTML page • using well known graphics concepts • access it through JavaScript API and techniques • Canvas bitmap = array of colored pixels

• Originally defined and implemented • Today implemented in all major by Apple in 2004 browsers • subsequently proposed as standard • polyfills • e.g., ExplorerCanvas and Google Chrome Frame • support backward compatibility with older browsers SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 25 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 26

Canvas API (5) Canvas API (6)

• Capabilities of Canvas in detail • Uses stateful context • for determining properties of graphical elements drawn • Canvas 2D context includes following variables • …

Canvas API (7) Canvas API (8)

• Variables: • 1. set context variables • Include table/list of var’s or refer to • 2. draw paths the canvas survey paper • use variable values • push + pop functions • context variables to/from stack, • context.save, context.restore

• Canvas element: functions • Bezier curves support • draw • styles • lines, polygons, arcs, text • line caps, joins • fill paths • comprehensive text drawing API • solid colors, gradients (linear &radial), repeated image patterns

Canvas API (11) Canvas API (12)

• Exposes bitmap pixels directly • Image data can be • ==> allow construction of filters • exported • e.g., blur, invert, emboss • uploaded to a server • also enables implementation of computer vision algorithms

Canvas API (13) Canvas API (14)

• Compositing API • Can load and draw on Canvas • composition of multiple bitmaps • external image data • using various techniques • frames from HTML5 Video element • Shadows • severe performance penalty

• Alone not enough to build interactive • Events provided by HTML DOM APIs graphics applications • mouse and keyboard • multi-touch from mobile devices • must • be accessed through JavaScript • drive graphic manipulations

Canvas API (17) Canvas API (18)

• requestAnimationFrame: special • Better alternative to setInterval for function for implementing smooth animation loop animations • because provides timing of animation • … frames • synchronized with refresh rate of graphics HW • … SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 39 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 40

Canvas API (19) Canvas API (20)

• Implementations can provide • JavaScript single threaded additional optimizations • ==> intensive computations that • e.g., do not execute animations would slow animation loop when page not visible • e.g., physics /image processing • can offload to other threads • using Web Workers

• Library for immediate-mode graphics • uses HTML5 Canvas • based on original Java-based Processing project

Processing language Processing.js (2) parser • Two majore components: • Transforms source code written in • Processing language parser … Java-like Processing language into • Processing graphics API JavaScript implementation • ==> enables developers to execute Processing programs in HTML5 environment • with little /no code modification

Processing graphics API In practice implementation • Uses • Many developers prefer to use • HTML5 Canvas -- for 2D features Processing API from JavaScript • WebGL -- for 3D features • rather than write software in Processing language • because …

• Because introduction of another • Scene graph library for 2D graphics language into project introduces • uses HTML5 Canvas complexity, and • provides vector graphics scene graph • because Processing programs are • similar to Adobe Illustrator difficult to debug

Paper.js (2) Paper.js (3)

• API inspired by and mostly compatible • Provides support for common with Adobe Scriptographer graphics primitives • JavaScript scripting plugin for • groups, layers, paths (Bezier curves) Adobe Illustrator with outline drawing, mouse and • created by developers of Paper.js keyboard interaction, working with raster images, vector geometry operations

Paper.js (4) 3D

• Main contribution: • brings vector graphics model of Adobe Illustrator to Web • ==> giving developers straightforward cross-browser retained-mode 2D graphics API

SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 53 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 54 WebGL brings capabilities WebGL to WWW • JavaScript API for OpenGL ES 2.0 • Render 3D scenes • simplified OpenGL • lighting • immediate-mode 3D graphics • textures capability • shaders definition • not fundamentally new • hardware accelerated 3D graphics to HTML and JavaScript SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 55 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 56

WebGL vs. OpenGL ES Scene graph libraries

• ES (= Embedded Systems) • e.g., Three.js ==> • retained-mode 3D graphics implementations

Physics libraries 3D Graphics Libraries

• e.g., Box2D.js ==> • physics simulation

• = JavaScript library for 3D graphics • Introduces standard 3D graphics • abstracts rendering layer abstractions • ==> allowing developers to write • scene graph, cameras, meshes, same code for rendering 3D graphics materials, textures, lighting models, using either WebGL, HTML5 Canvas, common objects or SVG • e.g. cubes and spheres

Three.js (3) Three.js (4)

• Support for loading 3D models in • Physics engine for Three.js Collada file format, and more • Physijs • Plenty examples using Three.js • built on Ammo.js published on Web • direct port of Bullet physics engine to JavaScript

Data Driven Documents D3 (2) (D3) • JavaScript library • Solves problem of performing • written by Mike Bostock Document Object Model (DOM) • for 2D Web-based interactive data manipulation based on data visualization • declarative approach leveraging functional programming techniques • ==> …

• Developers write concise statements • Does not introduce to manipulate DOM • own graphics API • based on data • scene graph model • rather than write verbose and • provides developers powerful tool for convoluted data transformation code leveraging Web standards using raw DOM AP • e.g., SVG and CSS

D3 (5) HTML5 wrap up

• Successor to ProtoVis • Being deployed now • visualization library • not finished standard • introduced own graphics • adoption varies vocabulary

HTML5 wrap up (2) HTML5 wrap up (3)

• E.g., no single standard for • Video standards • video compression (codec) • Flash = NOT! • streaming protocol • Apple iOS ==> • digital rights management (DRM) • no Flash, yes HTML5 • MS, Google – diff approaches • Browsers’ coverage varies

• “But the individual specifications are at different maturity levels and will become standards at different times” • -- W3C

Developing and • HTML5: “write once, run anywhere” • for developing mobile applications deploying mobile apps

Native app alternative HTML5 alternative

• Android -- Java • Write apps once w/HTML5 • iOS -- Objective-C • + mobile-specific compatibility • Windows Mobile -- MS tools tricks • ==> duplicate • deploy as native apps to all platforms • effort • available tools … • cost • skill sets SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 3 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 4

HTML5 --> native tools Native apps

• PhoneGap • Capabilities not available to Web • Appcelerator Titanium pages • ==> • due to browser sandboxing • powerful force HTML5 --> most widely used app dev approach

• Access to local file system • Camera • native OS features • compass • e.g., launch sub-applications with • geolocation Android intents • notifications with alerts, sound, or • hardware accelerated graphics vibration • accelerometer

Native apps (4) Native apps (5)

• Access to • High cost for multi platform dev • user’s contacts • repeat for each platform • revenue-generating app • learn proprietary app dev tools marketplaces • port code to different programming • Apple Store languages • Android Market • alternative? …

HTML5 apps Deploy as

• HTML5 : viable cross-platform • Web page for solution • desktop browsers • Dev app in HTML5 • Web page for mobile browsers • Deploy … • native application for • each mobile platforms • ==> …

• Apps originally developed w/ HTML5 • HTML5 apps can “feel” like native • ==> reach much larger audience • but not automatic • ==> much larger profit • tricks to achieve native app “look- • lower cost of application development and-feel” …

“Native” tricks “Native” tricks (2)

• Detect display resolution • If possible, scale user interface • render appropriately scaled version of elements to match the display app • simpler • different layouts for different display resolutions • …

“Native” tricks (3) “Native” tricks (4)

• Detect device orientation changes • Hide browser “telltales” • render accordingly • mobile browser’s navigation bar • user interface elements might need to • other distracting visual elements have different layout for different • ==> user only sees app full screen orientations • ensure all work well

• Use icon at correct resolution to look • Resolution properly configured like app icon • ==> icon on mobile desktop looks • shortcuts to saved pages ==> icon exactly like native app icon on mobile desktop • can be specified by page • …

“Native” tricks (7) “Native” tricks (8)

• Use HTML5 local storage API storage • Even with cloud-driven apps • ==> app can function without • local storage temporary holds Internet connection changes • read & write state • synch’d with cloud service • when connection re-established • ==> …

Cross platform dev tools “Native” tricks (9) and environments • ==> difference between usable app and unusable app

• Most mobile dev platforms • This means it is possible to build • have UI component native apps that • can display Web page within native • simply load a full screen view of a app Web page, which can itself • be a full featured HTML5 app. In this way, HTML5 apps can • be deployed as native apps SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 25 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 26

Cross platform dev (3) Cross platform dev (4)

• Native mobile app can displays Web • ==> arbitrary functionality available page to native apps • can modify page’s runtime • can be exposed to Web pages • ==> arbitrary functions in native • via JavaScript API app available to JavaScript code on • ==> HTML5 apps can gain all loaded Web page functionality available to native apps

Cross platform dev (5) Cross platform dev (6)

• ==> pattern for deploying HTML5 • 1. Build native app that displays Web apps as native apps: page • 2. Expose native functionality to Web page via JavaScript API • …

• 3. Build HTML5 app as Web page • Pattern HTML5 apps --> native embedded within native app mobile apps straightforward • take advantage of native • ==> can be automated functionality • mobile device capabilities similar • via JavaScript API • across platforms • ==> JavaScript API to native functionality can be standardized SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 31 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 32

PhoneGap (2) PhoneGap (3)

• PhoneGap project automates HTML5 • Using PhoneGap, developers can apps --> native mobile apps • author HTML5 apps once • specifies single JavaScript API for • derive native apps for multiple accessing native capabilities across platforms in single automated step many mobile platforms

PhoneGap (3) Appcellerator Titanium

• Once native apps created • Comprehensive mobile app • author can go through typical steps development platform • deploy app to revenue generating • based on app marketplace • JavaScript APIs • Apple App Store • HTML5 • Android Market • includes …

• Software SDK for developing mobile • Titanium Mobile SDK ==> apps • write app once in JavaScript • Eclipse-based IDE equipped with • automatically deploy to platform-specific tooling severalmobile platforms • suite of cloud-based services to • native apps speed app development • mobile Web • app analytics service SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 37 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 38

Appcellerator Titanium (4) Appcellerator Titanium (5)

• Process similar to PhoneGap • ==> develop, test, deploy mobile • Appcellerator dev env apps • Titanium Studio • Eclipse-based IDE • platform-specific functionality

Appcellerator Cloud Appcellerator analytics Services • Scalable back end features • Service aids app developers • common to many apps • collect • normally needed to be implemented • analyze by app developers • data about • cloud features: user management, • users, sessions, actions taken logins, photo uploads, push within apps notifications, status updates SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 41 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 42 Deploying apps Deploying apps

• 1) Android Market … • How to • 2) iOS App Store … • “get your app out there” • 3) Chrome Web Store … • generate revenue • 4) Amazon App Store … • 5) Zeewe … • 6) TapJS …

Google Play (= “Android June 2012 stats Market”) • 51.8% of smart phones -- Android • Publish apps in Android Market • 34.3% -- iOS • payment and deployment to client • ==> corresponding app marketplaces devices managed by platform • Google Play (= “Android Market”) • now, new interface, new app • iOS App Store discoverability features • • ==> reach most users e.g., recommendation system

Google Play (= “Android https://play.google.com/store Market”) • Sells also music, books, movies • Publishing = automated process • free for developers • straightforward app preparation and Web-based publishing process • ==> app appea in marketplace within minutes SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 47 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 48 iOS App Store iOS App Store (2)

• For iOS devices • Apps rejected for nebulous reasons • iPhone, iPod Touch, iPad • very strictly moderated, • costs developers $99 per year • submit, & keep apps available at Store

Chrome Web Store

• App marketplace from Google • Many apps free • aimed at Chrome users • collecting payment possible • “Chrome App” = Web app • when Chrome Web Store Payments • additional metadata stored by system used Chrome • $5 to become app author on Chrome • icon to launch app from Chrome Web Store home page • no recurring fees SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 51 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 52

https://developers.google.com/ chrome/web-store/docs/ Other markets

• Amazon’s App Store • Zeewe, a marketplace for Mobile Web Apps • TapJS, a game hosting service • Playtomic

• Based on market share statistics • reach largest majority of potential audience by focusing on • Apple’s App Store (the only marketplace for iOS apps) • Google Play for Android apps

SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 55 Overview • Interactive visualization Google Body Browser … • Google Maps … on mobile+cloud • TileMill … • Tableau Public … • D3 examples built for mobile devices … • more examples?

Google Body Browser Google Body Browser (2)

• Google Body Browser • peel away layers • now known as Zygote Body • skin, muscles, bones, vascular • http://www.zygotebody.com/ system, nervous system • virtual human body • zoom & rotate 3D model • “visible human” project • click on body part • exemplary example application of • ==> highlight & get name WebGL SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 3 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 4

Google Body Browser (4) Google Body Browser (3)

• search box ==> find body parts by name

• has been ported to native Android • Other similar projects app • such BioDigital Human • http://code.google.com/p/gdc2011- • https:// android-opengl/wiki/TalkTranscript www.biodigitalhuman.com/ • ==> take advantage of native app home/ features, e.g., data bundling, accelerated graphics

Google Maps

• Web-based mapping service perfect example of visual computing app • graphics “heavy lifting” of rendering map images • on server side

Google Maps (2) Google Maps (3)

• navigate maps • exists as • several views • Web app for • satellite imagery, street maps, • desktop browsers bicycle maps, terrain, weather, • mobile devices photos, traffic, more • optimized native app for mobile • find directions from place to place platforms • share map configurations SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 11 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 12 Google Maps (4) Google Maps (5)

• native apps take advantage of • Google also provides API device’s geolocation • ==> developers effortlessly embed • provide additional features maps within their apps • “show me where I am now” • live driving and walking directions

TileMill

• http://mapbox.com/tilemill/ • open source project • managed by MapBox • render map tiles on server side • styled using own CSS-like map styling • language called CartoCSS

TileMill (2)

• based on open source projects • Mapnik and Node.js • powerful tool for developing interactive map applications • can, e.g., overlay data vis on map, or • re-style existing map data • e.g., OpenStreetMap SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 17 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 18 Tableau Public Tableau Public (2)

• http://www.tableausoftware.com/ • Web-based version of Tableau public/ • enable interactive visual analysis • adaptation of Tableau visualization • relay user interactions to server-side SW to social Web instances of Tableau • render visualization on server • send rendered image to client

Tableau Public (3)

• gallery feature • ==> users can post visualizations • comment on them • share them across Web • embed them into their own pages

Web-based development OpenProcessing showcases • OpenProcessing … • http://www.openprocessing.org/ • Google Chrome Experiments … • online community platform • devoted to sharing and discussing Processing sketches • in collaborative, open-source environment

Google Chrome Experiments (2) • Examples: • The Wilderness Downtown … • 3D Water Waves … • Progressive Julia Fractal … • WebGL Experiments …

The Wilderness 3D Water Waves Downtown • interactive music video • demonstration of • enter your address when it starts • fluid simulation • then builds dynamic scenes • advanced 3D graphics techniques • based on images from that address • using WebGL • taken from Google Maps and Google Street view

• lightning fast implementation of Julia set rendering • uses shader-based image distortion • by Felix Woitze • demonstrates potential for general purpose GPU computing using WebGL

http://www.chromeexperiments.com/ WebGL Experiments webgl/

• http://www.chromeexperiments.com/ webgl/ • demonstrations of several other WebGL-based implementation

Conclusion

• Some examples of interactive mobile +cloud visualizations

• Semantic Web and the Data on the Web democratization of data • public data initiatives

Semantic Web and the democratization of data • Mobile+Cloud will democratize data analysis! • technologies & initiatives • already launched • demonstrate this evolution

Growth of Linked Open Data Cloud, from 12 datasets in 2007 to 295 in 2011 [http://richard.cyganiak.de/2007/10/lod/] Semantic Web

• “Web of Data” • “Giant Global Graph” • “Data Web” • “Linked Data Web” • “Enterprise Information Web”

Web Ontology Language Vocabularies (OWL) • Higher level of Semantic Web • provide data publishers common • represent domain ontologies and means to express domain concepts inference rules • RDF Data Cube Vocabulary • enable integration of statistical data from many data providers

Semantic Web + Web-based Visual Computing • More public data sets published within Semantic Web • Visual data analysis tools built to consume SW data • ==> able to access data as it becomes available

• ==> rich Web-based visualization • UN, World Bank, US Federal Data, and analysis tools more … • applied to SW information • Example visualizations of public data: • GapMinder • Italy budget Viz using D3 • BLS employment vis

UN, World Bank, US Int’l Human Dev Federal Data, more Indicators • UN published • http://hdr.undp.org/en/data/ • 34 databases explorer/ • 60 million records • Public Data Explorer • indicators over years / countries • economics, energy, human dev, crime, health, tourism, telecomm, +

Data.gov

• access to many data sets

Fedstats (fedstats.gov)

• links to numerous public data sets • hosted by US government organizations

Example visualizations of GapMinder World public data: • GapMinder World … • http://www.gapminder.org/world/ • Italy budget Viz using D3 … • Web-based visualization • BLS employment vis … • important trends around the World

• http://www.gapminder.org/world/ • interactive visualization • Web-based visualization • ==> move through time • important trends around the World • 1800 – today • interact with various parameters • ==> see how trends changed over time, geo location + other categories

GapMinder World (3) GapMinder World (4)

• Parameters: • Each parameter --> additional choice • income / person, children / woman, granularity child mortality, life expectancy, • Advanced users ==> advanced economy, society, education, features • energy, environment, health, • ==>further enrich interaction, infrastructure, population and work exploration

GapMinder World (5) GapMinder World (6)

• Some examples: • “Africa is not a country!” • “Wealth & Health of Nations” • huge diff’s among African countries • how long people live • “Is child mortality falling” • how much money they make • “Where is HIV decreasing?”

• “CO2 emissions since 1820” • changes in the number of people • … living with HIV

• http://www.visup.it/misc/workshop/ index.htm • Web-based visualization • access to public data • Italy’s administrative expenses during • 2002-2008

Italy budget Viz (2) BLS employment vis

• hover ==> • CNN Economy Tracker — US Bureau of • effortless and effective navigation Labor Statistics (BLS) employment • across 3 dims of data cube visualization • • D3 implementation http://www.cnn.com/SPECIALS/ • map.economy/index.html? access via Firefox, Safari, Opera mapIndex=3&hpt=C2 • not Internet Explorer

• interactive Choropleth map • colored in proportion to economic variable explored • probe states + get detailed values • colored relative to variables • data spans across US • states, industry, time SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 35 HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 36 Conclusion

• Public data • + Semantic Web • + Web-based interactive visualization • ==> access to information • “what if”? • info-based learning + decision making SIBGRAPI 2012 Tutorial: Cloud and mobile Web-based graphics and visualization Human Information Interaction for Knowledge Extraction, Interaction, Utilization, Decision making HI-I-KEIUD © Copyright 2012 Haim Levkowitz! 37 Conclusions

Conclusions, final • Mobile and cloud ==> prevailing computing paradigm thoughts • Graphics, visualization to take advantage • HTML5 + Canvas + WebGL + SVG + other libraries

Final thoughts

• Visual (and other) computing: • Local interaction – on mobile • “Horsepower” – on cloud • Need • balancing • synchronization