Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

AJAX Performance and Monitoring Ron Bodkin, Glassbox Leader [email protected]

Rod Bodkin — AJAX Performance and Monitoring Page 1 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Agenda ƒ AJAX Overview ƒ AJAX Technology ƒ Challenges ƒ Solutions

Rod Bodkin — AJAX Performance and Monitoring Page 2 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

AJAX…

Rod Bodkin — AJAX Performance and Monitoring Page 3 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

AJAX Defined ƒ Asynchronous JavaScript And XML ƒ A group of technologies: HTML, JavaScript and XML ƒ Still HTML-based, but with JavaScript packages that make web pages feel more responsive ƒ Parts of a page fetch data and update without the major re-load everyone is used to ƒ Used correctly, Ajax brings the interactivity of good desktop apps to the Web, such as a spell checker that checks on the fly like Outlook. ƒ Ajax can be chatty, so it needs a good network and thoughtful design

Rod Bodkin — AJAX Performance and Monitoring Page 4 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Other Options ƒ Fat Client ¾ Most interactive ¾ Hard to distribute, maintain ƒ HTML ¾ Easy to write, most portable ¾ Less interactive especially on WAN ƒ Flash ¾ Wide deployment (>95%)… still mostly used for animation ¾ Harder to develop, integrate ƒ ¾ applets had early promise, limited to a niche ƒ Microsoft XAML/Avalon ¾ Windows Vista only… ask me in 2009

Rod Bodkin — AJAX Performance and Monitoring Page 5 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Why AJAX? ƒ Rich interactivity ¾Responsiveness like a desktop app ¾Lively site ¾Ease of use ƒ Browser-based ¾Standards ¾Ubiquitous ¾Easy update ƒ Productive, usable, networked applications

Rod Bodkin — AJAX Performance and Monitoring Page 6 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Why Now? ƒ Technology matured ƒ Successful large-scale applications ƒ Emergence of tools and frameworks ƒ Even as IE went from 95% to 85% share

Rod Bodkin — AJAX Performance and Monitoring Page 7 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

New Interaction Patterns

Source: Adaptive Path

Rod Bodkin — AJAX Performance and Monitoring Page 8 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Agenda ƒ AJAX Overview ƒ AJAX Technology ƒ Challenges ƒ Solutions

Rod Bodkin — AJAX Performance and Monitoring Page 9 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Enabling Technologies ƒ XMLHttpRequest ¾ Allows asynchronous communication with server ¾ Generates events as data is received ƒ JavaScript ¾ Better portability across major browsers ¾ Still a lot of conditional logic required ƒ Dynamic updates to page ¾ DOM gives full object access ¾ innerHTML attribute for modifying HTML inside a tag

Rod Bodkin — AJAX Performance and Monitoring Page 10 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Javascript (Browser) Libraries ƒ dojo (widgets, packaging, utilities, persistence) ƒ Prototype (framework) ƒ script.aculo.us (effects, widgets) ƒ AjaxTK/Apache Kabuki - Zimbra (widgets) ƒ Yahoo! User Interface Library (widgets) ƒ TIBCO General Interface (framework & tools)

Rod Bodkin — AJAX Performance and Monitoring Page 11 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Browser + Server Frameworks ƒ Java ƒ .NET ¾ DWR (Direct Web ¾ Atlas Remoting) ¾ Ajax.NET ¾ Google Web Toolkit ¾ MagicAjax.NET ¾ ZK ƒ Ruby on Rails ¾ AjaxTags ƒ Python: Django ¾ Ajax4JSF ƒ Perl ƒ PHP ¾ Catalyst ¾ Symfony ¾ CGI::Ajax ¾ AjaxAC

Rod Bodkin — AJAX Performance and Monitoring Page 12 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Example: DWR 1.0 AJAX

Source: Getahead

Rod Bodkin — AJAX Performance and Monitoring Page 13 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation Reverse AJAX: Automatic Updates

Server Browser

Event

Update ? Script

Rod Bodkin — AJAX Performance and Monitoring Page 14 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Reverse AJAX: Polling

Server Browser

Event

Update Script

Rod Bodkin — AJAX Performance and Monitoring Page 15 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Reverse AJAX: Comet

Server Browser Start

Event

Update Script

Rod Bodkin — AJAX Performance and Monitoring Page 16 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Reverse AJAX: Piggy Back

Server Browser

Event

Request

Reply Script Update

Often discussed, but who is implementing it?

Rod Bodkin — AJAX Performance and Monitoring Page 17 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Widgets, e.g., Google Web Toolkit

Rod Bodkin — AJAX Performance and Monitoring Page 18 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Direct Service Integration

Web AJAX Mash-ups… page engine

Google Maps

Credit Agency

Order Service

Rod Bodkin — AJAX Performance and Monitoring Page 19 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Indirect Service Integration

Web AJAX page engine

Google Maps

Credit Agency

Aggregation Order server Service

Rod Bodkin — AJAX Performance and Monitoring Page 20 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Proxy Web Service Integration

Web AJAX page engine

Google Maps

Credit Agency

Proxy Order server Service

Rod Bodkin — AJAX Performance and Monitoring Page 21 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Agenda ƒ AJAX Overview ƒ AJAX Technology ƒ Challenges ƒ Solutions

Rod Bodkin — AJAX Performance and Monitoring Page 22 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

AJAX Problem Areas ƒ Excess server requests ƒ JavaScript problems ƒ Server load ƒ Service fragility ƒ Integration server problems

Rod Bodkin — AJAX Performance and Monitoring Page 23 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Excess Server Requests ƒ Adds complexity ¾ One single action can generate a flood of requests ¾ Or a handful of slower ones ¾ What happens if some are unreliable? ¾ One missing timeout can produce a huge backlog ƒ Hard to Monitor End User Performance ¾ Initial page load time isn’t the issue… ¾ Can’t determine cause of requests without page analysis ƒ Hard to correlate Requests with Pages let alone User Actions

Rod Bodkin — AJAX Performance and Monitoring Page 24 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

JavaScript Problems ƒ Not as fast as compiled languages! ƒ Processing large data sets is hard ƒ Engines can do a lot of work ƒ Browser portability is still an issue ¾ May require digging into framework code ƒ Stateful UI’s: what interactions led to errors? ¾ Debugging nested DOM/CSS/JS… ƒ Accessibility ƒ HTTP limit of 2 simultaneous connections per server by default (configurable)

Rod Bodkin — AJAX Performance and Monitoring Page 25 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Server Load: Pre-AJAX ƒ Short spikes of activity ƒ Long pauses of inactivity “think time” ƒ Efficient for use of server sockets and threads ƒ Can tax CPU ƒ Dynamic pages integrate big static chunks

Rod Bodkin — AJAX Performance and Monitoring Page 26 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Server Load: AJAX ƒ Initial download of engine (static content) ƒ Bursts of activity (change of context) ƒ With more frequent updates ƒ Typically shorter less intensive interactions ƒ Frequent requests and polling use network, CPU, sockets, and threads ƒ Comet instead uses much more sockets and typically threads and buffers ƒ More demanding of client too ƒ More static content

Rod Bodkin — AJAX Performance and Monitoring Page 27 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Service Fragility ƒ Each service is a possible point of failure ¾Outages ¾Configuration problems ¾Changes to service ƒ Cascading slowness possible ¾Not meeting SLA ¾Tests robustness/race conditions in browser

Rod Bodkin — AJAX Performance and Monitoring Page 28 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Integration Server Problems ƒ Slow query ƒ Too many queries ƒ Bottlenecks from aggregating many services ¾memory, sockets, … ƒ Contention for locks in server… ¾lots of threads for one session ƒ Outages/misconfiguration ƒ … Traditional problems in a new context

Rod Bodkin — AJAX Performance and Monitoring Page 29 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Typical Application Problems

Rod Bodkin — AJAX Performance and Monitoring Page 30 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Agenda ƒ AJAX Overview ƒ AJAX Technology ƒ Challenges ƒ Solutions

Rod Bodkin — AJAX Performance and Monitoring Page 31 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

The Performance Lifecycle

Operate • Monitor Define • Service level • Update

Design Test • Load test • Load test often architecture • Multi-platform • Create budget

Develop • Code & debug • Unit tests

Troubleshoot

Rod Bodkin — AJAX Performance and Monitoring Page 32 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Architecture Is Key ƒ AJAX shuffles the deck ¾ Changed workloads ¾ Emerging, immature technologies ƒ Allocate work sensibly ¾ Informs basic goals and SLA’s ƒ Budgets for both latency and scalability ¾ Interactive events (heavy & light) ¾ Update callbacks ƒ Prototype basic elements ¾ Many new technologies at play ƒ Extrapolate to model performance

Rod Bodkin — AJAX Performance and Monitoring Page 33 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Server Options ƒ The usual suspects… ¾ Java EE (Tomcat, JBoss, Jetty, GlassFish, WebSphere*, WebLogic*…) ¾ .NET for Windows* ¾ LAMP ƒ Emerging networking options ¾ Grizzly event-based HTTP Connector (in GlassFish) ¾ COMETd (Perl) ¾ Twisted (Python) ¾ Scala, Erlang/OTP (Concurrency Oriented Languages) ¾ Apache 2 Event MPM ¾ Continuation support…

* Indicates cost to use.

Rod Bodkin — AJAX Performance and Monitoring Page 34 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Emerging Standards? The Jetty open source Java Web container: ƒ Uses NIO to only use threads when processing I/O ƒ Server requests set up a continuation ¾ If waiting for data, they suspend by throwing ¾ When data is available, they are resumed ¾ The server replays the processing, but the next time they will continue ƒ Continuations allow using one thread per active request ƒ Also uses NIO split buffers to limit buffers for many requests ƒ DWR 2’s Reverse AJAX ¾ works with Jetty continuations ¾ Supports polling & COMET

Rod Bodkin — AJAX Performance and Monitoring Page 35 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Continuations ƒ Object to represent the state of program execution ¾ Native support in dynamic languages like Ruby, Smalltalk ¾ Java options rely on specialized state ƒ Jetty implementation supports AJAX scalability ¾ Requires no side effects in code ¾ Replays state based on request state & session state ¾ Doesn’t checkpoint session state… ƒ Other frameworks use to simplify conversations ¾ RIFE framework for Java ¾ Seaside framework for Smalltalk

Rod Bodkin — AJAX Performance and Monitoring Page 36 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

IDE Support ƒ ¾ATF (editor, debugger) ¾MyEclipse * (editor, code assist, debugger) ¾ * (code completion) ƒ IntelliJ IDEA* (refactoring editor)

* Indicates cost to use.

Rod Bodkin — AJAX Performance and Monitoring Page 37 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Browser Development Tools ƒ Debugging ¾ Mozilla Venkman debugger & profiler ¾ Microsoft Script Debugger for IE ƒ Inspectors ¾ WebDeveloper toolbar for Firefox: CSS, inspector ¾ FireBug console viewer, inspector, debugger ¾ Mouseover DOM Inspector ƒ Request monitoring ¾ FireBug ¾ GreaseMonkey: XMLHttpRequest Tracing/Debugging ¾ Eclipse ATF ƒ Logging: MochiKit, dojo… server-side?

* Indicates cost to use.

Rod Bodkin — AJAX Performance and Monitoring Page 38 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Venkman Profiling Output 33 util.js: 750 - 2500 milliseconds Function Name: anonymous (Lines 294 - 375) Total Calls: 6930 (max recurse 0) Total Time: 2093.01 (min/max/avg 0/10.02/0.3) Time (ex. calls): 1011.45 (min/max/avg 0/10.02/0.15)

35 troubleshooter.js: 750 - 2500 milliseconds Function Name: loadRowInfo (Lines 78 - 83) Total Calls: 99 (max recurse 0) Total Time: 2293.3 (min/max/avg 20.03/40.06/23.16) Time (ex. calls): 0 (min/max/avg 0/0/0)

Rod Bodkin — AJAX Performance and Monitoring Page 39 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Unit Test

ƒ Baseline test for functionality AND performance ƒ In browser ¾ Script.aculo.us, JsUnit, ASTUce ƒ JavaScript unit tests ¾ Rhino, MS Windows Script Host ¾ JavaScript Coverage Validator (beta)* ƒ On server ¾ JUnit, NUnit ¾ Java Coverage: Emma, Cobertura, Clover*

* Indicates cost to try or use.

Rod Bodkin — AJAX Performance and Monitoring Page 40 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

System Test ƒ Functional system tests (in browser) ¾Script recording and playback validation • Selenium, Squish/Web* ¾Script code objects (e.g., for JUnit) • Waitij/Waitir (Win/IE scripting for Java/Ruby) ƒ Load and Stress tests: simulate traffic ¾OpenSTA (http/s only… confused by gmail) ¾JMeter (http/s only) ¾LoadRunner (http/s, script objects too)* * Indicates cost to try or use.

Rod Bodkin — AJAX Performance and Monitoring Page 41 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Configuration ƒ Tune based on Load Test ƒ Network ¾ Load balancers, firewalls, routers ƒ OS level ¾ Maximum sockets ¾ Maximum threads ƒ Web server ¾ Timeouts ƒ Application server ¾ Threads ¾ Timeouts ƒ VM ¾ Memory settings (e.g., -Xss thread stack size)

Rod Bodkin — AJAX Performance and Monitoring Page 42 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Systems Monitoring ƒ Network-level ¾Router, sniffer metrics: throughput, latency ¾End-user response time (not just http pings!) ¾Customer Experience Management complicated: Keynote*, TeaLeaf* ƒ OS-level : sockets, IO, processes, CPU… ¾netstat, top, Nagios, Hyperic, HP Openview*, Tivoli*

* Indicates cost to try or use.

Rod Bodkin — AJAX Performance and Monitoring Page 43 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Using Fiddler: Mail Login & View Yahoo Mail Beta Gmail ƒ Request Count: 52 ƒ Request Count: 48 ƒ Bytes Sent: 61,069 ƒ Bytes Sent: 55,443 ƒ Bytes Received: 165,677 ƒ Bytes Received: 51,044 … … ƒ US West Coast (DSL - ƒ US West Coast (DSL - 30KB/sec) 30KB/sec) ƒ Round trip cost: 5.20s ƒ Round trip cost: 4.80s ƒ Elapsed Time: 12.20s ƒ Elapsed Time: 7.80s

Rod Bodkin — AJAX Performance and Monitoring Page 44 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

App Server Monitoring ƒ Container ¾Server JMX: request queue, pools, throughput ƒ Java VM with Java 5 JMX Data ¾All threads, memory ƒ Key elements ¾threads, memory, request queues, throughput, etc.

* Indicates cost to try or use.

Rod Bodkin — AJAX Performance and Monitoring Page 45 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

App Monitoring with Aspects ƒ Aspects run automatically at well-defined points at runtime ƒ No need to instrument code ƒ Allows low overhead tracking ƒ Easy to update monitoring policies ¾ Enable and disable, even sampling ƒ Standardized support ¾ AspectJ load-time weaving avoids changes to build process ¾ Popular extensible language for Java 5 javaagents ¾ Spring AOP allows proxy-based option for coarse-grained components ƒ Flexibility ¾ Reuse open source monitors for common APIs ¾ Easy to extend for custom monitoring

Rod Bodkin — AJAX Performance and Monitoring Page 46 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Glassbox Open Source

Browser Web ƒ Java 1.4+ Server HTTP ƒ Discovers and tracks operations as they execute Client Machine ƒ Load-time Java Application Application weaving ƒ Low overhead

Java Virtual AOP ƒ Detectsproblems common Agent Machine , e.g. Application Server Machine ¾ AJAX latency, load ¾ Excess queries ¾ Slow/broken Web services Web Any Database Services

Rod Bodkin — AJAX Performance and Monitoring Page 47 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation Troubleshooting Tools Provide

1. Aggressive data filtering can reduce the data haystack early

2. Troubleshooting intelligence means users need not be experts in the tool or have written the system

3. Cleaner data and more accurate diagnoses streamline communication Benefits ƒ Fix problems faster and cheaper by streamlining communication. ƒ Fix more problems by enabling non-experts to track them down.

Rod Bodkin — AJAX Performance and Monitoring Page 48 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Bringing It Together

DEMO

Rod Bodkin — AJAX Performance and Monitoring Page 49 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Conclusions ƒ AJAX is now ready for prime time ¾ Emphasize high-value interactivity ƒ Supporting technologies becoming mainstream ¾ But test carefully and watch for shifts ƒ Focus on architecture up front ¾ Benchmark latency and throughput ƒ Integrate monitoring and troubleshooting up front ƒ We’re looking for collaborators to build better AJAX monitoring & troubleshooting ƒ Please leave your business card to get updates ƒ These slides will be updated on the post-conference CD

Rod Bodkin — AJAX Performance and Monitoring Page 50 Colorado Software Summit: October 22 – 27, 2006 © Copyright 2006, Glassbox Corporation

Resources ƒ Ajaxian for News: www.ajaxian.com ƒ Ajax Patterns for Technologies: ajaxpatterns.org ƒ Ajax: A New Approach to Web Applications: http://adaptivepath.com/publications/essays/archives/000385.php ƒ Ajax Impact on Server Scaling: www.zimbra.com/blog/archives/2006/04/ajax_impact_on.html ƒ Scaling Connections for AJAX with Jetty 6: www.mortbay.com/MB/log/gregw/?permalink=ScalingConnections.html ƒ Not There Yet: COMET with Apache and Jetty: http://blogs.pathf.com/agileajax/2006/05/not_there_yet_c.html ƒ What I Didn’t Know About XHR: http://www.oreillynet.com/xml/blog/2006/10/what_i_didnt_know_about_xhr.html ƒ Glassbox: www.glassbox.com

Rod Bodkin — AJAX Performance and Monitoring Page 51