Magpie: Distributed request tracking for realistic performance modelling
Rebecca Isaacs Paul Barham Richard Mortier Dushyanth Narayanan Microsoft Research Cambridge
James Bulpin University of Cambridge
12 November 2003 Performance in distributed systems
l Faults in distributed systems are notoriously hard to diagnose
l Performance problems are even more subtle to debug l Often transient or affect only a subset of requests / users l Frequently involve complex interactions between multiple machines l Aggregate statistics (e.g. utilization) may look perfectly normal
12 November 2003 Magpie Approach l Track individual requests end to end l Observe control flow (causality) l Monitor resource consumption: CPU, bandwidth, disk l Debug performance “in the small” l Build a probabilistic workload model from the aggregate requests l Cluster similar requests according to their observed behaviour l Debug performance “in the large”
12 November 2003 How do we use this information? l Performance debugging l Why did this request take much longer than that request? l Fault detection l Configuration and management l Performance prediction
l Realistic workload models for capacity planning l Obtain automatically on a “live” system
12 November 2003 Magpie components l Instrumentation l System activity recorded to logs l Generic request parser l Extract individual requests from logs according to an event schema l Model construction l Behavioural clusters l Probabilistic state machine
12 November 2003 Outline l Introduction l What is a request? l Instrumentation l Request extraction l Modelling l Current status
12 November 2003 What is a request? l System activity which takes place in response to an action initiated by the application being traced l HTTP request l Database query l File open request l We describe a request as l The sequence of application components involved in its processing l The resource consumed at each stage l CPU, bandwidth, disk transfer size, (latency)
12 November 2003 A typical e-commerce site (1)
Internet
Storage SQL Servers Web Front Ends
12 November 2003 A typical e-commerce site (2)
Web Server SQL Server IIS CLR Application Static Logic Stored ContentFilter procedures
ASP.NET ADO.NET Data
WinSock2 API WinSock2 API http.sys Kernel Kernel
12 November 2003 HTTP request: detailed view
IIS worker thread picks up request ASP.NET thread blocks after from http.sys Sync WinSock send IIS worker thread RPC to database to SQL Server wakes up to write log !
WEB.eec - + + + - - - + - - -
WEB.398 HTTP request Disk HTTP response packets packet ASP.NET worker TDS request and reply packets sent and sent back to client Net RX thread takes over received Net TX
10.051s 10.100s 10.155s
Net TX
Net RX
Disk - -
SQL thread - SQL.9c4 unblocks
10.051s 10.100s 10.155s
KEY: Blocked IIS ASP.NET SQL Disk Other 12 November 2003 Why is request tracking hard? l Many components, multiple machines l Must track control flow across machines l No globally unique request ID l Components are developed independently l Multiple thread pools l Many threads participate in processing a request l Asynchronous communication l Must match send/recvs between threads/machines l Hand-rolled synchronization primitives l SQL server has user-mode scheduler
12 November 2003 Outline l Introduction l What is a request? l Instrumentation l Request extraction l Modelling l Current status
12 November 2003 Event Tracing for Windows l Low-overhead event mechanism l Events timestamped with cycle counter l Global ordering on events on a single machine l Can enable/disable sets of events at runtime l Using ETW in Magpie l Each instrumentation point posts an event l Events are logged to disk l Logs are post-processed to extract requests l Can also consume events in real time
12 November 2003 Instrumentation points l Existing ETW event providers l IIS, kernel l App-specific hooks l IIS, ASP.NET, SQL Server l Detours l Wrap dlls to trap Win32 and WinSock2 calls l WinPcap l Capture packets on the wire
12 November 2003 CPU usage from kernel events l The ETW kernel logger records every context switch l How do we know which cycles are used for which request? l We can attribute cycles to a request by l An application-specific event which occurs within a delimited sector of CPU time, or l The current context of execution, eg thread id
12 November 2003 Example: protocol processing in a DPC
DPC pkt DPC Events: cswitch start recv end cswitch
Request 1 time cycle count
Request 2 cycle count
12 November 2003 Application and middleware events l Cover points where flow of control moves between components l Cover points where resources are multiplexed and demultiplexed l E.g. user-level scheduling primitives l Propagation of a global request id is not required! l Magpie used to do this but not any more
12 November 2003 Instrumenting a web service
Web Server SQL Server IIS CLR Application Extended SPs Static Logic Stored ContentFilter procedures
HTTPModule Wrappers
ASP.NET ADO.NET
ISAPI Filter CLR profiler Data
Intercept Intercept WinSock2 API WinSock2 API http.sys Kernel Kernel
Event Tracing for Windows Event Tracing for Windows Packet Packet capture capture
12 November 2003 Outline l Introduction l What is a request? l Instrumentation l Request extraction l Modelling l Current status
12 November 2003 Generic request extraction l No inbuilt assumptions about the system or the application l No common unique identifier l Schema specifies semantics of events l Easy to add new event types l Parser stitches events into requests based on event semantics
12 November 2003 Terminology l Namespace l Event parameter which references an entity in the system, eg thread id l Timeline l Instantiation of a namespace with a unique value, eg thread id = 0xa l Events bind or unbind requests to timelines l Bindings capture the semantics of each event for a particular request type
12 November 2003 Example: connecting eventsRecv returns Enter Recv DPC start DPC end TCP pkt cswitch cswitch
Cpuid=0
Tid=0xa
Tid=0xb
Connid=0xd
Request 1 12 November 2003 Request 2 End-to-end request extraction l An instance of the request parser runs on each machine in the distributed system l Online or offline mode l Offline post-processing connects request fragments from each node according to a globally unique namespace, e.g. packet IP identifier
12 November 2003 Outline l Introduction l What is a request? l Instrumentation l Request extraction l Modelling l Current status
12 November 2003 Clustering for workload generation l Target the Indy performance modelling tool l Calculates throughput, bottlenecks l Needs transaction mix, resource consumption l Previously: microbenchmark approach l Run 10000 of each “transaction type” (URL) l Divide aggregate resource usage by 10000 l Aim: provide realistic workload models l From real, mixed workloads l Derive transaction “types” automatically
12 November 2003 Single request: cartoon view l Partial ordering of events l Annotated with resource usage
1k 12k
5ms 6ms 3ms 24k 6ms 1ms 6ms read
1k 6k
2ms 192k 3ms read
IIS CPU ASP.NET CPU SQL Server CPU
Network Disk 12 November 2003 Behavioural clustering of requests l Represent requests as event strings l “Flatten” out any concurrency l Use Levenshtein string edit distance l Modified to factor in resource usage vectors l Cluster requests based on this distance l Linear-time algorithm l Each cluster is a request “type” l Select representative from near centroid
12 November 2003 Build a workload model by clustering similar requests
1k 30k
2ms 10ms 5ms 14ms 24ms 1ms 5ms
0.2k 0.2k 0.2k 0.1k 6k 2k
A 5ms 11ms 5ms Requests in the same 7%
1k cluster often have 11k 14ms 27ms 2ms 1ms 2ms 7ms different URLs, and B 10% one URL may appear A 1k B 12k 5ms 6ms 3ms 24k 6ms 1ms 6ms in many clusters read 1k C 6k 2ms 192k 3ms C read E 15%
D 1k 11k
2ms 13ms 11ms 3ms 2ms 5ms
0.3k D 0.3k 5% 5ms
0.6k 1k E 5ms 11ms 12 November 2003 63% Taking it further: work-in- progress l Online and incremental modelling: l Detect component failure l Detect sudden shifts in workload l More sophisticated models l Learn the probabilistic state machine for each request l c.f. flowcharts annotated with performance information l “Bayesian watchdogs” l Compute the likelihood of a request’s behaviour as it moves through the system l Deal with “unlikely” requests appropriately
12 November 2003 Outline l Introduction l What is a request? l Instrumentation l Request extraction l Modelling l Current status
12 November 2003 Current status l Recent focus has been developing a generic request extraction scheme l Prototype for 2-machine e-commerce site l TPC-W style workload l Prototype for single machine SQL Server 2000 l Challenge is user mode scheduler l TPC-C workload l Other applications on the way l Large-scale l “Real” systems with “real” performance problems
12 November 2003 Conclusion l Magpie is a tool for performance analysis in a distributed system l Bottom up, per-request approach l Complementary to existing techniques: l Performance counters l Program profiling l Feeds into performance debugging and prediction tools
12 November 2003 Work-in-progress: learning the probabilistic state machine l Infer a stochastic context free grammar from a sample set of strings l Each state transition emits a character and has an associated probability l Use the Alergia algorithm (Carrasco & Oncina ‘94) l Construct a prefix tree from the sample set l Merge similar subtrees l Apply to Magpie requests l “Just” event strings…
12 November 2003 Ongoing work with Alergia
l Tuning the similarity criterion l Factoring in resource usage information l Can we identify event sequences with suspiciously low probability l Run online for anomaly detection?
12 November 2003