SCREAM Rob King Introduction The Problem The Base64 Static Compilation of Regular Expressions Algorithm Regular Expressions for Analysis and Modification The Algorithm Ways of Solving the Problem Encoding Operations Performance Rob King Expression Optimization Performance Analysis DVLabs Implementation TippingPoint Technologies and Usage Common Use Cases Caveats March 25, 2010 Summary OUTLINE SCREAM 1 Introduction Rob King The Problem Introduction The Base64 Algorithm The Problem The Base64 Regular Expressions Algorithm Regular Expressions 2 The Algorithm The Algorithm Ways of Solving the Ways of Solving the Problem Problem Encoding Operations Encoding Operations Performance Expression 3 Performance Optimization Performance Expression Optimization Analysis Implementation Performance Analysis and Usage Common Use Cases 4 Implementation and Usage Caveats Common Use Cases Summary Caveats 5 Summary THE PROBLEM In which we discover purpose of the whole thing... SCREAM Rob King This talk is about inspecting streams of data for Introduction interesting patterns, even when that stream of data has The Problem The Base64 been encoded. Algorithm Regular Expressions We focus on the Base64 encoding scheme, and The Algorithm Ways of Solving the discuss a tool that can be used when dealing with Problem Encoding Operations Base64. Performance Expression However, most portions of the algorithm are applicable Optimization Performance to other position-dependent bitwise block encodings Analysis (and, potentially, self-synchronizing encodings). Implementation and Usage We also want to talk about how Erlang made Common Use Cases Caveats development of the tool much easier. Summary This tool also provided an interesting “back door” to get Erlang accepted in DVLabs. THE PROBLEM In which we discover purpose of the whole thing... SCREAM Rob King This talk is about inspecting streams of data for Introduction interesting patterns, even when that stream of data has The Problem The Base64 been encoded. Algorithm Regular Expressions We focus on the Base64 encoding scheme, and The Algorithm Ways of Solving the discuss a tool that can be used when dealing with Problem Encoding Operations Base64. Performance Expression However, most portions of the algorithm are applicable Optimization Performance to other position-dependent bitwise block encodings Analysis (and, potentially, self-synchronizing encodings). Implementation and Usage We also want to talk about how Erlang made Common Use Cases Caveats development of the tool much easier. Summary This tool also provided an interesting “back door” to get Erlang accepted in DVLabs. THE PROBLEM In which we discover purpose of the whole thing... SCREAM Rob King This talk is about inspecting streams of data for Introduction interesting patterns, even when that stream of data has The Problem The Base64 been encoded. Algorithm Regular Expressions We focus on the Base64 encoding scheme, and The Algorithm Ways of Solving the discuss a tool that can be used when dealing with Problem Encoding Operations Base64. Performance Expression However, most portions of the algorithm are applicable Optimization Performance to other position-dependent bitwise block encodings Analysis (and, potentially, self-synchronizing encodings). Implementation and Usage We also want to talk about how Erlang made Common Use Cases Caveats development of the tool much easier. Summary This tool also provided an interesting “back door” to get Erlang accepted in DVLabs. THE PROBLEM In which we discover purpose of the whole thing... SCREAM Rob King This talk is about inspecting streams of data for Introduction interesting patterns, even when that stream of data has The Problem The Base64 been encoded. Algorithm Regular Expressions We focus on the Base64 encoding scheme, and The Algorithm Ways of Solving the discuss a tool that can be used when dealing with Problem Encoding Operations Base64. Performance Expression However, most portions of the algorithm are applicable Optimization Performance to other position-dependent bitwise block encodings Analysis (and, potentially, self-synchronizing encodings). Implementation and Usage We also want to talk about how Erlang made Common Use Cases Caveats development of the tool much easier. Summary This tool also provided an interesting “back door” to get Erlang accepted in DVLabs. THE PROBLEM In which we discover purpose of the whole thing... SCREAM Rob King This talk is about inspecting streams of data for Introduction interesting patterns, even when that stream of data has The Problem The Base64 been encoded. Algorithm Regular Expressions We focus on the Base64 encoding scheme, and The Algorithm Ways of Solving the discuss a tool that can be used when dealing with Problem Encoding Operations Base64. Performance Expression However, most portions of the algorithm are applicable Optimization Performance to other position-dependent bitwise block encodings Analysis (and, potentially, self-synchronizing encodings). Implementation and Usage We also want to talk about how Erlang made Common Use Cases Caveats development of the tool much easier. Summary This tool also provided an interesting “back door” to get Erlang accepted in DVLabs. THE PROBLEM In which we discover purpose of the whole thing... SCREAM Rob King This talk is about inspecting streams of data for Introduction interesting patterns, even when that stream of data has The Problem The Base64 been encoded. Algorithm Regular Expressions We focus on the Base64 encoding scheme, and The Algorithm Ways of Solving the discuss a tool that can be used when dealing with Problem Encoding Operations Base64. Performance Expression However, most portions of the algorithm are applicable Optimization Performance to other position-dependent bitwise block encodings Analysis (and, potentially, self-synchronizing encodings). Implementation and Usage We also want to talk about how Erlang made Common Use Cases Caveats development of the tool much easier. Summary This tool also provided an interesting “back door” to get Erlang accepted in DVLabs. CHALLENGES OF DATA STREAM INSPECTION SCREAM Rob King Introduction The Problem The Base64 Algorithm Regular Expressions When looking for patterns in streams of data, several things The Algorithm must be kept in mind: Ways of Solving the Problem Encoding Operations There is no “luxury of time”. Performance Context is limited. Expression Optimization Performance Resources are limited. Analysis Implementation and Usage Common Use Cases Caveats Summary CHALLENGES OF DATA STREAM INSPECTION SCREAM Rob King Introduction The Problem The Base64 Algorithm Regular Expressions When looking for patterns in streams of data, several things The Algorithm must be kept in mind: Ways of Solving the Problem Encoding Operations There is no “luxury of time”. Performance Context is limited. Expression Optimization Performance Resources are limited. Analysis Implementation and Usage Common Use Cases Caveats Summary CHALLENGES OF DATA STREAM INSPECTION SCREAM Rob King Introduction The Problem The Base64 Algorithm Regular Expressions When looking for patterns in streams of data, several things The Algorithm must be kept in mind: Ways of Solving the Problem Encoding Operations There is no “luxury of time”. Performance Context is limited. Expression Optimization Performance Resources are limited. Analysis Implementation and Usage Common Use Cases Caveats Summary CHALLENGES OF DATA STREAM INSPECTION SCREAM Rob King Introduction The Problem The Base64 Algorithm Regular Expressions When looking for patterns in streams of data, several things The Algorithm must be kept in mind: Ways of Solving the Problem Encoding Operations There is no “luxury of time”. Performance Context is limited. Expression Optimization Performance Resources are limited. Analysis Implementation and Usage Common Use Cases Caveats Summary ENCODED STREAMS SCREAM Rob King Introduction The Problem The Base64 Algorithm Regular Expressions The Algorithm Sometimes, the streams we’re inspecting will be Ways of Solving the Problem encoded. Encoding Operations Performance This means that we’re going to have to be (more!) Expression Optimization clever when looking for patterns in these streams. Performance Analysis Implementation and Usage Common Use Cases Caveats Summary DEALING WITH ENCODED STREAMS SCREAM Rob King Introduction The Problem The Base64 Algorithm Regular Expressions The Algorithm Ways of Solving the Problem There are several general strategies for dealing with Encoding Operations encoded streams. Performance Expression Optimization Performance Analysis Implementation and Usage Common Use Cases Caveats Summary DEALING WITH ENCODED STREAMS Strategy 1: Ignore the Encoding SCREAM Rob King Introduction The Problem The easiest thing to do is simply pretend the stream is The Base64 Algorithm Regular Expressions not encoded at all. The Algorithm Advantages: Ways of Solving the Problem We’re already done. Encoding Operations Performance Disadvantages: Expression Optimization We’re essentially admitting defeat. Performance Analysis Whatever we were looking for is not going to be found. Implementation We’re stil burdening our analysis engine with lots of and Usage Common Use Cases data with which we can do nothing. Caveats Summary DEALING WITH ENCODED STREAMS Strategy 1: Ignore the Encoding SCREAM Rob King Introduction The Problem The easiest thing to do is simply pretend the stream is The Base64 Algorithm Regular Expressions not encoded at all. The Algorithm Advantages: Ways of Solving the Problem We’re