synapse Documentation Release 0.1.0 The Vertex Project Feb 12, 2019 Contents: 1 Synapse User Guide 1 1.1 Background - Data Model and Terminology...............................1 1.2 Background - Storm Query Language..................................1 1.3 Synapse Tools - cmdr..........................................1 1.4 Synapse Tools - feed...........................................4 1.5 Synapse Tools - csvtool.........................................4 1.6 Synapse - Permissions..........................................4 1.7 Synapse - Triggers............................................4 1.8 Synapse - Layers.............................................4 1.9 Synapse Reference - Synapse Commands................................4 1.10 Storm Reference - Introduction..................................... 17 1.11 Storm Reference - Document Syntax Conventions........................... 19 1.12 Storm Reference - Lifting........................................ 20 1.13 Storm Reference - Filtering....................................... 27 1.14 Storm Reference - Pivoting....................................... 37 1.15 Storm Reference - Data Modification.................................. 45 1.16 Storm Reference - Subquery Syntax................................... 51 1.17 Storm Reference - Variables....................................... 51 1.18 Storm Reference - Type-Specific Storm Behavior............................ 51 1.19 Storm Reference - Storm Commands.................................. 51 2 Synapse DevOps Guide 65 2.1 Synapse Performance - 1......................................... 65 2.2 Synapse Performance - 2......................................... 69 2.3 Synapse Performance - 3......................................... 73 3 Synapse Contributors Guide 79 3.1 Contributing to Synapse......................................... 79 3.2 Synapse Doc Mastering......................................... 86 3.3 Synapse Release Process......................................... 91 4 Synapse Data Model - Types 95 4.1 Base Types................................................ 95 4.2 Types................................................... 100 5 Synapse Data Model - Forms 131 5.1 Forms................................................... 131 i 5.2 Universal Properties........................................... 217 6 synapse 219 6.1 synapse package............................................. 219 7 Indices and tables 375 Python Module Index 377 ii CHAPTER 1 Synapse User Guide This User Guide is written by and for Synapse users and is intended to provide a general overview of Synapse con- cepts and operations. Technical documentation appropriate for Synapse deployment and development can be found elsewhere in the Document Index. The User Guide is a living document and will continue to be updated and expanded as appropriate. The current sections are: 1.1 Background - Data Model and Terminology TBD 1.2 Background - Storm Query Language TBD 1.3 Synapse Tools - cmdr The Synapse command line interface (CLI) is a text-based interpreter used to communicate with a Synapse Cortex. The Synapse cmdr module is a command line tool used to connect and provide an interactive CLI to an existing local or remote Cortex. This section will cover the following Synapse CLI topics: • Obtaining a Command Line Interface • Command Line Interface Basics See the Synapse Reference - Synapse Commands for a list of available Synapse commands. 1 synapse Documentation, Release 0.1.0 1.3.1 Obtaining a Command Line Interface In order to obtain access to the Synapse CLI you must use the cmdr module connected to a local or remote Cortex. If you have access to an existing local or remote Cortex, proceed to Connecting to an Existing Cortex for instructions on how to connect to the Cortex. However, if you do not have access to an existing Cortex, proceed to Connecting to a Temporary Cortex for instructions on creating and connecting to a temporary Cortex on your local machine. Connecting to an Existing Cortex To connect to an existing local or remote Cortex, run the Synapse cmdr module by executing the following Python command from a terminal window, where the <url> parameter is the URL path to the Cortex. python -m synapse.tools.cmdr <url> The URL in the above usage statement is the path to the Cortex, and has the following format: <scheme>://<server>:<port>/<cortex> Example URL paths: • tcp://synapse.woot.com:1234/cortex01 • ssl://synapse.woot.com:1234/cortex01 Once connected the Cortex, you will be presented with the following Synapse CLI command prompt: cli> Connecting to a Temporary Cortex In the event that you do not have access to an existing Cortex, you can optionally use the Synapse feed module (Synapse Tools - feed) to access the CLI. The feed module is a command line tool that allows you to ingest data into a Cortex. However, it can also be used to create a temporary local Cortex for testing and debugging. To create and connect to a temporary local Cortex using the feed module, execute the following Python command from a terminal window: python -m synapse.tools.feed --test --debug Once connected the Cortex, you will be presented with the following Synapse CLI command prompt: cli> 1.3.2 Command Line Interface Basics Before we delve into Synapse commands, let’s discuss Synapse CLI command parsing and syntax conventions. This section will cover: • Using Whitespace Characters • Entering Literals Using Whitespace Characters Whitespace characters (i.e., space) are used within the Synapse CLI to delimit command line arguments. Specifically, whitespace characters are used to separate CLI commands, command arguments, command operators, variables and literals. 2 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0 Quotation marks are used to preserve whitespace characters in literals entered during variable assignment and compar- ison. If quotation marks are not used to quote whitespace characters, the whitespace characters will be used to delimit command line arguments. When entering a query/command on the Synapse CLI, one or more whitespace characters are required between the following command line arguments: • A command and command line parameters: cli> log --off cli> storm inet:fqdn=vertex.link inet:email=support@vertex. link • An unquoted literal and any subsequent CLI argument: cli> storm inet:[email protected] | count cli> storm inet:[email protected] -> * Whitespace characters can optionally be used when performing the following CLI operations: • Assignment operations using the equals sign assignment operator: cli> storm [inet:ipv4=192.168.0.1] cli> storm [inet:ipv4 = 192.168.0.1] • Comparison operations: cli> storm inet:ipv4=192.168.0.1 cli> storm inet:ipv4 = 192.168.0.1 • Pivot operations: cli> inet:ipv4 -> * cli> inet:ipv4->* Whitespace characters cannot be used between reserved characters when performing the following CLI operations: • Add and remove tag operations. The plus ( + ) and minus ( - ) sign characters are used to add and remove tags to and from nodes in the graph respectively. When performing tag operations using these characters, a whitespace character cannot be used between the actual character and the tag name (e.g., +#<tag>). cli> storm inet:ipv4 = 192.168.0.1 [-#oldtag +#newtag] Entering Literals Single ( '' ) or double ( "" ) quotation marks can be used when entering a literal on the CLI during an assignment or comparison operation. Enclosing a literal in quotation marks is required when the literal: • begins with a non-alphanumeric character, • contains a space ( \s ), tab ( \t ) or newline( \n ) character, or • contains a reserved Synapse character (e.g., \ ) , = ] } |). Enclosing a literal in single quotation marks will preserve the literal meaning of each character. Enclosing literals in double quotation marks will preserve the literal meaning of all characters except for the backslash ( \ ) character. The commands below demonstrate assignment and comparison operations that do not require quotation marks: • Lifting the domain vtx.lk: cli> storm inet:fqdn = vtx.lk • Lifting the file name windowsupdate.exe: cli> storm file:base = windowsupdate.exe The commands below demonstrate assignment and comparison operations that require the use of quotation marks. Failing to enclose the literals below in quotation marks will results in a syntax exception. 1.3. Synapse Tools - cmdr 3 synapse Documentation, Release 0.1.0 • Lift the file name windows update.exe which contains a whitespace character: cli> storm file:base = "windows update.exe" • Lift the file name windows,update.exe which contains the comma special character: cli> storm file:base = "windows,update.exe" 1.4 Synapse Tools - feed TBD 1.5 Synapse Tools - csvtool TBD 1.6 Synapse - Permissions TBD 1.7 Synapse - Triggers TBD 1.8 Synapse - Layers TBD 1.9 Synapse Reference - Synapse Commands The Synapse CLI contains a set of built-in commands that can be used to interact with a Synapse Cortex. This section details the usage for each built-in Synapse command. See Synapse Tools - cmdr for background on using cmdr and interacting with the Synapse CLI. The following Synapse commands are currently supported: • help • at • cron • kill • locs • log • ps 4 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0 • quit • storm • trigger 1.9.1 help The help command displays the list of available built-in commands and a brief message describing each command. Help on individual commands is available via help <command>. Syntax: cli> help at - Adds a non-recurring cron job. cron - Manages cron