sign in sign up
<<
Home , Mobile Country Code

synapse Documentation Release 0.1.0

The Vertex Project

Feb 12, 2019

Contents:

1 Synapse User Guide 1 1.1 Background - Data Model and Terminology...... 1 1.2 Background - Storm Query Language...... 1 1.3 Synapse Tools - cmdr...... 1 1.4 Synapse Tools - feed...... 4 1.5 Synapse Tools - csvtool...... 4 1.6 Synapse - Permissions...... 4 1.7 Synapse - Triggers...... 4 1.8 Synapse - Layers...... 4 1.9 Synapse Reference - Synapse Commands...... 4 1.10 Storm Reference - Introduction...... 17 1.11 Storm Reference - Document Syntax Conventions...... 19 1.12 Storm Reference - Lifting...... 20 1.13 Storm Reference - Filtering...... 27 1.14 Storm Reference - Pivoting...... 37 1.15 Storm Reference - Data Modification...... 45 1.16 Storm Reference - Subquery Syntax...... 51 1.17 Storm Reference - Variables...... 51 1.18 Storm Reference - Type-Specific Storm Behavior...... 51 1.19 Storm Reference - Storm Commands...... 51

2 Synapse DevOps Guide 65 2.1 Synapse Performance - 1...... 65 2.2 Synapse Performance - 2...... 69 2.3 Synapse Performance - 3...... 73

3 Synapse Contributors Guide 79 3.1 Contributing to Synapse...... 79 3.2 Synapse Doc Mastering...... 86 3.3 Synapse Release Process...... 91

4 Synapse Data Model - Types 95 4.1 Base Types...... 95 4.2 Types...... 100

5 Synapse Data Model - Forms 131 5.1 Forms...... 131

i 5.2 Universal Properties...... 217

6 synapse 219 6.1 synapse package...... 219

7 Indices and tables 375

Python Module Index 377

ii CHAPTER 1

Synapse User Guide

This User Guide is written by and for Synapse users and is intended to provide a general overview of Synapse con- cepts and operations. Technical documentation appropriate for Synapse deployment and development can be found elsewhere in the Document Index. The User Guide is a living document and will continue to be updated and expanded as appropriate. The current sections are:

1.1 Background - Data Model and Terminology

TBD

1.2 Background - Storm Query Language

TBD

1.3 Synapse Tools - cmdr

The Synapse command line interface (CLI) is a text-based interpreter used to communicate with a Synapse Cortex. The Synapse cmdr module is a command line tool used to connect and provide an interactive CLI to an existing local or remote Cortex. This section will cover the following Synapse CLI topics: • Obtaining a Command Line Interface • Command Line Interface Basics See the Synapse Reference - Synapse Commands for a list of available Synapse commands.

1 synapse Documentation, Release 0.1.0

1.3.1 Obtaining a Command Line Interface

In order to obtain access to the Synapse CLI you must use the cmdr module connected to a local or remote Cortex. If you have access to an existing local or remote Cortex, proceed to Connecting to an Existing Cortex for instructions on how to connect to the Cortex. However, if you do not have access to an existing Cortex, proceed to Connecting to a Temporary Cortex for instructions on creating and connecting to a temporary Cortex on your local machine.

Connecting to an Existing Cortex

To connect to an existing local or remote Cortex, run the Synapse cmdr module by executing the following Python command from a terminal window, where the parameter is the URL path to the Cortex. python -m synapse.tools.cmdr The URL in the above usage statement is the path to the Cortex, and has the following format: ://:/ Example URL paths: • tcp://synapse.woot.com:1234/cortex01 • ssl://synapse.woot.com:1234/cortex01 Once connected the Cortex, you will be presented with the following Synapse CLI command prompt: cli>

Connecting to a Temporary Cortex

In the event that you do not have access to an existing Cortex, you can optionally use the Synapse feed module (Synapse Tools - feed) to access the CLI. The feed module is a command line tool that allows you to ingest data into a Cortex. However, it can also be used to create a temporary local Cortex for testing and debugging. To create and connect to a temporary local Cortex using the feed module, execute the following Python command from a terminal window: python -m synapse.tools.feed --test --debug Once connected the Cortex, you will be presented with the following Synapse CLI command prompt: cli>

1.3.2 Command Line Interface Basics

Before we delve into Synapse commands, let’s discuss Synapse CLI command parsing and syntax conventions. This section will cover: • Using Whitespace Characters • Entering Literals

Using Whitespace Characters

Whitespace characters (i.e., space) are used within the Synapse CLI to delimit command line arguments. Specifically, whitespace characters are used to separate CLI commands, command arguments, command operators, variables and literals.

2 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

Quotation marks are used to preserve whitespace characters in literals entered during variable assignment and compar- ison. If quotation marks are not used to quote whitespace characters, the whitespace characters will be used to delimit command line arguments. When entering a query/command on the Synapse CLI, one or more whitespace characters are required between the following command line arguments: • A command and command line parameters: cli> log --off cli> storm inet:fqdn=vertex.link inet:email=support@vertex. link • An unquoted literal and any subsequent CLI argument: cli> storm inet:[email protected] | count cli> storm inet:[email protected] -> * Whitespace characters can optionally be used when performing the following CLI operations: • Assignment operations using the equals sign assignment operator: cli> storm [inet:ipv4=192.168.0.1] cli> storm [inet:ipv4 = 192.168.0.1] • Comparison operations: cli> storm inet:ipv4=192.168.0.1 cli> storm inet:ipv4 = 192.168.0.1 • Pivot operations: cli> inet:ipv4 -> * cli> inet:ipv4->* Whitespace characters cannot be used between reserved characters when performing the following CLI operations: • Add and remove tag operations. The plus ( + ) and minus ( - ) sign characters are used to add and remove tags to and from nodes in the graph respectively. When performing tag operations using these characters, a whitespace character cannot be used between the actual character and the tag name (e.g., +#). cli> storm inet:ipv4 = 192.168.0.1 [-#oldtag +#newtag]

Entering Literals

Single ( '' ) or double ( "" ) quotation marks can be used when entering a literal on the CLI during an assignment or comparison operation. Enclosing a literal in quotation marks is required when the literal: • begins with a non-alphanumeric character, • contains a space ( \s ), tab ( \t ) or newline( \n ) character, or • contains a reserved Synapse character (e.g., \ ) , = ] } |). Enclosing a literal in single quotation marks will preserve the literal meaning of each character. Enclosing literals in double quotation marks will preserve the literal meaning of all characters except for the backslash ( \ ) character. The commands below demonstrate assignment and comparison operations that do not require quotation marks: • Lifting the domain vtx.lk: cli> storm inet:fqdn = vtx.lk • Lifting the file name windowsupdate.exe: cli> storm file:base = windowsupdate.exe The commands below demonstrate assignment and comparison operations that require the use of quotation marks. Failing to enclose the literals below in quotation marks will results in a syntax exception.

1.3. Synapse Tools - cmdr 3 synapse Documentation, Release 0.1.0

• Lift the file name windows update.exe which contains a whitespace character: cli> storm file:base = "windows update.exe" • Lift the file name windows,update.exe which contains the comma special character: cli> storm file:base = "windows,update.exe"

1.4 Synapse Tools - feed

TBD

1.5 Synapse Tools - csvtool

TBD

1.6 Synapse - Permissions

TBD

1.7 Synapse - Triggers

TBD

1.8 Synapse - Layers

TBD

1.9 Synapse Reference - Synapse Commands

The Synapse CLI contains a set of built-in commands that can be used to interact with a Synapse Cortex. This section details the usage for each built-in Synapse command. See Synapse Tools - cmdr for background on using cmdr and interacting with the Synapse CLI. The following Synapse commands are currently supported: • help • at • cron • kill • locs • log • ps

4 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

• quit • storm • trigger

1.9.1 help

The help command displays the list of available built-in commands and a brief message describing each command. Help on individual commands is available via help . Syntax: cli> help at - Adds a non-recurring cron job. cron - Manages cron jobs in a cortex. help - List commands and display help output. hive - Manipulates values in a cell's Hive. kill - Kill a running task/query within the cortex. locs - List the current locals for a given CLI object. log - Add a storm log to the local command session. ps - List running tasks in the cortex. quit - Quit the current command line interpreter. storm - Execute a storm query. trigger - Manipulate triggers in a cortex.

1.9.2 at

The at command allows you to schedule a storm query to execute within a Cortex at one or more specified times. Once created, tasks / queries scheduled with at are managed using the cron command. At jobs, like cron jobs, remain in a Cortex until explicitly removed. Syntax: cli> help at === at

Adds a non-recurring cron job.

It will execute a Storm query at one or more specified times.

List/details/deleting cron jobs created with 'at' use the same commands as other cron jobs: cron list/stat/del respectively.

Syntax: at (time|+time delta)+ {query}

Notes: This command accepts one or more time specifications followed by exactly one storm query in curly braces. Each time specification may be in synapse time delta format (e.g + 1 day) or synapse time format (e.g. 20501217030432101). Seconds will be ignored, as cron jobs' granularity is limited to minutes.

All times are interpreted as UTC.

(continues on next page)

1.9. Synapse Reference - Synapse Commands 5 synapse Documentation, Release 0.1.0

(continued from previous page) The other option for time specification is a relative time from now. This consists of a plus sign, a positive integer, then one of 'minutes, hours, days'.

Note that the record for a cron job is stored until explicitly deleted via "cron del".

Examples: # Run a storm query in 5 minutes at +5 minutes {[inet:ipv4=1]}

# Run a storm query tomorrow and in a week at +1 day +7 days {[inet:ipv4=1]}

# Run a query at the end of the year Zulu at 20181231Z2359 {[inet:ipv4=1]}

Example: TBD

1.9.3 cron

The cron command allows you to schedule a storm query to execute within a Cortex on a recurring basis. cron has multiple subcommands, including: • cron help • cron add • cron list • cron stat • cron mod • cron del Syntax: cli> help cron === cron

Manages cron jobs in a cortex.

Cron jobs are rules persistently stored in a cortex such that storm queries automatically run on a time schedule.

Cron jobs may be be recurring or one-time. Use the 'at' command to add one-time jobs.

A subcommand is required. Use 'cron -h' for more detailed help. cron help cron includes detailed help describing its individual subcommands. Syntax:

6 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

cli> cron -h usage: cron [-h] {list,add,del,stat,mod} ...

Manages cron jobs in a cortex.

Cron jobs are rules persistently stored in a cortex such that storm queries automatically run on a time schedule.

Cron jobs may be be recurring or one-time. Use the 'at' command to add one-time jobs.

A subcommand is required. Use 'cron -h' for more detailed help. optional arguments: -h, --help show this help message and exit subcommands: {list,add,del,stat,mod} list List cron jobs you're allowed to manipulate add add a cron job del delete a cron job stat details a cron job mod change an existing cron jobquery cron add cron add adds a cron job to a Cortex. Syntax: cli> cron add -h usage: Add a recurring cron job to a cortex.

Syntax: cron add [optional arguments] {query}

--minute, -M int[,int...][=] --hour, -H --day, -d --month, -m --year, -y

or:

[--hourly | --daily : | --monthly :: | --yearly :::]

Notes: All times are interpreted as UTC.

All arguments are interpreted as the job period, unless the value ends in

1.9. Synapse Reference - Synapse Commands 7 synapse Documentation, Release 0.1.0

an equals sign, in which case the argument is interpreted as the

˓→recurrence period. Only one recurrence period parameter may be specified.

Currently, a fixed unit must not be larger than a specified recurrence period. i.e. '--hour 7 --minute +15' (every 15 minutes from 7-8am?) is

˓→not supported.

Value values for fixed hours are 0-23 on a 24-hour clock where midnight

˓→is 0.

If the --day parameter value does not start with in '+' and is an integer,

˓→ it is interpreted as a fixed day of the month. A negative integer may be specified to count from the end of the month with -1 meaning the last day of the month. All fixed day values are clamped to valid days, so for example '-d 31' will run on February 28.

If the fixed day parameter is a value in ([Mon, Tue, Wed, Thu, Fri, Sat, Sun] if locale is set to English) it is interpreted as a fixed day of the week.

Otherwise, if the parameter value starts with a '+', then it is

˓→interpreted as an recurrence interval of that many days.

If no plus-sign-starting parameter is specified, the recurrence period defaults to the unit larger than all the fixed parameters. e.g. '-M 5' means every hour at 5 minutes past, and -H 3, -M 1 means 3:01 every day.

At least one optional parameter must be provided.

All parameters accept multiple comma-separated values. If multiple parameters have multiple values, all combinations of those values are

˓→used.

All fixed units not specified lower than the recurrence period default to the lowest valid value, e.g. -m +2 will be scheduled at 12:00am the first

˓→of every other month. One exception is the largest fixed value is day of the week, then the default period is set to be a week.

A month period with a day of week fixed value is not currently supported.

Fixed-value year (i.e. --year 2019) is not supported. See the 'at' command for one-time cron jobs.

As an alternative to the above options, one may use exactly one of --hourly, --daily, --monthly, --yearly with a colon-separated list of fixed parameters for the value. It is an error to use both the individual options and these aliases at the same time.

Examples:

8 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

Run a query every last day of the month at 3 am cron add -H 3 -d-1 {#foo}

Run a query every 8 hours cron add -H +8 {#foo}

Run a query every Wednesday and Sunday at midnight and noon cron add -H 0,12 -d Wed,Sun {#foo}

Run a query every other day at 3:57pm cron add -d +2 -M 57 -H 15 {#foo} positional arguments: query Storm query in curly braces optional arguments: -h, --help show this help message and exit --minute MINUTE, -M MINUTE --hour HOUR, -H HOUR --day DAY, -d DAY day of week, day of month or number of days --month MONTH, -m MONTH --year YEAR, -y YEAR --hourly HOURLY --daily DAILY --monthly MONTHLY --yearly YEARLY Example: TBD cron list cron list lists existing cron jobs in a Cortex that the current user can view / modify based on their permissions. Syntax: cli> cron list -h usage: List existing cron jobs in a cortex.

Syntax: cron list

Example: cli> cron list user recurs? now? # start last start last end query 4ad2218a.. N N 1 2018-12-14T15:53 2018-12-14T15:53 #foo f6b6aebd.. Y N 3 2018-12-14T16:25 2018-12-14T16:25 #foo optional arguments: -h, --help show this help message and exit

Example: TBD

1.9. Synapse Reference - Synapse Commands 9 synapse Documentation, Release 0.1.0 cron stat cron stat displays statistics about a cron job. cron stat requires the iden (ID, identifier) prefix of the cron job to be displayed, which can be obtained with the cron list command. Syntax: cli> cron stat -h usage: Gives detailed information about a single cron job.

Syntax: cron stat

Notes: Any prefix that matches exactly one valid cron job iden is accepted. positional arguments: prefix Cron job iden prefix optional arguments: -h, --help show this help message and exit

Example: TBD cron mod cron mod allows you to modify the storm query executed by a cron job. cron mod requires the iden (ID, identifier) prefix of the cron job to be modified, which can be obtained with the cron list command. Once created, a cron job’s schedule (including jobs created with at ) cannot be modified. A new job must be added and the old job removed. Syntax: cli> cron mod -h usage: Changes an existing cron job's query.

Syntax: cron mod

Notes: Any prefix that matches exactly one valid cron iden is accepted. positional arguments: prefix Cron job iden prefix query New Storm query in curly braces optional arguments: -h, --help show this help message and exit

Example: TBD

10 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0 cron del cron del deletes the specified cron job. Cron jobs remain in a Cortex until explicitly removed. cron del requires the iden (ID, identifier) prefix of the cron job to be removed, which can be obtained with the cron list command. Syntax: cli> cron del -h usage: Deletes a single cron job.

Syntax: cron del

Notes: Any prefix that matches exactly one valid cron job iden is accepted. positional arguments: prefix Cron job iden prefix optional arguments: -h, --help show this help message and exit

Example: TBD

1.9.4 kill

The kill command terminates a task/query executing within a Cortex. kill requires the iden (ID, identifier) or iden prefix of the task to be terminated, which can be obtained with the ps command. Syntax: cli> help kill === kill

Kill a running task/query within the cortex.

Syntax: kill

Users may specify a partial iden GUID in order to kill exactly one matching process based on the partial guid.

Example: TBD

1.9.5 locs

The locs command prints a json-compatible dictionary of local CLI variables where the value is a repr of the object. Syntax:

1.9. Synapse Reference - Synapse Commands 11 synapse Documentation, Release 0.1.0

cli> help locs === locs

List the current locals for a given CLI object.

Example: TBD

1.9.6 log

The log command creates a local log of storm commands executed during your current session. Syntax: cli> help log === log

Add a storm log to the local command session.

Syntax: log (--on|--off) [--splices-only] [--format (mpk|jsonl)] [--path /path/to/

˓→file]

Required Arguments: --on: Enables logging of storm messages to a file. --off: Disables message logging and closes the current storm file.

Optional Arguments: --splices-only: Only records splices. Does not record any other messages. --format: The format used to save messages to disk. Defaults to msgpack (mpk). --path: The path to the log file. This will append messages to a existing

˓→file.

Notes: By default, the log file contains all messages received from the execution of a Storm query by the current CLI. By default, these messages are saved to a file located in ~/.syn/stormlogs/storm_(date).(format).

Examples: # Enable logging all messages to mpk files (default) log --on

# Disable logging and close the current file log --off

# Enable logging, but only log splices. Log them as jsonl instead of mpk. log --on --splices-only --format jsonl

# Enable logging, but log to a custom path: log --on --path /my/aweome/log/directory/storm20010203.mpk

Example: TBD

12 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

1.9.7 ps

The ps command displays the tasks/queries currently running in a Cortex. Syntax: cli> help ps === ps

List running tasks in the cortex.

Example: TBD

1.9.8 quit

The quit command terminates the current Synapse session and exits from the command line interpreter. Syntax: cli> help quit === quit

Quit the current command line interpreter.

Example:

quit

1.9.9 storm

The storm command executes a Synapse Storm query. Storm is the native Synapse query language used to lift, modify, model and analyze data in a Cortex and execute any loaded Synapse modules. The Storm query language is covered in detail starting with the Storm Reference - Introduction section of the Synapse User Guide. Syntax: cli> help storm === storm

Execute a storm query.

Syntax: storm

Arguments: query: The storm query

Optional Arguments: --hide-tags: Do not print tags --hide-props: Do not print secondary properties --hide-unknown: Do not print messages which do not have known handlers. --raw: Print the nodes in their raw format (overrides --hide-tags and --hide-props) --debug: Display cmd debug information along with nodes in raw format (continues on next page)

1.9. Synapse Reference - Synapse Commands 13 synapse Documentation, Release 0.1.0

(continued from previous page) (overrides --hide-tags, --hide-props and raw) --path: Get path information about returned nodes. --graph: Get graph information about returned nodes.

Examples: storm inet:ipv4=1.2.3.4 storm --debug inet:ipv4=1.2.3.4

1.9.10 trigger

The trigger command manipulates triggers in a Cortex. A trigger is a rule stored in a Cortex that enables the automatic execution of a Storm query when a particular event occurs (e.g., an IP address being added to the Cortex). For a detailed discussion of triggers and their use, see Synapse - Triggers. trigger has multiple subcommands, including: • trigger help • trigger add • trigger list • trigger mod • trigger del Syntax: cli> help trigger === trigger

Manipulate triggers in a cortex.

Triggers are rules persistently stored in a cortex such that storm queries automatically run when a particular event happens.

A subcommand is required. Use trigger -h for more detailed help. trigger help trigger includes detailed help describing its individual subcommands. Syntax: cli> trigger -h usage: trigger [-h] {list,add,del,mod} ...

Manipulate triggers in a cortex.

Triggers are rules persistently stored in a cortex such that storm queries automatically run when a particular event happens.

A subcommand is required. Use trigger -h for more detailed help. optional arguments: -h, --help show this help message and exit

14 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

subcommands: {list,add,del,mod} list List triggers you're allowed to manipulate add add a trigger del delete a trigger mod change an existing trigger query trigger add trigger add adds a new trigger to a Cortex. Syntax: cli> trigger add -h usage: Add triggers in a cortex.

Syntax: trigger add condition [#tag] query

Notes: Valid values for condition are: * tag:add * tag:del * node:add * node:del * prop:set

When condition is tag:add or tag:del, you may optionally provide a form name to restrict the trigger to fire only on tags added or deleted from nodes of those forms.

Tag names must start with #.

The added tag is provided to the query as an embedded variable '$tag'.

Simple one level tag globbing is supported, only at the end after a period, that is aka.* matches aka.foo and aka.bar but not aka.foo.bar. aka* is not supported.

Examples: # Adds a tag to every inet:ipv4 added trigger add node:add inet:ipv4 {[ +#mytag ]}

# Adds a tag #todo to every node as it is tagged #aka trigger add tag:add #aka {[ +#todo ]}

# Adds a tag #todo to every inet:ipv4 as it is tagged #aka trigger add tag:add inet:ipv4 #aka {[ +#todo ]} positional arguments: {node:del,node:add,prop:set,tag:del,tag:add} Condition on which to trigger arguments [form] [#tag] [prop] {query} optional arguments: -h, --help show this help message and exit

1.9. Synapse Reference - Synapse Commands 15 synapse Documentation, Release 0.1.0

Example: trigger list trigger list lists the current triggers in a Cortex. Syntax: cli> trigger list -h usage: List existing triggers in a cortex.

Syntax: trigger list

Example: cli> trigger list user iden cond object storm query 739719ff.. prop:set testtype10.intprop [ testint=6 ] optional arguments: -h, --help show this help message and exit

Example: trigger mod trigger mod allows you to modify the storm query associated with a given trigger. trigger mod requires the iden (ID, identifier) prefix of the cron job to be modified, which can be obtained with the trigger list command. Once created, a trigger’s condition, object, and tag parameters cannot be modified. To change these parameters, a new trigger must be added and the old trigger removed. Syntax: cli> trigger mod -h usage: Changes an existing trigger's query.

Syntax: trigger mod

Notes: Any prefix that matches exactly one valid trigger iden is accepted. positional arguments: prefix Trigger iden prefix query Storm query in curly braces optional arguments: -h, --help show this help message and exit

Example:

16 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

trigger del

trigger del removes the specified trigger from a Cortex. trigger del requires the iden (ID, identifier) prefix of the cron job to be modified, which can be obtained with the trigger list command. Syntax:

cli> trigger del -h usage: Delete an existing trigger.

Syntax: trigger del

Notes: Any prefix that matches exactly one valid trigger iden is accepted.

positional arguments: prefix Trigger iden prefix

optional arguments: -h, --help show this help message and exit

Example:

1.10 Storm Reference - Introduction

Storm (Background - Storm Query Language) is the query language used to interact with data in a Cortex. Storm allows you to ask about, retrieve, annotate, add, modify, and delete data from a Cortex. Most Synapse users (e.g., those conducting analysis on the data) will access Storm via the command-line interface (CLI), using the Synapse storm command to invoke a Storm query: cli> storm This section covers the following important Storm background concepts: • Storm Operations • Lift, Filter, and Pivot Criteria • Operation Chaining • Node Consumption

1.10.1 Storm Operations

Storm allows users to perform all of the standard operations used to interact with a Cortex: • Lift: – retrieve data based on specified criteria. (Storm Reference - Lifting) • Filter: – take a set of lifted nodes and refine your results by including or excluding a subset of nodes based on specified criteria. (Storm Reference - Filtering) • Pivot: – take a set of lifted nodes and identify other nodes that share one or more properties or property values with the lifted set. (Storm Reference - Pivoting)

1.10. Storm Reference - Introduction 17 synapse Documentation, Release 0.1.0

• Data modification: – add, modify, annotate, and delete nodes from a Cortex. (Storm Reference - Data Modifi- cation) Most operations (other than those used solely to lift or add data) require an existing data set on which to operate. This data set is typically the output of a previous Storm operation whose results are the nodes you want to modify or otherwise work with. In addition to these operations, the Storm query language supports an extensible set of Storm commands (Storm Reference - Storm Commands). Commands such as limit, noderefs, or uniq support specific functionality to further extend the power of Storm. Available commands can be displayed with storm help. Storm also supports powerful features such as the use of variables (Storm Reference - Variables) in queries and the ability to issue subqueries (Storm Reference - Subquery Syntax) within Storm itself.

1.10.2 Lift, Filter, and Pivot Criteria

The main operations carried out with Storm are lifting, filtering, and pivoting. When conducting these operations, you need to be able to clearly specify the data you are interested in – your selection criteria. In most cases, the criteria you specify will be based on one or more of the following: •A property (primary or secondary) on a node. • A specific value for a property (

= or = ) on a node. •A tag on a node. All of the above elements – nodes, properties, values, and tags – are the fundamental building blocks of the Synapse data model (Background - Data Model and Terminology). As such, an understanding of the Synapse data model is essential to effective use of Storm.

1.10.3 Operation Chaining

Storm allows multiple operations to be chained together to form increasingly complex queries. Storm operations are processed in order from left to right with each operation (lift, filter, or pivot) acting on the current result set (e.g., the output of the previous operation). From an analysis standpoint, this feature means that Storm syntax can parallel an analyst’s natural thought process: “show me X data. . . that’s interesting, take a subset of X data and show me the Y data that relates to X. . . hm, now take the results from Y and show me any relationship to Z data. . . ” and so on. From a practical standpoint, it means that order matters when constructing a Storm query. A lengthy Storm query is not evaluated as a whole. Instead, Synapse parses each component of the query in order, evaluating each component individually.

1.10.4 Node Consumption

Most Storm operations consume nodes when the operation occurs. That is, the set of nodes input into a particular Storm operation is typically transformed by that operation in some way. With few exceptions (such as the join operator (see Storm Reference - Pivoting) and the Storm count command), the nodes input to the operation are not retained - they are “consumed” during processing. Storm outputs only those nodes that result from carrying out the specified operation. If you lift a set of nodes and then filter the results, only those nodes captured by the filter are retained - the other nodes are consumed (discarded). In this way the operations performed in sequence may add or remove nodes from Storm’s working set, or clear the set entirely. The set is continually changing based on the last-performed operation or last-issued command. Particularly

18 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0 when first learning Storm, users are encouraged to break down lengthy queries into their component parts, and to validate the output (results) after the addition of each operation to the overall query.

1.11 Storm Reference - Document Syntax Conventions

This section covers the following important conventions used within the Storm Reference Documents: • Storm and Layers • Storm Syntax Conventions • Usage Statements vs. Specific Storm Queries • Type-Specific Behavior • Whitespace

1.11.1 Storm and Layers

The Storm Reference documentation provides basic syntax examples that assume a simple Storm environment - that is, a Cortex with a single Layer. For multi-Layer Cortexes, the effects of specific Storm commands - particularly data modification commands - may vary based on the specific arrangement of read / write Layers, the Layer in which the command is executed, and the permissions of the user (see Synapse - Permissions). See Synapse - Layers for additional details on Layers.

1.11.2 Storm Syntax Conventions

The Storm Reference documentation provides numerous examples of both abstract Storm syntax (usage statements) and specific Storm queries. The following conventions are used for Storm usage statements: • Items that must be entered literally on the command line are in bold. These items include command names and literal characters. • Items that represent “variables” that must be replaced with a name or value are placed within angle brackets ( < > ) in italics. Most “variables” are self-explanatory, however a few commonly used variable terms are defined here for convenience: – refers to a form / node primary property, such as inet:fqdn. – refers to the value of a primary property, such as woot.com in inet:fqdn=woot.com. – refers to a node secondary property (including universal properties) such as inet:ipv4:asn or inet:ipv4.created. – refers to the value of a secondary property, such as 4808 in inet:ipv4:asn=4808. – refers to a Storm query. – refers to a Storm query whose results contain the specified form(s) – refers to a tag (#sometag as opposed to a syn:tag form). • Bold brackets are literal characters. Parameters enclosed in non-bolded brackets are optional. • Parameters not enclosed in brackets are required. • A vertical bar signifies that you choose only one parameter. For example: – a | b indicates that you must choose a or b.

1.11. Storm Reference - Document Syntax Conventions 19 synapse Documentation, Release 0.1.0

– [ a | b ] indicates that you can choose a, b, or nothing (the non-bolded brackets indicate the parameter is optional). • Ellipses ( ... ) signify the parameter can be repeated on the command line. • The storm command that must precede a Storm query is assumed and is omitted from examples. Example: [ = [ : = ...] ] The Storm query above adds a new node. • The outer brackets are in bold and are required literal characters to specify a data modification (add) operation. Similarly, the equals signs are in bold to indicate literal characters. • and would need to be replaced by the specific form (such as inet:ipv4) and primary property value (such as 1.2.3.4) for the node being created. • The inner brackets are not bolded and indicate that one or more secondary properties can optionally be specified. • and would need to be replaced by the specific secondary property and value to add to the node, such as :loc = us. • The ellipsis ( ... ) indicate that additional secondary properties can optionally be specified.

1.11.3 Usage Statements vs. Specific Storm Queries

Examples of specific queries represent fully literal input, but are not shown in bold for readability. For example: Usage statement: [ = [ : = ...] ] Example query: [ inet:ipv4 = 1.2.3.4 :loc = us ]

1.11.4 Type-Specific Behavior

Some data types within the Synapse data model have been optimized in ways that impact their behavior within Storm queries (e.g., how types can be input, lifted, filtered, etc.) See Storm Reference - Type-Specific Storm Behavior for details.

1.11.5 Whitespace

Whitespace may be used in the examples for formatting and readability. See the section Synapse Tools - Cmdr on Using Whitespace Characters for how whitespace is used in the Synapse CLI, including within the Storm command.

1.12 Storm Reference - Lifting

Lift operations retrieve a set of nodes from a Synapse Cortex based on specified criteria. While all lift operations are retrieval operations, they can be broken down into “types” of lifts based on the criteria, comparison operator, or special handler used: • Simple Lifts • Lifts Using Standard Comparison Operators

20 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

• Lifts Using Extended Comparison Operators See Storm Reference - Document Syntax Conventions for an explanation of the syntax format used below. See Storm Reference - Type-Specific Storm Behavior for details on special syntax or handling for specific data types.

1.12.1 Simple Lifts

“Simple” lifts refers to the most “basic” lift operations. That is, operations to retrieve a set of nodes based on: • The presence of a specific primary or secondary property. • The presence of a specific primary property value or secondary property value. • The presence of a specific tag. The only difference between “simple” lifts and “lifts using comparison operators” is that we have defined simple lifts as those that use the equals ( = ) comparator, which is the easiest comparator to use to explain basic lift concepts. Syntax: | = | | | = | # Examples: Lift by primary property (): • Lift all domain nodes:

inet:fqdn

• Lift all mutex nodes:

it:dev:mutex

Lift a specific node ( = ): • Lift the node for the domain google.com:

inet:fqdn = google.com

• Lift the node for a specific MD5 hash:

hash:md5 = d41d8cd98f00b204e9800998ecf8427e

Lift a specific compound node: • Lift the DNS A record showing that domain woot.com resolved to IP 1.2.3.4:

inet:dns:a = (woot.com, 1.2.3.4)

Lift a specific GUID node: • Lift the organization node with the specified GUID:

ou:org=2f92bc913918f6598bcf310972ebf32e

Lift a specific digraph (edge) node: • Lift the has node linking the person node representing “Bob Smith” to his email address:

1.12. Storm Reference - Lifting 21 synapse Documentation, Release 0.1.0

has=((ps:person,12af06294ddf1a0ac8d6da34e1dabee4),(inet:email, [email protected]))

Lift by primary property value alone (): • Lift the domain woot.com: woot.com

Lift by the presence of a secondaray property (): • Lift the DNS SOA record nodes that have an email property: inet:dns:soa:email

Lift by a specific property value ( = ): • Lift the organization node with the alias vertex: ou:org:alias = vertex

• Lift all DNS A records for the domain blackcake.net: inet:dns:a:fqdn = blackcake.net

• Lift all the files with a PE compiled time of 1992-06-19 22:22:17: file:bytes:mime:pe:compiled = "1992/06/19 22:22:17"

Lift all nodes with a specific tag:

#cno.infra.anon.tor

Usage Notes: • Lifting nodes by form alone (e.g., lifting all inet:fqdn nodes or all inet:email nodes) is possible but generally impractical / undesirable as it will potentially return an extremely large data set. • Because of the risk of accidentally lifting all nodes of a given form, the Storm query planner will automatically optimize lifts that: – specify a form with no value (e.g., attempt to lift by form alone); and – are immediately followed by a positive tag filter (+#sometag). – For example, the following queries are executed in the same manner by the Storm runtime:

* Lift followed by tag filter: inet:fqdn +#hehe.haha * Lift by tag (described below): inet:fqdn#hehe.haha • Lifting by form alone when piped to the Storm limit command may be useful for returning a small number of “exemplar” nodes. • Lifting nodes by = is the most common method of lifting a single node. • When lifting a form whose consists of multiple components (e.g., a compound node or digraph node), the components must be passed as a comma-separated list enclosed in parentheses. • For certain common forms, if you specify a without a , the Storm parser will attempt to determine the correct form based on the type. This means it is possible (for example) to lift do- mains by alone. However, this may also have unexpected results; attempting to lift the email address

22 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

[email protected] without specifying the form inet:email will lift both the email address and the domain gmail.com. • Lifting nodes by the presence of a secondary property alone () may be impractical / undesirable (similar to lifting by form alone), but may be feasible in limited cases (i.e., where it is known that only a relatively small number of nodes have a given secondary property). • Lifting nodes by the value of a secondary property ( = ) is useful for lifting all nodes that share a secondary property with the same value; and may be used to lift individual nodes with unique or relatively unique secondary properties in cases where entering the primary property is impractical (such as for GUID nodes). • Lifting nodes by tag alone (#) lifts nodes of all forms with that tag. To lift specific forms only, use Lift by Tag (#) or an additional filter (see Storm Reference - Filtering).

1.12.2 Lifts Using Standard Comparison Operators

Lift operations can be performed using most of the standard mathematical / logical comparison operators (compara- tors), as well as lifting via regular expression: • = : equals (described above) • < : less than • > : greater than • <= : less than or equal to • >= : greater than or equal to Lifting by “not equal to” (!=) is not currently supported. Syntax: Examples: Lift using less than comparator: • Lift domain WHOIS records where the domain’s registration (created) date was before June 1, 2014:

inet:whois:rec:created < 2014/06/01

Lift using greater than comparator: • Lift files whose size is larger than 1MB:

file:bytes:size > 1048576

Lift using less than or equal to comparator: • Lift people (person nodes) born on or before January 1, 1980:

ps:person:dob <= 1980/01/01

Lift using greater than or equal to comparator: • Lift WHOIS records retrieved on or after December 1, 2018 at 12:00:

inet:whois:rec:asof >= "2018/12/01 12:00"

1.12. Storm Reference - Lifting 23 synapse Documentation, Release 0.1.0

1.12.3 Lifts Using Extended Comparison Operators

Storm supports a set of extended comparison operators (comparators) for specialized lift operations. In most cases, the same extended comparators are available for both lifting and filtering: • Lift by Regular Expression (~=) • Lift by Prefix (^=) • Lift by Range (*range=) • Lift by Set Membership (*in=) • Lift by Proximity (*near=) • Lift by Tag (#) • Recursive Tag Lift (##)

Lift by Regular Expression (~=)

The extended comparator ~= is used to lift nodes based on standard regular expressions.

Warning: While lifting using regular expressions is possible, matching is performed via brute force comparison of the relevant properties. Lifting by regex may thus be time consuming when lifting over large data sets. Lift by Prefix (^=) is supported for string types and should be considered as a more efficient alternative when possible.

Syntax: ~= Example: • Lift files with PDB paths containing the string rouji: file:bytes:mime:pe:pdbpath ~= "rouji"

Lift by Prefix (^=)

Synapse performs prefix indexing on string types, which optimizes lifting nodes whose or starts with a given prefix. This improves performance by avoiding regex brute-forcing. The extended comparator ^= is used to lift nodes by prefix. Syntax: [ : ] ^= Examples: Lift primary property by prefix: • Lift all usernames that start with “pinky”: inet:user^=pinky

Lift secondary property by prefix: • Lift all organizations whose name starts with “International”:

24 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

ou:org:name^=international

Usage Notes: • Extended string types that support dotted notation (such as the loc or syn:tag types) have custom behaviors with respect to lifting and filtering by prefix. See the respective sections in Storm Reference - Type-Specific Storm Behavior for additional details.

Lift by Range (*range=)

The range extended comparator (*range=) supports lifting nodes whose = or = fall within a specified range of values. The comparator can be used with types such as integers and times (including types that are extensions of those types, such as IP addresses). Syntax: [ : ] *range = ( , ) Examples: Lift by primary property in range: • Lift all IP addresses between 192.168.0.0 and 192.168.0.10: inet:ipv4*range=(192.168.0.0, 192.168.0.10)

Lift by secondary property in range: • Lift files whose size is between 1000 and 100000 bytes: file:bytes:size*range=(1000,100000)

• Lift WHOIS records that were captured between November 29, 2013 and June 14, 2016: inet:whois:rec:asof*range=(2013/11/29, 2016/06/14)

• Lift DNS requests made within one day of 12/01/2018: inet:dns:request:time*range=(2018/12/01, "+-1 day")

Usage Notes: • When specifying a range, both the minimum and maximum values are included in the range (the equivalent of “greater than or equal to and less than or equal to ”).

Lift by Set Membership (*in=)

The set membership extended comparator (*in=) supports lifting nodes whose = or = matches any of a set of specified values. The comparator can be used with any type. Syntax: [ : ] *in = ( , , ... ) Examples: Lift by primary property in a set: • Lift IP addresses matching any of the specified values:

1.12. Storm Reference - Lifting 25 synapse Documentation, Release 0.1.0

inet:ipv4*in=(127.0.0.1, 192.168.0.100, 255.255.255.254)

Lift by secondary property in a set: • Lift files whose size in bytes matches any of the specified values:

file:bytes:size*in=(4096, 16384, 65536)

• Lift tags that end in foo, bar, or baz:

syn:tag:base*in=(foo,bar,baz)

Lift by Proximity (*near=)

The proximity extended comparator (*near=) supports lifting nodes by “nearness” to another node based on a spec- ified property type. Currently, *near= supports proximity based on geospatial location (that is, nodes within a given radius of a specified latitude / longitude). Syntax: [ : ] *near = (( , ), ) Examples: • Lift locations (geo:place nodes) within 500 meters of the Eiffel Tower:

geo:place:latlong*near=((48.8583701,2.2944813),500m)

Usage Notes: • In the example above, the latitude and longitude of the desired location (i.e., the Eiffel Tower) are explicitly specified as parameters to *near=. • Radius can be specified in the following metric units. Values of less than 1 (e.g., 0.5km) must be specified with a leading zero: – Kilometers (km) – Meters (m) – Centimeters (cm) – Millimeters (mm) • The *near= comparator works for geospatial data by lifting nodes within a square bounding box centered at ,, then filters the nodes to be returned by ensuring that they are within the great-circle distance given by the argument.

Lift by Tag (#)

The tag extended comparator (#) supports lifting nodes based on a given tag being applied to the node. Syntax: [ ] # Examples: Lift all nodes associated with Tor infrastructure:

26 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

#cno.infra.anon.tor

• Lift the domains that Palo Alto Networks says are associated with the OilRig threat group: inet:fqdn#aka.paloalto.thr.oilrig

Recursive Tag Lift (##)

The recursive tag extended comparator (##) supports lifting nodes tagged with any tag that is itself tagged with a given tag. Tags can be applied to syn:tag nodes; that is, tags can be used to tag other tags. The ability to “tag the tags” can be used to represent certain types of analytical relationships. For example: • syn:tag nodes representing threat groups can be tagged to indicate their assessed country of origin. • syn:tag nodes representing malware or tools can be tagged with their assessed availability (e.g., public, private, private but shared, etc.) A recursive tag lift performs the following actions: 1. For the specified tag (##), lift the nodes that have that tag (i.e., the equivalent of #), including any syn:tag nodes. 2. For any lifted syn:tag nodes, lift all nodes tagged with those tags (including any additional syn:tag nodes). 3. Repeat #2 until no more syn:tag nodes are lifted. 4. Return the tagged nodes. Note that syn:tag nodes themselves are not returned. Syntax: ## Examples: • Lift all nodes tagged with any tags (such as threat group tags) that FireEye claims are associated with :

##aka.feye.cc.ru

Usage Notes: In the example above, the tag aka.feye.cc.ru could be applied to syn:tag nodes representing FireEye’s “Rus- sian” threat groups (e.g., aka.feye.thr.apt28, aka.feye.thr.apt29, etc.) Using a recursive tag lift allows you to easily lift all nodes tagged by any of those tags.

1.13 Storm Reference - Filtering

Filter operations are performed on the output of a previous Storm query. A filter operation downselects from the working set of nodes by either including or excluding a subset of nodes based on a set of criteria. • + specifies an inclusion filter. The filter downselects the working set to only those nodes that match the specified criteria. • - specifies an exclusion filter. The filter downselects the working set to all nodes except those that match the specified criteria. The types of filter operations within Storm are highly flexible and consist of the following: • Simple Filters

1.13. Storm Reference - Filtering 27 synapse Documentation, Release 0.1.0

• Filters Using Standard Comparison Operators • Filters Using Extended Comparison Operators • Compound Filters • Subquery Filters In most cases, the criteria and available comparators for lift operations (Storm Reference - Lifting) are also available for filter operations.

Note: When filtering based on a secondary property () or secondary property value ( = ), the property can be specified using the relative property name only (:baz vs. foo:bar:baz) unless the full property name is required for disambiguation. In the examples below, both syntaxes (i.e., using the full property name and the relative property name) are provided where appropriate for completeness. See the Background - Data Model and Terminology for additional discussion of properties.

See Storm Reference - Document Syntax Conventions for an explanation of the syntax format used below. See Storm Reference - Type-Specific Storm Behavior for details on special syntax or handling for specific data types.

1.13.1 Simple Filters

“Simple” filters refers to the most “basic” filter operations: that is, operations to include ( + ) or exclude ( - ) a subset of nodes based on: • The presence of a specific primary or secondary property in the working set. • The presence of a specific primary property value or secondary property value in the working set. • The presence of a specific tag on nodes in the working set. The only difference between “simple” filters and “filters using comparison operators” is that we have defined simple filters as those that use the equals ( = ) comparator, which is the easiest comparator to use to explain basic filtering concepts. Syntax: + | - | = | | = | Examples: Filter by Form (): • Downselect to include only domains:

+inet:fqdn

Filter by Primary Property Value: • Downselect to exclude the domain google.com:

-inet:fqdn=google.com

Filter by Presence of Secondary Property: • Downselect to exclude DNS SOA records with an “email” property:

-inet:dns:soa:email

28 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

-:email

Filter by Secondary Property Value: • Downselect to include only those domains that are also logical zones:

+inet:fqdn:iszone=1

+:iszone=1

Filter by Presence of Universal Property: • Downselect to include only those domains with a .seen property:

+inet:fqdn.seen

+.seen

Filter by tag: • Downselect to exclude nodes tagged as associated with Tor:

-#cno.infra.anon.tor

Usage Notes: • The comparator (comparison operator) specifies how or is evaluted with respect to or . The most common comparator is equals (=), although other comparators are available (see below).

1.13.2 Filters Using Standard Comparison Operators

Filter operations can be performed using any of the standard mathematical / logical comparison operators (compara- tors): • =: equals (described above) • != : not equals • < : less than • > : greater than • <= : less than or equal to • >= : greater than or equal to Syntax: + | - | | Examples: Filter by Not Equals: • Downselect to exclude the domain google.com:

+inet:fqdn != google.com

Filer by Less Than: • Downselect to include only WHOIS records collected prior to January 1, 2017:

1.13. Storm Reference - Filtering 29 synapse Documentation, Release 0.1.0

+inet:whois:rec:asof < 2017/01/01

+:asof < 2017/01/01

Filter by Greater Than: • Downselect to exclude files larger than 4096 bytes:

-file:bytes:size > 4096

-:size > 4096

Filter by Less Than or Equal To: • Downlselect to include only WHOIS nodes for domains created on or before noon on January 1, 2018:

+inet:whois:rec:created <= "2018/01/01 12:00"

+:created <= "2018/01/01 12:00"

Filter by Greater Than or Equal To: • Downlselect to include only people born on or after January 1, 1980:

+ps:person:dob >= 1980/01/01

+:dob >= 1980/01/01

Usage Notes: • Storm supports both equals ( = ) and not equals ( != ) comparators for filtering, although use of not equals is not strictly necessary. Because filters are either inclusive ( + ) or exclusive ( - ), equivalent filter logic for “not equals” can be performed with “equals”. That is, “include domains not equal to google.com” (+inet:fqdn != google.com) is equivalent to “exclude the domain google.com” (-inet:fqdn = google.com).

1.13.3 Filters Using Extended Comparison Operators

Storm supports a set of extended comparison operators (comparators) for specialized filter operations. In most cases, the same extended comparators are available for both lifting and filtering: • Filter by Regular Expression (~=) • Filter by Prefix (^=) • Filter by Time or Interval (@=) • Filter by Range (*range=) • Filter by Set Membership (*in=) • Filter by Proximity (*near=) • Filter by Tag (#)

30 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

Filter by Regular Expression (~=)

The extended comparator ~= is used to filter nodes based on standard regular expressions. Syntax: + | - | ~= Examples: Filter by Regular Expression: • Downselect to include only mutexes that start with the string “Net”:

+it:dev:mutex ~= "^Net"

Usage Notes: • Filtering using regular expressions is performed by matching the regex against the relevant property of each node in the working set. Because filtering is performed on a subset of data from the Cortex (i.e., the working set) there should be no noticeable performance impact with a regex filter. However, prefix filtering (see below) is supported for string types and can be used as a more efficient alternative in some cases.

Filter by Prefix (^=)

Synapse performs prefix indexing on string types, which optimizes filtering nodes whose or starts with a given prefix. The extended comparator ^= is used to filter nodes by prefix. Syntax: + | - [ : ] ^= Examples: Filter by primary property by prefix: • Downselect to include only usernames that start with “pinky”:

+inet:user ^= pinky

Filter by secondary property by prefix: • Downselect to include only organizations whose name starts with “International”:

+ou:org:name ^= international

+:name ^= international

Usage Notes: • Extended string types that support dotted notation (such as the loc or syn:tag types) have custom behaviors with respect to lifting and filtering by prefix. See the respective sections in Storm Reference - Type-Specific Storm Behavior for additional details.

Filter by Time or Interval (@=)

The time extended comparator (@=) supports filtering nodes based on comparisons among various combinations of timestamps and date/time ranges (intervals). See Storm Reference - Type-Specific Storm Behavior for additional detail on the use of time and ival data types.

1.13. Storm Reference - Filtering 31 synapse Documentation, Release 0.1.0

Syntax: + | - @=( , ) + | - @=

+inet:dns:a.seen@=(2018/07/01, 2018/08/01)

+.seen@=(2018/07/01, 2018/08/01)

• Downselect to include only those nodes (e.g., IP addresses) that were associated with Tor between June 1, 2016 and September 30, 2016 (note the interval here applies to the tag representing Tor):

+#cno.infra.anon.tor@=(2016/06/01, 2016/09/30)

Filter by comparing a timestamp to an interval: • Downselect to include only those DNS request nodes whose requests occurred between 2:00 PM November 12, 2017 and 9:30 AM November 14, 2017:

+inet:dns:request:time@=("2017/11/12 14:00:00", "2017/11/14 09:30:00")

+:time@=("2017/11/12 14:00:00", "2017/11/14 09:30:00")

Filter by comparing an interval to a timestamp: • Downselect to include only those DNS A records whose resolution time windows include the date December 1, 2017:

+inet:dns:a.seen@=2017/12/01

+.seen@=2017/12/01

Filter by comparing a timestamp to a timestamp: • Downselect to include only those WHOIS records whose domain was registered (created) on March 19, 1986 at 5:00 AM:

+inet:whois:rec:created@="1986/03/19 05:00:00"

+:created@="1986/03/19 05:00:00"

Filter using an interval with relative times: • Downselect to include only those inet:whois:email nodes that were observed between January 1, 2018 and the present:

+inet:whois:email.seen@=(2018/01/01, now)

+.seen@=(2018/01/01, now)

• Downselect to include only DNS requests whose requests occurred within one week after October 15, 2018:

32 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

+inet:dns:request:time@=(2018/10/15, "+ 7 days")

+:time@=(2018/10/15, "+ 7 days")

Usage Notes: • When specifying an interval, the minimum value is included in the interval but the maximum value is not (the equivalent of “greater than or equal to and less than ”). This behavior is slightly different than that for *range=, which includes both the minimum and maximum. • When comparing an interval to an interval, Storm will return nodes whose interval has any overlap with the specified interval. – For example, a filter interval of September 1, 2018 to October 1, 2018 (2018/09/01, 2018/10/01) will match nodes with any of the following intervals:

* August 12, 2018 to September 6, 2018 (2018/08/12, 2018/09/06). * September 13, 2018 to September 17, 2018 (2018/09/13, 2018/09/17). * September 30, 20180 to November 5, 2018 (2018/09/30, 2018/11/05). • When comparing a timestamp to an interval, Storm will return nodes whose timestamp falls within the speci- fied interval. • When comparing an interval to a timestamp, Storm will return nodes whose interval encompasses the specified timestamp. • When comparing a timestamp to a timestamp, interval ( @= ) syntax is supported, although the equals com- parator ( = ) can simply be used. • Because tags can be given timestamps (min / max interval values), interval filters can also be used with tags.

Filter by Range (*range=)

The range extended comparator (*range=) supports filtering nodes whose = or = fall within a specified range of values. The comparator can be used with types such as integers and times, including types that are extensions of those types, such as IP addresses. Syntax: | *range = ( , ) Examples: Filter by primary property in range: • Downselect to include all IP addresses between 192.168.0.0 and 192.168.0.10:

+inet:ipv4*range=(192.168.0.0, 192.168.0.10)

Filter by secondary property in range: • Downselect to include files whose size in bytes is within the specified range:

+file:bytes:size*range=(1000, 100000)

+:size*range=(1000, 100000)

• Downselect to include WHOIS records that were captured between the specified dates:

1.13. Storm Reference - Filtering 33 synapse Documentation, Release 0.1.0

+inet:whois:rec:asof*range=(2013/11/29, 2016/06/14)

+:asof*range=(2013/11/29, 2016/06/14)

• Downselect to include DNS requests made within 1 day of 12/01/2018:

+inet:dns:request:time*range=(2018/12/01, "+-1 day")

+:time*range=(2018/12/01, "+-1 day")

Usage Notes: • When specifying a range (*range=), both the minimum and maximum values are included in the range (the equivalent of “greater than or equal to and less than or equal to ”). This behavior is slightly different than that for time interval (@=), which includes the minimum but not the maximum. • The *range= extended comparator can be used with time types, although the time / interval extended com- parator ( @= ) is preferred.

Filter by Set Membership (*in=)

The set membership extended comparator (*in=) supports filtering nodes whose = or = matches any of a set of specified values. The comparator can be used with any type. Syntax: + | - | *in = ( , , ... ) Examples: Filter by primary property in set: • Downselect to include IP addresses matching any of the specified values: cli> storm [inet:ipv4=127.0.0.1 inet:ipv4=192.168.0.100 inet:ipv4=255.255.255.254] inet:ipv4=127.0.0.1 .created = 2019/02/12 15:25:50.493 :asn = 0 :loc = ?? :type = loopback inet:ipv4=192.168.0.100 .created = 2019/02/12 15:25:49.013 :asn = 0 :loc = ?? :type = private inet:ipv4=255.255.255.254 .created = 2019/02/12 15:25:50.494 :asn = 0 :loc = ?? :type = private complete. 3 nodes in 31 ms (96/sec).

+inet:ipv4*in=(127.0.0.1, 192.168.0.100, 255.255.255.254)

Filter by secondary property in set: • Downselect to include files whose size in bytes matches any of the specified values:

34 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

+file:bytes:size*in=(4096, 16384, 65536)

+:size*in=(4096, 16384, 65536)

• Downselect to exclude tags that end in foo, bar, or baz:

-syn:tag:base*in=(foo, bar, baz)

-:base*in=(foo, bar, baz)

Filter by Proximity (*near=)

The proximity extended comparator (*near=) supports filtering nodes by “nearness” to another node based on a specified property type. Currently, *near= supports proximity based on geospatial location (that is, nodes within a given radius of a specified latitude / longitude). Syntax: + | - | *near = (( , ), ) Examples: Filter by proximity: • Downselect to include only Foo Corporation offices within 1km of a specific coffee shop:

+geo:place:latlong*near=((47.6050632,-122.3339756),1km)

+:latlong*near=((47.6050632,-122.3339756),1km)

Usage Notes: • In the example above, the latitude and longitude of the desired location (i.e., the coffee shop) are explicitly specified as parameters to *near=. • Radius can be specified in the following metric units. Values of less than 1 (e.g., 0.5km) must be specified with a leading zero: – Kilometers (km) – Meters (m) – Centimeters (cm) – Millimeters (mm) • The *near= comparator works by identifying nodes within a square bounding box centered at , , then filters the nodes to be returned by ensuring that they are within the great-circle distance given by the argument.

Filter by Tag (#)

The tag extended comparator (#) supports filtering nodes based on a given tag being applied to the node. Syntax: + | -# Examples:

1.13. Storm Reference - Filtering 35 synapse Documentation, Release 0.1.0

• Downselect to include only nodes that FireEye says are part of the GREENCAT malware family:

+#aka.feye.mal.greencat

• Downselect to exclude nodes tagged as sinkholes:

-#cno.infra.sink.hole

Usage Notes: • When filtering by tag, only a single tag can be specified. To filter on multiple tags, use Compound Filters.

1.13.4 Compound Filters

Storm allows the use of the logical operators and, or, and not (including and not) to construct compound filters. Parentheses can be used to group portions of the filter statement to indicate order of precedence and clarify logical operations when evaluating the filter. Syntax: + | -( <filter> and | or | not | and not ... ) Examples: • Downselect to exclude files that are less than or equal to 16384 bytes in size and were compiled prior to January 1, 2014:

-(file:bytes:size <= 16384 and file:bytes:mime:pe:compiled < 2014/01/01)

-(:size <= 16384 and :mime:pe:compiled < 2014/01/01)

• Downselect to include only files or domains that FireEye claims are associated with APT1:

+((file:bytes or inet:fqdn) and #aka.feye.thr.apt1)

• Downselect to include only files and domains that FireEye claims are associated with APT1 that are not sink- holed:

+((file:bytes or inet:fqdn) and (#aka.feye.thr.apt1 and not #cno.infra.sink.

˓→hole))

Usage Notes: • Logical operators must be specified in lower case. • Parentheses should be used to logically group portions of the filter statement for clarity.

1.13.5 Subquery Filters

Storm’s subquery syntax (Storm Reference - Subquery Syntax) can be used to create filters. A subquery (denoted by curly braces ( {} ) ) can be placed anywhere within a larger Storm query. When nodes are passed to a subquery filter: • Nodes are consumed (i.e., are not returned by the subquery) if they evaluate false. • Nodes are not consumed (i.e., are returned by the subquery) if they evaluate true.

36 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

In this way subqueries act as complex filters, allowing the formation of advanced queries that would otherwise require methods such as saving the results of an initial query off to the side while running a second query, then loading the results of the first query back to the results of the second query. Syntax: + | -{ } Examples: • From an initial set of domains, return only those domains that resolve to an IP address that Trend Micro asso- ciates with the Pawn Storm threat group (i.e., an IP address tagged #aka.trend.thr.pawnstorm):

+{ -> inet:dns:a:fqdn :ipv4 -> inet:ipv4 +#aka.trend.thr.pawnstorm }

From an initial set of IP addresses, return only those IPs registered to an Autonomous System (AS) whois name starts with “makonix”:

+{ :asn -> inet:asn +:name^="makonix" }

1.14 Storm Reference - Pivoting

Pivot operations are performed on the output of a Storm query. Pivot operators are used to navigate from one set of nodes to another based a specified relationship. The pivot operations available within Storm are: • Pivot Out Operator • Pivot In Operator • Pivot With Join • Pivot to Digraph (Edge) Nodes • Pivot Across Digraph (Edge) Nodes • Pivot to Tags • Pivot from Tags • Implicit Pivot Syntax

Note: When pivoting from a secondary property ( = ), the secondary property must be specified using the relative property name only (:baz vs. foo:bar:baz). Specifying the full property name before the pivot would be interpreted as an additional lift (i.e., inet:dns:a:fqdn -> inet:fqdn would be interpreted as “take a set of inet:dns:a records from an initial query, lift all inet:dns:a records with an :fqdn property (i.e., every inet:dns:a node in the Cortex), and then pivot to the associated inet:fqdn nodes”).

See Storm Reference - Document Syntax Conventions for an explanation of the syntax format used below. See Storm Reference - Type-Specific Storm Behavior for details on special syntax or handling for specific data types.

1.14.1 Pivot Out Operator

The pivot out operator ( -> ) is the primary Storm pivot operator. The pivot out operator pivots from a primary or secondary property value of the current set of nodes to a primary or secondary property value of another set of nodes. This operator is also referred to as the “reference out” operator as it is used to pivot to nodes that are referenced by the current node set.

1.14. Storm Reference - Pivoting 37 synapse Documentation, Release 0.1.0

The pivot out operator is used to: • pivot from the primary property of the inbound set of nodes to the equivalent secondary property of another set of nodes, • pivot from a secondary property of the inbound set of nodes to the equivalent primary property of another set of nodes, • pivot from any / all secondary properties of the inbound set of nodes to the equivalent primary property of any / all nodes (“wildcard” pivot out), and • pivot from a secondary property of the inbound set of nodes to the equivalent secondary property of another set of nodes. Pivot to Digraph (Edge) Nodes and Pivot Across Digraph (Edge) Nodes are covered separately below. Syntax: -> : : -> -> * : -> : Examples: Pivot from primary property ( = ) to secondary property ( = ): • Pivot from a set of domains to all of their subdomains regardless of depth:

-> inet:fqdn:zone

• Pivot from a set of domains to their DNS A records:

-> inet:dns:a:fqdn

Pivot from secondary property ( = ) to primary property ( = ): • Pivot from a set of DNS A records to the resolution IP addresses contained in those records:

:ipv4 -> inet:ipv4

Pivot from all secondary properties to all forms ( = to = ): • Pivot from a set of WHOIS records to all nodes whose primary property equals any of the secondary properties of the WHOIS record (the asterisk * is a wildcard that indicates pivot to any applicable node):

-> *

Pivot from secondary property ( = ) to secondary property ( = ): • Pivot from the WHOIS records for a set of domains to the DNS A records for the same domains:

:fqdn -> inet:dns:a:fqdn

Usage Notes: • Pivoting out using the asterisk wildcard ( * ) is sometimes called a refs out pivot because it pivots from all secondary properties to all nodes referenced by those properties. • Pivoting using the wildcard is based on strong data typing within the Synapse data model, so will only pivot out to properties that match both and / . This means that the following nodes will not be returned by a wildcard pivot out:

38 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

– Nodes with matching / but of different . For example, if a node’s secondary property is a string (type ) that happens to contain a valid domain (type ), a wildcard pivot out from the node with the string value will not return the inet:fqdn node. – Digraph (edge) nodes, whose properties are of type (node definition, or , tuples). See Pivot to Digraph (Edge) Nodes and Pivot Across Digraph (Edge) Nodes for details on pivoting to / through those forms. • It is possible to perform an explicit pivot between properties of different types. For example: :name -> inet:fqdn

1.14.2 Pivot In Operator

The pivot in ( <-) operator is similar to but separate from the pivot out ( ->) operator. Instead of pivoting to the set of nodes the current set references, the pivot in operator pivots to the set of nodes that references the current set of nodes. Logically, any pivot in operation can be expressed as an equivalent pivot out operation. For example, the following two pivots would be functionally equivalent: • Pivot out from a set of domains to the DNS A records referenced by the domains: -> inet:dns:a:fqdn • Pivot in to a set of domains from the DNS A records that reference the domains: <- inet:dns:a:fqdn Because of this equivalence, and because “left to right” logic is generally more intuitive, only pivot out has been fully implemented in Storm. (The second example, above, will actually return an error.) The pivot in operator exists, but is only used to simplify certain special case pivot operations: • pivot from any / all primary properties of the inbound set of nodes to the equivalent secondary property of any / all nodes (“wildcard” pivot in), and • reverse Pivot to Digraph (Edge) Nodes and reverse Pivot Across Digraph (Edge) Nodes (covered separately below). Syntax: <- * Example: Pivot from all primary properties to all nodes with an equivalent secondary property ( = to = ): • Pivot from a set of domains to all nodes with a secondary property that references the domains:

<- *

Usage Notes: • Pivoting in using the asterisk wildcard ( * ) is sometimes called a refs in pivot because it pivots from all nodes to all nodes that reference those nodes. • Pivoting in using the wildcard will return an instance of a node for each matching secondary property. For example, where a node may have the same for two different secondary properties (such as :domain and :zone on an inet:fqdn node), the pivot in will return two copies of the node. Results can be de- duplicated using the Storm uniq command.

1.14. Storm Reference - Pivoting 39 synapse Documentation, Release 0.1.0

• Pivoting using the wildcard is based on strong data typing within the Synapse data model, so will only pivot in from properties that match both and / . This means that the following nodes will not be returned by a wildcard pivot in: – Nodes with matching / but of different . For example, if a node’s primary property (such as a domain, type ) - happens to be referenced as as a different type (such as a string, type ) as a secondary property of another node, a wildcard pivot in to the inet:fqdn node will not return the node with the string value. – Digraph (edge) nodes, whose properties are of type (node definition, or , tuples). See Pivot to Digraph (Edge) Nodes and Pivot Across Digraph (Edge) Nodes for details on pivoting to / through those forms. • Other than digraph (edge) node navigation / traversal, pivot in can only be used with the wildcard ( * ). That is, pivot in does not support specifying a particular target form: inet:fqdn=woot.com <- inet:dns:a:fqdn The above query will return an error. A filter operation (see Storm Reference - Filtering) can be used to downs- elect the results of a wildcard pivot in operation to a specific set of forms: inet:fqdn=woot.com <- * +inet:dns:a Note that when attempting to specify a target form using Implicit Pivot Syntax, Storm currently (and incor- rectly) returns 0 nodes (even if nodes exist) instead of generating an error: inet:fqdn=woot.com <- inet:dns:a

1.14.3 Pivot With Join

The pivot and join operator ( -+> ) performs the specified pivot operation but joins the results with the inbound set of nodes. That is, the inbound nodes are retained and combined with the results of the pivot. Another way to look at the difference between a pivot and a join is that a pivot operation consumes nodes (the inbound set is discarded and only nodes resulting from the pivot operation are returned) but a pivot and join does not consume the inbound nodes. The pivot and join operator is used to: • retain the inbound nodes and pivot from the primary property of the inbound set of nodes to the equivalent secondary property of another set of nodes, • retain the inbound nodes and pivot from a secondary property of the inbound set of nodes to the equivalent primary property of another set of nodes, • retain the inbound nodes and pivot from any / all secondary properties of the inbound set of nodes to the equivalent primary property of any / all nodes (“wildcard” pivot out), and • retain the inbound nodes and pivot from a secondary property of the inbound set of nodes to the equivalent secondary property of another set of nodes. Syntax: -+> : : -+> -+> * : -+> : Examples: Pivot and join from primary property ( = ) to secondary property ( = ):

40 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

• Return a set of domains and all of their immediate subdomains:

-+> inet:fqdn:domain

Pivot and join from secondary property ( = ) to primary property ( = ): • Return a set of DNS A records and their associated IP addresses:

:ipv4 -+> inet:ipv4

Pivot and join from all secondary properties to all forms ( = to = ): • Return a set of WHOIS records and all nodes whose primary property equals any of the secondary properties of the WHOIS record (the asterisk ( * ) is a wildcard that indicates pivot to any applicable node):

-+> *

Pivot and join from secondary property ( = ) to secondary property ( = ): • Return the WHOIS records for a set of domains and the DNS A records for the same domains:

:fqdn -+> inet:dns:a:fqdn

Usage Notes: • A pivot and join using the wildcard ( * ) will pivot to all nodes whose primary property ( = ) matches a secondary property ( = ) of the inbound nodes. This excludes digraph nodes (such as refs or has nodes) because their primary property is a pair of ndefs (node definitions, or , tuples).

1.14.4 Pivot to Digraph (Edge) Nodes

Digraph (edge) nodes are of type edge or timeedge. These nodes (forms) are unique in that their primary property value is a pair of node definitions (type ndef) - that is, , tuples. (timeedge forms are comprised of two , tuples and an additional

-> has

• Return an article and the set of digraph nodes representing things “referenced” by the article:

-+> refs

1.14. Storm Reference - Pivoting 41 synapse Documentation, Release 0.1.0

• Pivot out from a person node to the set of timeedge digraph nodes representing places that person has been to (and when):

-> wentto

• Pivot out from a set of domains to the set of digraph nodes representing things that reference the domains:

-> refs:n2

Pivot in from a set of nodes whose ndefs (, ) are the second element (:n2) in a set of a digraph nodes: • Pivot in from an article to the set of digraph nodes representing things that “have” the article (e.g., people or organizations who authored the article):

<- has

Usage Notes: - The pivot out and pivot in operators have been optimized for digraph nodes. Because digraphs use ndef properties, Storm makes the following assumptions: • When pivoting to or from a set of nodes to a set of digraph nodes, pivot using the ndef (,) of the inbound nodes and not their primary property () alone. • When pivoting out to a digraph node, the inbound nodes’ , ndef will be the first element (:n1) of the digraph. You must explicitly specify :n2 to pivot to the second element. • When pivoting in to a digraph node, the inbound nodes’ , ndef will be the second element (:n2) of the digraph. It is not possible to pivot in to :n1. • Pivoting to / from digraph nodes is one of the specialized use cases for the pivot in ( <-) operator, however the primary use case of pivot in with digraph nodes is reverse edge traversal (see Pivot Across Digraph (Edge) Nodes). See Pivot In Operator for general limitations of the pivot in operator.

1.14.5 Pivot Across Digraph (Edge) Nodes

Because digraph nodes represent generic edge relationships, analytically we are often more interested in the nodes on “either side” of the edge than in the digraph node itself. For this reason, the pivot operators have been optimized to allow a syntax for easily navigating “across” these digraphs (edges). Syntax: -> | -> * | <- | <- * | Examples: • Traverse a set of has nodes to pivot from a person to all the things the person “has”:

-> has -> *

• Traverse a set of refs nodes to pivot from a set of domains to the articles that “reference” the domain:

<- refs <- media:news

• Traverse a set of wentto nodes to pivot from a person to the locations the person has visited:

-> wentto -> *

Usage Notes:

42 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

• Storm makes the following assumptions to optimize the two pivots: – For pivots out, the first pivot is to the digraph nodes’ :n1 property and the second pivot is from the digraph nodes’ :n2 property. – For pivots in, the first pivot is to the digraph nodes’ :n2 property and the second pivot is from the digraph nodes’ :n1 property. • Pivoting “across” the digraph nodes still performs two pivot operations (i.e., to the digraph nodes and then from them). As such it is still possible to apply an optional filter to the digraph nodes themselves before the second pivot.

1.14.6 Pivot to Tags

Pivot to tags syntax allows you to pivot from a set of nodes to the set of syn:tag nodes for the tags applied to those nodes. This includes: • pivot to all leaf tag nodes, • pivot to all tag nodes, • pivot to all tag nodes matching a specified prefix, and • pivot to tag nodes matching an exact tag. See the Synapse background documents for additional discussion of tags and syn:tag nodes. Syntax: -> # [ * | # .* | # ] Examples: Pivot to all leaf tag nodes: • Pivot from a set of domains to the syn:tag nodes for all leaf tags applied to those domains:

-> #

Pivot to ALL tag nodes: • Pivot from a set of files to the syn:tag nodes for all tags applied to those files:

-> #*

Pivot to all tag nodes matching the specified prefix: • Pivot from a set of IP addresses to the syn:tag nodes for all tags applied to those IPs that are part of the anonymized infrastructure tag tree:

-> #cno.infra.anon.*

Pivot to tag nodes exactly matching the specified tag: • Pivot from a set of nodes to the syn:tag node for #foo.bar (if present on the inbound set of nodes):

-> #foo.bar

Usage Notes: • Pivot to all tags ( #* ) and pivot by prefix matching ( #.* ) will match all tags in the relevant tag trees from the inbound nodes, not just the leaf tags. For example, for an inbound node with tag #foo.bar.baz, #* will return the syn:tag nodes for foo, foo.bar, and foo.bar.baz.

1.14. Storm Reference - Pivoting 43 synapse Documentation, Release 0.1.0

1.14.7 Pivot from Tags

Pivot from tags syntax allows you to pivot from a set of syn:tag nodes to the set of nodes that have those tags. Syntax: -> * | Examples: • Pivot to all domains tagged with tags from any of the inbound syn:tag nodes:

-> inet:fqdn

• Pivot to all nodes tagged with tags from any of the inbound syn:tag nodes:

-> *

Usage Notes: • In many cases, pivot from tags is functionally equivalent to Lift by Tag (#). That is, the following queries will both return all nodes tagged with #aka.feye.thr.apt1: syn:tag=aka.feye.thr.apt1 -> * #aka.feye.thr.apt1 Pivoting from tags is most useful when used in conjunction with Pivot to Tags - that is, taking a set of inbound nodes, pivoting to the syn:tag nodes for any associated tags (pivot to tags), and then pivoting out again to other nodes tagged with some or all of those tags (pivot from tags).

1.14.8 Implicit Pivot Syntax

If the target or source property of a pivot is readily apparent - that is, given the inbound and target forms, only one set of properties makes sense for that pivot - the properties do not have to be explicitly specified. This implicit pivot syntax allows users to enter more concise pivot queries in some cases. Implicit pivot syntax can be used to pivot from a primary property to a secondary property, as well as from a secondary property to a primary property. Examples: Pivot from primary property ( = ) to implicit secondary property ( = ): • Pivot from a set of domains to their associated DNS A records: Regular (full) syntax:

-> inet:dns:a:fqdn

Implicit syntax:

-> inet:dns:a

With implicit syntax, the target property :fqdn can be omitted because it is the only logical target given a set of inet:fqdn nodes as the source. Pivot from implicit secondary property ( = ) to primary property ( = ): • Pivot from a set of DNS A records to their associated IP addresses: Regular (full) syntax:

44 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

:ipv4 -> inet:ipv4

Implicit syntax:

-> inet:ipv4

With implicit syntax, the source property :ipv4 can be omitted because it is the only logical source given a set of inet:ipv4 nodes as the target. Use of multiple implicit pivots: • Pivot from a set of domains to their DNS A records and then to the associated IP addresses: Regular (full) syntax:

-> inet:dns:a:fqdn :ipv4 -> inet:ipv4

Implicit syntax:

-> inet:dns:a -> inet:ipv4

1.15 Storm Reference - Data Modification

Storm can be used to directly modify the Synapse hypergraph by: • adding or deleting nodes; • setting, modifying, or deleting properties on nodes; and • adding or deleting tags from nodes. While storm and the Synapse command line (cmdr - see Synapse Reference - Synapse Commands) are not optimal for adding or modifying large amounts of data, users gain a powerful degree of flexibility and efficiency through the ability to create or modify data on the fly. For adding or modifying larger amounts of data, it is preferable to use the Synapse feed utility (Synapse Tools - feed), the Synapse CSV tool (Synapse Tools - csvtool), or programmatic ingest of data to help automate the process.

Warning: The ability to add and modify data directly from Storm is powerful and convenient, but also means users can inadvertently modify (or even delete) data inappropriately through mistyped syntax or premature striking of the “enter” key. While some built-in protections exist within Synapse itself it is important to remember that there is no “are you ?” prompt before a Storm query executes. The following recommended best practices will help prevent inadvertent changes to the hypergraph: • Use the Synapse permissions system (see Synapse - Permissions) to enforce least privilege. Limit users to permissions appropriate for tasks they have been trained for / are responsible for. • Limit potentially destructive permissions even for trained / trusted users. Require the use of the Storm sudo command for significant / critical changes (such as the deletion of nodes). • Use extreme caution when constructing complex Storm queries that may modify (or delete) large numbers of nodes. It is strongly recommended that you validate the output of a query by first running the query on its own to ensure it returns the expected results (set of nodes) before permanently modifying (or deleting) those nodes.

See Storm Reference - Document Syntax Conventions for an explanation of the syntax format used below.

1.15. Storm Reference - Data Modification 45 synapse Documentation, Release 0.1.0

See Storm Reference - Type-Specific Storm Behavior for details on special syntax or handling for specific data types.

1.15.1 Edit Mode

To modify data in a Cortex using Storm, you must enter “edit mode”. The use of square brackets ( [] ) within a Storm query can be thought of as entering edit mode, with the data in the brackets specifying the changes to be made. This is true for changes involving nodes, properties, and tags. The only exception is the deletion of nodes, which is done using the Storm :ref:‘storm-delnode‘ command. The square brackets used for the Storm data modification syntax indicate “perform the enclosed changes” in a generic way. The brackets are shorthand to request any of the following: • Add Nodes • Add or Modify Properties • Delete Properties • Add Tags • Remove Tags This means that all of the above directives can be specified within a single set of brackets, in any combination and in any order. The only caveat is that a node must exist before it can be modified, so you must add a node before you add a secondary property or a tag. See Combining Data Modification Operations below for examples.

Warning: It is critical to remember that the brackets are NOT a boundary that segregates nodes; the brackets simply indicate the start and end of data modification operations. They do NOT separate “nodes the modifications should apply to” from “nodes they should not apply to”. Storm Operation Chaining with left-to-right processing order still applies. Any modification request that operates on previous Storm output will operate on EV- ERYTHING to the left of the modify operation, regardless of whether those nodes are within or outside the brackets.

Consider the following examples: • inet:fqdn#aka.feye.thr.apt1 [ inet:fqdn=somedomain.com +#aka.eset.thr. sednit ] The above Storm query will: – Lift all of the domains tagged #aka.feye.thr.apt1. – Create the node for domain somedomain.com (if it does not exist), or lift it if it does. – Apply the tag aka.eset.thr.sednit to the domain somedomain.com and all of the domains tagged aka.feye.thr.apt1 • [inet:ipv4=1.2.3.4 :asn=1111 inet:ipv4=5.6.7.8 :asn=2222] The above Storm query will: – Create (or lift) the node for IP 1.2.3.4. – Set the node’s :asn property to 1111. – Create (or lift) the node for IP 5.6.7.8. – Set the :asn property for both IPs to 2222.

46 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

1.15.2 Add Nodes

Operation to add the specified node(s) to a Cortex. Syntax: [ = . . . ] Examples: Create a simple node:

[ inet:fqdn=woot.com ]

Create a composite (comp) node:

[ inet:dns:a=(woot.com, 12.34.56.78) ]

Create a GUID node:

[ ou:org=2f92bc913918f6598bcf310972ebf32e]

[ ou:org="*"]

Create a digraph (edge) node:

[ refs=((media:news, 00a1f0d928e25729b9e86e2d08c127ce), (inet:fqdn, woot.com)) ]

Create multiple nodes:

[ inet:fqdn=woot.com inet:ipv4=12.34.56.78 hash:md5=d41d8cd98f00b204e9800998ecf8427e ]

Usage Notes: • Storm can create as many nodes as are specified within the brackets. It is not necessary to create only one node at a time. • For nodes specified within the brackets that do not already exist, Storm will create and return the node. For nodes that already exist, Storm will simply return that node. • When creating a whose consists of multiple components, the components must be passed as a comma-separated list enclosed in parentheses. • Once a node is created, its primary property ( = ) cannot be modified. The only way to “change” a node’s primary property is to create a new node (and optionally delete the old node). “Modifying” nodes therefore consists of adding, modifying, or deleting secondary properties (including universal properties) or adding or removing tags.

1.15.3 Add or Modify Properties

Operation to add (set) or change one or more properties on the specified node(s). The same syntax is used to apply a new property or modify an existing property. Syntax: [: = ... ] Examples: Add (or modify) secondary property:

1.15. Storm Reference - Data Modification 47 synapse Documentation, Release 0.1.0

[ :loc=us.oh.wilmington ]

Add (or modify) universal property:

[ .seen=("2017/08/01 01:23", "2017/08/01 04:56") ]

Add (or modify) a string property to a null value:

[ :summary="" ]

Usage Notes: • Additions or modifications to properties are performed on the output of a previous Storm query. • Storm will set or change the specified properties for all nodes in the current working set (i.e., all nodes resulting from Storm syntax to the left of the = statement(s)) for which that property is valid, whether those nodes are within or outside of the brackets. • Specifying a property will set the = if it does not exist, or modify (overwrite) the = if it already exists. There is no prompt to confirm overwriting of an existing property. • Storm will return an error if the inbound set of nodes contains any forms for which is not a valid property. For example, attempting to set a :loc property when the inbound nodes contain both domains and IP addresses will return an error as :loc is not a valid secondary property for a domain (inet:fqdn). • Secondary properties must be specified by their relative property name. For example, for the form foo:bar with the property baz (i.e., foo:bar:baz) the relative property name is specified as :baz. • Storm can set or modify any secondary property (including universal properties) except those explicitly defined as read-only ('ro' : 1) in the data model. Attempts to modify read only properties will return an error.

1.15.4 Delete Properties

Operation to delete (fully remove) one or more properties from the specified node(s).

Warning: Storm syntax to delete properties has the potential to be destructive if executed following an incorrect, badly formed, or mistyped query. Users are strongly encouraged to validate their query by first executing it on its own (without the delete property operation) to confirm it returns the expected nodes before adding the delete syntax. While the property deletion syntax cannot fully remove a node from the hypergraph, it is possible for a bad property deletion operation to irreversibly damage hypergraph pivoting and traversal.

Syntax: [ -: ... ] Examples: Delete a property:

[ -:loc ]

Delete multiple properties:

[ -:author -:summary ]

Delete property using elevated privileges:

48 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

sudo | [ -:phone ]

Usage Notes: • Property deletions are performed on the output of a previous Storm query. • Storm will delete the specified property / properties for all nodes in the current working set (i.e., all nodes resulting from Storm syntax to the left of the -: statement), whether those nodes are within or outside of the brackets. • Deleting a property fully removes the property from the node; it does not set the property to a null value. • Properties which are read-only ( 'ro' : 1 ) as specified in the data model cannot be deleted. • Storm edit operations may need to be executed using the Storm sudo command to succeed. (As a best practice, we strongly recommend requiring administrator permissions activated using sudo to carry out delete opera- tions.)

1.15.5 Delete Nodes

Nodes can be deleted from a Cortex using the Storm delnode command.

1.15.6 Add Tags

Operation to add one or more tags to the specified node(s). Syntax: [ +# [ =( , ) ]... ] Examples: Add tags:

[ +#aka.feye.thr.apt1 +#cno.infra.sink.hole ]

Add tag with timestamps:

[ +#cno.infra.sink.hole=(2014/11/06, 2016/11/06) ]

Usage Notes: • Tag additions are performed on the output of a previous Storm query. • Storm will add the specified tag(s) to all nodes in the current working set (i.e., all nodes resulting from Storm syntax to the left of the +# statement) whether those nodes are within or outside of the brackets. • Timestamps can be added to a tag to show a point in time or an interval during which the tag was known to be valid or applicable to the node in question. In the second example above, the timestamps on the tag cno. infra.sink.hole are meant to indicate that the domain was sinkholed between 11/6/2014 and 11/6/2016.) • Timestamps are applied only to the tags to which they are explicitly added. For example, adding a timestamp to the tag #foo.bar.baz does not add the timestamp to tags #foo.bar and #foo. • See the sections on time

1.15. Storm Reference - Data Modification 49 synapse Documentation, Release 0.1.0

1.15.7 Modify Tags

Tags are “binary” in that they are either applied to a node or they are not. The only modification that can be made to an existing tag is to add or update any associated timestamp, which can be done using the same syntax as Add Tags. To “change” the tag applied to a node, you must add the new tag and delete the old one. The Storm movetag command can be used to modify tags in bulk - that is, rename an entire set of tags, or move a tag to a different tag tree.

1.15.8 Remove Tags

Operation to delete one or more tags from the specified node(s). Removing a tag from a node differs from deleting the node representing a tag (a syn:tag node), which can be done using the Storm delnode command.

Warning: Storm syntax to remove tags has the potential to be destructive if executed on an incorrect, badly formed, or mistyped query. Users are strongly encouraged to validate their query by first executing it on its own to confirm it returns the expected nodes before adding the tag deletion syntax. In addition, it is essential to understand how removing a tag at a given position in a tag tree affects other tags within that tree. Otherwise, tags may be improperly left in place (“orphaned”) or inadvertently removed.

Syntax: [ -# ... ] Examples: Remove a tag:

[ -#cno.infra.anon.tor ]

Remove a tag using elevated privileges: sudo | [ -#cno.infra.anon.tor ]

Usage Notes: • Tag deletions are performed on the output of a previous Storm query. • Storm will delete the specified tag(s) from all nodes in the current working set (i.e., all nodes resulting from Storm syntax to the left of the -# statement), whether those nodes are within or outside of the brackets. • Deleting a leaf tag deletes only the leaf tag from the node. For example, [ -#foo.bar.baz ] will delete the tag #foo.bar.baz but leave the tags #foo.bar and #foo on the node. • Deleting a non-leaf tag deletes that tag and all tags below it in the tag hierarchy from the node. For example, [ -#foo ] used on a node with tags #foo.bar.baz and #foo.hurr.derp will remove all of the following tags: – #foo.bar.baz – #foo.hurr.derp – #foo.bar – #foo.hurr

50 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

– #foo • Storm edit operations may need to be executed using the Storm sudo command to succeed. (As a best practice, we strongly recommend requiring administrator permissions activated using sudo to carry out delete opera- tions.)

1.15.9 Combining Data Modification Operations

The square brackets representing edit mode are used for a wide range of operations, meaning it is possible to combine operations within a single set of brackets. Examples: Create a node and add secondary properties:

[ inet:ipv4=94.75.194.194 :loc=nl :asn=60781 ]

Create a node and add a tag:

[ inet:fqdn=blackcake.net +#aka.feye.thr.apt1 ]

1.16 Storm Reference - Subquery Syntax

TBD

1.17 Storm Reference - Variables

TBD

1.18 Storm Reference - Type-Specific Storm Behavior

TBD

1.19 Storm Reference - Storm Commands

Storm commands are built-in or custom commands that can be used with the Synapse storm command itself. As such, Storm commands must be executed following the storm command: storm or storm | [ | ] The pipe symbol ( | ) is used to send (pipe) the output from a Storm query to any of the Storm commands, and to send the output from a Storm command back to a Storm query. Built-in commands are native to the Storm library and loaded by default within a given Cortex. Built-in commands comprise a set of helper commands that perform a variety of specialized tasks that are useful regardless of the types of data stored in the Cortex or the types of analysis performed.

1.16. Storm Reference - Subquery Syntax 51 synapse Documentation, Release 0.1.0

Custom commands are Storm commands that have been added to a Cortex to invoke the execution of dynamically loaded modules. Dynamically loaded modules are typically custom modules that have been added to Synapse to support domain-specific analysis. For example, a knowledge domain that requires tracking of IP addresses might have access to a third-party service such as Maxmind to obtain up-to-date data on the assigned Autonomous System (AS) or geographical location of a given IP address. A custom maxmind module and associated Storm command could be added to Synapse to query the Maxmind database and update the appropriate secondary properties on the associated inet:ipv4 nodes directly from Storm. The full list of storm commands (built-in and custom) available in a given Cortex can be displayed with storm help. Help for a specific Storm command can be displayed with storm --help. This section details the usage and syntax for built-in Storm commands: • help • count • delnode • graph • iden • limit • max • min • movetag • noderefs • reindex • sleep • spin • sudo • uniq See Storm Reference - Document Syntax Conventions for an explanation of the syntax format used below. The Storm query language is covered in detail starting with the Storm Reference - Introduction section of the Synapse User Guide.

1.19.1 help

The help command (storm help) displays the list of available built-in commands and a brief message describing each command. Help on individual commands is available via --help. Syntax: cli> storm help count: Iterate through query results, and print the resulting number of nodes delnode: Delete nodes produced by the previous query logic. graph: Generate a subgraph from the given input nodes and command line options. help: List available commands and a brief description for each. iden: Lift nodes by iden. limit: Limit the number of nodes generated by the query in the given position. (continues on next page)

52 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

(continued from previous page) max: Consume nodes and yield only the one node with the highest value for a property. min: Consume nodes and yield only the one node with the lowest value for a property. movetag: Rename an entire tag tree and preserve time intervals. noderefs: Get nodes adjacent to inbound nodes, up to n degrees away. reindex: Use admin privileges to re index/normalize node properties. sleep: Introduce a delay between returning each result for the storm query. spin: Iterate through all query results, but do not yield any. sudo: Use admin privileges to bypass standard query permissions. uniq: Filter nodes by their uniq iden values.

For detailed help on any command, use --help complete. 0 nodes in 3 ms (0/sec).

1.19.2 count

The count command enumerates the number of nodes returned from a given Storm query and displays the resultant nodes and associated node count. Syntax cli> storm count --help usage: count [-h]

Iterate through query results, and print the resulting number of nodes which were lifted. This does yield the nodes counted.

Example:

foo:bar:size=20 | count

optional arguments: -h, --help show this help message and exit complete. 0 nodes in 3 ms (0/sec).

Examples: • Count the number of email address nodes: inet:email | count

• Count the number of DNS A records for the domain woot.com: inet:dns:a:fqdn=woot.com | count

Usage Notes: • count does not consume nodes, so Storm will stream the nodes being counted to the CLI output while the command executes. To count nodes without streaming the output, count can be piped to the spin command (i.e., | count | spin). Spin consumes nodes and so will prevent nodes processed by the count command from streaming.

1.19. Storm Reference - Storm Commands 53 synapse Documentation, Release 0.1.0

1.19.3 delnode

The delnode command deletes a node or set of nodes from a Cortex.

Warning: The Storm delnode command has the potential to be destructive if executed on an incorrect, badly formed, or mistyped query. Users are strongly encouraged to validate their query by first executing it on its own to confirm it returns the expected nodes before piping the query to the delnode command.

Syntax: cli> storm delnode --help usage: delnode [-h] [--force]

Delete nodes produced by the previous query logic.

(no nodes are returned)

Example

inet:fqdn=vertex.link | delnode optional arguments: -h, --help show this help message and exit --force Force delete even if it causes broken references (requires admin). complete. 0 nodes in 3 ms (0/sec).

Examples: • Delete the node for the domain woowoo.com: inet:fqdn=woowoo.com | delnode

• Forcibly delete all nodes with the #testing tag:

#testing | delnode --force

Usage Notes: • delnode operates on the output of a previous Storm query. • delnode will attempt to perform some basic sanity-checking to help prevent egregious mistakes. For example, delnode will return an error if you attempt to delete a node that is still referenced by another node (such as an inet:fqdn that is referenced by an inet:dns:a node). Similarly, delnode will return an error if you attempt to delete a syn:tag node if that tag is still applied to other nodes. However, delnode cannot prevent all mistakes. • The --force parameter will forcibly delete the nodes input to the command, regardless of any sanity-checking errors or other conditions. This parameter should be used with extreme caution as it may result in broken references within the Cortex.

54 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

1.19.4 graph

The graph command generates a subgraph based on a specified set of nodes and parameters. Syntax: cli> storm graph --help usage: graph [-h] [--degrees DEGREES] [--pivot PIVOT] [--filter FILTER] [--form-pivot FORM_PIVOT FORM_PIVOT] [--form-filter FORM_FILTER FORM_FILTER]

Generate a subgraph from the given input nodes and command line options. optional arguments: -h, --help show this help message and exit --degrees DEGREES How many degrees to graph out. --pivot PIVOT Specify a storm pivot for all nodes. (must quote) --filter FILTER Specify a storm filter for all nodes. (must quote) --form-pivot FORM_PIVOT FORM_PIVOT Specify a form specific pivot. --form-filter FORM_FILTER FORM_FILTER Specify a form specific filter. complete. 0 nodes in 4 ms (0/sec).

Examples: TBD

1.19.5 iden

The iden command lifts one or more nodes by their node identifier (node ID / iden). Syntax: cli> storm iden --help usage: iden [-h] [iden [iden ...]]

Lift nodes by iden.

Example:

iden b25bc9eec7e159dce879f9ec85fb791f83b505ac55b346fcb64c3c51e98d1175 | count positional arguments: iden Iden to lift nodes by. May be specified multiple times. optional arguments: -h, --help show this help message and exit complete. 0 nodes in 3 ms (0/sec).

Example: • Lift the node with node ID d7fb3ae625e295c9279c034f5d91a7ad9132c79a9c2b16eecffc8d1609d75849:

1.19. Storm Reference - Storm Commands 55 synapse Documentation, Release 0.1.0

iden d7fb3ae625e295c9279c034f5d91a7ad9132c79a9c2b16eecffc8d1609d75849

Usage Notes: • The node ID (iden) for a given node can be obtained by lifting the node using the --raw option to the storm command: – storm --raw inet:fqdn=woot.com

1.19.6 limit

The limit command restricts the number of nodes returned from a given Storm query to the specified number of nodes. Syntax:

cli> storm limit --help

usage: limit [-h] count

Limit the number of nodes generated by the query in the given position.

Example:

inet:ipv4 | limit 10

positional arguments: count The maximum number of nodes to yield.

optional arguments: -h, --help show this help message and exit

complete. 0 nodes in 3 ms (0/sec).

Example: • Lift ten IP address nodes:

inet:ipv4 | limit 10

Usage Notes: • If the limit number specified (i.e., limit 100) is greater than the total number of nodes returned from the Storm query, no limit will be applied to the resultant nodes (i.e., all nodes will be returned). • By design, limit imposes an artificial limit on the nodes returned by a query, which may impair effective analysis of data by restricting results. As such, limit is most useful for viewing a subset of a large result set or an exemplar node for a given form. • While limit returns a sampling of nodes, it is not statistically random for the purposes of population sampling for algorithmic use.

1.19.7 max

The max command returns the node from a given set that contains the highest value for a specified secondary property. Syntax:

56 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

cli> storm max --help usage: max [-h] propname

Consume nodes and yield only the one node with the highest value for a property.

Examples:

file:bytes +#foo.bar | max :size

file:bytes +#foo.bar | max file:bytes:size

positional arguments: propname optional arguments: -h, --help show this help message and exit complete. 0 nodes in 3 ms (0/sec).

Examples: • Return the DNS A record for woot.com with the most recent .seen value: inet:dns:a:fqdn=woot.com | max .seen

• Return the most recent WHOIS record for domain woot.com: inet:whois:rec:fqdn=woot.com | max :asof

• Return the most recent WHOIS record for domain woot.com using full property syntax: inet:whois:rec:fqdn=woot.com | max inet:whois:rec:asof

1.19.8 min

The min command returns the node from a given set that contains the lowest value for a specified secondary property. Syntax: cli> storm min --help usage: min [-h] propname

Consume nodes and yield only the one node with the lowest value for a property.

Examples:

file:bytes +#foo.bar | min :size

file:bytes +#foo.bar | min file:bytes:size

(continues on next page)

1.19. Storm Reference - Storm Commands 57 synapse Documentation, Release 0.1.0

(continued from previous page) positional arguments: propname optional arguments: -h, --help show this help message and exit complete. 0 nodes in 3 ms (0/sec).

Examples: • Return the DNS A record for woot.com with the oldest .seen value: inet:dns:a:fqdn=woot.com | min .seen

• Return the oldest WHOIS record for domain woot.com: inet:whois:rec:fqdn=woot.com | min :asof

• Return the oldest WHOIS record for domain woot.com using full property syntax: inet:whois:rec:fqdn=woot.com | min inet:whois:rec:asof

1.19.9 movetag

The movetag command moves a Synapse tag and its associated tag tree from one location in a tag hierarcy to another location. It is equivalent to “renaming” a given tag and all of its subtags. Moving a tag consists of: • Creating the new syn:tag node(s). • Copying the definitions (:title and :doc properties) from the old syn:tag node to the new syn:tag node. • Applying the new tag(s) to the nodes with the old tag(s). – If the old tag(s) have associated timestamps / time intervals, they will be applied to the new tag(s). • Deleting the old tag(s) from the nodes. • Setting the :isnow property of the old syn:tag node(s) to reference the new syn:tag node. – The old syn:tag nodes are not deleted. – Once the :isnow property is set, attempts to apply the old tag will automatically result in the new tag being applied. Syntax: cli> storm movetag --help usage: movetag [-h] oldtag newtag

Rename an entire tag tree and preserve time intervals.

Example:

movetag #foo.bar #baz.faz.bar

(continues on next page)

58 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

(continued from previous page) positional arguments: oldtag The tag tree to rename. newtag The new tag tree name. optional arguments: -h, --help show this help message and exit complete. 0 nodes in 3 ms (0/sec).

Examples: • Move the tag named #research to #internal.research: movetag #research #internal.research

• Move the tag tree #aka.fireeye.malware to #aka.feye.mal: movetag aka.fireeye.malware aka.feye.mal

Usage Notes:

Warning: movetag should be used with caution as when used incorrectly it can result in “deleted” (inadvertently moved / removed) or orphaned (inadvertently retained) tags. For example, in the second example query above, all aka.fireeye.malware tags are renamed aka.feye.mal, but the tag aka.fireeye still exists and is still applied to all of the original nodes. In other words, the result of the above command will be that nodes previously tagged aka.fireeye.malware will now be tagged both aka.feye.mal and aka.fireeye. Users may wish to test the command on sample data first to understand its effects before applying it in a live Cortex.

1.19.10 noderefs

The noderefs command returns all nodes that are adjacent to a given set of nodes (one pivot away by default). “Adjacent” means nodes whose primary or secondary properties reference either a primary or secondary property of the set of input nodes. Syntax: cli> storm noderefs --help usage: noderefs [-h] [-d DEGREES] [-te] [-j] [-otf OMIT_TRAVERSAL_FORM] [-ott OMIT_TRAVERSAL_TAG] [-of OMIT_FORM] [-ot OMIT_TAG] [-u]

Get nodes adjacent to inbound nodes, up to n degrees away.

Examples: The following examples show long-form options. Short form options exist and should be easier for regular use.

Get all nodes 1 degree away from a input node:

ask inet:ipv4=1.2.3.4 | noderefs

Get all nodes 1 degree away from a input node and include the source node:

(continues on next page)

1.19. Storm Reference - Storm Commands 59 synapse Documentation, Release 0.1.0

(continued from previous page) ask inet:ipv4=1.2.3.4 | noderefs --join

Get all nodes 3 degrees away from a input node and include the source node:

ask inet:ipv4=1.2.3.4 | noderefs --join --degrees 3

Do not include nodes of a given form in the output or traverse across them:

ask inet:ipv4=1.2.3.4 | noderefs --omit-form inet:dns:a

Do not traverse across nodes of a given form (but include them in the output):

ask inet:ipv4=1.2.3.4 | noderefs --omit-traversal-form inet:dns:a

Do not include nodes with a specific tag in the output or traverse across

˓→them:

ask inet:ipv4=1.2.3.4 | noderefs --omit-tag omit.nopiv

Do not traverse across nodes with a sepcific tag (but include them in the

˓→output):

ask inet:ipv4=1.2.3.4 | noderefs --omit-traversal-tag omit.nopiv

Accept multiple inbound nodes, and unique the output set of nodes across all

˓→input nodes:

ask inet:ipv4=1.2.3.4 inet:ipv4=1.2.3.5 | noderefs --degrees 4 --unique

optional arguments: -h, --help show this help message and exit -d DEGREES, --degrees DEGREES Number of degrees to traverse from the source node. -te, --traverse-edge Traverse Edge type nodes, if encountered, to the opposite side of them, if the opposite side has not yet been encountered. -j, --join Include source nodes in the output of the refs command. -otf OMIT_TRAVERSAL_FORM, --omit-traversal-form OMIT_TRAVERSAL_FORM Form to omit traversal of. Nodes of forms will still be the output. -ott OMIT_TRAVERSAL_TAG, --omit-traversal-tag OMIT_TRAVERSAL_TAG Tags to omit traversal of. Nodes with these tags will still be in the output. -of OMIT_FORM, --omit-form OMIT_FORM Forms which will not be included in the output or traversed. -ot OMIT_TAG, --omit-tag OMIT_TAG Forms which have these tags will not not be included in the output or traversed. -u, --unique Unique the output across ALL input nodes, instead of each input node at a time. complete. 0 nodes in 4 ms (0/sec).

60 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

Examples: TBD

1.19.11 reindex

The reindex command reindexes a given node property. This is an administrative command that is typically used when data model updates have been pushed to a Cortex and existing node properties must be migrated to the new model. Syntax: cli> storm reindex --help usage: reindex [-h] (--type TYPE | --subs | --form-counts | --fire-handler FIRE_HANDLER)

Use admin privileges to re index/normalize node properties.

Example:

foo:bar | reindex --subs

reindex --type inet:ipv4

NOTE: This is mostly for model updates and migrations. Use with caution and be very sure of what you are doing. optional arguments: -h, --help show this help message and exit --type TYPE Re-index all properties of a specified type. --subs Re-parse and set sub props. --form-counts Re-calculate all form counts. --fire-handler FIRE_HANDLER Fire onAdd/wasSet/runTagAdd commands for a fully qualified form/property or tag name on inbound nodes. complete. 0 nodes in 3 ms (0/sec).

Examples: TBD

1.19.12 sleep

The sleep command adds a delay in returning each result for a given Storm query. By default, query results are streamed back and displayed as soon as they arrive for optimal performance. A sleep delay effectively slows the display of results. Syntax: cli> storm sleep --help usage: sleep [-h] delay

(continues on next page)

1.19. Storm Reference - Storm Commands 61 synapse Documentation, Release 0.1.0

(continued from previous page) Introduce a delay between returning each result for the storm query.

NOTE: This is mostly used for testing / debugging.

Example:

#foo.bar | sleep 0.5

positional arguments: delay Delay in floating point seconds. optional arguments: -h, --help show this help message and exit complete. 0 nodes in 4 ms (0/sec).

Example: • Retrieve domain nodes from a Cortex every second: inet:email | sleep 1.0

1.19.13 spin

The spin command is used to suppress the output of a Storm query. Spin simply consumes all nodes sent to the command, so no nodes are output to the CLI. This allows you to execute a Storm query and view messages and results without displaying the associated nodes. Syntax: cli> storm spin --help usage: spin [-h]

Iterate through all query results, but do not yield any. This can be used to operate on many nodes without returning any.

Example:

foo:bar:size=20 [ +#hehe ] | spin

optional arguments: -h, --help show this help message and exit complete. 0 nodes in 3 ms (0/sec).

Examples: • Count the number of email addresses without displaying the inet:email nodes: inet:email | count | spin

• Add the tag #int.research to any domain containing the string “firefox” but do not display the nodes.

62 Chapter 1. Synapse User Guide synapse Documentation, Release 0.1.0

inet:fqdn~=firefox [+#int.research] | spin

1.19.14 sudo

The sudo command executes a Storm query with elevated privileges. The Synapse permissions system can be used to grant or restrict permissions to users and groups. As a best practice, we strongly recommend restricting potentially dangerous Storm operations (e.g., movetag or deln- ode) and requiring the use of sudo to perform these actions. Syntax: cli> storm sudo --help usage: sudo [-h]

Use admin privileges to bypass standard query permissions.

Example:

sudo | [ inet:fqdn=vertex.link ] optional arguments: -h, --help show this help message and exit complete. 0 nodes in 3 ms (0/sec).

Example: • Delete a node using elevated privileges: sudo | inet:fqdn=mydomain.com | delnode

1.19.15 uniq

The uniq command removes duplicate results from a Storm query. Results are uniqued based on each node’s node identifier (node ID / iden) so that only the first node with a given node ID is returned. Syntax: cli> storm uniq --help usage: uniq [-h]

Filter nodes by their uniq iden values. When this is used a Storm pipeline, only the first instance of a given node is allowed through the pipeline.

Examples:

#badstuff +inet:ipv4 ->* | uniq

(continues on next page)

1.19. Storm Reference - Storm Commands 63 synapse Documentation, Release 0.1.0

(continued from previous page) optional arguments: -h, --help show this help message and exit complete. 0 nodes in 2 ms (0/sec).

Examples: • Lift all of the unique IP addresses that domains associated with the Fancy Bear threat group have resolved to: inet:fqdn#aka.threatconnect.thr.fancybear -> inet:dns:a -> inet:ipv4 | uniq

Many of the concepts above are closely related and this outline represents a reasonable effort to introduce concepts in a logical order. However, it is difficult to fully understand the potential of Synapse and hypergraphs without grasping the power of the Storm query language to understand, manipulate, and annotate data. Similarly, it’s hard to understand the effectiveness of Storm without knowledge of the underlying data model. The outline above is our suggested order but readers are encouraged to skip around or revisit earlier sections after digesting later sections to better see how these topics are tied together.

64 Chapter 1. Synapse User Guide CHAPTER 2

Synapse DevOps Guide

This DevOps Guide is written by and for Synapse developers and operators. This is focused towards individuals who are looking to deploy Synapse (such as Cortex Hypergraphs or Axon blob stores); looking to use Synapse components in their own projects; or those looking to contribute back to the Synapse project. The DevOps Guide is a living document and will continue to be updated and expanded. The current sections are:

Note: This document was generated from a single .rst file

2.1 Synapse Performance - 1

Measuring the performance of a synapse cortex is a complex undertaking which depends not only on the test config- urations, but the “shape” of the data being ingested and queried. These tests attempt to take an accurate measure of a set of “bookend” data sets selected specifically to demonstrate best case and worst case ingest performance.

2.1.1 A Note on Comparisons

“It doesn’t matter that a 747 can carry more passengers than the Space Shuttle, when the mission is to repair a ” -visi When comparing these numbers to benchmarks published by various big data systems such as Hadoop and Elastic Search, it is critical to keep in mind the fundamental difference between a knowledge system like a synapse cortex versus a simple indexer such as Elastic Search. A knowledge system is required to deconflict all new data against what is already known. This means for each new node added to the hypergraph, it must atomically determine if that node already exists so there is only ever one node which represents a particular thing. While many big data systems claim this type of ACID compliant deconfliction is possible, our testing has shown that the claimed performance of these types of systems is drastically reduced when required to carry out atomic check-and-add operations. However, there is also an advantage for a deconflicted knowledge system. When encountering a node which has been previously observed, the system does not create a new node. This has the counterintuitive effect of making a cortex

65 synapse Documentation, Release 0.1.0 typically become faster as it ingests more data. This performance increase is especially true when ingesting data with many recurrent nodes.

2.1.2 Test Data Sets

Majestic Million

The Majestic Million is a ranked list of a million FQDNs which is freely available at http://downloads.majestic.com/ majestic_million.csv. This dataset was selected for performance benchmarking due to it being a “worst case” bookend. Within a cor- tex, inet:fqdn nodes undergo extensive normalization and often the creation of one inet:fqdn node causes the cre- ation of several others. For example, creating inet:fqdn=www.woot.com would subsequently cause the creation of inet:fqdn=woot.com and inet:fqdn=com. Additionally, the FQDNs within the Majestic Million dataset are already deconflicted, meaning each FQDN only occurs once within the dataset. This causes every record to deconflict and create new nodes.

Non-Deconflicted

A synapse cortex is also capable of ingestion and indexing of “instance knowledge” which is not typically deconflicted. The primary property for such nodes is typically a system generated GUID and insert times are typically higher. This test is intended to be close to a “best case” scenario where node insertion is not being atomically deconflicted and node properties are not subject to extensive normalization rules.

2.1.3 Cortex Configurations

Each of the supported storage technologies used by a synapse cortex are tested. Where possible, tests are executed with minimal or no specialized configuration in an attempt to show performance capabilities without the use of exotic configuration. All cortex instances are configured without caching in an attempt to measure the speed of the storage layer implementations rather than the caching subsystem. A production cortex configured with caches is likely to perform queries much faster than these results. ram-1

The RAM storage backing provides cortex storage and indexing using native python data structures such as dictionaries and lists. This configuration is a highly performant cortex typically used for hypergraph data which can fit in system memory. For these tests, the RAM cortex is initialized with default configuration options. lmdb-1

The LMDB storage backing provides cortex storage and indexing using the Symas Lightning DB available here: https://symas.com/lightning-memory-mapped-database/ For these tests, the lmdb cortex is initialized with default configuration options. sqlite-1

The sqlite storage layer provides cortex storage and indexing using the SQLite embedded database library which is built into the python interpreter. For these tests, the sqlite-1 cortex is intitialized with default values.

66 Chapter 2. Synapse DevOps Guide synapse Documentation, Release 0.1.0 postgres-1

The Postgres storage layer provides cortex storage and indexing using the Postgresql Database available here: https: //www.postgresql.org/. For these tests, the Postgresql cortex is initialized with default values communicating with a default Postgresql 9.5 database on Ubuntu 16.04 LTS. telepath-x3

The Telepath cluster test is designed to measure the scalability of a multi-cortex federation which is operating with the assumption of shard-based division of node creation across several cortex hypergraphs. The primary purpose of the test is to determine the expected overhead of cluster logic and network protocol efficiency. The remote cortexes are simple RAM cortexes.

2.1.4 Test Systems

The current benchmark testing environment is a cluster of 3 hosts with the following hardware: • Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz (8 cores) • 256 GB Memory • 1000 base T network interface ( 1 Gbps ) • 8x 7200 RPM Seagate ST8000NM0055-1RM112 disks

2.1.5 Results

Each of the test results below shows how the various test configurations perform under the different benchmark tests. In sections below, we discuss results for individual configurations and what that means when selecting how to select a cortex configuration for a given purpose.

2.1. Synapse Performance - 1 67 synapse Documentation, Release 0.1.0

benchmark ram-1 lmdb-1 sqlite-1 postgresql-1 telepath-x3 add w/deconf (nodes/sec) 3,347 1,478 385 336 query node (queries/sec ) 21,296 7,610 8681 1,304 add w/o deconf (nodes/sec) 11,460 6,310 911 2,473 32,779 ram-1

As expected, the ram-1 cortex has the advantage of speed. In instances where it is possible to dedicate enough memory to a Cortex, this configuration can be deployed to achieve maximum deconfliction performance as well as being the fastest to query. However, for large data sets this definitely puts a limit on use of 32 bit platforms and lower memory server configurations. Another potential limitation in the use of a ram cortex is startup time. In the case of a ram cortex which has enabled persistence, there may be a very long startup time during a reboot due to the cortex needing to playback change events on disk to reach the current state. Further testing will profile how much ram is expected for use by a cortex with a given population of nodes. lmdb-1

The lmdb-1 configuration strikes a powerful balance between performance and persistence. It is highly likely that this configuration will become the recommended default configuration for production deployments. sqlite-1

The sqlite-1 configuration demonstrates the powerful query speed of the underlying sqlite db backend. However, transactional limits and insert speeds are quite a bit slower. This makes an sqlite Cortex most ideal for use with large datasets which grow a bit more slowly but are expected to be queried very quickly.

68 Chapter 2. Synapse DevOps Guide synapse Documentation, Release 0.1.0 postgres-1

The postgres-1 configuration is the only test configuration which uses a storage mechanism which resides outside the python process running the Cortex. This requires the use of Inter Process Communication (IPC) using local pipes, and forces many context switches in comparison with the other configurations. However, future testing of parallel query execution is expected to demonstrate postgres as a highly capable parallel query platform, potentially making it more attractive for use with a larger number of querying clients. telepath x3

Current results show highly efficient scale gains when using multiple cortexes in a federated configuration. However, the current testing environment involves the use of only 3 systems. Future scale testing using additional hardware will be a better estimate of performance in a truly production scale cluster. That being said, current results are promising. • scale efficiency: 2.8 / 3.0 Future testing of parallel query execution is expected to achieve even higher levels of efficiency.

2.1.6 Additional Tests

Over the course of subsequent releases, a table will be added here showing the performance of releases over time using line graphs showing the various test results over time. Additionally, tests which measure parallel query performance will be implemented to demonstrate storage backings which operate well or poorly under highly parallel query load.

Note: This document was generated from a notebook with interspersed markdown (only) cells. This alert came from its own raw nbconvert cell. Of note - Jupyter markdown prefers to use # characters to denote headers. These, when converted to RST can come across rather awkward. It’s important to only have a single header with a single # character otherwise the toctree directive used for indexing will render things quite awkwardly.

2.2 Synapse Performance - 2

Measuring the performance of a synapse cortex is a complex undertaking which depends not only on the test config- urations, but the “shape” of the data being ingested and queried. These tests attempt to take an accurate measure of a set of “bookend” data sets selected specifically to demonstrate best case and worst case ingest performance.

2.2.1 A Note on Comparisons

“It doesn’t matter that a 747 can carry more passengers than the Space Shuttle, when the mission is to repair a satellite” -visi When comparing these numbers to benchmarks published by various big data systems such as Hadoop and Elastic Search, it is critical to keep in mind the fundamental difference between a knowledge system like a synapse cortex versus a simple indexer such as Elastic Search. A knowledge system is required to deconflict all new data against what is already known. This means for each new node added to the hypergraph, it must atomically determine if that node already exists so there is only ever one node which represents a particular thing. While many big data systems claim this type of ACID compliant deconfliction is possible, our testing has shown that the claimed performance of these types of systems is drastically reduced when required to carry out atomic check-and-add operations.

2.2. Synapse Performance - 2 69 synapse Documentation, Release 0.1.0

However, there is also an advantage for a deconflicted knowledge system. When encountering a node which has been previously observed, the system does not create a new node. This has the counterintuitive effect of making a cortex typically become faster as it ingests more data. This performance increase is especially true when ingesting data with many recurrent nodes.

2.2.2 Test Data Sets

Majestic Million

The Majestic Million is a ranked list of a million FQDNs which is freely available at http://downloads.majestic.com/ majestic_million.csv. This dataset was selected for performance benchmarking due to it being a “worst case” bookend. Within a cor- tex, inet:fqdn nodes undergo extensive normalization and often the creation of one inet:fqdn node causes the cre- ation of several others. For example, creating inet:fqdn=www.woot.com would subsequently cause the creation of inet:fqdn=woot.com and inet:fqdn=com. Additionally, the FQDNs within the Majestic Million dataset are already deconflicted, meaning each FQDN only occurs once within the dataset. This causes every record to deconflict and create new nodes.

Non-Deconflicted

A synapse cortex is also capable of ingestion and indexing of “instance knowledge” which is not typically deconflicted. The primary property for such nodes is typically a system generated GUID and insert times are typically higher. This test is intended to be close to a “best case” scenario where node insertion is not being atomically deconflicted and node properties are not subject to extensive normalization rules.

2.2.3 Cortex Configurations

Each of the supported storage technologies used by a synapse cortex are tested. Where possible, tests are executed with minimal or no specialized configuration in an attempt to show performance capabilities without the use of exotic configuration. All cortex instances are configured without caching in an attempt to measure the speed of the storage layer implementations rather than the caching subsystem. A production cortex configured with caches is likely to perform queries much faster than these results. ram-1

The RAM storage backing provides cortex storage and indexing using native python data structures such as dictionaries and lists. This configuration is a highly performant cortex typically used for hypergraph data which can fit in system memory. For these tests, the RAM cortex is initialized with default configuration options. lmdb-1

The LMDB storage backing provides cortex storage and indexing using the Symas Lightning DB available here: https://symas.com/lightning-memory-mapped-database/ For these tests, the lmdb cortex is initialized with default configuration options. sqlite-1

The sqlite storage layer provides cortex storage and indexing using the SQLite embedded database library which is built into the python interpreter. For these tests, the sqlite-1 cortex is intitialized with default values.

70 Chapter 2. Synapse DevOps Guide synapse Documentation, Release 0.1.0 postgres-1

The Postgres storage layer provides cortex storage and indexing using the Postgresql Database available here: https: //www.postgresql.org/. For these tests, the Postgresql cortex is initialized with default values communicating with a default Postgresql 9.5 database on Ubuntu 16.04 LTS. telepath-x3

The Telepath cluster test is designed to measure the scalability of a multi-cortex federation which is operating with the assumption of shard-based division of node creation across several cortex hypergraphs. The primary purpose of the test is to determine the expected overhead of cluster logic and network protocol efficiency. The remote cortexes are simple RAM cortexes.

2.2.4 Test Systems

The current benchmark testing environment is a cluster of 3 hosts with the following hardware: • Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz (8 cores) • 256 GB Memory • 1000 base T network interface ( 1 Gbps ) • 8x 7200 RPM Seagate ST8000NM0055-1RM112 disks

Note: The following results section shows a broken link for a image. This image will be shown at the end.

2.2.5 Results

Each of the test results below shows how the various test configurations perform under the different benchmark tests. In sections below, we discuss results for individual configurations and what that means when selecting how to select a cortex configuration for a given purpose.

synapse/devops/attachment:synapse_bench.png

Fig. 1: synapse_bench.png

benchmark ram-1 lmdb-1 sqlite-1 postgresql-1 telepath-x3 add w/deconf (nodes/sec) 3,347 1,478 385 336 query node (queries/sec) 21,296 7,160 8,681 1,304 add w/o deconf (nodes/sec) 11,460 6,310 911 2,473 32,779

2.2. Synapse Performance - 2 71 synapse Documentation, Release 0.1.0 ram-1

As expected, the ram-1 cortex has the advantage of speed. In instances where it is possible to dedicate enough memory to a Cortex, this configuration can be deployed to achieve maximum deconfliction performance as well as being the fastest to query. However, for large data sets this definitely puts a limit on use of 32 bit platforms and lower memory server configurations. Another potential limitation in the use of a ram cortex is startup time. In the case of a ram cortex which has enabled persistence, there may be a very long startup time during a reboot due to the cortex needing to playback change events on disk to reach the current state. Further testing will profile how much ram is expected for use by a cortex with a given population of nodes. lmdb-1

The lmdb-1 configuration strikes a powerful balance between performance and persistence. It is highly likely that this configuration will become the recommended default configuration for production deployments. sqlite-1

The sqlite-1 configuration demonstrates the powerful query speed of the underlying sqlite db backend. However, transactional limits and insert speeds are quite a bit slower. This makes an sqlite Cortex most ideal for use with large datasets which grow a bit more slowly but are expected to be queried very quickly. postgres-1

The postgres-1 configuration is the only test configuration which uses a storage mechanism which resides outside the python process running the Cortex. This requires the use of Inter Process Communication (IPC) using local pipes, and forces many context switches in comparison with the other configurations. However, future testing of parallel query execution is expected to demonstrate postgres as a highly capable parallel query platform, potentially making it more attractive for use with a larger number of querying clients. telepath x3

Current results show highly efficient scale gains when using multiple cortexes in a federated configuration. However, the current testing environment involves the use of only 3 systems. Future scale testing using additional hardware will be a better estimate of performance in a truly production scale cluster. That being said, current results are promising. • scale efficiency: 2.8 / 3.0 Future testing of parallel query execution is expected to achieve even higher levels of efficiency.

Note: This image is shown using a rawNbConvert cell and the RST image code

72 Chapter 2. Synapse DevOps Guide synapse Documentation, Release 0.1.0

2.2.6 Additional Tests

Over the course of subsequent releases, a table will be added here showing the performance of releases over time using line graphs showing the various test results over time. Additionally, tests which measure parallel query performance will be implemented to demonstrate storage backings which operate well or poorly under highly parallel query load.

Note: This document was generated from a single raw nbconvert cell containing only RST

2.3 Synapse Performance - 3

Measuring the performance of a synapse cortex is a complex undertaking which depends not only on the test config- urations, but the “shape” of the data being ingested and queried. These tests attempt to take an accurate measure of a set of “bookend” data sets selected specifically to demonstrate best case and worst case ingest performance.

2.3.1 A Note on Comparisons

“It doesn’t matter that a 747 can carry more passengers than the Space Shuttle, when the mission is to repair a satellite” -visi When comparing these numbers to benchmarks published by various big data systems such as Hadoop and Elastic Search, it is critical to keep in mind the fundamental difference between a knowledge system like a synapse cortex versus a simple indexer such as Elastic Search. A knowledge system is required to deconflict all new data against what is already known. This means for each new node added to the hypergraph, it must atomically determine if that node already exists so there is only ever one node which represents a particular thing. While many big data systems claim

2.3. Synapse Performance - 3 73 synapse Documentation, Release 0.1.0 this type of ACID compliant deconfliction is possible, our testing has shown that the claimed performance of these types of systems is drastically reduced when required to carry out atomic check-and-add operations. However, there is also an advantage for a deconflicted knowledge system. When encountering a node which has been previously observed, the system does not create a new node. This has the counterintuitive effect of making a cortex typically become faster as it ingests more data. This performance increase is especially true when ingesting data with many recurrent nodes.

2.3.2 Test Data Sets

Majestic Million

The Majestic Million is a ranked list of a million FQDNs which is freely available at http://downloads.majestic.com/ majestic_million.csv. This dataset was selected for performance benchmarking due to it being a “worst case” bookend. Within a cor- tex, inet:fqdn nodes undergo extensive normalization and often the creation of one inet:fqdn node causes the cre- ation of several others. For example, creating inet:fqdn=www.woot.com would subsequently cause the creation of inet:fqdn=woot.com and inet:fqdn=com. Additionally, the FQDNs within the Majestic Million dataset are already deconflicted, meaning each FQDN only occurs once within the dataset. This causes every record to deconflict and create new nodes.

Non-Deconflicted

A synapse cortex is also capable of ingestion and indexing of “instance knowledge” which is not typically deconflicted. The primary property for such nodes is typically a system generated GUID and insert times are typically higher. This test is intended to be close to a “best case” scenario where node insertion is not being atomically deconflicted and node properties are not subject to extensive normalization rules.

2.3.3 Cortex Configurations

Each of the supported storage technologies used by a synapse cortex are tested. Where possible, tests are executed with minimal or no specialized configuration in an attempt to show performance capabilities without the use of exotic configuration. All cortex instances are configured without caching in an attempt to measure the speed of the storage layer implementations rather than the caching subsystem. A production cortex configured with caches is likely to perform queries much faster than these results. ram-1

The RAM storage backing provides cortex storage and indexing using native python data structures such as dictionaries and lists. This configuration is a highly performant cortex typically used for hypergraph data which can fit in system memory. For these tests, the RAM cortex is initialized with default configuration options. lmdb-1

The LMDB storage backing provides cortex storage and indexing using the Symas Lightning DB available here: https://symas.com/lightning-memory-mapped-database/ For these tests, the lmdb cortex is initialized with default configuration options.

74 Chapter 2. Synapse DevOps Guide synapse Documentation, Release 0.1.0 sqlite-1

The sqlite storage layer provides cortex storage and indexing using the SQLite embedded database library which is built into the python interpreter. For these tests, the sqlite-1 cortex is intitialized with default values. postgres-1

The Postgres storage layer provides cortex storage and indexing using the Postgresql Database available here: https: //www.postgresql.org/. For these tests, the Postgresql cortex is initialized with default values communicating with a default Postgresql 9.5 database on Ubuntu 16.04 LTS. telepath-x3

The Telepath cluster test is designed to measure the scalability of a multi-cortex federation which is operating with the assumption of shard-based division of node creation across several cortex hypergraphs. The primary purpose of the test is to determine the expected overhead of cluster logic and network protocol efficiency. The remote cortexes are simple RAM cortexes.

2.3.4 Test Systems

The current benchmark testing environment is a cluster of 3 hosts with the following hardware: • Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz (8 cores) • 256 GB Memory • 1000 base T network interface ( 1 Gbps ) • 8x 7200 RPM Seagate ST8000NM0055-1RM112 disks

2.3.5 Results

Each of the test results below shows how the various test configurations perform under the different benchmark tests. In sections below, we discuss results for individual configurations and what that means when selecting how to select a cortex configuration for a given purpose.

2.3. Synapse Performance - 3 75 synapse Documentation, Release 0.1.0

benchmark ram-1 lmdb-1 sqlite-1 postgresql-1 telepath-x3 add w/deconf (nodes/sec) 3,347 1,478 385 336 query node (queries/sec ) 21,296 7,610 8681 1,304 add w/o deconf (nodes/sec) 11,460 6,310 911 2,473 32,779 ram-1

As expected, the ram-1 cortex has the advantage of speed. In instances where it is possible to dedicate enough memory to a Cortex, this configuration can be deployed to achieve maximum deconfliction performance as well as being the fastest to query. However, for large data sets this definitely puts a limit on use of 32 bit platforms and lower memory server configurations. Another potential limitation in the use of a ram cortex is startup time. In the case of a ram cortex which has enabled persistence, there may be a very long startup time during a reboot due to the cortex needing to playback change events on disk to reach the current state. Further testing will profile how much ram is expected for use by a cortex with a given population of nodes. lmdb-1

The lmdb-1 configuration strikes a powerful balance between performance and persistence. It is highly likely that this configuration will become the recommended default configuration for production deployments. sqlite-1

The sqlite-1 configuration demonstrates the powerful query speed of the underlying sqlite db backend. However, transactional limits and insert speeds are quite a bit slower. This makes an sqlite Cortex most ideal for use with large datasets which grow a bit more slowly but are expected to be queried very quickly.

76 Chapter 2. Synapse DevOps Guide synapse Documentation, Release 0.1.0 postgres-1

The postgres-1 configuration is the only test configuration which uses a storage mechanism which resides outside the python process running the Cortex. This requires the use of Inter Process Communication (IPC) using local pipes, and forces many context switches in comparison with the other configurations. However, future testing of parallel query execution is expected to demonstrate postgres as a highly capable parallel query platform, potentially making it more attractive for use with a larger number of querying clients. telepath x3

Current results show highly efficient scale gains when using multiple cortexes in a federated configuration. However, the current testing environment involves the use of only 3 systems. Future scale testing using additional hardware will be a better estimate of performance in a truly production scale cluster. That being said, current results are promising. • scale efficiency: 2.8 / 3.0 Future testing of parallel query execution is expected to achieve even higher levels of efficiency.

2.3.6 Additional Tests

Over the course of subsequent releases, a table will be added here showing the performance of releases over time using line graphs showing the various test results over time. Additionally, tests which measure parallel query performance will be implemented to demonstrate storage backings which operate well or poorly under highly parallel query load. The following sections are still under development: • Doc 1 • Doc 2

2.3. Synapse Performance - 3 77 synapse Documentation, Release 0.1.0

78 Chapter 2. Synapse DevOps Guide CHAPTER 3

Synapse Contributors Guide

This Contributors Guide is written for people who will be working on the Synapse code base, contributing to it via code patches, or maintaining written documentation. The Contributors Guide is a living document and will continue to be updated and expanded. The current sections are:

3.1 Contributing to Synapse

• Project Style Guide. • Git Hook & Syntax Checking. • Contribution Process.

3.1.1 Project Style Guide

The following items should be considered when contributing to Synapse: • The project is not currently strictly PEP8 compliant. Compliant sections include the following: – Whitespace in Expressions and Statements. – Programming Recommendations regarding singleton comparison (use ‘is’ instead of equality operators). • Please keep line lengths under 120 characters. • Use single quotes for string constants (including docstrings) unless double quotes are required.

# Do this foo='1234' # NOT this foo="1234"

• Use a single line break between top level functions and class definitions, and class methods. This helps conserve vertical space.

79 synapse Documentation, Release 0.1.0

– Do this

import foo import duck

def bar(): return True

def baz(): return False

class Obj(object):

def __init__(self, a): self.a=a

def gimmeA(self): return self.a

* NOT this import foo import duck

def bar(): return True

def baz(): return False

class Obj(object):

def __init__(self, a): self.a=a

def gimmeA(self): return self.a

• Use Google style Python docstrings. This format is very readable and will allow type hinting for IDE users. See the following notes below about our slight twist on this convention. – Use ‘” quotes instead of “”” for starting/stoping doc strings. – Google Style typically has the summary line after the opening ‘” marker. Place this summary value on the new line following the opening ‘” marker. – More information about Google Style docstrings (and examples) can be found at the examples here. – We use Napoleon for parsing these doc strings. More info here. – Synapse as a project is not written using the Napoleon format currently but all new modules should audhere to that format. – Synapse acceptable example:

80 Chapter 3. Synapse Contributors Guide synapse Documentation, Release 0.1.0

def fooTheBar(param1, param2, **kwargs): ''' Summary line goes first.

Longer description lives here. It can be a bunch of stuff across multiple blocks if necessary.

Example: Examples should be given using either the ``Example`` section. Sections support any reStructuredText formatting, including literal blocks::

woah = fooTheBar('a', 'b', duck='quacker')

Section breaks are created by resuming unindented text. Section breaks are also implicitly created anytime a new section starts.

`PEP 484`_ type annotations are supported. If attribute, parameter, and return types are annotated according to `PEP 484`_, they do not need to be included in the docstring:

Args: param1 (int): The first parameter. param2 (str): The second parameter.

Keyword Arguments: duck (str): Optional keyword args which come in via **kwargs call ˓→conventions, which modify function behavior, should be documented

˓→under the Keyword Args section.

Returns: bool: The return value. True for success, False otherwise.

The ``Returns`` section supports any reStructuredText

˓→formatting, including literal blocks::

{ 'param1': param1, 'param2': param2 }

Raises: AttributeError: The ``Raises`` section is a list of all exceptions that are relevant to the interface. ValueError: If `param2` is equal to `param1`.

.. _PEP 484: https://www.python.org/dev/peps/pep-0484/

''' # Do stuff the with args...

• Imports should first be sorted in order of shortest to longest import, then by alphabetical order (when lengths match). Imports should be ordered starting from the Python standard library first, then any third party packages,

3.1. Contributing to Synapse 81 synapse Documentation, Release 0.1.0

then any Synapse specific imports. The following example shows the recommended styling for imports:

# Stdlib import logging import collections # Third Party Code import barlib.duck as b_duck import foolib.thing as f_thing # Synapse Code import synapse.common as s_common import synapse.compat as s_compat import synapse.cortex as s_cortex import synapse.lib.config as s_config

• Previously we used * imports in the Synapse codebase (especially around synapse.exc and synapse.common). If common functions or exceptions are needed, import synapse.common as noted above, and both the common functions and the entirety of synapse.exc exceptions will be available. This provides a consistent manner for referencing common functions and Synapse specific exception classes. New code should generally not use * imports. Here is an example:

# Do this import synapse.common as s_common tick= s_common.now() if tick< 1000000000: raise s_common.HitMaxTime(mesg='We have gone too far!')

# NOT this from synapse.common import * tick= now() if tick< 1000000000: raise HitMaxTime(mesg='We have gone too far!')

• Function names should follow the mixedCase format for anything which is exposed as a externally facing API on a object or module.

# Do this fooTheBar() # NOT this foo_the_bar()

• Private methods should be marked as such with a proceeding underscore.

# Do this _internalThing() # NOT this privateInternalThingDontUseMe()

– The corralary to this is that any function which is not private may be called arbitrarily at any time, so avoid public API functions which are tightly bound to instance state. For example, if a processing routine is broken into smaller subroutines for readability or testability, these routines are likely private and should not be exposed to outside callers. • Function calls with mandatory arguments should be called with positional arguments. Do not use keyword arguments unless neccesary.

def foo(a, b, duck=None): print(a, b, duck) (continues on next page)

82 Chapter 3. Synapse Contributors Guide synapse Documentation, Release 0.1.0

(continued from previous page)

# Do this foo('a','b', duck='quacker') # Not this foo(a='a', b='b', duck='quacker')

• Avoid the use of @property decorators. They do not reliably work over the telepath RMI. • Logging should be setup on a per-module basis, with loggers created using calls to log- ging.getLogger(__name__). This allows for module level control of loggers as neccesary. – Logger calls should use logging string interpolation, instead of using % or .format() methods. See Python Logging module docs for reference. – Example:

# Get the module level logger logger= logging.getLogger(__name__) # Do this - it only forms the final string if the message is # actually going to be logged logger.info('I am a message from %s about %s','bob','a duck') # NOT this - it performs the string format() call regardless of # whether or not the message is going to be logged. logger.info('I am a message from {} about {}'.format('bob','a duck'))

• It may be neccesary from time to time to include non-ASCII characters. Use UTF8 formatting for such source files and use the following encoding declaration at the top of the source file.

#-*- coding: utf-8 -*-

• Convenience methods are available for unit tests, primarily through the SynTest class. This is a subclass of unittest.TestCase and provides many short aliases for the assert* functions that TestCase provides. – Ensure you are closing resources which may be open with test cases. Many Synapse objects may be used as content managers which make this easy for test authors. • Avoid the use of the built-in re module. Instead use the third-party regex module. regex is preferred due to known bugs with unicode in the re module. Additionally, regex does provide some performance benefits over re, especially when using pre-compiled regular expression statements. • Whenever possible, regular expressions should be pre-compiled. String matches/comparisons should be per- formed against the pre-compiled regex instance.

# Do this fqdnre= regex.compile(r'^[\w._-]+$', regex.U)

def checkValue(valu): if not fqdnre.match(valu): self._raiseBadValu(valu)

# NOT this def checkValue(valu): if not regex.match(r'^[\w._-]+$', valu, regex.U) self._raiseBadValu(valu)

• Return values should be preferred over raising exceptions. Functions/methods that return a value should return None (or a default value) in the case of an error. The logic behind this is that it is much easier, cleaner, faster to check a return value than to handle an exception.

3.1. Contributing to Synapse 83 synapse Documentation, Release 0.1.0

Raising exceptions is reserved for “exceptional circumstances” and should NEVER be used for normal program flow.

# Do this def getWidgetById(self, wid): widget_hash= self._index.get(wid) if widget_hash is None: return None

widget= self._widgets.get(widget_hash) return widget

# NOT this def getWidgetById(self, wid): widget_hash= self._index.get(wid) if widget_hash is None: raise NotFoundError

widget= self._widgets.get(widget_hash) if widget is None: raise NotFoundError

return widget

Contributions to Synapse which do not follow the project style guidelines may not be accepted.

3.1.2 Git Hook & Syntax Checking

A set of helper scripts are available for doing python syntax checking. These include a script to do generic syntax checking of all synapse files; a git pre-commit hook; and a script to run autopep8 on staged git files. The pre-commit hook does syntax checking on .py files which contain invalid syntax. The hook will ALSO run nbstripout on .ipynb files to remove output data from cells. This results in cleaner diffs for .ipynb files over time. 1. An example of running the generic syntax check script is seen below:

~/git/synapse$ ./scripts/syntax_check.py PEP8 style violations have been detected.

./synapse/tests/test_lib_types.py:397: [E226] missing whitespace around

˓→arithmetic operator ./synapse/tests/test_lib_types.py:398: [E226] missing whitespace around

˓→arithmetic operator

2. Installing the git hook is easy:

cp scripts/githooks/pre-commit.git/hooks/pre-commit chmod+x.git/hooks/pre-commit

3. After installing the hook, attempting a commit with a syntax error will fail

~/git/synapse$ git commit -m "Demo commit" PEP8 style violations have been detected. Please fix them or force the commit with "git commit --no-verify".

(continues on next page)

84 Chapter 3. Synapse Contributors Guide synapse Documentation, Release 0.1.0

(continued from previous page) ./synapse/tests/test_lib_types.py:397: [E226] missing whitespace around

˓→arithmetic operator ./synapse/tests/test_lib_types.py:398: [E226] missing whitespace around

˓→arithmetic operator

4. This may be automatically fixed for you using the pep8_staged_files.py script. Note that most, but not all syntax errors may be fixed with the helper script.

# Run the pep8_staged_files.py script ~/git/synapse$ ./scripts/pep8_staged_files.py # Check the diff ~/git/synapse$ git diff synapse/tests/test_lib_types.py diff --git a/synapse/tests/test_lib_types.py b/synapse/tests/test_lib_types.py index 0e3a7498..b81575ef 100644 --- a/synapse/tests/test_lib_types.py +++ b/synapse/tests/test_lib_types.py class TypesTest(s_t_utils.SynTest):

def test_type(self): @@ -397,8 +395,8 @@ class TypesTest(s_t_utils.SynTest): self.eq({node.ndef[1] for node in nodes}, {'m'}) nodes = await alist(core.eval('testcomp +testcomp*range=((1024, ˓→grinch), (4096, zemeanone))')) self.eq({node.ndef[1] for node in nodes}, {(2048, 'horton'), (4096,

˓→'whoville')}) - guid0 = 'B'*32 - guid1 = 'D'*32 + guid0 = 'B' * 32 + guid1 = 'D' * 32 nodes = await alist(core.eval(f'testguid +testguid*range=({guid0}, ˓→{guid1})')) self.eq({node.ndef[1] for node in nodes}, {'c' * 32}) nodes = await alist(core.eval('testint | noderefs | ˓→+testcomp*range=((1000, grinch), (4000, whoville))'))

# Add the file and commit ~/git/synapse$ git add synapse/tests/test_lib_types.py ~/git/synapse$ git commit -m "Demo commit" [some-branch f254f5bf] Demo commit 1 file changed, 3 insertions(+), 2 deletions(-)

3.1.3 Contribution Process

The Vertex Project welcomes contributions to the Synapse Hypergraph framework in order to continue its growth! In order to contribute to the project, do the following: 1. Fork the Synapse repository from the Vertex Project. Make a new branch in git with a descriptive name for your change. For example:

git checkout-b foohuman_new_widget

2. Make your changes. Changes should include the following information: • Clear documentation for new features or changed behavior • Unit tests for new features or changed behaviors

3.1. Contributing to Synapse 85 synapse Documentation, Release 0.1.0

• If possible, unit tests should also show minimal use examples of new features. 3. Ensure that both your tests and existing Synapse tests successfully run. You can do that manually via the python unittest module, or you can set up CircleCI to run tests for your fork (this is a exercise for the reader). The following examples shows manual test runs:

pytest-v pytest-v synapse/tests/your_test_file.py

If test coverage is desired, you can use the provided testrunner.sh shell script to run a test. This script will generate HTML coverage reports and attempt to open those reports using xdg-open. This requires the pytest, pytest-cov, pytest-xdist packages to be installed.

./scripts/testrunner.sh ./scripts/testrunner.sh synapse/tests/your_test_file.py ./scripts/testrunner.sh synapse/tests/your_test_file.py::YourTestClass ./scripts/testrunner.sh synapse/tests/your_test_file.py::YourTestClass::test_

˓→function

4. Rebase your feature branch on top of the latest master branch of the Vertex Project Synapse repository. This may require you to add the Vertex Project repository to your git remotes. The following example of rebasing can be followed:

# Add the Vertex project repository as a remote named "upstream". git remote add upstream https://github.com/vertexproject/synapse.git # Grab data from the upstream repository git fetch--all # Change to your local git master branch git checkout master # Merge changes from upstream/master to your local master git merge upstream/master # Move back to your feature branch git checkout foohuman_new_feature # Rebase your feature branch ontop of master. # This may require resolving merge conflicts. git rebase master # Push your branch up to to your fork - this may require a --force # flag if you had previously pushed the branch prior to the rebase. git push

5. Ensure your tests still pass with the rebased feature branch. 6. If your changes require extensive documentation, please very your API documentation builds properly and any additional user or devops docs are created as needed. See Synapse Doc Mastering for documentation mastering notes. 7. Create the Pull Request in Github, from your fork’s feature branch to the master branch of the Vertex Project Synapse repository. Include a description and a reference to any open issues related to the PR.

3.2 Synapse Doc Mastering

Documentation for creation and generation of documentation for Synapse.

86 Chapter 3. Synapse Contributors Guide synapse Documentation, Release 0.1.0

3.2.1 Generating Docs Locally

API documentation is automatically generated from docstrings, and additional docs may also be added to Synapse as well for more detailed discussions of Syanpse subsystems. This is currently done via readthedocs. In order to do local doc generation you can do the following steps: 1. Install the following packages (preferably in a virtualenv):

# cd to your synapse checkout cd synapse # Install additional packages - this assumes the environment already has # any additional packages required for executing synapse code in it. python-m pip install-U-r requirements_doc.txt # Alternativly, you can install synapse directly in develop mode with pip # python -m pip install .[docs]

# Install pandoc package, required for building HTML. # This may require sudo access depending on your environment. apt install pandoc

2. Build the docs using sphinx. A makefile is provided which makes this easy.

# Go to your synapse repo cd synapse # Go to the docs folder cd docs # Use the make command to build the HTML docs make html

3. Now you can open the HTML docs for browsing them.

xdg-open _build/html/index.html

4. To rebuild documentation from scratch you can delete the _build directory and the api directories. Deleting the api directory will cause the automatic Synapse API documentation to be rebuilt.

# Delete the _build directory make clean # Delete the API directory rm-rf api

3.2.2 Mastering Docs

Synapse documents are mastered using either raw ReStructuredText (.rst) files or as Jupyter Notebooks (.ipynb). Notebooks should be used for documenting anything which may include Storm or code examples, so that the examples can be written in a manner that can be asserted, so the documentation can be tested in the CI pipeline. Notebooks are also executed during sphinx document build steps, so any output is current as of document build time. Text in Notebooks should be mastered as RST using raw NbConvert cells. In general, docs for Synapse fall into two categories: User guides and devops guides. User guides should be mastered in ./docs/synapse/userguides and devops guides should be mastered in ./docs/synapse/devops. Additional top level sections may be added over time. In order to master Notebooks, you will need to setup the hide_code extension for Jupyter. That is used to selectively hide code and output blocks as needed. For example, this allows use to hide the code used to run a Storm command and show the output.

3.2. Synapse Doc Mastering 87 synapse Documentation, Release 0.1.0

The following steps are a high level overview of the process to setup Jupyter and add or edit notebooks for documen- tation purposes. • Setup the hide_code extension:

# Then install & enable the Jupyter hide-code extension # This only has to be run once. jupyter nbextension install--py--user hide_code jupyter nbextension enable--py--user hide_code jupyter serverextension enable--py--user hide_code

• Launch Jupyter to run a local notebook server:

# Go to your synapse repo cd synapse # Launch the notebook server jupyter notebook

• Navigate to the docs directory in Jupyter. Create a new notebook or open an existing notebook as needed. This will likely be located under the docs/synapse/userguides or docs/synapse/devops directories. • For Storm CLI integration, you can add the following code block into the first code cell in order to get some Synapse Jupyter helpers: import os, sys try: from synapse.lib.jupyter import * except ImportError as e: # Insert the root path of the repository to sys.path. # This assumes the notebook is located three directories away # From the root synapse directory. It may need to be varied synroot= os.path.abspath('../../../') sys.path.insert(0, synroot) from synapse.lib.jupyter import *

• You can use helpers to execute storm commands in the following fashion to get a CoreCmdr object, execute a storm query printing the CLI ouput to screen, while asserting the number of nodes returned, and then closing the object.

# Get a CoreCmdr object corecmdr= await getTempCoreCmdr() # Execute the query and get the packed nodes. podes= await corecmdr.eval('[inet:ipv4=1.2.3.4]', num=1, cmdr=True) cli> storm [inet:ipv4=1.2.3.4] inet:ipv4=1.2.3.4 .created = 2019/02/12 15:26:37.995 :asn = 0 :loc = ?? :type = unicast complete. 1 nodes in 20 ms (50/sec).

• We have a helper function available from the synapse.lib.jupyter imported earlier called getDocData(fn). It will look for a given filename in the docs/docdata directory; and get its data. If the file ends with .json, .jsonl, .yaml, or .mpk we will return the decoded data, otherwise we will

88 Chapter 3. Synapse Contributors Guide synapse Documentation, Release 0.1.0

return the raw bytes. This uses a function called getDocPath(fn) which will find and return a file under the docs\docdata directory. There is an example below showing the use of this to load a json file located at docs/docdata/ mastering_example_ingest.json, and adding the data to the Cortex via the addFeedData() func- tion. fn='mastering_example_ingest.json' data= getDocData(fn) await corecmdr.addFeedData('syn.ingest', data) podes= await corecmdr.eval('#example', num=2, cmdr=True) cli> storm #example inet:ipv4=0.0.0.1 .created = 2019/02/12 15:26:38.028 :asn = 0 :loc = ?? :type = private #example inet:fqdn=woot.com .created = 2019/02/12 15:26:38.030 :domain = com :host = woot :issuffix = False :iszone = True :zone = woot.com #example complete. 2 nodes in 3 ms (666/sec).

• Since the Code cells are persistent, you can reuse the objects from earlier cells until a resource has been closed (.fini()’d). The following example shows using the corecmdr object from the above code section to lift a node and print it to the screen. from pprint import pprint # We want to make our nodes pretty podes= await(corecmdr.eval('inet:ipv4')) for pode in podes: pprint(pode)

(('inet:ipv4',1), {'iden':'2f70f448adcc6e9b9846aecfd034efc4f9d583e614f1b3489d1cf1d32fb64667', 'path': {}, 'props':{'.created': 1549985198028, 'asn':0, 'loc':'??', 'type':'private'}, 'tags':{'example':(None, None)}}) (('inet:ipv4', 16909060), {'iden':'20153b758f9d5eaaa38e4f4a65c36da797c3e59e549620fa7c4895e1a920991f', 'path': {}, 'props':{'.created': 1549985197995, 'asn':0, 'loc':'??', 'type':'unicast'}, 'tags': {}})

• We can also execute a line of text in the CLI directly with the runCmdLine() function. For example, we can use this to execute the help command and see all available commands to the raw CLI object. This will always

3.2. Synapse Doc Mastering 89 synapse Documentation, Release 0.1.0

print the CLI output to the Jupyter cell output.

# Run the help command. text='help' await corecmdr.runCmdLine(text) cli> help at- Adds a non-recurring cron job. cron- Manages cron jobs in a cortex. help- List commands and display help output. hive- Manipulates values in a cell's Hive. kill- Kill a running task/query within the cortex. locs- List the current locals for a given CLI object. log- Add a storm log to the local command session. ps- List running tasks in the cortex. quit- Quit the current command line interpreter. storm- Execute a storm query. trigger- Manipulate triggers in a cortex.

• In the above example, there is some Python syntax highlighting occuring. This may not be desired. In order to disable that, add the following to the first line of the RST body of a document: .. highlight:: none This will disable all code highlighting in a given document, until another highlight directive is encountered. • The following code and output will have their highlighting disabled, via the use of a pair of highlight directives before and after the code cell. The first directive disabled highlighting, and the subsequent directive re-enabled it for python3 highlighting. Read the Sphinx Literal documentation for additional information about highlighting controls.

# Run the help command again. text = 'help' await corecmdr.runCmdLine(text) cli> help at - Adds a non-recurring cron job. cron - Manages cron jobs in a cortex. help - List commands and display help output. hive - Manipulates values in a cell's Hive. kill - Kill a running task/query within the cortex. locs - List the current locals for a given CLI object. log - Add a storm log to the local command session. ps - List running tasks in the cortex. quit - Quit the current command line interpreter. storm - Execute a storm query. trigger - Manipulate triggers in a cortex.

• When we are done with the CoreCmdr object, we should fini() is to remove any resources it may have created. This is done below.

# Close the object. _= await corecmdr.fini()

• You can enable the hide_code options by selecting the “View -> Cell Toolbar -> Hide code” option. This will allow you to optionally hide code or output blocks.

90 Chapter 3. Synapse Contributors Guide synapse Documentation, Release 0.1.0

• After adding text and code to a notebook, ensure that it runs properly and any produces the expected outputs. You can then mark any code cells for hiding as necessary; then save your notebook. You can then follow the earlier instructions for how to build and view the docs locally. • Once new documents are made, they will needto be added to the appropriate toctree directive. There are three index documents: – index.rst - This controls top-level documentation ordering. It generally should not need to be edited unless adding a new top level document or adding an additional section to the second level Synapse directory. – synapse/userguide.rst - This controls the TOC ordering for user guides. – synapse/devops.rst - The controls the TOC ordering for devops guides. • Add notebooks to the repository using git add ..path/to/notebook.ipynb. You can then com- mit the notebook using git commit. If you have the git pre-commit hook from scripts/githooks/ pre-commit, this will strip any output from the notebook upon commit time. This will result in cleaner git diff views over time. See Git Hook & Syntax Checking

3.2.3 Under the hood

Docs are built from Notebooks using a custom conf.py file which executes the notebooks, converting them to RST and using a custom template (vertex.tpl) which looks for flags set by the hide_code extension.

3.2.4 ReadTheDocs

Building documents on ReadTheDocs.org using cPython 3.7 is currently an unsupported operation. This is accom- plished using a readthedocs.yml file, which uses environment_docs.yml to configure an 3.7 Anaconda environment. This is the environment which is used do actually execute sphinx and the associated ipython notebooks to generate RST documents.

3.3 Synapse Release Process

This doc details the release process we use for Synapse.

3.3.1 Github Milestone Management

The current milestone and the next milestone should be created in github. For example, if the current release is v0.0.20, we should have a v0.0.21 and v0.0.22 milestones created. When PRs are created or issues are addressed (via PR), they should be added to the milestone. This allows us to easily pull stories and PRs for release note generation.

3.3.2 Release Notes

Release notes should be compiled from the issues and PRs assigned to the milestone being released. These can all be obtained via a issue search in github. For example, if we’re releasing v0.0.20, we can pull all the stories via the following query in github: milestone:v0.0.20

Release notes should break things out by the following categories: 1. New Features in Synapse

3.3. Synapse Release Process 91 synapse Documentation, Release 0.1.0

2. Enhancements to existing features 3. Bugfixes 4. Major documentation updates Short text form is fine for describing these. These notes will be posted up on github on the releases page for consump- tion.

Markdown Template

The following can be used as a markdown template for Github release notes:

# Synapse Release Notes

## New Features - item1 - item2

## Enhancements - item1 - item2

## Bugs - item1 - item2

## Documentation - item1 - item2

3.3.3 Cutting the Release

This includes four parts: 1. Preparing the release notes/changelog information. 2. Tagging the release and pushing to github. 3. Publishing the release on pypi. 4. Publishing new docker images on dockerhub.

Preparing The Release Notes

Release notes are to be prepared as per the release notes format noted above. The markdown template also needs to be added to the top of the CHANGELOG.md file. This allows us to keep the changes in repository as well. This file needs to be updated prior to the release tagging. The formatting for adding the content to the file is the following:

- YYYY-MM-DD ------

## New Features - item1 - item2 (continues on next page)

92 Chapter 3. Synapse Contributors Guide synapse Documentation, Release 0.1.0

(continued from previous page)

## Enhancements - item1 - item2

## Bugs - item1 - item2

## Documentation - item1 - item

This also allows for machine parseable notes so that pyup.io can show our changelogs.

Tagging the Release

Version tagging in Synapse is managed by bumpversion. This handles updating the .py files containing the version number in them, as well as creating git tags and commit messages. There should not be a need to manually edit version numbers or do git commits. bumpversion is a python application, and can be installed via pip:

python-m pip install bumpversion

Bumpversion is designed for projects which do semantic versioning. Since synapse is not yet in that state, we’ll be using bumpversion to do patch releases. This can be done via the following (assuming the vertexproject/synapse remote is called ‘upstream’):

# Ensure we're on master with the latest version git checkout master&& git fetch--all&& git merge upstream/master # Do a dry-run to ensure that we're updating things properly bumpversion--dry-run--verbose patch # Bump the patch version bumpversion--verbose patch # Push the new commit and tag up to github git push upstream git push upstream--tags

Next, go to github at https://github.com/vertexproject/synapse/tags and edit the release notes for the tag that was pushed up. Add the release notes compiled from the previous step.

Publishing on Pypi

Coming soon (this may be automated very soon)

Updating Docker images

Coming soon

3.3. Synapse Release Process 93 synapse Documentation, Release 0.1.0

94 Chapter 3. Synapse Contributors Guide CHAPTER 4

Synapse Data Model - Types

4.1 Base Types

Base types are defined via Python classes.

4.1.1 bool

The base boolean type. It is implemented by the following class: synapse.lib.types.Bool.

4.1.2 comp

The base type for compound node fields. It is implemented by the following class: synapse.lib.types.Comp.

4.1.3 data

Arbitrary msgpack compatible data stored without an index. It is implemented by the following class: synapse. lib.types.Data.

4.1.4 edge

An digraph edge base type. It is implemented by the following class: synapse.lib.types.Edge.

4.1.5 file:base

A file name with no path. It is implemented by the following class: synapse.models.files.FileBase. A example of file:base: • woot.exe

95 synapse Documentation, Release 0.1.0

4.1.6 file:bytes

The file bytes type with SHA256 based primary property. It is implemented by the following class: synapse. models.files.FileBytes.

4.1.7 file:path

A normalized file path. It is implemented by the following class: synapse.models.files.FilePath. A example of file:path: • c:/windows/system32/calc.exe

4.1.8 geo:dist

A geographic distance (base unit is mm). It is implemented by the following class: synapse.models.geospace. Dist. A example of geo:dist: • 10 km

4.1.9 geo:latitude

BaseType has no doc string. It is implemented by the following class: synapse.models.geospace.Latitude.

4.1.10 geo:latlong

A Lat/Long string specifying a point on Earth. It is implemented by the following class: synapse.models. geospace.LatLong. A example of geo:latlong: • -12.45,56.78

4.1.11 geo:longitude

BaseType has no doc string. It is implemented by the following class: synapse.models.geospace. Longitude.

4.1.12 guid

The base GUID type. It is implemented by the following class: synapse.lib.types.Guid.

4.1.13 hex

The base hex type. It is implemented by the following class: synapse.lib.types.Hex. The base type hex has the following default options set: • size: 0

96 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.1.14 inet:addr

A network layer URL-like format to represent tcp/udp/icmp clients and servers. It is implemented by the following class: synapse.models.inet.Addr. A example of inet:addr: • tcp://1.2.3.4:80

4.1.15 inet:cidr4

An IPv4 address block in Classless Inter-Domain Routing (CIDR) notation. It is implemented by the following class: synapse.models.inet.Cidr4. A example of inet:cidr4: • 1.2.3.0/24

4.1.16 inet:dns:name

A DNS query name string. Likely an FQDN but not always. It is implemented by the following class: synapse. models.dns.DnsName. A example of inet:dns:name: • XXX

4.1.17 inet:email

An e-mail address. It is implemented by the following class: synapse.models.inet.Email.

4.1.18 inet:fqdn

A Fully Qualified Domain Name (FQDN). It is implemented by the following class: synapse.models.inet. Fqdn. A example of inet:fqdn: • vertex.link

4.1.19 inet:ipv4

An IPv4 address. It is implemented by the following class: synapse.models.inet.IPv4. A example of inet:ipv4: • 1.2.3.4

4.1.20 inet:ipv6

An IPv6 address. It is implemented by the following class: synapse.models.inet.IPv6. A example of inet:ipv6: • 2607:f8b0:4004:809::200e

4.1. Base Types 97 synapse Documentation, Release 0.1.0

4.1.21 inet:ipv6range

An IPv6 address range. It is implemented by the following class: synapse.models.inet.IPv6Range. A example of inet:ipv6range: • (2607:f8b0:4004:809::200e, 2607:f8b0:4004:809::2011)

4.1.22 inet:rfc2822:addr

An RFC 2822 Address field. It is implemented by the following class: synapse.models.inet.Rfc2822Addr. A example of inet:rfc2822:addr: • "Visi Kenshoto"

4.1.23 inet:url

A Universal Resource Locator (URL). It is implemented by the following class: synapse.models.inet.Url. A example of inet:url: • http://www.woot.com/files/index.html

4.1.24 int

The base 64 bit signed integer type. It is implemented by the following class: synapse.lib.types.Int. The base type int has the following default options set: • size: 8 • signed: True • fmt: %d • min: None • max: None • ismin: False • ismax: False

4.1.25 it:semver

Semantic Version type. It is implemented by the following class: synapse.models.infotech.SemVer.

4.1.26 ival

A time window/interval. It is implemented by the following class: synapse.lib.types.Ival.

4.1.27 loc

The base geo political location type. It is implemented by the following class: synapse.lib.types.Loc.

98 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.1.28 ndef

The node definition type for a (form,valu) compound field. It is implemented by the following class: synapse. lib.types.Ndef.

4.1.29 nodeprop

The nodeprop type for a (prop,valu) compound field. It is implemented by the following class: synapse.lib. types.NodeProp.

4.1.30 range

A base range type. It is implemented by the following class: synapse.lib.types.Range. The base type range has the following default options set: • type: ('int', {})

4.1.31 str

The base string type. It is implemented by the following class: synapse.lib.types.Str. The base type str has the following default options set: • enums: None • regex: None • lower: False • strip: False • onespace: False

4.1.32 syn:tag

The base type for a synapse tag. It is implemented by the following class: synapse.lib.types.Tag.

4.1.33 tel:mob:imei

An International Mobile Equipment Id. It is implemented by the following class: synapse.models.telco. Imei. A example of tel:mob:imei: • 490154203237518

4.1.34 tel:mob:imsi

An International Mobile Subscriber Id. It is implemented by the following class: synapse.models.telco. Imsi. A example of tel:mob:imsi: • 310150123456789

4.1. Base Types 99 synapse Documentation, Release 0.1.0

4.1.35 tel:phone

A phone number. It is implemented by the following class: synapse.models.telco.Phone. A example of tel:phone: • +15558675309

4.1.36 time

A date/time value. It is implemented by the following class: synapse.lib.types.Time. The base type time has the following default options set: • ismin: False • ismax: False

4.1.37 timeedge

An digraph edge base type with a unique time. It is implemented by the following class: synapse.lib.types. TimeEdge.

4.2 Types

Regular types are derived from BaseTypes.

4.2.1 cluster

A generic node, used in conjunction with Edge types, to cluster arbitrary nodes to a single node in the model. The cluster type is derived from the base type: guid.

4.2.2 event

A generic event node to represent events outside the model. The event type is derived from the base type: guid.

4.2.3 file:filepath

The fused knowledge of the association of a file:bytes node and a file:path. The file:filepath type is derived from the base type: comp. The type file:filepath has the following options set: • fields: (('file', 'file:bytes'), ('path', 'file:path'))

100 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.2.4 file:ismime

Records one, of potentially multiple, mime types for a given file. The file:ismime type is derived from the base type: comp. The type file:ismime has the following options set: • fields: (('file', 'file:bytes'), ('mime', 'file:mime'))

4.2.5 file:mime

A file mime name string. The file:mime type is derived from the base type: str. A example of file:mime: • text/plain The type file:mime has the following options set: • lower: 1

4.2.6 file:mime:pe:export

The fused knowledge of a file:bytes node containing a pe named export. The file:mime:pe:export type is derived from the base type: comp. The type file:mime:pe:export has the following options set: • fields: (('file', 'file:bytes'), ('name', 'str'))

4.2.7 file:mime:pe:resource

The fused knowledge of a file:bytes node containing a pe resource. The file:mime:pe:resource type is derived from the base type: comp. The type file:mime:pe:resource has the following options set: • fields: (('file', 'file:bytes'), ('type', 'pe:resource:type'), ('langid', 'pe:langid'), ('resource', 'file:bytes'))

4.2.8 file:mime:pe:section

The fused knowledge a file:bytes node containing a pe section. The file:mime:pe:section type is derived from the base type: comp. The type file:mime:pe:section has the following options set: • fields: (('file', 'file:bytes'), ('name', 'str'), ('sha256', 'hash:sha256'))

4.2.9 file:mime:pe:vsvers:info knowledge of a file:bytes node containing vsvers info. The file:mime:pe:vsvers:info type is derived from the base type: comp. The type file:mime:pe:vsvers:info has the following options set: • fields: (('file', 'file:bytes'), ('keyval', 'file:mime:pe:vsvers:keyval'))

4.2. Types 101 synapse Documentation, Release 0.1.0

4.2.10 file:mime:pe:vsvers:keyval

A key value pair found in a PE vsversion info structure. The file:mime:pe:vsvers:keyval type is derived from the base type: comp. The type file:mime:pe:vsvers:keyval has the following options set: • fields: (('name', 'str'), ('value', 'str'))

4.2.11 file:ref

A file that contains reference to the specified node. The file:ref type is derived from the base type: comp. The type file:ref has the following options set: • fields: (('file', 'file:bytes'), ('node', 'ndef'))

4.2.12 file:string

The fused knowledge of a file:bytes node containing a string. The file:string type is derived from the base type: comp. The type file:string has the following options set: • fields: (('file', 'file:bytes'), ('string', 'str'))

4.2.13 file:subfile

A parent file that fully contains the specified child file. The file:subfile type is derived from the base type: comp. The type file:subfile has the following options set: • fields: (('parent', 'file:bytes'), ('child', 'file:bytes'))

4.2.14 geo:nloc

Records a node latitude/longitude in space-time. The geo:nloc type is derived from the base type: comp. The type geo:nloc has the following options set: • fields: (('ndef', 'ndef'), ('latlong', 'geo:latlong'), ('time', 'time'))

4.2.15 geo:place

A GUID for a geographic place. The geo:place type is derived from the base type: guid.

4.2.16 gov:cn:icp

A Chinese Content Provider ID. The gov:cn:icp type is derived from the base type: int.

102 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.2.17 gov:cn:mucd

A Chinese PLA MUCD. The gov:cn:mucd type is derived from the base type: int.

4.2.18 gov:intl:un:m49

UN M49 Numeric Country Code. The gov:intl:un:m49 type is derived from the base type: int. The type gov:intl:un:m49 has the following options set: • max: 999 • min: 1

4.2.19 gov:us:cage

A Commercial and Government Entity (CAGE) code. The gov:us:cage type is derived from the base type: str. The type gov:us:cage has the following options set: • lower: True

4.2.20 gov:us:ssn

A US Social Security Number (SSN). The gov:us:ssn type is derived from the base type: int.

4.2.21 gov:us:zip

A US Zip Code. The gov:us:zip type is derived from the base type: int.

4.2.22 graph:link

A generic digraph edge to show relationships outside the model. The graph:link type is derived from the base type: edge.

4.2.23 graph:node

A generic node used to represent objects outside the model. The graph:node type is derived from the base type: guid.

4.2.24 graph:timelink

A generic digraph time edge to show relationships outside the model. The graph:timelink type is derived from the base type: timeedge.

4.2.25 has

A digraph edge which records that N1 has N2. The has type is derived from the base type: edge.

4.2. Types 103 synapse Documentation, Release 0.1.0

4.2.26 hash:lm

A hex encoded Microsoft Windows LM password hash. The hash:lm type is derived from the base type: hex. A example of hash:lm: • d41d8cd98f00b204e9800998ecf8427e The type hash:lm has the following options set: • size: 32

4.2.27 hash:md5

A hex encodeded MD5 hash. The hash:md5 type is derived from the base type: hex. A example of hash:md5: • d41d8cd98f00b204e9800998ecf8427e The type hash:md5 has the following options set: • size: 32

4.2.28 hash:ntlm

A hex encoded Microsoft Windows NTLM password hash. The hash:ntlm type is derived from the base type: hex. A example of hash:ntlm: • d41d8cd98f00b204e9800998ecf8427e The type hash:ntlm has the following options set: • size: 32

4.2.29 hash:sha1

A hex encoded SHA1 hash. The hash:sha1 type is derived from the base type: hex. A example of hash:sha1: • da39a3ee5e6b4b0d3255bfef95601890afd80709 The type hash:sha1 has the following options set: • size: 40

4.2.30 hash:sha256

A hex encoded SHA256 hash. The hash:sha256 type is derived from the base type: hex. A example of hash:sha256: • ad9f4fe922b61e674a09530831759843b1880381de686a43460a76864ca0340c The type hash:sha256 has the following options set: • size: 64

104 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.2.31 hash:sha384

A hex encoded SHA384 hash. The hash:sha384 type is derived from the base type: hex. A example of hash:sha384: • d425f1394e418ce01ed1579069a8bfaa1da8f32cf823982113ccbef531fa36bda9987f389c5af05b5e28035242efab6c The type hash:sha384 has the following options set: • size: 96

4.2.32 hash:sha512

A hex encoded SHA512 hash. The hash:sha512 type is derived from the base type: hex. A example of hash:sha512: • ca74fe2ff2d03b29339ad7d08ba21d192077fece1715291c7b43c20c9136cd132788239189f3441a87eb23ce2660aa243f334295902c904b5520f6e80ab91f11 The type hash:sha512 has the following options set: • size: 128

4.2.33 inet:asn

An Autonomous System Number (ASN). The inet:asn type is derived from the base type: int.

4.2.34 inet:asnet4

An Autonomous System Number (ASN) and its associated IPv4 address range. The inet:asnet4 type is derived from the base type: comp. A example of inet:asnet4: • (54959, (1.2.3.4, 1.2.3.20)) The type inet:asnet4 has the following options set: • fields: (('asn', 'inet:asn'), ('net4', 'inet:net4'))

4.2.35 inet:banner

A network protocol banner string presented by a server. The inet:banner type is derived from the base type: comp. The type inet:banner has the following options set: • fields: (('server', 'inet:server'), ('text', 'it:dev:str'))

4.2.36 inet:client

A network client address. The inet:client type is derived from the base type: inet:addr.

4.2. Types 105 synapse Documentation, Release 0.1.0

4.2.37 inet:dns:a

The result of a DNS A record lookup. The inet:dns:a type is derived from the base type: comp. A example of inet:dns:a: • (vertex.link,1.2.3.4) The type inet:dns:a has the following options set: • fields: (('fqdn', 'inet:fqdn'), ('ipv4', 'inet:ipv4'))

4.2.38 inet:dns:aaaa

The result of a DNS AAAA record lookup. The inet:dns:aaaa type is derived from the base type: comp. A example of inet:dns:aaaa: • (vertex.link,2607:f8b0:4004:809::200e) The type inet:dns:aaaa has the following options set: • fields: (('fqdn', 'inet:fqdn'), ('ipv6', 'inet:ipv6'))

4.2.39 inet:dns:answer

A single answer from within a DNS reply. The inet:dns:answer type is derived from the base type: guid.

4.2.40 inet:dns:cname

The result of a DNS CNAME record lookup. The inet:dns:cname type is derived from the base type: comp. A example of inet:dns:cname: • (foo.vertex.link,vertex.link) The type inet:dns:cname has the following options set: • fields: (('fqdn', 'inet:fqdn'), ('cname', 'inet:fqdn'))

4.2.41 inet:dns:mx

The result of a DNS MX record lookup. The inet:dns:mx type is derived from the base type: comp. A example of inet:dns:mx: • (vertex.link,mail.vertex.link) The type inet:dns:mx has the following options set: • fields: (('fqdn', 'inet:fqdn'), ('mx', 'inet:fqdn'))

106 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.2.42 inet:dns:ns

The result of a DNS NS record lookup. The inet:dns:ns type is derived from the base type: comp. A example of inet:dns:ns: • (vertex.link,ns.dnshost.com) The type inet:dns:ns has the following options set: • fields: (('zone', 'inet:fqdn'), ('ns', 'inet:fqdn'))

4.2.43 inet:dns:query

A DNS query unique to a given client. The inet:dns:query type is derived from the base type: comp. A example of inet:dns:query: • (1.2.3.4, woot.com, 1) The type inet:dns:query has the following options set: • fields: (('client', 'inet:client'), ('name', 'inet:dns:name'), ('type', 'int'))

4.2.44 inet:dns:request

A single instance of a DNS resolver request and optional reply info. The inet:dns:request type is derived from the base type: guid.

4.2.45 inet:dns:rev

The transformed result of a DNS PTR record lookup. The inet:dns:rev type is derived from the base type: comp. A example of inet:dns:rev: • (1.2.3.4,vertex.link) The type inet:dns:rev has the following options set: • fields: (('ipv4', 'inet:ipv4'), ('fqdn', 'inet:fqdn'))

4.2.46 inet:dns:rev6

The transformed result of a DNS PTR record for an IPv6 address. The inet:dns:rev6 type is derived from the base type: comp. A example of inet:dns:rev6: • (2607:f8b0:4004:809::200e,vertex.link) The type inet:dns:rev6 has the following options set: • fields: (('ipv6', 'inet:ipv6'), ('fqdn', 'inet:fqdn'))

4.2.47 inet:dns:soa

The result of a DNS SOA record lookup. The inet:dns:soa type is derived from the base type: guid.

4.2. Types 107 synapse Documentation, Release 0.1.0

4.2.48 inet:dns:txt

The result of a DNS MX record lookup. The inet:dns:txt type is derived from the base type: comp. A example of inet:dns:txt: • (hehe.vertex.link,"fancy TXT record") The type inet:dns:txt has the following options set: • fields: (('fqdn', 'inet:fqdn'), ('txt', 'str'))

4.2.49 inet:dns:type

A DNS query/answer type integer. The inet:dns:type type is derived from the base type: int.

4.2.50 inet:dns:wild:a

A DNS A wild card record and the IPv4 it resolves to. The inet:dns:wild:a type is derived from the base type: comp. The type inet:dns:wild:a has the following options set: • fields: (('fqdn', 'inet:fqdn'), ('ipv4', 'inet:ipv4'))

4.2.51 inet:dns:wild:aaaa

A DNS AAAA wild card record and the IPv6 it resolves to. The inet:dns:wild:aaaa type is derived from the base type: comp. The type inet:dns:wild:aaaa has the following options set: • fields: (('fqdn', 'inet:fqdn'), ('ipv6', 'inet:ipv6'))

4.2.52 inet:download

An instance of a file downloaded from a server. The inet:download type is derived from the base type: guid.

4.2.53 inet:email:header

A unique email message header. The inet:email:header type is derived from the base type: comp. The type inet:email:header has the following options set: • fields: (('name', 'inet:email:header:name'), ('value', 'str'))

4.2.54 inet:email:header:name

An email header name. The inet:email:header:name type is derived from the base type: str. A example of inet:email:header:name: • subject The type inet:email:header:name has the following options set:

108 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

• lower: True

4.2.55 inet:email:message

A unique email message. The inet:email:message type is derived from the base type: guid.

4.2.56 inet:email:message:attachment

A file which was attached to an email message. The inet:email:message:attachment type is derived from the base type: comp. The type inet:email:message:attachment has the following options set: • fields: (('message', 'inet:email:message'), ('file', 'file:bytes'))

4.2.57 inet:email:message:link

A url/link embedded in an email message. The inet:email:message:link type is derived from the base type: comp. The type inet:email:message:link has the following options set: • fields: (('message', 'inet:email:message'), ('url', 'inet:url'))

4.2.58 inet:flow

An individual network connection between a given source and destination. The inet:flow type is derived from the base type: guid.

4.2.59 inet:group

A group name string. The inet:group type is derived from the base type: str.

4.2.60 inet:http:cookie

An HTTP cookie string. The inet:http:cookie type is derived from the base type: str.

4.2.61 inet:http:header

An HTTP protocol header key/value. The inet:http:header type is derived from the base type: comp. The type inet:http:header has the following options set: • fields: (('name', 'inet:http:header:name'), ('value', 'str'))

4.2. Types 109 synapse Documentation, Release 0.1.0

4.2.62 inet:http:header:name

The base string type. The inet:http:header:name type is derived from the base type: str. The type inet:http:header:name has the following options set: • lower: True

4.2.63 inet:http:param

An HTTP request path query parameter. The inet:http:param type is derived from the base type: comp. The type inet:http:param has the following options set: • fields: (('name', 'str'), ('value', 'str'))

4.2.64 inet:http:request

A single HTTP request. The inet:http:request type is derived from the base type: guid.

4.2.65 inet:http:request:header

An HTTP request header. The inet:http:request:header type is derived from the base type: inet:http:header.

4.2.66 inet:http:response:header

An HTTP response header. The inet:http:response:header type is derived from the base type: inet:http:header.

4.2.67 inet:iface

A network interface with a set of associated protocol addresses. The inet:iface type is derived from the base type: guid.

4.2.68 inet:mac

A 48-bit Media Access Control (MAC) address. The inet:mac type is derived from the base type: str. A example of inet:mac: • aa:bb:cc:dd:ee:ff The type inet:mac has the following options set: • lower: True • regex: ^([0-9a-f]{2}[:]){5}([0-9a-f]{2})$

110 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.2.69 inet:net4

An IPv4 address range. The inet:net4 type is derived from the base type: range. A example of inet:net4: • (1.2.3.4, 1.2.3.20) The type inet:net4 has the following options set: • type: ('inet:ipv4', {})

4.2.70 inet:net6

An IPv6 address range. The inet:net6 type is derived from the base type: inet:ipv6range. A example of inet:net6: • ('ff::00', 'ff::30')

4.2.71 inet:passwd

A password string. The inet:passwd type is derived from the base type: str.

4.2.72 inet:port

A network port. The inet:port type is derived from the base type: int. A example of inet:port: • 80 The type inet:port has the following options set: • max: 65535 • min: 0

4.2.73 inet:search:query

An instance of a search query issued to a search engine. The inet:search:query type is derived from the base type: guid.

4.2.74 inet:search:result

A single result from a web search. The inet:search:result type is derived from the base type: guid.

4.2.75 inet:server

A network server address. The inet:server type is derived from the base type: inet:addr.

4.2. Types 111 synapse Documentation, Release 0.1.0

4.2.76 inet:servfile

A file hosted on a server for access over a network protocol. The inet:servfile type is derived from the base type: comp. The type inet:servfile has the following options set: • fields: (('server', 'inet:server'), ('file', 'file:bytes'))

4.2.77 inet:ssl:cert

An SSL certificate file served by a server. The inet:ssl:cert type is derived from the base type: comp. A example of inet:ssl:cert: • (1.2.3.4:443, guid:ff....fff) The type inet:ssl:cert has the following options set: • fields: (('server', 'inet:server'), ('file', 'file:bytes'))

4.2.78 inet:urlfile

A file hosted at a specific Universal Resource Locator (URL). The inet:urlfile type is derived from the base type: comp. The type inet:urlfile has the following options set: • fields: (('url', 'inet:url'), ('file', 'file:bytes'))

4.2.79 inet:urlredir

A URL that redirects to another URL, such as via a URL shortening service or an HTTP 302 response. The inet:urlredir type is derived from the base type: comp. A example of inet:urlredir: • (http://foo.com/,http://bar.com/) The type inet:urlredir has the following options set: • fields: (('src', 'inet:url'), ('dst', 'inet:url'))

4.2.80 inet:user

A username string. The inet:user type is derived from the base type: str. The type inet:user has the following options set: • lower: True

4.2.81 inet:web:acct

An account with a given Internet-based site or service. The inet:web:acct type is derived from the base type: comp. A example of inet:web:acct:

112 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

• twitter.com/invisig0th The type inet:web:acct has the following options set: • fields: (('site', 'inet:fqdn'), ('user', 'inet:user'))

4.2.82 inet:web:action

An instance of an account performing an action at an Internet-based site or service. The inet:web:action type is derived from the base type: guid.

4.2.83 inet:web:actref

A web action that references a given node. The inet:web:actref type is derived from the base type: comp. The type inet:web:actref has the following options set: • fields: (('act', 'inet:web:action'), ('node', 'ndef'))

4.2.84 inet:web:chprofile

A change to a web account. Used to capture historical properties associated with an account, as opposed to current data in the inet:web:acct node. The inet:web:chprofile type is derived from the base type: guid.

4.2.85 inet:web:file

A file posted by a web account. The inet:web:file type is derived from the base type: comp. The type inet:web:file has the following options set: • fields: (('acct', 'inet:web:acct'), ('file', 'file:bytes'))

4.2.86 inet:web:follows

A web account follows or is connected to another web account. The inet:web:follows type is derived from the base type: comp. The type inet:web:follows has the following options set: • fields: (('follower', 'inet:web:acct'), ('followee', 'inet:web:acct'))

4.2.87 inet:web:group

A group hosted within or registered with a given Internet-based site or service. The inet:web:group type is derived from the base type: comp. A example of inet:web:group: • somesite.com/mycoolgroup The type inet:web:group has the following options set: • fields: (('site', 'inet:fqdn'), ('id', 'inet:group'))

4.2. Types 113 synapse Documentation, Release 0.1.0

4.2.88 inet:web:logon

An instance of an account authenticating to an Internet-based site or service. The inet:web:logon type is derived from the base type: guid.

4.2.89 inet:web:memb

A web account that is a member of a web group. The inet:web:memb type is derived from the base type: comp. The type inet:web:memb has the following options set: • fields: (('acct', 'inet:web:acct'), ('group', 'inet:web:group'))

4.2.90 inet:web:mesg

A message sent from one web account to another web account. The inet:web:mesg type is derived from the base type: comp. A example of inet:web:mesg: • twitter.com/invisig0th|twitter.com/gobbles|20041012130220 The type inet:web:mesg has the following options set: • fields: (('from', 'inet:web:acct'), ('to', 'inet:web:acct'), ('time', 'time'))

4.2.91 inet:web:post

A post made by a web account. The inet:web:post type is derived from the base type: guid.

4.2.92 inet:web:postref

A web post that references a given node. The inet:web:postref type is derived from the base type: comp. The type inet:web:postref has the following options set: • fields: (('post', 'inet:web:post'), ('node', 'ndef'))

4.2.93 inet:whois:contact

An individual contact from a domain whois record. The inet:whois:contact type is derived from the base type: comp. The type inet:whois:contact has the following options set: • fields: (('rec', 'inet:whois:rec'), ('type', ('str', {'lower': True})))

4.2.94 inet:whois:email

An email address associated with an FQDN via whois registration text. The inet:whois:email type is derived from the base type: comp. The type inet:whois:email has the following options set:

114 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

• fields: (('fqdn', 'inet:fqdn'), ('email', 'inet:email'))

4.2.95 inet:whois:rar

A domain registrar. The inet:whois:rar type is derived from the base type: str. A example of inet:whois:rar: • godaddy, inc. The type inet:whois:rar has the following options set: • lower: True

4.2.96 inet:whois:rec

A domain whois record. The inet:whois:rec type is derived from the base type: comp. The type inet:whois:rec has the following options set: • fields: (('fqdn', 'inet:fqdn'), ('asof', 'time'))

4.2.97 inet:whois:recns

A nameserver associated with a domain whois record. The inet:whois:recns type is derived from the base type: comp. The type inet:whois:recns has the following options set: • fields: (('ns', 'inet:fqdn'), ('rec', 'inet:whois:rec'))

4.2.98 inet:whois:reg

A domain registrant. The inet:whois:reg type is derived from the base type: str. A example of inet:whois:reg: • woot hostmaster The type inet:whois:reg has the following options set: • lower: True

4.2.99 inet:whois:regmail

An association between a domain and a registrant email address. The inet:whois:regmail type is derived from the base type: comp. The type inet:whois:regmail has the following options set: • fields: (('fqdn', 'inet:fqdn'), ('email', 'inet:email'))

4.2. Types 115 synapse Documentation, Release 0.1.0

4.2.100 inet:wifi:ap

An SSID/MAC address combination for a access point. The inet:wifi:ap type is derived from the base type: comp. The type inet:wifi:ap has the following options set: • fields: (('ssid', 'inet:wifi:ssid'), ('bssid', 'inet:mac'))

4.2.101 inet:wifi:ssid

A WiFi service set identifier (SSID) name. The inet:wifi:ssid type is derived from the base type: str. A example of inet:wifi:ssid: • The Vertex Project

4.2.102 it:app:snort:hit

An instance of a snort rule hit. The it:app:snort:hit type is derived from the base type: guid.

4.2.103 it:app:snort:rule

A snort rule unique identifier. The it:app:snort:rule type is derived from the base type: guid.

4.2.104 it:app:yara:match

A yara rule match to a file. The it:app:yara:match type is derived from the base type: comp. The type it:app:yara:match has the following options set: • fields: (('rule', 'it:app:yara:rule'), ('file', 'file:bytes'))

4.2.105 it:app:yara:rule

A yara rule unique identifier. The it:app:yara:rule type is derived from the base type: guid.

4.2.106 it:auth:passwdhash

An instance of a password hash. The it:auth:passwdhash type is derived from the base type: guid.

4.2.107 it:av:filehit

A file that triggered an alert on a specific antivirus signature. The it:av:filehit type is derived from the base type: comp. The type it:av:filehit has the following options set: • fields: (('file', 'file:bytes'), ('sig', 'it:av:sig'))

116 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.2.108 it:av:sig

A signature name within the namespace of an antivirus engine name. The it:av:sig type is derived from the base type: comp. The type it:av:sig has the following options set: • fields: (('soft', 'it:prod:soft'), ('name', ('str', {'lower': True})))

4.2.109 it:dev:int

A developer selected integer constant. The it:dev:int type is derived from the base type: int.

4.2.110 it:dev:mutex

A string representing a mutex. The it:dev:mutex type is derived from the base type: str.

4.2.111 it:dev:pipe

A string representing a named pipe. The it:dev:pipe type is derived from the base type: str.

4.2.112 it:dev:regkey

A Windows registry key. The it:dev:regkey type is derived from the base type: str. A example of it:dev:regkey: • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run

4.2.113 it:dev:regval

A Windows registry key/value pair. The it:dev:regval type is derived from the base type: guid.

4.2.114 it:dev:str

A developer-selected string. The it:dev:str type is derived from the base type: str.

4.2.115 it:exec:bind

An instance of a host binding a listening port. The it:exec:bind type is derived from the base type: guid.

4.2.116 it:exec:file:add

An instance of a host adding a file to a filesystem. The it:exec:file:add type is derived from the base type: guid.

4.2. Types 117 synapse Documentation, Release 0.1.0

4.2.117 it:exec:file:del

An instance of a host deleting a file from a filesystem. The it:exec:file:del type is derived from the base type: guid.

4.2.118 it:exec:file:read

An instance of a host reading a file from a filesystem. The it:exec:file:read type is derived from the base type: guid.

4.2.119 it:exec:file:write

An instance of a host writing a file to a filesystem. The it:exec:file:write type is derived from the base type: guid.

4.2.120 it:exec:mutex

A mutex created by a process at runtime. The it:exec:mutex type is derived from the base type: guid.

4.2.121 it:exec:pipe

A named pipe created by a process at runtime. The it:exec:pipe type is derived from the base type: guid.

4.2.122 it:exec:proc

A process executing on a host. May be an actual (e.g., endpoint) or virtual (e.g., malware sandbox) host. The it:exec:proc type is derived from the base type: guid.

4.2.123 it:exec:reg:del

An instance of a host deleting a registry key. The it:exec:reg:del type is derived from the base type: guid.

4.2.124 it:exec:reg:get

An instance of a host getting a registry key. The it:exec:reg:get type is derived from the base type: guid.

4.2.125 it:exec:reg:set

An instance of a host creating or setting a registry key. The it:exec:reg:set type is derived from the base type: guid.

4.2.126 it:exec:url

A instance of a host requesting a URL. The it:exec:url type is derived from the base type: guid.

118 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.2.127 it:fs:file

A file on a host. The it:fs:file type is derived from the base type: guid.

4.2.128 it:host

A GUID that represents a host or system. The it:host type is derived from the base type: guid.

4.2.129 it:hostname

The name of a host or sytsem. The it:hostname type is derived from the base type: str. The type it:hostname has the following options set: • lower: True • strip: True

4.2.130 it:hostsoft

A version of a software product which is present on a given host. The it:hostsoft type is derived from the base type: comp. The type it:hostsoft has the following options set: • fields: (('host', 'it:host'), ('softver', 'it:prod:softver'))

4.2.131 it:hosturl

A url hosted on or served by a host or system. The it:hosturl type is derived from the base type: comp. The type it:hosturl has the following options set: • fields: (('host', 'it:host'), ('url', 'inet:url'))

4.2.132 it:os:android:aaid

An android advertising identification string. The it:os:android:aaid type is derived from the base type: str. The type it:os:android:aaid has the following options set: • lower: 1

4.2.133 it:os:android:ibroadcast

The given software broadcasts the given Android intent. The it:os:android:ibroadcast type is derived from the base type: comp. The type it:os:android:ibroadcast has the following options set: • fields: (('app', 'it:prod:soft'), ('intent', 'it:os:android:intent'))

4.2. Types 119 synapse Documentation, Release 0.1.0

4.2.134 it:os:android:ilisten

The given software listens for an android intent. The it:os:android:ilisten type is derived from the base type: comp. The type it:os:android:ilisten has the following options set: • fields: (('app', 'it:prod:soft'), ('intent', 'it:os:android:intent'))

4.2.135 it:os:android:intent

An android intent string. The it:os:android:intent type is derived from the base type: str.

4.2.136 it:os:android:perm

An android permission string. The it:os:android:perm type is derived from the base type: str.

4.2.137 it:os:android:reqperm

The given software requests the android permission. The it:os:android:reqperm type is derived from the base type: comp. The type it:os:android:reqperm has the following options set: • fields: (('app', 'it:prod:soft'), ('perm', 'it:os:android:perm'))

4.2.138 it:os:ios:idfa

An iOS advertising identification string. The it:os:ios:idfa type is derived from the base type: str. The type it:os:ios:idfa has the following options set: • lower: 1

4.2.139 it:prod:soft

A arbitrary, unversioned software product. The it:prod:soft type is derived from the base type: guid.

4.2.140 it:prod:softfile

A file is distributed by a specific software version. The it:prod:softfile type is derived from the base type: comp. The type it:prod:softfile has the following options set: • fields: (('soft', 'it:prod:softver'), ('file', 'file:bytes'))

120 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.2.141 it:prod:softlib

A software version contains a library software version. The it:prod:softlib type is derived from the base type: comp. The type it:prod:softlib has the following options set: • fields: (('soft', 'it:prod:softver'), ('lib', 'it:prod:softver'))

4.2.142 it:prod:softos

The software version is known to be compatible with the given os software version. The it:prod:softos type is derived from the base type: comp. The type it:prod:softos has the following options set: • fields: (('soft', 'it:prod:softver'), ('os', 'it:prod:softver'))

4.2.143 it:prod:softver

A specific version of a software product. The it:prod:softver type is derived from the base type: guid.

4.2.144 it:sec:cve

A vulnerability as designated by a Common Vulnerabilities and Exposures (CVE) number. The it:sec:cve type is derived from the base type: str. A example of it:sec:cve: • cve-2012-0158 The type it:sec:cve has the following options set: • lower: True • regex: (?i)^CVE-[0-9]{4}-[0-9]{4,}$

4.2.145 lang:idiom

A subcultural idiom. The lang:idiom type is derived from the base type: str.

4.2.146 lang:trans

Raw text with a documented translation. The lang:trans type is derived from the base type: str.

4.2.147 mat:item

A GUID assigned to a material object. The mat:item type is derived from the base type: guid.

4.2. Types 121 synapse Documentation, Release 0.1.0

4.2.148 mat:itemimage

The base type for compound node fields. The mat:itemimage type is derived from the base type: comp. The type mat:itemimage has the following options set: • fields: (('item', 'mat:item'), ('file', 'file:bytes'))

4.2.149 mat:spec

A GUID assigned to a material specification. The mat:spec type is derived from the base type: guid.

4.2.150 mat:specimage

The base type for compound node fields. The mat:specimage type is derived from the base type: comp. The type mat:specimage has the following options set: • fields: (('spec', 'mat:spec'), ('file', 'file:bytes'))

4.2.151 media:news

A GUID for a news article or report. The media:news type is derived from the base type: guid.

4.2.152 ou:alias

An alias for the org GUID. The ou:alias type is derived from the base type: str. A example of ou:alias: • vertexproject The type ou:alias has the following options set: • lower: True • regex: ^[0-9a-z]+$

4.2.153 ou:conference

A conference with a name and sponsoring org. The ou:conference type is derived from the base type: guid.

4.2.154 ou:conference:attendee

Represents a person attending a conference represented by an ou:conference node. The ou:conference:attendee type is derived from the base type: comp. The type ou:conference:attendee has the following options set: • fields: (('conference', 'ou:conference'), ('person', 'ps:person'))

122 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.2.155 ou:hasalias

The knowledge that an organization has an alias. The ou:hasalias type is derived from the base type: comp. The type ou:hasalias has the following options set: • fields: (('org', 'ou:org'), ('alias', 'ou:alias'))

4.2.156 ou:meet

A informal meeting of people which has no title or sponsor. See also: ou:conference. The ou:meet type is derived from the base type: guid.

4.2.157 ou:meet:attendee

Represents a person attending a meeting represented by an ou:meet node. The ou:meet:attendee type is derived from the base type: comp. The type ou:meet:attendee has the following options set: • fields: (('meet', 'ou:meet'), ('person', 'ps:person'))

4.2.158 ou:member

A person who is (or was) a member of an organization. The ou:member type is derived from the base type: comp. The type ou:member has the following options set: • fields: (('org', 'ou:org'), ('person', 'ps:person'))

4.2.159 ou:naics

The five or six digit North American Industry Classification System code. The ou:naics type is derived from the base type: str. A example of ou:naics: • 541715 The type ou:naics has the following options set: • regex: ^[1-9][0-9]{4}[0-9]?$

4.2.160 ou:name

The name of an organization. This may be a formal name or informal name of the organization. The ou:name type is derived from the base type: str. A example of ou:name: • acme corporation The type ou:name has the following options set: • lower: True • strip: True

4.2. Types 123 synapse Documentation, Release 0.1.0

4.2.161 ou:org

A GUID for a human organization such as a company or military unit. The ou:org type is derived from the base type: guid.

4.2.162 ou:org:has

An org owns, controls, or has exclusive use of an object or resource, potentially during a specific period of time. The ou:org:has type is derived from the base type: comp. The type ou:org:has has the following options set: • fields: (('org', 'ou:org'), ('node', 'ndef'))

4.2.163 ou:orgnet4

An organization’s IPv4 netblock. The ou:orgnet4 type is derived from the base type: comp. The type ou:orgnet4 has the following options set: • fields: (('org', 'ou:org'), ('net', 'inet:net4'))

4.2.164 ou:orgnet6

An organization’s IPv6 netblock. The ou:orgnet6 type is derived from the base type: comp. The type ou:orgnet6 has the following options set: • fields: (('org', 'ou:org'), ('net', 'inet:net6'))

4.2.165 ou:sic

The four digit Standard Industrial Classification Code. The ou:sic type is derived from the base type: str. A example of ou:sic: • 0111 The type ou:sic has the following options set: • regex: ^[0-9]{4}$

4.2.166 ou:suborg

Any parent/child relationship between two orgs. May represent ownership, organizational structure, etc. The ou:suborg type is derived from the base type: comp. The type ou:suborg has the following options set: • fields: (('org', 'ou:org'), ('sub', 'ou:org'))

124 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.2.167 ou:user

A user name within an organization. The ou:user type is derived from the base type: comp. The type ou:user has the following options set: • fields: (('org', 'ou:org'), ('user', 'inet:user'))

4.2.168 pe:langid

The PE language id. The pe:langid type is derived from the base type: int. The type pe:langid has the following options set: • enums: ((3072, 'custom default'), (5120, 'ui_custom_default'), (127, 'invariant'), (0, 'neutral'), (2048, 'sys default'), (4096, 'custom unspecified'), (1024, 'default'), (1078, 'af-ZA'), (1052, 'sq-AL'), (1156, 'gsw-FR'), (1118, 'am-ET'), (5121, 'ar-DZ'), (15361, 'ar-BH'), (3073, 'ar-EG'), (2049, 'ar-IQ'), (11265, 'ar-JO'), (13313, 'ar-KW'), (12289, 'ar-LB'), (4097, 'ar-LY'), (6145, 'ar-MA'), (8193, 'ar-OM'), (16385, 'ar-QA'), (1025, 'ar-SA'), (10241, 'ar-SY'), (7169, 'ar-TN'), (14337, 'ar-AE'), (9217, 'ar-YE'), (1067, 'hy-AM'), (1101, 'as-IN'), (2092, 'az-AZ'), (1068, 'az-AZ'), (1093, 'bn-Bangledesh'), (2117, 'bn-IN'), (1133, 'ba-RU'), (1069, 'Basque-Basque'), (1059, 'be-BY'), (30746, 'bs-neutral'), (8218, 'bs-BA'), (5146, 'bs-BA'), (1150, 'br-FR'), (1026, 'bg-BG'), (1170, 'ku-IQ'), (1116, 'chr-Cher'), (1027, 'ca-ES'), (3076, 'zh-HK'), (5124, 'zh-MO'), (4100, 'zh-SG'), (4, 'zh-Hans'), (31748, 'zh-Hant'), (1155, 'co-FR'), (26, 'hr'), (4122, 'hr-BA'), (1050, 'hr-HR'), (1029, 'cs-CZ'), (1030, 'da-DK'), (1164, 'prs-AF'), (1125, 'dv-MV'), (2067, 'nl-BE'), (1043, 'nl-NL'), (3081, 'en-AU'), (10249, 'en-BZ'), (4105, 'en-CA'), (9225, 'en-029'), (16393, 'en-IN'), (6153, 'en-IE'), (8201, 'en-JM'), (17417, 'en-MY'), (5129, 'en-NZ'), (13321, 'en-PH'), (18441, 'en-SG'), (7177, 'en-ZA'), (11273, 'en-TT'), (2057, 'en-GB'), (1033, 'en-US'), (12297, 'en-ZW'), (1061, 'et-EE'), (1080, 'fo-FO'), (1124, 'fil-PH'), (1035, 'fi-FI'), (2060, 'fr-BE'), (3084, 'fr-CA'), (1036, 'fr-FR'), (5132, 'fr-LU'), (6156, 'fr-MC'), (4108, 'fr-CH'), (1122, 'fy-NL'), (1110, 'gl-ES'), (1079, 'ka-GE'), (3079, 'de-AT'), (1031, 'de-DE'), (5127, 'de-LI'), (4103, 'de-LU'), (2055, 'de-CH'), (1032, 'el-GR'), (1135, 'kl-GL'), (1095, 'gu-IN'), (1128, 'ha-NG'), (1141, 'haw-US'), (1037, 'he-IL'), (1081, 'hi-IN'), (1038, 'hu-HU'), (1039, 'is-IS'), (1136, 'ig-NG'), (1057, 'id-ID'), (2141, 'iu-CA-Latin'), (1117, 'iu-CA'), (2108, 'ga-IE'), (1076, 'xh-ZA'), (1077, 'zu-ZA'), (1040, 'it-IT'), (2064, 'it-CH'), (1041, 'ja-JP'), (1099, 'kn-IN'), (1087, 'kk-KZ'), (1107, 'kh-KH'), (1158, 'qut-GT'), (1159, 'rw-RW'), (1111, 'kok-IN'), (1042, 'ko-KR'), (1088, 'ky-KG'), (1108, 'lo-LA'), (1062, 'lv-LV'), (1063, 'lt-LT'), (2094, 'dsb-DE'), (1134, 'lb-LU'), (1071, 'mk-MK'), (2110, 'ms-BN'), (1086, 'ms-MY'), (1100, 'ml-IN'), (1082, 'mt-MT'), (1153, 'mi-NZ'), (1146, 'arn-CL'), (1102, 'mr-IN'), (1148, 'moh-CA'), (1104, 'mn-MN-Cyrllic'), (2128, 'mn-MN-Prc'), (1121, 'ne-NP'), (1044, 'no-NO'), (2068, 'no-NO'), (1154, 'oc-FR'), (1096, 'or-IN'), (1123, 'ps-AF'), (1065, 'fa-IR'), (1045, 'pl-PL'), (1046, 'pt-BR'), (2070, 'pt-PT'), (2151, 'ff-SN'), (1094, 'pa-IN'), (2118, 'pa-PK'), (1131, 'quz-BO'), (2155, 'quz-EC'), (3179, 'quz-PE'), (1048, 'ro-RO'), (1047, 'rm-CH'), (1049, 'ru-RU'), (1157, 'sah-RU'), (9275, 'smn-FIl'), (4155, 'smj-NO'), (5179, 'smj-SE'), (3131, 'se-FI'), (1083,

4.2. Types 125 synapse Documentation, Release 0.1.0

'se-NO'), (2107, 'se-SE'), (8251, 'sms-FI'), (6203, 'sma-NO'), (7227, 'sma-SE'), (1103, 'sa-IN'), (31770, 'sr-Neutral'), (7194, 'sr-BA'), (6170, 'sr-code-Latin'), (3098, 'sr-CS-Cyrillic'), (2074, 'sr-CS-Latin'), (1132, 'nso-ZA'), (2098, 'tn-BW'), (1074, 'tn-ZA'), (2137, 'sd-PK'), (1115, 'si-LK'), (1051, 'sk-SK'), (1060, 'sl-SI'), (11274, 'es-AR'), (16394, 'es-BO'), (13322, 'es-CL'), (9226, 'es-CO'), (5130, 'es-CR'), (7178, 'es-DO'), (12298, 'es-EC'), (17418, 'es-SV'), (4106, 'es-GT'), (18442, 'es-HN'), (2058, 'es-MX'), (19466, 'es-NI'), (6154, 'es-PA'), (15370, 'es-PY'), (10250, 'es-PE'), (20490, 'es-PR'), (3082, 'es-ES-modern'), (1034, 'es-ES-traditional'), (21514, 'es-US'), (14346, 'es-UY'), (8202, 'es-VE'), (1089, 'sw-KE'), (2077, 'sv-FI'), (1053, 'sv-SE'), (1114, 'syr-SY'), (1064, 'tg-TJ'), (2143, 'tzm-DZ'), (1097, 'ta-IN'), (2121, 'ta-LK'), (1092, 'tt-RU'), (1098, 'te-IN'), (1054, 'th-TH'), (1105, 'bo-CN'), (2163, 'ti-ER'), (1139, 'ti-ET'), (2163, 'tigrigna-'), (1055, 'tr-TR'), (1090, 'tk-TM'), (1058, 'uk-UA'), (1070, 'hsb-DE'), (2080, 'ur-IN'), (1056, 'ur-PK'), (1152, 'ug-CN'), (2115, 'uz-UZ'), (1091, 'uz-UZ'), (2051, 'ca-ES-Valencia'), (1066, 'vi-VN'), (1106, 'cy-GB'), (1160, 'wo-SN'), (1144, 'ii-CN'), (1130, 'yo-NG'))

4.2.169 pe:resource:type

The typecode for the resource. The pe:resource:type type is derived from the base type: int. The type pe:resource:type has the following options set: • enums: ((1, 'RT_CURSOR'), (2, 'RT_BITMAP'), (3, 'RT_ICON'), (4, 'RT_MENU'), (5, 'RT_DIALOG'), (6, 'RT_STRING'), (7, 'RT_FONTDIR'), (8, 'RT_FONT'), (9, 'RT_ACCELERATOR'), (10, 'RT_RCDATA'), (11, 'RT_MESSAGETABLE'), (12, 'RT_GROUP_CURSOR'), (14, 'RT_GROUP_ICON'), (16, 'RT_VERSION'), (17, 'RT_DLGINCLUDE'), (19, 'RT_PLUGPLAY'), (20, 'RT_VXD'), (21, 'RT_ANICURSOR'), (22, 'RT_ANIICON'), (23, 'RT_HTML'), (24, 'RT_MANIFEST'))

4.2.170 pol:country

A GUID for a country. The pol:country type is derived from the base type: guid.

4.2.171 pol:iso2

The 2 digit ISO country code. The pol:iso2 type is derived from the base type: str. A example of pol:iso2: • us The type pol:iso2 has the following options set: • lower: True • nullval: ?? • regex: ^[a-z0-9]{2}$

126 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.2.172 pol:iso3

The 3 digit ISO country code. The pol:iso3 type is derived from the base type: str. A example of pol:iso3: • usa The type pol:iso3 has the following options set: • lower: True • nullval: ?? • regex: ^[a-z0-9]{3}$

4.2.173 pol:isonum

The ISO integer country code. The pol:isonum type is derived from the base type: int. A example of pol:isonum: • 840

4.2.174 ps:contact

A GUID for a contact info record. The ps:contact type is derived from the base type: guid.

4.2.175 ps:name

An arbitrary, lower spaced string with normalized whitespace. The ps:name type is derived from the base type: str. A example of ps:name: • robert grey The type ps:name has the following options set: • lower: True • onespace: True

4.2.176 ps:person

A GUID for a person. The ps:person type is derived from the base type: guid.

4.2.177 ps:person:has

A person owns, controls, or has exclusive use of an object or resource, potentially during a specific period of time. The ps:person:has type is derived from the base type: comp. The type ps:person:has has the following options set: • fields: (('person', 'ps:person'), ('node', 'ndef'))

4.2. Types 127 synapse Documentation, Release 0.1.0

4.2.178 ps:persona

A GUID for a suspected person. The ps:persona type is derived from the base type: guid.

4.2.179 ps:persona:has

A persona owns, controls, or has exclusive use of an object or resource, potentially during a specific period of time. The ps:persona:has type is derived from the base type: comp. The type ps:persona:has has the following options set: • fields: (('persona', 'ps:persona'), ('node', 'ndef'))

4.2.180 ps:tokn

A single name element (potentially given or sur). The ps:tokn type is derived from the base type: str. A example of ps:tokn: • robert The type ps:tokn has the following options set: • lower: True • strip: True

4.2.181 refs

A digraph edge which records that N1 refers to or contains N2. The refs type is derived from the base type: edge.

4.2.182 rsa:key

An RSA keypair modulus and public exponent. The rsa:key type is derived from the base type: comp. The type rsa:key has the following options set: • fields: (('mod', 'hex'), ('pub:exp', 'int'))

4.2.183 seen

Annotates that the data in a node was obtained from or observed by a given source. The seen type is derived from the base type: comp. The type seen has the following options set: • fields: (('source', 'source'), ('node', 'ndef'))

4.2.184 source

A data source unique identifier. The source type is derived from the base type: guid.

128 Chapter 4. Synapse Data Model - Types synapse Documentation, Release 0.1.0

4.2.185 syn:form

A Synapse form used for representing nodes in the graph. The syn:form type is derived from the base type: str. The type syn:form has the following options set: • strip: True

4.2.186 syn:prop

A Synapse property. The syn:prop type is derived from the base type: str. The type syn:prop has the following options set: • strip: True

4.2.187 syn:type

A Synapse type used for normalizing nodes and properties. The syn:type type is derived from the base type: str. The type syn:type has the following options set: • strip: True

4.2.188 tel:mob:carrier

The fusion of a MCC/MNC. The tel:mob:carrier type is derived from the base type: comp. The type tel:mob:carrier has the following options set: • fields: (('mcc', 'tel:mob:mcc'), ('mnc', 'tel:mob:mnc'))

4.2.189 tel:mob:cell

A mobile cell site which a phone may connect to. The tel:mob:cell type is derived from the base type: comp. The type tel:mob:cell has the following options set: • fields: (('carrier', 'tel:mob:carrier'), ('lac', ('int', {})), ('cid', ('int', {})))

4.2.190 tel:mob:imid

Fused knowledge of an IMEI/IMSI used together. The tel:mob:imid type is derived from the base type: comp. A example of tel:mob:imid: • (490154203237518, 310150123456789) The type tel:mob:imid has the following options set: • fields: (('imei', 'tel:mob:imei'), ('imsi', 'tel:mob:imsi'))

4.2. Types 129 synapse Documentation, Release 0.1.0

4.2.191 tel:mob:imsiphone

Fused knowledge of an IMSI assigned phone number. The tel:mob:imsiphone type is derived from the base type: comp. A example of tel:mob:imsiphone: • (310150123456789, "+7(495) 124-59-83") The type tel:mob:imsiphone has the following options set: • fields: (('imsi', 'tel:mob:imsi'), ('phone', 'tel:phone'))

4.2.192 tel:mob:mcc

ITU Mobile Country Code. The tel:mob:mcc type is derived from the base type: str. The type tel:mob:mcc has the following options set: • regex: ^[0-9]{3}$ • strip: 1

4.2.193 tel:mob:mnc

ITU Mobile Network Code. The tel:mob:mnc type is derived from the base type: str. The type tel:mob:mnc has the following options set: • regex: ^[0-9]{2,3}$ • strip: 1

4.2.194 tel:mob:tac

A mobile Type Allocation Code. The tel:mob:tac type is derived from the base type: int. A example of tel:mob:tac: • 49015420

4.2.195 tel:mob:telem

A single mobile telemetry measurement. The tel:mob:telem type is derived from the base type: guid.

4.2.196 wentto

A digraph edge which records that N1 went to N2 at a specific time. The wentto type is derived from the base type: timeedge.

130 Chapter 4. Synapse Data Model - Types CHAPTER 5

Synapse Data Model - Forms

5.1 Forms

Forms are derived from types, or base types. Forms represent node types in the graph.

5.1.1 cluster

A generic node, used in conjunction with Edge types, to cluster arbitrary nodes to a single node in the model. Properties: :name / cluster:name A human friendly name for the cluster. The property type is str. Its type has the following options set: • lower: True :desc / cluster:desc A human friendly long form description for the cluster. The property type is str. Its type has the following options set: • lower: True :type / cluster:type An optional type field used to group clusters. The property type is str. Its type has the following options set: • lower: True

5.1.2 event

A generic event node to represent events outside the model. Properties:

131 synapse Documentation, Release 0.1.0

:time / event:time The time of the event. The property type is time. :type / event:type A arbitrary type string for the event. The property type is str. :name / event:name A name for the event. The property type is str. :data / event:data Aribtrary non-indexed msgpack data attached to the event. The property type is data.

5.1.3 file:base

A file name with no path. Properties: :ext / file:base:ext The file extension (if any). It has the following property options set: • Read Only: 1 The property type is str.

5.1.4 file:bytes

The file bytes type with SHA256 based primary property. Properties: :size / file:bytes:size The file size in bytes. The property type is int. :md5 / file:bytes:md5 The md5 hash of the file. It has the following property options set: • Read Only: 1 The property type is hash:md5. :sha1 / file:bytes:sha1 The sha1 hash of the file. It has the following property options set: • Read Only: 1 The property type is hash:sha1. :sha256 / file:bytes:sha256 The sha256 hash of the file. It has the following property options set: • Read Only: 1 The property type is hash:sha256. :sha512 / file:bytes:sha512 The sha512 hash of the file. It has the following property options set: • Read Only: 1 The property type is hash:sha512. :name / file:bytes:name The best known base name for the file. The property type is file:base. :mime / file:bytes:mime The “best” mime type name for the file. It has the following property options set:

132 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

• Default Value: ?? The property type is file:mime. :mime:x509:cn / file:bytes:mime:x509:cn The Common Name (CN) attribute of the x509 Subject. The property type is str. :mime:pe:size / file:bytes:mime:pe:size The size of the executable file according to the PE file header. The property type is int. :mime:pe:imphash / file:bytes:mime:pe:imphash The PE import hash of the file as calculated by pefile; https:// github.com/erocarrera/pefile. The property type is guid. :mime:pe:compiled / file:bytes:mime:pe:compiled The compile time of the file according to the PE header. The property type is time. :mime:pe:pdbpath / file:bytes:mime:pe:pdbpath The PDB string according to the PE. The property type is file:path. :mime:pe:exports:time / file:bytes:mime:pe:exports:time The export time of the file according to the PE. The property type is time. :mime:pe:exports:libname / file:bytes:mime:pe:exports:libname The export library name according to the PE. The property type is str. :mime:pe:richhdr / file:bytes:mime:pe:richhdr The sha256 hash of the rich header bytes. The property type is hash:sha256.

5.1.5 file:filepath

The fused knowledge of the association of a file:bytes node and a file:path. Properties: :file / file:filepath:file The file seen at a path. It has the following property options set: • Read Only: True The property type is file:bytes. :path / file:filepath:path The path a file was seen at. It has the following property options set: • Read Only: True The property type is file:path. :path:dir / file:filepath:path:dir The parent directory. It has the following property options set: • Read Only: True The property type is file:path. :path:base / file:filepath:path:base The name of the file. It has the following property options set: • Read Only: True The property type is file:base. :path:base:ext / file:filepath:path:base:ext The extension of the file name. It has the following property options set:

5.1. Forms 133 synapse Documentation, Release 0.1.0

• Read Only: True The property type is str.

5.1.6 file:ismime

Records one, of potentially multiple, mime types for a given file. Properties: :file / file:ismime:file The file node that is an instance of the named mime type. It has the following property options set: • Read Only: True The property type is file:bytes. :mime / file:ismime:mime The mime type of the file. It has the following property options set: • Read Only: True The property type is file:mime.

5.1.7 file:mime

A file mime name string.

5.1.8 file:mime:pe:export

The fused knowledge of a file:bytes node containing a pe named export. Properties: :file / file:mime:pe:export:file The file containing the export. It has the following property options set: • Read Only: True The property type is file:bytes. :name / file:mime:pe:export:name The name of the export in the file. It has the following property options set: • Read Only: True The property type is str.

5.1.9 file:mime:pe:resource

The fused knowledge of a file:bytes node containing a pe resource. Properties: :file / file:mime:pe:resource:file The file containing the resource. It has the following property options set: • Read Only: True The property type is file:bytes. :type / file:mime:pe:resource:type The typecode for the resource. It has the following property options set: • Read Only: True

134 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

The property type is pe:resource:type. :langid / file:mime:pe:resource:langid The language code for the resource. It has the following property options set: • Read Only: True The property type is pe:langid. :resource / file:mime:pe:resource:resource The sha256 hash of the resource bytes. It has the following property options set: • Read Only: True The property type is file:bytes.

5.1.10 file:mime:pe:section

The fused knowledge a file:bytes node containing a pe section. Properties: :file / file:mime:pe:section:file The file containing the section. It has the following property options set: • Read Only: True The property type is file:bytes. :name / file:mime:pe:section:name The textual name of the section. It has the following property options set: • Read Only: True The property type is str. :sha256 / file:mime:pe:section:sha256 The sha256 hash of the section. Relocations must be zeroed before hashing. It has the following property options set: • Read Only: True The property type is hash:sha256.

5.1.11 file:mime:pe:vsvers:info knowledge of a file:bytes node containing vsvers info. Properties: :file / file:mime:pe:vsvers:info:file The file containing the vsversion keyval pair. It has the following property op- tions set: • Read Only: True The property type is file:bytes. :keyval / file:mime:pe:vsvers:info:keyval The vsversion info keyval in this file:bytes node. It has the following property options set: • Read Only: True The property type is file:mime:pe:vsvers:keyval.

5.1. Forms 135 synapse Documentation, Release 0.1.0

5.1.12 file:mime:pe:vsvers:keyval

A key value pair found in a PE vsversion info structure. Properties: :name / file:mime:pe:vsvers:keyval:name The key for the vsversion keyval pair. It has the following property op- tions set: • Read Only: True The property type is str. :value / file:mime:pe:vsvers:keyval:value The value for the vsversion keyval pair. It has the following property options set: • Read Only: True The property type is str.

5.1.13 file:path

A normalized file path. Properties: :dir / file:path:dir The parent directory. It has the following property options set: • Read Only: 1 The property type is file:path. :base / file:path:base The file base name. It has the following property options set: • Read Only: 1 The property type is file:base. :base:ext / file:path:base:ext The file extension. It has the following property options set: • Read Only: 1 The property type is str.

5.1.14 file:ref

A file that contains reference to the specified node. Properties: :file / file:ref:file The file that refers to a node. It has the following property options set: • Read Only: 1 The property type is file:bytes. :node / file:ref:node The node referenced by the file. It has the following property options set: • Read Only: 1 The property type is ndef . :node:form / file:ref:node:form The form of node which is referenced. It has the following property options set: • Read Only: 1

136 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

The property type is str. :type / file:ref:type A convention based name for the type of reference. The property type is str. Its type has the following options set: • lower: 1

5.1.15 file:string

The fused knowledge of a file:bytes node containing a string. Properties: :file / file:string:file The file containing the string. It has the following property options set: • Read Only: True The property type is file:bytes. :string / file:string:string The string contained in this file:bytes node. It has the following property options set: • Read Only: True The property type is str.

5.1.16 file:subfile

A parent file that fully contains the specified child file. Properties: :parent / file:subfile:parent The parent file containing the child file. It has the following property options set: • Read Only: True The property type is file:bytes. :child / file:subfile:child The child file contained in the parent file. It has the following property options set: • Read Only: True The property type is file:bytes. :name / file:subfile:name The name of the child file. Because a given set of bytes can have any number of arbitrary names, this field is used for display purposes only. The property type is file:base.

5.1.17 geo:nloc

Records a node latitude/longitude in space-time. Properties: :ndef / geo:nloc:ndef The node with location in geo/time. It has the following property options set: • Read Only: True The property type is ndef . :ndef:form / geo:nloc:ndef:form The form of node referenced by the ndef. It has the following property options set: • Read Only: True

5.1. Forms 137 synapse Documentation, Release 0.1.0

The property type is str. :latlong / geo:nloc:latlong The latitude/longitude the node was observed. It has the following property options set: • Read Only: True The property type is geo:latlong. :time / geo:nloc:time The time the node was observed at location. It has the following property options set: • Read Only: True The property type is time.

5.1.18 geo:place

A GUID for a geographic place. Properties: :name / geo:place:name The name of the place. The property type is str. Its type has the following options set: • lower: 1 • onespace: 1 :desc / geo:place:desc A long form description of the place. The property type is str. :loc / geo:place:loc The geo-political location string for the node. The property type is loc. :latlong / geo:place:latlong The lat/long position for the place. The property type is geo:latlong. :radius / geo:place:radius An approximate radius to use for bounding box calculation. The property type is geo:dist.

5.1.19 gov:cn:icp

A Chinese Internet Content Provider ID. Properties: :org / gov:cn:icp:org The org with the Internet Content Provider ID. The property type is ou:org.

5.1.20 gov:cn:mucd

A Chinese PLA MUCD.

138 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

5.1.21 gov:us:cage

A Commercial and Government Entity (CAGE) code. Properties: :name0 / gov:us:cage:name0 The name of the organization. The property type is ou:name. :name1 / gov:us:cage:name1 Name Part 1. The property type is str. Its type has the following options set: • lower: True :street / gov:us:cage:street The base string type. The property type is str. Its type has the following options set: • lower: True :city / gov:us:cage:city The base string type. The property type is str. Its type has the following options set: • lower: True :state / gov:us:cage:state The base string type. The property type is str. Its type has the following options set: • lower: True :zip / gov:us:cage:zip A US Zip Code. The property type is gov:us:zip. :cc / gov:us:cage:cc The 2 digit ISO country code. The property type is pol:iso2. :country / gov:us:cage:country The base string type. The property type is str. Its type has the following options set: • lower: True :phone0 / gov:us:cage:phone0 A phone number. The property type is tel:phone. :phone1 / gov:us:cage:phone1 A phone number. The property type is tel:phone.

5.1.22 gov:us:ssn

A US Social Security Number (SSN).

5.1.23 gov:us:zip

A US Zip Code.

5.1. Forms 139 synapse Documentation, Release 0.1.0

5.1.24 graph:link

A generic digraph edge to show relationships outside the model. Properties: :n1 / graph:link:n1 The node definition type for a (form,valu) compound field. It has the following property options set: • Read Only: 1 The property type is ndef . :n1:form / graph:link:n1:form The base string type. It has the following property options set: • Read Only: 1 The property type is str. :n2 / graph:link:n2 The node definition type for a (form,valu) compound field. It has the following property options set: • Read Only: 1 The property type is ndef . :n2:form / graph:link:n2:form The base string type. It has the following property options set: • Read Only: 1 The property type is str.

5.1.25 graph:node

A generic node used to represent objects outside the model. Properties: :type / graph:node:type The type name for the non-model node. The property type is str. :name / graph:node:name A human readable name for this record. The property type is str. :data / graph:node:data Aribtrary non-indexed msgpack data attached to the node. The property type is data.

5.1.26 graph:timelink

A generic digraph time edge to show relationships outside the model. Properties: :time / graph:timelink:time A date/time value. It has the following property options set: • Read Only: 1 The property type is time. :n1 / graph:timelink:n1 The node definition type for a (form,valu) compound field. It has the following property options set:

140 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

• Read Only: 1 The property type is ndef . :n1:form / graph:timelink:n1:form The base string type. It has the following property options set: • Read Only: 1 The property type is str. :n2 / graph:timelink:n2 The node definition type for a (form,valu) compound field. It has the following property options set: • Read Only: 1 The property type is ndef . :n2:form / graph:timelink:n2:form The base string type. It has the following property options set: • Read Only: 1 The property type is str.

5.1.27 has

A digraph edge which records that N1 has N2. Properties: :n1 / has:n1 The node definition type for a (form,valu) compound field. It has the following property options set: • Read Only: 1 The property type is ndef . :n1:form / has:n1:form The base string type. It has the following property options set: • Read Only: 1 The property type is str. :n2 / has:n2 The node definition type for a (form,valu) compound field. It has the following property options set: • Read Only: 1 The property type is ndef . :n2:form / has:n2:form The base string type. It has the following property options set: • Read Only: 1 The property type is str.

5.1.28 hash:md5

A hex encodeded MD5 hash.

5.1.29 hash:sha1

A hex encoded SHA1 hash.

5.1. Forms 141 synapse Documentation, Release 0.1.0

5.1.30 hash:sha256

A hex encoded SHA256 hash.

5.1.31 hash:sha384

A hex encoded SHA384 hash.

5.1.32 hash:sha512

A hex encoded SHA512 hash.

5.1.33 inet:asn

An Autonomous System Number (ASN). Properties: :name / inet:asn:name The name of the organization currently responsible for the ASN. It has the following property options set: • Default Value: ?? The property type is str. Its type has the following options set: • lower: True :owner / inet:asn:owner The guid of the organization currently responsible for the ASN. The property type is ou:org.

5.1.34 inet:asnet4

An Autonomous System Number (ASN) and its associated IPv4 address range. Properties: :asn / inet:asnet4:asn The Autonomous System Number (ASN) of the netblock. It has the following property options set: • Read Only: True The property type is inet:asn. :net4 / inet:asnet4:net4 The IPv4 address range assigned to the ASN. It has the following property options set: • Read Only: True The property type is inet:net4. :net4:min / inet:asnet4:net4:min The first IPv4 in the range assigned to the ASN. It has the following property options set: • Read Only: True The property type is inet:ipv4. :net4:max / inet:asnet4:net4:max The last IPv4 in the range assigned to the ASN. It has the following property options set:

142 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

• Read Only: True The property type is inet:ipv4.

5.1.35 inet:banner

A network protocol banner string presented by a server. Properties: :server / inet:banner:server The server which presented the banner string. It has the following property options set: • Read Only: 1 The property type is inet:server. :server:ipv4 / inet:banner:server:ipv4 The IPv4 address of the server. It has the following property options set: • Read Only: 1 The property type is inet:ipv4. :server:ipv6 / inet:banner:server:ipv6 The IPv6 address of the server. It has the following property options set: • Read Only: 1 The property type is inet:ipv6. :server:port / inet:banner:server:port The network port. It has the following property options set: • Read Only: 1 The property type is inet:port. :text / inet:banner:text The banner text. It has the following property options set: • Read Only: 1 The property type is it:dev:str.

5.1.36 inet:cidr4

An IPv4 address block in Classless Inter-Domain Routing (CIDR) notation. Properties: :broadcast / inet:cidr4:broadcast The broadcast IP address from the CIDR notation. It has the following property options set: • Read Only: True The property type is inet:ipv4. :mask / inet:cidr4:mask The mask from the CIDR notation. It has the following property options set: • Read Only: True The property type is int. :network / inet:cidr4:network The network IP address from the CIDR notation. It has the following property options set: • Read Only: True The property type is inet:ipv4.

5.1. Forms 143 synapse Documentation, Release 0.1.0

5.1.37 inet:client

A network client address. Properties: :proto / inet:client:proto The network protocol of the client. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True :ipv4 / inet:client:ipv4 The IPv4 of the client. It has the following property options set: • Read Only: True The property type is inet:ipv4. :ipv6 / inet:client:ipv6 The IPv6 of the client. It has the following property options set: • Read Only: True The property type is inet:ipv6. :host / inet:client:host The it:host node for the client. It has the following property options set: • Read Only: True The property type is it:host. :port / inet:client:port The client tcp/udp port. The property type is inet:port.

5.1.38 inet:dns:a

The result of a DNS A record lookup. Properties: :fqdn / inet:dns:a:fqdn The domain queried for its DNS A record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn. :ipv4 / inet:dns:a:ipv4 The IPv4 address returned in the A record. It has the following property options set: • Read Only: 1 The property type is inet:ipv4.

5.1.39 inet:dns:aaaa

The result of a DNS AAAA record lookup. Properties: :fqdn / inet:dns:aaaa:fqdn The domain queried for its DNS AAAA record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn.

144 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:ipv6 / inet:dns:aaaa:ipv6 The IPv6 address returned in the AAAA record. It has the following property options set: • Read Only: 1 The property type is inet:ipv6.

5.1.40 inet:dns:answer

A single answer from within a DNS reply. Properties: :ttl / inet:dns:answer:ttl The base 64 bit signed integer type. The property type is int. :request / inet:dns:answer:request A single instance of a DNS resolver request and optional reply info. The property type is inet:dns:request. :a / inet:dns:answer:a The DNS A record returned by the lookup. It has the following property options set: • Read Only: True The property type is inet:dns:a. :ns / inet:dns:answer:ns The DNS NS record returned by the lookup. It has the following property options set: • Read Only: True The property type is inet:dns:ns. :rev / inet:dns:answer:rev The DNS PTR record returned by the lookup. It has the following property options set: • Read Only: True The property type is inet:dns:rev. :aaaa / inet:dns:answer:aaaa The DNS AAAA record returned by the lookup. It has the following property options set: • Read Only: True The property type is inet:dns:aaaa. :rev6 / inet:dns:answer:rev6 The DNS PTR record returned by the lookup of a IPv6 address. It has the following property options set: • Read Only: True The property type is inet:dns:rev6. :cname / inet:dns:answer:cname The DNS CNAME record returned by the lookup. It has the following property options set: • Read Only: True The property type is inet:dns:cname. :mx / inet:dns:answer:mx The DNS MX record returned by the lookup. It has the following property options set: • Read Only: True The property type is inet:dns:mx. :soa / inet:dns:answer:soa The domain queried for its SOA record. It has the following property options set: • Read Only: True

5.1. Forms 145 synapse Documentation, Release 0.1.0

The property type is inet:dns:soa. :txt / inet:dns:answer:txt The DNS TXT record returned by the lookup. It has the following property options set: • Read Only: True The property type is inet:dns:txt.

5.1.41 inet:dns:cname

The result of a DNS CNAME record lookup. Properties: :fqdn / inet:dns:cname:fqdn The domain queried for its CNAME record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn. :cname / inet:dns:cname:cname The domain returned in the CNAME record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn.

5.1.42 inet:dns:mx

The result of a DNS MX record lookup. Properties: :fqdn / inet:dns:mx:fqdn The domain queried for its MX record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn. :mx / inet:dns:mx:mx The domain returned in the MX record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn.

5.1.43 inet:dns:ns

The result of a DNS NS record lookup. Properties: :zone / inet:dns:ns:zone The domain queried for its DNS NS record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn. :ns / inet:dns:ns:ns The domain returned in the NS record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn.

146 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

5.1.44 inet:dns:query

A DNS query unique to a given client. Properties: :client / inet:dns:query:client A network client address. The property type is inet:client. :name / inet:dns:query:name A DNS query name string. Likely an FQDN but not always. The property type is inet:dns:name. :name:ipv4 / inet:dns:query:name:ipv4 An IPv4 address. The property type is inet:ipv4. :name:ipv6 / inet:dns:query:name:ipv6 An IPv6 address. The property type is inet:ipv6. :name:fqdn / inet:dns:query:name:fqdn A Fully Qualified Domain Name (FQDN). The property type is inet:fqdn. :type / inet:dns:query:type The base 64 bit signed integer type. The property type is int.

5.1.45 inet:dns:request

A single instance of a DNS resolver request and optional reply info. Properties: :time / inet:dns:request:time A date/time value. The property type is time. :query / inet:dns:request:query A DNS query unique to a given client. The property type is inet:dns:query. :query:name / inet:dns:request:query:name A DNS query name string. Likely an FQDN but not always. The property type is inet:dns:name. :query:name:ipv4 / inet:dns:request:query:name:ipv4 An IPv4 address. The property type is inet:ipv4. :query:name:ipv6 / inet:dns:request:query:name:ipv6 An IPv6 address. The property type is inet:ipv6. :query:name:fqdn / inet:dns:request:query:name:fqdn A Fully Qualified Domain Name (FQDN). The property type is inet:fqdn. :query:type / inet:dns:request:query:type The base 64 bit signed integer type. The property type is int. :server / inet:dns:request:server A network server address. The property type is inet:server.

5.1. Forms 147 synapse Documentation, Release 0.1.0

:reply:code / inet:dns:request:reply:code The DNS server response code. The property type is int. :exe / inet:dns:request:exe The file containing the code that attempted the DNS lookup. The property type is file:bytes. :proc / inet:dns:request:proc The process that attempted the DNS lookup. The property type is it:exec:proc. :host / inet:dns:request:host The host that attempted the DNS lookup. The property type is it:host.

5.1.46 inet:dns:rev

The transformed result of a DNS PTR record lookup. Properties: :ipv4 / inet:dns:rev:ipv4 The IPv4 address queried for its DNS PTR record. It has the following property options set: • Read Only: 1 The property type is inet:ipv4. :fqdn / inet:dns:rev:fqdn The domain returned in the PTR record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn.

5.1.47 inet:dns:rev6

The transformed result of a DNS PTR record for an IPv6 address. Properties: :ipv6 / inet:dns:rev6:ipv6 The IPv6 address queried for its DNS PTR record. It has the following property options set: • Read Only: 1 The property type is inet:ipv6. :fqdn / inet:dns:rev6:fqdn The domain returned in the PTR record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn.

5.1.48 inet:dns:soa

The result of a DNS SOA record lookup. Properties: :fqdn / inet:dns:soa:fqdn The domain queried for its SOA record. It has the following property options set: • Read Only: 1

148 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

The property type is inet:fqdn. :ns / inet:dns:soa:ns The domain (MNAME) returned in the SOA record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn. :email / inet:dns:soa:email The email address (RNAME) returned in the SOA record. It has the following property options set: • Read Only: 1 The property type is inet:email.

5.1.49 inet:dns:txt

The result of a DNS MX record lookup. Properties: :fqdn / inet:dns:txt:fqdn The domain queried for its TXT record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn. :txt / inet:dns:txt:txt The string returned in the TXT record. It has the following property options set: • Read Only: 1 The property type is str.

5.1.50 inet:dns:wild:a

A DNS A wild card record and the IPv4 it resolves to. Properties: :fqdn / inet:dns:wild:a:fqdn The domain containing a wild card record. It has the following property options set: • Read Only: 1 The property type is inet:fqdn. :ipv4 / inet:dns:wild:a:ipv4 The IPv4 address returned by wild card resolutions. It has the following property options set: • Read Only: 1 The property type is inet:ipv4.

5.1.51 inet:dns:wild:aaaa

A DNS AAAA wild card record and the IPv6 it resolves to. Properties: :fqdn / inet:dns:wild:aaaa:fqdn The domain containing a wild card record. It has the following property options set: • Read Only: 1

5.1. Forms 149 synapse Documentation, Release 0.1.0

The property type is inet:fqdn. :ipv6 / inet:dns:wild:aaaa:ipv6 The IPv6 address returned by wild card resolutions. It has the following property options set: • Read Only: 1 The property type is inet:ipv6.

5.1.52 inet:download

An instance of a file downloaded from a server. Properties: :time / inet:download:time The time the file was downloaded. The property type is time. :fqdn / inet:download:fqdn The FQDN used to resolve the server. The property type is inet:fqdn. :file / inet:download:file The file that was downloaded. The property type is file:bytes. :server / inet:download:server The inet:addr of the server. The property type is inet:server. :server:host / inet:download:server:host The it:host node for the server. The property type is it:host. :server:ipv4 / inet:download:server:ipv4 The IPv4 of the server. The property type is inet:ipv4. :server:ipv6 / inet:download:server:ipv6 The IPv6 of the server. The property type is inet:ipv6. :server:port / inet:download:server:port The server tcp/udp port. The property type is inet:port. :server:proto / inet:download:server:proto The server network layer protocol. The property type is str. Its type has the following options set: • lower: True :client / inet:download:client The inet:addr of the client. The property type is inet:client. :client:host / inet:download:client:host The it:host node for the client. The property type is it:host. :client:ipv4 / inet:download:client:ipv4 The IPv4 of the client. The property type is inet:ipv4. :client:ipv6 / inet:download:client:ipv6 The IPv6 of the client. The property type is inet:ipv6.

150 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:client:port / inet:download:client:port The client tcp/udp port. The property type is inet:port. :client:proto / inet:download:client:proto The client network layer protocol. The property type is str. Its type has the following options set: • lower: True

5.1.53 inet:email

An e-mail address. Properties: :user / inet:email:user The username of the email address. It has the following property options set: • Read Only: True The property type is inet:user. :fqdn / inet:email:fqdn The domain of the email address. It has the following property options set: • Read Only: True The property type is inet:fqdn.

5.1.54 inet:email:header

A unique email message header. Properties: :name / inet:email:header:name The name of the email header. It has the following property options set: • Read Only: True The property type is inet:email:header:name. :value / inet:email:header:value The value of the email header. It has the following property options set: • Read Only: True The property type is str.

5.1.55 inet:email:message

A unique email message. Properties: :to / inet:email:message:to The email address of the recipient. The property type is inet:email. :from / inet:email:message:from The email address of the sender. The property type is inet:email. :replyto / inet:email:message:replyto The email address from the reply-to header. The property type is inet:email.

5.1. Forms 151 synapse Documentation, Release 0.1.0

:subject / inet:email:message:subject The email message subject line. The property type is str. :body / inet:email:message:body The body of the email message. The property type is str. :date / inet:email:message:date The time the email message was received. The property type is time. :bytes / inet:email:message:bytes The file bytes which contain the email message. The property type is file:bytes.

5.1.56 inet:email:message:attachment

A file which was attached to an email message. Properties: :message / inet:email:message:attachment:message The message containing the attached file. It has the following property options set: • Read Only: True The property type is inet:email:message. :file / inet:email:message:attachment:file The attached file. It has the following property options set: • Read Only: True The property type is file:bytes. :name / inet:email:message:attachment:name The name of the attached file. The property type is file:base.

5.1.57 inet:email:message:link

A url/link embedded in an email message. Properties: :message / inet:email:message:link:message The message containing the embedded link. It has the following prop- erty options set: • Read Only: True The property type is inet:email:message. :url / inet:email:message:link:url The url contained within the email message. It has the following property options set: • Read Only: True The property type is inet:url.

152 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

5.1.58 inet:flow

An individual network connection between a given source and destination. Properties: :time / inet:flow:time The time the network connection was initiated. The property type is time. :duration / inet:flow:duration The duration of the flow in seconds. The property type is int. :from / inet:flow:from The ingest source file/iden. Used for reparsing. The property type is guid. :dst / inet:flow:dst The destination address / port for a connection. It has the following property options set: • Read Only: True The property type is inet:server. :dst:ipv4 / inet:flow:dst:ipv4 The destination IPv4 address. It has the following property options set: • Read Only: True The property type is inet:ipv4. :dst:ipv6 / inet:flow:dst:ipv6 The destination IPv6 address. It has the following property options set: • Read Only: True The property type is inet:ipv6. :dst:port / inet:flow:dst:port The destination port. It has the following property options set: • Read Only: True The property type is inet:port. :dst:proto / inet:flow:dst:proto The destination port. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True :dst:host / inet:flow:dst:host The guid of the destination host. The property type is it:host. :dst:proc / inet:flow:dst:proc The guid of the destination process. The property type is it:exec:proc. :dst:exe / inet:flow:dst:exe The file (executable) that received the connection. It has the following property options set: • Read Only: True The property type is file:bytes. :dst:txbytes / inet:flow:dst:txbytes The number of bytes sent by the destination host / process / file. It has the following property options set: • Read Only: True

5.1. Forms 153 synapse Documentation, Release 0.1.0

The property type is int. :src / inet:flow:src The source address / port for a connection. It has the following property options set: • Read Only: True The property type is inet:client. :src:ipv4 / inet:flow:src:ipv4 The source IPv4 address. It has the following property options set: • Read Only: True The property type is inet:ipv4. :src:ipv6 / inet:flow:src:ipv6 The source IPv6 address. It has the following property options set: • Read Only: True The property type is inet:ipv6. :src:port / inet:flow:src:port The source port. It has the following property options set: • Read Only: True The property type is inet:port. :src:proto / inet:flow:src:proto The source port. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True :src:host / inet:flow:src:host The guid of the source host. It has the following property options set: • Read Only: True The property type is it:host. :src:proc / inet:flow:src:proc The guid of the source process. It has the following property options set: • Read Only: True The property type is it:exec:proc. :src:exe / inet:flow:src:exe The file (executable) that created the connection. It has the following property options set: • Read Only: True The property type is file:bytes. :src:txbytes / inet:flow:src:txbytes The number of bytes sent by the source host / process / file. It has the following property options set: • Read Only: True The property type is int.

5.1.59 inet:fqdn

A Fully Qualified Domain Name (FQDN). Properties: :created / inet:fqdn:created The earliest known registration (creation) date for the fqdn. The property type is time. Its type has the following options set:

154 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

• ismin: True :domain / inet:fqdn:domain The parent domain for the FQDN. It has the following property options set: • Read Only: True The property type is inet:fqdn. :expires / inet:fqdn:expires The current expiration date for the fqdn. The property type is time. Its type has the following options set: • ismax: True :host / inet:fqdn:host The host part of the FQDN. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True :issuffix / inet:fqdn:issuffix True if the FQDN is considered a suffix. It has the following property options set: • Default Value: 0 The property type is bool. :iszone / inet:fqdn:iszone True if the FQDN is considered a zone. It has the following property options set: • Default Value: 0 The property type is bool. :updated / inet:fqdn:updated The last known updated date for the fqdn. The property type is time. Its type has the following options set: • ismax: True :zone / inet:fqdn:zone The zone level parent for this FQDN. The property type is inet:fqdn.

5.1.60 inet:group

A group name string.

5.1.61 inet:http:cookie

An HTTP cookie string.

5.1.62 inet:http:param

An HTTP request path query parameter. Properties: :name / inet:http:param:name The name of the HTTP query parameter. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set:

5.1. Forms 155 synapse Documentation, Release 0.1.0

• lower: True :value / inet:http:param:value The value of the HTTP query parameter. It has the following property options set: • Read Only: True The property type is str.

5.1.63 inet:http:request

A single HTTP request. Properties: :flow / inet:http:request:flow An individual network connection between a given source and destination. The property type is inet:flow. :client / inet:http:request:client A network client address. It has the following property options set: • Read Only: True The property type is inet:client. :client:ipv4 / inet:http:request:client:ipv4 An IPv4 address. It has the following property options set: • Read Only: True The property type is inet:ipv4. :client:ipv6 / inet:http:request:client:ipv6 An IPv6 address. It has the following property options set: • Read Only: True The property type is inet:ipv6. :server / inet:http:request:server A network server address. It has the following property options set: • Read Only: True The property type is inet:server. :server:ipv4 / inet:http:request:server:ipv4 An IPv4 address. It has the following property options set: • Read Only: True The property type is inet:ipv4. :server:ipv6 / inet:http:request:server:ipv6 An IPv6 address. It has the following property options set: • Read Only: True The property type is inet:ipv6. :server:port / inet:http:request:server:port A network port. It has the following property options set: • Read Only: True The property type is inet:port. :time / inet:http:request:time The time that the HTTP request was sent. The property type is time. :method / inet:http:request:method The HTTP request method string. The property type is str.

156 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:path / inet:http:request:path The requested HTTP path (without query parameters). The property type is str. :url / inet:http:request:url The reconstructed URL for the request if known. The property type is inet:url. :query / inet:http:request:query The HTTP query string which optionally follows the path. The property type is str. :body / inet:http:request:body The body of the HTTP request. The property type is file:bytes. :response:time / inet:http:request:response:time A date/time value. The property type is time. :response:code / inet:http:request:response:code The base 64 bit signed integer type. The property type is int. :response:reason / inet:http:request:response:reason The base string type. The property type is str. :response:body / inet:http:request:response:body The file bytes type with SHA256 based primary property. The property type is file:bytes.

5.1.64 inet:http:request:header

An HTTP request header. Properties: :name / inet:http:request:header:name The name of the HTTP request header. It has the following property options set: • Read Only: True The property type is inet:http:header:name. :value / inet:http:request:header:value The value of the HTTP request header. It has the following property options set: • Read Only: True The property type is str.

5.1.65 inet:http:response:header

An HTTP response header. Properties: :name / inet:http:response:header:name The name of the HTTP response header. It has the following property options set: • Read Only: True The property type is inet:http:header:name.

5.1. Forms 157 synapse Documentation, Release 0.1.0

:value / inet:http:response:header:value The value of the HTTP response header. It has the following property options set: • Read Only: True The property type is str.

5.1.66 inet:iface

A network interface with a set of associated protocol addresses. Properties: :host / inet:iface:host The guid of the host the interface is associated with. The property type is it:host. :type / inet:iface:type The free-form interface type. The property type is str. Its type has the following options set: • lower: True :mac / inet:iface:mac The (MAC) address of the interface. The property type is inet:mac. :ipv4 / inet:iface:ipv4 The IPv4 address of the interface. The property type is inet:ipv4. :ipv6 / inet:iface:ipv6 The IPv6 address of the interface. The property type is inet:ipv6. :phone / inet:iface:phone The telephone number of the interface. The property type is tel:phone. :wifi:ssid / inet:iface:wifi:ssid The wifi SSID of the interface. The property type is inet:wifi:ssid. :wifi:bssid / inet:iface:wifi:bssid The wifi BSSID of the interface. The property type is inet:mac. :mob:imei / inet:iface:mob:imei The IMEI of the interface. The property type is tel:mob:imei. :mob:imsi / inet:iface:mob:imsi The IMSI of the interface. The property type is tel:mob:imsi.

5.1.67 inet:ipv4

An IPv4 address. Properties: :asn / inet:ipv4:asn The ASN to which the IPv4 address is currently assigned. It has the following property options set: • Default Value: 0

158 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

The property type is inet:asn. :latlong / inet:ipv4:latlong The best known latitude/longitude for the node. The property type is geo:latlong. :loc / inet:ipv4:loc The geo-political location string for the IPv4. It has the following property options set: • Default Value: ?? The property type is loc. :type / inet:ipv4:type The type of IP address (e.g., private, multicast, etc.). It has the following property options set: • Default Value: ?? The property type is str. :dns:rev / inet:ipv4:dns:rev The most current DNS reverse lookup for the IPv4. The property type is inet:fqdn.

5.1.68 inet:ipv6

An IPv6 address. Properties: :asn / inet:ipv6:asn The ASN to which the IPv6 address is currently assigned. It has the following property options set: • Default Value: 0 The property type is inet:asn. :ipv4 / inet:ipv6:ipv4 The mapped ipv4. The property type is inet:ipv4. :latlong / inet:ipv6:latlong The last known latitude/longitude for the node. The property type is geo:latlong. :dns:rev / inet:ipv6:dns:rev The most current DNS reverse lookup for the IPv6. The property type is inet:fqdn. :loc / inet:ipv6:loc The geo-political location string for the IPv6. It has the following property options set: • Default Value: ?? The property type is loc.

5.1.69 inet:mac

A 48-bit Media Access Control (MAC) address. Properties: :vendor / inet:mac:vendor The vendor associated with the 24-bit prefix of a MAC address. It has the following property options set: • Default Value: ?? The property type is str.

5.1. Forms 159 synapse Documentation, Release 0.1.0

5.1.70 inet:passwd

A password string. Properties: :md5 / inet:passwd:md5 The MD5 hash of the password. It has the following property options set: • Read Only: True The property type is hash:md5. :sha1 / inet:passwd:sha1 The SHA1 hash of the password. It has the following property options set: • Read Only: True The property type is hash:sha1. :sha256 / inet:passwd:sha256 The SHA256 hash of the password. It has the following property options set: • Read Only: True The property type is hash:sha256.

5.1.71 inet:rfc2822:addr

An RFC 2822 Address field. Properties: :name / inet:rfc2822:addr:name The name field parsed from an RFC 2822 address string. It has the following property options set: • Read Only: True The property type is ps:name. :email / inet:rfc2822:addr:email The email field parsed from an RFC 2822 address string. It has the following property options set: • Read Only: True The property type is inet:email.

5.1.72 inet:search:query

An instance of a search query issued to a search engine. Properties: :text / inet:search:query:text The search query text. The property type is str. :time / inet:search:query:time The time the web search was issued. The property type is time. :engine / inet:search:query:engine A simple name for the search engine used. It has the following property options set: • Example: google The property type is str. Its type has the following options set:

160 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

• lower: True

5.1.73 inet:search:result

A single result from a web search. Properties: :query / inet:search:result:query The search query that produced the result. The property type is inet:search:query. :title / inet:search:result:title The title of the matching web page. The property type is str. Its type has the following options set: • lower: True :rank / inet:search:result:rank The rank/order of the query result. The property type is int. :url / inet:search:result:url The URL hosting the matching content. The property type is inet:url. :text / inet:search:result:text Extracted/matched text from the matched content. The property type is str. Its type has the following options set: • lower: True

5.1.74 inet:server

A network server address. Properties: :proto / inet:server:proto The network protocol of the server. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True :ipv4 / inet:server:ipv4 The IPv4 of the server. It has the following property options set: • Read Only: True The property type is inet:ipv4. :ipv6 / inet:server:ipv6 The IPv6 of the server. It has the following property options set: • Read Only: True The property type is inet:ipv6. :host / inet:server:host The it:host node for the server. It has the following property options set: • Read Only: True The property type is it:host. :port / inet:server:port The server tcp/udp port. The property type is inet:port.

5.1. Forms 161 synapse Documentation, Release 0.1.0

5.1.75 inet:servfile

A file hosted on a server for access over a network protocol. Properties: :file / inet:servfile:file The file hosted by the server. It has the following property options set: • Read Only: True The property type is file:bytes. :server / inet:servfile:server The inet:addr of the server. It has the following property options set: • Read Only: True The property type is inet:server. :server:proto / inet:servfile:server:proto The network protocol of the server. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True :server:ipv4 / inet:servfile:server:ipv4 The IPv4 of the server. It has the following property options set: • Read Only: True The property type is inet:ipv4. :server:ipv6 / inet:servfile:server:ipv6 The IPv6 of the server. It has the following property options set: • Read Only: True The property type is inet:ipv6. :server:host / inet:servfile:server:host The it:host node for the server. It has the following property options set: • Read Only: True The property type is it:host. :server:port / inet:servfile:server:port The server tcp/udp port. The property type is inet:port.

5.1.76 inet:ssl:cert

An SSL certificate file served by a server. Properties: :file / inet:ssl:cert:file The file bytes for the SSL certificate. It has the following property options set: • Read Only: True The property type is file:bytes. :server / inet:ssl:cert:server The file bytes for the SSL certificate. It has the following property options set: • Read Only: True The property type is inet:server. :server:ipv4 / inet:ssl:cert:server:ipv4 The SSL server IPv4 address. It has the following property options set:

162 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

• Read Only: True The property type is inet:ipv4. :server:ipv6 / inet:ssl:cert:server:ipv6 The SSL server IPv6 address. It has the following property options set: • Read Only: True The property type is inet:ipv6. :server:port / inet:ssl:cert:server:port The SSL server listening port. It has the following property options set: • Read Only: True The property type is inet:port.

5.1.77 inet:url

A Universal Resource Locator (URL). Properties: :fqdn / inet:url:fqdn The fqdn used in the URL (e.g., http://www.woot.com/page.html). It has the following property options set: • Read Only: True The property type is inet:fqdn. :ipv4 / inet:url:ipv4 The IPv4 address used in the URL (e.g., http://1.2.3.4/page.html). It has the following property options set: • Read Only: True The property type is inet:ipv4. :ipv6 / inet:url:ipv6 The IPv6 address used in the URL. It has the following property options set: • Read Only: True The property type is inet:ipv6. :passwd / inet:url:passwd The optional password used to access the URL. It has the following property options set: • Read Only: True The property type is inet:passwd. :base / inet:url:base The base scheme, user/pass, fqdn, port and path w/o parameters. It has the following property options set: • Read Only: True The property type is str. :path / inet:url:path The path in the URL w/o parameters. It has the following property options set: • Read Only: True The property type is str. :params / inet:url:params The URL parameter string. It has the following property options set: • Read Only: True The property type is str.

5.1. Forms 163 synapse Documentation, Release 0.1.0

:port / inet:url:port The port of the URL. URLs prefixed with http will be set to port 80 and URLs prefixed with https will be set to port 443 unless otherwise specified. It has the following property options set: • Read Only: True The property type is inet:port. :proto / inet:url:proto The protocol in the URL. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True :user / inet:url:user The optional username used to access the URL. It has the following property options set: • Read Only: True The property type is inet:user.

5.1.78 inet:urlfile

A file hosted at a specific Universal Resource Locator (URL). Properties: :url / inet:urlfile:url The URL where the file was hosted. It has the following property options set: • Read Only: True The property type is inet:url. :file / inet:urlfile:file The file that was hosted at the URL. It has the following property options set: • Read Only: True The property type is file:bytes.

5.1.79 inet:urlredir

A URL that redirects to another URL, such as via a URL shortening service or an HTTP 302 response. Properties: :src / inet:urlredir:src The original/source URL before redirect. It has the following property options set: • Read Only: True The property type is inet:url. :src:fqdn / inet:urlredir:src:fqdn The FQDN within the src URL (if present). It has the following property options set: • Read Only: True The property type is inet:fqdn. :dst / inet:urlredir:dst The redirected/destination URL. It has the following property options set: • Read Only: True The property type is inet:url. :dst:fqdn / inet:urlredir:dst:fqdn The FQDN within the dst URL (if present). It has the following property options set:

164 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

• Read Only: True The property type is inet:fqdn.

5.1.80 inet:user

A username string.

5.1.81 inet:web:acct

An account with a given Internet-based site or service. Properties: :avatar / inet:web:acct:avatar The file representing the avatar (e.g., profile picture) for the account. The property type is file:bytes. :dob / inet:web:acct:dob A self-declared date of birth for the account (if the account belongs to a person). The property type is time. :email / inet:web:acct:email The email address associated with the account. The property type is inet:email. :latlong / inet:web:acct:latlong The last known latitude/longitude for the node. The property type is geo:latlong. :loc / inet:web:acct:loc A self-declared location for the account. The property type is loc. :name / inet:web:acct:name The localized name associated with the account (may be different from the account identifier, e.g., a display name). The property type is inet:user. :name:en / inet:web:acct:name:en The English version of the name associated with the (may be different from the account identifier, e.g., a display name). The property type is inet:user. :occupation / inet:web:acct:occupation A self-declared occupation for the account. The property type is str. Its type has the following options set: • lower: True :passwd / inet:web:acct:passwd The current password for the account. The property type is inet:passwd. :phone / inet:web:acct:phone The phone number associated with the account. The property type is tel:phone. :realname / inet:web:acct:realname The localized version of the real name of the account owner / registrant. The property type is ps:name. :realname:en / inet:web:acct:realname:en The English version of the real name of the account owner / registrant. The property type is ps:name.

5.1. Forms 165 synapse Documentation, Release 0.1.0

:signup / inet:web:acct:signup The date and time the account was registered. The property type is time. :signup:client / inet:web:acct:signup:client The client address used to sign up for the account. The property type is inet:client. :signup:client:ipv4 / inet:web:acct:signup:client:ipv4 The IPv4 address used to sign up for the account. The property type is inet:ipv4. :signup:client:ipv6 / inet:web:acct:signup:client:ipv6 The IPv6 address used to sign up for the account. The property type is inet:ipv4. :site / inet:web:acct:site The site or service associated with the account. It has the following property options set: • Read Only: True The property type is inet:fqdn. :tagline / inet:web:acct:tagline The text of the account status or tag line. The property type is str. :url / inet:web:acct:url The service provider URL where the account is hosted. The property type is inet:url. :user / inet:web:acct:user The unique identifier for the account (may be different from the common name or display name). It has the following property options set: • Read Only: True The property type is inet:user. :webpage / inet:web:acct:webpage A related URL specified by the account (e.g., a personal or company web page, blog, etc.). The property type is inet:url.

5.1.82 inet:web:action

An instance of an account performing an action at an Internet-based site or service. Properties: :act / inet:web:action:act The action performed by the account. The property type is str. Its type has the following options set: • lower: True • strip: True :acct / inet:web:action:acct The web account associated with the action. It has the following property options set: • Read Only: True The property type is inet:web:acct. :acct:site / inet:web:action:acct:site The site or service associated with the account. It has the following property options set: • Read Only: True The property type is inet:fqdn.

166 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:acct:user / inet:web:action:acct:user The unique identifier for the account. It has the following property options set: • Read Only: True The property type is inet:user. :time / inet:web:action:time The date and time the account performed the action. The property type is time. :client / inet:web:action:client The source client address of the action. The property type is inet:client. :client:ipv4 / inet:web:action:client:ipv4 The source IPv4 address of the action. The property type is inet:ipv4. :client:ipv6 / inet:web:action:client:ipv6 The source IPv6 address of the action. The property type is inet:ipv6.

5.1.83 inet:web:actref

A web action that references a given node. Properties: :act / inet:web:actref:act The action that references the given node. It has the following property options set: • Read Only: True The property type is inet:web:action. :node / inet:web:actref:node The ndef that is referenced as part of the action. It has the following property options set: • Read Only: True The property type is ndef . :node:form / inet:web:actref:node:form The form of node that is referenced as part of the action. It has the follow- ing property options set: • Read Only: True The property type is str.

5.1.84 inet:web:chprofile

A change to a web account. Used to capture historical properties associated with an account, as opposed to current data in the inet:web:acct node. Properties: :acct / inet:web:chprofile:acct The web account associated with the change. It has the following property options set: • Read Only: True The property type is inet:web:acct. :acct:site / inet:web:chprofile:acct:site The site or service associated with the account. It has the following property options set:

5.1. Forms 167 synapse Documentation, Release 0.1.0

• Read Only: True The property type is inet:fqdn. :acct:user / inet:web:chprofile:acct:user The unique identifier for the account. It has the following property options set: • Read Only: True The property type is inet:user. :client / inet:web:chprofile:client The source address used to make the account change. The property type is inet:client. :client:ipv4 / inet:web:chprofile:client:ipv4 The source IPv4 address used to make the account change. The property type is inet:ipv4. :client:ipv6 / inet:web:chprofile:client:ipv6 The source IPv6 address used to make the account change. The property type is inet:ipv6. :time / inet:web:chprofile:time The date and time when the account change occurred. The property type is time. :pv / inet:web:chprofile:pv The prop=valu of the account property that was changed. Valu should be the old / original value, while the new value should be updated on the inet:web:acct form. It has the following property options set: • Read Only: True The property type is nodeprop. :pv:prop / inet:web:chprofile:pv:prop The property that was changed. It has the following property options set: • Read Only: True The property type is str.

5.1.85 inet:web:file

A file posted by a web account. Properties: :acct / inet:web:file:acct The account that owns or is associated with the file. It has the following property options set: • Read Only: True The property type is inet:web:acct. :acct:site / inet:web:file:acct:site The site or service associated with the account. It has the following property op- tions set: • Read Only: True The property type is inet:fqdn. :acct:user / inet:web:file:acct:user The unique identifier for the account. It has the following property options set: • Read Only: True The property type is inet:user. :file / inet:web:file:file The file owned by or associated with the account. It has the following property options set:

168 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

• Read Only: True The property type is file:bytes. :name / inet:web:file:name The name of the file owned by or associated with the account. The property type is file:base. :posted / inet:web:file:posted The date and time the file was posted / submitted. The property type is time. :client / inet:web:file:client The source client address used to post or submit the file. The property type is inet:client. :client:ipv4 / inet:web:file:client:ipv4 The source IPv4 address used to post or submit the file. The property type is inet:ipv4. :client:ipv6 / inet:web:file:client:ipv6 The source IPv6 address used to post or submit the file. The property type is inet:ipv6.

5.1.86 inet:web:follows

A web account follows or is connected to another web account. Properties: :follower / inet:web:follows:follower The account following an account. It has the following property options set: • Read Only: True The property type is inet:web:acct. :followee / inet:web:follows:followee The account followed by an account. It has the following property options set: • Read Only: True The property type is inet:web:acct.

5.1.87 inet:web:group

A group hosted within or registered with a given Internet-based site or service. Properties: :site / inet:web:group:site The site or service associated with the group. It has the following property options set: • Read Only: True The property type is inet:fqdn. :id / inet:web:group:id The site-specific unique identifier for the group (may be different from the common name or display name). It has the following property options set: • Read Only: True The property type is inet:group. :name / inet:web:group:name The localized name associated with the group (may be different from the account identifier, e.g., a display name). The property type is inet:group.

5.1. Forms 169 synapse Documentation, Release 0.1.0

:name:en / inet:web:group:name:en The English version of the name associated with the group (may be different from the localized name). The property type is inet:group. :url / inet:web:group:url The service provider URL where the group is hosted. The property type is inet:url. :avatar / inet:web:group:avatar The file representing the avatar (e.g., profile picture) for the group. The property type is file:bytes. :desc / inet:web:group:desc The text of the description of the group. The property type is str. :webpage / inet:web:group:webpage A related URL specified by the group (e.g., primary web site, etc.). The property type is inet:url. :loc / inet:web:group:loc A self-declared location for the group. The property type is str. Its type has the following options set: • lower: True :latlong / inet:web:group:latlong The last known latitude/longitude for the node. The property type is geo:latlong. :signup / inet:web:group:signup The date and time the group was created on the site. The property type is time. :signup:client / inet:web:group:signup:client The client address used to create the group. The property type is inet:client. :signup:client:ipv4 / inet:web:group:signup:client:ipv4 The IPv4 address used to create the group. The property type is inet:ipv4. :signup:client:ipv6 / inet:web:group:signup:client:ipv6 The IPv6 address used to create the group. The property type is inet:ipv6.

5.1.88 inet:web:logon

An instance of an account authenticating to an Internet-based site or service. Properties: :acct / inet:web:logon:acct The web account associated with the logon event. It has the following property options set: • Read Only: True The property type is inet:web:acct. :acct:site / inet:web:logon:acct:site The site or service associated with the account. It has the following property options set: • Read Only: True The property type is inet:fqdn.

170 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:acct:user / inet:web:logon:acct:user The unique identifier for the account. It has the following property options set: • Read Only: True The property type is inet:user. :time / inet:web:logon:time The date and time the account logged into the service. It has the following property options set: • Read Only: True The property type is time. :client / inet:web:logon:client The source address of the logon. The property type is inet:client. :client:ipv4 / inet:web:logon:client:ipv4 The source IPv4 address of the logon. The property type is inet:ipv4. :client:ipv6 / inet:web:logon:client:ipv6 The source IPv6 address of the logon. The property type is inet:ipv6. :logout / inet:web:logon:logout The date and time the account logged out of the service. It has the following property options set: • Read Only: True The property type is time.

5.1.89 inet:web:memb

A web account that is a member of a web group. Properties: :acct / inet:web:memb:acct The account that is a member of the group. It has the following property options set: • Read Only: True The property type is inet:web:acct. :group / inet:web:memb:group The group that the account is a member of. It has the following property options set: • Read Only: True The property type is inet:web:group. :title / inet:web:memb:title The title or status of the member (e.g., admin, new member, etc.). The property type is str. Its type has the following options set: • lower: True :joined / inet:web:memb:joined The date / time the account joined the group. The property type is time.

5.1. Forms 171 synapse Documentation, Release 0.1.0

5.1.90 inet:web:mesg

A message sent from one web account to another web account. Properties: :from / inet:web:mesg:from The web account that sent the message. It has the following property options set: • Read Only: True The property type is inet:web:acct. :to / inet:web:mesg:to The web account that received the message. It has the following property options set: • Read Only: True The property type is inet:web:acct. :time / inet:web:mesg:time The date and time at which the message was sent. It has the following property options set: • Read Only: True The property type is time. :url / inet:web:mesg:url The URL where the message is posted / visible. The property type is inet:url. :text / inet:web:mesg:text The text of the message. The property type is str. :file / inet:web:mesg:file The file attached to or sent with the message. The property type is file:bytes.

5.1.91 inet:web:post

A post made by a web account. Properties: :acct / inet:web:post:acct The web account that made the post. It has the following property options set: • Read Only: True The property type is inet:web:acct. :acct:site / inet:web:post:acct:site The site or service associated with the account. It has the following property options set: • Read Only: True The property type is inet:fqdn. :acct:user / inet:web:post:acct:user The unique identifier for the account. It has the following property options set: • Read Only: True The property type is inet:user. :text / inet:web:post:text The text of the post. It has the following property options set: • Read Only: True The property type is str.

172 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:time / inet:web:post:time The date and time that the post was made. The property type is time. :url / inet:web:post:url The URL where the post is published / visible. The property type is inet:url. :file / inet:web:post:file The file that was attached to the post. The property type is file:bytes. :replyto / inet:web:post:replyto The post that this post is in reply to. The property type is inet:web:post. :repost / inet:web:post:repost The original post that this is a repost of. The property type is inet:web:post.

5.1.92 inet:web:postref

A web post that references a given node. Properties: :post / inet:web:postref:post The web post that references the given node. It has the following property options set: • Read Only: True The property type is inet:web:post. :node / inet:web:postref:node The ndef that is referenced as part of the web post. It has the following property options set: • Read Only: True The property type is ndef . :node:form / inet:web:postref:node:form The form of node that is referenced as part of the web post. It has the following property options set: • Read Only: True The property type is str.

5.1.93 inet:whois:contact

An individual contact from a domain whois record. Properties: :rec / inet:whois:contact:rec The whois record containing the contact data. It has the following property options set: • Read Only: True The property type is inet:whois:rec. :rec:fqdn / inet:whois:contact:rec:fqdn The domain associated with the whois record. It has the following property options set: • Read Only: True The property type is inet:fqdn. :rec:asof / inet:whois:contact:rec:asof The date of the whois record. It has the following property options set:

5.1. Forms 173 synapse Documentation, Release 0.1.0

• Read Only: True The property type is time. :type / inet:whois:contact:type The contact type (e.g., registrar, registrant, admin, billing, tech, etc.). The property type is str. Its type has the following options set: • lower: True :id / inet:whois:contact:id The ID associated with the contact. The property type is str. Its type has the following options set: • lower: True :name / inet:whois:contact:name The name of the contact. The property type is str. Its type has the following options set: • lower: True :email / inet:whois:contact:email The email address of the contact. The property type is inet:email. :orgname / inet:whois:contact:orgname The name of the contact organization. The property type is ou:name. :address / inet:whois:contact:address The content of the street address field(s) of the contact. The property type is str. Its type has the following options set: • lower: True :city / inet:whois:contact:city The content of the city field of the contact. The property type is str. Its type has the following options set: • lower: True :state / inet:whois:contact:state The content of the state field of the contact. The property type is str. Its type has the following options set: • lower: True :country / inet:whois:contact:country The two-letter country code of the contact. The property type is str. Its type has the following options set: • lower: True :phone / inet:whois:contact:phone The content of the phone field of the contact. The property type is tel:phone. : / inet:whois:contact:fax The content of the fax field of the contact. The property type is tel:phone. :url / inet:whois:contact:url The URL specified for the contact. The property type is inet:url. :whois:fqdn / inet:whois:contact:whois:fqdn The whois server FQDN for the given contact (most likely a registrar). The property type is inet:fqdn.

174 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

5.1.94 inet:whois:email

An email address associated with an FQDN via whois registration text. Properties: :fqdn / inet:whois:email:fqdn The domain with a whois record containing the email address. It has the following property options set: • Read Only: True The property type is inet:fqdn. :email / inet:whois:email:email The email address associated with the domain whois record. It has the following property options set: • Read Only: True The property type is inet:email.

5.1.95 inet:whois:rar

A domain registrar.

5.1.96 inet:whois:rec

A domain whois record. Properties: :fqdn / inet:whois:rec:fqdn The domain associated with the whois record. It has the following property options set: • Read Only: True The property type is inet:fqdn. :asof / inet:whois:rec:asof The date of the whois record. It has the following property options set: • Read Only: True The property type is time. :text / inet:whois:rec:text The full text of the whois record. The property type is str. Its type has the following options set: • lower: True :created / inet:whois:rec:created The “created” time from the whois record. The property type is time. :updated / inet:whois:rec:updated The “last updated” time from the whois record. The property type is time. :expires / inet:whois:rec:expires The “expires” time from the whois record. The property type is time. :registrar / inet:whois:rec:registrar The registrar name from the whois record. It has the following property options set: • Default Value: ??

5.1. Forms 175 synapse Documentation, Release 0.1.0

The property type is inet:whois:rar. :registrant / inet:whois:rec:registrant The registrant name from the whois record. It has the following property options set: • Default Value: ?? The property type is inet:whois:reg.

5.1.97 inet:whois:recns

A nameserver associated with a domain whois record. Properties: :ns / inet:whois:recns:ns A nameserver for a domain as listed in the domain whois record. It has the following property options set: • Read Only: True The property type is inet:fqdn. :rec / inet:whois:recns:rec The whois record containing the nameserver data. It has the following property options set: • Read Only: True The property type is inet:whois:rec. :rec:fqdn / inet:whois:recns:rec:fqdn The domain associated with the whois record. It has the following property options set: • Read Only: True The property type is inet:fqdn. :rec:asof / inet:whois:recns:rec:asof The date of the whois record. It has the following property options set: • Read Only: True The property type is time.

5.1.98 inet:whois:reg

A domain registrant.

5.1.99 inet:whois:regmail

An association between a domain and a registrant email address. Properties: :fqdn / inet:whois:regmail:fqdn The domain associated with the registrant email address. It has the following prop- erty options set: • Read Only: True The property type is inet:fqdn. :email / inet:whois:regmail:email The registrant email address associated with the domain. It has the following property options set:

176 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

• Read Only: True The property type is inet:email.

5.1.100 inet:wifi:ap

An SSID/MAC address combination for a wireless access point. Properties: :ssid / inet:wifi:ap:ssid The SSID for the wireless access point. The property type is inet:wifi:ssid. :bssid / inet:wifi:ap:bssid The MAC address for the wireless access point. The property type is inet:mac. :latlong / inet:wifi:ap:latlong The best known latitude/longitude for the wireless access point. The property type is geo:latlong. :loc / inet:wifi:ap:loc The geo-political location string for the wireless access point. It has the following property options set: • Default Value: ?? The property type is loc.

5.1.101 inet:wifi:ssid

A WiFi service set identifier (SSID) name.

5.1.102 it:app:snort:hit

An instance of a snort rule hit. Properties: :rule / it:app:snort:hit:rule The yara rule that matched the file. The property type is it:app:snort:rule. :flow / it:app:snort:hit:flow The inet:flow that matched the snort rule. The property type is inet:flow. :src / it:app:snort:hit:src The source address of flow that caused the hit. The property type is inet:addr. :src:ipv4 / it:app:snort:hit:src:ipv4 The source IPv4 address of the flow that caused the hit. The property type is inet:ipv4. :src:ipv6 / it:app:snort:hit:src:ipv6 The source IPv6 address of the flow that caused the hit. The property type is inet:ipv6. :src:port / it:app:snort:hit:src:port The source port of the flow that caused the hit. The property type is inet:port.

5.1. Forms 177 synapse Documentation, Release 0.1.0

:dst / it:app:snort:hit:dst The destination address of the trigger. The property type is inet:addr. :dst:ipv4 / it:app:snort:hit:dst:ipv4 The destination IPv4 address of the flow that caused the hit. The property type is inet:ipv4. :dst:ipv6 / it:app:snort:hit:dst:ipv6 The destination IPv4 address of the flow that caused the hit. The property type is inet:ipv6. :dst:port / it:app:snort:hit:dst:port The destination port of the flow that caused the hit. The property type is inet:port. :time / it:app:snort:hit:time The time of the network flow that caused the hit. The property type is time. :sensor / it:app:snort:hit:sensor The sensor host node that produced the hit. The property type is it:host. :version / it:app:snort:hit:version The version of the rule at the time of match. The property type is it:semver.

5.1.103 it:app:snort:rule

A snort rule unique identifier. Properties: :text / it:app:snort:rule:text The snort rule text. The property type is str. :name / it:app:snort:rule:name The name of the snort rule. The property type is str. :version / it:app:snort:rule:version The current version of the rule. The property type is it:semver.

5.1.104 it:app:yara:match

A yara rule match to a file. Properties: :rule / it:app:yara:match:rule The yara rule that matched the file. It has the following property options set: • Read Only: True The property type is it:app:yara:rule. :file / it:app:yara:match:file The file that matched the yara rule. It has the following property options set: • Read Only: True The property type is file:bytes. :version / it:app:yara:match:version The most recent version of the rule evaluated as a match. The property type is it:semver.

178 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

5.1.105 it:app:yara:rule

A yara rule unique identifier. Properties: :text / it:app:yara:rule:text The yara rule text. The property type is str. :name / it:app:yara:rule:name The name of the yara rule. The property type is str. :version / it:app:yara:rule:version The current version of the rule. The property type is it:semver.

5.1.106 it:auth:passwdhash

An instance of a password hash. Properties: :salt / it:auth:passwdhash:salt The (optional) hex encoded salt value used to calculate the password hash. The property type is hex. :hash:md5 / it:auth:passwdhash:hash:md5 The MD5 password hash value. The property type is hash:md5. :hash:sha1 / it:auth:passwdhash:hash:sha1 The SHA1 password hash value. The property type is hash:sha1. :hash:sha256 / it:auth:passwdhash:hash:sha256 The SHA256 password hash value. The property type is hash:sha256. :hash:sha512 / it:auth:passwdhash:hash:sha512 The SHA512 password hash value. The property type is hash:sha512. :hash:lm / it:auth:passwdhash:hash:lm The LM password hash value. The property type is hash:lm. :hash:ntlm / it:auth:passwdhash:hash:ntlm The NTLM password hash value. The property type is hash:ntlm. :passwd / it:auth:passwdhash:passwd The (optional) clear text password for this password hash. The property type is inet:passwd.

5.1.107 it:av:filehit

A file that triggered an alert on a specific antivirus signature. Properties: :file / it:av:filehit:file The file that triggered the signature hit. It has the following property options set: • Read Only: True

5.1. Forms 179 synapse Documentation, Release 0.1.0

The property type is file:bytes. :sig / it:av:filehit:sig The signature that the file triggered on. It has the following property options set: • Read Only: True The property type is it:av:sig. :sig:name / it:av:filehit:sig:name The signature name. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True :sig:soft / it:av:filehit:sig:soft The anti-virus product which contains the signature. It has the following property options set: • Read Only: True The property type is it:prod:soft.

5.1.108 it:av:sig

A signature name within the namespace of an antivirus engine name. Properties: :soft / it:av:sig:soft The anti-virus product which contains the signature. It has the following property options set: • Read Only: True The property type is it:prod:soft. :name / it:av:sig:name The signature name. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True :desc / it:av:sig:desc A free-form description of the signature. The property type is str. :url / it:av:sig:url A reference URL for information about the signature. The property type is inet:url.

5.1.109 it:dev:int

A developer selected integer constant.

5.1.110 it:dev:mutex

A string representing a mutex.

5.1.111 it:dev:pipe

A string representing a named pipe.

180 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

5.1.112 it:dev:regkey

A Windows registry key.

5.1.113 it:dev:regval

A Windows registry key/value pair. Properties: :key / it:dev:regval:key The Windows registry key. The property type is it:dev:regkey. :str / it:dev:regval:str The value of the registry key, if the value is a string. The property type is it:dev:str. :int / it:dev:regval:int The value of the registry key, if the value is an integer. The property type is it:dev:int. :bytes / it:dev:regval:bytes The file representing the value of the registry key, if the value is binary data. The property type is file:bytes.

5.1.114 it:dev:str

A developer-selected string. Properties: :norm / it:dev:str:norm Lower case normalized version of the it:dev:str. The property type is str. Its type has the following options set: • lower: True

5.1.115 it:exec:bind

An instance of a host binding a listening port. Properties: :proc / it:exec:bind:proc The main process executing code that bound the listening port. The property type is it:exec:proc. :host / it:exec:bind:host The host running the process that bound the listening port. Typically the same host refer- enced in :proc, if present. The property type is it:host. :exe / it:exec:bind:exe The specific file containing code that bound the listening port. May or may not be the same :exe specified in :proc, if present. The property type is file:bytes. :time / it:exec:bind:time The time the port was bound. The property type is time.

5.1. Forms 181 synapse Documentation, Release 0.1.0

:server / it:exec:bind:server The inet:addr of the server when binding the port. The property type is inet:server. :server:ipv4 / it:exec:bind:server:ipv4 The IPv4 address specified to bind(). The property type is inet:ipv4. :server:ipv6 / it:exec:bind:server:ipv6 The IPv6 address specified to bind(). The property type is inet:ipv6. :server:port / it:exec:bind:server:port The bound (listening) TCP port. The property type is inet:port.

5.1.116 it:exec:file:add

An instance of a host adding a file to a filesystem. Properties: :proc / it:exec:file:add:proc The main process executing code that created the new file. The property type is it:exec:proc. :host / it:exec:file:add:host The host running the process that created the new file. Typically the same host referenced in :proc, if present. The property type is it:host. :exe / it:exec:file:add:exe The specific file containing code that created the new file. May or may not be the same :exe specified in :proc, if present. The property type is file:bytes. :time / it:exec:file:add:time The time the file was created. The property type is time. :path / it:exec:file:add:path The path where the file was created. The property type is file:path. :path:dir / it:exec:file:add:path:dir The parent directory of the file path (parsed from :path). It has the following property options set: • Read Only: True The property type is file:path. :path:ext / it:exec:file:add:path:ext The file extension of the file name (parsed from :path). It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True • strip: True :path:base / it:exec:file:add:path:base The final component of the file path (parsed from :path). It has the following property options set: • Read Only: True The property type is file:base.

182 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:file / it:exec:file:add:file The file that was created. The property type is file:bytes.

5.1.117 it:exec:file:del

An instance of a host deleting a file from a filesystem. Properties: :proc / it:exec:file:del:proc The main process executing code that deleted the file. The property type is it:exec:proc. :host / it:exec:file:del:host The host running the process that deleted the file. Typically the same host referenced in :proc, if present. The property type is it:host. :exe / it:exec:file:del:exe The specific file containing code that deleted the file. May or may not be the same :exe specified in :proc, if present. The property type is file:bytes. :time / it:exec:file:del:time The time the file was deleted. The property type is time. :path / it:exec:file:del:path The path where the file was deleted. The property type is file:path. :path:dir / it:exec:file:del:path:dir The parent directory of the file path (parsed from :path). It has the following property options set: • Read Only: True The property type is file:path. :path:ext / it:exec:file:del:path:ext The file extension of the file name (parsed from :path). It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True • strip: True :path:base / it:exec:file:del:path:base The final component of the file path (parsed from :path). It has the following property options set: • Read Only: True The property type is file:base. :file / it:exec:file:del:file The file that was deleted. The property type is file:bytes.

5.1. Forms 183 synapse Documentation, Release 0.1.0

5.1.118 it:exec:file:read

An instance of a host reading a file from a filesystem. Properties: :proc / it:exec:file:read:proc The main process executing code that read the file. The property type is it:exec:proc. :host / it:exec:file:read:host The host running the process that read the file. Typically the same host referenced in :proc, if present. The property type is it:host. :exe / it:exec:file:read:exe The specific file containing code that read the file. May or may not be the same :exe specified in :proc, if present. The property type is file:bytes. :time / it:exec:file:read:time The time the file was read. The property type is time. :path / it:exec:file:read:path The path where the file was read. The property type is file:path. :path:dir / it:exec:file:read:path:dir The parent directory of the file path (parsed from :path). It has the following property options set: • Read Only: True The property type is file:path. :path:ext / it:exec:file:read:path:ext The file extension of the file name (parsed from :path). It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True • strip: True :path:base / it:exec:file:read:path:base The final component of the file path (parsed from :path). It has the following property options set: • Read Only: True The property type is file:base. :file / it:exec:file:read:file The file that was read. The property type is file:bytes.

5.1.119 it:exec:file:write

An instance of a host writing a file to a filesystem. Properties: :proc / it:exec:file:write:proc The main process executing code that wrote to / modified the existing file. The property type is it:exec:proc.

184 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:host / it:exec:file:write:host The host running the process that wrote to the file. Typically the same host referenced in :proc, if present. The property type is it:host. :exe / it:exec:file:write:exe The specific file containing code that wrote to the file. May or may not be the same :exe specified in :proc, if present. The property type is file:bytes. :time / it:exec:file:write:time The time the file was written to/modified. The property type is time. :path / it:exec:file:write:path The path where the file was written to/modified. The property type is file:path. :path:dir / it:exec:file:write:path:dir The parent directory of the file path (parsed from :path). It has the following property options set: • Read Only: True The property type is file:path. :path:ext / it:exec:file:write:path:ext The file extension of the file name (parsed from :path). It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True • strip: True :path:base / it:exec:file:write:path:base The final component of the file path (parsed from :path). It has the follow- ing property options set: • Read Only: True The property type is file:base. :file / it:exec:file:write:file The file that was modified. The property type is file:bytes.

5.1.120 it:exec:mutex

A mutex created by a process at runtime. Properties: :proc / it:exec:mutex:proc The main process executing code that created the mutex. The property type is it:exec:proc. :host / it:exec:mutex:host The host running the process that created the mutex. Typically the same host referenced in :proc, if present. The property type is it:host. :exe / it:exec:mutex:exe The specific file containing code that created the mutex. May or may not be the same :exe specified in :proc, if present. The property type is file:bytes.

5.1. Forms 185 synapse Documentation, Release 0.1.0

:time / it:exec:mutex:time The time the mutex was created. The property type is time. :name / it:exec:mutex:name The mutex string. The property type is it:dev:mutex.

5.1.121 it:exec:pipe

A named pipe created by a process at runtime. Properties: :proc / it:exec:pipe:proc The main process executing code that created the named pipe. The property type is it:exec:proc. :host / it:exec:pipe:host The host running the process that created the named pipe. Typically the same host referenced in :proc, if present. The property type is it:host. :exe / it:exec:pipe:exe The specific file containing code that created the named pipe. May or may not be the same :exe specified in :proc, if present. The property type is file:bytes. :time / it:exec:pipe:time The time the named pipe was created. The property type is time. :name / it:exec:pipe:name The named pipe string. The property type is it:dev:pipe.

5.1.122 it:exec:proc

A process executing on a host. May be an actual (e.g., endpoint) or virtual (e.g., malware sandbox) host. Properties: :host / it:exec:proc:host The host that executed the process. May be an actual or a virtual / notional host. The property type is it:host. :exe / it:exec:proc:exe The file considered the “main” executable for the process. For example, rundll32.exe may be considered the “main” executable for DLLs loaded by that program. The property type is file:bytes. :cmd / it:exec:proc:cmd The command string used to launch the process, including any command line parameters. The property type is str. :pid / it:exec:proc:pid The process ID. The property type is int. :time / it:exec:proc:time The start time for the process. The property type is time. :user / it:exec:proc:user The user name of the process owner. The property type is inet:user.

186 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:path / it:exec:proc:path The path to the executable of the process. The property type is file:path. :src:exe / it:exec:proc:src:exe The path to the executable which started the process. The property type is file:path. :src:proc / it:exec:proc:src:proc The process which created the process. The property type is it:exec:proc.

5.1.123 it:exec:reg:del

An instance of a host deleting a registry key. Properties: :proc / it:exec:reg:del:proc The main process executing code that deleted data from the registry. The property type is it:exec:proc. :host / it:exec:reg:del:host The host running the process that deleted data from the registry. Typically the same host referenced in :proc, if present. The property type is it:host. :exe / it:exec:reg:del:exe The specific file containing code that deleted data from the registry. May or may not be the same :exe referenced in :proc, if present. The property type is file:bytes. :time / it:exec:reg:del:time The time the data from the registry was deleted. The property type is time. :reg / it:exec:reg:del:reg The registry key or value that was deleted. The property type is it:dev:regval. :reg:key / it:exec:reg:del:reg:key The registry key that was deleted (parsed from :reg). It has the following property options set: • Read Only: True The property type is it:dev:regkey. :reg:str / it:exec:reg:del:reg:str The string value that was deleted (parsed from :reg). It has the following property options set: • Read Only: True The property type is it:dev:str. :reg:int / it:exec:reg:del:reg:int The integer value that was deleted (parsed from :reg). It has the following property options set: • Read Only: True The property type is it:dev:int. :reg:bytes / it:exec:reg:del:reg:bytes The binary data that was deleted (parsed from :reg). It has the following prop- erty options set: • Read Only: True The property type is file:bytes.

5.1. Forms 187 synapse Documentation, Release 0.1.0

5.1.124 it:exec:reg:get

An instance of a host getting a registry key. Properties: :proc / it:exec:reg:get:proc The main process executing code that read the registry. The property type is it:exec:proc. :host / it:exec:reg:get:host The host running the process that read the registry. Typically the same host referenced in :proc, if present. The property type is it:host. :exe / it:exec:reg:get:exe The specific file containing code that read the registry. May or may not be the same :exe referenced in :proc, if present. The property type is file:bytes. :time / it:exec:reg:get:time The time the registry was read. The property type is time. :reg / it:exec:reg:get:reg The registry key or value that was read. The property type is it:dev:regval. :reg:key / it:exec:reg:get:reg:key The registry key that was read (parsed from :reg). It has the following property options set: • Read Only: True The property type is it:dev:regkey. :reg:str / it:exec:reg:get:reg:str The string value that was read (parsed from :reg). It has the following property options set: • Read Only: True The property type is it:dev:str. :reg:int / it:exec:reg:get:reg:int The integer value that was read (parsed from :reg). It has the following property options set: • Read Only: True The property type is it:dev:int. :reg:bytes / it:exec:reg:get:reg:bytes The binary data that was read (parsed from :reg). It has the following property options set: • Read Only: True The property type is file:bytes.

5.1.125 it:exec:reg:set

An instance of a host creating or setting a registry key. Properties: :proc / it:exec:reg:set:proc The main process executing code that wrote to the registry. The property type is it:exec:proc.

188 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:host / it:exec:reg:set:host The host running the process that wrote to the registry. Typically the same host referenced in :proc, if present. The property type is it:host. :exe / it:exec:reg:set:exe The specific file containing code that wrote to the registry. May or may not be the same :exe referenced in :proc, if present. The property type is file:bytes. :time / it:exec:reg:set:time The time the registry was written to. The property type is time. :reg / it:exec:reg:set:reg The registry key or value that was written to. The property type is it:dev:regval. :reg:key / it:exec:reg:set:reg:key The registry key that was written (parsed from :reg). It has the following property options set: • Read Only: True The property type is it:dev:regkey. :reg:str / it:exec:reg:set:reg:str The string value that was written (parsed from :reg). It has the following property options set: • Read Only: True The property type is it:dev:str. :reg:int / it:exec:reg:set:reg:int The integer value that was written (parsed from :reg). It has the following property options set: • Read Only: True The property type is it:dev:int. :reg:bytes / it:exec:reg:set:reg:bytes The binary data that was written (parsed from :reg). It has the following prop- erty options set: • Read Only: True The property type is file:bytes.

5.1.126 it:exec:url

A instance of a host requesting a URL. Properties: :proc / it:exec:url:proc The main process executing code that requested the URL. The property type is it:exec:proc. :host / it:exec:url:host The host running the process that requested the URL. Typically the same host referenced in :proc, if present. The property type is it:host. :exe / it:exec:url:exe The specific file containing code that requested the URL. May or may not be the same :exe specified in :proc, if present. The property type is file:bytes.

5.1. Forms 189 synapse Documentation, Release 0.1.0

:time / it:exec:url:time The time the URL was requested. The property type is time. :url / it:exec:url:url The URL that was requested. The property type is inet:url. :client / it:exec:url:client The address of the client during the URL retrieval. The property type is inet:client. :client:ipv4 / it:exec:url:client:ipv4 The IPv4 of the client during the URL retrieval.. The property type is inet:ipv4. :client:ipv6 / it:exec:url:client:ipv6 The IPv6 of the client during the URL retrieval.. The property type is inet:ipv6. :client:port / it:exec:url:client:port The client port during the URL retrieval.. The property type is inet:port.

5.1.127 it:fs:file

A file on a host. Properties: :host / it:fs:file:host The host containing the file. The property type is it:host. :path / it:fs:file:path The path for the file. The property type is file:path. :path:dir / it:fs:file:path:dir The parent directory of the file path (parsed from :path). It has the following property options set: • Read Only: True The property type is file:path. :path:ext / it:fs:file:path:ext The file extension of the file name (parsed from :path). It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • lower: True • strip: True :path:base / it:fs:file:path:base The final component of the file path (parsed from :path). It has the following prop- erty options set: • Read Only: True The property type is file:base. :file / it:fs:file:file The file on the host. The property type is file:bytes.

190 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:ctime / it:fs:file:ctime The file creation time. The property type is time. :mtime / it:fs:file:mtime The file modification time. The property type is time. :atime / it:fs:file:atime The file access time. The property type is time. :user / it:fs:file:user The owner of the file. The property type is inet:user. :group / it:fs:file:group The group owner of the file. The property type is inet:user.

5.1.128 it:host

A GUID that represents a host or system. Properties: :name / it:host:name The name of the host or system. The property type is it:hostname. :desc / it:host:desc A free-form description of the host. The property type is str. :ipv4 / it:host:ipv4 The last known ipv4 address for the host. The property type is inet:ipv4. :latlong / it:host:latlong The last known location for the host. The property type is geo:latlong. :os / it:host:os The operating system of the host. The property type is it:prod:softver.

5.1.129 it:hostname

The name of a host or sytsem.

5.1.130 it:hostsoft

A version of a software product which is present on a given host. Properties: :host / it:hostsoft:host Host with the software. It has the following property options set: • Read Only: True The property type is it:host. :softver / it:hostsoft:softver Software on the host. It has the following property options set:

5.1. Forms 191 synapse Documentation, Release 0.1.0

• Read Only: True The property type is it:prod:softver.

5.1.131 it:hosturl

A url hosted on or served by a host or system. Properties: :host / it:hosturl:host Host serving a url. It has the following property options set: • Read Only: True The property type is it:host. :url / it:hosturl:url URL available on the host. It has the following property options set: • Read Only: True The property type is inet:url.

5.1.132 it:os:android:aaid

An android advertising identification string.

5.1.133 it:os:android:ibroadcast

The given software broadcasts the given Android intent. Properties: :app / it:os:android:ibroadcast:app The app software which broadcasts the android intent. It has the following property options set: • Read Only: True The property type is it:prod:softver. :intent / it:os:android:ibroadcast:intent The android intent which is broadcast by the app. It has the following property options set: • Read Only: True The property type is it:os:android:intent.

5.1.134 it:os:android:ilisten

The given software listens for an android intent. Properties: :app / it:os:android:ilisten:app The app software which listens for the android intent. It has the following property options set: • Read Only: True The property type is it:prod:softver. :intent / it:os:android:ilisten:intent The android intent which is listened for by the app. It has the following property options set:

192 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

• Read Only: True The property type is it:os:android:intent.

5.1.135 it:os:android:intent

An android intent string.

5.1.136 it:os:android:perm

An android permission string.

5.1.137 it:os:android:reqperm

The given software requests the android permission. Properties: :app / it:os:android:reqperm:app The android app which requests the permission. It has the following property options set: • Read Only: True The property type is it:prod:softver. :perm / it:os:android:reqperm:perm The android permission requested by the app. It has the following property options set: • Read Only: True The property type is it:os:android:perm.

5.1.138 it:os:ios:idfa

An iOS advertising identification string.

5.1.139 it:prod:soft

A arbitrary, unversioned software product. Properties: :name / it:prod:soft:name Name of the software. The property type is str. Its type has the following options set: • lower: True • strip: True :desc / it:prod:soft:desc A description of the software. The property type is str. :desc:short / it:prod:soft:desc:short A short description of the software. The property type is str. Its type has the following options set: • lower: True

5.1. Forms 193 synapse Documentation, Release 0.1.0

:author:org / it:prod:soft:author:org Organization which authored the software. The property type is ou:org. :author:acct / it:prod:soft:author:acct Web account of the software author. The property type is inet:web:acct. :author:email / it:prod:soft:author:email Email address of the sofware author. The property type is inet:email. :author:person / it:prod:soft:author:person Person who authored the software. The property type is ps:person. :url / it:prod:soft:url URL relevant for the software. The property type is inet:url. :isos / it:prod:soft:isos Set to True if the software is an operating system. The property type is bool. :islib / it:prod:soft:islib Set to True if the software is a library. The property type is bool.

5.1.140 it:prod:softfile

A file is distributed by a specific software version. Properties: :soft / it:prod:softfile:soft The software which distributes the file. It has the following property options set: • Read Only: True The property type is it:prod:softver. :file / it:prod:softfile:file The file distributed by the software. It has the following property options set: • Read Only: True The property type is file:bytes.

5.1.141 it:prod:softlib

A software version contains a library software version. Properties: :soft / it:prod:softlib:soft The software version that contains the library. It has the following property options set: • Read Only: True The property type is it:prod:softver. :lib / it:prod:softlib:lib The library software version. It has the following property options set: • Read Only: True The property type is it:prod:softver.

194 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

5.1.142 it:prod:softos

The software version is known to be compatible with the given os software version. Properties: :soft / it:prod:softos:soft The software which can run on the operating system. It has the following property options set: • Read Only: True The property type is it:prod:softver. :os / it:prod:softos:os The operating system which the software can run on. It has the following property options set: • Read Only: True The property type is it:prod:softver.

5.1.143 it:prod:softver

A specific version of a software product. Properties: :software / it:prod:softver:software Software associated with this version instance. The property type is it:prod:soft. :software:name / it:prod:softver:software:name The name of the software at a particular version. The property type is str. Its type has the following options set: • lower: True • strip: True :vers / it:prod:softver:vers Version string associated with this version instance. The property type is it:dev:str. :vers:norm / it:prod:softver:vers:norm Normalized version of the version string. The property type is str. Its type has the following options set: • lower: True :arch / it:prod:softver:arch Software architecture. The property type is it:dev:str. :semver / it:prod:softver:semver System normalized semantic version number. The property type is it:semver. :semver:major / it:prod:softver:semver:major Version major number. The property type is int. :semver:minor / it:prod:softver:semver:minor Version minor number. The property type is int. :semver:patch / it:prod:softver:semver:patch Version patch number. The property type is int.

5.1. Forms 195 synapse Documentation, Release 0.1.0

:semver:pre / it:prod:softver:semver:pre Semver prerelease string. The property type is str. :semver:build / it:prod:softver:semver:build Semver build string. The property type is str. :url / it:prod:softver:url URL where a specific version of the software is available from. The property type is inet:url.

5.1.144 it:sec:cve

A vulnerability as designated by a Common Vulnerabilities and Exposures (CVE) number. Properties: :desc / it:sec:cve:desc A free-form description of the CVE vulnerability. The property type is str.

5.1.145 lang:idiom

A subcultural idiom. Properties: :url / lang:idiom:url Authoritative URL for the idiom. The property type is inet:url. :desc:en / lang:idiom:desc:en English description. The property type is str.

5.1.146 lang:trans

Raw text with a documented translation. Properties: :text:en / lang:trans:text:en English translation. The property type is str. :desc:en / lang:trans:desc:en English description. The property type is str.

5.1.147 mat:item

A GUID assigned to a material object. Properties: :name / mat:item:name The human readable name of the material item. The property type is str. Its type has the following options set: • lower: True

196 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:latlong / mat:item:latlong The last known lat/long location of the node. The property type is geo:latlong.

5.1.148 mat:itemimage

The base type for compound node fields. Properties: :item / mat:itemimage:item The item contained within the image file. The property type is mat:item. :file / mat:itemimage:file The file containing an image of the item. The property type is file:bytes.

5.1.149 mat:spec

A GUID assigned to a material specification. Properties: :name / mat:spec:name The human readable name of the material spec. The property type is str. Its type has the following options set: • lower: True

5.1.150 mat:specimage

The base type for compound node fields. Properties: :spec / mat:specimage:spec The spec contained within the image file. The property type is mat:spec. :file / mat:specimage:file The file containing an image of the spec. The property type is file:bytes.

5.1.151 media:news

A GUID for a news article or report. Properties: :url / media:news:url The (optional) URL where the news was published. It has the following property options set: • Example: http://cnn.com/news/mars-lander.html The property type is inet:url. :url:fqdn / media:news:url:fqdn The FQDN within the news URL. It has the following property options set: • Example: cnn.com The property type is inet:fqdn.

5.1. Forms 197 synapse Documentation, Release 0.1.0

:file / media:news:file The (optional) file blob containing or published as the news. The property type is file:bytes. :title / media:news:title Title/Headline for the news. It has the following property options set: • Default Value: ?? • Example: mars lander reaches mars The property type is str. Its type has the following options set: • lower: True :summary / media:news:summary A brief summary of the news item. It has the following property options set: • Default Value: ?? • Example: lorum ipsum The property type is str. :published / media:news:published The date the news item was published. It has the following property options set: • Default Value: 0 • Example: 20161201180433 The property type is time. :org / media:news:org The org alias which published the news. It has the following property options set: • Example: microsoft The property type is ou:alias. :author / media:news:author The free-form author of the news. It has the following property options set: • Default Value: ?,? • Example: stark,anthony The property type is ps:name.

5.1.152 ou:conference

A conference with a name and sponsoring org. Properties: :org / ou:conference:org The org which created/managed the conference. The property type is ou:org. :name / ou:conference:name The full name of the conference. It has the following property options set: • Example: decfon 2017 The property type is str. Its type has the following options set: • lower: True :base / ou:conference:base The base name which is shared by all conference instances. It has the following property options set: • Example: defcon The property type is str. Its type has the following options set:

198 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

• lower: True • strip: True :start / ou:conference:start The conference start date / time. The property type is time. :end / ou:conference:end The conference end date / time. The property type is time. :place / ou:conference:place The geo:place node where the conference was held. The property type is geo:place.

5.1.153 ou:conference:attendee

Represents a person attending a conference represented by an ou:conference node. Properties: :conference / ou:conference:attendee:conference The conference which was attended. It has the following property options set: • Read Only: True The property type is ou:conference. :person / ou:conference:attendee:person The person who attended the conference. It has the following property options set: • Read Only: True The property type is ps:person. :arrived / ou:conference:attendee:arrived The time when a person arrived to the conference. The property type is time. :departed / ou:conference:attendee:departed The time when a person departed from the conference. The property type is time. :role:staff / ou:conference:attendee:role:staff The person worked as staff at the conference. The property type is bool. :role:speaker / ou:conference:attendee:role:speaker The person was a speaker or presenter at the conference. The property type is bool.

5.1.154 ou:hasalias

The knowledge that an organization has an alias. Properties: :org / ou:hasalias:org Org guid. It has the following property options set: • Read Only: True The property type is ou:org. :alias / ou:hasalias:alias Alias for the organization. It has the following property options set:

5.1. Forms 199 synapse Documentation, Release 0.1.0

• Read Only: True The property type is ou:alias.

5.1.155 ou:meet

A informal meeting of people which has no title or sponsor. See also: ou:conference. Properties: :name / ou:meet:name A human friendly name for the meeting. The property type is str. Its type has the following options set: • lower: True :start / ou:meet:start The date / time the meet starts. The property type is time. :end / ou:meet:end The date / time the meet ends. The property type is time. :place / ou:meet:place The geo:place node where the meet was held. The property type is geo:place.

5.1.156 ou:meet:attendee

Represents a person attending a meeting represented by an ou:meet node. Properties: :meet / ou:meet:attendee:meet The meeting which was attended. It has the following property options set: • Read Only: True The property type is ou:meet. :person / ou:meet:attendee:person The person who attended the meeting. It has the following property options set: • Read Only: True The property type is ps:person. :arrived / ou:meet:attendee:arrived The time when a person arrived to the meeting. The property type is time. :departed / ou:meet:attendee:departed The time when a person departed from the meeting. The property type is time.

5.1.157 ou:member

A person who is (or was) a member of an organization. Properties: :org / ou:member:org The GUID of the org the person is a member of. It has the following property options set: • Read Only: True

200 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

The property type is ou:org. :person / ou:member:person The GUID of the person that is a member of an org. It has the following property options set: • Read Only: True The property type is ps:person. :title / ou:member:title The persons normalized title. The property type is str. Its type has the following options set: • lower: True • strip: True :start / ou:member:start Earliest known association of the person with the org. The property type is time. Its type has the following options set: • ismin: True :end / ou:member:end Most recent known association of the person with the org. The property type is time. Its type has the following options set: • ismax: True

5.1.158 ou:name

The name of an organization. This may be a formal name or informal name of the organization.

5.1.159 ou:org

A GUID for a human organization such as a company or military unit. Properties: :loc / ou:org:loc Location for an organization. The property type is loc. :name / ou:org:name The localized name of an organization. The property type is ou:name. :alias / ou:org:alias The default alias for an organization. The property type is ou:alias. :phone / ou:org:phone The primary phone number for the organization. The property type is tel:phone. :sic / ou:org:sic The Standard Industrial Classification code for the organization. The property type is ou:sic. :naics / ou:org:naics The North American Industry Classification System code for the organization. The property type is ou:naics. :us:cage / ou:org:us:cage The Commercial and Government Entity (CAGE) code for the organization. The property type is gov:us:cage.

5.1. Forms 201 synapse Documentation, Release 0.1.0

:founded / ou:org:founded The date on which the org was founded. The property type is time. :disolved / ou:org:disolved The date on which the org was disolved. The property type is time. :url / ou:org:url The primary url for the organization. The property type is inet:url.

5.1.160 ou:org:has

An org owns, controls, or has exclusive use of an object or resource, potentially during a specific period of time. Properties: :org / ou:org:has:org The org who owns or controls the object or resource. It has the following property options set: • Read Only: True The property type is ou:org. :node / ou:org:has:node The object or resource that is owned or controlled by the org. It has the following property options set: • Read Only: True The property type is ndef . :node:form / ou:org:has:node:form The form of the object or resource that is owned or controlled by the org. It has the following property options set: • Read Only: True The property type is str.

5.1.161 ou:orgnet4

An organization’s IPv4 netblock. Properties: :org / ou:orgnet4:org Org guid. It has the following property options set: • Read Only: True The property type is ou:org. :net / ou:orgnet4:net Netblock owned by the organization. It has the following property options set: • Read Only: True The property type is inet:net4. :name / ou:orgnet4:name The name that the organization assigns to this netblock. The property type is str. Its type has the following options set: • lower: True • strip: True

202 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

5.1.162 ou:orgnet6

An organization’s IPv6 netblock. Properties: :org / ou:orgnet6:org Org guid. It has the following property options set: • Read Only: True The property type is ou:org. :net / ou:orgnet6:net Netblock owned by the organization. It has the following property options set: • Read Only: True The property type is inet:net6. :name / ou:orgnet6:name The name that the organization assigns to this netblock. The property type is str. Its type has the following options set: • lower: True • strip: True

5.1.163 ou:suborg

Any parent/child relationship between two orgs. May represent ownership, organizational structure, etc. Properties: :org / ou:suborg:org The org which owns the sub organization. It has the following property options set: • Read Only: True The property type is ou:org. :sub / ou:suborg:sub The sub org which owned by the org. It has the following property options set: • Read Only: True The property type is ou:org. :perc / ou:suborg:perc The optional percentage of sub which is owned by org. The property type is int. Its type has the following options set: • min: 0 • max: 100 :current / ou:suborg:current Bool indicating if the suborg relationship still current. The property type is bool.

5.1.164 ou:user

A user name within an organization. Properties: :org / ou:user:org Org guid. It has the following property options set: • Read Only: True

5.1. Forms 203 synapse Documentation, Release 0.1.0

The property type is ou:org. :user / ou:user:user The username associated with the organization. It has the following property options set: • Read Only: True The property type is inet:user.

5.1.165 pol:country

A GUID for a country. Properties: :flag / pol:country:flag The file bytes type with SHA256 based primary property. The property type is file:bytes. :founded / pol:country:founded A date/time value. It has the following property options set: • Default Value: 0 The property type is time. :iso2 / pol:country:iso2 The 2 digit ISO country code. The property type is pol:iso2. :iso3 / pol:country:iso3 The 3 digit ISO country code. The property type is pol:iso3. :isonum / pol:country:isonum The ISO integer country code. The property type is pol:isonum. :name / pol:country:name The base string type. The property type is str. Its type has the following options set: • lower: True :pop / pol:country:pop The base 64 bit signed integer type. It has the following property options set: • Default Value: 0 The property type is int. :tld / pol:country:tld A Fully Qualified Domain Name (FQDN). The property type is inet:fqdn.

5.1.166 ps:contact

A GUID for a contact info record. Properties: :org / ps:contact:org The ou:org GUID which owns this contact. The property type is ou:org. :asof / ps:contact:asof A date/time value. It has the following property options set: • date: The time this contact was created or modified. The property type is time.

204 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:person / ps:contact:person The ps:person GUID which owns this contact. The property type is ps:person. :name / ps:contact:name The person name listed for the contact. The property type is ps:name. :title / ps:contact:title The job/org title listed for this contact. The property type is str. Its type has the following options set: • lower: True • strip: True :photo / ps:contact:photo The photo listed for this contact. The property type is file:bytes. :orgname / ps:contact:orgname The listed org/company name for this contact. The property type is ou:name. :user / ps:contact:user The username or handle for this contact. The property type is inet:user. :web:acct / ps:contact:web:acct The account for this contact. The property type is inet:web:acct. :dob / ps:contact:dob The Date of Birth (DOB) for this contact. The property type is time. :url / ps:contact:url The home or main site for this contact. The property type is inet:url. :email / ps:contact:email The main email address for this contact. The property type is inet:email. :email:work / ps:contact:email:work The work email address for this contact. The property type is inet:email. :loc / ps:contact:loc Best known contact geopolitical location. The property type is loc. :address / ps:contact:address The free-form address listed for the contact. The property type is str. Its type has the following options set: • lower: True • strip: True :phone / ps:contact:phone The main phone number for this contact. The property type is tel:phone. :phone:fax / ps:contact:phone:fax The fax number for this contact. The property type is tel:phone. :phone:work / ps:contact:phone:work The work phone number for this contact. The property type is tel:phone.

5.1. Forms 205 synapse Documentation, Release 0.1.0

5.1.167 ps:name

An arbitrary, lower spaced string with normalized whitespace. Properties: :sur / ps:name:sur The surname part of the name. The property type is ps:tokn. :middle / ps:name:middle The middle name part of the name. The property type is ps:tokn. :given / ps:name:given The given name part of the name. The property type is ps:tokn.

5.1.168 ps:person

A GUID for a person. Properties: :dob / ps:person:dob The Date of Birth (DOB) if known. The property type is time. :img / ps:person:img The primary image of a person. The property type is file:bytes. :nick / ps:person:nick A username commonly used by the person. The property type is inet:user. :name / ps:person:name The localized name for the person. The property type is ps:name. :name:sur / ps:person:name:sur The surname of the person. The property type is ps:tokn. :name:middle / ps:person:name:middle The middle name of the person. The property type is ps:tokn. :name:given / ps:person:name:given The given name of the person. The property type is ps:tokn.

5.1.169 ps:person:has

A person owns, controls, or has exclusive use of an object or resource, potentially during a specific period of time. Properties: :person / ps:person:has:person The person who owns or controls the object or resource. It has the following prop- erty options set: • Read Only: True The property type is ps:person.

206 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:node / ps:person:has:node The object or resource that is owned or controlled by the person. It has the following property options set: • Read Only: True The property type is ndef . :node:form / ps:person:has:node:form The form of the object or resource that is owned or controlled by the person. It has the following property options set: • Read Only: True The property type is str.

5.1.170 ps:persona

A GUID for a suspected person. Properties: :dob / ps:persona:dob The Date of Birth (DOB) if known. The property type is time. :img / ps:persona:img The primary image of a suspected person. The property type is file:bytes. :nick / ps:persona:nick A username commonly used by the suspected person. The property type is inet:user. :name / ps:persona:name The localized name for the suspected person. The property type is ps:name. :name:sur / ps:persona:name:sur The surname of the suspected person. The property type is ps:tokn. :name:middle / ps:persona:name:middle The middle name of the suspected person. The property type is ps:tokn. :name:given / ps:persona:name:given The given name of the suspected person. The property type is ps:tokn.

5.1.171 ps:persona:has

A persona owns, controls, or has exclusive use of an object or resource, potentially during a specific period of time. Properties: :persona / ps:persona:has:persona The persona who owns or controls the object or resource. It has the following property options set: • Read Only: True The property type is ps:persona. :node / ps:persona:has:node The object or resource that is owned or controlled by the persona. It has the following property options set: • Read Only: True

5.1. Forms 207 synapse Documentation, Release 0.1.0

The property type is ndef . :node:form / ps:persona:has:node:form The form of the object or resource that is owned or controlled by the per- sona. It has the following property options set: • Read Only: True The property type is str.

5.1.172 ps:tokn

A single name element (potentially given or sur).

5.1.173 refs

A digraph edge which records that N1 refers to or contains N2. Properties: :n1 / refs:n1 The node definition type for a (form,valu) compound field. It has the following property options set: • Read Only: 1 The property type is ndef . :n1:form / refs:n1:form The base string type. It has the following property options set: • Read Only: 1 The property type is str. :n2 / refs:n2 The node definition type for a (form,valu) compound field. It has the following property options set: • Read Only: 1 The property type is ndef . :n2:form / refs:n2:form The base string type. It has the following property options set: • Read Only: 1 The property type is str.

5.1.174 rsa:key

An RSA keypair modulus and public exponent. Properties: :mod / rsa:key:mod The RSA key modulus. It has the following property options set: • Read Only: 1 The property type is hex. :pub:exp / rsa:key:pub:exp The public exponent. It has the following property options set: • Read Only: 1 The property type is int. :bits / rsa:key:bits The length of the modulus in bits. The property type is int.

208 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:priv:exp / rsa:key:priv:exp The private exponent. The property type is hex. :priv:p / rsa:key:priv:p One of the two private primes. The property type is hex. :priv:q / rsa:key:priv:q One of the two private primes. The property type is hex.

5.1.175 seen

Annotates that the data in a node was obtained from or observed by a given source. Properties: :source / seen:source The source which observed or provided the node. It has the following property options set: • Read Only: 1 The property type is source. :node / seen:node The node which was observed by or received from the source. It has the following property options set: • Read Only: 1 The property type is ndef .

5.1.176 source

A data source unique identifier. Properties: :name / source:name A human friendly name for the source. The property type is str. Its type has the following options set: • lower: True :type / source:type An optional type field used to group sources. The property type is str. Its type has the following options set: • lower: True

5.1.177 syn:form

A Synapse form used for representing nodes in the graph. Properties: :doc / syn:form:doc The docstring for the form. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • strip: True :type / syn:form:type Synapse type for this form. It has the following property options set:

5.1. Forms 209 synapse Documentation, Release 0.1.0

• Read Only: True The property type is syn:type. :runt / syn:form:runt Whether or not the form is runtime only. It has the following property options set: • Read Only: True The property type is bool.

5.1.178 syn:prop

A Synapse property. Properties: :doc / syn:prop:doc Description of the property definition. The property type is str. Its type has the following options set: • strip: True :form / syn:prop:form The form of the property. It has the following property options set: • Read Only: True The property type is syn:form. :type / syn:prop:type The synapse type for this property. It has the following property options set: • Read Only: True The property type is syn:type. :relname / syn:prop:relname Relative property name. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • strip: True :univ / syn:prop:univ Specifies if a prop is universal. It has the following property options set: • Read Only: True The property type is bool. :defval / syn:prop:defval Set to the python repr of the default value for this property. It has the following property options set: • Read Only: True The property type is str. :base / syn:prop:base Base name of the property. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • strip: True :ro / syn:prop:ro If the property is read-only after being set. It has the following property options set: • Read Only: True The property type is bool.

210 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

5.1.179 syn:tag

The base type for a synapse tag. Properties: :up / syn:tag:up The parent tag for the tag. It has the following property options set: • Read Only: 1 The property type is syn:tag. :isnow / syn:tag:isnow Set to an updated tag if the tag has been renamed. The property type is syn:tag. :doc / syn:tag:doc A short definition for the tag. It has the following property options set: • Default Value: '' The property type is str. :depth / syn:tag:depth How deep the tag is in the hierarchy. It has the following property options set: • Read Only: 1 The property type is int. :title / syn:tag:title A display title for the tag. It has the following property options set: • Default Value: '' The property type is str. :base / syn:tag:base The tag base name. Eg baz for foo.bar.baz. It has the following property options set: • Read Only: 1 The property type is str.

5.1.180 syn:type

A Synapse type used for normalizing nodes and properties. Properties: :doc / syn:type:doc The docstring for the type. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • strip: True :ctor / syn:type:ctor The python ctor path for the type object. It has the following property options set: • Read Only: True The property type is str. Its type has the following options set: • strip: True :subof / syn:type:subof Type which this inherits from. It has the following property options set: • Read Only: True The property type is syn:type.

5.1. Forms 211 synapse Documentation, Release 0.1.0

:opts / syn:type:opts Arbitrary type options. It has the following property options set: • Read Only: True The property type is data.

5.1.181 tel:mob:carrier

The fusion of a MCC/MNC. Properties: :mcc / tel:mob:carrier:mcc ITU Mobile Country Code. It has the following property options set: • Read Only: 1 The property type is tel:mob:mcc. :mnc / tel:mob:carrier:mnc ITU Mobile Network Code. It has the following property options set: • Read Only: 1 The property type is tel:mob:mnc. :org / tel:mob:carrier:org Organization operating the carrier. The property type is ou:org. :loc / tel:mob:carrier:loc Location the carrier operates from. The property type is loc.

5.1.182 tel:mob:cell

A mobile cell site which a phone may connect to. Properties: :carrier / tel:mob:cell:carrier Mobile carrier. The property type is tel:mob:carrier. :carrier:mcc / tel:mob:cell:carrier:mcc Mobile Country Code. The property type is tel:mob:mcc. :carrier:mnc / tel:mob:cell:carrier:mnc Mobile Network Code. The property type is tel:mob:mnc. :lac / tel:mob:cell:lac Location Area Code. LTE networks may call this a TAC. The property type is int. :cid / tel:mob:cell:cid Cell ID. The property type is int. :radio / tel:mob:cell:radio Cell radio type. The property type is str. Its type has the following options set: • lower: 1 • onespace: 1

212 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

:latlong / tel:mob:cell:latlong Last known location of the cell site. The property type is geo:latlong. :loc / tel:mob:cell:loc Location the cell is operated at. The property type is loc.

5.1.183 tel:mob:imei

An International Mobile Equipment Id. Properties: :tac / tel:mob:imei:tac The Type Allocate Code within the IMEI. It has the following property options set: • Read Only: 1 The property type is tel:mob:tac. :serial / tel:mob:imei:serial The serial number within the IMEI. It has the following property options set: • Read Only: 1 The property type is int.

5.1.184 tel:mob:imid

Fused knowledge of an IMEI/IMSI used together. Properties: :imei / tel:mob:imid:imei The IMEI for the phone hardware. It has the following property options set: • Read Only: 1 The property type is tel:mob:imei. :imsi / tel:mob:imid:imsi The IMSI for the phone subscriber. It has the following property options set: • Read Only: 1 The property type is tel:mob:imsi.

5.1.185 tel:mob:imsi

An International Mobile Subscriber Id. Properties: :mcc / tel:mob:imsi:mcc The Mobile Country Code. It has the following property options set: • Read Only: 1 The property type is tel:mob:mcc.

5.1. Forms 213 synapse Documentation, Release 0.1.0

5.1.186 tel:mob:imsiphone

Fused knowledge of an IMSI assigned phone number. Properties: :phone / tel:mob:imsiphone:phone The phone number assigned to the IMSI. It has the following property options set: • Read Only: 1 The property type is tel:phone. :imsi / tel:mob:imsiphone:imsi The IMSI with the assigned phone number. It has the following property options set: • Read Only: 1 The property type is tel:mob:imsi.

5.1.187 tel:mob:mcc

ITU Mobile Country Code. Properties: :loc / tel:mob:mcc:loc Location assigned to the MCC. The property type is loc.

5.1.188 tel:mob:tac

A mobile Type Allocation Code. Properties: :org / tel:mob:tac:org The org guid for the manufacturer. The property type is ou:org. :manu / tel:mob:tac:manu The TAC manufacturer name. It has the following property options set: • Default Value: ?? The property type is str. Its type has the following options set: • lower: 1 :model / tel:mob:tac:model The TAC model name. It has the following property options set: • Default Value: ?? The property type is str. Its type has the following options set: • lower: 1 :internal / tel:mob:tac:internal The TAC internal model name. It has the following property options set: • Default Value: ?? The property type is str. Its type has the following options set: • lower: 1

214 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

5.1.189 tel:mob:telem

A single mobile telemetry measurement. Properties: :time / tel:mob:telem:time A date/time value. The property type is time. :latlong / tel:mob:telem:latlong A Lat/Long string specifying a point on Earth. The property type is geo:latlong. :cell / tel:mob:telem:cell A mobile cell site which a phone may connect to. The property type is tel:mob:cell. :cell:carrier / tel:mob:telem:cell:carrier The fusion of a MCC/MNC. The property type is tel:mob:carrier. :imsi / tel:mob:telem:imsi An International Mobile Subscriber Id. The property type is tel:mob:imsi. :imei / tel:mob:telem:imei An International Mobile Equipment Id. The property type is tel:mob:imei. :phone / tel:mob:telem:phone A phone number. The property type is tel:phone. :mac / tel:mob:telem:mac A 48-bit Media Access Control (MAC) address. The property type is inet:mac. :ipv4 / tel:mob:telem:ipv4 An IPv4 address. The property type is inet:ipv4. :ipv6 / tel:mob:telem:ipv6 An IPv6 address. The property type is inet:ipv6. :wifi:ssid / tel:mob:telem:wifi:ssid A WiFi service set identifier (SSID) name. The property type is inet:wifi:ssid. :wifi:bssid / tel:mob:telem:wifi:bssid A 48-bit Media Access Control (MAC) address. The property type is inet:mac. :aaid / tel:mob:telem:aaid An android advertising identification string. The property type is it:os:android:aaid. :idfa / tel:mob:telem:idfa An iOS advertising identification string. The property type is it:os:ios:idfa. :name / tel:mob:telem:name An arbitrary, lower spaced string with normalized whitespace. The property type is ps:name. :email / tel:mob:telem:email An e-mail address. The property type is inet:email.

5.1. Forms 215 synapse Documentation, Release 0.1.0

:acct / tel:mob:telem:acct An account with a given Internet-based site or service. The property type is inet:web:acct. :app / tel:mob:telem:app A specific version of a software product. The property type is it:prod:softver. :data / tel:mob:telem:data Arbitrary msgpack compatible data stored without an index. The property type is data.

5.1.190 tel:phone

A phone number. Properties: :loc / tel:phone:loc The location associated with the number. It has the following property options set: • Default Value: ?? The property type is loc.

5.1.191 wentto

A digraph edge which records that N1 went to N2 at a specific time. Properties: :n1 / wentto:n1 The node definition type for a (form,valu) compound field. It has the following property options set: • Read Only: 1 The property type is ndef . :n1:form / wentto:n1:form The base string type. It has the following property options set: • Read Only: 1 The property type is str. :n2 / wentto:n2 The node definition type for a (form,valu) compound field. It has the following property options set: • Read Only: 1 The property type is ndef . :n2:form / wentto:n2:form The base string type. It has the following property options set: • Read Only: 1 The property type is str. :time / wentto:time A date/time value. It has the following property options set: • Read Only: 1 The property type is time.

216 Chapter 5. Synapse Data Model - Forms synapse Documentation, Release 0.1.0

5.2 Universal Properties

Universal props are system level properties which may be present on every node. These properties are not specific to a particular form and exist outside of a particular namespace.

5.2.1 .created

The time the node was created in the cortex. It has the following property options set: • Read Only: True The universal property type is time.

5.2.2 .seen

The time interval for first/last observation of the node. The universal property type is ival.

5.2. Universal Properties 217 synapse Documentation, Release 0.1.0

218 Chapter 5. Synapse Data Model - Forms CHAPTER 6

synapse

6.1 synapse package

The synapse distributed key-value hypergraph analysis framework.

6.1.1 Subpackages synapse.cmds package

Submodules synapse.cmds.cortex module class synapse.cmds.cortex.KillCmd(cli, **opts) Bases: synapse.lib.cli.Cmd Kill a running task/query within the cortex. Syntax: kill Users may specify a partial iden GUID in order to kill exactly one matching process based on the partial guid. runCmdOpts(opts) Perform the command actions. Must be implemented by Cmd implementers. Parameters opts (dict) – Options dictionary. class synapse.cmds.cortex.Log(cli, **opts) Bases: synapse.lib.cli.Cmd Add a storm log to the local command session. Syntax: log (–on|–off) [–splices-only] [–format (mpk|jsonl)] [–path /path/to/file]

219 synapse Documentation, Release 0.1.0

Required Arguments: –on: Enables logging of storm messages to a file. –off: Disables message logging and closes the current storm file. Optional Arguments: –splices-only: Only records splices. Does not record any other messages. –format: The format used to save messages to disk. Defaults to msgpack (mpk). –path: The path to the log file. This will append messages to a existing file.

Notes

By default, the log file contains all messages received from the execution of a Storm query by the current CLI. By default, these messages are saved to a file located in ~/.syn/stormlogs/storm_(date).(format).

Examples

# Enable logging all messages to mpk files (default) log –on # Disable logging and close the current file log –off # Enable logging, but only log splices. Log them as jsonl instead of mpk. log –on –splices-only –format jsonl # Enable logging, but log to a custom path: log –on –path /my/aweome/log/directory/storm20010203.mpk closeLogFd() encodeMsg(mesg) Get byts for a message onStormMesg(mesg) openLogFd(opts) queueLoop() runCmdOpts(opts) Perform the command actions. Must be implemented by Cmd implementers. Parameters opts (dict) – Options dictionary. save(mesg) splicetypes = ('tag:add', 'tag:del', 'node:add', 'node:del', 'prop:set', 'prop:del') class synapse.cmds.cortex.PsCmd(cli, **opts) Bases: synapse.lib.cli.Cmd List running tasks in the cortex. runCmdOpts(opts) Perform the command actions. Must be implemented by Cmd implementers. Parameters opts (dict) – Options dictionary. class synapse.cmds.cortex.StormCmd(cli, **opts) Bases: synapse.lib.cli.Cmd Execute a storm query. Syntax: storm

Parameters query – The storm query

220 Chapter 6. synapse synapse Documentation, Release 0.1.0

Optional Arguments: –hide-tags: Do not print tags –hide-props: Do not print secondary properties –hide- unknown: Do not print messages which do not have known handlers. –raw: Print the nodes in their raw format (overrides –hide-tags and –hide-props)

–debug: Display cmd debug information along with nodes in raw format (overrides –hide-tags, –hide-props and raw)

–path: Get path information about returned nodes. –graph: Get graph information about returned nodes.

Examples

storm inet:ipv4=1.2.3.4 storm –debug inet:ipv4=1.2.3.4 runCmdOpts(opts) Perform the command actions. Must be implemented by Cmd implementers. Parameters opts (dict) – Options dictionary. synapse.cmds.cron module class synapse.cmds.cron.At(cli, **opts) Bases: synapse.lib.cli.Cmd Adds a non-recurring cron job. It will execute a Storm query at one or more specified times. List/details/deleting cron jobs created with ‘at’ use the same commands as other cron jobs: cron list/stat/del respectively. Syntax: at (time|+time delta)+ {query}

Notes

This command accepts one or more time specifications followed by exactly one storm query in curly braces. Each time specification may be in synapse time delta format (e.g + 1 day) or synapse time format (e.g. 20501217030432101). Seconds will be ignored, as cron jobs’ granularity is limited to minutes. All times are interpreted as UTC. The other option for time specification is a relative time from now. This consists of a plus sign, a positive integer, then one of ‘minutes, hours, days’. Note that the record for a cron job is stored until explicitly deleted via “cron del”.

Examples

# Run a storm query in 5 minutes at +5 minutes {[inet:ipv4=1]} # Run a storm query tomorrow and in a week at +1 day +7 days {[inet:ipv4=1]} # Run a query at the end of the year Zulu at 20181231Z2359 {[inet:ipv4=1]} runCmdOpts(opts) Perform the command actions. Must be implemented by Cmd implementers.

6.1. synapse package 221 synapse Documentation, Release 0.1.0

Parameters opts (dict) – Options dictionary. class synapse.cmds.cron.Cron(cli, **opts) Bases: synapse.lib.cli.Cmd Manages cron jobs in a cortex. Cron jobs are rules persistently stored in a cortex such that storm queries automatically run on a time schedule. Cron jobs may be be recurring or one-time. Use the ‘at’ command to add one-time jobs. A subcommand is required. Use ‘cron -h’ for more detailed help. runCmdOpts(opts) Perform the command actions. Must be implemented by Cmd implementers. Parameters opts (dict) – Options dictionary. synapse.cmds.hive module class synapse.cmds.hive.HiveCmd(cli, **opts) Bases: synapse.lib.cli.Cmd Manipulates values in a cell’s Hive. A Hive is a hierarchy persistent storage mechanism typically used for configuration data. static parsepath(path) Turn a slash-delimited path into a list that hive takes runCmdOpts(opts) Perform the command actions. Must be implemented by Cmd implementers. Parameters opts (dict) – Options dictionary. synapse.cmds.hive.tuplify(obj) synapse.cmds.trigger module class synapse.cmds.trigger.Trigger(cli, **opts) Bases: synapse.lib.cli.Cmd Manipulate triggers in a cortex. Triggers are rules persistently stored in a cortex such that storm queries automatically run when a particular event happens. A subcommand is required. Use trigger -h for more detailed help. runCmdOpts(opts) Perform the command actions. Must be implemented by Cmd implementers. Parameters opts (dict) – Options dictionary. synapse.data package synapse.data.get(name, defval=None) Return an object from the embedded synapse data folder.

222 Chapter 6. synapse synapse Documentation, Release 0.1.0

Example

for tld in syanpse.data.get(‘iana.tlds’): dostuff(tld)

NOTE: Files are named synapse/data/.mpk synapse.lib package

Subpackages synapse.lib.crypto package

Submodules synapse.lib.crypto.ecc module class synapse.lib.crypto.ecc.PriKey(priv) Bases: object A helper class for using ECC private keys. dump() Get the private key bytes in DER/PKCS8 format. Returns The DER/PKCS8 encoded private key. Return type bytes exchange(pubkey) Perform a ECDH key exchange with a public key. Parameters pubkey (PubKey) – A PubKey to perform the ECDH with. Returns The ECDH bytes. This is deterministic for a given pubkey and private key. Return type bytes static generate() Generate a new ECC PriKey instance. Returns A new PriKey instance. Return type PriKey iden() Return a SHA256 hash for the public key (to be used as a GUID). Returns The SHA256 hash of the public key bytes. Return type str static load(byts) Create a PriKey instance from DER/PKCS8 encoded bytes. Parameters byts (bytes) – Bytes to load Returns A new PubKey instance. Return type PriKey

6.1. synapse package 223 synapse Documentation, Release 0.1.0

public() Get the PubKey which corresponds to the ECC PriKey. Returns A new PubKey object whose key corresponds to the private key. Return type PubKey sign(byts) Compute the ECC signature for the given bytestream. Parameters byts (bytes) – The bytes to sign. Returns The RSA Signature bytes. Return type bytes class synapse.lib.crypto.ecc.PubKey(publ) Bases: object A helper class for using ECC public keys. dump() Get the public key bytes in DER/SubjectPublicKeyInfo format. Returns The DER/SubjectPublicKeyInfo encoded public key. Return type bytes iden() Return a SHA256 hash for the public key (to be used as a GUID). Returns The SHA256 hash of the public key bytes. Return type str static load(byts) Create a PubKey instance from DER/PKCS8 encoded bytes. Parameters byts (bytes) – Bytes to load Returns A new PubKey instance. Return type PubKey verify(byts, sign) Verify the signature for the given bytes using the ECC public key. Parameters • byts (bytes) – The data bytes. • sign (bytes) – The signature bytes. Returns True if the data was verified, False otherwise. Return type bool synapse.lib.crypto.ecc.doECDHE(statprv_u, statpub_v, ephmprv_u, ephmpub_v, length=64, salt=None, info=None) Perform one side of an Ecliptic Curve Diffie Hellman Ephemeral key exchange. Parameters • statprv_u (PriKey) – Static Private Key for U • (PubKey (statpub_v) – Static Public Key for V • ephmprv_u (PriKey) – Ephemeral Private Key for U

224 Chapter 6. synapse synapse Documentation, Release 0.1.0

• ephmpub_v (PubKey) – Ephemeral Public Key for V • length (int) – Number of bytes to return • salt (bytes) – Salt to use when computing the key. • info (bytes) – Additional information to use when computing the key.

Notes

This makes no assumption about the reuse of the Ephemeral keys passed to the function. It is the caller’s responsibility to destroy the keys after they are used for doing key generation. This implementation is the dhHybrid1 scheme described in NIST 800-56A Revision 2. Returns The derived key. Return type bytes synapse.lib.crypto.tinfoil module class synapse.lib.crypto.tinfoil.CryptSeq(rx_key, tx_key, initial_rx_seq=0, ini- tial_tx_seq=0) Bases: object Applies and verifies sequence numbers of encrypted messages coming and going Parameters • rx_key (bytes) – TX key (used with TinFoilHat). • tx_key (bytes) – RX key (used with TinFoilHat). • initial_rx_seq (int) – Starting rx sequence number. • initial_tx_seq (int) – Starting tx sequence number. decrypt(ciphertext) Decrypt a message, validating its sequence number is as we expect. Parameters ciphertext (bytes) – The message to decrypt and verify. Returns A mesg. Return type mesg Raises s_exc.CryptoErr – If the message decryption fails or the sequence number was unexpected. encrypt(mesg) Wrap a message with a sequence number and encrypt it. Parameters mesg – The mesg to encrypt. Returns The encrypted message. Return type bytes class synapse.lib.crypto.tinfoil.TinFoilHat(ekey) Bases: object The TinFoilHat class implements a GCM-AES encryption/decryption class. Parameters

6.1. synapse package 225 synapse Documentation, Release 0.1.0

• ekey (bytes) – A 32 byte key used for doing encryption & decryption. It • assumed the caller has generated the key in a safe manner. (is)– dec(byts) Decode an envelope dict and decrypt the given bytes. Parameters byts (bytes) – Bytes to decrypt. Returns Decrypted message. Return type bytes enc(byts, asscd=None) Encrypt the given bytes and return an envelope dict in msgpack form. Parameters • byts (bytes) – The message to be encrypted. • asscd (bytes) – Extra data that needs to be authenticated (but not encrypted). Returns The encrypted message. This is a msgpacked dictionary containing the IV, ciphertext, and associated data. Return type bytes synapse.lib.crypto.tinfoil.newkey() Generate a new, random 32 byte key. Returns 32 random bytes Return type bytes synapse.lib.crypto.vault module class synapse.lib.crypto.vault.Cert(cert, rkey=None) Bases: object A Certificate helper object for signing / verifying data. Parameters • cert ((str, dict)) – Certificate tufo • rkey (s_ecc.PriKey) – Private ECC Key addsigner(sign) Append a new signature tuple to the current Cert’s signers. Parameters sign ((str, bytes, bytes)) – Signature tuple to add the Cert. Returns None dump() Get the certificate in bytes for storage. Returns A msgpack encoded form of the Cert. Return type bytes getkey() Get the private ECC key for the certificate. Returns Private ECC Key. If not present, this returns None.

226 Chapter 6. synapse synapse Documentation, Release 0.1.0

Return type s_ecc.PriKey iden() Get the iden for the certificate. Returns Iden of the certificate. Return type str static load(byts, rkey=None) Create a Cert object from the bytes. Parameters • byts (bytes) – Bytes from a previously saved Cert • rkey (s_ecc.PriKey) – The ECC Private Key associated with the Cert. Returns A Cert object for the bytes and ECC private key. Return type Cert public() Get the Public ECC key for the Cert Returns The Public ECC key. Return type s_ecc.PubKey sign(cert, **info) Sign a certificate with the current Cert. Parameters • cert (Cert) – Certificate to sign with the current Cert. • **info – Additional data to include in the signed message. Returns None signed(cert) Check if this cert signed the given Cert and return the info. Parameters cert (Cert) – A Cert to confirm that we signed. Returns The signer info dict ( or None if not signed ). Return type dict signers() Get the signing chain for the Cert. Returns A tuple of tuples; the inner tuples contain iden, data bytes and signature bytes. Return type tuple toknbytes() Get the token bytes for the certificate. Returns The msgpack encoded certificate token dictionary. Return type bytes verify(byts, sign) Verify that the the Cert signed a set of bytes. Parameters • byts (bytes) – Data to check.

6.1. synapse package 227 synapse Documentation, Release 0.1.0

• sign (bytes) – Signature to verify. Returns True if the Cert signed the byts, False otherwise. Return type bool class synapse.lib.crypto.vault.Vault(path) Bases: synapse.eventbus.EventBus A Certificate / Key vault. User tokens are stored in the following format:

tokn: { 'user':, 'ecdsa:pubkey':, }

Certs are stored in the following format:

cert: (,{ "signers":( , ), })

Sigs are stored as tuples of the following:

(,,),

The iden used for signing data must only be used for public key lookup. Parameters path (str) – Path to the DB backing the vault.

Notes

The Vault does store ECC Private Key material and should be treated as a sensitive file by users. addRootCert(cert) Add a certificate to the Vault as a root certificate. Parameters cert (Cert) – Certificate to add to the Vault. Returns None addUserAuth(auth) Store a private user auth tufo. Parameters auth ((str, dict)) – A user auth tufo obtained via the genUserAuth API.

Notes

This is a sensitive API. Auth tufos should only be loaded from trusted sources. This API is primarily designed for provisioning automation. Returns Cert object derived from the auth tufo. Return type Cert

228 Chapter 6. synapse synapse Documentation, Release 0.1.0

delRootCert(cert) Delete a root certificate from the Vault. Parameters cert (Cert) – Certificate for the root CA to remove. Returns None static genCertTokn(rpub, **info) Generate a public key certificate token. Parameters • rpub (s_ecc.PubKey)– • **info – Additional key/value data to be added to the certificate token. Returns A msgpack encoded dictionary. Return type bytes genEccKey() Generate a new ECC key and store it in the vault. Returns The new ECC key. Return type s_ecc.PriKey genRootCert() Get or generate the primary root cert for this vault. Returns A cert helper object Return type Cert genToknCert(tokn, rkey=None) Generate Cert object for a given token and ECC Private key. Parameters • tokn (bytes) – Token which will be signed. • s_ecc.PriKey (rkey) – ECC Private key used to sign the token. Returns A Cert object Return type Cert genUserAuth(user) Generate a sensitive user auth data structure. Parameters user (str) – The user name to generate the auth data for.

Notes

The data returned by this API contains the user certificate and private key material. It is a sensitive data structure and care should be taken as to what happens with the output of this API. Returns A tufo containing the user name and a dictionary of certificate and key material. Return type ((str, dict)) genUserCert(name) Generate a key/cert for the given user. Parameters name (str) – The user name.

6.1. synapse package 229 synapse Documentation, Release 0.1.0

Returns A newly generated user certificate. Return type Cert getCert(iden) Get a certificate by iden. Parameters iden (str) – The cert iden. Returns The Cert or None. Return type Cert getCertKey(iden) Get the ECC Private Key for a given iden. Parameters iden (str) – Iden to retrieve Returns The ECC Private Key object. Return type s_ecc.PriKey getRootCerts() Get a list of the root certificates from the Vault. Returns A list of root certificates as Cert objects. Return type list getUserCert(name) Retrieve a cert tufo for the given user. Parameters name (str) – Name of the user to retrieve the certificate for. Returns User’s certificate object, or None. Return type Cert isValidCert(cert) Check if a Vault can validate a Cert against its root certificates. Parameters cert (Cert) – Cert to check against. Returns True if the certificate is valid, False otherwise. Return type bool synapse.lib.crypto.vault.shared(path) A context manager for locking a potentially shared vault. Parameters path (str) – Path to the vault.

Example

with s_vault.shared(‘~/.syn/vault’) as vault: dostuff()

Yields Vault – A Vault object. synapse.lib.platforms package

Home for platform specific code such as thishost info. all platform modules must be importable from any platform.

230 Chapter 6. synapse synapse Documentation, Release 0.1.0

( guard any platform specific code with appropriate conditionals )

Submodules synapse.lib.platforms.common module synapse.lib.platforms.common.daemonize() For unix platforms, form a new process group using fork(). synapse.lib.platforms.common.getLibC() Return a ctypes reference to libc synapse.lib.platforms.common.getVolInfo(*paths) Retrieve volume usage info for the given path. synapse.lib.platforms.common.inet_ntop(afam, byts) synapse.lib.platforms.common.inet_pton(afam, text) synapse.lib.platforms.common.initHostInfo() synapse.lib.platforms.common.setProcName(name) Set the process title/name for process listing. synapse.lib.platforms.darwin module synapse.lib.platforms.darwin.initHostInfo() synapse.lib.platforms.freebsd module synapse.lib.platforms.freebsd.initHostInfo() synapse.lib.platforms.linux module synapse.lib.platforms.linux.initHostInfo() synapse.lib.platforms.windows module synapse.lib.platforms.windows.daemonize() synapse.lib.platforms.windows.getLibC() Override to account for python on windows not being able to find libc sometimes. . . synapse.lib.platforms.windows.initHostInfo() class synapse.lib.platforms.windows.sockaddr Bases: _ctypes.Structure ipv4 Structure/Union member ipv6 Structure/Union member

6.1. synapse package 231 synapse Documentation, Release 0.1.0

sa_family Structure/Union member

Submodules synapse.lib.agenda module class synapse.lib.agenda.Agenda Bases: synapse.lib.base.Base Organize and execute all the scheduled storm queries in a cortex. add(username, query: str, reqs, incunit=None, incvals=None) Persistently adds an appointment Parameters • query (str) – storm query to run • reqs (Union[None, Dict[TimeUnit, Union[int, Tuple[int]], List[..]) – one or more dicts of the fixed aspects of the appointment. dict value may be a single or multiple. May be an empty dict or None. • incunit (Union[None, TimeUnit]) – the unit that changes for recurring, or None for non-recurring. It is an error for this value to match a key in reqdict. • incvals (Union[None, int, Iterable[int]) – count of units of incunit or explicit day of week or day of month. Not allowed for incunit == None, required for others (1 would be a typical value)

Notes

For values in reqs that are lists and incvals if a list, all combinations of all values (the product) are used Returns iden of new appointment delete(iden) Delete an appointment enable() Enable cron jobs to start running, start the scheduler loop Go through all the appointments, making sure the query is valid, and remove the ones that aren’t. (We can’t evaluate queries until enabled because not all the modules are loaded yet.) list() mod(iden, query) Change the query of an appointment class synapse.lib.agenda.ApptRec(reqdict, incunit=None, incval=1) Bases: object Represents a single element of a single combination of an appointment nexttime(lastts) Returns next timestamp that meets requirements, incrementing by (self.incunit * incval) if not increasing, or 0.0 if there are no future matches

232 Chapter 6. synapse synapse Documentation, Release 0.1.0

pack() Make ApptRec json/msgpack-friendly classmethod unpack(val) Convert from json/msgpack-friendly class synapse.lib.agenda.TimeUnit Bases: enum.IntEnum Unit of time that recurring and required parts of appointments are made of DAY = 5 DAYOFMONTH = 3 DAYOFWEEK = 4 HOUR = 6 MINUTE = 7 MONTH = 2 YEAR = 1 fromString = > synapse.lib.ast module class synapse.lib.ast.AbsProp(valu, kids=()) Bases: synapse.lib.ast.Value repr() class synapse.lib.ast.AbsPropCond(kids=()) Bases: synapse.lib.ast.Cond getCondEval(runt) class synapse.lib.ast.AndCond(kids=()) Bases: synapse.lib.ast.Cond and getCondEval(runt) getLiftHints() class synapse.lib.ast.AstNode(kids=()) Bases: object Base class for all nodes in the STORM abstract syntax tree. addKid(astn) format(depth=0) getRuntVars(runt) init(core) iterright() Yield “rightward” siblings until None. optimize()

6.1. synapse package 233 synapse Documentation, Release 0.1.0

prepare() replace(astn) repr() setKid(indx, astn) sibling(offs=1) Return sibling node by relative offset from self. class synapse.lib.ast.BreakOper(kids=()) Bases: synapse.lib.ast.AstNode run(runt, genr) class synapse.lib.ast.CallArgs(kids=()) Bases: synapse.lib.ast.RunValue compute(path) runtval(runt) class synapse.lib.ast.CaseEntry(kids=()) Bases: synapse.lib.ast.AstNode class synapse.lib.ast.CmdOper(kids=()) Bases: synapse.lib.ast.Oper run(runt, genr) class synapse.lib.ast.Cmpr(valu, kids=()) Bases: synapse.lib.ast.Value repr() class synapse.lib.ast.CompValue(kids=()) Bases: synapse.lib.ast.AstNode A computed value which requires a runtime, node, and path. compute(path) isRuntSafe(runt) class synapse.lib.ast.Cond(kids=()) Bases: synapse.lib.ast.AstNode getCondEval(runt) getLiftHints() class synapse.lib.ast.Const(valu, kids=()) Bases: synapse.lib.ast.Value repr() class synapse.lib.ast.ContinueOper(kids=()) Bases: synapse.lib.ast.AstNode run(runt, genr) class synapse.lib.ast.Edit(kids=()) Bases: synapse.lib.ast.Oper class synapse.lib.ast.EditNodeAdd(kids=()) Bases: synapse.lib.ast.Edit

234 Chapter 6. synapse synapse Documentation, Release 0.1.0

run(runt, genr) class synapse.lib.ast.EditPropDel(kids=()) Bases: synapse.lib.ast.Edit run(runt, genr) class synapse.lib.ast.EditPropSet(kids=()) Bases: synapse.lib.ast.Edit run(runt, genr) class synapse.lib.ast.EditTagAdd(kids=()) Bases: synapse.lib.ast.Edit run(runt, genr) class synapse.lib.ast.EditTagDel(kids=()) Bases: synapse.lib.ast.Edit run(runt, genr) class synapse.lib.ast.EditUnivDel(kids=()) Bases: synapse.lib.ast.Edit run(runt, genr) class synapse.lib.ast.FiltOper(kids=()) Bases: synapse.lib.ast.Oper getLiftHints() run(runt, genr) class synapse.lib.ast.ForLoop(kids=()) Bases: synapse.lib.ast.Oper getRuntVars(runt) run(runt, genr) class synapse.lib.ast.FormPivot(kids=(), isjoin=False) Bases: synapse.lib.ast.PivotOper run(runt, genr) class synapse.lib.ast.FuncCall(kids=()) Bases: synapse.lib.ast.RunValue compute(path) runtval(runt) class synapse.lib.ast.HasAbsPropCond(kids=()) Bases: synapse.lib.ast.Cond getCondEval(runt) class synapse.lib.ast.HasRelPropCond(kids=()) Bases: synapse.lib.ast.Cond getCondEval(runt) class synapse.lib.ast.LiftByScrape(ndefs) Bases: synapse.lib.ast.LiftOper lift(runt)

6.1. synapse package 235 synapse Documentation, Release 0.1.0

class synapse.lib.ast.LiftFormTag(kids=()) Bases: synapse.lib.ast.LiftOper lift(runt) class synapse.lib.ast.LiftOper(kids=()) Bases: synapse.lib.ast.Oper run(runt, genr) class synapse.lib.ast.LiftProp(kids=()) Bases: synapse.lib.ast.LiftOper lift(runt) class synapse.lib.ast.LiftPropBy(kids=()) Bases: synapse.lib.ast.LiftOper lift(runt) class synapse.lib.ast.LiftTag(kids=()) Bases: synapse.lib.ast.LiftOper lift(runt) class synapse.lib.ast.LiftTagTag(kids=()) Bases: synapse.lib.ast.LiftOper ##foo.bar lift(runt) class synapse.lib.ast.List(valu, kids=()) Bases: synapse.lib.ast.Value compute(path) repr() runtval(runt) value() class synapse.lib.ast.NotCond(kids=()) Bases: synapse.lib.ast.Cond not getCondEval(runt) class synapse.lib.ast.Oper(kids=()) Bases: synapse.lib.ast.AstNode class synapse.lib.ast.OrCond(kids=()) Bases: synapse.lib.ast.Cond or getCondEval(runt) class synapse.lib.ast.PivotIn(kids=(), isjoin=False) Bases: synapse.lib.ast.PivotOper <- * run(runt, genr)

236 Chapter 6. synapse synapse Documentation, Release 0.1.0

class synapse.lib.ast.PivotInFrom(kids=(), isjoin=False) Bases: synapse.lib.ast.PivotOper run(runt, genr) class synapse.lib.ast.PivotOper(kids=(), isjoin=False) Bases: synapse.lib.ast.Oper class synapse.lib.ast.PivotOut(kids=(), isjoin=False) Bases: synapse.lib.ast.PivotOper -> * run(runt, genr) class synapse.lib.ast.PivotToTags(kids=(), isjoin=False) Bases: synapse.lib.ast.PivotOper -> # pivot to all leaf tag nodes -> #* pivot to all tag nodes -> #cno.* pivot to all tag nodes which match cno.* -> #foo.bar pivot to the tag node foo.bar if present run(runt, genr) class synapse.lib.ast.PropPivot(kids=(), isjoin=False) Bases: synapse.lib.ast.PivotOper run(runt, genr) class synapse.lib.ast.PropPivotOut(kids=(), isjoin=False) Bases: synapse.lib.ast.PivotOper run(runt, genr) class synapse.lib.ast.PropValue(kids=()) Bases: synapse.lib.ast.CompValue compute(path) getPropAndValu(path) prepare() class synapse.lib.ast.Query(kids=()) Bases: synapse.lib.ast.AstNode iterNodePaths(runt) run(runt, genr) class synapse.lib.ast.RelProp(valu, kids=()) Bases: synapse.lib.ast.Value repr() class synapse.lib.ast.RelPropCond(kids=()) Bases: synapse.lib.ast.Cond :foo:bar getCondEval(runt) class synapse.lib.ast.RelPropValue(kids=()) Bases: synapse.lib.ast.PropValue class synapse.lib.ast.RunValue(kids=()) Bases: synapse.lib.ast.CompValue

6.1. synapse package 237 synapse Documentation, Release 0.1.0

A computed value that requires a runtime. compute(path) isRuntSafe(runt) runtval(runt) exception synapse.lib.ast.StormBreak(item=None) Bases: synapse.lib.ast.StormCtrlFlow exception synapse.lib.ast.StormContinue(item=None) Bases: synapse.lib.ast.StormCtrlFlow exception synapse.lib.ast.StormCtrlFlow(item=None) Bases: Exception class synapse.lib.ast.SubGraph(rules) Bases: object An Oper like object which generates a subgraph. rules = { ‘degrees’: 1, ‘filters’: [ ‘-(#foo or #bar)’, ‘-(foo:bar or baz:faz)’, ], ‘pivots’: [ ‘-> * | limit 100’, ‘<- * | limit 100’, ] ‘forms’: { ‘inet:fqdn’:{ ‘filters’: [], ‘pivots’: [], } ‘*’: { ‘filters’: [], ‘pivots’: [], }, }, } # nodes which were original seeds have path.meta(‘graph:seed’) # all nodes have path.meta(‘edges’) which is a list of (iden, info) tuples. omit(node) pivots(node) run(runt, genr) class synapse.lib.ast.SubQuery(kids=()) Bases: synapse.lib.ast.Oper inline(runt, genr) run(runt, genr) class synapse.lib.ast.SubqCond(kids=()) Bases: synapse.lib.ast.Cond getCondEval(runt)

238 Chapter 6. synapse synapse Documentation, Release 0.1.0

class synapse.lib.ast.SwitchCase(kids=()) Bases: synapse.lib.ast.Oper prepare() run(runt, genr) class synapse.lib.ast.TagCond(kids=()) Bases: synapse.lib.ast.Cond #foo.bar getCondEval(runt) getLiftHints() class synapse.lib.ast.TagMatch(valu, kids=()) Bases: synapse.lib.ast.Value class synapse.lib.ast.TagName(valu, kids=()) Bases: synapse.lib.ast.Value class synapse.lib.ast.TagPropValue(kids=()) Bases: synapse.lib.ast.CompValue compute(path) prepare() class synapse.lib.ast.TagValuCond(kids=()) Bases: synapse.lib.ast.Cond getCondEval(runt) class synapse.lib.ast.UnivProp(valu, kids=()) Bases: synapse.lib.ast.Value repr() class synapse.lib.ast.UnivPropValue(kids=()) Bases: synapse.lib.ast.PropValue class synapse.lib.ast.Value(valu, kids=()) Bases: synapse.lib.ast.RunValue A fixed/constant value. compute(path) runtval(runt) value() class synapse.lib.ast.VarDeref(kids=()) Bases: synapse.lib.ast.RunValue compute(path) runtval(runt) class synapse.lib.ast.VarList(valu, kids=()) Bases: synapse.lib.ast.Value class synapse.lib.ast.VarListSetOper(kids=()) Bases: synapse.lib.ast.Oper getRuntVars(runt)

6.1. synapse package 239 synapse Documentation, Release 0.1.0

run(runt, genr) class synapse.lib.ast.VarSetOper(kids=()) Bases: synapse.lib.ast.Oper getRuntVars(runt) run(runt, genr) class synapse.lib.ast.VarValue(kids=()) Bases: synapse.lib.ast.RunValue compute(path) isRuntSafe(runt) prepare() runtval(runt) synapse.lib.ast.agen(*items)

synapse.lib.auth module

class synapse.lib.auth.Auth Bases: synapse.lib.cell.Cell An authentication / authorization management helper. addRole(name) Add a new role to the auth system. Parameters name (str) – The role name. Returns The newly created role. Return type Role Raises s_exc.DupRoleName – If the role already exists. addUser(name, passwd=None) Add a new user to the auth system. Parameters name (str) – The user name. Returns The newly created user. Return type User Raises s_exc.DupUserName – If the user already exists. cellapi alias of AuthApi confdefs = (('lmdb:mapsize', {'type': 'int', 'defval': 1073741824, 'doc': 'Memory map size for the auth LMDB.'}),) delRole(name) Delete a role from the auth system. Parameters name (str) – The user name to delete. Returns True if the operation succeeded. Return type True Raises s_exc.NoSuchRole – If the role does not exist.

240 Chapter 6. synapse synapse Documentation, Release 0.1.0

delUser(name) Delete a user from the auth system. Parameters name (str) – The user name to delete. Returns True if the operation succeeded. Return type True Raises s_exc.NoSuchUser – If the user did not exist. getRoles() Get a list of roles. Returns List of role names. Return type list getUsers() Get a list of user names. Returns List of user names. Return type list initCellAuth() initConfDefs() reqRole(role) Get a role object. Parameters role (str) – Name of the role object to get. Returns Role object. Return type Role Raises s_exc.NoSuchRole – If the role does not exist. reqUser(user) Get a user object. Parameters user (str) – Username to request. Returns User object. Return type User Raises s_exc.NoSuchUser – If the user does not exist. class synapse.lib.auth.AuthApi Bases: synapse.lib.cell.CellApi tryTeleAuth(auth) Return a (name, info, roles) tuple for a User or None. class synapse.lib.auth.Role(auth, name, info) Bases: object A Role within the auth system. addRule(rule, indx=None) Add an allow rule. Parameters • rule (bool, tuple) – Add an allow/deny and path tuple.

6.1. synapse package 241 synapse Documentation, Release 0.1.0

• indx (int) – The index for where to insert the rule. Returns True if the rule was added. False otherwise. Return type bool allowed(perm, elev=True) Check if the user/role is allowed the given permission. Parameters • perm ((str,)) – A permission path tuple. • elev (bool) – If true, allow admin status. Returns True if the permission is allowed. False otherwise. Return type bool delRule(indx) Remove an allow rule. Parameters indx (int) – The rule number to remove. Returns Return type True Raises s_exc.NoSuchRule – If the rule did not exist. initRuleTree() save() setAdmin(admin) Set the admin value to True/False. Parameters admin (bool) – Value to set the admin value too. Returns The current AuthBase admin value. Return type bool class synapse.lib.auth.User(auth, name, info) Bases: synapse.lib.auth.Role addRole(name) Grant a role to a user. Parameters name (str) – The name of the role to grant. Returns Return type True Raises s_exc.NoSuchRole – If the role does not exist. allowed(perm, elev=True) Check if a user is allowed the given permission. Parameters • perm (tuple) – A permission path tuple. • elev (bool) – If true, allow admin status. Returns True if the permission is allowed. False otherwise. Return type bool

242 Chapter 6. synapse synapse Documentation, Release 0.1.0

delRole(name) Revoke a role from a user. Parameters name (str) – The name of the role to revoke. Returns True if the role was removed; False if the role was not on the user. Return type bool save() setLocked(locked) setPasswd(passwd) tryPasswd(passwd) synapse.lib.auth.reqAdmin(f, attr=’auth’) A Decorator to wrap a function to require it to be executed in a admin user context. Parameters • f – Function being wrapped. • attr (str) – Name of Auth local.

Notes

This decorator should only be placed on methods on a class since it relies on having access to a local instance of a Auth object. Returns Function results. Raises • s_exc.ReqConfOpt – If the auth local is not found on the object. • s_exc.NoSuchUser – If the Auth local does not have a instance of the current user. • s_exc.AuthDeny – If the user in scope is not a admin user. synapse.lib.auth.runas(user) Construct and return a with-block object which runs as the given synapse user name.

Example

import synapse.lib.auth as s_auth s_auth.runas(‘visi@localhost’): # calls from here down may use check user/perms dostuff() synapse.lib.auth.whoami() Return the name of the current synapse user for this thread.

Example

name = s_auth.whoami()

6.1. synapse package 243 synapse Documentation, Release 0.1.0 synapse.lib.base module class synapse.lib.base.Base Bases: object Base class for Synapse objects. Acts as an observable, enables async init and fini.

Example

class Foo(Base): async def __anit__(self, x, y): await Base.__anit__(self) await stuff(x, y) foo = await Foo.anit(10)

Note: One should not create instances directly via its initializer, i.e. Base(). One shall always use the class method anit.

classmethod anit(*args, **kwargs) dist(mesg) Distribute an existing event tuple. Parameters mesg ((str,dict)) – An event tuple.

Example

await base.dist( (‘foo’,{‘bar’:’baz’}) ) fini() Shut down the object and notify any onfini() coroutines. Returns Remaining ref count fire(evtname, **info) Fire the given event name on the Base. Returns a list of the return values of each callback.

Example

for ret in d.fire(‘woot’,foo=’asdf’): print(‘got: %r’ % (ret,))

incref() Increment the reference count for this base. This API may be optionally used to control fini(). link(func) Add a callback function to receive all events.

244 Chapter 6. synapse synapse Documentation, Release 0.1.0

Example

base1 = Base() base2 = Base() base1.link( base2.dist ) # all events on base1 are also propagated on base2 main() Helper function to block until shutdown ( and handle ctrl-c and SIGTERM).

Examples

Run a base, wait until main() has returned, then do other stuff:

foo= Base() foo.main() dostuff()

Notes

This fires a ‘ebus:main’ event prior to entering the waitfini() loop. Returns None off(evnt, func) Remove a previously registered event handler function.

Example

base.off( ‘foo’, onFooFunc ) on(evnt, func, base=None) Add an base function callback for a specific event with optional filtering. If the function returns a coroutine, it will be awaited. Parameters • evnt (str) – An event name • func (function) – A callback function to receive event tufo

Examples

Add a callback function and fire it: async def baz(event): x = event[1].get(‘x’) y = event[1].get(‘y’) return x + y d.on(‘foo’, baz) # this fire triggers baz. . . await d.fire(‘foo’, x=10, y=20)

Returns Return type None

onfini(func) Add a function/coroutine/Base to be called on fini().

6.1. synapse package 245 synapse Documentation, Release 0.1.0

schedCallSafe(func, *args, **kwargs) schedCoro(coro) Schedules a free-running coroutine to run on this base’s event loop. Kills the coroutine if Base is fini’d. It does not pend on coroutine completion. Precondition: This function is not threadsafe and must be run on the Base’s event loop

Returns An asyncio.Task

schedCoroSafe(coro) Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on This function does not pend on coroutine completion.

Note: This method may be run outside the event loop on a different thread.

schedCoroSafePend(coro) Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on

Note: This method may not be run inside an event loop

unlink(func) Remove a callback function previously added with link()

Example

base.unlink( callback ) waiter(count, *names) Construct and return a new Waiter for events on this base.

Example

# wait up to 3 seconds for 10 foo:bar events. . . waiter = base.waiter(10,’foo:bar’) # .. fire thread that will cause foo:bar events events = waiter.wait(timeout=3) if events == None: # handle the timout case. . . for event in events: # parse the events if you need. . . NOTE: use with caution. . . it’s easy to accidentally construct race conditions with this mechanism ;) waitfini(timeout=None) Wait for the base to fini() Returns None if timed out, True if fini happened

246 Chapter 6. synapse synapse Documentation, Release 0.1.0

Example

base.waitfini(timeout=30) class synapse.lib.base.BaseRef Bases: synapse.lib.base.Base An object for managing multiple Base instances by name. gen(name) Atomically get/gen a Base and incref. (requires ctor during BaseRef init) Parameters name (str) – The name/iden of the Base instance. get(name) Retrieve a Base instance by name. Parameters name (str) – The name/iden of the Base Returns The Base instance (or None) Return type (Base) items() pop(name) Remove and return a Base from the BaseRef. Parameters name (str) – The name/iden of the Base instance Returns The named base ( or None ) Return type (Base) put(name, base) Add a Base (or sub-class) to the BaseRef by name. Parameters • name (str) – The name/iden of the Base • base (Base) – The Base instance Returns (None) vals() class synapse.lib.base.Waiter(base, count, *names) Bases: object A helper to wait for a given number of events on a Base. fini() wait(timeout=None) Wait for the required number of events and return them or None on timeout.

Example

evnts = waiter.wait(timeout=30) if evnts == None: handleTimedOut() return for evnt in evnts: doStuff(evnt)

6.1. synapse package 247 synapse Documentation, Release 0.1.0 synapse.lib.boss module class synapse.lib.boss.Boss Bases: synapse.lib.base.Base An object to track “promoted” async tasks. execute(coro, name, user, info=None) Create a synapse task from the given coroutine. get(iden) promote(name, user, info=None) Promote the currently running task. ps() synapse.lib.cache module

A few speed optimized (lockless) cache helpers. Use carefully. class synapse.lib.cache.FixedCache(callback, size=10000) Bases: object aget(key) clear() get(key) pop(key) put(key, val) class synapse.lib.cache.TagGlobs Bases: object An object that manages multiple tag globs and values for caching. add(name, valu, base=None) get(name) rem(name, valu) synapse.lib.cache.getTagGlobRegx(*args) synapse.lib.cache.memoize(size=10000) synapse.lib.cell module class synapse.lib.cell.Cell Bases: synapse.lib.base.Base, synapse.telepath.Aware A Cell() implements a synapse micro-service. addCellCmd(name, func) Add a Cmdr() command to the cell. cellapi alias of CellApi confbase = ()

248 Chapter 6. synapse synapse Documentation, Release 0.1.0

confdefs = () classmethod deploy(dirn) getCellIden() getCellType() getLocalProxy() Creates a local telepath daemon, shares this object, and returns the telepath proxy of this object TODO: currently, this will fini self if the created dmon is fini’d getTeleApi(link, mesg) Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict) initCellAuth() class synapse.lib.cell.CellApi Bases: synapse.lib.base.Base addAuthRole(**kwargs) addAuthRule(**kwargs) addAuthUser(**kwargs) addUserRole(**kwargs) delAuthRule(**kwargs) delUserRole(**kwargs) getAuthInfo(**kwargs) getAuthRoles(**kwargs) getAuthUsers(**kwargs) getCellIden() getCellType() getHiveKey(**kwargs) kill(iden) listHiveKey(**kwargs) popHiveKey(**kwargs) ps() setAuthAdmin(**kwargs) setHiveKey(**kwargs) setUserLocked(**kwargs) setUserPasswd(**kwargs) class synapse.lib.cell.PassThroughApi Bases: synapse.lib.cell.CellApi Class that passes through methods made on it to its cell. allowed_methods = []

6.1. synapse package 249 synapse Documentation, Release 0.1.0 synapse.lib.cell.SLAB_MAP_SIZE = 134217728 Base classes for the synapse “cell” microservice architecture. synapse.lib.cell.adminapi(f ) synapse.lib.certdir module class synapse.lib.certdir.CertDir(path=None) Bases: object Certificate loading/generation/signing utilities. Features: • Locates and load certificates, keys, and certificate signing requests (CSRs). • Generates keypairs for users, hosts, and certificate authorities (CAs), supports both signed and self- signed. • Generates certificate signing requests (CSRs) for users, hosts, and certificate authorities (CAs). • Signs certificate signing requests (CSRs). • Generates PKCS#12 archives for use in browser.

Parameters path (str) – Optional path which can override the default path directory.

Notes

• All certificates will be loaded from and written to ~/.syn/certs by default. Set the envvar SYN_CERT_DIR to

override. * All certificate generation methods create 4096 bit RSA keypairs. * All certificate signing methods use sha256 as the signature algorithm. * CertDir does not currently support signing CA CSRs. genCaCert(name, signas=None, outp=None, save=True) Generates a CA keypair. Parameters • name (str) – The name of the CA keypair. • signas (str) – The CA keypair to sign the new CA with. • outp (synapse.lib.output.Output) – The output buffer.

Examples

Make a CA named “myca”: mycakey, mycacert = cdir.genCaCert(‘myca’)

Returns Tuple containing the private key and certificate objects. Return type ((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genClientCert(name, outp=None) Generates a user PKCS #12 archive. Please note that the resulting file will contain private key material. Parameters

250 Chapter 6. synapse synapse Documentation, Release 0.1.0

• name (str) – The name of the user keypair. • outp (synapse.lib.output.Output) – The output buffer.

Examples

Make the PKC12 object for user “myuser”: myuserpkcs12 = cdir.genClientCert(‘myuser’)

Returns The PKCS #12 archive. Return type OpenSSL.crypto.PKCS12

genHostCert(name, signas=None, outp=None, csr=None, sans=None) Generates a host keypair. Parameters • name (str) – The name of the host keypair. • signas (str) – The CA keypair to sign the new host keypair with. • outp (synapse.lib.output.Output) – The output buffer. • csr (OpenSSL.crypto.PKey) – The CSR public key when generating the keypair from a CSR. • sans (list) – List of subject alternative names.

Examples

Make a host keypair named “myhost”: myhostkey, myhostcert = cdir.genHostCert(‘myhost’)

Returns Tuple containing the private key and certificate objects. Return type ((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genHostCsr(name, outp=None) Generates a host certificate signing request. Parameters • name (str) – The name of the host CSR. • outp (synapse.lib.output.Output) – The output buffer.

Examples

Generate a CSR for the host key named “myhost”: cdir.genHostCsr(‘myhost’)

Returns None

genUserCert(name, signas=None, outp=None, csr=None) Generates a user keypair.

6.1. synapse package 251 synapse Documentation, Release 0.1.0

Parameters • name (str) – The name of the user keypair. • signas (str) – The CA keypair to sign the new user keypair with. • outp (synapse.lib.output.Output) – The output buffer. • csr (OpenSSL.crypto.PKey) – The CSR public key when generating the keypair from a CSR.

Examples

Generate a user cert for the user “myuser”: myuserkey, myusercert = cdir.genUserCert(‘myuser’)

Returns Tuple containing the key and certificate objects. Return type ((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genUserCsr(name, outp=None) Generates a user certificate signing request. Parameters • name (str) – The name of the user CSR. • outp (synapse.lib.output.Output) – The output buffer.

Examples

Generate a CSR for the user “myuser”: cdir.genUserCsr(‘myuser’)

Returns None

getCaCert(name) Loads the X509 object for a given CA. Parameters name (str) – The name of the CA keypair.

Examples

Get the certificate for the CA “myca” mycacert = cdir.getCaCert(‘myca’)

Returns The certificate, if exists. Return type OpenSSL.crypto.X509

getCaCertPath(name) Gets the path to a CA certificate. Parameters name (str) – The name of the CA keypair.

252 Chapter 6. synapse synapse Documentation, Release 0.1.0

Examples

Get the path to the CA certificate for the CA “myca”: mypath = cdir.getCACertPath(‘myca’)

Returns The path if exists. Return type str

getCaCerts() Return a list of CA certs from the CertDir. Returns List of CA certificates. Return type [OpenSSL.crypto.X509] getCaKey(name) Loads the PKey object for a given CA keypair. Parameters name (str) – The name of the CA keypair.

Examples

Get the private key for the CA “myca”: mycakey = cdir.getCaKey(‘myca’)

Returns The private key, if exists. Return type OpenSSL.crypto.PKey

getCaKeyPath(name) Gets the path to a CA key. Parameters name (str) – The name of the CA keypair.

Examples

Get the path to the private key for the CA “myca”: mypath = cdir.getCAKeyPath(‘myca’)

Returns The path if exists. Return type str

getClientCert(name) Loads the PKCS12 archive object for a given user keypair. Parameters name (str) – The name of the user keypair.

Examples

Get the PKCS12 object for the user “myuser”: mypkcs12 = cdir.getClientCert(‘myuser’)

6.1. synapse package 253 synapse Documentation, Release 0.1.0

Notes

The PKCS12 archive will contain private key material if it was created with CertDir or the easycert tool Returns The PKCS12 archive, if exists. Return type OpenSSL.crypto.PKCS12 getClientCertPath(name) Gets the path to a client certificate. Parameters name (str) – The name of the client keypair.

Examples

Get the path to the client certificate for “myuser”: mypath = cdir.getClientCertPath(‘myuser’)

Returns The path if exists. Return type str

getClientSSLContext() Returns an ssl.SSLContext appropriate for initiating a TLS session getHostCaPath(name) Gets the path to the CA certificate that issued a given host keypair. Parameters name (str) – The name of the host keypair.

Examples

Get the path to the CA cert which issue the cert for “myhost”: mypath = cdir.getHostCaPath(‘myhost’)

Returns The path if exists. Return type str

getHostCert(name) Loads the X509 object for a given host keypair. Parameters name (str) – The name of the host keypair.

Examples

Get the certificate object for the host “myhost”: myhostcert = cdir.getHostCert(‘myhost’)

Returns The certificate, if exists. Return type OpenSSL.crypto.X509

getHostCertPath(name) Gets the path to a host certificate.

254 Chapter 6. synapse synapse Documentation, Release 0.1.0

Parameters name (str) – The name of the host keypair.

Examples

Get the path to the host certificate for the host “myhost”: mypath = cdir.getHostCertPath(‘myhost’)

Returns The path if exists. Return type str

getHostKey(name) Loads the PKey object for a given host keypair. Parameters name (str) – The name of the host keypair.

Examples

Get the private key object for the host “myhost”: myhostkey = cdir.getHostKey(‘myhost’)

Returns The private key, if exists. Return type OpenSSL.crypto.PKey

getHostKeyPath(name) Gets the path to a host key. Parameters name (str) – The name of the host keypair.

Examples

Get the path to the host key for the host “myhost”: mypath = cdir.getHostKeyPath(‘myhost’)

Returns The path if exists. Return type str

getServerSSLContext(hostname=None) Returns an ssl.SSLContext appropriate to listen on a socket Parameters • hostname – if None, the value from socket.gethostname is used to find the key in the servers directory. • name should match the not-suffixed part of two files ending in .key and .crt in the hosts subdirectory (This)– getUserCaPath(name) Gets the path to the CA certificate that issued a given user keypair. Parameters name (str) – The name of the user keypair.

6.1. synapse package 255 synapse Documentation, Release 0.1.0

Examples

Get the path to the CA cert which issue the cert for “myuser”: mypath = cdir.getUserCaPath(‘myuser’)

Returns The path if exists. Return type str

getUserCert(name) Loads the X509 object for a given user keypair. Parameters name (str) – The name of the user keypair.

Examples

Get the certificate object for the user “myuser”: myusercert = cdir.getUserCert(‘myuser’)

Returns The certificate, if exists. Return type OpenSSL.crypto.X509

getUserCertPath(name) Gets the path to a user certificate. Parameters name (str) – The name of the user keypair.

Examples

Get the path for the user cert for “myuser”: mypath = cdir.getUserCertPath(‘myuser’)

Returns The path if exists. Return type str

getUserForHost(user, host) Gets the name of the first existing user cert for a given user and host. Parameters • user (str) – The name of the user. • host (str) – The name of the host.

Examples

Get the name for the “myuser” user cert at “cool.vertex.link”: usercertname = cdir.getUserForHost(‘myuser’, ‘cool.vertex.link’)

Returns The cert name, if exists. Return type str

256 Chapter 6. synapse synapse Documentation, Release 0.1.0

getUserKey(name) Loads the PKey object for a given user keypair. Parameters name (str) – The name of the user keypair.

Examples

Get the key object for the user key for “myuser”: myuserkey = cdir.getUserKey(‘myuser’)

Returns The private key, if exists. Return type OpenSSL.crypto.PKey

getUserKeyPath(name) Gets the path to a user key. Parameters name (str) – The name of the user keypair.

Examples

Get the path to the user key for “myuser”: mypath = cdir.getUserKeyPath(‘myuser’)

Returns The path if exists. Return type str

importFile(path, mode, outp=None) Imports certs and keys into the Synapse cert directory Parameters • path (str) – The path of the file to be imported. • mode (str) – The certdir subdirectory to import the file into.

Examples

Import CA certifciate ‘mycoolca.crt’ to the ‘cas’ directory. certdir.importFile(‘mycoolca.crt’, ‘cas’)

Notes

importFile does not perform any validation on the files it imports. Returns None isCaCert(name) Checks if a CA certificate exists. Parameters name (str) – The name of the CA keypair.

6.1. synapse package 257 synapse Documentation, Release 0.1.0

Examples

Check if the CA certificate for “myca” exists: exists = cdir.isCaCert(‘myca’)

Returns True if the certificate is present, False otherwise. Return type bool

isClientCert(name) Checks if a user client certificate (PKCS12) exists. Parameters name (str) – The name of the user keypair.

Examples

Check if the client certificate “myuser” exists: exists = cdir.isClientCert(‘myuser’)

Returns True if the certificate is present, False otherwise. Return type bool

isHostCert(name) Checks if a host certificate exists. Parameters name (str) – The name of the host keypair.

Examples

Check if the host cert “myhost” exists: exists = cdir.isUserCert(‘myhost’)

Returns True if the certificate is present, False otherwise. Return type bool

isUserCert(name) Checks if a user certificate exists. Parameters name (str) – The name of the user keypair.

Examples

Check if the user cert “myuser” exists: exists = cdir.isUserCert(‘myuser’)

Returns True if the certificate is present, False otherwise. Return type bool

selfSignCert(cert, pkey) Self-sign a certificate.

258 Chapter 6. synapse synapse Documentation, Release 0.1.0

Parameters • cert (OpenSSL.crypto.X509) – The certificate to sign. • pkey (OpenSSL.crypto.PKey) – The PKey with which to sign the certificate.

Examples

Sign a given certificate with a given private key: cdir.selfSignCert(mycert, myotherprivatekey)

Returns None

signCertAs(cert, signas) Signs a certificate with a CA keypair. Parameters • cert (OpenSSL.crypto.X509) – The certificate to sign. • signas (str) – The CA keypair name to sign the new keypair with.

Examples

Sign a certificate with the CA “myca”: cdir.signCertAs(mycert, ‘myca’)

Returns None

signHostCsr(xcsr, signas, outp=None, sans=None) Signs a host CSR with a CA keypair. Parameters • cert (OpenSSL.crypto.X509Req) – The certificate signing request. • signas (str) – The CA keypair name to sign the CSR with. • outp (synapse.lib.output.Output) – The output buffer. • sans (list) – List of subject alternative names.

Examples

Sign a host key with the CA “myca”: cdir.signHostCsr(mycsr, ‘myca’)

Returns Tuple containing the public key and certificate objects. Return type ((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

signUserCsr(xcsr, signas, outp=None) Signs a user CSR with a CA keypair. Parameters • cert (OpenSSL.crypto.X509Req) – The certificate signing request.

6.1. synapse package 259 synapse Documentation, Release 0.1.0

• signas (str) – The CA keypair name to sign the CSR with. • outp (synapse.lib.output.Output) – The output buffer.

Examples

cdir.signUserCsr(mycsr, ‘myca’) Returns Tuple containing the public key and certificate objects. Return type ((OpenSSL.crypto.PKey, OpenSSL.crypto.X509)) valUserCert(byts, cacerts=None) Validate the PEM encoded x509 user certificate bytes and return it. Parameters • byts (bytes) – The bytes for the User Certificate. • cacerts (tuple) – A tuple of OpenSSL.crypto.X509 CA Certificates. Raises OpenSSL.crypto.X509StoreContextError – If the certificate is not valid. Returns The certificate, if it is valid. Return type OpenSSL.crypto.X509 synapse.lib.certdir.iterFqdnUp(fqdn) synapse.lib.chop module synapse.lib.chop.digits(text) synapse.lib.chop.hexstr(text) Ensure a string is valid hex. Parameters text (str) – String to normalize.

Examples

Norm a few strings: hexstr(‘0xff00’) hexstr(‘ff00’)

Notes

Will accept strings prefixed by ‘0x’ or ‘0X’ and remove them. Returns Normalized hex string. Return type str synapse.lib.chop.intrange(text) synapse.lib.chop.intstr(text) synapse.lib.chop.mergeRanges(x, y) Merge two ranges into one. synapse.lib.chop.onespace(text)

260 Chapter 6. synapse synapse Documentation, Release 0.1.0 synapse.lib.chop.printables(text) synapse.lib.cli module class synapse.lib.cli.Cli(item, outp=None, **locs) Bases: synapse.eventbus.EventBus A modular / event-driven CLI base object. addCmdClass(ctor, **opts) Add a Cmd subclass to this cli. get(name, defval=None) getCmdByName(name) Return a Cmd instance by name. getCmdNames() Return a list of all the known command names for the CLI. getCmdPrompt() Get the command prompt. Returns Configured command prompt Return type str get_input(prompt=None) Get the input string to parse. Parameters prompt (str) – Optional string to use as the prompt. Otherwise self.cmdprompt is used. Returns A string to process. Return type str printf(mesg, addnl=True) reflectItem() runCmdLine(line) Run a single command line. Parameters line (str) – Line to execute.

Examples

Execute the ‘woot’ command with the ‘help’ switch: await cli.runCmdLine(‘woot –help’)

Returns Arbitrary data from the cmd class. Return type object

runCmdLoop() Run commands from a user in an interactive fashion until fini() or EOFError is raised. set(name, valu)

6.1. synapse package 261 synapse Documentation, Release 0.1.0 class synapse.lib.cli.Cmd(cli, **opts) Bases: object Base class for modular commands in the synapse CLI. FIXME: document the _cmd_syntax definitions. getCmdBrief() Return the single-line description for this command. getCmdDoc() Return the help/doc output for this command. getCmdItem() Get a reference to the object we are commanding. getCmdName() getCmdOpts(text) Use the _cmd_syntax def to split/parse/normalize the cmd line. Parameters text (str) – Command to process.

Notes

This is implemented independent of argparse (et al) due to the need for syntax aware argument splitting. Also, allows different split per command type Returns An opts dictionary. Return type dict printf(mesg, addnl=True) runCmdLine(line) Run a line of command input for this command. Parameters line (str) – Line to execute

Examples

Run the foo command with some arguments: await foo.runCmdLine(‘foo –opt baz woot.com’) runCmdOpts(opts) Perform the command actions. Must be implemented by Cmd implementers. Parameters opts (dict) – Options dictionary. class synapse.lib.cli.CmdHelp(cli, **opts) Bases: synapse.lib.cli.Cmd List commands and display help output.

Example

help foocmd runCmdOpts(opts) Perform the command actions. Must be implemented by Cmd implementers.

262 Chapter 6. synapse synapse Documentation, Release 0.1.0

Parameters opts (dict) – Options dictionary. class synapse.lib.cli.CmdLocals(cli, **opts) Bases: synapse.lib.cli.Cmd List the current locals for a given CLI object. runCmdOpts(opts) Perform the command actions. Must be implemented by Cmd implementers. Parameters opts (dict) – Options dictionary. class synapse.lib.cli.CmdQuit(cli, **opts) Bases: synapse.lib.cli.Cmd Quit the current command line interpreter.

Example

quit runCmdOpts(opts) Perform the command actions. Must be implemented by Cmd implementers. Parameters opts (dict) – Options dictionary. synapse.lib.cli.get_input(text) Get input from a user via stdin.

Notes

This is just a wrapper for input() so mocking does not have to replace builtin functions for testing runCmdLoop. Parameters text (str) – Text displayed prior to the input prompt. Returns String of text from the user. Return type str synapse.lib.cmd module class synapse.lib.cmd.Parser(prog=None, outp=, **kwargs) Bases: argparse.ArgumentParser exit(status=0, message=None) Argparse expects exit() to be a terminal function and not return. As such, this function must raise an exception instead. synapse.lib.cmdr module synapse.lib.cmdr.getItemCmdr(cell, outp=None, **opts) Construct and return a cmdr for the given remote cell.

6.1. synapse package 263 synapse Documentation, Release 0.1.0

Example

cmdr = await getItemCmdr(foo) synapse.lib.cmdr.runItemCmdr(item, outp=None, **opts) Create a cmdr for the given item and run the cmd loop.

Example

runItemCmdr(foo) synapse.lib.const module synapse.lib.coro module

Async/Coroutine related utilities. class synapse.lib.coro.AsyncToSyncCMgr(func, *args, **kwargs) Bases: object Wraps an async context manager as a sync one class synapse.lib.coro.GenrHelp(genr) Bases: object list() spin() synapse.lib.coro.event_wait(event: asyncio.locks.Event, timeout=None) Wait on an an asyncio event with an optional timeout Returns true if the event got set, None if timed out synapse.lib.coro.executor(func, *args, **kwargs) synapse.lib.coro.genr2agenr(func, *args, qsize=100, **kwargs) Returns an async generator that receives a stream of messages from a sync generator func(*args, **kwargs) synapse.lib.coro.genrhelp(f ) synapse.lib.coro.iscoro(item) synapse.lib.coro.ornot(func, *args, **kwargs) Calls func and awaits it if a returns a coroutine.

Note: This is useful for implementing a function that might take a telepath proxy object or a local object, and you must call a non-async method on that object. This is also useful when calling a callback that might either be a coroutine function or a regular function.

Usage: ok = await s_coro.ornot(maybeproxy.allowed, ‘path’)

264 Chapter 6. synapse synapse Documentation, Release 0.1.0 synapse.lib.datapath module class synapse.lib.datapath.DataElem(item, name=None, parent=None) Bases: object iter(path) Iterate sub elements using the given path. Parameters path (Union[str, Datapath]) – the datapath string or a pre-computed one

Example

data = { ‘foo’:[ {‘bar’:’lol’}, {‘bar’:’heh’} ] } root = s_datapath.initelem(data) for elem in root.iter(‘foo/*/bar’): dostuff(elem) # elem is at value “lol” and “heh” name() step(path) Step to the given DataElem within the tree. vals(path) Iterate the given path elements and yield values. Parameters path (Union[str, Datapath]) – the datapath string or a pre-computed one

Example

data = { ‘foo’:[ {‘bar’:’lol’}, {‘bar’:’heh’} ] } root = s_datapath.initelem(data) for elem in root.iter(‘foo/*/bar’): dostuff(elem) # elem is at value “lol” and “heh” valu(path) Return the value of the element at the given path. class synapse.lib.datapath.DataPath(path) Bases: object A pre-computation of a datapath. class synapse.lib.datapath.XmlDataElem(item, name=None, parent=None) Bases: synapse.lib.datapath.DataElem synapse.lib.datapath.initelem(item, name=None, parent=None) Construct a new DataElem from the given item using which ever DataElem class is most correct for the type.

Example

elem = initelem(

6.1. synapse package 265 synapse Documentation, Release 0.1.0

synapse.lib.datfile module

Utilities for handling data files embedded within python packages. synapse.lib.datfile.openDatFile(datpath) Open a file-like object using a pkg relative path.

Example

fd = openDatFile(‘foopkg.barpkg/wootwoot.bin’)

synapse.lib.db module

class synapse.lib.db.Pool(size, ctor) Bases: synapse.eventbus.EventBus The Pool allows generic db connection pooling using a factory/ctor method and a python queue. avail() xact() class synapse.lib.db.Xact(pool, db) Bases: synapse.eventbus.EventBus A unified helper for managing db transaction behavior. commit() Perform an incremental commit(). See DB API execute(*args) Execute a query on the cursor for this xact. NOTE: This will acquire the write-lock for the db Pool() until your transaction is complete. Use se- lect() for read queries.

Parameters *args (list) – The args to hand to cursor execute(). Returns see DB API Return type Cursor

executemany(*args) select(*args) Read-only optimized cursor execute method. Parameters *args (list) – The args to hand to cursor execute(). Returns see DB API Return type Cursor update(*args) wrlock() Acquire the pool write lock for the remainder of this transaction.

266 Chapter 6. synapse synapse Documentation, Release 0.1.0 synapse.lib.editatom module class synapse.lib.editatom.EditAtom(allbldgbuids) Bases: object A simple utility class to track all the changes for adding a node or setting a property before committing them all at once. addNode(node) Update the shared map with my in-construction node commit(snap) Push the recorded changes to disk, notify all the listeners getNodeBeingMade(buid) Return a node if it is currently being made, mark as a dependency, else None if none found rendevous() Wait until all my adjacent editatoms are also at this point synapse.lib.encoding module synapse.lib.encoding.addFormat(name, fn, opts) Add an additional ingest file format synapse.lib.encoding.decode(name, byts, **opts) Decode the given byts with the named decoder. If name is a comma separated list of decoders, loop through and do them all.

Example

byts = s_encoding.decode(‘base64’,byts) Note: Decoder names may also be prefixed with + to encode for that name/layer. synapse.lib.encoding.encode(name, item, **opts) synapse.lib.encoding.iterdata(fd, close_fd=True, **opts) Iterate through the data provided by a file like object. Optional parameters may be used to control how the data is deserialized.

Examples

The following example show use of the iterdata function.:

with open('foo.csv','rb') as fd: for row in iterdata(fd, format='csv', encoding='utf8'): dostuff(row)

Parameters • fd (file) – File like object to iterate over. • close_fd (bool) – Default behavior is to close the fd object. If this is not true, the fd will not be closed.

6.1. synapse package 267 synapse Documentation, Release 0.1.0

• **opts (dict) – Ingest open directive. Causes the data in the fd to be parsed according to the ‘format’ key and any additional arguments. Yields An item to process. The type of the item is dependent on the format parameters.

synapse.lib.filepath module

class synapse.lib.filepath.FpFile(pparts, idx, parent=None, fd=None) Bases: object close() Closes the file-like object if it has one isfile() Returns a boolean. If it returns False, it may be assumed to be a directory next() This is the workhorse method that can contain path specific processing of children. The object should consume as much as possible of the path before creating the child class NOTE: Override for container formats nexts() open(mode=’rb’) Returns a file-like object for this path This should return None if it doesn’t make sense to open, i.e. a directory path() class synapse.lib.filepath.FpOpener(fpobj) Bases: object close() read(*args) seek(*args) class synapse.lib.filepath.FpTar(pparts, idx, parent=None, fd=None) Bases: synapse.lib.filepath.FpFile close() Closes the file-like object if it has one innerLs(path) innrEnum() enumerate the files and directory paths in the container exactly once! creates a nested dict and a set of files and dirs. . . memory inefficient FIXME: consider abandoning sets innrExists(path) innrIsdir(path) innrIsfile(path) innrOpen(*parts) innrTmpExtract(path) Extract a file from within the container to a named temporary file

268 Chapter 6. synapse synapse Documentation, Release 0.1.0

next() This is the workhorse method for path specific processing of container children. The object should con- sume as much as possible of the path before instantiating a new object At a minimum, each container should override: innrOpen(path) innrEnum(path) nexts() open(mode=’rb’) Returns a file-like object for the path inside the container This should return None if it doesn’t make sense to open, i.e. a directory or if the container doesn’t contain the end of the path path() class synapse.lib.filepath.FpZip(pparts, idx, parent=None, fd=None) Bases: synapse.lib.filepath.FpTar innrEnum() enumerate the files and directory paths in the container exactly once! innrOpen(*parts) synapse.lib.filepath.exists(*paths) Determines if the path exists even if the path terminates inside a container file. If a list of paths are provided, they are joined first. Returns a boolean. synapse.lib.filepath.getPathParts(path) Returns the elements of a path in order, w/o regard to their original separators synapse.lib.filepath.isdir(*paths) Determines if the path is a directory, even if the path terminates inside a container file. If a list of paths are provided, they are joined first. Returns a boolean. synapse.lib.filepath.isfile(*paths) Determines if the path is a file, even if the path terminates inside a container file. If a list of paths are provided, they are joined first. Returns a boolean. synapse.lib.filepath.max_temp_sz = 104857600 Provide a generic opener API for paths that cross into supported container files. For example: /dir0/dir1/foo.zip/d0/bar The above represents a file named bar located in the d0 directory inside the foo.zip zip archive located on the filesystem in the /dir0/dir1 directory. synapse.lib.filepath.normpath(*paths) Normalizes a path: 1. uses forward-slashes 2. removes leading slashes 3. removes trailing slashes This is useful for container path enumeration synapse.lib.filepath.openfile(*paths, **kwargs) Returns a read-only file-like object even if the path terminates inside a container file. If the path is a regular os accessible path mode may be passed through as a keyword argument. If the path terminates in a container file, mode is ignored. If req=True (Default) NoSuchPath will also be raised if the path exists, but is a directory

Example

fd = openfile(‘/foo/bar/baz.egg/path/inside/zip/to/file’) if fd == None: return fbuf = fd.read() fd.close()

6.1. synapse package 269 synapse Documentation, Release 0.1.0 synapse.lib.filepath.openfiles(*paths, **kwargs) Yields a read-only file-like object for each path even if the path terminates inside a container file. Paths may use python’s fnmatch glob matching If the path is a regular os accessible path mode may be passed through as a keyword argument. If the path terminates in a container file, mode is ignored. If req=True (Default) NoSuchPath will also be raised if ANY matching path exists, but is a directory

Example

for fd in openfiles(‘/foo/bar/.egg/dir0/zz/nest.zip’): fbuf = fd.read() synapse.lib.filepath.parsePath(*paths) function to parse the incoming path. lists of paths are joined prior to parsing synapse.lib.filepath.parsePaths(*paths) function to parse the incoming path. lists of paths are joined prior to parsing The path supports python’s fnmatch glob matching synapse.lib.filepath.subpaths(path) Returns a list of subpaths in a path, one for each level in the path This is an internal function used for ONLY for iterating over paths in a container As such it should not be used for filesystem paths since separators will vary across platforms synapse.lib.gis module synapse.lib.gis.bbox(lat, lon, dist) Calculate a min/max bounding box for the circle defined by lalo/dist. Parameters • lat (float) – The latitude in degrees • lon (float) – The longitude in degrees • dist (int) – A distance in geo:dist base units (mm) Returns (latmin, latmax, lonmin, lonmax) Return type (float,float,float,float) synapse.lib.gis.dms2dec(degs, mins, secs) Convert degrees, minutes, seconds lat/long form to degrees float. Parameters • degs (int) – Degrees • mins (int) – Minutes • secs (int) – Seconds Returns Degrees Return type (float) synapse.lib.gis.haversine(px, py, r=6371008800.0) Calculate the haversine distance between two points defined by (lat,lon) tuples. Parameters

270 Chapter 6. synapse synapse Documentation, Release 0.1.0

• px ((float,float)) – lat/long position 1 • py ((float,float)) – lat/long position 2 • r (float) – Radius of sphere Returns Distance in mm. Return type (int) synapse.lib.gis.latlong(text) Chop a latlong string and return (float,float). Does not perform validation on the coordinates. Parameters text (str) – A longitude,latitude string. Returns A longitude, latitude float tuple. Return type (float,float) synapse.lib.gis.near(point, dist, points) Determine if the given point is within dist of any of points. Parameters • point ((float,float)) – A latitude, longitude float tuple. • dist (int) – A distance in mm ( base units ) • points (list) – A list of latitude, longitude float tuples to compare against. synapse.lib.hashitem module synapse.lib.hashitem.hashitem(item) Generate a uniq hash for the JSON compatible primitive data structure. synapse.lib.hashitem.normdict(item) synapse.lib.hashitem.normitem(item) synapse.lib.hashitem.normiter(item) synapse.lib.hashset module class synapse.lib.hashset.HashSet Bases: object digests() Get a list of (name, bytes) tuples for the hashes in the hashset. eatfd(fd) Consume all the bytes from a file like object.

Example

hset = HashSet() hset.eatfd(fd) guid() Use elements from this hash set to create a unique (re)identifier. update(byts) Update all the hashes in the set with the given bytes.

6.1. synapse package 271 synapse Documentation, Release 0.1.0

synapse.lib.hive module

class synapse.lib.hive.Hive Bases: synapse.lib.base.Base, synapse.telepath.Aware An optionally persistent atomically accessed tree which implements primitives for use in making dis- tributed/clustered services. add(full, valu) Atomically increments a node’s value. dict(full) Open a HiveDict at the given full path. dir(full) get(full) getHiveAuth() getTeleApi(link, mesg) Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict) onedit(path, func, base=None) open(full) Open and return a hive Node(). pop(full) Remove and return the value for the given node. set(full, valu) A set operation at the hive level (full path). storNodeValu(full, valu) class synapse.lib.hive.HiveApi Bases: synapse.lib.base.Base addAndSync(path, valu, iden) edits() get(full) setAndSync(path, valu, iden) treeAndSync(path, iden) class synapse.lib.hive.HiveAuth Bases: synapse.lib.base.Base addRole(name) addUser(name) getRoleByName(name) getUserByName(name) role(iden) user(iden) class synapse.lib.hive.HiveDict Bases: synapse.lib.base.Base

272 Chapter 6. synapse synapse Documentation, Release 0.1.0

get(name, onedit=None) items() pop(name) set(name, valu) setdefault(name, valu) class synapse.lib.hive.HiveIden Bases: synapse.lib.base.Base addRule(rule, indx=None) delRule(rule) class synapse.lib.hive.HiveRole Bases: synapse.lib.hive.HiveIden A role within the Hive authorization subsystem. The HiveRole mainly exists to contain rules. class synapse.lib.hive.HiveUser Bases: synapse.lib.hive.HiveIden allowed(perm, elev=True) grant(name, indx=None) revoke(name) setAdmin(admin) setLocked(locked) setPasswd(passwd) tryPasswd(passwd) class synapse.lib.hive.Node Bases: synapse.lib.base.Base A single node within the Hive tree. add(valu) Increments existing node valu dict() dir() get(name) name() open(*path) parent() set(valu) class synapse.lib.hive.SlabHive Bases: synapse.lib.hive.Hive storNodeDele(full) storNodeValu(full, valu)

6.1. synapse package 273 synapse Documentation, Release 0.1.0

class synapse.lib.hive.TeleHive Bases: synapse.lib.hive.Hive add(path, valu) Atomically increments a node’s value. get(path) open(path) Open and return a hive Node(). set(path, valu) A set operation at the hive level (full path). synapse.lib.hive.iterpath(path) synapse.lib.hive.opendir(dirn, conf=None) synapse.lib.hive.openurl(url, **opts)

synapse.lib.ingest module

synapse.lib.interval module

A few utilities for dealing with intervals. synapse.lib.interval.fold(*vals) Initialize a new (min,max) tuple interval from values. Parameters *vals ([int,..]) – A list of values (or Nones) Returns A (min,max) interval tuple or None Return type ((int,int)) synapse.lib.interval.overlap(ival0, ival1) Determine if two interval tuples have overlap. Parameters • iv0 ((int,int)) – An interval tuple • iv1 ((int,int))– Returns True if the intervals overlap, otherwise False Return type (bool) synapse.lib.interval.parsetime(text) Parse an interval time string and return a (min,max) tuple. Parameters text (str) – A time interval string Returns A epoch millis epoch time string Return type ((int,int))

synapse.lib.jupyter module

class synapse.lib.jupyter.CmdrCore Bases: synapse.lib.base.Base A helper for jupyter/storm CLI interaction

274 Chapter 6. synapse synapse Documentation, Release 0.1.0

addFeedData(name, items, seqn=None) Add feed data to the cortex. eval(text, opts=None, num=None, cmdr=False) A helper for executing a storm command and getting a list of packed nodes. Parameters • text (str) – Storm command to execute. • opts (dict) – Opt to pass to the cortex during execution. • num (int) – Number of nodes to expect in the output query. Checks that with an assert statement. • cmdr (bool) – If True, executes the line via the Cmdr CLI and will send output to outp.

Notes

The opts dictionary will not be used if cmdr=True. Returns A list of packed nodes. Return type list runCmdLine(text) Run a line of text directly via cmdr. storm(text, opts=None, num=None, cmdr=False) A helper for executing a storm command and getting a list of storm messages. Parameters • text (str) – Storm command to execute. • opts (dict) – Opt to pass to the cortex during execution. • num (int) – Number of nodes to expect in the output query. Checks that with an assert statement. • cmdr (bool) – If True, executes the line via the Cmdr CLI and will send output to outp.

Notes

The opts dictionary will not be used if cmdr=True. Returns A list of storm messages. Return type list synapse.lib.jupyter.genTempCoreProxy(mods=None) Get a temporary cortex proxy. synapse.lib.jupyter.getDocData(fp, root=None) Parameters • fn (str) – Name of the file to retrieve the data of. • root (str) – Optional root path to look for a docdata directory in.

6.1. synapse package 275 synapse Documentation, Release 0.1.0

Notes

Will detect json/jsonl/yaml/mpk extensions and automatically decode that data if found; otherwise it returns bytes. Defaults to looking for the docdata directory in the current working directory. This behavior works fine for notebooks nested in the docs directory of synapse; but this root directory that is looked for may be overridden by providing an alternative root. Returns May be deserialized data or bytes. Return type data Raises ValueError if the file does not exist or directory traversal attempted.. synapse.lib.jupyter.getDocPath(fn, root=None) Helper for getting a documentation data file paths. Parameters • fn (str) – Name of the file to retrieve the full path for. • root (str) – Optional root path to look for a docdata in.

Notes

Defaults to looking for the docdata directory in the current working directory. This behavior works fine for notebooks nested in the docs directory of synapse; but this root directory that is looked for may be overridden by providing an alternative root. Returns A file path. Return type str Raises ValueError if the file does not exist or directory traversal attempted.. synapse.lib.jupyter.getItemCmdr(prox, outp=None, locs=None) Get a Cmdr instance with prepopulated locs synapse.lib.jupyter.getTempCoreCmdr(mods=None, outp=None) Get a CmdrCore instance which is backed by a temporary Cortex. Parameters • mods (list) – A list of additional CoreModules to load in the Cortex. • outp – A output helper. Will be used for the Cmdr instance.

Notes

The CmdrCore returned by this should be fini()’d to tear down the temporary Cortex. Returns A CmdrCore instance. Return type CmdrCore synapse.lib.jupyter.getTempCoreProx(mods=None) Get a Telepath Proxt to a Cortex instance which is backed by a temporary Cortex. Parameters mods (list) – A list of additional CoreModules to load in the Cortex.

276 Chapter 6. synapse synapse Documentation, Release 0.1.0

Notes

The Proxy returned by this should be fini()’d to tear down the temporary Cortex. Returns s_telepath.Proxy synapse.lib.kv module class synapse.lib.kv.KvDict(stor, iden) Bases: object A KvDict uses the KvStor to implement a pythonic dict-like object. Unlike the KvLook object, the KvDict keeps all items in the dictionary in memory, so retrieval is fast; and only updates needs to be written to the the underlying KvStor object. Note: set() must be called to persist changes to mutable values like dicts or lists get(prop, defval=None) Get a property from the KvDict. Parameters • prop (str) – The property name. • defval (obj) – The default value to return. Returns The return value, or None. Return type (obj) items() Return a tuple of (prop, valu) tuples from the KvDict. Returns Tuple of (prop, valu) tuples. Return type (((str, object), ..)) pop(prop) Pop a property from the KvDict. Parameters prop (str) – The property name. Returns The object stored in the KvDict, or None if the object was not present. Return type object set(prop, valu) Set a property in the KvDict. Parameters • prop (str) – The property name. • valu (obj) – A msgpack compatible value. Returns None class synapse.lib.kv.KvLook(stor, iden) Bases: object A KvLook uses the KvStor to implement key=valu lookup namespace.

6.1. synapse package 277 synapse Documentation, Release 0.1.0

The primary APIs, get() and set(), will use msgpack to decode and encode objects retrieved from the store. This allows storing complex data structures in the KV store. getraw() and setraw() APIs exist for purely bytes in / bytes out interfaces. get(prop, defval=None) Lookup a property from the KvLook. Parameters • prop (str) – The property name. • defval (obj) – The default value to return. Returns The valu, aftering being unpacked via msgpack, or None. Return type object getraw(lkey) Retrieve a value directly by bytes. Parameters lkey (bytes) – Byte value to retrieve. Returns Bytes for a given key, or None if it does not exist. Return type bytes items() Iterate over items stored in the KvLook namespace. Yields ((str, object)) – The name and object for items in KvLook namespace. set(prop, valu) Set a property in the KvLook. Parameters • prop (str) – The property name to set. • valu (obj) – A msgpack compatible value. Returns None setraw(lkey, lval) Set the value directly by bytes. Parameters • lkey (bytes) – Byte value to set. • lval (bytes) – Bytes to set to the value. Returns None class synapse.lib.kv.KvSet(stor, iden) Bases: object A KvSet uses the KvStor to implement a pythonic set-like object. Unlike the KvLook object, the KvSet keeps all items in the set in memory, so access is fast; and only updates needs to be written to the the underlying KvStor object. add(valu) Add a value to the KvSet. Parameters valu (obj) – A msgpack value to add. Returns None

278 Chapter 6. synapse synapse Documentation, Release 0.1.0

remove(valu) Remove a value from the KvSet. Parameters valu (obj) – A msgpack value to remove. Returns True if the value was found and removed. Return type (bool) update(vals) Extend the KvSet by adding any new values from vals to the set. Parameters vals (list) – A list of msgpack values to add to the set. Returns None class synapse.lib.kv.KvStor(path) Bases: synapse.eventbus.EventBus The KvStor uses an LMDB key-value database to implement persistence and indexing for primitive data struc- tures. addKvDup(lkey, lval) Add a (potentially duplicate) key=valu to the KvStor. Parameters • lkey (bytes) – The kv key. • lval (bytes) – The kv val. Returns None addKvDups(dups) Add a list of (lkey,lval) dups to the KvStor. Parameters dups (list) – A list of (lkey,lval) tuples. delKvDup(lkey, lval) Delete a single key=value pair from the KvStor. Parameters • lkey – (bytes): The kv key. • lval – (bytes): The kv value. Returns True if at least one key was deleted, False otherwise Return type bool delKvProp(lkey) Delete a key=val prop from the KvStor. Parameters lkey (bytes) – key to delete Returns True if at least one key was deleted, False otherwise Return type bool genKvAlias Resolve or create a new object alias by name. Parameters name (str) – String to create or resolve an alias for.

6.1. synapse package 279 synapse Documentation, Release 0.1.0

Notes

The iden returned as an alias is randomly generated the first time that genKvAlias is called for a given name. Returns The iden for the name Return type bytes getKvDict(name) Create or retrieve a KvDict by name from the KvStor. Parameters name (str) – The name of the KvDict. Returns The KvDict helper instance. Return type KvDict getKvLook(name) Create or retrieve a KvLook by name from the KvStor. Parameters name (str) – The name of the KvLook. Returns The KvLook helper instance. Return type KvLook getKvProp(lkey) Retrieve the lval bytes for a key. Parameters lkey (bytes) – The kv key. Returns The kv value, or None if it does not exist. Return type bytes getKvSet(name) Create or retrieve a KvSet by name from the KvStor. Parameters name (str) – The name of the KvSet. Returns The KvSet helper instance. Return type KvSet hasKvDups(lkey) Returns True if the number of values for lkey is greater than 0. Parameters lkey (bytes) – The kv key. Returns True if the dups key exists, False otherwise. Return type bool iterKvDups(lkey) Yield lkey, lval tuples for the given dup key. Parameters lkey (bytes) – The kv key. Yields bytes – The value of the dups for a given key. iterKvProps(lkey) Yield lkey, lval tuples for the given prop prefix. Parameters lkey (bytes) – The kv key prefix. Yields ((bytes, bytes)) – A tuple of key, value pairs which start with the prefix.

280 Chapter 6. synapse synapse Documentation, Release 0.1.0

setKvProp(lkey, lval) Set a non-duplicate key=valu to the KvStor. Parameters • lkey (bytes) – The kv key. • lval (bytes) – The kv val. Returns None setKvProps(props) Set a multiple non-duplicate key=valu props in the KvStor. Parameters props (dict) – A dict of lkey: lvalu pairs. Returns None

synapse.lib.layer module

The layer library contains the base Layer object and helpers used for cortex construction. class synapse.lib.layer.Encoder Bases: collections.defaultdict class synapse.lib.layer.Layer Bases: synapse.lib.cell.Cell A layer implements btree indexed storage for a cortex. abort() cellapi alias of LayerApi commit() getBuidProps(buid) getLiftRows(lops) getModelVers() getNodeNdef(buid) getOffset(iden) iterFormRows(form) Iterate (buid, valu) rows for the given form in this layer. iterPropRows(form, prop) Iterate (buid, valu) rows for the given form:prop in this layer. iterUnivRows(prop) Iterate (buid, valu) rows for the given universal prop setModelVers(vers) setOffset(iden, offs) splicelistAppend(mesg) splices(offs, size) stat()

6.1. synapse package 281 synapse Documentation, Release 0.1.0

stor(sops) Execute a series of storage operations. class synapse.lib.layer.LayerApi Bases: synapse.lib.cell.PassThroughApi allowed_methods = ['getLiftRows', 'stor', 'commit', 'abort', 'getBuidProps', 'iterFormRows', 'iterPropRows', 'iterUnivRows', 'getOffset', 'setOffset', 'initdb', 'splicelistAppend', 'splices', 'stat'] getModelVers() class synapse.lib.layer.Utf8er Bases: collections.defaultdict synapse.lib.link module class synapse.lib.link.Link Bases: synapse.lib.base.Base A Link() is created to wrap a socket reader/writer. feed(byts) Used by Plex() to unpack bytes. get(name, defval=None) Get a property from the Link info. recv(size) recvsize(size) rx() send(byts) set(name, valu) Set a property in the Link info. tx(mesg) Async transmit routine which will wait for writer drain(). synapse.lib.link.connect(host, port, ssl=None) Async connect and return a Link(). synapse.lib.link.listen(host, port, onlink, ssl=None) Listen on the given host/port and fire onlink(Link). Returns a server object that contains the listening sockets

synapse.lib.lmdb module

class synapse.lib.lmdb.Metrics(lenv, name=b’metrics’) Bases: object A helper for recording metrics about an Environment. Parameters • lenv (lmdb.Environment) – The LMDB Environment. • name (str) – The name of the metrics instance.

282 Chapter 6. synapse synapse Documentation, Release 0.1.0

inc(xact, prop, step=1) Increment the value of a global metric. Parameters • xact (lmdb.Transaction) – An LMDB write transaction. • prop (str) – The property to increment. • step (int) – The value by which to increment the property. Returns None iter(xact, offs) Iterate over metrics items from a given offset. Parameters • xact (lmdb.Transaction) – An LMDB transaction. • offs (int) – The offset to begin iterating from. Yields ((indx, object)) – The index and sample. record(xact, info) Record metrics info. Parameters • xact (Transaction) – An LMDB write transaction. • info (dict) – A dictionary of sample info to save. Returns None stat() Return the metrics info. Returns The dictionary of recorded metrics. Return type dict class synapse.lib.lmdb.Offs(lenv, db) Bases: object get(iden) set(iden, offs) xget(xact, iden) xset(xact, iden, offs) class synapse.lib.lmdb.PropSetr(psto, xact) Bases: object A helper for setting properties. Most to cache cursors. Parameters • ptso (PropStor) – The PropStore. • xact (lmdb.Transaction) – An LMDB write transaction. has(penc, byts) Check for the existence of an encoded prop, valu pair in a PropStor. Parameters

6.1. synapse package 283 synapse Documentation, Release 0.1.0

• penc (bytes) – The encoded property name. • byts (bytes) – The valu bytes. Returns True if the pair exists, False otherwise. Return type bool put(items) Put a list of items into the PropStor. Parameters items (list) – A list of (buid, ((penv, lval, flags),. . . )) tuples to put. Yields ((int, (bytes, list))) – Yields the item number, buid and list of changed props. set(buid, penc, lval, flags=0) Set a row in a PropStor. Parameters • buid (bytes) – The binary GUID like sequence of 32 bytes. • penc (bytes) – The encoded property name. • lval (bytes) – The valu bytes. Returns True if the row was set, False otherwise. Return type bool class synapse.lib.lmdb.PropStor(lenv, name=b’stor’) Bases: object A property store. Parameters • lenv (lmdb.Environment) – The LMDB Environment. • name (str) – The name of property store. eq(xact, penc, pval) Yield (buid, pkey, pval) rows by prop=valu. Parameters • xact (lmdb.Transaction) – An LMDB transaction. • penc (bytes) – The encoded property name. • pval (bytes) – The encoded property value. Yields ((bytes, bytes, bytes)) – A buid, penc, pval row. getPropSetr(xact) Return a new PropSetr helper. Parameters xact (lmdb.Transaction) – An LMDB transaction. Returns The property setter helper. Return type PropSetr has(xact, penc, byts) Check for the existence of an encoded prop, valu pair in a PropStor. Parameters • xact (lmdb.Transaction) – An LMDB transaction.

284 Chapter 6. synapse synapse Documentation, Release 0.1.0

• penc (bytes) – The encoded property name. • byts (bytes) – The valu bytes. Returns True if the pair exists, False otherwise. Return type bool pref(xact, penc, byts) Perform a prefix search and yield (buid, penc, pval) rows. Parameters • xact (lmdb.Transaction) – An LMDB transaction. • penc (bytes) – The encoded property name. • byts (bytes) – The valu bytes. Yields ((bytes, bytes, bytes)) – A buid, penc, pval row. range(xact, penc, bval, nval) Perform a range search and yield (buid, penc, pval) rows. Parameters • xact (lmdb.Transaction) – An LMDB transaction. • penc (bytes) – The encoded property name. • bval (bytes) – The lower bound to search. • nval (bytes) – The upper bound to search. Yields ((bytes, bytes, bytes)) – A buid, penc, pval row. recs(xact, rows) Yields full (buid, (props..)) records from rows. Parameters • xact (lmdb.Transaction) – An LMDB transaction. • rows (list) – A list of ((buid, penc, pval)) rows. Yields ((bytes, list)) – A set of (buid, (props. . . )) for the rows. class synapse.lib.lmdb.Seqn(lenv, name) Bases: object An append optimized sequence of byte blobs. Parameters • lenv (lmdb.Environment) – The LMDB Environment. • name (str) – The name of the sequence. index() Return the current index to be used iter(xact, offs) Iterate over items in a sequence from a given offset. Parameters • xact (lmdb.Transaction) – An LMDB transaction. • offs (int) – The offset to begin iterating from.

6.1. synapse package 285 synapse Documentation, Release 0.1.0

Yields (indx, valu) – The index and valu of the item. nextindx(xact) Determine the next insert offset according to storage. Parameters xact (lmdb.Transaction) – An LMDB transaction. Returns The next insert offset. Return type int save(xact, items) Save a series of items to a sequence. Parameters • xact (lmdb.Transaction) – An LMDB write transaction. • items (tuple) – The series of items to save into the sequence. Returns None slice(xact, offs, size) synapse.lib.lmdb.encodeValAsKey(v, isprefix=False) Encode a value (int or str) as used in a key into bytes so that prefix searches on strings and range searches on ints work. The first encoded byte indicates Integers are 8-byte little endian - MIN_INT_VAL(this ensures that all negative values sort before all nonnegative values) Strings are UTF-8 encoded NULL-terminated unless isprefix is True. If string length > LARGE_STRING_SIZE, just the first 128 bytes are written and a non-cryptographically hash is appended, and isprefix is disregarded. Note that this scheme prevents interleaving of value types: all string encodings compare larger than all integer encodings. Parameters • (Union[str, int] (v) – the value. • isprefix – whether to interpret v as a prefix. If true, strings will not be appended with a NULL. synapse.lib.lmdblayer module

The layer library contains the base Layer object and helpers used for cortex construction. class synapse.lib.lmdblayer.LmdbLayer Bases: synapse.lib.layer.Layer A layer implements btree indexed storage for a cortex. commit() confdefs = (('lmdb:mapsize', {'type': 'int', 'defval': 536870912}), ('lmdb:maxsize', {'type': 'int', 'defval': None, 'doc': 'The largest the DB file will grow to'}), ('lmdb:growsize', {'type': 'int', 'defval': None, 'doc': 'The amount in bytes to grow the DB file when full. Defaults to doubling'}), ('lmdb:readahead', {'type': 'bool', 'defval': True})) getBuidProps(buid) getModelVers() getNodeNdef(buid)

286 Chapter 6. synapse synapse Documentation, Release 0.1.0

getOffset(iden) Note: this method doesn’t need to be async, but it is probable that future layer implementations would need it to be async initdb(name, dupsort=False) iterFormRows(form) Iterate (buid, valu) rows for the given form in this layer. iterPropRows(form, prop) Iterate (buid, valu) rows for the given form:prop in this layer. iterUnivRows(prop) Iterate (buid, valu) rows for the given universal prop setModelVers(vers) setOffset(iden, offs) Note: this method doesn’t need to be async, but it is probable that future layer implementations would need it to be async splices(offs, size) stat() synapse.lib.lmdbslab module class synapse.lib.lmdbslab.Scan(slab, db) Bases: object A state-object used by Slab. Not to be instantiated directly. bump() first() iternext() last_key() Return the last key in the database. Returns none if database is empty. set_key(lkey) set_range(lkey) class synapse.lib.lmdbslab.Slab Bases: synapse.lib.base.Base A “monolithic” LMDB instance for use in a asyncio loop thread. COMMIT_PERIOD = 1.0 aborted() delete(lkey, val=None, db=None) forcecommit() get(lkey, db=) grow(size=None) Close out the current transaction and resize the memory map. initdb(name, dupsort=False)

6.1. synapse package 287 synapse Documentation, Release 0.1.0

pop(lkey, db=None) put(lkey, lval, dupdata=False, db=None) putmulti(kvpairs, dupdata=False, append=False, db=) replace(lkey, lval, db=None) Like put, but returns the previous value if existed scanByDups(lkey, db=None) scanByFull(db=None) scanByPref(byts, db=None) scanByRange(lmin, lmax=None, db=None) synchold() Hold this across small/fast multi-writes to delay commit evaluation. This allows commit() boundaries to occur when the underlying db is coherent.

Example

with dude.writer(): dude.put(foo, bar) dude.put(baz, faz) synapse.lib.migrate module class synapse.lib.migrate.Migration Bases: synapse.lib.base.Base A migration instance provides a resume-capable workspace for large data migrations within a cortex. editNdefProps(oldv, newv) Change all ndef props from oldv to newv. getFormTodo(name) getLayers() setNodeBuid(form, oldb, newb) Carry out the rewrite of a node buid in all layers. setNodeForm(layr, buid, name, oldv, newv) Reset the primary property for the given buid. setPropsByType(name, oldv, newv, info) Update secondary props of the given type.

synapse.lib.mixins module

synapse.lib.mixins.addSynMixin(subsys, name, cname=None) Add a mixin class to the specified subsystem.

Example

s_mixins.addSynMixin(‘foo’,’synapse.foo.FooMixin’)

288 Chapter 6. synapse synapse Documentation, Release 0.1.0

synapse.lib.mixins.getSynMixins(subsys, name) Return a list of mixin classes for the given subsystem class.

Example

for clas in getSynMixins(‘telepath’,’foo.bar.Baz’): dostuff() synapse.lib.mixins.ldict() synapse.lib.modelrev module class synapse.lib.modelrev.ModelRev(core) Bases: object revCoreLayers() synapse.lib.module module class synapse.lib.module.CoreModule(core) Bases: object confdefs = () getConfPath() Get the path to the module specific config file (conf.yaml).

Notes

This creates the parent directory for the conf.yaml file if it does not exist. This API exists to allow a implementor to get the conf path during initCoreModule and drop a example config if needed. One use case of that is for missing configuration values, an example config can be written to the file and a exception raised. Returns Path to where the conf file is located at. Return type str getModDir() Get the path to the module specific directory.

Notes

This creates the directory if it did not previously exist. Returns The filepath to the module specific directory. Return type str getModName() Return the lowercased name of this module.

6.1. synapse package 289 synapse Documentation, Release 0.1.0

Notes

This pulls the mod_name attribute on the class. This allows an implementer to set a arbitrary name for the module. If this attribute is not set, it defaults to self.__class__.__name__. Returns The module name. Return type (str) getModPath(*paths) Construct a path relative to this module’s working directory. Parameters (*paths) – A list of path strings

Notes

This creates the module specific directory if it does not exist. Returns The full path (or None if no cortex dir is configured). Return type (str) getModelDefs() getModelRevs() initCoreModule() Module implementers may over-ride this method to initialize the module during initial construction. Any exception raised within this method will be raised from the constructor and mark the module as failed. Args: Returns None mod_name = None synapse.lib.modules module

Module which implements the synapse module API/convention. synapse.lib.msgpack module class synapse.lib.msgpack.Unpk Bases: object An extension of the msgpack streaming Unpacker which reports sizes.

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings. feed(byts) Feed bytes to the unpacker and return completed objects. Parameters byts (bytes) – Bytes to unpack.

290 Chapter 6. synapse synapse Documentation, Release 0.1.0

Notes

It is intended that this function is called multiple times with bytes from some sort of a stream, as it will unpack and return objects as they are available. Returns List of tuples containing the item size and the unpacked item. Return type list synapse.lib.msgpack.dumpfile(item, path) Dump an object to a file by path. Parameters • item (object) – The object to serialize. • path (str) – The file path to save. Returns None synapse.lib.msgpack.en(item) Use msgpack to serialize a compatible python object. Parameters item (obj) – The object to serialize

Notes

String objects are encoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow encoding bad input strings. Returns The serialized bytes in msgpack format. Return type bytes synapse.lib.msgpack.iterfd(fd) Generator which unpacks a file object of msgpacked content. Parameters fd – File object to consume data from.

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings. Yields Objects from a msgpack stream. synapse.lib.msgpack.iterfile(path, since=-1) Generator which yields msgpack objects from a file path. Parameters path – File path to open and consume data from.

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings. Yields Objects from a msgpack stream. synapse.lib.msgpack.loadfile(path) Load and upack the msgpack bytes from a file by path.

6.1. synapse package 291 synapse Documentation, Release 0.1.0

Parameters path (str) – The file path to a message pack file. Raises msgpack.exceptions.ExtraData – If the file contains multiple objects. Returns The decoded python object. Return type (obj) synapse.lib.msgpack.un(byts) Use msgpack to de-serialize a python object. Parameters byts (bytes) – The bytes to de-serialize

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings. Returns The de-serialized object Return type obj synapse.lib.node module class synapse.lib.node.Node(snap, buid=None, rawprops=None) Bases: object A Cortex hypergraph node. NOTE: This object is for local Cortex use during a single Xact. addTag(tag, valu=(None, None)) delTag(tag, init=False) Delete a tag from the node. delete(force=False) Delete a node from the cortex. The following tear-down operations occur in order: • validate that you have permissions to delete the node • validate that you have permissions to delete all tags • validate that there are no remaining references to the node. • delete all the tags (bottom up) – fire onDelTag() handlers – delete tag properties from storage – log tag:del splices • delete all secondary properties – fire onDelProp handler – delete secondary property from storage – log prop:del splices • delete the primary property

292 Chapter 6. synapse synapse Documentation, Release 0.1.0

– fire onDel handlers for the node – delete primary property from storage – log node:del splices filter(text, opts=None, user=None) get(name) Return a secondary property value from the Node. Parameters name (str) – The name of a secondary property. Returns The secondary property value or None. Return type (obj) getNodeRefs() Return a list of (prop, (form, valu)) refs out for the node. getTag(name, defval=None) getTags(leaf=False) has(name) hasTag(name) iden() pack(dorepr=False) Return the serializable/packed version of the node. Returns An (iden, info) node tuple. Return type (tuple) pop(name, init=False) repr(name=None) reprs() seen(tick, source=None) Update the .seen interval and optionally a source specific seen node. set(name, valu, init=False) Set a property on the node. Parameters • name (str) – The name of the property. • valu (obj) – The value of the property. • init (bool) – Set to True to disable read-only enforcement Returns True if the property was changed. Return type (bool) storm(text, opts=None, user=None) class synapse.lib.node.Path(runt, vars, nodes) Bases: object A path context tracked through the storm runtime. fork(node)

6.1. synapse package 293 synapse Documentation, Release 0.1.0

get(name, defv=) meta(name, valu) Add node specific metadata to be returned with the node. pack(path=False) set(name, valu) synapse.lib.node.iden(pode) Return the iden (buid) of the packed node. Parameters pode (tuple) – A packed node. Returns The node iden. Return type str synapse.lib.node.ndef(pode) Return a node definition (, tuple from the node. Parameters node (tuple) – A packed node. Returns The (,) tuple for the node Return type ((str,obj)) synapse.lib.node.prop(pode, prop) Return the valu of a given property on the node. Parameters • pode (tuple) – A packed node. • prop (str) – Property to retrieve.

Notes

The prop argument may be the full property name (foo:bar:baz), relative property name (:baz) , or the unadorned property name (baz). Returns: synapse.lib.node.props(pode) Get the props from the node. Parameters pode (tuple) – A packed node.

Notes

This will include any universal props present on the node. Returns A dictionary of properties. Return type dict synapse.lib.node.tagged(pode, tag) Check if a packed node has a given tag. Parameters • pode (tuple) – A packed node. • tag (str) – The tag to check.

294 Chapter 6. synapse synapse Documentation, Release 0.1.0

Examples

Check if a node is tagged with “woot” and dostuff if it is. if s_node.tagged(node,’woot’): dostuff()

Notes

If the tag starts with #, this is removed prior to checking. Returns True if the tag is present. False otherwise. Return type bool synapse.lib.node.tags(pode, leaf=False) Get all the tags for a given node. Parameters • pode (tuple) – A packed node. • leaf (bool) – If True, only return the full tags. Returns A list of tag strings. Return type list synapse.lib.output module

Tools for easily hookable output from cli-like tools. class synapse.lib.output.OutPut Bases: object printf(mesg, addnl=True) class synapse.lib.output.OutPutBytes Bases: synapse.lib.output.OutPutFd class synapse.lib.output.OutPutFd(fd, enc=’utf8’) Bases: synapse.lib.output.OutPut class synapse.lib.output.OutPutStr Bases: synapse.lib.output.OutPut synapse.lib.queue module class synapse.lib.queue.AQueue Bases: synapse.lib.base.Base An async queue with chunk optimized sync compatible consumer. put(item) Add an item to the queue. slice() class synapse.lib.queue.AsyncQueue Bases: synapse.lib.base.Base Multi-async producer, single sync finite consumer queue with blocking at empty and full.

6.1. synapse package 295 synapse Documentation, Release 0.1.0

get() Pending retrieve on the queue put(item) Put onto the queue. It will async pend if the queue is full or draining. class synapse.lib.queue.Queue(items=()) Bases: synapse.eventbus.EventBus A simple custom queue to address python Queue() issues. done() Gracefully mark this Queue as done. This still allows a Queue consumer to finish consuming it. The Queue functions get(), slice() and slices() will not block when .done() has been called on a Queue. Returns None get(timeout=None) Get the next item from the queue. Parameters timeout (int) – Duration, in seconds, to wait for items to be available to the queue before returning.

Notes

This will block if the queue is empty and no timeout value is specified, or .done() has not been called on the Queue.

Examples

Get an item and do stuff with it:

item=q.get(timeout=30) dostuff(item)

Returns Item from the queue, or None if the queue is fini() or timeout occurs.

getn(timeout=None) Get the next item using the (ok, retn) convention. put(item) Add an item to the queue and wake any consumers waiting on the queue. Parameters item – Item to add to the queue.

Notes

This will not add the item or wake any consumers if .done() has not been called on the Queue.

Examples

Put a string in a queue:

296 Chapter 6. synapse synapse Documentation, Release 0.1.0

q.put('woot')

Returns None

size() Return the number of entries in the Queue. Returns The number of entries. Return type int slice(size, timeout=None) Get a slice of the next items from the queue. Parameters • size (int) – Maximum number of items to get from the queue. • timeout (int) – Duration, in seconds, to wait for items to be available to the queue before returning.

Examples

Return up to 3 items on a 30 second timeout from the queue:

items=q.slice(3, timeout=30)

Notes

This will block if the queue is empty and no timeout value is specified, or .done() has not been called on the Queue. Returns A list of items from the queue. This will return None on fini() or timeout. Return type list Raises • synapse.exc.IsFini – Once the queue is fini • synapse.exc.TimeOut – If timeout it specified and has passed. slices(size, timeout=None) Yields slices of items from the queue. Parameters • size (int) – Maximum number of items to yield at a time. • timeout (int) – Duration, in seconds, to wait for items to be added to the queue before exiting.

6.1. synapse package 297 synapse Documentation, Release 0.1.0

Examples

Yield 2 items at a time with a 1 second time:

for items in q.slices(2, timeout=1): dostuff(items)

Notes

This will block if the queue is empty and no timeout value is specified, or .done() has not been called on the Queue. Yields list – This generator yields a list of items. class synapse.lib.queue.S2AQueue Bases: synapse.lib.base.Base Sync single producer, async single consumer finite queue with blocking at empty and full. get() Async pend retrieve on the queue put(item) Put onto the queue. Pend if the queue is full or draining. synapse.lib.ratelimit module class synapse.lib.ratelimit.RateLimit(rate, per) Bases: object A RateLimit class may be used to detect/enforce rate limits.

Example

# allow 20 uses per 10 sec ( 2/sec ) rlimit = RateLimit(20,10)

Notes

It is best ( even in a “calls per day” type config ) to specify a smaller “per” to force rate “smoothing”. allows() Returns True if the rate limit has not been reached.

Example

if not rlimit.allows(): rasie RateExceeded()

# ok to go. . . synapse.lib.reflect module synapse.lib.reflect.getClsNames(item) Return a list of “fully qualified” class names for an instance.

298 Chapter 6. synapse synapse Documentation, Release 0.1.0

Example

for name in getClsNames(foo): print(name) synapse.lib.reflect.getItemInfo(item) Get “reflection info” dict for the given object. Parameters item – Item to inspect.

Examples

Find out what classes a Telepath Proxy object inherits:

info= getItemInfo(prox) classes= info.get('inherits')

Notes

Classes may implement a def _syn_reflect(self): function in order to return explicit values. The Telepath Proxy object is one example of doing this, in order to allow a remote caller to identify what classes the Proxy object represents. Returns Dictionary of reflection information. Return type dict synapse.lib.reflect.getItemLocals(item) Iterate the locals of an item and yield (name,valu) pairs.

Example

for name,valu in getItemLocals(item): dostuff() synapse.lib.reflect.getMethName(meth) Return a fully qualified string for the .. name of a given method. synapse.lib.remotelayer module

The layer that exists locally (as a client) passing data requests to a remote layer class synapse.lib.remotelayer.RemoteLayer Bases: synapse.lib.layer.Layer A layer retrieved over telepath. confdefs = (('remote:telepath', {'type': 'str', 'doc': 'Path to remote layer'}),) getLiftRows(*args, **kwargs) getModelVers() iterFormRows(*args, **kwargs) Iterate (buid, valu) rows for the given form in this layer. iterPropRows(*args, **kwargs) Iterate (buid, valu) rows for the given form:prop in this layer.

6.1. synapse package 299 synapse Documentation, Release 0.1.0

iterUnivRows(*args, **kwargs) Iterate (buid, valu) rows for the given universal prop setModelVers(vers) splices(*args, **kwargs) synapse.lib.scope module class synapse.lib.scope.Scope(*frames, **vals) Bases: object The Scope object assists in creating nested varible scopes.

Example

with Scope() as scope: scope.set(‘foo’,10) with scope: scope.set(‘foo’,20) dostuff(scope) # ‘foo’ is 20. . . dostuff(scope) # ‘foo’ is 10 again. . . add(name, *vals) Add values as iter() compatible items in the current scope frame. ctor(name, func, *args, **kwargs) Add a constructor to be called when a specific property is not present.

Example

scope.ctor(‘foo’,FooThing) . . . foo = scope.get(‘foo’) enter(vals=None) Add an additional scope frame. get(name, defval=None) Retrieve a value from the closest scope frame. iter(name) Iterate through values added with add() from each scope frame. leave() Pop the current scope frame. pop(name, defval=None) Pop and return a value (from the last frame) of the scope. Parameters name (str) – The name of the scope variable. Returns The scope variable value or None Return type obj set(name, valu) Set a value in the current scope frame. update(vals) Set multiple values in the current scope frame.

300 Chapter 6. synapse synapse Documentation, Release 0.1.0 synapse.lib.scope.ctor(name, func, *args, **kwargs) Add a ctor callback to the global scope. synapse.lib.scope.enter(vals=None) Return the task’s local scope for use in a with block synapse.lib.scope.get(name, defval=None) Access this task’s scope with default values from glob. synapse.lib.scope.pop(name) Pop and return a task scope variable. :param name: The task scope variable name. :type name: str Returns The scope value or None Return type obj synapse.lib.scope.set(name, valu) Set a value in the current frame of the local task scope. synapse.lib.scope.update(vals) synapse.lib.scrape module synapse.lib.scrape.scrape(text) Scrape types from a blob of text and return node tuples. synapse.lib.share module class synapse.lib.share.Share Bases: synapse.lib.base.Base Class to wrap a dynamically shared object. synapse.lib.slaboffs module class synapse.lib.slaboffs.SlabOffs(slab: synapse.lib.lmdbslab.Slab, db) Bases: object get(iden) set(iden, offs) synapse.lib.slabseqn module class synapse.lib.slabseqn.SlabSeqn(slab: synapse.lib.lmdbslab.Slab, name: str) Bases: object An append optimized sequence of byte blobs. Parameters • lenv (lmdb.Environment) – The LMDB Environment. • name (str) – The name of the sequence. index() Return the current index to be used

6.1. synapse package 301 synapse Documentation, Release 0.1.0

iter(offs) Iterate over items in a sequence from a given offset. Parameters offs (int) – The offset to begin iterating from. Yields (indx, valu) – The index and valu of the item. nextindx() Determine the next insert offset according to storage. Parameters xact (lmdb.Transaction) – An LMDB transaction. Returns The next insert offset. Return type int save(items) Save a series of items to a sequence. Parameters items (tuple) – The series of items to save into the sequence. Returns None slice(offs, size) synapse.lib.snap module class synapse.lib.snap.Snap Bases: synapse.lib.base.Base A “snapshot” is a transaction across multiple Cortex layers. The Snap object contains the bulk of the Cortex API to facilitate performance through careful use of transaction boundaries. Transactions produce the following EventBus events: (. . . any splice. . . ) (‘log’, {‘level’: ‘mesg’: }) (‘print’, {}), addFeedData(name, items, seqn=None) addFeedNodes(name, items) Call a feed function and return what it returns (typically yields Node()s). Parameters • name (str) – The name of the feed record type. • items (list) – A list of records of the given feed type. Returns The return value from the feed function. Typically Node() generator. Return type (object) addNode(name, valu, props=None) Add a node by form name and value with optional props. Parameters • name (str) – The form of node to add. • valu (obj) – The value for the node. • props (dict) – Optional secondary properties for the node.

302 Chapter 6. synapse synapse Documentation, Release 0.1.0

addNodes(nodedefs) Add/merge nodes in bulk. The addNodes API is designed for bulk adds which will also set properties and add tags to existing nodes. Nodes are specified as a list of the following tuples: ( (form, valu), {‘props’:{}, ‘tags’:{}})

Parameters nodedefs (list) – A list of nodedef tuples. Returns A list of xact messages. Return type (list)

addTagNode(name) Ensure that the given syn:tag node exists. eval(text, opts=None, user=None) Run a storm query and yield Node() objects. getLiftNodes(lops, rawprop, cmpr=None) getLiftRows(lops) Yield row tuples from a series of lift operations. Row tuples only requirement is that the first element be the binary id of a node. Parameters lops (list) – A list of lift operations. Yields (tuple) – (layer_indx, (buid, . . . )) rows. getNodeByBuid(buid) Retrieve a node tuple by binary id. Parameters buid (bytes) – The binary ID for the node. Returns The node object or None. Return type Optional[s_node.Node] getNodeByNdef(ndef ) Return a single Node by (form,valu) tuple. Parameters ndef ((str,obj)) – A (form,valu) ndef tuple. Returns The Node or None. Return type (synapse.lib.node.Node) getNodesBy(full, valu=None, cmpr=’=’) The main function for retrieving nodes by prop. Parameters • full (str) – The property/tag name. • valu (obj) – A lift compatible value for the type. • cmpr (str) – An optional alternate comparator. Yields (synapse.lib.node.Node) – Node instances. getOffset(iden, offs)

6.1. synapse package 303 synapse Documentation, Release 0.1.0

getRowNodes(rows, rawprop, cmpr=None) Join a row generator into (row, Node()) tuples. A row generator yields tuple rows where the first valu is the buid of a node. Parameters • rows – A generator of (layer_idx, (buid, . . . )) tuples. • rawprop (str) – “raw” propname i.e. if a tag, starts with “#”. Used for filtering so that we skip the props for a buid if we’re asking from a higher layer than the row was from (and hence, we’ll presumable get/have gotten the row when that layer is lifted. • cmpr (func) – A secondary comparison function used to filter nodes. Yields (tuple) – (row, node) getRuntNodes(full, valu=None, cmpr=’=’) getStormRuntime(opts=None, user=None) iterStormPodes(text, opts=None, user=None) Yield packed node tuples for the given storm query text. printf(mesg) setOffset(iden, offs) setUser(user) splice(name, **info) Construct and log a splice record to be saved on commit(). stor(sops) storm(text, opts=None, user=None) Execute a storm query and yield (Node(), Path()) tuples. warn(mesg, **info) synapse.lib.storm module class synapse.lib.storm.Cmd(argv) Bases: object A one line description of the command. Command usage details and long form description.

Example

cmd –help execStormCmd(runt, genr) Abstract base method getArgParser() classmethod getCmdBrief() hasValidOpts(snap) name = 'cmd'

304 Chapter 6. synapse synapse Documentation, Release 0.1.0

class synapse.lib.storm.CountCmd(argv) Bases: synapse.lib.storm.Cmd Iterate through query results, and print the resulting number of nodes which were lifted. This does yield the nodes counted.

Example

foo:bar:size=20 | count execStormCmd(runt, genr) Abstract base method name = 'count' class synapse.lib.storm.DelNodeCmd(argv) Bases: synapse.lib.storm.Cmd Delete nodes produced by the previous query logic. (no nodes are returned) Example inet:fqdn=vertex.link | delnode execStormCmd(runt, genr) Abstract base method getArgParser() name = 'delnode' class synapse.lib.storm.GraphCmd(argv) Bases: synapse.lib.storm.Cmd Generate a subgraph from the given input nodes and command line options. execStormCmd(runt, genr) Abstract base method getArgParser() name = 'graph' class synapse.lib.storm.HelpCmd(argv) Bases: synapse.lib.storm.Cmd List available commands and a brief description for each. execStormCmd(runt, genr) Abstract base method getArgParser() name = 'help' class synapse.lib.storm.IdenCmd(argv) Bases: synapse.lib.storm.Cmd Lift nodes by iden.

6.1. synapse package 305 synapse Documentation, Release 0.1.0

Example

iden b25bc9eec7e159dce879f9ec85fb791f83b505ac55b346fcb64c3c51e98d1175 | count execStormCmd(runt, genr) Abstract base method getArgParser() name = 'iden' class synapse.lib.storm.LimitCmd(argv) Bases: synapse.lib.storm.Cmd Limit the number of nodes generated by the query in the given position.

Example

inet:ipv4 | limit 10 execStormCmd(runt, genr) Abstract base method getArgParser() name = 'limit' class synapse.lib.storm.MaxCmd(argv) Bases: synapse.lib.storm.Cmd Consume nodes and yield only the one node with the highest value for a property.

Examples

file:bytes +#foo.bar | max :size file:bytes +#foo.bar | max file:bytes:size execStormCmd(runt, genr) Abstract base method getArgParser() name = 'max' class synapse.lib.storm.MinCmd(argv) Bases: synapse.lib.storm.Cmd Consume nodes and yield only the one node with the lowest value for a property.

Examples

file:bytes +#foo.bar | min :size file:bytes +#foo.bar | min file:bytes:size execStormCmd(runt, genr) Abstract base method getArgParser() name = 'min'

306 Chapter 6. synapse synapse Documentation, Release 0.1.0

class synapse.lib.storm.MoveTagCmd(argv) Bases: synapse.lib.storm.Cmd Rename an entire tag tree and preserve time intervals.

Example

movetag #foo.bar #baz.faz.bar execStormCmd(runt, genr) Abstract base method getArgParser() name = 'movetag' class synapse.lib.storm.NoderefsCmd(argv) Bases: synapse.lib.storm.Cmd Get nodes adjacent to inbound nodes, up to n degrees away.

Examples

The following examples show long-form options. Short form options exist and should be easier for regular use. Get all nodes 1 degree away from a input node: ask inet:ipv4=1.2.3.4 | noderefs Get all nodes 1 degree away from a input node and include the source node: ask inet:ipv4=1.2.3.4 | noderefs –join Get all nodes 3 degrees away from a input node and include the source node: ask inet:ipv4=1.2.3.4 | noderefs –join –degrees 3 Do not include nodes of a given form in the output or traverse across them: ask inet:ipv4=1.2.3.4 | noderefs –omit-form inet:dns:a Do not traverse across nodes of a given form (but include them in the output): ask inet:ipv4=1.2.3.4 | noderefs –omit-traversal-form inet:dns:a Do not include nodes with a specific tag in the output or traverse across them: ask inet:ipv4=1.2.3.4 | noderefs –omit-tag omit.nopiv Do not traverse across nodes with a sepcific tag (but include them in the output): ask inet:ipv4=1.2.3.4 | noderefs –omit-traversal-tag omit.nopiv Accept multiple inbound nodes, and unique the output set of nodes across all input nodes: ask inet:ipv4=1.2.3.4 inet:ipv4=1.2.3.5 | noderefs –degrees 4 –unique doRefs(srcnode, srcpath, visited) execStormCmd(runt, genr) Abstract base method getArgParser() getRefs(srcnode, srcpath)

6.1. synapse package 307 synapse Documentation, Release 0.1.0

name = 'noderefs' class synapse.lib.storm.Parser(prog=None, descr=None) Bases: argparse.ArgumentParser exit(status=0, message=None) Argparse expects exit() to be a terminal function and not return. As such, this function must raise an exception which will be caught by Cmd.hasValidOpts. class synapse.lib.storm.ReIndexCmd(argv) Bases: synapse.lib.storm.Cmd Use admin privileges to re index/normalize node properties.

Example

foo:bar | reindex –subs reindex –type inet:ipv4 NOTE: This is mostly for model updates and migrations. Use with caution and be very sure of what you are doing. execStormCmd(runt, genr) Abstract base method getArgParser() name = 'reindex' class synapse.lib.storm.Runtime(snap, opts=None, user=None) Bases: object A Runtime represents the instance of a running query. addInput(node) Add a Node() object as input to the query runtime. allowed() cancel() elevate() execStormQuery(query) getInput() getOpt(name, defval=None) getVar(name, defv=None) initPath(node) isRuntVar(name) iterStormQuery(query) printf(mesg) setOpt(name, valu) setVar(name, valu) tick() warn(mesg, **info)

308 Chapter 6. synapse synapse Documentation, Release 0.1.0 class synapse.lib.storm.SleepCmd(argv) Bases: synapse.lib.storm.Cmd Introduce a delay between returning each result for the storm query. NOTE: This is mostly used for testing / debugging.

Example

#foo.bar | sleep 0.5 execStormCmd(runt, genr) Abstract base method getArgParser() name = 'sleep' class synapse.lib.storm.SpinCmd(argv) Bases: synapse.lib.storm.Cmd Iterate through all query results, but do not yield any. This can be used to operate on many nodes without returning any.

Example

foo:bar:size=20 [ +#hehe ] | spin execStormCmd(runt, genr) Abstract base method name = 'spin' class synapse.lib.storm.SudoCmd(argv) Bases: synapse.lib.storm.Cmd Use admin privileges to bypass standard query permissions.

Example

sudo | [ inet:fqdn=vertex.link ] execStormCmd(runt, genr) Abstract base method name = 'sudo' class synapse.lib.storm.UniqCmd(argv) Bases: synapse.lib.storm.Cmd Filter nodes by their uniq iden values. When this is used a Storm pipeline, only the first instance of a given node is allowed through the pipeline.

Examples

#badstuff +inet:ipv4 ->* | uniq execStormCmd(runt, genr) Abstract base method

6.1. synapse package 309 synapse Documentation, Release 0.1.0

getArgParser() name = 'uniq' synapse.lib.stormtypes module class synapse.lib.stormtypes.Dict(valu, path=None) Bases: synapse.lib.stormtypes.Prim deref(name) class synapse.lib.stormtypes.Lib(runt, name=()) Bases: synapse.lib.stormtypes.StormType addLibFuncs() deref(name) class synapse.lib.stormtypes.LibTime(runt, name=()) Bases: synapse.lib.stormtypes.Lib addLibFuncs() fromunix(secs) Normalize a timestamp from a unix epoch time.

Example

[ :time = $lib.time.fromunix($epoch) ] class synapse.lib.stormtypes.Node(node, path=None) Bases: synapse.lib.stormtypes.Prim Implements the STORM api for a node instance. class synapse.lib.stormtypes.Prim(valu, path=None) Bases: synapse.lib.stormtypes.StormType The base type for all STORM primitive values. value() class synapse.lib.stormtypes.StormType(path=None) Bases: object The base type for storm runtime value objects. deref(name) class synapse.lib.stormtypes.Str(valu, path=None) Bases: synapse.lib.stormtypes.Prim synapse.lib.stormtypes.fromprim(valu, path=None) synapse.lib.syntax module class synapse.lib.syntax.Parser(parseinfo, text, offs=0) Bases: object must be quoted: , ) = must be quoted at beginning: . : # @ ( $ etc. . . .

310 Chapter 6. synapse synapse Documentation, Release 0.1.0

absprop() foo:bar casevalu() cmdargv() cmdname() cmdvalu(until={’\t’, ’\n’, ’’, ’|’, ’}’}) Consume and return one command argument, stopping when it hits a character (not in a quotation) in until. cmpr() cond() :foo :foo=20 :foo:bar=$baz :foo:bar=:foo:baz foo:bar foo:bar:baz=20 #foo.bar #foo.bar@2013 (:foo=10 and ( #foo or #bar )) condexpr() condsubq() eat(size, ignore=None) editnodeadd() foo:bar = hehe editoper() editpropdel() editpropset() :foo=10 edittagadd() edittagdel() editunivdel() editunivset() .foo = bar expect(text) filtoper() forloop() formjoin() formjoinin() <+- * / <+- prop formpivot() -> * -> #tag.match -> form:prop -> form formpivotin() <- * / <- prop getSortedDict()

6.1. synapse package 311 synapse Documentation, Release 0.1.0

ignore(charset) ignorespace() Ignore whitespace as well as comment syntax // and /* */ liftbytag() lifttagtag() more() nextchar() nextmust(text) nextstr(text) nextstrs(*texts) noms(chars=None, until=None, ignore=None) oper() peek(chars) propjoin(prop) :foo:bar -+> baz:faz proppivot(prop) :foo:bar -> baz:faz query() quoted() relprop() :foo:bar relpropvalu() singlequoted() stormcmd() A storm sub-query aware command line splitter. ( not for storm commands, but for commands which may take storm ) subquery() switchcase() tagmatch() tagname() univprop() .foo univpropvalu() valu() valulist() varcall(varv) varderef(varv) varlist()

312 Chapter 6. synapse synapse Documentation, Release 0.1.0

varname() vartokn() varvalu() $foo $foo.bar $foo.bar() $foo[0] $foo.bar(10) synapse.lib.syntax.is_literal(text, off ) synapse.lib.syntax.isquote(text, off ) synapse.lib.syntax.meh(txt, off, cset) synapse.lib.syntax.nextchar(text, off, valu) synapse.lib.syntax.nextin(text, off, vals) synapse.lib.syntax.nextstr(text, off, valu) synapse.lib.syntax.nom(txt, off, cset, trim=True) Consume chars in set from the string and return (subtxt,offset).

Example

text = “foo(bar)” chars = set(‘abcdefghijklmnopqrstuvwxyz’) name,off = nom(text,0,chars) synapse.lib.syntax.nom_whitespace(text, off ) synapse.lib.syntax.parse_cmd_kwarg(text, off=0) Parse a foo:bar= kwarg into (prop,valu),off synapse.lib.syntax.parse_cmd_kwlist(text, off=0) Parse a foo:bar=[,. . . ] kwarg list into (prop,valu),off synapse.lib.syntax.parse_cmd_string(text, off, trim=True) Parse in a command line string which may be quoted. synapse.lib.syntax.parse_float(text, off, trim=True) synapse.lib.syntax.parse_int(text, off, trim=True) synapse.lib.syntax.parse_list(text, off=0, trim=True) Parse a list (likely for comp type) coming from a command line input. The string elements within the list may optionally be quoted. synapse.lib.syntax.parse_literal(text, off, trim=True) synapse.lib.syntax.parse_string(text, off, trim=True) synapse.lib.syntax.parse_valu(text, off=0) Special syntax for the right side of equals in a macro synapse.lib.task module class synapse.lib.task.Task Bases: synapse.lib.base.Base The synapse Task object implements concepts similar to process trees for asyncio.Task instances. kill()

6.1. synapse package 313 synapse Documentation, Release 0.1.0

pack() worker(coro, name=’worker’) synapse.lib.task.current() Return the current synapse task. synapse.lib.task.executor(func, *args, **kwargs) Execute a function in an executor thread. Parameters todo ((func,args,kwargs)) – A todo tuple. synapse.lib.task.loop() synapse.lib.task.user() Return the current task user. synapse.lib.task.username() Return the current task user name. synapse.lib.thishost module synapse.lib.thishost.get(prop) Retrieve a property from the hostinfo dictionary.

Example

import synapse.lib.thishost as s_thishost if s_thishost.get(‘platform’) == ‘windows’: dostuff() synapse.lib.thishost.hostaddr(dest=’8.8.8.8’) Retrieve the ipv4 address for this host ( optionally as seen from dest ). .. rubric:: Example addr = s_socket.hostaddr() synapse.lib.thisplat module synapse.lib.threads module class synapse.lib.threads.Pool(size=3, maxsize=None) Bases: synapse.eventbus.EventBus A thread pool for firing and cleaning up threads. The Pool() class can be used to keep persistant threads for work processing as well as optionally spin up new threads to handle “bursts” of activity. # fixed pool of 16 worker threads pool = Pool(size=16) # dynamic pool of 5-10 workers pool = Pool(size=5, maxsize=10) # dynamic pool of 8- workers pool = Pool(size=8, maxsize=-1) call(func, *args, **kwargs) Call the given func(*args,**kwargs) in the pool. wrap(func) Wrap a function to transparently dispatch via the pool.

314 Chapter 6. synapse synapse Documentation, Release 0.1.0

Example

# dispatch the message handler from a pool bus.on(‘foo’, pool.wrap( doFooThing ) ) class synapse.lib.threads.RWLock Bases: object A multi-reader/exclusive-writer lock. reader() Acquire a multi-reader lock.

Example

lock = RWLock() with lock.reader(): # other readers can be here too. . . dowrites() release(holder) Used to release an RWWith holder ( you probably shouldn’t use this ) writer() Acquire an exclusive-write lock.

Example

lock = RWLock() with lock.writer(): # no readers or other writers but us! dowrites() class synapse.lib.threads.RWWith(rwlock) Bases: object The RWWith class implements “with block” syntax for RWLock. class synapse.lib.threads.RetnWait Bases: object Emulate synchronous callback waiting with a thread local event.

Example

Do a thing in a thread and wait for the thread to return: with retnwait() as retn: dothing(callback=retn.retn) isset, valu = retn.wait(timeout=3) capture(*args, **kwargs) This can be used as a generic callback function to capture callback arguments.

Notes

This will capture the args and kwargs passed to it. This sets the retn_evnt under the hood, so a caller which is blocked on a wait() call will return the *args, **kwargs. Returns None

6.1. synapse package 315 synapse Documentation, Release 0.1.0

errx(exc) Set the exception information for the current RetnWait object. Parameters exc (Exception) – An Exception, or an Exception subclass.

Notes

This is used by a caller to signal that an exception has occured. This sets the retn_evnt under the hood, so a caller which is blocked on a wait() call will return the excfo tufo. Returns None retn(valu) An ease-of-use API for single value callbacks. Parameters valu (object) – The object to set the return value too.

Notes

This sets the retn_evnt under the hood, so a caller which is blocked on a wait() call will return the valu. Returns None wait(timeout=None) Wait for an async callback to complete. Parameters timeout (int/float) – Timeout in seconds. Returns A Boolean flag indicating if the operation finished or had a timeout or error condition set. The object is either the return value from the callback or an excfo tufo. Return type ((bool, object)) class synapse.lib.threads.SyncTask Bases: object done(valu) err(e) wait(timeout=None) class synapse.lib.threads.Thread(func, *args, **kwargs) Bases: threading.Thread, synapse.eventbus.EventBus A thread / EventBus to allow fini() etc. run() Method representing the thread’s activity. You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively. synapse.lib.threads.current() synapse.lib.threads.iden() synapse.lib.threads.isfini() synapse.lib.threads.worker(func, *args, **kwargs) Fire a worker thread to run the given func(*args,**kwargs)

316 Chapter 6. synapse synapse Documentation, Release 0.1.0 synapse.lib.time module

Time related utilities for synapse “epoch millis” time values. synapse.lib.time.delta(text) Parse a simple time delta string and return the delta. synapse.lib.time.ival(*times) synapse.lib.time.parse(text, base=None, chop=False) Parse a time string into an epoch millis value. synapse.lib.time.repr(tick, pack=False) Return a date string for an epoch-millis timestamp. Parameters tick (int) – The timestamp in milliseconds since the epoch. Returns A date time string Return type (str) synapse.lib.trees module

A simple implementation of an interval tree to lookup potentially overlapping intervals from a point. class synapse.lib.trees.IntervalTree(ivals) Bases: object Construct an interval tree from the inputs. https://en.wikipedia.org/wiki/Interval_tree

Example

ivals = ( ((1,30),{}), ((18,33),{}), . . . ) itree = IntervalTree(ivals) for ival in itree.get(12): dostuff(ival) get(valu) Return intervals which contain the specified value.

Example

for ival in itree.get(valu): dostuff(ival) class synapse.lib.trees.Tree Bases: object A simple step tree with deep and shallow access. first(path) Return the first non-None value in the path. get(path) Return the value for the node at the given path. last(path) Return the last non-None value in the path.

6.1. synapse package 317 synapse Documentation, Release 0.1.0

put(path, valu) synapse.lib.trigger module class synapse.lib.trigger.Triggers(core) Bases: object class Rule(ver: int, cond: str, user: str, storm: str, form: Union[str, NoneType] = None, tag: Union[str, NoneType] = None, prop: Union[str, NoneType] = None) Bases: object en() execute(node, vars=None) Actually execute the query form = None prop = None tag = None add(username, condition, query, info) delete(iden) get(iden) list() mod(iden, query) runNodeAdd(node) runNodeDel(node) runPropSet(node, prop, oldv) runTagAdd(node, tag) runTagDel(node, tag) synapse.lib.types module class synapse.lib.types.Bool(modl, name, info, opts) Bases: synapse.lib.types.Type indx(norm) Return the property index bytes for the given normalized value. postTypeInit() repr(valu, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.lib.types.Comp(modl, name, info, opts) Bases: synapse.lib.types.Type getCompOffs(name) If this type is a compound, return the field offset for the given property name or None.

318 Chapter 6. synapse synapse Documentation, Release 0.1.0

indx(norm) Return the property index bytes for the given normalized value. postTypeInit() repr(valu, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.lib.types.Data(modl, name, info, opts) Bases: synapse.lib.types.Type indx(norm) Return the property index bytes for the given normalized value. norm(valu) Normalize the value for a given type. Parameters valu (obj) – The value to normalize. Returns The normalized valu, info tuple. Return type ((obj,dict))

Notes

The info dictionary uses the following key conventions: subs (dict): The normalized sub-fields as name: valu entries.

class synapse.lib.types.Edge(modl, name, info, opts) Bases: synapse.lib.types.Type getCompOffs(name) If this type is a compound, return the field offset for the given property name or None. indx(norm) Return the property index bytes for the given normalized value. postTypeInit() repr(norm, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.lib.types.FieldHelper(modl, fields) Bases: collections.defaultdict Helper for Comp types. Performs Type lookup/creation upon first use. class synapse.lib.types.Guid(modl, name, info, opts) Bases: synapse.lib.types.Type indx(norm) Return the property index bytes for the given normalized value. postTypeInit() class synapse.lib.types.Hex(modl, name, info, opts) Bases: synapse.lib.types.Type indx(norm) Return the property index bytes for the given normalized value.

6.1. synapse package 319 synapse Documentation, Release 0.1.0

indxByEq(valu) postTypeInit() class synapse.lib.types.Int(modl, name, info, opts) Bases: synapse.lib.types.IntBase indx(valu) Return the property index bytes for the given normalized value. merge(oldv, newv) Allow types to “merge” data from two sources based on value precidence. Parameters • valu (object) – The current value. • newv (object) – The updated value. Returns The merged value. Return type (object) postTypeInit() repr(norm, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.lib.types.IntBase(modl, name, info, opts) Bases: synapse.lib.types.Type indxByGe(valu) indxByGt(valu) indxByLe(valu) indxByLt(valu) class synapse.lib.types.Ival(modl, name, info, opts) Bases: synapse.lib.types.Type An interval, i.e. a range, of times getLiftOps(tabl, cmpr, oper) If this type has special lift operations it needs to do (like a regex search), that will be handled by a sub class. Base types with no special needs can let the Prop/Univ/Form classes handle the generic lift case. indx(norm) Return the property index bytes for the given normalized value. merge(oldv, newv) Allow types to “merge” data from two sources based on value precidence. Parameters • valu (object) – The current value. • newv (object) – The updated value. Returns The merged value. Return type (object) postTypeInit()

320 Chapter 6. synapse synapse Documentation, Release 0.1.0

repr(norm, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.lib.types.Loc(modl, name, info, opts) Bases: synapse.lib.types.Type indx(norm) Return the property index bytes for the given normalized value. indxByPref(valu) postTypeInit() stems() class synapse.lib.types.Ndef(modl, name, info, opts) Bases: synapse.lib.types.Type indx(norm) Return the property index bytes for the given normalized value. postTypeInit() class synapse.lib.types.NodeProp(modl, name, info, opts) Bases: synapse.lib.types.Type indx(norm) Return the property index bytes for the given normalized value. postTypeInit() class synapse.lib.types.Range(modl, name, info, opts) Bases: synapse.lib.types.Type indx(norm) Return the property index bytes for the given normalized value. postTypeInit() repr(norm, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.lib.types.Str(modl, name, info, opts) Bases: synapse.lib.types.StrBase indxByPref(valu) postTypeInit() class synapse.lib.types.StrBase(modl, name, info, opts) Bases: synapse.lib.types.Type Base class for types which index/behave like strings. indx(norm) Return the property index bytes for the given normalized value. indxByPref(valu) postTypeInit() class synapse.lib.types.Tag(modl, name, info, opts) Bases: synapse.lib.types.StrBase

6.1. synapse package 321 synapse Documentation, Release 0.1.0

indxByPref(valu) postTypeInit() class synapse.lib.types.Time(modl, name, info, opts) Bases: synapse.lib.types.IntBase getTickTock(vals) Get a tick, tock time pair. Parameters vals (list) – A pair of values to norm. Returns A pair of integers, sorted so that it the first is less than or equal to the second int. Return type (int, int) indx(norm) Return the property index bytes for the given normalized value. indxByEq(valu) indxByIval(valu) indxByRange(valu) Override default *range= handler to account for relative computation. merge(oldv, newv) Allow types to “merge” data from two sources based on value precidence. Parameters • valu (object) – The current value. • newv (object) – The updated value. Returns The merged value. Return type (object) postTypeInit() repr(valu, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.lib.types.TimeEdge(modl, name, info, opts) Bases: synapse.lib.types.Edge getCompOffs(name) If this type is a compound, return the field offset for the given property name or None. postTypeInit() repr(norm, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.lib.types.Type(modl, name, info, opts) Bases: object clone(opts) Create a new instance of this type with the specified options. Parameters opts (dict) – The type specific options for the new instance. cmpr(val1, name, val2) Compare the two values using the given type specific comparator.

322 Chapter 6. synapse synapse Documentation, Release 0.1.0

extend(name, opts, info) Extend this type to construct a sub-type. Parameters • name (str) – The name of the new sub-type. • opts (dict) – The type options for the sub-type. • info (dict) – The type info for the sub-type. Returns A new sub-type instance. Return type (synapse.types.Type) getCmprCtor(name) getCompOffs(name) If this type is a compound, return the field offset for the given property name or None. getIndxOps(valu, cmpr=’=’) Return a list of index operation tuples to lift values in a table. Valid index operations include: (‘eq’, ) (‘pref’, ) (‘range’, (, )) getLiftHintCmpr(valu, cmpr) getLiftHintCmprCtor(name) getLiftOps(tabl, cmpr, oper) If this type has special lift operations it needs to do (like a regex search), that will be handled by a sub class. Base types with no special needs can let the Prop/Univ/Form classes handle the generic lift case. getTypeVals(valu) indx(norm) Return the property index bytes for the given normalized value. indxByEq(valu) indxByIn(vals) indxByRange(valu) merge(oldv, newv) Allow types to “merge” data from two sources based on value precidence. Parameters • valu (object) – The current value. • newv (object) – The updated value. Returns The merged value. Return type (object) norm(valu) Normalize the value for a given type. Parameters valu (obj) – The value to normalize. Returns The normalized valu, info tuple. Return type ((obj,dict))

6.1. synapse package 323 synapse Documentation, Release 0.1.0

Notes

The info dictionary uses the following key conventions: subs (dict): The normalized sub-fields as name: valu entries.

pack() postTypeInit() repr(norm, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. setCmprCtor(name, func) Set a comparator ctor for a given named comparison operation. Parameters • name (str) – Name of the comparison operation. • func – Function which returns a comparator.

Notes

Comparator ctors should expect to get the right-hand-side of the comparison as their argument, and the returned function should expect to get the left hand side of the comparison and return a boolean from there. setLiftHintCmprCtor(name, func) setNormFunc(typo, func) Register a normalizer function for a given python type. Parameters • typo (type) – A python type/class to normalize. • func (function) – A callback which normalizes a python value. synapse.lib.urlhelp module synapse.lib.urlhelp.chopurl(url) A sane “stand alone” url parser.

Example

info = chopurl(url) synapse.lib.version module

Synapse utilites for dealing with Semvar versioning. This includes the Synapse version information. synapse.lib.version.fmtVersion(*vsnparts) Join a string of parts together with a . separator. Parameters *vsnparts – Returns:

324 Chapter 6. synapse synapse Documentation, Release 0.1.0

synapse.lib.version.packVersion(major, minor=0, patch=0) Pack a set of major/minor/patch integers into a single integer for storage. Parameters • major (int) – Major version level integer. • minor (int) – Minor version level integer. • patch (int) – Patch version level integer. Returns System normalized integer value to represent a software version. Return type int synapse.lib.version.parseSemver(text) Parse a Semantic Version string into is component parts. Parameters • text (str) – A text string to parse into semver components. This string has whitespace and leading ‘v’ • stripped off of it. (characters)–

Examples

Parse a string into it semvar parts:

parts= parseSemver('v1.2.3')

Returns The dictionary will contain the keys ‘major’, ‘minor’ and ‘patch’ pointing to integer values. The dictionary may also contain keys for ‘build’ and ‘pre’ information if that data is parsed out of a semver string. None is returned if the string is not a valid Semver string. Return type dict

synapse.lib.version.parseVersionParts(text, seps=(’.’, ’-’, ’_’, ’+’)) Extract a list of major/minor/version integer strings from a string. Parameters • text (str) – String to parse • seps (tuple) – A tuple or list of separators to use when parsing the version string.

Examples

Parse a simple version string into a major and minor parts:

parts= parseVersionParts('1.2')

Parse a complex version string into a major and minor parts:

parts= parseVersionParts('wowsoft_1.2')

Parse a simple version string into a major, minor and patch parts. Parts after the “3.” are dropped from the results:

6.1. synapse package 325 synapse Documentation, Release 0.1.0

parts= parseVersionParts('1.2.3.4.5')

Notes

This attempts to brute force out integers from the version string by stripping any leading ascii letters and part separators, and then regexing out numeric parts optionally followed by part separators. It will stop at the first mixed-character part encountered. For example, “1.2-3a” would only parse out the “1” and “2” from the string. Returns Either a empty dictionary or dictionary containing up to three keys, ‘major’, ‘minor’ and ‘patch’. Return type dict synapse.lib.version.unpackVersion(ver) Unpack a system normalized integer representing a softare version into its component parts. Parameters ver (int) – System normalized integer value to unpack into a tuple. Returns A tuple containing the major, minor and patch values shifted out of the integer. Return type (int, int, int)

synapse.lookup package

Submodules

synapse.lookup.iana module

synapse.lookup.iso3166 module

Provides data for the ISO 3166-1 Country codes. Reference: https://en.wikipedia.org/wiki/ISO_3166 synapse.lookup.iso3166.makeColLook(rows, scol, dcol)

synapse.lookup.pe module

synapse.lookup.pe.getLangCodes() synapse.lookup.pe.getRsrcTypes() synapse.lookup.phonenum module synapse.lookup.phonenum.formPhoneNode(node, valu) synapse.lookup.phonenum.getPhoneInfo(numb) Walk the phone info tree to find the best-match info for the given number.

326 Chapter 6. synapse synapse Documentation, Release 0.1.0

Example

info = getPhoneInfo(17035551212) country = info.get(‘cc’) synapse.lookup.phonenum.initPhoneTree() synapse.lookup.phonenum.phnode(valu) synapse.models package

Subpackages synapse.models.gov package

Submodules synapse.models.gov.cn module class synapse.models.gov.cn.GovCnModule(core) Bases: synapse.lib.module.CoreModule getModelDefs() synapse.models.gov.intl module class synapse.models.gov.intl.GovIntlModule(core) Bases: synapse.lib.module.CoreModule getModelDefs() synapse.models.gov.us module class synapse.models.gov.us.GovUsModule(core) Bases: synapse.lib.module.CoreModule getModelDefs()

Submodules synapse.models.base module class synapse.models.base.BaseModule(core) Bases: synapse.lib.module.CoreModule getModelDefs() synapse.models.crypto module class synapse.models.crypto.CryptoModule(core) Bases: synapse.lib.module.CoreModule

6.1. synapse package 327 synapse Documentation, Release 0.1.0

getModelDefs() synapse.models.dns module class synapse.models.dns.DnsModule(core) Bases: synapse.lib.module.CoreModule getModelDefs() class synapse.models.dns.DnsName(modl, name, info, opts) Bases: synapse.lib.types.StrBase indxByPref(valu) postTypeInit() synapse.models.files module class synapse.models.files.FileBase(modl, name, info, opts) Bases: synapse.lib.types.StrBase indxByPref(valu) postTypeInit() class synapse.models.files.FileBytes(modl, name, info, opts) Bases: synapse.lib.types.Type indx(norm) Return the property index bytes for the given normalized value. postTypeInit() class synapse.models.files.FileModule(core) Bases: synapse.lib.module.CoreModule getModelDefs() initCoreModule() Module implementers may over-ride this method to initialize the module during initial construction. Any exception raised within this method will be raised from the constructor and mark the module as failed. Args: Returns None class synapse.models.files.FilePath(modl, name, info, opts) Bases: synapse.lib.types.StrBase indxByPref(valu) postTypeInit() synapse.models.geopol module class synapse.models.geopol.PolModule(core) Bases: synapse.lib.module.CoreModule getModelDefs()

328 Chapter 6. synapse synapse Documentation, Release 0.1.0

synapse.models.geospace module

class synapse.models.geospace.Dist(modl, name, info, opts) Bases: synapse.lib.types.Type indx(norm) Return the property index bytes for the given normalized value. postTypeInit() repr(norm) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.models.geospace.GeoModule(core) Bases: synapse.lib.module.CoreModule getModelDefs() class synapse.models.geospace.LatLong(modl, name, info, opts) Bases: synapse.lib.types.Type indx(valu) Return the property index bytes for the given normalized value. postTypeInit() repr(norm, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.models.geospace.Latitude(modl, name, info, opts) Bases: synapse.lib.types.Type SCALE = 100000000 SPACE = 9000000000 indx(norm) Return the property index bytes for the given normalized value. postTypeInit() class synapse.models.geospace.Longitude(modl, name, info, opts) Bases: synapse.lib.types.Type SCALE = 100000000 SPACE = 18000000000 indx(norm) Return the property index bytes for the given normalized value. postTypeInit() synapse.models.inet module class synapse.models.inet.Addr(modl, name, info, opts) Bases: synapse.lib.types.StrBase postTypeInit()

6.1. synapse package 329 synapse Documentation, Release 0.1.0 class synapse.models.inet.Cidr4(modl, name, info, opts) Bases: synapse.lib.types.StrBase postTypeInit() class synapse.models.inet.Email(modl, name, info, opts) Bases: synapse.lib.types.StrBase postTypeInit() class synapse.models.inet.Fqdn(modl, name, info, opts) Bases: synapse.lib.types.Type indx(norm) Return the property index bytes for the given normalized value. indxByEq(valu) postTypeInit() repr(valu, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.models.inet.IPv4(modl, name, info, opts) Bases: synapse.lib.types.Type The base type for an IPv4 address. getCidrRange(text) getNetRange(text) getTypeVals(text) indx(norm) Return the property index bytes for the given normalized value. indxByEq(valu) postTypeInit() repr(norm, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.models.inet.IPv6(modl, name, info, opts) Bases: synapse.lib.types.Type indx(norm) Return the property index bytes for the given normalized value. postTypeInit() class synapse.models.inet.IPv6Range(modl, name, info, opts) Bases: synapse.lib.types.Range postTypeInit() class synapse.models.inet.InetModule(core) Bases: synapse.lib.module.CoreModule getModelDefs()

330 Chapter 6. synapse synapse Documentation, Release 0.1.0

initCoreModule() Module implementers may over-ride this method to initialize the module during initial construction. Any exception raised within this method will be raised from the constructor and mark the module as failed. Args: Returns None class synapse.models.inet.Rfc2822Addr(modl, name, info, opts) Bases: synapse.lib.types.StrBase An RFC 2822 compatible email address parser indxByPref(valu) postTypeInit() class synapse.models.inet.Url(modl, name, info, opts) Bases: synapse.lib.types.StrBase postTypeInit() synapse.models.inet.getAddrType(ip) synapse.models.infotech module class synapse.models.infotech.ItModule(core) Bases: synapse.lib.module.CoreModule bruteVersionStr(valu) Brute force the version out of a string. Parameters valu (str) – String to attempt to get version information for.

Notes

This first attempts to parse strings using the it:semver normalization before attempting to extract version parts out of the string. Returns The system normalized version integer and a subs dictionary. Return type int, dict getModelDefs() initCoreModule() Module implementers may over-ride this method to initialize the module during initial construction. Any exception raised within this method will be raised from the constructor and mark the module as failed. Args: Returns None class synapse.models.infotech.SemVer(modl, name, info, opts) Bases: synapse.lib.types.Type Provides support for parsing a semantic version string into its component parts. This normalizes a version string into an integer to allow version ordering. Prerelease information is disregarded for integer comparison purposes, as we cannot map an arbitrary pre-release version into a integer value Major, minor and patch levels are represented as integers, with a max width of 20 bits. The comparable integer value representing the semver is the bitwise concatenation of the major, minor and patch levels.

6.1. synapse package 331 synapse Documentation, Release 0.1.0

Prerelease and build information will be parsed out and available as strings if that information is present. indx(valu) Return the property index bytes for the given normalized value. postTypeInit() repr(valu, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. synapse.models.language module class synapse.models.language.LangModule(core) Bases: synapse.lib.module.CoreModule getModelDefs() synapse.models.material module

A data model focused on material objects. class synapse.models.material.MatModule(core) Bases: synapse.lib.module.CoreModule getModelDefs() synapse.models.media module class synapse.models.media.MediaModule(core) Bases: synapse.lib.module.CoreModule getModelDefs() synapse.models.orgs module class synapse.models.orgs.OuModule(core) Bases: synapse.lib.module.CoreModule getModelDefs() synapse.models.person module class synapse.models.person.PsModule(core) Bases: synapse.lib.module.CoreModule getModelDefs() synapse.models.syn module class synapse.models.syn.SynModule(core) Bases: synapse.lib.module.CoreModule

332 Chapter 6. synapse synapse Documentation, Release 0.1.0

getModelDefs() initCoreModule() Module implementers may over-ride this method to initialize the module during initial construction. Any exception raised within this method will be raised from the constructor and mark the module as failed. Args: Returns None synapse.models.telco module class synapse.models.telco.Imei(modl, name, info, opts) Bases: synapse.lib.types.Type indx(valu) Parameters valu (int)– Returns Return type bytes postTypeInit() class synapse.models.telco.Imsi(modl, name, info, opts) Bases: synapse.lib.types.Type indx(valu) Parameters valu (int)– Returns Return type bytes postTypeInit() class synapse.models.telco.Phone(modl, name, info, opts) Bases: synapse.lib.types.Type indx(valu) Parameters valu (str) – Value to encode Returns Encoded value Return type bytes indxByEq(valu) postTypeInit() repr(valu, defval=None) Return a printable representation for the value. For types which need no display processing, the normalized value should be returned. class synapse.models.telco.TelcoModule(core) Bases: synapse.lib.module.CoreModule getModelDefs() synapse.models.telco.chop_imei(imei) synapse.models.telco.digits(text)

6.1. synapse package 333 synapse Documentation, Release 0.1.0 synapse.models.telco.imeicsum(text) Calculate the imei check byte. synapse.servers package

Submodules synapse.servers.cortex module synapse.servers.cortex.main(argv, outp=) synapse.servers.cortex.mainopts(opts, outp=) synapse.tools package

Subpackages synapse.tools.cryo package

Submodules synapse.tools.cryo.cat module synapse.tools.cryo.cat.main(argv, outp=) synapse.tools.cryo.list module synapse.tools.cryo.list.main(argv, outp=)

Submodules synapse.tools.autodoc module class synapse.tools.autodoc.DocHelp(ctors, types, forms, props, univs) Bases: object Helper to pre-compute all doc strings hierarchically class synapse.tools.autodoc.RstHelp Bases: object addHead(name, lvl=0, link=None) addLines(*lines) getRstText() synapse.tools.autodoc.docModel(outp, core) synapse.tools.autodoc.main(argv, outp=None) synapse.tools.autodoc.makeargparser()

334 Chapter 6. synapse synapse Documentation, Release 0.1.0

synapse.tools.autodoc.processCtors(rst, dochelp, ctors) Parameters • rst (RstHelp)– • dochelp (DocHelp)– • ctors (list)– Returns None synapse.tools.autodoc.processFormsProps(rst, dochelp, forms) synapse.tools.autodoc.processTypes(rst, dochelp, types) Parameters • rst (RstHelp)– • dochelp (DocHelp)– • ctors (list)– Returns None synapse.tools.autodoc.processUnivs(rst, dochelp, univs)

synapse.tools.backup module

synapse.tools.backup.backup(srcdir, dstdir) synapse.tools.backup.backup_lmdb(envpath, dstdir) synapse.tools.backup.main(argv) synapse.tools.backup.parse_args(argv) synapse.tools.cellauth module synapse.tools.cellauth.handleList(opts) synapse.tools.cellauth.handleModify(opts) synapse.tools.cellauth.main(argv, outprint=None) synapse.tools.cellauth.makeargparser() synapse.tools.cellauth.printuser(user) synapse.tools.cellauth.reprrule(rule) synapse.tools.cmdr module synapse.tools.cmdr.main(argv) synapse.tools.csvtool module synapse.tools.csvtool.main(argv, outp=) synapse.tools.csvtool.makeargparser()

6.1. synapse package 335 synapse Documentation, Release 0.1.0

synapse.tools.deploy module

synapse.tools.deploy.main(argv, outp=None) synapse.tools.deploy.makeargpaser()

synapse.tools.dmon module

synapse.tools.dmon.getArgParser() synapse.tools.dmon.main(argv, outp=)

synapse.tools.easycert module

synapse.tools.easycert.main(argv, outp=None)

synapse.tools.feed module

synapse.tools.feed.addFeedData(core, outp, feedformat, debug=False, *paths, chunksize=1000, offset=0) synapse.tools.feed.getItems(*paths) synapse.tools.feed.main(argv, outp=None) synapse.tools.feed.makeargparser() synapse.tools.guid module synapse.tools.guid.main(argv, outp=None) synapse.tools.json2mpk module synapse.tools.json2mpk.getArgParser() synapse.tools.json2mpk.main(argv, outp=None) synapse.tools.pullfile module synapse.tools.pullfile.main(argv, outp=None) synapse.tools.pullfile.setup() synapse.tools.pushfile module synapse.tools.pushfile.main(argv, outp=None) synapse.tools.pushfile.makeargparser()

336 Chapter 6. synapse synapse Documentation, Release 0.1.0

6.1.2 Submodules

6.1.3 synapse.axon module class synapse.axon.Axon Bases: synapse.lib.cell.Cell addBlobStor(blobstorpath) Causes an axon to start using a particular blobstor. This is persistently stored; on Axon restart, it will automatically reconnect to the blobstor at the specified path. bulkput(files, proxykeeper=None) Save a list of files to the axon. Parameters files ([bytes]) – A list of files as bytes blobs. Returns The number of files saved. Return type int cellapi alias of AxonApi confdefs = (('mapsize', {'type': 'int', 'defval': 1099511627776, 'doc': 'The size of the LMDB memory map'}),) get(hashval, proxykeeper=None) Yield bytes for the given SHA256. Parameters hashval (str) – The SHA256 hash bytes. Yields bytes – Bytes of the file requested. Raises RetnErr – If the file requested does not exist. getBlobStors() Returns A list of all the watched blobstors locs(hashval) Get the blobstor bsids for a given sha256 value Parameters hashval (bytes) – The sha256 digest to look up Returns A list of BlobStor IDs Return type list metrics(offs=0) putone(bytz, hashval=None, proxykeeper=None) If hashval is None and or not None and not already in the axon, stores bytz as a single blob Returns The hash of bytz Return type bytes stat() unwatchBlobStor(blobstorpath) Cause an axon to stop using a particular blobstor by path. This is persistently stored. wants(hashvals) Given a list of hashvals, returns a list of the hashes not available class synapse.axon.AxonApi Bases: synapse.lib.cell.PassThroughApi

6.1. synapse package 337 synapse Documentation, Release 0.1.0

allowed_methods = ['get', 'locs', 'stat', 'wants', 'metrics', 'putone', 'addBlobStor', 'unwatchBlobStor', 'getBlobStors'] get(hashval) startput() class synapse.axon.BlobStor Bases: synapse.lib.cell.Cell The blob store maps sha256 values to sequences of bytes stored in a LMDB database. bulkput(blocs) Save items from an iterator of (sha256, chunk #, ). Parameters • blocs – An iterator of (sha256, chunk #, ). Every 0 chunk # represents a new file. • sha256 must be None, except if the chunk # is 0, in which it may optionally be a sha256 hash. If (The)– • and the blobstor contains a value with a matching hash, all bytes will be skipped until the next (present)– • with a chunk # of 0 is encountered. (bloc)– Returns None cellapi alias of BlobStorApi clone(offset: int, include_contents=True, timeout=0) Yield (offset, (sha256, chunknum, bytes)) tuples to clone this BlobStor. Parameters • offset (int) – Offset to start yielding rows from. • include_contents (bool) – Whether to include the blob value in the results stream Yields ((bytes, (bytes, int, bytes))) – tuples of (index, (sha256,chunknum,bytes)) data. confdefs = (('mapsize', {'type': 'int', 'doc': 'LMDB mapsize value', 'defval': 1099511627776}), ('cloneof', {'type': 'str', 'doc': 'The telepath of a blob cell to clone from', 'defval': None})) get(hashval) Load and yield the bytes blocks for a given hash. Parameters hashval (bytes) – Hash to retrieve bytes for. getCloneProgress() Get the next offset to retrieve for the clone:index of the BlobStor. Returns The offset value Return type int metrics(offs=0) Get metrics for the BlobStor. These can be aggregated to compute the storage stats. Parameters offs (int) – Offset to start collecting stats from. Yields ((int, dict)) – Yields index, sample data from the metrics sequence. putmany(items) putone(item)

338 Chapter 6. synapse synapse Documentation, Release 0.1.0

stat() Get storage stats for the BlobStor. Returns A dictionary containing the total bytes and blocks store in the BlobStor. Return type dict class synapse.axon.BlobStorApi Bases: synapse.lib.cell.PassThroughApi allowed_methods = ['clone', 'stat', 'metrics', 'offset', 'bulkput', 'putone', 'putmany', 'get', '_complete', '_cancel', '_partialsubmit', 'getCloneProgress'] startput() class synapse.axon.IncrementalTransaction(lenv) Bases: synapse.eventbus.EventBus An lmdb write transaction that commits if the number of outstanding bytes to be commits grows too large. Naturally, this breaks transaction atomicity. MAX_OUTSTANDING = 1073741824 commit() cursor(db=None) guarantee() Make an LMDB transaction if we don’t already have kone, and return it put(key, value, db) Write data to the database, committing if too many bytes are uncommitted class synapse.axon.Uploader Bases: synapse.lib.share.Share A remotely shareable object used for streaming uploads to a blobstor cancelFile() Cancel the current blob. Will still listen for new blobs. finish() Conclude an uploading session finishFile() Finish and commit the existing file, keeping the uploader active for more files. typename = 'uploader' write(bytz) Upload some data Parameters bytz (bytes) – a chunk of data. It does not have to be an entire blob. class synapse.axon.UploaderProxy Bases: synapse.lib.share.Share A proxy to a blobstor uploader living with the axon finish() finishFile() typename = 'uploaderproxy' write(bytz)

6.1. synapse package 339 synapse Documentation, Release 0.1.0

synapse.axon.to_aiter(it) Take either a sync or async iteratable and yields as an async generator

6.1.4 synapse.cells module

Constructors for the various cells. ( used for dmon config automation) synapse.cells.add(name, ctor) Add a Cell() constructor alias. Parameters • name (str) – Name of the cell alias. • ctor – Function used to create the Cell().

Notes

Third party modules which implement synapse.lib.cell.Cell classes should import synapse. cells and register an alias and class path for their Cell using this function. This can be done in a module __init__.py file. synapse.cells.deploy(name, dirn, boot=None) Deploy a cell of the named type to the specified directory. synapse.cells.getCells() Get a list of registered cell aliases and their fully qualified paths. synapse.cells.init(name, dirn, *args, **kwargs) Initialize and return a Cell() object by alias. synapse.cells.initFromDirn(dirn, *args, **kwargs) As above, but retrieves type from boot.yaml in dirn

6.1.5 synapse.common module

class synapse.common.NoValu Bases: object synapse.common.addpref(pref, info) Add the given prefix to all elements in the info dict. synapse.common.aspin(genr) Async version of spin synapse.common.buid(valu=None) A binary GUID like sequence of 32 bytes. Parameters • valu (object) – Optional, if provided, the hash of the msgpack • form of the object is returned. This can be used to (encoded)– • stable buids. (create)–

340 Chapter 6. synapse synapse Documentation, Release 0.1.0

Notes

By default, this returns a random 32 byte value. Returns A 32 byte value. Return type bytes synapse.common.chunks(item, size) Divide an iterable into chunks. Parameters • item – Item to slice • size (int) – Maximum chunk size.

Notes

This supports Generator objects and objects which support calling the __getitem__() method with a slice object. Yields Slices of the item containing up to “size” number of items. synapse.common.config(conf, confdefs) Initialize a config dict using the given confdef tuples. synapse.common.debase64(b) synapse.common.ehex(byts) Encode a set of bytes to a string using binascii.hexlify. Parameters byts (bytes) – Bytes to encode. Returns A string representing the bytes. Return type str synapse.common.enbase64(b) synapse.common.err(e) synapse.common.errinfo(name, mesg) synapse.common.excinfo(e) Populate err,errmsg,errtrace info from exc. synapse.common.firethread(f ) A decorator for making a function fire a thread. synapse.common.gendir(*paths, **opts) synapse.common.genfile(*paths) Create or open ( for read/write ) a file path join. Parameters *paths – A list of paths to join together to make the file.

Notes

If the file already exists, the fd returned is opened in r+b mode. Otherwise, the fd is opened in w+b mode. Returns A file-object which can be read/written too. Return type io.BufferedRandom

6.1. synapse package 341 synapse Documentation, Release 0.1.0 synapse.common.genpath(*paths) synapse.common.gentask(func, *args, **kwargs) synapse.common.getSynDir(*paths) synapse.common.getSynPath(*paths) synapse.common.getTempDir() synapse.common.getbytes(*paths, **opts) synapse.common.getexcfo(e) Get an err tufo from an exception. Parameters e (Exception) – An Exception (or Exception subclass).

Notes

This can be called outside of the context of an exception handler, however details such as file, line, function name and source may be missing. Returns Return type ((str, dict)) synapse.common.getfile(*paths, **opts) synapse.common.guid(valu=None) Get a 16 byte guid value. By default, this is a random guid value. Parameters valu – Object used to construct the guid valu from. This must be able to be msgpack’d. Returns 32 character, lowercase ascii string. Return type str synapse.common.int64en(i) Encode a 64-bit int into 8 byte big-endian bytes synapse.common.int64un(b) Decode a 64-bit int from 8 byte big-endian synapse.common.intify(x) Ensure ( or coerce ) a value into being an integer or None. Parameters x (obj) – An object to intify Returns The int value ( or None ) Return type (int) synapse.common.isguid(text) synapse.common.iterfd(fd, size=10000000) Generator which yields bytes from a file descriptor. Parameters • fd (file) – A file-like object to read bytes from. • size (int) – Size, in bytes, of the number of bytes to read from the • at a given time. (fd)–

342 Chapter 6. synapse synapse Documentation, Release 0.1.0

Notes

If the first read call on the file descriptor is a empty bytestring, that zero length bytestring will be yielded and the generator will then be exhuasted. This behavior is intended to allow the yielding of contents of a zero byte file. Yields bytes – Bytes from the file descriptor. synapse.common.iterzip(*args) synapse.common.jsload(*paths) synapse.common.jssave(js, *paths) synapse.common.listdir(*paths, glob=None) List the (optionally glob filtered) full paths from a dir. Parameters • *paths ([str,..]) – A list of path elements • glob (str) – An optional fnmatch glob str synapse.common.lockfile(path) A file lock with-block helper. Parameters path (str) – A path to a lock file.

Examples

Get the lock on a file and dostuff while having the lock: path = ‘/hehe/haha.lock’ with lockfile(path): dostuff()

Notes

This is curently based on fcntl.lockf(), and as such, it is purely advisory locking. If multiple processes are attempting to obtain a lock on the same file, this will block until the process which has the current lock releases it. Yields None synapse.common.makedirs(path, mode=511) synapse.common.now() Get the current epoch time in milliseconds. This relies on time.time(), which is system-dependent in terms of resolution.

Examples

Get the current time and make a row for a Cortex:

tick= now() row= (someiden,'foo:prop',1, tick) core.addRows([row])

Returns Epoch time in milliseconds.

6.1. synapse package 343 synapse Documentation, Release 0.1.0

Return type int

synapse.common.reqbytes(*paths) synapse.common.reqdir(*paths) synapse.common.reqfile(*paths, **opts) synapse.common.reqlines(*paths, **opts) Open a file and yield lines of text.

Example

for line in reqlines(‘foo.txt’): dostuff(line) NOTE: This API is used as a performance optimization over the standard fd line iteration mechanism.

synapse.common.reqok(ok, retn) Raise exception from retn if not ok. synapse.common.reqpath(*paths) synapse.common.result(retn) Return a value or raise an exception from a retn tuple. synapse.common.retnexc(e) Construct a retn tuple for the given exception. synapse.common.rowstotufos(rows) Convert rows into tufos. Parameters rows (list) – List of rows containing (i, p, v, t) tuples. Returns List of tufos. Return type list synapse.common.setlogging(mlogger, defval=None) Configure synapse logging. Parameters • mlogger (logging.Logger) – Reference to a logging.Logger() • defval (str) – Default log level

Notes

This calls logging.basicConfig and should only be called once per process. Returns None synapse.common.spin(genr) Crank through a generator but discard the yielded values. Parameters genr – Any generator or iterable valu.

344 Chapter 6. synapse synapse Documentation, Release 0.1.0

Notes

This generator is exhausted via the collections.dequeue() constructor with a maxlen=0, which will quickly exhaust an iterator staying in C code as much as possible. Returns None synapse.common.synerr(excname, **info) Return a SynErr exception. If the given name is not known, fall back on the base class. synapse.common.to_bytes(valu, size) synapse.common.to_int(byts) synapse.common.uhex(text) Decode bytes to a string using binascii.unhexlify. :param text: Text to decode. :type text: str Returns The decoded bytes. Return type bytes synapse.common.verstr(vtup) Convert a version tuple to a string. synapse.common.vertup(vstr) Convert a version string to a tuple.

Example

ver = vertup(‘1.3.30’) synapse.common.worker(meth, *args, **kwargs) synapse.common.yamlload(*paths) synapse.common.yamlmod(obj, *paths) Combines/creates a yaml file and combines with obj. obj and file must be maps/dict or empty. synapse.common.yamlsave(obj, *paths)

6.1.6 synapse.cortex module class synapse.cortex.CoreApi Bases: synapse.lib.cell.CellApi The CoreApi is exposed over telepath. addCronJob(query, reqs, incunit=None, incval=1) Add a cron job to the cortex A cron job is a persistently-stored item that causes storm queries to be run in the future. The specification for the times that the queries run can be one-shot or recurring. Parameters • query (str) – The storm query to execute in the future • reqs (Union[Dict[str, Union[int, List[int]]], List[Dict[.. ]]]) – Either a dict of the fixed time fields or a list of such dicts. The keys are in the set (‘year’, ‘month’, ‘dayofmonth’, ‘dayofweek’, ‘hour’, ‘minute’. The values must be positive integers, except for the key of ‘dayofmonth’ in which it may also be a

6.1. synapse package 345 synapse Documentation, Release 0.1.0

negative integer which represents the number of days from the end of the month with -1 representing the last day of the month. All values may also be lists of valid values. • incunit (Optional[str]) – A member of the same set as above, with an additional member ‘day’. If is None (default), then the appointment is one-shot and will not recur. • incval (Union[int, List[int]) – A integer or a list of integers of the number of units

Returns (bytes): An iden that can be used to later modify, query, and delete the job.

Notes

reqs must have fields present or incunit must not be None (or both) The incunit if not None it must be larger in unit size than all the keys in all reqs elements. addFeedData(name, items, seqn=None) addNode(form, valu, props=None) addNodeTag(iden, tag, valu=(None, None)) Add a tag to a node specified by iden. Parameters • iden (str) – A hex encoded node BUID. • tag (str) – A tag string. • valu (tuple) – A time interval tuple or (None, None). addNodes(nodes) Add a list of packed nodes to the cortex. Parameters nodes (list) – [ ( (form, valu), {‘props’:{}, ‘tags’:{}}), . . . ] Yields (tuple) – Packed node tuples ((form,valu), {‘props’: {}, ‘tags’:{}}) addTrigger(condition, query, *, info) Adds a trigger to the cortex allowed(*path) count(text, opts=None) Count the number of nodes which result from a storm query. Parameters • text (str) – Storm query text. • opts (dict) – Storm query options. Returns The number of nodes resulting from the query. Return type (int) delCronJob(iden) Delete a cron job Parameters iden (bytes) – The iden of the cron job to be deleted delNodeTag(iden, tag) Delete a tag from the node specified by iden. Parameters

346 Chapter 6. synapse synapse Documentation, Release 0.1.0

• iden (str) – A hex encoded node BUID. • tag (str) – A tag string. delTrigger(iden) Deletes a trigger from the cortex eval(text, opts=None) Evalute a storm query and yield packed nodes. getCoreInfo() Return static generic information about the cortex including model definition getCoreMods(**kwargs) getFeedOffs(iden) getModelDict() Return a dictionary which describes the data model. Returns A model description dictionary. Return type (dict) getNodesBy(full, valu, cmpr=’=’) Yield Node.pack() tuples which match the query. listCronJobs() Get information about all the cron jobs accessible to the current user listTriggers() Lists all the triggers that the current user is authorized to access setFeedOffs(**kwargs) setNodeProp(iden, name, valu) splices(**kwargs) stat(**kwargs) storm(text, opts=None) Evaluate a storm query and yield result messages. :Yields: ((str,dict)) – Storm messages. updateCronJob(iden, query) Change an existing cron job’s query Parameters iden (bytes) – The iden of the cron job to be changed updateTrigger(iden, query) Change an existing trigger’s query class synapse.cortex.Cortex Bases: synapse.lib.cell.Cell A Cortex implements the synapse hypergraph. The bulk of the Cortex API lives on the Snap() object which can be obtained by calling Cortex.snap() in a with block. This allows callers to manage transaction boundaries explicitly and dramatically increases performance. addCoreMods(mods) Add a list of (name,conf) module tuples to the cortex. addFeedData(name, items, seqn=None) Add data using a feed/parser function. Parameters

6.1. synapse package 347 synapse Documentation, Release 0.1.0

• name (str) – The name of the feed record format. • items (list) – A list of items to ingest. • seqn ((str,int)) – An (iden, offs) tuple for this feed chunk. Returns The next expected offset (or None) if seqn is None. Return type (int) addNodes(nodedefs) Quickly add/modify a list of nodes from node definition tuples. This API is the simplest/fastest way to add nodes, set node props, and add tags to nodes remotely. Parameters nodedefs (list) – A list of node definition tuples. See below. A node definition tuple is defined as: ( (form, valu), {‘props’:{}, ‘tags’:{}) The “props” or “tags” keys may be omitted. addRuntLift(prop, func) Register a runt lift helper for a given prop. Parameters • prop (str) – Full property name for the prop to register the helper for. • func – Returns None. Return type None addRuntPropDel(prop, func) Register a prop set helper for a runt form addRuntPropSet(prop, func) Register a prop set helper for a runt form addStormCmd(ctor) Add a synapse.lib.storm.Cmd class to the cortex. addStormLib(path, ctor) cellapi alias of CoreApi confdefs = (('layer:lmdb:mapsize', {'type': 'int', 'defval': None, 'doc': 'The default size for a new LMDB layer map.'}), ('modules', {'type': 'list', 'defval': (), 'doc': 'A list of module classes to load.'}), ('storm:log', {'type': 'bool', 'defval': False, 'doc': 'Log storm queries via system logger.'}), ('storm:log:level', {'type': 'int', 'defval': 30, 'doc': 'Logging log level to emit storm logs at.'}), ('splice:sync', {'type': 'str', 'defval': None, 'doc': 'A telepath URL for an upstream cortex.'}), ('splice:cryotank', {'type': 'str', 'defval': None, 'doc': 'A telepath URL for a cryotank used to archive splices.'}), ('feeds', {'type': 'list', 'defval': (), 'doc': 'A list of feed dictionaries.'}), ('cron:enable', {'type': 'bool', 'defval': True, 'doc': 'Enable cron jobs running.'})) eval(text, opts=None, user=None) Evaluate a storm query and yield Nodes only. getCoreInfo() getCoreMod(name) getCoreMods() getFeedFunc(name) Get a data ingest function. getFeedOffs(iden) getNodeByNdef(ndef ) Return a single Node() instance by (form,valu) tuple.

348 Chapter 6. synapse synapse Documentation, Release 0.1.0

getNodesBy(full, valu, cmpr=’=’) Get nodes by a property value or lift syntax. Parameters • full (str) – The full name of a property :. • valu (obj) – A value that the type knows how to lift by. • cmpr (str) – The comparison operator you are lifting by. Some node property types allow special syntax here.

Examples

# simple lift by property equality core.getNodesBy(‘file:bytes:size’, 20) # The inet:ipv4 type knows about cidr syntax core.getNodesBy(‘inet:ipv4’, ‘1.2.3.0/24’) getStormCmd(name) getStormCmds() getStormLib(path) getStormQuery() iterStormPodes(text, opts=None, user=None) loadCoreModule(ctor, conf=None) Load a cortex module with the given ctor and conf. Parameters • ctor (str) – The python module class path • conf (dict) – Config dictionary for the module offTagAdd(name, func) Unregister a callback for tag addition. Parameters • name (str) – The name of the tag. • func (function) – The callback func(node, tagname, tagval). offTagDel(name, func) Unregister a callback for tag deletion. Parameters • name (str) – The name of the tag. • func (function) – The callback func(node, tagname, tagval). onTagAdd(name, func) Register a callback for tag addition. Parameters • name (str) – The name of the tag. • func (function) – The callback func(node, tagname, tagval). onTagDel(name, func) Register a callback for tag deletion.

6.1. synapse package 349 synapse Documentation, Release 0.1.0

Parameters • name (str) – The name of the tag. • func (function) – The callback func(node, tagname, tagval). pokeFormCount(form, valu) runRuntLift(full, valu=None, cmpr=None) Execute a runt lift function. Parameters • full (str) – Property to lift by. • valu – • cmpr – Returns Yields bytes, list tuples where the list contains a series of key/value pairs which are used to construct a Node object. Return type bytes, list runRuntPropDel(node, prop) runRuntPropSet(node, prop, valu) runTagAdd(node, tag, valu) runTagDel(node, tag, valu) setFeedFunc(name, func) Set a data ingest function. def func(snap, items): loaditems. . . setFeedOffs(iden, offs) snap(user=None) Return a transaction object for the default view. Parameters write (bool) – Set to True for a write transaction. Returns (synapse.lib.snap.Snap) NOTE: This must be used in a with block. stat() storm(text, opts=None, user=None) Evaluate a storm query and yield (node, path) tuples. :Yields: (Node, Path) tuples streamstorm(text, opts=None, user=None) Evaluate a storm query and yield result messages. :Yields: ((str,dict)) – Storm messages. class synapse.cortex.View(core, layers) Bases: object A view represents a cortex as seen from a specific set of layers. The view class is used to implement Copy-On-Write layers as well as interact with a subset of the layers con- figured in a Cortex. snap()

350 Chapter 6. synapse synapse Documentation, Release 0.1.0 synapse.cortex.cmdre = regex.Regex('^[\\w\\.]+$', flags=regex.V0) A Cortex implements the synapse hypergraph object. Many Cortex APIs operate on nodes which consist of primitive data structures which can be serialized with msgpack/json Example Node ( (, ), { “props”: { : , . . . }, “tags”: { “foo”:

Notes

The cortex and temporary directory are town down on exit. This should only be called from synchronous code. Returns Proxy to the cortex.

6.1.7 synapse.cryotank module class synapse.cryotank.CryoApi Bases: synapse.lib.cell.CellApi The CryoCell API as seen by a telepath proxy. This is the API to reference for remote CryoCell use. delete(**kwargs) init(name, conf=None) last(name) list() metrics(name, offs, size=None) offset(name, iden) puts(name, items, seqn=None) rows(name, offs, size, iden=None) slice(name, offs, size, iden=None) class synapse.cryotank.CryoCell Bases: synapse.lib.cell.Cell cellapi alias of CryoApi confdefs = (('tankdefaults', {'defval': {}, 'doc': 'Default config over-rides for a new tank.'}),)

6.1. synapse package 351 synapse Documentation, Release 0.1.0

delete(name) init(name, conf=None) Generate a new CryoTank with a given name or get an reference to an existing CryoTank. Parameters name (str) – Name of the CryoTank. Returns A CryoTank instance. Return type CryoTank list() Get a list of (name, info) tuples for the CryoTanks. Returns A list of tufos. Return type list onTeleOpen(link, path) Allow a telepath share to create a new sub-share. class synapse.cryotank.CryoTank Bases: synapse.lib.cell.Cell A CryoTank implements a stream of structured data. cellapi alias of TankApi confdefs = (('mapsize', {'type': 'int', 'doc': 'LMDB mapsize value', 'defval': 1099511627776}), ('noindex', {'type': 'bool', 'doc': 'Disable indexing', 'defval': 0})) getOffset(iden) info() Returns information about the CryoTank instance. Returns A dict containing items and metrics indexes. Return type dict last() Return the last item stored in this CryoTank. metrics(offs, size=None) Yield metrics rows starting at offset. Parameters • offs (int) – The index offset. • size (int) – The maximum number of records to yield. Yields ((int, dict)) – An index offset, info tuple for metrics. puts(items, seqn=None) Add the structured data from items to the CryoTank. Parameters • items (list) – A list of objects to store in the CryoTank. • seqn (iden, offs) – An iden / offset pair to record. Returns The ending offset of the items or seqn. Return type int

352 Chapter 6. synapse synapse Documentation, Release 0.1.0

rows(offs, size, iden=None) Yield a number of raw items from the CryoTank starting at a given offset. Parameters • offs (int) – The index of the desired datum (starts at 0) • size (int) – The max number of items to yield. Yields ((indx, bytes)) – Index and msgpacked bytes. setOffset(iden, offs) slice(offs, size, iden=None) Yield a number of items from the CryoTank starting at a given offset. Parameters • offs (int) – The index of the desired datum (starts at 0) • size (int) – The max number of items to yield.

Notes

This API performs msgpack unpacking on the bytes, and could be slow to call remotely. Yields ((index, object)) – Index and item values. class synapse.cryotank.CryoTankIndexer(cryotank: synapse.cryotank.CryoTank) Bases: object Manages indexing of a single cryotank’s records This implements a lazy indexer that indexes a cryotank in a separate thread. Cryotank entries are msgpack-encoded values. An index consists of a property name, one or more datapaths (i.e. what field out of the entry), and a synapse type. The type specifies the function that normalizes the output of the datapath query into a string or integer. Indices can be added and deleted asynchronously from the indexing thread via CryotankIndexer.addIndex and CryotankIndexer.delIndex. Indexes can be queried with queryNormValu, queryNormRecords, queryRows. To harmonize with LMDB requirements, writing only occurs on a singular indexing thread. Reading indices takes place in the caller’s thread. Both reading and writing index metadata (that is, information about which indices are running) take place on the indexer’s thread.

Note: The indexer cannot detect when a type has changed from underneath itself. Operators must explicitly delete and re-add the index to avoid mixed normalized data.

MAX_WAIT_S = 10 addIndex(prop, syntype, datapaths) Add an index to the cryotank Parameters • prop (str) – the name of the property this will be stored as in the normalized record • syntype (str) – the synapse type this will be interpreted as

6.1. synapse package 353 synapse Documentation, Release 0.1.0

• datapaths (Iterable[str]) – datapath specs against which the raw record is run to extract a single field that is passed to the type normalizer. These will be tried in order until one succeeds. At least one must be present. Returns None

Note: Additional datapaths will only be tried if prior datapaths are not present, and not if the normalization fails.

delIndex(prop) Delete an index Parameters prop (str) – the (normalized) property name Returns None getIndices() Get information about all the indices Parameters None – Returns Any]]: all the indices with progress and statistics Return type List[Dict[str pauseIndex(prop=None) Temporarily stop indexing one or all indices. Parameters prop – (Optional[str]): the index to stop indexing, or if None, indicate to stop all indices Returns None

Note: Pausing is not persistent. Restarting the process will resume indexing.

queryNormRecords(prop: str, valu: Union[int, str, None] = None, exact=False) → Iter- able[Tuple[int, Dict[str, Union[str, int]]]] Query for normalized property values grouped together in dicts Parameters • prop – The name of the indexed property • valu – The normalized value. If not present, all records with prop present, sorted by prop will be returned. It will be considered a prefix if exact is False. • exact – Indicates that the result must match exactly. Conversely, if False, indicates a prefix match. Returns A generator of offset, dictionary tuples queryNormValu(prop: str, valu: Union[int, str, None] = None, exact=False) Query for normalized individual property values Parameters • prop – The name of the indexed property • valu – The normalized value. If not present, all records with prop present, sorted by prop will be returned. It will be considered a prefix if exact is False.

354 Chapter 6. synapse synapse Documentation, Release 0.1.0

• exact (bool) – Indicates that the result must match exactly. Conversely, if False, indi- cates a prefix match. Returns A generator of offset, normalized value tuples queryRows(prop: str, valu: Union[int, str, None] = None, exact=False) → Iterable[Tuple[int, bytes]] Query for raw (i.e. from the cryotank itself) records Parameters • prop – The name of the indexed property • valu – The normalized value. If not present, all records with prop present, • by prop will be returned. It will be considered a prefix if exact is False. (sorted)– • exact – Indicates that the result must match exactly. Conversely, if False, indicates a prefix match. Returns A generator of tuple (offset, messagepack encoded) raw records Return type Iterable[Tuple[int, bytes]] resumeIndex(prop=None) Undo a pauseIndex Parameters prop – (Optional[str]): the index to start indexing, or if None, indicate to resume all indices Returns None class synapse.cryotank.TankApi Bases: synapse.lib.cell.CellApi addIndex(prop, syntype, datapaths) Add an index to the cryotank Parameters • prop (str) – the name of the property this will be stored as in the normalized record • syntype (str) – the synapse type this will be interpreted as • datapaths (Iterable[str]) – datapath specs against which the raw record is run to extract a single field that is passed to the type normalizer. These will be tried in order until one succeeds. At least one must be present. Returns None

Note: Additional datapaths will only be tried if prior datapaths are not present, and not if the normalization fails.

delIndex(prop) Delete an index Parameters prop (str) – the (normalized) property name Returns None getIndices() Get information about all the indices Parameters None –

6.1. synapse package 355 synapse Documentation, Release 0.1.0

Returns Any]]: all the indices with progress and statistics Return type List[Dict[str metrics(offs, size=None) offset(iden) pauseIndex(prop=None) Temporarily stop indexing one or all indices Parameters prop – (Optional[str]): the index to stop indexing, or if None, indicate to stop all indices Returns None

Note: Pausing is not persistent. Restarting the process will resume indexing.

puts(items, seqn=None) queryNormRecords(prop, valu=None, exact=False) Query for normalized property values grouped together in dicts Parameters • name (str) – name of the Cryotank • prop (str) – The name of the indexed property • valu (Optional[Union[int, str]]) – The normalized value. If not present, all records with prop present, • by prop will be returned. It will be considered a prefix if exact is False. (sorted)– • exact (bool) – Indicates that the result must match exactly. Conversely, if False, indi- cates a prefix match. Returns A generator of offset, dictionary tuples Return type Iterable[Tuple[int, Dict[str, Union[str, int]]]] queryNormValu(prop, valu=None, exact=False) Query for normalized individual property values Parameters • name (str) – name of the Cryotank • prop (str) – The name of the indexed property • valu (Optional[Union[int, str]]) – The normalized value. If not present, all records with prop present, • by prop will be returned. It will be considered a prefix if exact is False. (sorted)– • exact (bool) – Indicates that the result must match exactly. Conversely, if False, indi- cates a prefix match. Returns A generator of offset, normalized value tuples. Return type Iterable[Tuple[int, Union[str, int]]]

356 Chapter 6. synapse synapse Documentation, Release 0.1.0

queryRows(prop, valu=None, exact=False) Query for raw (i.e. from the cryotank itself) records Parameters • name (str) – name of the Cryotank • prop (str) – The name of the indexed property • valu (Optional[Union[int, str]]) – The normalized value. If not present, all records with prop present, • by prop will be returned. It will be considered a prefix if exact is False. (sorted)– • exact (bool) – Indicates that the result must match exactly. Conversely, if False, indi- cates a prefix match. Returns A generator of tuple (offset, messagepack encoded) raw records Return type Iterable[Tuple[int, bytes]] resumeIndex(prop=None) Undo a pauseIndex Parameters prop (Optional[str]) – the index to start indexing, or if None, indicate to resume all indices Returns None slice(size, offs, iden=None)

6.1.8 synapse.daemon module class synapse.daemon.AsyncGenr Bases: synapse.lib.share.Share typename = 'genr' class synapse.daemon.Daemon Bases: synapse.lib.base.Base confdefs = (('listen', {'defval': 'tcp://127.0.0.1:27492', 'doc': 'The default listen host/port'}), ('modules', {'defval': (), 'doc': 'A list of python modules to import before Cell construction.'})) listen(url, **opts) Bind and listen on the given host/port with possible SSL. Parameters • host (str) – A hostname or IP address. • port (int) – The TCP port to bind. loadDmonCell(name) share(name, item) Share an object via the telepath protocol. Parameters • name (str) – Name of the shared object • item (object) – The object to share over telepath. class synapse.daemon.Genr Bases: synapse.lib.share.Share

6.1. synapse package 357 synapse Documentation, Release 0.1.0

typename = 'genr' class synapse.daemon.Sess Bases: synapse.lib.base.Base getSessItem(name) popSessItem(name) setSessItem(name, item)

6.1.9 synapse.datamodel module

An API to assist with the creation and enforcement of cortex data models. class synapse.datamodel.Form(modl, name, info) Bases: object The Form class implements data model logic for a node form. getDelOps(buid) getLiftOps(valu, cmpr=’=’) Get a set of lift operations for use with an Xact. getSetOps(buid, norm) onAdd(func) Add a callback for adding this type of node. The callback is executed after node construction. Parameters func (function) – A node add callback. The callback is called with the current transaction and the new node. def func(xact, node): dostuff() onDel(func) pack() prop(name: str) Return a secondary property for this form by relative prop name. Parameters name (str) – The relative property name. Returns The property or None. Return type (synapse.datamodel.Prop) wasAdded(node) Fire the onAdd() callbacks for node creation. wasDeleted(node) Fire the onDel() callbacks for node deletion. class synapse.datamodel.Model Bases: object The data model used by a Cortex hypergraph. addBaseType(item) Add a Type instance to the data model.

358 Chapter 6. synapse synapse Documentation, Release 0.1.0

addDataModels(mods) Add a list of (name, mdef) tuples. A model definition (mdef) is structured as follows: { “ctors”:( (‘name’, ‘class.path.ctor’, {}, {‘doc’: ‘The foo thing.’}), ), “types”:( (‘name’, (‘basetype’, {typeopts}), {info}), ), “forms”:( (formname, (typename, typeopts), {info}, ( (propname, (typename, typeopts), {info}), )), ), “univs”:( (propname, (typename, typeopts), {info}), ) } Parameters mods (list)– addUnivProp(name, tdef, info) form(name) getModelDef() Returns A list of one model definition compatible with addDataModels that represents the cur- rent data model getModelDict() getModelInfo() getPropsByType(name) getTypeClone(typedef ) prop(name) type(name) Return a synapse.lib.types.Type by name. univ(name) class synapse.datamodel.ModelInfo Bases: object A summary of the information in a DataModel, sufficent for parsing storm queries. addDataModels(mods) Adds a model definition (same format as input to Model.addDataModels and output of Model.getModelDef). addUnivForm(univname, form) addUnivName(univname) isform(name)

6.1. synapse package 359 synapse Documentation, Release 0.1.0

isprop(name) isuniv(name) class synapse.datamodel.Prop(modl, form, name, typedef, info) Bases: synapse.datamodel.PropBase The Prop class represents a property defined within the data model. getCompOffs() Return the offset of this field within the compound primary prop or None. getDelOps(buid) Get a list of storage operations to delete this property from the buid. Parameters buid (bytes) – The node buid. Returns The storage operations Return type (tuple) getLiftOps(valu, cmpr=’=’) getSetOps(buid, norm) pack() class synapse.datamodel.PropBase Bases: object onDel(func) Add a callback for deleting this property. The callback is executed after the property is deleted. Parameters func (function) – A prop del callback. The callback is called within the current transaction, with the node, and the old property value (or None). def func(node, oldv): dostuff() onSet(func) Add a callback for setting this property. The callback is executed after the property is set. Parameters func (function) – A prop set callback. The callback is called within the current transaction, with the node, and the old property value (or None). def func(node, oldv): dostuff() wasDel(node, oldv) wasSet(node, oldv) Fire the onset() handlers for this property. Parameters • node (synapse.lib.node.Node) – The node whose property was set. • oldv (obj) – The previous value of the property. class synapse.datamodel.Univ(modl, name, typedef, propinfo) Bases: synapse.datamodel.PropBase A property-like object that can lift without Form(). getLiftOps(valu, cmpr=’=’)

360 Chapter 6. synapse synapse Documentation, Release 0.1.0

6.1.10 synapse.dyndeps module synapse.dyndeps.getDynLocal(name) Dynamically import a python module and return a local.

Example

cls = getDynLocal(‘foopkg.barmod.BlahClass’) blah = cls() synapse.dyndeps.getDynMeth(name) Retrieve and return an unbound method by python path. synapse.dyndeps.getDynMod(name) Dynamically import a python module and return a ref (or None).

Example

mod = getDynMod(‘foo.bar’) synapse.dyndeps.runDynTask(task) Run a dynamic task and return the result.

Example

foo = runDynTask( (‘baz.faz.Foo’, (), {} ) ) synapse.dyndeps.tryDynFunc(name, *args, **kwargs) Dynamically import a module and call a function or raise an exception. synapse.dyndeps.tryDynLocal(name) Dynamically import a module and return a module local or raise an exception. synapse.dyndeps.tryDynMod(name) Dynamically import a python module or exception.

6.1.11 synapse.eventbus module class synapse.eventbus.BusRef(ctor=None) Bases: synapse.eventbus.EventBus An object for managing multiple EventBus instances. gen(name) Atomically get/gen an EventBus and incref. (requires ctor during BusRef init) Parameters name (str) – The name/iden of the EventBus instance. get(name) Retrieve an EventBus instance by name. Parameters name (str) – The name/iden of the EventBus Returns The EventBus instance (or None) Return type (EventBus) items()

6.1. synapse package 361 synapse Documentation, Release 0.1.0

pop(name) Remove and return an EventBus from the BusRef. Parameters name (str) – The name/iden of the EventBus instance Returns The named event bus ( or None ) Return type (EventBus) put(name, ebus) Add an EventBus (or sub-class) to the BusRef by name. Parameters • name (str) – The name/iden of the EventBus • ebus (EventBus) – The EventBus instance Returns (None) vals() class synapse.eventbus.EventBus Bases: object A synapse EventBus provides an easy way manage callbacks. consume(gtor) Feed the event bus from a generator.

Example

bus.consume( getAllEvents() ) dist(mesg) Distribute an existing event tuple. Parameters mesg ((str,dict)) – An event tuple.

Example

ebus.dist( (‘foo’,{‘bar’:’baz’}) ) distall(events) Distribute multiple events on the event bus. exc(exc, **info) Implements the exception log convention for EventBus. A caller is expected to be within the except frame. Parameters exc (Exception) – The exception to log Returns None fini() Fire the ‘fini’ handlers and set self.isfini.

Example

d.fini() fire(evtname, **info) Fire the given event name on the EventBus. Returns a list of the return values of each callback.

362 Chapter 6. synapse synapse Documentation, Release 0.1.0

Example

for ret in d.fire(‘woot’,foo=’asdf’): print(‘got: %r’ % (ret,))

incref() Increment the reference count for this event bus. This API may be optionally used to control fini(). link(func) Add a callback function to receive all events.

Example

bus1 = EventBus() bus2 = EventBus() bus1.link( bus2.dist ) # all events on bus1 are also propigated on bus2 log(level, mesg, **info) Implements the log event convention for an EventBus. Parameters • level (int) – A python logger level for the event • mesg (str) – A log message • **info – Additional log metadata main() Helper function to block until shutdown ( and handle ctrl-c and SIGTERM).

Examples

Run a event bus, wait until main() has returned, then do other stuff:

foo= EventBus() foo.main() dostuff()

Notes

This does fire a ‘ebus:main’ event prior to entering the waitfini() loop. Returns None off(evnt, func) Remove a previously registered event handler function.

Example

bus.off( ‘foo’, onFooFunc ) on(evnt, func, **filts) Add an event bus callback for a specific event with optional filtering. Parameters

6.1. synapse package 363 synapse Documentation, Release 0.1.0

• evnt (str) – An event name • func (function) – A callback function to receive event tufo • **filts – Optional positive filter values for the event tuple.

Examples

Add a callback function and fire it: def baz(event): x = event[1].get(‘x’) y = event[1].get(‘y’) return x + y d.on(‘foo’, baz, x=10) # this fire triggers baz. . . d.fire(‘foo’, x=10, y=20) # this fire does not ( due to filt ) d.fire(‘foo’, x=30, y=20)

Returns Return type None

onWith(evnt, func, **filts) A context manager which can be used to add a callback and remove it when using a with statement. Parameters • evnt (str) – An event name • func (function) – A callback function to receive event tufo • **filts – Optional positive filter values for the event tuple.

Examples

Temporarily add the baz callback function and use it. def baz(event): x = event[1].get(‘x’) y = event[1].get(‘y’) return x + y with d.onWith(‘foo’, baz, x=10): # this fire triggers baz. . . d.fire(‘foo’, x=10, y=20) # this does NOT fire triggers baz since it is outside # of the context manager. d.fire(‘foo’, x=10, y=30) onfini(func) Register a handler to fire when this EventBus shuts down. unlink(func) Remove a callback function previously added with link()

Example

bus.unlink( callback ) waiter(count, *names) Construct and return a new Waiter for events on this bus.

364 Chapter 6. synapse synapse Documentation, Release 0.1.0

Example

# wait up to 3 seconds for 10 foo:bar events. . . waiter = bus.waiter(10,’foo:bar’) # .. fire thread that will cause foo:bar events events = waiter.wait(timeout=3) if events == None: # handle the timout case. . . for event in events: # parse the events if you need. . . NOTE: use with caution. . . it’s easy to accidentally construct race conditions with this mechanism ;) waitfini(timeout=None) Wait for the event bus to fini()

Example

bus.waitfini(timeout=30) class synapse.eventbus.Waiter(bus, count, *names) Bases: object A helper to wait for a given number of events on an EventBus. fini() wait(timeout=None) Wait for the required number of events and return them or None on timeout.

Example

evnts = waiter.wait(timeout=30) if evnts == None: handleTimedOut() return for evnt in evnts: doStuff(evnt)

6.1.12 synapse.exc module exception synapse.exc.AlreadyInAsync(*args, **info) Bases: synapse.exc.SynErr Raised when an attempt to pend on getting the value back from a coroutine, when already in the event loop thread exception synapse.exc.AuthDeny(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.AxonBadChunk(*args, **info) Bases: synapse.exc.AxonErr exception synapse.exc.AxonBlobStorBsidChanged(*args, **info) Bases: synapse.exc.AxonErr exception synapse.exc.AxonBlobStorDisagree(*args, **info) Bases: synapse.exc.AxonErr

6.1. synapse package 365 synapse Documentation, Release 0.1.0 exception synapse.exc.AxonErr(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.AxonNoBlobStors(*args, **info) Bases: synapse.exc.AxonErr exception synapse.exc.AxonUnknownBsid(*args, **info) Bases: synapse.exc.AxonErr exception synapse.exc.AxonUploaderFinished(*args, **info) Bases: synapse.exc.AxonErr exception synapse.exc.BadCmdName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadCmprValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadConfValu(*args, **info) Bases: synapse.exc.SynErr The configuration value provided is not valid. This should contain the config name, valu and mesg. exception synapse.exc.BadCoreName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadCoreStore(*args, **info) Bases: synapse.exc.SynErr The storage layer has encountered an error exception synapse.exc.BadCtorType(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadEccExchange(*args, **info) Bases: synapse.exc.CryptoErr Raised when there is an issue doing a ECC Key Exchange exception synapse.exc.BadFileExt(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadIndxValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadInfoValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadLiftValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadMesgVers(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadOperArg(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadOptValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadPropConf(*args, **info) Bases: synapse.exc.SynErr

366 Chapter 6. synapse synapse Documentation, Release 0.1.0

The configuration for the property is invalid. exception synapse.exc.BadPropDef(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadPropName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadPropValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadRuleValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadStorValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadStorageVersion(*args, **info) Bases: synapse.exc.SynErr Stored persistent data is incompatible with running software exception synapse.exc.BadStormSyntax(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadSyntaxError(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadThreadIden(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadTime(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadTypeDef(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadTypeValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadUrl(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.Canceled(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.CantDelNode(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.CantDelProp(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.CantRevLayer(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.CliFini(*args, **info) Bases: synapse.exc.SynErr Raised when the CLI is to exit. exception synapse.exc.CorruptDatabase(*args, **info) Bases: synapse.exc.SynErr

6.1. synapse package 367 synapse Documentation, Release 0.1.0

exception synapse.exc.CryptoErr(*args, **info) Bases: synapse.exc.SynErr Raised when there is a synapse.lib.crypto error. exception synapse.exc.DbOutOfSpace(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.DupFileName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.DupIndx(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.DupPropName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.DupRoleName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.DupTypeName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.DupUserName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.FileExists(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.InconsistentStorage(*args, **info) Bases: synapse.exc.SynErr Stored persistent data is inconsistent exception synapse.exc.IsFini(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.IsReadOnly(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.IsRuntForm(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.JobErr(job) Bases: synapse.exc.SynErr Used for remote exception propagation. exception synapse.exc.LinkErr(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.LinkShutDown(*args, **info) Bases: synapse.exc.LinkErr exception synapse.exc.MustBeLocal(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoCertKey(*args, **info) Bases: synapse.exc.SynErr Raised when a Cert object requires a RSA Private Key to perform an operation and the key is not present. exception synapse.exc.NoCurrTask(*args, **info) Bases: synapse.exc.SynErr

368 Chapter 6. synapse synapse Documentation, Release 0.1.0 exception synapse.exc.NoModIden(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoRevPath(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchAct(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchCmpr(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchCond(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchCtor(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchDecoder(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchDir(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchDyn(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchEncoder(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchFile(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchForm(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchFunc(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchIden(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchImpl(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchIndx(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchLift(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchMeth(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchObj(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchOper(*args, **info) Bases: synapse.exc.SynErr

6.1. synapse package 369 synapse Documentation, Release 0.1.0 exception synapse.exc.NoSuchOpt(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchPath(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchPivot(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchProp(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchRole(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchRule(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchSeq(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchStor(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchType(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchUser(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchVar(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NotReady(*args, **info) Bases: synapse.exc.Retry exception synapse.exc.ParserExit(*args, **info) Bases: synapse.exc.SynErr Raised by synapse.lib.cmd.Parser on Parser exit() exception synapse.exc.ReadOnlyProp(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.RecursionLimitHit(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.ReqConfOpt(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.Retry(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.StepTimeout(*args, **info) Bases: synapse.exc.SynErr Raised when a TestStep.wait() call times out. exception synapse.exc.StormRuntimeError(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.StormVarListError(*args, **info) Bases: synapse.exc.StormRuntimeError

370 Chapter 6. synapse synapse Documentation, Release 0.1.0 exception synapse.exc.SynErr(*args, **info) Bases: Exception get(name) Return a value from the errinfo dict.

Example

try: foothing() except SynErr as e: blah = e.get(‘blah’)

items() exception synapse.exc.TimeOut(*args, **info) Bases: synapse.exc.SynErr

6.1.13 synapse.glob module synapse.glob.coroToTask(coro) Schedule a coro to run on the global loop and return a task. Parameters coro (coroutine) – The coroutine instance.

Notes

This API is thread safe. Returns A Future to wait on. Return type concurrent.futures.Future synapse.glob.iAmLoop() synapse.glob.initloop() synapse.glob.sync(coro, timeout=None) Schedule a coroutine to run on the global loop and return it’s result. Parameters coro (coroutine) – The coroutine instance.

Notes

This API is thread safe and should only be called by non-loop threads. synapse.glob.synchelp(f ) The synchelp decorator allows the transparent execution of a coroutine using the global loop from a thread other than the event loop: @s_glob.synchelp async def stuff(x, y): ... # From within the global event loop, the standard await: valu = await stuff(x, y) # From a worker thread, outside the event loop:

6.1. synapse package 371 synapse Documentation, Release 0.1.0

valu = stuff(x, y) # In both cases, the actual work is done by the global loop.

6.1.14 synapse.mindmeld module

6.1.15 synapse.reactor module class synapse.reactor.Reactor Bases: object A class for registration of one-to-one callbacks. ( much like a switch-case in C ) Unlike an EventBus, only one action may be registered for a given mesg type and the function is expected to return a result. rtor = Reactor() def doFooBar(mesg): return 20 + mesg[1].get(‘x’) rtor.act(‘foo:bar’, doFooBar) y = rtor.react( tufo(‘foo:bar’, x=30) ) # y is now 50. . . act(name, func) Register a handler for an action by name.

Example

rtor.act(‘foo:bar’, doFooBar) react(mesg, name=None) Dispatch to the handler and return his response.

Example

resp = rtor.react(mesg)

Notes

• Handler exceptions will propagate upward

6.1.16 synapse.telepath module

An RMI framework for synapse. class synapse.telepath.Aware Bases: object The telepath.Aware mixin allows shared objects to handle individual links managed by the Daemon. getTeleApi(link, mesg) Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)

372 Chapter 6. synapse synapse Documentation, Release 0.1.0

onTeleOpen(link, path) Allow a telepath share to create a new sub-share. onTeleShare(dmon, name) class synapse.telepath.Genr Bases: synapse.telepath.Share class synapse.telepath.Method(proxy, name, share=None) Bases: object The telepath Method is used to provide proxy method calls. class synapse.telepath.Proxy Bases: synapse.lib.base.Base A telepath Proxy is used to call remote APIs on a shared object.

Example

import synapse.telepath as s_telepath # open the “foo” object shared in a dmon on localhost:3344 async def doFooThing(): proxy = await s_telepath.openurl(‘tcp://127.0.0.1:3344/foo’) valu = await proxy.getFooValu(x, y) The proxy (and openurl function) may also be used from sync code: proxy = s_telepath.openurl(‘tcp://127.0.0.1:3344/foo’) valu = proxy.getFooValu(x, y) call(methname, *args, **kwargs) Call a remote method by name. Parameters • methname (str) – The name of the remote method. • *args – Arguments to the method call. • **kwargs – Keyword arguments to the method call. Most use cases will likely use the proxy methods directly: The following two are effectively the same: valu = proxy.getFooBar(x, y) valu = proxy.call(‘getFooBar’, x, y) getPoolLink() handshake(auth=None) task(todo, name=None) taskv2(todo, name=None) class synapse.telepath.Share Bases: synapse.lib.base.Base The telepath client side of a dynamically shared object.

6.1. synapse package 373 synapse Documentation, Release 0.1.0 class synapse.telepath.Task Bases: object A telepath Task is used to internally track calls/responses. reply(retn) result() synapse.telepath.alias(name) Resolve a telepath alias via ~/.syn/aliases.yaml Parameters name (str) – Name of the alias to resolve.

Notes

An exact match against the aliases will always be returned first. If no exact match is found and the name contains a ‘/’ in it, the value before the slash is looked up and the remainder of the path is joined to any result. This is done to support dynamic Telepath share names. Returns The url string, if present in the alias. None will be returned if there are no matches. Return type str

374 Chapter 6. synapse CHAPTER 7

Indices and tables

• genindex • modindex • search

375 synapse Documentation, Release 0.1.0

376 Chapter 7. Indices and tables Python Module Index

s synapse.lib.datfile, 266 synapse, 219 synapse.lib.db, 266 synapse.axon, 337 synapse.lib.editatom, 267 synapse.cells, 340 synapse.lib.encoding, 267 synapse.cmds, 219 synapse.lib.filepath, 268 synapse.cmds.cortex, 219 synapse.lib.gis, 270 synapse.cmds.cron, 221 synapse.lib.hashitem, 271 synapse.cmds.hive, 222 synapse.lib.hashset, 271 synapse.cmds.trigger, 222 synapse.lib.hive, 272 synapse.common, 340 synapse.lib.ingest, 274 synapse.cortex, 345 synapse.lib.interval, 274 synapse.cryotank, 351 synapse.lib.jupyter, 274 synapse.daemon, 357 synapse.lib.kv, 277 synapse.data, 222 synapse.lib.layer, 281 synapse.datamodel, 358 synapse.lib.link, 282 synapse.dyndeps, 361 synapse.lib.lmdb, 282 synapse.eventbus, 361 synapse.lib.lmdblayer, 286 synapse.exc, 365 synapse.lib.lmdbslab, 287 synapse.glob, 371 synapse.lib.migrate, 288 synapse.lib, 223 synapse.lib.mixins, 288 synapse.lib.agenda, 232 synapse.lib.modelrev, 289 synapse.lib.ast, 233 synapse.lib.module, 289 synapse.lib.auth, 240 synapse.lib.modules, 290 synapse.lib.base, 244 synapse.lib.msgpack, 290 synapse.lib.boss, 248 synapse.lib.node, 292 synapse.lib.cache, 248 synapse.lib.output, 295 synapse.lib.cell, 248 synapse.lib.platforms, 230 synapse.lib.certdir, 250 synapse.lib.platforms.common, 231 synapse.lib.chop, 260 synapse.lib.platforms.darwin, 231 synapse.lib.cli, 261 synapse.lib.platforms.freebsd, 231 synapse.lib.cmd, 263 synapse.lib.platforms.linux, 231 synapse.lib.cmdr, 263 synapse.lib.platforms.windows, 231 synapse.lib.const, 264 synapse.lib.queue, 295 synapse.lib.coro, 264 synapse.lib.ratelimit, 298 synapse.lib.crypto, 223 synapse.lib.reflect, 298 synapse.lib.crypto.ecc, 223 synapse.lib.remotelayer, 299 synapse.lib.crypto.tinfoil, 225 synapse.lib.scope, 300 synapse.lib.crypto.vault, 226 synapse.lib.scrape, 301 synapse.lib.datapath, 265 synapse.lib.share, 301 synapse.lib.slaboffs, 301

377 synapse Documentation, Release 0.1.0 synapse.lib.slabseqn, 301 synapse.tools.deploy, 336 synapse.lib.snap, 302 synapse.tools.dmon, 336 synapse.lib.storm, 304 synapse.tools.easycert, 336 synapse.lib.stormtypes, 310 synapse.tools.feed, 336 synapse.lib.syntax, 310 synapse.tools.guid, 336 synapse.lib.task, 313 synapse.tools.json2mpk, 336 synapse.lib.thishost, 314 synapse.tools.pullfile, 336 synapse.lib.thisplat, 314 synapse.tools.pushfile, 336 synapse.lib.threads, 314 synapse.lib.time, 317 synapse.lib.trees, 317 synapse.lib.trigger, 318 synapse.lib.types, 318 synapse.lib.urlhelp, 324 synapse.lib.version, 324 synapse.lookup, 326 synapse.lookup.iana, 326 synapse.lookup.iso3166, 326 synapse.lookup.pe, 326 synapse.lookup.phonenum, 326 synapse.mindmeld, 372 synapse.models, 327 synapse.models.base, 327 synapse.models.crypto, 327 synapse.models.dns, 328 synapse.models.files, 328 synapse.models.geopol, 328 synapse.models.geospace, 329 synapse.models.gov, 327 synapse.models.gov.cn, 327 synapse.models.gov.intl, 327 synapse.models.gov.us, 327 synapse.models.inet, 329 synapse.models.infotech, 331 synapse.models.language, 332 synapse.models.material, 332 synapse.models.media, 332 synapse.models.orgs, 332 synapse.models.person, 332 synapse.models.syn, 332 synapse.models.telco, 333 synapse.reactor, 372 synapse.servers, 334 synapse.servers.cortex, 334 synapse.telepath, 372 synapse.tools, 334 synapse.tools.autodoc, 334 synapse.tools.backup, 335 synapse.tools.cellauth, 335 synapse.tools.cmdr, 335 synapse.tools.cryo, 334 synapse.tools.cryo.cat, 334 synapse.tools.cryo.list, 334 synapse.tools.csvtool, 335

378 Python Module Index Index

A addFeedData() (synapse.lib.jupyter.CmdrCore abort() (synapse.lib.layer.Layer method), 281 method), 274 aborted() (synapse.lib.lmdbslab.Slab method), 287 addFeedData() (synapse.lib.snap.Snap method), 302 AbsProp (class in synapse.lib.ast), 233 addFeedNodes() (synapse.lib.snap.Snap method), absprop() (synapse.lib.syntax.Parser method), 310 302 AbsPropCond (class in synapse.lib.ast), 233 addFormat() (in module synapse.lib.encoding), 267 act() (synapse.reactor.Reactor method), 372 addHead() (synapse.tools.autodoc.RstHelp method), add() (in module synapse.cells), 340 334 add() (synapse.lib.agenda.Agenda method), 232 addIndex() (synapse.cryotank.CryoTankIndexer add() (synapse.lib.cache.TagGlobs method), 248 method), 353 add() (synapse.lib.hive.Hive method), 272 addIndex() (synapse.cryotank.TankApi method), 355 add() (synapse.lib.hive.Node method), 273 addInput() (synapse.lib.storm.Runtime method), 308 add() (synapse.lib.hive.TeleHive method), 274 addKid() (synapse.lib.ast.AstNode method), 233 add() (synapse.lib.kv.KvSet method), 278 addKvDup() (synapse.lib.kv.KvStor method), 279 add() (synapse.lib.scope.Scope method), 300 addKvDups() (synapse.lib.kv.KvStor method), 279 add() (synapse.lib.trigger.Triggers method), 318 addLibFuncs() (synapse.lib.stormtypes.Lib method), addAndSync() (synapse.lib.hive.HiveApi method), 310 272 addLibFuncs() (synapse.lib.stormtypes.LibTime addAuthRole() (synapse.lib.cell.CellApi method), method), 310 249 addLines() (synapse.tools.autodoc.RstHelp method), addAuthRule() (synapse.lib.cell.CellApi method), 334 249 addNode() (synapse.cortex.CoreApi method), 346 addAuthUser() (synapse.lib.cell.CellApi method), addNode() (synapse.lib.editatom.EditAtom method), 249 267 addBaseType() (synapse.datamodel.Model method), addNode() (synapse.lib.snap.Snap method), 302 358 addNodes() (synapse.cortex.CoreApi method), 346 addBlobStor() (synapse.axon.Axon method), 337 addNodes() (synapse.cortex.Cortex method), 348 addCellCmd() (synapse.lib.cell.Cell method), 248 addNodes() (synapse.lib.snap.Snap method), 302 addCmdClass() (synapse.lib.cli.Cli method), 261 addNodeTag() (synapse.cortex.CoreApi method), 346 addCoreMods() (synapse.cortex.Cortex method), 347 addpref() (in module synapse.common), 340 addCronJob() (synapse.cortex.CoreApi method), 345 Addr (class in synapse.models.inet), 329 addDataModels() (synapse.datamodel.Model addRole() (synapse.lib.auth.Auth method), 240 method), 358 addRole() (synapse.lib.auth.User method), 242 addDataModels() (synapse.datamodel.ModelInfo addRole() (synapse.lib.hive.HiveAuth method), 272 method), 359 addRootCert() (synapse.lib.crypto.vault.Vault addFeedData() (in module synapse.tools.feed), 336 method), 228 addFeedData() (synapse.cortex.CoreApi method), addRule() (synapse.lib.auth.Role method), 241 346 addRule() (synapse.lib.hive.HiveIden method), 273 addFeedData() (synapse.cortex.Cortex method), 347 addRuntLift() (synapse.cortex.Cortex method), 348 addRuntPropDel() (synapse.cortex.Cortex method),

379 synapse Documentation, Release 0.1.0

348 Auth (class in synapse.lib.auth), 240 addRuntPropSet() (synapse.cortex.Cortex method), AuthApi (class in synapse.lib.auth), 241 348 AuthDeny, 365 addsigner() (synapse.lib.crypto.vault.Cert method), avail() (synapse.lib.db.Pool method), 266 226 Aware (class in synapse.telepath), 372 addStormCmd() (synapse.cortex.Cortex method), 348 Axon (class in synapse.axon), 337 addStormLib() (synapse.cortex.Cortex method), 348 AxonApi (class in synapse.axon), 337 addSynMixin() (in module synapse.lib.mixins), 288 AxonBadChunk, 365 addTag() (synapse.lib.node.Node method), 292 AxonBlobStorBsidChanged, 365 addTagNode() (synapse.lib.snap.Snap method), 303 AxonBlobStorDisagree, 365 addTrigger() (synapse.cortex.CoreApi method), 346 AxonErr, 365 addUnivForm() (synapse.datamodel.ModelInfo AxonNoBlobStors, 366 method), 359 AxonUnknownBsid, 366 addUnivName() (synapse.datamodel.ModelInfo AxonUploaderFinished, 366 method), 359 addUnivProp() (synapse.datamodel.Model method), B 359 backup() (in module synapse.tools.backup), 335 addUser() (synapse.lib.auth.Auth method), 240 backup_lmdb() (in module synapse.tools.backup), addUser() (synapse.lib.hive.HiveAuth method), 272 335 addUserAuth() (synapse.lib.crypto.vault.Vault BadCmdName, 366 method), 228 BadCmprValu, 366 addUserRole() (synapse.lib.cell.CellApi method), BadConfValu, 366 249 BadCoreName, 366 adminapi() (in module synapse.lib.cell), 250 BadCoreStore, 366 agen() (in module synapse.lib.ast), 240 BadCtorType, 366 Agenda (class in synapse.lib.agenda), 232 BadEccExchange, 366 aget() (synapse.lib.cache.FixedCache method), 248 BadFileExt, 366 alias() (in module synapse.telepath), 374 BadIndxValu, 366 allowed() (synapse.cortex.CoreApi method), 346 BadInfoValu, 366 allowed() (synapse.lib.auth.Role method), 242 BadLiftValu, 366 allowed() (synapse.lib.auth.User method), 242 BadMesgVers, 366 allowed() (synapse.lib.hive.HiveUser method), 273 BadOperArg, 366 allowed() (synapse.lib.storm.Runtime method), 308 BadOptValu, 366 allowed_methods (synapse.axon.AxonApi attribute), BadPropConf, 366 337 BadPropDef, 367 allowed_methods (synapse.axon.BlobStorApi BadPropName, 367 attribute), 339 BadPropValu, 367 allowed_methods (synapse.lib.cell.PassThroughApi BadRuleValu, 367 attribute), 249 BadStorageVersion, 367 allowed_methods (synapse.lib.layer.LayerApi BadStormSyntax, 367 attribute), 282 BadStorValu, 367 allows() (synapse.lib.ratelimit.RateLimit method), BadSyntaxError, 367 298 BadThreadIden, 367 AlreadyInAsync, 365 BadTime, 367 AndCond (class in synapse.lib.ast), 233 BadTypeDef, 367 anit() (synapse.lib.base.Base class method), 244 BadTypeValu, 367 ApptRec (class in synapse.lib.agenda), 232 BadUrl, 367 AQueue (class in synapse.lib.queue), 295 Base (class in synapse.lib.base), 244 aspin() (in module synapse.common), 340 BaseModule (class in synapse.models.base), 327 AstNode (class in synapse.lib.ast), 233 BaseRef (class in synapse.lib.base), 247 AsyncGenr (class in synapse.daemon), 357 bbox() (in module synapse.lib.gis), 270 AsyncQueue (class in synapse.lib.queue), 295 BlobStor (class in synapse.axon), 338 AsyncToSyncCMgr (class in synapse.lib.coro), 264 BlobStorApi (class in synapse.axon), 339 At (class in synapse.cmds.cron), 221 Bool (class in synapse.lib.types), 318

380 Index synapse Documentation, Release 0.1.0

Boss (class in synapse.lib.boss), 248 CmdLocals (class in synapse.lib.cli), 263 BreakOper (class in synapse.lib.ast), 234 cmdname() (synapse.lib.syntax.Parser method), 311 bruteVersionStr() CmdOper (class in synapse.lib.ast), 234 (synapse.models.infotech.ItModule method), CmdQuit (class in synapse.lib.cli), 263 331 CmdrCore (class in synapse.lib.jupyter), 274 buid() (in module synapse.common), 340 cmdre (in module synapse.cortex), 350 bulkput() (synapse.axon.Axon method), 337 cmdvalu() (synapse.lib.syntax.Parser method), 311 bulkput() (synapse.axon.BlobStor method), 338 Cmpr (class in synapse.lib.ast), 234 bump() (synapse.lib.lmdbslab.Scan method), 287 cmpr() (synapse.lib.syntax.Parser method), 311 BusRef (class in synapse.eventbus), 361 cmpr() (synapse.lib.types.Type method), 322 commit() (synapse.axon.IncrementalTransaction C method), 339 call() (synapse.lib.threads.Pool method), 314 commit() (synapse.lib.db.Xact method), 266 call() (synapse.telepath.Proxy method), 373 commit() (synapse.lib.editatom.EditAtom method), 267 CallArgs (class in synapse.lib.ast), 234 commit() (synapse.lib.layer.Layer method), 281 cancel() (synapse.lib.storm.Runtime method), 308 commit() (synapse.lib.lmdblayer.LmdbLayer method), Canceled, 367 286 cancelFile() (synapse.axon.Uploader method), 339 COMMIT_PERIOD (synapse.lib.lmdbslab.Slab attribute), CantDelNode, 367 287 CantDelProp, 367 Comp (class in synapse.lib.types), 318 CantRevLayer, 367 compute() (synapse.lib.ast.CallArgs method), 234 capture() (synapse.lib.threads.RetnWait method), 315 compute() (synapse.lib.ast.CompValue method), 234 CaseEntry (class in synapse.lib.ast), 234 compute() (synapse.lib.ast.FuncCall method), 235 casevalu() (synapse.lib.syntax.Parser method), 311 compute() (synapse.lib.ast.List method), 236 Cell (class in synapse.lib.cell), 248 compute() (synapse.lib.ast.PropValue method), 237 CellApi (class in synapse.lib.cell), 249 compute() (synapse.lib.ast.RunValue method), 238 cellapi (synapse.axon.Axon attribute), 337 compute() (synapse.lib.ast.TagPropValue method), cellapi (synapse.axon.BlobStor attribute), 338 239 cellapi (synapse.cortex.Cortex attribute), 348 compute() (synapse.lib.ast.Value method), 239 cellapi (synapse.cryotank.CryoCell attribute), 351 compute() (synapse.lib.ast.VarDeref method), 239 cellapi (synapse.cryotank.CryoTank attribute), 352 compute() (synapse.lib.ast.VarValue method), 240 cellapi (synapse.lib.auth.Auth attribute), 240 CompValue (class in synapse.lib.ast), 234 cellapi (synapse.lib.cell.Cell attribute), 248 Cond (class in synapse.lib.ast), 234 cellapi (synapse.lib.layer.Layer attribute), 281 cond() (synapse.lib.syntax.Parser method), 311 Cert (class in synapse.lib.crypto.vault), 226 condexpr() (synapse.lib.syntax.Parser method), 311 CertDir (class in synapse.lib.certdir), 250 condsubq() (synapse.lib.syntax.Parser method), 311 chop_imei() (in module synapse.models.telco), 333 confbase (synapse.lib.cell.Cell attribute), 248 chopurl() (in module synapse.lib.urlhelp), 324 confdefs (synapse.axon.Axon attribute), 337 chunks() (in module synapse.common), 341 confdefs (synapse.axon.BlobStor attribute), 338 Cidr4 (class in synapse.models.inet), 329 confdefs (synapse.cortex.Cortex attribute), 348 clear() (synapse.lib.cache.FixedCache method), 248 confdefs (synapse.cryotank.CryoCell attribute), 351 Cli (class in synapse.lib.cli), 261 confdefs (synapse.cryotank.CryoTank attribute), 352 CliFini, 367 confdefs (synapse.daemon.Daemon attribute), 357 clone() (synapse.axon.BlobStor method), 338 confdefs (synapse.lib.auth.Auth attribute), 240 clone() (synapse.lib.types.Type method), 322 confdefs (synapse.lib.cell.Cell attribute), 249 close() (synapse.lib.filepath.FpFile method), 268 confdefs (synapse.lib.lmdblayer.LmdbLayer at- close() (synapse.lib.filepath.FpOpener method), 268 tribute), 286 close() (synapse.lib.filepath.FpTar method), 268 confdefs (synapse.lib.module.CoreModule attribute), closeLogFd() (synapse.cmds.cortex.Log method), 289 220 confdefs (synapse.lib.remotelayer.RemoteLayer Cmd (class in synapse.lib.cli), 261 attribute), 299 Cmd (class in synapse.lib.storm), 304 config() (in module synapse.common), 341 cmdargv() (synapse.lib.syntax.Parser method), 311 connect() (in module synapse.lib.link), 282 CmdHelp (class in synapse.lib.cli), 262 Const (class in synapse.lib.ast), 234

Index 381 synapse Documentation, Release 0.1.0

consume() (synapse.eventbus.EventBus method), 362 delIndex() (synapse.cryotank.CryoTankIndexer ContinueOper (class in synapse.lib.ast), 234 method), 354 CoreApi (class in synapse.cortex), 345 delIndex() (synapse.cryotank.TankApi method), 355 CoreModule (class in synapse.lib.module), 289 delKvDup() (synapse.lib.kv.KvStor method), 279 coroToTask() (in module synapse.glob), 371 delKvProp() (synapse.lib.kv.KvStor method), 279 CorruptDatabase, 367 DelNodeCmd (class in synapse.lib.storm), 305 Cortex (class in synapse.cortex), 347 delNodeTag() (synapse.cortex.CoreApi method), 346 count() (synapse.cortex.CoreApi method), 346 delRole() (synapse.lib.auth.Auth method), 240 CountCmd (class in synapse.lib.storm), 304 delRole() (synapse.lib.auth.User method), 242 Cron (class in synapse.cmds.cron), 222 delRootCert() (synapse.lib.crypto.vault.Vault CryoApi (class in synapse.cryotank), 351 method), 228 CryoCell (class in synapse.cryotank), 351 delRule() (synapse.lib.auth.Role method), 242 CryoTank (class in synapse.cryotank), 352 delRule() (synapse.lib.hive.HiveIden method), 273 CryoTankIndexer (class in synapse.cryotank), 353 delta() (in module synapse.lib.time), 317 CryptoErr, 367 delTag() (synapse.lib.node.Node method), 292 CryptoModule (class in synapse.models.crypto), 327 delTrigger() (synapse.cortex.CoreApi method), 347 CryptSeq (class in synapse.lib.crypto.tinfoil), 225 delUser() (synapse.lib.auth.Auth method), 240 ctor() (in module synapse.lib.scope), 300 delUserRole() (synapse.lib.cell.CellApi method), ctor() (synapse.lib.scope.Scope method), 300 249 current() (in module synapse.lib.task), 314 deploy() (in module synapse.cells), 340 current() (in module synapse.lib.threads), 316 deploy() (synapse.lib.cell.Cell class method), 249 cursor() (synapse.axon.IncrementalTransaction deref() (synapse.lib.stormtypes.Dict method), 310 method), 339 deref() (synapse.lib.stormtypes.Lib method), 310 deref() (synapse.lib.stormtypes.StormType method), D 310 Daemon (class in synapse.daemon), 357 Dict (class in synapse.lib.stormtypes), 310 daemonize() (in module dict() (synapse.lib.hive.Hive method), 272 synapse.lib.platforms.common), 231 dict() (synapse.lib.hive.Node method), 273 daemonize() (in module digests() (synapse.lib.hashset.HashSet method), 271 synapse.lib.platforms.windows), 231 digits() (in module synapse.lib.chop), 260 Data (class in synapse.lib.types), 319 digits() (in module synapse.models.telco), 333 DataElem (class in synapse.lib.datapath), 265 dir() (synapse.lib.hive.Hive method), 272 DataPath (class in synapse.lib.datapath), 265 dir() (synapse.lib.hive.Node method), 273 DAY (synapse.lib.agenda.TimeUnit attribute), 233 Dist (class in synapse.models.geospace), 329 DAYOFMONTH (synapse.lib.agenda.TimeUnit attribute), dist() (synapse.eventbus.EventBus method), 362 233 dist() (synapse.lib.base.Base method), 244 DAYOFWEEK (synapse.lib.agenda.TimeUnit attribute), distall() (synapse.eventbus.EventBus method), 362 233 dms2dec() (in module synapse.lib.gis), 270 DbOutOfSpace, 368 DnsModule (class in synapse.models.dns), 328 debase64() (in module synapse.common), 341 DnsName (class in synapse.models.dns), 328 dec() (synapse.lib.crypto.tinfoil.TinFoilHat method), DocHelp (class in synapse.tools.autodoc), 334 226 docModel() (in module synapse.tools.autodoc), 334 decode() (in module synapse.lib.encoding), 267 doECDHE() (in module synapse.lib.crypto.ecc), 224 decrypt() (synapse.lib.crypto.tinfoil.CryptSeq done() (synapse.lib.queue.Queue method), 296 method), 225 done() (synapse.lib.threads.SyncTask method), 316 delAuthRule() (synapse.lib.cell.CellApi method), doRefs() (synapse.lib.storm.NoderefsCmd method), 249 307 delCronJob() (synapse.cortex.CoreApi method), 346 dump() (synapse.lib.crypto.ecc.PriKey method), 223 delete() (synapse.cryotank.CryoApi method), 351 dump() (synapse.lib.crypto.ecc.PubKey method), 224 delete() (synapse.cryotank.CryoCell method), 351 dump() (synapse.lib.crypto.vault.Cert method), 226 delete() (synapse.lib.agenda.Agenda method), 232 dumpfile() (in module synapse.lib.msgpack), 291 delete() (synapse.lib.lmdbslab.Slab method), 287 DupFileName, 368 delete() (synapse.lib.node.Node method), 292 DupIndx, 368 delete() (synapse.lib.trigger.Triggers method), 318 DupPropName, 368

382 Index synapse Documentation, Release 0.1.0

DupRoleName, 368 err() (synapse.lib.threads.SyncTask method), 316 DupTypeName, 368 errinfo() (in module synapse.common), 341 DupUserName, 368 errx() (synapse.lib.threads.RetnWait method), 315 eval() (synapse.cortex.CoreApi method), 347 E eval() (synapse.cortex.Cortex method), 348 eat() (synapse.lib.syntax.Parser method), 311 eval() (synapse.lib.jupyter.CmdrCore method), 275 eatfd() (synapse.lib.hashset.HashSet method), 271 eval() (synapse.lib.snap.Snap method), 303 Edge (class in synapse.lib.types), 319 event_wait() (in module synapse.lib.coro), 264 Edit (class in synapse.lib.ast), 234 EventBus (class in synapse.eventbus), 362 EditAtom (class in synapse.lib.editatom), 267 exc() (synapse.eventbus.EventBus method), 362 editNdefProps() (synapse.lib.migrate.Migration exchange() (synapse.lib.crypto.ecc.PriKey method), method), 288 223 EditNodeAdd (class in synapse.lib.ast), 234 excinfo() (in module synapse.common), 341 editnodeadd() (synapse.lib.syntax.Parser method), execStormCmd() (synapse.lib.storm.Cmd method), 311 304 editoper() (synapse.lib.syntax.Parser method), 311 execStormCmd() (synapse.lib.storm.CountCmd EditPropDel (class in synapse.lib.ast), 235 method), 305 editpropdel() (synapse.lib.syntax.Parser method), execStormCmd() (synapse.lib.storm.DelNodeCmd 311 method), 305 EditPropSet (class in synapse.lib.ast), 235 execStormCmd() (synapse.lib.storm.GraphCmd editpropset() (synapse.lib.syntax.Parser method), method), 305 311 execStormCmd() (synapse.lib.storm.HelpCmd edits() (synapse.lib.hive.HiveApi method), 272 method), 305 EditTagAdd (class in synapse.lib.ast), 235 execStormCmd() (synapse.lib.storm.IdenCmd edittagadd() (synapse.lib.syntax.Parser method), method), 306 311 execStormCmd() (synapse.lib.storm.LimitCmd EditTagDel (class in synapse.lib.ast), 235 method), 306 edittagdel() (synapse.lib.syntax.Parser method), execStormCmd() (synapse.lib.storm.MaxCmd 311 method), 306 EditUnivDel (class in synapse.lib.ast), 235 execStormCmd() (synapse.lib.storm.MinCmd editunivdel() (synapse.lib.syntax.Parser method), method), 306 311 execStormCmd() (synapse.lib.storm.MoveTagCmd editunivset() (synapse.lib.syntax.Parser method), method), 307 311 execStormCmd() (synapse.lib.storm.NoderefsCmd ehex() (in module synapse.common), 341 method), 307 elevate() (synapse.lib.storm.Runtime method), 308 execStormCmd() (synapse.lib.storm.ReIndexCmd Email (class in synapse.models.inet), 330 method), 308 en() (in module synapse.lib.msgpack), 291 execStormCmd() (synapse.lib.storm.SleepCmd en() (synapse.lib.trigger.Triggers.Rule method), 318 method), 309 enable() (synapse.lib.agenda.Agenda method), 232 execStormCmd() (synapse.lib.storm.SpinCmd enbase64() (in module synapse.common), 341 method), 309 enc() (synapse.lib.crypto.tinfoil.TinFoilHat method), execStormCmd() (synapse.lib.storm.SudoCmd 226 method), 309 encode() (in module synapse.lib.encoding), 267 execStormCmd() (synapse.lib.storm.UniqCmd encodeMsg() (synapse.cmds.cortex.Log method), 220 method), 309 Encoder (class in synapse.lib.layer), 281 execStormQuery() (synapse.lib.storm.Runtime encodeValAsKey() (in module synapse.lib.lmdb), method), 308 286 execute() (synapse.lib.boss.Boss method), 248 encrypt() (synapse.lib.crypto.tinfoil.CryptSeq execute() (synapse.lib.db.Xact method), 266 method), 225 execute() (synapse.lib.trigger.Triggers.Rule method), enter() (in module synapse.lib.scope), 301 318 enter() (synapse.lib.scope.Scope method), 300 executemany() (synapse.lib.db.Xact method), 266 eq() (synapse.lib.lmdb.PropStor method), 284 executor() (in module synapse.lib.coro), 264 err() (in module synapse.common), 341 executor() (in module synapse.lib.task), 314

Index 383 synapse Documentation, Release 0.1.0

exists() (in module synapse.lib.filepath), 269 FpOpener (class in synapse.lib.filepath), 268 exit() (synapse.lib.cmd.Parser method), 263 FpTar (class in synapse.lib.filepath), 268 exit() (synapse.lib.storm.Parser method), 308 FpZip (class in synapse.lib.filepath), 269 expect() (synapse.lib.syntax.Parser method), 311 Fqdn (class in synapse.models.inet), 330 extend() (synapse.lib.types.Type method), 322 fromprim() (in module synapse.lib.stormtypes), 310 fromString (synapse.lib.agenda.TimeUnit attribute), F 233 feed() (synapse.lib.link.Link method), 282 fromunix() (synapse.lib.stormtypes.LibTime method), feed() (synapse.lib.msgpack.Unpk method), 290 310 FieldHelper (class in synapse.lib.types), 319 FuncCall (class in synapse.lib.ast), 235 FileBase (class in synapse.models.files), 328 FileBytes (class in synapse.models.files), 328 G FileExists, 368 gen() (synapse.eventbus.BusRef method), 361 FileModule (class in synapse.models.files), 328 gen() (synapse.lib.base.BaseRef method), 247 FilePath (class in synapse.models.files), 328 genCaCert() (synapse.lib.certdir.CertDir method), filter() (synapse.lib.node.Node method), 293 250 FiltOper (class in synapse.lib.ast), 235 genCertTokn() (synapse.lib.crypto.vault.Vault static filtoper() (synapse.lib.syntax.Parser method), 311 method), 229 fini() (synapse.eventbus.EventBus method), 362 genClientCert() (synapse.lib.certdir.CertDir fini() (synapse.eventbus.Waiter method), 365 method), 250 fini() (synapse.lib.base.Base method), 244 gendir() (in module synapse.common), 341 fini() (synapse.lib.base.Waiter method), 247 genEccKey() (synapse.lib.crypto.vault.Vault method), finish() (synapse.axon.Uploader method), 339 229 finish() (synapse.axon.UploaderProxy method), 339 generate() (synapse.lib.crypto.ecc.PriKey static finishFile() (synapse.axon.Uploader method), 339 method), 223 finishFile() (synapse.axon.UploaderProxy genfile() (in module synapse.common), 341 method), 339 genHostCert() (synapse.lib.certdir.CertDir method), fire() (synapse.eventbus.EventBus method), 362 251 fire() (synapse.lib.base.Base method), 244 genHostCsr() (synapse.lib.certdir.CertDir method), firethread() (in module synapse.common), 341 251 first() (synapse.lib.lmdbslab.Scan method), 287 genKvAlias (synapse.lib.kv.KvStor attribute), 279 first() (synapse.lib.trees.Tree method), 317 genpath() (in module synapse.common), 341 FixedCache (class in synapse.lib.cache), 248 Genr (class in synapse.daemon), 357 fmtVersion() (in module synapse.lib.version), 324 Genr (class in synapse.telepath), 373 fold() (in module synapse.lib.interval), 274 genr2agenr() (in module synapse.lib.coro), 264 forcecommit() (synapse.lib.lmdbslab.Slab method), GenrHelp (class in synapse.lib.coro), 264 287 genrhelp() (in module synapse.lib.coro), 264 fork() (synapse.lib.node.Path method), 293 genRootCert() (synapse.lib.crypto.vault.Vault ForLoop (class in synapse.lib.ast), 235 method), 229 forloop() (synapse.lib.syntax.Parser method), 311 gentask() (in module synapse.common), 342 Form (class in synapse.datamodel), 358 genTempCoreProxy() (in module form (synapse.lib.trigger.Triggers.Rule attribute), 318 synapse.lib.jupyter), 275 form() (synapse.datamodel.Model method), 359 genToknCert() (synapse.lib.crypto.vault.Vault format() (synapse.lib.ast.AstNode method), 233 method), 229 formjoin() (synapse.lib.syntax.Parser method), 311 genUserAuth() (synapse.lib.crypto.vault.Vault formjoinin() (synapse.lib.syntax.Parser method), method), 229 311 genUserCert() (synapse.lib.certdir.CertDir method), formPhoneNode() (in module 251 synapse.lookup.phonenum), 326 genUserCert() (synapse.lib.crypto.vault.Vault FormPivot (class in synapse.lib.ast), 235 method), 229 formpivot() (synapse.lib.syntax.Parser method), 311 genUserCsr() (synapse.lib.certdir.CertDir method), formpivotin() (synapse.lib.syntax.Parser method), 252 311 GeoModule (class in synapse.models.geospace), 329 FpFile (class in synapse.lib.filepath), 268 get() (in module synapse.data), 222

384 Index synapse Documentation, Release 0.1.0

get() (in module synapse.lib.scope), 301 getArgParser() (synapse.lib.storm.MinCmd get() (in module synapse.lib.thishost), 314 method), 306 get() (synapse.axon.Axon method), 337 getArgParser() (synapse.lib.storm.MoveTagCmd get() (synapse.axon.AxonApi method), 338 method), 307 get() (synapse.axon.BlobStor method), 338 getArgParser() (synapse.lib.storm.NoderefsCmd get() (synapse.eventbus.BusRef method), 361 method), 307 get() (synapse.exc.SynErr method), 371 getArgParser() (synapse.lib.storm.ReIndexCmd get() (synapse.lib.base.BaseRef method), 247 method), 308 get() (synapse.lib.boss.Boss method), 248 getArgParser() (synapse.lib.storm.SleepCmd get() (synapse.lib.cache.FixedCache method), 248 method), 309 get() (synapse.lib.cache.TagGlobs method), 248 getArgParser() (synapse.lib.storm.UniqCmd get() (synapse.lib.cli.Cli method), 261 method), 309 get() (synapse.lib.hive.Hive method), 272 getAuthInfo() (synapse.lib.cell.CellApi method), get() (synapse.lib.hive.HiveApi method), 272 249 get() (synapse.lib.hive.HiveDict method), 272 getAuthRoles() (synapse.lib.cell.CellApi method), get() (synapse.lib.hive.Node method), 273 249 get() (synapse.lib.hive.TeleHive method), 274 getAuthUsers() (synapse.lib.cell.CellApi method), get() (synapse.lib.kv.KvDict method), 277 249 get() (synapse.lib.kv.KvLook method), 278 getBlobStors() (synapse.axon.Axon method), 337 get() (synapse.lib.link.Link method), 282 getBuidProps() (synapse.lib.layer.Layer method), get() (synapse.lib.lmdb.Offs method), 283 281 get() (synapse.lib.lmdbslab.Slab method), 287 getBuidProps() (synapse.lib.lmdblayer.LmdbLayer get() (synapse.lib.node.Node method), 293 method), 286 get() (synapse.lib.node.Path method), 293 getbytes() (in module synapse.common), 342 get() (synapse.lib.queue.AsyncQueue method), 295 getCaCert() (synapse.lib.certdir.CertDir method), get() (synapse.lib.queue.Queue method), 296 252 get() (synapse.lib.queue.S2AQueue method), 298 getCaCertPath() (synapse.lib.certdir.CertDir get() (synapse.lib.scope.Scope method), 300 method), 252 get() (synapse.lib.slaboffs.SlabOffs method), 301 getCaCerts() (synapse.lib.certdir.CertDir method), get() (synapse.lib.trees.IntervalTree method), 317 253 get() (synapse.lib.trees.Tree method), 317 getCaKey() (synapse.lib.certdir.CertDir method), 253 get() (synapse.lib.trigger.Triggers method), 318 getCaKeyPath() (synapse.lib.certdir.CertDir get_input() (in module synapse.lib.cli), 263 method), 253 get_input() (synapse.lib.cli.Cli method), 261 getCellIden() (synapse.lib.cell.Cell method), 249 getAddrType() (in module synapse.models.inet), 331 getCellIden() (synapse.lib.cell.CellApi method), getArgParser() (in module synapse.tools.dmon), 249 336 getCells() (in module synapse.cells), 340 getArgParser() (in module getCellType() (synapse.lib.cell.Cell method), 249 synapse.tools.json2mpk), 336 getCellType() (synapse.lib.cell.CellApi method), getArgParser() (synapse.lib.storm.Cmd method), 249 304 getCert() (synapse.lib.crypto.vault.Vault method), getArgParser() (synapse.lib.storm.DelNodeCmd 230 method), 305 getCertKey() (synapse.lib.crypto.vault.Vault getArgParser() (synapse.lib.storm.GraphCmd method), 230 method), 305 getCidrRange() (synapse.models.inet.IPv4 method), getArgParser() (synapse.lib.storm.HelpCmd 330 method), 305 getClientCert() (synapse.lib.certdir.CertDir getArgParser() (synapse.lib.storm.IdenCmd method), 253 method), 306 getClientCertPath() (synapse.lib.certdir.CertDir getArgParser() (synapse.lib.storm.LimitCmd method), 254 method), 306 getClientSSLContext() getArgParser() (synapse.lib.storm.MaxCmd (synapse.lib.certdir.CertDir method), 254 method), 306 getCloneProgress() (synapse.axon.BlobStor method), 338

Index 385 synapse Documentation, Release 0.1.0

getClsNames() (in module synapse.lib.reflect), 298 getDynLocal() (in module synapse.dyndeps), 361 getCmdBrief() (synapse.lib.cli.Cmd method), 262 getDynMeth() (in module synapse.dyndeps), 361 getCmdBrief() (synapse.lib.storm.Cmd class getDynMod() (in module synapse.dyndeps), 361 method), 304 getexcfo() (in module synapse.common), 342 getCmdByName() (synapse.lib.cli.Cli method), 261 getFeedFunc() (synapse.cortex.Cortex method), 348 getCmdDoc() (synapse.lib.cli.Cmd method), 262 getFeedOffs() (synapse.cortex.CoreApi method), getCmdItem() (synapse.lib.cli.Cmd method), 262 347 getCmdName() (synapse.lib.cli.Cmd method), 262 getFeedOffs() (synapse.cortex.Cortex method), 348 getCmdNames() (synapse.lib.cli.Cli method), 261 getfile() (in module synapse.common), 342 getCmdOpts() (synapse.lib.cli.Cmd method), 262 getFormTodo() (synapse.lib.migrate.Migration getCmdPrompt() (synapse.lib.cli.Cli method), 261 method), 288 getCmprCtor() (synapse.lib.types.Type method), 323 getHiveAuth() (synapse.lib.hive.Hive method), 272 getCompOffs() (synapse.datamodel.Prop method), getHiveKey() (synapse.lib.cell.CellApi method), 249 360 getHostCaPath() (synapse.lib.certdir.CertDir getCompOffs() (synapse.lib.types.Comp method), method), 254 318 getHostCert() (synapse.lib.certdir.CertDir method), getCompOffs() (synapse.lib.types.Edge method), 319 254 getCompOffs() (synapse.lib.types.TimeEdge getHostCertPath() (synapse.lib.certdir.CertDir method), 322 method), 254 getCompOffs() (synapse.lib.types.Type method), 323 getHostKey() (synapse.lib.certdir.CertDir method), getCondEval() (synapse.lib.ast.AbsPropCond 255 method), 233 getHostKeyPath() (synapse.lib.certdir.CertDir getCondEval() (synapse.lib.ast.AndCond method), method), 255 233 getIndices() (synapse.cryotank.CryoTankIndexer getCondEval() (synapse.lib.ast.Cond method), 234 method), 354 getCondEval() (synapse.lib.ast.HasAbsPropCond getIndices() (synapse.cryotank.TankApi method), method), 235 355 getCondEval() (synapse.lib.ast.HasRelPropCond getIndxOps() (synapse.lib.types.Type method), 323 method), 235 getInput() (synapse.lib.storm.Runtime method), 308 getCondEval() (synapse.lib.ast.NotCond method), getItemCmdr() (in module synapse.lib.cmdr), 263 236 getItemCmdr() (in module synapse.lib.jupyter), 276 getCondEval() (synapse.lib.ast.OrCond method), getItemInfo() (in module synapse.lib.reflect), 299 236 getItemLocals() (in module synapse.lib.reflect), getCondEval() (synapse.lib.ast.RelPropCond 299 method), 237 getItems() (in module synapse.tools.feed), 336 getCondEval() (synapse.lib.ast.SubqCond method), getkey() (synapse.lib.crypto.vault.Cert method), 226 238 getKvDict() (synapse.lib.kv.KvStor method), 280 getCondEval() (synapse.lib.ast.TagCond method), getKvLook() (synapse.lib.kv.KvStor method), 280 239 getKvProp() (synapse.lib.kv.KvStor method), 280 getCondEval() (synapse.lib.ast.TagValuCond getKvSet() (synapse.lib.kv.KvStor method), 280 method), 239 getLangCodes() (in module synapse.lookup.pe), 326 getConfPath() (synapse.lib.module.CoreModule getLayers() (synapse.lib.migrate.Migration method), method), 289 288 getCoreInfo() (synapse.cortex.CoreApi method), getLibC() (in module synapse.lib.platforms.common), 347 231 getCoreInfo() (synapse.cortex.Cortex method), 348 getLibC() (in module synapse.lib.platforms.windows), getCoreMod() (synapse.cortex.Cortex method), 348 231 getCoreMods() (synapse.cortex.CoreApi method), getLiftHintCmpr() (synapse.lib.types.Type 347 method), 323 getCoreMods() (synapse.cortex.Cortex method), 348 getLiftHintCmprCtor() (synapse.lib.types.Type getDelOps() (synapse.datamodel.Form method), 358 method), 323 getDelOps() (synapse.datamodel.Prop method), 360 getLiftHints() (synapse.lib.ast.AndCond method), getDocData() (in module synapse.lib.jupyter), 275 233 getDocPath() (in module synapse.lib.jupyter), 276 getLiftHints() (synapse.lib.ast.Cond method), 234

386 Index synapse Documentation, Release 0.1.0 getLiftHints() (synapse.lib.ast.FiltOper method), getModelDefs() (synapse.models.orgs.OuModule 235 method), 332 getLiftHints() (synapse.lib.ast.TagCond method), getModelDefs() (synapse.models.person.PsModule 239 method), 332 getLiftNodes() (synapse.lib.snap.Snap method), getModelDefs() (synapse.models.syn.SynModule 303 method), 332 getLiftOps() (synapse.datamodel.Form method), getModelDefs() (synapse.models.telco.TelcoModule 358 method), 333 getLiftOps() (synapse.datamodel.Prop method), getModelDict() (synapse.cortex.CoreApi method), 360 347 getLiftOps() (synapse.datamodel.Univ method), 360 getModelDict() (synapse.datamodel.Model getLiftOps() (synapse.lib.types.Ival method), 320 method), 359 getLiftOps() (synapse.lib.types.Type method), 323 getModelInfo() (synapse.datamodel.Model getLiftRows() (synapse.lib.layer.Layer method), 281 method), 359 getLiftRows() (synapse.lib.remotelayer.RemoteLayer getModelRevs() (synapse.lib.module.CoreModule method), 299 method), 290 getLiftRows() (synapse.lib.snap.Snap method), 303 getModelVers() (synapse.lib.layer.Layer method), getLocalProxy() (synapse.lib.cell.Cell method), 281 249 getModelVers() (synapse.lib.layer.LayerApi getMethName() (in module synapse.lib.reflect), 299 method), 282 getModDir() (synapse.lib.module.CoreModule getModelVers() (synapse.lib.lmdblayer.LmdbLayer method), 289 method), 286 getModelDef() (synapse.datamodel.Model method), getModelVers() (synapse.lib.remotelayer.RemoteLayer 359 method), 299 getModelDefs() (synapse.lib.module.CoreModule getModName() (synapse.lib.module.CoreModule method), 290 method), 289 getModelDefs() (synapse.models.base.BaseModule getModPath() (synapse.lib.module.CoreModule method), 327 method), 290 getModelDefs() (synapse.models.crypto.CryptoModulegetn() (synapse.lib.queue.Queue method), 296 method), 327 getNetRange() (synapse.models.inet.IPv4 method), getModelDefs() (synapse.models.dns.DnsModule 330 method), 328 getNodeBeingMade() getModelDefs() (synapse.models.files.FileModule (synapse.lib.editatom.EditAtom method), method), 328 267 getModelDefs() (synapse.models.geopol.PolModule getNodeByBuid() (synapse.lib.snap.Snap method), method), 328 303 getModelDefs() (synapse.models.geospace.GeoModulegetNodeByNdef() (synapse.cortex.Cortex method), method), 329 348 getModelDefs() (synapse.models.gov.cn.GovCnModulegetNodeByNdef() (synapse.lib.snap.Snap method), method), 327 303 getModelDefs() (synapse.models.gov.intl.GovIntlModulegetNodeNdef() (synapse.lib.layer.Layer method), 281 method), 327 getNodeNdef() (synapse.lib.lmdblayer.LmdbLayer getModelDefs() (synapse.models.gov.us.GovUsModule method), 286 method), 327 getNodeRefs() (synapse.lib.node.Node method), 293 getModelDefs() (synapse.models.inet.InetModule getNodesBy() (synapse.cortex.CoreApi method), 347 method), 330 getNodesBy() (synapse.cortex.Cortex method), 348 getModelDefs() (synapse.models.infotech.ItModule getNodesBy() (synapse.lib.snap.Snap method), 303 method), 331 getOffset() (synapse.cryotank.CryoTank method), getModelDefs() (synapse.models.language.LangModule 352 method), 332 getOffset() (synapse.lib.layer.Layer method), 281 getModelDefs() (synapse.models.material.MatModule getOffset() (synapse.lib.lmdblayer.LmdbLayer method), 332 method), 286 getModelDefs() (synapse.models.media.MediaModule getOffset() (synapse.lib.snap.Snap method), 303 method), 332 getOpt() (synapse.lib.storm.Runtime method), 308

Index 387 synapse Documentation, Release 0.1.0

getPathParts() (in module synapse.lib.filepath), 269 getTeleApi() (synapse.lib.cell.Cell method), 249 getPhoneInfo() (in module getTeleApi() (synapse.lib.hive.Hive method), 272 synapse.lookup.phonenum), 326 getTeleApi() (synapse.telepath.Aware method), 372 getPoolLink() (synapse.telepath.Proxy method), 373 getTempCoreCmdr() (in module synapse.lib.jupyter), getPropAndValu() (synapse.lib.ast.PropValue 276 method), 237 getTempCoreProx() (in module synapse.lib.jupyter), getPropsByType() (synapse.datamodel.Model 276 method), 359 getTempCortex() (in module synapse.cortex), 351 getPropSetr() (synapse.lib.lmdb.PropStor method), getTempDir() (in module synapse.common), 342 284 getTickTock() (synapse.lib.types.Time method), 322 getraw() (synapse.lib.kv.KvLook method), 278 getTypeClone() (synapse.datamodel.Model getRefs() (synapse.lib.storm.NoderefsCmd method), method), 359 307 getTypeVals() (synapse.lib.types.Type method), 323 getRoleByName() (synapse.lib.hive.HiveAuth getTypeVals() (synapse.models.inet.IPv4 method), method), 272 330 getRoles() (synapse.lib.auth.Auth method), 241 getUserByName() (synapse.lib.hive.HiveAuth getRootCerts() (synapse.lib.crypto.vault.Vault method), 272 method), 230 getUserCaPath() (synapse.lib.certdir.CertDir getRowNodes() (synapse.lib.snap.Snap method), 303 method), 255 getRsrcTypes() (in module synapse.lookup.pe), 326 getUserCert() (synapse.lib.certdir.CertDir method), getRstText() (synapse.tools.autodoc.RstHelp 256 method), 334 getUserCert() (synapse.lib.crypto.vault.Vault getRuntNodes() (synapse.lib.snap.Snap method), method), 230 304 getUserCertPath() (synapse.lib.certdir.CertDir getRuntVars() (synapse.lib.ast.AstNode method), method), 256 233 getUserForHost() (synapse.lib.certdir.CertDir getRuntVars() (synapse.lib.ast.ForLoop method), method), 256 235 getUserKey() (synapse.lib.certdir.CertDir method), getRuntVars() (synapse.lib.ast.VarListSetOper 256 method), 239 getUserKeyPath() (synapse.lib.certdir.CertDir getRuntVars() (synapse.lib.ast.VarSetOper method), method), 257 240 getUsers() (synapse.lib.auth.Auth method), 241 getServerSSLContext() getVar() (synapse.lib.storm.Runtime method), 308 (synapse.lib.certdir.CertDir method), 255 getVolInfo() (in module getSessItem() (synapse.daemon.Sess method), 358 synapse.lib.platforms.common), 231 getSetOps() (synapse.datamodel.Form method), 358 GovCnModule (class in synapse.models.gov.cn), 327 getSetOps() (synapse.datamodel.Prop method), 360 GovIntlModule (class in synapse.models.gov.intl), getSortedDict() (synapse.lib.syntax.Parser 327 method), 311 GovUsModule (class in synapse.models.gov.us), 327 getStormCmd() (synapse.cortex.Cortex method), 349 grant() (synapse.lib.hive.HiveUser method), 273 getStormCmds() (synapse.cortex.Cortex method), GraphCmd (class in synapse.lib.storm), 305 349 grow() (synapse.lib.lmdbslab.Slab method), 287 getStormLib() (synapse.cortex.Cortex method), 349 guarantee() (synapse.axon.IncrementalTransaction getStormQuery() (synapse.cortex.Cortex method), method), 339 349 Guid (class in synapse.lib.types), 319 getStormRuntime() (synapse.lib.snap.Snap guid() (in module synapse.common), 342 method), 304 guid() (synapse.lib.hashset.HashSet method), 271 getSynDir() (in module synapse.common), 342 getSynMixins() (in module synapse.lib.mixins), 288 H getSynPath() (in module synapse.common), 342 handleList() (in module synapse.tools.cellauth), 335 getTag() (synapse.lib.node.Node method), 293 handleModify() (in module synapse.tools.cellauth), getTagGlobRegx() (in module synapse.lib.cache), 335 248 handshake() (synapse.telepath.Proxy method), 373 getTags() (synapse.lib.node.Node method), 293 has() (synapse.lib.lmdb.PropSetr method), 283

388 Index synapse Documentation, Release 0.1.0

has() (synapse.lib.lmdb.PropStor method), 284 indx() (synapse.lib.types.Guid method), 319 has() (synapse.lib.node.Node method), 293 indx() (synapse.lib.types.Hex method), 319 HasAbsPropCond (class in synapse.lib.ast), 235 indx() (synapse.lib.types.Int method), 320 hashitem() (in module synapse.lib.hashitem), 271 indx() (synapse.lib.types.Ival method), 320 HashSet (class in synapse.lib.hashset), 271 indx() (synapse.lib.types.Loc method), 321 hasKvDups() (synapse.lib.kv.KvStor method), 280 indx() (synapse.lib.types.Ndef method), 321 HasRelPropCond (class in synapse.lib.ast), 235 indx() (synapse.lib.types.NodeProp method), 321 hasTag() (synapse.lib.node.Node method), 293 indx() (synapse.lib.types.Range method), 321 hasValidOpts() (synapse.lib.storm.Cmd method), indx() (synapse.lib.types.StrBase method), 321 304 indx() (synapse.lib.types.Time method), 322 haversine() (in module synapse.lib.gis), 270 indx() (synapse.lib.types.Type method), 323 HelpCmd (class in synapse.lib.storm), 305 indx() (synapse.models.files.FileBytes method), 328 Hex (class in synapse.lib.types), 319 indx() (synapse.models.geospace.Dist method), 329 hexstr() (in module synapse.lib.chop), 260 indx() (synapse.models.geospace.Latitude method), Hive (class in synapse.lib.hive), 272 329 HiveApi (class in synapse.lib.hive), 272 indx() (synapse.models.geospace.LatLong method), HiveAuth (class in synapse.lib.hive), 272 329 HiveCmd (class in synapse.cmds.hive), 222 indx() (synapse.models.geospace.Longitude method), HiveDict (class in synapse.lib.hive), 272 329 HiveIden (class in synapse.lib.hive), 273 indx() (synapse.models.inet.Fqdn method), 330 HiveRole (class in synapse.lib.hive), 273 indx() (synapse.models.inet.IPv4 method), 330 HiveUser (class in synapse.lib.hive), 273 indx() (synapse.models.inet.IPv6 method), 330 hostaddr() (in module synapse.lib.thishost), 314 indx() (synapse.models.infotech.SemVer method), 332 HOUR (synapse.lib.agenda.TimeUnit attribute), 233 indx() (synapse.models.telco.Imei method), 333 indx() (synapse.models.telco.Imsi method), 333 I indx() (synapse.models.telco.Phone method), 333 iAmLoop() (in module synapse.glob), 371 indxByEq() (synapse.lib.types.Hex method), 319 iden() (in module synapse.lib.node), 294 indxByEq() (synapse.lib.types.Time method), 322 iden() (in module synapse.lib.threads), 316 indxByEq() (synapse.lib.types.Type method), 323 iden() (synapse.lib.crypto.ecc.PriKey method), 223 indxByEq() (synapse.models.inet.Fqdn method), 330 iden() (synapse.lib.crypto.ecc.PubKey method), 224 indxByEq() (synapse.models.inet.IPv4 method), 330 iden() (synapse.lib.crypto.vault.Cert method), 227 indxByEq() (synapse.models.telco.Phone method), iden() (synapse.lib.node.Node method), 293 333 IdenCmd (class in synapse.lib.storm), 305 indxByGe() (synapse.lib.types.IntBase method), 320 ignore() (synapse.lib.syntax.Parser method), 311 indxByGt() (synapse.lib.types.IntBase method), 320 ignorespace() (synapse.lib.syntax.Parser method), indxByIn() (synapse.lib.types.Type method), 323 312 indxByIval() (synapse.lib.types.Time method), 322 Imei (class in synapse.models.telco), 333 indxByLe() (synapse.lib.types.IntBase method), 320 imeicsum() (in module synapse.models.telco), 333 indxByLt() (synapse.lib.types.IntBase method), 320 importFile() (synapse.lib.certdir.CertDir method), indxByPref() (synapse.lib.types.Loc method), 321 257 indxByPref() (synapse.lib.types.Str method), 321 Imsi (class in synapse.models.telco), 333 indxByPref() (synapse.lib.types.StrBase method), inc() (synapse.lib.lmdb.Metrics method), 282 321 InconsistentStorage, 368 indxByPref() (synapse.lib.types.Tag method), 321 incref() (synapse.eventbus.EventBus method), 363 indxByPref() (synapse.models.dns.DnsName incref() (synapse.lib.base.Base method), 244 method), 328 IncrementalTransaction (class in synapse.axon), indxByPref() (synapse.models.files.FileBase 339 method), 328 index() (synapse.lib.lmdb.Seqn method), 285 indxByPref() (synapse.models.files.FilePath index() (synapse.lib.slabseqn.SlabSeqn method), 301 method), 328 indx() (synapse.lib.types.Bool method), 318 indxByPref() (synapse.models.inet.Rfc2822Addr indx() (synapse.lib.types.Comp method), 318 method), 331 indx() (synapse.lib.types.Data method), 319 indxByRange() (synapse.lib.types.Time method), 322 indx() (synapse.lib.types.Edge method), 319 indxByRange() (synapse.lib.types.Type method), 323

Index 389 synapse Documentation, Release 0.1.0

inet_ntop() (in module innrTmpExtract() (synapse.lib.filepath.FpTar synapse.lib.platforms.common), 231 method), 268 inet_pton() (in module Int (class in synapse.lib.types), 320 synapse.lib.platforms.common), 231 int64en() (in module synapse.common), 342 InetModule (class in synapse.models.inet), 330 int64un() (in module synapse.common), 342 info() (synapse.cryotank.CryoTank method), 352 IntBase (class in synapse.lib.types), 320 init() (in module synapse.cells), 340 IntervalTree (class in synapse.lib.trees), 317 init() (synapse.cryotank.CryoApi method), 351 intify() (in module synapse.common), 342 init() (synapse.cryotank.CryoCell method), 352 intrange() (in module synapse.lib.chop), 260 init() (synapse.lib.ast.AstNode method), 233 intstr() (in module synapse.lib.chop), 260 initCellAuth() (synapse.lib.auth.Auth method), 241 IPv4 (class in synapse.models.inet), 330 initCellAuth() (synapse.lib.cell.Cell method), 249 ipv4 (synapse.lib.platforms.windows.sockaddr at- initConfDefs() (synapse.lib.auth.Auth method), 241 tribute), 231 initCoreModule() (synapse.lib.module.CoreModule IPv6 (class in synapse.models.inet), 330 method), 290 ipv6 (synapse.lib.platforms.windows.sockaddr at- initCoreModule() (synapse.models.files.FileModule tribute), 231 method), 328 IPv6Range (class in synapse.models.inet), 330 initCoreModule() (synapse.models.inet.InetModule is_literal() (in module synapse.lib.syntax), 313 method), 330 isCaCert() (synapse.lib.certdir.CertDir method), 257 initCoreModule() (synapse.models.infotech.ItModule isClientCert() (synapse.lib.certdir.CertDir method), 331 method), 258 initCoreModule() (synapse.models.syn.SynModule iscoro() (in module synapse.lib.coro), 264 method), 333 isdir() (in module synapse.lib.filepath), 269 initdb() (synapse.lib.lmdblayer.LmdbLayer method), isfile() (in module synapse.lib.filepath), 269 287 isfile() (synapse.lib.filepath.FpFile method), 268 initdb() (synapse.lib.lmdbslab.Slab method), 287 IsFini, 368 initelem() (in module synapse.lib.datapath), 265 isfini() (in module synapse.lib.threads), 316 initFromDirn() (in module synapse.cells), 340 isform() (synapse.datamodel.ModelInfo method), 359 initHostInfo() (in module isguid() (in module synapse.common), 342 synapse.lib.platforms.common), 231 isHostCert() (synapse.lib.certdir.CertDir method), initHostInfo() (in module 258 synapse.lib.platforms.darwin), 231 isprop() (synapse.datamodel.ModelInfo method), 359 initHostInfo() (in module isquote() (in module synapse.lib.syntax), 313 synapse.lib.platforms.freebsd), 231 IsReadOnly, 368 initHostInfo() (in module IsRuntForm, 368 synapse.lib.platforms.linux), 231 isRuntSafe() (synapse.lib.ast.CompValue method), initHostInfo() (in module 234 synapse.lib.platforms.windows), 231 isRuntSafe() (synapse.lib.ast.RunValue method), initloop() (in module synapse.glob), 371 238 initPath() (synapse.lib.storm.Runtime method), 308 isRuntSafe() (synapse.lib.ast.VarValue method), 240 initPhoneTree() (in module isRuntVar() (synapse.lib.storm.Runtime method), synapse.lookup.phonenum), 327 308 initRuleTree() (synapse.lib.auth.Role method), 242 isuniv() (synapse.datamodel.ModelInfo method), 360 inline() (synapse.lib.ast.SubQuery method), 238 isUserCert() (synapse.lib.certdir.CertDir method), innerLs() (synapse.lib.filepath.FpTar method), 268 258 innrEnum() (synapse.lib.filepath.FpTar method), 268 isValidCert() (synapse.lib.crypto.vault.Vault innrEnum() (synapse.lib.filepath.FpZip method), 269 method), 230 innrExists() (synapse.lib.filepath.FpTar method), items() (synapse.eventbus.BusRef method), 361 268 items() (synapse.exc.SynErr method), 371 innrIsdir() (synapse.lib.filepath.FpTar method), 268 items() (synapse.lib.base.BaseRef method), 247 innrIsfile() (synapse.lib.filepath.FpTar method), items() (synapse.lib.hive.HiveDict method), 273 268 items() (synapse.lib.kv.KvDict method), 277 innrOpen() (synapse.lib.filepath.FpTar method), 268 items() (synapse.lib.kv.KvLook method), 278 innrOpen() (synapse.lib.filepath.FpZip method), 269 iter() (synapse.lib.datapath.DataElem method), 265

390 Index synapse Documentation, Release 0.1.0

iter() (synapse.lib.lmdb.Metrics method), 283 KvDict (class in synapse.lib.kv), 277 iter() (synapse.lib.lmdb.Seqn method), 285 KvLook (class in synapse.lib.kv), 277 iter() (synapse.lib.scope.Scope method), 300 KvSet (class in synapse.lib.kv), 278 iter() (synapse.lib.slabseqn.SlabSeqn method), 301 KvStor (class in synapse.lib.kv), 279 iterdata() (in module synapse.lib.encoding), 267 iterfd() (in module synapse.common), 342 L iterfd() (in module synapse.lib.msgpack), 291 LangModule (class in synapse.models.language), 332 iterfile() (in module synapse.lib.msgpack), 291 last() (synapse.cryotank.CryoApi method), 351 iterFormRows() (synapse.lib.layer.Layer method), last() (synapse.cryotank.CryoTank method), 352 281 last() (synapse.lib.trees.Tree method), 317 iterFormRows() (synapse.lib.lmdblayer.LmdbLayer last_key() (synapse.lib.lmdbslab.Scan method), 287 method), 287 Latitude (class in synapse.models.geospace), 329 iterFormRows() (synapse.lib.remotelayer.RemoteLayer LatLong (class in synapse.models.geospace), 329 method), 299 latlong() (in module synapse.lib.gis), 271 iterFqdnUp() (in module synapse.lib.certdir), 260 Layer (class in synapse.lib.layer), 281 iterKvDups() (synapse.lib.kv.KvStor method), 280 LayerApi (class in synapse.lib.layer), 282 iterKvProps() (synapse.lib.kv.KvStor method), 280 ldict() (in module synapse.lib.mixins), 289 iternext() (synapse.lib.lmdbslab.Scan method), 287 leave() (synapse.lib.scope.Scope method), 300 iterNodePaths() (synapse.lib.ast.Query method), Lib (class in synapse.lib.stormtypes), 310 237 LibTime (class in synapse.lib.stormtypes), 310 iterpath() (in module synapse.lib.hive), 274 lift() (synapse.lib.ast.LiftByScrape method), 235 iterPropRows() (synapse.lib.layer.Layer method), lift() (synapse.lib.ast.LiftFormTag method), 236 281 lift() (synapse.lib.ast.LiftProp method), 236 iterPropRows() (synapse.lib.lmdblayer.LmdbLayer lift() (synapse.lib.ast.LiftPropBy method), 236 method), 287 lift() (synapse.lib.ast.LiftTag method), 236 iterPropRows() (synapse.lib.remotelayer.RemoteLayer lift() (synapse.lib.ast.LiftTagTag method), 236 method), 299 LiftByScrape (class in synapse.lib.ast), 235 iterright() (synapse.lib.ast.AstNode method), 233 liftbytag() (synapse.lib.syntax.Parser method), 312 iterStormPodes() (synapse.cortex.Cortex method), LiftFormTag (class in synapse.lib.ast), 235 349 LiftOper (class in synapse.lib.ast), 236 iterStormPodes() (synapse.lib.snap.Snap method), LiftProp (class in synapse.lib.ast), 236 304 LiftPropBy (class in synapse.lib.ast), 236 iterStormQuery() (synapse.lib.storm.Runtime LiftTag (class in synapse.lib.ast), 236 method), 308 LiftTagTag (class in synapse.lib.ast), 236 iterUnivRows() (synapse.lib.layer.Layer method), lifttagtag() (synapse.lib.syntax.Parser method), 281 312 iterUnivRows() (synapse.lib.lmdblayer.LmdbLayer LimitCmd (class in synapse.lib.storm), 306 method), 287 Link (class in synapse.lib.link), 282 iterUnivRows() (synapse.lib.remotelayer.RemoteLayer link() (synapse.eventbus.EventBus method), 363 method), 299 link() (synapse.lib.base.Base method), 244 iterzip() (in module synapse.common), 343 LinkErr, 368 ItModule (class in synapse.models.infotech), 331 LinkShutDown, 368 Ival (class in synapse.lib.types), 320 List (class in synapse.lib.ast), 236 ival() (in module synapse.lib.time), 317 list() (synapse.cryotank.CryoApi method), 351 list() (synapse.cryotank.CryoCell method), 352 J list() (synapse.lib.agenda.Agenda method), 232 JobErr, 368 list() (synapse.lib.coro.GenrHelp method), 264 jsload() (in module synapse.common), 343 list() (synapse.lib.trigger.Triggers method), 318 jssave() (in module synapse.common), 343 listCronJobs() (synapse.cortex.CoreApi method), 347 K listdir() (in module synapse.common), 343 kill() (synapse.lib.cell.CellApi method), 249 listen() (in module synapse.lib.link), 282 kill() (synapse.lib.task.Task method), 313 listen() (synapse.daemon.Daemon method), 357 KillCmd (class in synapse.cmds.cortex), 219

Index 391 synapse Documentation, Release 0.1.0

listHiveKey() (synapse.lib.cell.CellApi method), makeargpaser() (in module synapse.tools.deploy), 249 336 listTriggers() (synapse.cortex.CoreApi method), makeColLook() (in module synapse.lookup.iso3166), 347 326 LmdbLayer (class in synapse.lib.lmdblayer), 286 makedirs() (in module synapse.common), 343 load() (synapse.lib.crypto.ecc.PriKey static method), MatModule (class in synapse.models.material), 332 223 MAX_OUTSTANDING (synapse.axon.IncrementalTransaction load() (synapse.lib.crypto.ecc.PubKey static method), attribute), 339 224 max_temp_sz (in module synapse.lib.filepath), 269 load() (synapse.lib.crypto.vault.Cert static method), MAX_WAIT_S (synapse.cryotank.CryoTankIndexer at- 227 tribute), 353 loadCoreModule() (synapse.cortex.Cortex method), MaxCmd (class in synapse.lib.storm), 306 349 MediaModule (class in synapse.models.media), 332 loadDmonCell() (synapse.daemon.Daemon method), meh() (in module synapse.lib.syntax), 313 357 memoize() (in module synapse.lib.cache), 248 loadfile() (in module synapse.lib.msgpack), 291 merge() (synapse.lib.types.Int method), 320 Loc (class in synapse.lib.types), 321 merge() (synapse.lib.types.Ival method), 320 lockfile() (in module synapse.common), 343 merge() (synapse.lib.types.Time method), 322 locs() (synapse.axon.Axon method), 337 merge() (synapse.lib.types.Type method), 323 Log (class in synapse.cmds.cortex), 219 mergeRanges() (in module synapse.lib.chop), 260 log() (synapse.eventbus.EventBus method), 363 meta() (synapse.lib.node.Path method), 294 Longitude (class in synapse.models.geospace), 329 Method (class in synapse.telepath), 373 loop() (in module synapse.lib.task), 314 Metrics (class in synapse.lib.lmdb), 282 metrics() (synapse.axon.Axon method), 337 M metrics() (synapse.axon.BlobStor method), 338 main() (in module synapse.servers.cortex), 334 metrics() (synapse.cryotank.CryoApi method), 351 main() (in module synapse.tools.autodoc), 334 metrics() (synapse.cryotank.CryoTank method), 352 main() (in module synapse.tools.backup), 335 metrics() (synapse.cryotank.TankApi method), 356 main() (in module synapse.tools.cellauth), 335 Migration (class in synapse.lib.migrate), 288 main() (in module synapse.tools.cmdr), 335 MinCmd (class in synapse.lib.storm), 306 main() (in module synapse.tools.cryo.cat), 334 MINUTE (synapse.lib.agenda.TimeUnit attribute), 233 main() (in module synapse.tools.cryo.list), 334 mod() (synapse.lib.agenda.Agenda method), 232 main() (in module synapse.tools.csvtool), 335 mod() (synapse.lib.trigger.Triggers method), 318 main() (in module synapse.tools.deploy), 336 mod_name (synapse.lib.module.CoreModule attribute), main() (in module synapse.tools.dmon), 336 290 main() (in module synapse.tools.easycert), 336 Model (class in synapse.datamodel), 358 main() (in module synapse.tools.feed), 336 ModelInfo (class in synapse.datamodel), 359 main() (in module synapse.tools.guid), 336 ModelRev (class in synapse.lib.modelrev), 289 main() (in module synapse.tools.json2mpk), 336 MONTH (synapse.lib.agenda.TimeUnit attribute), 233 main() (in module synapse.tools.pullfile), 336 more() (synapse.lib.syntax.Parser method), 312 main() (in module synapse.tools.pushfile), 336 MoveTagCmd (class in synapse.lib.storm), 306 main() (synapse.eventbus.EventBus method), 363 MustBeLocal, 368 main() (synapse.lib.base.Base method), 245 mainopts() (in module synapse.servers.cortex), 334 N makeargparser() (in module name (synapse.lib.storm.Cmd attribute), 304 synapse.tools.autodoc), 334 name (synapse.lib.storm.CountCmd attribute), 305 makeargparser() (in module name (synapse.lib.storm.DelNodeCmd attribute), 305 synapse.tools.cellauth), 335 name (synapse.lib.storm.GraphCmd attribute), 305 makeargparser() (in module synapse.tools.csvtool), name (synapse.lib.storm.HelpCmd attribute), 305 335 name (synapse.lib.storm.IdenCmd attribute), 306 makeargparser() (in module synapse.tools.feed), name (synapse.lib.storm.LimitCmd attribute), 306 336 name (synapse.lib.storm.MaxCmd attribute), 306 makeargparser() (in module synapse.tools.pushfile), name (synapse.lib.storm.MinCmd attribute), 306 336 name (synapse.lib.storm.MoveTagCmd attribute), 307

392 Index synapse Documentation, Release 0.1.0

name (synapse.lib.storm.NoderefsCmd attribute), 307 NoSuchEncoder, 369 name (synapse.lib.storm.ReIndexCmd attribute), 308 NoSuchFile, 369 name (synapse.lib.storm.SleepCmd attribute), 309 NoSuchForm, 369 name (synapse.lib.storm.SpinCmd attribute), 309 NoSuchFunc, 369 name (synapse.lib.storm.SudoCmd attribute), 309 NoSuchIden, 369 name (synapse.lib.storm.UniqCmd attribute), 310 NoSuchImpl, 369 name() (synapse.lib.datapath.DataElem method), 265 NoSuchIndx, 369 name() (synapse.lib.hive.Node method), 273 NoSuchLift, 369 Ndef (class in synapse.lib.types), 321 NoSuchMeth, 369 ndef() (in module synapse.lib.node), 294 NoSuchName, 369 near() (in module synapse.lib.gis), 271 NoSuchObj, 369 newkey() (in module synapse.lib.crypto.tinfoil), 226 NoSuchOper, 369 next() (synapse.lib.filepath.FpFile method), 268 NoSuchOpt, 369 next() (synapse.lib.filepath.FpTar method), 268 NoSuchPath, 370 nextchar() (in module synapse.lib.syntax), 313 NoSuchPivot, 370 nextchar() (synapse.lib.syntax.Parser method), 312 NoSuchProp, 370 nextin() (in module synapse.lib.syntax), 313 NoSuchRole, 370 nextindx() (synapse.lib.lmdb.Seqn method), 286 NoSuchRule, 370 nextindx() (synapse.lib.slabseqn.SlabSeqn method), NoSuchSeq, 370 302 NoSuchStor, 370 nextmust() (synapse.lib.syntax.Parser method), 312 NoSuchType, 370 nexts() (synapse.lib.filepath.FpFile method), 268 NoSuchUser, 370 nexts() (synapse.lib.filepath.FpTar method), 269 NoSuchVar, 370 nextstr() (in module synapse.lib.syntax), 313 NotCond (class in synapse.lib.ast), 236 nextstr() (synapse.lib.syntax.Parser method), 312 NotReady, 370 nextstrs() (synapse.lib.syntax.Parser method), 312 NoValu (class in synapse.common), 340 nexttime() (synapse.lib.agenda.ApptRec method), now() (in module synapse.common), 343 232 NoCertKey, 368 O NoCurrTask, 368 off() (synapse.eventbus.EventBus method), 363 Node (class in synapse.lib.hive), 273 off() (synapse.lib.base.Base method), 245 Node (class in synapse.lib.node), 292 Offs (class in synapse.lib.lmdb), 283 Node (class in synapse.lib.stormtypes), 310 offset() (synapse.cryotank.CryoApi method), 351 NodeProp (class in synapse.lib.types), 321 offset() (synapse.cryotank.TankApi method), 356 NoderefsCmd (class in synapse.lib.storm), 307 offTagAdd() (synapse.cortex.Cortex method), 349 nom() (in module synapse.lib.syntax), 313 offTagDel() (synapse.cortex.Cortex method), 349 nom_whitespace() (in module synapse.lib.syntax), omit() (synapse.lib.ast.SubGraph method), 238 313 on() (synapse.eventbus.EventBus method), 363 NoModIden, 368 on() (synapse.lib.base.Base method), 245 noms() (synapse.lib.syntax.Parser method), 312 onAdd() (synapse.datamodel.Form method), 358 NoRevPath, 369 onDel() (synapse.datamodel.Form method), 358 norm() (synapse.lib.types.Data method), 319 onDel() (synapse.datamodel.PropBase method), 360 norm() (synapse.lib.types.Type method), 323 onedit() (synapse.lib.hive.Hive method), 272 normdict() (in module synapse.lib.hashitem), 271 onespace() (in module synapse.lib.chop), 260 normitem() (in module synapse.lib.hashitem), 271 onfini() (synapse.eventbus.EventBus method), 364 normiter() (in module synapse.lib.hashitem), 271 onfini() (synapse.lib.base.Base method), 245 normpath() (in module synapse.lib.filepath), 269 onSet() (synapse.datamodel.PropBase method), 360 NoSuchAct, 369 onStormMesg() (synapse.cmds.cortex.Log method), NoSuchCmpr, 369 220 NoSuchCond, 369 onTagAdd() (synapse.cortex.Cortex method), 349 NoSuchCtor, 369 onTagDel() (synapse.cortex.Cortex method), 349 NoSuchDecoder, 369 onTeleOpen() (synapse.cryotank.CryoCell method), NoSuchDir, 369 352 NoSuchDyn, 369 onTeleOpen() (synapse.telepath.Aware method), 372

Index 393 synapse Documentation, Release 0.1.0 onTeleShare() (synapse.telepath.Aware method), parsePaths() (in module synapse.lib.filepath), 270 373 Parser (class in synapse.lib.cmd), 263 onWith() (synapse.eventbus.EventBus method), 364 Parser (class in synapse.lib.storm), 308 open() (synapse.lib.filepath.FpFile method), 268 Parser (class in synapse.lib.syntax), 310 open() (synapse.lib.filepath.FpTar method), 269 ParserExit, 370 open() (synapse.lib.hive.Hive method), 272 parseSemver() (in module synapse.lib.version), 325 open() (synapse.lib.hive.Node method), 273 parsetime() (in module synapse.lib.interval), 274 open() (synapse.lib.hive.TeleHive method), 274 parseVersionParts() (in module openDatFile() (in module synapse.lib.datfile), 266 synapse.lib.version), 325 opendir() (in module synapse.lib.hive), 274 PassThroughApi (class in synapse.lib.cell), 249 openfile() (in module synapse.lib.filepath), 269 Path (class in synapse.lib.node), 293 openfiles() (in module synapse.lib.filepath), 269 path() (synapse.lib.filepath.FpFile method), 268 openLogFd() (synapse.cmds.cortex.Log method), 220 path() (synapse.lib.filepath.FpTar method), 269 openurl() (in module synapse.lib.hive), 274 pauseIndex() (synapse.cryotank.CryoTankIndexer Oper (class in synapse.lib.ast), 236 method), 354 oper() (synapse.lib.syntax.Parser method), 312 pauseIndex() (synapse.cryotank.TankApi method), optimize() (synapse.lib.ast.AstNode method), 233 356 OrCond (class in synapse.lib.ast), 236 peek() (synapse.lib.syntax.Parser method), 312 ornot() (in module synapse.lib.coro), 264 phnode() (in module synapse.lookup.phonenum), 327 OuModule (class in synapse.models.orgs), 332 Phone (class in synapse.models.telco), 333 OutPut (class in synapse.lib.output), 295 PivotIn (class in synapse.lib.ast), 236 OutPutBytes (class in synapse.lib.output), 295 PivotInFrom (class in synapse.lib.ast), 236 OutPutFd (class in synapse.lib.output), 295 PivotOper (class in synapse.lib.ast), 237 OutPutStr (class in synapse.lib.output), 295 PivotOut (class in synapse.lib.ast), 237 overlap() (in module synapse.lib.interval), 274 pivots() (synapse.lib.ast.SubGraph method), 238 PivotToTags (class in synapse.lib.ast), 237 P pokeFormCount() (synapse.cortex.Cortex method), pack() (synapse.datamodel.Form method), 358 350 pack() (synapse.datamodel.Prop method), 360 PolModule (class in synapse.models.geopol), 328 pack() (synapse.lib.agenda.ApptRec method), 232 Pool (class in synapse.lib.db), 266 pack() (synapse.lib.node.Node method), 293 Pool (class in synapse.lib.threads), 314 pack() (synapse.lib.node.Path method), 294 pop() (in module synapse.lib.scope), 301 pack() (synapse.lib.task.Task method), 313 pop() (synapse.eventbus.BusRef method), 361 pack() (synapse.lib.types.Type method), 324 pop() (synapse.lib.base.BaseRef method), 247 packVersion() (in module synapse.lib.version), 324 pop() (synapse.lib.cache.FixedCache method), 248 parent() (synapse.lib.hive.Node method), 273 pop() (synapse.lib.hive.Hive method), 272 parse() (in module synapse.lib.time), 317 pop() (synapse.lib.hive.HiveDict method), 273 parse_args() (in module synapse.tools.backup), 335 pop() (synapse.lib.kv.KvDict method), 277 parse_cmd_kwarg() (in module synapse.lib.syntax), pop() (synapse.lib.lmdbslab.Slab method), 287 313 pop() (synapse.lib.node.Node method), 293 parse_cmd_kwlist() (in module pop() (synapse.lib.scope.Scope method), 300 synapse.lib.syntax), 313 popHiveKey() (synapse.lib.cell.CellApi method), 249 parse_cmd_string() (in module popSessItem() (synapse.daemon.Sess method), 358 synapse.lib.syntax), 313 postTypeInit() (synapse.lib.types.Bool method), parse_float() (in module synapse.lib.syntax), 313 318 parse_int() (in module synapse.lib.syntax), 313 postTypeInit() (synapse.lib.types.Comp method), parse_list() (in module synapse.lib.syntax), 313 319 parse_literal() (in module synapse.lib.syntax), postTypeInit() (synapse.lib.types.Edge method), 313 319 parse_string() (in module synapse.lib.syntax), 313 postTypeInit() (synapse.lib.types.Guid method), parse_valu() (in module synapse.lib.syntax), 313 319 parsePath() (in module synapse.lib.filepath), 270 postTypeInit() (synapse.lib.types.Hex method), 320 parsepath() (synapse.cmds.hive.HiveCmd static postTypeInit() (synapse.lib.types.Int method), 320 method), 222 postTypeInit() (synapse.lib.types.Ival method), 320

394 Index synapse Documentation, Release 0.1.0 postTypeInit() (synapse.lib.types.Loc method), 321 method), 333 postTypeInit() (synapse.lib.types.Ndef method), postTypeInit() (synapse.models.telco.Imsi 321 method), 333 postTypeInit() (synapse.lib.types.NodeProp postTypeInit() (synapse.models.telco.Phone method), 321 method), 333 postTypeInit() (synapse.lib.types.Range method), pref() (synapse.lib.lmdb.PropStor method), 285 321 prepare() (synapse.lib.ast.AstNode method), 233 postTypeInit() (synapse.lib.types.Str method), 321 prepare() (synapse.lib.ast.PropValue method), 237 postTypeInit() (synapse.lib.types.StrBase method), prepare() (synapse.lib.ast.SwitchCase method), 239 321 prepare() (synapse.lib.ast.TagPropValue method), postTypeInit() (synapse.lib.types.Tag method), 322 239 postTypeInit() (synapse.lib.types.Time method), prepare() (synapse.lib.ast.VarValue method), 240 322 PriKey (class in synapse.lib.crypto.ecc), 223 postTypeInit() (synapse.lib.types.TimeEdge Prim (class in synapse.lib.stormtypes), 310 method), 322 printables() (in module synapse.lib.chop), 260 postTypeInit() (synapse.lib.types.Type method), printf() (synapse.lib.cli.Cli method), 261 324 printf() (synapse.lib.cli.Cmd method), 262 postTypeInit() (synapse.models.dns.DnsName printf() (synapse.lib.output.OutPut method), 295 method), 328 printf() (synapse.lib.snap.Snap method), 304 postTypeInit() (synapse.models.files.FileBase printf() (synapse.lib.storm.Runtime method), 308 method), 328 printuser() (in module synapse.tools.cellauth), 335 postTypeInit() (synapse.models.files.FileBytes processCtors() (in module synapse.tools.autodoc), method), 328 334 postTypeInit() (synapse.models.files.FilePath processFormsProps() (in module method), 328 synapse.tools.autodoc), 335 postTypeInit() (synapse.models.geospace.Dist processTypes() (in module synapse.tools.autodoc), method), 329 335 postTypeInit() (synapse.models.geospace.Latitude processUnivs() (in module synapse.tools.autodoc), method), 329 335 postTypeInit() (synapse.models.geospace.LatLong promote() (synapse.lib.boss.Boss method), 248 method), 329 Prop (class in synapse.datamodel), 360 postTypeInit() (synapse.models.geospace.Longitude prop (synapse.lib.trigger.Triggers.Rule attribute), 318 method), 329 prop() (in module synapse.lib.node), 294 postTypeInit() (synapse.models.inet.Addr method), prop() (synapse.datamodel.Form method), 358 329 prop() (synapse.datamodel.Model method), 359 postTypeInit() (synapse.models.inet.Cidr4 PropBase (class in synapse.datamodel), 360 method), 330 propjoin() (synapse.lib.syntax.Parser method), 312 postTypeInit() (synapse.models.inet.Email PropPivot (class in synapse.lib.ast), 237 method), 330 proppivot() (synapse.lib.syntax.Parser method), 312 postTypeInit() (synapse.models.inet.Fqdn PropPivotOut (class in synapse.lib.ast), 237 method), 330 props() (in module synapse.lib.node), 294 postTypeInit() (synapse.models.inet.IPv4 method), PropSetr (class in synapse.lib.lmdb), 283 330 PropStor (class in synapse.lib.lmdb), 284 postTypeInit() (synapse.models.inet.IPv6 method), PropValue (class in synapse.lib.ast), 237 330 Proxy (class in synapse.telepath), 373 postTypeInit() (synapse.models.inet.IPv6Range ps() (synapse.lib.boss.Boss method), 248 method), 330 ps() (synapse.lib.cell.CellApi method), 249 postTypeInit() (synapse.models.inet.Rfc2822Addr PsCmd (class in synapse.cmds.cortex), 220 method), 331 PsModule (class in synapse.models.person), 332 postTypeInit() (synapse.models.inet.Url method), PubKey (class in synapse.lib.crypto.ecc), 224 331 public() (synapse.lib.crypto.ecc.PriKey method), 223 postTypeInit() (synapse.models.infotech.SemVer public() (synapse.lib.crypto.vault.Cert method), 227 method), 332 put() (synapse.axon.IncrementalTransaction method), postTypeInit() (synapse.models.telco.Imei 339

Index 395 synapse Documentation, Release 0.1.0

put() (synapse.eventbus.BusRef method), 362 release() (synapse.lib.threads.RWLock method), 315 put() (synapse.lib.base.BaseRef method), 247 RelProp (class in synapse.lib.ast), 237 put() (synapse.lib.cache.FixedCache method), 248 relprop() (synapse.lib.syntax.Parser method), 312 put() (synapse.lib.lmdb.PropSetr method), 284 RelPropCond (class in synapse.lib.ast), 237 put() (synapse.lib.lmdbslab.Slab method), 288 relpropvalu() (synapse.lib.syntax.Parser method), put() (synapse.lib.queue.AQueue method), 295 312 put() (synapse.lib.queue.AsyncQueue method), 296 RelPropValue (class in synapse.lib.ast), 237 put() (synapse.lib.queue.Queue method), 296 rem() (synapse.lib.cache.TagGlobs method), 248 put() (synapse.lib.queue.S2AQueue method), 298 RemoteLayer (class in synapse.lib.remotelayer), 299 put() (synapse.lib.trees.Tree method), 318 remove() (synapse.lib.kv.KvSet method), 278 putmany() (synapse.axon.BlobStor method), 338 rendevous() (synapse.lib.editatom.EditAtom putmulti() (synapse.lib.lmdbslab.Slab method), 288 method), 267 putone() (synapse.axon.Axon method), 337 replace() (synapse.lib.ast.AstNode method), 234 putone() (synapse.axon.BlobStor method), 338 replace() (synapse.lib.lmdbslab.Slab method), 288 puts() (synapse.cryotank.CryoApi method), 351 reply() (synapse.telepath.Task method), 374 puts() (synapse.cryotank.CryoTank method), 352 repr() (in module synapse.lib.time), 317 puts() (synapse.cryotank.TankApi method), 356 repr() (synapse.lib.ast.AbsProp method), 233 repr() (synapse.lib.ast.AstNode method), 234 Q repr() (synapse.lib.ast.Cmpr method), 234 Query (class in synapse.lib.ast), 237 repr() (synapse.lib.ast.Const method), 234 query() (synapse.lib.syntax.Parser method), 312 repr() (synapse.lib.ast.List method), 236 queryNormRecords() repr() (synapse.lib.ast.RelProp method), 237 (synapse.cryotank.CryoTankIndexer method), repr() (synapse.lib.ast.UnivProp method), 239 354 repr() (synapse.lib.node.Node method), 293 queryNormRecords() (synapse.cryotank.TankApi repr() (synapse.lib.types.Bool method), 318 method), 356 repr() (synapse.lib.types.Comp method), 319 queryNormValu() (synapse.cryotank.CryoTankIndexer repr() (synapse.lib.types.Edge method), 319 method), 354 repr() (synapse.lib.types.Int method), 320 queryNormValu() (synapse.cryotank.TankApi repr() (synapse.lib.types.Ival method), 320 method), 356 repr() (synapse.lib.types.Range method), 321 queryRows() (synapse.cryotank.CryoTankIndexer repr() (synapse.lib.types.Time method), 322 method), 355 repr() (synapse.lib.types.TimeEdge method), 322 queryRows() (synapse.cryotank.TankApi method), repr() (synapse.lib.types.Type method), 324 356 repr() (synapse.models.geospace.Dist method), 329 Queue (class in synapse.lib.queue), 296 repr() (synapse.models.geospace.LatLong method), queueLoop() (synapse.cmds.cortex.Log method), 220 329 quoted() (synapse.lib.syntax.Parser method), 312 repr() (synapse.models.inet.Fqdn method), 330 repr() (synapse.models.inet.IPv4 method), 330 R repr() (synapse.models.infotech.SemVer method), 332 repr() Range (class in synapse.lib.types), 321 (synapse.models.telco.Phone method), 333 reprrule() range() (synapse.lib.lmdb.PropStor method), 285 (in module synapse.tools.cellauth), 335 reprs() RateLimit (class in synapse.lib.ratelimit), 298 (synapse.lib.node.Node method), 293 reqAdmin() react() (synapse.reactor.Reactor method), 372 (in module synapse.lib.auth), 243 reqbytes() Reactor (class in synapse.reactor), 372 (in module synapse.common), 344 ReqConfOpt read() (synapse.lib.filepath.FpOpener method), 268 , 370 reqdir() reader() (synapse.lib.threads.RWLock method), 315 (in module synapse.common), 344 reqfile() ReadOnlyProp, 370 (in module synapse.common), 344 reqlines() record() (synapse.lib.lmdb.Metrics method), 283 (in module synapse.common), 344 reqok() recs() (synapse.lib.lmdb.PropStor method), 285 (in module synapse.common), 344 reqpath() RecursionLimitHit, 370 (in module synapse.common), 344 reqRole() recv() (synapse.lib.link.Link method), 282 (synapse.lib.auth.Auth method), 241 reqUser() recvsize() (synapse.lib.link.Link method), 282 (synapse.lib.auth.Auth method), 241 result() reflectItem() (synapse.lib.cli.Cli method), 261 (in module synapse.common), 344 result() ReIndexCmd (class in synapse.lib.storm), 308 (synapse.telepath.Task method), 374

396 Index synapse Documentation, Release 0.1.0

resumeIndex() (synapse.cryotank.CryoTankIndexer runCmdOpts() (synapse.cmds.cortex.PsCmd method), method), 355 220 resumeIndex() (synapse.cryotank.TankApi method), runCmdOpts() (synapse.cmds.cortex.StormCmd 357 method), 221 retn() (synapse.lib.threads.RetnWait method), 316 runCmdOpts() (synapse.cmds.cron.At method), 221 retnexc() (in module synapse.common), 344 runCmdOpts() (synapse.cmds.cron.Cron method), 222 RetnWait (class in synapse.lib.threads), 315 runCmdOpts() (synapse.cmds.hive.HiveCmd method), Retry, 370 222 revCoreLayers() (synapse.lib.modelrev.ModelRev runCmdOpts() (synapse.cmds.trigger.Trigger method), 289 method), 222 revoke() (synapse.lib.hive.HiveUser method), 273 runCmdOpts() (synapse.lib.cli.Cmd method), 262 Rfc2822Addr (class in synapse.models.inet), 331 runCmdOpts() (synapse.lib.cli.CmdHelp method), 262 Role (class in synapse.lib.auth), 241 runCmdOpts() (synapse.lib.cli.CmdLocals method), role() (synapse.lib.hive.HiveAuth method), 272 263 rows() (synapse.cryotank.CryoApi method), 351 runCmdOpts() (synapse.lib.cli.CmdQuit method), 263 rows() (synapse.cryotank.CryoTank method), 352 runDynTask() (in module synapse.dyndeps), 361 rowstotufos() (in module synapse.common), 344 runItemCmdr() (in module synapse.lib.cmdr), 264 RstHelp (class in synapse.tools.autodoc), 334 runNodeAdd() (synapse.lib.trigger.Triggers method), run() (synapse.lib.ast.BreakOper method), 234 318 run() (synapse.lib.ast.CmdOper method), 234 runNodeDel() (synapse.lib.trigger.Triggers method), run() (synapse.lib.ast.ContinueOper method), 234 318 run() (synapse.lib.ast.EditNodeAdd method), 234 runPropSet() (synapse.lib.trigger.Triggers method), run() (synapse.lib.ast.EditPropDel method), 235 318 run() (synapse.lib.ast.EditPropSet method), 235 runRuntLift() (synapse.cortex.Cortex method), 350 run() (synapse.lib.ast.EditTagAdd method), 235 runRuntPropDel() (synapse.cortex.Cortex method), run() (synapse.lib.ast.EditTagDel method), 235 350 run() (synapse.lib.ast.EditUnivDel method), 235 runRuntPropSet() (synapse.cortex.Cortex method), run() (synapse.lib.ast.FiltOper method), 235 350 run() (synapse.lib.ast.ForLoop method), 235 runTagAdd() (synapse.cortex.Cortex method), 350 run() (synapse.lib.ast.FormPivot method), 235 runTagAdd() (synapse.lib.trigger.Triggers method), run() (synapse.lib.ast.LiftOper method), 236 318 run() (synapse.lib.ast.PivotIn method), 236 runTagDel() (synapse.cortex.Cortex method), 350 run() (synapse.lib.ast.PivotInFrom method), 237 runTagDel() (synapse.lib.trigger.Triggers method), run() (synapse.lib.ast.PivotOut method), 237 318 run() (synapse.lib.ast.PivotToTags method), 237 Runtime (class in synapse.lib.storm), 308 run() (synapse.lib.ast.PropPivot method), 237 runtval() (synapse.lib.ast.CallArgs method), 234 run() (synapse.lib.ast.PropPivotOut method), 237 runtval() (synapse.lib.ast.FuncCall method), 235 run() (synapse.lib.ast.Query method), 237 runtval() (synapse.lib.ast.List method), 236 run() (synapse.lib.ast.SubGraph method), 238 runtval() (synapse.lib.ast.RunValue method), 238 run() (synapse.lib.ast.SubQuery method), 238 runtval() (synapse.lib.ast.Value method), 239 run() (synapse.lib.ast.SwitchCase method), 239 runtval() (synapse.lib.ast.VarDeref method), 239 run() (synapse.lib.ast.VarListSetOper method), 239 runtval() (synapse.lib.ast.VarValue method), 240 run() (synapse.lib.ast.VarSetOper method), 240 RunValue (class in synapse.lib.ast), 237 run() (synapse.lib.threads.Thread method), 316 RWLock (class in synapse.lib.threads), 315 runas() (in module synapse.lib.auth), 243 RWWith (class in synapse.lib.threads), 315 runCmdLine() (synapse.lib.cli.Cli method), 261 rx() (synapse.lib.link.Link method), 282 runCmdLine() (synapse.lib.cli.Cmd method), 262 runCmdLine() (synapse.lib.jupyter.CmdrCore S method), 275 S2AQueue (class in synapse.lib.queue), 298 runCmdLoop() (synapse.lib.cli.Cli method), 261 sa_family (synapse.lib.platforms.windows.sockaddr runCmdOpts() (synapse.cmds.cortex.KillCmd attribute), 231 method), 219 save() (synapse.cmds.cortex.Log method), 220 runCmdOpts() (synapse.cmds.cortex.Log method), save() (synapse.lib.auth.Role method), 242 220 save() (synapse.lib.auth.User method), 243

Index 397 synapse Documentation, Release 0.1.0

save() (synapse.lib.lmdb.Seqn method), 286 setAndSync() (synapse.lib.hive.HiveApi method), save() (synapse.lib.slabseqn.SlabSeqn method), 302 272 SCALE (synapse.models.geospace.Latitude attribute), setAuthAdmin() (synapse.lib.cell.CellApi method), 329 249 SCALE (synapse.models.geospace.Longitude attribute), setCmprCtor() (synapse.lib.types.Type method), 324 329 setdefault() (synapse.lib.hive.HiveDict method), Scan (class in synapse.lib.lmdbslab), 287 273 scanByDups() (synapse.lib.lmdbslab.Slab method), setFeedFunc() (synapse.cortex.Cortex method), 350 288 setFeedOffs() (synapse.cortex.CoreApi method), scanByFull() (synapse.lib.lmdbslab.Slab method), 347 288 setFeedOffs() (synapse.cortex.Cortex method), 350 scanByPref() (synapse.lib.lmdbslab.Slab method), setHiveKey() (synapse.lib.cell.CellApi method), 249 288 setKid() (synapse.lib.ast.AstNode method), 234 scanByRange() (synapse.lib.lmdbslab.Slab method), setKvProp() (synapse.lib.kv.KvStor method), 280 288 setKvProps() (synapse.lib.kv.KvStor method), 281 schedCallSafe() (synapse.lib.base.Base method), setLiftHintCmprCtor() (synapse.lib.types.Type 246 method), 324 schedCoro() (synapse.lib.base.Base method), 246 setLocked() (synapse.lib.auth.User method), 243 schedCoroSafe() (synapse.lib.base.Base method), setLocked() (synapse.lib.hive.HiveUser method), 273 246 setlogging() (in module synapse.common), 344 schedCoroSafePend() (synapse.lib.base.Base setModelVers() (synapse.lib.layer.Layer method), method), 246 281 Scope (class in synapse.lib.scope), 300 setModelVers() (synapse.lib.lmdblayer.LmdbLayer scrape() (in module synapse.lib.scrape), 301 method), 287 seek() (synapse.lib.filepath.FpOpener method), 268 setModelVers() (synapse.lib.remotelayer.RemoteLayer seen() (synapse.lib.node.Node method), 293 method), 300 select() (synapse.lib.db.Xact method), 266 setNodeBuid() (synapse.lib.migrate.Migration selfSignCert() (synapse.lib.certdir.CertDir method), 288 method), 258 setNodeForm() (synapse.lib.migrate.Migration SemVer (class in synapse.models.infotech), 331 method), 288 send() (synapse.lib.link.Link method), 282 setNodeProp() (synapse.cortex.CoreApi method), Seqn (class in synapse.lib.lmdb), 285 347 Sess (class in synapse.daemon), 358 setNormFunc() (synapse.lib.types.Type method), 324 set() (in module synapse.lib.scope), 301 setOffset() (synapse.cryotank.CryoTank method), set() (synapse.lib.cli.Cli method), 261 353 set() (synapse.lib.hive.Hive method), 272 setOffset() (synapse.lib.layer.Layer method), 281 set() (synapse.lib.hive.HiveDict method), 273 setOffset() (synapse.lib.lmdblayer.LmdbLayer set() (synapse.lib.hive.Node method), 273 method), 287 set() (synapse.lib.hive.TeleHive method), 274 setOffset() (synapse.lib.snap.Snap method), 304 set() (synapse.lib.kv.KvDict method), 277 setOpt() (synapse.lib.storm.Runtime method), 308 set() (synapse.lib.kv.KvLook method), 278 setPasswd() (synapse.lib.auth.User method), 243 set() (synapse.lib.link.Link method), 282 setPasswd() (synapse.lib.hive.HiveUser method), 273 set() (synapse.lib.lmdb.Offs method), 283 setProcName() (in module set() (synapse.lib.lmdb.PropSetr method), 284 synapse.lib.platforms.common), 231 set() (synapse.lib.node.Node method), 293 setPropsByType() (synapse.lib.migrate.Migration set() (synapse.lib.node.Path method), 294 method), 288 set() (synapse.lib.scope.Scope method), 300 setraw() (synapse.lib.kv.KvLook method), 278 set() (synapse.lib.slaboffs.SlabOffs method), 301 setSessItem() (synapse.daemon.Sess method), 358 set_key() (synapse.lib.lmdbslab.Scan method), 287 setup() (in module synapse.tools.pullfile), 336 set_range() (synapse.lib.lmdbslab.Scan method), setUser() (synapse.lib.snap.Snap method), 304 287 setUserLocked() (synapse.lib.cell.CellApi method), setAdmin() (synapse.lib.auth.Role method), 242 249 setAdmin() (synapse.lib.hive.HiveUser method), 273 setUserPasswd() (synapse.lib.cell.CellApi method), 249

398 Index synapse Documentation, Release 0.1.0

setVar() (synapse.lib.storm.Runtime method), 308 splicetypes (synapse.cmds.cortex.Log attribute), Share (class in synapse.lib.share), 301 220 Share (class in synapse.telepath), 373 startput() (synapse.axon.AxonApi method), 338 share() (synapse.daemon.Daemon method), 357 startput() (synapse.axon.BlobStorApi method), 339 shared() (in module synapse.lib.crypto.vault), 230 stat() (synapse.axon.Axon method), 337 sibling() (synapse.lib.ast.AstNode method), 234 stat() (synapse.axon.BlobStor method), 338 sign() (synapse.lib.crypto.ecc.PriKey method), 224 stat() (synapse.cortex.CoreApi method), 347 sign() (synapse.lib.crypto.vault.Cert method), 227 stat() (synapse.cortex.Cortex method), 350 signCertAs() (synapse.lib.certdir.CertDir method), stat() (synapse.lib.layer.Layer method), 281 259 stat() (synapse.lib.lmdb.Metrics method), 283 signed() (synapse.lib.crypto.vault.Cert method), 227 stat() (synapse.lib.lmdblayer.LmdbLayer method), signers() (synapse.lib.crypto.vault.Cert method), 227 287 signHostCsr() (synapse.lib.certdir.CertDir method), stems() (synapse.lib.types.Loc method), 321 259 step() (synapse.lib.datapath.DataElem method), 265 signUserCsr() (synapse.lib.certdir.CertDir method), StepTimeout, 370 259 stor() (synapse.lib.layer.Layer method), 281 singlequoted() (synapse.lib.syntax.Parser method), stor() (synapse.lib.snap.Snap method), 304 312 storm() (synapse.cortex.CoreApi method), 347 size() (synapse.lib.queue.Queue method), 297 storm() (synapse.cortex.Cortex method), 350 Slab (class in synapse.lib.lmdbslab), 287 storm() (synapse.lib.jupyter.CmdrCore method), 275 SLAB_MAP_SIZE (in module synapse.lib.cell), 249 storm() (synapse.lib.node.Node method), 293 SlabHive (class in synapse.lib.hive), 273 storm() (synapse.lib.snap.Snap method), 304 SlabOffs (class in synapse.lib.slaboffs), 301 StormBreak, 238 SlabSeqn (class in synapse.lib.slabseqn), 301 StormCmd (class in synapse.cmds.cortex), 220 SleepCmd (class in synapse.lib.storm), 308 stormcmd() (synapse.lib.syntax.Parser method), 312 slice() (synapse.cryotank.CryoApi method), 351 StormContinue, 238 slice() (synapse.cryotank.CryoTank method), 353 StormCtrlFlow, 238 slice() (synapse.cryotank.TankApi method), 357 StormRuntimeError, 370 slice() (synapse.lib.lmdb.Seqn method), 286 StormType (class in synapse.lib.stormtypes), 310 slice() (synapse.lib.queue.AQueue method), 295 StormVarListError, 370 slice() (synapse.lib.queue.Queue method), 297 storNodeDele() (synapse.lib.hive.SlabHive method), slice() (synapse.lib.slabseqn.SlabSeqn method), 302 273 slices() (synapse.lib.queue.Queue method), 297 storNodeValu() (synapse.lib.hive.Hive method), 272 Snap (class in synapse.lib.snap), 302 storNodeValu() (synapse.lib.hive.SlabHive method), snap() (synapse.cortex.Cortex method), 350 273 snap() (synapse.cortex.View method), 350 Str (class in synapse.lib.stormtypes), 310 sockaddr (class in synapse.lib.platforms.windows), Str (class in synapse.lib.types), 321 231 StrBase (class in synapse.lib.types), 321 SPACE (synapse.models.geospace.Latitude attribute), streamstorm() (synapse.cortex.Cortex method), 350 329 SubGraph (class in synapse.lib.ast), 238 SPACE (synapse.models.geospace.Longitude attribute), subpaths() (in module synapse.lib.filepath), 270 329 SubqCond (class in synapse.lib.ast), 238 spin() (in module synapse.common), 344 SubQuery (class in synapse.lib.ast), 238 spin() (synapse.lib.coro.GenrHelp method), 264 subquery() (synapse.lib.syntax.Parser method), 312 SpinCmd (class in synapse.lib.storm), 309 SudoCmd (class in synapse.lib.storm), 309 splice() (synapse.lib.snap.Snap method), 304 SwitchCase (class in synapse.lib.ast), 238 splicelistAppend() (synapse.lib.layer.Layer switchcase() (synapse.lib.syntax.Parser method), method), 281 312 splices() (synapse.cortex.CoreApi method), 347 synapse (module), 219 splices() (synapse.lib.layer.Layer method), 281 synapse.axon (module), 337 splices() (synapse.lib.lmdblayer.LmdbLayer synapse.cells (module), 340 method), 287 synapse.cmds (module), 219 splices() (synapse.lib.remotelayer.RemoteLayer synapse.cmds.cortex (module), 219 method), 300 synapse.cmds.cron (module), 221

Index 399 synapse Documentation, Release 0.1.0 synapse.cmds.hive (module), 222 synapse.lib.modules (module), 290 synapse.cmds.trigger (module), 222 synapse.lib.msgpack (module), 290 synapse.common (module), 340 synapse.lib.node (module), 292 synapse.cortex (module), 345 synapse.lib.output (module), 295 synapse.cryotank (module), 351 synapse.lib.platforms (module), 230 synapse.daemon (module), 357 synapse.lib.platforms.common (module), 231 synapse.data (module), 222 synapse.lib.platforms.darwin (module), 231 synapse.datamodel (module), 358 synapse.lib.platforms.freebsd (module), synapse.dyndeps (module), 361 231 synapse.eventbus (module), 361 synapse.lib.platforms.linux (module), 231 synapse.exc (module), 365 synapse.lib.platforms.windows (module), synapse.glob (module), 371 231 synapse.lib (module), 223 synapse.lib.queue (module), 295 synapse.lib.agenda (module), 232 synapse.lib.ratelimit (module), 298 synapse.lib.ast (module), 233 synapse.lib.reflect (module), 298 synapse.lib.auth (module), 240 synapse.lib.remotelayer (module), 299 synapse.lib.base (module), 244 synapse.lib.scope (module), 300 synapse.lib.boss (module), 248 synapse.lib.scrape (module), 301 synapse.lib.cache (module), 248 synapse.lib.share (module), 301 synapse.lib.cell (module), 248 synapse.lib.slaboffs (module), 301 synapse.lib.certdir (module), 250 synapse.lib.slabseqn (module), 301 synapse.lib.chop (module), 260 synapse.lib.snap (module), 302 synapse.lib.cli (module), 261 synapse.lib.storm (module), 304 synapse.lib.cmd (module), 263 synapse.lib.stormtypes (module), 310 synapse.lib.cmdr (module), 263 synapse.lib.syntax (module), 310 synapse.lib.const (module), 264 synapse.lib.task (module), 313 synapse.lib.coro (module), 264 synapse.lib.thishost (module), 314 synapse.lib.crypto (module), 223 synapse.lib.thisplat (module), 314 synapse.lib.crypto.ecc (module), 223 synapse.lib.threads (module), 314 synapse.lib.crypto.tinfoil (module), 225 synapse.lib.time (module), 317 synapse.lib.crypto.vault (module), 226 synapse.lib.trees (module), 317 synapse.lib.datapath (module), 265 synapse.lib.trigger (module), 318 synapse.lib.datfile (module), 266 synapse.lib.types (module), 318 synapse.lib.db (module), 266 synapse.lib.urlhelp (module), 324 synapse.lib.editatom (module), 267 synapse.lib.version (module), 324 synapse.lib.encoding (module), 267 synapse.lookup (module), 326 synapse.lib.filepath (module), 268 synapse.lookup.iana (module), 326 synapse.lib.gis (module), 270 synapse.lookup.iso3166 (module), 326 synapse.lib.hashitem (module), 271 synapse.lookup.pe (module), 326 synapse.lib.hashset (module), 271 synapse.lookup.phonenum (module), 326 synapse.lib.hive (module), 272 synapse.mindmeld (module), 372 synapse.lib.ingest (module), 274 synapse.models (module), 327 synapse.lib.interval (module), 274 synapse.models.base (module), 327 synapse.lib.jupyter (module), 274 synapse.models.crypto (module), 327 synapse.lib.kv (module), 277 synapse.models.dns (module), 328 synapse.lib.layer (module), 281 synapse.models.files (module), 328 synapse.lib.link (module), 282 synapse.models.geopol (module), 328 synapse.lib.lmdb (module), 282 synapse.models.geospace (module), 329 synapse.lib.lmdblayer (module), 286 synapse.models.gov (module), 327 synapse.lib.lmdbslab (module), 287 synapse.models.gov.cn (module), 327 synapse.lib.migrate (module), 288 synapse.models.gov.intl (module), 327 synapse.lib.mixins (module), 288 synapse.models.gov.us (module), 327 synapse.lib.modelrev (module), 289 synapse.models.inet (module), 329 synapse.lib.module (module), 289 synapse.models.infotech (module), 331

400 Index synapse Documentation, Release 0.1.0 synapse.models.language (module), 332 TelcoModule (class in synapse.models.telco), 333 synapse.models.material (module), 332 TeleHive (class in synapse.lib.hive), 273 synapse.models.media (module), 332 Thread (class in synapse.lib.threads), 316 synapse.models.orgs (module), 332 tick() (synapse.lib.storm.Runtime method), 308 synapse.models.person (module), 332 Time (class in synapse.lib.types), 322 synapse.models.syn (module), 332 TimeEdge (class in synapse.lib.types), 322 synapse.models.telco (module), 333 TimeOut, 371 synapse.reactor (module), 372 TimeUnit (class in synapse.lib.agenda), 233 synapse.servers (module), 334 TinFoilHat (class in synapse.lib.crypto.tinfoil), 225 synapse.servers.cortex (module), 334 to_aiter() (in module synapse.axon), 339 synapse.telepath (module), 372 to_bytes() (in module synapse.common), 345 synapse.tools (module), 334 to_int() (in module synapse.common), 345 synapse.tools.autodoc (module), 334 toknbytes() (synapse.lib.crypto.vault.Cert method), synapse.tools.backup (module), 335 227 synapse.tools.cellauth (module), 335 Tree (class in synapse.lib.trees), 317 synapse.tools.cmdr (module), 335 treeAndSync() (synapse.lib.hive.HiveApi method), synapse.tools.cryo (module), 334 272 synapse.tools.cryo.cat (module), 334 Trigger (class in synapse.cmds.trigger), 222 synapse.tools.cryo.list (module), 334 Triggers (class in synapse.lib.trigger), 318 synapse.tools.csvtool (module), 335 Triggers.Rule (class in synapse.lib.trigger), 318 synapse.tools.deploy (module), 336 tryDynFunc() (in module synapse.dyndeps), 361 synapse.tools.dmon (module), 336 tryDynLocal() (in module synapse.dyndeps), 361 synapse.tools.easycert (module), 336 tryDynMod() (in module synapse.dyndeps), 361 synapse.tools.feed (module), 336 tryPasswd() (synapse.lib.auth.User method), 243 synapse.tools.guid (module), 336 tryPasswd() (synapse.lib.hive.HiveUser method), 273 synapse.tools.json2mpk (module), 336 tryTeleAuth() (synapse.lib.auth.AuthApi method), synapse.tools.pullfile (module), 336 241 synapse.tools.pushfile (module), 336 tuplify() (in module synapse.cmds.hive), 222 sync() (in module synapse.glob), 371 tx() (synapse.lib.link.Link method), 282 synchelp() (in module synapse.glob), 371 Type (class in synapse.lib.types), 322 synchold() (synapse.lib.lmdbslab.Slab method), 288 type() (synapse.datamodel.Model method), 359 SyncTask (class in synapse.lib.threads), 316 typename (synapse.axon.Uploader attribute), 339 SynErr, 370 typename (synapse.axon.UploaderProxy attribute), synerr() (in module synapse.common), 345 339 SynModule (class in synapse.models.syn), 332 typename (synapse.daemon.AsyncGenr attribute), 357 typename (synapse.daemon.Genr attribute), 357 T Tag (class in synapse.lib.types), 321 U tag (synapse.lib.trigger.Triggers.Rule attribute), 318 uhex() (in module synapse.common), 345 TagCond (class in synapse.lib.ast), 239 un() (in module synapse.lib.msgpack), 292 tagged() (in module synapse.lib.node), 294 UniqCmd (class in synapse.lib.storm), 309 TagGlobs (class in synapse.lib.cache), 248 Univ (class in synapse.datamodel), 360 TagMatch (class in synapse.lib.ast), 239 univ() (synapse.datamodel.Model method), 359 tagmatch() (synapse.lib.syntax.Parser method), 312 UnivProp (class in synapse.lib.ast), 239 TagName (class in synapse.lib.ast), 239 univprop() (synapse.lib.syntax.Parser method), 312 tagname() (synapse.lib.syntax.Parser method), 312 univpropvalu() (synapse.lib.syntax.Parser method), TagPropValue (class in synapse.lib.ast), 239 312 tags() (in module synapse.lib.node), 295 UnivPropValue (class in synapse.lib.ast), 239 TagValuCond (class in synapse.lib.ast), 239 unlink() (synapse.eventbus.EventBus method), 364 TankApi (class in synapse.cryotank), 355 unlink() (synapse.lib.base.Base method), 246 Task (class in synapse.lib.task), 313 unpack() (synapse.lib.agenda.ApptRec class method), Task (class in synapse.telepath), 373 233 task() (synapse.telepath.Proxy method), 373 unpackVersion() (in module synapse.lib.version), taskv2() (synapse.telepath.Proxy method), 373 326

Index 401 synapse Documentation, Release 0.1.0

Unpk (class in synapse.lib.msgpack), 290 wait() (synapse.lib.base.Waiter method), 247 unwatchBlobStor() (synapse.axon.Axon method), wait() (synapse.lib.threads.RetnWait method), 316 337 wait() (synapse.lib.threads.SyncTask method), 316 update() (in module synapse.lib.scope), 301 Waiter (class in synapse.eventbus), 365 update() (synapse.lib.db.Xact method), 266 Waiter (class in synapse.lib.base), 247 update() (synapse.lib.hashset.HashSet method), 271 waiter() (synapse.eventbus.EventBus method), 364 update() (synapse.lib.kv.KvSet method), 279 waiter() (synapse.lib.base.Base method), 246 update() (synapse.lib.scope.Scope method), 300 waitfini() (synapse.eventbus.EventBus method), 365 updateCronJob() (synapse.cortex.CoreApi method), waitfini() (synapse.lib.base.Base method), 246 347 wants() (synapse.axon.Axon method), 337 updateTrigger() (synapse.cortex.CoreApi method), warn() (synapse.lib.snap.Snap method), 304 347 warn() (synapse.lib.storm.Runtime method), 308 Uploader (class in synapse.axon), 339 wasAdded() (synapse.datamodel.Form method), 358 UploaderProxy (class in synapse.axon), 339 wasDel() (synapse.datamodel.PropBase method), 360 Url (class in synapse.models.inet), 331 wasDeleted() (synapse.datamodel.Form method), User (class in synapse.lib.auth), 242 358 user() (in module synapse.lib.task), 314 wasSet() (synapse.datamodel.PropBase method), 360 user() (synapse.lib.hive.HiveAuth method), 272 whoami() (in module synapse.lib.auth), 243 username() (in module synapse.lib.task), 314 worker() (in module synapse.common), 345 Utf8er (class in synapse.lib.layer), 282 worker() (in module synapse.lib.threads), 316 worker() (synapse.lib.task.Task method), 314 V wrap() (synapse.lib.threads.Pool method), 314 vals() (synapse.eventbus.BusRef method), 362 write() (synapse.axon.Uploader method), 339 vals() (synapse.lib.base.BaseRef method), 247 write() (synapse.axon.UploaderProxy method), 339 vals() (synapse.lib.datapath.DataElem method), 265 writer() (synapse.lib.threads.RWLock method), 315 valu() (synapse.lib.datapath.DataElem method), 265 wrlock() (synapse.lib.db.Xact method), 266 valu() (synapse.lib.syntax.Parser method), 312 Value (class in synapse.lib.ast), 239 X value() (synapse.lib.ast.List method), 236 Xact (class in synapse.lib.db), 266 value() (synapse.lib.ast.Value method), 239 xact() (synapse.lib.db.Pool method), 266 value() (synapse.lib.stormtypes.Prim method), 310 xget() (synapse.lib.lmdb.Offs method), 283 valulist() (synapse.lib.syntax.Parser method), 312 XmlDataElem (class in synapse.lib.datapath), 265 valUserCert() (synapse.lib.certdir.CertDir method), xset() (synapse.lib.lmdb.Offs method), 283 260 varcall() (synapse.lib.syntax.Parser method), 312 Y VarDeref (class in synapse.lib.ast), 239 yamlload() (in module synapse.common), 345 varderef() (synapse.lib.syntax.Parser method), 312 yamlmod() (in module synapse.common), 345 VarList (class in synapse.lib.ast), 239 yamlsave() (in module synapse.common), 345 varlist() (synapse.lib.syntax.Parser method), 312 YEAR (synapse.lib.agenda.TimeUnit attribute), 233 VarListSetOper (class in synapse.lib.ast), 239 varname() (synapse.lib.syntax.Parser method), 312 VarSetOper (class in synapse.lib.ast), 240 vartokn() (synapse.lib.syntax.Parser method), 313 varvalu() (synapse.lib.syntax.Parser method), 313 VarValue (class in synapse.lib.ast), 240 Vault (class in synapse.lib.crypto.vault), 228 verify() (synapse.lib.crypto.ecc.PubKey method), 224 verify() (synapse.lib.crypto.vault.Cert method), 227 verstr() (in module synapse.common), 345 vertup() (in module synapse.common), 345 View (class in synapse.cortex), 350 W wait() (synapse.eventbus.Waiter method), 365

402 Index