Synapse Documentation Release 0.0.34

Nov 15, 2017

Contents

1 Synapse User Guide 3 1.1 Why Synapse?...... 3 1.2 Graphs and Hypergraphs...... 5 1.3 Data Model - Basics...... 9 1.4 Data Model - Concepts...... 14 1.5 Data Model - Node Concepts...... 15 1.6 Data Model - Node Properties...... 17 1.7 Data Model - Node Types...... 19 1.8 Data Model - Tag Concepts...... 26 1.9 Tags - Analytical Model...... 30 1.10 Using Synapse - Commands...... 37 1.11 Storm Query Language - Basics...... 40 1.12 Storm Reference - Data Modiﬁcation Operators...... 45 1.13 Storm Reference - Lift Operators...... 52 1.14 Storm Reference - Filter Operator, Comparators, and Helpers...... 56 1.15 Storm Reference - Pivot Operators...... 59 1.16 Storm Reference - “By” Handlers...... 61 1.17 Storm Reference - Statistical Operators...... 61 1.18 Storm Reference - Miscellaneous Operators...... 61 1.19 Ingest - Introduction...... 62 1.20 Ingest - Commandline Ingest Tool...... 62 1.21 Ingest - Embed Directives...... 63 1.22 Ingest - Parsing Lines...... 69 1.23 Ingest - Parsing CSV...... 71 1.24 Ingest - Parsing JSON...... 72 1.25 Ingest - Parsing JSONL...... 85 1.26 Ingest - Parsing XML...... 86 1.27 Ingest - file:bytes Nodes...... 87 1.28 Ingest - Sending Data to a Remote Cortex...... 87 1.29 Ingest - Format Details...... 87 1.30 Ingest - Conditional Expressions...... 88

2 Synapse DevOps Guide 89 2.1 Cortex: Synapse Hypergraph...... 89 2.2 Cortex: Built-In Storage Layers...... 93 2.3 Cortex: Storage Layer Details...... 94

i 2.4 Ingest: Structured Data Import...... 99 2.5 Axon: Synapse Blob Storage...... 99 2.6 Dmon: Synapse Deployment...... 99 2.7 Eventbus: Synapse Event Distribution...... 100 2.8 CoreModule: Model Deﬁnitions...... 100 2.9 Storm: Synapse Query Language...... 104 2.10 Telepath: Synapse Remote Method Invocation...... 104 2.11 Synapse Deployment Scenarios...... 104

3 Synapse Data Model 109 3.1 Types...... 109 3.2 Forms...... 132 3.3 Universal Props...... 195

4 synapse package 197 4.1 Subpackages...... 197 4.2 Submodules...... 332 4.3 synapse.async module...... 332 4.4 synapse.axon module...... 333 4.5 synapse.common module...... 339 4.6 synapse.cortex module...... 341 4.7 synapse.daemon module...... 342 4.8 synapse.datamodel module...... 344 4.9 synapse.dyndeps module...... 348 4.10 synapse.eventbus module...... 349 4.11 synapse.exc module...... 352 4.12 synapse.gene module...... 357 4.13 synapse.glob module...... 358 4.14 synapse.hivemind module...... 358 4.15 synapse.link module...... 358 4.16 synapse.mindmeld module...... 359 4.17 synapse.reactor module...... 359 4.18 synapse.statemach module...... 359 4.19 synapse.telepath module...... 359 4.20 Module contents...... 362

5 Synapse Conﬁgable Classes 363 5.1 synapse.axon.Axon...... 363 5.2 synapse.axon.AxonHost...... 365 5.3 synapse.cores.common.Cortex...... 366 5.4 synapse.cores.lmdb.LmdbStorage...... 369 5.5 synapse.cores.postgres.PsqlStorage...... 369 5.6 synapse.cores.ram.RamStorage...... 369 5.7 synapse.cores.sqlite.SqliteStorage...... 370 5.8 synapse.lib.ﬁfo.Fifo...... 370 5.9 synapse.lib.remcycle.Hypnos...... 371 5.10 synapse.lib.session.Curator...... 372 5.11 synapse.lib.storm.Runtime...... 372 5.12 synapse.swarm.runtime.Runtime...... 372

6 Synapse Performance 375 6.1 A Note on Comparisons...... 375 6.2 Test Data Sets...... 375 6.3 Cortex Conﬁgurations...... 376 6.4 Test Systems...... 377 ii 6.5 Results...... 377 6.6 Additional Tests...... 379

7 Contributing to Synapse 381 7.1 Project Style Guide...... 381 7.2 Docs Generation...... 386 7.3 Contribution Process...... 386 7.4 Running Postgres Tests...... 387

8 Synapse Release Process 389 8.1 Github Milestone Management...... 389 8.2 Release Notes...... 389 8.3 Cutting the Release...... 390

9 Indices and tables 393

Python Module Index 395

iii iv Synapse Documentation, Release 0.0.34

Contents:

Contents 1 Synapse Documentation, Release 0.0.34

2 Contents CHAPTER 1

Synapse User Guide

This User Guide is written by and for Synapse users. Additional technical documentation can be found elsewhere in the doc index. The User Guide is a living document and will continue to be updated and expanded. The current sections are:

1.1 Why Synapse?

Synapse is a distributed key-value hypergraph analysis framework. Synapse is designed to support analysis conducted over very large and disparate data sets. Analysis is predicated on the representation of data from a given knowledge domain in a structured data model that allows analysts to represent, annotate, and query across the collected data. Synapse offers several advantages: Free and Open Source Unlike costly commercial analysis systems, Synapse is available on Github under the Apache License 2.0. Performance Synapse was designed to address the performance limitations that constrain many large-scale analysis systems and eventually make them unworkable in practice. Synapse supports multiple storage and indexing options (RAM, LMDB, PostgreSQL, SQLite) that can be tailored to support the type and volume of data stored, as well as the proportion of read (query / retrieval) vs. write (node creation and deconﬂiction) operations. For detailed performance benchmarks, see Synapse Performance for more information. Extensible Comprehensive Data Model Synapse’s hypergraph framework allows for representation of more disparate data types and complex relationships. The extensible data model can be adapted to any knowledge domain. Powerful Analysis Framework Synapse provides a shared analytical workspace that consolidates research and prevents “stovepiping” of ﬁndings. An- alytical knowledge is preserved in a central representational framework instead of residing solely within a specialized research team or with an individual “subject matter expert” who may leave the organization.

3 Synapse Documentation, Release 0.0.34

Synapse’s shared workspace allows real-time annotation of data with analytical observations; the work of one analyst is immediately visible to all others in the same workspace. The ability to “analyze once, record for all” means that analysts no longer need to waste effort independently reviewing the same data to come to the same conclusions. Finally, the incorporation of new data or new analytical findings dynamically highlights supporting or conflicting analysis, allowing assessments and hypotheses to be revisited as needed. Proven Methodology Informed by Real-World Experience Synapse was not developed as a mathematical abstraction. Instead, Synapse grew out of a real-world need to track a complex, diverse, and very large data set: namely, cyber threat data. The developers and analysts who worked on early Synapse prototypes came from a variety of government and commercial backgrounds but shared a common goal: the desire to record, annotate, and track cyber threat activity (specifically, Advanced Persistent Threat activity) both reliably and at scale. At a time when government and industry were beginning to grasp the scope and scale of the problem, “tracking” this complex activity was largely being done using long-form reports, spreadsheets, or domain knowledge residing in an analyst’s mind. There was no way to effectively store and analyze large amounts of data, and critical analytical decisions – such as attribution – were either impossible, or being made based on loose correlation, analysts’ recollection, or generally accepted “truths” - and not on concrete, verifiable data whose source and analysis could be traced and either verified or questioned. In contrast, Synapse and its predecessors were designed from the beginning to support the following critical elements: • The use of a shared analytical workspace to give all analysts access to the same data in real time (as noted above). • The concept that the analysis captured within the system should “speak for itself”: that is, to the extent possible, data and analytical findings must be represented in such a way that relationships and conclusions should be self-evident. These features provide the following benefits: • It allows (and requires) analysts to “show their work” in a reasonably concise manner. Analysts should not have to refer to long-form reporting (or rely on the unquestioned word of a subject matter expert) to trace an analytical line of reasoning. • It allows analysts to better vet and verify each other’s findings. Conflicting analytical lines are highlighted through the structure of the data itself. Analysis can readily be questioned, reviewed, and deconflicted. The original Synapse prototype was designed to store a broad range of threat data, including: • Network infrastructure • Malware and malware behavior • Host- and network-based incident response data • Detection signatures and signature hits • Decoded network packet captures • Targeting of organizations, individuals, and data • Threat groups and threat actors • People and personas • Newsfeeds and reference materials Prototype systems eventually stored nearly one billion nodes, edges, and analyst annotations. Data modeled by this system was used to produce some of the most groundbreaking public reporting on APT activity to date. Synapse is the next generation of technology built on approximately five years of technical and analytical lessons learned. The new hypergraph design addresses many of the shortcomings identified with earlier directed graph systems. And because the experience of working with threat data taught us the power of a flexible analysis platform over

4 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

any large and disparate data set, Synapse has been designed to be ﬂexible, modular, and adaptable to any knowledge domain. Many of the real-world examples in this User Guide reference data from the ﬁeld of information technology or threat tracking, given Synapse’s history; but the structures, processes, and queries can be applied to other domains and data sets as well. The intent of Synapse is that any data that could be represented in a spreadsheet, database, or graph database, could be represented in a Synapse hypergraph using an appropriate data model.

1.2 Graphs and Hypergraphs

To understand the power of Synapse, it helps to have some additional background. Without delving into mathematical deﬁnitions, this section introduces key concepts related to a hypergraph, and contrasts them with those of a graph or a directed graph. Most people should be familiar with the concept of a graph – even if not in the strict mathematical sense – or with data that can be visually represented in graph form.

1.2.1 Graphs

A graph is a mathematical structure used to model pairwise relations between objects. Graphs consist of vertices (or nodes) that represent objects, and edges that connect two vertices in some type of relationship. Note that edges are speciﬁcally pairwise or two-dimensional: an edge connects exactly two nodes. Both nodes and edges may have properties that describe their relevant features. In this sense both nodes and edges can be thought of as representational objects within the graph, with nodes typically representing things (“nouns”) and edges representing relationships (“verbs”). A simple example of data that can be represented by a graph are cities connected by roads. If abstracted into graph format, each city would be a vertex or node, and a road connecting two cities would be an edge. Since you can travel from City A to City B or from City B to City A on the same road, the graph is “directionless” (or “undirected”). Another example would be social networks based on “connections”, such as Facebook or LinkedIn. In this case, each person would be a node, and the connection between two people would be an edge. Because basic connections in these networks are mutual (you can’t “friend” someone on Facebook without them agreeing to “friend” you in return), it can be considered a directionless graph. (This is a bit of a simpliﬁcation, but serves our purpose as an example.)

1.2.2 Directed Graphs

A directed graph is a graph where the edges have a direction associated with them. In other words, the relationship represented by the edge is “one-way”. Where an edge in an undirected graph is often represented by a straight line, an edge in a directed graph is represented by an arrow. In our cities-and-roads example, the graph would be a directed graph if the roads were all one-way streets; in such a case you can use a particular road to go from City A to City B, but not from City B to City A. In our social networks example, networks that support a “follows” relationship (such as Twitter) can be represented as directed graphs. Each person is still a node, but the “follows” relationship is one way – I can “follow” you, but you don’t have to follow me. If you choose to follow me, that would be a second, independent one-way edge in the opposite direction. (Again, this is a bit of a simpliﬁcation but works for a basic illustration.) Many other types of data can be represented with nodes and directed edges. In information security, for example, you can represent data and relationships such as: -- --> or -- -->

1.2. Graphs and Hypergraphs 5 Synapse Documentation, Release 0.0.34

In those cases, ﬁles, domains and IP addresses are nodes and “performed DNS lookup” or “has DNS A record” (e.g., “resolved to”) are edges (relationships). The edges are directed because a malware binary can contain programming to resolve a domain name, but a domain can’t “perform a lookup” for a malware binary; the relationship (edge) is one-way or directional. In addition to nodes and edges, some directed graph implementations may allow labeling or tagging of nodes and / or edges with additional information. These tags can act as metadata for various purposes, such as to create analytically relevant groupings. Many tools exist to visually represent various types of data in a directed graph format; Maltego (which bills itself as a “visual link analysis tool” that can represent information in a directed graph) is a well-known example.

1.2.3 Analysis with Graphs

Directed graphs have become increasingly popular for representing and conducting analysis across large data sets. Analysis using a directed graph can be highly generalized into four methods for interacting with the data: • Lifting or retrieving data. This simply asks about and returns specific nodes or edges from the graph. For example, you can ask about the node representing your Twitter account, or about the node representing IP address 1.2.3.4. You can also ask about sets of nodes that share some common feature – for example, all of the Twitter users who signed up for the service in January 2014, or all the PE executables whose compile date is 6/19/1992. • Filtering the results. Once you’ve asked about (lifted) an initial data set (a node or set of nodes), filtering allows you to further refine your results by either including or excluding data based on some criteria. For example, once you have your list of Twitter users who signed up in January 2014, you may decide to exclude users who list their location as the United States. Similarly, once you have your list of files compiled on 6/19/1992, you can filter those results to only include files whose size is greater than 26576 bytes. • Traversing the graph structure. Once you’ve asked about an initial data set, you can ask about relationships between your data set and other nodes by pathing (traversing) along the edges (relationships) that connect those nodes. For example, if you retrieve the node for your Twitter account, you can identify all of the accounts you are following on Twitter by traversing all of the “follows” edges from your node to the nodes of accounts that you follow. Similarly, if you retrieve the node for IP address 1.2.3.4, you can retrieve all of the domains that resolve to that IP by pathing backwards (remember, edges are directional) along the all of the “resolves to” edges that point to that IP. • Pivoting across like properties. Once you’ve asked about an initial data set, pivoting allows you to retrieve additional nodes or edges that share some property in common with your original data. For example, you can retrieve the node representing a PE executable, and then pivot to any other PE executables that share the same PE import hash or the same PE compile time. Despite their utility and increased use, directed graphs have certain limitations, most notably the “two-dimensionality” inherent in the concept of an edge. The fact that an edge can only connect exactly two nodes leads to a variety of consequences, including: • Performance. Even though a directed graph edge can only join two nodes, in theory there is no limit to the total number of edges to or from a given node – you can have 100,000 followers on Twitter, connected to you by 100,000 edges. However, these “edge dense” or “heavy” nodes represent a potential performance limitation when attempting to conduct analysis across a large or complex directed graph. The computational resources required to traverse all of those edges, hold the resulting set of nodes in memory, and then perform additional operations on the results (filtering, pivoting, additional traversals, etc.) can become prohibitive. • Data Representation. Some relationships involve more than two objects, which may require some creativity (or at the very least, additional overhead) to force them into a two-dimensional directed graph model. One side effect may be a multiplication of edges, as noted above; another may be the need to continually create specialized node types to represent arbitrary “clusters” of data or objects (basically combining what would normally be two or more nodes into a single node, simply so the cluster can be associated with another node via a single edge).

6 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

1.2.4 Hypergraphs

A hypergraph is a generalization of a graph in which an edge can join any number of nodes. Because an edge is no longer limited to joining exactly two nodes, edges in a hypergraph are often called hyperedges. If a directed graph where edges join exactly two nodes is “two-dimensional”, then a hypergraph where a hyperedge can join any number (n-number) of nodes can be considered “n-dimensional”. Looked at another way, they key features of a hypergraph are: • Everything is a node. “Things” (“nouns”) are still nodes in a hypergraph, similar to a regular graph. However, relationships (“verbs”) are now also represented as nodes. Looked at another way, where an edge in a graph consists of three objects (two nodes and the edge connecting them), in a hypergraph the same data is represented as a single multi-dimensional node. • Hyperedges connect arbitrary sets of nodes. An edge in a regular graph connects two nodes, and can be easily visualized in two dimensions as a line or arrow connecting two points. A hyperedge can connect an arbitrary number of nodes; one way to attempt to visualize this is as a “shape” (though this doesn’t entirely escape the “nodes and edges” concept, and becomes increasingly difﬁcult to visualize beyond three dimensions). The standard “visualization” for a hyperedge is as a set of disconnected nodes encircled by a boundary; the boundary represents the hyperedge “joining” the nodes into a related group. Just as there is no limit to the number of edges to or from a node in a directed graph, a node in a hypergraph can be joined by any number of hyperedges (e.g., be part of any number of “groups”). In Synapse, hyperedges are represented by tags, which can be thought of as labels applied to nodes.

1.2.5 Analysis with a Synapse Hypergraph

Synapse is a specific implementation of a hypergraph model. Within Synapse, an individual hypergraph is called a Cortex. A Cortex is a scalable hypergraph implementation which also includes key/value-based node properties and a data model which facilitates normalization. Analysis of data using a Cortex leverages some of the same methods as a directed graph: lifting nodes and filtering results are still part of the process. However, as noted above, in the absence of pairwise edges there is no traversal. Instead, all navigation is based on a pivot. (Technically, selecting a set of nodes from Synapse based on a tag could be considered “navigating” along a hyperedge. But mostly everything is a pivot.) Synapse optimizes this ability to pivot across properties through two key design features: type safety and property normalization. • Type safety ensures that node property types are explicitly declared and enforced across the data model. Where a property value is an IP address, for example, that IP address is declared and stored as an integer for consistency (as opposed to being stored as an integer in some instances and a dotted-decimal string in others). • Property normalization ensures that properties are represented in a consistent manner for both storage and display purposes, regardless of the format in which they are received. Synapse takes a “do what I mean” approach to input where possible, attempting to recognize common formats and normalize them on the user’s behalf. This allows users to work with data in a way that should feel “natural”. For example, a user can enter an IP address as an integer, a hex string, or a dotted decimal notation; Synapse will automatically store the IP as an integer and represent it back to the user as a dotted-decimal string. Similarly, a user can enter a directory path using either Windows (C:\foo\bar\baz.exe) or Linux (/home/user/foo/bar) format and using any combination of upper and lowercase letters; Synapse will automatically enforce normalization such as the use of forward slashes for directory separators and the use of all lower-case letters for drive, path, and file names. These features make pivoting highly effective because they ensure that data of the same type and / or with the same value is represented consistently throughout the Synapse hypergraph. In contrast, lack of consistency can cause analysts to miss relevant correlations - either because the same data is represented in multiple forms, or because the burden is placed on the analyst to “correctly” normalize their input when querying the system.

1.2. Graphs and Hypergraphs 7 Synapse Documentation, Release 0.0.34

Synapse’s optimized use of pivots, combined with the ability to represent relationships (including complex “multi- dimensional” relationships) as nodes, provides some significant advantages over a directed graph. These include: Performance “Asking questions” of a hypergraph may be less computationally intensive than in a directed graph. As a simple example, let’s say you want to know all of the domains that have resolved to a particular IP address. “Resolves to” (“has a DNS A record for”) is a relationship (edge) in a directed graph, so to answer this question you first need to lift the node for the IP address and then traverse an arbitrary number of edges to return the set of nodes represented by the endpoints of all those edges. For a handful of edges (domains), this traversal is not very difficult; but if thousands of domains have resolved to that IP, traversing all of those edges becomes more computationally intensive. Viewed another way - and depending on the specific implementation of the directed graph - a traversal may represent two pivots “under the hood”. Assume a generic representation of an edge as a tuple comprised of two nodes and the specific edge relationship ({n1,edge,n2}). “Traversing” from one set of nodes, along a specified edge, to a second set of nodes can be viewed as an initial pivot from a set of nodes to that set of edges where those nodes represent n1 of the edge; and then a second pivot from the set of n2s of those edges to the nodes that correspond to those n2s. In a Cortex, a single node represents the “resolves to” relationship, with the domain and IP address involved in the relationship both stored as properties on that node. So you simply need to lift the set of “resolves to” nodes where the value of the IP address property is the IP you are interested in. Once you have the relevant set of “resolves to” nodes, you simply pivot from the set of “domain” properties to the set of nodes representing those domains. No Loss of Granularity As noted above, the pairwise nature of edges in a directed graph may result in a loss of granularity for complex relationships that realistically involve three or more elements as opposed to the two elements supported by a directed edge. In order to “fit” those relationships into a directed graph model, one solution is to arbitrarily combine some of those elements into a single node in order to force the relationship to be pairwise. This results in some loss of detail as elements that should rightly be treated as independent components are artificially conflated. Discovery “Asking questions of” or “exploring” a directed graph has some inherent limitations. First, since relationships are represented by edges, an analyst is limited to asking about (traversing) “known relationships” (that is, edges that are already defined in the model). This may limit the discovery of new or unexpected patterns or correlations. Similarly, while directed graphs may support some navigation via pivots, analysts are often limited to pivoting via the same property / value on the same node type – for example, I can ask about all PE file nodes that have the same PE import hash value as a given PE file node because I am asking about the same value for the same property across the same node type. As noted above, Synapse’s type enforcement and property normalization remove this restriction. In a directed graph it is harder to ask about a value that may be present in different properties on different node types. For example, let’s say you have a malicious domain and you determine the set of IP addresses that the domain has resolved to. You want to know if any of those IP addresses have also been used to send spear phishing email messages. Speaking generically, there is no readily apparent relationship between an IP address as the resolution of a domain, and an IP address as the source of an email message, other than the fact that they are both IP addresses. This lack of an apparent relationship (edge) implies that you can’t get your answer using a few simple traversals. How you answer this question will vary depending on the specific implementation of the directed graph. However, if you assume an implementation with the following defined edges: -- --> and -- --> Then you may be able to obtain an answer through a multi-part query similar to the following: 1. Traverse the set of “has DNS A record” edges from the domain to obtain the set of IP addresses.

8 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

2. Traverse the set of “was source IP” edges from the resulting set of IP addresses to the set of RFC822 messages to get the messages (if any) associated with the IPs. 3. Traverse back along the “was source IP” edges from the RFC822 messages to get the IP addresses that were used to send email messages. If the above sounds messy and a bit redundant, it’s because to an extent, it is. There may be slightly more “elegant” solutions given alternate directed graph implementations (for example, if the source IP of an email message was stored as a property on the email message node as opposed to being associated with the message via an edge). But it still requires some creative navigation amongst nodes, edges, and properties to find the answer. In a Synapse hypergraph Cortex, the IP addresses appear as properties on both the set of “domain has DNS A record” nodes (as the “resolved to” property, for example) and the set of “spear phishing email nodes” (as the “source IP” property, for example). You can simply pivot between the two node types based on the value of those properties to find your answer. Not only is the navigation itself significantly easier, but you are able to readily ask questions across disparate or arbitrary data types (DNS records and email messages), as long as they share some value in common – even if that value represents a different property in each case.

1.2.6 Conclusions

Though hypergraphs may be less familiar conceptually than traditional graphs, they offer distinct performance and analytical advantages over directed graph models, addressing historical shortcomings in representation, navigation, and analytical capability. Synapse, as a speciﬁc implementation of a hypergraph model, incorporates additional design features (type safety, property normalization, and a robust query language, in addition to storage and indexing optimization for performance) that further enhance its power and ﬂexibility as an analysis tool.

1.3 Data Model - Basics

This section covers the basic “building blocks” of the Synapse hypergraph data model. It is intended to provide an overview of the fundamentals to get you started. Model elements are discussed in greater detail in the next section. (Note: This documentation presents the data model from a User or Analyst perspective; for detailed information, see the Data Model section or the Synapse source code.)

1.3.1 Background

Recall that Synapse is a distributed key-value hypergraph analysis framework. That is, Synapse is a particular implementation of a hypergraph model, where an instance of a hypergraph is called a Cortex. In our brief discussion of graphs and hypergraphs, we pointed out some fundamental concepts related to the Synapse hypergraph implementation: • Everything is a node. There are no pairwise (“two-dimensional”) edges in a hypergraph the way there are in a directed graph. • Tags act as “hyperedges”. In a directed graph, an edge connects exactly two nodes. In Synapse, tags are labels that can be applied to an arbitrary number of nodes. These tags effectively act as an n-dimensional edge that can connect any number of nodes – a hyperedge. • Every navigation of the graph is a pivot. Since there are no pairwise edges in a hypergraph, you can’t query or explore the graph by navigating (traversing) its edges. Instead, every navigation is a pivot from the properties of one node or set of nodes to the properties of another set of nodes. (Technically, selecting a set of nodes based on tag could be considered “navigating” along a hyperedge. But mostly everything is a pivot.) To start building on those concepts, you need to understand the basic elements of the Synapse data model.

1.3. Data Model - Basics 9 Synapse Documentation, Release 0.0.34

1.3.2 Data Model Terminology

The fundamental concepts you should be familiar with are: • Node • Form • Type • Tag Synapse uses a query language called Storm to interact with data in the hypergraph. Storm allows a user to ask about, ﬁlter, and pivot around data based in large part on node properties, values, and tags. Understanding these structures will signiﬁcantly improve your ability to use Storm and interact with Synapse data.

1.3.3 Nodes

A node can be thought of as an “object” within the hypergraph; although that is a bit of a misnomer, as nodes in Synapse can represent both “objects” (IP addresses, files, people, bank accounts, chemical formulas) and “relationships” (remember, what would have been an edge in a directed graph is now also a node in Synapse). It may be better to think of a node generically as a “thing”: any “thing” you want to model within Synapse (object, relationship, event) is represented as a node. Nodes are represented by data structures called “tufos” (short for “tuple form”; a form is described below). While “tufo” is used in the Synapse technical documentation and source code, “node” is more familiar and intuitive to most people, so this User Guide uses that term. However, the terms “tufo” and “node” are roughly interchangeable. (A high-level overview of tufos are provided here.) From a user perspective, you need to know that every node consists of the following components: •A globally unique identifier (GUID, identifier, or ID) that is unique across all nodes in a given hypergraph (e.g., a given Cortex). The way that node GUIDs are generated makes them “Cortex specific”: this means that the same node (e.g., inet:fqdn=woot.com) in two different Cortexes could have a different GUID in each. • One or more universal properties created automatically by Synapse whenever a new node is generated. The most important universal property for our purpose is the property tufo:form=

that lists the primary property type for that node. For example, a node representing an Internet domain (inet:fqdn) would have the universal property tufo:form=inet:fqdn. • A single primary property that consists of the form of the node plus its specific value. All primary properties (=) must be unique for a given form (node type). For example, the primary property of the node representing the domain “woot.com” would be inet:fqdn=woot.com. The uniqueness of the = pair ensures there can be only one node that represents the domain “woot.com”. Where the object represented by the node does not have a characteristic that is sufficiently unique to act as a primary property value – that is, any object where the possibility of a collision exists – the primary property is a machine-generated GUID. For example, in Synapse objects such as people, companies / organizations, or files (e.g., specific sets of bytes) are represented by GUIDs. Note that the GUID used as the primary property for some nodes is different from the GUID assigned to each node as its identifier. • Various secondary properties that are specific to that form. A node representing an Internet domain will have different secondary properties than a node representing a person, which will have different secondary properties than a node representing a bank account. Some secondary properties (and their values) are generated and set automatically when a node is created. Other secondary properties may be added or modified at a later time, unless they are defined as “read only” in the form definition.

10 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

Secondary properties form a relative namespace beneath the namespace of the node’s form. That is, if the node’s form type is tufo:form=foo:bar (e.g., this is a foo:bar node), then all secondary properties will be named in the format foo:bar:. In some circumstances (e.g., where the namespace context is clear) secondary properties can be referenced as relative properties – that is, as : as opposed to the fully qualiﬁed foo:bar:. So a secondary property foo:bar:baz can be referenced by its relative name :baz in some circumstances. • Optional ephemeral properties that may exist temporarily for some special purpose, and do not represent data stored permanently about that node. Ephemeral property names are preceded by a dot ( . ) to indicate they are ephemeral. The most common ephemeral property is .new, which indicates that a node is newly created. Example An example will help to illustrate the concepts above. The command below adds a node for the domain “woowoo.com” to a Cortex and displays the detailed properties of the newly-created node:

cli> ask --raw addnode(inet:fqdn,woowoo.com) [ [ "159eb804de045a47220dbde76984f2f4", { ".new": true, "inet:fqdn": "woowoo.com", "inet:fqdn:domain": "com", "inet:fqdn:host": "woowoo", "inet:fqdn:sfx": 0, "inet:fqdn:zone": 1, "tufo:form": "inet:fqdn" } ] ] (1 results)

In the output above: • 159eb804de045a47220dbde76984f2f4 is the GUID (identiﬁer) for the node. • ".new": True is the ephemeral property showing this is a newly created node. • "tufo:form": "inet:fqdn" lists the type of node (the form for the node). • "inet:fqdn": "woowoo.com" is the primary property of the node (=). The remaining entries are various node-speciﬁc secondary properties and their values (inet:fqdn:zone, inet:fqdn:domain, etc.)

1.3.4 Forms

A form is the definition of a Synapse hypergraph node. A form consists of the declaration of the primary property and its type, along with the form’s secondary properties (and their types). A form can be thought of as a template: if you want to create an inet:fqdn node in Synapse, the inet:fqdn form tells you the proper structure for the node and the properties it can contain. Forms are defined within the Synapse data model, and are declared within the model as tufos – that is, form definitions are themselves nodes in the hypergraph. Form definitions can be found here; those definitions are auto-generated from the Synapse source code. Forms are also documented within the source code of the appropriate Python module itself. (For example, the inet:fqdn form is defined within the inet.py module). The data model can be extended to include new forms or to modify existing forms (e.g., to add or change the secondary properties of a form) by:

1.3. Data Model - Basics 11 Synapse Documentation, Release 0.0.34

• creating new form nodes directly within the hypergraph to describe the updated data model; or • updating or extending the relevant Synapse source code. Because forms are nodes within the Synapse hypergraph, they can be created or modiﬁed directly within the Cortex, without the need to modify the Synapse source code. However, because the Synapse source code supports features such as model versioning and migration paths, it is preferable to maintain long-term or ofﬁcial model changes within the Synapse source. Below are examples of how a form (inet:fqdn) is represented and documented in both autogenerated docs and and the source code (in this case, inet.py). inet:fqdn - auto-generated from source code

**inet:fqdn = ** A Fully Qualified Domain Name (FQDN)

Properties: inet:fqdn:created = * Minimum time in millis since epoch inet:fqdn:domain = * The parent FQDN of the FQDN inet:fqdn:expires = * Maximum time in millis since epoch inet:fqdn:host = * The hostname of the FQDN inet:fqdn:sfx = (default: 0) * Set to 1 if this FQDN is considered a “suffix” inet:fqdn:updated = * Maximum time in millis since epoch inet:fqdn:zone = (default: 0) * Set to 1 if this FQDN is a logical zone (under a suffix)

Synapse source code (inet.py)

('inet:fqdn',{'ptype':'inet:fqdn'},[ ('sfx',{'ptype':'bool','defval':0,'doc':'Set to 1 if this FQDN is considered a

˓→"suffix"'}), ('zone',{'ptype':'bool','defval':0,'doc':'Set to 1 if this FQDN is a logical zone

˓→(under a suffix)'}), ('domain',{'ptype':'inet:fqdn','doc':'The parent FQDN of the FQDN'}), ('host',{'ptype':'str','doc':'The hostname of the FQDN'}), ('created',{'ptype':'time:min'}), ('updated',{'ptype':'time:max'}), ('expires',{'ptype':'time:max'}), ]),

Note that there are some minor differences between the the auto-generated documentation and the Synapse source code. Since either (or both together) can be helpful for analysts working with Synapse data, it helps to be aware of these differences. • Default values. Some nodes have properties that are automatically set to a specific value unless otherwise specified. If a property has a default value, it will be noted in both docs and the source code. • Read-only properties. Primary properties are unique and cannot be changed. Some secondary properties (typically those derived from the primary property) should also not be modified and are therefore implicitly read-only. In some cases, secondary properties are explicitly defined as read-only in the Synapse source code via the definition 'ro':1. However, these designations are not carried over to docs. (An example is the :port property of an inet:url node. A port number is generally not included in a URL that uses standard ports

12 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

for a given protocol (e.g., https://www.foo.com/bar/baz.html). Based on the presence of an “https” prefix in a URL, Synapse will set :port=443 as a read-only property, as specified in the source.) • Readability. While automatic docs are a bit more readable for the general user, the auto-generation process sorts and displays types, forms, and form secondary properties in alphabetical order. However, alphabetical order may not be the most intuitive order for grouping either forms or form-specific properties, based on how an analyst would typically view or work with the data. In contrast, the Synapse source code lists forms and form properties in an order that may be more “sensi- cal” for the given node type. The source code also tends to list secondary properties that can be automatically set by Synapse first in the source code (e.g., secondary properties that can be derived from the primary property’s value). For example, when creating the node inet:fqdn=woowoo.com, Synapse can parse that = and automatically set the secondary properties inet:fqdn:domain=com and inet:fqdn:host=woowoo. Secondary properties that require that an additional value be provided (e.g., inet:fqdn:created) are listed later in the source code.

1.3.5 Types

A type is the definition of an element within the data model, describing what the element is and how it should be normalized (if necessary) and structured to conform to the model. Synapse supports standard types (such as integers and strings) as well as extensions of these types. From a user standpoint, types are important primarily as they define the primary and secondary properties of forms. Like forms, types are also nodes within the Synapse hypergraph. Types within the data model can be modified or extended with new types by either creating new type nodes directly within the hypergraph or by updating or extending the relevant Synapse source code.

1.3.6 Tags

Tags are annotations applied to nodes. Broadly speaking, nodes represent “things” (objects, relationships, events – generally things that are “facts” or “observables”) while tags represent analytical observations – annotations that could change if the data or the assessment of the data changes. Tags can be applied to any number of relevant nodes, so in this sense tags act as hyperedges within the Synapse hypergraph, joining an arbitrary number of nodes in an “n-dimensional” relationship. A tag – like every other object in the Synapse data model – is also a form (syn:tag) that is declared in the Synapse data model (in datamodel.py) and represented within the hypergraph as a node. However, since the form (“template”) of a tag already exists within the data model, creating new tags does not require any changes to the Synapse source code. Analysts can create new tags “on the ﬂy” to record their analytical observations. Creating a new tag simply creates a new node of form syn:tag just as creating a new Internet domain creates a new node of form inet:fqdn. Tags can represent any observation that is analytically relevant to the knowledge domain modeled within the Synapse hypergraph. For example, in the knowledge domain of cyber threat data, analysts may wish to annotate observations such as: • “This malware binary is part of the threat cluster we track as Foobar Group.” (syn:tag=tc.foobar) • “This IP address is a TOR exit node.” (syn:tag=net.tor.exit) • “This domain has been sinkholed.” (syn:tag=cno.sink.hole) • “FooCorp Security says this indicator is part of activity they call Vicious Wombat.” (syn:tag=aka. foocorp.viciouswombat) • “This malware persists as a Windows service.” (syn:tag=persist.winreg.service)

1.3. Data Model - Basics 13 Synapse Documentation, Release 0.0.34

Note that tags can use a dotted “hierarchical” notation that allows analytical observations to be grouped by increasing levels of speciﬁcity. For example: • syn:tag=persist (malware persistence methods) • syn:tag=persist.winreg (malware persistence methods using the Windows registry) • syn:tag=persist.winreg.service (malware persistence methods using the Service keys of the Win- dows registry) Nodes, properties, and tags are discussed in greater detail in the next section.

1.4 Data Model - Concepts

The ability to conduct effective analysis across a Synapse hypergraph is highly dependent on the design and implementation of an appropriate data model for a given knowledge domain. The specifics of any data model (nodes, properties, tags) will vary based on both the knowledge domain and specific analytical needs. However, there are a few general principles and recommendations that should be kept in mind when developing a data model: • The model is an abstraction. Analysis often involves subtle distinctions and qualifications; this is why much analysis is provided in long-form reporting, where natural language can be used to convey variations in confi- dence or provide caveats surrounding conclusions. Capturing data and corresponding analysis in a formalized data model trades some of these subtleties for consistent representation and programmatic accessibility. A data model can never fully capture the richness and detail of a long-form report. But a well-designed data model can capture critical components in a way that conveys sufficient depth of analytical findings so that an analyst only rarely needs to refer to additional long-form reporting or original sourcing for clarification. • The model should be self-evident. While the model is necessarily an abstraction, it should not be abstracted to the point where the data and analysis in the model cannot stand on their own. That is, while there are times when supplemental external reports or explanatory notes may be helpful, they should not be required in order to understand the information in the hypergraph. The model should be designed to convey the maximum amount of information possible: objects, relationships, and analytical annotations should be unambiguous, well-defined, and clearly understood. An analyst with domain knowledge but no prior exposure to analytical findings should be able to look at the information represented in the hypergraph and understand the analytical line of thought. • The model should be driven by real-world analytical need and analytical relevance. Any model, regardless of knowledge domain, should be designed around the analytical questions that need to be answered by that model. Many data models are designed as academic abstractions (“how would we classify all possible exploitable vulnerabilities in software?”) without consideration for the practical questions that the data is intended to answer. Are some exploits theoretically possible, but never yet observed in the real world? Are some distinctions too fine-grained (or not fine-grained enough) for your analytical needs? Domain knowledge experts should have significant input into the type of data modeled, what analysis needs to be performed on the data, and how it should be represented. • The best models evolve in a cycle of forethought combined with real-world stress-testing. Creating a model “on the fly” with no prior consideration often leads to a narrowly-focused, fragmented data model – in the immediacy of detailed analysis, analysts may focus on the trees while missing the big picture of the forest. However, even the most well-thought-out model planned in the abstract will still fall short when faced with real- world data. Experience has shown that there are always edge cases, circumstances, or anomalies that cannot be anticipated. The most effective models are typically those that are planned carefully up front, then tested against real-world data and refined before being placed fully into production. • Err on the side of detail. No data model is set in stone – in fact, a good model will both expand and evolve with analytical need. That said, changes to the model may require revising or updating existing model elements and associated analysis, and some changes are easier to effect than others. Generally speaking, it is easier to combine two things (tags, nodes, properties) that you later decide are “the same” than it is to split up one large thing that you later decide should be two different things – if only because splitting things may require manual

14 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

review of each item to determine which category it should belong to. If there is a question regarding a design decision, more detail is often preferable until the design can be fully stress-tested.

1.5 Data Model - Node Concepts

Recall from Data Model – Basics that the key data model components within Synapse are nodes and tags. Broadly speaking: • Nodes commonly represent “facts” or “observables”: things that are objectively true or verifiable. • Tags commonly represent information that may be more changeable over time. Most commonly, tags represent “analytical assessments”: conclusions drawn from observables that may change if data or the assessment of the data changes. In some cases, tags represent “facts” that may change over time, such as the geolocation of an IP address (represented by a location tag #loc) or the association of an IP address with a specialized service, such as a TOR exit node or a VPN provider. Nodes, however, can represent a number of different types of observables: • Objects. Nodes can represent atomic “things”, whether real or abstract. In the knowledge domain of cyber threat data, “object” node types include domains, IP addresses (IPv4 or IPv6), hosts (computers / devices), usernames, passwords, accounts, files, social media posts, and so on. Other object types might include people, organizations, countries, bank or financial accounts, units of currency, chemical elements or formulas, telephone numbers, airline flights, and so on. Any “thing” can be represented as a node. In some cases, it may be useful to represent both generic and specific instances of a thing. For example, let’s say you wish to represent vehicles in your hypergraph. It can be useful to create nodes for the Honda Accord and the Ford Focus to represent different makes and models of vehicles. However, depending on the type of analysis being performed, it may also be useful to represent individual vehicles – Bob Smith’s Honda, or the Ford Focus with Vehicle Identification Number (VIN) 1HGBH41JXMN109186. (Synapse supports these concepts with the mat:spec (material specification) and mat:item (material item) forms.) Similarly, in the field of biology it might be useful to create a node representing the species orcinus orca as well as nodes representing individual killer whales. • Relationships. Nodes can represent specific relationships among objects. Recall that in a directed graph a relationship is represented as a directed edge (e) joining exactly two nodes (n1, n2); but in a hypergraph the entire relationship is represented by a single node, and the relationship can encompass any number of objects – not just two. In the knowledge domain of cyber threat data, relationships may include a domain resolving to an IP address, a malware binary writing a file to disk, or a threat actor moving laterally between two hosts. Other types of relationships could include a company being a subsidiary of another business, or the ownership of an object (such as a car) by an individual or organization. As an example of a “multi-dimensional” relationship, biological parentage could be represented by a three-way relationship among two genetic parents and an offspring. • Events. The term “event” implies that an object existed or a relationship occurred at a particular point in time. Nodes can represent time-based events; these can be thought of as “instance knowledge” as opposed to “fused” knowledge (which can be thought of knowledge that summarizes or “fuses” the data from any number of instance knowledge nodes). Examples include the results of a point-in-time domain lookup / domain resolution (DNS A record), the temperature reading in a city on a given date, or a domain registration (“whois”) record captured on a specific date. Another way to look at events in contrast to relationships is that each type of node provides a difference in data granularity. Whether to create a relationship-type node or an event-type node may depend on how much detail is required for a given analytical purpose. This consideration often applies to relationships that change over time, particularly those that may change frequently.

1.5. Data Model - Node Concepts 15 Synapse Documentation, Release 0.0.34

An example from cyber threat data is a DNS A record. The IP that a domain resolves to may change infrequently (e.g., for a website hosted on a stable server) or may change quite often (where the IP is dynamically assigned, or where load balancing is used, for example). One option to represent and track DNS A records would be to create an individual timestamped node (an event) every time you check the domain’s current resolution; this would represent a very high degree of granularity as the node would record the exact time a domain resolved to a given IP, potentially down to the millisecond. However, the number of such nodes could readily reach into the hundreds of millions, if not billions, if you create a node for every resolution of every domain you want to track. An alternative would be to decide that it is sufficient to know that a domain resolved to an IP address during a given period of time – an “earliest observed” (seen:min) and “last observed” (seen:max) range. A single node can be created to show that domain woot.com resolved to IP address 1.2.3.4, and that the earliest observed resolution was 8/6/2011 at 13:56, and the most recently observed resolution was 5/29/2016 at 7:32. These timestamps can be “moved out” (earlier or later) if additional data causes us to revise our observation boundaries. While this approach loses some granularity – the domain is not guaranteed to have resolved to that IP during the entire period, and we don’t know exactly when it resolved there during that period. But it may be sufficient for our needs, and may be preferable to creating thousands of nodes for individual resolutions. In essence, this type of time-bounded relationship node “fuses” what could have been thousands of individual timestamped DNS A record nodes into a single time-bounded node. Of course, a hybrid approach is also possible, where most DNS A record data is recorded in fused nodes (e.g., inet:dns:a nodes) but it is also possible to record a high-resolution, point-in-time event when needed (e.g., inet:dns:look nodes). Additional examples include malware behavior and environmental observations. In some circumstances, it may be enough to know that when a malware binary is executed, it drops (writes) a specific file (a set of bytes with a specific hash) to disk; this would represent relationship-type “fused knowledge” (e.g., “file1 writes file2”). In other circumstances, it may be important to know not only what file was dropped, but also the specific filename and directory path used, or the specific configuration of the computer or sandbox where the malware executed; this would represent specific event-based “instance knowledge”. Similarly, with environmental observations (temperature, humidity, barometric pressure, etc.) it may be sufficient to know that in a given location (city, latitude / longitude), the recorded temperature has varied between two upper and lower bounds; in other circumstances, it may be important to know the specific temperature observation at a specific point in time.

1.5.1 Deconﬂicted vs. Non-Deconﬂicted Nodes

For performance purposes, Synapse supports the use of both deconflicted and non-deconflicted nodes. Deconflicted nodes must be unique within the hypergraph; if you ask Synapse to create a node whose type must be deconflicted, Synapse will first check to see whether the node already exists. This initial read operation slightly delays node creation (write operation) where the node does not yet exist, but avoids the overhead of a full write if the node is already present. Most “object” and “relationship” nodes are deconflicted nodes. (One notable exception is nodes whose primary property is a randomly-generated GUID, such as people (ps:person) or organizations (ou:org). If the GUID is generated by Synapse itself using the “star” ( * ) syntax, Synapse guarantees the GUIDs are unique within a given Cortex, so the node does not need to be deconflicted before it is created. Essentially a request such as addnode(ps:person, *) translates to “create an arbitrary ps:person node with a randomly generated GUID and no predefined secondary properties”.) Non-deconflicted nodes are known or assumed to be unique a priori; as such there is no need to check for the node’s prior existence. Synapse skips any deconfliction check (read operation) and simply creates the node. Many “event” nodes are non-deconflicted nodes because they are assumed to be sufficiently high-resolution to represent data from a single, non-re-encounterable point in time. Some high-volume data types (e.g., streaming records, such as event logs) may be purposely designed as non-deconflicted to make the ingest of such data more performant. In addition, data that is deconflicted in advance (e.g., external to Synapse) and therefore known to be unique can be ingested as non-deconflicted data.

16 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

1.6 Data Model - Node Properties

1.6.1 Primary Properties

In Data Model – Basics, we noted that a node’s primary property consisted of the node’s form (its type or “template”) plus its specific value. More importantly, we noted that a primary property must be unique for a given form. From this standpoint, the design of a form and selection of its primary property are critical. Looked at another way, when creating a form (node type) for a given knowledge domain, the most important question you must ask is: what is it about this object / relationship / event that makes it unique and allows it to be deconflicted from all other instances of this form? That is, what gives it “thinghood”? That unique feature (or set of features) will determine an appropriate primary property. Whether that primary property is a simple property (e.g., a single =), a GUID, or a set of properties will determine the type of node (simple, composite, or cross-reference – described in Data Model - Node Types) that should be used. Once defined, the primary property of a form cannot be changed. To modify a form’s primary property, you would need to create a new node type and then migrate any existing nodes to use the new form.

1.6.2 Secondary Properties

Secondary properties are those characteristics that are not necessarily unique for a given form, but that provide relevant detail to help describe an individual node. Once you have determined the primary property for a given form, consider what additional details about that form should be captured as secondary properties. Since much navigation and analysis within a Synapse hypergraph is based on querying and pivoting across properties and / or values, a basic rule of thumb is that any piece of data that you might want to search on, pivot across, or otherwise correlate should be broken out as a secondary property (if not already a primary property). Secondary properties can be added or removed from a given form by updating the form within Synapse. Depending on the speciﬁc changes made, you may need to retroactively update previously-created nodes of the same form.

1.6.3 Secondary Properties vs. Relationship Nodes vs. Tags

One of the difﬁculties in designing an effective data model is making effective design choices as to whether data is best represented as a property (primary or secondary) of a node; as a component of the relationship represented by a relationship node; or as a tag applied to relevant nodes. While every modeling decision is unique, there are some basic principles that can be used to guide your choices: • One-to-one vs. one-to-many. Since properties (both primary and secondary) are structured as =, properties must have only a single value for a given node. Looked at another way: if there is a characteristic associated with a node that can have more than one value, it should not be represented as a secondary property and is a better candidate for a relationship node or a tag. • Intrinsic-ness. The more closely related or “intrinsic” a thing is to an object, the more likely the thing should be a secondary property (if it exists in a one-to-one relationship with the object) or a component of a relationship node (if the thing exists in a one-to-many relationship with the object). Things that are not highly intrinsic are better candidates for tags that can be applied to relevant nodes. A few examples will help to better illustrate these concepts. Example 1 - People and Email Addresses An email address is a common piece of data associated with a person. In creating a person node (ps:person), one’s ﬁrst thought may be to include the email address as a secondary property on the node (ps:person:email). However, it is common for people to have more than one email address, such as a work email and and a personal email. One option would be to create multiple secondary properties: ps:person:wemail for a work email

1.6. Data Model - Node Properties 17 Synapse Documentation, Release 0.0.34 and ps:person:pemail for a personal email. This is also problematic, for several reasons: first, an individual may have multiple work or personal email addresses. Also, those email addresses may change (for example, if the person changes jobs). This would potentially require creating an unlimited number of secondary properties (ps:person:wemail1, ps:person:wemail2...ps:person:wemailn), and also makes querying and pivoting extremely difficult if you have to account for dozens of differently-named secondary properties that may contain the data you’re looking for. Because “person” and “email address” is typically a one-to-many relationship (a person may have many email addresses), capturing the data in a secondary property is a poor choice. But should the data be captured in a relationship node or a tag? For “person has email address”, the relationship is best captured in a relationship node (ps:hasemail) for the following reasons: • “Email address” is a defined type within the Synapse data model (inet:email), making it suitable as a component of a primary property (e.g., in a composite node) and ideal for querying or pivoting. In other words, structuring this as a node makes it easier (more performant, syntactically simpler) to answer relevant analytical questions. • An email address is a fairly intrinsic characteristic of a person in the sense that the same email address cannot be assigned to two different people at the same time. This “intrinsic-ness” makes it a better candidate for a component of a relationship node than a tag. Looked at another way, if the email address were represented as a tag, you would be unlikely to apply that tag to more than one node. • While the data may change over time (a person may obtain a new email address), you likely don’t want to lose the historical knowledge of a previously used address. You can capture both current and historical email address associations in multiple relationship nodes (and time-bound the nodes with secondary seen:min and seen:max dates if necessary). Example 2 - Threat Cluster Indicators The ability to identify and correlate activity with a specific “threat group”, often to aid in the detection of malicious activity or to drive incident response, is central to the knowledge domain of cyber threat data. We refer to the process of associating indicators (domains, IP addresses, files representing tools or malware binaries) with a set of related activity as threat clustering, though others may refer to it as “attributing” indicators to a specific “threat group”. Let’s say you determine that an IP address (inet:ipv4) is associated with activity tracked as “Threat Cluster 12” (t12). One option would be to create a secondary property on the inet:ipv4 node for an associated threat cluster (tc) to show the IP is associated with Threat Cluster 12 (inet:ipv4:tc=t12). This is problematic for a few reasons: • Association with a set of malicious activity typically implies that an object (IP address, domain, file) is also “bad” or malicious. While threat actors may pay for infrastructure and thus control a “dedicated” IP (which therefore is likely “fully bad”), they may also compromise infrastructure that hosts legitimate content - such that the IP may be “bad” under some circumstances but “good” under others. The property does not allow us to make that distinction. • The IP address may not be used exclusively in activity associated with Threat Cluster 12, either over time or at a single point in time. For example, different threat actors may compromise the same vulnerable system, either concurrently or at different times. Alternately, an IP address may represent specialized infrastructure (a TOR exit node, a proxy, a VPN endpoint) designed to be used concurrently by multiple individuals. It should be clear that if we try to represent threat clustering as a secondary property, we run into similar issues as our previous example. In particular, for some indicators threat clustering may be a one-to-many relationship, and we quickly multiply the number of secondary properties we might need to track this information (:tc1, :tc2... :tcn). But should this “threat cluster” information be represented as a relationship node (perhaps “threat cluster has IP address”?) or as a tag? In this case, use of a tag is a better option for the following reasons: • “Threat cluster” is not intrinsic to an IP address (or a domain, or a file). It’s an analytical assessment specific to a given knowledge domain. This lack of “intrinsic-ness” makes it a poor candidate for a component property

18 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

of a relationship node. • Recording threat clustering data as a property (=, or :tc=t12) means you would need to ask about this data by lifting and / or pivoting. In particular if you wanted to ask the question “what are all the things associated with Threat Cluster 12?”, you would need to lift data where the secondary property on multiple node types (domains, ﬁles, IP addresses, email addresses) had a value of t12. “Lift by value” (as opposed to “lift by =”) is a computationally intense action. If the threat cluster data is stored as a tag (#tc.t12), it is much easier to lift the set of nodes that have that tag. The use of a tag better supports the ability to ask analytically relevant questions.

1.7 Data Model - Node Types

Recall that nodes can represent data such objects, relationships, and events. Synapse uses a set of common node (form) types to represent a broad range of information.

1.7.1 Simple Nodes

A “simple” node is one that represents an atomic object or entity. “Simple” is an informal descriptor meant to convey that forms for these types of nodes have a primary property that consists of a single =; the “uniqueness” or “thinghood” of a simple node can be represented by a single element. In addition, simple nodes are typically the most readily understood from a modeling perspective. Even for simple nodes, it may still be possible (and desirable) to break the node’s “basic” primary property into component parts so that those components are queryable or pivotable within the hypergraph. If Synapse can parse relevant subcomponents from the primary property, the associated secondary properties can be created automatically during node creation. In cases where these paresed secondary properties are themselves forms, Synapse will also automatically create nodes for those forms if they do not already exist. For example, for an inet:fqdn node, secondary properties include the inet:fqdn:host (the hostname of the fqdn, stored as a string) and inet:fqdn:domain (the parent fqdn of the fqdn, which is also an inet:fqdn form.). If you create the node inet:fqdn=foo.woot.com, Synapse will create: • an inet:fqdn node for foo.woot.com. • secondary proeprties inet:fqdn:host=foo and inet:fqdn:domain=woot.com (these do not need to be manually defined when creating the node). • an additional inet:fqdn node for woot.com, if it does not already exist in the Cortex. Examples of simple nodes include: • IP addresses. An IP address (IPv4 or IPv6) must be unique within its address space and can be defined by the address itself (inet:ipv4=1.2.3.4). Secondary properties (depending on need) may include things like the associated Autonomous System Number (ASN), geolocation data, and whether the IP belongs to a specialized or reserved group (e.g., private, multicast, etc.) • Domains. Similarly, an Internet domain must be globally unique (the same domain cannot be registered by two different entities) and can be defined by the domain string (inet:fqdn=woot.com). Additional secondary properties may include whether the domain is a top-level domain (TLD). • Files. Files are unique based on their content. Instead of doing a byte-by-byte comparison of content, files are typically “compared” by generating a hash of the file and comparing the hash of one file with the hash of another file; if the hashes match, the files are considered to be the same. However, improvements in both cryptanalysis and computational power have made hash collisions more likely – that is, it is possible to not only find but in some cases to deliberately generate two files with different content that nonetheless result

1.7. Data Model - Node Types 19 Synapse Documentation, Release 0.0.34

in the same hash. Thus, some hash algorithms (MD5, and increasingly SHA1) that were once considered unique no longer are, and there is no guarantee that this trend will not eventually hold true for additional hash algorithms. Because no hash can be considered reliably unique, the primary property of a file is represented by a GUID (file:bytes=). Specifically, if known, Synapse stores four hashes (MD5, SHA1, SHA256, SHA512) as secondary properties and generates a stable (deterministic) GUID from those hashes. For this reason a Synapse file:bytes GUID is sometimes referred to as a “superhash”. Additional secondary properties may include the file size in bytes or the file type (PE executable, plain text, GIF, etc.). Depending on analytical need, file type-specific secondary properties may also be defined (e.g., a compile date for PE executable, file-specific metadata for a JPEG or Word document, etc.) • Organizations. It is difficult to determine a unique characteristic for an organization (business, non-profit, government department, NGO). The organization name is one possibility, but names are not universally deconflicted – it is possible for more than one Acme Corporation to exist globally. Some type of business identification number is another possibility – a tax ID number, or a Dun & Bradstreet number, for example – but those may be country-specific and / or entity-specific (applying to businesses but not to government departments, for example). For these reasons, the primary property of an organization in the Synapse data model is represented by a randomly-generated GUID (ou:org=). Secondary properties may include an organization’s name, its location / country of origin, or the URL of the organization’s web site. • People. Similar to an organization, it is difficult to determine a universally unique characteristic for a person. Names are not guaranteed to be unique, as there can easily be more than one John Smith in the world. The same problem occurs with date of birth (multiple people can share the same birth date), and even the combination of name and date of birth cannot be guaranteed to be unique. For this reason, a person is also represented by a randomly-generated GUID (ps:person=). Secondary properties may include the person’s given, middle, and family names, date of birth, and so on. Aliases GUIDs are used as primary properties for forms which are not easily deconflictable (that is, which do not have readily identified characteristics that can serve as a unique primary property). For commonly used node types that are represented by GUIDs (such as organizations or people), it is impractical for an analyst to have to type (or even copy / paste) a GUID at the CLI every time they want to reference a given node. While nodes can be retrieved based on a more “human-friendly” secondary = (e.g., ou:org:alias=vertex), even the need to enter a full secondary property / value may be inconvenient. Synapse allows an alias to be defined as part of the type for the form’s primary property. The alias is typically one of the form’s secondary properties. For example, the Synapse data model defines the property ps:person:guidname as an alias for a person, and ou:org:alias as an alias for an organization. An example of the alias definition can be seen in this snippet of source code that defines the ou:org type (from orgs.py):

def getBaseModels(): modl={ 'types':( ('ou:org', {'subof':'guid', 'alias':'ou:org:alias', 'doc':'A GUID for a human organization such as a company or military

˓→unit' } ), ... ), ... }

Once deﬁned in the data model, the alias property’s value preceded by a dollar sign ( $ ) can be used as shorthand to refer to the node. As such, the following statements are equivalent:

20 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

ou:org=2f92bc913918f6598bcf310972ebf32e ou:org:alias=vertex ou:org=$vertex

1.7.2 Composite (Comp) Nodes

There are cases where the primary property of a node (that which makes the node unique or gives it “thinghood”) cannot be defined by a single element. However, these nodes may be defined as “unique” based on the combination of two or more elements. This is true, for example, for many relationship nodes (which makes sense, given that in a directed graph an edge is a relationship that joins two objects). A composite (comp) node supports this through a primary property that is composed of two or more = elements that collectively define a unique node for a given form. Comp nodes provide additional flexibility to the data model in that: • The elements of the comp node’s primary property can be any data or data type; that is, they can be GUIDs, lengthy blocks of text, or even other comp forms or seprarator (sepr) forms (described below). • Comp nodes may have primary properties that consist of a set of required elements as well as optional elements that can be included if known, or if an additional degree of granularity / uniqueness is required. For example, there may be cases where a form is defined as fully unique by a combination of five specific elements, but we may not always have data available to include all five. A subset of elements may be “sufficient” to create the comp node in the absence of complete data. Synapse uses the following conventions for comp nodes: • The elements of a comp node’s primary property are specified as a comma-separated ordered list within parentheses (e.g., =(,,...)). • Those elements that are mandatory for a given form must be present and listed in the order in which they are defined within the model. Since the mandatory elements are listed in their specified order, they can be listed by alone. • Optional elements can be included at the end of the list in the form =. • While the “primary property” is comprised of multiple elements, the elements can vary widely in number, length, and complexity. For performance reasons, the real primary property (used to store, index and reference a comp node) is a GUID that is generated as a function of the set of elements specified on node creation. Note that because the comp node GUID is “seeded” by the set of unique elements themselves, the GUID is deterministic: the same set of elements will result in the same GUID, including across different Cortexes. (Contrast this with node identifier GUIDs or randomly generated GUIDs used as primary properties, such as for ps:person nodes or ou:org nodes – such GUIDs are not deterministic and may vary across Cortexes.) For comp nodes, it is common to break out the individual elements of the primary property as secondary properties on the node so that they are searchable / pivotable. If Synapse can parse the values from the primary property, the secondary properties can be created automatically during node creation. In addition, for any secondary properties that are themselves forms, Synapse can create nodes for those forms if they do not already exist. Examples of comp nodes include: • Suborganization / subsidiary. The concept of “organization / sub-organization” (ou:suborg) is a straightforward relationship whose uniqueness is defined by the two entities involved. The relationship is generic enough that it can apply to a range of situations, from corporation and subsidiary to government and ministry within the government. The primary property consists of two elements, the GUID of the parent org (ou:org) and the GUID of the sub-org (ou:org). • Social networks. Social networks are comprised of individuals who establish relationships with other individuals. Such relationships may be “one-way” (you can “follow” someone on Twitter) or “two-way”

1.7. Data Model - Node Types 21 Synapse Documentation, Release 0.0.34

(you can mutually connect with someone on LinkedIn). The uniqueness of a social networking relationship (inet:follows) is defined by the individual user accounts involved. Even though there are only two elements that comprise the primary property, each of those elements is a complex node type (specifically, a sepr node defined by the combination of the service name and the username - e.g., inet:netuser=twitter. com/joeuser). (Note that within the Synapse data model, instead of creating two separate node types for “one-way” vs “two-way” social network connections, a “two-way” connection is represented by two “one-way” inet:follows nodes, with each user “following” the other.) • Bank or financial accounts. A bank or financial account is another candidate for a comp node. In considering what makes an account unique, an account number alone is insufficient, as the number is only guaranteed unique within a single financial institution. An account number combined with the account owner’s name seems like a possibility, although account ownership may change (e.g., an account may be transferred, or change from an individual to a joint account) and it is possible (however unlikely) that identical account numbers with identical owner names could exist at two different financial institutions. One option would be to combine the individual account number with a number that uniquely identifies the financial institution. Within the United States, this could be the institution’s ABA routing number combined with the individual account number (note that ABA numbers are specific to US financial institutions, though other countries or regions may use similar systems). In designing a form to represent a financial account it is worth considering the knowledge domain along with analytical need to decide whether a single form should represent any / all financial accounts (regardless of country of origin or account type – banking, investing, etc.), or whether it is preferable to create different forms for different account types (e.g., one form for US investment accounts, a different form for German banking accounts, etc.). Secondary properties for consideration may include the account type; date(s) the account was opened or closed; known minimum / maximum account balances (similar to :seen:min and :seen:max for date ranges); interest rate, if any; and so on. An example of a comp node with optional properties would be: • Files on computers. In cases of host-based computer forensics or cyber threat data analysis, it may be necessary to represent that a file was present on a specific computer (as opposed to representing the “location-less” existence of a file as a file:bytes node). “Interesting” files could include malware or tools used by threat actors, cached web content (such as a copy of a web-based exploit), host-specific logs, or files that provide other evidence of malicious or illegal activity (e.g., copies of stolen data). In considering what makes a “file on a computer” (it:hostfile) unique from all other files on all other computers (or the same computer), it is clear that multiple elements are involved: – The computer (host) (it:host). – The path and file name (file:path and / or file:base). – The file (file:bytes). – Timestamps associated with the file (created, modified, accessed), which may be operating-system and / or file system specific. While it is possible to create a comp node whose primary property is the combination of all of those elements, there is another challenge. In computer forensic or computer intrusion investigations, evidence is rarely perfect; that is, we are not guaranteed to have all of the above data available. Depending on the source of our evidence (forensic images, host-based logs, antivirus logs, network logs), we may have information about path and filename but no bytes; or a copy of the bytes (say from network traffic showing a file was downloaded to the host) but no path data; or the path and bytes but no timestamps. If we require all of the elements listed to form our primary property, we enforce high fidelity in our data model, but prevent ourselves from creating nodes with “partial” data that may still prove highly valuable for analysis. Alternatives include: – In defining our form, limit our primary property elements (for example, to it:host and file:bytes) and include the other components as secondary properties. However, this does not really solve our problem for several reasons: a given set of bytes could exist at two different locations on the same host, so the

22 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

combination of it:host and file:bytes are not guaranteed to be unique. In addition, we may not always have the bytes (or a hash that could be used to represent the bytes). Finally, things like the path that truly help define the “uniqueness” of a specific file on a specific host don’t belong as secondary properties. – Create multiple forms to represent various combinations of the above data. However, this leads to a plethora of forms that are essentially duplicative. Instead, we can leverage a single comp node (form) but make some of the elements of the primary property optional. In considering what element(s) are essential to the concept of “a file on a computer” (it:hostfile), the only element that is absolutely required is the computer (it:host). (This makes sense if you think about it; in the absence of a computer, a file is just a file (file:bytes).) While it would be rare to create an it:hostfile node without any reference to the file itself, the information we have on the file may vary - we may have the filename or path (file:base, file:path), the actual bytes (a file:bytes node with a complete “superhash” GUID), or simply a hash value (a file:bytes: secondary property that will be used to create a GUID based on the available hash). So none of those other properties can be considered to be required, but they can be included if the data is available. Comp node optional elements and node uniqueness Recall that while a comp node’s “primary property” (that which makes it unique) is a combination of two or more elements, the actual primary property stored and referenced in Synapse is a GUID generated as a function of the individual elements specified at the time the node is created. So if you have =(foo,bar,baz) the GUID is a function of foo, bar, and baz. The function is deterministic, so the same set of elements will always generate the same GUID. This has implications for the data model when some of the elements are optional. Let’s say you have a comp node =(foo,bar,baz,hurr,derp) where foo is required but the remaining elements are optional. If, when you first create the node, you only know foo, the node GUID will be based only on foo. Once created, a node’s primary property cannot be changed; so if you later identify baz, you can’t simply “add” it to the existing comp node; you would need to create a second comp node based on foo and baz, which would generate a different GUID. If you later learn bar and derp, a node created from foo, bar, baz, and derp would have yet another GUID. To provide a more concrete example, consider the it:hostfile node described above. Let’s say initially you determine that a suspicious file existed at the path C:\WINDOWS\system32\scvhost.exe on host MYHOST. You create the initial it:hostfile node based on those two properties, and Synapse generates the GUID 671993b20eb292dbd1dec63cbd26d3ce from that data. In the course of your analysis, you tag the it:hostfile node as being associated with Threat Cluster 12 (#tc.t12). You later recover the actual file bytes for scvhost.exe, a file:bytes node with the GUID (“superhash”) d385c823f1f5c64b5cec20c9e04adb32. You can’t add the file:bytes element (an optional component of the it:hostfile node’s primary property) to the existing node, so a new it:hostfile node is created with a different GUID based on the combination of the host, the path, and the file:bytes GUID. The new node has “higher resolution” (more information, greater specificity), but the two nodes are not automatically “combined” by Synapse, and tags on the existing node (such as the #tc.t12 tag) are not automatically copied over to the new node. (Note that not copying the tags may be a good thing; perhaps both Threat Cluster 12 and Threat Cluster 35 have been associated with the path C:\WINDOWS\system32\scvhost.exe - not an unreasonable assumption, as use of scvhost.exe to masquerade as the legitimate svchost.exe is fairly common. Perhaps both threat clusters are even associated with the same path on the same host at different times during a three-year period. But only that specific file (file:bytes) located at that specific path on that specific host is associated with Threat Cluster 12. In that case, it might be reasonable to tag the it:hostfile node based on the host and path alone with both #tc.t12 and #tc.t35 (both clusters are associated with that exact path on that exact host), but the it:hostfile node based on the host, path, and specific file with #tc.t12 (only Threat Cluster 12 is associated with that exact file at that exact path on that exact host). A similar issue exists for file:bytes nodes. While not a true comp node, the primary property GUID of a file:bytes node is based on the combination of the file’s MD5, SHA1, SHA256, and SHA512 hashes. In other words, the GUID is generally meant to be generated based on having an actual copy of the file (the actual bytes) where

1.7. Data Model - Node Types 23 Synapse Documentation, Release 0.0.34

the four hashes can be calculated and used to create a “complete” GUID (“superhash”). However, in some cases you may know one of the hashes of a ﬁle - say the file:bytes:md5 hash referenced in third-party reporting or log data - but not have the actual bytes. Synapse will still create a file:bytes node but the GUID will be generated based on the MD5 hash alone. If the bytes are later obtained, Synapse will create a different node with a different GUID for the “actual” bytes based on all four hashes. Analysts and developers should be aware of these restrictions. The use of optional elements in a comp node allows for the greatest ﬂexibility, particularly in cases where available data for a given form may vary; but it does have implications for analysis, and in particular for tagging nodes, that must be taken into account.

1.7.3 Cross-Reference (Xref) Nodes

As noted in Data Model Concepts, the model should be “self-evident” to the extent possible: nodes and tags should be well designed and unambiguous. In addition, analysts should rarely need to refer to external reporting or data to understand an analytical line of thought. Data and analysis required to support (or refute) a hypothesis should exist within the hypergraph itself, so that the hypergraph stands on its own. In addition, it is preferable for data in the hypergraph to consist of original or verifiable source material where possible. This follows the general analytical principle of primary sources: you can best verify your own data (or other original data) and related analysis. Third-party reporting raises questions of source reliability, accuracy, and so on. However, this presents several challenges. First, it is both impractical and unrealistic to assume that all data in a hypergraph can be originally sourced. Almost all analysis relies on some amount of research by others; this is why research papers provide references and cite sources. Let’s say that you are attempting to link a computer intrusion to the infamous Threat Cluster 12, but you don’t have direct knowledge of the intrusion or the intrusion investigation. However, a third-party source states that malware found during the investigation communciates with a domain that you have linked to Threat Cluster 12. How do you reference other reporting or sources within the hypergraph? Second, in conducting analysis across a broad range of data types, there are cases where information needed to support (or refute) a hypothesis is highly specialized, or cannot easily be broken down into pre-existing nodes (objects, relationships, or events). For example, let’s say you want to demonstrate that Alice and Bob know each other, but you don’t have evidence (such as social media connections) to demonstrate that. (Maybe you don’t have access to that data, or maybe Alice and Bob want to keep their relationship secret and so do not have social media connections.) However, you identify a photograph showing Alice and Bob together that helps support your assertion. How would you represent this in the hypergraph? Synapse supports these concepts through a specialized node type called an xref (short for “cross-reference”) node, which allows you to demonstrate that one object “references” another. So a photograph (file:bytes) can “reference” (contain) an image of a person (ps:person) or a particular place (geo:place); or a document (file:bytes) can reference anything from an atomic object (a security report referencing a malicious domain (inet:fqdn)) to a particular assertion (a report stating that malware found on Acme Corporation’s network com- municated with myevildomain.com, or a news article noting that Acme Corporation was in merger talks with Widgets, Inc. in March 2016). An xref node can be thought of as a specialized type of “relationship” node. The relationship nodes discussed previously can be clearly defined because the “participants” in the relationship are known in advance: a DNS A record consists of a domain (inet:fqdn) pointing to an IP address (inet:ipv4). Because those forms are known, they can be specified in the form for the inet:dns:a record, and that form can be represented as a sepr or comp node (in this case, a sepr node). With a “references” relationship, the participants are not known in advance. While the “thing containing the reference” may typically be some type of file (file:bytes) such as a report, a news article, or a photograph, the “thing being referenced” could be anything; its form may be arbitrary. One option would be to create multiple comp nodes to define each possible type of relationship: file:bytes references inet:fqdn, file:bytes references geo:place,

24 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

file:bytes references ps:person, etc. However it should be clear that this becomes inefficient if a new form needs to be defined every time a new “thing” needs to be referenced. A better solution is the xref node, which provides the flexibility to “reference” any type of object. An xref node’s primary property consists of: • the primary property of the “thing” referencing another thing (e.g., file:bytes); • the form of the thing being referenced (so Synapse knows whether the referenced object is a domain, a hash, a person, an airplane, a specific airplane, etc.) • the primary property of the “thing” being referenced. The Synapse data model currently includes two predefined xref-type nodes: • file:imgof (a file contains an image of something) • file:txtref (a file contains a “text reference” to something) Similar to comp nodes, the elements of an xref node’s primary property are specified as a comma-separated ordered list within parentheses (e.g., =(,,)).

1.7.4 Separator (Sepr) Nodes

Separator (sepr) nodes pre-date composite (comp) nodes and are subject to certain limitations that were addressed with the creation of the comp node. While some legacy sepr forms exist within the Synapse data model, comp nodes are preferred for future development. Sepr nodes are an early type of node that was developed to represent nodes with multi-element primary properties (typically various types of relationship nodes). They can be considered a subset of comp nodes and have been su- perseded by comp nodes. They are described here for completeness and to address some of the legacy forms present within the Synapse data model. Synapse uses the following conventions for sepr nodes: • Sepr nodes have primary properties that consist of two or more elements. (Most, if not all, sepr forms defined within Synapse to date consist of two elements.) • The elements of the primary property are separated with a designated character specified in the data model. Note that this imposes the restriction that whatever character is used as the separator cannot appear in any element of the primary property. (Comp nodes use a comma-separated list, which removes this “special character” limitation.) By convention, Synapse most often uses a forward slash ( / ) as the separator character (though pipe ( | ) and at ( @ ) are also used). If no character is specified, the model defaults to a comma ( , ). • Because the primary property of a sepr node is the string consisting of , the elements of a sepr primary property should ideally be “human readable” (and therefore “human type-able”, such as at the CLI). (Comp node elements can be any data or data type of any length as the true primary property is a GUID generated from the individual elements). Similar to comp nodes, it is common to break out the individual elements of the primary property of a sepr node as secondary properties on the node so that they are searchable / pivotable. If Synapse can parse the values from the primary property, the secondary properties can be created automatically during node creation. Examples of sepr nodes include: • DNS A records. A domain having a DNS A record for an IP address is a straightforward relationship. Within Synapse, this relationship has been defined as a sepr node (inet:dns:a) that consists of the unique combination of domain and IP address separated by a forward slash (inet:dns:a=woot.com/1.2.3.4). Synapse

1.7. Data Model - Node Types 25 Synapse Documentation, Release 0.0.34

is able to parse the domain and IP address from the primary property and automatically create them as secondary properties (e.g., inet:dns:a:fqdn=woot.com and inet:dns:a:ipv4=1.2.3.4). Similarly, because both components are also forms (inet:fqdn and inet:ipv4), Synapse will create the individual nodes if they do not already exist. • Social media or Internet service accounts. Service accounts are an example of an “object” type node that requires two components to uniquely define the node. A username by itself is not unique because someone (or two different people) could have the same username on two different services (such as LinkedIn and Twitter). However, usernames typically must be unique within a given service, so Synapse uses both elements (the service and the username, separated by a forward slash) to uniquely define an account (inet:netuser=twitter.com/joeuser). Similar to the previous example, Synapse is able to parse the service and username from the primary property and automatically create secondary properties for these elements (inet:netuser:site=twitter.com, inet:netuser:user=joeuser). Other secondary properties may depend on the types of account(s) being tracked and the specific analytical need. User profile data available from a given service may vary widely depending on the service purpose (software development vs. cloud storage service vs. social media) or on geography or culture. For example, some Asian web sites allow users to post their blood type, while western web sites may allow users to post their zodiacal sign; within different cultures, both are believed to reflect an individual’s personality.

1.8 Data Model - Tag Concepts

1.8.1 Tag Structure

Tags use a dotted naming convention, with the period ( . ) used as a separator character to delimit individual “components” of a tag if necessary. This dotted notation means it is possible to create tag hierarchies of arbitrary depth that support increasingly detailed or speciﬁc observations. For example, the top level tag foo can represent a broad set of observations, while foo.bar and foo.baz could represent subsets of foo or more speciﬁc observations related to foo. In a tag hierarchy, the full tag (down through and including the lowest or rightmost element) is known as the leaf tag for that hierarchy. For the tag foo.bar.baz, foo.bar.baz is the leaf. The “top” or leftmost component foo may be informally referred to as the root of that hierarchy. The lowest (rightmost) element in a tag hierarchy is known as the base of the tag; for tag foo.bar.baz, baz is the base.

26 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

When you apply a tag to a node, all of the tags in the tag hierarchy are applied; that is, if you apply tag foo.bar.baz to a node, Synapse automatically applies the tags foo.bar and foo as well. When you delete a tag from a node, the tag and all tags below it in the hierarchy are deleted. If you delete the tag foo.bar.baz from a node, the tags foo.bar and foo will remain. However, if you delete the tag foo from a node with the tag foo.bar.baz, then all three tags (foo, foo.bar, and foo.bar.baz) are deleted.

1.8.2 Tag Timestamps

Applying a tag to a node has a particular analytical meaning; it typically represents the recording of an assessment or judgment about the existing data within the hypergraph. Many assessments are “binary” in the sense that they are always either true or false; in these cases, the presence or absence of a tag is sufficient to accurately reflect the current analytical assessment, based on available data. There are other cases where an assessment may be true only for a period of time or within a specified time frame. Internet infrastructure is one example; whether an IP address is part of an anonymization service (such as TOR or an anonymous VPN provider) can be annotated using tags. However, this information can change over time as the IP address is reallocated to a different provider or repurposed for another use. Although the relevant tag (for example, anon.vpn) can be applied while the IP is part of an anonymous VPN service and removed when that is no longer true, completely removing the tag causes us to lose the historical knowledge that the IP was part of a VPN network at one time. To address these use cases, Synapse supports the use of timestamps with tags. When a tag is applied to a node, optional “first seen” and “last seen” timestamps can be used to indicate “when” the condition represented by that tag was valid. The timestamps can be “pushed out” if additional data indicates the condition was true earlier or later than initially observed. Tag timestamps represent a time range, and not necessarily specific occurrences (other than the “first known” and “last known” observations). This means that the assessment represented by the tag is not guaranteed to have been true throughout the entire date range. That said, the use of timestamps allows much greater granularity in recording analytical observations in cases where the timing of an assessment (“when” something was true) is relevant.

1.8.3 Tag Display

If any tags are applied to a node, detailed tag information will be displayed with the node properties regardless of “how” you ask about a node (e.g., ask and ask --props will display the same tag information, even though they display different details about the node properties). This includes: • the lowest (leaf) tags (e.g., for display purposes Synapse will list foo.bar.baz but not foo.bar and foo, even though they are also separate tags on the node). • the timestamp when the tag was ﬁrst applied to the node. • the minimum / maximum times during which the tag was applicable to the node, if present. For example:

cli> ask inet:fqdn=woot.com

inet:fqdn = woot.com #foo.bar (added 2017/06/20 19:59:02.854) #hurr.derp (added 2017/08/02 21:11:37.866) 2017/05/23 00:00:00.000 - 2017/08/03

˓→00:00:00.000 (1 results)

1.8. Data Model - Tag Concepts 27 Synapse Documentation, Release 0.0.34

In the output above, the timestamps in parentheses represent when the tags foo.bar and hurr.derp were applied to the node. The additional timestamps associated with hurr.derp show the minimum / maximum range when the tag was applicable or relevant for this node. Using the --raw option will display all tags (not just leaf tags) and properties in JSON format (timestamps are in Unix epoch time):

cli> ask --raw inet:fqdn=woot.com [ [ "a4d82cf025323796617ff57e884a4738", { "#foo": 1497988742854, "#foo.bar": 1497988742854, "#hurr": 1501707978035, "#hurr.derp": 1501708038579, "<#hurr.derp": 1501718400000, ">#hurr.derp": 1495497600000, "inet:fqdn": "woot.com", "inet:fqdn:created": 1433680424000, "inet:fqdn:domain": "com", "inet:fqdn:host": "woot", "inet:fqdn:sfx": 0, "inet:fqdn:zone": 1, "tufo:form": "inet:fqdn" } ] ] (1 results)

In the output above, timestamp associated with "#hurr.derp" is the time the tag was applied; the timestamps associated with "<#hurr.derp" and ">#hurr.derp" are the maximum and minimum timestamps associated with the tag.

1.8.4 Tags - Data Model

Tags can be thought of as labels applied to nodes. However, tags themselves are also nodes – that is, every tag is represented by a node (of form syn:tag) within the hypergraph. For a syn:tag node, the primary property (=) is the name of the tag; so for the tag foo.bar, the node’s primary property is syn:tag=foo.bar. Tag nodes can be created manually just like any other node; however, most often the creation of tag nodes is transparent to the user. That is, Synapse will create a syn:tag node on the fly when a tag is applied to a node for the first time. For example, applying the tag hurr.derp to the node inet:fqdn=woot.com will automatically create the node syn:tag=hurr.derp if it does not already exist. Note that if you delete the node associated with a tag (syn:tag=foo.bar.baz, as opposed to deleting the tag foo.bar.baz from a node) then not only is the syn:tag node itself removed from the Cortex, but the corresponding tag is removed from all nodes to which it was previously applied. From a strict data model perspective, tags are very simple since there is only one form (syn:tag) and one “helper” form (syn:tagform, the meaning of a tag when applied to a specific form) used to represent tags within the Synapse hypergraph. Any number of tags across any number of knowledge domains can be created based on those predefined forms. Tags and tagforms are fundamental components of the Synapse data model (hence the syn prefix in syn:tag and syn:tagform), independent of model elements that may be specific to a particular knowledge domain. These

28 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34 fundamental components are defined within the core Synapse source code (as opposed to source code for domain- specific data models). This means they are not covered by the Synapse automated document generation process, and do not appear in the standard data model documentation. The form structures can be found in the source code of datamodel.py and are documented here (in simplified format) for reference: syn:tag syn:tag = A tag or label that can be applied to one or more nodes; acts as a hyperedge to

˓→join any number of nodes into a related set.

Properties: syn:tag:base = The base element of the tag (right-most element; to the right of the final dot, if

˓→any. For tag foo.bar.baz, :base=baz.) syn:tag:up = The portion of the tag excluding the base (all content to the left of the final

˓→dot, if any. For tag foo.bar.baz, :up=foo.bar) syn:tag:depth = Number of levels down from the "top" of the tag. For tag foo.bar.baz, :depth=2.

˓→For tag foo, :depth=0. syn:tag:title = The short name or definition of what the tag means. syn:tag:doc = The long form description of what the tag means. syn:tagform syn:tagform = A multi-field composite type which generates a stable GUID from normalized fields.

˓→The composite type consists of the tag (syn:tag) and the form to which the tag

˓→applies (syn:prop).

Properties: syn:tagform:tag = The tag being documented syn:tagform:form = The form that the tag applies to syn:tagform:title = The short name for what the tag means when applied to the given node form. syn:tagform:doc = The long form description for what the tag means when applied to the given node

˓→form.

The :base, :up, and :depth properties of a syn:tag node facilitate analysis using the Storm query language by supporting: • querying or ﬁltering at arbitrary depth; • traversing the tag hierarchy; • pivoting between nodes and tags. The :title and :doc properties allow deﬁnitions for the tag and tagform to be documented on the tag node itself.

1.8. Data Model - Tag Concepts 29 Synapse Documentation, Release 0.0.34

1.9 Tags - Analytical Model

While the Synapse data model related to tags is straightforward (consisting of only two forms), the appropriate use of tags for analysis is more complex. Tags can be thought of as being part of an analytical model that relies on the Synapse data model, but that: • Exists independently from the data model (you do not need to write code to implement new tags or design a tag structure). • Is knowledge domain-specific (tags used for cyber threat analysis will be very different from tags used for biomedical research). • Is tightly coupled with the specific analytical questions or types of questions the hypergraph is intended to answer (the questions you want to answer should dictate the tags you create and apply). In short, effective use of Synapse to conduct analysis is dependent on: • the data model (how you define forms to represent data within your knowledge domain). • the analytical model (how you design a set of tags to annotate data within your knowledge domain). A well-designed tag structure should: • Represent relevant observations: tags should annotate assessments and conclusions that are important to your analysis. • Facilitate effective analysis: tags should be structured to allow you to ask meaningful questions of your data.

1.9.1 Tags as Assessments

Research and analysis consists of testing theories or hypotheses and drawing conclusions based on available data. As- suming data is initially collected and recorded accurately within a hypergraph, that underlying data (typically encoded in nodes and properties) should not change. However, as you collect more data or different types of data, or as you re-evaluate existing data, your assessment of that data (typically encoded in tags) may change. Nodes and properties are meant to be largely stable; tags are meant to be ﬂexible and evolving. If a tag represents the outcome of an assessment, then every tag can be considered to have an underlying question or hypothesis it is attempting to answer. Answering this question often involves the judgment of a human analyst; hence evaluating and tagging data within the hypergraph is one of the primary analyst tasks. Hypotheses may be simple or complex; most often individual tags represent relatively simple concepts that are then used collectively to support (or refute) more complex theories. Because the concept of encoding assessments, judg- ments, or analytical conclusions within a graph or hypergraph may be unfamiliar to some, a few examples may be helpful. Example 1 A common concept in tracking cyber threat data is the idea of determining whether an indicator (a ﬁle, domain, IP address, email address, etc.) is part of a “threat cluster” - that is, whether the indicator is associated with a set of related activity presumed to be carried out by some (generally unknown) set of malicious actors. For something to be part of a threat cluster, it must be considered to be unique to that threat cluster – that is, if the indicator (such as a domain) is observed on a network, it can be considered a sure sign that activity associated with the threat cluster is occurring in that network. An analyst reviewing new threat data – say a piece of malware containing a never-before-observed domain – will try to determine whether the activity can be associated with any known threat cluster. The broad question “can this be associated with any known cluster?” can be thought of as comprised of n number of individual hypotheses based on the number of known threat clusters (“This activity is associated with Threat Cluster 1...with Threat Cluster 2...with Threat Cluster n”).

30 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

Let’s say the analyst determines the activity is related to Threat Cluster 12 and therefore applies the tag tc.t12 (tc to indicate the “threat cluster” name space, t12 to indicate Threat Cluster 12) to the malicious domain (inet:fqdn). The presence of that tag indicates that the hypothesis “This domain is associated with Threat Cluster 12” has been assessed to be true, based on the available data. Example 2 The criteria used to evaluate whether an indicator is part of a threat cluster may be complex. Tags (and their underlying hypotheses) can also represent concepts that are much simpler (easier to evaluate). In tracking cyber threat data, let’s say that you want to know how often malicious domains (e.g., domains used in malware communications, or to host phishing or exploit sites) mimic the names of legitimate companies or services, and which companies or services are imitated most often. A tag such as mimic. could be used to indicate this. An analyst evaluating whether a domain mimics the name of a legitimate company or service may ﬁrst identify some similarity with a known company, and then determine whether the domain is legitimately registered to that company. For example, if the analyst determines that the domain g00gle.com bears a resemblance to google.com but is not a legitimate domain registered to Google, they may apply the tag mimic.google to the domain. The hypothesis “This domain mimics Google” has been assessed to be true. More complex hypotheses may not be explicitly annotated within the graph (that is, as individual tags), but may be supported (or refuted) by the presence of other tags or combinations of tags. For example, if your hypothesis is “Threat Cluster 12 frequently registers domains that imitate technology companies”, you could ask Synapse to show you all the domains (inet:fqdn nodes) associated with Threat Cluster 12 (tagged with tc.t12) and then show you which of those domains have a mimic tag: • Comparing the number of mimic domains to the total number of Threat Cluster 12 domains can indicate how often the actors associated with this activity attempt to imitate other services. • The companies or services reﬂected in the mimic tags can indicate the types of organizations imitated. This information will help you evaluate whether or not your hypothesis is true based on currently available data. A corresponding Storm query to help evaluate the above would be: cli> ask inet:fqdn*tag=tc.t12 +#mimic

1.9.2 Tag Considerations

The ability to use tags effectively to make assessments and facilitate further analysis depends on a well-designed analytical model – the set of tags used to annotate data, and the structure of those tags. The specific structure used may be highly specific to a given knowledge domain and given research purpose. However, the following points should be taken into consideration in designing your analytical model. Tag Hierarchies The structure of a tag hierarchy is an important consideration, as the “order” of the tags can affect the types of analytical questions you can most easily answer. Because hierarchies are generally structured from “less specific” to “more specific”, the hierarchy you choose affects how (or whether) you can narrow your focus effectively. Consider whether the structure you create allows you to increase specificity in a way that is analytically relevant or meaningful to the questions you’re trying to answer. For example, let’s say you are storing copies of articles from various news feeds within a Synapse Cortex. You want to use tags to annotate the subject matter of the articles. Two possible options would be: Hierarchy #1

...: us.economics.trade.gdp us.economics.trade.deficit

1.9. Tags - Analytical Model 31 Synapse Documentation, Release 0.0.34

us.economics.banking.lending us.economics.banking.regulatory us.politics.elections.national france.politics.elections.national france.politics.elections.local china.economics.banking.lending

Hierarchy #2

...: economics.trade.gdp.us economics.trade.deficit.us economics.banking.lending.us economics.banking.regulatory.us politics.elections.national.us politics.elections.national.france politics.elections.local.france economics.banking.lending.china

Using Synapse’s Storm query language, it is easy to ask about nodes that have a specific tag (ask #). Storm also allows you to ask about tag nodes (syn:tag forms) that share a common base element (:base secondary property) and then locate all nodes that have any of those tags. While this is a slightly more complex query, it is not overly difficult (ask syn:tag:base= fromtags()). Based on this, you can see how the choice of hierarchy makes it easier (or harder) to ask certain questions. (Note: examples are simplified for discussion purposes. See the Storm reference and Storm tutorial for detailed information on using Storm.) “Show me all the articles related to France”: • Hierarchy #1: ask #france • Hierarchy #2: ask syn:tag:base=france fromtags() “Show me all the articles on to banking within the US”: • Hierarchy #1: ask #us.economics.banking • Hierarchy #2: ask syn:tag:base=us fromtags() +#economics.banking or ask syn:tag:base=us +syn:tag~=banking fromtags() “Show me all the articles about global trade”: • Hierarchy #1: ask syn:tag:base=trade fromtags() • Hierarchy #2: ask #economics.trade “Show me all the articles about national elections”: • Hierarchy #1: ask syn:tag:base=national fromtags() • Hierarchy #2: ask #politics.elections.national Hierarchy #1 makes it easier to ask the first two questions; Hierarchy #2 makes it easier to ask the last two questions. As you can see, choosing one hierarchy over the other doesn’t necessarily prevent you from asking certain questions – if you choose the first hierarchy, you can still ask about global trade issues. However, asking that question (structuring an appropriate Storm query) is a bit harder, and the potential complexity of a query across a poorly-structured set of tags increases as both the tag depth and the total number of tags increases. While the differences in query structure may seem relatively minor, structuring your tags to make it “easier” to ask questions has two important effects:

32 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

• More efficient / performant for Synapse to return the requested data: in general, lifting data by tag will be more efficient than lifting nodes by property and then pivoting from tag nodes to nodes that have those tags. Efficiency may be further impacted if additional operations (filtering, additional pivots) are performed on the results. While these performance impacts may be measured in fractions of seconds or seconds at most, they still impact an analyst’s workflow. • Simpler for analysts to remember: you want analysts to spend their time analyzing data, not figuring out how to ask the right question to retrieve the data in the first place. This has a much bigger impact on an analyst’s workflow. Neither hierarchy is right or wrong; which is more suitable depends on the types of questions you want to answer. If your analysis focuses primarily on news content within a particular geography, the first option (which places “country” at the root of the hierarchy) is probably more suitable. If your analysis focuses more on global geopolitical topics, the second hierarchy is probably better. As a general rule, the analytical focus that you “care about most” should generally go at the top of the hierarchy in order to make it “easier” to ask those questions. Tag Definitions The form of a tag (syn:tag) allows both short-form and long-form definitions to be stored directly on the tag’s node. Consistently using these definition fields to clearly define a tag’s meaning is extremely helpful for analysis. Recall from Data Model – Concepts that a well-designed Synapse data model should be “self-evident”: the structure of the hypergraph (data model) combined with the set of associated tags (analytical model) is able to convey key relationships and assessments in a concise way. In other words, understanding nodes and tags is meant to be simpler (and faster) than reading a long form report about why an analyst interprets X to mean Y. That said, a data model is still an abstraction: it trades the precision and detail of long-form reporting for the power of a consistent model and programmatic access to data and analysis. Within this framework, tags are the “shorthand” for analytical observations and annotations. Nuances of meaning that may be essential for proper analysis get lost if a complex observation is reduced to the tag foo.bar.baz. There is a risk that different analysts may interpret and use the same tag in different ways, particularly as the number of analysts using the system increases. The risk also increases as the number of tags increases, as there may be hundreds or even thousands of tags being used to annotate the data. By convention, the :title secondary property has been used for a “short” definition for the tag – a phrase or sentence at most – while :doc has been used for a detailed definition to more completely explain the meaning of a given tag. The idea is that :title would be suitable to be exposed via an API or UI as a simple definition (such as a label or hover-over), while :doc would be suitable for display on request by a user who wanted more detailed information or clarification. Storing a tag’s definition directly within the Synapse data model helps to make Synapse “self-documenting”: an analyst can view the tag’s definition at any time directly within Synapse simply by viewing the tag node’s properties (ask --props syn:tag=). There is no need to refer to an external application or dictionary to look up a tag’s precise meaning and appropriate use. The same principle applies to syn:tagform (“tagform”) nodes, which were created to document the precise meaning of a tag when it is applied to a specific form (node type). Tagforms support use cases where a tag embodying a particular concept may still have subtle differences in meaning when the tag is applied to different node types – say an inet:ipv4 vs. an inet:fqdn. While these nuances could be documented on the syn:tag node itself, it could make for a very lengthy definition. In those cases it may be preferable to create syn:tagform nodes to separately document the various meanings for a given tag / form combination. Tag Governance Because tags are simply nodes, any user with the ability to create nodes can create a new tag. On one hand, this ability to create tags on the fly makes tags extremely powerful, flexible, and convenient for analysts – they can create annotations to reflect their observations as they are conducting analysis, without the need to wait for code changes or approval cycles.

1.9. Tags - Analytical Model 33 Synapse Documentation, Release 0.0.34

However, there is also risk to this approach, particularly with large numbers of analysts, as analysts may create tags in an uncoordinated and haphazard fashion. The creation of arbitrary (and potentially duplicative or contradictory) tags can work against effective analysis. A middle ground between tag free-for-all and tight tag restrictions (“no new tags without prior approval”) is usually the best approach. It is useful for an analyst to be able to create a tag on demand to record an observation in the moment. However, it is also helpful to have some type of regular governance or review process to ensure the tags are being used in a consistent manner and that any newly created tags fit appropriately into the overall analytical model. This governance and consistency is important across all analysts using a specific instance of Synapse, but is especially important within a broader community. If you plan to exchange data, analysis, or annotations with other groups with their own instances of Synapse, you should use an agreed-upon, consistent data model as well as an agreed-upon set of tags. Level of Detail Tag hierarchies can be arbitrarily deep. If one function of hierarchies is to represent an increasing level of detail, then deep hierarchies have the potential to represent extremely fine-grained analytical observations. More detail is often better; however, tag hierarchies should reflect the level of detail that is relevant for your analysis, and no more. That is, the analysis being performed should drive the set of tags being used and the level of detail they support. (Contrast that approach with taking an arbitrary taxonomy and using it to create tags without consideration for the taxonomy’s relevance or applicability.) Not only is an excess of detail potentially unnecessary to the analysis at hand, it can actually create more work and be detrimental to the analysis you are trying to conduct. Tags typically represent an analytical assertion, which means in most cases a human analyst needs to evaluate the data, make an assessment, and subsequently annotate data with the appropriate tag(s). Using an excessive number of tags or excessively detailed tags means an analyst needs to do more work (keystrokes or mouse clicks) to annotate the data. There is also a certain amount of overhead associated with tag creation itself, particularly if newly created tags need to be reviewed for governance, or if administrative tasks (such as ensuring tags have associated definitions) need to be performed. More importantly, while the physical act of applying a tag to a node may be “easy”, the analytical decision to apply the tag often requires careful review and evaluation of the evidence. If tags are overly detailed, representing shades of meaning that aren’t really relevant, analysts may get bogged down splitting hairs – worrying about whether tag A or tag B is more precise or appropriate when that distiction doesn’t matter to the analysis at hand. In that situation, the analysis is being driven by the overly detailed tags, instead of the tag structure being driven by the analytical need. Where detail is necessary or helpful it should be used; but beware of becoming overly detailed where it isn’t relevant, as the act of annotating can take over from real analysis. Flexibility Just as a good data model will evolve and adapt to meet changing analytical needs, the analytical model represented by a set of tags or tag hierarchies should be able to evolve and adapt. No matter how well-thought-out your tag structure is, you will identify exceptions, edge cases, and observations you didn’t realize you wanted to capture. To the extent possible, your tag structure should be flexible enough to account for future changes. Note that it is relatively easy to “bulk change” tags (to decide a tag should have a different name or structure, and to re-tag existing nodes with the new tag) as long as the change is one-to-one. That is, while the tag name may change, the meaning of the tag does not, so that everything tagged with the old name should remain tagged with the new name. For example, if you decide that foo.bar.baz.hurr and foo.bar.baz.derp provide too much granularity and should both be rolled up into foo.bar.baz, the change is relatively easy. Similarly, if you create the tag foo.bar and later decide that tag should reside under a top-level tag wut, you can rename foo.bar to wut.foo.bar and re-tag the relevant nodes. (Note: Changing the tags is still a manual process as Synapse does not currently support “mass renaming” of tags. However, it is relatively straightforward to lift all nodes that have a given tag, apply the new “renamed” tag to all the nodes, and then delete the syn:tag node for the original tag, which will also remove the old tag from any nodes.)

34 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

This flexibility provides a safety net when designing tag hierarchies, as it allows some freedom to “not get it right” the first time. Particularly when implementing a new tag or set of tags, it can be helpful to test them out on real-world data before finalizing the tags or tag structure. The ability to say “if we don’t get it quite right we can rename it later” can free up analysts or developers to experiment. It is harder to modify tags through means such as “splitting” tags. For example, if you create the tag foo.bar and later decide that bar should really be tracked as two variants (foo.bar.um and foo.bar.wut), it can be painstaking to separate those out, particularly if the set of nodes currently tagged foo.bar is large. For the sake of flexibility it is often preferable to err on the side of “more detail”, particularly during early testing. Consistency of Use Creating a well-thought out set of tags to support your analytical model is ineffective if those tags aren’t used consistently – that is, by a majority of analysts across a majority of relevant data. It’s true that 100% visibility into a given data set and 100% analyst review and annotation of that data is an unrealistic goal. However, for data and annotations that represent your most pressing analytical questions, you should strive for as much completeness as possible. Looked at another way, inconsistent use of tags can result in gaps that can skew your assessment of the data. At best, this can lead to the inability to draw conclusions; at worst, to faulty analysis. This inconsistency often occurs as both the number of analysts and the number of tags used for analysis increase. The larger the team of analysts, the more difficult it is for that team to work closely and consistently together. Similarly, the more tags available to represent different assessments, the fewer tags an analyst can work with and apply within a given time frame. In both cases, analysts may tend to “drift” towards analytical tasks that are most immediately relevant to their work, or most interesting to them – thus losing sight of the collective analytical goals of the entire team. Consider the example above of tracking Internet domains that mimic legitimate companies. If some analysts are annotating this data but others are not, your ability to answer questions about this data is skewed. Let’s say Threat Cluster 12 is associated with 200 domains, and 173 of them imitate real companies, but only 42 have been annotated with mimic tags. If you try to use the data to answer the question “does Threat Cluster 12 consistently register domains that imitate valid companies?”, your assessment is likely to be “no” based on the incompletely annotated data. There are gaps in your analysis because the information to answer this question has only been partially recorded. As the scope of analysis within a given instance of Synapse increases, it is essential to recognize these gaps as a potential shortcoming that may need to be addressed. Options include establishing policy around which analytical tasks (and associated observations) are essential (perhaps even required) and which are secondary (“as time allows”); or designating individual analysts to be responsible for particular analytical tasks. Tag Example It may be helpful to walk through an example of designing a tag structure. While somewhat simplified, it illustrates some of the considerations taken into account. Internet domains (inet:fqdn) used for malicious activity are often taken over by security researchers in a process known as “sinkholing”. The security firm takes control of the domain, either after it expires or in coordination with a domain registrar, and updates the domain’s DNS A record to point to the IP address of a server controlled by the security firm. This allows the security firm to help identify (and ideally notify) victims who are attempting to communicate with the malicious domain. It may also provide insight into the individuals or organizations being targeted by the malicious actors. The process of sinkholing also requires supporting infrastructure used by the security firm. This typically includes (at minimum): • The DNS name servers (inet:fqdn) used to resolve the sinkholed domains. • The IP address(es) (inet:ipv4) the name servers resolve to. • The IP address(es) that the sinkholed domains resolve to. • Any email address(es) (inet:email) used by the security firm to register the sinkholed domains.

1.9. Tags - Analytical Model 35 Synapse Documentation, Release 0.0.34

For cyber threat data purposes, it is useful to know when a domain has been “sinkholed” and is no longer under direct control of a threat group. It is also useful to identify sinkhole infrastructure, which can then be used to identify other sinkholed domains. All of the objects listed above are associated with sinkhole operations, so one option would be to simply use a single tag sinkhole (or sink for short, if you want to save on keystrokes) to denote they are associated with this activity. However, a single tag is not useful if you want to be able to distinguish (and ask about) sinkholed domains separately from legitimate domains associated with the security firm’s sinkhole name servers. A second set of tag elements can be used in combination with sink to distinguish these different components: • dom – the sinkholed domain • ns – the name server used to resolve the domain • nsip – the name server IP address • domip – the sinkhole domain IP address • reg – the email used to register the sinkhole domain Use of a second tag element helps draw better distinctions among the different components, but creates a larger number of tags. However, the sinkholed domain and its IP (as well as the sinkhole name server and its IP) can be considered two aspects of the same concept (“sinkhole domain” and “sinkhole name server”). This could allow you to consolidate some of the tags because the combination of tag plus form allows you to distinguish between “sinkholed domains” (inet:fqdn) and “IP addresses hosting sinkholed domains” (inet:ipv4) even if you use the same tag for both: • dom – a sinkholed domain or the IP address the domain resolves to • ns – a sinkhole name server or the IP address the name server resolves to • reg – the email used to register the sinkhole domain Another consideration is the “order” in which to structure these elements. Does dom.sink make more sense, or sink.dom? Placing dom (and ns and reg) first makes sense if, in your analysis, you are most interested in domains (in general) followed by sinkholed domains (in particular). In this case, the purpose is to track sinkhole operations (in general) and then to be able to distinguish among the different types of infrastructure associated with these operations; so sink. dom makes more sense to allow you to go from “more general” to “more specific”. As a small tweak, because the term “sinkhole” is widely recognized within the security community, changing sink.dom to sink.hole may be a bit more intuitive. Additional information that may be interesting to note is the specific organization responsible for the sinkholed domains and associated infrastructure. In some cases it may be possible to identify the responsible organization (through domain registration records or reverse IP lookups). An additional optional element could be placed at the end of the tag for cases where the organization is known (e.g., sink.hole.kaspersky for Kaspersky Lab). That gives you the following tag structure:

sink sink.hole sink.ns sink.reg sink.hole.kaspersky sink.hole.microsoft sink.ns.microsoft

...etc. This structure allows you to use Storm to ask questions such as: “Show me all of the domains sinkholed by Kaspersky”:

36 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

• ask inet:fqdn*tag=sink.hole.kaspersky “Show me all of the IP addresses associated with sinkhole name servers”: • ask inet:ipv4*tag=sink.ns “Show me all of the Threat Group 12 domains sinkholed by Microsoft”: • ask inet:fqdn*tag=sink.hole.microsoft +#tc.t12 For each of these tags, the corresponding syn:tag nodes can be given a deﬁnition (secondary property :title and / or :doc) within Synapse. Since we are using sink.hole and sink.ns with two different node types (inet:fqdn and inet:ipv4), we can also optionally create syn:tagform nodes with custom deﬁnitions for the meaning of the tag when used on each type of node. A syn:tag node might look like this: cli> ask --props syn:tag=sink.hole syn:tag = sink.hole :base = hole :depth = 1 :doc = A malicious domain that has been sinkholed, or an IP address to which

˓→sinkholed domains resolve. :title = A sinkholed domain or associated IP address :up = sink (1 results)

An optional syn:tagform node representing sink.hole speciﬁcally when applied to inet:ipv4 nodes might look like this: cli> ask --props syn:tagform:tag=sink.hole +syn:tagform:form=inet:ipv4

syn:tagform = 6343cfbdb736d988a72801be48ea07e2 :doc = An IP address used as the DNS A record for a sinkholed domain. :form = inet:ipv4 :tag = sink.hole :title = IP address of a sinkholed domain (1 results)

1.10 Using Synapse - Commands

Synapse includes a command line interface (CLI) for interacting with a Synapse Cortex. The cortex module is used to connect directly to a Cortex. The cmdr module is used to connect to a remote Cortex, which is the most common scenario for users of a production Cortex. To connect to a remote Cortex, pass the path to the Cortex as an argument to the cmdr module: python.exe -m synapse.tools.cmdr Where should look similar to: ://:/ For example: tcp://synapse.woot.com:1234/cortex01 or ssh://[email protected]:1234/cortex01

1.10. Using Synapse - Commands 37 Synapse Documentation, Release 0.0.34

Once you connect to the Cortex, you should have a Synapse command prompt: cli>

1.10.1 Synapse Commands

The Synapse CLI supports various commands for interacting with the Synapse Cortex. The list of currently available commands can be viewed by running help at the command prompt:

cli> help ask - Execute a query. guid - Generate a new guid help - List commands and display help output. quit - Quit the current command line interpreter.

Individual commands are documented in greater detail below. ask – executes a Storm query. Storm is the query language used to interact with data in a Synapse hypergraph. The Synapse ask command indicates that the input following the command and any command options represents a Storm query. Syntax: ask [--debug --[props|raw]] • Running ask with no arguments will display help and usage information. • ask displays the set of nodes returned by in = form, along with their associated tags, if any. For hierarchical (dotted) tags, only the ﬁnal (leaf) tags are displayed. • --debug displays the same data as ask with the addition of informational / diagnostic data about the execution of • --props displays the same data as ask with the addition of the = properties from each node. Output is formatted for readability (e.g., epoch timestamps are displayed in YYYY/ MM/DD hh:mm:ss.mmmm format, IPv4 addresses are displayed as dotted-decimal strings). • --raw displays all properties and tags associated with the node in JSON format, including universal properties (e.g., tufo:form=), ephemeral properties (if any), and all tags (not simply leaf tags). Examples: Retrieve (“lift”) the node representing the domain woot.com. Note that the results indicate the node has been labeled with the tag foo.bar (#foo.bar).

cli> ask inet:fqdn=woot.com

inet:fqdn = woot.com #foo.bar (added 2017/06/20 19:59:02.854) (1 results)

cli> ask --props inet:fqdn=woot.com

inet:fqdn = woot.com #foo.bar (added 2017/06/20 19:59:02.854) :created = 2015/06/07 12:33:44.000 :domain = com :host = woot :sfx = False

38 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

:zone = True (1 results) cli> ask --raw inet:fqdn=woot.com

[ [ "a4d82cf025323796617ff57e884a4738", { "#foo": 1497988742854, "#foo.bar": 1497988742854, "inet:fqdn": "woot.com", "inet:fqdn:created": 1433680424000, "inet:fqdn:domain": "com", "inet:fqdn:host": "woot", "inet:fqdn:sfx": 0, "inet:fqdn:zone": 1, "tufo:form": "inet:fqdn" } ] ] (1 results) cli> ask --debug inet:fqdn=woot.com oplog: lift (took:0) {'sub': 0, 'took': 0, 'mnem': 'lift', 'add': 1} options: limit = None uniq = 1 limits: lift = None time = None touch = None inet:fqdn = woot.com #foo.bar (added 2017/06/20 19:59:02.854) (1 results) guid - generates a Globally Unique Identiﬁer (GUID). guid simply generates a 32-byte random number. One use for this command would be to generate a value that could be used as the primary property for a node that does not have a characteristic (or set of characteristics) that can act as a unique primary property. Syntax: guid (does not take any parameters) Example: cli> guid new guid: '5ed3cf8f1e903f24dacfa5e97aa15878'

1.10. Using Synapse - Commands 39 Synapse Documentation, Release 0.0.34

1.11 Storm Query Language - Basics

1.11.1 Background

Storm is the query language used to interact with data in a Synapse hypergraph. Storm allows you to ask about, retrieve, annotate, add, modify, and delete data from a Cortex. Most Synapse users (e.g., those conducting analysis on the data) will access Storm via the command-line interface (CLI), using the Synapse ask command to invoke a Storm query: cli> ask Storm is based on an underlying set of operators that allow interaction with a Synapse Cortex and its data. The Synapse CLI can invoke Storm operators directly – that is, calling the operator and passing appropriate parameters: cli> ask ( , ... ) For example: cli> ask lift(inet:fqdn,woot.com) That said, Storm is meant to be usable by analysts from a variety of knowledge domains who are not necessarily programmers and who may not be comfortable using operators in what feels like a “programming language”. For this reason, Storm has been designed to mask some of the underlying programmatic complexity. The intent is for Storm to act more like a “data language”, allowing knowledge domain users to: • Reference data and data types in an intuitive form. Through features such as type safety and property normalization, Storm tries to take a “do what I mean” approach, removing the burden of translating or standardizing data from the user where possible. • Use a simplified syntax to run Storm queries. In addition to the standard operator-based Storm syntax (operator syntax), most common operators support the use of a short-form macro syntax to make queries both more intuitive and more efficient (by allowing common queries to be executed with fewer keystrokes). As an example of this simplification, analysts can ask Synapse about a node simply by specifying the node’s form and primary property value (=): cli> ask inet:ipv4=1.2.3.4 Note that Storm accepts the IP address in its “intuitive” form (dotted decimal notation), even though Synapse stores IP addresses as integers (IP 1.2.3.4 is stored as integer 16909060). The analyst does not need to convert the IP to integer form to run the query, nor do they need to escape the IP with quotes ("1.2.3.4") to indicate it is a string representation of the data. (Generally speaking, double quotes only need to be used in limited situations, such as numeric strings that may be interpreted as integers, or when input contains characters that may be interpreted as specialized input (escape characters or characters such as a space or comma that otherwise indicate “end of data”) by the Synapse parser.) In addition, the Storm syntax should feel intuitive to the user, like asking a question: “Tell me about (ask about) the IP address 1.2.3.4”. Analysts still need to learn the Storm “language” and master enough command-line syntax to perform tasks and find help when necessary. However, the intent is for Storm to function more like “how do I ask this question of the data?” and not “how do I write a program to get the data I need?” Finally – and most importantly – giving analysts direct access to Storm to allow them to create arbitrary queries provides them with an extraordinarily powerful analytical tool. Analysts are not constrained by working with a set of predefined queries provided to them through a GUI or an API. Instead, they can follow their analysis wherever it takes them, creating queries as needed and working with the data in whatever manner is most appropriate to their research.

40 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

1.11.2 Storm Operators

Operators implement various Storm functions such as retrieving nodes, applying tags, or pivoting across data. Opera- tors can be divided into broad categories based on their typical use: • Data modification – add, modify, annotate, and delete nodes from a Cortex. • Lift (query) operators – retrieve data based on specified criteria. • Filter operators – take a set of lifted nodes and refine your results by including or excluding a subset of nodes based on specified criteria. • Pivot operators - take a set of lifted nodes and identify other nodes that share one or more properties or property values with the lifted set. • Lift and filter (“by” handlers) – optimize certain queries by lifting and filtering nodes concurrently. • Statistical operators – specialized operators to calculate statistics over a set of nodes. • Miscellaneous operators – various special purpose operators that do not fit into one of the above categories. Most operators (other than those used solely to lift data) require an existing data set on which to operate. This data set is typically the output of a previous Storm operator whose results are the nodes you want to modify or otherwise work with.

1.11.3 Lift, Filter, and Pivot Criteria

Working with Synapse data commonly involves three broad types of operations: • Lifting data (selecting a set of nodes). • Filtering data (down-selecting a subset of nodes from an existing set of nodes). • Pivoting across data (“navigating” the hypergraph by moving from an existing set of nodes to another set of nodes that share some property and / or value with the original set). Whether lifting, ﬁltering, or pivoting across data in a Cortex, you need to be able to clearly specify the data you’re interested in – your selection criteria. In most cases, the criteria you specify will be based on one or more of the following: •A property (primary or secondary) on a node. •A speciﬁc value for a property (=) on a node. •A tag on a node. All of the above elements – nodes, properties, values, and tags – are the fundamental building blocks of the Synapse data model. As such, an understanding of the Synapse data model is essential to effective use of Storm.

1.11.4 Operator Chaining and Node Consumption

Storm allows multiple operators to be chained together to form increasingly complex queries. Storm operators are processed in order from left to right with each operator acting on the current result set (e.g., the output of the previous operator). From an analysis standpoint, this feature means that Storm can parallel an analyst’s natural thought process: “show me X data...that’s interesting, show me the Y data that relates to X...hm, take only this subset of results from Y and show me any relationship to Z data. . . ” and so on.

1.11. Storm Query Language - Basics 41 Synapse Documentation, Release 0.0.34

From a practical standpoint, it means that order matters when constructing a Storm query. A lengthy Storm query is not evaluated as a whole. Instead, Synapse parses each component of the query in order, evaluating each component individually as it goes. The Storm runtime(s) executing the query keep a list of lifted nodes in memory while performing the requested operations (lifts, pivots, data modification, and so on.) Most Storm operators “consume” nodes upon input. That is, the set of nodes input into a particular Storm operator is typically transformed by that operator in some way. With few exceptions, the nodes input to the operator are not retained - they are consumed during processing. Storm outputs and retains in memory only those nodes that result from carrying out the specified operation. In this way the operators used may add or remove nodes from Storm’s in-memory “working set”, or clear the set entirely; as such this set is continually changing based on the last-used operator. Particularly when first learning Storm, users are encouraged to break down complex queries into their component parts, and to validate the output (results) after the addition of each operator to the overall query.

1.11.5 Syntax Conventions

The Synapse documentation provides numerous examples of both abstract Storm syntax (usage statements) and specific Storm queries. The following conventions are used for Storm usage statements: • Items that must be entered literally on the command line are in bold. These items include the command name and literal characters. • Items representing variables that must be replaced by a name are in italics. • Bold brackets are literal characters. Parameters enclosed in non-bolded brackets are optional. • Parameters not enclosed in brackets are required. • A vertical bar signifies that you choose only one parameter. For example, [ a | b ] indicates that you can choose a, b, or nothing. • Ellipses ( ... ) signify the parameter can be repeated on the command line. Whitespace may be used in the examples for formatting and readability. Synapse will parse Storm input with or without whitespace (e.g., the Synapse parser will strip / ignore whitespace in Storm queries; the exception is that whitespace within double-quoted strings is preserved, such as the timestamp in the example below). For example, the following Storm queries are equivalent to the Synapse parser: addnode( inet:fqdn , woot.com , : created = "2017-08-15 01:23" ) addnode(inet:fqdn,woot.com,:created="2017-08-15 01:23") Examples of specific queries represent fully literal input, but are not shown in bold for readability. For example: Usage statement: addnode( , , [ : = , ...] ) Specific query: addnode(inet:fqdn,woot.com)

1.11.6 Operator Syntax vs. Macro Syntax

Storm operators function similar to a programming language, where the operator acts as a function and the operator’s parameters act as input to that function. With very few exceptions, all Storm operators can be used at the Synapse command line by invoking the Synapse ask command, calling the appropriate Storm operator, and passing appropriate parameters to the operator; this is known as operator syntax and provides the most complete access to Storm’s functionality.

42 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

While Storm’s operator syntax is both detailed and complete, it has a few drawbacks: • It can feel very “code-like”, particularly to analysts or other Synapse users who are not programmers. • It has few optimizations, meaning that every operator and its associated parameters must be typed in full. This can become tedious for users who interact heavily with Synapse using Storm. To address these issues, Storm also supports what is known as macro syntax. Macro syntax acts as a sort of “shorthand” through techniques such as: • Replacing operators with equivalent intuitive symbols. • Allowing the omission of explicit operator names or parameters where there is an obvious default value. The macro syntax is meant to be both more efficient (requiring fewer keystrokes) and more intuitive, a “data language” for asking questions of the data as opposed to a programming language for retrieving data from a data store. While not every operator has a macro syntax equivalent, the most commonly used operators have been implemented both ways. When Storm macro syntax is used at the CLI, Synapse automatically “translates” the macro syntax to the equivalent operator syntax in order to execute the requested query. Two examples – one simple, one more complex – illustrate the differences between the two. Example 1 The most basic Storm query simply lifts (retrieves) a single node (such as the domain woot.com) using the lift() operator: cli> ask lift(inet:fqdn,woot.com) The same query can be executed as follows using macro syntax: cli> ask inet:fqdn=woot.com Note that in macro syntax, the lift() operator – the most fundamental Storm operator – is eliminated entirely; macro syntax assumes you want to retrieve (lift) nodes unless you specify otherwise. Similarly, instead of entering comma-separated parameters as input to the operator, macro syntax supports the use of the simple = pattern to ask about the node in question. Example 2 The usefulness of macro syntax is even more apparent with longer, more complex queries. Storm allows users to chain operators together to lift a set of nodes and perform a series of additional filter and pivot operations that follow a line of analysis across the data. In the knowledge domain of cyber threat data, there is a common analytical workflow used to research potentially malicious infrastructure. This line of analysis takes a set of “known bad” domains (for example, those associated with a known threat cluster), identifies the IP addresses those domains have resolved to, excludes some potentially irrelevant IPs, and then identifies other domains that have resolved to those IPs. Domains that resolved to the same IP address(es) as the “known bad” domains during the same time period may be associated with the same threat. The full query for this line of analytical reasoning using operator syntax would be: cli> ask lift(inet:fqdn,by=tag,tc.t12) pivot(inet:fqdn,inet:dns:a:fqdn) pivot(inet:dns:a:ipv4,inet:ipv4) -#anon.tor -#anon.vpn pivot(inet:ipv4,inet:dns:a:ipv4) pivot(inet:dns:a:fqdn,inet:fqdn)

The same query using macro syntax would be: cli> ask inet:fqdn*tag=tc.t12 -> inet:dns:a:fqdn :ipv4 -> inet:ipv4 -#anon.tor -#anon. ˓→vpn -> inet:dns:a:ipv4 :fqdn -> inet:fqdn

1.11. Storm Query Language - Basics 43 Synapse Documentation, Release 0.0.34

The components of the query are broken down below; note how each new component builds on the previous query to follow the line of analysis and reﬁne results: Request Operator & Macro Syntax Macro Syntax Notes List all domains tagged as Threat Operator lift(inet:fqdn, • Omit “lift(...)” operator Cluster 12 by=tag,tc.t12) • Asterisk ( * ) substitutes for Macro inet:fqdn*tag=tc. “by” parameter 12

Pivot from these domains to DNS Operator pivot(inet:fqdn, • Omit “from” parameter in A record nodes that have those do- inet:dns:a:fqdn) pivot (inet:fqdn) as it is mains Macro -> inet:dns:a:fqdn the primary property of our working result set (i.e., default input value) • Arrow ( -> ) substitutes for “pivot” operator

Pivot from those DNS A record Operator • “From” parameter spec- nodes to the IP addresses those do- pivot(inet:dns:a:ipv4, iﬁed as relative vs full mains have resolved too inet:ipv4) property (:ipv4 vs Macro :ipv4 -> inet:ipv4 inet:dns:a:ipv4) • Arrow ( -> ) substitutes for “pivot” operator

Remove any IP addresses tagged as Operator No Operator syntax • Filter operation; the minus ( - TOR exit nodes available ) represents an exclusion ﬁlter Macro -#anon.tor • Hash ( # ) substitutes for “tag”

Remove any IP addresses tagged as Operator No Operator syntax • Filter operation; the minus ( - anonymous VPN infrastructure. available ) represents an exclusion ﬁlter Macro -#anon.vpn • Hash ( # ) substitutes for “tag”

Pivot from those remaining IP ad- Operator pivot(inet:ipv4, • Omit “from” parameter in dresses to any DNS A records where inet:dns:a:ipv4) pivot (inet:ipv4) as it is those IPs were present Macro -> inet:dns:a:ipv4 the primary property of our working result set (i.e., default input value) • Arrow ( -> ) substitutes for “pivot” operator

Pivot from those DNS A records to Operator • “From” parameter spec- the domains associated with those pivot(inet:dns:a:fqdn, iﬁed as relative vs full records inet:fqdn) property (:fqdn vs Macro :fqdn -> inet:fqdn inet:dns:a:fqdn) • Arrow ( -> ) substitutes for “pivot” operator

Note: Filter operations at the command line (CLI) are performed using macro syntax; there is no equivalent operator

44 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34 syntax. See the Storm reference guides or a detailed discussion of individual operators and their operator and / or macro syntax.

1.11.7 Query Optimization - “Good” and “Bad” Queries

Storm is meant to be flexible as well as performant across large and diverse data sets. There is no single “right” way to use Storm to ask a question of the hypergraph data. However, there are definitely “better” (more efficient or more performant) ways to ask a question. Given that there is typically more than one “path” to an answer (more than one way to ask the question), analysts should consider which path may be more optimal (or at least consider which path is not optimal) when formulating a Storm query. Crafting an optimal query can mean the difference between quickly receiving a meaningful response and waiting for Synapse to return a response because it is processing an excessive amount of data. Synapse currently has no built-in timeouts or other limits (such as total number of nodes lifted) on Storm queries, though these “safety nets” are planned for a future release. Asking a “bad” (non-performant) question will not harm Synapse, but it may frustrate analysts waiting for their CLI to return a response. As a simple example of a “bad” vs “good” query, let’s say you want to lift all of the IP addresses that are associated with Threat Cluster 12. There are two key components to the data you want to ask about: IP addresses (inet:ipv4), represented by a set of nodes; and the activity (set of related indicators) known as Threat Cluster 12, represented by a tag (tc.t12) applied to the relevant nodes. Two ways to ask that question using Storm are: • Lift all of the IP addresses in Synapse, then filter down to only those tagged as part of Threat Cluster 12: cli> ask inet:ipv4 +#tc.t12 • Lift all of the nodes tagged as part of Threat Cluster 12, then filter down to only IP address nodes: cli> ask #tc.t12 +inet:ipv4 The first query is problematic because it asks Storm to return all inet:ipv4 nodes within the hypergraph – potentially hundreds of thousands, or even millions of nodes, depending on how densely populated the hypergraph is (math- ematically speaking, there are over four billion possible IPv4 addresses). Synapse has to lift all of those inet:ipv4 nodes into memory and then select only those nodes with the tc.t12 tag. The query is likely to take an extremely long time to return (at least until query limits are incorporated into Synapse), and therefore represents a “bad” query. The second query asks Storm to return all nodes tagged with tc.t12. This may still be a large number depending on how much analysis and annotation has been performed related to Threat Cluster 12. However, the number of nodes tagged tc.t12 will still be much smaller than the number of inet:ipv4 nodes within a hypergraph. As such, the second query is more efficient or performant, and represents a “good” (or at least “better” query). (Note: The previous example is used for simple illustrative purposes. Technically, the “best” way to ask this particular question would be to use what is called a Storm “by” handler (represented by the asterisk ( * )) to “lift by tag”: cli> ask inet:ipv4*tag=tc.t12 “By” handlers are specifically designed to further optimize certain queries by lifting and filtering nodes concurrently, as opposed to lifting nodes and then filtering the results.)

1.12 Storm Reference - Data Modiﬁcation Operators

The operators below can be used to modify the Synapse hypergraph by: • adding or deleting nodes

1.12. Storm Reference - Data Modiﬁcation Operators 45 Synapse Documentation, Release 0.0.34

• adding cross-reference (xref) nodes • setting, modifying, or deleting properties on nodes • adding or deleting tags from nodes All of these operators are deﬁned in storm.py.

Warning: Synapse does not have an “are you sure?” prompt. Caution should be used with operators that can modify Synapse data, especially when used on the output of complex queries that may modify (or delete) large numbers of nodes. It is strongly recommended that you validate the output of a query by ﬁrst running the query on its own to ensure it returns the expected results before applying any operator that will modify that data.

See the Storm Syntax Conventions for an explanation of the usage format used below. Where speciﬁc query examples are given, they are commonly provided in pairs using operator syntax followed by the equivalent macro syntax (if available).

1.12.1 addnode()

Adds the speciﬁed node(s) to a Cortex. Operator Syntax: addnode( , , [ : = , ...] ) addnode( ,( , , ... ), [ : = , ...] ) Macro Syntax: [ = ... [ : = ...] ]

[ = ( , , ... ) [ : = ...] ] Examples: Simple Node: addnode( inet:fqdn , woot.com )

[ inet:fqdn = woot.com ]

Separator (sepr) Node: addnode( inet:dns:a , ( woot.com , 1.2.3.4 ) )

[ inet:dns:a = ( woot.com , 1.2.3.4 ) ]

Composite (comp) Node: addnode( inet:follows , (twitter.com/ernie , twitter.com/bert ) )

[ inet:follows = ( twitter.com/ernie , twitter.com/bert ) ]

Comp Node with Optional Values: Todo

46 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

Cross-reference (xref) Node: addnode( file:txtref , ( d41d8cd98f00b204e9800998ecf8427e , inet:fqdn , woot.com ) )

[ file:txtref = ( d41d8cd98f00b204e9800998ecf8427e , inet:fqdn , woot.com ) ]

Node with Properties: addnode( inet:dns:a , ( woot.com , 1.2.3.4 ) , :seen:min = "2017-08-01 01:23" , :seen:max = "2017-08-10 04:56" )

[ inet:dns:a = ( woot.com , 1.2.3.4 ) :seen:min = "2017-08-01 01:23" :seen:max = "2017-08-10 04:56" ]

Usage Notes: • addnode() used at the Synapse CLI is most suitable for adding a relatively small number of nodes. For larger amounts of data, it is preferable to use the Synapse ingest subsystem to automate the process. • When creating a whose consists of multiple components, the components must be passed as a comma-separated list enclosed in parentheses. • addnode() will create non-deconflictable node types. • addnode() will check whether a deconflictable node type already exists and either create it or return information on the existing node. • Secondary properties must be specified by their relative property name. For the form foo:bar and the property baz (e.g., foo:bar:baz) the relative property name is specified as :baz. • Specifying one or more secondary properties will set the = if it does not exist, or modify (overwrite) the = if it already exists. Operator Syntax Notes: • The operator syntax can only create only one node at a time. Macro Syntax Notes: • The macro syntax can create as many nodes as are specified within the brackets. • All nodes specified within the brackets that do not already exist will be created. For nodes that already exist, Synapse will return data for that node.

1.12.2 addxref()

Create one or more cross-reference (xref) nodes that reference a speciﬁed =. Operator Syntax: addxref( , , ) Macro Syntax: None. Examples: file:bytes = d41d8cd98f00b204e9800998ecf8427e addxref( file:txtref , inet:fqdn , woot.

˓→com )

Usage Notes:

1.12. Storm Reference - Data Modiﬁcation Operators 47 Synapse Documentation, Release 0.0.34

• addxref() operates on the output of a previous Storm query. • There are currently two valid values, file:txtref and file:imgof. For both of those types, the Storm query should return one or more file:bytes nodes. • Xref nodes can also be created with addnode() using the syntax for creating a comp node type (e.g., addnode( file:txtref , ( , , ) )). Note that addnode() can only create one xref node at a time (e.g., from a single file:bytes node to a single =). • addxref() may be useful if you want to create multiple xref nodes from multiple file:bytes nodes to the same = at once (e.g., if you have eight photographs of the same object).

1.12.3 setprop()

Sets one or more property values on the speciﬁed node(s). Operator Syntax: setprop( : = , ... ) Macro Syntax: [: = ... ] Examples:

inet:dns:a = woot.com/1.2.3.4 setprop( :seen:min = "2017-08-01 01:23" , :seen:max = "2017-08-10 04:56" )

inet:dns:a = woot.com/1.2.3.4 [ :seen:min = "2017-08-01 01:23" :seen:max = "2017-08-10 04:56" ]

Usage Notes: • setprop() operates on the output of a previous Storm query. • Secondary properties must be specified by their relative property name. For the form foo:bar and the property baz (e.g., foo:bar:baz) the relative property name is specified as :baz. • Synapse will set the secondary propert(ies) for all nodes returned by for which that secondary property is a valid property. Nodes for which that property is not a valid secondary property will be ignored. • setprop() will create and set the property if it does not exist, or overwrite the existing = if it does exist. • setprop() can set or modify any property not explicitly defined as read only ('ro' : 1) in the data model. Attempts to modify read only properties will fail silently (e.g., the property value will not be overwritten, but the user will not be notified that the request failed). • setprop() cannot be used to remove (delete) a property entirely. Operator Syntax Notes: • N/A Macro Syntax Notes: • Synapse will attempt to set the specified propert(ies) for all previously referenced nodes (e.g., to the left of the = statement) for which that property is valid, whether those nodes are within or outside of the macro syntax brackets. See Special Note on Macro Syntax.

48 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

1.12.4 addtag()

Adds one or more tags to the speciﬁed node(s). Operator Syntax: addtag( [ @ - , ... ] ) Macro Syntax: [# @ - ... ] Examples: Add Tags inet:fqdn = woot.com addtag( foo.bar , baz.faz ) inet:fqdn = woot.com [ #foo.bar #baz.faz ]

Add Tag with Single Timestamp inet:fqdn = woot.com addtag( baz.faz@201708151330 ) inet:fqdn = woot.com [ #baz.faz@201708151330 ]

Add Tag with Time Boundaries inet:fqdn = woot.com addtag( baz.faz@20160101-20160131 ) inet:fqdn = woot.com [ #baz.faz@20160101-20160131 ]

Usage Notes: • addtag() operates on the output of a previous Storm query. • Synapse will apply the specified tag(s) to all nodes returned by . • Timestamps (in the format YYYYMMDDHHMMSS) can be added to a tag to show a point in time or a range during which the tag was known to be valid (equivalent to :seen:min and :seen:max for the tag). • Timestamps must have a minimum resolution of YYYY. • If one timestamp is provided and no timestamps currently exist on the tag, Synapse will set both the minimum and maximum timestamps as specified. • If a two timestamps are provided and no timestamps currently exist on the tag, Synapse will set the minimum and maximum timestamps as specified. • If timestamps already exist on the tag, Synapse will check the timestamp argument(s) provided against the existing timestamps: – If a timestamp argument is earlier than the current minimum timestamp, Synapse will update the minimum time with the new value. – If a timestamp argument is later than the current maximum timestamp, Synapse will update the maximum time with the new value. – If timestamp arguments fall between the existing minimum and maximum, no updates will be made. • In short, the timestamp window on a given tag can be updated by being “pushed out” from the current values, but there is currently no way to “decrease” the window (other than deleting the tag from the node and recreating it).

1.12. Storm Reference - Data Modiﬁcation Operators 49 Synapse Documentation, Release 0.0.34

Operator Syntax Notes: • N/A Macro Syntax Notes: • Synapse will set the speciﬁed tag(s) for all previously referenced nodes (e.g., to the left of the statement) whether those nodes are within or outside of the macro syntax brackets. See Special Note on Macro Syntax.

1.12.5 delnode()

Deletes the speciﬁed node(s) from a Cortex. Operator Syntax: delnode( [ force=1 ] ) Macro Syntax: None. Examples:

inet:fqdn = woot.com delnode()

inet:fqdn = woot.com delnode(force=1)

Usage Notes: • delnode() operates on the output of a previous Storm query. • delnode() can be executed with no parameters, although this effectively does nothing (i.e., the operator will consume input, but not actually delete the nodes). • Use of the force=1 parameter will delete the nodes input to the operator. The need to enter force=1 is meant to require the user to think about what they’re doing before executing the delnode() command (there is no “are you sure?” prompt). Future releases of Synapse will support a permissions structure that will limit the users who are able to execute this operator.

Warning: delnode() has the potential to be destructive if executed on an incorrect, badly formed, or mistyped query. Users are strongly encouraged to validate their query by ﬁrst executing it on its own to conﬁrm it returns the expected nodes before executing delnode(). Consider the difference between running inet:fqdn=woot. com delnode(force=1) (which deletes the single node for the domain woot.com and accidentally running inet:fqdn delnode(force=1) (which deletes ALL inet:fqdn nodes).

1.12.6 delprop()

Delete a property from the speciﬁed node(s). Operator Syntax: delprop( : ,[ force=1 ] ) Macro Syntax: [ -: ... ] Examples:

50 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

# Operator examples inet:fqdn = vertex.link delprop(:created) inet:fqdn = vertex.link delprop(:created, force=1)

# Macro examples inet:fqdn = vertex.link [ -:created ] inet:fqdn = vertex.link [ -:created -:updated ]

Usage Notes: • Props which are read-only (ro=1) or have a default value (defval) on them cannot be deleted. Operator Syntax Notes: • Use of the force=1 parameter will delete the prop on the input nodes to the operator. The need to enter force=1 is meant to require the user to think about what they’re doing before executing the delprop() command (there is no “are you sure?” prompt). Future releases of Synapse will support a permissions structure that will limit the users who are able to execute this operator. This is only required when using the operator syntax. Macro Syntax Notes: • Synapse will delete the speciﬁed props(s) from all previously referenced nodes (e.g., to the left of the -: statement), whether those nodes are within or outside of the macro syntax brackets. See Special Note on Macro Syntax.

Warning: delprop() has the potential to be destructive if executed on an incorrect, badly formed, or mistyped query. Users are strongly encouraged to validate their query by ﬁrst executing it on its own to conﬁrm it returns the expected nodes before executing delprop() or using the macro delprop syntax. While this cannot remove a node from the graph, it is possible that a bad delprop call can irreversibly damage graph traversal.

1.12.7 deltag()

Deletes one or more tags from the speciﬁed node(s). Operator Syntax: deltag( [ , ... ] ) Macro Syntax: [ -# ... ] Examples: inet:fqdn = woot.com deltag( baz.faz ) inet:fqdn = woot.com [ -#baz.faz ]

Usage Notes: • deltag() operates on the output of a previous query. • Deleting a leaf tag deletes only the leaf tag.

1.12. Storm Reference - Data Modiﬁcation Operators 51 Synapse Documentation, Release 0.0.34

• Deleting a non-leaf tag deletes that tag and all tags below it in the tag hierarchy. Operator Syntax Notes: • N/A Macro Syntax Notes: • Synapse will delete the speciﬁed tag(s) from all previously referenced nodes (e.g., to the left of the statement), whether those nodes are within or outside of the macro syntax brackets. See Special Note on Macro Syntax.

1.12.8 Special Note on Macro Syntax

The square brackets ( [] ) used for the Storm macro syntax indicate “perform the enclosed data modifications” in a generic way. As such, the brackets are shorthand to request any of the following: • Add nodes (addnode()). • Add or modify properties (setprop()). • Delete properties (delprop()). • Add tags (addtag()). • Delete tags (deltag()). This means that all of the above directives can be specified within a single set of macro syntax brackets, in any combination and in any order. However, it is important to keep in mind that the brackets are NOT a boundary that segregates nodes. The brackets simply indicate the start and end of data modification shorthand. They do NOT separate “nodes these modifications should apply to” from “nodes they should not apply to”. The Storm operator chaining with left-to-right processing order still applies. Any modification request that operates on previous Storm output will operate on the output of everything “leftwards” of the modifier, regardless of whether that content is within or outside of the macro syntax brackets. For example:

inet:ipv4 = 12.34.56.78 inet:fqdn = woot.com [ inet:ipv4 = 1.2.3.4 :created = "2016-12-18 00:35" inet:fqdn = woowoo.com #my.tag ]

The above statement will: • Lift the nodes for IP 12.34.56.78 and domain woot.com (if they exist); • Create the node for IP 1.2.3.4 (if it does not exist), or retrieve it if it does; • Set the :created property for domain woot.com; • Create the node for domain woowoo.com (if it does not exist), or retrieve it if it does; • Apply the tag my.tag to IP 12.34.56.78 and domain woot.com (if they exist) and to IP 1.2.3.4 and domain woowoo.com.

1.13 Storm Reference - Lift Operators

The operators below can be used to retrieve a set of nodes from a Synapse hypergraph based on a set of speciﬁed criteria. All of these operators are deﬁned in storm.py. See the Storm Syntax Conventions for an explanation of the usage format used below.

52 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

Where speciﬁc query examples are given, they are commonly provided in pairs using operator syntax followed by the equivalent macro syntax (if available).

1.13.1 lift()

Lifts (retrieves) a set of nodes from a Cortex based on specified criteria. Optional parameters: • “by” handler: a “by” handler is a modifier that limits the lift criteria in various ways. Use of a “by” handler can be thought of as similar to performing a filter operation concurrent with lifting instead of after lifting. – by= (operator syntax) – * (macro syntax) “By” handlers have specific use cases and are documented separately. • Return limit: specify the maximum number of nodes to be returned by the lift query. – limit= (operator syntax) – ^ (macro syntax) Lift operations are highly flexible. Only basic examples of lift() usage are provided here. See the section on by handlers for specialized lift operations using the “by” parameter. Operator Syntax: lift( [ , , limit= , by= ] ) Macro Syntax: [ ^ * = ] Examples: Lift by primary property (form): • Lift all domain nodes:

lift( inet:fqdn )

inet:fqdn

Lift by secondary property: • Lift all inet:fqdn nodes that have an :updated secondary property:

lift( inet:fqdn:updated )

inet:fqdn:updated

Lift by primary prop=valu: • Lift the node for domain woot.com:

lift( inet:fqdn , woot.com )

inet:fqdn = woot.com

Lift by secondary prop=valu: • Lift all DNS A record nodes where the domain is woot.com:

1.13. Storm Reference - Lift Operators 53 Synapse Documentation, Release 0.0.34

lift( inet:dns:a:fqdn , woot.com )

inet:dns:a:fqdn = woot.com

Lift with limit: • Lift 10 domain nodes:

lift( inet:fqdn , limit=10 )

inet:fqdn^10

• Lift 25 DNS A record nodes where the IP is 127.0.0.1:

lift( inet:dns:a:ipv4 , 127.0.0.1 , limit=25)

inet:dns:a:ipv4^25=127.0.0.1

Lift with “by” handlers – lift by tag A single example using a “by” handler is provided for illustrative purposes. Individual “by” handlers may use their own custom syntax. See the Storm By Handler reference for additional details and usage. • Lift by tag: lift all inet:fqdn nodes that have the tag my.tag:

lift( inet:fqdn , by=tag , my.tag)

inet:fqdn*tag=my.tag

• Lift 10 inet:fqdn nodes with the tag my.tag:

lift(inet:fqdn,limit=10,by=tag,my.tag)

inet:fqdn^10*tag=my.tag

Usage Notes: • Nodes are typically lifted by: – Primary property (form): Lifting by form is possible but often impractical. Specifying a form alone attempts to lift all nodes of that form. – Primary prop=valu: Lifts a node by its primary property value. This is the most common method for lifting a single node. – Secondary property: Similar to lifting by form, lifting by a secondary property alone will lift all nodes with that property, regardless of the property’s specific value. It is often impractical but may be feasible in limited cases (e.g., where only a relatively small number of nodes have an given secondary property). – Secondary prop=valu: Lifts all nodes that have the secondary property with the specified value. • When lifting by prop + valu, additional comparison operators can be used besides just equals ( = ); these include ‘not equals’, ‘greater than or equal to’, etc. Use of these comparison operators is covered under Storm By Handlers. • For lift() operations at the CLI, it is generally simpler to use macro syntax. • The limit= option (^ in macro syntax) restricts the number of nodes returned, regardless of the total number of nodes that would otherwise be returned by the query. The specific nodes returned are non-deterministic. Limiting the results of a query is generally not useful for analysis (it artificially restricts results) but may be

54 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

useful for troubleshooting queries or returning “exemplar” nodes (e.g., to examine their structure, properties, etc.) • The number of nodes returned by any query can also be restricted by using the limit() operator. The ﬁrst set of examples below uses the limit parameter to the lift() operator (in both operator and macro syntax). The second set of examples uses the limit() operator (in both operator and macro syntax - note that the macro syntax is equivalent in each case).

lift ( inet:fqdn , limit=10 )

inet:fqdn^10

vs.

lift ( inet:fqdn ) limit( 10 )

inet:fqdn^10

1.13.2 guid()

Lifts one or more nodes based on each node’s Globally Unique Identiﬁer (GUID). Operator Syntax: guid( [ , ... ] ) Macro Syntax: N/A Examples: guid( a4d82cf025323796617ff57e884a4738 ) guid( 6472c5f038b0a4e5b1853c49e688fc74 , 5413b2ae7632a0909d63d31a33ec0807 )

Usage Notes: • The GUID is a unique identiﬁer assigned to every node. (This identiﬁer is not the GUID value used as a primary property by some forms). This GUID is frequently referred to as the iden in API documentation. • The GUID for a node or set of nodes can be displayed at the Synapse CLI by using the ask --raw option pre- ceding a Storm query. For example, in the query output below, b19fe2a26bbe4a6c74b051142d0e5316 is the GUID for the requested node:

ask --raw inet:ipv4=1.2.3.4 [ [ "b19fe2a26bbe4a6c74b051142d0e5316", { "inet:ipv4": 16909060, "inet:ipv4:asn": 0, "inet:ipv4:cc": "??", "inet:ipv4:type": "??", "tufo:form": "inet:ipv4" } ] ] (1 results)

1.13. Storm Reference - Lift Operators 55 Synapse Documentation, Release 0.0.34

1.13.3 alltag()

Lifts a set of nodes based on one or more tags. Operator Syntax: alltag( [ , ...] ) Macro Syntax: # ... Examples: Lifts all nodes that have the tag foo.bar or the tag baz.faz. alltag( foo.bar , baz.faz )

#foo.bar #baz.faz

Usage Notes: • alltag() retrieves all nodes that have any of the speciﬁed tags.

1.14 Storm Reference - Filter Operator, Comparators, and Helpers

The filter operator operates on the output of a previous Storm query. The filter operator, along with various comparison operators (comparators) and helper functions can be used to filter (refine, downselect) the set of nodes returned by the Storm query, based on specified criteria. The filter operator, comparators, and helper functions are all defined in storm.py. See the Storm Syntax Conventions for an explanation of the usage format used below. Filter operations differ in that, while there is a filt() operator within Synapse, the operator syntax is not available via Storm. In other words, all filter operations within Storm are performed using macro syntax.

1.14.1 ﬁlt()

Filters the set of nodes output by a Storm query by either including ( + ) or excluding ( - ) a set of nodes based on speciﬁed criteria. Macro Syntax + | - | | # | ( ) Examples: These basic examples all use the “equals” comparator. See below for additional examples using other comparators. Filter by primary property (form): • Downselect to include only domains:

+inet:fqdn

Filter by primary / : • Downselect to exclude the domain google.com:

-inet:fqdn = google.com

56 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

Filter by secondary property: • Downselect to exclude domains with an “expires” property:

-inet:fqdn:expires

Filter by secondary / : • Downselect to include only those domains that are also logical zones:

+inet:fqdn:zone = 1

Filter by tag: • Downselect to exclude nodes tagged as associated with Tor:

-#anon.tor

Usage Notes: • A filter operation downselects from the working set of nodes by either including or excluding a subset of nodes based on a set of criteria. –+ specifies an inclusion filter. The filter downselects the working set to only those nodes that match the specified criteria. –- specifies an exclusion filter. The filter downselects the working set to all nodes except those that match the specified criteria. • The comparator (comparison operator) specifies how is evaulated with respect to . The most common comparator is equals ( = ), although other standard comparators are available. •A helper function is a Storm-specific feature that optimizes certain types of filter operations. Helpers are discussed separately.

1.14.2 Filter Comparators

Filter operations can be performed using any of the standard mathematical / logical comparison operators (comparators): • = : equals • < : less than • > : greater than • <= : less than or equal to • >= : greater than or equal to • ~= : regular expression Examples: Less than: • Downselect to include only domains created before January 1, 2017:

+inet:fqdn:created < "20170101"

Greater than: • Downselect to exclude ﬁles larger than 4096 bytes:

1.14. Storm Reference - Filter Operator, Comparators, and Helpers 57 Synapse Documentation, Release 0.0.34

-file:bytes:size > 4096

Less than or equal to: • Downselect to include only DNS A records whose most recent observed time was on or before March 15, 2014 at 12:00 UTC:

+inet:dns:a:seen:max <= "201403151200"

Greater than or equal to: • Downselect to include only people born on or after January 1, 1980:

+ps:person:dob >= "19800101"

Regular expression: • Downselect to include only domains that start with the string “serve”:

+inet:fqdn ~= "serve*"

Usage Notes: • Storm does not include a “not equal to” ( != ) comparator. Since ﬁltering is either an inclusive ( + ) or exclusive ( - ) operation, equivalent logic can be performed using “equals” ( = ): – “exclude things not equal to ” is equivalent to “include things equal to “ – “include things not equal to ” is equivanelt to “exclude things equal to “

1.14.3 Filter Helper Functions

Storm includes a number of filter helper functions. These helpers are designed to optimize queries that would otherwise require multiple filter operations (such as querying for multiple values, or a range of values, for a specified property). Storm also includes a set of by handlers that are used in conjunction with lift() operations (as in “lift by...”). While filter helpers optimize certain filter operations, they are carried out after an initial lift() operation. By handlers are similar to filter helpers but optimize certain lift() operations by effectively lifting and filtering nodes in a single operation. With respect to the use of by handlers vs. filter helper functions, neither is “more correct” than the other. Because they perform similar functions (lift and filter vs. lift then filter) the set of by handlers and the set of filter functions largely parallel each other. In other words, you can “lift by X or Y” using a by handler or you can “filter by X or Y” using a filter helper. By handlers are typically “more efficient” because they filter during the lift operation as opposed to after; however, the performance impact will typically be insignificant except in the case of very large lift() operations. Individual filter helper functions are documented below. re() Todo and() Todo or() Todo

58 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34 in() Todo has() Todo seen() Todo range() Todo tag() Todo ival() Todo ivalival() Todo

1.15 Storm Reference - Pivot Operators

The operators below can be used to pivot from one set of nodes to another set of nodes within a Synapse hypergraph based on a set of specified criteria. All of these operators are defined in storm.py. See the Storm Syntax Conventions for an explanation of the usage format used below. Where specific query examples are given, they are commonly provided in pairs using operator syntax followed by the equivalent macro syntax (if available).

1.15.1 pivot()

Todo

1.15.2 join()

Todo

1.15.3 refs()

Todo

1.15.4 fromtags()

Todo

1.15. Storm Reference - Pivot Operators 59 Synapse Documentation, Release 0.0.34

1.15.5 totags()

Todo

1.15.6 jointags()

Todo

1.15.7 tree()

The tree() operator acts as a recursive pivot for forms which reference their own types or multiple duplicate ptypes. Operator Syntax: tree( , , [ recurnlim= ] )

tree( , , [ recurnlim= ] )

tree( , [ recurnlim= ] ) Macro Syntax: There is no macro syntax for the tree() operator. Examples:

# Full form - traversing to all of the woot.com children nodes inet:fqdn= woot.com tree( inet:fqdn, inet:fqdn:domain )

# Relative source only form - traversing ou:suborg relationships ou:org:alias= someorg-> ou:suborg:org tree( :sub, ou:suborg:org ) :sub-> ou:org

# Destination only form - traversing all of the woot.com children nodes. inet:fqdn= woot.com tree( inet:fqdn:domain )

# Select the entire syn:tage=foo tree. syn:tag=foo tree(syn:tag, syn:tag:up)

# tree() up - select all parent fqdns of mx.somebox.woot.com inet:fqdn= mx.somebox.woot.com tree( inet:fqdn:domain, inet:fqdn )

Usage Notes: • The tree() operator acts as a recursive pivot. This allows a user to build a set of nodes which have self- referencing forms. For example, in the syn:tag form, the syn:tag:up ptype is a syn:tag, so we can recursively pivot on it. • The recurlim option may be set to limit the depth of the number of lookups performed by the tree() operator. This can be used to only grab a portion of a node tree. This value defaults to 20; and can be set to zero (recurlim=0) in order to disable this limit. • The tree() operator does consume all of the nodes present in the source query it uses to start pivoting from, and only returns the nodes from the resulting pivots. Operator Syntax Notes: • N/A Macro Syntax Notes:

60 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

• tree() has no Macro syntax implementation.

1.16 Storm Reference - “By” Handlers

Todo See the Storm Syntax Conventions for an explanation of the usage format used below. Where speciﬁc query examples are given, they are commonly provided in pairs using operator syntax followed by the equivalent macro syntax (if available).

1.17 Storm Reference - Statistical Operators

1.17.1 stat()

Todo

1.18 Storm Reference - Miscellaneous Operators

1.18.1 show:cols()

Todo

1.18.2 limit()

Todo

1.18.3 opts()

Todo

1.18.4 save()

Todo

1.16. Storm Reference - “By” Handlers 61 Synapse Documentation, Release 0.0.34

1.18.5 load()

Todo

1.18.6 clear()

Todo

1.18.7 nexttag()

Todo

1.18.8 dset()

Todo

1.19 Ingest - Introduction

The Synapse Ingest subsystem was designed to help users load data into the Synapse hypergraph (Cortex). The design principle of the ingest system was that users should be able to load data into a Cortex without needing to write code to do so. The Ingest system can also be used to parse data from structured data sources; for example, it can be used to parse data from web APIs and store the results in Cortex. Since Ingest is designed to be friendly to non-programmers, Ingest deﬁnitions are typically written in JSON. Writing an Ingest deﬁnition, either for a static set of data or for parsing data several times over, requires familiarity with the Synapse model. Documentation on the built in models can be found at Data Model. Additional modeling documentation can be found at the Synapse User Guide.

1.19.1 Follow Along

The examples shown here in the user guide can also be executed directly, so readers may follow along if they have a copy of the Synapse git repository checked out. These examples show running the ingest tool and querying a Cortex to see the results of running the examples.

1.20 Ingest - Commandline Ingest Tool

While the Ingest subsystem in Synapse lives at synapse.lib.ingest, most users may use the standalone ingest tool directly. This can be invoked with the following command: python -m synapse.tools.ingest . If invoked with the –help ﬂag, the following options are listed:

~/synapse$ python -m synapse.tools.ingest --help usage: ingest [-h] [--core CORE] [--progress] [--sync SYNC] [--save SAVE] [--debug] [--verbose] [files [files ...]]

Command line tool for ingesting data into a cortex positional arguments:

62 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

files JSON ingest definition files

optional arguments: -h, --help show this help message and exit --core CORE Cortex to use for ingest deconfliction --progress Print loading progress --sync SYNC Sync to an additional cortex --save SAVE Save cortex sync events to a file --debug Drop to interactive prompt to inspect cortex --verbose Show changes to local cortex incrementally

These options control what we are ingesting, where it is going, and various logging details. --core This specifies which Cortex to connect to and add the ingest data to. By default, this is a ram cortex (ram://), but could be any supported Cortex url or a Telepath url to a Cortex. --progress Display the progress of the ingest process every second. This expects no arguments. --sync This can be used to sync events from the Cortex specified in the --core option with a remote Cortex via a splice pump. See the Syncing Data section below for more details. --save This creates a savefile for changes made to the Cortex specified in the --core option. This can be used to replay events to another Cortex. --debug This drops the user into a cmdr session for the Cortex specified in the --core option after the ingest processing is complete. It accepts no arguments. --verbose This prints the nodes added to the Cortex specified in the --core option as the nodes are created in the Cortex. It accpts no arguments.

1.20.1 Source File Path Location

In many of the Ingest examples, the ingest data files themselves have been located in the docs/synapse/ examples/ directory of the Synapse git repository. They specify source files by name - such as ingest_structured_dnsa2.jsonl. The Ingest subsystem uses a helper (loadfile) which sets a “basedir” value where the ingest definition file resides. This basedir is where the full file path for source files made with, using os.path.join(). In other words, the path to the source file in a Ingest definition is relative to the path of the definition file loaded by the Ingest tool.

1.21 Ingest - Embed Directives

The simplest example is ingesting static data which is located in the ingest file itself. This is done via an “embed” directive. It allows us to embed nodes directly into a ingest file. We can also include secondary properties in the files. Here is a brief example showing an ingest containing two inet:fqdn nodes:

1.21. Ingest - Embed Directives 63 Synapse Documentation, Release 0.0.34

{ "embed":[ { "nodes":[ [ "inet:fqdn", [ "woot.com", "vertex.link" ] ] ] } ] }

The items in the “nodes” key are a list of two-value pairs. The ﬁrst item is the form we are creating. The second item is a list of objects that will be used to make the nodes. In this case, we simply have two inet:fqdn‘s listed. If we ingest this ﬁle, if would be the equivalent of either adding nodes via Storm (ask [inet:fqdn=vertex.link inet:fqdn=woot.com]) or via the Cortex formTufoByProp() API. We can use the ingest tool (located at synapse.tools.ingest) to ingest this into a Cortex:

~/synapse$ python -m synapse.tools.ingest --core sqlite:///ingest_examples.db --

˓→verbose docs/synapse/examples/ingest_embed1.json add: inet:fqdn=com :host = com :sfx = 1 :zone = 0 add: inet:fqdn=woot.com :domain = com :host = woot :sfx = 0 :zone = 1 add: inet:fqdn=link :host = link :sfx = 1 :zone = 0 add: inet:fqdn=vertex.link :domain = link :host = vertex :sfx = 0 :zone = 1 ingest took: 0.031163692474365234 sec

Then we can open up the Cortex and see that we have made those nodes:

~/synapse$ python -m synapse.cortex sqlite:///ingest_examples.db cli> ask --props inet:fqdn=vertex.link inet:fqdn = vertex.link :domain = link :host = vertex :sfx = False :zone = True (1 results)

Expanding on the previous example, we can add additional forms in the embed directive - we are not limited to just a single type of node. Here is an example showing the addition of two inet:netuser nodes - one with a single

64 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34 primary property, and one with multiple secondary properties:

{ "embed":[ { "nodes":[ [ "inet:fqdn", [ "woot.com", "vertex.link" ] ], [ "inet:netuser", [ [ "github.com/bobtheuser", { "props":{ "seen:min": "2014", "seen:max": "2018", "email": "[email protected]" } } ], "google.com/bobtheuser" ] ] ] } ] }

This adds the two inet:netuser nodes to our Cortex. We can run that with the following command to add the nodes to our example core:

~/synapse$ python -m synapse.tools.ingest --core sqlite:///ingest_examples.db --

˓→verbose docs/synapse/examples/ingest_embed2.json add: inet:netuser=github.com/bobtheuser :email = [email protected] :seen:max = 1514764800000 :seen:min = 1388534400000 :site = github.com :user = bobtheuser add: inet:user=bobtheuser add: inet:fqdn=github.com :domain = com :host = github :sfx = 0 :zone = 1 add: inet:[email protected] :fqdn = gmail.com :user = bobtheuser add: inet:fqdn=gmail.com :domain = com :host = gmail :sfx = 0 :zone = 1

1.21. Ingest - Embed Directives 65 Synapse Documentation, Release 0.0.34

add: inet:netuser=google.com/bobtheuser :site = google.com :user = bobtheuser add: inet:fqdn=google.com :domain = com :host = google :sfx = 0 :zone = 1 ingest took: 0.021549463272094727 sec

Since we are using verbose mode we can see the inet:netuser nodes were created; while the existing inet:fqdn nodes were not. The default behavior for creating new nodes is to also create nodes for secondary properties if they are also a node type. In the example above we saw the creation of the inet:email, inet:netuser and other nodes which were not explicitly defined in the ingest definition. We can confirm those via the cmdr interface as well:

~/synapse$ python -m synapse.cortex sqlite:///ingest_examples.db cli> ask inet:netuser inet:netuser = github.com/bobtheuser inet:netuser = google.com/bobtheuser (2 results) cli> ask --props inet:user refs() inet:user = bobtheuser inet:email = [email protected] :fqdn = gmail.com :user = bobtheuser inet:netuser = github.com/bobtheuser :email = [email protected] :seen:max = 2018/01/01 00:00:00.000 :seen:min = 2014/01/01 00:00:00.000 :site = github.com :user = bobtheuser inet:netuser = google.com/bobtheuser :site = google.com :user = bobtheuser (4 results)

Besides adding properties, we can also add Tags to the ingest ﬁles. An example below shows adding some tags to the nodes in the embed directive. These tags can apply to either the entire set of nodes in the embed directive (#story.bob) or to a single node (the one #src.commercial tag).

{ "embed":[ { "nodes":[ [ "inet:fqdn", [ [ "woot.com", { "tags":[ "story.bob.infrastructure", "src.osint" ] } ],

66 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

[ "vertex.link", { "tags":[ "story.bob.infrastructure", "src.commercial" ] } ] ] ], [ "inet:netuser", [ [ "github.com/bobtheuser", { "tags":[ "story.bob.accounts", "src.osint" ] } ], [ "google.com/bobtheuser", { "tags":[ "story.bob.accounts", "src.osint" ] } ] ] ] ], "tags":[ "story.bob" ] } ] }

We can then apply this ingest with the following command (output omitted - it is rather long):

~/synapse$ python -m synapse.tools.ingest --core sqlite:///ingest_examples.db --

˓→verbose docs/synapse/examples/ingest_embed3.json

Back in cmdr we can lift the nodes via the tags we just added:

~/synapse$ python -m synapse.cortex sqlite:///ingest_examples.db cli> ask #src.osint inet:netuser = github.com/bobtheuser #src.osint (added 2017/08/16 02:16:35.409) #story.bob.accounts (added 2017/08/16 02:16:35.409) inet:netuser = google.com/bobtheuser #src.osint (added 2017/08/16 02:16:35.409) #story.bob.accounts (added 2017/08/16 02:16:35.409) inet:fqdn = woot.com

1.21. Ingest - Embed Directives 67 Synapse Documentation, Release 0.0.34

#src.osint (added 2017/08/16 02:16:35.409) #story.bob.infrastructure (added 2017/08/16 02:16:35.409) (3 results) cli> ask #src.commercial inet:fqdn = vertex.link #src.commercial (added 2017/08/16 02:16:35.409) #story.bob.infrastructure (added 2017/08/16 02:16:35.409) (1 results) cli> ask #story.bob inet:netuser = github.com/bobtheuser #src.osint (added 2017/08/16 02:16:35.409) #story.bob.accounts (added 2017/08/16 02:16:35.409) inet:netuser = google.com/bobtheuser #src.osint (added 2017/08/16 02:16:35.409) #story.bob.accounts (added 2017/08/16 02:16:35.409) inet:fqdn = vertex.link #src.commercial (added 2017/08/16 02:16:35.409) #story.bob.infrastructure (added 2017/08/16 02:16:35.409) inet:fqdn = woot.com #src.osint (added 2017/08/16 02:16:35.409) #story.bob.infrastructure (added 2017/08/16 02:16:35.409) (4 results) cli> ask #story.bob.accounts inet:netuser = github.com/bobtheuser #src.osint (added 2017/08/16 02:16:35.409) #story.bob.accounts (added 2017/08/16 02:16:35.409) inet:netuser = google.com/bobtheuser #src.osint (added 2017/08/16 02:16:35.409) #story.bob.accounts (added 2017/08/16 02:16:35.409) (2 results) cli>

A complete example of this example embed ingest is shown below. While the previous three ingests demonstrated different parts of the ingest system, this is close to how the ingest ﬁle would look for longer term storage or for doing a one-time load of data into a Cortex.

{ "embed":[ { "nodes":[ [ "inet:fqdn", [ [ "woot.com", { "tags":[ "story.bob.infrastructure", "src.osint" ] } ], [ "vertex.link", { "tags":[ "story.bob.infrastructure", "src.commercial"

68 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

] } ] ] ], [ "inet:netuser", [ [ "github.com/bobtheuser", { "props":{ "seen:min": "2014", "seen:max": "2018", "email": "[email protected]" }, "tags":[ "story.bob.accounts", "src.osint" ] } ], [ "google.com/bobtheuser", { "tags":[ "story.bob.accounts", "src.osint" ] } ] ] ] ], "tags":[ "story.bob" ] } ] }

This can be found at the ﬁle path docs/synapse/examples/ingest_embed4.json and ingested like the previous examples were. However, since there is nothing new to add here, there will be no new nodes created as a result of ingesting it into sqlite:///ingest_examples.db.

1.22 Ingest - Parsing Lines

Parsing structured data is the bread and butter of the Ingest system, allowing us to extract data from various sources and insert them into a Cortex. Ths simplest form of structured data is where the structure is simply “parse one line out of a ﬁle at a time”. The example we’ll work with is a list of Top Level Domains (TLDs) in ﬁle like so:

# Version 2017012300, Last Updated Mon Jan 23 07:07:01 2017 UTC COM NET ORG

1.22. Ingest - Parsing Lines 69 Synapse Documentation, Release 0.0.34

NINJA

We can convert each line into a inet:fqdn node representing that TLD. The following ingest shows an example of how to do that:

{ "sources":[ [ "ingest_structured_tlds.txt", { "open":{ "encoding": "utf8", "format": "lines" }, "ingest":{ "forms":[ [ "inet:fqdn", { "props":{ "sfx":{ "value":1 } } } ] ] } } ] ] }

The structure of this ingest file differs from the previous example showing the “embed” directive. This uses the “sources” directive. This directive specifies a source file and a dictionary continaing “open” and “ingest” directives. The open directive is below and tells us how to open the file and how it is shaped:

"open":{ "encoding":"utf8", "format":"lines" },

It specifies the file encoding (utf-8) and the format of the file (lines). There are a few file formats which Synapse will natively parse; they are noted below. The formats are extensible at runtime as well, so an API user could register their own formats. By default, the lines starting with # are ignored as comment lines. The ingest directive is below and tells us how to process each line within the file:

"ingest":{ "forms":[ [ "inet:fqdn", { "props":{ "sfx":{ "value":1 } }

70 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

} ] ] }

This definition includes a “forms” directive. This instructs ingest on which types of nodes to make as it processes the lines of data. In this flat file example, each line of text is a single item. Without any other directive, that line of text is used as the primary property for creating the inet:fqdn node. The “props” dictionary specifies additional properties which will be added to the node; here we are setting the sfx property to equal 1. This ingest can be run via the ingest tool:

python-m synapse.tools.ingest--core sqlite:///ingest_examples.db docs/synapse/

˓→examples/ingest_structured_tlds1.json

After ingesting this, we can see the inet:fqdn nodes have been added to our Cortex:

~/synapse$ python -m synapse.cortex sqlite:///ingest_examples.db cli> ask --props inet:fqdn=ninja inet:fqdn = ninja :host = ninja :sfx = True :zone = False (1 results)

1.23 Ingest - Parsing CSV

Now that we’ve seen how to do an ingest of a simple ﬁle containing some lines of text, we can move onto more complicated structures. Comma Separated Value (CSV) support is provided via the built-in csv module. If we look at the ﬁle below we can see a few FQDN and IP address pairs:

domain,ipv4 vertex.link,127.0.0.1 vertex.ninja,1.2.3.4 vertex.ninja,127.0.0.1

When the CSV helper parses this ﬁle, it will return each line as though we were iterating over a csv.reader object. For the sake of demonstration we’ll ingest these in as as inet:dns:a records. We’ll use the following ingest for doing that:

{ "sources":[ [ "ingest_structured_dnsa1.csv", { "open":{ "encoding": "utf8", "format": "csv" }, "ingest":{ "forms":[ [ "inet:dns:a", { "template": "{{domain}}/{{ipv4}}"

1.23. Ingest - Parsing CSV 71 Synapse Documentation, Release 0.0.34

} ] ], "vars":[ [ "domain", { "path": "0" } ], [ "ipv4", { "path": "1" } ] ] } } ] ] }

This ingest adds a “vars” section, which denotes variables which are extracted out of the data as we iterate over the CSV file. The CSV data is just a list of values, and we denote which element of the list to associate with which variable. That is done with the “path” directive - it can be used to extract specific items out of the data we are iterating over. Since we do not have a single string we can use as the primary property to make a node, we’ve added the “template” directive to the “forms” section. This allows us to construct the primary property using a string template. The vars we extracted are substituted into the {{domain}} and {{ipv4}} fields using str.replace, after calling str() on the vars. This ingest can be run via the ingest tool:

python-m synapse.tools.ingest--verbose--core sqlite:///ingest_examples.db docs/

˓→synapse/examples/ingest_structured_dnsa1.json

After ingesting this, we can see the inet:dns:a nodes have been added to our Cortex:

~/synapse$ python -m synapse.cortex sqlite:///ingest_examples.db /cli> ask --props inet:dns:a limit(1) inet:dns:a = vertex.link/127.0.0.1 :fqdn = vertex.link :ipv4 = 127.0.0.1 (1 results)

1.24 Ingest - Parsing JSON

JSON is a very commonly encountered data format, since it is so portable and easy to use. JSON support is provided via the built-in json module. For this, we’ll look at a more complicated example - a blob of information about a social media account for a user at the made up site, socialnetwork.ninja. The data we want to ingest can be seen below:

{ "account_created": "20170502010203",

72 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

"last_login": "20170817010203", "user": "bobtheuser", "site": "socialnetwork.ninja", "user_friends":[ "alicethefriend", "mallorythespy" ], "user_associations":[ "goodguyinc", "sillyhackers" ], "recent_posts":[ { "text": "Just got back from the concert \n had a great time", "time": "20170817010203" }, { "text": "Just did hax with nine bit bytes @ defcon", "time": "20170801020304" }, { "text": "hallo!", "time": "20170502010603" } ] }

There are several nodes we can create here from this data. First, the site and user can be made into a inet:netuser form, and we can set some time-based properties there. We have some friendships which are noted, which we’ll consider to be bidirectional relationships, and which we can use the inet:follows node type to represent. Similarly, the user has some organizations they are a part of on the site which can be treated as inet:netmemb nodes. Lastly, there is some post information from the user which we can use to make inet:netpost nodes. We’ll look at each of these separately, then together as a single document.

"forms":[ [ "inet:netuser", { "template":"{{domain}}/{{user}}", "props":{ "seen:max":{ "var":"last_login" }, "signup":{ "var":"account_created" } } } ] ], "vars":[ [ "domain", { "path":"site" } ],

1.24. Ingest - Parsing JSON 73 Synapse Documentation, Release 0.0.34

[ "user", { "path":"user" } ], [ "account_created", { "path":"account_created" } ], [ "last_login", { "path":"last_login" } ] ],

The above deﬁnition shows the extraction of a few vars from the json object, using their names as the path. This sets the variables domain, user, account_created and last_login. Those are used to create the inet:netuser node for the user. The following sections require the “iter” directive, which is used to iterate over a set of data which is structured in a object. This allows us to handle nested data structures in a clean fashion. There is one important concept to consider when dealing with “iter” directives - variable scope. When an iter is encounted, the Ingest process enters into a new scope and the variables set in the parent section are available to the child scopes. When an iter is exhausted, it leaves the scope and the variables it set are no longer available. A simple iter, going over the user’s organizations, can be seen below:

"iters":[ [ "user_associations/*", { "forms":[ [ "inet:netmemb", { "template":"({{domain}}/{{user}},{{domain}}/{{org}})" } ] ], "vars":[ [ "org", { "path":0 } ] ] } ],

Since we are iterating over the list of strings, if we want to set that string as a variable we can do so with the path set to the value 0 (not "0" as in the CSV example). Since we’re in a child scope, we still have access the parent “vars”, so we can use those when making the inet:netmemb nodes. This is seen in the template string:

74 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

"template":"({{domain}}/{{user}},{{domain}}/{{org}})"

This makes the string form of the inet:netmemb comp type using the org var extracted during the current iter, and the domain and user vars from the parent. Likewise, the domain and user are reused again in the iter which is used to make the inet:follows nodes.

[ "user_friends/*", { "forms":[ [ "inet:follows", { "template":"({{domain}}/{{user}},{{domain}}/{{friend}})" } ], [ "inet:follows", { "template":"({{domain}}/{{friend}},{{domain}}/{{user}})" } ] ], "vars":[ [ "friend", { "path":0 } ] ] } ],

In the above example, we’re making two inet:follows nodes for each friend, under the assumption that the friend relationship of socialnetwork.ninja is bi-directional. This is important since the inet:follows node itself is a one-way relationship (think of a Twitter follow), not a a two-way relationship. Next, we need to iterate over the dictionary objects containing the users posts. That is seen below:

[ "recent_posts/*", { "forms":[ [ "inet:netpost", { "template":"( \"{{domain}}/{{user}}\",\"{{text}}\")", "props":{ "time":{ "var":"time" } } } ] ], "vars":[

1.24. Ingest - Parsing JSON 75 Synapse Documentation, Release 0.0.34

[ "text", { "path":"text" } ], [ "time", { "path":"time" } ] ] } ]

The big difference in this section is that, unlike the previous two, we are accessing variables like in the parent scope (using key names). We then create the inet:netpost in a similar manner to other sections, in order to make those nodes. Putting this all together gives us the following ingest document:

{ "sources":[ [ "ingest_structured_nested_data.json", { "open":{ "encoding": "utf8", "format": "json" }, "ingest":{ "forms":[ [ "inet:netuser", { "template": "{{domain}}/{{user}}", "props":{ "seen:max":{ "var": "last_login" }, "signup":{ "var": "account_created" } } } ] ], "vars":[ [ "domain", { "path": "site" } ], [ "user", {

76 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

"path": "user" } ], [ "account_created", { "path": "account_created" } ], [ "last_login", { "path": "last_login" } ] ], "iters":[ [ "user_associations/*", { "forms":[ [ "inet:netmemb", { "template": "({{domain}}/{{user}},{{domain}}/{{org}})" } ] ], "vars":[ [ "org", { "path":0 } ] ] } ], [ "user_friends/*", { "forms":[ [ "inet:follows", { "template": "({{domain}}/{{user}},{{domain}}/{{friend}})" } ], [ "inet:follows", { "template": "({{domain}}/{{friend}},{{domain}}/{{user}})" } ] ], "vars":[ [ "friend",

1.24. Ingest - Parsing JSON 77 Synapse Documentation, Release 0.0.34

{ "path":0 } ] ] } ], [ "recent_posts/*", { "forms":[ [ "inet:netpost", { "template": "(\"{{domain}}/{{user}}\",\"{{text}}\")", "props":{ "time":{ "var": "time" } } } ] ], "vars":[ [ "text", { "path": "text" } ], [ "time", { "path": "time" } ] ] } ] ] } } ] ] }

This ingest can be run via the ingest tool: python-m synapse.tools.ingest--verbose--core sqlite:///ingest_examples.db docs/

˓→synapse/examples/ingest_structured_nested_def.json

After ingesting this, we can see the various nodes have been added to our Cortex:

~/synapse$ python -m synapse.cortex sqlite:///ingest_examples.db cli> ask inet:netuser:site=socialnetwork.ninja inet:netuser = socialnetwork.ninja/alicethefriend inet:netuser = socialnetwork.ninja/bobtheuser inet:netuser = socialnetwork.ninja/mallorythespy

78 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

(3 results) cli> ask --props inet:netuser=socialnetwork.ninja/bobtheuser inet:netuser->

˓→inet:follows:follower show:cols(inet:follows:follower, inet:follows:followee) socialnetwork.ninja/bobtheuser socialnetwork.ninja/mallorythespy socialnetwork.ninja/bobtheuser socialnetwork.ninja/alicethefriend (2 results) cli> ask --props inet:netuser=socialnetwork.ninja/bobtheuser inet:netuser->

˓→inet:netpost:netuser inet:netpost = 0e89cf5db1fd8c426daa8b01e58cd2dd :netuser = socialnetwork.ninja/bobtheuser :netuser:site = socialnetwork.ninja :netuser:user = bobtheuser :text = hallo! :time = 2017/05/02 01:06:03.000 inet:netpost = 16a550ca28e7e62e0b47b482cc02b58e :netuser = socialnetwork.ninja/bobtheuser :netuser:site = socialnetwork.ninja :netuser:user = bobtheuser :text = Just got back from the concert had a great time :time = 2017/08/17 01:02:03.000 inet:netpost = e6b5d03f398a02548d9201efbdc58a06 :netuser = socialnetwork.ninja/bobtheuser :netuser:site = socialnetwork.ninja :netuser:user = bobtheuser :text = Just did hax with nine bit bytes @ defcon :time = 2017/08/01 02:03:04.000 (3 results)

When ingesting data in this format, it is common to also apply tags to a ingest deﬁnition. This can be useful for adding additional analytical data or categorization to the nodes as they are created in the Cortex. This can be done by adding a “tags” directive in line with other ingest directives. An example of modifying the above ingest to add a #src.socialnetwork tag can be seen highlighted below:

{ "sources":[ [ "ingest_structured_nested_data.json", { "open":{ "encoding": "utf8", "format": "json" }, "ingest":{ "tags":[ "src.socialnetwork" ], "forms":[ [ "inet:netuser", { "template": "{{domain}}/{{user}}", "props":{ "seen:max":{ "var": "last_login" }, "signup":{ "var": "account_created"

1.24. Ingest - Parsing JSON 79 Synapse Documentation, Release 0.0.34

} } } ] ], "vars":[ [ "domain", { "path": "site" } ], [ "user", { "path": "user" } ], [ "account_created", { "path": "account_created" } ], [ "last_login", { "path": "last_login" } ] ], "iters":[ [ "user_associations/*", { "forms":[ [ "inet:netmemb", { "template": "({{domain}}/{{user}},{{domain}}/{{org}})" } ] ], "vars":[ [ "org", { "path":0 } ] ] } ], [ "user_friends/*", { "forms":[ [

80 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

"inet:follows", { "template": "({{domain}}/{{user}},{{domain}}/{{friend}})" } ], [ "inet:follows", { "template": "({{domain}}/{{friend}},{{domain}}/{{user}})" } ] ], "vars":[ [ "friend", { "path":0 } ] ] } ], [ "recent_posts/*", { "forms":[ [ "inet:netpost", { "template": "(\"{{domain}}/{{user}}\",\"{{text}}\")", "props":{ "time":{ "var": "time" } } } ] ], "vars":[ [ "text", { "path": "text" } ], [ "time", { "path": "time" } ] ] } ] ] } } ]

1.24. Ingest - Parsing JSON 81 Synapse Documentation, Release 0.0.34

] }

This will add the #src.socialnetwork tag to all of the nodes created directly by the ingest process. This ingest can be run via the ingest tool:

python-m synapse.tools.ingest--verbose--core sqlite:///ingest_examples.db docs/

˓→synapse/examples/ingest_structured_nested_def2.json

Then we can lift nodes from this ingest via tags. The following example shows lifting all of the inet:netuser nodes made with this ingest:

~/synapse$ python -m synapse.cortex sqlite:///ingest_examples.db cli> ask inet:netuser*tag=src.socialnetwork inet:netuser = socialnetwork.ninja/bobtheuser #src.socialnetwork (added 2017/08/17 23:53:30.024) (1 results)

This result may seem odd - in the previous query ask inet:netuser:site=socialnetwork.ninja we had three inet:netuser accounts found; but ask inet:netuser*tag=src.socialnetwork only gave us one account. This is because we only created one inet:netuser account directly in the ingest deﬁnition, the other accounts were created as a result of creating the inet:netfollows nodes. If we wanted to be thorough, we could also create the inet:netuser accounts as well. See the highlighted section of the updated ingest below:

82 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

"path": "site" } ], [ "user", { "path": "user" } ], [ "account_created", { "path": "account_created" } ], [ "last_login", { "path": "last_login" } ] ], "iters":[ [ "user_associations/*", { "forms":[ [ "inet:netmemb", { "template": "({{domain}}/{{user}},{{domain}}/{{org}})" } ] ], "vars":[ [ "org", { "path":0 } ] ] } ], [ "user_friends/*", { "forms":[ [ "inet:netuser", { "template": "{{domain}}/{{friend}}" } ], [ "inet:follows", { "template": "({{domain}}/{{user}},{{domain}}/{{friend}})"

1.24. Ingest - Parsing JSON 83 Synapse Documentation, Release 0.0.34

} ], [ "inet:follows", { "template": "({{domain}}/{{friend}},{{domain}}/{{user}})" } ] ], "vars":[ [ "friend", { "path":0 } ] ] } ], [ "recent_posts/*", { "forms":[ [ "inet:netpost", { "template": "(\"{{domain}}/{{user}}\",\"{{text}}\")", "props":{ "time":{ "var": "time" } } } ] ], "vars":[ [ "text", { "path": "text" } ], [ "time", { "path": "time" } ] ] } ] ] } } ] ] }

84 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

This updated ingest will make the inet:netuser nodes for the friends and tag them prior to making the inet:netfollows nodes. It can be run via the ingest tool: python-m synapse.tools.ingest--verbose--core sqlite:///ingest_examples.db docs/

˓→synapse/examples/ingest_structured_nested_def3.json

Repeating our earlier Storm query gives us all three inet:netuser nodes:

~/synapse$ python -m synapse.cortex sqlite:///ingest_examples.db cli> ask inet:netuser*tag=src.socialnetwork inet:netuser = socialnetwork.ninja/alicethefriend #src.socialnetwork (added 2017/08/17 23:59:57.797) inet:netuser = socialnetwork.ninja/bobtheuser #src.socialnetwork (added 2017/08/17 23:53:30.024) inet:netuser = socialnetwork.ninja/mallorythespy #src.socialnetwork (added 2017/08/17 23:59:57.797) (3 results)

1.25 Ingest - Parsing JSONL

The “jsonl” format is a combination of the lines and JSON formats. It assumes the data it is processing is a multi-line document, and each line of the document is a single json blob which needs to be deserialized and ingested. A simple jsonl ﬁle is shown below:

[{"ipv4":"192.168.0.1","domain":"foobar.com"},{"ipv4":"192.168.0.100","domain":

˓→"w00t.foobar.com"}] [{"ipv4":"127.0.0.1","domain":"knight.net"}] [{"ipv4":"10.20.30.40","domain":"knight.net"},{"ipv4":"8.8.8.8","domain":

˓→"acmecorp.knight.net"}]

This is similar to the earlier CSV data; except each line may have multiple objects we want to consume, and we’ve got to treat the data as a list of JSON dictionaries. This will require directly iterating on the data we are consuming and creating the nodes from that. A wilcard * can be used to indicate to iterate directly over the object being processed. This ingest can be seen below:

{ "sources":[ [ "ingest_structured_dnsa2.jsonl", { "open":{ "encoding": "utf8", "format": "jsonl" }, "ingest":{ "iters":[ [ "*", { "tags":[ "src.jsonldemo" ], "forms":[ [ "inet:dns:a",

1.25. Ingest - Parsing JSONL 85 Synapse Documentation, Release 0.0.34

{ "template": "{{domain}}/{{ipv4}}" } ] ], "vars":[ [ "domain", { "path": "domain" } ], [ "ipv4", { "path": "ipv4" } ] ] } ] ] } } ] ] }

It can be run via the ingest tool: python-m synapse.tools.ingest--verbose--core sqlite:///ingest_examples.db docs/

˓→synapse/examples/ingest_structured_dnsa2.json

After ingesting this, we can see the various nodes have been added to our Cortex:

~/synapse$ python -m synapse.cortex sqlite:///ingest_examples.db cli> ask #src.jsonldemo inet:dns:a = acmecorp.knight.net/8.8.8.8 #src.jsonldemo (added 2017/08/18 00:36:21.414) inet:dns:a = foobar.com/192.168.0.1 #src.jsonldemo (added 2017/08/18 00:36:21.414) inet:dns:a = knight.net/10.20.30.40 #src.jsonldemo (added 2017/08/18 00:36:21.414) inet:dns:a = knight.net/127.0.0.1 #src.jsonldemo (added 2017/08/18 00:36:21.414) inet:dns:a = w00t.foobar.com/192.168.0.100 #src.jsonldemo (added 2017/08/18 00:36:21.414) (5 results)

1.26 Ingest - Parsing XML

Todo

86 Chapter 1. Synapse User Guide Synapse Documentation, Release 0.0.34

1.27 Ingest - file:bytes Nodes

Todo

1.28 Ingest - Sending Data to a Remote Cortex

It is easy to send data processed by an Ingest to a remote Cortex. This is done using the --sync option when invoking the ingest tool. An example can be seen below: python-m synapse.lib.ingest--sync tcp://some.place.com/core/path/too/ingestdef.json

The above invocation doesn’t specify a Cortex with the --core option - instead it runs the ingest into a ram:// backed Cortex by default. This can be useful so that the local ram Cortex can quickly deduplicate data from the ingest, and the resulting events will be sent up to the remote Cortex as splice events. The remote Cortex will then apply any changes it needs in order to add anything new (nodes, props, tags, etc) from the ingest processing. When syncing data to a remote Cortex, the are pros and cons between the different type of local Cortex’s to use. A ram cortex is very quick to process and deduplicate the ingest data; but it is recreated during each run. This results in data always being resent to the remote Cortex during each run. Using a persistent Cortex, such as a sqlite///, is slower locally, but prevents resending duplicate data up to the remote Cortex when when it is re-encountered.gitg

1.29 Ingest - Format Details

The following format options are built into the Ingest system by default. These common formats allow users to quickly get started with Ingest in order to start parsing content into a Cortex. These helpers have various options which can be used to control how they generate objects which are processed by the ingest deﬁnitions.

1.29.1 csv

Todo

1.29.2 xml

Todo

1.29.3 json

Todo

1.29.4 jsonl

Todo

1.29.5 lines

Todo

1.27. Ingest - file:bytes Nodes 87 Synapse Documentation, Release 0.0.34

1.30 Ingest - Conditional Expressions

Todo The following sections are still under development: • Storm Reference • Storm Tutorial Many of the concepts above are closely interrelated, and the outline above represents a reasonable effort at introducing concepts in a logical order. However, it is difﬁcult to fully understand the potential of Synapse and hypergraphs without grasping the power of the Storm query language to understand, manipulate, and annotate data. Similarly, it’s hard to understand the effectiveness of Storm without knowledge of the underlying data model. The outline above is our suggested order, but readers are encouraged to skip around, or revisit earlier sections after digesting later sections to better see how these topics are tied together.

88 Chapter 1. Synapse User Guide CHAPTER 2

Synapse DevOps Guide

This DevOps Guide is written by and for Synapse developers and operators. This is focused towards individuals who are looking to deploy Synapse (such as Cortex Hypergraphs or Axon blob stores); looking to use Synapse components in their own projects; or those looking to contribute back to the Synapse project. The DevOps Guide is a living document and will continue to be updated and expanded. The current sections are:

2.1 Cortex: Synapse Hypergraph

2.1.1 Introduction

Within Synapse, a Cortex is a “key-value based linkless hypergraph” implementation. A Cortex allows the creation and retrieval of nodes which adhere to a data model which is itself inspectable and represented within the Cortex. Each node consists of a Globally Unique Identiﬁer (GUID) and a group of properties, all of which are indexed to allow fast retrieval. The Cortex API has a distinct storage-layer separation which facilitates a modular approach to data persistence and indexing. While each storage implementation may have unique advantages, users of a given Cortex need not be aware of which type they are requesting information from. Additionally, a Cortex is the fundamental hypergraph storage behind a Swarm cluster which allows many Cortexes to be queried in parallel and the results to be intersected and fused as one.

2.1.2 Cortex Quick Start

A quick example of forming a RAM Cortex and creating a few nodes: import synapse.cortex as s_cortex core= s_cortex.openurl('ram://') tufo0= core.formTufoByProp('foo:bar','woot', baz=30) tufo1= core.formTufoByProp('foo:bar','haha', baz=10)

89 Synapse Documentation, Release 0.0.34

# Access a primary property from a tufo if tufo0[1].get('foo:bar') =='woot': print('woot')

# Access a secondary property from a tufo if tufo0[1].get('foo:bar:baz') == 30: print('woot')

# retrieve all foo:bar tufos tufs= core.getTufosByProp('foo:bar')

# retrieve all tufos where foo:bar:baz=30, but no more than 100 tufs= core.getTufosByProp('foo:bar:baz', valu=30, limit=100)

Various storage backings may be accessed with different URL schemes. To create a Cortex which is backed by an SQLite3 ﬁle: core= s_cortex.openurl('sqlite://myfile.db')

To create a cortex which is backed by a local PostgreSQL table: core= s_cortex.openurl('postgres:///mydbname/mytable')

To create a cortex which is backed by a remote PostgreSQL table: core= s_cortex.openurl('postgres://visi:[email protected]/mydbname/mytable')

2.1.3 Introducing the TuFo

A Cortex represents hypergraph nodes in a data structure we call a “tuple form” or tufo. A tufo data structure consists of an identiﬁer and a dictionary of properties:

(,{"prop":"valu",...})

Each tufo contains a single primary property frequently referred to as the “form” of the tufo. Additional secondary properties which are specific to the form (or type) of tufo are prefixed with the form to create a specific name space for each property. For example, if the model contained a form named “foo:bar”, the additional properties would all begin with “foo:bar:”:

(,{"foo:bar":"woot","foo:bar:len":4})

Universal TuFo Properties

Tufos created by a Cortex have several “universal” properties which are automatically populated during creation. For example, each tufo will contain a “tufo:form”= property which allows tufo consumers to know which property is primary and how to determine the preﬁx for secondary properties:

(,{"foo:bar":"woot","tufo:form":"foo:bar"})

TuFo Secondary Properties

Secondary properties for a given tufo will be preﬁxed with the form to create a unique name space for the secondary property. The data model for a Cortex may enforce that all secondary properties must adhere to speciﬁc type nor-

90 Chapter 2. Synapse DevOps Guide Synapse Documentation, Release 0.0.34 malization and may optionally include a default value if the property is not speciﬁed during formTufoByProp. For example, within the inet:fqdn form, the secondary property “sfx” will default to 0 if unspeciﬁed during formation: formTufoByProp('inet:fqdn','vertex.link')->(,{'inet:fqdn':'vertex.link',

˓→'inet:fqdn:sfx':0,...})

Secondary properties may be used to lift nodes from the Cortex using the getTufosByProp API which is discussed below.

TuFo Ephemeral Properties

It is occasionally useful for a tufo to carry properties which are ephemeral and do not represent data which is stored or indexed within the Cortex. These ephemeral properties are often used to represent run time knowledge or context and rarely have anything to do with the “truth” of the form. Ephemeral properties may be identiﬁed by beginning with the ”.” character. For example, the Cortex API formTufoByProp will retrieve an existing tufo or create a new instance if the requested = pair is not found. It may be useful for an application to know if the tufo was just created by the formTufoByProp call or already existed within the Cortex. To prevent API users from needing to call getTufoByProp to pre-test for existence, the formTufoByProp API adds the property ”.new”=1 when the tufo did not previously exist.:

(,{"foo:bar":"woot",".new":1})

Form a TuFo

The formTufoByProp API facilitates the creation of tufos and implements tufo deconﬂiction. The primary property of a tufo created using the formTufoByProp API is atomically deconﬂicted within the Cortex to ensure there is only ever one instance of that = pair. Additionally, calling formTufoByProp on an existing = pair will retrieve and return the already existing tufo if present. This allows a single atomic mechanism for the check-and- create primitive and provides a safe way to re-ingest data without worrying about creating duplicate representations of the same knowledge: tuf0= core.formTufoByProp('foo:bar','woot')

Additional secondary properties may be speciﬁed as kwargs to formTufoByProp: tuf0= core.formTufoByProp('foo:bar','woot', baz=30)

Tufos may also be created for non-deconflicted forms which represent a unique occurrence or observation in time. Tufos which are not meant to be deconflicted are declared with a GUID as their primary property and may be formed using the the value None. tuf0 = core.addTufoEvent(‘hurr:durr’, blah=30, gronk=’lol’) The addTufoEvent API handles the generation of a GUID for the tufo’s primary property and is able to insert the tufo without incurring the overhead of deconfliction.

Retrieving TuFos By Property

A list of tufos which have a given property and optionally a speciﬁc value may be retrieved from a Cortex using the getTufosByProp API: tufs= core.getTufosByProp('foo:bar:baz', valu=30)

2.1. Cortex: Synapse Hypergraph 91 Synapse Documentation, Release 0.0.34

2.1.4 Cortex Datamodels

A Cortex may optionally store and enforce a data model which declares types, forms, and properties. The model’s knowledge of types will be used to ensure that properties are correctly normalized. This knowledge is also used by the getTufosByPropType API to allow the retrieval of tufos which contain a property type without knowing about the form in advance. The data model is stored as tufos within the Cortex which allows them to be inspected like any other tufos:

# gather all the forms with a property of type "inet:fqdn" and print the form name. for tufo in core.getTufosByProp('syn:prop:ptype', valu='inet:fqdn'): prop= tufo[1].get('syn:prop') form= tufo[1].get('syn:prop:form') print('tufo form: %s has prop: %s'% ( form, prop )) syn:type

The syn:type tufos are used to declare all types which the Cortex is aware of. Each type must either be implemented by an existing python class or extend an existing type. syn:form

The syn:form tufos are used to declare each form which the Cortex data model is aware of. A syn:form tufo will contain knowledge about the type for the primary property. syn:prop

Every declared property for a given tufo form is represented via a syn:prop tufo which contains knowledge of both the parent form as well as type information.

2.1.5 Cortex Rows

A row level API exists within the Cortex API to allow access to the underlying storage abstraction for individual properties. These APIs are detailed in Cortex Storage Details.

2.1.6 Calculating Statistics

INPROG

2.1.7 Cortex Storage Details

The Cortex can be back by one of several different storage layers. Detail information about storage layers which are available for Cortex can be found here Cortex Storage Types. Additional information about the Storage layer API itself can be found here Cortex Storage Details.

92 Chapter 2. Synapse DevOps Guide Synapse Documentation, Release 0.0.34

2.2 Cortex: Built-In Storage Layers

There are multiple different storage layers available for use with the Cortex hypgeraph. The choice of storage layer may affect performance characteristics of Cortex. XXX Notes about open_* handlers in cortex.py XXX Notes about cortex ctor helper in each storage deﬁnition.

2.2.1 ram://

The ram:// storage implementation is implemented in put python data structures resulting in a mind bending level of performance. However, this storage backing is only appropriate for data which does not cause the Cortex to exceed the available amount of memory on the system. With the addition of the saveﬁle= URL parameter, a RAM cortex can persist to disk. However, for large or frequently changed data this saveﬁle can grow very large due to storing all changes since the beginning and may take a long time to start up and apply all changes before the Cortex comes online.

2.2.2 sqlite://

The SQLite3 storage implementation uses a single SQLite3 db ﬁle to store a Cortex. They are reasonably fast for medium sized data sets and very simple to create and manage.

2.2.3 postgres://

The PostgreSQL storage backing implements storage for a Cortex as a single table within a PostgreSQL database. While slower than a ram:// Cortex, a PostgreSQL core has the advantage of being backed by PSQL and having associated PSQL management tools available for administrators to use.

2.2.4 lmdb://

A cortex backed by the Symas Lightning DB (lmdb).

2.2.5 Cortex Storage Compatibility Notes

Due to issues with Python serialization, the data stored and accessed from a Cortex is not guaranteed across Cortexes created in Python 2.7 and Python 3.x. In short, if a Cortex or saveﬁle was created in Python 2.7; its use in a Python3 environment isn’t guaranteed. The inverse is also true; a Cortex created in 3.x may not work in Python2.7 as expected. This is known to affect the LMDB Cortex implementation, which heavily relies on using msgpack for doing key/value serialization, which has issues across python 2/3 with string handling. The Blob store APIs may also be affected by this, since the stored data is stored as a sgpack’d object. If there is a need for doing a data migration in order to ensure that your able to access a Cortex created on 2.7 with python 3.x, we have plans to provide a row level dump/backup tool in the near future that can be used to migrate data.

2.2. Cortex: Built-In Storage Layers 93 Synapse Documentation, Release 0.0.34

2.3 Cortex: Storage Layer Details

The storage layer for a Cortex is a standalone object which can be instantiated on its own. This allows the creation of empty Cortex’s, raw data manipulation or reuse of the Storage object for other purposes. In addition, the clean API separation between the Cortex and Storage classes allows for additional storage layer backings to be implemented.

2.3.1 What is a Storage Row

Most fundamentally, the Storage layer implements a way to store rows of data with simple prop/valu indexing. A row consists of four values: iden| prop| valu| time

These rows values are what we call a row from an API perspective. The field definitions are the following: • iden: This is a str value which acts as unique identifer for a given collection of related rows. A lift of all the rows with the same iden in a Storage layer is considered a join operation; and the resulting rows may be then converted into a tufo object by the Cortex or an external caller. By convention, the iden is typically a random, 16 byte guid value. An example is the following: “e2ac3afddab9394490d55f37a21f013d” (with lowercase characters). While it is possible to create and add rows with arbitrary iden values, tools and code made by the Vertex Project may not support them in all use cases. Suitable iden values can be made using the synapse.common.guid() function. • prop: This is a str value which represents a property. When rows are lifted together by a join operation and folder into a tufo, these become the keys of the tufo dictionary. • valu: This is either a str or int value which is associated with the prop. When rows are folded into a tufo, this valu becomes the value of the corresponding prop in the tufo dictionary. If the storage implementation allows for multiple values for the same prop the tufos may be inconsistent. The storage layer should enforce iden/prop/valu combination uniqueness. • time: This is a int containing the epoch timestamp in milliseconds. This should record when row was created. Suitable time values can be made using the synapse.common.time() function. Through synapse documentation, when row level API is referenced, it is referring to either adding or retrieving rows in this (iden, prop, valu, time) format. In many cases this is shortened to (i, p, v, t) in both documentation and code. It is possible for a storage implementation to store these rows across multiple columns, DBs or indices, as long as they adhere to the (i, p, v, t) format going in and coming out. For example the SQL based stores stores these rows in a five column table, with a separate columns for valus which are strings and valus which are integers.

2.3.2 Storage Layer APIs

The complete list of Storage layer APIs can be found at Cortex Storage API.

2.3.3 Implementing a Storage Layer

When implementing a new storage layer ﬁve things must be done. 1. The Storage class from synapse/cores/storage.py must be subclassed and multiple methods overridden. Some of these are private methods since they are wrapped by public APIs or called by other functions. 2. The StoreXact class from synapse/cores/xact.py must be subclassed and multiple methods overridden.

94 Chapter 2. Synapse DevOps Guide Synapse Documentation, Release 0.0.34

3. Additional Storage and StoreXact optional methods may be overridden to provided storage layer speciﬁc functionality for these objects which overrides default behaviors. This may be the case where a storage layer may have a more optimized way to do something, such as via a speciﬁc type of query or lift. 4. The Storage class should implement a helper function to allow creating a Cortex with the Storage implementation as the backing store. 5. The Cortex should pass the basic Cortex tests provided in the test_cortex.py test suite. Each of these is actions is detailed below.

Overriding Required Storage APIs

The following APIs must be implemented:

Initializtion Type APIs

• _initCoreStor(self): • getStoreType(self):

Transaction APIs

These are used to provide transaction safety around a Storage object. • getStoreXact(self, size=None):

Blob Storage APIs

• _getBlobValu(self, key): • _setBlobValu(self, key, valu): • _hasBlobValu(self, key): • _delBlobValu(self, key): • _getBlobKeys(self):

Adding Data to the Store

• _addRows(self, rows):

Deleting Data from the Store

• _delRowsById(self, iden): • _delRowsByProp(self, prop, valu=None, mintime=None, maxtime=None): • _delRowsByIdProp(self, iden, prop, valu=None):

2.3. Cortex: Storage Layer Details 95 Synapse Documentation, Release 0.0.34

Getting Data From the Store

• getRowsById(self, iden): • getRowsByProp(self, prop, valu=None, mintime=None, maxtime=None, limit=None): • getRowsByIdProp(self, iden, prop, valu=None): • getSizeByProp(self, prop, valu=None, mintime=None, maxtime=None): • rowsByRange(self, prop, valu, limit=None): • rowsByGe(self, prop, valu, limit=None): • rowsByLe(self, prop, valu, limit=None): • sizeByGe(self, prop, valu, limit=None): • sizeByLe(self, prop, valu, limit=None): • sizeByRange(self, prop, valu, limit=None): • joinsByGe(self, prop, valu, limit=None): • joinsByLe(self, prop, valu, limit=None): • _genStoreRows(self, **kwargs):

Override the StoreXact APIs

The following APIs must be overridden: • _coreXactBegin(self): • _coreXactCommit(self):

Optional Storage APIs to Override

Some of the APIs provided in the Storage and StoreXact classes provide default implementations which will generically work but may not be the best choice for a given storage layer.

Initializtion Type APIs

These allow for the the storage layer to close resources on teardown and allow it to do custom function/helper registration when a Cortex class is registered with a Storage object. • _ﬁniCoreStore(self): • _setSaveFd(self, fd, load=True, ﬁni=False):

Row Level APIs

These are row level APIs which may be overridden. • _setRowsByIdProp(self, iden, prop, valu): • _delJoinByProp(self, prop, valu=None, mintime=None, maxtime=None): • getJoinByProp(self, prop, valu=None, mintime=None, maxtime=None, limit=None):

96 Chapter 2. Synapse DevOps Guide Synapse Documentation, Release 0.0.34

• rowsByLt(self, prop, valu, limit=None): • rowsByGt(self, prop, valu, limit=None):

Join Level APIs

These APIs return rows which can be turned into complete tufos. They are broken out so that the Storage layer can provide optimized methods which may be quicker than the default implementations. These are expected to return lists of rows which the Cortex can turn into tufos as needed. • getRowsById(self, iden): • getRowsByIdens(self, idens): The default implementations of these functions are just wrappers for joinsByLe / joinsByGt, respectively. • joinsByLt(self, prop, valu, limit=None): • joinsByGt(self, prop, valu, limit=None):

Optional StorXact APIs

These APIs may be used to acquire/release resources needed for the transaction: • _coreXactAcquire(self): • _coreXactRelease(self): These APIs may be used to perform work during __enter__ and __exit__ calls: • _coreXactInit(self): • _coreXactFini(self):

Implementing a helper function

A helper function for making a Cortex with your storage layer should be provided. It should match the following call signature and return a Cortex class which uses your storage layer for backing. A simple example is seen below: def initMyStorageCortex(link, conf=None, storconf=None): ''' Initialize a MyStore based Cortex from a link tufo.

Args: link ((str, dict)): Link tufo. conf (dict): Configable opts for the Cortex object. storconf (dict): Configable opts for the storage object.

Returns: s_cores_common.Cortex: Cortex created from the link tufo. ''' if not conf: conf={} if not storconf: storconf={}

store= MyStorage(link, **storconf) return s_cores_common.Cortex(link, store, **conf)

2.3. Cortex: Storage Layer Details 97 Synapse Documentation, Release 0.0.34

Then, in synapse/cortex.py, a few changes need to be made. We have to import the ﬁle containing the Storage object implementation and the helper function, as well as updating a pair of dictionaries to register URL handlers for making either raw Storage objects or making a Cortex backed by the new Storage implementation. The storectors dictionary should contain the path of your Storage class implementation, and the corctors should contain the path to the helper function. Assuming the storage object was implemented in synaspe/cores/mystorage.py, these would look like the following: import synapse.cores.ram import synapse.cores.lmdb import synapse.cores.sqlite import synapse.cores.postgres import synapse.cores.mystorage

... storectors={ 'lmdb': synapse.cores.lmdb.LmdbStorage, 'sqlite': synapse.cores.sqlite.SqliteStorage, 'ram': synapse.cores.ram.RamStorage, 'postgres': synapse.cores.postgres.PsqlStorage, 'mystorage': synapse.cores.mystorage.MyStorage, } corctors={ 'lmdb': synapse.cores.lmdb.initLmdbCortex, 'sqlite': synapse.cores.sqlite.initSqliteCortex, 'ram': synapse.cores.ram.initRamCortex, 'postgres': synapse.cores.postgres.initPsqlCortex, 'mystorage': synapse.cores.mystorage.initMyStorageCortex, }

With these registered, users can easily make raw storage objects or Cortexs using the openstorage() and openurl() functions provided in synapse/cortex.py. Examples of that are below: import synapse.cortex as s_cortex stor= s_cortex.openstore('mystorage:///./some/path') # Now you have a raw Storage object available. # This may be useful for various tests or direct storage layer activity. core= s_cortex.openurl('mystorage:///./some/other/path') # Now you have a Cortex available which has the Hypergraph data model loaded in it so

˓→you actually # store nodes using prop normalization, join a swarm instance, ask queries via storm,

˓→etc.

Basic Cortex Test Suite

Adding a new storage layer implementation to the test suite is fairly straightforward. In the synapse/tests/test_cortex.py ﬁle, add the following test to the CortexBaseTest class (this assumes you registered the handler as “mystore”): def test_cortex_mystore(self): with s_cortex.openurl('mystore:///./store/path') as core: self.basic_core_expectations(core,'mystoretype')

Then you can run the Cortex tests using the following command to ensure your Cortex works properly:

98 Chapter 2. Synapse DevOps Guide Synapse Documentation, Release 0.0.34

python-m unittest synapse.tests.test_cortex.CortexBaseTest.test_cortex_mystore

2.4 Ingest: Structured Data Import

API / Developer level details for the Ingest Subsystem.

2.4.1 Ingesting Data via API

Todo

2.4.2 Ingesting Data from the Web

Todo

2.4.3 Registering Format Helpers

Todo

2.4.4 Ingest Under the Hood

Todo

2.5 Axon: Synapse Blob Storage

2.6 Dmon: Synapse Deployment purpose / deployment arch as data

2.4. Ingest: Structured Data Import 99 Synapse Documentation, Release 0.0.34

2.6.1 Dmon Quick Start

2.6.2 Basic Dmon Conﬁguration

2.6.3 Dmon Deployment

2.7 Eventbus: Synapse Event Distribution

2.7.1 EventBus Quick Start

2.7.2 EventBus Basics

2.7.3 Event Bus Linking

2.7.4 Async Event Bus Consumer

2.8 CoreModule: Model Deﬁnitions

2.8.1 Quick Start

This is intended as a reference for developers working on Synapse who are creating, or adding models to the base models or for developers who are looking to extend Synapse for their own use by the creation of new models. This is not intended as a ‘how-to’ guide for Synapse hypergraph modeling, rather a document for how models may be deﬁned and loaded into a Cortex.

2.8.2 Models As Code

The the core models used in Synapse hypergraph implementations are a part of the codebase which makes up Synapse. This allows the model definitions to grow along with the codebase over time. One feature we have in Synapse is the ability to track model revisions over time. By using the Cortex extensions class, CoreModule, we can implement our modules, and specific handlers for them, and migration steps over time in a single place. By placing some conventions around how the CoreModule is used for modeling, we can ensure consistency across the Cortex, DataModel and TypeLib classes; as well as ensuring that consistent data migrations are available to Cortex users. An example of a simple model definition file is the following import synapse.lib.module as s_module class FooBarModule(s_module.CoreModule):

# getBaseModels is a method designed to be overridden in # CoreModules which implement Synapse models. It must return # an iterable of name, model tuples. @staticmethod def getBaseModels(): modl={ 'types':( ('foo:bar',{'subof':'str','doc':'A foo bar!'}), ), 'forms':( ('foo:bar',

100 Chapter 2. Synapse DevOps Guide Synapse Documentation, Release 0.0.34

{'ptype':'foo:bar'}, []) ) } name='foobar' return ((name, modl), )

This example module inherits from the the Synapse CoreModule. The initial model is defined as a tuple of (name, dict) pairs; which are returned by the getBaseModels() function. One @staticmethod, getBaseModels, is defined which returns a iterable of name, model tuples. This serves two purposes: 1. The @staticmethod nature of the getBaseModels function allows for the data model to directly available. This allows the path to be loaded as a dynamic Synapse module; which is used by the TypeLib and DataModel classes. This way, those classes may always have the latest versions of the model available. 2. Upon loading the the CoreModule implementation into a Cortex, each name & model pair is considered as the ‘revision 0’ for that named model. This allows new Cortexes to always be created with the latest base model. On existing Cortexes which do not have nodes storing the model revision, the revision 0 of the model will be created and the base model automatically loaded. When model updates need to be done, these updates are to be delivered as functions in the CoreModule sublcass. These functions must be decorated with the @s_module.modelrev function. This decorator takes two arguments: 1. The name of the model to be updated. 2. The timestamp of the model revision, in integer form. This is validated using the “%Y%m%d%H%M” format string. These decorated functions should implement any changes neccesary for the data model; and perform any neccesary data migrations. For performance reasons, these migrations should be performed using the storage layer APIs. The use of node (tufo) level APIs when doing data migrations may result in terrible performance, causing migrations to take a significant amount of time. Under the hood, when the model is loaded by the Cortex, the rev0 function and any @modelrev decorated functions are sorted by model version, and these functions are then executed when CoreModule initialization is completed. This has the following effects: • On new Cortexes, the rev0 function will be loaded (adding the complete data model). Then subsequent functions will be executed. These should perform any model changes and data migrations as needed. • On existing Cortexes which have already had the base model loaded; the rev0 function will be skipped and subsequent functions executed until the last function has been executed. These functions should perform a model addition of the basemodel contents, and perform any required migration actions using the storage layer APIs. The following is a second revision of the earlier FooBarModule - it add a property for the foo:bar type, and performs a storage layer migration of nodes to set a default value for the new property. The model revision for the foobar model will be updated to 201707210101 in the Cortex.: import synapse.lib.module as s_module class FooBarModule(s_module.CoreModule):

@staticmethod def getBaseModels(): modl={ 'types':( ('foo:bar',{'subof':'str','doc':'A foo bar!'}), ), 'forms':(

2.8. CoreModule: Model Deﬁnitions 101 Synapse Documentation, Release 0.0.34

('foo:bar', {'ptype':'foo:bar'}, [('duck',{'defval':'mallard','ptype':'str','doc':'Duck type.'}

˓→)] ), ), } name='foobar' return ((name, modl), )

@s_module.modelrev('foobar', 201707210101) def _testRev1(self): ''' This revision adds the 'duck' property to our foo:bar nodes with its default

˓→value. ''' self.core.addPropDef('foo:bar:duck', form='foo:bar', defval='mallard', ptype=

˓→'str', doc='Duck value!') # Now lets migrate existing nodes to accommodate model changes. rows=[] tick= s_common.now() for iden, p, v, t in self.core.getRowsByProp('foo:bar'): rows.append((iden,'foo:bar:duck','mallard', tick)) self.core.addRows(rows)

It is highly encouraged for model developers to write unit tests for any migrations which they do, in order to ensure that their migration functions are working correctly.

2.8.3 Advanced CoreModule Usage

The CoreModule class can also be used to extend the functionality of the Cortex beyond simply adding additional model deﬁnitions. The CoreModule has access to the Cortex is loaded with, for example, we can use it to add additional event handlers; type casts; or other functionality. The Cortex on method (from eventbus.py) can be used to quickly strap in additional actions, and the CoreModule class itself has speciﬁc event helpers as well (with more coming soon). An example of extending the previous example is shown below (minus migration functions). import logging import synpase.eventbus as s_eventbus import synapse.lib.module as s_module logger= logging.getLogger(__name__) class FooBarModule(s_module.CoreModule):

# Override the default initCoreModule function def initCoreModule(self):

# Define a function used for helping out during node creation. self.onFormNode('foo:knight', self.onTufoFormKnight)

# Calling self.revCoreModl() is required by classes which override # initCoreModule and define module revisions. self.revCoreModl()

def onTufoFormKnight(self, form, valu, props, mesg): if valu in ['erec','lancelot','blumenthal']:

102 Chapter 2. Synapse DevOps Guide Synapse Documentation, Release 0.0.34

props['foo:knight:court']='round table'

# Use an eventhandler to do an action during the property set. @s_eventbus.on('node:prop:set', prop='foo:bar:duck') def onTufoSetDuck(self, mesg): newv= mesg[1].get('newv') for tufo in self.core.getTufosByProp('foo:bar:duck', newv): msg='Already seen duck {} on {}'.format(newv, tufo[1].get('foo:bar')) logger.info(msg)

@staticmethod def getBaseModels(): modl={ 'types':( ('foo:bar',{'subof':'str','doc':'A foo bar!'}), ('foo:knight',{'subof':'str','doc':'A knight!'} ), 'forms':( ('foo:bar', {'ptype':'foo:bar'}, [('duck',{'defval':'mallard','ptype':'str','doc':'Duck type.'}

˓→)] ), ('foo:knight', {'ptype':'foo:knight'}, [('court',{'ptype':'str','doc':'Knight court'})] ), ), } name='foobar' return ((name, modl), )

This example shows the overriding of the initCoreModule() function, which registers a single function as a helper during node creation, and calls the revCoreModl() to cache the model revision functions for model initalization use by the Cortex. The helper is used to set a secondary property based on the primary property of the node. In addition, the @s_eventbus.on decorator is used to perform any action when an event is ﬁred in the Cortex attached to the class. In the example, a message is logged; but other data could be retrieved, or looked up or modiﬁed; etc.

2.8.4 Core Synapse Model Conventions

The core Synapse modules are defined in the synapse/__init__.py file, in the BASE_MODELS list. This is a list of tuple values; containing the path to the CoreModule ctor, and the options. The base modules typically do not have options in them. New modules which contain new models should be added to the BASE_MODELs list. During the import process of Synapse, the python modules will be loaded and cached by the synapse.lib.modules.load_ctor() function. In addition, any ctors present in the environmental variable SYN_CORE_MODULES will also be loaded. The models contained in these ctors will be used to populate model information for instances of the TypeLib and DataModel classes, as well as serve as the CoreModules loaded into Cortexes upon creation. The convention for CoreModules which implement data models within the core Synapse codebase shall maintain a single CoreModule subclass per file, and this subclass will be responsible for maintaining a single named model.

2.8. CoreModule: Model Deﬁnitions 103 Synapse Documentation, Release 0.0.34

2.8.5 Gotchas

The following modeling gotchas exist: • The implementation of getBaseModels should be a @staticMethod, since it may be called directly by TypeLib or DataModel creation if the ctor has been loaded by synapse.lib.modules.load_ctor(). • It is possible for a single CoreModule to implement multiple named models, and revision them separately with @s_module.modelrev() decorators. The core Synapse modules will not be implemented in such a manner for the sake of simplicity in the codebase. • While it is possible for the model revision functions to simply add the base model data; it should really only do the changes neccesary to support the model changes. Currently, there are self.core.addTufoForm, self.core.addPropDef, and self.core.addType functions available for doing model additions. These functions may throw exceptions - see their docstrings for more information. We anticipate adding additional functions for doing removal of types, forms and props soon.

2.9 Storm: Synapse Query Language

Placeholder for future Storm documentation.

2.10 Telepath: Synapse Remote Method Invocation

2.10.1 Telepath Quick Start

2.10.2 Telepath Link URLs tcp:// ssl://

2.10.3 The Telepath Proxy

2.10.4 Telepath Protocol Modularity

2.10.5 Telepath Protocol Portability

2.11 Synapse Deployment Scenarios

2.11.1 Hardware

Currently Synapse may be run either natively or in a docker container. Both options will be covered. If performance is a consideration in testing please do not use a virtual machine and make sure the resources allocated are appropriate for the performance needed.

Experimental Hardware

For an experimental setup we recommend at least 8GB ram and several terabytes of disk space. Performance of the Synapse Hypergraph will drop considerably if the hardware resources are out paced by the data being loaded.

104 Chapter 2. Synapse DevOps Guide Synapse Documentation, Release 0.0.34

2.11.2 Install

Ubuntu 16.04 LTS is the recommended platform for installation. Installation via Docker is also supported. Synapse is available at the following places: 1. PyPi https://pypi.python.org/pypi/synapse 2. Github https://github.com/vertexproject/synapse 3. DockerHub https://hub.docker.com/r/vertexproject/synapse/

Ubuntu

Install the following prerequisites prior to using Synapse:

$ sudo apt update

$ sudo apt install -yq build-essential libffi-dev libssl-dev python3 python3-dev

˓→python3-pip python3-setuptools

$ sudo -H pip3 install --upgrade pip setuptools wheel

The following commands assume your Synapse checkout will be in ‘~/synapse’:

$ cd ~/ $ sudo apt install unzip wget $ wget https://github.com/vertexproject/synapse/archive/master.zip $ unzip master.zip $ mv synapse-master synapse $ cd synapse $ sudo python3 setup.py develop

An exemplar dmon conﬁguration ﬁle is located at synapse/docker/cortex/sqlite_dmon.json:

$ python3 -m synapse.tools.dmon synapse/docker/cortex/sqlite_dmon.json

Docker

Synapse docker images are also based on Ubuntu 16.04 and install all relevant dependencies. Unless otherwise stated Synapse tracks the latest stable release of Docker engine for Ubuntu 16.04 LTS. General steps: 1. Create base Synapse image 2. Create a Postgresql backed Cortex image 3. Start Synapse

Synapse image

This image is intended to serve 2 functions 1. Provide a simple sandbox to get started with synapse 2. Base image to facilitate building other synapse ecosystem images

2.11. Synapse Deployment Scenarios 105 Synapse Documentation, Release 0.0.34

• build

$ docker build -t vertexproject/synapse -f /synapse/docker/

˓→synapse_dockerfile

• volumes – /syndata is exposed by default • ports – no ports are exposed by default

Cortex image - Postgresql

This image will provide a synapse daemon config driven cortex backed into a postgres(9.5) database by default. It is the general image used for experimentation as it can also be easily configured to start additional Cortexes with alternate storage backings as well. By default each Cortex image is configured to expose a Cortex object. • build

$ docker build -t vertexproject/core_pg -f /synapse/docker/

˓→cortex/postgres_9.5_dockerfile

• volumes – /syndata – /var/lib/postgresql/data for postgres data • ports – 47322 - listener in the default /syndata/dmon.json – 5432 - for postgres • use – /syndata/dmon.json is the synapse dmon conf ﬁle used by the image. This can be modiﬁed or mapped in at container startup:

$ docker run vertexproject/core_pg

Start Docker Cortex

Start a container using the Posgresql Cortex image just created:

$ docker run vertexproject/core_pg

General Cortex Use

Connecting to a Cortex will be a variant of: import synapse.telepath as s_telepath host='172.17.0.2' port= 47322

106 Chapter 2. Synapse DevOps Guide Synapse Documentation, Release 0.0.34

core= s_telepath.openurl('tcp:///core', host=host, port=port)

At this point ‘core’ is a proxy object to the Cortex being shared by the Synapse daemon running in the Docker container. The normal Cortex apis can now be called:

# make sure proxy is working normally... # this should return *something* forms= core.getTufosByProp('syn:core')

# create an fqdn and store it fqdn='woot.com' new_tufo= core.formTufoByProp('fqdn', fqdn)

# retrieve the shiny new fqdn ret_tufo= core.getTufosByProp('fqdn', fqdn)[0] print('formed, stored and retrieved a form: %r'% (new_tufo[0] == ret_tufo[0],))

2.11.3 Other Cortex Docker images

The other Docker images listed below are simpler examples of running a more basic Cortex without Postgresql. core_ram

Provides a synapse daemon conﬁg driven cortex backed into ram. • build

$ docker build -t vertexproject/core_ram -f /synapse/docker/

˓→cortex/ram_dockerfile

• volumes – /syndata • ports – 47322 - listener in the default /syndata/dmon.json • use – /syndata/dmon.json is the synapse dmon conf ﬁle used by the image. This can be modiﬁed or mapped in at container startup

$ docker run vertexproject/core_ram core_sqlite

Provides a synapse daemon conﬁg driven cortex backed into a sqlite database by default. • build

2.11. Synapse Deployment Scenarios 107 Synapse Documentation, Release 0.0.34

$ docker build -t vertexproject/core_sqlite -f /synapse/docker/

˓→cortex/sqlite_dockerfile

$ docker run vertexproject/core_sqlite

The following sections are still under development: • Doc 1 • Doc 2

108 Chapter 2. Synapse DevOps Guide CHAPTER 3

Synapse Data Model

3.1 Types

3.1.1 ﬁle:base

A ﬁle or directory name (without a full path), such as system32 or foo.exe. Type Hierarchy: ﬁle:base

3.1.2 ﬁle:bytes

A unique file identifier Type Hierarchy: guid -> file:bytes

3.1.3 ﬁle:imgof

A file that contains an image of the specified node. Type Hierarchy: xref -> file:imgof

3.1.4 ﬁle:path

A file path that has been normalized by Synapse. Can consist of a directory path, a path and file name, or a file name. Type Hierarchy: file:path

109 Synapse Documentation, Release 0.0.34

3.1.5 ﬁle:rawpath

A raw file path in its default (non-normalized) form. Can consist of a directory path, a path and file name, or a file name. Type Hierarchy: file:rawpath

3.1.6 ﬁle:sub

A parent file that fully contains the specified child file. Type Hierarchy: sepr -> file:sub Sepr Fields: <file:bytes>/<file:bytes>

3.1.7 ﬁle:txtref

A file that contains a reference to the specified node. Type Hierarchy: xref -> file:txtref

3.1.8 geo:alias

An alias for the place GUID Type Hierarchy: str -> str:lwr -> geo:alias Examples: • repr mode: ‘foobar’ Type Constraints: • regex: ^[0-9a-z]+$ Type Transforms: • case: lower

3.1.9 geo:latlong

A Lat/Long string specifying a point Type Hierarchy: str -> geo:latlong Type Constraints: • regex: ^[-+]?([1-8]?d(.d+)?|90(.0+)?),s*[-+]?(180(.0+)?|((1[0-7]d)|([1-9]?d))(.d+)?)$ • null value: ??

3.1.10 geo:place

A GUID for a speciﬁc place Type Hierarchy: guid -> geo:place

110 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.1.11 gov:cn:icp

A Chinese Internet Content Provider ID Type Hierarchy: int -> gov:cn:icp

3.1.12 gov:cn:mucd

A Chinese PLA MUCD Type Hierarchy: int -> gov:cn:mucd

3.1.13 gov:cn:orgicp

A multi-ﬁeld composite type which uses separated repr values Type Hierarchy: sepr -> gov:cn:orgicp Sepr Fields: /

3.1.14 gov:intl:un:m49

UN M4 Numeric Country Code Type Hierarchy: int -> gov:intl:un:m49

3.1.15 gov:us:cage

A Commercial and Government Entity (CAGE) code Type Hierarchy: str -> gov:us:cage Type Transforms: • case: lower

3.1.16 gov:us:ssn

A US Social Security Number (SSN) Type Hierarchy: int -> gov:us:ssn

3.1.17 gov:us:zip

A US Zip Code Type Hierarchy: int -> gov:us:zip

3.1. Types 111 Synapse Documentation, Release 0.0.34

3.1.18 hash:md5

An MD5 hash Type Hierarchy: str -> hash:md5 Examples: • repr mode: ‘d41d8cd98f00b204e9800998ecf8427e’ Type Constraints: • regex: ^[0-9a-f]{32}$ Type Transforms: • case: lower

3.1.19 hash:sha1

A SHA1 hash Type Hierarchy: str -> hash:sha1 Examples: • repr mode: ‘da39a3ee5e6b4b0d3255bfef95601890afd80709’ Type Constraints: • regex: ^[0-9a-f]{40}$ Type Transforms: • case: lower

3.1.20 hash:sha256

A SHA256 hash Type Hierarchy: str -> hash:sha256 Examples: • repr mode: ‘ad9f4fe922b61e674a09530831759843b1880381de686a43460a76864ca0340c’ Type Constraints: • regex: ^[0-9a-f]{64}$ Type Transforms: • case: lower

3.1.21 hash:sha384

A SHA384 hash Type Hierarchy: str -> hash:sha384 Examples: • repr mode: ‘d425f1394e418ce01ed1579069a8bfaa1da8f32cf823982113ccbef531fa36bda9987f389c5af05b5e28035242efab6c’

112 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

Type Constraints: • regex: ^[0-9a-f]{96}$ Type Transforms: • case: lower

3.1.22 hash:sha512

A SHA512 hash Type Hierarchy: str -> hash:sha512 Examples: • repr mode: ‘ca74fe2ff2d03b29339ad7d08ba21d192077fece1715291c7b43c20c9136cd132788239189f3441a87eb23ce2660aa243f334295902c904b5520f6e80ab91f11’ Type Constraints: • regex: ^[0-9a-f]{128}$ Type Transforms: • case: lower

3.1.23 inet:addr

An IPv4 or IPv6 address Type Hierarchy: inet:addr Examples: • repr mode: ‘1.2.3.4’

3.1.24 inet:asn

An Autonomous System Number (ASN). Type Hierarchy: int -> inet:asn

3.1.25 inet:asnet4

An Autonomous System Number (ASN) and its associated IPv4 address range. Type Hierarchy: sepr -> inet:asnet4 Examples: • repr mode: ‘54959/1.2.3.4-1.2.3.20’ Sepr Fields: /

3.1. Types 113 Synapse Documentation, Release 0.0.34

3.1.26 inet:cidr4

An IPv4 address block in Classless Inter-Domain Routing (CIDR) notation. Type Hierarchy: inet:cidr4 Examples: • repr mode: ‘1.2.3.0/24’

3.1.27 inet:dns:a

The result of a DNS A record lookup. Type Hierarchy: sepr -> inet:dns:a Examples: • repr mode: ‘vertex.link/1.2.3.4’ Sepr Fields: /

3.1.28 inet:dns:aaaa

The result of a DNS AAAA record lookup. Type Hierarchy: sepr -> inet:dns:aaaa Examples: • repr mode: ‘vertex.link/2607:f8b0:4004:809::200e’ Sepr Fields: /

3.1.29 inet:dns:cname

The result of a DNS CNAME record lookup. Type Hierarchy: sepr -> inet:dns:cname Examples: • repr mode: ‘foo.vertex.link/vertex.link’ Sepr Fields: /

3.1.30 inet:dns:look

The instance (point-in-time) result of a DNS record lookup. Type Hierarchy: guid -> inet:dns:look

114 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.1.31 inet:dns:mx

The result of a DNS MX record lookup. Type Hierarchy: sepr -> inet:dns:mx Examples: • repr mode: ‘vertex.link/mail.vertex.link’ Sepr Fields: /

3.1.32 inet:dns:ns

The result of a DNS NS record lookup. Type Hierarchy: sepr -> inet:dns:ns Examples: • repr mode: ‘vertex.link/ns.dnshost.com’ Sepr Fields: /

3.1.33 inet:dns:req

A fused DNS request record Type Hierarchy: comp -> inet:dns:req

3.1.34 inet:dns:rev

The transformed result of a DNS PTR record lookup. Type Hierarchy: sepr -> inet:dns:rev Examples: • repr mode: ‘1.2.3.4/vertex.link’ Sepr Fields: /

3.1.35 inet:dns:rev6

The transformed result of a DNS PTR record for an IPv6 address. Type Hierarchy: sepr -> inet:dns:rev6 Examples: • repr mode: ‘2607:f8b0:4004:809::200e/vertex.link’ Sepr Fields: /

3.1.36 inet:dns:soa

The result of a DNS SOA record lookup. Type Hierarchy: comp -> inet:dns:soa

3.1. Types 115 Synapse Documentation, Release 0.0.34

3.1.37 inet:dns:txt

The result of a DNS TXT record lookup. Type Hierarchy: comp -> inet:dns:txt

3.1.38 inet:dns:type

A DNS request type enum Type Hierarchy: str -> inet:dns:type Type Transforms: • case: lower

3.1.39 inet:email

An email address. Type Hierarchy: inet:email Examples: • repr mode: ‘[email protected]‘

3.1.40 inet:ﬂow

An individual network connection between a given source and destination. Type Hierarchy: guid -> inet:ﬂow

3.1.41 inet:fqdn

A fully qualiﬁed domain name (FQDN). Type Hierarchy: inet:fqdn Examples: • repr mode: ‘vertex.link’

3.1.42 inet:iface

A network interface with a set of associated protocol addresses. Type Hierarchy: guid -> inet:iface

3.1.43 inet:ipv4

An IPv4 address. Type Hierarchy: inet:ipv4 Examples:

116 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

• repr mode: ‘1.2.3.4’

3.1.44 inet:ipv6

An IPv6 address. Type Hierarchy: inet:ipv6 Examples: • repr mode: ‘2607:f8b0:4004:809::200e’

3.1.45 inet:mac

A 48-bit Media Access Control (MAC) address. Type Hierarchy: str -> inet:mac Examples: • repr mode: ‘aa:bb:cc:dd:ee:ff’ Type Constraints: • regex: ^([0-9a-f]{2}[:]){5}([0-9a-f]{2})$ • null value: ?? Type Transforms: • case: lower

3.1.46 inet:net4

An IPv4 address range. Type Hierarchy: sepr -> inet:net4 Examples: • repr mode: ‘1.2.3.4-1.2.3.20’ Sepr Fields: -

3.1.47 inet:net6

An IPv6 address range. Type Hierarchy: sepr -> inet:net6 Examples: • repr mode: ‘ff::00-ff::30’ Sepr Fields: -

3.1. Types 117 Synapse Documentation, Release 0.0.34

3.1.48 inet:passwd

A password string. Type Hierarchy: str -> inet:passwd

3.1.49 inet:port

A network port. Type Hierarchy: int -> inet:port Examples: • repr mode: ‘80’ Type Constraints: • min value: 0 (0x0) • max value: 65535 (0xffff)

3.1.50 inet:srv4

An IPv4 address and port. Type Hierarchy: inet:srv4 Examples: • repr mode: ‘1.2.3.4:80’

3.1.51 inet:srv6

An IPv6 address and port. Type Hierarchy: inet:srv6 Examples: • repr mode: ‘[2607:f8b0:4004:809::200e]:80’

3.1.52 inet:ssl:tcp4cert

An SSL certificate file served by an IPv4 TCP server. Type Hierarchy: sepr -> inet:ssl:tcp4cert Sepr Fields: /<file:bytes>

3.1.53 inet:tcp4

A TCP server listening on an IPv4 address and port. Type Hierarchy: inet:srv4 -> inet:tcp4 Examples: • repr mode: ‘1.2.3.4:80’

118 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.1.54 inet:tcp6

A TCP server listening on a speciﬁc IPv6 address and port. Type Hierarchy: inet:srv6 -> inet:tcp6 Examples: • repr mode: ‘[2607:f8b0:4004:809::200e]:80’

3.1.55 inet:udp4

A UDP server listening on an IPv4 address and port. Type Hierarchy: inet:srv4 -> inet:udp4 Examples: • repr mode: ‘1.2.3.4:80’

3.1.56 inet:udp6

A UDP server listening on a speciﬁc IPv6 address and port. Type Hierarchy: inet:srv6 -> inet:udp6 Examples: • repr mode: ‘[2607:f8b0:4004:809::200e]:80’

3.1.57 inet:url

A Universal Resource Locator (URL). Type Hierarchy: inet:url Examples: • repr mode: ‘http://www.woot.com/ﬁles/index.html‘

3.1.58 inet:urlﬁle

A file hosted at a specific Universal Resource Locator (URL). Type Hierarchy: comp -> inet:urlfile

3.1.59 inet:urlredir

A URL that redirects to another URL, such as via a URL shortening service or an HTTP 302 response. Type Hierarchy: comp -> inet:urlredir Examples: • repr mode: ‘(http://foo.com/,http://bar.com/)’

3.1. Types 119 Synapse Documentation, Release 0.0.34

3.1.60 inet:user

A username string. Type Hierarchy: str -> str:lwr -> inet:user Type Transforms: • case: lower

3.1.61 inet:web:acct

An account with a given Internet-based site or service. Type Hierarchy: sepr -> inet:web:acct Examples: • repr mode: ‘twitter.com/invisig0th’ Sepr Fields: /

3.1.62 inet:web:action

An instance of an account performing an action at an Internet-based site or service. Type Hierarchy: guid -> inet:web:action

3.1.63 inet:web:actref

A web action that references a given node. Type Hierarchy: xref -> inet:web:actref

3.1.64 inet:web:chproﬁle

A change to a web account. Used to capture historical properties associated with an account, as opposed to current data in the inet:web:acct node. Type Hierarchy: guid -> inet:web:chproﬁle

3.1.65 inet:web:ﬁle

A ﬁle posted by a web account. Type Hierarchy: comp -> inet:web:ﬁle

3.1.66 inet:web:follows

A web account follows or is connected to another web account. Type Hierarchy: comp -> inet:web:follows

120 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.1.67 inet:web:group

A group hosted within or registered with a given Internet-based site or service. Type Hierarchy: sepr -> inet:web:group Examples: • repr mode: ‘somesite.com/mycoolgroup’ Sepr Fields: /

3.1.68 inet:web:logon

An instance of an account authenticating to an Internet-based site or service. Type Hierarchy: guid -> inet:web:logon

3.1.69 inet:web:memb

A web account that is a member of a web group. Type Hierarchy: comp -> inet:web:memb

3.1.70 inet:web:mesg

A message sent from one web account to another web account. Type Hierarchy: comp -> inet:web:mesg Examples: • repr mode: ‘twitter.com/invisig0th|twitter.com/gobbles|20041012130220’

3.1.71 inet:web:post

A post made by a web account. Type Hierarchy: comp -> inet:web:post

3.1.72 inet:web:postref

A web post that references a given node. Type Hierarchy: xref -> inet:web:postref

3.1.73 inet:whois:contact

An individual contact from a domain whois record. Type Hierarchy: comp -> inet:whois:contact

3.1. Types 121 Synapse Documentation, Release 0.0.34

3.1.74 inet:whois:rar

A domain registrar. Type Hierarchy: str -> str:lwr -> inet:whois:rar Examples: • repr mode: ‘godaddy, inc.’ Type Transforms: • case: lower

3.1.75 inet:whois:rec

A domain whois record Type Hierarchy: sepr -> inet:whois:rec Sepr Fields: @

3.1.76 inet:whois:recns

A nameserver associated with a domain whois record. Type Hierarchy: comp -> inet:whois:recns

3.1.77 inet:whois:reg

A domain registrant. Type Hierarchy: str -> str:lwr -> inet:whois:reg Examples: • repr mode: ‘woot hostmaster’ Type Transforms: • case: lower

3.1.78 inet:whois:regmail

An association between a domain and a registrant email address. Type Hierarchy: comp -> inet:whois:regmail

3.1.79 inet:wiﬁ:ssid

A WiFi service set identiﬁer (SSID) name. Type Hierarchy: str -> inet:wiﬁ:ssid Examples: • repr mode: ‘The Vertex Project’

122 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.1.80 it:av:ﬁlehit

A file that triggered an alert on a specific antivirus signature. Type Hierarchy: sepr -> it:av:filehit Sepr Fields: <file:bytes>/

3.1.81 it:av:sig

A vendor- or organization-speciﬁc antivirus signature name. Type Hierarchy: sepr -> it:av:sig Sepr Fields: /

3.1.82 it:dev:int

A developer-selected integer constant. Type Hierarchy: int -> it:dev:int

3.1.83 it:dev:mutex

A string representing a mutex. Type Hierarchy: str -> it:dev:str -> it:dev:mutex

3.1.84 it:dev:pipe

A string representing a named pipe. Type Hierarchy: str -> it:dev:str -> it:dev:pipe

3.1.85 it:dev:regkey

A Windows registry key. Type Hierarchy: str -> it:dev:str -> it:dev:regkey Examples: • repr mode: ‘HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run’

3.1.86 it:dev:regval

A Windows registry key/value pair. Type Hierarchy: comp -> it:dev:regval

3.1. Types 123 Synapse Documentation, Release 0.0.34

3.1.87 it:dev:str

A developer-selected string. Type Hierarchy: str -> it:dev:str

3.1.88 it:exec:proc

A process executing on a host. May be an actual (e.g., endpoint) or virtual (e.g., malware sandbox) host. Type Hierarchy: guid -> it:exec:proc

3.1.89 it:host

A GUID that represents a host or system. Type Hierarchy: guid -> it:host

3.1.90 it:hostname

The name of a host or system. Type Hierarchy: str -> str:lwr -> it:hostname Type Transforms: • case: lower

3.1.91 it:hostsoft

A version of a software product which is present on a given host. Type Hierarchy: comp -> it:hostsoft

3.1.92 it:hosturl

A URL hosted on or served by a host or system. Type Hierarchy: comp -> it:hosturl

3.1.93 it:prod:soft

A arbitrary, unversioned software product. Type Hierarchy: guid -> it:prod:soft

3.1.94 it:prod:softver

A version of a particular software product. Type Hierarchy: guid -> it:prod:softver

124 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.1.95 it:sec:cve

A vulnerability as designated by a Common Vulnerabilities and Exposures (CVE) number. Type Hierarchy: str -> str:lwr -> it:sec:cve Examples: • repr mode: ‘CVE-2012-0158’ Type Constraints: • regex: (?i)^CVE-[0-9]{4}-[0-9]{4,}$ Type Transforms: • case: lower

3.1.96 it:semver

Semantic Version type. Type Hierarchy: it:semver

3.1.97 lang:idiom

A subcultural idiom Type Hierarchy: str -> str:txt -> lang:idiom

3.1.98 lang:trans

Raw text with a documented translation Type Hierarchy: str -> str:txt -> lang:trans

3.1.99 mat:item

A GUID assigned to a material object Type Hierarchy: guid -> mat:item

3.1.100 mat:itemimage

A multi-ﬁeld composite type which uses separated repr values Type Hierarchy: sepr -> mat:itemimage Sepr Fields: /<ﬁle:bytes>

3.1.101 mat:spec

A GUID assigned to a material speciﬁcation Type Hierarchy: guid -> mat:spec

3.1. Types 125 Synapse Documentation, Release 0.0.34

3.1.102 mat:specimage

A multi-ﬁeld composite type which uses separated repr values Type Hierarchy: sepr -> mat:specimage Sepr Fields: /<ﬁle:bytes>

3.1.103 media:news

A GUID for a news article or report Type Hierarchy: guid -> media:news

3.1.104 ou:alias

An alias for the org GUID Type Hierarchy: str -> str:lwr -> ou:alias Examples: • repr mode: ‘vertexproj’ Type Constraints: • regex: ^[0-9a-z]+$ Type Transforms: • case: lower

3.1.105 ou:hasemail

A multi-ﬁeld composite type which generates a stable guid from normalized ﬁelds Type Hierarchy: comp -> ou:hasemail

3.1.106 ou:hasﬁle

A multi-field composite type which generates a stable guid from normalized fields Type Hierarchy: comp -> ou:hasfile

3.1.107 ou:hasfqdn

A multi-ﬁeld composite type which generates a stable guid from normalized ﬁelds Type Hierarchy: comp -> ou:hasfqdn

3.1.108 ou:hashost

A multi-ﬁeld composite type which generates a stable guid from normalized ﬁelds Type Hierarchy: comp -> ou:hashost

126 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.1.109 ou:hasipv4

A multi-ﬁeld composite type which generates a stable guid from normalized ﬁelds Type Hierarchy: comp -> ou:hasipv4

3.1.110 ou:hasphone

A multi-ﬁeld composite type which generates a stable guid from normalized ﬁelds Type Hierarchy: comp -> ou:hasphone

3.1.111 ou:haswebacct

A multi-ﬁeld composite type which generates a stable guid from normalized ﬁelds Type Hierarchy: comp -> ou:haswebacct

3.1.112 ou:naics

North American Industry Classiﬁcation System Type Hierarchy: int -> ou:naics

3.1.113 ou:name

The base string type Type Hierarchy: str -> str:lwr -> ou:name Type Transforms: • case: lower

3.1.114 ou:org

A GUID for a human organization such as a company or military unit Type Hierarchy: guid -> ou:org

3.1.115 ou:sic

Standard Industrial Classiﬁcation Code Type Hierarchy: int -> ou:sic

3.1.116 ou:suborg

An org which owns a sub org Type Hierarchy: comp -> ou:suborg

3.1. Types 127 Synapse Documentation, Release 0.0.34

3.1.117 ou:user

A user name within an organization Type Hierarchy: sepr -> ou:user Sepr Fields: /

3.1.118 pol:country

A GUID for a country Type Hierarchy: guid -> pol:country

3.1.119 pol:iso2

The 2 digit ISO country code Type Hierarchy: str -> pol:iso2 Examples: • repr mode: ‘us’ Type Constraints: • regex: ^[a-z0-9]{2}$ • null value: ?? Type Transforms: • case: lower

3.1.120 pol:iso3

The 3 digit ISO country code Type Hierarchy: str -> pol:iso3 Examples: • repr mode: ‘usa’ Type Constraints: • regex: ^[a-z0-9]{3}$ • null value: ?? Type Transforms: • case: lower

3.1.121 pol:isonum

The ISO integer country code Type Hierarchy: int -> pol:isonum Examples:

128 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

• repr mode: ‘840’

3.1.122 ps:contact

A GUID for a contact info record Type Hierarchy: guid -> ps:contact

3.1.123 ps:hasalias

A multi-ﬁeld composite type which uses separated repr values Type Hierarchy: sepr -> ps:hasalias Sepr Fields: /

3.1.124 ps:hasemail

A multi-ﬁeld composite type which uses separated repr values Type Hierarchy: sepr -> ps:hasemail Sepr Fields: /

3.1.125 ps:hashost

A multi-ﬁeld composite type which uses separated repr values Type Hierarchy: sepr -> ps:hashost Sepr Fields: /

3.1.126 ps:hasphone

A multi-ﬁeld composite type which uses separated repr values Type Hierarchy: sepr -> ps:hasphone Sepr Fields: /

3.1.127 ps:hasuser

A multi-ﬁeld composite type which uses separated repr values Type Hierarchy: sepr -> ps:hasuser Sepr Fields: /

3.1.128 ps:haswebacct

A multi-ﬁeld composite type which uses separated repr values Type Hierarchy: sepr -> ps:haswebacct Sepr Fields: /

3.1. Types 129 Synapse Documentation, Release 0.0.34

3.1.129 ps:image

A multi-ﬁeld composite type which uses separated repr values Type Hierarchy: sepr -> ps:image Sepr Fields: /<ﬁle:bytes>

3.1.130 ps:name

A last,ﬁrst person full name Type Hierarchy: ps:name Examples: • repr mode: ‘smith,bob’

3.1.131 ps:person

A GUID for a person or suspected person Type Hierarchy: guid -> ps:person

3.1.132 ps:tokn

A single name element (potentially given or sur) Type Hierarchy: str -> str:lwr -> ps:tokn Examples: • repr mode: ‘mike’ Type Transforms: • case: lower

3.1.133 rsa:key

An RSA keypair modulus and public exponent Type Hierarchy: sepr -> rsa:key Sepr Fields: /

3.1.134 syn:auth:role

The base string type Type Hierarchy: str -> syn:auth:role

3.1.135 syn:auth:user

The base string type Type Hierarchy: str -> syn:auth:user

130 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.1.136 syn:auth:userrole

A multi-ﬁeld composite type which generates a stable guid from normalized ﬁelds Type Hierarchy: comp -> syn:auth:userrole

3.1.137 syn:splice

A Globally Unique Identiﬁer type Type Hierarchy: guid -> syn:splice

3.1.138 syn:tagform

A multi-ﬁeld composite type which generates a stable guid from normalized ﬁelds Type Hierarchy: comp -> syn:tagform Examples: • repr mode: ‘(foo.bar,baz:faz)’

3.1.139 tel:mob:imei

An International Mobile Equipment Id Type Hierarchy: tel:mob:imei

3.1.140 tel:mob:imsi

An International Mobile Subscriber Id Type Hierarchy: tel:mob:imsi

3.1.141 tel:mob:tac

A mobile Type Allocation Code Type Hierarchy: int -> tel:mob:tac

3.1.142 tel:phone

Type Hierarchy: tel:phone

3.1.143 time:epoch

Timestamp in seconds since epoch (deprecated) Type Hierarchy: time:epoch Examples: • repr mode: ‘20161216084632’

3.1. Types 131 Synapse Documentation, Release 0.0.34

3.1.144 time:epoch:max

Maximum time in seconds (depricated) Type Hierarchy: time:epoch -> time:epoch:max Examples: • repr mode: ‘20161216084632’

3.1.145 time:epoch:min

Minimum time in seconds (depricated) Type Hierarchy: time:epoch -> time:epoch:min Examples: • repr mode: ‘20161216084632’

3.1.146 time:max

Maximum time in millis since epoch Type Hierarchy: time -> time:max Examples: • repr mode: ‘20161216084632’

3.1.147 time:min

Minimum time in millis since epoch Type Hierarchy: time -> time:min Examples: • repr mode: ‘20161216084632’

3.2 Forms

3.2.1 ﬁle:base = <ﬁle:base>

A file or directory name (without a full path), such as system32 or foo.exe. Properties: • file:base=<file:base> – A file or directory name (without a full path), such as system32 or foo.exe.

132 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.2 ﬁle:bytes = <ﬁle:bytes>

A unique file identifier Properties: • file:bytes=<file:bytes> – A unique file identifier • file:bytes:md5 = – The md5 hash of the file. • file:bytes:mime = (default: ‘??’) – The MIME type of the file. • file:bytes:mime:* = – Namespace for high-value mime details • file:bytes:mime:pe:compiled = – The compile time of the file according to the PE header. • file:bytes:mime:pe:imphash = – The PE import hash of the file as calculated by Vivisect; this method excludes imports referenced as ordinals and may fail to calculate an import hash for files that use ordinals. • file:bytes:mime:pe:size = – The size of the executable file according to the file headers. • file:bytes:mime:x509:cn = – The Common Name (CN) attribute of the x509 Subject. • file:bytes:name=<file:base> – The name of the file. Because a given set of bytes can have any number of arbitrary names, this field is used for display purposes only. • file:bytes:sha1 = – The sha1 hash of the file. • file:bytes:sha256 = – The sha256 hash of the file. • file:bytes:sha512 = – The sha512 hash of the file. • file:bytes:size = – The size of the file in bytes.

3.2.3 ﬁle:imgof = <ﬁle:imgof>

A file that contains an image of the specified node. Properties: • file:imgof=<file:imgof>

3.2. Forms 133 Synapse Documentation, Release 0.0.34

– A file that contains an image of the specified node. • file:imgof:file=<file:bytes> – The guid of the file containing the image. • file:imgof:xref = – The form=valu of the object referenced in the image, e.g., geo:place=. • file:imgof:xref:intval = – The value of the property of the referenced object, as specified by the propvalu, if the value is an integer. • file:imgof:xref:prop = – The property (form) of the referenced object, as specified by the propvalu. • file:imgof:xref:strval = – The value of the property of the referenced object, as specified by the propvalu, if the value is a string.

3.2.4 ﬁle:path = <ﬁle:path>

A file path that has been normalized by Synapse. Can consist of a directory path, a path and file name, or a file name. Properties: • file:path=<file:path> – A file path that has been normalized by Synapse. Can consist of a directory path, a path and file name, or a file name. • file:path:base=<file:base> – The final component of the file path. Can be a file name (if present) or the final directory. • file:path:dir=<file:path> – The parent directory of the file path. • file:path:ext = – The file extension of the file name, (if present); for example: exe, bat, py, docx.

3.2.5 file:subfile = <file:sub>

A parent file that fully contains the specified child file. Properties: • file:subfile=<file:sub> – A parent file that fully contains the specified child file. • file:subfile:child=<file:bytes> – The guid of the child file. • file:subfile:name=<file:base> – The name of the child file. Because a given set of bytes can have any number of arbitrary names, this field is used for display purposes only. • file:subfile:parent=<file:bytes>

134 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

– The guid of the parent ﬁle.

3.2.6 ﬁle:txtref = <ﬁle:txtref>

A file that contains a reference to the specified node. Properties: • file:txtref=<file:txtref> – A file that contains a reference to the specified node. • file:txtref:file=<file:bytes> – The guid of the file containing the reference. • file:txtref:xref = – The form=valu of the object referenced in the file, e.g., inet:fqdn=foo.com. • file:txtref:xref:intval = – The value of the property of the referenced object, as specified by the propvalu, if the value is an integer. • file:txtref:xref:prop = – The property (form) of the referenced object, as specified by the propvalu. • file:txtref:xref:strval = – The value of the property of the referenced object, as specified by the propvalu, if the value is a string.

3.2.7 geo:place =

A GUID for a speciﬁc place Properties: • geo:place = – A GUID for a speciﬁc place • geo:place:alias = – An alias for the place GUID • geo:place:latlong = (default: ‘??’) – The location of the place • geo:place:name = – The name of the place

3.2.8 gov:cn:icp =

A Chinese Internet Content Provider ID Properties: • gov:cn:icp = – A Chinese Internet Content Provider ID

3.2. Forms 135 Synapse Documentation, Release 0.0.34

3.2.9 gov:cn:mucd =

A Chinese PLA MUCD Properties: • gov:cn:mucd = – A Chinese PLA MUCD

3.2.10 gov:us:cage =

A Commercial and Government Entity (CAGE) code Properties: • gov:us:cage = – A Commercial and Government Entity (CAGE) code • gov:us:cage:cc = – The 2 digit ISO country code • gov:us:cage:city = – The base string type • gov:us:cage:country = – The base string type • gov:us:cage:name0 = – The name of the organization • gov:us:cage:name1 = – Name Part 1 • gov:us:cage:phone0 = • gov:us:cage:phone1 = • gov:us:cage:state = – The base string type • gov:us:cage:street = – The base string type • gov:us:cage:zip = – A US Zip Code

3.2.11 gov:us:ssn =

A US Social Security Number (SSN) Properties: • gov:us:ssn = – A US Social Security Number (SSN)

136 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.12 gov:us:zip =

A US Zip Code Properties: • gov:us:zip = – A US Zip Code

3.2.13 hash:md5 =

An MD5 hash Properties: • hash:md5 = – An MD5 hash

3.2.14 hash:sha1 =

A SHA1 hash Properties: • hash:sha1 = – A SHA1 hash

3.2.15 hash:sha256 =

A SHA256 hash Properties: • hash:sha256 = – A SHA256 hash

3.2.16 hash:sha384 =

A SHA384 hash Properties: • hash:sha384 = – A SHA384 hash

3.2.17 hash:sha512 =

A SHA512 hash Properties: • hash:sha512 = – A SHA512 hash

3.2. Forms 137 Synapse Documentation, Release 0.0.34

3.2.18 inet:asn =

An Autonomous System Number (ASN). Properties: • inet:asn = – An Autonomous System Number (ASN). • inet:asn:name = (default: ‘??’) – The name of the organization currently responsible for the ASN. • inet:asn:owner = – The guid of the organization currently responsible for the ASN.

3.2.19 inet:asnet4 =

An Autonomous System Number (ASN) and its associated IPv4 address range. Properties: • inet:asnet4 = – An Autonomous System Number (ASN) and its associated IPv4 address range. • inet:asnet4:asn = – The Autonomous System Number (ASN) of the netblock. • inet:asnet4:net4 = – The IPv4 address range assigned to the ASN. • inet:asnet4:net4:max = – The last IPv4 in the range assigned to the ASN. • inet:asnet4:net4:min = – The ﬁrst IPv4 in the range assigned to the ASN.

3.2.20 inet:cidr4 =

An IPv4 address block in Classless Inter-Domain Routing (CIDR) notation. Properties: • inet:cidr4 = – An IPv4 address block in Classless Inter-Domain Routing (CIDR) notation. • inet:cidr4:ipv4 = – The IP address from the CIDR notation. • inet:cidr4:mask = – The mask from the CIDR notation.

138 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.21 inet:dns:a =

Fused knowledge of a DNS A record. Properties: • inet:dns:a = – Fused knowledge of a DNS A record. • inet:dns:a:fqdn = – The domain queried for its DNS A record. • inet:dns:a:ipv4 = – The IPv4 address returned in the A record. • inet:dns:a:seen:max = – The most recent observed time for the data in the A record. • inet:dns:a:seen:min = – The earliest observed time for the data in the A record.

3.2.22 inet:dns:aaaa =

Fused knowledge of a DNS AAAA record. Properties: • inet:dns:aaaa = – Fused knowledge of a DNS AAAA record. • inet:dns:aaaa:fqdn = – The domain queried for its DNS AAAA record. • inet:dns:aaaa:ipv6 = – The IPv6 address returned in the AAAA record. • inet:dns:aaaa:seen:max = – The most recent observed time for the data in the AAAA record. • inet:dns:aaaa:seen:min = – The earliest observed time for the data in the AAAA record.

3.2.23 inet:dns:cname =

Consolidated knowledge of a DNS CNAME record. Properties: • inet:dns:cname = – Consolidated knowledge of a DNS CNAME record. • inet:dns:cname:cname = – The domain returned in the CNAME record.

3.2. Forms 139 Synapse Documentation, Release 0.0.34

• inet:dns:cname:fqdn = – The domain queried for its CNAME record. • inet:dns:cname:seen:max = – The most recent observed time for the data in the CNAME record. • inet:dns:cname:seen:min = – The earliest observed time for the data in the CNAME record.

3.2.24 inet:dns:look =

Instance knowledge of a DNS record lookup. Properties: • inet:dns:look = – Instance knowledge of a DNS record lookup. • inet:dns:look:a = – The DNS A record returned by the lookup. • inet:dns:look:a:fqdn = – The domain queried for its A record. • inet:dns:look:a:ipv4 = – The IPv4 address returned in the A record. • inet:dns:look:aaaa = – The DNS AAAA record returned by the lookup. • inet:dns:look:aaaa:fqdn = – The domain queried for its AAAA record. • inet:dns:look:aaaa:ipv6 = – The IPv6 address returned in the AAAA record. • inet:dns:look:cname = – The DNS CNAME record returned by the lookup. • inet:dns:look:cname:cname = – The domain returned in the CNAME record. • inet:dns:look:cname:fqdn = – The domain queried for its CNAME record. • inet:dns:look:mx = – The DNS MX record returned by the lookup. • inet:dns:look:mx:fqdn = – The domain queried for its MX record. • inet:dns:look:mx:mx = – The domain returned in the MX record.

140 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

• inet:dns:look:ns = – The DNS NS record returned by the lookup. • inet:dns:look:ns:ns = – The domain returned in the NS record. • inet:dns:look:ns:zone = – The domain queried for its NS record. • inet:dns:look:rev = – The DNS PTR record returned by the lookup. • inet:dns:look:rev:fqdn = – The domain returned in the PTR record. • inet:dns:look:rev:ipv4 = – The IPv4 address queried for its PTR record. • inet:dns:look:soa = – The DNS SOA record returned by the lookup. • inet:dns:look:soa:email = – The normalized email address (RNAME) returned in the SOA record. • inet:dns:look:soa:expire = – The EXPIRE value returned in the SOA record. • inet:dns:look:soa:fqdn = – The domain queried for its SOA record. • inet:dns:look:soa:min = – The MINIMUM value returned in the SOA record. • inet:dns:look:soa:ns = – The domain (MNAME) returned in the SOA record. • inet:dns:look:soa:refresh = – The REFRESH value returned in the SOA record. • inet:dns:look:soa:retry = – The RETRY value returned in the SOA record. • inet:dns:look:soa:serial = – The SERIAL value returned in the SOA record. • inet:dns:look:time = – The date and time that the lookup occurred. • inet:dns:look:txt = – The DNS TXT record returned by the lookup. • inet:dns:look:txt:fqdn = – The domain queried for its TXT record.

3.2. Forms 141 Synapse Documentation, Release 0.0.34

• inet:dns:look:txt:txt = – The string returned in the TXT record.

3.2.25 inet:dns:mx =

Consolidated knowledge of a DNS MX record. Properties: • inet:dns:mx = – Consolidated knowledge of a DNS MX record. • inet:dns:mx:fqdn = – The domain queried for its MX record. • inet:dns:mx:mx = – The domain returned in the MX record. • inet:dns:mx:seen:max = – The most recent observed time for the data in the MX record. • inet:dns:mx:seen:min = – The earliest observed time for the data in the MX record.

3.2.26 inet:dns:ns =

Fused knowledge of a DNS NS record. Properties: • inet:dns:ns = – Fused knowledge of a DNS NS record. • inet:dns:ns:ns = – The domain returned in the NS record. • inet:dns:ns:seen:max = – The most recent observed time for the data in the NS record. • inet:dns:ns:seen:min = – The earliest observed time for the data in the NS record. • inet:dns:ns:zone = – The domain queried for its DNS NS record.

3.2.27 inet:dns:req =

Fused knowledge of a DNS request origin Properties: • inet:dns:req =

142 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

– Fused knowledge of a DNS request origin • inet:dns:req:addr = – The IPv4 address which requested the FQDN • inet:dns:req:addr:ipv4 = – The IPv4 address which requested the FQDN • inet:dns:req:fqdn = – The requested FQDN • inet:dns:req:type = – The type of DNS record requested

3.2.28 inet:dns:rev =

Fused knowledge of a DNS PTR record. Properties: • inet:dns:rev = – Fused knowledge of a DNS PTR record. • inet:dns:rev:fqdn = – The domain returned in the PTR record. • inet:dns:rev:ipv4 = – The IPv4 address queried for its DNS PTR record. • inet:dns:rev:seen:max = – The most recent observed time for the data in the PTR record. • inet:dns:rev:seen:min = – The earliest observed time for the data in the PTR record.

3.2.29 inet:dns:rev6 =

Fused knowledge of a DNS PTR record for IPv6. Properties: • inet:dns:rev6 = – Fused knowledge of a DNS PTR record for IPv6. • inet:dns:rev6:fqdn = – The domain returned in the PTR record. • inet:dns:rev6:ipv6 = – The IPv6 address queried for its DNS PTR record. • inet:dns:rev6:seen:max = – The most recent observed time for the data in the PTR record. • inet:dns:rev6:seen:min =

3.2. Forms 143 Synapse Documentation, Release 0.0.34

– The earliest observed time for the data in the PTR record.

3.2.30 inet:dns:soa =

Consolidated knowledge of a DNS SOA record. Properties: • inet:dns:soa = – Consolidated knowledge of a DNS SOA record. • inet:dns:soa:email = – The email address (RNAME) returned in the SOA record. • inet:dns:soa:fqdn = – The domain queried for its SOA record. • inet:dns:soa:ns = – The domain (MNAME) returned in the SOA record • inet:dns:soa:seen:max = – The most recent observed time for the data in the SOA record. • inet:dns:soa:seen:min = – The earliest observed time for the data in the SOA record.

3.2.31 inet:dns:txt =

Consolidated knowledge of a DNS TXT record. Properties: • inet:dns:txt = – Consolidated knowledge of a DNS TXT record. • inet:dns:txt:fqdn = – The domain queried for its TXT record. • inet:dns:txt:seen:max = – The most recent observed time for the data in the TXT record. • inet:dns:txt:seen:min = – The earliest observed time for the data in the TXT record. • inet:dns:txt:txt = – The string returned in the TXT record.

144 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.32 inet:email =

An email address. Properties: • inet:email = – An email address. • inet:email:fqdn = – The domain of the email address. • inet:email:user = – The username of the email address.

3.2.33 inet:ﬂow =

An individual network connection between a given source and destination. Properties: • inet:flow = – An individual network connection between a given source and destination. • inet:flow:dst:exe = <file:bytes> – The file (executable) that received the connection. • inet:flow:dst:host = – The guid of the destination host. • inet:flow:dst:proc = – The guid of the destination process. • inet:flow:dst:tcp4 = – The destination IPv4 address / port for an IPv4 TCP connection. • inet:flow:dst:tcp4:ipv4 = – The destination IPv4 address. • inet:flow:dst:tcp4:port = – The destination IPv4 port. • inet:flow:dst:tcp6 = – The destination IPv6 address / port for an IPv6 TCP connection. • inet:flow:dst:tcp6:ipv6 = – The destination IPv6 address. • inet:flow:dst:tcp6:port = – The destination IPv6 port. • inet:flow:dst:txbytes = – The number of bytes sent by the destination host / process / file.

3.2. Forms 145 Synapse Documentation, Release 0.0.34

• inet:flow:dst:udp4 = – The destination IPv4 address / port for an IPv4 UDP connection. • inet:flow:dst:udp4:ipv4 = – The destination IPv4 address. • inet:flow:dst:udp4:port = – The destination IPv4 port. • inet:flow:dst:udp6 = – The destination IPv6 address / port for an IPv6 UDP connection. • inet:flow:dst:udp6:ipv6 = – The destination IPv6 address. • inet:flow:dst:udp6:port = – The destination IPv6 port. • inet:flow:duration = – The duration of the flow in seconds. • inet:flow:from = – The ingest source file/iden. Used for reparsing. • inet:flow:src:exe = <file:bytes> – The file (executable) that created the connection. • inet:flow:src:host = – The guid of the source host. • inet:flow:src:proc = – The guid of the source process. • inet:flow:src:tcp4 = – The source IPv4 address / port for an IPv4 TCP connection. • inet:flow:src:tcp4:ipv4 = – The source IPv4 address. • inet:flow:src:tcp4:port = – The source IPv4 port. • inet:flow:src:tcp6 = – The source IPv6 address / port for an IPv6 TCP connection. • inet:flow:src:tcp6:ipv6 = – The source IPv6 address. • inet:flow:src:tcp6:port = – The source IPv6 port. • inet:flow:src:txbytes = – The number of bytes sent by the source host / process / file.

146 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

• inet:flow:src:udp4 = – The source IPv4 address / port for an IPv4 UDP connection. • inet:flow:src:udp4:ipv4 = – The source IPv4 address. • inet:flow:src:udp4:port = – The source IPv4 port. • inet:flow:src:udp6 = – The source IPv6 address / port for an IPv6 UDP connection. • inet:flow:src:udp6:ipv6 = – The source IPv6 address. • inet:flow:src:udp6:port = – The source IPv6 port. • inet:flow:time = – The time the network connection was initiated.

3.2.34 inet:fqdn =

A fully qualified domain name (FQDN). Properties: • inet:fqdn = – A fully qualified domain name (FQDN). • inet:fqdn:created = – The earliest known registration (creation) date for the fqdn. • inet:fqdn:domain = – The parent fqdn of the fqdn. • inet:fqdn:expires = – The current expiration date for the fqdn. • inet:fqdn:host = – The host portion of the fqdn. • inet:fqdn:sfx = (default: 0) – Set to 1 if the fqdn is considered a “suffix”. • inet:fqdn:updated = – The last known updated date for the fqdn. • inet:fqdn:zone = (default: 0) – Set to 1 if the fqdn is a logical zone (under a suffix).

3.2. Forms 147 Synapse Documentation, Release 0.0.34

3.2.35 inet:iface =

A network interface with a set of associated protocol addresses. Properties: • inet:iface = – A network interface with a set of associated protocol addresses. • inet:iface:host = – The guid of the host the interface is associated with. • inet:iface:ipv4 = – The IPv4 address of the interface. • inet:iface:ipv6 = – The IPv6 address of the interface. • inet:iface:mac = – The ethernet (MAC) address of the interface. • inet:iface:mob:imei = – The IMEI of the interface. • inet:iface:mob:imsi = – The IMSI of the interface. • inet:iface:phone = – The telephone number of the interface. • inet:iface:wifi:bssid = – The wifi BSSID of the interface. • inet:iface:wifi:ssid = – The wifi SSID of the interface.

3.2.36 inet:ipv4 =

An IPv4 address. Properties: • inet:ipv4 = – An IPv4 address. • inet:ipv4:asn = (default: -1) – The ASN to which the IPv4 address is currently assigned. • inet:ipv4:cc = (default: ‘??’) – The country where the IPv4 address is currently located. • inet:ipv4:type = (default: ‘??’) – The type of IP address (e.g., private, multicast, etc.).

148 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.37 inet:ipv6 =

An IPv6 address. Properties: • inet:ipv6 = – An IPv6 address. • inet:ipv6:asn = (default: -1) – The ASN to which the IPv6 address is currently assigned. • inet:ipv6:cc = (default: ‘??’) – The country where the IPv6 address is currently located.

3.2.38 inet:mac =

A 48-bit Media Access Control (MAC) address. Properties: • inet:mac = – A 48-bit Media Access Control (MAC) address. • inet:mac:vendor = (default: ‘??’) – The vendor associated with the 24-bit preﬁx of a MAC address.

3.2.39 inet:passwd =

A password string. Properties: • inet:passwd = – A password string. • inet:passwd:md5 = – The computed MD5 hash of the password. • inet:passwd:sha1 = – The computed SHA1 hash of the password. • inet:passwd:sha256 = – The computed SHA256 hash of the password.

3.2.40 inet:ssl:tcp4cert =

An SSL certificate file served by an IPv4 TCP server. Properties: • inet:ssl:tcp4cert = – An SSL certificate file served by an IPv4 TCP server.

3.2. Forms 149 Synapse Documentation, Release 0.0.34

• inet:ssl:tcp4cert:cert = <file:bytes> – The SSL certificate. • inet:ssl:tcp4cert:tcp4 = – The IPv4 TCP server where the certificate was observed. • inet:ssl:tcp4cert:tcp4:ipv4 = – The IPv4 address associated with the TCP server.

3.2.41 inet:tcp4 =

An IPv4 address and port. Properties: • inet:tcp4 = – An IPv4 address and port. • inet:tcp4:ipv4 = – The IPv4 address of the TCP server. • inet:tcp4:port = – The port of the IPv4 TCP server.

3.2.42 inet:tcp6 =

An IPv6 address and port. Properties: • inet:tcp6 = – An IPv6 address and port. • inet:tcp6:ipv6 = – The IPv6 address of the TCP server. • inet:tcp6:port = – The port of the IPv6 TCP server.

3.2.43 inet:udp4 =

An IPv4 address and port. Properties: • inet:udp4 = – An IPv4 address and port. • inet:udp4:ipv4 = – The IPv4 address of the UDP server. • inet:udp4:port =

150 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

– The port of the IPv4 UDP server.

3.2.44 inet:udp6 =

An IPv6 address and port. Properties: • inet:udp6 = – An IPv6 address and port. • inet:udp6:ipv6 = – The IPv6 address of the UDP server. • inet:udp6:port = – The port of the IPv6 UDP server.

3.2.45 inet:url =

A Universal Resource Locator (URL). Properties: • inet:url = – A Universal Resource Locator (URL). • inet:url:fqdn = – The fqdn used in the URL (e.g., http://www.woot.com/page.html). • inet:url:ipv4 = – The IPv4 address used in the URL (e.g., http://1.2.3.4/page.html). • inet:url:ipv6 = – The IPv6 address used in the URL. • inet:url:passwd = – The optional password used to access the URL. • inet:url:port = – The port of the URL. URLs prefixed with http will be set to port 80 and URLs prefixed with https will be set to port 443 unless otherwise specified. • inet:url:user = – The optional username used to access the URL.

3.2.46 inet:urlﬁle =

A file hosted at a specific Universal Resource Locator (URL). Properties: • inet:urlfile = – A file hosted at a specific Universal Resource Locator (URL).

3.2. Forms 151 Synapse Documentation, Release 0.0.34

• inet:urlfile:file = <file:bytes> – The file that was hosted at the URL. • inet:urlfile:seen:max = – The most recent known time the file was hosted at the URL. • inet:urlfile:seen:min = – The earliest known time the file was hosted at the URL. • inet:urlfile:url = – The URL where the file was hosted.

3.2.47 inet:urlredir =

A URL that redirects to another URL, such as via a URL shortening service or an HTTP 302 response. Properties: • inet:urlredir = – A URL that redirects to another URL, such as via a URL shortening service or an HTTP 302 response. • inet:urlredir:dst = – The redirected/destination URL • inet:urlredir:dst:fqdn = – The FQDN within the dst URL (if present) • inet:urlredir:src = – The original/source URL before redirect • inet:urlredir:src:fqdn = – The FQDN within the src URL (if present)

3.2.48 inet:user =

A username string. Properties: • inet:user = – A username string.

3.2.49 inet:web:acct =

An account with a given Internet-based site or service. Properties: • inet:web:acct = – An account with a given Internet-based site or service. • inet:web:acct:avatar = <ﬁle:bytes>

152 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

– The file representing the avatar (e.g., profile picture) for the account. • inet:web:acct:dob = – A self-declared date of birth for the account (if the account belongs to a person). • inet:web:acct:email = – The email address associated with the account. • inet:web:acct:loc = – A self-declared location for the account. • inet:web:acct:name = – The name associated with the account (may be different from the account identifier, e.g., a display name). • inet:web:acct:occupation = – A self-declared occupation for the account. • inet:web:acct:passwd = – The current password for the account. • inet:web:acct:phone = – The phone number associated with the account. • inet:web:acct:realname = – The real name of the account owner / registrant. • inet:web:acct:seen:max = – The most recent known date of activity for the account. • inet:web:acct:seen:min = – The earliest known date of activity for the account. • inet:web:acct:signup = – The date and time the account was registered. • inet:web:acct:signup:ipv4 = – The IPv4 address used to sign up for the account. • inet:web:acct:site = – The site or service associated with the account. • inet:web:acct:tagline = – The text of the account status or tag line. • inet:web:acct:url = – The service provider URL where the account is hosted. • inet:web:acct:user = – The unique identifier for the account (may be different from the common name or display name). • inet:web:acct:webpage = – A related URL specified by the account (e.g., a personal or company web page, blog, etc.).

3.2. Forms 153 Synapse Documentation, Release 0.0.34

3.2.50 inet:web:action =

An instance of an account performing an action at an Internet-based site or service. Properties: • inet:web:action = – An instance of an account performing an action at an Internet-based site or service. • inet:web:action:acct = – The web account associated with the action. • inet:web:action:acct:site = – The site or service associated with the account. • inet:web:action:acct:user = – The unique identiﬁer for the account. • inet:web:action:act = – The action performed by the account. • inet:web:action:info = – Any other data associated with the action. • inet:web:action:ipv4 = – The source IPv4 address of the action. • inet:web:action:ipv6 = – The source IPv6 address of the action. • inet:web:action:time = – The date and time the account performed the action.

3.2.51 inet:web:actref =

A web action that references a given node. Properties: • inet:web:actref = – A web action that references a given node. • inet:web:actref:act = – The action that references the given node. • inet:web:actref:xref = – The prop=valu that is referenced as part of the action. • inet:web:actref:xref:intval = – The normed value of the form that was referenced, if the value is an integer. • inet:web:actref:xref:prop = – The property (form) of the referenced object, as speciﬁed by the propvalu.

154 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

• inet:web:actref:xref:strval = – The normed value of the form that was referenced, if the value is a string.

3.2.52 inet:web:chproﬁle =

A change to a web account. Used to capture historical properties associated with an account, as opposed to current data in the inet:web:acct node. Properties: • inet:web:chprofile = – A change to a web account. Used to capture historical properties associated with an account, as opposed to current data in the inet:web:acct node. • inet:web:chprofile:acct = – The web account associated with the change. • inet:web:chprofile:acct:site = – The site or service associated with the account. • inet:web:chprofile:acct:user = – The unique identifier for the account. • inet:web:chprofile:ipv4 = – The source IPv4 address used to make the account change. • inet:web:chprofile:ipv6 = – The source IPv6 address used to make the account change. • inet:web:chprofile:pv = – The prop=valu of the account property that was changed. Valu should be the old / original value, while the new value should be updated on the inet:web:acct form. • inet:web:chprofile:pv:intval = – The normed value of the property (specified by pv), if the property is an integer. • inet:web:chprofile:pv:prop = – The property that was changed. • inet:web:chprofile:pv:strval = – The normed value of the property (specified by pv), if the property is a string. • inet:web:chprofile:time = – The date and time when the account change occurred.

3.2.53 inet:web:ﬁle =

A file posted by a web account. Properties: • inet:web:file = – A file posted by a web account.

3.2. Forms 155 Synapse Documentation, Release 0.0.34

• inet:web:file:acct = – The account that owns or is associated with the file. • inet:web:file:acct:site = – The site or service associated with the account. • inet:web:file:acct:user = – The unique identifier for the account. • inet:web:file:file = <file:bytes> – The file owned by or associated with the account. • inet:web:file:ipv4 = – The source IPv4 address used to post or submit the file. • inet:web:file:ipv6 = – The source IPv6 address used to post or submit the file. • inet:web:file:name = <file:base> – The name of the file owned by or associated with the account. • inet:web:file:posted = – The date and time the file was posted / submitted. • inet:web:file:seen:max = – The most recent known date the file was posted / submitted / associatedwith the account. • inet:web:file:seen:min = – The earliest known date the file was posted / submitted / associated with the account.

3.2.54 inet:web:follows =

A web account follows or is connected to another web account. Properties: • inet:web:follows = – A web account follows or is connected to another web account. • inet:web:follows:followee = – The account followed by an account. • inet:web:follows:follower = – The account following an account. • inet:web:follows:seen:max = – The most recent known date when the “follows” relationship existed. • inet:web:follows:seen:min = – The earliest known date when the “follows” relationship existed.

156 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.55 inet:web:group =

A group hosted within or registered with a given Internet-based site or service. Properties: • inet:web:group = – A group hosted within or registered with a given Internet-based site or service. • inet:web:group:avatar = <file:bytes> – The file representing the avatar (e.g., profile picture) for the group. • inet:web:group:desc = – The text of the description of the group. • inet:web:group:name = – The base string type • inet:web:group:site = – The site or service associated with the group. • inet:web:group:url = – The service provider URL where the group is hosted. • inet:web:group:webpage = – A related URL specified by the group (e.g., primary web site, etc.).

3.2.56 inet:web:logon =

An instance of an account authenticating to an Internet-based site or service. Properties: • inet:web:logon = – An instance of an account authenticating to an Internet-based site or service. • inet:web:logon:acct = – The web account associated with the logon event. • inet:web:logon:acct:site = – The site or service associated with the account. • inet:web:logon:acct:user = – The unique identiﬁer for the account. • inet:web:logon:ipv4 = – The source IPv4 address of the logon. • inet:web:logon:ipv6 = – The source IPv6 address of the logon. • inet:web:logon:logout = – The date and time the account logged out of the service.

3.2. Forms 157 Synapse Documentation, Release 0.0.34

• inet:web:logon:time = – The date and time the account logged into the service.

3.2.57 inet:web:memb =

A web account that is a member of a web group. Properties: • inet:web:memb = – A web account that is a member of a web group. • inet:web:memb:acct = – The account that is a member of the group. • inet:web:memb:group = – The group that the account is a member of. • inet:web:memb:joined = – The date / time the account joined the group. • inet:web:memb:seen:max = – The most recent known date when the account was a member of the group. • inet:web:memb:seen:min = – The earliest known date when the account was a member of the group. • inet:web:memb:title = – The title or status of the member (e.g., admin, new member, etc.).

3.2.58 inet:web:mesg =

A message sent from one web account to another web account. Properties: • inet:web:mesg = – A message sent from one web account to another web account. • inet:web:mesg:file = <file:bytes> – The file attached to or sent with the message. • inet:web:mesg:from = – The web account that sent the message. • inet:web:mesg:text = – The text of the message. • inet:web:mesg:time = – The date and time at which the message was sent. • inet:web:mesg:to = – The web account that received the message.

158 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

• inet:web:mesg:url = – The URL where the message is posted / visible.

3.2.59 inet:web:post =

A post made by a web account. Properties: • inet:web:post = – A post made by a web account. • inet:web:post:acct = – The web account that made the post. • inet:web:post:acct:site = – The site or service associated with the account. • inet:web:post:acct:user = – The unique identifier for the account. • inet:web:post:file = <file:bytes> – The file that was attached to the post. • inet:web:post:replyto = – The post that this post is in reply to. • inet:web:post:repost = – The original post that this is a repost of. • inet:web:post:text = – The text of the post. • inet:web:post:time = – The date and time that the post was made. • inet:web:post:url = – The URL where the post is published / visible.

3.2.60 inet:web:postref =

A web post that references a given node. Properties: • inet:web:postref = – A web post that references a given node. • inet:web:postref:post = – The web post that references the given node. • inet:web:postref:xref = – The prop=valu that is referenced by the post.

3.2. Forms 159 Synapse Documentation, Release 0.0.34

• inet:web:postref:xref:intval = – The normed value of the form that was referenced, if the value is an integer. • inet:web:postref:xref:prop = – The property (form) of the referenced object, as speciﬁed by the propvalu. • inet:web:postref:xref:strval = – The normed value of the form that was referenced, if the value is a string.

3.2.61 inet:whois:contact =

An individual contact from a domain whois record. Properties: • inet:whois:contact = – An individual contact from a domain whois record. • inet:whois:contact:address = – The content of the street address field(s) of the contract. • inet:whois:contact:city = – The content of the city field of the contact. • inet:whois:contact:country = – The two-letter country code of the contact. • inet:whois:contact:email = – The email address of the contact. • inet:whois:contact:fax = – The content of the fax field of the contact. • inet:whois:contact:id = – The ID associated with the contact. • inet:whois:contact:name = – The name of the contact. • inet:whois:contact:orgname = – The name of the contact organization. • inet:whois:contact:phone = – The content of the phone field of the contact. • inet:whois:contact:rec = – The whois record containing the contact data. • inet:whois:contact:rec:asof = – The date of the whois record. • inet:whois:contact:rec:fqdn = – The domain associated with the whois record.

160 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

• inet:whois:contact:state = – The content of the state ﬁeld of the contact. • inet:whois:contact:type = – The contact type (e.g., registrar, registrant, admin, billing, tech, etc.).

3.2.62 inet:whois:rar =

A domain registrar. Properties: • inet:whois:rar = – A domain registrar.

3.2.63 inet:whois:rec =

A domain whois record Properties: • inet:whois:rec = – A domain whois record • inet:whois:rec:asof = – The date of the whois record. • inet:whois:rec:created = – The “created” time from the whois record. • inet:whois:rec:expires = – The “expires” time from the whois record. • inet:whois:rec:fqdn = – The domain associated with the whois record. • inet:whois:rec:registrant = (default: ‘??’) – The registrant name from the whois record. • inet:whois:rec:registrar = (default: ‘??’) – The registrar name from the whois record. • inet:whois:rec:text = – The full text of the whois record. • inet:whois:rec:updated = – The “last updated” time from the whois record.

3.2. Forms 161 Synapse Documentation, Release 0.0.34

3.2.64 inet:whois:recns =

A nameserver associated with a domain whois record. Properties: • inet:whois:recns = – A nameserver associated with a domain whois record. • inet:whois:recns:ns = – A nameserver for a domain as listed in the domain whois record. • inet:whois:recns:rec = – The whois record containing the nameserver data. • inet:whois:recns:rec:asof = – The date of the whois record. • inet:whois:recns:rec:fqdn = – The domain associated with the whois record.

3.2.65 inet:whois:reg =

A domain registrant. Properties: • inet:whois:reg = – A domain registrant.

3.2.66 inet:whois:regmail =

An association between a domain and a registrant email address. Properties: • inet:whois:regmail = – An association between a domain and a registrant email address. • inet:whois:regmail:email = – The registrant email address associated with the domain. • inet:whois:regmail:fqdn = – The domain associated with the registrant email address. • inet:whois:regmail:seen:max = – The most recent known date the registrant email was associated with the domain. • inet:whois:regmail:seen:min = – The earliest known date the registrant email was associated with the domain.

162 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.67 inet:wiﬁ:ssid =

A WiFi service set identifier (SSID) name. Properties: • inet:wifi:ssid = – A WiFi service set identifier (SSID) name.

3.2.68 it:av:ﬁlehit =

A file that triggered an alert on a specific antivirus signature. Properties: • it:av:filehit = – A file that triggered an alert on a specific antivirus signature. • it:av:filehit:file = <file:bytes> – The file that triggered the signature hit. • it:av:filehit:sig = – The signature that the file triggered on.

3.2.69 it:av:sig =

A vendor- or organization-speciﬁc antivirus signature name. Properties: • it:av:sig = – A vendor- or organization-speciﬁc antivirus signature name. • it:av:sig:desc = – A free-form description of the signature. • it:av:sig:org = – The organization responsible for the signature. • it:av:sig:sig = – The signature name. • it:av:sig:url = – A reference URL for information about the signature.

3.2.70 it:dev:int =

A developer-selected integer constant. Properties: • it:dev:int = – A developer-selected integer constant.

3.2. Forms 163 Synapse Documentation, Release 0.0.34

3.2.71 it:dev:mutex =

A string representing a mutex. Properties: • it:dev:mutex = – A string representing a mutex.

3.2.72 it:dev:pipe =

A string representing a named pipe. Properties: • it:dev:pipe = – A string representing a named pipe.

3.2.73 it:dev:regkey =

A Windows registry key. Properties: • it:dev:regkey = – A Windows registry key.

3.2.74 it:dev:regval =

A Windows registry key/value pair. Properties: • it:dev:regval = – A Windows registry key/value pair. • it:dev:regval:bytes = <ﬁle:bytes> – The ﬁle representing the value of the registry key, if the value is binary data. • it:dev:regval:int = – The value of the registry key, if the value is an integer. • it:dev:regval:key = – The Windows registry key. • it:dev:regval:str = – The value of the registry key, if the value is a string.

164 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.75 it:dev:str =

A developer-selected string. Properties: • it:dev:str = – A developer-selected string. • it:dev:str:norm = – Lower case normalized version of it:dev:str

3.2.76 it:exec:bind:tcp =

A Globally Unique Identifier type Properties: • it:exec:bind:tcp = – A Globally Unique Identifier type • it:exec:bind:tcp:exe = <file:bytes> – The specific file containing code that bound the port. May or may not be the same :exe specified in :proc, if present. • it:exec:bind:tcp:host = – The host running the process that bound the port. Typically the same host referenced in :proc, if present. • it:exec:bind:tcp:ipv4 = – The IPv4 address specified to bind(). • it:exec:bind:tcp:ipv6 = – The IPv6 specified to bind(). • it:exec:bind:tcp:port = – The bound (listening) TCP port. • it:exec:bind:tcp:proc = – The main process executing code that bound the listening TCP port. • it:exec:bind:tcp:time = – The time the port was bound.

3.2.77 it:exec:bind:udp =

A Globally Unique Identifier type Properties: • it:exec:bind:udp = – A Globally Unique Identifier type • it:exec:bind:udp:exe = <file:bytes>

3.2. Forms 165 Synapse Documentation, Release 0.0.34

– The specific file containing code that bound the port. May or may not be the same :exe specified in :proc, if present. • it:exec:bind:udp:host = – The host running the process that bound the port. Typically the same host referenced in :proc, if present. • it:exec:bind:udp:ipv4 = – The IPv4 specified to bind(). • it:exec:bind:udp:ipv6 = – The IPv6 specified to bind(). • it:exec:bind:udp:port = – The bound (listening) UDP port. • it:exec:bind:udp:proc = – The main process executing code that bound the listening UDP port. • it:exec:bind:udp:time = – The time the port was bound.

3.2.78 it:exec:ﬁle:add =

A Globally Unique Identifier type Properties: • it:exec:file:add = – A Globally Unique Identifier type • it:exec:file:add:exe = <file:bytes> – The specific file containing code that created the new file. May or may not be the same :exe specified in :proc, if present. • it:exec:file:add:file = <file:bytes> – The file that was created. • it:exec:file:add:host = – The host running the process that created the new file. Typically the same host referenced in :proc, if present. • it:exec:file:add:path = <file:path> – The path where the file was created. • it:exec:file:add:path:base = <file:base> – The final component of the file path (parsed from :path). • it:exec:file:add:path:dir = <file:path> – The parent directory of the file path (parsed from :path). • it:exec:file:add:path:ext = – The file extension of the file name (parsed from :path).

166 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

• it:exec:file:add:proc = – The main process executing code that created the new file. • it:exec:file:add:time = – The time the file was created.

3.2.79 it:exec:ﬁle:del =

A Globally Unique Identifier type Properties: • it:exec:file:del = – A Globally Unique Identifier type • it:exec:file:del:exe = <file:bytes> – The specific file containing code that deleted the file. May or may not be the same :exe specified in :proc if present. • it:exec:file:del:file = <file:bytes> – The file that was deleted. • it:exec:file:del:host = – The host running the process that deleted the file. Typically the same host referenced in :proc, if present. • it:exec:file:del:path = <file:path> – The path where the file was deleted. • it:exec:file:del:path:base = <file:base> – The final component of the file path (parsed from :path). • it:exec:file:del:path:dir = <file:path> – The parent directory of the file path (parsed from :path). • it:exec:file:del:path:ext = – The file extension of the file name (parsed from :path). • it:exec:file:del:proc = – The main process executing code that deleted the file. • it:exec:file:del:time = – The time the file was deleted.

3.2.80 it:exec:ﬁle:read =

A Globally Unique Identifier type Properties: • it:exec:file:read = – A Globally Unique Identifier type

3.2. Forms 167 Synapse Documentation, Release 0.0.34

• it:exec:file:read:exe = <file:bytes> – The specific file containing code that read the file. May or may not be the same :exe specified in :proc, if present. • it:exec:file:read:file = <file:bytes> – The file that was read. • it:exec:file:read:host = – The host running the process that read the file. Typically the same host referenced in :proc, if present. • it:exec:file:read:path = <file:path> – The path where the file was read. • it:exec:file:read:path:base = <file:base> – The final component of the file path (parsed from :path). • it:exec:file:read:path:dir = <file:path> – The parent directory of the file path (parsed from :path). • it:exec:file:read:path:ext = – The file extension of the file name (parsed from :path). • it:exec:file:read:proc = – The main process executing code that read the file. • it:exec:file:read:time = – The time the file was read.

3.2.81 it:exec:ﬁle:write =

A Globally Unique Identifier type Properties: • it:exec:file:write = – A Globally Unique Identifier type • it:exec:file:write:exe = <file:bytes> – The specific file containing code that wrote to the file. May or may not be the same :exe referenced in :proc, if present. • it:exec:file:write:file = <file:bytes> – The file that was modified. • it:exec:file:write:host = – The host running the process that wrote to the file. Typically the same host referenced in :proc, if present. • it:exec:file:write:path = <file:path> – The path where the file was modified. • it:exec:file:write:path:base = <file:base> – The final component of the file path (parsed from :path).

168 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

• it:exec:file:write:path:dir = <file:path> – The parent directory of the file path (parsed from :path). • it:exec:file:write:path:ext = – The file extension of the file name (parsed from :path). • it:exec:file:write:proc = – The main process executing code that wrote to / modified the existing file. • it:exec:file:write:time = – The time the file was written to / modified.

3.2.82 it:exec:mutex =

A Globally Unique Identifier type Properties: • it:exec:mutex = – A Globally Unique Identifier type • it:exec:mutex:exe = <file:bytes> – The specific file containing code that created the mutex. May or may not be the same :exe specified in :proc, if present. • it:exec:mutex:host = – The host running the process that created the mutex. Typically the same host referenced in :proc, if present. • it:exec:mutex:name = – The mutex string. • it:exec:mutex:proc = – The main process executing code that created the mutex. • it:exec:mutex:time = – The time the mutex was created.

3.2.83 it:exec:pipe =

A Globally Unique Identifier type Properties: • it:exec:pipe = – A Globally Unique Identifier type • it:exec:pipe:exe = <file:bytes> – The specific file containing code that created the named pipe. May or may not be the same :exe specified in :proc, if present. • it:exec:pipe:host =

3.2. Forms 169 Synapse Documentation, Release 0.0.34

– The host running the process that created the named pipe. Typically the same host referenced in :proc, if present. • it:exec:pipe:name = – The named pipe string. • it:exec:pipe:proc = – The main process executing code that created the named pipe. • it:exec:pipe:time = – The time the named pipe was created.

3.2.84 it:exec:proc =

A Globally Unique Identifier type Properties: • it:exec:proc = – A Globally Unique Identifier type • it:exec:proc:cmd = – The command string used to launch the process, including any command line parameters. • it:exec:proc:exe = <file:bytes> – The file considered the “main” executable for the process. For example, rundll32.exe may be considered the “main” executable for DLLs loaded by that program. • it:exec:proc:host = – The host that executed the process. May be an actual or a virtual / notional host. • it:exec:proc:path = <file:path> – The path to the executable of the process. • it:exec:proc:pid = – The process ID. • it:exec:proc:src:exe = <file:path> – The executable which created the process. • it:exec:proc:src:proc = – The process which created the process. • it:exec:proc:time = – The start time for the process. • it:exec:proc:user = – The user name of the process owner.

170 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.85 it:exec:reg:del =

A Globally Unique Identifier type Properties: • it:exec:reg:del = – A Globally Unique Identifier type • it:exec:reg:del:exe = <file:bytes> – The specific file containing code that deleted data from the registry. May or may not be the same :exe referenced in :proc, if present. • it:exec:reg:del:host = – The host running the process that deleted data from the registry. Typically the same host referenced in :proc, if present. • it:exec:reg:del:proc = – The main process executing code that deleted data from the registry. • it:exec:reg:del:reg = – The registry key or value that was deleted. • it:exec:reg:del:reg:bytes = <file:bytes> – The binary data that was deleted (parsed from :reg). • it:exec:reg:del:reg:int = – The integer value that was deleted (parsed from :reg). • it:exec:reg:del:reg:key = – The registry key that was deleted (parsed from :reg). • it:exec:reg:del:reg:str = – The string value that was deleted (parsed from :reg). • it:exec:reg:del:time = – The time the data from the registry was deleted.

3.2.86 it:exec:reg:get =

A Globally Unique Identifier type Properties: • it:exec:reg:get = – A Globally Unique Identifier type • it:exec:reg:get:exe = <file:bytes> – The specific file containing code that read the registry. May or may not be the same :exe referenced in :proc, if present. • it:exec:reg:get:host = – The host running the process that read the registry. Typically the same host referenced in :proc, if present.

3.2. Forms 171 Synapse Documentation, Release 0.0.34

• it:exec:reg:get:proc = – The main process executing code that read the registry. • it:exec:reg:get:reg = – The registry key or value that was read. • it:exec:reg:get:reg:bytes = <ﬁle:bytes> – The binary data that was read (parsed from :reg). • it:exec:reg:get:reg:int = – The integer value that was read (parsed from :reg). • it:exec:reg:get:reg:key = – The registry key that was read (parsed from :reg). • it:exec:reg:get:reg:str = – The string value that was read (parsed from :reg). • it:exec:reg:get:time = – The time the registry was read.

3.2.87 it:exec:reg:set =

A Globally Unique Identifier type Properties: • it:exec:reg:set = – A Globally Unique Identifier type • it:exec:reg:set:exe = <file:bytes> – The specific file containing code that wrote to the registry. May or may not be the same :exe referenced in :proc, if present. • it:exec:reg:set:host = – The host running the process that wrote to the registry. Typically the same host referenced in :proc, if present. • it:exec:reg:set:proc = – The main process executing code that wrote to the registry. • it:exec:reg:set:reg = – The registry key or value that was written. • it:exec:reg:set:reg:bytes = <file:bytes> – The binary data that was written (parsed from :reg). • it:exec:reg:set:reg:int = – The integer value that was written (parsed from :reg). • it:exec:reg:set:reg:key = – The registry key that was written (parsed from :reg). • it:exec:reg:set:reg:str =

172 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

– The string value that was written (parsed from :reg). • it:exec:reg:set:time = – The time the registry was written to.

3.2.88 it:exec:url =

A Globally Unique Identifier type Properties: • it:exec:url = – A Globally Unique Identifier type • it:exec:url:exe = <file:bytes> – The specific file containing code that requested the URL. May or may not be the same :exe specified in :proc, if present. • it:exec:url:host = – The host running the process that requested the URL. Typically the same host referenced in :proc, if present. • it:exec:url:ipv4 = – The IPv4 address of the host during URL retrieval. • it:exec:url:ipv6 = – The IPv6 address of the host during URL retrieval. • it:exec:url:proc = – The main process executing code that requested the URL. • it:exec:url:time = – The time the URL was requested. • it:exec:url:url = – The URL that was requested.

3.2.89 it:fs:ﬁle =

A Globally Unique Identifier type Properties: • it:fs:file = – A Globally Unique Identifier type • it:fs:file:atime = – The file access time. • it:fs:file:ctime = – The file creation time. • it:fs:file:file = <file:bytes>

3.2. Forms 173 Synapse Documentation, Release 0.0.34

– The file on the host. • it:fs:file:group = – The group owner of the file. • it:fs:file:host = – The host containing the file. • it:fs:file:mtime = – The file modification time. • it:fs:file:path = <file:path> – The path for the file. • it:fs:file:path:base = <file:base> – The final component of the file path (parsed from :path). • it:fs:file:path:dir = <file:path> – The parent directory of the file path (parsed from :path). • it:fs:file:path:ext = – The file extension of the file name (parsed from :path). • it:fs:file:user = – The owner of the file.

3.2.90 it:host =

A GUID that represents a host or system. Properties: • it:host = – A GUID that represents a host or system. • it:host:desc = – A free-form description of the host. • it:host:ipv4 = – The last known ipv4 address for the host. • it:host:name = – The name of the host or system.

3.2.91 it:hostname =

The name of a host or system. Properties: • it:hostname = – The name of a host or system.

174 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.92 it:hostsoft =

A version of a software product which is present on a given host. Properties: • it:hostsoft = – A version of a software product which is present on a given host. • it:hostsoft:host = – Host with the software • it:hostsoft:seen:max = – Maximum time the software was seen on the host • it:hostsoft:seen:min = – Minimum time the software was seen on the host • it:hostsoft:softver = – Software on the host

3.2.93 it:prod:soft =

A arbitrary, unversioned software product. Properties: • it:prod:soft = – A arbitrary, unversioned software product. • it:prod:soft:author:acct = – Web user responsible for the software • it:prod:soft:author:org = – Organization responsible for the software • it:prod:soft:author:person = – Person responsible for the software • it:prod:soft:desc = – A description of the software • it:prod:soft:desc:short = – A short description of the software • it:prod:soft:name = – Name of the software • it:prod:soft:url = – URL relevant for the software

3.2. Forms 175 Synapse Documentation, Release 0.0.34

3.2.94 it:prod:softver =

A version of a particular software product. Properties: • it:prod:softver = – A version of a particular software product. • it:prod:softver:arch = – Software architecture. • it:prod:softver:semver = – System normalized semantic version number. • it:prod:softver:semver:build = – Semver build string. • it:prod:softver:semver:major = – Version major number • it:prod:softver:semver:minor = – Version minor number • it:prod:softver:semver:patch = – Version patch number • it:prod:softver:semver:pre = – Semver prerelease string. • it:prod:softver:software = – Software associated with this version instance. • it:prod:softver:software:name = – The name of the software at a particular version. • it:prod:softver:url = – URL where a speciﬁc version of the software is available from • it:prod:softver:vers = – A developer-selected string. • it:prod:softver:vers:norm = – Normalized version of the version string.

3.2.95 it:sec:cve =

A vulnerability as designated by a Common Vulnerabilities and Exposures (CVE) number. Properties: • it:sec:cve = – A vulnerability as designated by a Common Vulnerabilities and Exposures (CVE) number.

176 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

• it:sec:cve:desc = – A free-form description of the CVE vulnerability.

3.2.96 lang:idiom =

A subcultural idiom Properties: • lang:idiom = – A subcultural idiom • lang:idiom:desc:en = – English description • lang:idiom:url = – Authoritative URL for the idiom

3.2.97 lang:trans =

Raw text with a documented translation Properties: • lang:trans = – Raw text with a documented translation • lang:trans:desc:en = – English description • lang:trans:text:en = – English translation

3.2.98 mat:item =

A GUID assigned to a material object Properties: • mat:item = – A GUID assigned to a material object • mat:item:name = – The human readable name of the material item

3.2.99 mat:itemimage =

A multi-ﬁeld composite type which uses separated repr values Properties: • mat:itemimage =

3.2. Forms 177 Synapse Documentation, Release 0.0.34

– A multi-field composite type which uses separated repr values • mat:itemimage:file = <file:bytes> – The file containing an image of the item • mat:itemimage:item = – The item contained within the image file

3.2.100 mat:spec =

A GUID assigned to a material speciﬁcation Properties: • mat:spec = – A GUID assigned to a material speciﬁcation • mat:spec:name = – The human readable name of the material spec

3.2.101 mat:specimage =

A multi-field composite type which uses separated repr values Properties: • mat:specimage = – A multi-field composite type which uses separated repr values • mat:specimage:file = <file:bytes> – The file containing an image of the spec • mat:specimage:spec = – The spec contained within the image file

3.2.102 media:news =

A published news item, report, or article GUID Properties: • media:news = – A published news item, report, or article GUID • media:news:author = (default: ‘?,?’) – The free-form author of the news • media:news:file = <file:bytes> – The (optional) file blob containing or published as the news • media:news:org = (default: ‘??’) – The org alias which published the news

178 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

• media:news:published = (default: 0) – The date the news item was published • media:news:summary = (default: ‘??’) – A brief summary of the news item • media:news:title = (default: ‘??’) – Title/Headline for the news • media:news:url = – The (optional) URL where the news was published • media:news:url:fqdn = – The FQDN within the news URL

3.2.103 ou:hasemail =

A multi-field composite type which generates a stable guid from normalized fields Properties: • ou:hasemail = – A multi-field composite type which generates a stable guid from normalized fields • ou:hasemail:email = – An email address. • ou:hasemail:org = – A GUID for a human organization such as a company or military unit • ou:hasemail:seen:max = – Maximum time in millis since epoch • ou:hasemail:seen:min = – Minimum time in millis since epoch

3.2.104 ou:hasﬁle =

A multi-field composite type which generates a stable guid from normalized fields Properties: • ou:hasfile = – A multi-field composite type which generates a stable guid from normalized fields • ou:hasfile:file = <file:bytes> – A unique file identifier • ou:hasfile:org = – A GUID for a human organization such as a company or military unit • ou:hasfile:seen:max = – Maximum time in millis since epoch

3.2. Forms 179 Synapse Documentation, Release 0.0.34

• ou:hasﬁle:seen:min = – Minimum time in millis since epoch

3.2.105 ou:hasfqdn =

A multi-field composite type which generates a stable guid from normalized fields Properties: • ou:hasfqdn = – A multi-field composite type which generates a stable guid from normalized fields • ou:hasfqdn:fqdn = – A fully qualified domain name (FQDN). • ou:hasfqdn:org = – A GUID for a human organization such as a company or military unit • ou:hasfqdn:seen:max = – Maximum time in millis since epoch • ou:hasfqdn:seen:min = – Minimum time in millis since epoch

3.2.106 ou:hashost =

A multi-field composite type which generates a stable guid from normalized fields Properties: • ou:hashost = – A multi-field composite type which generates a stable guid from normalized fields • ou:hashost:host = – A GUID that represents a host or system. • ou:hashost:org = – A GUID for a human organization such as a company or military unit • ou:hashost:seen:max = – Maximum time in millis since epoch • ou:hashost:seen:min = – Minimum time in millis since epoch

3.2.107 ou:hasipv4 =

A multi-ﬁeld composite type which generates a stable guid from normalized ﬁelds Properties: • ou:hasipv4 =

180 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

– A multi-ﬁeld composite type which generates a stable guid from normalized ﬁelds • ou:hasipv4:ipv4 = – An IPv4 address. • ou:hasipv4:org = – A GUID for a human organization such as a company or military unit • ou:hasipv4:seen:max = – Maximum time in millis since epoch • ou:hasipv4:seen:min = – Minimum time in millis since epoch

3.2.108 ou:hasphone =

A multi-field composite type which generates a stable guid from normalized fields Properties: • ou:hasphone = – A multi-field composite type which generates a stable guid from normalized fields • ou:hasphone:org = – A GUID for a human organization such as a company or military unit • ou:hasphone:phone = • ou:hasphone:seen:max = – Maximum time in millis since epoch • ou:hasphone:seen:min = – Minimum time in millis since epoch

3.2.109 ou:haswebacct =

A multi-field composite type which generates a stable guid from normalized fields Properties: • ou:haswebacct = – A multi-field composite type which generates a stable guid from normalized fields • ou:haswebacct:org = – A GUID for a human organization such as a company or military unit • ou:haswebacct:seen:max = – Maximum time in millis since epoch • ou:haswebacct:seen:min = – Minimum time in millis since epoch • ou:haswebacct:web:acct = – An account with a given Internet-based site or service.

3.2. Forms 181 Synapse Documentation, Release 0.0.34

3.2.110 ou:member =

A multi-ﬁeld composite type which uses separated repr values Properties: • ou:member = – A multi-ﬁeld composite type which uses separated repr values • ou:member:start = – Minimum time in millis since epoch • ou:member:title = (default: ‘??’) – The base string type

3.2.111 ou:org =

A GUID for a human organization such as a company or military unit Properties: • ou:org = – A GUID for a human organization such as a company or military unit • ou:org:alias = – An alias for the org GUID • ou:org:cc = – The 2 digit ISO country code • ou:org:naics = – North American Industry Classiﬁcation System • ou:org:name = – The base string type • ou:org:name:en = – The base string type • ou:org:phone = – The primary phone number for the organization • ou:org:sic = – Standard Industrial Classiﬁcation Code • ou:org:url = – A Universal Resource Locator (URL). • ou:org:us:cage = – A Commercial and Government Entity (CAGE) code

182 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.112 ou:owns =

A multi-ﬁeld composite type which uses separated repr values Properties: • ou:owns = – A multi-ﬁeld composite type which uses separated repr values

3.2.113 ou:suborg =

An org which owns a sub org Properties: • ou:suborg = – An org which owns a sub org • ou:suborg:current = (default: 1) – Is the suborg relationship still current • ou:suborg:org = – The org which owns sub • ou:suborg:perc = – The optional percentage of sub which is owned by org • ou:suborg:seen:max = – The optional time the suborg relationship ended • ou:suborg:seen:min = – The optional time the suborg relationship began • ou:suborg:sub = – The the sub which is owned by org

3.2.114 ou:user =

A user name within an organization Properties: • ou:user = – A user name within an organization • ou:user:org = – A GUID for a human organization such as a company or military unit • ou:user:user = – A username string.

3.2. Forms 183 Synapse Documentation, Release 0.0.34

3.2.115 pol:country =

A GUID for a country Properties: • pol:country = – A GUID for a country • pol:country:founded = (default: 0) – Timestamp in milliseconds since epoch • pol:country:iso2 = – The 2 digit ISO country code • pol:country:iso3 = – The 3 digit ISO country code • pol:country:isonum = – The ISO integer country code • pol:country:name = – The base string type • pol:country:pop = (default: 0) – The base integer type • pol:country:tld = – A fully qualiﬁed domain name (FQDN).

3.2.116 pol:ﬂag = <ﬁle:bytes>

The flag image SHA256 Properties: • pol:flag = <file:bytes> – The flag image SHA256 • pol:flag:cc = – The (optional) ISO2 country code for the flag • pol:flag:orgalias = – The (optional) org alias for the flat

3.2.117 pol:hist =

A GUID for a country Properties: • pol:hist = – A GUID for a country

184 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.118 ps:contact =

A collection of contact information in a single record Properties: • ps:contact = – A collection of contact information in a single record • ps:contact:address = – The free-form address listed for the contact • ps:contact:asof = – The time this contact was created or modiﬁed • ps:contact:dob = – The Date of Birth (DOB) for the contact • ps:contact:email = – The main email address for this contact • ps:contact:email:work = – The contact work email address • ps:contact:name = – The person name listed for the contact • ps:contact:org = – The ou:org GUID which owns this contact • ps:contact:orgname = – The listed org/company name for this contact • ps:contact:person = – The ps:person GUID which owns this contact • ps:contact:phone = – The “main” phone number for this contact • ps:contact:phone:fax = – The contact fax phone number • ps:contact:phone:work = – The contact work phone number • ps:contact:photo = <ﬁle:bytes> – The photo listed for this contact • ps:contact:title = – The job/org title listed for this contact • ps:contact:url = – The home/main site for this contact • ps:contact:user =

3.2. Forms 185 Synapse Documentation, Release 0.0.34

– The username or handle for the contact • ps:contact:web:acct = – The social media account for this contact

3.2.119 ps:hasalias =

A multi-field composite type which uses separated repr values Properties: • ps:hasalias = – A multi-field composite type which uses separated repr values • ps:hasalias:alias = – A last,first person full name • ps:hasalias:person = – A GUID for a person or suspected person

3.2.120 ps:hasemail =

A multi-ﬁeld composite type which uses separated repr values Properties: • ps:hasemail = – A multi-ﬁeld composite type which uses separated repr values • ps:hasemail:email = – An email address. • ps:hasemail:person = – A GUID for a person or suspected person

3.2.121 ps:hasphone =

A multi-ﬁeld composite type which uses separated repr values Properties: • ps:hasphone = – A multi-ﬁeld composite type which uses separated repr values • ps:hasphone:person = – A GUID for a person or suspected person • ps:hasphone:phone =

186 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.122 ps:hasuser =

A multi-ﬁeld composite type which uses separated repr values Properties: • ps:hasuser = – A multi-ﬁeld composite type which uses separated repr values • ps:hasuser:person = – A GUID for a person or suspected person • ps:hasuser:user = – A username string.

3.2.123 ps:haswebacct =

A multi-ﬁeld composite type which uses separated repr values Properties: • ps:haswebacct = – A multi-ﬁeld composite type which uses separated repr values • ps:haswebacct:person = – A GUID for a person or suspected person • ps:haswebacct:web:acct = – An account with a given Internet-based site or service.

3.2.124 ps:image =

A multi-field composite type which uses separated repr values Properties: • ps:image = – A multi-field composite type which uses separated repr values • ps:image:file = <file:bytes> – A unique file identifier • ps:image:person = – A GUID for a person or suspected person

3.2.125 ps:name =

A last,ﬁrst person full name Properties: • ps:name = – A last,ﬁrst person full name

3.2. Forms 187 Synapse Documentation, Release 0.0.34

• ps:name:given = – The “given name” part of ps:name • ps:name:middle = – The “middle name” part of ps:name • ps:name:sur = – The “surname” part of ps:name

3.2.126 ps:person =

A GUID for a person or suspected person Properties: • ps:person = – A GUID for a person or suspected person • ps:person:dob = – The Date of Birth (DOB) if known • ps:person:guidname = – The GUID resolver alias for this person • ps:person:img = <ﬁle:bytes> – The “primary” image of a person • ps:person:name = – A last,ﬁrst person full name • ps:person:name:given = – A single name element (potentially given or sur) • ps:person:name:sur = – A single name element (potentially given or sur) • ps:person:nick = – A username string.

3.2.127 ps:tokn =

A single name element (potentially given or sur) Properties: • ps:tokn = – A single name element (potentially given or sur)

188 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.128 rsa:key =

An RSA keypair modulus and public exponent Properties: • rsa:key = – An RSA keypair modulus and public exponent • rsa:key:bits = – The length of the modulus in bits • rsa:key:mod = – The modulus • rsa:key:priv:exp = – The private exponent • rsa:key:priv:p = – One of the two private primes • rsa:key:priv:q = – One of the two private primes • rsa:key:pub:exp = – The public exponent

3.2.129 syn:auth:role =

The base string type Properties: • syn:auth:role = – The base string type • syn:auth:role:desc = – The base string type

3.2.130 syn:auth:user =

The base string type Properties: • syn:auth:user = – The base string type • syn:auth:user:storm:limit:lift = (default: 10000) – The storm query lift limit for the user • syn:auth:user:storm:limit:time = (default: 120) – The storm query time limit for the user

3.2. Forms 189 Synapse Documentation, Release 0.0.34

3.2.131 syn:auth:userrole =

A multi-field composite type which generates a stable guid from normalized fields Properties: • syn:auth:userrole = – A multi-field composite type which generates a stable guid from normalized fields • syn:auth:userrole:role = – The base string type • syn:auth:userrole:user = – The base string type

3.2.132 syn:core =

A node representing a unique Cortex Properties: • syn:core = – A node representing a unique Cortex

3.2.133 syn:ﬁfo =

A Globally Unique Identifier type Properties: • syn:fifo = – A Globally Unique Identifier type • syn:fifo:desc = – The fifo description • syn:fifo:name = – The name of this fifo

3.2.134 syn:form =

The base form type. Properties: • syn:form = – The base form type. • syn:form:doc = – basic form deﬁnition • syn:form:local = (default: 0) – Flag used to determine if a form should not be included in splices

190 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

• syn:form:model = – which model deﬁnes a given form • syn:form:ptype = – Synapse type for this form • syn:form:ver = – form version within the model

3.2.135 syn:model = prefix for all forms with in the model Properties: • syn:model = – prefix for all forms with in the model • syn:model:hash = – version hash for the current model • syn:model:prefix = – Prefix used by teh types/forms in the model

3.2.136 syn:prop =

The base property type. Properties: • syn:prop = – The base property type. • syn:prop:base = – Base name of the property • syn:prop:defval = – Set to the default value for this property • syn:prop:doc = – Description of the property definition. • syn:prop:form = – The form of the property. • syn:prop:glob = (default: 0) – Set to 1 if this property defines a glob • syn:prop:ptype = – Synapse type for this field • syn:prop:relname = – Relative name of the property

3.2. Forms 191 Synapse Documentation, Release 0.0.34

• syn:prop:req = – Set to 1 if this property is required to form teh node. • syn:prop:title = – A short description of the property deﬁnition. • syn:prop:univ = – Speciﬁes if a prop is universal and has no form associated with it.

3.2.137 syn:seq =

A sequential id generation tracker Properties: • syn:seq = – A sequential id generation tracker • syn:seq:nextvalu = (default: 0) – The next sequential value • syn:seq:width = (default: 0) – How many digits to use to represent the number

3.2.138 syn:splice =

A Globally Unique Identifier type Properties: • syn:splice = – A Globally Unique Identifier type • syn:splice:act = – The base string type • syn:splice:form = – The base string type • syn:splice:node = – A Globally Unique Identifier type • syn:splice:tag = – The base string type • syn:splice:time = – Timestamp in milliseconds since epoch • syn:splice:user = – The base string type • syn:splice:valu = – The base string type

192 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.139 syn:tag =

The base form for a synapse tag. Properties: • syn:tag = – The base form for a synapse tag. • syn:tag:base = • syn:tag:depth = (default: 0) – How deep the tag is in the hierarchy • syn:tag:doc = (default: ‘’) – The base string type • syn:tag:title = (default: ‘’) • syn:tag:up =

3.2.140 syn:tagform =

A node describing the meaning of a tag on a speciﬁc form Properties: • syn:tagform = – A node describing the meaning of a tag on a speciﬁc form • syn:tagform:doc = (default: ‘??’) – The long form description for what the tag means on the given node form • syn:tagform:form = – The form that the tag applies too • syn:tagform:tag = – The tag being documented • syn:tagform:title = (default: ‘??’) – The short name for what the tag means the given node form

3.2.141 syn:trigger =

A Globally Unique Identiﬁer type Properties: • syn:trigger = – A Globally Unique Identiﬁer type • syn:trigger:en = (default: 0) – Is the trigger currently enabled • syn:trigger:on =

3.2. Forms 193 Synapse Documentation, Release 0.0.34

– A synapse permission string • syn:trigger:run = – A synapse storm query string • syn:trigger:user = – The base string type

3.2.142 syn:type =

The base type type. Properties: • syn:type = – The base type type. • syn:type:* = • syn:type:ctor = – Python path to the class used to instantiate the type. • syn:type:subof = – Type which this inherits from.

3.2.143 tel:mob:imei =

An International Mobile Equipment Id Properties: • tel:mob:imei= – An International Mobile Equipment Id • tel:mob:imei:serial = – The serial number within the IMEI • tel:mob:imei:tac= – The Type Allocate Code within the IMEI

3.2.144 tel:mob:imsi =

An International Mobile Subscriber Id Properties: • tel:mob:imsi= – An International Mobile Subscriber Id • tel:mob:imsi:mcc = – The Mobile Country Code

194 Chapter 3. Synapse Data Model Synapse Documentation, Release 0.0.34

3.2.145 tel:mob:tac =

A mobile Type Allocation Code Properties: • tel:mob:tac= – A mobile Type Allocation Code • tel:mob:tac:internal = (default: ‘??’) – The TAC internal model name • tel:mob:tac:manu = (default: ‘??’) – The TAC manufacturer name • tel:mob:tac:model = (default: ‘??’) – The TAC model name • tel:mob:tac:org = – The org guid for the manufacturer

3.2.146 tel:phone =

Properties: • tel:phone= • tel:phone:cc = (default: ‘??’) – The 2 digit ISO country code

3.2.147 tel:preﬁx =

Properties: • tel:prefix= • tel:prefix:cc = (default: ‘??’) – The 2 digit ISO country code • tel:prefix:tag = – A synapse tag

3.3 Universal Props

Universal props are system level properties which are generally present on every node. These properties are not speciﬁc to a particular form and exist outside of a particular namespace.

3.3. Universal Props 195 Synapse Documentation, Release 0.0.34

3.3.1 node:created

• node:created = – The time the node was created

3.3.2 node:ndef

• node:ndef = – The unique guid representing the combination of the node form and primary property.

3.3.3 tufo:form

• tufo:form = – The form of the node

196 Chapter 3. Synapse Data Model CHAPTER 4

synapse package

4.1 Subpackages

4.1.1 synapse.cmds package

Submodules synapse.cmds.common module class synapse.cmds.common.GuidCmd(cli, **opts) Bases: synapse.lib.cli.Cmd Generate a new guid Examples: guid runCmdOpts(opts) class synapse.cmds.common.PyCmd(cli, **opts) Bases: synapse.lib.cli.Cmd Evaluate a line of python code with the cmd item. Examples: py item.getFooThingByBar(‘baz’) runCmdOpts(opts) synapse.cmds.cortex module class synapse.cmds.cortex.AskCmd(cli, **opts) Bases: synapse.lib.cli.Cmd

197 Synapse Documentation, Release 0.0.34

Execute a query. Examples: ask optional: –debug –[props|raw] ask –debug inet:ipv4=0 ask –props inet:ipv4=”0.0.0.0” ask –raw inet:ipv4=”0.0.0.0” ask –debug – props inet:ipv4=0x01020304 runCmdOpts(opts) class synapse.cmds.cortex.NextSeqCmd(cli, **opts) Bases: synapse.lib.cli.Cmd Generate and display the next id in the named sequence. Usage: nextseq runCmdOpts(opts)

Module contents

4.1.2 synapse.cores package

Submodules synapse.cores.common module class synapse.cores.common.Cortex(link, store, **conf ) Bases: synapse.eventbus.EventBus, synapse.datamodel.DataModel, synapse.lib. storm.Runtime, synapse.lib.ingest.IngestApi Top level Cortex key/valu storage object. ackCoreFifo(name, seqn) Acknowledge transmission of fifo items. Args: name (str): The syn:fifo:name of the fifo. nseq (int): The next expected sequence. addCoreMods(mods) Add a list of (name,conf) tuples to the cortex. addJsonItem(form, item, tstamp=None) Add and fully index a JSON compatible data structure ( not in text form ). Example: foo = { ‘bar’:10, ‘baz’:{ ‘faz’:’hehe’, ‘woot’:30 } } core.addJsonItem(‘foo’,foo) addJsonItems(form, items, tstamp=None) Add and fully index a list of JSON compatible data structures. Example: core.addJsonItems(‘foo’, foolist) addJsonText(form, text) Add and fully index a blob of JSON text. Example:

198 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

addListRows(prop, *vals) Add rows by making a guid for each and using now(). Example: core.addListRows(‘foo:bar’,[ 1, 2, 3, 4]) addPermDefs(defs) Add a set of permission definitions for use with the cortex auth subsystem. A perm definition tuple consists of: (perm, info, fields) ex: (‘node:add’, {‘doc’: ‘The permission to add nodes’}, ( (‘form’, {‘doc’: ‘The form of the node being created’}), ) Args: defs (list): A list of permission defs Returns: (None) addRoleRule(role, rule, indx=None) Add a rule tuple for the given role (optionally at index). Args: role (str): The role name rule (tuple): The rule tuple indx (int): The index at which to insert the rule addRows(rows) Add (iden,prop,valu,time) rows to the Cortex. Example: import time tick = now() rows = [ (id1,’baz’,30,tick), (id1,’foo’,’bar’,tick), ] core.addRows(rows) addRuntNode(form, valu, props=None) Add a “runtime” node which does not persist. This is used for ephemeral node “look aside” registration. Args: form (str): The primary property for the node valu (obj): The primary value for the node props (dict): The node secondary properties ( if modeled, they will be indexed ) Returns: ((None,dict)): The ephemeral node addSaveLink(func) Add an event callback to receive save events for this cortex. Example: def savemesg(mesg): dostuff() core.addSaveLink(savemesg) addSeedCtor(name, func) Add a “seed constructor” to the cortex. This allows modules to register functions to construct nodes by a “seed name” which they transform into an existing node from the model. Example: def seedOrgName(name, valu,**props): orgn = core.getTufoByProp(‘org:iden:name’,valu) if orgn == None:

4.1. Subpackages 199 Synapse Documentation, Release 0.0.34

orgn = core.formTufoByProp(‘org:iden’,guid(),name=valu) return orgn core.addSeedCtor(‘org:iden:name’, seedOrgName) core.formTufoByProp(‘org:iden:name’,’The Vertex Project, LLC’) addSpliceFd(fd) Write all cortex splice events to the speciﬁed ﬁle-like object. Example: fd = open(‘audit.mpk’,’r+b’) core.addSpliceFd(fd) addStatFunc(name, func) Add a callback function to implement a statistic type. Example: def calcwoot(rows): sum([ r[2] for r in rows ]) + 99 ... core.addStatFunc(‘woot’, calcwoot) # later.. woot = core.getStatByProp(‘woot’,’haha’) addTufoDark(tufo, name, valu) Add a dark row to a tufo with a given name and value. Dark rows get their own index and can be used to quickly pull tufos. While similar to dsets, these are primarily intended for implementing features inside of Synapse directly. Args: tufo ((str, dict)): Tufo to add the dark row too. name (str): Dark row name. value (str): Value to set on the dark property. May be any data type which may stored in a cortex. Returns: None: Returns None. addTufoDset(tufo, name) Add the tufo to a named dataset. addTufoList(tufo, name, *vals) Add “list” rows to a tufo. Example: core.addTufoList(tufo, ‘counts’, 99, 300, 1773) Notes: • this creates the tufo:list: prop on the tufo to indicate that the list is present. addTufoTag(tufo, tag, times=()) Add a tag (and optionally time box) to a tufo. Args: tufo ((str,dict)): A node in tuple form. tag (str): A synapse tag string times ((int,)): A list of time stamps in milli epoch Examples: Form a node, and add the baz.faz tag to it:

node= core.formTufoByProp('foo','bar') node= core.addTufoTag(tufo,'baz.faz')

Add a timeboxed tag to a node:

200 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

node= core.addTufoTag(tufo,'foo.bar@2012-2016')

Add a list of times sample times to a tag to create a timebox timeslist = (1513382400000, 1513468800000) node = core.addTufoTag(tufo,’hehe.haha’, times=timelist)

Returns: ((str,dict)): The node in tuple form (with updated props)

addTufoTags(tufo, tags) Add multiple tags to a tufo. Example: core.addTufoTags(tufo,[’foo.bar’,’baz.faz’]) addUserRule(user, rule, indx=None) Add a rule tuple for the given user (optionally at index). Args: user (str): The user name rule (tuple): The rule tuple indx (int): The index at which to insert the rule Returns: (None) NOTE: see synapse.lib.auth.Rules for rule tuple deﬁnition allowed(perm, user=None) Returns True if the user is allowed the given perms/info. Args: perm (str): The permission tufo user (str): The user name (or None for “current user”) Returns: (bool): True if the user is allowed delAddRows(iden, rows) Delete rows by iden and add rows as a single operation. Example: core.delAddRows( iden, rows ) Notes: This API is oriented toward “reparse” cases where iden will be the same for the new rows. delBlobValu(key) Remove and return a value from the blob store. Args: key (str): Key to remove. Returns: Content in the blob store for a given key. Raises: NoSuchName: If the key is not present in the store. delJoinByProp(prop, valu=None, mintime=None, maxtime=None) Delete a group of rows by selecting for property and joining on iden. Example: core.delJoinByProp(‘foo’,valu=10) delRoleRule(role, indx) Delete a rule at index for the given role. Args: role (str): The role name indx (int): The index of the rule to remove Returns: (None)

4.1. Subpackages 201 Synapse Documentation, Release 0.0.34

delRowsById(iden) Delete all the rows for a given iden. Example: core.delRowsById(iden) delRowsByIdProp(iden, prop, valu=None) Delete rows with the givin combination of iden and prop[=valu]. Example: core.delRowsByIdProp(id, ‘foo’) delRowsByProp(prop, valu=None, mintime=None, maxtime=None) Delete rows with a given prop[=valu]. Example: core.delRowsByProp(‘foo’,valu=10) delTufo(tufo) Delete a tufo and it’s associated props/lists/etc. Example: core.delTufo(foob) delTufoByProp(form, valu) Delete a tufo from the cortex by prop=valu. Example: core.delTufoByProp(‘foo’,’bar’) delTufoDark(tufo, name, valu=None) Remove dark rows from a tufo for a given name and optional value. Args: tufo ((str, dict)): Tufo to remove data dark rows from. name (str): Speciﬁc dark rows to remove. valu (str): Value to remove (optional). Returns: None: Returns None. delTufoDset(tufo, name) delTufoIval(tufo, name) Remove an iterval from a node. Args: tufo ((str,dict)): The node in tuple form name (str): The name of the interval Returns: ((str,dict)): The updated node in tuple form delTufoListValu(tufo, name, valu) delTufoProp(tufo, name) Delete a property from a node in tufo format. Args: tufo ((str,dict)): The node in tufo form name (str): The relative property name to delete Returns: ((str,dict)) The updated node in tufo form delTufoTag(tufo, tag) Delete a tag from a tufo. Args: tufo ((str,dict)): The node in tuple form. tag (str): The tag to remove Example: Remove the tag baz tag (and its subtags) from all tufos tagged baz.faz:

202 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

for tufo in core.getTufosByTag('baz.faz'): core.delTufoTag(tufo,'baz')

Remove a tag from a tufo and then do something with the tufo:

tufo= core.delTufoTag(tufo,'hehe.haha') dostuff(tufo)

Returns: ((str,dict)): The node in tuple form (with updated props) delTufosByProp(prop, valu=None) Delete multiple tufos by a single property. Example: core.delTufosByProp(‘foo:bar’,valu=10) delUserRule(user, indx) Delete a rule at index for the given user. Args: user (str): The user name indx (int): The index of the rule to remove Returns: (None) NOTE: see synapse.lib.auth.Rules for rule tuple definition eatSpliceFd(fd) Consume all cortex splice events from the given file-like object. The file bytes are expected to be msgpack encoded (str,dict) splice tuples. Example: fd = open(‘saved.mpk’,’rb’) core.eatSyncFd(fd) finiSnap(snap) Cancel a tufo snapshot. formNodeByBytes(byts, stor=True, **props) Form a new file:bytes node by passing bytes and optional props. If stor=False is specified, the cortex will create the file:bytes node even if it is not configured with access to an axon to store the bytes. Args: byts (bytes): The bytes for a file:bytes node stor (bool): If True, attempt to store the bytes in an axon **props: Additional props for the file:bytes node Example: core.formNodeByBytes(byts,name=’foo.exe’) formNodeByFd(fd, stor=True, **props) Form a new file:bytes node by passing a file object and optional props. Args: fd (file): A file-like object to read file:bytes from. stor (bool): If True, attempt to store the bytes in an axon **props: Additional props for the file:bytes node formTufoByProp(prop, valu, **props) Form an (iden,info) tuple by atomically deconflicting the existence of prop=valu and creating it if not present. Args: prop (str): The primary property (or form) of the node valu (obj): The primary valu for the node **props: Additional secondary properties for the node

4.1. Subpackages 203 Synapse Documentation, Release 0.0.34

Example: Make a node for the FQDN woot.com:

tufo= core.formTufoByProp('inet:fqdn','woot.com')

Notes: When forming nodes whose primary property is derived from the GuidType, deconfliction can be skipped if the value is set to None. This allows for high-speed ingest of event type data which does note require deconfliction. This API will fire a node:form event prior to creating rows, allowing callbacks to populate any additional properties on the node. After node creation is finished, node:add events are fired on for the Cortex event bus, splices and triggers. Returns: ((str, dict)): The newly formed tufo, or the existing tufo if the node already exists. The ephemeral property ”.new” can be checked to see if the node was newly created or not.

formTufoByTufo(tufo) Form an (iden,info) tufo by extracting information from an existing one. Args: tufo ((str, dict)): An existing tufo to form a new tufo from. Examples: Create an IPv4 node from an existing node:

t0=( None,{'inet:ipv4':0x01020304,'inet:ipv4:asn': 1024}) new_tufo= core.formTufoByTufo(t0)

Notes: This API uses the formTufoByProp API to form the new tufo; after extracting the form, primary property and sub properties from the input tufo. In addition, this API does not utilize the iden value present in the ﬁrst element of the tuple when making a new node. Returns: ((str, dict)): The new tufo, or an existing tufo if the tufo already existed. formTufosByProps(items) Forms tufos by prop, given a tuple of (form, valu, props) tuples. Args: items (tuple): A tuple of tuples of (form, valu, props) Returns: tuple: Tuple containing tufos, either with the node or error data Example: items = ( (‘foo:thing’, ‘hehe’, {‘a’: 1}), (‘bar:thing’, ‘haha’, {‘b’: 2}), ) results = core.formTufosByProps(items) getBlobKeys() Get a list of keys in the blob key/value store. Returns: list: List of keys in the store. getBlobValu(key, default=None) Get a value from the blob key/value (KV) store. This resides below the tufo storage layer and is Cortex implementation dependent. In purely memory backed cortexes, this KV store may not be persistent, even if the tufo-layer is persistent, through something such as the saveﬁle mechanism. Notes: Data which is retrieved from the KV store is msgpacked, so caveats with that apply. Args: key (str): Value to retrieve default: Value returned if the key is not present in the blob store. Returns: The value from the KV store or the default valu (None).

204 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

getCoreFifo(name) Return a Fifo object by name. Args: name (str): The :name of the syn:fifo node. Returns: (synapse.lib.fifo.Fifo): The Fifo object. getCorePath(*paths) Construct a path relative to the cortex metadata dir (or None). Args:* paths ([str,]): A set of path elements Returns: (str): The full path ( or None ). getCoreXact(size=1000) Get a Storage transaction context for use in a with block. This object allows bulk storage layer optimization and proper ordering of events. The context manager created through this function supports firing splice events. Args: size (int): Number of transactions to cache before starting to execute storage layer events. Examples: Get a context manager, use it to do stuff and fire splices:

with core.getCoreXact() as xact: result= dostuff() xact.spliced('some:slice:evt', **result)

Notes: This API does not work over a Telepath proxy object and it will raise an exception. Managing a transaction with from a remote caller is inherently difﬁcult since the transaction can be opened, the caller then go away, unfortunately leaving the system in a weird state. Returns: s_xact.StoreXact: Transaction context manager. getJoinByProp(prop, valu=None, mintime=None, maxtime=None, limit=None) Similar to getRowsByProp but also lifts all other rows for iden. Example: for row in core.getRowsByProp(‘foo’,valu=20): stuff(row) Notes: • See getRowsByProp for options getJsonItems(prop, valu=None, mintime=None, maxtime=None, limit=None) Return a list of (iden,item) tuples (similar to tufos, but with hierarchical structure ) Example: x = { ‘bar’:10, } core.addJsonItem(‘foo’,x) # ... later ... for prim in core.getJsonsByProp(‘foo:bar’, 10): dostuff(tufo) getModlVers(name) Retrieve the model version for the given model name. Args: name (str): The name of the model

4.1. Subpackages 205 Synapse Documentation, Release 0.0.34

Returns: (int): The model version ( linear version number ) or -1 getPermDef(name) Return a permission definition tuple for the given perm name. Args: name (str): The permission name Returns: (tuple): A (name,info,fields) tuple for the perm. getPivotRows(prop, byprop, valu=None, mintime=None, maxtime=None, limit=None) Similar to getRowsByProp but pivots through “iden” to a different property. This can be a light way to return a single property from a tufo rather than lifting the whole. Example: # return rows for the foo:bar prop by lifting foo:baz=10 and pivoting through iden. for row in core.getPivotRows(‘foo:bar’, ‘foo:baz’, valu=10): dostuff() getRoleAuth(role) Return an auth dict for the given role. Args: role (str): The role name Returns: (dict): The role auth dict getRoleRules(role) Get a list of rule tuples for the given role. Args: role (str): The role name Returns: (list): A list of rule tuples getRoleUsers(role) Get a list of the users for the specified role. Args: role (str): The role name Returns: ([str,...]): The users for the role getRowsBy(name, prop, valu, limit=None) Retrieve rows by a specialized index within the cortex. Example: rows = core.getRowsBy(‘range’,’foo:bar’,(20,30))

Notes: • most commonly used to facilitate range searches

getRowsById(iden) Return all the rows for a given iden. Example: for row in core.getRowsById(iden): stuff() getRowsByIdProp(iden, prop, valu=None) Return rows with the given ,. Example: for row in core.getRowsByIdProp(iden,’foo:bar’): dostuff(row)

206 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

getRowsByProp(prop, valu=None, mintime=None, maxtime=None, limit=None) Return a tuple of (iden,prop,valu,time) rows by prop[=valu]. Example: for row in core.getRowsByProp(‘foo’,valu=20): stuff(row) Notes: • Specify limit= to limit return size • Specify mintime= in epoch to filter rows • Specify maxtime= in epoch to filter rows getSeqNode(name) API helper/wrapper to form a syn:seq sequential id tracker getSizeBy(name, prop, valu, limit=None) Retrieve row count by a specialized index within the cortex. Example: size = core.getSizeBy(‘range’,’foo:bar’,(20,30)) print(‘there are %d rows where 20 <= foo < 30 ‘ % (size,)) getSizeByProp(prop, valu=None, mintime=None, maxtime=None) Return the count of matching rows by prop[=valu] Example: if core.getSizeByProp(‘foo’,valu=10) > 30: stuff() getSnapNext(snap) Get the next block of tufos for the given snapshot. getSplicePump(core) Return a splice pump for the remote cortex. #Example: with core.getSplicePump(prox): core.formTufoByProp(‘inet:fqdn’,’vertex.link’) getStatByProp(stat, prop, valu=None, mintime=None, maxtime=None, limit=None) Calculate and return a statistic for the specified rows. ( See getRowsByProp docs for most args ) Various statistics types are builtin. sum - total of all row values count - number of rows histo - histogram of values average - sum / count min - minimum value max - maximum value any - True if any value is true all - True if every value is true Example: minval = core.getStatByProp(‘min’,’foo:bar’) getStoreType() getTufoByIden(iden) Retrieve a tufo by id. Example: tufo = core.getTufoByIden(iden)

4.1. Subpackages 207 Synapse Documentation, Release 0.0.34

getTufoByProp(prop, valu=None) Return an (iden,info) tuple by joining rows based on a property. Example: tufo = core.getTufoByProp(‘fqdn’,’woot.com’) Notes: • This must be used only for rows added with formTufoByProp! getTufoDarkNames(tufo) Get a list of dark row names on a tufo. Args: tufo ((str, dict)): Tufo to look up. Returns: list: List of (name, time) tuples for a given tufos dark rows. getTufoDarkValus(tufo, name) Get a list of dark row values on a given tufo with a speciﬁc name. Args: tufo ((str, dict)): Tufo to look up. name (str): Speciﬁc dark rows to look up. Returns: list: List of (value, time) tuples for a given tufos dark rows. getTufoDsets(tufo) Return a list of (name,time) tuples for dset membership. getTufoList(tufo, name) Retrieve “list” values from a tufo. Example: for val in core.getTufoList(item,’foolist’): dostuff(val) getTufosBy(name, prop, valu, limit=None) Retrieve tufos by either a specialized method or the lower getRowsBy api Specialized methods will be dependant on the storage backing and the data indexed Example: tufos = core.getTufosBy(‘in’, ‘foo’, (47,3,8,22)) getTufosByDark(name, valu=None, mintime=None, maxtime=None, limit=None) Get a list of tufos with the named dark rows and optional values. Args: name (str): Dark row name to retrieve tufos by. valu (str): Value to retrieve. mintime (int): Mini- mum timevalue on tufos to return. maxtime (int): Maximum timevalue on tufos to return. limit (int): Maximum number of tufos to return. Examples: Get a list of tufos by a tag:

for tufo in getTufosByDark('tag','foo.bar.baz'): dostuff(tufo)

Returns: list: List of tufos getTufosByDset(name, mintime=None, maxtime=None, limit=None) Return a list of the tufos in the named dataset. Example: for tufo in getTufosByDset(‘woot’): dostuff(tufo)

208 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

getTufosByIdens(idens) Return a list of tufos for the given iden GUIDs. Exmple: tufos = core.getTufosByIdens(idens) getTufosByProp(prop, valu=None, mintime=None, maxtime=None, limit=None) Return a list of tufos by property. Example: for tufo in core.getTufosByProp(‘foo:bar’, 10): dostuff(tufo) getTufosByPropType(name, valu=None, mintime=None, maxtime=None, limit=None) Return tufos by interrogating the data model to find fields of the given type. Example: # return all tufos with an inet:email type property with value [email protected] for tufo in core.getTufosByPropType(‘inet:email’, valu=’[email protected]’): dostuff(tufo) getTufosByTag(tag, form=None, limit=None) Retrieve a list of tufos by tag and optionally form. Args: tag (str): A synapse tag name form (str): A synapse node form limit (int): A limit for the query Example: for node in core.getTufosByTag(‘foo.bar’): dostuff(tufo) getUserAuth(user) Return an auth dict for the given user. Args: user (str): The user name Returns: (dict): The user auth dict getUserRoles(user) Get a list of the roles for the specified user. Args: user (str): The user name Returns: ([str,...]): The roles for the user getUserRules(user) Get a list of rule tuples for the given user. Args: user (str): The user name Returns: (list): A list of rule tuples hasBlobValu(key) Check the blob store to see if a key is present. Args: key (str): Key to check Returns: bool: If the key is present, returns True, otherwise False. incTufoProp(tufo, prop, incval=1) Atomically increment/decrement the value of a given tufo property. Example: tufo = core.incTufoProp(tufo,prop)

4.1. Subpackages 209 Synapse Documentation, Release 0.0.34

initCoreModule(ctor, conf ) Load a cortex module with the given ctor and conf. Args: ctor (str): The python module class path conf (dict):Conﬁg dictionary for the module initTufosBy(name, meth) Initialize a “tufos by” handler for the Cortex. This is useful when the index or storage backing can optimize tufo creation from raw rows. Example: def getbyin(prop,valus,limit=None): ret = [] for valu in valus: res = self.getTufosByProp(prop, valu=valu, limit=limit) ret.extend(res) if limit != None: limit -= len(res) if limit <= 0: break return ret core.initTufos(‘in’,getbyin) Notes: • Used by Cortex implementers to facilitate getTufosBy(...) isOk() An API allowing Cortex to test for internal error states. isRuntForm(prop) Returns True if the given property name is a runtime node form. Args: prop (str): The property name Returns: (bool): True if the property is a runtime node form. isRuntProp(prop) Return True if the given property name is a runtime node prop. Args: prop (str): The property name Returns: (bool): True if the property is a runtime node property. isSetPropOk(prop, isadd=False) Check for enforcement and validity of a full prop name. This can be used to determine if a property name may be set on a node, given the data models currently loaded in a Cortex. Args: prop (str): Full property name to check. isadd (bool): Bool indicating that the property check is being done on a property which has not yet been set on a node. Examples: Check if a value is valid before calling a function.:

prop='foo:bar:baz' if core.isSetPropOk(prop): doSomething(...)

Returns: bool: True if the property can be set on the node; False if it cannot be set. nextSeqValu(name) Return the next sequence identiﬁer for the given name. Example: name = core.nextSeqValu(‘foo’) # name is now foo0 or fooN from sequence

210 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

popTufosByProp(prop, valu=None) Delete and return multiple tufos by a property. Example: items = core.popTufosByProp(‘foo:bar’,valu=10) putCoreFifo(name, item) Add an item to a cortex fifo. Args: name (str): The syn:fifo:name of the fifo. item (obj): The object to put in the fifo. reqCorePath(*paths) Use getCorePath and raise if dir is not set. Args: paths ([str,...]): A list of path elements to join. Returns: (str): The full path for the cortex directory. Raises: NoSuchOpt reqPermDef(name) Require (or raise) a permission definition tuple Args: name (str): The permission name Returns: (tuple): A (name,info,fields) tuple for the perm. Raises: (NoSuchPerm): The permission name was not found reqperm(perm, user=None) Require a given permission (or raise AuthDeny) Args: perm ((str,dict)): A perm tuple user (str): The user to check (or self) Raises: AuthDeny: The user is not allowed revModlVers(name, vers, func) Update and track a model version using a callback function. Args: name (str): The name of the model vers (int): The version int ( in YYYYMMDDHHMM int format ) func (function): The version update function Returns: (None) Each specified function is expected to update the cortex including data migration. setBlobValu(key, valu) Set a value from the blob key/value (KV) store. This resides below the tufo storage layer and is Cortex implementation dependent. In purely memory backed cortexes, this KV store may not be persistent, even if the tufo-layer is persistent, through something such as the savefile mechanism. Notes: Data which is stored in the KV store is msgpacked, so caveats with that apply. Args: key (str): Name of the value to store. valu: Value to store in the KV store. Returns: The input value, unchanged. setModlVers(name, vers) Set the version number for a specific model. Args: name (str): The name of the model vers (int): The new (linear) version number Returns: (None)

4.1. Subpackages 211 Synapse Documentation, Release 0.0.34

setRoleRules(role, rules) Set the rules for a given role. Rules are documented in synapse.lib.auth.Rules. Args: role (str): The role name rules (list): A list of rule tuples Returns: (None) Raises: (synapse.exc.BadRuleValu) setRowsByIdProp(iden, prop, valu) Update/insert the value of the row(s) with iden and prop to valu. Example: core.setRowsByIdProp(iden,’foo’,10) setSaveFd(fd, load=True, fini=False) Set a save fd for the cortex and optionally load. Args: fd (file): A file like object to save splice events to using msgpack load (bool): If True, load splice event from fd before starting to record fini (bool): If True, close() the fd automatically on cortex fini() Returns: (None) Example: core.setSaveFd(fd)

NOTE: This save ﬁle is allowed to be storage layer speciﬁc. If you want to store cortex splice events, use addSpliceFd().

setTufoIval(tufo, name, ival) Add an interval to a node. Args: tufo ((str,dict)): The node to add the interval to name (str): The name of the interval ival ((int,int)): The interval tuple Returns: ((str,dict)): The updated tufo setTufoProp(tufo, prop, valu) Set a single tufo property. Example: core.setTufoProp(tufo, ‘woot’, ‘hehe’) setTufoProps(tufo, **props) Set ( with de-duplication ) the given tufo props. Args: tufo ((str, dict)): The tufo to set properties on. **props: Properties to set on the tufo. Examples:

tufo= core.setTufoProps(tufo, woot='hehe', blah=10)

Returns: ((str, dict)): The source tufo, with any updated properties. setUserRules(user, rules) Set the rules for a given user. Args: user (str): The user name rules (list): The list of rule tuples Returns: (None)

212 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

snapTufosByDark(name, valu=None, mintime=None, maxtime=None, limit=None) Create a snapshot of tufos by dark name/values. Args: name (str): Dark row name to snapshot tufos by. valu (str): Optional value to retrieve tufos by. mintime (int): Minimum timevalue on tufos to return. maxtime (int): Maximum timevalue on tufos to return. limit (int): Maximum number of tufos to return. Returns: dict: Snapshot generator for getting tufos. snapTufosByDset(name, mintime=None, maxtime=None, limit=None) snapTufosByProp(prop, valu=None, mintime=None, maxtime=None, limit=None) splice(mesg) Feed the cortex a splice event to make changes to the hypergraph. Args: mesg ((str,dict)): The (name,info) for the splice event. Returns: None Example: core.splice(mesg) splices(msgs) Process a list of splices in bulk (see splice()). Args: msgs (list): The list of splices. subCoreFifo(name, xmit=None) Provde an xmit function for a given core fifo. Args: name (str): The name of the fifo. xmit (func): A fifo xmit func. NOTE: if xmit is None, it is assumed that the caller is a remote telepath client and the socket.tx function is used. synapse.cores.common.chunked(n, iterable) synapse.cores.common.reqiden(tufo) Raise an exception if the given tufo is ephemeral. synapse.cores.lmdb module class synapse.cores.lmdb.LmdbStorage(link, **conf ) Bases: synapse.cores.storage.Storage getRowsById(iden) getRowsByIdProp(iden, prop, valu=None) getRowsByProp(prop, valu=None, limit=None, mintime=None, maxtime=None, do_count_only=False) getSizeByProp(prop, valu=None, limit=None, mintime=None, maxtime=None) getStoreType() getStoreXact(size=None, core=None) rowsByGe(prop, valu, limit=None) rowsByLe(prop, valu, limit=None) rowsByRange(prop, valu, limit=None)

4.1. Subpackages 213 Synapse Documentation, Release 0.0.34

sizeByGe(prop, valu, limit=None) sizeByLe(prop, valu, limit=None) sizeByLt(prop, valu, limit=None) sizeByRange(prop, valu, limit=None) class synapse.cores.lmdb.LmdbXact(store, size=None, core=None) Bases: synapse.cores.xact.StoreXact synapse.cores.lmdb.initLmdbCortex(link, conf=None, storconf=None) Initialize a LMDB based Cortex from a link tufo. Args: link ((str, dict)): Link tufo. conf (dict): Configable opts for the Cortex object. storconf (dict): Configable opts for the storage object. Returns: s_cores_common.Cortex: Cortex created from the link tufo. synapse.cores.postgres module class synapse.cores.postgres.PsqlStorage(link, **conf ) Bases: synapse.cores.sqlite.SqliteStorage dblim = None getRowsByIdens(idens) getStoreType() synapse.cores.postgres.initPsqlCortex(link, conf=None, storconf=None) Initialize a Sqlite based Cortex from a link tufo. Args: link ((str, dict)): Link tufo. conf (dict): Configable opts for the Cortex object. storconf (dict): Configable opts for the storage object. Returns: s_cores_common.Cortex: Cortex created from the link tufo. synapse.cores.postgres.md5(x) synapse.cores.ram module class synapse.cores.ram.RamStorage(link, **conf ) Bases: synapse.cores.storage.Storage getRowsById(iden) getRowsByIdProp(iden, prop, valu=None) getRowsByIdens(idens) getRowsByProp(prop, valu=None, mintime=None, maxtime=None, limit=None) getSizeByProp(prop, valu=None, mintime=None, maxtime=None) getStoreType() getStoreXact(size=None, core=None) rowsByGe(prop, valu, limit=None) rowsByLe(prop, valu, limit=None) rowsByRange(prop, valu, limit=None)

214 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

sizeByGe(prop, valu, limit=None) sizeByLe(prop, valu, limit=None) sizeByRange(prop, valu, limit=None) class synapse.cores.ram.RamXact(store, size=None, core=None) Bases: synapse.cores.xact.StoreXact synapse.cores.ram.initRamCortex(link, conf=None, storconf=None) Initialize a RAM based Cortex from a link tufo. The path element of the link tufo, if present, is used to cache the Cortex instance. Subsequent calls with the same path will return the existing Cortex instance. Args: link ((str, dict)): Link tufo. conf (dict): Conﬁgable opts for the Cortex object. storconf (dict): Conﬁgable opts for the storage object. Returns: s_cores_common.Cortex: Cortex created from the link tufo. synapse.cores.sqlite module class synapse.cores.sqlite.DbPool(size, ctor) Bases: object The DbPool allows generic db connection pooling using a factory/ctor method and a python queue. Example: def connectdb(): # do stuff return db pool = DbPool(3, connectdb) get() put(db) Add/Return a db connection to the pool. class synapse.cores.sqlite.SqlXact(store, size=None, core=None) Bases: synapse.cores.xact.StoreXact class synapse.cores.sqlite.SqliteStorage(link, **conf ) Bases: synapse.cores.storage.Storage dblim = -1 delete(q, **args) getJoinByProp(prop, valu=None, mintime=None, maxtime=None, limit=None) getRowsById(iden) getRowsByIdProp(iden, prop, valu=None) getRowsByProp(prop, valu=None, limit=None, mintime=None, maxtime=None) getSizeByProp(prop, valu=None, limit=None, mintime=None, maxtime=None) getStoreType() getStoreXact(size=None, core=None) rowsByGe(prop, valu, limit=None) rowsByLe(prop, valu, limit=None) rowsByRange(prop, valu, limit=None)

4.1. Subpackages 215 Synapse Documentation, Release 0.0.34

select(q, **args) sizeByGe(prop, valu, limit=None) sizeByLe(prop, valu, limit=None) sizeByRange(prop, valu, limit=None) update(q, **args) synapse.cores.sqlite.initSqliteCortex(link, conf=None, storconf=None) Initialize a Sqlite based Cortex from a link tufo. Args: link ((str, dict)): Link tufo. conf (dict): Configable opts for the Cortex object. storconf (dict): Configable opts for the storage object. Returns: s_cores_common.Cortex: Cortex created from the link tufo. synapse.cores.storage module synapse - storage.py Created on 7/19/17. Base storage layer implementation for Synapse Cortex class. See Storage class for more information. class synapse.cores.storage.Storage(link, **conf ) Bases: synapse.lib.config.Config Base class for Cortex storage layer backends. This implements functionality which is needed for a storage layer to operate, as well as providing stubs for the storage layer implementations to override. See the Synapse Documentation for more details. Args: link ((str, dict)): Link tufo containing information for creating the Storage object. This may include path information, authentication information, etc. **conf (dict): Additional configible options for the storage layer. addRows(rows) Add (iden, prop, valu, time) rows to the Storage object. Args: rows (list): List of rows containing (i, p, v, t) tuples. Examples: Adding a pair of rows to the storage object:

import time tick= now()

rows=[ (id1,'baz',30,tick), (id1,'foo','bar',tick), ]

store.addRows(rows)

Notes: The general convention for the iden value is a 16 byte hex string, such as “e190d108bdd30a035a15764313f4c397”. These can be made with the synapse.common.guid() function. While the storage layer is free to STORE these idens however it sees ﬁt, some tools may expect that, at the public row level APIs, idens may conform to that shape. If other types of idens are put into the system, that could cause unintentional issues.

216 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

This does ﬁre a “core:save:add:rows” event on the savebus to save the raw rows which are being send to the storage layer. Returns: None addSaveLink(func) Add an event callback to receive save events for this Storage object. Args: func: Function to receive events from the Storage savebus. Examples: Register a function to receive events:

def savemesg(mesg): dostuff()

core.addSaveLink(savemesg)

Returns: None delBlobValu(key) Remove and return a value from the blob store. Args: key (str): Key to remove. Returns: Content in the blob store for a given key. Raises: NoSuchName: If the key is not present in the store. delRowsById(iden) Delete all the rows for a given iden. Args: iden (str): Iden to delete rows for. Examples: Delete the rows for a given iden:

store.delRowsById(iden)

Notes: This does ﬁre a “core:save:del:rows:by:iden” event on the savebus to record which rows were deleted. Returns: None delRowsByIdProp(iden, prop, valu=None) Delete rows with the given combination of iden and prop[=valu]. Args: iden (str): Iden to delete rows for. prop (str): Prop to delete rows for. valu: Optional value to check. If present, only delete iden/prop rows with this value. Examples: Delete all ‘foo’ rows for a given iden:

store.delRowsByIdProp(iden,'foo')

Notes: This does ﬁre a “core:save:del:rows:by:idprop” event on the savebus to record which rows were deleted. Returns: None delRowsByProp(prop, valu=None, mintime=None, maxtime=None) Delete rows with a given property (and optional valu) combination. Args: prop (str): Property to delete. valu: Optional value to constrain the property deletion by. mintime (int): Optional, minimum time in which to constrain the

4.1. Subpackages 217 Synapse Documentation, Release 0.0.34

deletion by.

maxtime (int): Optional, maximum time in which to constrain the deletion by.

Examples: Delete all ‘foo’ rows with the valu=10:

store.delRowsByProp('foo',valu=10)

Notes: This does ﬁre a “core:save:del:rows:by:prop” event on the savebus to record which rows were deleted. Returns: None genJoinByProp(prop, valu=None, mintime=None, maxtime=None, limit=None) genStoreRows(**kwargs) A generator which yields raw rows from the storage layer. Args: **kwargs: Arguments which are passed to the storage layer implementation of _gen- StoreRows(). Notes: Since this is intended for use as a backup mechanism for a Storage object, it is not to be considered a performant API. Yields: list: List of rows. The number of rows and contents will vary by implementation. getBlobKeys() Get a list of keys in the blob key/value store. Returns: list: List of keys in the store. getBlobValu(key, default=None) Get a value from the blob key/value (KV) store. This resides below the tufo storage layer and is Cortex implementation dependent. In purely memory backed cortexes, this KV store may not be persistent, even if the tufo-layer is persistent, through something such as the saveﬁle mechanism. Notes: Data which is retrieved from the KV store is msgpacked, so caveats with that apply. Args: key (str): Value to retrieve default: Value returned if the key is not present in the blob store. Returns: The value from the KV store or the default valu (None). getCoreXact(size=1000, core=None) Get a Storage transaction context for use in a with block. This object allows bulk storage layer optimization and proper ordering of events. Args: size (int): Number of transactions to cache before starting to execute storage layer events. core: Cortex to attach to the StoreXact. Required for splice event support. Examples: Get a StoreXact object and use it:

with store.getCoreXact() as xact: store.dostuff()

Returns: s_xact.StoreXact: A StoreXact object for the current thread. getJoinByProp(prop, valu=None, mintime=None, maxtime=None, limit=None)

218 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

getJoinsBy(name, prop, valu, limit=None) Retrieve joined rows by either a sepeciﬁed method or by falling back to the rowsBy handlers. Specialized methods will be dependent on the storage backind and the data indexed. Args: name (str): Name of the method to look up. prop (str): Prop to ﬁlter by. valu: Value (or values) to pass to the helper method. limit (int): Limit on the join. Limit meaning may vary by implementation or named helper. Returns: getRowsById(iden) Return all the rows for a given iden. Args: iden (str): Iden to get rows from the storage object for. Examples: Getting rows by iden and doing stuff:

for row in store.getRowsById(iden): stuff()

Getting rows by iden and making a tufo out of them:

rows= store.getRowsById(iden) tufo= (iden, {p: v for (i, p, v, t) in rows})

Returns: list: List of rows for a given iden. getRowsByIdProp(iden, prop, valu=None) Return rows with the given ,. Args: iden (str): Iden to get rows from the storage object for. prop (str): Prop to constrain the lift by. valu: Optional, value to constrain the lift by. Examples: Getting rows by iden, prop and doing stuff:

for row in core.getRowsByIdProp(iden,'foo:bar'): dostuff(row)

Returns: list: List of rows for a given iden, prop, value comtination. getRowsByIdens(idens) Return all the rows for a given list of idens. Args: idens (list): Idens to get rows from the storage object for. Examples: Getting rows by idens and doing stuff:

for row in store.getRowsByIdens(idens): stuff(row)

Getting rows by idens and making a tufos out of them:

rows= store.getRowsById(iden) tufos= s_common.rowstotufos(rows)

Returns: list: List of rows for the given idens. getRowsByProp(prop, valu=None, mintime=None, maxtime=None, limit=None) Get rows from the Storage layer based on their property value and other, optional, constraints.

4.1. Subpackages 219 Synapse Documentation, Release 0.0.34

Args: prop (str): Property to retrieve rows based on. valu: Optional, str or integer value to constrain the retrieval by. mintime (int): Optional, minimum (inclusive) time to constrain the retrieval by. maxtime (int): Optiona, maximum (exclusive) time to constrain the retrieval by. limit (int): Maximum number of rows to return. Returns: list: List of (i, p, v, t) rows. getSizeByProp(prop, valu=None, mintime=None, maxtime=None) getStoreType() Get the Store type. This may be used by the Cortex to determine what its backing store is. Returns: str: String indicating what the backing storage layer is. getStoreXact(size=None, core=None) Get a StoreXact object. This is normally called by the getCoreXact function. Args: size (int): Number of events to cache in the transaction before executing them. core: Cortex to attach to the transaction. Required for splice event support. Returns: s_xact.StoreXact: A storage layer speciﬁc StoreXact object. hasBlobValu(key) Check the blob store to see if a key is present. Args: key (str): Key to check Returns: bool: If the key is present, returns True, otherwise False. initJoinsBy(name, meth) Initialize a “joins by” handler for the Storage layer. These helpers are used by the Cortex to do special types of lifts. This allows different Storage layers to implement certain lifts in a optimized fashion. Args: name (str): Named handler to register. meth (func): Function to register. Examples: Registering a ‘woot’ handler:

def getbywoot(prop,valu,limit=None): return stuff() # list of rows

core.initJoinsBy('woot',getbywoot)

Returns: None initRowsBy(name, meth) Initialize a “rows by” handler for the Storage layer. These helpers are used by the Cortex to do special types of lifts. This allows different Storage layers to implement certain lifts in a optimized fashion. Args: name (str): Named handler to register. meth (func): Function to register. Examples: Registering a ‘woot’ handler:

220 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

def getbywoot(prop,valu,limit=None): return stuff() # list of rows

core.initRowsBy('woot',getbywoot)

Returns: None initSizeBy(name, meth) Initialize a “size by” handler for the Storage layer. These helpers are used by the Cortex to do size by lifts. This allows different Storage layers to implement certain lifts in a optimized fashion. Args: name (str): Named handler to register. meth (func): Function to register. Examples: Registering a ‘woot’ handler:

def sizebywoot(prop,valu,limit=None): return stuff() # size of rows

core.initSizeBy('woot',meth)

Returns: None reqJoinByMeth(name) Get a handler for a JoinBy lift. Args: name (str): Name of the registered handler to retrieve. Returns: Function used to lift joined rows. Raises: NoSuchGetBy: If the named handler does not exist. reqRowsByMeth(name) Get a handler for a RowsBy lift. Args: name (str): Name of the registered handler to retrieve. Returns: Function used to lift by rows. Raises: NoSuchGetBy: If the named handler does not exist. reqSizeByMeth(name) Get a handler for a SizeBy lift. Args: name (str): Name of the registered handler to retrieve. Returns: Function used to lift by size. Raises: NoSuchGetBy: If the named handler does not exist. rowsByGe(prop, valu, limit=None) rowsByGt(prop, valu, limit=None) rowsByLe(prop, valu, limit=None) rowsByLt(prop, valu, limit=None) rowsByRange(prop, valu, limit=None) setBlobValu(key, valu) Set a value from the blob key/value (KV) store.

4.1. Subpackages 221 Synapse Documentation, Release 0.0.34

This resides below the tufo storage layer and is Cortex implementation dependent. In purely memory backed cortexes, this KV store may not be persistent, even if the tufo-layer is persistent, through something such as the savefile mechanism. Notes: Data which is stored in the KV store is msgpacked, so caveats with that apply. Args: key (str): Name of the value to store. valu: Value to store in the KV store. Returns: The input value, unchanged. setModlVers(name, vers) Set the version number for a specific model. Args: name (str): The name of the model vers (int): The new (linear) version number Returns: (None) setRowsByIdProp(iden, prop, valu) Update or insert the value of the row(s) with iden and prop to valu. Args: iden (str): Iden to update. prop (str): Property to update. valu: Value to set. Examples: Set the foo=10 value on a given iden: Notes: This does fire a “core:save:set:rows:by:idprop” event on the savebus to save the changes which are being sent to the storage layer. Returns: None setSaveFd(fd, load=True, fini=False) Set a save fd for the cortex and optionally load. Args: fd (file): A file like object to save splice events to using msgpack load (bool): If True, load splice event from fd before starting to record fini (bool): If True, close() the fd automatically on cortex fini() Returns: (None) Example: store.setSaveFd(fd)

NOTE: This save ﬁle is allowed to be storage layer speciﬁc. If you want to store cortex splice events, use addSpliceFd() from the Cortex class.

sizeByGe(prop, valu, limit=None) sizeByLe(prop, valu, limit=None) sizeByRange(prop, valu, limit=None) updateProperty(oldprop, newprop) Do a wholesale replacement of one property with another property. Args: oldprop (str): The orginal property which is removed. newprop (str): The property that is updated in place. Examples: Rename “inet:tcp4:port” to “inet:tcp4:foobar”:

nrows= store.updateProperty('inet:tcp4:port','inet:tcp4:foobar')

Notes: This API does ﬁre syn:core:store:up:prop:pre and syn:core:store:up:prop:post events with the old and new property names in it, before and after the property update is done. This API is primarily designed for assisting with Cortex data migrations.

222 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

Returns: int: Number of rows updated in place. updatePropertyValu(prop, oldval, newval) Do a wholesale update of one property=valu combination with a new valu. Args: prop (str): Property to select by for updating. oldval: Old valu to select rows by. newval: Valu to set the the prop=oldval rows to be. Examples: Rename “tufo:form”=”inet:tcp4” to instead be “tufo:form”=”inet:tcp4000”:

nrows= store.updatePropertyValu('tufo:form','inet:tcp4','inet:tcp4000

˓→')

Notes: This API does fire syn:core:store:up:propval:pre and syn:core:store:up:propval:post events with the property, old value and new values in it; before and after update is done. This API is primarily designed for assisting with Cortex data migrations. The oldval and newval must be of the same type. Returns: int: Number of rows updated in place. synapse.cores.xact module synapse - xact.py.py Created on 8/1/17. StoreXact implementation. This is designed to be subclassed by Storage layer implementors. class synapse.cores.xact.StoreXact(store, size=None, core=None) Bases: object A context manager for a storage “transaction”. acquire() begin() cedetime() commit() Commit the results thus far ( without closing / releasing ) fire(name, **props) Pend an event to fire when the transaction next commits. fireall() release() spliced(act, **info) Fire a splice event from the transaction. Args: act (str): Splice action. **info: Event values. Returns: None sync() Loop commiting and syncing events until there are no more events that need to fire. trigger(node, name, **info) Fire a trigger from the transaction Args: node ((str,dict)): The node for the trigger name (str): The trigger permission string info (dict): The trigger permission metadata

4.1. Subpackages 223 Synapse Documentation, Release 0.0.34

Module contents

4.1.3 synapse.data package

Module contents synapse.data.get(name, defval=None) Return an object from the embedded synapse data folder. Example: for tld in syanpse.data.get(‘iana.tlds’): dostuff(tld) NOTE: Files are named synapse/data/.mpk

4.1.4 synapse.docker package

Module contents

4.1.5 synapse.lib package

Subpackages synapse.lib.platforms package

Submodules synapse.lib.platforms.common module synapse.lib.platforms.common.daemonize() For unix platforms, form a new process group using fork(). synapse.lib.platforms.common.getLibC() Return a ctypes reference to libc synapse.lib.platforms.common.getVolInfo(*paths) Retrieve volume usage info for the given path. synapse.lib.platforms.common.inet_ntop(afam, byts) synapse.lib.platforms.common.inet_pton(afam, text) synapse.lib.platforms.common.initHostInfo() synapse.lib.platforms.common.setProcName(name) Set the process title/name for process listing. synapse.lib.platforms.darwin module synapse.lib.platforms.darwin.initHostInfo() synapse.lib.platforms.freebsd module synapse.lib.platforms.freebsd.initHostInfo()

224 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

synapse.lib.platforms.linux module

synapse.lib.platforms.linux.initHostInfo()

synapse.lib.platforms.windows module

synapse.lib.platforms.windows.daemonize() synapse.lib.platforms.windows.getLibC() Override to account for python on windows not being able to ﬁnd libc sometimes... synapse.lib.platforms.windows.initHostInfo() class synapse.lib.platforms.windows.sockaddr Bases: _ctypes.Structure ipv4 Structure/Union member ipv6 Structure/Union member sa_family Structure/Union member

Module contents

Home for platform speciﬁc code such as thishost info. all platform modules must be importable from any platform. ( guard any platform speciﬁc code with appropriate conditionals )

Submodules synapse.lib.atomfile module class synapse.lib.atomfile.AtomFile(fd, **opts) Bases: synapse.eventbus.EventBus Implement generic atomic “seek and read” behavior which some platforms override. flush() Request that all changes are flushed to disk. Example: atom.flush() readoff(off, size) Atomically read size bytes at the given offset. Example: byts = atom.readoff(off,size)

4.1. Subpackages 225 Synapse Documentation, Release 0.0.34

resize(size) Resize the underlying file (currently only supports growth). Example: atom.resize(newsize) writeoff(off, byts) Atomically write bytes at the given offset. class synapse.lib.atomfile.FastAtom(fd, **opts) Bases: synapse.lib.atomfile.AtomFile An AtomFile which uses pread/pwrite to avoid locking. class synapse.lib.atomfile.MemAtom(fd, **opts) Bases: synapse.lib.atomfile.AtomFile An AtomFile which uses a contiguous memory map for high-speed file IO. synapse.lib.atomfile.getAtomFile(fd, memok=True) Factory to construct the most optimal AtomFile for this platform. Example: atom = getAtomFile(fd) # provides thread safe routines for most optimal # offset based file I/O for this platform. # read 20 bytes at offset 300 byts = atom.readoff(300,20) # write byts at offset 400 atom.writeoff(400,byts) synapse.lib.atomfile.openAtomFile(path, memok=True) Open the given file path as an AtomFile. Args: path (str): A file path memok (bool): If True, allow use of mmap files. synapse.lib.atomic module class synapse.lib.atomic.CmpSet(valu) Bases: object The CmpSet class facilitates atomic compare and set. set(valu) Atomically set the valu and return change status. Args: valu (obj): The new value Returns: (bool): True if the value changed. synapse.lib.auth module class synapse.lib.auth.Rules(rules) Bases: object Rules provides an abstraction for metadata based filtration of events and tufos. Each “rule” is a tuple of: (allow, perm): (bool, (str, dict)) allow(perm) Returns True if the given perm/info is allowed by the rules.

226 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

Args: perm ((str,dict)): The requested permission tuple Returns: (bool): True if the rules allow the perm/info synapse.lib.auth.runas(user) Construct and return a with-block object which runs as the given synapse user name. Example: import synapse.lib.auth as s_auth s_auth.runas(‘visi@localhost‘): # calls from here down may use check user/perms dostuff() synapse.lib.auth.whoami() Return the name of the current synapse user for this thread. Example: name = s_auth.whoami() synapse.lib.cache module class synapse.lib.cache.Cache(maxtime=None, onmiss=None) Bases: synapse.eventbus.EventBus A callback driven cache with options for timeout / maxsize. Example: cache = Cache(maxtime=60) cache.put(iden,thing) # ... some time later valu = cache.get(iden) Notes: The maxtime option is used to specify cache time based flushing. Cache accesses continue to bump a timestamp forward to facilite flushing the cache entries which were least recently requested. clear() Flush and clear the entire cache. flush(key) Flush a key:val within the cache. Example: cache.flush(‘woot’) Notes: • Mostly used to trigger “cache:flush” events get(key) Return a val from the cache. Example: val = cache.get(key) has(key) Checks if the given key is in the cache (mostly used for testing) Args: key (str): The key being checked in the cache Returns: (bool): True if the key is in the cache

4.1. Subpackages 227 Synapse Documentation, Release 0.0.34

keys() Return a list of the keys in the cache. Example: for key in cache.keys(): stuff(key) pop(key) Remove and return a val from the cache. Example: cache.pop(‘woot’) put(key, val) Put a key:val into the cache. Example: cache.put(‘woot’,10) setMaxTime(valu) setOnMiss(onmiss) Set a callback function to use on cache miss. Example: def onmiss(key): return stuff.get(key) cache.setOnMiss( onmiss ) values() class synapse.lib.cache.FixedCache(maxsize=10000, onmiss=None) Bases: synapse.eventbus.EventBus Implements a fixed-size cache. For implementation speed, the cache will flush oldest values first regardless of last cache hit. clear() Remove all entries from the FixedCache. get(key) Return the value from the cache. If onmiss is set, lookup entry on cache miss and add to cache. Example: valu = cache.get(‘foo’) if valu != None: dostuff(valu) class synapse.lib.cache.KeyCache(lookmeth) Bases: collections.defaultdict A fast key/val lookup cache. Example: cache = KeyCache( getFooThing ) valu = cache[x] get(key) pop(key) put(key, val)

228 Chapter 4. synapse package Synapse Documentation, Release 0.0.34 class synapse.lib.cache.MatchCache Bases: synapse.lib.cache.Cache The MatchCache allows caching fnmatch results for perf. match(valu, must) Check if a given value matches an fnmatch pattern Args: valu (str): The string being checked must (str): The fnmatch pattern to check against Returns: (bool): True if valu matches must class synapse.lib.cache.OnDem Bases: collections.defaultdict A dictionary based caching on-demand resolver. Example: class Woot(OnDem): @keymeth(‘foo’) def _getFooThing(self): # only called once return FooThing() woot = Woot() foo = woot.get(‘foo’) add(name, func, *args, **kwargs) Add a key lookup function callback to the OnDem dict. Example: def getfoo(): return FooThing() od = OnDem() od.add(‘foo’, getfoo) foo = od.get(‘foo’) get(name) Return the value for the given OnDem key. Example: woot = od.get(‘woot’) class synapse.lib.cache.RefDict Bases: synapse.eventbus.EventBus Allow reference counted ( and instance folded ) cache. bumpref(key, val=True) Increment and decrement the given ref to potentially trigger events. This API will be most commonly used by leaf references who only seek to trigger potential events. Args: key (str): The key to reference val (obj): The value for the key Returns: (None) clear() count(key) Returns the number of references to the speciﬁed key. Args: key (str): The key to check for references Returns: (int): The number of references

4.1. Subpackages 229 Synapse Documentation, Release 0.0.34

get(key) holdref(key, val=True) Provides with-block syntax for holding a reference increment. Args: key (str): The key to reference val (obj): The value for the key Returns: contextmanager Example: refd = RefDict() with refd.holdref(‘foo’,’bar’): dostuff() pop(key) pops(keys) put(key, val=True) puts(items) class synapse.lib.cache.TufoCache(core, maxtime=None) Bases: synapse.lib.cache.Cache class synapse.lib.cache.TufoPropCache(core, prop, maxtime=None) Bases: synapse.lib.cache.TufoCache getTufoByValu(valu) synapse.lib.cache.keymeth(name) Decorator for use with OnDem to add key callback methods. synapse.lib.certdir module class synapse.lib.certdir.CertDir(path=None) Bases: object genCaCert(name, signas=None, outp=None) genHostCert(name, signas=None, outp=None, pkey=None, sans=None) genHostCsr(name, outp=None) genUserCert(name, signas=None, outp=None, pkey=None) genUserCsr(name, outp=None) getCaCert(name) getCaCertPath(name) getCaKey(name) getCaKeyPath(name) getClientCert(name) Loads the PKCS12 object for a given client certificate. Example: mypkcs12 = cdir.getClientCert(‘mycert’) Args: name (str): The name of the client certificate. Returns: OpenSSL.crypto.PKCS12: The certificate if exists.

230 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

getClientCertPath(name) Gets the path to a client certificate. Example: mypath = cdir.getClientCertPath(‘mycert’) Args: name (str): The name of the client certificate. Returns: str: The path if exists. getHostCaPath(name) getHostCert(name) getHostCertPath(name) getHostKey(name) getHostKeyPath(name) getPathJoin(*paths) getUserCaPath(name) getUserCert(name) getUserCertPath(name) getUserForHost(user, host) getUserKey(name) getUserKeyPath(name) isCaCert(name) isClientCert(name) Checks if a client certificate exists. Example: exists = cdir.isClientCert(‘mycert’) Args: name (str): The name of the client certificate. Returns: bool: True if the certificate is present, False otherwise. isHostCert(name) isUserCert(name) selfSignCert(cert, pkey) signCertAs(cert, signas) signHostCsr(xcsr, signas, outp=None, sans=None) signUserCsr(xcsr, signas, outp=None) synapse.lib.certdir.iterFqdnUp(fqdn) synapse.lib.cli module class synapse.lib.cli.Cli(item, outp=None, **locs) Bases: synapse.eventbus.EventBus A modular / event-driven CLI base object. addCmdClass(ctor, **opts) Add a Cmd subclass to this cli.

4.1. Subpackages 231 Synapse Documentation, Release 0.0.34

get(name, defval=None) getCmdByName(name) Return a Cmd instance by name. getCmdNames() Return a list of all the known command names for the CLI. printf(mesg, addnl=True) runCmdLine(line) Run a single command line. Example: cli.runCmdLine(‘woot –help’) runCmdLoop() Run commands from stdin until close or ﬁni(). set(name, valu) exception synapse.lib.cli.CliFini Bases: Exception class synapse.lib.cli.Cmd(cli, **opts) Bases: object Base class for modular commands in the synapse CLI. FIXME: document the _cmd_syntax deﬁnitions. getCmdBrief() Return the single-line description for this command. getCmdDoc() Return the help/doc output for this command. getCmdItem() Get a reference to the object we are commanding. getCmdName() getCmdOpts(text) Use the _cmd_syntax def to split/parse/normalize the cmd line. NOTE: This is implemented indepedent of argparse (et.al) due to the need for syntax aware argument splitting. ( also, allows different split per command type ) printf(mesg, addnl=True) runCmdLine(line) Run a line of command input for this command. Example: foo.runCmdLine(‘foo –opt baz woot.com’) class synapse.lib.cli.CmdHelp(cli, **opts) Bases: synapse.lib.cli.Cmd List commands and display help output. Example: help foocmd

232 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

runCmdOpts(opts) class synapse.lib.cli.CmdQuit(cli, **opts) Bases: synapse.lib.cli.Cmd Quit the current command line interpreter. Example: quit runCmdOpts(opts) synapse.lib.cli.get_input(text) Get input from a user via stdin. Notes: This is just a wrapper for input() so mocking does not have to replace builtin functions for testing runCmdLoop. Args: text (str): Text displayed prior to the input prompt. Returns: str: String of text from the user. synapse.lib.cmdr module synapse.lib.cmdr.getItemCmdr(item, outp=None, **opts) Construct and return a cmdr for the given item. Example: cmdr = getItemCmdr(foo) synapse.lib.cmdr.runItemCmdr(item, outp=None, **opts) Create a cmdr for the given item and run the cmd loop. Example: runItemCmdr(foo) synapse.lib.conﬁg module

Tools for providing a central API for configurable objects within Synapse. class synapse.lib.config.Config(opts=None, defs=()) Bases: synapse.lib.config.Configable, synapse.eventbus.EventBus A EventBus classs which has the Configable mixin already added. class synapse.lib.config.Configable(opts=None, defs=()) Bases: object Config object base mixin to allow addition to objects which already inherit from the EventBus. addConfDef(name, **info) Add a configable option to the object. Args: name (str): Name of configuration option to set. **info: A list of options for the Configable option. The following options have specific meanings: • type: A Synapse type which is used to normalize the value. • doc: A docstring (used for doing automatic document generation within Synapse)

4.1. Subpackages 233 Synapse Documentation, Release 0.0.34

• defval: A default value for the option. This must be copyable using copy.deepcopy(). This is done to avoid mutable default values from being changed. • asloc: A string, if present, will set the a local object attribute to the name of the string which is equal to the valu of the conﬁguration option. Examples: Add a deﬁnition for a option to a object with a local attribute:

item.addConfDef('cache:expiretime', type='int', doc='Time to expire data', defval=60, asloc='cache_expiretime')

Add a untype option deﬁnition to a object with a mutable defval. This sets the value to a copy of the defval, leaving the default object untouched:

item.addConfDef('foobars', defval=[], doc='A list of foobars we care

˓→about.')

Notes: This does not have to be explicitly called if the @confdef decorator is used to define the options for a class. Returns: None addConfDefs(defs) Add multiple configuration definitions for this object via addConfDef(). Args: defs ((str, dict),): A tuple containing multiple configuration definitions. Examples: Set a pair of options definitions related to caching on a object:

defs=( ('caching',{'type':'bool','doc':'Enable caching on this object'}

˓→), ('cache:expiretime',{'type':'int','doc':'Time to expire data',

˓→'defval': 60}) ) item.addConfDefs(defs)

Notes: This does not have to be explicitly called if the @confdef decorator is used to define the options for a class. Returns: None getConfDef(name) Get the defitition for a given Args: name (str): Name to get the definition of. Returns: dict: Dictionary containing the configuration definition for the given named option. Raises: NoSuchOpt: If the name is not a valid option. getConfDefs() Get the configuration definitions for this object. Returns: dict: Dictionary of option, values for the object. getConfNorm(name, valu) Return a normalized version of valu based on type knowledge for name. Args: name (str): The name of the config option valu (obj): The valu of the config option Returns: (obj): The normalized form for valu

234 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

getConfOpt(name) Get the configured value for a given option. Args: name (str): Name of the option to retrieve. Returns: Value stored in the configuration dict, or None if the name is not present. getConfOpts() Get the current configuration for this object. Returns: dict: Dictionary of option and configured value for the object. onConfOptSet(name, func) Utility function for dynamically handling updates to config options. Args: name (str): Option name to respond too. func: Function to execute. This should take one parameter, the changed value. Examples: React to a (arbitrary) cache configuration option change:

def setCacheEnabled(self, en): dostuff() item.onConfOptSet('caching', setCacheEnabled)

Notes: The callback is fired using the syn:conf:set: event. Many places through Synapse (or third party applications which use Synapse) may set multiple opts at once using the setConfOpts() API, typically during a objects __init__ function. This sets values in a random order, due to dictionary iteration; and relying on other options being set during these callbacks can cause race conditions, leading to differences between expected and observed behaviors. In order to ensure that callbacks are free of such race conditions, do not write callback functions which rely on other Configable options being set to a particular value. Returns: None reqConfOk(opts) Check that that config values pass validation or raise. Args: opts (dict): Dictionary containing name, valu pairs to validate. Raises: NoSuchType: If the specified type of the option is non-existent. BadTypeValu: If a bad valu is encountered during type normalization. reqConfOpts() Check for the presense of required config options and raise if missing. Raises: ReqConfOpt setConfOpt(name, valu) Set a single config option for the object. This will perform type normalization if the configration option has a ‘type’ value set. Args: name (str): Configuration name valu: Value to set to the configuration option. Notes: This fires the following events, so that the EventBus can react to configuration changes. Each event includes the name, new valu and oldvalu. • syn:conf:set • syn:conf:set: Returns: None setConfOpts(opts) Use settings from the given dict to update the object config.

4.1. Subpackages 235 Synapse Documentation, Release 0.0.34

Args: opts (dict): Examples: Set a pair of keys on a object using a dictionary:

opts={ 'foo:enabled': True, 'foo:size': 1000 } item.setConfOpts(opts)

Returns: None synapse.lib.config.confdef(name) A decorator used to flag configable definition functions. The options returned by the decorated functions are automatically loaded into the class upon initialization of the Configable mixin. This decorator must be used AFTER a @staticmethod decorator for autodoc generation to work properly. Args: name (str): Identifier for a given function. This is used to prevent reloading configable options multiple times in the case of multi-class inheritance or mixin use. Examples: Example class using the confdef decorator to define and (automatically load) a set of options into a Configable class:

class Foo(s_config.Config):

@staticmethod @s_config.confdef(name='foo') def foodefs(): defs=( ('fooval',{'type':'int','doc':'what is foo val?','defval':

˓→99}), ('enabled',{'type':'bool','doc':'is thing enabled?','defval

˓→':0}), ) return defs synapse.lib.dark module synapse.lib.dark.genDarkRows(iden, name, valus) Generate dark rows in bulk for a given set of values. This can be used generate rows in order to do bulk insertion of rows into a Cortex without the overhead of continually going through the addTufoDark() API. Args: iden (str): Iden to reverse. name (str): Dark row name. valus : Iterator to generate values to make in the rows. May be any data type which may stored in a Cortex. Example: Example use with a core:

rows= list(genDarkRows('1234','hidden',['garden','server','clown]) core.addRows(rows)

Yields: tuple: A cortex row, containing a iden, prop, value and timestamp.

236 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

synapse.lib.datapath module

class synapse.lib.datapath.DataElem(item, name=None, parent=None) Bases: object iter(path) Iterate sub elements using the given path. Example: data = { ‘foo’:[ {‘bar’:’lol’}, {‘bar’:’heh’} ] } root = s_datapath.initelem(data) for elem in root.iter(‘foo/*/bar’): dostuff(elem) # elem is at value “lol” and “heh” name() step(path) Step to the given DataElem within the tree. vals(path) Iterate the given path elements and yield values. Example: data = { ‘foo’:[ {‘bar’:’lol’}, {‘bar’:’heh’} ] } root = s_datapath.initelem(data) for elem in root.iter(‘foo/*/bar’): dostuff(elem) # elem is at value “lol” and “heh” valu(path) Return the value of the element at the given path. class synapse.lib.datapath.XmlDataElem(item, name=None, parent=None) Bases: synapse.lib.datapath.DataElem synapse.lib.datapath.initelem(item, name=None, parent=None) Construct a new DataElem from the given item using which ever DataElem class is most correct for the type. Example: elem = initelem( synapse.lib.datﬁle module

Utilities for handling data ﬁles embedded within python packages. synapse.lib.datfile.openDatFile(datpath) Open a ﬁle-like object using a pkg relative path. Example: fd = openDatFile(‘foopkg.barpkg/wootwoot.bin’) synapse.lib.encoding module synapse.lib.encoding.decode(name, byts, **opts) Decode the given byts with the named decoder. If name is a comma separated list of decoders, loop through and do them all. Example:

4.1. Subpackages 237 Synapse Documentation, Release 0.0.34

byts = s_encoding.decode(‘base64’,byts)

Note: Decoder names may also be prefixed with + to encode for that name/layer. synapse.lib.encoding.encode(name, item, **opts) synapse.lib.fifo module class synapse.lib.fifo.Fifo(conf, xmit=None) Bases: synapse.lib.config.Config ack(seqn) Acknowledge (and cull) an item in the sequence. flush() Flush any file buffers associated with this Fifo. put(item) Put a new item into the Fifo. Args: item (obj): The object to serialize into the Fifo. resync(xmit=None) Re-synchronize this Fifo by sending all window entries. (optionally specify a new xmit callback) Args: xmit (func): The fifo xmit() callback. Example: def xmit(qent): seqn, nseq, item = qent dostuff() fifo.resync(xmit=xmit) class synapse.lib.fifo.Window(fdir, path, xmit=None) Bases: synapse.eventbus.EventBus A read window within a Fifo. ack(seqn) Ack a sequence from this Fifo. Args: seqn (int): The sequence number to acknowledge. fill() Possibly trigger filling the fifo window from bytes. flush() Flush the current AtomFile contents. resync(xmit=None) Re-synchronize this Fifo (assuming an xmit restart). ( see Fifo.resync ) synapse.lib.filepath module class synapse.lib.filepath.FpFile(pparts, idx, parent=None, fd=None) Bases: object close() Closes the file-like object if it has one isfile() Returns a boolean. If it returns False, it may be assumed to be a directory

238 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

next() This is the workhorse method that can contain path specific processing of children. The object should consume as much as possible of the path before creating the child class NOTE: Override for container formats nexts() open(mode=’rb’) Returns a file-like object for this path This should return None if it doesn’t make sense to open, i.e. a directory path() class synapse.lib.filepath.FpOpener(fpobj) Bases: object close() read(*args) seek(*args) class synapse.lib.filepath.FpTar(pparts, idx, parent=None, fd=None) Bases: synapse.lib.filepath.FpFile close() Closes the file-like object if it has one innerLs(path) innrEnum() enumerate the files and directory paths in the container exactly once! creates a nested dict and a set of files and dirs... memory inefficient FIXME: consider abandoning sets innrExists(path) innrIsdir(path) innrIsfile(path) innrOpen(*parts) innrTmpExtract(path) Extract a file from within the container to a named temporary file next() This is the workhorse method for path specific processing of container children. The object should consume as much as possible of the path before instantiating a new object At a minimum, each container should override: innrOpen(path) innrEnum(path) nexts() open(mode=’rb’) Returns a file-like object for the path inside the container This should return None if it doesn’t make sense to open, i.e. a directory or if the container doesn’t contain the end of the path path() class synapse.lib.filepath.FpZip(pparts, idx, parent=None, fd=None) Bases: synapse.lib.filepath.FpTar innrEnum() enumerate the files and directory paths in the container exactly once!

4.1. Subpackages 239 Synapse Documentation, Release 0.0.34

innrOpen(*parts) synapse.lib.filepath.exists(*paths) Determines if the path exists even if the path terminates inside a container file. If a list of paths are provided, they are joined first. Returns a boolean. synapse.lib.filepath.getPathParts(path) Returns the elements of a path in order, w/o regard to their original separators synapse.lib.filepath.isdir(*paths) Determines if the path is a directory, even if the path terminates inside a container file. If a list of paths are provided, they are joined first. Returns a boolean. synapse.lib.filepath.isfile(*paths) Determines if the path is a file, even if the path terminates inside a container file. If a list of paths are provided, they are joined first. Returns a boolean. synapse.lib.filepath.max_temp_sz = 104857600 Provide a generic opener API for paths that cross into supported container files. For example: /dir0/dir1/foo.zip/d0/bar The above represents a file named bar located in the d0 directory inside the foo.zip zip archive located on the filesystem in the /dir0/dir1 directory. synapse.lib.filepath.normpath(*paths) Normalizes a path: 1. uses forward-slashes 2. removes leading slashes 3. removes trailing slashes This is useful for container path enumeration synapse.lib.filepath.openfile(*paths, **kwargs) Returns a read-only file-like object even if the path terminates inside a container file. If the path is a regular os accessible path mode may be passed through as a keyword argument. If the path terminates in a container file, mode is ignored. If req=True (Default) NoSuchPath will also be raised if the path exists, but is a directory Example: fd = openfile(‘/foo/bar/baz.egg/path/inside/zip/to/file’) if fd == None: return fbuf = fd.read() fd.close() synapse.lib.filepath.openfiles(*paths, **kwargs) Yields a read-only file-like object for each path even if the path terminates inside a container file. Paths may use python’s fnmatch glob matching If the path is a regular os accessible path mode may be passed through as a keyword argument. If the path terminates in a container file, mode is ignored. If req=True (Default) NoSuchPath will also be raised if ANY matching path exists, but is a directory Example: for fd in openfiles(‘/foo/bar/.egg/dir0/zz/nest.zip’): fbuf = fd.read() synapse.lib.filepath.parsePath(*paths) function to parse the incoming path. lists of paths are joined prior to parsing synapse.lib.filepath.parsePaths(*paths) function to parse the incoming path. lists of paths are joined prior to parsing The path supports python’s fnmatch glob matching synapse.lib.filepath.subpaths(path) Returns a list of subpaths in a path, one for each level in the path This is an internal function used for ONLY for

240 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

iterating over paths in a container As such it should not be used for filesystem paths since separators will vary across platforms synapse.lib.hashitem module synapse.lib.hashitem.hashitem(item) Generate a uniq hash for the JSON compatible primitive data structure. synapse.lib.hashitem.normdict(item) synapse.lib.hashitem.normitem(item) synapse.lib.hashitem.normiter(item) synapse.lib.hashset module class synapse.lib.hashset.HashSet Bases: object digests() Return a list of (name,digest) tuples for the hashes in the set. eatfd(fd) Consume all the bytes from a file like object. Example: hset = HashSet() hset.eatfd(fd) guid() Use elements from this hash set to create a unique (re)identifier. update(byts) Update all the hashes in the set with the given bytes. synapse.lib.heap module class synapse.lib.heap.Heap(fd, **opts) Bases: synapse.eventbus.EventBus A persistant heap object. alloc(size) Allocate a block within the Heap and return the offset. Example: off = heap.alloc(len(foo)) readiter(off, size, itersize=10000000) Yield back byts chunks for the given off/size. Example: for byts in heap.readiter(off,size): dostuff() readoff(off, size) Read and return bytes from the heap at an offset. Example:

4.1. Subpackages 241 Synapse Documentation, Release 0.0.34

head = heap.readoff(off,headsize) size() sync(mesg) Consume a heap:sync event. syncs(msgs) Consume a list of heap:sync events. writeoff(off, byts) Write bytes at an offset within the heap. Example: off = heap.alloc(size) heap.writeoff(off,byts) synapse.lib.ingest module class synapse.lib.ingest.Ingest(info, axon=None) Bases: synapse.eventbus.EventBus An Ingest allows modular data acquisition and cortex loading. get(name, defval=None) Retrieve a value from self._i_info ingest(core, data=None) Ingest the data from this definition into the specified cortex. set(name, valu) Set a value in self._i_info class synapse.lib.ingest.IngestApi(core) Bases: object An API mixin which may be used to wrap a cortex and send along ingest data. addGestData(name, data) Ingest data according to a previously registered ingest format. Example: data = getDataFromThing() # use the foo:bar ingest iapi.addGestData(‘foo:bar’, data) setGestDef(name, idef ) Set an ingest definition by storing it in the cortex. Example: idef = {... ingest def json ... } iapi.setGestDef(‘foo:bar’, idef) setGestFunc(name, func) Set an ingest function to handle a custom data format. Args: name (str): The name of the ingest format func (func): The callback function to parse the data synapse.lib.ingest.addFormat(name, fn, opts) Add an additional ingest file format

242 Chapter 4. synapse package Synapse Documentation, Release 0.0.34 synapse.lib.ingest.iterdata(fd, close_fd=True, **opts) Iterate through the data provided by a ﬁle like object. Optional parameters may be used to control how the data is deserialized. Examples: The following example show use of the iterdata function.:

with open('foo.csv','rb') as fd: for row in iterdata(fd, format='csv', encoding='utf8'): dostuff(row)

Args: fd (ﬁle) : File like object to iterate over. close_fd (bool) : Default behavior is to close the fd object. If this is not true, the fd will not be closed.

**opts (dict): Ingest open directive. Causes the data in the fd to be parsed according to the ‘format’ key and any additional arguments.

Yields: An item to process. The type of the item is dependent on the format parameters. synapse.lib.ingest.loadfile(*paths) Load a json ingest def from file and construct an Ingest class. This routine is useful because it implements the convention for adding runtime info to the ingest json to facilitate path relative file opening etc... synapse.lib.ingest.register_ingest(core, gest, evtname, ret_func=False) Register an ingest class with a cortex eventbus with a given name. When events are fired, they are expected to have the argument “data” which is passed along to the Ingest.ingest() function. Parameters • core – Cortex to register the Ingest with • gest – Ingest to register • evtname – Event name to register the ingest with. • ret_func – Bool, if true, return the ingest function. synapse.lib.interval module

A few utilities for dealing with intervals. synapse.lib.interval.fold(*vals) Initialize a new (min,max) tuple interval from values. Args:* vals ([int,...]): A list of values (or Nones) Returns: ((int,int)): A (min,max) interval tuple or None synapse.lib.interval.overlap(ival0, ival1) Determine if two interval tuples have overlap. Args: iv0 ((int,int)): An interval tuple iv1 ((int,int)); An interval tuple Returns: (bool): True if the intervals overlap, otherwise False synapse.lib.interval.parsetime(text) Parse an interval time string and return a (min,max) tuple. Args: text (str): A time interval string Returns: ((int,int)): A epoch millis epoch time string

4.1. Subpackages 243 Synapse Documentation, Release 0.0.34 synapse.lib.iq module synapse - iq.py Created on 10/21/17. The IQ module contains the core test helper code used in Synapse. This gives the opportunity for third-party users of Synapse to test their code using some of the same of the same helpers used to test Synapse. The core class, synapse.lib.iq.SynTest is a subclass of unittest.TestCase, with several wrapper functions to allow for easier calls to assert* functions, with less typing. There are also Synapse speciﬁc helpers, to load both Ram and PSQL Cortexes. Since SynTest is built from unittest.TestCase, the use of SynTest is compatible with the unittest, nose and pytest frameworks. This does not lock users into a particular test framework; while at the same time allowing base use to be invoked via the built-in Unittest library. class synapse.lib.iq.SynTest(methodName=’runTest’) Bases: unittest.case.TestCase static addTstForms(core) Add test forms to the cortex. Args: core (s_cores_common.Cortex): Core to prep. Returns: None eq(x, y) Assert X is equal to Y false(x) Assert X is False ge(x, y) Assert that X is greater than or equal to Y getDmonCore() Context manager to make a ram:/// cortex which has test models loaded into it and shared via daemon. Yields: s_cores_common.Cortex: A proxy object to the Ram backed cortex with test models. getLoggerStream(logname) Get a logger and attach a io.StringIO object to the logger to capture log messages. Args: logname (str): Name of the logger to get. Examples: Do an action and get the stream of log messages to check against:

with self.getLoggerStream('synapse.foo.bar') as stream: # Do something that triggers a log message doSomthing() stream.seek(0) mesgs= stream.read() # Do something with messages

Yields: io.StringIO: A io.StringIO object getPgConn() Get a psycopg2 connection object. The PG database connected to is derived from the SYN_TEST_PG_DB environmental variable. Returns: psycopg2.connection: Raw psycopg2 connection object.

244 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

getPgCore(table=’‘, persist=False, **opts) Get a Postgresql backed Cortex. This will grab the SYN_TEST_PG_DB environmental variable, and use it to construct a string to connect to a PSQL server and create a Cortex. By default, the Cortex DB tables will be dropped when onﬁni() is called on the Cortex. Some example values for this envar are shown below:

# From our .drone.yml file root@database:5432/syn_test # An example which may be used with a local docker image # after having created the syn_test database postgres:1234@localhost:5432/syn_test

Args: table (str): The PSQL table name to use. If the table name is not provided by URL or argument; a random table name will be created. persist (bool): If set to True, keep the tables created by the Cortex creation. opts: Additional options passed to openlink call. Returns: s_cores_common.Cortex: A PSQL backed cortex. Raises: unittest.SkipTest: if there is no SYN_TEST_PG_DB envar set.

getRamCore() Context manager to make a ram:/// cortex which has test models loaded into it. Yields: s_cores_common.Cortex: Ram backed cortex with test models. getTestDir() Get a temporary directory for test purposes. This destroys the directory afterwards. Yields: str: The path to a temporary directory. getTestOutp() Get a Output instance with a expects() function. Returns: TstOutPut: A TstOutPut instance. getTestWait(bus, size, *evts) gt(x, y) Assert that X is greater than Y isin(member, container) Assert a member is inside of a container. isinstance(obj, cls) Assert a object is the instance of a given class or tuple of classes. le(x, y) Assert that X is less than or equal to Y len(x, obj) Assert that the length of an object is equal to X lt(x, y) Assert that X is less than Y ne(x, y) Assert X is not equal to Y

4.1. Subpackages 245 Synapse Documentation, Release 0.0.34

nn(x) Assert X is not None none(x) Assert X is None noprop(info, prop) Assert a property is not present in a dictionary. notin(member, container) Assert a member is not inside of a container. raises(*args, **kwargs) Assert a function raises an exception. setTstEnvars(**props) Set Environmental variables for the purposes of running a speciﬁc test. Args:** props: A kwarg list of envars to set. The values set are run through str() to ensure we’re setting strings. Examples: Run a test while a envar is set:

with self.setEnvars(magic='haha') as nop: ret= dostuff() self.true(ret)

Notes: This helper explicitly sets and unsets values in os.environ, as os.putenv does not automatically updates the os.environ object. Yields: None. This context manager yields None. Upon exiting, envars are either removed from os.environ or reset to their previous values. skipIfNoInternet() Allow skipping a test if SYN_TEST_SKIP_INTERNET envar is set. Raises: unittest.SkipTest if SYN_TEST_SKIP_INTERNET envar is set to a integer greater than 1. skipLongTest() Allow skipping a test if SYN_TEST_SKIP_LONG envar is set. Raises: unittest.SkipTest if SYN_TEST_SKIP_LONG envar is set to a integer greater than 1. sorteq(x, y) Assert two sorted sequences are the same. thisHostMust(**props) Requires a host having a speciﬁc property. Args:** props: Raises: unittest.SkipTest if the required property is missing. thisHostMustNot(**props) Requires a host to not have a speciﬁc property. Args:** props: Raises: unittest.SkipTest if the required property is missing. true(x) Assert X is True class synapse.lib.iq.TstEnv Bases: object

246 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

add(name, item, fini=False) fini() class synapse.lib.iq.TstOutPut Bases: synapse.lib.output.OutPutStr expect(substr) synapse.lib.mixins module synapse.lib.mixins.addSynMixin(subsys, name, cname=None) Add a mixin class to the specified subsystem. Example: s_mixins.addSynMixin(‘foo’,’synapse.foo.FooMixin’) synapse.lib.mixins.getSynMixins(subsys, name) Return a list of mixin classes for the given subsystem class. Example: for clas in getSynMixins(‘telepath’,’foo.bar.Baz’): dostuff() synapse.lib.mixins.ldict() synapse.lib.module module class synapse.lib.module.CoreModule(core, conf ) Bases: synapse.eventbus.EventBus, synapse.lib.config.Configable The CoreModule base class from which cortex modules must extend. This module interface implements helper APIs to facilitate cortex extensions. To load a module within a cortex, add it to the list of modules in the cortex config ( mostly likely within your dmon config ) as shown: # example cortex config { “modules”:[ [”foopkg.barmod.modctor”, { “foo:opt”:10, “bar:opt”:”http://www.vertex.link” }] ] } Modules may extend the cortex in various ways such as: • Implement and enforce data model additions • Enrich properties during node creation / modification • Add “by” handlers and side-pocket indexes to extend queries • Add custom storm/swarm operators to the query language • etc etc etc...

NOTE: The cortex which loads the module plumbs all events into the CoreModule instance using Event- Bus.link().

4.1. Subpackages 247 Synapse Documentation, Release 0.0.34

form(form, valu, **props) A module shortcut for core.formTufoByProp() Args: form (str): The node form to retrieve/create valu (obj): The node value **props: Additional node properties static getBaseModels() Get a tuple containing name, model values associated with the CoreModule. Any models which are returned by this function are considered revision 0 models for the name, and will be automatically loaded into a Cortex if the model does not currently exist. Note: While this may return multiple tuples, internal Synapse convention is to deﬁne a single model in a single CoreModule subclass in a single ﬁle, for consistency. Returns: ((str, dict)): A tuple containing name, model pairs. getModName() Return the name of this module. Returns: (str): The module name. getModPath(*paths) Construct a path relative to this module’s working directory. Args: (*paths): A list of path strings Returns: (str): The full path getModlRevs() Generate a list of ( name, vers, func ) tuples for model revisions in this module. Returns: ([ (str, int, func), ... ]) Example: for name, vers, func in modu.getModlRevs(): core.revModlVers(name,revs) initCoreModule() Module implementers may over-ride this method to initialize the module during initial construction. Any exception raised within this method will be raised from the constructor and mark the module as failed. Args: Returns: (None) onFormNode(form, func) Register a callback to run during node formation. This callback will be able to set properties on the node prior to construction. Args: form (str): The name of the node creation func (function): A callback Returns: (None) Example: def myFormFunc(form, valu, props, mesg): props[’foo:bar:baz’] = 10 self.onFormNode(‘foo:bar’, myFormFunc) NOTE: This may not be used for a module loaded with a remote cortex. onNodeAdd(func, form=None) Register a callback to run when a node is added. Args: func (function): The callback form (str): The form of node to watch for (or all!)

248 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

Returns: (None) Example: def callback(node): dostuff(node) self.onNodeAdd(callback, form=’inet:fqdn’) onNodeDel(func, form=None) Register a callback to run when a node is deleted. Args: func (function): The callback form (str): The form of node to watch for (or all!) Returns: (None) Example: def callback(node): dostuff(node) self.onNodeDel(callback, form=’inet:fqdn’) postCoreModule() Module implementers may over-ride this method to initialize the module after the conﬁguration data has been loaded. Returns: (None) reqModPath(*paths) Require a path relative to this module’s working directory. Args: (*paths): A list of path strings Returns: (str): The full path synapse.lib.module.modelrev(name, vers) A decorator used to ﬂag model revision functions. Args: name (str): Name of the model. vers (int): Revision of the model. It is validated using vali- date_revnumber. synapse.lib.module.validate_revnumber(revision) Validate a model revision number matches the time format ‘%Y%m%d%H%M’ Args: revision (int): Revision to validate. Raises: BadRevValu: If the integer does not match the time format. synapse.lib.modules module

Module which implements the synapse module API/convention. synapse.lib.modules.call(name, *args, **kwargs) Call the given function on all loaded synapse modules. Returns a list of name,ret,exc tuples where each module which implements the given function returns either ret on successful execution or exc in the event of an exception. Args: name (str): Name of the function to execute. *args (tuple): Additional positional args to use when executing the function. **kwargs (dict): Additional keyword args to use when executing the function. Example: Call getFooByBar with a single positional argument:

import synapse.lib.modules as s_modules for name,ret,exc in s_modules.call('getFooByBar',bar): dostuff()

4.1. Subpackages 249 Synapse Documentation, Release 0.0.34

Returns: list: List of name, returnval, exception information for each registered module. synapse.lib.modules.call_ctor(name, *args, **kwargs) Call the given function on all ctors loaded by load_ctor. Returns a list of name,ret,exc tuples where each ctor which implements the given function returns either ret on successful execution or exc in the event of an exception. Args: name (str): Name of the function to execute. *args (tuple): Additional positional args to use when executing the function. **kwargs (dict): Additional keyword args to use when executing the function. Example: Call getFooByBar on all loaded ctors.:

import synapse.lib.modules as s_modules for name,ret,exc in s_modules.call_ctor('getFooByBar'): dostuff()

Notes: This function does not create instances of the classes. It is best used when called against @staticmethod or @classmethod functions. Returns: list: List of name, returnval, exception information for each registered ctor. synapse.lib.modules.load(name) Load the given module path as a synapse module. Args: name (str): Python path to load. Example: Load the foopkg.barmod module.:

import synapse.lib.modules as s_modules s_modules.load('foopkg.barmod')

Notes: Users should be aware that the import process can perform arbitrary code execution by imported modules. Returns: The loaded module is returned. synapse.lib.modules.load_ctor(name, opts) Load the given ctor path as a synapse CoreModule for extending the Cortex implementation. Args: name (str): Python path to a class ctor to load. opts (dict): Dictionary of conﬁguration options. Example: Load the foopkg.barmod.Baz ctor:

import synapse.lib.modules as s_modules s_modules.load_ctor('foopkg.barmod.Baz', {})

Notes: This can only be used to dynamically load a subclass of the CoreModule class. Users should be aware that the import process can perform arbitrary code execution by imported modules. Returns: The loaded class is returned. Raises: NoSuchCtor: If the imported module does not have the listed ctor BadCtorType: If the ctor is not a subclass of the CoreModule class. synapse.lib.msgpack module class synapse.lib.msgpack.Unpk Bases: object

250 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

An extension of the msgpack streaming Unpacker which reports sizes. feed(byts) Feed bytes to the unpacker and return completed objects. synapse.lib.msgpack.en(item) Use msgpack to serialize a compatible python object. Args: item (obj): The object to serialize Returns: bytes: The serialized bytes synapse.lib.msgpack.iterfd(fd) Generator which unpacks a ﬁle object of msgpacked content. Args: fd: File object to consume data from. Yields: Objects from a msgpack stream. synapse.lib.msgpack.un(byts) Use msgpack to de-serialize a python object. Args: byts (bytes): The bytes to de-serialize Returns: obj: The de-serialized object synapse.lib.openﬁle module

A single entry point for various protocol and open behaviors to allow all ﬁle open (for read) requests to support URLs and encapsulation. synapse.lib.openfile.openfd(*paths, **opts) Open and return a ﬁle like object for the given path/url. Example: with openfd(‘http://vertex.link/foo.csv‘) as fd: dostuff(fd) with openfd(‘foo/bar.txt’) as fd: fd.read() synapse.lib.output module

Tools for easily hookable output from cli-like tools. class synapse.lib.output.OutPut Bases: object printf(mesg, addnl=True) class synapse.lib.output.OutPutBytes Bases: synapse.lib.output.OutPutFd class synapse.lib.output.OutPutFd(fd, enc=’utf8’) Bases: synapse.lib.output.OutPut class synapse.lib.output.OutPutStr Bases: synapse.lib.output.OutPut

4.1. Subpackages 251 Synapse Documentation, Release 0.0.34 synapse.lib.persist module

Tools for persisting msgpack compatible objects. class synapse.lib.persist.Dir(path, **opts) Bases: synapse.eventbus.EventBus A persistence dir may be used similar to a Write-Ahead-Log to sync objects based on events ( and allow desync- catch-up ) add(item) Add an object to the persistant store. Returns (off,size) tuple within the persistance stream. Example: off,size = pers.add(item) getIdenOffset(iden) getPumpOffs() Return a dict of { iden:offset } info for the running pumps. Example: for iden,noff in pdir.getPumpOffs(): dostuff() items(off ) Yield (nextoff,item) tuples from the file backlog and real-time once caught up. NOTE: because this is a legitimate yield generator it may not be used across a telepath proxy. Example: for noff,item in pers.items(0): stuff(item) pump(iden, func) Fire a new pump thread to call the given function. Example: pdir.firePumpThread( iden, core.sync ) class synapse.lib.persist.File(fd=None, **opts) Bases: synapse.eventbus.EventBus A single fd based persistence stream. This is mostly a helper for Dir(). All consume/resume behavior should be facilitated by the Dir() object. add(item) Add an item to the persistance storage. readoff(off, size) Read size bytes form the given offset. class synapse.lib.persist.Offset(*paths) Bases: synapse.eventbus.EventBus A file backed persistant offset calculator. Example: poff = Offset(dirpath,’foo.off’): for off,item in pers.items(): dostuff(item) poff.set(off)

252 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

get() Return the current offset. set(valu) Set (and save) the current offset. synapse.lib.persist.opendir(*paths, **opts) Open a persistance directory by path name with options. synapse.lib.queue module class synapse.lib.queue.Queue(items=()) Bases: synapse.eventbus.EventBus A simple custom queue to address python Queue() issues. done() Gracefully mark this Queue as done. This still allows a Queue consumer to ﬁnish consuming it. The Queue functions get(), slice() and slices() will not block when .done() has been called on a Queue. Returns: None get(timeout=None) Get the next item from the queue. Args: timeout (int): Duration, in seconds, to wait for items to be available to the queue before returning. Notes: This will block if the queue is empty and no timeout value is speciﬁed, or .done() has not been called on the Queue. Examples: Get an item and do stuff with it:

item=q.get(timeout=30) dostuff(item)

Returns: Item from the queue, or None if the queue is ﬁni() or timeout occurs. put(item) Add an item to the queue and wake any consumers waiting on the queue. Args: item: Item to add to the queue. Examples: Put a string in a queue:

q.put('woot')

Returns: None slice(size, timeout=None) Get a slice of the next items from the queue. Args: size (int): Maximum number of items to get from the queue. timeout (int): Duration, in seconds, to wait for items to be available to the queue before returning. Examples: Return up to 3 items on a 30 second timeout from the queue:

4.1. Subpackages 253 Synapse Documentation, Release 0.0.34

items=q.slice(3, timeout=30)

Notes: This will block if the queue is empty and no timeout value is speciﬁed, or .done() has not been called on the Queue. Returns: list: A list of items from the queue. This will return None on ﬁni() or timeout. slices(size, timeout=None) Yields slices of items from the queue. Args: size (int): Maximum number of items to yield at a time. timeout (int): Duration, in seconds, to wait for items to be added to the queue before exiting. Examples: Yield 2 items at a time with a 1 second time:

for items in q.slices(2, timeout=1): dostuff(items)

Notes: This will block if the queue is empty and no timeout value is specified, or .done() has not been called on the Queue. Yields: list: This generator yields a list of items. exception synapse.lib.queue.QueueShutdown Bases: Exception synapse.lib.ratelimit module class synapse.lib.ratelimit.RateLimit(rate, per) Bases: object A RateLimit class may be used to detect/enforce rate limits. Example: # allow 20 uses per 10 sec ( 2/sec ) rlimit = RateLimit(20,10) Notes: It is best ( even in a “calls per day” type config ) to specify a smaller “per” to force rate “smoothing”. allows() Returns True if the rate limit has not been reached. Example: if not rlimit.allows(): rasie RateExceeded() # ok to go... synapse.lib.reflect module synapse.lib.reflect.getClsNames(item) Return a list of “fully qualified” class names for an instance. Example: for name in getClsNames(foo): print(name)

254 Chapter 4. synapse package Synapse Documentation, Release 0.0.34 synapse.lib.reflect.getItemInfo(item) Get “reﬂection info” dict for the given object. Args: item: Item to inspect. Examples: Find out what classes a Telepath Proxy object inherits:

info= getItemInfo(prox) classes= info.get('inherits')

Notes: Classes may implement a def _syn_reflect(self): function in order to return explicit values. The Telepath Proxy object is one example of doing this, in order to allow a remote caller to identify what classes the Proxy object represents. Returns: dict: Dictionary of reﬂection information. synapse.lib.reflect.getItemLocals(item) Iterate the locals of an item and yield (name,valu) pairs. Example: for name,valu in getItemLocals(item): dostuff() synapse.lib.reflect.getMethName(meth) Return a fully qualiﬁed string for the .. name of a given method. synapse.lib.remcycle module

Remcycle provides a mechanism for kicking off asynchronous HTTP(S) requests via Tornado’s AsyncHTTPClient. A method for templating URL endpoints, providing default configuration values and user set configuration values, per request, is available. In addition, data can also be ingested into a cortex (provided, or a default ram cortex) for immediate consumption. These requests are handled by Hypnos objects, which can have a single or multiple definitions snapped into them in order to grab data on demand. Hypnos grabs these requests with a single IO thread, and offloads the consumption of the data to multiple worker threads. class synapse.lib.remcycle.Hypnos(core=None, opts=None, *args, **kwargs) Bases: synapse.lib.config.Config Object for grabbing a bunch of HTTP(S) data and consuming it via callbacks or ingest definitions. Users can register multiple namespaces, each with their own set of API endpoints configured with them. See the fire_api() function for details on retrieving data with Hypnos. The Hypnos object inherits from the Config object, and as such has both configable parameters and an EventBus available for message passing. Notes: The following items may be passed via kwargs to change the Hypnos object behavior: • ioloop: Tornado ioloop used by the IO thread. This would normally be left unset, and an ioloop will be created for the io thread. This is provided as a helper for testing. • content_type_skip: A list of content-type values which will not have any attempts to decode data done on them. The following values may be passed via configable opts: • web:worker:threads:min: Minimum number of worker threads to spawn. • web:worker:threads:max: Maximum number of worker threads to spawn.

4.1. Subpackages 255 Synapse Documentation, Release 0.0.34

• web:ingest:max_spool_size: Maximum spoolfile size, in bytes, to use for storing responses associated withAPIs that have ingest definitions. • web:cache:enable: Enable caching of job results for a period of time, retrievable by jobid. • web:cache:timeout: Timeout value, in seconds, that the results will persist in the cache. • web:tornado:max_clients: Maximum number of concurrent requests which can be made at one time. Args: core (synapse.cores.common.Cortex): A cortex used to store ingest data. By default a ram cortex is used. opts (dict): Optional configuration data for the Config mixin. addWebConfig(config, reload_config=True) Register a configuration into a Hypnos object. The Hypnos object can accept a configuration object shaped like the following:

{ "apis":[ [ "geoloc", { "api_args":[ "someplace" ], "api_optargs":{ "domore":0 }, "doc":"api example", "http":{ "headers":{ "token-goodness":"sekrittoken" } }, "ingest":{ "definition":{ "ingest":{ "forms":[ [ "inet:ipv4", { "var":"ip" } ] ], "vars":[ [ "ip", { "path":"ip" } ] ] }, "open":{ "format":"json" } }, "name":"geolocv4" }, "url":"http://vertex.link/api/v4/geoloc/{{someplace}}/info?domore={

˓→{domore}}&apikey={APIKEY}",

256 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

"vars":[ [ "APIKEY", "8675309" ] ] } ], [ "https", { "doc":"Get the vertex project landingpage.", "http":{ "validate_cert": false }, "url":"https://vertex.link/" } ] ], "doc":"GrabVertex.linkstuff", "http":{ "user_agent":"Totally Not a Python application." }, "namespace":"vertexproject" }

The following config keys are required: • namespace: String identifier for all APIs present in the configuration. Must be locally unique. • doc: Simple string describing the overall namespace. • apis: Sequence containing containing configuration values for API endpoints. See Nyx object for details of how this data should be shaped. The sequence should contain two value pairs of data. The first value should be the name of the API, while the second value is the actual API configuration. The name of the API, when joined with the namespace, forms the name the API can be called with for for later use. Given the example above, the following APIs would be registered: – vertexproject:geoloc – vertexproject:https The following config keys are optional: • http: Global HTTP Request arguments which will be the basis for creating HTTPRequest objects. These values should conform to the Tornado HTTPRequest constructor. An example of a generic configuration for getting arbitrary endpoints is shown below:

{ "apis":[ [ "fqdn", { "api_args":[ "fqdn" ], "api_optargs":{ "endpoint":"" },

4.1. Subpackages 257 Synapse Documentation, Release 0.0.34

"doc":"Get arbitrary domain name.", "http":{ "validate_cert": false }, "url":"https://{{fqdn}}/{{endpoint}}" } ] ], "doc":"Definition for getting an arbitrary domain.", "http":{ "user_agent":"Some.UserAgent" }, "namespace":"generic" }

Args: config (dict): Dictionary containing the configuration information. reload_config (bool): If true and the namespace is already registered, the existing namespace will be removed and the new config added. Otherwise a NameError will be thrown. Returns: None Raises: Other exceptions are possible, likely as a result of ducktyping. NameError: If the existing namespace is registered or a invalid HTTP value is provided.

delWebConf(namespace) Safely remove a namespace. Removes a given namespace, APIs and any corresponding event handlers which have been snapped into the Hypnos object and its cortex via the addWebConfig API. Args: namespace (str): Namespace to remove. Returns: None Raises: NoSuchName: If the namespace requested does not exist. fireWebApi(name, *args, **kwargs) Fire a request to a registered API. The API response is serviced by a thread in the Hypnos thread pool, which will fire either an event on the Hypnos service bus or a caller provided callback function. The default action is to fire an event on the service bus with the same name as the API itself. A flattened version of the response, error information and the Boss job id will be stamped into the kwargs passed along to the the callbacks. If the API name has a ingest associated with it, the response data will be pushed into a generator created according to the ingest open directive. The flattened response is a dictionary, accessed from kwargs using the ‘resp’ key. It contains the following information: • request: A dictionary containing the requested URL and headers. This is guaranteed to exist. It has the following values: – url: URL requested by the remote server. – headers: Headers passed to the remote server.

258 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

• code: HTTP Response code. This will only be present on a successfull request or if a HTTPError is encountered. • data: This may be one of three values: – A SpooledTemporaryFile containing the raw bytes of the response. This will be present if there is a ingest associated with the named response. A corresponding generator will be created and placed in the “ingdata” field and consumed by the ingest. Post-consumption, seek(0) will be called on the file-like object. If there are multiple post-ingest consumers of the job, each one may want to call seek(0) on the file object before consuming it. – The decoded data as a string or a decoded json blob. We will attempt to parse the data based on the Content-Type header. This is a best effort decoding. – In the event that the best effort decoding fails, the response will be available as raw bytes. • effective_url: The effective url returned by the server. By default, Tornado will follow redirects, so this URL may differ from the request URL. It will only be present on a successful request or if a HTTPError is encountered. • headers: The response headers. It will only be present on a successful request or if a HTTPError is encountered. The flattened error is a dictionary, accessed from kwargs using the ‘errinfo’ key. It mimics the synapse excinfo output, but without investigating a stack trace for performance reasons. It contains the following information: • err: The Exception class raised during the request. • errmsg: The str() representation of the exception. • errfile: Empty string. • errline: Empty string. The Hypnos boss job id is a str which can be accessed from kwargs using the ‘jid’ key. Notes: The following items may be used via kwargs to set request parameters: • api_args: This should be a dictionary containing any required or optional arguments the API rquires. The following items may be passed via kwargs to change the job execution parameters: • callback: A function which will be called by the servicing thread. By default, this will be wrapped to fire boss.err() if excinfo is present in the callback’s kwargs. • ondone: A function to be executed by the job:fini handler when the job has been completed. If the api we’re firing has an ingest associated with it, the response data may not be available to be consumed by the ondone handler. • job_timeout: A timeout on how long the job can run from the perspective of the boss. This isn’t related to the request or connect timeouts. • wrap_callback: By default, the callback function is wrapped to perform error checking (and fast job failure) in the event of an error encountered during the request, and additional processing of the HTTP response data to perform decoding and content-type processing. If this value is set to false, the decorator will not be applied to a provided callback function, and the error handling and additional data procesing will be the responsibility of any event handlers or the provided callback function. The fast failure behavior is handled by boss.err() on the job associated with the API call.

4.1. Subpackages 259 Synapse Documentation, Release 0.0.34

A HTTP body can be provided to the request by passing its contents in by adding the “req_body” value to api_args argument. See the Nyx object documentation for more details. If caching is enabled, the caching will be performed as the ﬁrst thing done by the worker thread handling the response data. This is done separately from the wrap_callback step mentioned above.

Args: name (str): Name of the API to send a request for. *args: Additional args passed to the callback functions. **kwargs: Additional args passed to the callback functions or for changing the job execution. Returns: str: String containing a Job ID which can be used to look up a job against the Hyp- nos.web_boss object. Raises: NoSuchName: If the requested API name does not exist.

getNyxApi(name) Get the Nyx object corresponding to a given API name. Args: name (str): Name of the API to get the object for. Returns: Nyx: A Nyx object. getWebDescription() Get a dictionary containing all namespaces, their docstrings, and registered api data. Returns: dict: Dictionary describing the regsistered namespace API data. webCacheClear() Clear all the contents of the web cache. webCacheGet(jid) Retrieve the cached web response for a given job id. Args: jid (str): Job ID to retrieve. Returns: dict: A dictionary containing the job response data. It will have the following keys: • web_api_name: Name of the API • resp: Dictionary containing response data. The raw data is not decoded or processed in any fashion, and is available in the ‘data’ key of this dictionary (if present). • api_args: Args used when crafting the HTTPRequest with Nyx • err (optional): Error type if a error is encountered. • errmsg (optional): Error message if a error is encountered. • errﬁle (optional): Empty string if a error is encountered. • errline (optional): Empty string if a error is encountered. webCachePop(jid) Retrieve the cached web response for a given job id and remove it from the cache. Args: jid (str): Job ID to retrieve.

Returns: dict: A dictionary containing the job response data. See the docs for webCacheGet for the dictionary details.

260 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

webContentTypeSkipAdd(content_type) Add a content-type value to be skipped from any sort of decoding attempts. Args: content_type (str): Content-type value to skip. webContentTypeSkipDel(content_type) Removes a content-type value from the set of values to be skipped from any sort of decoding attempts. Args: content_type (str): Content-type value to remove. webJobWait(jid, timeout=None) Proxy function for the self.web_boss wait() call, in order to allow RMI callers to wait for a job to be completed if they wish. Args: jid (str): Job id to wait for. timeout: Time to wait for. Returns: bool: async.Boss.wait() result. class synapse.lib.remcycle.Nyx(config) Bases: object Configuration parser & request generator for a REST service. This class is responsible for doing the actual HTTPRequest generation in a parametrized fashion for a given input. The API configuration is expected to be a dictionary with the expected values: • doc: Human readable description of the current API endpoint. • url: This is the actual URL which will be used to connect to a service. This string will be run through format() twice - once during the construction of the Nyx object, and the second time during the construction of the per-request url. As such, any values set by the api_args and api_optargs configuration methods noted below should be enclosed with double curly brackets. The following configuration values are optional for a Nyx configuration. • api_args: This is a list of values which must be provided by the user when they call buildHttpRequest. These would represent URL parameters which are required to be provided by the user each time a new HTTPRequest object is built. • api_optargs: This is a dictionary of URL parameters which are are optional for the user to provide when calling buildHttpRequest. This dictionary represents the parameter names and default values for them. A user may provide alternative values when calling buildHttpRequest, but sensible defaults should be provided here. • http: A dictionary of key/value items which can provide per-api specific arguements for the creation of HTTPRequest objects. These values should conform to the Tornado HTTPRequest constructor. • ingest: A dictionary containing a Synapse ingest definition which will be used to create an Ingest objects. During registration of a Nyx object with Hypnos, these will be registered into the Hypnos cortex. This dictionary should contain the key “name” which will be used to create a unique name for the ingest events, and the key “definition” which must contain the ingest definition. The ingest definition must contain a “open” directive which is used with the ingest iterdata() function to process the API data prior to ingest. • vars: This is a dictionary of items which are stamped into the url template during the construction of the Nyx object using format(). Some API endpoints (typically PUT/POST/PATCH) may require additional content which is provided via the HTTP body. The api_arg value “req_body” is reserved in order to support passing body data when making the HTTPRequest object. A consequence of pulling the body from the api_args is that the ‘body’ argument is not allowed to be present in the “http” kv dictionary used when constructing the non-URL portions of the

4.1. Subpackages 261 Synapse Documentation, Release 0.0.34

HTTPRequest For use cases where a caller needs to make body requests with a default set of content, they are responsible for provided that content in the req_body api_args value when calling buildHttpRequest. See a complete example below:

{ "api_args":[ "someplace" ], "api_optargs":{ "domore":0 }, "doc":"api example", "http":{ "headers":{ "token-goodness":"sekrit token" } }, "ingest":{ "definition":{ "ingest":{ "forms":[ [ "inet:ipv4", { "var":"ip" } ] ], "vars":[ [ "ip", { "path":"ip" } ] ] }, "open":{ "format":"json" } }, "name":"geolocv4" }, "url":"http://vertex.link/api/v4/geoloc/{{someplace}}/info?domore={{domore}}&

˓→apikey={APIKEY}", "vars":{ "APIKEY":"8675309" } }

This example should be interpreted as the following: 1. The APIKEY value in the ‘vars’ will be set in the URL, resulting in the following default url:

"http://vertex.link/api/v4/geoloc/{someplace}/info?domore={domore}&

˓→apikey=8675309"

2. The HTTP request will have the header “token-goodness” set to “sekrit token” for the request.

262 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

3. The caller must provide the “someplace” value in the api_args when calling buildHttpRequest. The caller may provide the “domore” value if they want to override the default value of “0”. 4. An Ingest will be created for parsing the data from the API and made available to the Hypnos object.

Parameters config – API Endpoint conﬁguration outlined above.

buildHttpRequest(api_args=None) Build the HTTPRequest object for a given configuration and arguments. Args: api_args (dict): Arguments support either required or optional URL values. Notes: A HTTP body can be provided to the request by passing its contents in by adding the “req_body” value to api_args argument. Returns: tornado.httpclient.HTTPRequest: HTTPRequest object with the configured url and attributes. Raises: NoSuchName: If the api_args is missing a required API value. description() Get a dictionary containing an objects docstring, required api_args and optional api args. Returns: dict: Dictionary containing data. synapse.lib.sched module class synapse.lib.sched.Sched(pool=None) Bases: synapse.eventbus.EventBus at(ts, func, *args, **kwargs) Schedule a function to run at a specific time. Example: # call foo(bar,baz=10) at ts sched.at(ts, foo, bar, baz=10) cancel(item) Cancel a previously scheduled call. Example: def woot(x,y): stuff() sched = Sched() item = sched.insec(10, woot, 10, 20) sched.cancel(item) insec(delay, func, *args, **kwargs) Schedule a callback to occur in delay seconds. Example: def woot(x,y): stuff() sched = Sched() e = sched.insec(10, woot, 10, 20) # woot will be called in 10 seconds.. loop(secs, func, *args, **kwargs) Call the given function in a delay loop.

4.1. Subpackages 263 Synapse Documentation, Release 0.0.34

Args: secs (int): Seconds between loop calls (can be ﬂoat) func (function): The function to call args (list): The call arguments kwargs (dict): The call keyword arguments Examples: Scheduled a function to be called once every 10 seconds:

def tensec(x,y=None): blah()

sched= Sched() sched.loop(10, tensec, 10, y='woot')

Notes: If the function returns False, the loop will explicitly break. If the task object is isﬁni’d, the loop will explicitly break. In either of those scenarios, the task will not be scheduled for further execution. Returns: s_task.Task: A Task object representing the object’s execution. persec(count, func, *args, **kwargs) Schedule a callback to occur count times per second. Args: count: Number of times per second for this to occur. Either an int or a ﬂoat. func: Function to execute. *args: Args passed to the function. **kwargs: Kwargs passed to the function. Examples: Scheduled a function to be called 10 times per second:

def tenpersec(x,y=None): blah()

sched= Sched() sched.persec(10, tenpersec, 10, y='woot')

Notes: This indefinitely calls the scheduled function until the function returns False or the Task is fini’d. See the Sched.loop function for more details. Returns: s_task.Task: A Task object representing the object’s execution. yieldTimeTasks() synapse.lib.scope module class synapse.lib.scope.Scope(*frames, **vals) Bases: object The Scope object assists in creating nested varible scopes. Example: with Scope() as scope: scope.set(‘foo’,10) with scope: scope.set(‘foo’,20) dostuff(scope) # ‘foo’ is 20... dostuff(scope) # ‘foo’ is 10 again... add(name, *vals) Add values as iter() compatible items in the current scope frame. ctor(name, func, *args, **kwargs) Add a constructor to be called when a specific property is not present. Example: scope.ctor(‘foo’,FooThing) ... foo = scope.get(‘foo’)

264 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

enter(vals=None) Add an additional scope frame. get(name, defval=None) Retrieve a value from the closest scope frame. iter(name) Iterate through values added with add() from each scope frame. leave() Pop the current scope frame. set(name, valu) Set a value in the current scope frame. update(vals) Set multiple values in the current scope frame. synapse.lib.scope.ctor(name, func, *args, **kwargs) Add a ctor callback to the global scope. synapse.lib.scope.enter(vals=None) Return the thread’s local scope for use in a with block synapse.lib.scope.get(name, defval=None) Access this thread’s scope with default values from glob. synapse.lib.scope.set(name, valu) Set a value in the current frame of the local thread scope. synapse.lib.scope.update(vals) synapse.lib.scrape module synapse.lib.scrape.scrape(text) Scrape types from a blob of text and return an ingest compatible dict. synapse.lib.scrape.splices(text, tags=()) Return a list of splice events for the give scrape output. synapse.lib.service module class synapse.lib.service.IdenProxy(svcprox, svcfo) Bases: synapse.lib.service.SvcBase class synapse.lib.service.SvcBase(svcprox) Bases: object class synapse.lib.service.SvcBus Bases: synapse.eventbus.EventBus getSynSvcs() Retrieve a list of the services on the service bus. Example: for name,info in sbus.getSynSvcs(): dostuff(name,info) getSynSvcsByTag(tag) Return a list of synapse services by hierarchical tag. Example:

4.1. Subpackages 265 Synapse Documentation, Release 0.0.34

for name,props in sbus.getSynSvcsByTag(‘foo.bar’): dostuff(name,props) iAmAlive(iden) “heartbeat” API for services. Example: sbus.iAmAlive(iden) Notes: This API is generally called by a scheduled loop within the service object. iAmSynSvc(iden, props) API used by synapse service to register with the bus. Example: sbus.iAmSynSvc(‘syn.blah’, foo=’bar’, baz=10) class synapse.lib.service.SvcMeth(svcbase, name) Bases: object class synapse.lib.service.SvcNameMeth(nameprox, name) Bases: object class synapse.lib.service.SvcNameProxy(svcprox, name) Bases: object Constructed by SvcProxy for simplifying callByName use. class synapse.lib.service.SvcProxy(sbus, timeout=None) Bases: synapse.eventbus.EventBus A client-side helper for service dispatches. Mostly exists to wrap functionality for calling multiple services by tag. callByIden(iden, func, *args, **kwargs) Call a speciﬁc object on the service bus by iden. Example: ret = svcprox.callByIden(iden,’getFooByBar’,bar) callByName(name, dyntask, timeout=None) Call a speciﬁc object on the service bus by name. Example: # dyntask tuple is (name,args,kwargs) dyntask = gentask(‘getFooByBar’,bar) ret = svcprox.callByName(‘foo0’, dyntask) callByTag(tag, dyntask, timeout=None) Call a method on all services with the given tag. Yields (svcfo,job) tuples for the results. Example: dyntask = gentask(‘getFooThing’) for svcfo,retval in svcprox.callByTag(‘foo.bar’,dyntask): dostuff(svcfo,retval) getNameProxy(name) Construct and return a SvcNameProxy to simplify callByName use. Example:

266 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

foosbars = svcprox.getNameProxy(‘foos_bars’) valu = foosbars.getBlahThing() dostuff(valu) getSynSvc(iden) Return the tufo for the speciﬁed svc iden ( or None ). Example: svcfo = svcprox.getSynSvc(iden) if svcfo != None: dostuff(svcfo) getSynSvcByName(name) getSynSvcs() Return the current list of known service tufos. Example: for svcfo in svcprox.getSynSvcs(): dostuff(svcfo) getSynSvcsByTag(tag) Return a list of service tufos by tag. Example: for svcfo in svcprox.getSynSvcsByTag(tag): dostuff(svcfo) getTagProxy(tag) Construct and return a SvcTagProxy to simplify callByTag use. Example: foosbars = svcprox.getTagProxy(‘foos.bars’) for valu in foosbars.getBlahThing(): dostuff(valu) runSynSvc(name, item, tags=(), **props) Publish an object to the service bus with the given tags. Example: foo = Foo() svcprox.runSynSvc(‘foo0’, foo, tags=(‘foos.foo0’,)) setSynSvcTimeout(timeout) class synapse.lib.service.SvcTagMeth(tagprox, name) Bases: object class synapse.lib.service.SvcTagProxy(svcprox, tag) Bases: object Constructed by SvcProxy for simplifying callByTag use. synapse.lib.service.openurl(url, **opts) Open a remote service bus and return a SvcProxy class. Example: svcprox = openbus(‘tcp://svcbus.com/mybus‘) synapse.lib.service.runSynSvc(name, item, sbus, tags=(), **props) Add an object as a synapse service. Example:

4.1. Subpackages 267 Synapse Documentation, Release 0.0.34

woot = Woot() sbus = s_telepath.openurl(‘tcp://1.2.3.4:90/syn.svcbus‘) runSynSvc(‘syn.woot’, woot, sbus)

synapse.lib.session module

class synapse.lib.session.Curator(conf=None) Bases: synapse.lib.config.Config The Curator class manages sessions. get(iden=None) Return a Session by iden. Args: iden (str): The guid (None creates a new sess). Returns: (Sess) class synapse.lib.session.Sess(iden, **props) Bases: synapse.eventbus.EventBus A synapse session to store prop/vals. get(prop) Retrieve a session property by name. Args: prop (str): The property name to retrieve. Returns: (obj): The property valu (or None) set(prop, valu) Set a session property to the given value. Args: prop (str): The name of the session property valu (obj): The property valu synapse.lib.socket module class synapse.lib.socket.Plex Bases: synapse.eventbus.EventBus Manage multiple Sockets using a multi-plexor IO thread. addPlexSock(sock) Add a Socket to the Plex() Args: sock (Socket): Socket to add. Example: plex.addPlexSock(sock) getPlexSocks() Return a list of the Socket()s managed by the Plex(). Returns: ([Socket(),...]): The list of Socket() instances. wake() class synapse.lib.socket.Socket(sock, **info) Bases: synapse.eventbus.EventBus Wrapper for the builtin socket.Socket class. Args: sock socket.socket: socket to wrap **info:

268 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

accept() close() Hook the socket close() function to trigger ﬁni() get(prop, defval=None) Retrieve a property from the socket’s info dict. Example: if sock.get(‘listen’): dostuff() recv(size) Slightly modiﬁed recv function which masks socket errors. ( makes them look like a simple close ) Additionally, any non-blocking recv’s with no available data will return None! recvall(size) Recieve the exact number of bytes requested. Returns None on if socket closes early. Example: byts = sock.recvall(300) if byts == None: return dostuff(byts)

Notes: • this API will trigger ﬁni() on close

recvobj() runTxLoop() Run a pass through the non-blocking tx loop. Returns: (bool): True if there is still more work to do rx() Yield any completed mesg tufos (type,info) in the recv buffer. Example: for mesg in sock.rx(): dostuff(mesg) send(byts) set(prop, valu) Set a property on the Socket by name. Example: sock.set(‘woot’, 30) setblocking(valu) Set the socket’s blocking mode to True/False. Args: valu (bool): False to set socket non-blocking tx(mesg) Transmit a mesg tufo ( type, info ) via the socket using msgpack. If present this API is safe for use with a socket in a Plex(). txbytes(byts)

4.1. Subpackages 269 Synapse Documentation, Release 0.0.34 synapse.lib.socket.connect(sockaddr, **sockinfo) Simpliﬁed connected TCP socket constructor. synapse.lib.socket.hostaddr(dest=‘8.8.8.8’) Retrieve the ipv4 address for this host ( optionally as seen from dest ). Example: addr = s_socket.hostaddr() synapse.lib.socket.inet_ntop(afam, byts) Implements classic socket.inet_ntop regardless of platform. (aka windows) synapse.lib.socket.inet_pton(afam, text) Implements classic socket.inet_pton regardless of platform. (aka windows) synapse.lib.socket.listen(sockaddr, **sockinfo) Simpliﬁed listening socket contructor. synapse.lib.socket.socketpair() Standard sockepair() on posix systems, and pure shinanegans on windows. synapse.lib.socket.sockgzip(byts) synapse.lib.storm module class synapse.lib.storm.LimitHelp(limit) Bases: object dec(size=1) get() reached() class synapse.lib.storm.OperWith(query, oper) Bases: object class synapse.lib.storm.Query(data=(), maxtime=None) Bases: object add(tufo) Add a tufo to the current query result set. cancel() Cancel the current query ( occurs at next tick() call ). clear() data() load(name) log(**info) Log execution metadata for the current oper. opt(name) Return the current value of a query option. result() retn() Return the results data or raise exception. save(name, data)

270 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

setOpt(name, valu) Set a query option to the given value. size() Get the number of tufos currently in the query. take() Return and clear the current result set. ( used by ﬁltration operators ) tick(touch=1) withop(oper) exception synapse.lib.storm.QueryCancelled Bases: synapse.lib.storm.QueryKilled exception synapse.lib.storm.QueryKilled Bases: Exception exception synapse.lib.storm.QueryLimitTime Bases: synapse.lib.storm.QueryKilled exception synapse.lib.storm.QueryLimitTouch Bases: synapse.lib.storm.QueryKilled class synapse.lib.storm.Runtime(**opts) Bases: synapse.lib.config.Configable ask(text, data=(), timeout=None) Run a storm query and return the query result dict. user= stdin= eval(text, data=(), timeout=None) Run a storm query and return only the result data. Example: for tufo in runt.eval(‘foo:bar=10 +foo:baz<30’): dostuff(tufo) getCmprFunc(oper) Return a comparison function for the given operator. getLiftLimit(*limits) Return the lift() result limit for the current user/runtime. Args:* limits: Optional list of requested int limit values Returns: (int): The lift limit value or None getLiftLimitHelp(*limits) Return a LimitHelp object for the speciﬁed limits or defaults. Args: limits (list): A list of int/None limits Returns: (LimitHelp) getStormCore(name=None) Return the (optionally named) cortex for use in direct storm operators. parse(text) plan(opers) Review a list of opers and return a potentially optimized list. Args: opers ([(str,dict),]): List of opers in tufo form Returns: ([(str,dict),]): Optimized list

4.1. Subpackages 271 Synapse Documentation, Release 0.0.34

run(opers, data=(), timeout=None) Execute a pre-parsed set of opers. Example: opers = runt.parse(‘foo:bar=20 +foo:baz | foo:faz=30’) # ... some time later... res0 = runt.run(opers) setCmprCtor(name, func) Add a comparitor constructor function for use in the “filt” operator. Example: def substr(oper): prop = oper[1].get(‘prop’) valu = oper[1].get(‘valu’) def cmpr(tufo): tufo[1].get(prop).find(valu) != -1 return cmpr runt.setCmprCtor(‘substr’,substr) # storm syntax now suports # +foo:bar*substring=”baz” and -foo:bar*substring=”baz” setCmprFunc(name, func) Helper function for adding simple comparitors. Example: def substr(x,y): return x.find(y) != -1 runt.setCmprFunc(‘substr’, substr)

NOTE: Under the hood, this API dynamically generates a cmpr ctor add adds it using addCmprCtor.

setOperFunc(name, func) Add a handler function for a given operator. The function must implement the convention: def func(query,oper): args = oper[1].get(‘args’) opts = dict(oper[1].get(‘kwlist’)) dostuff() Where query is a synapse.lib.storm.Query instance and oper is a (,) tufo. stormTufosBy(by, prop, valu=None, limit=None) A STORM runtime speciﬁc version of the cortex function getTufosBy which allows sub-classes to override the default behavior for operators like lift/join/pivot. class synapse.lib.storm.ShowHelp(core, show) Bases: object The ShowHelp class implements routines for formatting storm query output based on the embedded “show” directive within the query results. pad(rows) Pad a series of column values for aligned display. rows(nodes) Return a list of display columns for the given nodes. Args: nodes (((str,dict),...)): A list of nodes in tuple form. Returns: ( [(str,...), ...] ): A list of column lists containing strings synapse.lib.storm.eq(x, y) synapse.lib.storm.ge(x, y)

272 Chapter 4. synapse package Synapse Documentation, Release 0.0.34 synapse.lib.storm.gt(x, y) synapse.lib.storm.invert(func) synapse.lib.storm.le(x, y) synapse.lib.storm.lt(x, y) synapse.lib.storm.setkw(oper, name, valu) Set a keyword argument in a given operator. Args: oper ((str,dict)): A storm operator tufo name (str): A kwarg name valu (obj): A kwarg value Returns: (None) synapse.lib.syntax module synapse.lib.syntax.is_literal(text, off ) synapse.lib.syntax.isquote(text, off ) synapse.lib.syntax.meh(txt, off, cset) synapse.lib.syntax.nextchar(text, off, valu) synapse.lib.syntax.nextin(text, off, vals) synapse.lib.syntax.nextstr(text, off, valu) synapse.lib.syntax.nom(txt, off, cset, trim=True) Consume chars in set from the string and return (subtxt,offset). Example: text = “foo(bar)” chars = set(‘abcdefghijklmnopqrstuvwxyz’) name,off = nom(text,0,chars) synapse.lib.syntax.nom_whitespace(text, off ) synapse.lib.syntax.oper(name, *args, **kwargs) synapse.lib.syntax.parse(text, off=0) Parse and return a set of instruction tufos. synapse.lib.syntax.parse_cmd_kwarg(text, off=0) Parse a foo:bar= kwarg into (prop,valu),off synapse.lib.syntax.parse_cmd_kwlist(text, off=0) Parse a foo:bar=[,...] kwarg list into (prop,valu),off synapse.lib.syntax.parse_cmd_string(text, off, trim=True) Parse in a command line string which may be quoted. synapse.lib.syntax.parse_int(text, off, trim=True) synapse.lib.syntax.parse_list(text, off=0, trim=True) Parse a list (likely for comp type) coming from a command line input. The string elements within the list may optionally be quoted. synapse.lib.syntax.parse_literal(text, off, trim=True) synapse.lib.syntax.parse_macro_filt(text, off=0, trim=True, mode=’must’) synapse.lib.syntax.parse_macro_join(text, off=0) &foo:bar &foo:bar=baz:faz &hehe.haha/foo:bar=baz:faz

4.1. Subpackages 273 Synapse Documentation, Release 0.0.34 synapse.lib.syntax.parse_macro_lift(text, off=0, trim=True) Parse a “lift” macro and return an inst,off tuple. synapse.lib.syntax.parse_oper(text, off=0) Returns an inst,off tuple by parsing an operator expression. Example: inst,off = parse_oper(‘foo(“bar”,baz=20)’) synapse.lib.syntax.parse_opts(text, off=0) synapse.lib.syntax.parse_perm(text, off=0) Parse a permission string [=...] synapse.lib.syntax.parse_ques(text, off=0, trim=True) Parse “query” syntax: tag/prop[@][^][*][=valu] synapse.lib.syntax.parse_string(text, off, trim=True) synapse.lib.syntax.parse_time(text, off ) synapse.lib.syntax.parse_valu(text, off=0) Special syntax for the right side of equals in a macro synapse.lib.tags module class synapse.lib.tags.ByTag Bases: object A dictionary style put/get API using tags. get(tag) Retrieve items by a tag. Example: for item in btag.get(‘foo.bar’): dostuff(item) pop(item, dval=None) Remove an item previously added to the ByTag. put(item, tags) Add an item for ByTag lookup with tags. Example: btag.put( woot, (‘woots.woot0’, ‘foo’) ) synapse.lib.tags.getTufoSubs(tufo, tag) Return a list of tufo props for the given tag (and down). Args: tufo ((str,dict)): A node in tuple form tag (str): A tag name Returns: synapse.lib.tags.iterTagDown(tag, div=’.’) Yield tags from top to bottom. Example: iterTagDown(‘foo.bar.baz’) -> (‘foo’,’foo.bar’,’foo.bar.baz’)

274 Chapter 4. synapse package Synapse Documentation, Release 0.0.34 synapse.lib.tags.iterTagUp(tag, div=’.’) Yield tags from top to bottom. Example: iterTagUp(‘foo.bar.baz’) -> (‘foo.bar.baz’,’foo.bar’,’foo’) synapse.lib.tags.tufoHasTag(tufo, tag) Returns True if the tufo has the given tag. Example: if tufoHasTag(tufo,’woot’): dostuff() synapse.lib.task module class synapse.lib.task.CallTask(call) Bases: synapse.lib.task.Task An extension for a runnable task. Args: call ((func,[],{})): A tuple of call details. class synapse.lib.task.Task(iden=None) Bases: synapse.eventbus.EventBus A cancelable Task abstraction which operates much like a Future but with some additional features. err(info) Fire an error return value for the task. Args: info (dict): Exception info dict (see synapse.common.excinfo ) get(prop, defval=None) Get a value from the info dict for the task. Args: prop (str): The name of the info value. defval (obj): The default value to return if not found

Returns: (obj): The object from the info dict (or None)

onretn(func) Provide a function to receive return values. The speciﬁfed callback will be called with a retn tuple deﬁned as (ok,valu). If ok is True, valu is a return valu, if ok is False, valu is an excinfo dictionary. Args: func (function): Callback for a retn tuple. retn(valu) Fire a result return value for the task. Args: valu (obj): The return value run() Execute the task. set(prop, valu) Set a value in the info dict for the task. Args: prop (str): The name of the info dict value valu (obj): The value to set in the info dict

4.1. Subpackages 275 Synapse Documentation, Release 0.0.34 synapse.lib.thishost module synapse.lib.thishost.get(prop) Retrieve a property from the hostinfo dictionary. Example: import synapse.lib.thishost as s_thishost if s_thishost.get(‘platform’) == ‘windows’: dostuff() synapse.lib.thisplat module synapse.lib.threads module class synapse.lib.threads.Pool(size=3, maxsize=None) Bases: synapse.eventbus.EventBus A thread pool for ﬁring and cleaning up threads. The Pool() class can be used to keep persistant threads for work processing as well as optionally spin up new threads to handle “bursts” of activity. # ﬁxed pool of 16 worker threads pool = Pool(size=16) # dynamic pool of 5-10 workers pool = Pool(size=5, maxsize=10) # dynamic pool of 8- workers pool = Pool(size=8, maxsize=-1) call(func, *args, **kwargs) Call the given func(*args,**kwargs) in the pool. task(func, *args, **kwargs) Call the given function in the pool with a task. NOTE: Callers must use with-block syntax. Example: def foo(x): dostuff() def onretn(valu): otherstuff() with pool.task(foo, 10) as task: task.onretn(onretn) # the task is queued for execution after we # leave the with block. wrap(func) Wrap a function to transparently dispatch via the pool. Example: # dispatch the message handler from a pool bus.on(‘foo’, pool.wrap( doFooThing ) ) class synapse.lib.threads.RWLock Bases: object A multi-reader/exclusive-writer lock. reader() Acquire a multi-reader lock. Example: lock = RWLock()

276 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

with lock.reader(): # other readers can be here too... dowrites() release(holder) Used to release an RWWith holder ( you probably shouldn’t use this ) writer() Acquire an exclusive-write lock. Example: lock = RWLock() with lock.writer(): # no readers or other writers but us! dowrites() class synapse.lib.threads.RWWith(rwlock) Bases: object The RWWith class implements “with block” syntax for RWLock. class synapse.lib.threads.Thread(func, *args, **kwargs) Bases: threading.Thread, synapse.eventbus.EventBus A thread / EventBus to allow ﬁni() etc. run() class synapse.lib.threads.cancelable(func, *args, **kwargs) Bases: object Use these to allow cancelation of blocking calls (where possible) to shutdown threads. Example: with cancelable(sock.close): byts = sock.recv(100) synapse.lib.threads.current() synapse.lib.threads.iCantWait(name=None) Mark the current thread as a no-wait thread. Any no-wait thread will raise MustNotWait on blocking calls within synapse APIs to prevent deadlock bugs. Example: iCantWait(name=’FooThread’) synapse.lib.threads.iMayWait() Function for no-wait aware APIs to use while handling no-wait threads. Example: def mayWaitThing(): if not iMayWait(): return False waitForThing() synapse.lib.threads.iWillWait() Check if the current thread is a marked no-wait thead and raise MustNotWait. Example: def doBlockingThing(): iWillWait() waitForThing() synapse.lib.threads.iden() synapse.lib.threads.isfini() synapse.lib.threads.newtask(func, *args, **kwargs) synapse.lib.threads.withlock(lock)

4.1. Subpackages 277 Synapse Documentation, Release 0.0.34 synapse.lib.threads.worker(func, *args, **kwargs) Fire a worker thread to run the given func(*args,**kwargs) synapse.lib.time module

Time related utilities for synapse “epoch millis” time values. synapse.lib.time.parse(text, base=None, chop=False) Parse a time string into an epoch millis value. synapse.lib.time.repr(tick, pack=False) Return a date string for an epoch-millis timestamp. Args: tick (int): The timestamp in milliseconds since the epoch. Returns: (str): A date time string synapse.lib.trees module

A simple implementation of an interval tree to lookup potentially overlapping intervals from a point. class synapse.lib.trees.IntervalTree(ivals) Bases: object Construct an interval tree from the inputs. https://en.wikipedia.org/wiki/Interval_tree Example: ivals = ( ((1,30),{}), ((18,33),{}), ... ) itree = IntervalTree(ivals) for ival in itree.get(12): dostuff(ival) get(valu) Return intervals which contain the speciﬁed value. Example: for ival in itree.get(valu): dostuff(ival) synapse.lib.trigger module class synapse.lib.trigger.Triggers Bases: object add(func, perm) Add a new callback to the triggers. Args: func (function): The function to call perm (str,dict): The permission tufo Returns: (None) clear() Clear all previously registered triggers trigger(perm, *args, **kwargs) Fire any matching trigger functions for the given perm.

278 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

Args: perm ((str,dict)): The perm tufo to trigger *args (list): args list to use calling the trigger function **kwargs (dict): kwargs dict to use calling the trigger function Returns: (None) synapse.lib.tufo module

Some common utility functions for dealing with tufos. synapse.lib.tufo.ephem(form, fval, **props) synapse.lib.tufo.equal(tuf0, tuf1) Since dicts are not comparible, this implements equality comparison for a given tufo by comparing and orders list of (prop,valu) pairs. Example: import synapse.lib.tufo as s_tufo tuf0 = s_tufo.tufo(‘foo’,bar=10,baz=20) tuf1 = s_tufo.tufo(‘foo’,baz=20,bar=10) if s_tufo.equal(tuf0,tuf1): print(‘woot’)

NOTE: This API is not particularly fast and is mostly for implementing tests. synapse.lib.tufo.ival(tufo, name) Return a min,max interval tuple or None for the node. Args: tufo ((str,dict)): A node in tuple form name (str): The name of the interval to return Returns: (int,int) An interval value ( or None ) synapse.lib.tufo.ndef(tufo) Return a node deﬁnition (, tuple from the tufo. Args: tufo ((str,dict)): A node in tuple form Returns: ((str,obj)): The (,) tuple for the node synapse.lib.tufo.props(tufo, pref=None) Return the relative props from the given tufo preﬁx. ( or from the form name by default ) Example: import synapse.tufo as s_tufo tuf0 = s_tufo.tufo(‘bar’, **{‘baz’:’faz’, ‘derp’: 20, ‘namespace:sound’: ‘quack’}) # tuf0 = (‘bar’, {‘namespace:ducksound’: ‘quack’, ‘derp’: 20, ‘baz’: ‘faz’}) info = s_tufo.props(tuf0, pref=’namespace’) # info = {‘ducksound’: ‘quack’} tuf1 = s_tufo.tufo(‘duck’, **{‘tufo:form’: ‘animal’, ‘animal:sound’:’quack’, ‘animal:stype’: ‘duck’}) # tuf1 = (‘duck’, {‘tufo:form’: ‘animal’, ‘animal:stype’: ‘duck’, ‘animal:sound’: ‘quack’}) info = s_tufo.props(tuf1) # info = {‘stype’: ‘duck’, ‘sound’: ‘quack’} synapse.lib.tufo.tagged(tufo, tag) synapse.lib.tufo.tags(tufo, leaf=False) synapse.lib.tufo.tufo(name, **props) Convenience / syntax sugar for tufo construction. Example: tuf0 = s_tufo.tufo(‘bar’,baz=’faz’,derp=20) # tuf0 = (‘bar’,{‘baz’:’faz’, ‘derp’:20})

4.1. Subpackages 279 Synapse Documentation, Release 0.0.34 synapse.lib.types module class synapse.lib.types.BoolType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) repr(valu) class synapse.lib.types.CompType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) class synapse.lib.types.DataType(tlib, name, **info) Bases: object extend(name, **info) Construct a new subtype from this instance. get(prop, defval=None) Retrieve a type info property from this type or parent types. Example: ex = item.get(‘doc’) parse(text, oldval=None) Parse input text and return the system mode (normalized) value for the type. Example: valu = tobj.parse(text) repr(valu) subprops = () subs() Implement if the presence of a property with this type requires sub props. class synapse.lib.types.GuidType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) class synapse.lib.types.IntType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) repr(valu) class synapse.lib.types.JsonType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) class synapse.lib.types.MultiFieldType(tlib, name, **info) Bases: synapse.lib.types.DataType class synapse.lib.types.NDefType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None)

280 Chapter 4. synapse package Synapse Documentation, Release 0.0.34 class synapse.lib.types.PermType(tlib, name, **info) Bases: synapse.lib.types.DataType Enforce that the permission string and options are known. norm(valu, oldval=None) class synapse.lib.types.PropValuType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) class synapse.lib.types.SeprType(tlib, name, **info) Bases: synapse.lib.types.MultiFieldType norm(valu, oldval=None) class synapse.lib.types.StormType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) class synapse.lib.types.StrType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) class synapse.lib.types.TagType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) class synapse.lib.types.TimeType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) repr(valu) class synapse.lib.types.TypeLib(load=True) Bases: object An extensible type library for use in cortex data models. addDataModel(name, modl) addDataModels(modtups) Load a list of (name,model) tuples. Args: modtups ([(str,dict)]): A list of (name,modl) tuples. Returns: (None) NOTE: This API loads all types ﬁrst and may therefor be used to prevent type dependency ordering issues between multiple models. addType(name, **info) Add a type to the cached types. Args: name (str): Name of the type to add. **info (dict): Type properties to include. Example: Add a new foo:bar type:

tlib.addType('foo:bar', subof='str', doc='A foo bar.')

Raises: DupTypeName: If the type already exists.

4.1. Subpackages 281 Synapse Documentation, Release 0.0.34

addTypeCast(name, func) Add a “cast” function to do normalization without creating a complete type. getDataType(name) Return the DataType subclass for the given type name. getTypeBases(name) Return a list of type inheritence names beginning with the base type. Example: for base in tlib.getTypeBases(‘foo:minval’): print(‘base type: %s’ % (name,)) getTypeCast(name, valu) Use either a type or a registered “cast” name to normalize the given input. Example: valu = tlib.getTypeCast(“foo:bar”,”hehe”) getTypeDef(name) Get the deﬁnition for a given type. Args: name (str): Name of the type to look up. Examples: Do stuff with the type deﬁnition of ‘int’:

tdef= tlib.getTypeDef('int') dostuff(tdef)

Returns: ((str, dict)): The type deﬁnition tufo. The str is the name of the type, and the dictionary are any type options (ctor and subof values). If the name is not a registered type, this is None. getTypeDefs() Return a list of (name,info) tuples for all the types. Returns: ([(name,info)]): The loaded types getTypeInfo(name, prop, defval=None) A helper to return an info prop for the type or it’s parents. Example: ex = tlib.getTypeInfo(‘inet:tcp4’,’ex’) getTypeInst(name) Return the DataType instance for the given type name. Example: dtype = tlib.getTypeInst(‘foo:bar’)

NOTE: This API returns non-primitive objects and can not be used over telepath RMI.

getTypeNorm(name, valu, oldval=None) Normalize a type speciﬁc value in system mode. Example: fqdn,subs = tlib.getTypeNorm(‘inet:fqdn’,’Foo.Com’)

282 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

getTypeParse(name, text) Parse input text for the given type into it’s system form. Example: ipv4,subs = tlib.getTypeParse(‘inet:ipv4’,‘1.2.3.4’) getTypeRepr(name, valu) Return the humon readable form of the given type value. Example: print( tlib.getTypeRepr(‘inet:ipv4’, ipv4addr) ) isDataModl(name) Return True if the given data model name exists. Args: name (str): The name of the data model Returns: (boolean): True if the model exists isDataType(name) Return boolean which is true if the given name is a data type. Example: if tlib.isDataType(‘foo:bar’): dostuff() isSubType(name, base) Returns True if the given type name is a sub-type of the base name. Example: if tlib.isSubType(‘foo’,’str’): dostuff() loadModModels() reqDataType(name) Return a reference to the named DataType or raise NoSuchType. Args: name (str): Name of the type to get a reference for. Returns: DataType: Instance of a DataType for that name. Raises: NoSuchType: If the type is not valid. class synapse.lib.types.XrefType(tlib, name, **info) Bases: synapse.lib.types.DataType The XrefType allows linking a speciﬁc type of node to an inspeciﬁc set of node forms. Example Sub Type: addType(‘foo:barrefs’, subof=’xref’, source=’bar,foo:bar’) norm(valu, oldval=None) synapse.lib.types.deMsgB64(text) synapse.lib.types.enMsgB64(item) synapse.lib.types.islist(x)

4.1. Subpackages 283 Synapse Documentation, Release 0.0.34 synapse.lib.urlhelp module synapse.lib.urlhelp.chopurl(url) A sane “stand alone” url parser. Example: info = chopurl(url) synapse.lib.version module

Synapse utilites for dealing with Semvar versioning. This includes the Synapse version information. synapse.lib.version.fmtVersion(*vsnparts) Join a string of parts together with a . separator. Args:* vsnparts: Returns: synapse.lib.version.packVersion(major, minor=0, patch=0) Pack a set of major/minor/patch integers into a single integer for storage. Args: major (int): Major version level integer. minor (int): Minor version level integer. patch (int): Patch version level integer. Returns: int: System normalized integer value to represent a software version. synapse.lib.version.parseSemver(text) Parse a Semantic Version string into is component parts. Args: text (str): A text string to parse into semver components. This string has whitespace and leading ‘v’ characters stripped off of it. Examples: Parse a string into it semvar parts:

parts= parseSemver('v1.2.3')

Returns: dict: The dictionary will contain the keys ‘major’, ‘minor’ and ‘patch’ pointing to integer values. The dictionary may also contain keys for ‘build’ and ‘pre’ information if that data is parsed out of a semver string. None is returned if the string is not a valid Semver string. synapse.lib.version.parseVersionParts(text, seps=(‘.’, ‘-‘, ‘_’, ‘+’)) Extract a list of major/minor/version integer strings from a string. Args: text (str): String to parse seps (tuple): A tuple or list of separators to use when parsing the version string. Examples: Parse a simple version string into a major and minor parts:

parts= parseVersionParts('1.2')

Parse a complex version string into a major and minor parts:

parts= parseVersionParts('wowsoft_1.2')

Parse a simple version string into a major, minor and patch parts. Parts after the “3.” are dropped from the results:

parts= parseVersionParts('1.2.3.4.5')

284 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

Notes: This attempts to brute force out integers from the version string by stripping any leading ascii letters and part separators, and then regexing out numeric parts optionally followed by part separators. It will stop at the ﬁrst mixed-character part encountered. For example, “1.2-3a” would only parse out the “1” and “2” from the string. Returns: dict: Either a empty dictionary or dictionary containing up to three keys, ‘major’, ‘minor’ and ‘patch’. synapse.lib.version.unpackVersion(ver) Unpack a system normalized integer representing a softare version into its component parts. Args: ver (int): System normalized integer value to unpack into a tuple. Returns: (int, int, int): A tuple containing the major, minor and patch values shifted out of the integer. synapse.lib.webapp module class synapse.lib.webapp.BaseHand(application, request, **kwargs) Bases: tornado.web.RequestHandler get(*args) initialize(**globs) post(*args) sendHttpResp(code, headers, content) class synapse.lib.webapp.WebApp(**conf ) Bases: tornado.web.Application, synapse.daemon.DmonConf, synapse.lib.config. Config The WebApp class allows easy publishing of python methods as HTTP APIs. Example: class Woot: def getFooByBar(self, bar): stuff() woot = Woot() wapp = WebApp() wapp.addApiPath(‘/v1/foo/bybar/(.*)’, woot.getFooByBar ) wapp.listen(8080) wapp.main() addApiPath(regex, func, host=’.*’, perm=None) Add a path regex to allow access to a function. Example: def getFooByBar(self, bar): stuff() wapp.addApiPath(‘/v1/foo/(.+)’, getFooByBar) Notes: See parsetypes decorator for adding type safety addFilePath(regex, path, host=’.*’) Add a static ﬁle path ( or directory path ) to the WebApp. Example: wapp.addFilePath(‘/js/(.*)’, ‘/path/to/js’ )

4.1. Subpackages 285 Synapse Documentation, Release 0.0.34

addHandPath(regex, handler, host=’.*’, **globs) Add a BaseHand derived handler. getServBinds() Get a list of sockaddr tuples for the bound listeners. ( mostly used to facilitate unit testing ) listen(port, host=‘0.0.0.0’) Add a listener to the tornado HTTPServer. Example: wapp.listen(8080)

Module contents

4.1.6 synapse.links package

Submodules synapse.links.common module exception synapse.links.common.ImplementMe Bases: Exception class synapse.links.common.LinkRelay(link) Bases: object connect() Create, connect, and return a new client Socket() getLinkProp(name, defval=None) listen() Create and return a new server Socket() proto = None exception synapse.links.common.NoSuchProto Bases: Exception exception synapse.links.common.TooManyTries Bases: Exception synapse.links.common.raiseSockError(link, e) synapse.links.local module class synapse.links.local.LocalRelay(link) Bases: synapse.links.common.LinkRelay Implements the PF_UNIX/PF_LOCAL protocol for synapse. ( and named pipes on windows platforms ) local:/// proto = ‘local’

286 Chapter 4. synapse package Synapse Documentation, Release 0.0.34 synapse.links.ssh module class synapse.links.ssh.SshRelay(link) Bases: synapse.links.common.LinkRelay Implements the SSH link protocol for synapse. ssh://{[}user{[}:passwd]@][:port]/?forward=[&keyﬁle=] proto = ‘ssh’ synapse.links.ssl module class synapse.links.ssl.Socket(sock, **info) Bases: synapse.lib.socket.Socket send(byts) class synapse.links.ssl.SslRelay(link) Bases: synapse.links.common.LinkRelay proto = ‘ssl’ synapse.links.tcp module class synapse.links.tcp.TcpRelay(link) Bases: synapse.links.common.LinkRelay Implements the TCP protocol for synapse. tcp://{[}user{[}:passwd]@][:port]/ proto = ‘tcp’

Module contents

The package for the built in link protocol handlers.

4.1.7 synapse.lookup package

Submodules synapse.lookup.iana module synapse.lookup.iso3166 module

Provides data for the ISO 3166-1 Country codes. Reference: https://en.wikipedia.org/wiki/ISO_3166 synapse.lookup.iso3166.makeColLook(rows, scol, dcol)

4.1. Subpackages 287 Synapse Documentation, Release 0.0.34

synapse.lookup.phonenum module

synapse.lookup.phonenum.formPhoneNode(node, valu) synapse.lookup.phonenum.getPhoneInfo(numb) Walk the phone info tree to ﬁnd the best-match info for the given number. Example: info = getPhoneInfo(17035551212) country = info.get(‘cc’) synapse.lookup.phonenum.initPhoneTree() synapse.lookup.phonenum.phnode(valu)

Module contents

4.1.8 synapse.models package

Subpackages synapse.models.gov package

Submodules synapse.models.gov.cn module class synapse.models.gov.cn.GovCnMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() initCoreModule() onAddMucd(node) synapse.models.gov.intl module class synapse.models.gov.intl.GovIntlMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() synapse.models.gov.us module class synapse.models.gov.us.GovUsMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels()

288 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

Module contents

Submodules

synapse.models.axon module

class synapse.models.axon.AxonMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() initCoreModule() onFormAxonPath(form, valu, props, mesg) class synapse.models.axon.AxonPathType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None)

synapse.models.biology module

class synapse.models.biology.BioMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels()

synapse.models.chemistry module

class synapse.models.chemistry.ChemMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels()

synapse.models.crypto module

class synapse.models.crypto.CryptoMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() synapse.models.dns module class synapse.models.dns.DnsMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() synapse.models.ﬁles module class synapse.models.files.FileBaseType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) repr(valu)

4.1. Subpackages 289 Synapse Documentation, Release 0.0.34 class synapse.models.files.FileMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() initCoreModule() seedFileGoodHash(prop, valu, **props) Hashes that we consider “cardinal enough” to pivot. seedFileMd5(prop, valu, **props) seedFileSha1(prop, valu, **props) class synapse.models.files.FilePathType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) class synapse.models.files.FileRawPathType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) synapse.models.ﬁnance module class synapse.models.finance.FinMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() synapse.models.geopol module class synapse.models.geopol.PolMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() synapse.models.geospace module class synapse.models.geospace.GeoMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() synapse.models.inet module class synapse.models.inet.AddrType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) class synapse.models.inet.CidrType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) repr(valu)

290 Chapter 4. synapse package Synapse Documentation, Release 0.0.34 class synapse.models.inet.EmailType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) repr(valu) subprops = ((‘user’, {‘ptype’: ‘inet:user’}), (‘fqdn’, {‘ptype’: ‘inet:fqdn’})) class synapse.models.inet.FqdnType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) repr(valu) subprops = ((‘sfx’, {‘ptype’: ‘bool’}), (‘zone’, {‘ptype’: ‘bool’}), (‘domain’, {‘ptype’: ‘inet:fqdn’}), (‘host’, {‘ptype’: ‘str’})) class synapse.models.inet.IPv4Type(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) repr(valu) class synapse.models.inet.IPv6Type(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) repr(valu) class synapse.models.inet.InetMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() initCoreModule() onSetFqdnSfx(mesg) onTufoFormFqdn(form, valu, props, mesg) onTufoFormPasswd(form, valu, props, mesg) class synapse.models.inet.Srv4Type(tlib, name, **info) Bases: synapse.lib.types.DataType Base type for : format. norm(valu, oldval=None) repr(valu) subprops = ((‘port’, {‘ptype’: ‘inet:port’}), (‘ipv4’, {‘ptype’: ‘inet:ipv4’})) class synapse.models.inet.Srv6Type(tlib, name, **info) Bases: synapse.lib.types.DataType Base type for [IPv6]:port format. norm(valu, oldval=None) repr(valu) subprops = ((‘port’, {‘ptype’: ‘inet:port’}), (‘ipv6’, {‘ptype’: ‘inet:ipv6’})) class synapse.models.inet.UrlType(tlib, name, **info) Bases: synapse.lib.types.DataType

4.1. Subpackages 291 Synapse Documentation, Release 0.0.34

norm(valu, oldval=None) repr(valu) subprops = ((‘proto’, {‘ptype’: ‘str’}), (‘path’, {‘ptype’: ‘str’}), (‘fqdn’, {‘ptype’: ‘inet:fqdn’}), (‘ipv4’, {‘ptype’: ‘inet:ipv4’}), (‘ipv6’, {‘ptype’: ‘inet:ipv6’}), (‘port’, {‘ptype’: ‘inet:port’}), (‘user’, {‘ptype’: ‘inet:user’}), (‘passwd’, {‘ptype’: ‘inet:passwd’})) synapse.models.inet.castInetDeFang(valu) synapse.models.inet.ipv4cidr(valu) synapse.models.inet.ipv4int(valu) synapse.models.inet.ipv4mask(ipv4, mask) synapse.models.inet.ipv4str(valu) synapse.models.inet.ipv6norm(text) Normalize an IPv6 address into RFC5952 canonical form. Example: text = ipv6norm(text)

synapse.models.infotech module

class synapse.models.infotech.ItMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() initCoreModule() class synapse.models.infotech.SemverType(tlib, name, **info) Bases: synapse.lib.types.DataType Provides support for parsing a semantic version string into its component parts. This normalizes a version string into a integer to allow version ordering. Prerelease information is disregarded for integer comparison purposes, as we cannot map an arbitrary pre-release version into a integer value Major, minor and patch levels are represented as integers, with a max width of 20 bits. The comparable integer value representing the semver is the bitwise concatenation of the major, minor and patch levels. Prerelease and build information will be parsed out and available as strings if that information is present. norm(valu, oldval=None) repr(valu) subprops = ((‘major’, {‘ptype’: ‘int’}), (‘minor’, {‘ptype’: ‘int’}), (‘patch’, {‘ptype’: ‘int’}), (‘build’, {‘ptype’: ‘str:txt’}), (‘prerelease’, {‘ptype’: ‘str:txt’})) synapse.models.infotech.bruteStr(valu) Brute force the version out of a string. Args: valu (str): String to attempt to get version information for. Notes: This ﬁrst attempts to parse strings using the it:semver normalization before attempting to extract version parts out of the string. Returns: int, dict: The system normalized version integer and a subs dictionary. synapse.models.infotech.bruteVersion(valu) Attempt to brute force a valu into a semantic version string and its components Args: valu: A string or integer to attempt to obtain a system normalized version valu and subs for. Returns: int, dict: The system normalized version integer and a subs dictionary.

292 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

synapse.models.infotech.bruteVersionValu(valu) Return the system normalized version integer for a given input. Args: valu: String or integer to normalize. Returns: int: System normalized version value.

synapse.models.language module

class synapse.models.language.LangMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() synapse.models.material module

A data model focused on material objects. class synapse.models.material.MatMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() synapse.models.media module class synapse.models.media.MediaMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() synapse.models.mime module class synapse.models.mime.MimeMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() synapse.models.money module class synapse.models.money.MoneyMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() synapse.models.orgs module class synapse.models.orgs.OuMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() initCoreModule() seedOrgAlias(prop, valu, **props) seedOrgName(prop, valu, **props)

4.1. Subpackages 293 Synapse Documentation, Release 0.0.34

synapse.models.person module

class synapse.models.person.Name(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) subprops = ((‘sur’, {‘ptype’: ‘ps:tokn’}), (‘given’, {‘ptype’: ‘ps:tokn’}), (‘middle’, {‘ptype’: ‘ps:tokn’}), (‘parts’, {‘doc’: ‘Number of ps:tokn elements in ps:name’, ‘ptype’: ‘int’})) class synapse.models.person.PsMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() initCoreModule() seedPersonGuidName(prop, valu, **props)

synapse.models.science module

class synapse.models.science.SciMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels()

synapse.models.syn module

class synapse.models.syn.SynMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels()

synapse.models.telco module

class synapse.models.telco.ImeiType(tlib, name, **info) Bases: synapse.lib.types.DataType https://en.wikipedia.org/wiki/International_Mobile_Equipment_Identity norm(valu, oldval=None) class synapse.models.telco.ImsiType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) class synapse.models.telco.PhoneType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) repr(valu) class synapse.models.telco.TelMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() initCoreModule() synapse.models.telco.digits(text)

294 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

synapse.models.telco.genTelLocCast(iso2, cc, idd, size) Generate a generic phone canonicalizer for numbers which may reside within an arbitrary country’s local exchange. synapse.models.telco.imeicsum(text) Calculate the imei check byte. synapse.models.temporal module class synapse.models.temporal.EpochType(tlib, name, **info) Bases: synapse.lib.types.DataType norm(valu, oldval=None) repr(valu) class synapse.models.temporal.TimeMod(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() initCoreModule() synapse.models.temporal.fromUnixEpoch(valu)

Module contents

4.1.9 synapse.swarm package

Submodules synapse.swarm.runtime module class synapse.swarm.runtime.Runtime(svcbus, **opts) Bases: synapse.lib.storm.Runtime, synapse.eventbus.EventBus A STORM runtime capable of using a swarm cluster

Module contents

4.1.10 synapse.tests package

Submodules synapse.tests.common module exception synapse.tests.common.TooFewEvents Bases: Exception synapse.tests.common.getIngestCore(path, core=None) synapse.tests.common.getTestPath(*paths)

4.1. Subpackages 295 Synapse Documentation, Release 0.0.34

synapse.tests.nopmod module

A python module used for testing dmon dynamic module loading.

synapse.tests.test_async module

class synapse.tests.test_async.AsyncTests(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_async_basics() test_async_custom_pool_basics() test_async_ondone() test_async_pool_basics() test_async_sugar() test_async_sync() test_async_timeout() test_async_wait_syntimeout() test_async_wait_timeout() synapse.tests.test_axon module class synapse.tests.test_axon.AxonClusterTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_axon_cluster() test_axon_cluster_cortex() class synapse.tests.test_axon.AxonFSTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_axon__fs_isdir(*args, **kwargs) test_axon__fs_isfile(*args, **kwargs) test_axon_fs_create() test_axon_fs_getattr() test_axon_fs_getxattr() test_axon_fs_mkdir() test_axon_fs_read(*args, **kwargs) test_axon_fs_readdir(*args, **kwargs) test_axon_fs_rename(*args, **kwargs) test_axon_fs_rmdir(*args, **kwargs) test_axon_fs_truncate(*args, **kwargs) test_axon_fs_unlink(*args, **kwargs) test_axon_fs_utimens(*args, **kwargs)

296 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

test_axon_get_renameprops(*args, **kwargs) class synapse.tests.test_axon.AxonHostTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_axon_autorun() test_axon_clone_large() test_axon_host() test_axon_host_clone() test_axon_host_free_limit() test_axon_host_maxsize_limit() test_axon_host_spinbackup() class synapse.tests.test_axon.AxonTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_axon_basics() test_axon_bytesize() test_axon_eatbytes() test_axon_restrictions() test_axon_telepath() synapse.tests.test_cmds_cortex module class synapse.tests.test_cmds_cortex.SynCmdCoreTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest getCoreCmdr(core) test_cmds_ask() test_cmds_ask_debug() test_cmds_ask_exc_response() test_cmds_ask_multilift() test_cmds_ask_mutual_exclusive() test_cmds_ask_noopts() test_cmds_ask_null_response() test_cmds_ask_props() test_cmds_ask_raw() test_cmds_ask_tagtime() test_cmds_guid() test_cmds_help() test_cmds_py() test_cmds_quit()

4.1. Subpackages 297 Synapse Documentation, Release 0.0.34

synapse.tests.test_common module

class synapse.tests.test_common.CommonTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_common_genfile() test_common_guid() test_common_intify() test_common_isguid() test_common_vertup() test_compat_canstor() synapse.tests.test_cortex module class synapse.tests.test_cortex.CoreTestDataModelModuleV0(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() class synapse.tests.test_cortex.CoreTestDataModelModuleV1(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() class synapse.tests.test_cortex.CoreTestModule(core, conf ) Bases: synapse.lib.module.CoreModule static getBaseModels() initCoreModule() class synapse.tests.test_cortex.CortexBaseTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest basic_core_expectations(core, storetype) Run basic tests against a Cortex instance. This should be a minimal test too ensure that a Storage and Cortex combination is working properly. Args: core (s_cores_common.Cortex): Cortex instance to test. storetype (str): String to check the get- StoreType api against. runblob(core) runcore(core) rundarks(core) rundsets(core) runidens(core) runjson(core) runrange(core) runsnaps(core) runstore(core) Run generic storage layer tests on the core.store object

298 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

runtufobydefault(core) test_cortex_lmdb() test_cortex_postgres() test_cortex_ram() test_cortex_sqlite3() class synapse.tests.test_cortex.CortexTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_cortex_addmodel() test_cortex_auth() test_cortex_by_type() test_cortex_bytype() test_cortex_caching() test_cortex_caching_add_tufo() test_cortex_caching_atlimit() test_cortex_caching_del_tufo() test_cortex_caching_del_tufo_prop() test_cortex_caching_disable() test_cortex_caching_new() test_cortex_caching_oneref() test_cortex_caching_set() test_cortex_caching_tags() test_cortex_caching_under_limit() test_cortex_choptag() test_cortex_comp() test_cortex_datamodel_runt_consistency() test_cortex_dict() test_cortex_enforce() test_cortex_fifos() test_cortex_fire_set() test_cortex_formtufobytufo() test_cortex_formtufosbyprops() test_cortex_getbytag() test_cortex_ingest() test_cortex_isnew() test_cortex_lift_by_cidr() test_cortex_local()

4.1. Subpackages 299 Synapse Documentation, Release 0.0.34

test_cortex_minmax() test_cortex_minmax_epoch() test_cortex_modlrevs() test_cortex_modlvers() test_cortex_module() test_cortex_module_datamodel_migration() test_cortex_module_datamodel_migration_persistent() test_cortex_norm_fail() test_cortex_ramhost() test_cortex_reqprops() test_cortex_reqstor() test_cortex_runts() test_cortex_savefd() test_cortex_seed() test_cortex_seq() test_cortex_splice_propdel() test_cortex_splicefd() test_cortex_splicepump() test_cortex_splices() test_cortex_splices_errs() test_cortex_stats() test_cortex_tag_ival() test_cortex_tagform() test_cortex_tags() test_cortex_trigger() test_cortex_tufo_del() test_cortex_tufo_list() test_cortex_tufo_pop() test_cortex_tufo_setprop() test_cortex_tufo_setprops() test_cortex_tufo_tag() test_cortex_universal_props() test_cortex_xact_deadlock() test_pg_encoding() class synapse.tests.test_cortex.FakeType(tlib, name, **info) Bases: synapse.lib.types.IntType

300 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

class synapse.tests.test_cortex.StorageTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_nonexist_ctor() test_storage_confopts() test_storage_handler_defaults() test_storage_handler_misses() test_storage_row_manipulation() test_storage_rowmanipulation() test_storage_xact_spliced() synapse.tests.test_daemon module class synapse.tests.test_daemon.Blah(woot) Bases: object class synapse.tests.test_daemon.DaemonTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_daemon_conf() test_daemon_conf_fork() test_daemon_conf_onfini() test_daemon_ctor_config() test_daemon_ctor_dmonurl() test_daemon_ctor_nonurl() test_daemon_error() test_daemon_on() test_daemon_timeout() class synapse.tests.test_daemon.Woot Bases: object foo(x, y=20) pid() synapse.tests.test_data module class synapse.tests.test_data.DataTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_data_iana_tlds() synapse.tests.test_datamodel module class synapse.tests.test_datamodel.DataModelTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_datamodel_badprop()

4.1. Subpackages 301 Synapse Documentation, Release 0.0.34

test_datamodel_bool() test_datamodel_cast_json() test_datamodel_cortex() test_datamodel_fail_noprop() test_datamodel_fail_notype() test_datamodel_filebase() test_datamodel_filepath() test_datamodel_formbase() test_datamodel_forms() test_datamodel_getPropDef() test_datamodel_getPropInfo() test_datamodel_glob() test_datamodel_hash() test_datamodel_inet() test_datamodel_istufoform() test_datamodel_parsetypes() test_datamodel_reqpropnorm() test_datamodel_subs() test_datamodel_time() test_datamodel_typefns() test_datamodel_types() test_datatype_syn_prop() test_datatype_syn_tag() synapse.tests.test_dyndeps module class synapse.tests.test_dyndeps.DynDepsTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_dyndeps_alias() test_dyndeps_dynloc() test_dyndeps_dynmod() test_dyndeps_dyntask() test_dyndeps_eval() test_dyndeps_expected_aliases() test_dyndeps_meth() test_dyndeps_nosuchdyn() class synapse.tests.test_dyndeps.Foo Bases: object

302 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

bar() synapse.tests.test_dyndeps.woot(x, y=30)

synapse.tests.test_eventbus module

class synapse.tests.test_eventbus.EventBusTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_evenbus_unlink() test_eventbus_basics() test_eventbus_busref() test_eventbus_busref_gen() test_eventbus_consume() test_eventbus_exc() test_eventbus_filt() test_eventbus_finionce() test_eventbus_link() test_eventbus_log() test_eventbus_main_sigint() test_eventbus_main_sigterm() test_eventbus_off() test_eventbus_refcount() test_eventbus_waiter() test_eventbus_waitfini() test_eventbus_withfini() synapse.tests.test_eventbus.block_processing(evt1, evt2) Function to make an eventbus and call main(). Used as a Process target. Args: evt1 (multiprocessing.Event): event to twiddle evt2 (multiprocessing.Event): event to twiddle synapse.tests.test_eventbus.send_sig(pid, sig) Sent a signal to a process. Args: pid (int): Process id to send the signal too. sig (int): Signal to send. Returns: None

synapse.tests.test_gene module

class synapse.tests.test_gene.GeneTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_gene_bool() test_gene_borkcall() test_gene_borkparn()

4.1. Subpackages 303 Synapse Documentation, Release 0.0.34

test_gene_call() test_gene_call_noargs() test_gene_callcall() test_gene_comp() test_gene_empty_value() test_gene_eval() test_gene_hangcall() test_gene_lab() test_gene_landlor() test_gene_newpparn() test_gene_noname() test_gene_notoks() test_gene_null() test_gene_pow() test_gene_prec() test_gene_shift() synapse.tests.test_glob module class synapse.tests.test_glob.GlobTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_glob_call() test_glob_insec() test_glob_persec() test_glob_task() synapse.tests.test_hivemind module synapse.tests.test_lib_atomﬁle module class synapse.tests.test_lib_atomfile.AtomTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_atomfile_base() test_atomfile_get() test_atomfile_pread()

304 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

synapse.tests.test_lib_atomic module

class synapse.tests.test_lib_atomic.AtomTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_atomic_xchg() synapse.tests.test_lib_auth module class synapse.tests.test_lib_auth.AuthTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_auth_rules() test_auth_runas() synapse.tests.test_lib_cache module class synapse.tests.test_lib_cache.CacheTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_cache_clearing() test_cache_defval() test_cache_fini() test_cache_fixed() test_cache_magic() test_cache_match() test_cache_miss() test_cache_set_maxtime() test_cache_timeout() test_cache_tufo() test_cache_tufo_prop() test_keycache_lookup() test_ondem_add() test_ondem_class() test_refdict() test_refdict_with() synapse.tests.test_lib_certdir module class synapse.tests.test_lib_certdir.CertDirTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest getCertDir() test_certdir_ca() test_certdir_host()

4.1. Subpackages 305 Synapse Documentation, Release 0.0.34

test_certdir_hostca() test_certdir_hostcsr() test_certdir_user() test_certdir_userca() synapse.tests.test_lib_cli module class synapse.tests.test_lib_cli.CliTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_cli_cmd() test_cli_cmd_loop_quit() test_cli_cmdret() test_cli_help() test_cli_opts_defval() test_cli_opts_flag() test_cli_opts_glob() test_cli_opts_list() test_cli_quit() class synapse.tests.test_lib_cli.Hehe Bases: object cmd_hehe(cli, line) The hehe cmd. synapse.tests.test_lib_conﬁg module class synapse.tests.test_lib_config.ConfTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_conf_asloc() test_conf_base() test_conf_defval() test_confdef_decorator() test_configable_proxymethod() test_lib_config_req() class synapse.tests.test_lib_config.Foo(opts=None, defs=()) Bases: synapse.lib.config.Config static foodefs()

306 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

synapse.tests.test_lib_datapath module

class synapse.tests.test_lib_datapath.DataPathTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_datapath_iter() test_datapath_parent() test_datapath_quoted() test_datapath_self() test_datapath_valu()

synapse.tests.test_lib_datﬁle module

class synapse.tests.test_lib_datfile.DatFileTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_datfile_basic()

synapse.tests.test_lib_encoding module

class synapse.tests.test_lib_encoding.EncTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_lib_encoding_de() test_lib_encoding_en() synapse.tests.test_lib_ﬁfo module class synapse.tests.test_lib_fifo.FifoTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_fifo_ack_neg1() test_fifo_fifo() test_fifo_flush() test_fifo_nack_past()

synapse.tests.test_lib_ﬁlepath module

class synapse.tests.test_lib_filepath.TestFilePath(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_filepath_glob() test_filepath_regular() test_filepath_tar() test_filepath_tar_open() test_filepath_zip() test_filepath_zip_open()

4.1. Subpackages 307 Synapse Documentation, Release 0.0.34 synapse.tests.test_lib_hashitem module class synapse.tests.test_lib_hashitem.HashItemTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_lib_hashitem() synapse.tests.test_lib_heap module class synapse.tests.test_lib_heap.HeapTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_heap_base() test_heap_readiter() test_heap_resize() test_heap_save() synapse.tests.test_lib_ingest module class synapse.tests.test_lib_ingest.IngTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_ingest_addFormat() test_ingest_basic() test_ingest_basic_bufio() test_ingest_cast() test_ingest_condform() test_ingest_condform_with_missing_var() test_ingest_condtag() test_ingest_cortex_registration() test_ingest_csv() test_ingest_embed_nodes() test_ingest_embed_pernode_tagsprops() test_ingest_embed_props() test_ingest_embed_tags() test_ingest_files() test_ingest_func() test_ingest_gen_guid() test_ingest_iter_object() test_ingest_iter_objectish_array() test_ingest_iterdata() test_ingest_iteriter()

308 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

test_ingest_json() test_ingest_jsoncast() test_ingest_jsonl() test_ingest_lines() test_ingest_pivot() test_ingest_reqprops() test_ingest_savevar() test_ingest_tag_template_whif() test_ingest_taginfo() test_ingest_tagiter() test_ingest_template() test_ingest_varprop() test_ingest_xml() test_ingest_xml_search() test_ingest_xref() synapse.tests.test_lib_interval module class synapse.tests.test_lib_interval.IvalTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_ival_fold() test_ival_overlap() test_ival_parsetime() synapse.tests.test_lib_iq module synapse - test_lib_iq.py.py Created on 10/21/17. Test for synapse.lib.iq classes class synapse.tests.test_lib_iq.IqTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_iq_outp() test_iq_syntest_envars() test_iq_syntest_helpers() test_iq_syntest_logstream() test_iq_syntest_psql() test_iq_testenv()

4.1. Subpackages 309 Synapse Documentation, Release 0.0.34

synapse.tests.test_lib_mixins module

class synapse.tests.test_lib_mixins.Bar Bases: object class synapse.tests.test_lib_mixins.Foo Bases: object class synapse.tests.test_lib_mixins.MixTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_lib_mixins_basics() synapse.tests.test_lib_module module class synapse.tests.test_lib_module.BarMod(core, conf ) Bases: synapse.lib.module.CoreModule class synapse.tests.test_lib_module.CoreModTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_lib_module_modname() test_lib_module_modpath() class synapse.tests.test_lib_module.FooMod(core, conf ) Bases: synapse.lib.module.CoreModule synapse.tests.test_lib_msgpack module class synapse.tests.test_lib_msgpack.MsgPackTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_msgpack_byte() test_msgpack_en() test_msgpack_un() test_msgpack_unpk() synapse.tests.test_lib_openﬁle module class synapse.tests.test_lib_openfile.OpenFileTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_openfile_abs() test_openfile_http() test_openfile_relative() synapse.tests.test_lib_output module class synapse.tests.test_lib_output.TestOutPut(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_output_bytes()

310 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

test_output_str() synapse.tests.test_lib_persist module class synapse.tests.test_lib_persist.PersistTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_persist_dir() test_persist_file() test_persist_offset() test_persist_pump() synapse.tests.test_lib_queue module class synapse.tests.test_lib_queue.QueueTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_queue_base() test_queue_multislice() test_queue_postfini() test_queue_slice() test_queue_timeout() synapse.tests.test_lib_ratelimit module class synapse.tests.test_lib_ratelimit.RateLimitTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_ratelimit_allow() synapse.tests.test_lib_reﬂect module class synapse.tests.test_lib_reflect.Bar Bases: object class synapse.tests.test_lib_reflect.Foo Bases: synapse.eventbus.EventBus class synapse.tests.test_lib_reflect.ReflectTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_reflect_getClsNames() test_reflect_getItemInfo() test_reflect_syn_reflect()

4.1. Subpackages 311 Synapse Documentation, Release 0.0.34

synapse.tests.test_lib_remcycle module

class synapse.tests.test_lib_remcycle.BytsNommer Bases: object noms(*args, **kwargs) class synapse.tests.test_lib_remcycle.FakeIpify Bases: object IPS = [‘1.2.3.4’, ‘8.8.8.8’, ‘10.0.0.1’, ‘127.0.0.1’, ‘192.168.1.1’, ‘255.255.255.255’] random_ip(*args, **kwargs) Get a random IP class synapse.tests.test_lib_remcycle.HypnosTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest, tornado.testing.AsyncTestCase classmethod setUpClass() Spin up the fake ipify server on a random port classmethod tearDownClass() test_hypnos_automatic_ingest() test_hypnos_cache_job() test_hypnos_cache_with_failure() test_hypnos_cache_with_ingest() test_hypnos_callback_ondone() test_hypnos_config_bounds() test_hypnos_config_register_deregister() test_hypnos_content_type_skips() test_hypnos_default_callback() test_hypnos_default_callback_null() test_hypnos_fini() test_hypnos_fini_core() test_hypnos_fire_api_callback() test_hypnos_manual_ingest_via_eventbus() test_hypnos_post_byts() test_hypnos_simple_fail() test_hypnos_with_telepath() class synapse.tests.test_lib_remcycle.NyxTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest setUp() test_hypnos_make_body() test_nyx_make_request() test_nyx_quoted_values() test_nyx_simple_config()

312 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

test_nyx_tornado_http_check() class synapse.tests.test_lib_remcycle.StandaloneTestServer(port=40000) Bases: synapse.eventbus.EventBus Creates a Synapse webapp in its own thread, so there is no conﬂict between the AsyncTestCase ioloop and the ioloop used by the webapp. fireWebApp() onServerFini() synapse.tests.test_lib_remcycle.get_fake_ipify_global_config(port=40000) synapse.tests.test_lib_remcycle.get_fake_ipify_ingest_global_config(port=40000) synapse.tests.test_lib_remcycle.get_vertex_global_config() synapse.tests.test_lib_sched module class synapse.tests.test_lib_sched.SchedTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_sched_base() test_sched_cancel() test_sched_loop() test_sched_persec() synapse.tests.test_lib_scope module class synapse.tests.test_lib_scope.GeneTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_lib_scope() test_lib_scope_enter() test_lib_scope_get_defval() synapse.tests.test_lib_scrape module class synapse.tests.test_lib_scrape.ScrapeTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_scrape_sync() synapse.tests.test_lib_service module class synapse.tests.test_lib_service.SvcTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_service_base() test_service_byname() test_service_dmon_conf() test_service_getNameProxy()

4.1. Subpackages 313 Synapse Documentation, Release 0.0.34

test_service_getTagProxy() test_service_proxy() test_service_proxysugar() class synapse.tests.test_lib_service.Woot Bases: object foo(x, y=10)

synapse.tests.test_lib_session module

class synapse.tests.test_lib_session.SessTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_lib_session_cura() synapse.tests.test_lib_socket module class synapse.tests.test_lib_socket.SocketTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_sock_plex() test_sock_plex_txbuf() test_socket_glob_plex() test_socket_hostaddr() synapse.tests.test_lib_storm module class synapse.tests.test_lib_storm.LimitTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_limit_behavior() test_limit_behavior_negatives() test_limit_default() class synapse.tests.test_lib_storm.StormTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_storm_addnode() test_storm_addtag() test_storm_alltag() test_storm_cmpr_norm() test_storm_delnode() test_storm_delnode_caching() test_storm_delprop() test_storm_deltag() test_storm_edit_end()

314 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

test_storm_editmode() test_storm_filt_regex() test_storm_guid_operator() test_storm_guid_stablegen() test_storm_lift() test_storm_lift_limit() test_storm_lifts_by() test_storm_limit() test_storm_pivot() test_storm_pivot_runt() test_storm_query_log() test_storm_refs() test_storm_setprop() test_storm_show_help() test_storm_tag_fromtag() test_storm_tag_glob() test_storm_tag_ival() test_storm_tag_jointag() test_storm_tag_query() test_storm_tag_totag() test_storm_task() test_storm_task_telepath() test_storm_tree() synapse.tests.test_lib_syntax module class synapse.tests.test_lib_syntax.StormSyntaxTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_storm_syntax_basic() test_storm_syntax_by() test_storm_syntax_comp_opts() test_storm_syntax_edit() test_storm_syntax_gele() test_storm_syntax_lifteq() test_storm_syntax_liftlift() test_storm_syntax_macro_eq() test_storm_syntax_oper_args()

4.1. Subpackages 315 Synapse Documentation, Release 0.0.34

test_storm_syntax_pivot() test_storm_syntax_regex() test_storm_syntax_uppercase_and_underscore() test_storm_syntax_whites()

synapse.tests.test_lib_tags module

class synapse.tests.test_lib_tags.TagTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_aspect_adddel() test_aspect_bytag() test_aspect_iter_down() test_aspect_iter_up() test_tags_subs()

synapse.tests.test_lib_task module

class synapse.tests.test_lib_task.TaskTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_task_base() test_task_call()

synapse.tests.test_lib_thishost module

class synapse.tests.test_lib_thishost.ThisTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_thishost_hostname() test_thishost_platform() test_thishost_ptrsize() synapse.tests.test_lib_threads module class synapse.tests.test_lib_threads.ThreadsTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_threads_cancelable() test_threads_cantwait() test_threads_pool() test_threads_pool_wrap() synapse.tests.test_lib_threads.newtask(func, *args, **kwargs)

316 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

synapse.tests.test_lib_time module

class synapse.tests.test_lib_time.TimeTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_time_parse()

synapse.tests.test_lib_trees module

class synapse.tests.test_lib_trees.TreeTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_lib_tree_interval() synapse.tests.test_lib_trigger module class synapse.tests.test_lib_trigger.TrigTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_trigger_base() synapse.tests.test_lib_tufo module class synapse.tests.test_lib_tufo.TestTufoProps(methodName=’runTest’) Bases: synapse.lib.iq.SynTest setUp() test_default_props() Ensure that when no preﬁx is provided, the properties are taken from the form. test_named_props() Ensure that provided preﬁxes are used. class synapse.tests.test_lib_tufo.TufoCreateTests(methodName=’runTest’) Bases: synapse.lib.iq.SynTest setUp() test_kwargs_tufo_creation() Ensure that tufos’ can be created via **kwargs. test_simple_tufo_creation() Ensure that tufos can be created with explicit arguments. class synapse.tests.test_lib_tufo.TufoEqualityTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest setUp() test_equals() Ensure that tufo equality works where expected. test_not_equals() Ensure that tufo equality works where expected.

4.1. Subpackages 317 Synapse Documentation, Release 0.0.34 synapse.tests.test_lib_types module class synapse.tests.test_lib_types.DataTypesTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_datatype_basics() test_datatype_bool() test_datatype_dup() test_datatype_fqdn() test_datatype_guid() test_datatype_guid_sub() test_datatype_hash_md5() test_datatype_int() test_datatype_int_minmax() test_datatype_int_repr() test_datatype_str() test_datatype_str_enums() test_datatype_str_hex() test_type_bases() test_type_cast() test_type_comp() test_type_getTypeInfo() test_type_gettdef() test_type_issub() test_type_json() test_type_pend() test_type_sepr() test_type_sepr_parse() test_type_sepr_reverse() test_type_str_nullval() test_type_str_strip() test_type_time_timeepoch() test_type_xref() test_types_comp_opt_only() test_types_comp_optfields() test_types_guid() test_types_guid_resolver() test_types_ndef()

318 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

test_types_perm() test_types_propvalu() test_types_storm() test_types_tagtime()

synapse.tests.test_lib_version module

synapse - test_lib_version.py Created on 10/6/17. class synapse.tests.test_lib_version.VersionTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_version_basics() test_version_extract_parts() test_version_fmt() test_version_pack() test_version_parseSemver() test_version_unpack()

synapse.tests.test_lib_webapp module

class synapse.tests.test_lib_webapp.Foo Bases: object addup(x, y=0) bar() horked() exception synapse.tests.test_lib_webapp.Horked Bases: Exception class synapse.tests.test_lib_webapp.WebAppTest(methodName=’runTest’) Bases: tornado.testing.AsyncTestCase, synapse.lib.iq.SynTest test_webapp_body() test_webapp_dmon() test_webapp_publish() test_webapp_ssl() test_webapp_static() synapse.tests.test_link module class synapse.tests.test_link.FooBar Bases: object foo() class synapse.tests.test_link.LinkTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest

4.1. Subpackages 319 Synapse Documentation, Release 0.0.34

test_link_chopurl() test_link_fromurl() test_link_invalid() test_link_local() test_link_pool() test_link_refused() test_link_ssl_auth() test_link_ssl_basic() test_link_ssl_nocheck() test_link_ssl_nofile() synapse.tests.test_link_ssl module class synapse.tests.test_link_ssl.FooBar Bases: object bar() class synapse.tests.test_link_ssl.SslTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_ssl_hugetx()

synapse.tests.test_lookup_iso3166 module

class synapse.tests.test_lookup_iso3166.CountryLookTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_lookup_countries()

synapse.tests.test_lookup_phonenum module

class synapse.tests.test_lookup_phonenum.PhLookTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_lookup_phonenum()

synapse.tests.test_mindmeld module

synapse.tests.test_model_axon module

class synapse.tests.test_model_axon.AxonModelTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_axonblob() test_axonpath()

320 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

synapse.tests.test_model_crypto module

class synapse.tests.test_model_crypto.CryptoModelTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_form_rsakey() test_model_crypto_201708231712() synapse.tests.test_model_dns module class synapse.tests.test_model_dns.DnsModelTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_model_dns_a() test_model_dns_aaaa() test_model_dns_look() test_model_dns_ns() test_model_dns_req() test_model_dns_rev() test_model_dns_rev6() synapse.tests.test_model_ﬁles module class synapse.tests.test_model_files.FileModelTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_filebase() test_filepath() test_model_file_bytes() test_model_file_bytes_axon() test_model_file_seeds() test_model_file_seeds_capitalization() test_model_filepath_complex() test_model_files_imgof() test_model_files_txtref()

synapse.tests.test_model_geopol module

class synapse.tests.test_model_geopol.GeopolModelTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_model_geopol_country() test_model_geopol_iso2() test_model_geopol_iso3() test_model_geopol_isonum()

4.1. Subpackages 321 Synapse Documentation, Release 0.0.34

synapse.tests.test_model_gov_cn module

class synapse.tests.test_model_gov_cn.CnGovTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_models_cngov_mucd() synapse.tests.test_model_gov_us module class synapse.tests.test_model_gov_us.UsGovTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_models_usgov_cage() synapse.tests.test_model_inet module class synapse.tests.test_model_inet.InetModelTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest tes_model_type_inet_tcp4() test_model_fqdn_punycode() test_model_inet_201706121318() test_model_inet_201706201837() test_model_inet_201709181501() test_model_inet_201709271521() test_model_inet_201710111553() test_model_inet_addr() test_model_inet_asnet4() test_model_inet_cast_defang() test_model_inet_chprofile() test_model_inet_cidr4() test_model_inet_email() test_model_inet_fqdn() test_model_inet_fqdn_set_sfx() test_model_inet_fqdn_unicode() test_model_inet_iface() test_model_inet_ipv4() test_model_inet_ipv4_raise() test_model_inet_ipv6() test_model_inet_mac() test_model_inet_passwd() test_model_inet_postref()

322 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

test_model_inet_postref_postmissingprops() test_model_inet_srv4_types() test_model_inet_srv6_types() test_model_inet_url_fields() test_model_inet_urlfile() test_model_inet_urlredir() test_model_inet_web_acct() test_model_inet_web_action() test_model_inet_web_actref() test_model_inet_web_follows() test_model_inet_web_logon() test_model_inet_web_memb() test_model_inet_web_mesg() test_model_inet_web_post() test_model_inet_whois_recns() test_model_inet_whoisemail() test_model_inet_wifi() test_model_type_inet_cidr() test_model_type_inet_email() test_model_type_inet_ipv6() test_model_type_inet_mac() test_model_type_inet_port() test_model_type_inet_udp4() test_model_type_inet_url() test_model_typeinet_ipv4() test_model_whois_contact() synapse.tests.test_model_infotech module class synapse.tests.test_model_infotech.InfoTechTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_model_infotech_av() test_model_infotech_brutecast() test_model_infotech_cve() test_model_infotech_host() test_model_infotech_hostexec() test_model_infotech_hostname()

4.1. Subpackages 323 Synapse Documentation, Release 0.0.34

test_model_infotech_itdev() test_model_infotech_semvertype() test_model_infotech_software() synapse.tests.test_model_infotech.initcomp(*args, **kwargs)

synapse.tests.test_model_language module

class synapse.tests.test_model_language.LangTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_model_language_idiom() test_model_language_trans()

synapse.tests.test_model_material module

class synapse.tests.test_model_material.MatTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_model_mat_spec_item() synapse.tests.test_model_media module class synapse.tests.test_model_media.MediaTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_models_media_news()

synapse.tests.test_model_orgs module

class synapse.tests.test_model_orgs.OrgTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_model_org_has_webacct() test_model_orgs_seed_alias() test_model_orgs_seed_name() synapse.tests.test_model_person module class synapse.tests.test_model_person.PersonTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_model_person() test_model_person_2() test_model_person_contact() test_model_person_guidname() test_model_person_has_alias() test_model_person_has_email()

324 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

test_model_person_has_phone() test_model_person_has_user() test_model_person_has_webacct() test_model_person_name() test_model_person_tokn() synapse.tests.test_model_syn module class synapse.tests.test_model_syn.SynModelTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_model_syn_201709051630() test_model_syn_201709191412() test_model_syn_201710191144() test_model_syn_201711012123() synapse.tests.test_model_telco module class synapse.tests.test_model_telco.TelcoTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_model_telco_cast_loc_us() test_model_telco_imei() test_model_telco_imsi() test_model_telco_phone() test_model_type_phone() synapse.tests.test_model_temporal module class synapse.tests.test_model_temporal.InetModelTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_model_time_from_unix() test_model_time_minmax() synapse.tests.test_reactor module class synapse.tests.test_reactor.ReactorTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_reactor_base()

4.1. Subpackages 325 Synapse Documentation, Release 0.0.34 synapse.tests.test_runtime module class synapse.tests.test_runtime.StormRunTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest getStormCore() prepStormCore(core) runStormBasics(core) runStormStats(core) test_storm_postgres() test_storm_ram() test_storm_sqlite() synapse.tests.test_statemach module class synapse.tests.test_statemach.StateMachTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest getFooMachine(fd) test_statemach_loadnsave() test_statemach_nofd() synapse.tests.test_swarm_runtime module class synapse.tests.test_swarm_runtime.SwarmRunBase(methodName=’runTest’) Bases: synapse.lib.iq.SynTest getSwarmEnv() class synapse.tests.test_swarm_runtime.SwarmRunTest(methodName=’runTest’) Bases: synapse.tests.test_swarm_runtime.SwarmRunBase test_swarm_runtime_and() test_swarm_runtime_by() test_swarm_runtime_clear() test_swarm_runtime_eq() test_swarm_runtime_frob() test_swarm_runtime_gele() test_swarm_runtime_has() test_swarm_runtime_join() test_swarm_runtime_maxtime() test_swarm_runtime_opts() test_swarm_runtime_opts_uniq() test_swarm_runtime_or() test_swarm_runtime_pivot()

326 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

test_swarm_runtime_regex() test_swarm_runtime_saveload()

synapse.tests.test_telepath module

class synapse.tests.test_telepath.Foo Bases: synapse.eventbus.EventBus bar(x, y) baz(x, y) echo(x) localthing(x) speed() class synapse.tests.test_telepath.TelePathTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest getFooEnv(url=’tcp://127.0.0.1:0/foo’) getFooServ() test_telepath_basics() test_telepath_call() test_telepath_callx() test_telepath_chop() test_telepath_clientside() test_telepath_eval() test_telepath_events() test_telepath_fakesync() test_telepath_gzip() test_telepath_nosuchobj() test_telepath_push() test_telepath_reconnect() test_telepath_reminder() test_telepath_reqproxy() test_telepath_reshare() test_telepath_server_badvers() test_telepath_yielder() synapse.tests.test_tools_autodoc module class synapse.tests.test_tools_autodoc.TestAutoDoc(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_tools_autodoc()

4.1. Subpackages 327 Synapse Documentation, Release 0.0.34

test_tools_autodoc_configable() synapse.tests.test_tools_dmon module class synapse.tests.test_tools_dmon.TestArgParser(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_getArgParser_bools() test_getArgParser_logLevel() test_getArgParser_logLevel_exception() class synapse.tests.test_tools_dmon.TestMain(methodName=’runTest’) Bases: synapse.lib.iq.SynTest getTempConfig() test_main_lsboot() test_main_onboot() synapse.tests.test_tools_dumprows_loadrows module synapse - test_tools_dumprows_loadrows.py Created on 7/26/17. Unittests for the dumprows and loadrows tools. class synapse.tests.test_tools_dumprows_loadrows.DumpRowsTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest make_sql_genrows_json(fp) test_blob_dump() test_dump_force() test_dump_largecore() test_simple_compress() test_simple_use() class synapse.tests.test_tools_dumprows_loadrows.LoadRowsTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest make_sql_genrows_json(fp) test_dumprows_load() test_dumprows_load_compressed() test_savefile_load() synapse.tests.test_tools_easycert module class synapse.tests.test_tools_easycert.TestEasyCert(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_easycert_csr() test_easycert_server_sign()

328 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

test_easycert_user_sign() synapse.tests.test_tools_ingest module class synapse.tests.test_tools_ingest.TestIngest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_tools_ingest() synapse.tests.test_tools_json2mpk module class synapse.tests.test_tools_json2mpk.Json2MpkTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_tools_json2mpk() synapse.tests.test_tools_pushﬁle module class synapse.tests.test_tools_pushfile.TestPushFile(methodName=’runTest’) Bases: synapse.lib.iq.SynTest test_tools_pushfile() synapse.tests.test_tools_superhash module class synapse.tests.test_tools_superhash.SuperhashTest(methodName=’runTest’) Bases: synapse.lib.iq.SynTest check_json_blob(obj, fn, edict) setUp() test_bad_file() test_ingest() test_ingest_multiple_files() test_plain()

Module contents

4.1.11 synapse.tools package

Submodules synapse.tools.autodoc module class synapse.tools.autodoc.RstHelp Bases: object addHead(name, lvl=0) addLines(*lines) getRstText()

4.1. Subpackages 329 Synapse Documentation, Release 0.0.34 synapse.tools.autodoc.inspect_mod(mod, cls) Find Config classes in a module which has @confdef decorated functions in them. synapse.tools.autodoc.main(argv, outp=None) synapse.tools.autodoc.reprvalu(valu) synapse.tools.cmdr module synapse.tools.cmdr.main(argv) synapse.tools.dmon module synapse.tools.dmon.getArgParser() synapse.tools.dmon.initboot() Initialize an @reboot config to fire a dmon. synapse.tools.dmon.initconf() Load ( create if needed ) the dmon onboot json config. ( ~/.syn.dmon/onboot.json ) Example: conf = initconf() # ... modify stuff ... saveconf(conf) synapse.tools.dmon.lsboot() synapse.tools.dmon.main(argv, outp=None) synapse.tools.dmon.noboot(path) Remove a config from the list of onboot dmon configs. synapse.tools.dmon.onboot(path) Install a config by confirming a cron @reboot dmon config and addin the given path to the includes for the startup config. Note: The original file path will be read in synapse.tools.dmon.saveconf(conf ) Save the config dict to the onboot json config. synapse.tools.dumprows module synapse - dumprows.py Created on 7/24/17. Dump all of the rows from a cortex storage instance. synapse.tools.dumprows.dump_blobs(outp, fd, store) synapse.tools.dumprows.dump_rows(outp, fd, store, compress=False, genrows_kwargs=None) synapse.tools.dumprows.dump_store(outp, fd, store, compress=False, dump_blobstore=False, genrows_kwargs=None) synapse.tools.dumprows.gen_backup_tufo(options) synapse.tools.dumprows.main(argv, outp=None)

330 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

synapse.tools.dumprows.makeargpaser()

synapse.tools.easycert module

synapse.tools.easycert.main(argv, outp=None)

synapse.tools.ingest module

synapse.tools.ingest.main(argv, outp=None)

synapse.tools.json2mpk module

synapse.tools.json2mpk.getArgParser() synapse.tools.json2mpk.main(argv, outp=None)

synapse.tools.loadrows module

synapse - loadrows.py Created on 7/26/17. Load rows from a saveﬁle (or a dumprows ﬁle) into a Cortex Storage object. synapse.tools.loadrows.main(argv, outp=None) synapse.tools.loadrows.makeargpaser()

synapse.tools.pushﬁle module

synapse.tools.pushfile.getArgParser() synapse.tools.pushfile.main(argv, outp=None)

synapse.tools.superhash module

synapse - superhash.py Created on 4/26/17. Generate the hashsets and guid (superhash) for a file. synapse.tools.superhash.compute_hashes(fp) Compute the superhash information for a file path :param fp: Path to compute :return: Tuple containing the superhash (guid) and other hashes for the file. synapse.tools.superhash.main(argv, outp=None) synapse.tools.superhash.makeargpaser()

4.1. Subpackages 331 Synapse Documentation, Release 0.0.34

Module contents

4.2 Submodules

4.3 synapse.async module class synapse.async.Boss Bases: synapse.eventbus.EventBus A Boss manages asynchonous jobs. ( If given a pool, he will execute them as well ) Additionally, if given a pool, the boss will assume he is responsible for calling fini() on the pool when he is notified of fini() done(jid, ret) err(jid, **excinfo) initJob(jid=None, **info) Initialize and return a new job tufo. Example: jid = jobid() task = (meth,args,kwargs) job = boss.initJob(jid,task=task,timeout=60) Notes: Info Conventions: * task=(meth,args,kwargs) * timeout= # max runtime * ondone= # def ondone(job): job(jid) Return a job tuple by ID. Example: job = boss.job(jid) jobs() Return a list of job tuples. Example: for job in boss.jobs(): dostuff(job) runBossPool(size, maxsize=None) Create and run a thread pool for this Boss() Example: boss.runBossPool(3) setBossPool(pool) Set a thread pool object for the boss. Example: pool = s_threads.Pool(size=3) boss.setBossPool(pool) sync(job, timeout=None) Wait and return the value for the job.

332 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

wait(jid, timeout=None) Wait for a job to complete by id. Example: boss.wait( jid, timeout=10 ) synapse.async.jobDoneMesg(job) Construct a job:done message for the given job. Example: def ondone(job): otherguy.dist( jobDoneMesg(job) ) synapse.async.jobid() synapse.async.jobret(job) Return job result or raise on error. Example: return jobret(job) synapse.async.newtask(meth, *args, **kwargs)

4.4 synapse.axon module class synapse.axon.Axon(axondir, **opts) Bases: synapse.lib.config.Config, synapse.axon.AxonMixin An Axon acts as a binary blob store with hash based indexing/retrieval. alloc(size) Initialize a new blob upload context within this axon. Args: size (int): Size of the blob to allocate space for. Examples: Allocate a blob for a set of bytes and write it too the axon:

iden= axon.alloc(len(byts)) for b in chunks(byts,10240): axon.chunk(iden,b)

Returns: str: Identiﬁer for a given upload. Raises: byiden(iden) Get a axon:blob node by iden (superhash) value. Args: iden (str): Iden to look up. Returns: ((str, dict)): Blob tufo returned by the Axon’s cortex. bytes(htype, hvalu) Yield chunks of bytes for the given hash value. Args: htype (str): Hash type. hvalu (str): Hash value. Examples: Get the bytes for a given guid and do stuff with them:

for byts in axon.bytes('guid', axonblobguid): dostuff(byts)

4.4. synapse.axon module 333 Synapse Documentation, Release 0.0.34

Iteratively write bytes to a ﬁle for a given md5sum:

for byts in axon.bytes('md5', md5sum): fd.write(byts)

Form a contiguous bytes object for a given sha512sum. This is not recommended for large ﬁles.:

byts=b''.join((_byts for _byts in axon.bytes('sha512', sha512sum)))

Notes: This API will raise an exception to the caller if the requested hash is not present in the axon. This is contrasted against the Axon.iterblob() API, which first requires the caller to first obtain an axon:blob tufo in order to start retrieving bytes from the axon. Yields: bytes: A chunk of bytes for a given hash. Raises: NoSuchFile: If the requested hash is not present in the axon. This is raised when the generator is first consumed. chunk(iden, byts) Save a chunk of a blob allocated with alloc(). Args: iden (str): Iden to save bytes too byts (bytes): Bytes to write to the blob. Returns: ((str, dict)): axon:blob node if the upload is complete; otherwise None. Raises: NoSuchIden: If the iden is not in progress. find(htype, hvalu) Returns a list of blob tufos for hashes in the axon. Args: htype (str): Hash type. hvalu (str): Hash value. Examples: Find all blobs for a given md5sum:

blobs= axon.find('md5', md5hash)

Returns: list: List of tufos for a given hash value. fs_create(path, mode) Forms an axon:path node and sets its properties based on a given ﬁle mode. Args: path (str): Path to form. mode (int): Path mode. Examples: Creating a directory:

axon.fs_create('/mydir', 0o775)

Creating a ﬁle

axon.fs_create('/mydir/myfile', 0x81B4)

Returns: None fs_getattr(path) Return the file attributes for a given file path. Args: path (str): Path to look up. Examples: Get the attributes for a given file:

axon.fs_getattr('/foo/bar/baz.faz')

Returns: dict: Attribute dictionary.

334 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

fs_getxattr(path, name) Return a file attribute value for a given file path and attr name. Args: path (str): Path to look up. name (str): Attribute name to retrieve. Examples: Get the size of a file:

axon.fs_getxattr('/foo/bar/baz.faz','st_size')

Returns: Requested attribute value, or None if the attribute does not exist. Raises: NoSuchData if the requested path does not exist. fs_mkdir(path, mode) Creates a new directory at the given path. Args: path (str): Path to create. mode (int): Mode for any created path nodes. Example: axon.fs_mkdir(‘/mydir’, 0o775)

Returns: None Raises: FileExists: If the path already exists.

fs_read(path, size, offset) Reads a ﬁle. Args: path (str): Path to read size (int): Number of bytes to read. offset (int): File offset to retrieve. Examples: Get the bytes of a ﬁle:

byts= axon.fs_read('/dir/file1', 100,0)

Returns: bytes: Bytes read from the path. Raises: NoSuchEntity: If the path does not exist. fs_readdir(path) Reads a directory and gets a list of files in the directory. Args: path (str): Path to get a list of files for. Examples:d Read the files in the root directory:

files= axon.fs_readdir('/')

Returns: list: List of files / folders under the path fs_rename(src, dst) Rename a file: Args: src (str): Full path to the source file. dst (str): Full path to teh destination. Examples: Rename a file:

axon.fs_rename('/myfile','/mycoolerfile')

Returns: None Raises: NoSuchEntity: If the source does not exist. NoSuchDir: If the source or destination parent path does not exist. NotEmpty: If the destination already exists.

4.4. synapse.axon module 335 Synapse Documentation, Release 0.0.34

fs_rmdir(path) Removes a directory Args: path (str): Path to remove. Examples: Remove a directory:

axon.fs_rmdir('/mydir')

Returns: None Raises: NoSuchEntity: If the path does not exist. NotEmpty: If the path is not empty. fs_truncate(path) Truncates a ﬁle by setting its st_size to zero. Args: path (str): Path to truncate. Examples: Truncate a ﬁle:

axon.fs_truncate('/myfile')

Returns: None Raises: NoSuchEntity: If the path does not exist. fs_unlink(path) Unlink (delete) a ﬁle. Args: path (str): Path to unlink. Examples: Delete a ﬁle:

axon.fs_unlink('/myfile')

Returns: None Raises: NoSuchFile: If the path does not exist. fs_utimens(path, times=None) Change ﬁle timestamps (st_atime, st_mtime). Args: path (str): Path to ﬁle to change. times (tuple): Tuple containing two integers - st_atime and st_mtime. Examples: Set the timestamps to epoch 0:

axon.fs_utimens('/myfile',(0,0))

Returns: None getAxonInfo() Return a dictionary of salient info about an axon. has(htype, hvalu) Check if the Axon has a given hash type/valu combination stored in it. Args: htype (str): Hash type. hvalu (str): Hash value. Examples: Check if a sha256 value is present:

if not axon.has('sha256', shaval): stuff()

336 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

Check if a known superhash iden is present:

if axon.has('guid', guidval): stuff()

Returns: ((str, dict)): axon:blob tufo if the axon has the hash or guid. None otherwise. iterblob(blob) Yield bytes blocks from the give blob until complete. Args: blob ((str, dict)): axon:blob tufo to yield bytes from. Examples: Get the bytes from a blob and do stuff with them:

for byts in axon.iterblob(blob): dostuff(byts)

Iteratively write bytes to a ﬁle for a given blob:

fd= file('foo.bin','wb') for byts in axon.iterblob(blob): fd.write(byts)

Form a contiguous bytes object for a given blob. This is not recommended for large ﬁles.:

byts=b''.join((_byts for _byts in axon.iterblob(blob)))

Yields: bytes: A chunk of bytes setAxonInfo(prop, valu) sync(mesg) Consume an axon:sync event (only if we are a clone). wants(htype, hvalu, size) Single round trip call to Axon.has() and possibly Axon.alloc(). Args: htype (str): Hash type. hvalu (str): Hash value. size (int): Number of bytes to allocate. Examples: Check if a sha256 value is present in the Axon, and if not, create the node for a set of bytes:

iden= axon.wants('sha256',valu,size) if iden != None: for byts in chunks(filebytes,onemeg): axon.chunk(iden,byts)

Returns: None if the hvalu is present; otherwise the iden is returned for writing. class synapse.axon.AxonCluster(svcprox) Bases: synapse.axon.AxonMixin Present a singular axon API from an axon cluster. alloc(size, bytag=’class.synapse.axon.Axon’) Allocate a new block within an axon to save size bytes. Returns an iden to use for subsequent calls to axon.chunk() byiden(iden, bytag=’class.synapse.axon.Axon’) Get a axon:blob node by iden (superhash) value. Args: iden (str): Iden to look up.

4.4. synapse.axon module 337 Synapse Documentation, Release 0.0.34

Returns: ((str, dict)): Blob tufo returned by the Axon’s cortex. bytes(htype, hvalu, bytag=’class.synapse.axon.Axon’) chunk(iden, byts) find(htype, hvalu, bytag=’class.synapse.axon.Axon’) Find and return any blobs with the given hash. Example: blobs = axon.ﬁnd(‘sha256’,valu) has(htype, hvalu, bytag=’class.synapse.axon.Axon’) Returns True if any of the axons in the cluster contain the given hash. Example: if not axapi.has(‘sha256’,ﬁlehash): dostuff() iterblob(blob) wants(htype, hvalu, size, bytag=’class.synapse.axon.Axon’) class synapse.axon.AxonHost(datadir, **opts) Bases: synapse.lib.config.Config Manage multiple axons on a given host. add(**opts) Add a new axon to the AxonHost. Args:** opts: kwarg values which supersede the defaults of the AxonHost when making the Axon. Examples: Add another Axon to the host with defaults:

axfo= host.add()

Returns: ((str, dict)): A Axon information tuple containing configuration and link data. info() Return info for attempting to allocate clones and check health. makeAxonOpts() Make a configable dictionary from the current AxonHost options. Returns: dict: Configable dict of valid options which can be passed to a Axon() usage() Return volume usage info. class synapse.axon.AxonMixin Bases: object The parts of the Axon which must be executed locally in proxy cases. ( used as mixin for both Axon and AxonCluster ) eatbytes(byts) Consume a buffer of bytes into the axon as a blob. Example: tufo = axon.eatbytes(byts) eatfd(fd) Consume the contents of a file object into the axon as a blob.

338 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

Example: tufo = axon.eatfd(fd)

4.5 synapse.common module

class synapse.common.NoValu Bases: object synapse.common.addpref(pref, info) Add the given prefix to all elements in the info dict. synapse.common.canstor(s) synapse.common.chunks(item, size) Divide an iterable into chunks. synapse.common.debase64(b) synapse.common.enbase64(b) synapse.common.errinfo(name, mesg) synapse.common.excinfo(e) Populate err,errmsg,errtrace info from exc. synapse.common.firethread(f ) A decorator for making a function fire a thread. synapse.common.gendir(*paths, **opts) synapse.common.genfile(*paths) Create or open ( for read/write ) a file path join. synapse.common.genpath(*paths) synapse.common.gentask(func, *args, **kwargs) synapse.common.getbytes(*paths, **opts) synapse.common.getfile(*paths, **opts) synapse.common.guid(valu=None) Get a 16 byte guid value. By default, this is a random guid value. Args: valu: Object used to construct the guid valu from. This must be able to be msgpack’d. Returns: str: 32 character, lowercase ascii string. synapse.common.intify(x) Ensure ( or coerce ) a value into being an integer or None. Args: x (obj): An object to intify Returns: (int): The int value ( or None ) synapse.common.isguid(text) synapse.common.iterfd(fd, size=10000000) synapse.common.iterzip(*args)

4.5. synapse.common module 339 Synapse Documentation, Release 0.0.34

synapse.common.jsload(*paths) synapse.common.jssave(js, *paths) synapse.common.makedirs(path, mode=511) synapse.common.now() Get the current epoch time in milliseconds. This relies on time.time(), which is system-dependent in terms of resolution. Examples: Get the current time and make a row for a Cortex:

tick= now() row= (someiden,'foo:prop',1, tick) core.addRows([row])

Returns: int: Epoch time in milliseconds. synapse.common.reqStorDict(x) Raises BadStorValu if any value in the dict is not compatible with being stored in a cortex. synapse.common.reqbytes(*paths) synapse.common.reqdir(*paths) synapse.common.reqfile(*paths, **opts) synapse.common.reqlines(*paths, **opts) Open a ﬁle and yield lines of text. Example: for line in reqlines(‘foo.txt’): dostuff(line)

NOTE: This API is used as a performance optimization over the standard fd line iteration mechanism.

synapse.common.reqpath(*paths) synapse.common.reqstor(name, valu) Check to see if a value can be stored in a Cortex. Args: name (str): Property name. valu: Value to check. Returns: The valu is returned if it can be stored in a Cortex. Raises: BadPropValu if the value is not Cortex storable. synapse.common.rowstotufos(rows) Convert rows into tufos. Args: rows (list): List of rows containing (i, p, v, t) tuples. Returns: list: List of tufos. synapse.common.splice(act, **info) synapse.common.synerr(excname, **info) Return a SynErr exception. If the given name is not known, fall back on the base class. synapse.common.to_bytes(valu, size) synapse.common.to_int(byts) synapse.common.tufo(typ, **kwargs)

340 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

synapse.common.verstr(vtup) Convert a version tuple to a string. synapse.common.vertup(vstr) Convert a version string to a tuple. Example: ver = vertup(‘1.3.30’) synapse.common.worker(meth, *args, **kwargs)

4.6 synapse.cortex module

exception synapse.cortex.InvalidParam(name, msg) Bases: Exception synapse.cortex.choptag(tag) Chop a tag into hierarchal levels. synapse.cortex.fromdir(path, conf=None) Initialize a cortex from a directory. Args: path (str): The path to the directory conf (dict): An optional set of config info Returns: (synapse.cores.common.Cortex) synapse.cortex.fromstore(stor, **conf ) Create and return a Cortex for the given Storage layer object. Args: stor (Storage): A synapse.cores.storage.Storage instance. conf (dict): A cortex config dictionary Returns: (synapse.cores.common.Cortex): The Cortex hypergraph instance. synapse.cortex.openlink(link, conf=None, storconf=None) Open a cortex via a link tuple. synapse.cortex.openstore(url, storconf=None, **opts) Opens or creates a Cortex Storage object by URL. This does not attempt to open Storage objects over telepath. Args: url (str): URL which is parsed in order to connect to. storconf (dict): Configable options passed to the storage layer. **opts (dict): Additional options added to the link tufo from the parsed URL. Example: Opening a object and adding a row:

tick= s_common.now() url='sqlite:///./derry.db' store= openstore(url) store.addRows([('1234','syn:test','what', tick)])

Returns: synapse.cores.storage.Storage: A storage object implementing a speciﬁc backend. Raises: NoSuchImpl: If the requested protocol has no storage implementation. synapse.cortex.openurl(url, conf=None, storconf=None, **opts) Construct or reference a cortex by url. This will open a cortex if there is a registered handler for the URL otherwise it will attempt to connect to the URL via Telepath.

4.6. synapse.cortex module 341 Synapse Documentation, Release 0.0.34

If telepath is used, any configable options passed via openurl will not be automatically set. Args: url (str): URL which is parsed in order to connect to. conf (dict): Configable options passed to the Cortex. storconf (dict): Configable options passed to the storage layer. **opts (dict): Additional options added to the link tufo from the parsed URL. Examples: Open up a ram backed cortex:

core= openurl('ram:///')

Open up a remote cortex over telepath:

core= openurl('tcp://1.2.3.4:10000/core)

Notes: The following handlers are registerd by default: • ram:// • sqlite:/// • lmdb:/// • postgres://[[:]@][]/[][/

] For SQL databases, the default table name is “syncortex” Returns: s_cores_common.Cortex: Cortex object or a telepath proxy for a Cortex.

4.7 synapse.daemon module class synapse.daemon.Daemon(pool=None) Bases: synapse.eventbus.EventBus, synapse.daemon.DmonConf links() Return a list of the link tufos the Daemon is listening on. listen(linkurl, **opts) Create and run a link server by url. Example: link = dmon.listen(‘tcp://127.0.0.1:8888‘) Notes: • Returns the parsed link tufo loadDmonConf(conf ) Process Daemon specific dmon config elements. # examples of additional config elements { “share”:( (‘fooname’,{‘optname’:optval}), ... ), “listen”:( ‘tcp://0.0.0.0:8899‘, ...

342 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

), } setMesgFunc(name, func) setUserAuth(auth) share(name, item, fini=False) Share an object via the telepath protocol. Example: dmon.share(‘foo’, Foo()) class synapse.daemon.DmonConf Bases: object A mixin class for configuring a daemon by dict/json. Note: it is assumed that DmonConf is mixed into a class which inherits from EventBus. dmoneval(url) Evaluate ( and cache ) a URL (or local var/ctor name). This allows multiple instances of the same URL to resolve to the same object. killDmonFork(name, perm=False) Kill ( not gracefully ) a daemon fork process by name. loadDmonConf(conf ) { ‘title’:’The Foo Thing’, ‘vars’:{ ‘foo’:10, ‘bar’:[”baz”,”faz”] } ‘forks’:( (‘name’,{ # nested config for forked dmon }, ), ‘includes’:( ‘~/foo/config.json’, ), ‘ctors’:( (‘baz’, ‘ctor://foo.bar.Baz()’), (‘faz’, ‘ctor://foo.bar.Baz()’, {‘config’:’woot’}), (‘mybus’, ‘tcp://host.com/syn.svcbus’), ), ‘services’:( (‘mybus’,( (‘baz’,{‘tags’:(‘foo.bar’,’baz’)}), ), ), ‘addons’:( (‘auth’,’tcp://host.com:8899/synauth‘), (‘logging’,’ctor://mypkg.mymod.MyLogger()’), ), ‘configs’:{ ‘woot’:{ ‘fooopt’:10,

4.7. synapse.daemon module 343 Synapse Documentation, Release 0.0.34

}, }, } loadDmonFile(path) loadDmonJson(text) setDmonConf(prop, valu) class synapse.daemon.OnHelp Bases: object A class used by Daemon to deal with on() handlers and ﬁlters. addOnInst(sock, iden, ﬁlt) delOnInst(sock, iden) dist(mesg) synapse.daemon.checkConfDict(conf ) synapse.daemon.checkConfFile(path) synapse.daemon.forkdmon(conf )

4.8 synapse.datamodel module

An API to assist with the creation and enforcement of cortex data models. class synapse.datamodel.DataModel(load=True) Bases: synapse.lib.types.TypeLib addPropDef(prop, **info) Add a property deﬁnition to the DataModel. Example: model.addPropDef(‘foo:bar’, ptype=’int’, defval=30)

Returns: ((str, dict)): Retuns the prop, property deﬁnition tuple. Raises: DupPropName: If the property name is already present in the data model. BadPropConf: If the propety has an invalid conﬁguration.

addTufoForm(form, **info) Add a tufo form to the data model Example: # must add tufo before adding tufo props model.addTufoForm(‘woot’)

Raises: BadPropName: If the property name is poorly formed.

addTufoProp(form, prop, **info) Add a property to the data model. Example:

344 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

# all foo tufos must have a foo:bar property model.addTufoProp(‘foo’, ‘bar’, ptype=’int’, defval=0) getFormDefs(form) Return a list of (prop,valu) tuples for the default values of a form. getFormReqs(form) Return a list of prop values which are required form a form. Args: form (str): Form to request values for. Returns: list: List of required properties needed for making the given form. getModelDict() Returns a dictionary which represents the data model. getPropDef(prop, glob=True) Return a property definition tufo by property name. Example: pdef = model.getPropDef(‘foo:bar’) getPropFormBase(prop) Return a form,base tuple for the name parts of a given property. Example: Args: prop (str): The fully qualified property name Returns: ((str,str)): The (form,base) name tuple for the prop. getPropInfo(prop, name) A helper function to resolve a prop info from either the property itself or the first type it inherits from which contains the info. Example: ex = modl.getPropInfo(‘inet:dns:a:fqdn’,’ex’) getPropNorm(prop, valu, oldval=None) Return a normalized system mode value for the given property. Args: prop (str): Property to normalize. valu: Input value to normalize. oldval: Optional previous version of the value. Examples: Normalize an IPV4 address:

valu, subs= model.getPropNorm('inet:ipv4','1.2.3.4') # valu = 16909060

Normalize a DNS A record:

valu, subs= model.getPropNorm('inet:dns:a','woot.com/1.2.3.4') # valu = 'woot.com/1.2.3.4' # subs['fqdn'] = 'woot.com' # subs['fqdn:domain'] = 'com' # subs['fqdn:host'] = 'woot' # subs['ipv4'] = 16909060

Notes: If the requested property is not part of the data model, this returns the input valu. If this is not desired behavior, the reqPropNorm() function can be used to throw a NoSuchProp exception. Returns:

4.8. synapse.datamodel module 345 Synapse Documentation, Release 0.0.34

tuple: A tuple of two items. The ﬁrst item is the system normalized valu, as an integer or string. The second item is a dictionary of subproperties for the input. getPropParse(prop, valu) Parse a humon input string into a system mode property value. Example: valu,subs = model.getPropParse(prop, text) getPropRepr(prop, valu) Return the humon readable representation for a property. Example: valu = tufo[1].get(prop) x = model.getPropRepr(prop, valu) getPropType(prop) Return the data model type instance for the given property, or None if the data model doesn’t have an entry for the property. Args: prop (str): Property to get the DataType instance for. Returns: s_types.DataType: A DataType for a given property. getPropTypeName(prop) Retrieve the name of the type for the given property. Args: prop (str): The property Returns: (str): The type name (or None) getPropsByType(name) Return a list of prop def tuples (name,info) for all props of the given type. Example: for prop,info in modl.getPropsByType(‘guid’): dostuff() getSubPropDefs(prop) Return a dict of defvals for props under prop. getSubProps(prop) Return a list of (name,info) prop defs for all sub props. Example: for pdef in model.getSubProps(‘foo:bar’): dostuff(pdef) getTufoForms() Get a list of the currently loaded tufo forms. Returns: list: List of forms. getTypeOfs(name) Return a list of type inheritence (including speciﬁed name). Args: name (str): The name of a type Returns: ([str, ...]): The list of type names it inherits from getUnivProps() Get a list of the universal tufo props. Returns: list: List of universal tufo props isTufoForm(name) Check if a form is a valid form.

346 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

Args: name (str): Form to check Returns: bool: True if the form is a valid form. False otherwise. reqPropNorm(prop, valu, oldval=None) Return a normalized system mode value for the given property. This throws an exception if the property does not exist. Args: prop (str): Property to normalize. valu: Input value to normalize. oldval: Optional previous version of the value. Examples: Normalize an IPV4 address:

valu, subs= model.reqPropNorm('inet:ipv4','1.2.3.4') # valu = 16909060

Normalize a DNS A record:

valu, subs= model.reqPropNorm('inet:dns:a','woot.com/1.2.3.4') # valu = 'woot.com/1.2.3.4' # subs['fqdn'] = 'woot.com' # subs['fqdn:domain'] = 'com' # subs['fqdn:host'] = 'woot' # subs['ipv4'] = 16909060

Notes: This is similar to the getPropNorm() function, however it throws an exception on a missing property instead of returning the valu to the caller. Returns: tuple: A tuple of two items. The ﬁrst item is the system normalized valu, as an integer or string. The second item is a dictionary of subproperties for the input. Raises: NoSuchProp: If the requested property is not part of the data model. reqTufoForm(name) Check if a form is a valid form, raise an exception otherwise. Args: name (str): Form to check Raises: NoSuchForm: If the form does not exist in the datamodel. synapse.datamodel.getTypeNorm(name, valu) Normalize a type speciﬁc value in system mode. synapse.datamodel.getTypeParse(name, text) Parse input text for the given type into it’s system form. synapse.datamodel.getTypeRepr(name, valu) Return the humon readable form of the given type value. synapse.datamodel.parsetypes(*atypes, **kwtypes) Decorator to parse input args from humon to system values. Example: class Woot: @parsetypes(‘int’,’hash:md5’) def getFooBar(self, size, md5): # size will be an int and md5 will be lower dostuff() woot = Woot() # call with user input strings... woot.getFooBar(‘20’,‘0a0a0a0a0B0B0B0B0c0c0c0c0D0D0D0D’)

4.8. synapse.datamodel module 347 Synapse Documentation, Release 0.0.34

synapse.datamodel.rebuildTlib() Rebuild the datamodel’s global TypeLib instance. The datamodel.py module maintains a instance of the TypeLib object. If there are new models dynamically loaded into Synapse, this can be used to rebuild the TypeLib object with those additions. Returns: (None): Returns None.

4.9 synapse.dyndeps module class synapse.dyndeps.CallCapt Bases: object synapse.dyndeps.addDynAlias(name, item) Add an “alias” to the dyndeps resolver system. Example: addDynAlias(‘foobar’,FooBar) # ... subsequently allows ... x = getDynLocal(‘foobar’) synapse.dyndeps.delDynAlias(name) Remove (and return) a dyndeps “alias” previously registered with addDynAlias() Example: delDynAlias(‘foobar’) synapse.dyndeps.getDynLocal(name) Dynamically import a python module and return a local. Example: cls = getDynLocal(‘foopkg.barmod.BlahClass’) blah = cls() synapse.dyndeps.getDynMeth(name) Retrieve and return an unbound method by python path. synapse.dyndeps.getDynMod(name) Dynamically import a python module and return a ref (or None). Example: mod = getDynMod(‘foo.bar’) synapse.dyndeps.runDynEval(text, locs=None) Run a “dyn eval” string returning the result. Example: # dyn imports foo.bar and calls foo.bar.baz(‘woot’, y=30) valu = run- DynEval(“foo.bar.baz(‘woot’,y=30)”) WARNING: duh. this executes arbitrary code. trusted inputs only! synapse.dyndeps.runDynTask(task) Run a dynamic task and return the result. Example: foo = runDynTask( (‘baz.faz.Foo’, (), {} ) )

348 Chapter 4. synapse package Synapse Documentation, Release 0.0.34 synapse.dyndeps.tryDynFunc(name, *args, **kwargs) Dynamically import a module and call a function or raise an exception. synapse.dyndeps.tryDynLocal(name) Dynamically import a module and return a module local or raise an exception. synapse.dyndeps.tryDynMod(name) Dynamically import a python module or exception.

4.10 synapse.eventbus module class synapse.eventbus.BusRef(ctor=None) Bases: synapse.eventbus.EventBus An object for managing multiple EventBus instances. gen(name) Atomically get/gen an EventBus and incref. (requires ctor during BusRef init) Args: name (str): The name/iden of the EventBus instance. get(name) Retrieve an EventBus instance by name. Args: name (str): The name/iden of the EventBus Returns: (EventBus): The EventBus instance (or None) pop(name) Remove and return an EventBus from the BusRef. Args: name (str): The name/iden of the EventBus instance Returns: (EventBus): The named event bus ( or None ) put(name, ebus) Add an EventBus (or sub-class) to the BusRef by name. Args: name (str): The name/iden of the EventBus ebus (EventBus): The EventBus instance Returns: (None) vals() class synapse.eventbus.EventBus Bases: object A synapse EventBus provides an easy way manage callbacks. consume(gtor) Feed the event bus from a generator. Example: bus.consume( getAllEvents() ) dist(mesg) Distribute an existing event tuple. Args: mesg ((str,dict)): An event tuple. Example: ebus.dist( (‘foo’,{‘bar’:’baz’}) )

4.10. synapse.eventbus module 349 Synapse Documentation, Release 0.0.34

distall(events) Distribute multiple events on the event bus. exc(exc, **info) Implements the exception log convention for EventBus. A caller is expected to be within the except frame. Args: exc (Exception): The exception to log Returns: None fini() Fire the ‘fini’ handlers and set self.isfini. Example: d.fini() fire(evtname, **info) Fire the given event name on the EventBus. Returns a list of the return values of each callback. Example: for ret in d.fire(‘woot’,foo=’asdf’): print(‘got: %r’ % (ret,)) incref() Increment the reference count for this event bus. This API may be optionally used to control fini(). link(func) Add a callback function to receive all events. Example: bus1 = EventBus() bus2 = EventBus() bus1.link( bus2.dist ) # all events on bus1 are also propigated on bus2 log(level, mesg, **info) Implements the log event convention for an EventBus. Args: level (int): A python logger level for the event mesg (str): A log message **info: Additional log metadata main() Helper function to block until shutdown ( and handle ctrl-c and SIGTERM). Examples: Run a event bus, wait until main() has returned, then do other stuff:

foo= EventBus() foo.main() dostuff()

Notes: This does fire a ‘ebus:main’ event prior to entering the waitfini() loop. Returns: None off(evnt, func) Remove a previously registered event handler function. Example: bus.off( ‘foo’, onFooFunc ) on(evnt, func, **filts) Add an event bus callback for a specific event with optional filtering.

350 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

Args: evnt (str): An event name func (function): A callback function to receive event tufo **filts: Op- tional positive filter values for the event tuple. Returns: (None) Example: def baz(event): x = event[1].get(‘x’) y = event[1].get(‘y’) return x + y d.on(‘foo’, baz, x=10) # this fire triggers baz... d.fire(‘foo’, x=10, y=20) # this fire does not ( due to filt ) d.fire(‘foo’, x=30, y=20) onfini(func) Register a handler to fire when this EventBus shuts down. unlink(func) Remove a callback function previously added with link() Example: bus.unlink( callback ) waiter(count, *names) Construct and return a new Waiter for events on this bus. Example: # wait up to 3 seconds for 10 foo:bar events... waiter = bus.waiter(10,’foo:bar’) # .. fire thread that will cause foo:bar events events = waiter.wait(timeout=3) if events == None: # handle the timout case... for event in events: # parse the events if you need...

NOTE: use with caution... it’s easy to accidentally construct race conditions with this mechanism ;)

waitfini(timeout=None) Wait for the event bus to ﬁni() Example: bus.waitﬁni(timeout=30) class synapse.eventbus.Waiter(bus, count, *names) Bases: object A helper to wait for a given number of events on an EventBus. fini() wait(timeout=None) Wait for the required number of events and return them or None on timeout. Example: evnts = waiter.wait(timeout=30) if evnts == None: handleTimedOut() return for evnt in evnts: doStuff(evnt)

4.10. synapse.eventbus module 351 Synapse Documentation, Release 0.0.34

4.11 synapse.exc module

exception synapse.exc.AuthDeny(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadAtomFile(*args, **info) Bases: synapse.exc.SynErr Raised when there is a internal issue with an atomfile. exception synapse.exc.BadConfValu(*args, **info) Bases: synapse.exc.SynErr The configuration value provided is not valid. This should contain the config name, valu and mesg. exception synapse.exc.BadCoreName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadCoreStore(*args, **info) Bases: synapse.exc.SynErr The storage layer has encountered an error exception synapse.exc.BadCtorType(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadFifoSeq(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadInfoValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadJson Bases: Exception exception synapse.exc.BadMesgResp Bases: Exception exception synapse.exc.BadMesgVers(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadOperArg(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadPropConf(*args, **info) Bases: synapse.exc.SynErr The configuration for the property is invalid. exception synapse.exc.BadPropName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadPropValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadPySource Bases: Exception exception synapse.exc.BadRevValu(*args, **info) Bases: synapse.exc.SynErr

352 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

exception synapse.exc.BadRuleSyntax(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadRuleValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadStorValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadSyntaxError(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadTypeValu(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.BadUrl Bases: Exception exception synapse.exc.CantDelProp(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.CantSetProp(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.DupFileName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.DupOpt Bases: Exception exception synapse.exc.DupPropName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.DupRole Bases: Exception exception synapse.exc.DupTypeName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.DupUser Bases: Exception exception synapse.exc.FileExists(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.HitCoreLimit(*args, **info) Bases: synapse.exc.SynErr You’ve reached some limit of the storage layer. exception synapse.exc.HitMaxRetry Bases: Exception exception synapse.exc.HitMaxTime Bases: Exception exception synapse.exc.HitStormLimit(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.IsFini Bases: Exception exception synapse.exc.IsRuntProp(*args, **info) Bases: synapse.exc.SynErr

4.11. synapse.exc module 353 Synapse Documentation, Release 0.0.34

exception synapse.exc.JobErr(job) Bases: Exception Used for remote exception propagation. exception synapse.exc.LinkErr(link, mesg=’‘) Bases: Exception retry = False exception synapse.exc.LinkNotAuth(link, mesg=’‘) Bases: synapse.exc.LinkErr exception synapse.exc.LinkRefused(link, mesg=’‘) Bases: synapse.exc.LinkErr retry = True exception synapse.exc.MustBeLocal(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.MustBeProxy(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.MustNotWait Bases: Exception exception synapse.exc.NoAuthUser(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoCurrSess Bases: Exception exception synapse.exc.NoInitCore Bases: Exception exception synapse.exc.NoRevAllow(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuch Bases: Exception exception synapse.exc.NoSuchAct(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchCmpr(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchConf(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchCore(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchCtor(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchData(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchDecoder(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchDir(*args, **info) Bases: synapse.exc.SynErr

354 Chapter 4. synapse package Synapse Documentation, Release 0.0.34 exception synapse.exc.NoSuchDyn(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchEncoder(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchEntity(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchFifo(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchFile(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchForm(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchFunc(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchGetBy(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchIden Bases: Exception exception synapse.exc.NoSuchImpl(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchJob Bases: Exception exception synapse.exc.NoSuchMeth(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchMod(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchName(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchObj(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchOper(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchOpt(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchPath(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchPeer Bases: Exception exception synapse.exc.NoSuchPerm(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchProp(*args, **info) Bases: synapse.exc.SynErr

4.11. synapse.exc module 355 Synapse Documentation, Release 0.0.34 exception synapse.exc.NoSuchProto Bases: Exception exception synapse.exc.NoSuchRole(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchRule(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchSeq(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchSess Bases: Exception exception synapse.exc.NoSuchStat(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchTufo(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchType(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoSuchUser(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NoWritableAxons(*args, **info) Bases: synapse.exc.SynErr There are no writable axons available for the required operation. exception synapse.exc.NotEmpty(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.NotEnoughFree(*args, **info) Bases: synapse.exc.SynErr There is not enough disk space free for the required operation. exception synapse.exc.NotSupported(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.PropNotFound(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.ReqConfOpt(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.SidNotFound Bases: Exception exception synapse.exc.SynErr(*args, **info) Bases: Exception get(name) Return a value from the errinfo dict. Example: try: foothing() except SynErr as e: blah = e.get(‘blah’) items()

356 Chapter 4. synapse package Synapse Documentation, Release 0.0.34 exception synapse.exc.TeleClientSide(*args, **info) Bases: synapse.exc.SynErr exception synapse.exc.WebAppErr(*args, **info) Bases: synapse.exc.SynErr

4.12 synapse.gene module class synapse.gene.CallNode(tokn, kids=()) Bases: synapse.gene.GeneNode class synapse.gene.GeneLab(globs=None) Bases: object clear() getGeneExpr(text) class synapse.gene.GeneNode(tokn, kids=()) Bases: object eval(syms) getNodePrec() class synapse.gene.ListNode(tokn, kids=()) Bases: synapse.gene.GeneNode class synapse.gene.OperNode(tokn, kids=()) Bases: synapse.gene.GeneNode class synapse.gene.UndefinedValue Bases: object class synapse.gene.ValuNode(tokn, kids=()) Bases: synapse.gene.GeneNode class synapse.gene.VarNode(tokn, kids=()) Bases: synapse.gene.GeneNode synapse.gene.asint(f ) synapse.gene.bint(x) Return a boolean integer from a normal integer. ( ie 0 if x == 0, else 1 ) synapse.gene.eval(text, syms=None) Evaluate the given expression (with speciﬁed symbols). Example: eval(‘foo:bar <= baz + 10’, syms=tufo[1]) synapse.gene.exprbase(toks, off=0) Parse a simple base expression. ( ) (...) synapse.gene.expression(toks, off=0) Parse an expression from the token stream. Returns node,off. synapse.gene.exprlist(toks, off=0) Parse an expression list from the token stream. Returns a list of expression nodes and offset.

4.12. synapse.gene module 357 Synapse Documentation, Release 0.0.34 synapse.gene.istok(toks, off, c) Returns True if the tokn at offset is c. synapse.gene.istokoper(toks, off ) Returns True if the tokn at offset is an oper. synapse.gene.logical_and(x, y) synapse.gene.logical_or(x, y) synapse.gene.nexttok(toks, off ) Return the next tokn,off (or None,off) from toks at off. synapse.gene.raisetok(tokn, mesg) synapse.gene.tokenize(text) Produce a token list from the given text string. [ (,), ... ]

4.13 synapse.glob module

4.14 synapse.hivemind module

4.15 synapse.link module

Provides access to the synapse link protocols. synapse.link.addLinkProto(name, ctor) Add a custom LinkRelay by name. Example: class MyRelay(LinkRelay): # ... addLinkProto(‘mine’,MyRelay()) synapse.link.chopLinkUrl(url) Parse a link tuple from a url. Example: link = chopLinkUrl(‘tcp://1.2.3.4:80/‘) Notes: • url parameters become link properties • user:passwd@host syntax is used for authdata synapse.link.getLinkRelay(link) Get a LinkRelay for the given link tufo. Example: relay = getLinkRelay(link) synapse.link.listen(url, **info) Chop a link url and form a listening socket.

358 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

4.16 synapse.mindmeld module

4.17 synapse.reactor module

class synapse.reactor.Reactor Bases: object A class for registration of one-to-one callbacks. ( much like a switch-case in C ) Unlike an EventBus, only one action may be registered for a given mesg type and the function is expected to return a result. rtor = Reactor() def doFooBar(mesg): return 20 + mesg[1].get(‘x’) rtor.act(‘foo:bar’, doFooBar) y = rtor.react( tufo(‘foo:bar’, x=30) ) # y is now 50... act(name, func) Register a handler for an action by name. Example: rtor.act(‘foo:bar’, doFooBar) react(mesg, name=None) Dispatch to the handler and return his response. Example: resp = rtor.react(mesg) Notes: • Handler exceptions will propagate upward

4.18 synapse.statemach module class synapse.statemach.StateMachine(statefd=None, load=True) Bases: object A class which can be used to save/replay API calls to allow saving the “state” of an object as a sequence of calls.

addStateDelta(name, args, kwargs) loadStateFd(fd) synapse.statemach.keepstate(f )

4.19 synapse.telepath module

An RMI framework for synapse. class synapse.telepath.Method(proxy, meth) Bases: object

4.16. synapse.mindmeld module 359 Synapse Documentation, Release 0.0.34

class synapse.telepath.Proxy(relay, plex=None, sock=None) Bases: synapse.eventbus.EventBus The telepath proxy provides “pythonic” access to remote objects. ( you most likely want openurl() or openlink() ) NOTE: all locals in this class must have _tele_ prefixes to prevent accidental deref of something with the same name in code using it under the assumption it’s something else.... call(methname, *args, **kwargs) Call a shared method as a job. Example: job = proxy.call(‘getFooByBar’,bar) # ... do other stuff ... ret = proxy.syncjob(job) callx(name, task, ondone=None) Call a method on a specific shared object as a job. Example: # task is (,,) task = (‘getFooByBar’, (bar,), {} ) job = proxy.callx(‘woot’,task) ret = proxy.syncjob(job) fire(evntname, **info) off(name, func) on(evnt, func, **filts) push(name, item) Push access to an object to the daemon, allowing other clients access. Example: prox = s_telepath.openurl(‘tcp://127.0.0.1/‘) prox.push( ‘bar’, Bar() ) syncjob(job, timeout=None) Wait on a given job and return/raise it’s result. Example: job = proxy.call(‘woot’, 10, bar=20) ret = proxy.syncjob(job) synapse.telepath.clientside(f ) A function decorator which causes the given function to be run on the telepath client side. Args: f: The function being decorated. Notes: A decorated function must use APIs within the function to access object locals, variables, etc. In other words, the method must be able to have self be a Telepath Proxy object. Example: A class with a decorated clientside function:

class Foob: def __init__(self, data): self.locals= data

def getLocals():

360 Chapter 4. synapse package Synapse Documentation, Release 0.0.34

return self.locals

@s_telepath.clientside def foo(self, bar):

# We use an API to get data from self.locals instead # accessing it directly, since that would cause failures. object_locals= self.getLocals()

# Now do stuff on the client side. dostuff(object_locals)

Returns: The decorated function. synapse.telepath.evalurl(url, **opts) Construct either a local object or a telepath proxy. WARNING: this API enables ctor:// proto which uses eval! ( trusted inputs only ) Example: item0 = evalurl(‘tcp://1.2.3.4:90/foo‘) item1 = evalurl(‘ctor://foo.bar.baz(“woot”,y=20)’) synapse.telepath.getClientSides(item) Return a dict of name:path pairs for any clientside functions in item. synapse.telepath.isProxy(item) Check to see if a object is a telepath proxy object or not. Args: item (object): Object to inspect. Returns: bool: True if the object is a telepath object; otherwise False. synapse.telepath.openlink(link) Construct a telepath proxy from a link tufo. Example: foo = openlink(link) foo.bar(20) synapse.telepath.openurl(url, **opts) Construct a telepath proxy from a url. Example: foo = openurl(‘tcp://1.2.3.4:90/foo‘) foo.dostuff(30) # call remote method synapse.telepath.reminder(evnt, **info) May be used on server side to set a telepath reconnect reminder. Args: name (str): The event name to fire on reconnect **info: The event metadata Returns: (bool): True if the telepath caller was notified. NOTE: This requests that the current telepath caller fire the given event back to us in the event of a socket reconnect. synapse.telepath.reqIsProxy(item) Check if the item is a proxy and raise MustBeProxy if not. Args: item (obj): The object to test for being a telepath proxy Returns: (None)

4.19. synapse.telepath module 361 Synapse Documentation, Release 0.0.34

Example: reqIsProxy(foo) # foo is def a proxy here... synapse.telepath.reqNotProxy(item) Check if the item is a proxy and raise MustBeProxy if so. Args: item (obj): The object to test for being a telepath proxy Returns: (None) Example: reqNotProxy(foo) # foo is def not a proxy here... synapse.telepath.teleSynAck(sock, name=None, sid=None)

4.20 Module contents

The synapse distributed key-value hypergraph analysis framework.

362 Chapter 4. synapse package CHAPTER 5

Synapse Conﬁgable Classes

The following objects are Conﬁgable objects. They have settings which may be provided at runtime or during object initialization which may change their behavior.

5.1 synapse.axon.Axon

5.1.1 axon:axonbus

• defval: • doc: Axon servicebus used for making clones of a Axon. • type: str

5.1.2 axon:bytemax

• defval: 1024000000000 • doc: Max size of data this axon is allowed to store. • type: int

5.1.3 axon:clone

• defval: 0 • doc: Flag to indicate the axon is to be a clone axon or not. Not usually directly setby the user. • type: bool

363 Synapse Documentation, Release 0.0.34

5.1.4 axon:clone:iden

• defval: • doc: Iden of the axon that this is a clone of (only applies to clones). Not usually directly set by the user. • type: str

5.1.5 axon:clones

• defval: 2 • doc: Number of clones to make of this axon on the axonbus. • type: int

5.1.6 axon:hostname

• defval: build-6273767-project-71807-vertexprojectsynapse • doc: Hostname associated with an Axon. • type: str

5.1.7 axon:iden

• defval: None • doc: Unique identiﬁer for the axon. Not usually directly set by the user. • type: str

5.1.8 axon:listen

• defval: tcp://0.0.0.0:0/axon • doc: Default listener URL for the axon • type: str

5.1.9 axon:ro

• defval: 0 • doc: Axon Read-only mode. Prevents allocating new space for writing data to the heap ﬁle. • type: bool

5.1.10 axon:syncopts

• defval: {} • doc: kwarg Options used when making a persistent sync directory.

364 Chapter 5. Synapse Conﬁgable Classes Synapse Documentation, Release 0.0.34

5.1.11 axon:tags

• defval: () • doc: Tuple of tag values for the axon over a Axon servicebus.

5.2 synapse.axon.AxonHost

5.2.1 axon:axonbus

• defval: • doc: URL to an axonbus • type: str

5.2.2 axon:bytemax

• defval: 1024000000000 • doc: Max size of each axon created by the host. • type: int

5.2.3 axon:clones

• defval: 2 • doc: The default number of clones for a axon. • type: int

5.2.4 axon:hostname

• defval: build-6273767-project-71807-vertexprojectsynapse • doc: AxonHost hostname • type: str

5.2.5 axon:listen

• defval: tcp://0.0.0.0:0/axon • doc: Default listener URLs for the axons created by this host • type: str

5.2.6 axon:syncopts

• defval: {} • doc: kwarg Options used when making a persistent sync directory for axons.

5.2. synapse.axon.AxonHost 365 Synapse Documentation, Release 0.0.34

5.2.7 axon:tags

• defval: () • doc: Tuple of tag values for the axon to add when sharing over a Axon servicebus.

5.2.8 axonhost:autorun

• defval: 0 • doc: Number of Axons to autostart. • type: int

5.2.9 axonhost:maxsize

• defval: 0 • doc: Max total allocations for Axons created by the host. Only applies if set to a positive integer. • type: int

5.3 synapse.cores.common.Cortex

5.3.1 auth:en

• asloc: _auth_en • defval: 0 • doc: Enable auth/perm enforcement for splicing/storm • type: bool

5.3.2 auth:url

• doc: Optional remote auth cortex (restart required) • type: inet:url

5.3.3 autoadd

• asloc: autoadd • defval: 1 • doc: Automatically add forms for props where type is form • type: bool

366 Chapter 5. Synapse Conﬁgable Classes Synapse Documentation, Release 0.0.34

5.3.4 axon:dirmode

• defval: 509 • doc: Default mode used to make axon:path nodes for directories. • type: int

5.3.5 axon:url

• doc: Allows cortex to be aware of an axon blob store • type: str

5.3.6 cache:maxsize

• asloc: cache_maxsize • defval: 1000 • doc: Enables caching layer in the cortex • type: int

5.3.7 caching

• asloc: caching • defval: 0 • doc: Enables caching layer in the cortex • type: bool

5.3.8 dir

• doc: The cortex metadata directory • type: str

5.3.9 enforce

• asloc: enforce • defval: 1 • doc: Enables data model enforcement • type: bool

5.3.10 ﬁfo:defs

• defval: {} • doc: Conﬁg defaults for core ﬁfos

5.3. synapse.cores.common.Cortex 367 Synapse Documentation, Release 0.0.34

5.3.11 log:level

• asloc: loglevel • defval: 0 • doc: Filters log events to >= level • type: int

5.3.12 log:save

• asloc: logsave • defval: 0 • doc: Enables saving exceptions to the cortex as syn:log nodes • type: bool

5.3.13 modules

• defval: () • doc: An optional list of (pypath,conf) tuples for synapse modules to load

5.3.14 rev:model

• defval: 1 • doc: Set to 0 to disallow model version updates • type: bool

5.3.15 rev:storage

• defval: 1 • doc: Set to 0 to disallow storage version updates • type: bool

5.3.16 storm:limit:lift

• asloc: limlift • defval: None • doc: Global lift limit

368 Chapter 5. Synapse Conﬁgable Classes Synapse Documentation, Release 0.0.34

5.3.17 storm:query:log:en

• asloc: querylog • defval: 0 • doc: words • ptype: bool

5.3.18 storm:query:log:level

• asloc: queryloglevel • defval: 10 • doc: Logging level to ﬁre query log messages at. • ptype: int

5.4 synapse.cores.lmdb.LmdbStorage

5.4.1 rev:storage

• defval: 1 • doc: Set to 0 to disallow storage version updates • type: bool

5.5 synapse.cores.postgres.PsqlStorage

5.5.1 rev:storage

• defval: 1 • doc: Set to 0 to disallow storage version updates • type: bool

5.6 synapse.cores.ram.RamStorage

5.6.1 rev:storage

• defval: 1 • doc: Set to 0 to disallow storage version updates • type: bool

5.4. synapse.cores.lmdb.LmdbStorage 369 Synapse Documentation, Release 0.0.34

5.7 synapse.cores.sqlite.SqliteStorage

5.7.1 rev:storage

• defval: 1 • doc: Set to 0 to disallow storage version updates • type: bool

5.8 synapse.lib.ﬁfo.Fifo

5.8.1 dir

• doc: Path to the FIFO directory • req: 1 • type: str

5.8.2 ﬁle:maxsize

• asloc: maxsize • defval: 1000000000 • doc: Max ﬁfo ﬁle size • type: int

5.8.3 window:ﬁll

• defval: 10000000 • doc: Window ﬁll read size • type: int

5.8.4 window:max

• defval: 2000 • doc: Maximum window size • type: int

5.8.5 window:min

• defval: 1000 • doc: Minimum window size • type: int

370 Chapter 5. Synapse Conﬁgable Classes Synapse Documentation, Release 0.0.34

5.9 synapse.lib.remcycle.Hypnos

5.9.1 web:cache:enable

• defval: False • doc: Enable caching of job results for a period of time, retrievable by jobid. • type: bool

5.9.2 web:cache:timeout

• defval: 300 • doc: Timeout value, in seconds, that the results will persist in the cache. • type: int

5.9.3 web:ingest:max_spool_size

• defval: 2048000 • doc: Maximum spoolﬁle size, in bytes, to use for storing responses associated with APIs that have ingest deﬁnitions. • type: int

5.9.4 web:tornado:max_clients

• defval: 10 • doc: Maximum number of concurrent requests which can be made at one time. • type: int

5.9.5 web:worker:threads:max

• defval: 64 • doc: Maximum number of worker threads to spawn. • type: int

5.9.6 web:worker:threads:min

• defval: 8 • doc: Minimum number of worker threads to spawn. • type: int

5.9. synapse.lib.remcycle.Hypnos 371 Synapse Documentation, Release 0.0.34

5.10 synapse.lib.session.Curator

5.10.1 timeout

• defval: 3600 • doc: Session timeout in seconds • type: int

5.11 synapse.lib.storm.Runtime

5.11.1 storm:limit:lift

• asloc: limlift • defval: None • doc: Global lift limit

5.11.2 storm:query:log:en

• asloc: querylog • defval: 0 • doc: words • ptype: bool

5.11.3 storm:query:log:level

• asloc: queryloglevel • defval: 10 • doc: Logging level to ﬁre query log messages at. • ptype: int

5.12 synapse.swarm.runtime.Runtime

5.12.1 storm:limit:lift

• asloc: limlift • defval: None • doc: Global lift limit

372 Chapter 5. Synapse Conﬁgable Classes Synapse Documentation, Release 0.0.34

5.12.2 storm:query:log:en

• asloc: querylog • defval: 0 • doc: words • ptype: bool

5.12.3 storm:query:log:level

• asloc: queryloglevel • defval: 10 • doc: Logging level to ﬁre query log messages at. • ptype: int

5.12.4 svcbus:deftag

• asloc: deftag • defval: class.synapse.cores.common.Cortex • doc: Default tag for cores • type: syn:tag

5.12.5 svcbus:timeout

• asloc: svctime • doc: SvcBus Telepath Link Tufo • type: int

5.12. synapse.swarm.runtime.Runtime 373 Synapse Documentation, Release 0.0.34

374 Chapter 5. Synapse Conﬁgable Classes CHAPTER 6

Synapse Performance

Measuring the performance of a synapse cortex is a complex undertaking which depends not only on the test conﬁg- urations, but the “shape” of the data being ingested and queried. These tests attempt to take an accurate measure of a set of “bookend” data sets selected speciﬁcally to demonstrate best case and worst case ingest performance.

6.1 A Note on Comparisons

“It doesn’t matter that a 747 can carry more passengers than the Space Shuttle, when the mission is to repair a satellite” -visi When comparing these numbers to benchmarks published by various big data systems such as Hadoop and Elastic Search, it is critical to keep in mind the fundamental difference between a knowledge system like a synapse cortex versus a simple indexer such as Elastic Search. A knowledge system is required to deconflict all new data against what is already known. This means for each new node added to the hypergraph, it must atomically determine if that node already exists so there is only ever one node which represents a particular thing. While many big data systems claim this type of ACID compliant deconfliction is possible, our testing has shown that the claimed performance of these types of systems is drastically reduced when required to carry out atomic check-and-add operations. However, there is also an advantage for a deconflicted knowledge system. When encountering a node which has been previously observed, the system does not create a new node. This has the counterintuitive effect of making a cortex typically become faster as it ingests more data. This performance increase is especially true when ingesting data with many recurrent nodes.

6.2 Test Data Sets

6.2.1 Majestic Million

The Majestic Million is a ranked list of a million FQDNs which is freely available at http://downloads.majestic.com/ majestic_million.csv.

375 Synapse Documentation, Release 0.0.34

This dataset was selected for performance benchmarking due to it being a “worst case” bookend. Within a cortex, inet:fqdn nodes undergo extensive normalization and often the creation of one inet:fqdn node causes the creation of several others. For example, creating inet:fqdn=www.woot.com would subsequently cause the creation of inet:fqdn=woot.com and inet:fqdn=com. Additionally, the FQDNs within the Majestic Million dataset are already deconﬂicted, meaning each FQDN only occurs once within the dataset. This causes every record to deconﬂict and create new nodes.

6.2.2 Non-Deconﬂicted

A synapse cortex is also capable of ingestion and indexing of “instance knowledge” which is not typically deconﬂicted. The primary property for such nodes is typically a system generated GUID and insert times are typically higher. This test is intended to be close to a “best case” scenario where node insertion is not being atomically deconﬂicted and node properties are not subject to extensive normalization rules.

6.3 Cortex Conﬁgurations

Each of the supported storage technologies used by a synapse cortex are tested. Where possible, tests are executed with minimal or no specialized configuration in an attempt to show performance capabilities without the use of exotic configuration. All cortex instances are configured without caching in an attempt to measure the speed of the storage layer implementations rather than the caching subsystem. A production cortex configured with caches is likely to perform queries much faster than these results.

6.3.1 ram-1

The RAM storage backing provides cortex storage and indexing using native python data structures such as dictionaries and lists. This configuration is a highly performant cortex typically used for hypergraph data which can fit in system memory. For these tests, the RAM cortex is initialized with default configuration options.

6.3.2 lmdb-1

The LMDB storage backing provides cortex storage and indexing using the Symas Lightning DB available here: https://symas.com/lightning-memory-mapped-database/ For these tests, the lmdb cortex is initialized with default conﬁguration options.

6.3.3 sqlite-1

The sqlite storage layer provides cortex storage and indexing using the SQLite embedded database library which is built into the python interpreter. For these tests, the sqlite-1 cortex is intitialized with default values.

6.3.4 postgres-1

The Postgres storage layer provides cortex storage and indexing using the Postgresql Database available here: https: //www.postgresql.org/. For these tests, the Postgresql cortex is initialized with default values communicating with a default Postgresql 9.5 database on Ubuntu 16.04 LTS.

376 Chapter 6. Synapse Performance Synapse Documentation, Release 0.0.34

6.3.5 telepath-x3

The Telepath cluster test is designed to measure the scalability of a multi-cortex federation which is operating with the assumption of shard-based division of node creation across several cortex hypergraphs. The primary purpose of the test is to determine the expected overhead of cluster logic and network protocol efﬁciency. The remote cortexes are simple RAM cortexes.

6.4 Test Systems

The current benchmark testing environment is a cluster of 3 hosts with the following hardware: • Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz (8 cores) • 256 GB Memory • 1000 base T network interface ( 1 Gbps ) • 8x 7200 RPM Seagate ST8000NM0055-1RM112 disks

6.5 Results

Each of the test results below shows how the various test configurations perform under the different benchmark tests. In sections below, we discuss results for individual configurations and what that means when selecting how to select a cortex configuration for a given purpose.

6.4. Test Systems 377 Synapse Documentation, Release 0.0.34

benchmark ram-1 lmdb-1 sqlite-1 postgresql-1 telepath-x3 add w/deconf (nodes/sec) 3,347 1,478 385 336 query node (queries/sec ) 21,296 7,610 8681 1,304 add w/o deconf (nodes/sec) 11,460 6,310 911 2,473 32,779

6.5.1 ram-1

As expected, the ram-1 cortex has the advantage of speed. In instances where it is possible to dedicate enough memory to a Cortex, this configuration can be deployed to achieve maximum deconfliction performance as well as being the fastest to query. However, for large data sets this definitely puts a limit on use of 32 bit platforms and lower memory server configurations. Another potential limitation in the use of a ram cortex is startup time. In the case of a ram cortex which has enabled persistence, there may be a very long startup time during a reboot due to the cortex needing to playback change events on disk to reach the current state. Further testing will profile how much ram is expected for use by a cortex with a given population of nodes.

6.5.2 lmdb-1

The lmdb-1 configuration strikes a powerful balance between performance and persistence. It is highly likely that this configuration will become the recommended default configuration for production deployments.

6.5.3 sqlite-1

The sqlite-1 conﬁguration demonstrates the powerful query speed of the underlying sqlite db backend. However, transactional limits and insert speeds are quite a bit slower. This makes an sqlite Cortex most ideal for use with large datasets which grow a bit more slowly but are expected to be queried very quickly.

6.5.4 postgres-1

The postgres-1 configuration is the only test configuration which uses a storage mechanism which resides outside the python process running the Cortex. This requires the use of Inter Process Communication (IPC) using local pipes, and forces many context switches in comparison with the other configurations. However, future testing of parallel query execution is expected to demonstrate postgres as a highly capable parallel query platform, potentially making it more attractive for use with a larger number of querying clients.

6.5.5 telepath x3

Current results show highly efficient scale gains when using multiple cortexes in a federated configuration. However, the current testing environment involves the use of only 3 systems. Future scale testing using additional hardware will be a better estimate of performance in a truly production scale cluster. That being said, current results are promising. • scale efficiency: 2.8 / 3.0 Future testing of parallel query execution is expected to achieve even higher levels of efficiency.

378 Chapter 6. Synapse Performance Synapse Documentation, Release 0.0.34

6.6 Additional Tests

Over the course of subsequent releases, a table will be added here showing the performance of releases over time using line graphs showing the various test results over time. Additionally, tests which measure parallel query performance will be implemented to demonstrate storage backings which operate well or poorly under highly parallel query load.

6.6. Additional Tests 379 Synapse Documentation, Release 0.0.34

380 Chapter 6. Synapse Performance CHAPTER 7

Contributing to Synapse

• Project Style Guide. • Docs Generation. • Contribution Process. • Running Postgres Tests.

7.1 Project Style Guide

The following items should be considered when contributing to Synapse: • The project is not currently strictly PEP8 compliant. Compliant sections include the following: – Whitespace in Expressions and Statements. – Programming Recommendations regarding singleton comparison (use ‘is’ instead of equality operators). • Please keep line lengths under 120 characters. • Use single quotes for string constants (including docstrings) unless double quotes are required.

# Do this foo='1234' # NOT this foo="1234"

• Use a single line break between top level functions and class deﬁnitions, and class methods. This helps conserve vertical space. – Do this

import foo import duck

def bar():

381 Synapse Documentation, Release 0.0.34

return True

def baz(): return False

class Obj(object):

def __init__(self, a): self.a=a

def gimmeA(self): return self.a

* NOT this import foo import duck

def bar(): return True

def baz(): return False

class Obj(object):

def __init__(self, a): self.a=a

def gimmeA(self): return self.a

• Use Google style Python docstrings. This format is very readable and will allow type hinting for IDE users. See the following notes below about our slight twist on this convention. – Use ‘” quotes instead of “”” for starting/stoping doc strings. – Google Style typically has the summary line after the opening ‘” marker. Place this summary value on the new line following the opening ‘” marker. – More information about Google Style docstrings (and examples) can be found here. – We use Napoleon for parsing these doc strings. More info here. – Synapse as a project is not written using the Napoleon format currently but all new modules should audhere to that format. – Synapse acceptable example:

def fooTheBar(param1, param2, **kwargs): ''' Summary line goes first.

Longer description lives here. It can be a bunch of stuff across multiple blocks if neccesary.

382 Chapter 7. Contributing to Synapse Synapse Documentation, Release 0.0.34

Example: Examples should be given using either the ``Example`` section. Sections support any reStructuredText formatting, including literal blocks::

woah = fooTheBar('a', 'b', duck='quacker')

Section breaks are created by resuming unindented text. Section breaks are also implicitly created anytime a new section starts.

`PEP 484`_ type annotations are supported. If attribute, parameter, and return types are annotated according to `PEP 484`_, they do not need to

˓→be included in the docstring:

Args: param1 (int): The first parameter. param2 (str): The second parameter.

Keyword Arguments: duck (str): Optional keyword args which come in via **kwargs call ˓→conventions, which modify function behavior, should be documented

˓→under the Keyword Args section.

Returns: bool: The return value. True for success, False otherwise.

The ``Returns`` section supports any reStructuredText

˓→formatting, including literal blocks::

{ 'param1': param1, 'param2': param2 }

Raises: AttributeError: The ``Raises`` section is a list of all exceptions that are relevant to the interface. ValueError: If `param2` is equal to `param1`.

.. _PEP 484: https://www.python.org/dev/peps/pep-0484/

''' # Do stuff the with args...

• Imports should first be sorted in order of shortest to longest import, then by alphabetical order (when lengths match). Imports should be ordered starting from the Python standard library first, then any third party packages, then any Synapse specific imports. The following example shows the recommended styling for imports:

# Stdlib import logging import collections # Third Party Code

7.1. Project Style Guide 383 Synapse Documentation, Release 0.0.34

import barlib.duck as b_duck import foolib.thing as f_thing # Synapse Code import synapse.common as s_common import synapse.compat as s_compat import synapse.cortex as s_cortex import synapse.lib.config as s_config

• Previously we used * imports in the Synapse codebase (especially around synapse.exc and synapse.common). If common functions or exceptions are needed, import synapse.common as noted above, and both the common functions and the entirety of synapse.exc exceptions will be available. This provides a consistent manner for referencing common functions and Synapse speciﬁc exception classes. New code should generally not use * imports. Here is an example:

# Do this import synapse.common as s_common tick= s_common.now() if tick< 1000000000: raise s_common.HitMaxTime(mesg='We have gone too far!')

# NOT this from synapse.common import * tick= now() if tick< 1000000000: raise HitMaxTime(mesg='We have gone too far!')

• Function names should follow the mixedCase format for anything which is exposed as a externally facing API on a object or module.

# Do this fooTheBar() # NOT this foo_the_bar()

• Private methods should be marked as such with a proceeding underscore.

# Do this _internalThing() # NOT this privateInternalThingDontUseMe()

– The corralary to this is that any function which is not private may be called arbitrarily at any time, so avoid public API functions which are tightly bound to instance state. For example, if a processing routine is broken into smaller subroutines for readability or testability, these routines are likely private and should not be exposed to outside callers. • Function calls with mandatory arguments should be called with positional arguments. Do not use keyword arguments unless neccesary.

def foo(a, b, duck=None): print(a, b, duck)

# Do this foo('a','b', duck='quacker') # Not this foo(a='a', b='b', duck='quacker')

• Avoid the use of @property decorators. They do not reliably work over the telepath RMI.

384 Chapter 7. Contributing to Synapse Synapse Documentation, Release 0.0.34

• Logging should be setup on a per-module basis, with loggers created using calls to logging.getLogger(__name__). This allows for module level control of loggers as neccesary. – Logger calls should use logging string interpolation, instead of using % or .format() methods. See Python Logging module docs for reference. – Example:

# Get the module level logger logger= logging.getLogger(__name__) # Do this - it only forms the final string if the message is # actually going to be logged logger.info('I am a message from %s about %s','bob','a duck') # NOT this - it performs the string format() call regardless of # whether or not the message is going to be logged. logger.info('I am a message from {} about {}'.format('bob','a duck'))

• It may be neccesary from time to time to include non-ASCII characters. Use UTF8 formatting for such source ﬁles and use the following encoding declaration at the top of the source ﬁle.

#-*- coding: utf-8 -*-

• Convenience methods are availible for unit tests, primarily through the SynTest class. This is a subclass of unittest.TestCase and provides many short aliases for the assert* functions that TestCase provides. – Ensure you are closing resources which may be open with test cases. Many Synapse objects may be used as content managers which make this easy for test authors. • Avoid the use of the built-in re module. Instead use the third-party regex module. regex is preferred due to known bugs with unicode in the re module. Additionally, regex does provide some performance beneﬁts over re, especially when using pre-compiled regular expression statements. • Whenever possible, regular expressions should be pre-compiled. String matches/comparisons should be performed against the pre-compiled regex instance.

# Do this fqdnre= regex.compile(r'^[\w._-]+$', regex.U)

def checkValue(valu): if not fqdnre.match(valu): self._raiseBadValu(valu)

# NOT this def checkValue(valu): if not regex.match(r'^[\w._-]+$', valu, regex.U) self._raiseBadValu(valu)

• Return values should be preferred over raising exceptions. Functions/methods that return a value should return None (or a default value) in the case of an error. The logic behind this is that it is much easier, cleaner, faster to check a return value than to handle an exception. Raising exceptions is reserved for “exceptional circumstances” and should NEVER be used for normal program ﬂow.

# Do this def getWidgetById(self, wid): widget_hash= self._index.get(wid) if widget_hash is None: return None

7.1. Project Style Guide 385 Synapse Documentation, Release 0.0.34

widget= self._widgets.get(widget_hash) return widget

# NOT this def getWidgetById(self, wid): widget_hash= self._index.get(wid) if widget_hash is None: raise NotFoundError

widget= self._widgets.get(widget_hash) if widget is None: raise NotFoundError

return widget

Contributions to Synapse which do not follow the project style guidelines may not be accepted.

7.2 Docs Generation

API documentation is automatically generated from docstrings, and additional docs may also be added to Synapse as well for more detailed discussions of Syanpse subsystems. This is currently done via readthedocs. In order to do local doc generation you can do the following steps: 1. Install the following packages (preferably in a virtualenv):

pip install sphinx==1.5.3 Pygments==2.2.0 setuptools==28.8.0 docutils==0.13.1

˓→mkdocs==0.15.0 mock==1.0.1 pillow==2.6.1 git+https://github.com/rtfd/

˓→[email protected]#egg=readthedocs-sphinx-ext sphinx_rtd_

˓→theme commonmark==0.5.4 recommonmark==0.4.0

2. Build the docs using sphinx. A makeﬁle is provided which makes this painless.

# Go to your synapse repo cd synapse # Go to the docs folder cd docs # Use the make command to build the HTML docs make html

3. Now you can open the HTML docs for browsing them.

xdg-open _build/html/index.html

If you need to write explicit docs for Synapse, they should be added to the repository at docs/synapse and a reference added to the docs in docs/index.rst in order for the documentation

7.3 Contribution Process

The Vertex Project welcomes contributions to the Synapse Hypergraph framework in order to continue its growth! In order to contribute to the project, do the following: 1. Fork the Synapse repository from the Vertex Project. Make a new branch in git with a descriptive name for your change. For example:

386 Chapter 7. Contributing to Synapse Synapse Documentation, Release 0.0.34

git checkout-b foohuman_new_widget

2. Make your changes. Changes should include the following information: • Clear documentation for new features or changed behavior • Unit tests for new features or changed behaviors • If possible, unit tests should also show minimal use examples of new features. 3. Ensure that both your tests and existing Synapse tests succesfully run. You can do that manually via the python unittest module, or you can set up Travis CI to run tests for your fork. The following examples show manual test runs:

python-m unittest discover-v python-m unittest synapse.tests.your_test_file-v

If test coverage is desired, you can use the provided testrunner.sh shell script to run a test. This script will generate HTML coverage reports and attempt to open those reports using xdg-open. This requires the pytest, pytest-cov, pytest-xdist packages to be installed.

./scripts/testrunner.sh ./scripts/testrunner.sh synapse/tests/your_test_file.py ./scripts/testrunner.sh synapse/tests/your_test_file.py::YourTestClass ./scripts/testrunner.sh synapse/tests/your_test_file.py::YourTestClass::test_

˓→function

4. Rebase your feature branch on top of the latest master branch of the Vertex Project Synapse repository. This may require you to add the Vertex Project repository to your git remotes. The following example of rebasing can be followed:

# Add the Vertex project repository as a remote named "upstream". git remote add upstream https://github.com/vertexproject/synapse.git # Grab data from the upstream repository git fetch--all # Change to your local git master branch git checkout master # Merge changes from upstream/master to your local master git merge upstream/master # Move back to your feature branch git checkout foohuman_new_feature # Rebase your feature branch ontop of master. # This may require resolving merge conflicts. git rebase master # Push your branch up to to your fork - this may require a --force # flag if you had previously pushed the branch prior to the rebase. git push

5. Ensure your tests still pass with the rebased feature branch. 6. Create the Pull Request in Github, from your fork’s feature branch to the master branch of the Vertex Project Synapse repository. Include a description and a reference to any open issues related to the PR.

7.4 Running Postgres Tests

Changes which involve modifying Postgres storage layer may require additional testing during local development. It is easy to use a dockerized version of Postgres in order to do this. The following instructions show how to start and stop

7.4. Running Postgres Tests 387 Synapse Documentation, Release 0.0.34 a docker-compose ﬁle provided in Synapse in order to run these tests, as well as conﬁguring shell aliases to start/stop the container. This does require docker and docker-compose to be installed and working. 1. Start the container via docker-compose from the root checkout of the Synapse repository. This will daemonize the docker container so its runs in the background.

docker-compose-f synapse/tests/docker-compose.yml up-d--force-recreate

2. Conﬁgure test environment variable so the unittest runner knows where to connect too for getting a PSQL Cortex connection from.

export SYN_TEST_PG_DB=postgres:hehe@localhost:5432/syn_test

3. Run a PSQL Cortex unit test to make sure it works properly.

synapse$ python -m unittest synapse.tests.test_cortex.CortexBaseTest.test_cortex_

˓→postgres -v test_cortex_postgres (synapse.tests.test_cortex.CortexBaseTest) ... ok

------Ran 1 test in 12.006s

4. Tear down the PSQL container when done with it.

docker-compose-f synapse/tests/docker-compose.yml down

5. These commands can be set as bash aliases to easily run them. These can be added to a ~/.bash_aliases file or other shell configuration file as appropriate.

alias start_syn_psql='docker-compose -f synapse/tests/docker-compose.yml up -d --

˓→force-recreate && export SYN_TEST_PG_DB=postgres:hehe@localhost:5432/syn_test' alias stop_syn_psql='docker-compose -f synapse/tests/docker-compose.yml down'

388 Chapter 7. Contributing to Synapse CHAPTER 8

Synapse Release Process

This doc details the release process we use for Synapse.

8.1 Github Milestone Management

The current milestone and the next milestone should be created in github. For example, if the current release is v0.0.20, we should have a v0.0.21 and v0.0.22 milestones created. When PRs are created or issues are addressed (via PR), they should be added to the milestone. This allows us to easily pull stories and PRs for release note generation.

8.2 Release Notes

Release notes should be compiled from the issues and PRs assigned to the milestone being released. These can all be obtained via a issue search in github. For example, if we’re releasing v0.0.20, we can pull all the stories via the following query in github: milestone:v0.0.20

Release notes should break things out by the following categories: 1. New Features in Synapse 2. Enhancements to existing features 3. Bugﬁxes 4. Major documentation updates Short text form is ﬁne for describing these. These notes will be posted up on github on the releases page for consumption.

389 Synapse Documentation, Release 0.0.34

8.2.1 Markdown Template

The following can be used as a markdown template for Github release notes:

# Synapse Release Notes

## New Features - item1 - item2

## Enhancements - item1 - item2

## Bugs - item1 - item2

## Documentation - item1 - item2

8.3 Cutting the Release

This includes three parts: 1. Tagging the release and pushing to github. 2. Publishing the release on pypi. 3. Publishing new docker images on dockerhub. Version tagging in Synapse is managed by bumpversion. This handles updating the .py ﬁles containing the version number in them, as well as creating git tags and commit messages. There should not be a need to manually edit version numbers or do git commits. bumpversion is a python application, and can be installed via pip: python-m pip install bumpversion

Bumpversion is designed for projects which do semantic versioning. Since synapse is not yet in that state, we’ll be using bumpversion to do patch releases. This can be done via the following (assuming the vertexproject/synapse remote is called ‘upstream’):

# Ensure we're on master with the latest version git checkout master&& git fetch--all&& git merge upstream/master # Do a dry-run to ensure that we're updating things properly bumpversion--dry-run--verbose patch # Bump the patch version bumpversion--verbose patch # Push the new commit and tag up to github git push upstream git push upstream--tags

Next, go to github at https://github.com/vertexproject/synapse/tags and edit the release notes for the tag that was pushed up. Add the release notes compiled from the previous step. Coming soon (this may be automated very soon)

390 Chapter 8. Synapse Release Process Synapse Documentation, Release 0.0.34

Coming soon

8.3. Cutting the Release 391 Synapse Documentation, Release 0.0.34

392 Chapter 8. Synapse Release Process CHAPTER 9

Indices and tables

• genindex • modindex • search

393 Synapse Documentation, Release 0.0.34

394 Chapter 9. Indices and tables Python Module Index

s synapse.lib.encoding, 237 synapse, 362 synapse.lib.fifo, 238 synapse.async, 332 synapse.lib.filepath, 238 synapse.axon, 333 synapse.lib.hashitem, 241 synapse.cmds, 198 synapse.lib.hashset, 241 synapse.cmds.common, 197 synapse.lib.heap, 241 synapse.cmds.cortex, 197 synapse.lib.ingest, 242 synapse.common, 339 synapse.lib.interval, 243 synapse.cores, 224 synapse.lib.iq, 244 synapse.cores.common, 198 synapse.lib.mixins, 247 synapse.cores.lmdb, 213 synapse.lib.module, 247 synapse.cores.postgres, 214 synapse.lib.modules, 249 synapse.cores.ram, 214 synapse.lib.msgpack, 250 synapse.cores.sqlite, 215 synapse.lib.openfile, 251 synapse.cores.storage, 216 synapse.lib.output, 251 synapse.cores.xact, 223 synapse.lib.persist, 252 synapse.cortex, 341 synapse.lib.platforms, 225 synapse.daemon, 342 synapse.lib.platforms.common, 224 synapse.data, 224 synapse.lib.platforms.darwin, 224 synapse.datamodel, 344 synapse.lib.platforms.freebsd, 224 synapse.docker, 224 synapse.lib.platforms.linux, 225 synapse.dyndeps, 348 synapse.lib.platforms.windows, 225 synapse.eventbus, 349 synapse.lib.queue, 253 synapse.exc, 352 synapse.lib.ratelimit, 254 synapse.gene, 357 synapse.lib.reflect, 254 synapse.glob, 358 synapse.lib.remcycle, 255 synapse.hivemind, 358 synapse.lib.sched, 263 synapse.lib, 286 synapse.lib.scope, 264 synapse.lib.atomfile, 225 synapse.lib.scrape, 265 synapse.lib.atomic, 226 synapse.lib.service, 265 synapse.lib.auth, 226 synapse.lib.session, 268 synapse.lib.cache, 227 synapse.lib.socket, 268 synapse.lib.certdir, 230 synapse.lib.storm, 270 synapse.lib.cli, 231 synapse.lib.syntax, 273 synapse.lib.cmdr, 233 synapse.lib.tags, 274 synapse.lib.config, 233 synapse.lib.task, 275 synapse.lib.dark, 236 synapse.lib.thishost, 276 synapse.lib.datapath, 237 synapse.lib.thisplat, 276 synapse.lib.datfile, 237 synapse.lib.threads, 276 synapse.lib.time, 278

395 Synapse Documentation, Release 0.0.34

synapse.lib.trees, 278 synapse.tests.test_async, 296 synapse.lib.trigger, 278 synapse.tests.test_axon, 296 synapse.lib.tufo, 279 synapse.tests.test_cmds_cortex, 297 synapse.lib.types, 280 synapse.tests.test_common, 298 synapse.lib.urlhelp, 284 synapse.tests.test_cortex, 298 synapse.lib.version, 284 synapse.tests.test_daemon, 301 synapse.lib.webapp, 285 synapse.tests.test_data, 301 synapse.link, 358 synapse.tests.test_datamodel, 301 synapse.links, 287 synapse.tests.test_dyndeps, 302 synapse.links.common, 286 synapse.tests.test_eventbus, 303 synapse.links.local, 286 synapse.tests.test_gene, 303 synapse.links.ssh, 287 synapse.tests.test_glob, 304 synapse.links.ssl, 287 synapse.tests.test_hivemind, 304 synapse.links.tcp, 287 synapse.tests.test_lib_atomfile, 304 synapse.lookup, 288 synapse.tests.test_lib_atomic, 305 synapse.lookup.iana, 287 synapse.tests.test_lib_auth, 305 synapse.lookup.iso3166, 287 synapse.tests.test_lib_cache, 305 synapse.lookup.phonenum, 288 synapse.tests.test_lib_certdir, 305 synapse.mindmeld, 359 synapse.tests.test_lib_cli, 306 synapse.models, 295 synapse.tests.test_lib_config, 306 synapse.models.axon, 289 synapse.tests.test_lib_datapath, 307 synapse.models.biology, 289 synapse.tests.test_lib_datfile, 307 synapse.models.chemistry, 289 synapse.tests.test_lib_encoding, 307 synapse.models.crypto, 289 synapse.tests.test_lib_fifo, 307 synapse.models.dns, 289 synapse.tests.test_lib_filepath, 307 synapse.models.files, 289 synapse.tests.test_lib_hashitem, 308 synapse.models.finance, 290 synapse.tests.test_lib_heap, 308 synapse.models.geopol, 290 synapse.tests.test_lib_ingest, 308 synapse.models.geospace, 290 synapse.tests.test_lib_interval, 309 synapse.models.gov, 289 synapse.tests.test_lib_iq, 309 synapse.models.gov.cn, 288 synapse.tests.test_lib_mixins, 310 synapse.models.gov.intl, 288 synapse.tests.test_lib_module, 310 synapse.models.gov.us, 288 synapse.tests.test_lib_msgpack, 310 synapse.models.inet, 290 synapse.tests.test_lib_openfile, 310 synapse.models.infotech, 292 synapse.tests.test_lib_output, 310 synapse.models.language, 293 synapse.tests.test_lib_persist, 311 synapse.models.material, 293 synapse.tests.test_lib_queue, 311 synapse.models.media, 293 synapse.tests.test_lib_ratelimit, 311 synapse.models.mime, 293 synapse.tests.test_lib_reflect, 311 synapse.models.money, 293 synapse.tests.test_lib_remcycle, 312 synapse.models.orgs, 293 synapse.tests.test_lib_sched, 313 synapse.models.person, 294 synapse.tests.test_lib_scope, 313 synapse.models.science, 294 synapse.tests.test_lib_scrape, 313 synapse.models.syn, 294 synapse.tests.test_lib_service, 313 synapse.models.telco, 294 synapse.tests.test_lib_session, 314 synapse.models.temporal, 295 synapse.tests.test_lib_socket, 314 synapse.reactor, 359 synapse.tests.test_lib_storm, 314 synapse.statemach, 359 synapse.tests.test_lib_syntax, 315 synapse.swarm, 295 synapse.tests.test_lib_tags, 316 synapse.swarm.runtime, 295 synapse.tests.test_lib_task, 316 synapse.telepath, 359 synapse.tests.test_lib_thishost, 316 synapse.tests, 329 synapse.tests.test_lib_threads, 316 synapse.tests.common, 295 synapse.tests.test_lib_time, 317 synapse.tests.nopmod, 296 synapse.tests.test_lib_trees, 317

396 Python Module Index Synapse Documentation, Release 0.0.34 synapse.tests.test_lib_trigger, 317 synapse.tests.test_lib_tufo, 317 synapse.tests.test_lib_types, 318 synapse.tests.test_lib_version, 319 synapse.tests.test_lib_webapp, 319 synapse.tests.test_link, 319 synapse.tests.test_link_ssl, 320 synapse.tests.test_lookup_iso3166, 320 synapse.tests.test_lookup_phonenum, 320 synapse.tests.test_mindmeld, 320 synapse.tests.test_model_axon, 320 synapse.tests.test_model_crypto, 321 synapse.tests.test_model_dns, 321 synapse.tests.test_model_files, 321 synapse.tests.test_model_geopol, 321 synapse.tests.test_model_gov_cn, 322 synapse.tests.test_model_gov_us, 322 synapse.tests.test_model_inet, 322 synapse.tests.test_model_infotech, 323 synapse.tests.test_model_language, 324 synapse.tests.test_model_material, 324 synapse.tests.test_model_media, 324 synapse.tests.test_model_orgs, 324 synapse.tests.test_model_person, 324 synapse.tests.test_model_syn, 325 synapse.tests.test_model_telco, 325 synapse.tests.test_model_temporal, 325 synapse.tests.test_reactor, 325 synapse.tests.test_runtime, 326 synapse.tests.test_statemach, 326 synapse.tests.test_swarm_runtime, 326 synapse.tests.test_telepath, 327 synapse.tests.test_tools_autodoc, 327 synapse.tests.test_tools_dmon, 328 synapse.tests.test_tools_dumprows_loadrows, 328 synapse.tests.test_tools_easycert, 328 synapse.tests.test_tools_ingest, 329 synapse.tests.test_tools_json2mpk, 329 synapse.tests.test_tools_pushfile, 329 synapse.tests.test_tools_superhash, 329 synapse.tools, 332 synapse.tools.autodoc, 329 synapse.tools.cmdr, 330 synapse.tools.dmon, 330 synapse.tools.dumprows, 330 synapse.tools.easycert, 331 synapse.tools.ingest, 331 synapse.tools.json2mpk, 331 synapse.tools.loadrows, 331 synapse.tools.pushfile, 331 synapse.tools.superhash, 331

Python Module Index 397 Synapse Documentation, Release 0.0.34

398 Python Module Index Index

A addJsonItems() (synapse.cores.common.Cortex method), accept() (synapse.lib.socket.Socket method), 268 198 ack() (synapse.lib.fifo.Fifo method), 238 addJsonText() (synapse.cores.common.Cortex method), ack() (synapse.lib.fifo.Window method), 238 198 ackCoreFifo() (synapse.cores.common.Cortex method), addLines() (synapse.tools.autodoc.RstHelp method), 329 198 addLinkProto() (in module synapse.link), 358 acquire() (synapse.cores.xact.StoreXact method), 223 addListRows() (synapse.cores.common.Cortex method), act() (synapse.reactor.Reactor method), 359 198 add() (synapse.axon.AxonHost method), 338 addOnInst() (synapse.daemon.OnHelp method), 344 add() (synapse.lib.cache.OnDem method), 229 addPermDefs() (synapse.cores.common.Cortex method), add() (synapse.lib.iq.TstEnv method), 246 199 add() (synapse.lib.persist.Dir method), 252 addPlexSock() (synapse.lib.socket.Plex method), 268 add() (synapse.lib.persist.File method), 252 addpref() (in module synapse.common), 339 add() (synapse.lib.scope.Scope method), 264 addPropDef() (synapse.datamodel.DataModel method), add() (synapse.lib.storm.Query method), 270 344 add() (synapse.lib.trigger.Triggers method), 278 addRoleRule() (synapse.cores.common.Cortex method), addApiPath() (synapse.lib.webapp.WebApp method), 285 199 addCmdClass() (synapse.lib.cli.Cli method), 231 addRows() (synapse.cores.common.Cortex method), 199 addConfDef() (synapse.lib.config.Configable method), addRows() (synapse.cores.storage.Storage method), 216 233 AddrType (class in synapse.models.inet), 290 addConfDefs() (synapse.lib.config.Configable method), addRuntNode() (synapse.cores.common.Cortex method), 234 199 addCoreMods() (synapse.cores.common.Cortex method), addSaveLink() (synapse.cores.common.Cortex method), 198 199 addDataModel() (synapse.lib.types.TypeLib method), addSaveLink() (synapse.cores.storage.Storage method), 281 217 addDataModels() (synapse.lib.types.TypeLib method), addSeedCtor() (synapse.cores.common.Cortex method), 281 199 addDynAlias() (in module synapse.dyndeps), 348 addSpliceFd() (synapse.cores.common.Cortex method), addFilePath() (synapse.lib.webapp.WebApp method), 200 285 addStateDelta() (synapse.statemach.StateMachine addFormat() (in module synapse.lib.ingest), 242 method), 359 addGestData() (synapse.lib.ingest.IngestApi method), addStatFunc() (synapse.cores.common.Cortex method), 242 200 addHandPath() (synapse.lib.webapp.WebApp method), addSynMixin() (in module synapse.lib.mixins), 247 285 addTstForms() (synapse.lib.iq.SynTest static method), addHead() (synapse.tools.autodoc.RstHelp method), 329 244 addJsonItem() (synapse.cores.common.Cortex method), addTufoDark() (synapse.cores.common.Cortex method), 198 200 addTufoDset() (synapse.cores.common.Cortex method),

399 Synapse Documentation, Release 0.0.34

200 BadInfoValu, 352 addTufoForm() (synapse.datamodel.DataModel method), BadJson, 352 344 BadMesgResp, 352 addTufoList() (synapse.cores.common.Cortex method), BadMesgVers, 352 200 BadOperArg, 352 addTufoProp() (synapse.datamodel.DataModel method), BadPropConf, 352 344 BadPropName, 352 addTufoTag() (synapse.cores.common.Cortex method), BadPropValu, 352 200 BadPySource, 352 addTufoTags() (synapse.cores.common.Cortex method), BadRevValu, 352 201 BadRuleSyntax, 352 addType() (synapse.lib.types.TypeLib method), 281 BadRuleValu, 353 addTypeCast() (synapse.lib.types.TypeLib method), 281 BadStorValu, 353 addup() (synapse.tests.test_lib_webapp.Foo method), 319 BadSyntaxError, 353 addUserRule() (synapse.cores.common.Cortex method), BadTypeValu, 353 201 BadUrl, 353 addWebConfig() (synapse.lib.remcycle.Hypnos method), Bar (class in synapse.tests.test_lib_mixins), 310 256 Bar (class in synapse.tests.test_lib_reflect), 311 alloc() (synapse.axon.Axon method), 333 bar() (synapse.tests.test_dyndeps.Foo method), 302 alloc() (synapse.axon.AxonCluster method), 337 bar() (synapse.tests.test_lib_webapp.Foo method), 319 alloc() (synapse.lib.heap.Heap method), 241 bar() (synapse.tests.test_link_ssl.FooBar method), 320 allow() (synapse.lib.auth.Rules method), 226 bar() (synapse.tests.test_telepath.Foo method), 327 allowed() (synapse.cores.common.Cortex method), 201 BarMod (class in synapse.tests.test_lib_module), 310 allows() (synapse.lib.ratelimit.RateLimit method), 254 BaseHand (class in synapse.lib.webapp), 285 asint() (in module synapse.gene), 357 basic_core_expectations() ask() (synapse.lib.storm.Runtime method), 271 (synapse.tests.test_cortex.CortexBaseTest AskCmd (class in synapse.cmds.cortex), 197 method), 298 AsyncTests (class in synapse.tests.test_async), 296 baz() (synapse.tests.test_telepath.Foo method), 327 at() (synapse.lib.sched.Sched method), 263 begin() (synapse.cores.xact.StoreXact method), 223 AtomFile (class in synapse.lib.atomfile), 225 bint() (in module synapse.gene), 357 AtomTest (class in synapse.tests.test_lib_atomfile), 304 BioMod (class in synapse.models.biology), 289 AtomTest (class in synapse.tests.test_lib_atomic), 305 Blah (class in synapse.tests.test_daemon), 301 AuthDeny, 352 block_processing() (in module AuthTest (class in synapse.tests.test_lib_auth), 305 synapse.tests.test_eventbus), 303 Axon (class in synapse.axon), 333 BoolType (class in synapse.lib.types), 280 AxonCluster (class in synapse.axon), 337 Boss (class in synapse.async), 332 AxonClusterTest (class in synapse.tests.test_axon), 296 bruteStr() (in module synapse.models.infotech), 292 AxonFSTest (class in synapse.tests.test_axon), 296 bruteVersion() (in module synapse.models.infotech), 292 AxonHost (class in synapse.axon), 338 bruteVersionValu() (in module synapse.models.infotech), AxonHostTest (class in synapse.tests.test_axon), 297 292 AxonMixin (class in synapse.axon), 338 buildHttpRequest() (synapse.lib.remcycle.Nyx method), AxonMod (class in synapse.models.axon), 289 263 AxonModelTest (class in synapse.tests.test_model_axon), bumpref() (synapse.lib.cache.RefDict method), 229 320 BusRef (class in synapse.eventbus), 349 AxonPathType (class in synapse.models.axon), 289 byiden() (synapse.axon.Axon method), 333 AxonTest (class in synapse.tests.test_axon), 297 byiden() (synapse.axon.AxonCluster method), 337 ByTag (class in synapse.lib.tags), 274 B bytes() (synapse.axon.Axon method), 333 BadAtomFile, 352 bytes() (synapse.axon.AxonCluster method), 338 BadConfValu, 352 BytsNommer (class in synapse.tests.test_lib_remcycle), BadCoreName, 352 312 BadCoreStore, 352 BadCtorType, 352 C BadFifoSeq, 352 Cache (class in synapse.lib.cache), 227

400 Index Synapse Documentation, Release 0.0.34

CacheTest (class in synapse.tests.test_lib_cache), 305 CmdQuit (class in synapse.lib.cli), 233 call() (in module synapse.lib.modules), 249 CmpSet (class in synapse.lib.atomic), 226 call() (synapse.lib.threads.Pool method), 276 CnGovTest (class in synapse.tests.test_model_gov_cn), call() (synapse.telepath.Proxy method), 360 322 call_ctor() (in module synapse.lib.modules), 250 commit() (synapse.cores.xact.StoreXact method), 223 callByIden() (synapse.lib.service.SvcProxy method), 266 CommonTest (class in synapse.tests.test_common), 298 callByName() (synapse.lib.service.SvcProxy method), CompType (class in synapse.lib.types), 280 266 compute_hashes() (in module synapse.tools.superhash), callByTag() (synapse.lib.service.SvcProxy method), 266 331 CallCapt (class in synapse.dyndeps), 348 confdef() (in module synapse.lib.config), 236 CallNode (class in synapse.gene), 357 Config (class in synapse.lib.config), 233 CallTask (class in synapse.lib.task), 275 Configable (class in synapse.lib.config), 233 callx() (synapse.telepath.Proxy method), 360 ConfTest (class in synapse.tests.test_lib_config), 306 cancel() (synapse.lib.sched.Sched method), 263 connect() (in module synapse.lib.socket), 269 cancel() (synapse.lib.storm.Query method), 270 connect() (synapse.links.common.LinkRelay method), cancelable (class in synapse.lib.threads), 277 286 canstor() (in module synapse.common), 339 consume() (synapse.eventbus.EventBus method), 349 CantDelProp, 353 CoreModTest (class in synapse.tests.test_lib_module), CantSetProp, 353 310 castInetDeFang() (in module synapse.models.inet), 292 CoreModule (class in synapse.lib.module), 247 cedetime() (synapse.cores.xact.StoreXact method), 223 CoreTestDataModelModuleV0 (class in CertDir (class in synapse.lib.certdir), 230 synapse.tests.test_cortex), 298 CertDirTest (class in synapse.tests.test_lib_certdir), 305 CoreTestDataModelModuleV1 (class in check_json_blob() (synapse.tests.test_tools_superhash.SuperhashTest synapse.tests.test_cortex), 298 method), 329 CoreTestModule (class in synapse.tests.test_cortex), 298 checkConfDict() (in module synapse.daemon), 344 Cortex (class in synapse.cores.common), 198 checkConfFile() (in module synapse.daemon), 344 CortexBaseTest (class in synapse.tests.test_cortex), 298 ChemMod (class in synapse.models.chemistry), 289 CortexTest (class in synapse.tests.test_cortex), 299 chopLinkUrl() (in module synapse.link), 358 count() (synapse.lib.cache.RefDict method), 229 choptag() (in module synapse.cortex), 341 CountryLookTest (class in chopurl() (in module synapse.lib.urlhelp), 284 synapse.tests.test_lookup_iso3166), 320 chunk() (synapse.axon.Axon method), 334 CryptoMod (class in synapse.models.crypto), 289 chunk() (synapse.axon.AxonCluster method), 338 CryptoModelTest (class in chunked() (in module synapse.cores.common), 213 synapse.tests.test_model_crypto), 321 chunks() (in module synapse.common), 339 ctor() (in module synapse.lib.scope), 265 CidrType (class in synapse.models.inet), 290 ctor() (synapse.lib.scope.Scope method), 264 clear() (synapse.gene.GeneLab method), 357 Curator (class in synapse.lib.session), 268 clear() (synapse.lib.cache.Cache method), 227 current() (in module synapse.lib.threads), 277 clear() (synapse.lib.cache.FixedCache method), 228 clear() (synapse.lib.cache.RefDict method), 229 D clear() (synapse.lib.storm.Query method), 270 Daemon (class in synapse.daemon), 342 clear() (synapse.lib.trigger.Triggers method), 278 daemonize() (in module synapse.lib.platforms.common), Cli (class in synapse.lib.cli), 231 224 clientside() (in module synapse.telepath), 360 daemonize() (in module synapse.lib.platforms.windows), CliFini, 232 225 CliTest (class in synapse.tests.test_lib_cli), 306 DaemonTest (class in synapse.tests.test_daemon), 301 close() (synapse.lib.filepath.FpFile method), 238 data() (synapse.lib.storm.Query method), 270 close() (synapse.lib.filepath.FpOpener method), 239 DataElem (class in synapse.lib.datapath), 237 close() (synapse.lib.filepath.FpTar method), 239 DataModel (class in synapse.datamodel), 344 close() (synapse.lib.socket.Socket method), 269 DataModelTest (class in synapse.tests.test_datamodel), Cmd (class in synapse.lib.cli), 232 301 cmd_hehe() (synapse.tests.test_lib_cli.Hehe method), DataPathTest (class in synapse.tests.test_lib_datapath), 306 307 CmdHelp (class in synapse.lib.cli), 232 DataTest (class in synapse.tests.test_data), 301

Index 401 Synapse Documentation, Release 0.0.34

DataType (class in synapse.lib.types), 280 258 DataTypesTest (class in synapse.tests.test_lib_types), 318 deMsgB64() (in module synapse.lib.types), 283 DatFileTest (class in synapse.tests.test_lib_datﬁle), 307 description() (synapse.lib.remcycle.Nyx method), 263 dblim (synapse.cores.postgres.PsqlStorage attribute), 214 digests() (synapse.lib.hashset.HashSet method), 241 dblim (synapse.cores.sqlite.SqliteStorage attribute), 215 digits() (in module synapse.models.telco), 294 DbPool (class in synapse.cores.sqlite), 215 Dir (class in synapse.lib.persist), 252 debase64() (in module synapse.common), 339 dist() (synapse.daemon.OnHelp method), 344 dec() (synapse.lib.storm.LimitHelp method), 270 dist() (synapse.eventbus.EventBus method), 349 decode() (in module synapse.lib.encoding), 237 distall() (synapse.eventbus.EventBus method), 349 delAddRows() (synapse.cores.common.Cortex method), DmonConf (class in synapse.daemon), 343 201 dmoneval() (synapse.daemon.DmonConf method), 343 delBlobValu() (synapse.cores.common.Cortex method), DnsMod (class in synapse.models.dns), 289 201 DnsModelTest (class in synapse.tests.test_model_dns), delBlobValu() (synapse.cores.storage.Storage method), 321 217 done() (synapse.async.Boss method), 332 delDynAlias() (in module synapse.dyndeps), 348 done() (synapse.lib.queue.Queue method), 253 delete() (synapse.cores.sqlite.SqliteStorage method), 215 dump_blobs() (in module synapse.tools.dumprows), 330 delJoinByProp() (synapse.cores.common.Cortex dump_rows() (in module synapse.tools.dumprows), 330 method), 201 dump_store() (in module synapse.tools.dumprows), 330 delOnInst() (synapse.daemon.OnHelp method), 344 DumpRowsTest (class in delRoleRule() (synapse.cores.common.Cortex method), synapse.tests.test_tools_dumprows_loadrows), 201 328 delRowsById() (synapse.cores.common.Cortex method), DupFileName, 353 201 DupOpt, 353 delRowsById() (synapse.cores.storage.Storage method), DupPropName, 353 217 DupRole, 353 delRowsByIdProp() (synapse.cores.common.Cortex DupTypeName, 353 method), 202 DupUser, 353 delRowsByIdProp() (synapse.cores.storage.Storage DynDepsTest (class in synapse.tests.test_dyndeps), 302 method), 217 delRowsByProp() (synapse.cores.common.Cortex E method), 202 eatbytes() (synapse.axon.AxonMixin method), 338 delRowsByProp() (synapse.cores.storage.Storage eatfd() (synapse.axon.AxonMixin method), 338 method), 217 eatfd() (synapse.lib.hashset.HashSet method), 241 delTufo() (synapse.cores.common.Cortex method), 202 eatSpliceFd() (synapse.cores.common.Cortex method), delTufoByProp() (synapse.cores.common.Cortex 203 method), 202 echo() (synapse.tests.test_telepath.Foo method), 327 delTufoDark() (synapse.cores.common.Cortex method), EmailType (class in synapse.models.inet), 290 202 en() (in module synapse.lib.msgpack), 251 delTufoDset() (synapse.cores.common.Cortex method), enbase64() (in module synapse.common), 339 202 encode() (in module synapse.lib.encoding), 238 delTufoIval() (synapse.cores.common.Cortex method), EncTest (class in synapse.tests.test_lib_encoding), 307 202 enMsgB64() (in module synapse.lib.types), 283 delTufoListValu() (synapse.cores.common.Cortex enter() (in module synapse.lib.scope), 265 method), 202 enter() (synapse.lib.scope.Scope method), 264 delTufoProp() (synapse.cores.common.Cortex method), ephem() (in module synapse.lib.tufo), 279 202 EpochType (class in synapse.models.temporal), 295 delTufosByProp() (synapse.cores.common.Cortex eq() (in module synapse.lib.storm), 272 method), 203 eq() (synapse.lib.iq.SynTest method), 244 delTufoTag() (synapse.cores.common.Cortex method), equal() (in module synapse.lib.tufo), 279 202 err() (synapse.async.Boss method), 332 delUserRule() (synapse.cores.common.Cortex method), err() (synapse.lib.task.Task method), 275 203 errinfo() (in module synapse.common), 339 delWebConf() (synapse.lib.remcycle.Hypnos method), eval() (in module synapse.gene), 357

402 Index Synapse Documentation, Release 0.0.34 eval() (synapse.gene.GeneNode method), 357 Foo (class in synapse.tests.test_dyndeps), 302 eval() (synapse.lib.storm.Runtime method), 271 Foo (class in synapse.tests.test_lib_config), 306 evalurl() (in module synapse.telepath), 361 Foo (class in synapse.tests.test_lib_mixins), 310 EventBus (class in synapse.eventbus), 349 Foo (class in synapse.tests.test_lib_reflect), 311 EventBusTest (class in synapse.tests.test_eventbus), 303 Foo (class in synapse.tests.test_lib_webapp), 319 exc() (synapse.eventbus.EventBus method), 350 Foo (class in synapse.tests.test_telepath), 327 excinfo() (in module synapse.common), 339 foo() (synapse.tests.test_daemon.Woot method), 301 exists() (in module synapse.lib.filepath), 240 foo() (synapse.tests.test_lib_service.Woot method), 314 expect() (synapse.lib.iq.TstOutPut method), 247 foo() (synapse.tests.test_link.FooBar method), 319 exprbase() (in module synapse.gene), 357 FooBar (class in synapse.tests.test_link), 319 expression() (in module synapse.gene), 357 FooBar (class in synapse.tests.test_link_ssl), 320 exprlist() (in module synapse.gene), 357 foodefs() (synapse.tests.test_lib_config.Foo static extend() (synapse.lib.types.DataType method), 280 method), 306 FooMod (class in synapse.tests.test_lib_module), 310 F forkdmon() (in module synapse.daemon), 344 FakeIpify (class in synapse.tests.test_lib_remcycle), 312 form() (synapse.lib.module.CoreModule method), 248 FakeType (class in synapse.tests.test_cortex), 300 formNodeByBytes() (synapse.cores.common.Cortex false() (synapse.lib.iq.SynTest method), 244 method), 203 FastAtom (class in synapse.lib.atomfile), 226 formNodeByFd() (synapse.cores.common.Cortex feed() (synapse.lib.msgpack.Unpk method), 251 method), 203 Fifo (class in synapse.lib.fifo), 238 formPhoneNode() (in module FifoTest (class in synapse.tests.test_lib_fifo), 307 synapse.lookup.phonenum), 288 File (class in synapse.lib.persist), 252 formTufoByProp() (synapse.cores.common.Cortex FileBaseType (class in synapse.models.files), 289 method), 203 FileExists, 353 formTufoByTufo() (synapse.cores.common.Cortex FileMod (class in synapse.models.files), 290 method), 204 FileModelTest (class in synapse.tests.test_model_files), formTufosByProps() (synapse.cores.common.Cortex 321 method), 204 FilePathType (class in synapse.models.files), 290 FpFile (class in synapse.lib.filepath), 238 FileRawPathType (class in synapse.models.files), 290 FpOpener (class in synapse.lib.filepath), 239 fill() (synapse.lib.fifo.Window method), 238 FpTar (class in synapse.lib.filepath), 239 find() (synapse.axon.Axon method), 334 FpZip (class in synapse.lib.filepath), 239 find() (synapse.axon.AxonCluster method), 338 FqdnType (class in synapse.models.inet), 291 fini() (synapse.eventbus.EventBus method), 350 fromdir() (in module synapse.cortex), 341 fini() (synapse.eventbus.Waiter method), 351 fromstore() (in module synapse.cortex), 341 fini() (synapse.lib.iq.TstEnv method), 247 fromUnixEpoch() (in module synapse.models.temporal), finiSnap() (synapse.cores.common.Cortex method), 203 295 FinMod (class in synapse.models.finance), 290 fs_create() (synapse.axon.Axon method), 334 fire() (synapse.cores.xact.StoreXact method), 223 fs_getattr() (synapse.axon.Axon method), 334 fire() (synapse.eventbus.EventBus method), 350 fs_getxattr() (synapse.axon.Axon method), 334 fire() (synapse.telepath.Proxy method), 360 fs_mkdir() (synapse.axon.Axon method), 335 fireall() (synapse.cores.xact.StoreXact method), 223 fs_read() (synapse.axon.Axon method), 335 firethread() (in module synapse.common), 339 fs_readdir() (synapse.axon.Axon method), 335 fireWebApi() (synapse.lib.remcycle.Hypnos method), fs_rename() (synapse.axon.Axon method), 335 258 fs_rmdir() (synapse.axon.Axon method), 335 fireWebApp() (synapse.tests.test_lib_remcycle.StandaloneTestServerfs_truncate() (synapse.axon.Axon method), 336 method), 313 fs_unlink() (synapse.axon.Axon method), 336 FixedCache (class in synapse.lib.cache), 228 fs_utimens() (synapse.axon.Axon method), 336 flush() (synapse.lib.atomfile.AtomFile method), 225 flush() (synapse.lib.cache.Cache method), 227 G flush() (synapse.lib.fifo.Fifo method), 238 ge() (in module synapse.lib.storm), 272 flush() (synapse.lib.fifo.Window method), 238 ge() (synapse.lib.iq.SynTest method), 244 fmtVersion() (in module synapse.lib.version), 284 gen() (synapse.eventbus.BusRef method), 349 fold() (in module synapse.lib.interval), 243

Index 403 Synapse Documentation, Release 0.0.34 gen_backup_tufo() (in module synapse.tools.dumprows), get_vertex_global_config() (in module 330 synapse.tests.test_lib_remcycle), 313 genCaCert() (synapse.lib.certdir.CertDir method), 230 getArgParser() (in module synapse.tools.dmon), 330 genDarkRows() (in module synapse.lib.dark), 236 getArgParser() (in module synapse.tools.json2mpk), 331 gendir() (in module synapse.common), 339 getArgParser() (in module synapse.tools.pushfile), 331 GeneLab (class in synapse.gene), 357 getAtomFile() (in module synapse.lib.atomfile), 226 GeneNode (class in synapse.gene), 357 getAxonInfo() (synapse.axon.Axon method), 336 GeneTest (class in synapse.tests.test_gene), 303 getBaseModels() (synapse.lib.module.CoreModule static GeneTest (class in synapse.tests.test_lib_scope), 313 method), 248 genfile() (in module synapse.common), 339 getBaseModels() (synapse.models.axon.AxonMod static genHostCert() (synapse.lib.certdir.CertDir method), 230 method), 289 genHostCsr() (synapse.lib.certdir.CertDir method), 230 getBaseModels() (synapse.models.biology.BioMod static genJoinByProp() (synapse.cores.storage.Storage method), 289 method), 218 getBaseModels() (synapse.models.chemistry.ChemMod genpath() (in module synapse.common), 339 static method), 289 genStoreRows() (synapse.cores.storage.Storage method), getBaseModels() (synapse.models.crypto.CryptoMod 218 static method), 289 gentask() (in module synapse.common), 339 getBaseModels() (synapse.models.dns.DnsMod static genTelLocCast() (in module synapse.models.telco), 294 method), 289 genUserCert() (synapse.lib.certdir.CertDir method), 230 getBaseModels() (synapse.models.files.FileMod static genUserCsr() (synapse.lib.certdir.CertDir method), 230 method), 290 GeoMod (class in synapse.models.geospace), 290 getBaseModels() (synapse.models.finance.FinMod static GeopolModelTest (class in method), 290 synapse.tests.test_model_geopol), 321 getBaseModels() (synapse.models.geopol.PolMod static get() (in module synapse.data), 224 method), 290 get() (in module synapse.lib.scope), 265 getBaseModels() (synapse.models.geospace.GeoMod get() (in module synapse.lib.thishost), 276 static method), 290 get() (synapse.cores.sqlite.DbPool method), 215 getBaseModels() (synapse.models.gov.cn.GovCnMod get() (synapse.eventbus.BusRef method), 349 static method), 288 get() (synapse.exc.SynErr method), 356 getBaseModels() (synapse.models.gov.intl.GovIntlMod get() (synapse.lib.cache.Cache method), 227 static method), 288 get() (synapse.lib.cache.FixedCache method), 228 getBaseModels() (synapse.models.gov.us.GovUsMod get() (synapse.lib.cache.KeyCache method), 228 static method), 288 get() (synapse.lib.cache.OnDem method), 229 getBaseModels() (synapse.models.inet.InetMod static get() (synapse.lib.cache.RefDict method), 229 method), 291 get() (synapse.lib.cli.Cli method), 231 getBaseModels() (synapse.models.infotech.ItMod static get() (synapse.lib.ingest.Ingest method), 242 method), 292 get() (synapse.lib.persist.Offset method), 252 getBaseModels() (synapse.models.language.LangMod get() (synapse.lib.queue.Queue method), 253 static method), 293 get() (synapse.lib.scope.Scope method), 265 getBaseModels() (synapse.models.material.MatMod get() (synapse.lib.session.Curator method), 268 static method), 293 get() (synapse.lib.session.Sess method), 268 getBaseModels() (synapse.models.media.MediaMod get() (synapse.lib.socket.Socket method), 269 static method), 293 get() (synapse.lib.storm.LimitHelp method), 270 getBaseModels() (synapse.models.mime.MimeMod get() (synapse.lib.tags.ByTag method), 274 static method), 293 get() (synapse.lib.task.Task method), 275 getBaseModels() (synapse.models.money.MoneyMod get() (synapse.lib.trees.IntervalTree method), 278 static method), 293 get() (synapse.lib.types.DataType method), 280 getBaseModels() (synapse.models.orgs.OuMod static get() (synapse.lib.webapp.BaseHand method), 285 method), 293 get_fake_ipify_global_config() (in module getBaseModels() (synapse.models.person.PsMod static synapse.tests.test_lib_remcycle), 313 method), 294 get_fake_ipify_ingest_global_config() (in module getBaseModels() (synapse.models.science.SciMod static synapse.tests.test_lib_remcycle), 313 method), 294 get_input() (in module synapse.lib.cli), 233 getBaseModels() (synapse.models.syn.SynMod static

404 Index Synapse Documentation, Release 0.0.34

method), 294 getCorePath() (synapse.cores.common.Cortex method), getBaseModels() (synapse.models.telco.TelMod static 205 method), 294 getCoreXact() (synapse.cores.common.Cortex method), getBaseModels() (synapse.models.temporal.TimeMod 205 static method), 295 getCoreXact() (synapse.cores.storage.Storage method), getBaseModels() (synapse.tests.test_cortex.CoreTestDataModelModuleV0218 static method), 298 getDataType() (synapse.lib.types.TypeLib method), 282 getBaseModels() (synapse.tests.test_cortex.CoreTestDataModelModuleV1getDmonCore() (synapse.lib.iq.SynTest method), 244 static method), 298 getDynLocal() (in module synapse.dyndeps), 348 getBaseModels() (synapse.tests.test_cortex.CoreTestModulegetDynMeth() (in module synapse.dyndeps), 348 static method), 298 getDynMod() (in module synapse.dyndeps), 348 getBlobKeys() (synapse.cores.common.Cortex method), getfile() (in module synapse.common), 339 204 getFooEnv() (synapse.tests.test_telepath.TelePathTest getBlobKeys() (synapse.cores.storage.Storage method), method), 327 218 getFooMachine() (synapse.tests.test_statemach.StateMachTest getBlobValu() (synapse.cores.common.Cortex method), method), 326 204 getFooServ() (synapse.tests.test_telepath.TelePathTest getBlobValu() (synapse.cores.storage.Storage method), method), 327 218 getFormDefs() (synapse.datamodel.DataModel method), getbytes() (in module synapse.common), 339 345 getCaCert() (synapse.lib.certdir.CertDir method), 230 getFormReqs() (synapse.datamodel.DataModel method), getCaCertPath() (synapse.lib.certdir.CertDir method), 345 230 getGeneExpr() (synapse.gene.GeneLab method), 357 getCaKey() (synapse.lib.certdir.CertDir method), 230 getHostCaPath() (synapse.lib.certdir.CertDir method), getCaKeyPath() (synapse.lib.certdir.CertDir method), 230 231 getCertDir() (synapse.tests.test_lib_certdir.CertDirTest getHostCert() (synapse.lib.certdir.CertDir method), 231 method), 305 getHostCertPath() (synapse.lib.certdir.CertDir method), getClientCert() (synapse.lib.certdir.CertDir method), 230 231 getClientCertPath() (synapse.lib.certdir.CertDir method), getHostKey() (synapse.lib.certdir.CertDir method), 231 230 getHostKeyPath() (synapse.lib.certdir.CertDir method), getClientSides() (in module synapse.telepath), 361 231 getClsNames() (in module synapse.lib.reflect), 254 getIdenOffset() (synapse.lib.persist.Dir method), 252 getCmdBrief() (synapse.lib.cli.Cmd method), 232 getIngestCore() (in module synapse.tests.common), 295 getCmdByName() (synapse.lib.cli.Cli method), 232 getItemCmdr() (in module synapse.lib.cmdr), 233 getCmdDoc() (synapse.lib.cli.Cmd method), 232 getItemInfo() (in module synapse.lib.reflect), 254 getCmdItem() (synapse.lib.cli.Cmd method), 232 getItemLocals() (in module synapse.lib.reflect), 255 getCmdName() (synapse.lib.cli.Cmd method), 232 getJoinByProp() (synapse.cores.common.Cortex getCmdNames() (synapse.lib.cli.Cli method), 232 method), 205 getCmdOpts() (synapse.lib.cli.Cmd method), 232 getJoinByProp() (synapse.cores.sqlite.SqliteStorage getCmprFunc() (synapse.lib.storm.Runtime method), 271 method), 215 getConfDef() (synapse.lib.config.Configable method), getJoinByProp() (synapse.cores.storage.Storage method), 234 218 getConfDefs() (synapse.lib.config.Configable method), getJoinsBy() (synapse.cores.storage.Storage method), 234 218 getConfNorm() (synapse.lib.config.Configable method), getJsonItems() (synapse.cores.common.Cortex method), 234 205 getConfOpt() (synapse.lib.config.Configable method), getLibC() (in module synapse.lib.platforms.common), 234 224 getConfOpts() (synapse.lib.config.Configable method), getLibC() (in module synapse.lib.platforms.windows), 235 225 getCoreCmdr() (synapse.tests.test_cmds_cortex.SynCmdCoreTestgetLiftLimit() (synapse.lib.storm.Runtime method), 271 method), 297 getLiftLimitHelp() (synapse.lib.storm.Runtime method), getCoreFifo() (synapse.cores.common.Cortex method), 271 204 getLinkProp() (synapse.links.common.LinkRelay

Index 405 Synapse Documentation, Release 0.0.34

method), 286 206 getLinkRelay() (in module synapse.link), 358 getRowsBy() (synapse.cores.common.Cortex method), getLoggerStream() (synapse.lib.iq.SynTest method), 244 206 getMethName() (in module synapse.lib.reﬂect), 255 getRowsById() (synapse.cores.common.Cortex method), getModelDict() (synapse.datamodel.DataModel method), 206 345 getRowsById() (synapse.cores.lmdb.LmdbStorage getModlRevs() (synapse.lib.module.CoreModule method), 213 method), 248 getRowsById() (synapse.cores.ram.RamStorage method), getModlVers() (synapse.cores.common.Cortex method), 214 205 getRowsById() (synapse.cores.sqlite.SqliteStorage getModName() (synapse.lib.module.CoreModule method), 215 method), 248 getRowsById() (synapse.cores.storage.Storage method), getModPath() (synapse.lib.module.CoreModule method), 219 248 getRowsByIdens() (synapse.cores.postgres.PsqlStorage getNameProxy() (synapse.lib.service.SvcProxy method), method), 214 266 getRowsByIdens() (synapse.cores.ram.RamStorage getNodePrec() (synapse.gene.GeneNode method), 357 method), 214 getNyxApi() (synapse.lib.remcycle.Hypnos method), 260 getRowsByIdens() (synapse.cores.storage.Storage getPathJoin() (synapse.lib.certdir.CertDir method), 231 method), 219 getPathParts() (in module synapse.lib.ﬁlepath), 240 getRowsByIdProp() (synapse.cores.common.Cortex getPermDef() (synapse.cores.common.Cortex method), method), 206 206 getRowsByIdProp() (synapse.cores.lmdb.LmdbStorage getPgConn() (synapse.lib.iq.SynTest method), 244 method), 213 getPgCore() (synapse.lib.iq.SynTest method), 244 getRowsByIdProp() (synapse.cores.ram.RamStorage getPhoneInfo() (in module synapse.lookup.phonenum), method), 214 288 getRowsByIdProp() (synapse.cores.sqlite.SqliteStorage getPivotRows() (synapse.cores.common.Cortex method), method), 215 206 getRowsByIdProp() (synapse.cores.storage.Storage getPlexSocks() (synapse.lib.socket.Plex method), 268 method), 219 getPropDef() (synapse.datamodel.DataModel method), getRowsByProp() (synapse.cores.common.Cortex 345 method), 206 getPropFormBase() (synapse.datamodel.DataModel getRowsByProp() (synapse.cores.lmdb.LmdbStorage method), 345 method), 213 getPropInfo() (synapse.datamodel.DataModel method), getRowsByProp() (synapse.cores.ram.RamStorage 345 method), 214 getPropNorm() (synapse.datamodel.DataModel method), getRowsByProp() (synapse.cores.sqlite.SqliteStorage 345 method), 215 getPropParse() (synapse.datamodel.DataModel method), getRowsByProp() (synapse.cores.storage.Storage 346 method), 219 getPropRepr() (synapse.datamodel.DataModel method), getRstText() (synapse.tools.autodoc.RstHelp method), 346 329 getPropsByType() (synapse.datamodel.DataModel getSeqNode() (synapse.cores.common.Cortex method), method), 346 207 getPropType() (synapse.datamodel.DataModel method), getServBinds() (synapse.lib.webapp.WebApp method), 346 286 getPropTypeName() (synapse.datamodel.DataModel getSizeBy() (synapse.cores.common.Cortex method), method), 346 207 getPumpOffs() (synapse.lib.persist.Dir method), 252 getSizeByProp() (synapse.cores.common.Cortex getRamCore() (synapse.lib.iq.SynTest method), 245 method), 207 getRoleAuth() (synapse.cores.common.Cortex method), getSizeByProp() (synapse.cores.lmdb.LmdbStorage 206 method), 213 getRoleRules() (synapse.cores.common.Cortex method), getSizeByProp() (synapse.cores.ram.RamStorage 206 method), 214 getRoleUsers() (synapse.cores.common.Cortex method), getSizeByProp() (synapse.cores.sqlite.SqliteStorage

406 Index Synapse Documentation, Release 0.0.34

method), 215 getTestPath() (in module synapse.tests.common), 295 getSizeByProp() (synapse.cores.storage.Storage method), getTestWait() (synapse.lib.iq.SynTest method), 245 220 getTufoByIden() (synapse.cores.common.Cortex getSnapNext() (synapse.cores.common.Cortex method), method), 207 207 getTufoByProp() (synapse.cores.common.Cortex getSplicePump() (synapse.cores.common.Cortex method), 207 method), 207 getTufoByValu() (synapse.lib.cache.TufoPropCache getStatByProp() (synapse.cores.common.Cortex method), 230 method), 207 getTufoDarkNames() (synapse.cores.common.Cortex getStoreType() (synapse.cores.common.Cortex method), method), 208 207 getTufoDarkValus() (synapse.cores.common.Cortex getStoreType() (synapse.cores.lmdb.LmdbStorage method), 208 method), 213 getTufoDsets() (synapse.cores.common.Cortex method), getStoreType() (synapse.cores.postgres.PsqlStorage 208 method), 214 getTufoForms() (synapse.datamodel.DataModel method), getStoreType() (synapse.cores.ram.RamStorage method), 346 214 getTufoList() (synapse.cores.common.Cortex method), getStoreType() (synapse.cores.sqlite.SqliteStorage 208 method), 215 getTufosBy() (synapse.cores.common.Cortex method), getStoreType() (synapse.cores.storage.Storage method), 208 220 getTufosByDark() (synapse.cores.common.Cortex getStoreXact() (synapse.cores.lmdb.LmdbStorage method), 208 method), 213 getTufosByDset() (synapse.cores.common.Cortex getStoreXact() (synapse.cores.ram.RamStorage method), method), 208 214 getTufosByIdens() (synapse.cores.common.Cortex getStoreXact() (synapse.cores.sqlite.SqliteStorage method), 208 method), 215 getTufosByProp() (synapse.cores.common.Cortex getStoreXact() (synapse.cores.storage.Storage method), method), 209 220 getTufosByPropType() (synapse.cores.common.Cortex getStormCore() (synapse.lib.storm.Runtime method), 271 method), 209 getStormCore() (synapse.tests.test_runtime.StormRunTest getTufosByTag() (synapse.cores.common.Cortex method), 326 method), 209 getSubPropDefs() (synapse.datamodel.DataModel getTufoSubs() (in module synapse.lib.tags), 274 method), 346 getTypeBases() (synapse.lib.types.TypeLib method), 282 getSubProps() (synapse.datamodel.DataModel method), getTypeCast() (synapse.lib.types.TypeLib method), 282 346 getTypeDef() (synapse.lib.types.TypeLib method), 282 getSwarmEnv() (synapse.tests.test_swarm_runtime.SwarmRunBasegetTypeDefs() (synapse.lib.types.TypeLib method), 282 method), 326 getTypeInfo() (synapse.lib.types.TypeLib method), 282 getSynMixins() (in module synapse.lib.mixins), 247 getTypeInst() (synapse.lib.types.TypeLib method), 282 getSynSvc() (synapse.lib.service.SvcProxy method), 267 getTypeNorm() (in module synapse.datamodel), 347 getSynSvcByName() (synapse.lib.service.SvcProxy getTypeNorm() (synapse.lib.types.TypeLib method), 282 method), 267 getTypeOfs() (synapse.datamodel.DataModel method), getSynSvcs() (synapse.lib.service.SvcBus method), 265 346 getSynSvcs() (synapse.lib.service.SvcProxy method), 267 getTypeParse() (in module synapse.datamodel), 347 getSynSvcsByTag() (synapse.lib.service.SvcBus getTypeParse() (synapse.lib.types.TypeLib method), 282 method), 265 getTypeRepr() (in module synapse.datamodel), 347 getSynSvcsByTag() (synapse.lib.service.SvcProxy getTypeRepr() (synapse.lib.types.TypeLib method), 283 method), 267 getUnivProps() (synapse.datamodel.DataModel method), getTagProxy() (synapse.lib.service.SvcProxy method), 346 267 getUserAuth() (synapse.cores.common.Cortex method), getTempConﬁg() (synapse.tests.test_tools_dmon.TestMain 209 method), 328 getUserCaPath() (synapse.lib.certdir.CertDir method), getTestDir() (synapse.lib.iq.SynTest method), 245 231 getTestOutp() (synapse.lib.iq.SynTest method), 245 getUserCert() (synapse.lib.certdir.CertDir method), 231

Index 407 Synapse Documentation, Release 0.0.34 getUserCertPath() (synapse.lib.certdir.CertDir method), I 231 iAmAlive() (synapse.lib.service.SvcBus method), 266 getUserForHost() (synapse.lib.certdir.CertDir method), iAmSynSvc() (synapse.lib.service.SvcBus method), 266 231 iCantWait() (in module synapse.lib.threads), 277 getUserKey() (synapse.lib.certdir.CertDir method), 231 iden() (in module synapse.lib.threads), 277 getUserKeyPath() (synapse.lib.certdir.CertDir method), IdenProxy (class in synapse.lib.service), 265 231 iMayWait() (in module synapse.lib.threads), 277 getUserRoles() (synapse.cores.common.Cortex method), imeicsum() (in module synapse.models.telco), 295 209 ImeiType (class in synapse.models.telco), 294 getUserRules() (synapse.cores.common.Cortex method), ImplementMe, 286 209 ImsiType (class in synapse.models.telco), 294 getVolInfo() (in module synapse.lib.platforms.common), incref() (synapse.eventbus.EventBus method), 350 224 incTufoProp() (synapse.cores.common.Cortex method), getWebDescription() (synapse.lib.remcycle.Hypnos 209 method), 260 inet_ntop() (in module synapse.lib.platforms.common), GlobTest (class in synapse.tests.test_glob), 304 224 GovCnMod (class in synapse.models.gov.cn), 288 inet_ntop() (in module synapse.lib.socket), 270 GovIntlMod (class in synapse.models.gov.intl), 288 inet_pton() (in module synapse.lib.platforms.common), GovUsMod (class in synapse.models.gov.us), 288 224 gt() (in module synapse.lib.storm), 272 inet_pton() (in module synapse.lib.socket), 270 gt() (synapse.lib.iq.SynTest method), 245 InetMod (class in synapse.models.inet), 291 guid() (in module synapse.common), 339 InetModelTest (class in synapse.tests.test_model_inet), guid() (synapse.lib.hashset.HashSet method), 241 322 GuidCmd (class in synapse.cmds.common), 197 InetModelTest (class in GuidType (class in synapse.lib.types), 280 synapse.tests.test_model_temporal), 325 info() (synapse.axon.AxonHost method), 338 H InfoTechTest (class in has() (synapse.axon.Axon method), 336 synapse.tests.test_model_infotech), 323 has() (synapse.axon.AxonCluster method), 338 Ingest (class in synapse.lib.ingest), 242 has() (synapse.lib.cache.Cache method), 227 ingest() (synapse.lib.ingest.Ingest method), 242 hasBlobValu() (synapse.cores.common.Cortex method), IngestApi (class in synapse.lib.ingest), 242 209 IngTest (class in synapse.tests.test_lib_ingest), 308 hasBlobValu() (synapse.cores.storage.Storage method), initboot() (in module synapse.tools.dmon), 330 220 initcomp() (in module hashitem() (in module synapse.lib.hashitem), 241 synapse.tests.test_model_infotech), 324 HashItemTest (class in synapse.tests.test_lib_hashitem), initconf() (in module synapse.tools.dmon), 330 308 initCoreModule() (synapse.cores.common.Cortex HashSet (class in synapse.lib.hashset), 241 method), 209 Heap (class in synapse.lib.heap), 241 initCoreModule() (synapse.lib.module.CoreModule HeapTest (class in synapse.tests.test_lib_heap), 308 method), 248 Hehe (class in synapse.tests.test_lib_cli), 306 initCoreModule() (synapse.models.axon.AxonMod HitCoreLimit, 353 method), 289 HitMaxRetry, 353 initCoreModule() (synapse.models.ﬁles.FileMod HitMaxTime, 353 method), 290 HitStormLimit, 353 initCoreModule() (synapse.models.gov.cn.GovCnMod holdref() (synapse.lib.cache.RefDict method), 230 method), 288 Horked, 319 initCoreModule() (synapse.models.inet.InetMod horked() (synapse.tests.test_lib_webapp.Foo method), method), 291 319 initCoreModule() (synapse.models.infotech.ItMod hostaddr() (in module synapse.lib.socket), 270 method), 292 Hypnos (class in synapse.lib.remcycle), 255 initCoreModule() (synapse.models.orgs.OuMod method), HypnosTest (class in synapse.tests.test_lib_remcycle), 293 312 initCoreModule() (synapse.models.person.PsMod method), 294

408 Index Synapse Documentation, Release 0.0.34 initCoreModule() (synapse.models.telco.TelMod ipv4int() (in module synapse.models.inet), 292 method), 294 ipv4mask() (in module synapse.models.inet), 292 initCoreModule() (synapse.models.temporal.TimeMod ipv4str() (in module synapse.models.inet), 292 method), 295 IPv4Type (class in synapse.models.inet), 291 initCoreModule() (synapse.tests.test_cortex.CoreTestModuleipv6 (synapse.lib.platforms.windows.sockaddr attribute), method), 298 225 initelem() (in module synapse.lib.datapath), 237 ipv6norm() (in module synapse.models.inet), 292 initHostInfo() (in module IPv6Type (class in synapse.models.inet), 291 synapse.lib.platforms.common), 224 IqTest (class in synapse.tests.test_lib_iq), 309 initHostInfo() (in module synapse.lib.platforms.darwin), is_literal() (in module synapse.lib.syntax), 273 224 isCaCert() (synapse.lib.certdir.CertDir method), 231 initHostInfo() (in module synapse.lib.platforms.freebsd), isClientCert() (synapse.lib.certdir.CertDir method), 231 224 isDataModl() (synapse.lib.types.TypeLib method), 283 initHostInfo() (in module synapse.lib.platforms.linux), isDataType() (synapse.lib.types.TypeLib method), 283 225 isdir() (in module synapse.lib.filepath), 240 initHostInfo() (in module isfile() (in module synapse.lib.filepath), 240 synapse.lib.platforms.windows), 225 isfile() (synapse.lib.filepath.FpFile method), 238 initialize() (synapse.lib.webapp.BaseHand method), 285 IsFini, 353 initJob() (synapse.async.Boss method), 332 isfini() (in module synapse.lib.threads), 277 initJoinsBy() (synapse.cores.storage.Storage method), isguid() (in module synapse.common), 339 220 isHostCert() (synapse.lib.certdir.CertDir method), 231 initLmdbCortex() (in module synapse.cores.lmdb), 214 isin() (synapse.lib.iq.SynTest method), 245 initPhoneTree() (in module synapse.lookup.phonenum), isinstance() (synapse.lib.iq.SynTest method), 245 288 islist() (in module synapse.lib.types), 283 initPsqlCortex() (in module synapse.cores.postgres), 214 isOk() (synapse.cores.common.Cortex method), 210 initRamCortex() (in module synapse.cores.ram), 215 isProxy() (in module synapse.telepath), 361 initRowsBy() (synapse.cores.storage.Storage method), isquote() (in module synapse.lib.syntax), 273 220 isRuntForm() (synapse.cores.common.Cortex method), initSizeBy() (synapse.cores.storage.Storage method), 221 210 initSqliteCortex() (in module synapse.cores.sqlite), 216 IsRuntProp, 353 initTufosBy() (synapse.cores.common.Cortex method), isRuntProp() (synapse.cores.common.Cortex method), 210 210 innerLs() (synapse.lib.filepath.FpTar method), 239 isSetPropOk() (synapse.cores.common.Cortex method), innrEnum() (synapse.lib.filepath.FpTar method), 239 210 innrEnum() (synapse.lib.filepath.FpZip method), 239 isSubType() (synapse.lib.types.TypeLib method), 283 innrExists() (synapse.lib.filepath.FpTar method), 239 istok() (in module synapse.gene), 357 innrIsdir() (synapse.lib.filepath.FpTar method), 239 istokoper() (in module synapse.gene), 358 innrIsfile() (synapse.lib.filepath.FpTar method), 239 isTufoForm() (synapse.datamodel.DataModel method), innrOpen() (synapse.lib.filepath.FpTar method), 239 346 innrOpen() (synapse.lib.filepath.FpZip method), 239 isUserCert() (synapse.lib.certdir.CertDir method), 231 innrTmpExtract() (synapse.lib.filepath.FpTar method), items() (synapse.exc.SynErr method), 356 239 items() (synapse.lib.persist.Dir method), 252 insec() (synapse.lib.sched.Sched method), 263 iter() (synapse.lib.datapath.DataElem method), 237 inspect_mod() (in module synapse.tools.autodoc), 329 iter() (synapse.lib.scope.Scope method), 265 IntervalTree (class in synapse.lib.trees), 278 iterblob() (synapse.axon.Axon method), 337 intify() (in module synapse.common), 339 iterblob() (synapse.axon.AxonCluster method), 338 IntType (class in synapse.lib.types), 280 iterdata() (in module synapse.lib.ingest), 242 InvalidParam, 341 iterfd() (in module synapse.common), 339 invert() (in module synapse.lib.storm), 273 iterfd() (in module synapse.lib.msgpack), 251 IPS (synapse.tests.test_lib_remcycle.FakeIpify attribute), iterFqdnUp() (in module synapse.lib.certdir), 231 312 iterTagDown() (in module synapse.lib.tags), 274 ipv4 (synapse.lib.platforms.windows.sockaddr attribute), iterTagUp() (in module synapse.lib.tags), 274 225 iterzip() (in module synapse.common), 339 ipv4cidr() (in module synapse.models.inet), 292 ItMod (class in synapse.models.infotech), 292

Index 409 Synapse Documentation, Release 0.0.34 ival() (in module synapse.lib.tufo), 279 loadDmonConf() (synapse.daemon.Daemon method), IvalTest (class in synapse.tests.test_lib_interval), 309 342 iWillWait() (in module synapse.lib.threads), 277 loadDmonConf() (synapse.daemon.DmonConf method), 343 J loadDmonFile() (synapse.daemon.DmonConf method), job() (synapse.async.Boss method), 332 344 jobDoneMesg() (in module synapse.async), 333 loadDmonJson() (synapse.daemon.DmonConf method), JobErr, 353 344 jobid() (in module synapse.async), 333 loadﬁle() (in module synapse.lib.ingest), 243 jobret() (in module synapse.async), 333 loadModModels() (synapse.lib.types.TypeLib method), jobs() (synapse.async.Boss method), 332 283 jsload() (in module synapse.common), 339 LoadRowsTest (class in Json2MpkTest (class in synapse.tests.test_tools_dumprows_loadrows), synapse.tests.test_tools_json2mpk), 329 328 JsonType (class in synapse.lib.types), 280 loadStateFd() (synapse.statemach.StateMachine method), jssave() (in module synapse.common), 340 359 LocalRelay (class in synapse.links.local), 286 K localthing() (synapse.tests.test_telepath.Foo method), 327 keepstate() (in module synapse.statemach), 359 log() (synapse.eventbus.EventBus method), 350 KeyCache (class in synapse.lib.cache), 228 log() (synapse.lib.storm.Query method), 270 keymeth() (in module synapse.lib.cache), 230 logical_and() (in module synapse.gene), 358 keys() (synapse.lib.cache.Cache method), 227 logical_or() (in module synapse.gene), 358 killDmonFork() (synapse.daemon.DmonConf method), loop() (synapse.lib.sched.Sched method), 263 343 lsboot() (in module synapse.tools.dmon), 330 lt() (in module synapse.lib.storm), 273 L lt() (synapse.lib.iq.SynTest method), 245 LangMod (class in synapse.models.language), 293 LangTest (class in synapse.tests.test_model_language), M 324 main() (in module synapse.tools.autodoc), 330 ldict() (in module synapse.lib.mixins), 247 main() (in module synapse.tools.cmdr), 330 le() (in module synapse.lib.storm), 273 main() (in module synapse.tools.dmon), 330 le() (synapse.lib.iq.SynTest method), 245 main() (in module synapse.tools.dumprows), 330 leave() (synapse.lib.scope.Scope method), 265 main() (in module synapse.tools.easycert), 331 len() (synapse.lib.iq.SynTest method), 245 main() (in module synapse.tools.ingest), 331 LimitHelp (class in synapse.lib.storm), 270 main() (in module synapse.tools.json2mpk), 331 LimitTest (class in synapse.tests.test_lib_storm), 314 main() (in module synapse.tools.loadrows), 331 link() (synapse.eventbus.EventBus method), 350 main() (in module synapse.tools.pushﬁle), 331 LinkErr, 354 main() (in module synapse.tools.superhash), 331 LinkNotAuth, 354 main() (synapse.eventbus.EventBus method), 350 LinkRefused, 354 make_sql_genrows_json() LinkRelay (class in synapse.links.common), 286 (synapse.tests.test_tools_dumprows_loadrows.DumpRowsTest links() (synapse.daemon.Daemon method), 342 method), 328 LinkTest (class in synapse.tests.test_link), 319 make_sql_genrows_json() listen() (in module synapse.lib.socket), 270 (synapse.tests.test_tools_dumprows_loadrows.LoadRowsTest listen() (in module synapse.link), 358 method), 328 listen() (synapse.daemon.Daemon method), 342 makeargpaser() (in module synapse.tools.dumprows), 331 listen() (synapse.lib.webapp.WebApp method), 286 makeargpaser() (in module synapse.tools.loadrows), 331 listen() (synapse.links.common.LinkRelay method), 286 makeargpaser() (in module synapse.tools.superhash), 331 ListNode (class in synapse.gene), 357 makeAxonOpts() (synapse.axon.AxonHost method), 338 LmdbStorage (class in synapse.cores.lmdb), 213 makeColLook() (in module synapse.lookup.iso3166), 287 LmdbXact (class in synapse.cores.lmdb), 214 makedirs() (in module synapse.common), 340 load() (in module synapse.lib.modules), 250 match() (synapse.lib.cache.MatchCache method), 229 load() (synapse.lib.storm.Query method), 270 MatchCache (class in synapse.lib.cache), 228 load_ctor() (in module synapse.lib.modules), 250 MatMod (class in synapse.models.material), 293

410 Index Synapse Documentation, Release 0.0.34

MatTest (class in synapse.tests.test_model_material), 324 norm() (synapse.lib.types.IntType method), 280 max_temp_sz (in module synapse.lib.filepath), 240 norm() (synapse.lib.types.JsonType method), 280 md5() (in module synapse.cores.postgres), 214 norm() (synapse.lib.types.NDefType method), 280 MediaMod (class in synapse.models.media), 293 norm() (synapse.lib.types.PermType method), 281 MediaTest (class in synapse.tests.test_model_media), 324 norm() (synapse.lib.types.PropValuType method), 281 meh() (in module synapse.lib.syntax), 273 norm() (synapse.lib.types.SeprType method), 281 MemAtom (class in synapse.lib.atomfile), 226 norm() (synapse.lib.types.StormType method), 281 Method (class in synapse.telepath), 359 norm() (synapse.lib.types.StrType method), 281 MimeMod (class in synapse.models.mime), 293 norm() (synapse.lib.types.TagType method), 281 MixTest (class in synapse.tests.test_lib_mixins), 310 norm() (synapse.lib.types.TimeType method), 281 modelrev() (in module synapse.lib.module), 249 norm() (synapse.lib.types.XrefType method), 283 MoneyMod (class in synapse.models.money), 293 norm() (synapse.models.axon.AxonPathType method), MsgPackTest (class in synapse.tests.test_lib_msgpack), 289 310 norm() (synapse.models.files.FileBaseType method), 289 MultiFieldType (class in synapse.lib.types), 280 norm() (synapse.models.files.FilePathType method), 290 MustBeLocal, 354 norm() (synapse.models.files.FileRawPathType method), MustBeProxy, 354 290 MustNotWait, 354 norm() (synapse.models.inet.AddrType method), 290 norm() (synapse.models.inet.CidrType method), 290 N norm() (synapse.models.inet.EmailType method), 291 Name (class in synapse.models.person), 294 norm() (synapse.models.inet.FqdnType method), 291 name() (synapse.lib.datapath.DataElem method), 237 norm() (synapse.models.inet.IPv4Type method), 291 ndef() (in module synapse.lib.tufo), 279 norm() (synapse.models.inet.IPv6Type method), 291 NDefType (class in synapse.lib.types), 280 norm() (synapse.models.inet.Srv4Type method), 291 ne() (synapse.lib.iq.SynTest method), 245 norm() (synapse.models.inet.Srv6Type method), 291 newtask() (in module synapse.async), 333 norm() (synapse.models.inet.UrlType method), 291 newtask() (in module synapse.lib.threads), 277 norm() (synapse.models.infotech.SemverType method), newtask() (in module synapse.tests.test_lib_threads), 316 292 next() (synapse.lib.filepath.FpFile method), 239 norm() (synapse.models.person.Name method), 294 next() (synapse.lib.filepath.FpTar method), 239 norm() (synapse.models.telco.ImeiType method), 294 nextchar() (in module synapse.lib.syntax), 273 norm() (synapse.models.telco.ImsiType method), 294 nextin() (in module synapse.lib.syntax), 273 norm() (synapse.models.telco.PhoneType method), 294 nexts() (synapse.lib.filepath.FpFile method), 239 norm() (synapse.models.temporal.EpochType method), nexts() (synapse.lib.filepath.FpTar method), 239 295 NextSeqCmd (class in synapse.cmds.cortex), 198 normdict() (in module synapse.lib.hashitem), 241 nextSeqValu() (synapse.cores.common.Cortex method), normitem() (in module synapse.lib.hashitem), 241 210 normiter() (in module synapse.lib.hashitem), 241 nextstr() (in module synapse.lib.syntax), 273 normpath() (in module synapse.lib.filepath), 240 nexttok() (in module synapse.gene), 358 NoSuch, 354 nn() (synapse.lib.iq.SynTest method), 246 NoSuchAct, 354 NoAuthUser, 354 NoSuchCmpr, 354 noboot() (in module synapse.tools.dmon), 330 NoSuchConf, 354 NoCurrSess, 354 NoSuchCore, 354 NoInitCore, 354 NoSuchCtor, 354 nom() (in module synapse.lib.syntax), 273 NoSuchData, 354 nom_whitespace() (in module synapse.lib.syntax), 273 NoSuchDecoder, 354 noms() (synapse.tests.test_lib_remcycle.BytsNommer NoSuchDir, 354 method), 312 NoSuchDyn, 354 none() (synapse.lib.iq.SynTest method), 246 NoSuchEncoder, 355 noprop() (synapse.lib.iq.SynTest method), 246 NoSuchEntity, 355 NoRevAllow, 354 NoSuchFifo, 355 norm() (synapse.lib.types.BoolType method), 280 NoSuchFile, 355 norm() (synapse.lib.types.CompType method), 280 NoSuchForm, 355 norm() (synapse.lib.types.GuidType method), 280 NoSuchFunc, 355

Index 411 Synapse Documentation, Release 0.0.34

NoSuchGetBy, 355 onNodeDel() (synapse.lib.module.CoreModule method), NoSuchIden, 355 249 NoSuchImpl, 355 onretn() (synapse.lib.task.Task method), 275 NoSuchJob, 355 onServerFini() (synapse.tests.test_lib_remcycle.StandaloneTestServer NoSuchMeth, 355 method), 313 NoSuchMod, 355 onSetFqdnSfx() (synapse.models.inet.InetMod method), NoSuchName, 355 291 NoSuchObj, 355 onTufoFormFqdn() (synapse.models.inet.InetMod NoSuchOper, 355 method), 291 NoSuchOpt, 355 onTufoFormPasswd() (synapse.models.inet.InetMod NoSuchPath, 355 method), 291 NoSuchPeer, 355 open() (synapse.lib.filepath.FpFile method), 239 NoSuchPerm, 355 open() (synapse.lib.filepath.FpTar method), 239 NoSuchProp, 355 openAtomFile() (in module synapse.lib.atomfile), 226 NoSuchProto, 286, 355 openDatFile() (in module synapse.lib.datfile), 237 NoSuchRole, 356 opendir() (in module synapse.lib.persist), 253 NoSuchRule, 356 openfd() (in module synapse.lib.openfile), 251 NoSuchSeq, 356 openfile() (in module synapse.lib.filepath), 240 NoSuchSess, 356 openfiles() (in module synapse.lib.filepath), 240 NoSuchStat, 356 OpenFileTest (class in synapse.tests.test_lib_openfile), NoSuchTufo, 356 310 NoSuchType, 356 openlink() (in module synapse.cortex), 341 NoSuchUser, 356 openlink() (in module synapse.telepath), 361 NotEmpty, 356 openstore() (in module synapse.cortex), 341 NotEnoughFree, 356 openurl() (in module synapse.cortex), 341 notin() (synapse.lib.iq.SynTest method), 246 openurl() (in module synapse.lib.service), 267 NotSupported, 356 openurl() (in module synapse.telepath), 361 NoValu (class in synapse.common), 339 oper() (in module synapse.lib.syntax), 273 now() (in module synapse.common), 340 OperNode (class in synapse.gene), 357 NoWritableAxons, 356 OperWith (class in synapse.lib.storm), 270 Nyx (class in synapse.lib.remcycle), 261 opt() (synapse.lib.storm.Query method), 270 NyxTest (class in synapse.tests.test_lib_remcycle), 312 OrgTest (class in synapse.tests.test_model_orgs), 324 OuMod (class in synapse.models.orgs), 293 O OutPut (class in synapse.lib.output), 251 off() (synapse.eventbus.EventBus method), 350 OutPutBytes (class in synapse.lib.output), 251 off() (synapse.telepath.Proxy method), 360 OutPutFd (class in synapse.lib.output), 251 Offset (class in synapse.lib.persist), 252 OutPutStr (class in synapse.lib.output), 251 on() (synapse.eventbus.EventBus method), 350 overlap() (in module synapse.lib.interval), 243 on() (synapse.telepath.Proxy method), 360 onAddMucd() (synapse.models.gov.cn.GovCnMod P method), 288 packVersion() (in module synapse.lib.version), 284 onboot() (in module synapse.tools.dmon), 330 pad() (synapse.lib.storm.ShowHelp method), 272 onConfOptSet() (synapse.lib.config.Configable method), parse() (in module synapse.lib.syntax), 273 235 parse() (in module synapse.lib.time), 278 OnDem (class in synapse.lib.cache), 229 parse() (synapse.lib.storm.Runtime method), 271 onfini() (synapse.eventbus.EventBus method), 351 parse() (synapse.lib.types.DataType method), 280 onFormAxonPath() (synapse.models.axon.AxonMod parse_cmd_kwarg() (in module synapse.lib.syntax), 273 method), 289 parse_cmd_kwlist() (in module synapse.lib.syntax), 273 onFormNode() (synapse.lib.module.CoreModule parse_cmd_string() (in module synapse.lib.syntax), 273 method), 248 parse_int() (in module synapse.lib.syntax), 273 OnHelp (class in synapse.daemon), 344 parse_list() (in module synapse.lib.syntax), 273 onNodeAdd() (synapse.lib.module.CoreModule method), parse_literal() (in module synapse.lib.syntax), 273 248 parse_macro_filt() (in module synapse.lib.syntax), 273 parse_macro_join() (in module synapse.lib.syntax), 273

412 Index Synapse Documentation, Release 0.0.34 parse_macro_lift() (in module synapse.lib.syntax), 273 Proxy (class in synapse.telepath), 359 parse_oper() (in module synapse.lib.syntax), 274 PsMod (class in synapse.models.person), 294 parse_opts() (in module synapse.lib.syntax), 274 PsqlStorage (class in synapse.cores.postgres), 214 parse_perm() (in module synapse.lib.syntax), 274 pump() (synapse.lib.persist.Dir method), 252 parse_ques() (in module synapse.lib.syntax), 274 push() (synapse.telepath.Proxy method), 360 parse_string() (in module synapse.lib.syntax), 274 put() (synapse.cores.sqlite.DbPool method), 215 parse_time() (in module synapse.lib.syntax), 274 put() (synapse.eventbus.BusRef method), 349 parse_valu() (in module synapse.lib.syntax), 274 put() (synapse.lib.cache.Cache method), 228 parsePath() (in module synapse.lib.filepath), 240 put() (synapse.lib.cache.KeyCache method), 228 parsePaths() (in module synapse.lib.filepath), 240 put() (synapse.lib.cache.RefDict method), 230 parseSemver() (in module synapse.lib.version), 284 put() (synapse.lib.fifo.Fifo method), 238 parsetime() (in module synapse.lib.interval), 243 put() (synapse.lib.queue.Queue method), 253 parsetypes() (in module synapse.datamodel), 347 put() (synapse.lib.tags.ByTag method), 274 parseVersionParts() (in module synapse.lib.version), 284 putCoreFifo() (synapse.cores.common.Cortex method), path() (synapse.lib.filepath.FpFile method), 239 211 path() (synapse.lib.filepath.FpTar method), 239 puts() (synapse.lib.cache.RefDict method), 230 PermType (class in synapse.lib.types), 280 PyCmd (class in synapse.cmds.common), 197 persec() (synapse.lib.sched.Sched method), 264 PersistTest (class in synapse.tests.test_lib_persist), 311 Q PersonTest (class in synapse.tests.test_model_person), Query (class in synapse.lib.storm), 270 324 QueryCancelled, 271 PhLookTest (class in synapse.tests.test_lookup_phonenum),QueryKilled, 271 320 QueryLimitTime, 271 phnode() (in module synapse.lookup.phonenum), 288 QueryLimitTouch, 271 PhoneType (class in synapse.models.telco), 294 Queue (class in synapse.lib.queue), 253 pid() (synapse.tests.test_daemon.Woot method), 301 QueueShutdown, 254 plan() (synapse.lib.storm.Runtime method), 271 QueueTest (class in synapse.tests.test_lib_queue), 311 Plex (class in synapse.lib.socket), 268 PolMod (class in synapse.models.geopol), 290 R Pool (class in synapse.lib.threads), 276 raises() (synapse.lib.iq.SynTest method), 246 pop() (synapse.eventbus.BusRef method), 349 raiseSockError() (in module synapse.links.common), 286 pop() (synapse.lib.cache.Cache method), 228 raisetok() (in module synapse.gene), 358 pop() (synapse.lib.cache.KeyCache method), 228 RamStorage (class in synapse.cores.ram), 214 pop() (synapse.lib.cache.RefDict method), 230 RamXact (class in synapse.cores.ram), 215 pop() (synapse.lib.tags.ByTag method), 274 random_ip() (synapse.tests.test_lib_remcycle.FakeIpify pops() (synapse.lib.cache.RefDict method), 230 method), 312 popTufosByProp() (synapse.cores.common.Cortex RateLimit (class in synapse.lib.ratelimit), 254 method), 210 RateLimitTest (class in synapse.tests.test_lib_ratelimit), post() (synapse.lib.webapp.BaseHand method), 285 311 postCoreModule() (synapse.lib.module.CoreModule reached() (synapse.lib.storm.LimitHelp method), 270 method), 249 react() (synapse.reactor.Reactor method), 359 prepStormCore() (synapse.tests.test_runtime.StormRunTest Reactor (class in synapse.reactor), 359 method), 326 ReactorTest (class in synapse.tests.test_reactor), 325 printf() (synapse.lib.cli.Cli method), 232 read() (synapse.lib.filepath.FpOpener method), 239 printf() (synapse.lib.cli.Cmd method), 232 reader() (synapse.lib.threads.RWLock method), 276 printf() (synapse.lib.output.OutPut method), 251 readiter() (synapse.lib.heap.Heap method), 241 PropNotFound, 356 readoff() (synapse.lib.atomfile.AtomFile method), 225 props() (in module synapse.lib.tufo), 279 readoff() (synapse.lib.heap.Heap method), 241 PropValuType (class in synapse.lib.types), 281 readoff() (synapse.lib.persist.File method), 252 proto (synapse.links.common.LinkRelay attribute), 286 rebuildTlib() (in module synapse.datamodel), 347 proto (synapse.links.local.LocalRelay attribute), 286 recv() (synapse.lib.socket.Socket method), 269 proto (synapse.links.ssh.SshRelay attribute), 287 recvall() (synapse.lib.socket.Socket method), 269 proto (synapse.links.ssl.SslRelay attribute), 287 recvobj() (synapse.lib.socket.Socket method), 269 proto (synapse.links.tcp.TcpRelay attribute), 287 RefDict (class in synapse.lib.cache), 229

Index 413 Synapse Documentation, Release 0.0.34

ReflectTest (class in synapse.tests.test_lib_reflect), 311 reqStorDict() (in module synapse.common), 340 register_ingest() (in module synapse.lib.ingest), 243 reqTufoForm() (synapse.datamodel.DataModel method), release() (synapse.cores.xact.StoreXact method), 223 347 release() (synapse.lib.threads.RWLock method), 277 resize() (synapse.lib.atomfile.AtomFile method), 225 reminder() (in module synapse.telepath), 361 result() (synapse.lib.storm.Query method), 270 repr() (in module synapse.lib.time), 278 resync() (synapse.lib.fifo.Fifo method), 238 repr() (synapse.lib.types.BoolType method), 280 resync() (synapse.lib.fifo.Window method), 238 repr() (synapse.lib.types.DataType method), 280 retn() (synapse.lib.storm.Query method), 270 repr() (synapse.lib.types.IntType method), 280 retn() (synapse.lib.task.Task method), 275 repr() (synapse.lib.types.TimeType method), 281 retry (synapse.exc.LinkErr attribute), 354 repr() (synapse.models.files.FileBaseType method), 289 retry (synapse.exc.LinkRefused attribute), 354 repr() (synapse.models.inet.CidrType method), 290 revModlVers() (synapse.cores.common.Cortex method), repr() (synapse.models.inet.EmailType method), 291 211 repr() (synapse.models.inet.FqdnType method), 291 rows() (synapse.lib.storm.ShowHelp method), 272 repr() (synapse.models.inet.IPv4Type method), 291 rowsByGe() (synapse.cores.lmdb.LmdbStorage method), repr() (synapse.models.inet.IPv6Type method), 291 213 repr() (synapse.models.inet.Srv4Type method), 291 rowsByGe() (synapse.cores.ram.RamStorage method), repr() (synapse.models.inet.Srv6Type method), 291 214 repr() (synapse.models.inet.UrlType method), 292 rowsByGe() (synapse.cores.sqlite.SqliteStorage method), repr() (synapse.models.infotech.SemverType method), 215 292 rowsByGe() (synapse.cores.storage.Storage method), 221 repr() (synapse.models.telco.PhoneType method), 294 rowsByGt() (synapse.cores.storage.Storage method), 221 repr() (synapse.models.temporal.EpochType method), rowsByLe() (synapse.cores.lmdb.LmdbStorage method), 295 213 reprvalu() (in module synapse.tools.autodoc), 330 rowsByLe() (synapse.cores.ram.RamStorage method), reqbytes() (in module synapse.common), 340 214 reqConfOk() (synapse.lib.config.Configable method), 235 rowsByLe() (synapse.cores.sqlite.SqliteStorage method), ReqConfOpt, 356 215 reqConfOpts() (synapse.lib.config.Configable method), rowsByLe() (synapse.cores.storage.Storage method), 221 235 rowsByLt() (synapse.cores.storage.Storage method), 221 reqCorePath() (synapse.cores.common.Cortex method), rowsByRange() (synapse.cores.lmdb.LmdbStorage 211 method), 213 reqDataType() (synapse.lib.types.TypeLib method), 283 rowsByRange() (synapse.cores.ram.RamStorage reqdir() (in module synapse.common), 340 method), 214 reqfile() (in module synapse.common), 340 rowsByRange() (synapse.cores.sqlite.SqliteStorage reqiden() (in module synapse.cores.common), 213 method), 215 reqIsProxy() (in module synapse.telepath), 361 rowsByRange() (synapse.cores.storage.Storage method), reqJoinByMeth() (synapse.cores.storage.Storage 221 method), 221 rowstotufos() (in module synapse.common), 340 reqlines() (in module synapse.common), 340 RstHelp (class in synapse.tools.autodoc), 329 reqModPath() (synapse.lib.module.CoreModule method), Rules (class in synapse.lib.auth), 226 249 run() (synapse.lib.storm.Runtime method), 271 reqNotProxy() (in module synapse.telepath), 362 run() (synapse.lib.task.Task method), 275 reqpath() (in module synapse.common), 340 run() (synapse.lib.threads.Thread method), 277 reqperm() (synapse.cores.common.Cortex method), 211 runas() (in module synapse.lib.auth), 227 reqPermDef() (synapse.cores.common.Cortex method), runblob() (synapse.tests.test_cortex.CortexBaseTest 211 method), 298 reqPropNorm() (synapse.datamodel.DataModel method), runBossPool() (synapse.async.Boss method), 332 347 runCmdLine() (synapse.lib.cli.Cli method), 232 reqRowsByMeth() (synapse.cores.storage.Storage runCmdLine() (synapse.lib.cli.Cmd method), 232 method), 221 runCmdLoop() (synapse.lib.cli.Cli method), 232 reqSizeByMeth() (synapse.cores.storage.Storage runCmdOpts() (synapse.cmds.common.GuidCmd method), 221 method), 197 reqstor() (in module synapse.common), 340

414 Index Synapse Documentation, Release 0.0.34 runCmdOpts() (synapse.cmds.common.PyCmd method), seedFileGoodHash() (synapse.models.files.FileMod 197 method), 290 runCmdOpts() (synapse.cmds.cortex.AskCmd method), seedFileMd5() (synapse.models.files.FileMod method), 198 290 runCmdOpts() (synapse.cmds.cortex.NextSeqCmd seedFileSha1() (synapse.models.files.FileMod method), method), 198 290 runCmdOpts() (synapse.lib.cli.CmdHelp method), 232 seedOrgAlias() (synapse.models.orgs.OuMod method), runCmdOpts() (synapse.lib.cli.CmdQuit method), 233 293 runcore() (synapse.tests.test_cortex.CortexBaseTest seedOrgName() (synapse.models.orgs.OuMod method), method), 298 293 rundarks() (synapse.tests.test_cortex.CortexBaseTest seedPersonGuidName() (synapse.models.person.PsMod method), 298 method), 294 rundsets() (synapse.tests.test_cortex.CortexBaseTest seek() (synapse.lib.filepath.FpOpener method), 239 method), 298 select() (synapse.cores.sqlite.SqliteStorage method), 215 runDynEval() (in module synapse.dyndeps), 348 selfSignCert() (synapse.lib.certdir.CertDir method), 231 runDynTask() (in module synapse.dyndeps), 348 SemverType (class in synapse.models.infotech), 292 runidens() (synapse.tests.test_cortex.CortexBaseTest send() (synapse.lib.socket.Socket method), 269 method), 298 send() (synapse.links.ssl.Socket method), 287 runItemCmdr() (in module synapse.lib.cmdr), 233 send_sig() (in module synapse.tests.test_eventbus), 303 runjson() (synapse.tests.test_cortex.CortexBaseTest sendHttpResp() (synapse.lib.webapp.BaseHand method), method), 298 285 runrange() (synapse.tests.test_cortex.CortexBaseTest SeprType (class in synapse.lib.types), 281 method), 298 Sess (class in synapse.lib.session), 268 runsnaps() (synapse.tests.test_cortex.CortexBaseTest SessTest (class in synapse.tests.test_lib_session), 314 method), 298 set() (in module synapse.lib.scope), 265 runstore() (synapse.tests.test_cortex.CortexBaseTest set() (synapse.lib.atomic.CmpSet method), 226 method), 298 set() (synapse.lib.cli.Cli method), 232 runStormBasics() (synapse.tests.test_runtime.StormRunTestset() (synapse.lib.ingest.Ingest method), 242 method), 326 set() (synapse.lib.persist.Offset method), 253 runStormStats() (synapse.tests.test_runtime.StormRunTest set() (synapse.lib.scope.Scope method), 265 method), 326 set() (synapse.lib.session.Sess method), 268 runSynSvc() (in module synapse.lib.service), 267 set() (synapse.lib.socket.Socket method), 269 runSynSvc() (synapse.lib.service.SvcProxy method), 267 set() (synapse.lib.task.Task method), 275 Runtime (class in synapse.lib.storm), 271 setAxonInfo() (synapse.axon.Axon method), 337 Runtime (class in synapse.swarm.runtime), 295 setBlobValu() (synapse.cores.common.Cortex method), runtufobydefault() (synapse.tests.test_cortex.CortexBaseTest 211 method), 298 setBlobValu() (synapse.cores.storage.Storage method), runTxLoop() (synapse.lib.socket.Socket method), 269 221 RWLock (class in synapse.lib.threads), 276 setblocking() (synapse.lib.socket.Socket method), 269 RWWith (class in synapse.lib.threads), 277 setBossPool() (synapse.async.Boss method), 332 rx() (synapse.lib.socket.Socket method), 269 setCmprCtor() (synapse.lib.storm.Runtime method), 272 setCmprFunc() (synapse.lib.storm.Runtime method), 272 S setConfOpt() (synapse.lib.config.Configable method), sa_family (synapse.lib.platforms.windows.sockaddr at- 235 tribute), 225 setConfOpts() (synapse.lib.config.Configable method), save() (synapse.lib.storm.Query method), 270 235 saveconf() (in module synapse.tools.dmon), 330 setDmonConf() (synapse.daemon.DmonConf method), Sched (class in synapse.lib.sched), 263 344 SchedTest (class in synapse.tests.test_lib_sched), 313 setGestDef() (synapse.lib.ingest.IngestApi method), 242 SciMod (class in synapse.models.science), 294 setGestFunc() (synapse.lib.ingest.IngestApi method), 242 Scope (class in synapse.lib.scope), 264 setkw() (in module synapse.lib.storm), 273 scrape() (in module synapse.lib.scrape), 265 setMaxTime() (synapse.lib.cache.Cache method), 228 ScrapeTest (class in synapse.tests.test_lib_scrape), 313 setMesgFunc() (synapse.daemon.Daemon method), 343

Index 415 Synapse Documentation, Release 0.0.34 setModlVers() (synapse.cores.common.Cortex method), 216 211 sizeByGe() (synapse.cores.storage.Storage method), 222 setModlVers() (synapse.cores.storage.Storage method), sizeByLe() (synapse.cores.lmdb.LmdbStorage method), 222 214 setOnMiss() (synapse.lib.cache.Cache method), 228 sizeByLe() (synapse.cores.ram.RamStorage method), 215 setOperFunc() (synapse.lib.storm.Runtime method), 272 sizeByLe() (synapse.cores.sqlite.SqliteStorage method), setOpt() (synapse.lib.storm.Query method), 270 216 setProcName() (in module sizeByLe() (synapse.cores.storage.Storage method), 222 synapse.lib.platforms.common), 224 sizeByLt() (synapse.cores.lmdb.LmdbStorage method), setRoleRules() (synapse.cores.common.Cortex method), 214 211 sizeByRange() (synapse.cores.lmdb.LmdbStorage setRowsByIdProp() (synapse.cores.common.Cortex method), 214 method), 212 sizeByRange() (synapse.cores.ram.RamStorage method), setRowsByIdProp() (synapse.cores.storage.Storage 215 method), 222 sizeByRange() (synapse.cores.sqlite.SqliteStorage setSaveFd() (synapse.cores.common.Cortex method), 212 method), 216 setSaveFd() (synapse.cores.storage.Storage method), 222 sizeByRange() (synapse.cores.storage.Storage method), setSynSvcTimeout() (synapse.lib.service.SvcProxy 222 method), 267 skipIfNoInternet() (synapse.lib.iq.SynTest method), 246 setTstEnvars() (synapse.lib.iq.SynTest method), 246 skipLongTest() (synapse.lib.iq.SynTest method), 246 setTufoIval() (synapse.cores.common.Cortex method), slice() (synapse.lib.queue.Queue method), 253 212 slices() (synapse.lib.queue.Queue method), 254 setTufoProp() (synapse.cores.common.Cortex method), snapTufosByDark() (synapse.cores.common.Cortex 212 method), 212 setTufoProps() (synapse.cores.common.Cortex method), snapTufosByDset() (synapse.cores.common.Cortex 212 method), 213 setUp() (synapse.tests.test_lib_remcycle.NyxTest snapTufosByProp() (synapse.cores.common.Cortex method), 312 method), 213 setUp() (synapse.tests.test_lib_tufo.TestTufoProps sockaddr (class in synapse.lib.platforms.windows), 225 method), 317 Socket (class in synapse.lib.socket), 268 setUp() (synapse.tests.test_lib_tufo.TufoCreateTests Socket (class in synapse.links.ssl), 287 method), 317 socketpair() (in module synapse.lib.socket), 270 setUp() (synapse.tests.test_lib_tufo.TufoEqualityTest SocketTest (class in synapse.tests.test_lib_socket), 314 method), 317 sockgzip() (in module synapse.lib.socket), 270 setUp() (synapse.tests.test_tools_superhash.SuperhashTest sorteq() (synapse.lib.iq.SynTest method), 246 method), 329 speed() (synapse.tests.test_telepath.Foo method), 327 setUpClass() (synapse.tests.test_lib_remcycle.HypnosTest splice() (in module synapse.common), 340 class method), 312 splice() (synapse.cores.common.Cortex method), 213 setUserAuth() (synapse.daemon.Daemon method), 343 spliced() (synapse.cores.xact.StoreXact method), 223 setUserRules() (synapse.cores.common.Cortex method), splices() (in module synapse.lib.scrape), 265 212 splices() (synapse.cores.common.Cortex method), 213 share() (synapse.daemon.Daemon method), 343 SqliteStorage (class in synapse.cores.sqlite), 215 ShowHelp (class in synapse.lib.storm), 272 SqlXact (class in synapse.cores.sqlite), 215 SidNotFound, 356 Srv4Type (class in synapse.models.inet), 291 signCertAs() (synapse.lib.certdir.CertDir method), 231 Srv6Type (class in synapse.models.inet), 291 signHostCsr() (synapse.lib.certdir.CertDir method), 231 SshRelay (class in synapse.links.ssh), 287 signUserCsr() (synapse.lib.certdir.CertDir method), 231 SslRelay (class in synapse.links.ssl), 287 size() (synapse.lib.heap.Heap method), 242 SslTest (class in synapse.tests.test_link_ssl), 320 size() (synapse.lib.storm.Query method), 271 StandaloneTestServer (class in sizeByGe() (synapse.cores.lmdb.LmdbStorage method), synapse.tests.test_lib_remcycle), 313 213 StateMachine (class in synapse.statemach), 359 sizeByGe() (synapse.cores.ram.RamStorage method), StateMachTest (class in synapse.tests.test_statemach), 214 326 sizeByGe() (synapse.cores.sqlite.SqliteStorage method), step() (synapse.lib.datapath.DataElem method), 237

416 Index Synapse Documentation, Release 0.0.34

Storage (class in synapse.cores.storage), 216 synapse.cortex (module), 341 StorageTest (class in synapse.tests.test_cortex), 300 synapse.daemon (module), 342 StoreXact (class in synapse.cores.xact), 223 synapse.data (module), 224 StormRunTest (class in synapse.tests.test_runtime), 326 synapse.datamodel (module), 344 StormSyntaxTest (class in synapse.tests.test_lib_syntax), synapse.docker (module), 224 315 synapse.dyndeps (module), 348 StormTest (class in synapse.tests.test_lib_storm), 314 synapse.eventbus (module), 349 stormTufosBy() (synapse.lib.storm.Runtime method), synapse.exc (module), 352 272 synapse.gene (module), 357 StormType (class in synapse.lib.types), 281 synapse.glob (module), 358 StrType (class in synapse.lib.types), 281 synapse.hivemind (module), 358 subCoreFifo() (synapse.cores.common.Cortex method), synapse.lib (module), 286 213 synapse.lib.atomfile (module), 225 subpaths() (in module synapse.lib.filepath), 240 synapse.lib.atomic (module), 226 subprops (synapse.lib.types.DataType attribute), 280 synapse.lib.auth (module), 226 subprops (synapse.models.inet.EmailType attribute), 291 synapse.lib.cache (module), 227 subprops (synapse.models.inet.FqdnType attribute), 291 synapse.lib.certdir (module), 230 subprops (synapse.models.inet.Srv4Type attribute), 291 synapse.lib.cli (module), 231 subprops (synapse.models.inet.Srv6Type attribute), 291 synapse.lib.cmdr (module), 233 subprops (synapse.models.inet.UrlType attribute), 292 synapse.lib.config (module), 233 subprops (synapse.models.infotech.SemverType at- synapse.lib.dark (module), 236 tribute), 292 synapse.lib.datapath (module), 237 subprops (synapse.models.person.Name attribute), 294 synapse.lib.datfile (module), 237 subs() (synapse.lib.types.DataType method), 280 synapse.lib.encoding (module), 237 SuperhashTest (class in synapse.lib.fifo (module), 238 synapse.tests.test_tools_superhash), 329 synapse.lib.filepath (module), 238 SvcBase (class in synapse.lib.service), 265 synapse.lib.hashitem (module), 241 SvcBus (class in synapse.lib.service), 265 synapse.lib.hashset (module), 241 SvcMeth (class in synapse.lib.service), 266 synapse.lib.heap (module), 241 SvcNameMeth (class in synapse.lib.service), 266 synapse.lib.ingest (module), 242 SvcNameProxy (class in synapse.lib.service), 266 synapse.lib.interval (module), 243 SvcProxy (class in synapse.lib.service), 266 synapse.lib.iq (module), 244 SvcTagMeth (class in synapse.lib.service), 267 synapse.lib.mixins (module), 247 SvcTagProxy (class in synapse.lib.service), 267 synapse.lib.module (module), 247 SvcTest (class in synapse.tests.test_lib_service), 313 synapse.lib.modules (module), 249 SwarmRunBase (class in synapse.lib.msgpack (module), 250 synapse.tests.test_swarm_runtime), 326 synapse.lib.openfile (module), 251 SwarmRunTest (class in synapse.lib.output (module), 251 synapse.tests.test_swarm_runtime), 326 synapse.lib.persist (module), 252 synapse (module), 362 synapse.lib.platforms (module), 225 synapse.async (module), 332 synapse.lib.platforms.common (module), 224 synapse.axon (module), 333 synapse.lib.platforms.darwin (module), 224 synapse.cmds (module), 198 synapse.lib.platforms.freebsd (module), 224 synapse.cmds.common (module), 197 synapse.lib.platforms.linux (module), 225 synapse.cmds.cortex (module), 197 synapse.lib.platforms.windows (module), 225 synapse.common (module), 339 synapse.lib.queue (module), 253 synapse.cores (module), 224 synapse.lib.ratelimit (module), 254 synapse.cores.common (module), 198 synapse.lib.reflect (module), 254 synapse.cores.lmdb (module), 213 synapse.lib.remcycle (module), 255 synapse.cores.postgres (module), 214 synapse.lib.sched (module), 263 synapse.cores.ram (module), 214 synapse.lib.scope (module), 264 synapse.cores.sqlite (module), 215 synapse.lib.scrape (module), 265 synapse.cores.storage (module), 216 synapse.lib.service (module), 265 synapse.cores.xact (module), 223 synapse.lib.session (module), 268

Index 417 Synapse Documentation, Release 0.0.34 synapse.lib.socket (module), 268 synapse.models.temporal (module), 295 synapse.lib.storm (module), 270 synapse.reactor (module), 359 synapse.lib.syntax (module), 273 synapse.statemach (module), 359 synapse.lib.tags (module), 274 synapse.swarm (module), 295 synapse.lib.task (module), 275 synapse.swarm.runtime (module), 295 synapse.lib.thishost (module), 276 synapse.telepath (module), 359 synapse.lib.thisplat (module), 276 synapse.tests (module), 329 synapse.lib.threads (module), 276 synapse.tests.common (module), 295 synapse.lib.time (module), 278 synapse.tests.nopmod (module), 296 synapse.lib.trees (module), 278 synapse.tests.test_async (module), 296 synapse.lib.trigger (module), 278 synapse.tests.test_axon (module), 296 synapse.lib.tufo (module), 279 synapse.tests.test_cmds_cortex (module), 297 synapse.lib.types (module), 280 synapse.tests.test_common (module), 298 synapse.lib.urlhelp (module), 284 synapse.tests.test_cortex (module), 298 synapse.lib.version (module), 284 synapse.tests.test_daemon (module), 301 synapse.lib.webapp (module), 285 synapse.tests.test_data (module), 301 synapse.link (module), 358 synapse.tests.test_datamodel (module), 301 synapse.links (module), 287 synapse.tests.test_dyndeps (module), 302 synapse.links.common (module), 286 synapse.tests.test_eventbus (module), 303 synapse.links.local (module), 286 synapse.tests.test_gene (module), 303 synapse.links.ssh (module), 287 synapse.tests.test_glob (module), 304 synapse.links.ssl (module), 287 synapse.tests.test_hivemind (module), 304 synapse.links.tcp (module), 287 synapse.tests.test_lib_atomfile (module), 304 synapse.lookup (module), 288 synapse.tests.test_lib_atomic (module), 305 synapse.lookup.iana (module), 287 synapse.tests.test_lib_auth (module), 305 synapse.lookup.iso3166 (module), 287 synapse.tests.test_lib_cache (module), 305 synapse.lookup.phonenum (module), 288 synapse.tests.test_lib_certdir (module), 305 synapse.mindmeld (module), 359 synapse.tests.test_lib_cli (module), 306 synapse.models (module), 295 synapse.tests.test_lib_config (module), 306 synapse.models.axon (module), 289 synapse.tests.test_lib_datapath (module), 307 synapse.models.biology (module), 289 synapse.tests.test_lib_datfile (module), 307 synapse.models.chemistry (module), 289 synapse.tests.test_lib_encoding (module), 307 synapse.models.crypto (module), 289 synapse.tests.test_lib_fifo (module), 307 synapse.models.dns (module), 289 synapse.tests.test_lib_filepath (module), 307 synapse.models.files (module), 289 synapse.tests.test_lib_hashitem (module), 308 synapse.models.finance (module), 290 synapse.tests.test_lib_heap (module), 308 synapse.models.geopol (module), 290 synapse.tests.test_lib_ingest (module), 308 synapse.models.geospace (module), 290 synapse.tests.test_lib_interval (module), 309 synapse.models.gov (module), 289 synapse.tests.test_lib_iq (module), 309 synapse.models.gov.cn (module), 288 synapse.tests.test_lib_mixins (module), 310 synapse.models.gov.intl (module), 288 synapse.tests.test_lib_module (module), 310 synapse.models.gov.us (module), 288 synapse.tests.test_lib_msgpack (module), 310 synapse.models.inet (module), 290 synapse.tests.test_lib_openfile (module), 310 synapse.models.infotech (module), 292 synapse.tests.test_lib_output (module), 310 synapse.models.language (module), 293 synapse.tests.test_lib_persist (module), 311 synapse.models.material (module), 293 synapse.tests.test_lib_queue (module), 311 synapse.models.media (module), 293 synapse.tests.test_lib_ratelimit (module), 311 synapse.models.mime (module), 293 synapse.tests.test_lib_reflect (module), 311 synapse.models.money (module), 293 synapse.tests.test_lib_remcycle (module), 312 synapse.models.orgs (module), 293 synapse.tests.test_lib_sched (module), 313 synapse.models.person (module), 294 synapse.tests.test_lib_scope (module), 313 synapse.models.science (module), 294 synapse.tests.test_lib_scrape (module), 313 synapse.models.syn (module), 294 synapse.tests.test_lib_service (module), 313 synapse.models.telco (module), 294 synapse.tests.test_lib_session (module), 314

418 Index Synapse Documentation, Release 0.0.34 synapse.tests.test_lib_socket (module), 314 synapse.tools.dumprows (module), 330 synapse.tests.test_lib_storm (module), 314 synapse.tools.easycert (module), 331 synapse.tests.test_lib_syntax (module), 315 synapse.tools.ingest (module), 331 synapse.tests.test_lib_tags (module), 316 synapse.tools.json2mpk (module), 331 synapse.tests.test_lib_task (module), 316 synapse.tools.loadrows (module), 331 synapse.tests.test_lib_thishost (module), 316 synapse.tools.pushfile (module), 331 synapse.tests.test_lib_threads (module), 316 synapse.tools.superhash (module), 331 synapse.tests.test_lib_time (module), 317 sync() (synapse.async.Boss method), 332 synapse.tests.test_lib_trees (module), 317 sync() (synapse.axon.Axon method), 337 synapse.tests.test_lib_trigger (module), 317 sync() (synapse.cores.xact.StoreXact method), 223 synapse.tests.test_lib_tufo (module), 317 sync() (synapse.lib.heap.Heap method), 242 synapse.tests.test_lib_types (module), 318 syncjob() (synapse.telepath.Proxy method), 360 synapse.tests.test_lib_version (module), 319 SynCmdCoreTest (class in synapse.tests.test_lib_webapp (module), 319 synapse.tests.test_cmds_cortex), 297 synapse.tests.test_link (module), 319 syncs() (synapse.lib.heap.Heap method), 242 synapse.tests.test_link_ssl (module), 320 SynErr, 356 synapse.tests.test_lookup_iso3166 (module), 320 synerr() (in module synapse.common), 340 synapse.tests.test_lookup_phonenum (module), 320 SynMod (class in synapse.models.syn), 294 synapse.tests.test_mindmeld (module), 320 SynModelTest (class in synapse.tests.test_model_syn), synapse.tests.test_model_axon (module), 320 325 synapse.tests.test_model_crypto (module), 321 SynTest (class in synapse.lib.iq), 244 synapse.tests.test_model_dns (module), 321 synapse.tests.test_model_files (module), 321 T synapse.tests.test_model_geopol (module), 321 tagged() (in module synapse.lib.tufo), 279 synapse.tests.test_model_gov_cn (module), 322 tags() (in module synapse.lib.tufo), 279 synapse.tests.test_model_gov_us (module), 322 TagTest (class in synapse.tests.test_lib_tags), 316 synapse.tests.test_model_inet (module), 322 TagType (class in synapse.lib.types), 281 synapse.tests.test_model_infotech (module), 323 take() (synapse.lib.storm.Query method), 271 synapse.tests.test_model_language (module), 324 Task (class in synapse.lib.task), 275 synapse.tests.test_model_material (module), 324 task() (synapse.lib.threads.Pool method), 276 synapse.tests.test_model_media (module), 324 TaskTest (class in synapse.tests.test_lib_task), 316 synapse.tests.test_model_orgs (module), 324 TcpRelay (class in synapse.links.tcp), 287 synapse.tests.test_model_person (module), 324 tearDownClass() (synapse.tests.test_lib_remcycle.HypnosTest synapse.tests.test_model_syn (module), 325 class method), 312 synapse.tests.test_model_telco (module), 325 TelcoTest (class in synapse.tests.test_model_telco), 325 synapse.tests.test_model_temporal (module), 325 TeleClientSide, 356 synapse.tests.test_reactor (module), 325 TelePathTest (class in synapse.tests.test_telepath), 327 synapse.tests.test_runtime (module), 326 teleSynAck() (in module synapse.telepath), 362 synapse.tests.test_statemach (module), 326 TelMod (class in synapse.models.telco), 294 synapse.tests.test_swarm_runtime (module), 326 tes_model_type_inet_tcp4() synapse.tests.test_telepath (module), 327 (synapse.tests.test_model_inet.InetModelTest synapse.tests.test_tools_autodoc (module), 327 method), 322 synapse.tests.test_tools_dmon (module), 328 test_aspect_adddel() (synapse.tests.test_lib_tags.TagTest synapse.tests.test_tools_dumprows_loadrows (module), method), 316 328 test_aspect_bytag() (synapse.tests.test_lib_tags.TagTest synapse.tests.test_tools_easycert (module), 328 method), 316 synapse.tests.test_tools_ingest (module), 329 test_aspect_iter_down() (synapse.tests.test_lib_tags.TagTest synapse.tests.test_tools_json2mpk (module), 329 method), 316 synapse.tests.test_tools_pushfile (module), 329 test_aspect_iter_up() (synapse.tests.test_lib_tags.TagTest synapse.tests.test_tools_superhash (module), 329 method), 316 synapse.tools (module), 332 test_async_basics() (synapse.tests.test_async.AsyncTests synapse.tools.autodoc (module), 329 method), 296 synapse.tools.cmdr (module), 330 test_async_custom_pool_basics() synapse.tools.dmon (module), 330 (synapse.tests.test_async.AsyncTests method),

Index 419 Synapse Documentation, Release 0.0.34

296 method), 296 test_async_ondone() (synapse.tests.test_async.AsyncTests test_axon_fs_mkdir() (synapse.tests.test_axon.AxonFSTest method), 296 method), 296 test_async_pool_basics() test_axon_fs_read() (synapse.tests.test_axon.AxonFSTest (synapse.tests.test_async.AsyncTests method), method), 296 296 test_axon_fs_readdir() (synapse.tests.test_axon.AxonFSTest test_async_sugar() (synapse.tests.test_async.AsyncTests method), 296 method), 296 test_axon_fs_rename() (synapse.tests.test_axon.AxonFSTest test_async_sync() (synapse.tests.test_async.AsyncTests method), 296 method), 296 test_axon_fs_rmdir() (synapse.tests.test_axon.AxonFSTest test_async_timeout() (synapse.tests.test_async.AsyncTests method), 296 method), 296 test_axon_fs_truncate() (synapse.tests.test_axon.AxonFSTest test_async_wait_syntimeout() method), 296 (synapse.tests.test_async.AsyncTests method), test_axon_fs_unlink() (synapse.tests.test_axon.AxonFSTest 296 method), 296 test_async_wait_timeout() test_axon_fs_utimens() (synapse.tests.test_axon.AxonFSTest (synapse.tests.test_async.AsyncTests method), method), 296 296 test_axon_get_renameprops() test_atomfile_base() (synapse.tests.test_lib_atomfile.AtomTest (synapse.tests.test_axon.AxonFSTest method), method), 304 296 test_atomfile_get() (synapse.tests.test_lib_atomfile.AtomTesttest_axon_host() (synapse.tests.test_axon.AxonHostTest method), 304 method), 297 test_atomfile_pread() (synapse.tests.test_lib_atomfile.AtomTesttest_axon_host_clone() (synapse.tests.test_axon.AxonHostTest method), 304 method), 297 test_atomic_xchg() (synapse.tests.test_lib_atomic.AtomTesttest_axon_host_free_limit() method), 305 (synapse.tests.test_axon.AxonHostTest test_auth_rules() (synapse.tests.test_lib_auth.AuthTest method), 297 method), 305 test_axon_host_maxsize_limit() test_auth_runas() (synapse.tests.test_lib_auth.AuthTest (synapse.tests.test_axon.AxonHostTest method), 305 method), 297 test_axon__fs_isdir() (synapse.tests.test_axon.AxonFSTest test_axon_host_spinbackup() method), 296 (synapse.tests.test_axon.AxonHostTest test_axon__fs_isfile() (synapse.tests.test_axon.AxonFSTest method), 297 method), 296 test_axon_restrictions() (synapse.tests.test_axon.AxonTest test_axon_autorun() (synapse.tests.test_axon.AxonHostTest method), 297 method), 297 test_axon_telepath() (synapse.tests.test_axon.AxonTest test_axon_basics() (synapse.tests.test_axon.AxonTest method), 297 method), 297 test_axonblob() (synapse.tests.test_model_axon.AxonModelTest test_axon_bytesize() (synapse.tests.test_axon.AxonTest method), 320 method), 297 test_axonpath() (synapse.tests.test_model_axon.AxonModelTest test_axon_clone_large() (synapse.tests.test_axon.AxonHostTest method), 320 method), 297 test_bad_file() (synapse.tests.test_tools_superhash.SuperhashTest test_axon_cluster() (synapse.tests.test_axon.AxonClusterTest method), 329 method), 296 test_blob_dump() (synapse.tests.test_tools_dumprows_loadrows.DumpRowsTest test_axon_cluster_cortex() method), 328 (synapse.tests.test_axon.AxonClusterTest test_cache_clearing() (synapse.tests.test_lib_cache.CacheTest method), 296 method), 305 test_axon_eatbytes() (synapse.tests.test_axon.AxonTest test_cache_defval() (synapse.tests.test_lib_cache.CacheTest method), 297 method), 305 test_axon_fs_create() (synapse.tests.test_axon.AxonFSTest test_cache_fini() (synapse.tests.test_lib_cache.CacheTest method), 296 method), 305 test_axon_fs_getattr() (synapse.tests.test_axon.AxonFSTest test_cache_fixed() (synapse.tests.test_lib_cache.CacheTest method), 296 method), 305 test_axon_fs_getxattr() (synapse.tests.test_axon.AxonFSTesttest_cache_magic() (synapse.tests.test_lib_cache.CacheTest

420 Index Synapse Documentation, Release 0.0.34

method), 305 method), 297 test_cache_match() (synapse.tests.test_lib_cache.CacheTesttest_cmds_ask_mutual_exclusive() method), 305 (synapse.tests.test_cmds_cortex.SynCmdCoreTest test_cache_miss() (synapse.tests.test_lib_cache.CacheTest method), 297 method), 305 test_cmds_ask_noopts() (synapse.tests.test_cmds_cortex.SynCmdCoreTest test_cache_set_maxtime() method), 297 (synapse.tests.test_lib_cache.CacheTest test_cmds_ask_null_response() method), 305 (synapse.tests.test_cmds_cortex.SynCmdCoreTest test_cache_timeout() (synapse.tests.test_lib_cache.CacheTest method), 297 method), 305 test_cmds_ask_props() (synapse.tests.test_cmds_cortex.SynCmdCoreTest test_cache_tufo() (synapse.tests.test_lib_cache.CacheTest method), 297 method), 305 test_cmds_ask_raw() (synapse.tests.test_cmds_cortex.SynCmdCoreTest test_cache_tufo_prop() (synapse.tests.test_lib_cache.CacheTest method), 297 method), 305 test_cmds_ask_tagtime() test_certdir_ca() (synapse.tests.test_lib_certdir.CertDirTest (synapse.tests.test_cmds_cortex.SynCmdCoreTest method), 305 method), 297 test_certdir_host() (synapse.tests.test_lib_certdir.CertDirTesttest_cmds_guid() (synapse.tests.test_cmds_cortex.SynCmdCoreTest method), 305 method), 297 test_certdir_hostca() (synapse.tests.test_lib_certdir.CertDirTesttest_cmds_help() (synapse.tests.test_cmds_cortex.SynCmdCoreTest method), 305 method), 297 test_certdir_hostcsr() (synapse.tests.test_lib_certdir.CertDirTesttest_cmds_py() (synapse.tests.test_cmds_cortex.SynCmdCoreTest method), 306 method), 297 test_certdir_user() (synapse.tests.test_lib_certdir.CertDirTesttest_cmds_quit() (synapse.tests.test_cmds_cortex.SynCmdCoreTest method), 306 method), 297 test_certdir_userca() (synapse.tests.test_lib_certdir.CertDirTesttest_common_genfile() (synapse.tests.test_common.CommonTest method), 306 method), 298 test_cli_cmd() (synapse.tests.test_lib_cli.CliTest test_common_guid() (synapse.tests.test_common.CommonTest method), 306 method), 298 test_cli_cmd_loop_quit() test_common_intify() (synapse.tests.test_common.CommonTest (synapse.tests.test_lib_cli.CliTest method), method), 298 306 test_common_isguid() (synapse.tests.test_common.CommonTest test_cli_cmdret() (synapse.tests.test_lib_cli.CliTest method), 298 method), 306 test_common_vertup() (synapse.tests.test_common.CommonTest test_cli_help() (synapse.tests.test_lib_cli.CliTest method), 298 method), 306 test_compat_canstor() (synapse.tests.test_common.CommonTest test_cli_opts_defval() (synapse.tests.test_lib_cli.CliTest method), 298 method), 306 test_conf_asloc() (synapse.tests.test_lib_config.ConfTest test_cli_opts_flag() (synapse.tests.test_lib_cli.CliTest method), 306 method), 306 test_conf_base() (synapse.tests.test_lib_config.ConfTest test_cli_opts_glob() (synapse.tests.test_lib_cli.CliTest method), 306 method), 306 test_conf_defval() (synapse.tests.test_lib_config.ConfTest test_cli_opts_list() (synapse.tests.test_lib_cli.CliTest method), 306 method), 306 test_confdef_decorator() (synapse.tests.test_lib_config.ConfTest test_cli_quit() (synapse.tests.test_lib_cli.CliTest method), method), 306 306 test_configable_proxymethod() test_cmds_ask() (synapse.tests.test_cmds_cortex.SynCmdCoreTest (synapse.tests.test_lib_config.ConfTest method), 297 method), 306 test_cmds_ask_debug() (synapse.tests.test_cmds_cortex.SynCmdCoreTesttest_cortex_addmodel() (synapse.tests.test_cortex.CortexTest method), 297 method), 299 test_cmds_ask_exc_response() test_cortex_auth() (synapse.tests.test_cortex.CortexTest (synapse.tests.test_cmds_cortex.SynCmdCoreTest method), 299 method), 297 test_cortex_by_type() (synapse.tests.test_cortex.CortexTest test_cmds_ask_multilift() method), 299 (synapse.tests.test_cmds_cortex.SynCmdCoreTesttest_cortex_bytype() (synapse.tests.test_cortex.CortexTest

Index 421 Synapse Documentation, Release 0.0.34

method), 299 test_cortex_getbytag() (synapse.tests.test_cortex.CortexTest test_cortex_caching() (synapse.tests.test_cortex.CortexTest method), 299 method), 299 test_cortex_ingest() (synapse.tests.test_cortex.CortexTest test_cortex_caching_add_tufo() method), 299 (synapse.tests.test_cortex.CortexTest method), test_cortex_isnew() (synapse.tests.test_cortex.CortexTest 299 method), 299 test_cortex_caching_atlimit() test_cortex_lift_by_cidr() (synapse.tests.test_cortex.CortexTest method), (synapse.tests.test_cortex.CortexTest method), 299 299 test_cortex_caching_del_tufo() test_cortex_lmdb() (synapse.tests.test_cortex.CortexBaseTest (synapse.tests.test_cortex.CortexTest method), method), 299 299 test_cortex_local() (synapse.tests.test_cortex.CortexTest test_cortex_caching_del_tufo_prop() method), 299 (synapse.tests.test_cortex.CortexTest method), test_cortex_minmax() (synapse.tests.test_cortex.CortexTest 299 method), 299 test_cortex_caching_disable() test_cortex_minmax_epoch() (synapse.tests.test_cortex.CortexTest method), (synapse.tests.test_cortex.CortexTest method), 299 300 test_cortex_caching_new() test_cortex_modlrevs() (synapse.tests.test_cortex.CortexTest (synapse.tests.test_cortex.CortexTest method), method), 300 299 test_cortex_modlvers() (synapse.tests.test_cortex.CortexTest test_cortex_caching_oneref() method), 300 (synapse.tests.test_cortex.CortexTest method), test_cortex_module() (synapse.tests.test_cortex.CortexTest 299 method), 300 test_cortex_caching_set() test_cortex_module_datamodel_migration() (synapse.tests.test_cortex.CortexTest method), (synapse.tests.test_cortex.CortexTest method), 299 300 test_cortex_caching_tags() test_cortex_module_datamodel_migration_persistent() (synapse.tests.test_cortex.CortexTest method), (synapse.tests.test_cortex.CortexTest method), 299 300 test_cortex_caching_under_limit() test_cortex_norm_fail() (synapse.tests.test_cortex.CortexTest (synapse.tests.test_cortex.CortexTest method), method), 300 299 test_cortex_postgres() (synapse.tests.test_cortex.CortexBaseTest test_cortex_choptag() (synapse.tests.test_cortex.CortexTest method), 299 method), 299 test_cortex_ram() (synapse.tests.test_cortex.CortexBaseTest test_cortex_comp() (synapse.tests.test_cortex.CortexTest method), 299 method), 299 test_cortex_ramhost() (synapse.tests.test_cortex.CortexTest test_cortex_datamodel_runt_consistency() method), 300 (synapse.tests.test_cortex.CortexTest method), test_cortex_reqprops() (synapse.tests.test_cortex.CortexTest 299 method), 300 test_cortex_dict() (synapse.tests.test_cortex.CortexTest test_cortex_reqstor() (synapse.tests.test_cortex.CortexTest method), 299 method), 300 test_cortex_enforce() (synapse.tests.test_cortex.CortexTest test_cortex_runts() (synapse.tests.test_cortex.CortexTest method), 299 method), 300 test_cortex_ﬁfos() (synapse.tests.test_cortex.CortexTest test_cortex_savefd() (synapse.tests.test_cortex.CortexTest method), 299 method), 300 test_cortex_ﬁre_set() (synapse.tests.test_cortex.CortexTest test_cortex_seed() (synapse.tests.test_cortex.CortexTest method), 299 method), 300 test_cortex_formtufobytufo() test_cortex_seq() (synapse.tests.test_cortex.CortexTest (synapse.tests.test_cortex.CortexTest method), method), 300 299 test_cortex_splice_propdel() test_cortex_formtufosbyprops() (synapse.tests.test_cortex.CortexTest method), (synapse.tests.test_cortex.CortexTest method), 300 299 test_cortex_splicefd() (synapse.tests.test_cortex.CortexTest

422 Index Synapse Documentation, Release 0.0.34

method), 300 method), 301 test_cortex_splicepump() test_daemon_ctor_nonurl() (synapse.tests.test_cortex.CortexTest method), (synapse.tests.test_daemon.DaemonTest 300 method), 301 test_cortex_splices() (synapse.tests.test_cortex.CortexTest test_daemon_error() (synapse.tests.test_daemon.DaemonTest method), 300 method), 301 test_cortex_splices_errs() test_daemon_on() (synapse.tests.test_daemon.DaemonTest (synapse.tests.test_cortex.CortexTest method), method), 301 300 test_daemon_timeout() (synapse.tests.test_daemon.DaemonTest test_cortex_sqlite3() (synapse.tests.test_cortex.CortexBaseTest method), 301 method), 299 test_data_iana_tlds() (synapse.tests.test_data.DataTest test_cortex_stats() (synapse.tests.test_cortex.CortexTest method), 301 method), 300 test_datamodel_badprop() test_cortex_tag_ival() (synapse.tests.test_cortex.CortexTest (synapse.tests.test_datamodel.DataModelTest method), 300 method), 301 test_cortex_tagform() (synapse.tests.test_cortex.CortexTest test_datamodel_bool() (synapse.tests.test_datamodel.DataModelTest method), 300 method), 301 test_cortex_tags() (synapse.tests.test_cortex.CortexTest test_datamodel_cast_json() method), 300 (synapse.tests.test_datamodel.DataModelTest test_cortex_trigger() (synapse.tests.test_cortex.CortexTest method), 302 method), 300 test_datamodel_cortex() (synapse.tests.test_datamodel.DataModelTest test_cortex_tufo_del() (synapse.tests.test_cortex.CortexTest method), 302 method), 300 test_datamodel_fail_noprop() test_cortex_tufo_list() (synapse.tests.test_cortex.CortexTest (synapse.tests.test_datamodel.DataModelTest method), 300 method), 302 test_cortex_tufo_pop() (synapse.tests.test_cortex.CortexTesttest_datamodel_fail_notype() method), 300 (synapse.tests.test_datamodel.DataModelTest test_cortex_tufo_setprop() method), 302 (synapse.tests.test_cortex.CortexTest method), test_datamodel_filebase() 300 (synapse.tests.test_datamodel.DataModelTest test_cortex_tufo_setprops() method), 302 (synapse.tests.test_cortex.CortexTest method), test_datamodel_filepath() 300 (synapse.tests.test_datamodel.DataModelTest test_cortex_tufo_tag() (synapse.tests.test_cortex.CortexTest method), 302 method), 300 test_datamodel_formbase() test_cortex_universal_props() (synapse.tests.test_datamodel.DataModelTest (synapse.tests.test_cortex.CortexTest method), method), 302 300 test_datamodel_forms() (synapse.tests.test_datamodel.DataModelTest test_cortex_xact_deadlock() method), 302 (synapse.tests.test_cortex.CortexTest method), test_datamodel_getPropDef() 300 (synapse.tests.test_datamodel.DataModelTest test_daemon_conf() (synapse.tests.test_daemon.DaemonTest method), 302 method), 301 test_datamodel_getPropInfo() test_daemon_conf_fork() (synapse.tests.test_datamodel.DataModelTest (synapse.tests.test_daemon.DaemonTest method), 302 method), 301 test_datamodel_glob() (synapse.tests.test_datamodel.DataModelTest test_daemon_conf_onfini() method), 302 (synapse.tests.test_daemon.DaemonTest test_datamodel_hash() (synapse.tests.test_datamodel.DataModelTest method), 301 method), 302 test_daemon_ctor_config() test_datamodel_inet() (synapse.tests.test_datamodel.DataModelTest (synapse.tests.test_daemon.DaemonTest method), 302 method), 301 test_datamodel_istufoform() test_daemon_ctor_dmonurl() (synapse.tests.test_datamodel.DataModelTest (synapse.tests.test_daemon.DaemonTest method), 302

Index 423 Synapse Documentation, Release 0.0.34 test_datamodel_parsetypes() method), 318 (synapse.tests.test_datamodel.DataModelTest test_datatype_syn_prop() method), 302 (synapse.tests.test_datamodel.DataModelTest test_datamodel_reqpropnorm() method), 302 (synapse.tests.test_datamodel.DataModelTest test_datatype_syn_tag() (synapse.tests.test_datamodel.DataModelTest method), 302 method), 302 test_datamodel_subs() (synapse.tests.test_datamodel.DataModelTesttest_datﬁle_basic() (synapse.tests.test_lib_datﬁle.DatFileTest method), 302 method), 307 test_datamodel_time() (synapse.tests.test_datamodel.DataModelTesttest_default_props() (synapse.tests.test_lib_tufo.TestTufoProps method), 302 method), 317 test_datamodel_typefns() test_dump_force() (synapse.tests.test_tools_dumprows_loadrows.DumpRowsTest (synapse.tests.test_datamodel.DataModelTest method), 328 method), 302 test_dump_largecore() (synapse.tests.test_tools_dumprows_loadrows.DumpRowsTest test_datamodel_types() (synapse.tests.test_datamodel.DataModelTest method), 328 method), 302 test_dumprows_load() (synapse.tests.test_tools_dumprows_loadrows.LoadRowsTest test_datapath_iter() (synapse.tests.test_lib_datapath.DataPathTest method), 328 method), 307 test_dumprows_load_compressed() test_datapath_parent() (synapse.tests.test_lib_datapath.DataPathTest (synapse.tests.test_tools_dumprows_loadrows.LoadRowsTest method), 307 method), 328 test_datapath_quoted() (synapse.tests.test_lib_datapath.DataPathTesttest_dyndeps_alias() (synapse.tests.test_dyndeps.DynDepsTest method), 307 method), 302 test_datapath_self() (synapse.tests.test_lib_datapath.DataPathTesttest_dyndeps_dynloc() (synapse.tests.test_dyndeps.DynDepsTest method), 307 method), 302 test_datapath_valu() (synapse.tests.test_lib_datapath.DataPathTesttest_dyndeps_dynmod() (synapse.tests.test_dyndeps.DynDepsTest method), 307 method), 302 test_datatype_basics() (synapse.tests.test_lib_types.DataTypesTesttest_dyndeps_dyntask() (synapse.tests.test_dyndeps.DynDepsTest method), 318 method), 302 test_datatype_bool() (synapse.tests.test_lib_types.DataTypesTesttest_dyndeps_eval() (synapse.tests.test_dyndeps.DynDepsTest method), 318 method), 302 test_datatype_dup() (synapse.tests.test_lib_types.DataTypesTesttest_dyndeps_expected_aliases() method), 318 (synapse.tests.test_dyndeps.DynDepsTest test_datatype_fqdn() (synapse.tests.test_lib_types.DataTypesTest method), 302 method), 318 test_dyndeps_meth() (synapse.tests.test_dyndeps.DynDepsTest test_datatype_guid() (synapse.tests.test_lib_types.DataTypesTest method), 302 method), 318 test_dyndeps_nosuchdyn() test_datatype_guid_sub() (synapse.tests.test_dyndeps.DynDepsTest (synapse.tests.test_lib_types.DataTypesTest method), 302 method), 318 test_easycert_csr() (synapse.tests.test_tools_easycert.TestEasyCert test_datatype_hash_md5() method), 328 (synapse.tests.test_lib_types.DataTypesTest test_easycert_server_sign() method), 318 (synapse.tests.test_tools_easycert.TestEasyCert test_datatype_int() (synapse.tests.test_lib_types.DataTypesTest method), 328 method), 318 test_easycert_user_sign() test_datatype_int_minmax() (synapse.tests.test_tools_easycert.TestEasyCert (synapse.tests.test_lib_types.DataTypesTest method), 328 method), 318 test_equals() (synapse.tests.test_lib_tufo.TufoEqualityTest test_datatype_int_repr() (synapse.tests.test_lib_types.DataTypesTest method), 317 method), 318 test_evenbus_unlink() (synapse.tests.test_eventbus.EventBusTest test_datatype_str() (synapse.tests.test_lib_types.DataTypesTest method), 303 method), 318 test_eventbus_basics() (synapse.tests.test_eventbus.EventBusTest test_datatype_str_enums() method), 303 (synapse.tests.test_lib_types.DataTypesTest test_eventbus_busref() (synapse.tests.test_eventbus.EventBusTest method), 318 method), 303 test_datatype_str_hex() (synapse.tests.test_lib_types.DataTypesTesttest_eventbus_busref_gen()

424 Index Synapse Documentation, Release 0.0.34

(synapse.tests.test_eventbus.EventBusTest method), 307 method), 303 test_form_rsakey() (synapse.tests.test_model_crypto.CryptoModelTest test_eventbus_consume() method), 321 (synapse.tests.test_eventbus.EventBusTest test_gene_bool() (synapse.tests.test_gene.GeneTest method), 303 method), 303 test_eventbus_exc() (synapse.tests.test_eventbus.EventBusTesttest_gene_borkcall() (synapse.tests.test_gene.GeneTest method), 303 method), 303 test_eventbus_filt() (synapse.tests.test_eventbus.EventBusTesttest_gene_borkparn() (synapse.tests.test_gene.GeneTest method), 303 method), 303 test_eventbus_finionce() (synapse.tests.test_eventbus.EventBusTesttest_gene_call() (synapse.tests.test_gene.GeneTest method), 303 method), 303 test_eventbus_link() (synapse.tests.test_eventbus.EventBusTesttest_gene_call_noargs() (synapse.tests.test_gene.GeneTest method), 303 method), 304 test_eventbus_log() (synapse.tests.test_eventbus.EventBusTesttest_gene_callcall() (synapse.tests.test_gene.GeneTest method), 303 method), 304 test_eventbus_main_sigint() test_gene_comp() (synapse.tests.test_gene.GeneTest (synapse.tests.test_eventbus.EventBusTest method), 304 method), 303 test_gene_empty_value() test_eventbus_main_sigterm() (synapse.tests.test_gene.GeneTest method), (synapse.tests.test_eventbus.EventBusTest 304 method), 303 test_gene_eval() (synapse.tests.test_gene.GeneTest test_eventbus_off() (synapse.tests.test_eventbus.EventBusTest method), 304 method), 303 test_gene_hangcall() (synapse.tests.test_gene.GeneTest test_eventbus_refcount() (synapse.tests.test_eventbus.EventBusTest method), 304 method), 303 test_gene_lab() (synapse.tests.test_gene.GeneTest test_eventbus_waiter() (synapse.tests.test_eventbus.EventBusTest method), 304 method), 303 test_gene_landlor() (synapse.tests.test_gene.GeneTest test_eventbus_waitfini() (synapse.tests.test_eventbus.EventBusTest method), 304 method), 303 test_gene_newpparn() (synapse.tests.test_gene.GeneTest test_eventbus_withfini() (synapse.tests.test_eventbus.EventBusTest method), 304 method), 303 test_gene_noname() (synapse.tests.test_gene.GeneTest test_fifo_ack_neg1() (synapse.tests.test_lib_fifo.FifoTest method), 304 method), 307 test_gene_notoks() (synapse.tests.test_gene.GeneTest test_fifo_fifo() (synapse.tests.test_lib_fifo.FifoTest method), 304 method), 307 test_gene_null() (synapse.tests.test_gene.GeneTest test_fifo_flush() (synapse.tests.test_lib_fifo.FifoTest method), 304 method), 307 test_gene_pow() (synapse.tests.test_gene.GeneTest test_fifo_nack_past() (synapse.tests.test_lib_fifo.FifoTest method), 304 method), 307 test_gene_prec() (synapse.tests.test_gene.GeneTest test_filebase() (synapse.tests.test_model_files.FileModelTest method), 304 method), 321 test_gene_shift() (synapse.tests.test_gene.GeneTest test_filepath() (synapse.tests.test_model_files.FileModelTest method), 304 method), 321 test_getArgParser_bools() test_filepath_glob() (synapse.tests.test_lib_filepath.TestFilePath (synapse.tests.test_tools_dmon.TestArgParser method), 307 method), 328 test_filepath_regular() (synapse.tests.test_lib_filepath.TestFilePathtest_getArgParser_logLevel() method), 307 (synapse.tests.test_tools_dmon.TestArgParser test_filepath_tar() (synapse.tests.test_lib_filepath.TestFilePath method), 328 method), 307 test_getArgParser_logLevel_exception() test_filepath_tar_open() (synapse.tests.test_lib_filepath.TestFilePath (synapse.tests.test_tools_dmon.TestArgParser method), 307 method), 328 test_filepath_zip() (synapse.tests.test_lib_filepath.TestFilePathtest_glob_call() (synapse.tests.test_glob.GlobTest method), 307 method), 304 test_filepath_zip_open() (synapse.tests.test_lib_filepath.TestFilePathtest_glob_insec() (synapse.tests.test_glob.GlobTest

Index 425 Synapse Documentation, Release 0.0.34

method), 304 (synapse.tests.test_lib_remcycle.HypnosTest test_glob_persec() (synapse.tests.test_glob.GlobTest method), 312 method), 304 test_hypnos_post_byts() (synapse.tests.test_lib_remcycle.HypnosTest test_glob_task() (synapse.tests.test_glob.GlobTest method), 312 method), 304 test_hypnos_simple_fail() test_heap_base() (synapse.tests.test_lib_heap.HeapTest (synapse.tests.test_lib_remcycle.HypnosTest method), 308 method), 312 test_heap_readiter() (synapse.tests.test_lib_heap.HeapTest test_hypnos_with_telepath() method), 308 (synapse.tests.test_lib_remcycle.HypnosTest test_heap_resize() (synapse.tests.test_lib_heap.HeapTest method), 312 method), 308 test_ingest() (synapse.tests.test_tools_superhash.SuperhashTest test_heap_save() (synapse.tests.test_lib_heap.HeapTest method), 329 method), 308 test_ingest_addFormat() (synapse.tests.test_lib_ingest.IngTest test_hypnos_automatic_ingest() method), 308 (synapse.tests.test_lib_remcycle.HypnosTest test_ingest_basic() (synapse.tests.test_lib_ingest.IngTest method), 312 method), 308 test_hypnos_cache_job() test_ingest_basic_bufio() (synapse.tests.test_lib_remcycle.HypnosTest (synapse.tests.test_lib_ingest.IngTest method), method), 312 308 test_hypnos_cache_with_failure() test_ingest_cast() (synapse.tests.test_lib_ingest.IngTest (synapse.tests.test_lib_remcycle.HypnosTest method), 308 method), 312 test_ingest_condform() (synapse.tests.test_lib_ingest.IngTest test_hypnos_cache_with_ingest() method), 308 (synapse.tests.test_lib_remcycle.HypnosTest test_ingest_condform_with_missing_var() method), 312 (synapse.tests.test_lib_ingest.IngTest method), test_hypnos_callback_ondone() 308 (synapse.tests.test_lib_remcycle.HypnosTest test_ingest_condtag() (synapse.tests.test_lib_ingest.IngTest method), 312 method), 308 test_hypnos_config_bounds() test_ingest_cortex_registration() (synapse.tests.test_lib_remcycle.HypnosTest (synapse.tests.test_lib_ingest.IngTest method), method), 312 308 test_hypnos_config_register_deregister() test_ingest_csv() (synapse.tests.test_lib_ingest.IngTest (synapse.tests.test_lib_remcycle.HypnosTest method), 308 method), 312 test_ingest_embed_nodes() test_hypnos_content_type_skips() (synapse.tests.test_lib_ingest.IngTest method), (synapse.tests.test_lib_remcycle.HypnosTest 308 method), 312 test_ingest_embed_pernode_tagsprops() test_hypnos_default_callback() (synapse.tests.test_lib_ingest.IngTest method), (synapse.tests.test_lib_remcycle.HypnosTest 308 method), 312 test_ingest_embed_props() test_hypnos_default_callback_null() (synapse.tests.test_lib_ingest.IngTest method), (synapse.tests.test_lib_remcycle.HypnosTest 308 method), 312 test_ingest_embed_tags() test_hypnos_fini() (synapse.tests.test_lib_remcycle.HypnosTest (synapse.tests.test_lib_ingest.IngTest method), method), 312 308 test_hypnos_fini_core() (synapse.tests.test_lib_remcycle.HypnosTesttest_ingest_files() (synapse.tests.test_lib_ingest.IngTest method), 312 method), 308 test_hypnos_fire_api_callback() test_ingest_func() (synapse.tests.test_lib_ingest.IngTest (synapse.tests.test_lib_remcycle.HypnosTest method), 308 method), 312 test_ingest_gen_guid() (synapse.tests.test_lib_ingest.IngTest test_hypnos_make_body() method), 308 (synapse.tests.test_lib_remcycle.NyxTest test_ingest_iter_object() (synapse.tests.test_lib_ingest.IngTest method), 312 method), 308 test_hypnos_manual_ingest_via_eventbus() test_ingest_iter_objectish_array()

426 Index Synapse Documentation, Release 0.0.34

(synapse.tests.test_lib_ingest.IngTest method), test_ival_fold() (synapse.tests.test_lib_interval.IvalTest 308 method), 309 test_ingest_iterdata() (synapse.tests.test_lib_ingest.IngTest test_ival_overlap() (synapse.tests.test_lib_interval.IvalTest method), 308 method), 309 test_ingest_iteriter() (synapse.tests.test_lib_ingest.IngTest test_ival_parsetime() (synapse.tests.test_lib_interval.IvalTest method), 308 method), 309 test_ingest_json() (synapse.tests.test_lib_ingest.IngTest test_keycache_lookup() (synapse.tests.test_lib_cache.CacheTest method), 308 method), 305 test_ingest_jsoncast() (synapse.tests.test_lib_ingest.IngTest test_kwargs_tufo_creation() method), 309 (synapse.tests.test_lib_tufo.TufoCreateTests test_ingest_jsonl() (synapse.tests.test_lib_ingest.IngTest method), 317 method), 309 test_lib_config_req() (synapse.tests.test_lib_config.ConfTest test_ingest_lines() (synapse.tests.test_lib_ingest.IngTest method), 306 method), 309 test_lib_encoding_de() (synapse.tests.test_lib_encoding.EncTest test_ingest_multiple_files() method), 307 (synapse.tests.test_tools_superhash.SuperhashTesttest_lib_encoding_en() (synapse.tests.test_lib_encoding.EncTest method), 329 method), 307 test_ingest_pivot() (synapse.tests.test_lib_ingest.IngTest test_lib_hashitem() (synapse.tests.test_lib_hashitem.HashItemTest method), 309 method), 308 test_ingest_reqprops() (synapse.tests.test_lib_ingest.IngTesttest_lib_mixins_basics() (synapse.tests.test_lib_mixins.MixTest method), 309 method), 310 test_ingest_savevar() (synapse.tests.test_lib_ingest.IngTest test_lib_module_modname() method), 309 (synapse.tests.test_lib_module.CoreModTest test_ingest_tag_template_whif() method), 310 (synapse.tests.test_lib_ingest.IngTest method), test_lib_module_modpath() 309 (synapse.tests.test_lib_module.CoreModTest test_ingest_taginfo() (synapse.tests.test_lib_ingest.IngTest method), 310 method), 309 test_lib_scope() (synapse.tests.test_lib_scope.GeneTest test_ingest_tagiter() (synapse.tests.test_lib_ingest.IngTest method), 313 method), 309 test_lib_scope_enter() (synapse.tests.test_lib_scope.GeneTest test_ingest_template() (synapse.tests.test_lib_ingest.IngTest method), 313 method), 309 test_lib_scope_get_defval() test_ingest_varprop() (synapse.tests.test_lib_ingest.IngTest (synapse.tests.test_lib_scope.GeneTest method), 309 method), 313 test_ingest_xml() (synapse.tests.test_lib_ingest.IngTest test_lib_session_cura() (synapse.tests.test_lib_session.SessTest method), 309 method), 314 test_ingest_xml_search() test_lib_tree_interval() (synapse.tests.test_lib_trees.TreeTest (synapse.tests.test_lib_ingest.IngTest method), method), 317 309 test_limit_behavior() (synapse.tests.test_lib_storm.LimitTest test_ingest_xref() (synapse.tests.test_lib_ingest.IngTest method), 314 method), 309 test_limit_behavior_negatives() test_iq_outp() (synapse.tests.test_lib_iq.IqTest method), (synapse.tests.test_lib_storm.LimitTest 309 method), 314 test_iq_syntest_envars() (synapse.tests.test_lib_iq.IqTest test_limit_default() (synapse.tests.test_lib_storm.LimitTest method), 309 method), 314 test_iq_syntest_helpers() (synapse.tests.test_lib_iq.IqTest test_link_chopurl() (synapse.tests.test_link.LinkTest method), 309 method), 319 test_iq_syntest_logstream() test_link_fromurl() (synapse.tests.test_link.LinkTest (synapse.tests.test_lib_iq.IqTest method), method), 320 309 test_link_invalid() (synapse.tests.test_link.LinkTest test_iq_syntest_psql() (synapse.tests.test_lib_iq.IqTest method), 320 method), 309 test_link_local() (synapse.tests.test_link.LinkTest test_iq_testenv() (synapse.tests.test_lib_iq.IqTest method), 320 method), 309 test_link_pool() (synapse.tests.test_link.LinkTest

Index 427 Synapse Documentation, Release 0.0.34

method), 320 method), 321 test_link_refused() (synapse.tests.test_link.LinkTest test_model_fqdn_punycode() method), 320 (synapse.tests.test_model_inet.InetModelTest test_link_ssl_auth() (synapse.tests.test_link.LinkTest method), 322 method), 320 test_model_geopol_country() test_link_ssl_basic() (synapse.tests.test_link.LinkTest (synapse.tests.test_model_geopol.GeopolModelTest method), 320 method), 321 test_link_ssl_nocheck() (synapse.tests.test_link.LinkTest test_model_geopol_iso2() method), 320 (synapse.tests.test_model_geopol.GeopolModelTest test_link_ssl_nofile() (synapse.tests.test_link.LinkTest method), 321 method), 320 test_model_geopol_iso3() test_lookup_countries() (synapse.tests.test_lookup_iso3166.CountryLookTest(synapse.tests.test_model_geopol.GeopolModelTest method), 320 method), 321 test_lookup_phonenum() test_model_geopol_isonum() (synapse.tests.test_lookup_phonenum.PhLookTest (synapse.tests.test_model_geopol.GeopolModelTest method), 320 method), 321 test_main_lsboot() (synapse.tests.test_tools_dmon.TestMaintest_model_inet_201706121318() method), 328 (synapse.tests.test_model_inet.InetModelTest test_main_onboot() (synapse.tests.test_tools_dmon.TestMain method), 322 method), 328 test_model_inet_201706201837() test_model_crypto_201708231712() (synapse.tests.test_model_inet.InetModelTest (synapse.tests.test_model_crypto.CryptoModelTest method), 322 method), 321 test_model_inet_201709181501() test_model_dns_a() (synapse.tests.test_model_dns.DnsModelTest (synapse.tests.test_model_inet.InetModelTest method), 321 method), 322 test_model_dns_aaaa() (synapse.tests.test_model_dns.DnsModelTesttest_model_inet_201709271521() method), 321 (synapse.tests.test_model_inet.InetModelTest test_model_dns_look() (synapse.tests.test_model_dns.DnsModelTest method), 322 method), 321 test_model_inet_201710111553() test_model_dns_ns() (synapse.tests.test_model_dns.DnsModelTest (synapse.tests.test_model_inet.InetModelTest method), 321 method), 322 test_model_dns_req() (synapse.tests.test_model_dns.DnsModelTesttest_model_inet_addr() (synapse.tests.test_model_inet.InetModelTest method), 321 method), 322 test_model_dns_rev() (synapse.tests.test_model_dns.DnsModelTesttest_model_inet_asnet4() method), 321 (synapse.tests.test_model_inet.InetModelTest test_model_dns_rev6() (synapse.tests.test_model_dns.DnsModelTest method), 322 method), 321 test_model_inet_cast_defang() test_model_file_bytes() (synapse.tests.test_model_files.FileModelTest(synapse.tests.test_model_inet.InetModelTest method), 321 method), 322 test_model_file_bytes_axon() test_model_inet_chprofile() (synapse.tests.test_model_files.FileModelTest (synapse.tests.test_model_inet.InetModelTest method), 321 method), 322 test_model_file_seeds() (synapse.tests.test_model_files.FileModelTesttest_model_inet_cidr4() (synapse.tests.test_model_inet.InetModelTest method), 321 method), 322 test_model_file_seeds_capitalization() test_model_inet_email() (synapse.tests.test_model_inet.InetModelTest (synapse.tests.test_model_files.FileModelTest method), 322 method), 321 test_model_inet_fqdn() (synapse.tests.test_model_inet.InetModelTest test_model_filepath_complex() method), 322 (synapse.tests.test_model_files.FileModelTest test_model_inet_fqdn_set_sfx() method), 321 (synapse.tests.test_model_inet.InetModelTest test_model_files_imgof() method), 322 (synapse.tests.test_model_files.FileModelTest test_model_inet_fqdn_unicode() method), 321 (synapse.tests.test_model_inet.InetModelTest test_model_files_txtref() (synapse.tests.test_model_files.FileModelTestmethod), 322

428 Index Synapse Documentation, Release 0.0.34 test_model_inet_iface() (synapse.tests.test_model_inet.InetModelTestmethod), 323 method), 322 test_model_inet_web_post() test_model_inet_ipv4() (synapse.tests.test_model_inet.InetModelTest (synapse.tests.test_model_inet.InetModelTest method), 322 method), 323 test_model_inet_ipv4_raise() test_model_inet_whois_recns() (synapse.tests.test_model_inet.InetModelTest (synapse.tests.test_model_inet.InetModelTest method), 322 method), 323 test_model_inet_ipv6() (synapse.tests.test_model_inet.InetModelTesttest_model_inet_whoisemail() method), 322 (synapse.tests.test_model_inet.InetModelTest test_model_inet_mac() (synapse.tests.test_model_inet.InetModelTest method), 323 method), 322 test_model_inet_wifi() (synapse.tests.test_model_inet.InetModelTest test_model_inet_passwd() method), 323 (synapse.tests.test_model_inet.InetModelTest test_model_infotech_av() method), 322 (synapse.tests.test_model_infotech.InfoTechTest test_model_inet_postref() method), 323 (synapse.tests.test_model_inet.InetModelTest test_model_infotech_brutecast() method), 322 (synapse.tests.test_model_infotech.InfoTechTest test_model_inet_postref_postmissingprops() method), 323 (synapse.tests.test_model_inet.InetModelTest test_model_infotech_cve() method), 322 (synapse.tests.test_model_infotech.InfoTechTest test_model_inet_srv4_types() method), 323 (synapse.tests.test_model_inet.InetModelTest test_model_infotech_host() method), 323 (synapse.tests.test_model_infotech.InfoTechTest test_model_inet_srv6_types() method), 323 (synapse.tests.test_model_inet.InetModelTest test_model_infotech_hostexec() method), 323 (synapse.tests.test_model_infotech.InfoTechTest test_model_inet_url_fields() method), 323 (synapse.tests.test_model_inet.InetModelTest test_model_infotech_hostname() method), 323 (synapse.tests.test_model_infotech.InfoTechTest test_model_inet_urlfile() (synapse.tests.test_model_inet.InetModelTestmethod), 323 method), 323 test_model_infotech_itdev() test_model_inet_urlredir() (synapse.tests.test_model_infotech.InfoTechTest (synapse.tests.test_model_inet.InetModelTest method), 323 method), 323 test_model_infotech_semvertype() test_model_inet_web_acct() (synapse.tests.test_model_infotech.InfoTechTest (synapse.tests.test_model_inet.InetModelTest method), 324 method), 323 test_model_infotech_software() test_model_inet_web_action() (synapse.tests.test_model_infotech.InfoTechTest (synapse.tests.test_model_inet.InetModelTest method), 324 method), 323 test_model_language_idiom() test_model_inet_web_actref() (synapse.tests.test_model_language.LangTest (synapse.tests.test_model_inet.InetModelTest method), 324 method), 323 test_model_language_trans() test_model_inet_web_follows() (synapse.tests.test_model_language.LangTest (synapse.tests.test_model_inet.InetModelTest method), 324 method), 323 test_model_mat_spec_item() test_model_inet_web_logon() (synapse.tests.test_model_material.MatTest (synapse.tests.test_model_inet.InetModelTest method), 324 method), 323 test_model_org_has_webacct() test_model_inet_web_memb() (synapse.tests.test_model_orgs.OrgTest (synapse.tests.test_model_inet.InetModelTest method), 324 method), 323 test_model_orgs_seed_alias() test_model_inet_web_mesg() (synapse.tests.test_model_orgs.OrgTest (synapse.tests.test_model_inet.InetModelTest method), 324

Index 429 Synapse Documentation, Release 0.0.34 test_model_orgs_seed_name() (synapse.tests.test_model_telco.TelcoTest (synapse.tests.test_model_orgs.OrgTest method), 325 method), 324 test_model_time_from_unix() test_model_person() (synapse.tests.test_model_person.PersonTest (synapse.tests.test_model_temporal.InetModelTest method), 324 method), 325 test_model_person_2() (synapse.tests.test_model_person.PersonTesttest_model_time_minmax() method), 324 (synapse.tests.test_model_temporal.InetModelTest test_model_person_contact() method), 325 (synapse.tests.test_model_person.PersonTest test_model_type_inet_cidr() method), 324 (synapse.tests.test_model_inet.InetModelTest test_model_person_guidname() method), 323 (synapse.tests.test_model_person.PersonTest test_model_type_inet_email() method), 324 (synapse.tests.test_model_inet.InetModelTest test_model_person_has_alias() method), 323 (synapse.tests.test_model_person.PersonTest test_model_type_inet_ipv6() method), 324 (synapse.tests.test_model_inet.InetModelTest test_model_person_has_email() method), 323 (synapse.tests.test_model_person.PersonTest test_model_type_inet_mac() method), 324 (synapse.tests.test_model_inet.InetModelTest test_model_person_has_phone() method), 323 (synapse.tests.test_model_person.PersonTest test_model_type_inet_port() method), 324 (synapse.tests.test_model_inet.InetModelTest test_model_person_has_user() method), 323 (synapse.tests.test_model_person.PersonTest test_model_type_inet_udp4() method), 325 (synapse.tests.test_model_inet.InetModelTest test_model_person_has_webacct() method), 323 (synapse.tests.test_model_person.PersonTest test_model_type_inet_url() method), 325 (synapse.tests.test_model_inet.InetModelTest test_model_person_name() method), 323 (synapse.tests.test_model_person.PersonTest test_model_type_phone() method), 325 (synapse.tests.test_model_telco.TelcoTest test_model_person_tokn() method), 325 (synapse.tests.test_model_person.PersonTest test_model_typeinet_ipv4() method), 325 (synapse.tests.test_model_inet.InetModelTest test_model_syn_201709051630() method), 323 (synapse.tests.test_model_syn.SynModelTest test_model_whois_contact() method), 325 (synapse.tests.test_model_inet.InetModelTest test_model_syn_201709191412() method), 323 (synapse.tests.test_model_syn.SynModelTest test_models_cngov_mucd() method), 325 (synapse.tests.test_model_gov_cn.CnGovTest test_model_syn_201710191144() method), 322 (synapse.tests.test_model_syn.SynModelTest test_models_media_news() method), 325 (synapse.tests.test_model_media.MediaTest test_model_syn_201711012123() method), 324 (synapse.tests.test_model_syn.SynModelTest test_models_usgov_cage() method), 325 (synapse.tests.test_model_gov_us.UsGovTest test_model_telco_cast_loc_us() method), 322 (synapse.tests.test_model_telco.TelcoTest test_msgpack_byte() (synapse.tests.test_lib_msgpack.MsgPackTest method), 325 method), 310 test_model_telco_imei() (synapse.tests.test_model_telco.TelcoTesttest_msgpack_en() (synapse.tests.test_lib_msgpack.MsgPackTest method), 325 method), 310 test_model_telco_imsi() (synapse.tests.test_model_telco.TelcoTesttest_msgpack_un() (synapse.tests.test_lib_msgpack.MsgPackTest method), 325 method), 310 test_model_telco_phone() test_msgpack_unpk() (synapse.tests.test_lib_msgpack.MsgPackTest

430 Index Synapse Documentation, Release 0.0.34

method), 310 method), 311 test_named_props() (synapse.tests.test_lib_tufo.TestTufoPropstest_ratelimit_allow() (synapse.tests.test_lib_ratelimit.RateLimitTest method), 317 method), 311 test_nonexist_ctor() (synapse.tests.test_cortex.StorageTest test_reactor_base() (synapse.tests.test_reactor.ReactorTest method), 301 method), 325 test_not_equals() (synapse.tests.test_lib_tufo.TufoEqualityTesttest_refdict() (synapse.tests.test_lib_cache.CacheTest method), 317 method), 305 test_nyx_make_request() test_refdict_with() (synapse.tests.test_lib_cache.CacheTest (synapse.tests.test_lib_remcycle.NyxTest method), 305 method), 312 test_reflect_getClsNames() test_nyx_quoted_values() (synapse.tests.test_lib_reflect.ReflectTest (synapse.tests.test_lib_remcycle.NyxTest method), 311 method), 312 test_reflect_getItemInfo() test_nyx_simple_config() (synapse.tests.test_lib_reflect.ReflectTest (synapse.tests.test_lib_remcycle.NyxTest method), 311 method), 312 test_reflect_syn_reflect() (synapse.tests.test_lib_reflect.ReflectTest test_nyx_tornado_http_check() method), 311 (synapse.tests.test_lib_remcycle.NyxTest test_savefile_load() (synapse.tests.test_tools_dumprows_loadrows.LoadRowsTest method), 312 method), 328 test_ondem_add() (synapse.tests.test_lib_cache.CacheTest test_sched_base() (synapse.tests.test_lib_sched.SchedTest method), 305 method), 313 test_ondem_class() (synapse.tests.test_lib_cache.CacheTest test_sched_cancel() (synapse.tests.test_lib_sched.SchedTest method), 305 method), 313 test_openfile_abs() (synapse.tests.test_lib_openfile.OpenFileTesttest_sched_loop() (synapse.tests.test_lib_sched.SchedTest method), 310 method), 313 test_openfile_http() (synapse.tests.test_lib_openfile.OpenFileTesttest_sched_persec() (synapse.tests.test_lib_sched.SchedTest method), 310 method), 313 test_openfile_relative() (synapse.tests.test_lib_openfile.OpenFileTesttest_scrape_sync() (synapse.tests.test_lib_scrape.ScrapeTest method), 310 method), 313 test_output_bytes() (synapse.tests.test_lib_output.TestOutPuttest_service_base() (synapse.tests.test_lib_service.SvcTest method), 310 method), 313 test_output_str() (synapse.tests.test_lib_output.TestOutPut test_service_byname() (synapse.tests.test_lib_service.SvcTest method), 310 method), 313 test_persist_dir() (synapse.tests.test_lib_persist.PersistTest test_service_dmon_conf() method), 311 (synapse.tests.test_lib_service.SvcTest test_persist_file() (synapse.tests.test_lib_persist.PersistTest method), 313 method), 311 test_service_getNameProxy() test_persist_offset() (synapse.tests.test_lib_persist.PersistTest (synapse.tests.test_lib_service.SvcTest method), 311 method), 313 test_persist_pump() (synapse.tests.test_lib_persist.PersistTesttest_service_getTagProxy() method), 311 (synapse.tests.test_lib_service.SvcTest test_pg_encoding() (synapse.tests.test_cortex.CortexTest method), 314 method), 300 test_service_proxy() (synapse.tests.test_lib_service.SvcTest test_plain() (synapse.tests.test_tools_superhash.SuperhashTest method), 314 method), 329 test_service_proxysugar() test_queue_base() (synapse.tests.test_lib_queue.QueueTest (synapse.tests.test_lib_service.SvcTest method), 311 method), 314 test_queue_multislice() (synapse.tests.test_lib_queue.QueueTesttest_simple_compress() (synapse.tests.test_tools_dumprows_loadrows.DumpRowsTest method), 311 method), 328 test_queue_postfini() (synapse.tests.test_lib_queue.QueueTesttest_simple_tufo_creation() method), 311 (synapse.tests.test_lib_tufo.TufoCreateTests test_queue_slice() (synapse.tests.test_lib_queue.QueueTest method), 317 method), 311 test_simple_use() (synapse.tests.test_tools_dumprows_loadrows.DumpRowsTest test_queue_timeout() (synapse.tests.test_lib_queue.QueueTest method), 328

Index 431 Synapse Documentation, Release 0.0.34 test_sock_plex() (synapse.tests.test_lib_socket.SocketTest test_storm_ﬁlt_regex() (synapse.tests.test_lib_storm.StormTest method), 314 method), 315 test_sock_plex_txbuf() (synapse.tests.test_lib_socket.SocketTesttest_storm_guid_operator() method), 314 (synapse.tests.test_lib_storm.StormTest test_socket_glob_plex() (synapse.tests.test_lib_socket.SocketTest method), 315 method), 314 test_storm_guid_stablegen() test_socket_hostaddr() (synapse.tests.test_lib_socket.SocketTest (synapse.tests.test_lib_storm.StormTest method), 314 method), 315 test_ssl_hugetx() (synapse.tests.test_link_ssl.SslTest test_storm_lift() (synapse.tests.test_lib_storm.StormTest method), 320 method), 315 test_statemach_loadnsave() test_storm_lift_limit() (synapse.tests.test_lib_storm.StormTest (synapse.tests.test_statemach.StateMachTest method), 315 method), 326 test_storm_lifts_by() (synapse.tests.test_lib_storm.StormTest test_statemach_nofd() (synapse.tests.test_statemach.StateMachTest method), 315 method), 326 test_storm_limit() (synapse.tests.test_lib_storm.StormTest test_storage_confopts() (synapse.tests.test_cortex.StorageTest method), 315 method), 301 test_storm_pivot() (synapse.tests.test_lib_storm.StormTest test_storage_handler_defaults() method), 315 (synapse.tests.test_cortex.StorageTest method), test_storm_pivot_runt() (synapse.tests.test_lib_storm.StormTest 301 method), 315 test_storage_handler_misses() test_storm_postgres() (synapse.tests.test_runtime.StormRunTest (synapse.tests.test_cortex.StorageTest method), method), 326 301 test_storm_query_log() (synapse.tests.test_lib_storm.StormTest test_storage_row_manipulation() method), 315 (synapse.tests.test_cortex.StorageTest method), test_storm_ram() (synapse.tests.test_runtime.StormRunTest 301 method), 326 test_storage_rowmanipulation() test_storm_refs() (synapse.tests.test_lib_storm.StormTest (synapse.tests.test_cortex.StorageTest method), method), 315 301 test_storm_setprop() (synapse.tests.test_lib_storm.StormTest test_storage_xact_spliced() method), 315 (synapse.tests.test_cortex.StorageTest method), test_storm_show_help() (synapse.tests.test_lib_storm.StormTest 301 method), 315 test_storm_addnode() (synapse.tests.test_lib_storm.StormTesttest_storm_sqlite() (synapse.tests.test_runtime.StormRunTest method), 314 method), 326 test_storm_addtag() (synapse.tests.test_lib_storm.StormTesttest_storm_syntax_basic() method), 314 (synapse.tests.test_lib_syntax.StormSyntaxTest test_storm_alltag() (synapse.tests.test_lib_storm.StormTest method), 315 method), 314 test_storm_syntax_by() (synapse.tests.test_lib_syntax.StormSyntaxTest test_storm_cmpr_norm() method), 315 (synapse.tests.test_lib_storm.StormTest test_storm_syntax_comp_opts() method), 314 (synapse.tests.test_lib_syntax.StormSyntaxTest test_storm_delnode() (synapse.tests.test_lib_storm.StormTest method), 315 method), 314 test_storm_syntax_edit() (synapse.tests.test_lib_syntax.StormSyntaxTest test_storm_delnode_caching() method), 315 (synapse.tests.test_lib_storm.StormTest test_storm_syntax_gele() method), 314 (synapse.tests.test_lib_syntax.StormSyntaxTest test_storm_delprop() (synapse.tests.test_lib_storm.StormTest method), 315 method), 314 test_storm_syntax_lifteq() test_storm_deltag() (synapse.tests.test_lib_storm.StormTest (synapse.tests.test_lib_syntax.StormSyntaxTest method), 314 method), 315 test_storm_edit_end() (synapse.tests.test_lib_storm.StormTesttest_storm_syntax_liftlift() method), 314 (synapse.tests.test_lib_syntax.StormSyntaxTest test_storm_editmode() (synapse.tests.test_lib_storm.StormTest method), 315 method), 314 test_storm_syntax_macro_eq()

432 Index Synapse Documentation, Release 0.0.34

(synapse.tests.test_lib_syntax.StormSyntaxTest method), 326 method), 315 test_swarm_runtime_has() test_storm_syntax_oper_args() (synapse.tests.test_swarm_runtime.SwarmRunTest (synapse.tests.test_lib_syntax.StormSyntaxTest method), 326 method), 315 test_swarm_runtime_join() test_storm_syntax_pivot() (synapse.tests.test_swarm_runtime.SwarmRunTest (synapse.tests.test_lib_syntax.StormSyntaxTest method), 326 method), 315 test_swarm_runtime_maxtime() test_storm_syntax_regex() (synapse.tests.test_swarm_runtime.SwarmRunTest (synapse.tests.test_lib_syntax.StormSyntaxTest method), 326 method), 316 test_swarm_runtime_opts() test_storm_syntax_uppercase_and_underscore() (synapse.tests.test_swarm_runtime.SwarmRunTest (synapse.tests.test_lib_syntax.StormSyntaxTest method), 326 method), 316 test_swarm_runtime_opts_uniq() test_storm_syntax_whites() (synapse.tests.test_swarm_runtime.SwarmRunTest (synapse.tests.test_lib_syntax.StormSyntaxTest method), 326 method), 316 test_swarm_runtime_or() test_storm_tag_fromtag() (synapse.tests.test_swarm_runtime.SwarmRunTest (synapse.tests.test_lib_storm.StormTest method), 326 method), 315 test_swarm_runtime_pivot() test_storm_tag_glob() (synapse.tests.test_lib_storm.StormTest (synapse.tests.test_swarm_runtime.SwarmRunTest method), 315 method), 326 test_storm_tag_ival() (synapse.tests.test_lib_storm.StormTesttest_swarm_runtime_regex() method), 315 (synapse.tests.test_swarm_runtime.SwarmRunTest test_storm_tag_jointag() (synapse.tests.test_lib_storm.StormTest method), 326 method), 315 test_swarm_runtime_saveload() test_storm_tag_query() (synapse.tests.test_lib_storm.StormTest (synapse.tests.test_swarm_runtime.SwarmRunTest method), 315 method), 327 test_storm_tag_totag() (synapse.tests.test_lib_storm.StormTesttest_tags_subs() (synapse.tests.test_lib_tags.TagTest method), 315 method), 316 test_storm_task() (synapse.tests.test_lib_storm.StormTest test_task_base() (synapse.tests.test_lib_task.TaskTest method), 315 method), 316 test_storm_task_telepath() test_task_call() (synapse.tests.test_lib_task.TaskTest (synapse.tests.test_lib_storm.StormTest method), 316 method), 315 test_telepath_basics() (synapse.tests.test_telepath.TelePathTest test_storm_tree() (synapse.tests.test_lib_storm.StormTest method), 327 method), 315 test_telepath_call() (synapse.tests.test_telepath.TelePathTest test_swarm_runtime_and() method), 327 (synapse.tests.test_swarm_runtime.SwarmRunTesttest_telepath_callx() (synapse.tests.test_telepath.TelePathTest method), 326 method), 327 test_swarm_runtime_by() test_telepath_chop() (synapse.tests.test_telepath.TelePathTest (synapse.tests.test_swarm_runtime.SwarmRunTest method), 327 method), 326 test_telepath_clientside() test_swarm_runtime_clear() (synapse.tests.test_telepath.TelePathTest (synapse.tests.test_swarm_runtime.SwarmRunTest method), 327 method), 326 test_telepath_eval() (synapse.tests.test_telepath.TelePathTest test_swarm_runtime_eq() method), 327 (synapse.tests.test_swarm_runtime.SwarmRunTesttest_telepath_events() (synapse.tests.test_telepath.TelePathTest method), 326 method), 327 test_swarm_runtime_frob() test_telepath_fakesync() (synapse.tests.test_telepath.TelePathTest (synapse.tests.test_swarm_runtime.SwarmRunTest method), 327 method), 326 test_telepath_gzip() (synapse.tests.test_telepath.TelePathTest test_swarm_runtime_gele() method), 327 (synapse.tests.test_swarm_runtime.SwarmRunTesttest_telepath_nosuchobj()

Index 433 Synapse Documentation, Release 0.0.34

(synapse.tests.test_telepath.TelePathTest test_type_comp() (synapse.tests.test_lib_types.DataTypesTest method), 327 method), 318 test_telepath_push() (synapse.tests.test_telepath.TelePathTesttest_type_gettdef() (synapse.tests.test_lib_types.DataTypesTest method), 327 method), 318 test_telepath_reconnect() test_type_getTypeInfo() (synapse.tests.test_lib_types.DataTypesTest (synapse.tests.test_telepath.TelePathTest method), 318 method), 327 test_type_issub() (synapse.tests.test_lib_types.DataTypesTest test_telepath_reminder() (synapse.tests.test_telepath.TelePathTest method), 318 method), 327 test_type_json() (synapse.tests.test_lib_types.DataTypesTest test_telepath_reqproxy() (synapse.tests.test_telepath.TelePathTest method), 318 method), 327 test_type_pend() (synapse.tests.test_lib_types.DataTypesTest test_telepath_reshare() (synapse.tests.test_telepath.TelePathTest method), 318 method), 327 test_type_sepr() (synapse.tests.test_lib_types.DataTypesTest test_telepath_server_badvers() method), 318 (synapse.tests.test_telepath.TelePathTest test_type_sepr_parse() (synapse.tests.test_lib_types.DataTypesTest method), 327 method), 318 test_telepath_yielder() (synapse.tests.test_telepath.TelePathTesttest_type_sepr_reverse() (synapse.tests.test_lib_types.DataTypesTest method), 327 method), 318 test_thishost_hostname() test_type_str_nullval() (synapse.tests.test_lib_types.DataTypesTest (synapse.tests.test_lib_thishost.ThisTest method), 318 method), 316 test_type_str_strip() (synapse.tests.test_lib_types.DataTypesTest test_thishost_platform() (synapse.tests.test_lib_thishost.ThisTest method), 318 method), 316 test_type_time_timeepoch() test_thishost_ptrsize() (synapse.tests.test_lib_thishost.ThisTest (synapse.tests.test_lib_types.DataTypesTest method), 316 method), 318 test_threads_cancelable() test_type_xref() (synapse.tests.test_lib_types.DataTypesTest (synapse.tests.test_lib_threads.ThreadsTest method), 318 method), 316 test_types_comp_opt_only() test_threads_cantwait() (synapse.tests.test_lib_threads.ThreadsTest (synapse.tests.test_lib_types.DataTypesTest method), 316 method), 318 test_threads_pool() (synapse.tests.test_lib_threads.ThreadsTesttest_types_comp_optfields() method), 316 (synapse.tests.test_lib_types.DataTypesTest test_threads_pool_wrap() method), 318 (synapse.tests.test_lib_threads.ThreadsTest test_types_guid() (synapse.tests.test_lib_types.DataTypesTest method), 316 method), 318 test_time_parse() (synapse.tests.test_lib_time.TimeTest test_types_guid_resolver() method), 317 (synapse.tests.test_lib_types.DataTypesTest test_tools_autodoc() (synapse.tests.test_tools_autodoc.TestAutoDoc method), 318 method), 327 test_types_ndef() (synapse.tests.test_lib_types.DataTypesTest test_tools_autodoc_configable() method), 318 (synapse.tests.test_tools_autodoc.TestAutoDoc test_types_perm() (synapse.tests.test_lib_types.DataTypesTest method), 327 method), 318 test_tools_ingest() (synapse.tests.test_tools_ingest.TestIngesttest_types_propvalu() (synapse.tests.test_lib_types.DataTypesTest method), 329 method), 319 test_tools_json2mpk() (synapse.tests.test_tools_json2mpk.Json2MpkTesttest_types_storm() (synapse.tests.test_lib_types.DataTypesTest method), 329 method), 319 test_tools_pushfile() (synapse.tests.test_tools_pushfile.TestPushFiletest_types_tagtime() (synapse.tests.test_lib_types.DataTypesTest method), 329 method), 319 test_trigger_base() (synapse.tests.test_lib_trigger.TrigTest test_version_basics() (synapse.tests.test_lib_version.VersionTest method), 317 method), 319 test_type_bases() (synapse.tests.test_lib_types.DataTypesTesttest_version_extract_parts() method), 318 (synapse.tests.test_lib_version.VersionTest test_type_cast() (synapse.tests.test_lib_types.DataTypesTest method), 319 method), 318 test_version_fmt() (synapse.tests.test_lib_version.VersionTest

434 Index Synapse Documentation, Release 0.0.34

method), 319 TstEnv (class in synapse.lib.iq), 246 test_version_pack() (synapse.tests.test_lib_version.VersionTestTstOutPut (class in synapse.lib.iq), 247 method), 319 tufo() (in module synapse.common), 340 test_version_parseSemver() tufo() (in module synapse.lib.tufo), 279 (synapse.tests.test_lib_version.VersionTest TufoCache (class in synapse.lib.cache), 230 method), 319 TufoCreateTests (class in synapse.tests.test_lib_tufo), 317 test_version_unpack() (synapse.tests.test_lib_version.VersionTestTufoEqualityTest (class in synapse.tests.test_lib_tufo), method), 319 317 test_webapp_body() (synapse.tests.test_lib_webapp.WebAppTesttufoHasTag() (in module synapse.lib.tags), 275 method), 319 TufoPropCache (class in synapse.lib.cache), 230 test_webapp_dmon() (synapse.tests.test_lib_webapp.WebAppTesttx() (synapse.lib.socket.Socket method), 269 method), 319 txbytes() (synapse.lib.socket.Socket method), 269 test_webapp_publish() (synapse.tests.test_lib_webapp.WebAppTestTypeLib (class in synapse.lib.types), 281 method), 319 test_webapp_ssl() (synapse.tests.test_lib_webapp.WebAppTestU method), 319 un() (in module synapse.lib.msgpack), 251 test_webapp_static() (synapse.tests.test_lib_webapp.WebAppTestUndefinedValue (class in synapse.gene), 357 method), 319 unlink() (synapse.eventbus.EventBus method), 351 TestArgParser (class in synapse.tests.test_tools_dmon), unpackVersion() (in module synapse.lib.version), 285 328 Unpk (class in synapse.lib.msgpack), 250 TestAutoDoc (class in synapse.tests.test_tools_autodoc), update() (in module synapse.lib.scope), 265 327 update() (synapse.cores.sqlite.SqliteStorage method), 216 TestEasyCert (class in synapse.tests.test_tools_easycert), update() (synapse.lib.hashset.HashSet method), 241 328 update() (synapse.lib.scope.Scope method), 265 TestFilePath (class in synapse.tests.test_lib_filepath), 307 updateProperty() (synapse.cores.storage.Storage TestIngest (class in synapse.tests.test_tools_ingest), 329 method), 222 TestMain (class in synapse.tests.test_tools_dmon), 328 updatePropertyValu() (synapse.cores.storage.Storage TestOutPut (class in synapse.tests.test_lib_output), 310 method), 223 TestPushFile (class in synapse.tests.test_tools_pushfile), UrlType (class in synapse.models.inet), 291 329 usage() (synapse.axon.AxonHost method), 338 TestTufoProps (class in synapse.tests.test_lib_tufo), 317 UsGovTest (class in synapse.tests.test_model_gov_us), thisHostMust() (synapse.lib.iq.SynTest method), 246 322 thisHostMustNot() (synapse.lib.iq.SynTest method), 246 ThisTest (class in synapse.tests.test_lib_thishost), 316 V Thread (class in synapse.lib.threads), 277 validate_revnumber() (in module synapse.lib.module), ThreadsTest (class in synapse.tests.test_lib_threads), 316 249 tick() (synapse.lib.storm.Query method), 271 vals() (synapse.eventbus.BusRef method), 349 TimeMod (class in synapse.models.temporal), 295 vals() (synapse.lib.datapath.DataElem method), 237 TimeTest (class in synapse.tests.test_lib_time), 317 valu() (synapse.lib.datapath.DataElem method), 237 TimeType (class in synapse.lib.types), 281 values() (synapse.lib.cache.Cache method), 228 to_bytes() (in module synapse.common), 340 ValuNode (class in synapse.gene), 357 to_int() (in module synapse.common), 340 VarNode (class in synapse.gene), 357 tokenize() (in module synapse.gene), 358 VersionTest (class in synapse.tests.test_lib_version), 319 TooFewEvents, 295 verstr() (in module synapse.common), 340 TooManyTries, 286 vertup() (in module synapse.common), 341 TreeTest (class in synapse.tests.test_lib_trees), 317 trigger() (synapse.cores.xact.StoreXact method), 223 W trigger() (synapse.lib.trigger.Triggers method), 278 wait() (synapse.async.Boss method), 332 Triggers (class in synapse.lib.trigger), 278 wait() (synapse.eventbus.Waiter method), 351 TrigTest (class in synapse.tests.test_lib_trigger), 317 Waiter (class in synapse.eventbus), 351 true() (synapse.lib.iq.SynTest method), 246 waiter() (synapse.eventbus.EventBus method), 351 tryDynFunc() (in module synapse.dyndeps), 348 waitfini() (synapse.eventbus.EventBus method), 351 tryDynLocal() (in module synapse.dyndeps), 349 wake() (synapse.lib.socket.Plex method), 268 tryDynMod() (in module synapse.dyndeps), 349 wants() (synapse.axon.Axon method), 337

Index 435 Synapse Documentation, Release 0.0.34 wants() (synapse.axon.AxonCluster method), 338 WebApp (class in synapse.lib.webapp), 285 WebAppErr, 357 WebAppTest (class in synapse.tests.test_lib_webapp), 319 webCacheClear() (synapse.lib.remcycle.Hypnos method), 260 webCacheGet() (synapse.lib.remcycle.Hypnos method), 260 webCachePop() (synapse.lib.remcycle.Hypnos method), 260 webContentTypeSkipAdd() (synapse.lib.remcycle.Hypnos method), 260 webContentTypeSkipDel() (synapse.lib.remcycle.Hypnos method), 261 webJobWait() (synapse.lib.remcycle.Hypnos method), 261 whoami() (in module synapse.lib.auth), 227 Window (class in synapse.lib.ﬁfo), 238 withlock() (in module synapse.lib.threads), 277 withop() (synapse.lib.storm.Query method), 271 Woot (class in synapse.tests.test_daemon), 301 Woot (class in synapse.tests.test_lib_service), 314 woot() (in module synapse.tests.test_dyndeps), 303 worker() (in module synapse.common), 341 worker() (in module synapse.lib.threads), 277 wrap() (synapse.lib.threads.Pool method), 276 writeoff() (synapse.lib.atomﬁle.AtomFile method), 226 writeoff() (synapse.lib.heap.Heap method), 242 writer() (synapse.lib.threads.RWLock method), 277 X XmlDataElem (class in synapse.lib.datapath), 237 XrefType (class in synapse.lib.types), 283 Y yieldTimeTasks() (synapse.lib.sched.Sched method), 264

436 Index