Synapse Documentation Release 0.0.34 Nov 15, 2017 Contents 1 Synapse User Guide 3 1.1 Why Synapse?..............................................3 1.2 Graphs and Hypergraphs.........................................5 1.3 Data Model - Basics...........................................9 1.4 Data Model - Concepts.......................................... 14 1.5 Data Model - Node Concepts...................................... 15 1.6 Data Model - Node Properties...................................... 17 1.7 Data Model - Node Types........................................ 19 1.8 Data Model - Tag Concepts....................................... 26 1.9 Tags - Analytical Model......................................... 30 1.10 Using Synapse - Commands....................................... 37 1.11 Storm Query Language - Basics..................................... 40 1.12 Storm Reference - Data Modification Operators............................. 45 1.13 Storm Reference - Lift Operators.................................... 52 1.14 Storm Reference - Filter Operator, Comparators, and Helpers..................... 56 1.15 Storm Reference - Pivot Operators.................................... 59 1.16 Storm Reference - “By” Handlers.................................... 61 1.17 Storm Reference - Statistical Operators................................. 61 1.18 Storm Reference - Miscellaneous Operators............................... 61 1.19 Ingest - Introduction........................................... 62 1.20 Ingest - Commandline Ingest Tool.................................... 62 1.21 Ingest - Embed Directives........................................ 63 1.22 Ingest - Parsing Lines.......................................... 69 1.23 Ingest - Parsing CSV........................................... 71 1.24 Ingest - Parsing JSON.......................................... 72 1.25 Ingest - Parsing JSONL......................................... 85 1.26 Ingest - Parsing XML.......................................... 86 1.27 Ingest - file:bytes Nodes...................................... 87 1.28 Ingest - Sending Data to a Remote Cortex................................ 87 1.29 Ingest - Format Details.......................................... 87 1.30 Ingest - Conditional Expressions..................................... 88 2 Synapse DevOps Guide 89 2.1 Cortex: Synapse Hypergraph....................................... 89 2.2 Cortex: Built-In Storage Layers..................................... 93 2.3 Cortex: Storage Layer Details...................................... 94 i 2.4 Ingest: Structured Data Import...................................... 99 2.5 Axon: Synapse Blob Storage....................................... 99 2.6 Dmon: Synapse Deployment....................................... 99 2.7 Eventbus: Synapse Event Distribution.................................. 100 2.8 CoreModule: Model Definitions..................................... 100 2.9 Storm: Synapse Query Language.................................... 104 2.10 Telepath: Synapse Remote Method Invocation............................. 104 2.11 Synapse Deployment Scenarios..................................... 104 3 Synapse Data Model 109 3.1 Types................................................... 109 3.2 Forms................................................... 132 3.3 Universal Props.............................................. 195 4 synapse package 197 4.1 Subpackages............................................... 197 4.2 Submodules............................................... 332 4.3 synapse.async module.......................................... 332 4.4 synapse.axon module........................................... 333 4.5 synapse.common module........................................ 339 4.6 synapse.cortex module.......................................... 341 4.7 synapse.daemon module......................................... 342 4.8 synapse.datamodel module........................................ 344 4.9 synapse.dyndeps module......................................... 348 4.10 synapse.eventbus module........................................ 349 4.11 synapse.exc module........................................... 352 4.12 synapse.gene module........................................... 357 4.13 synapse.glob module........................................... 358 4.14 synapse.hivemind module........................................ 358 4.15 synapse.link module........................................... 358 4.16 synapse.mindmeld module........................................ 359 4.17 synapse.reactor module......................................... 359 4.18 synapse.statemach module........................................ 359 4.19 synapse.telepath module......................................... 359 4.20 Module contents............................................. 362 5 Synapse Configable Classes 363 5.1 synapse.axon.Axon............................................ 363 5.2 synapse.axon.AxonHost......................................... 365 5.3 synapse.cores.common.Cortex...................................... 366 5.4 synapse.cores.lmdb.LmdbStorage.................................... 369 5.5 synapse.cores.postgres.PsqlStorage................................... 369 5.6 synapse.cores.ram.RamStorage..................................... 369 5.7 synapse.cores.sqlite.SqliteStorage.................................... 370 5.8 synapse.lib.fifo.Fifo........................................... 370 5.9 synapse.lib.remcycle.Hypnos...................................... 371 5.10 synapse.lib.session.Curator........................................ 372 5.11 synapse.lib.storm.Runtime........................................ 372 5.12 synapse.swarm.runtime.Runtime..................................... 372 6 Synapse Performance 375 6.1 A Note on Comparisons......................................... 375 6.2 Test Data Sets.............................................. 375 6.3 Cortex Configurations.......................................... 376 6.4 Test Systems............................................... 377 ii 6.5 Results.................................................. 377 6.6 Additional Tests............................................. 379 7 Contributing to Synapse 381 7.1 Project Style Guide............................................ 381 7.2 Docs Generation............................................. 386 7.3 Contribution Process........................................... 386 7.4 Running Postgres Tests.......................................... 387 8 Synapse Release Process 389 8.1 Github Milestone Management..................................... 389 8.2 Release Notes.............................................. 389 8.3 Cutting the Release............................................ 390 9 Indices and tables 393 Python Module Index 395 iii iv Synapse Documentation, Release 0.0.34 Contents: Contents 1 Synapse Documentation, Release 0.0.34 2 Contents CHAPTER 1 Synapse User Guide This User Guide is written by and for Synapse users. Additional technical documentation can be found elsewhere in the doc index. The User Guide is a living document and will continue to be updated and expanded. The current sections are: 1.1 Why Synapse? Synapse is a distributed key-value hypergraph analysis framework. Synapse is designed to support analysis conducted over very large and disparate data sets. Analysis is predicated on the representation of data from a given knowledge domain in a structured data model that allows analysts to represent, annotate, and query across the collected data. Synapse offers several advantages: Free and Open Source Unlike costly commercial analysis systems, Synapse is available on Github under the Apache License 2.0. Performance Synapse was designed to address the performance limitations that constrain many large-scale analysis systems and eventually make them unworkable in practice. Synapse supports multiple storage and indexing options (RAM, LMDB, PostgreSQL, SQLite) that can be tailored to support the type and volume of data stored, as well as the proportion of read (query / retrieval) vs. write (node creation and deconfliction) operations. For detailed performance benchmarks, see Synapse Performance for more information. Extensible Comprehensive Data Model Synapse’s hypergraph framework allows for representation of more disparate data types and complex relationships. The extensible data model can be adapted to any knowledge domain. Powerful Analysis Framework Synapse provides a shared analytical workspace that consolidates research and prevents “stovepiping” of findings. An- alytical knowledge is preserved in a central representational framework instead of residing solely within a specialized research team or with an individual “subject matter expert” who may leave the organization. 3 Synapse Documentation, Release 0.0.34 Synapse’s shared workspace allows real-time annotation of data with analytical observations; the work of one analyst is immediately visible to all others in the same workspace. The ability to “analyze once, record for all” means that analysts no longer need to waste effort independently reviewing the same data to come to the same conclusions. Finally, the incorporation of new data or new analytical findings dynamically highlights supporting or conflicting analysis, allowing assessments and hypotheses to be revisited as needed. Proven Methodology Informed by Real-World Experience Synapse was not developed as a mathematical abstraction. Instead, Synapse grew out of a real-world need to track a complex, diverse, and very large data set: