Blacklynx Open API Library User Guide 9 Mar 2019
Total Page:16
File Type:pdf, Size:1020Kb
BlackLynx Open API Library User Guide 9 Mar 2019 BlackLynx Open API Library User Guide BLACKLYNX, INC. - Please note that BlackLynx APIs are SUBJECT TO CHANGE WITHOUT NOTICE 1 BlackLynx Open API Library User Guide 9 Mar 2019 Copyright (c) 2015-2019, BlackLynx, Inc. (formerly Ryft Systems, Inc.) All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. All advertising materials mentioning features or use of this software must display the following acknowledgement: This product includes software developed by BlackLynx, Inc. Neither the name of BlackLynx, Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY BLACKLYNX, INC. ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL BLACKLYNX, INC. BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. GNU General Public License, Version 2, June 1991 Copyright © 1989, 1991 Free Software Foundation, Inc. / 51 Franklin St. 5th Floor, Boston, MA 02110-1301, USA BLACKLYNX, INC. - Please note that BlackLynx APIs are SUBJECT TO CHANGE WITHOUT NOTICE 2 BlackLynx Open API Library User Guide 9 Mar 2019 Revision History Date Changes Minimum and/or Release Software Revision(s) 20190309 • In addition to combination LONG/LAT (or LAT/LONG) fields, PIP now supports separated LONG and LAT fields r1643 20190107 • Added ARP decodes to PCAP. r1513 • PIP vertex files now have more liberal number separator rules. 20181024 • Currency symbols are now optional in the numeric portions of the currency primitive’s query clause. r1344 • CSV, JSON, and XML are now escape-friendly according to RFC 4180, http://json.org (as of 20180822), and r1293 http://www.w3.org/TR/xml (as of 20180822), respectively. o An UNESCAPE_DATA=false option is added to not perform escape checking, but this is not recommended. o Search and replace operations now handle escaping rules when generating structured output as well. • Added the new EMPTY_FIELD primitive. • Added two new global options primarily targeting forensics use cases: find-input-recursively and input-is-filelist. • Various PCAP primitive enhancements and improvements: o Match-associated TCP stream dumps, UDP dumps, and payload dumps can now be generated. o New meta field family frame.time[_...] is now available. o Output index file generation is now supported, to include options enabling match qualifiers and packet number output. This resulted in an API call change, so existing user programs leveraging the PCAP primitive will need to be updated. o IP fragmentation now supports the capability to intelligently filter on the individual fragments for those query operations that do not revolve around payload searches. For payload searches, the properly combined fragmented payloads are leveraged for the search. In all cases, in the output data file that is generated upon request, fragmented source packets now properly appear in-order with respect to other source corpus packets. o Added ip.proto and ipv6.nxt PCAP filter support. • Extended and enhanced several primitive descriptions to be more complete. • Updated all examples and added a few new ones throughout this guide. • The Java and Python bindings have been revamped. Please note that existing Java and Python user programs may need to be modified to match the updated API bindings. BLACKLYNX, INC. - Please note that BlackLynx APIs are SUBJECT TO CHANGE WITHOUT NOTICE 3 BlackLynx Open API Library User Guide 9 Mar 2019 Table of Contents 1 Scope ..................................................................................................................................................................7 2 Overview .............................................................................................................................................................8 3 Available Functions ..............................................................................................................................................9 3.1 Open API Library Dataset ........................................................................................................................................ 9 3.2 Dataset Manipulation Functions ............................................................................................................................. 9 3.3 Primitives .............................................................................................................................................................. 10 3.3.1 Primitive Criteria Specification ...................................................................................................................... 10 3.3.2 Parameters and Options List ......................................................................................................................... 13 3.3.3 API Global Options List .................................................................................................................................. 21 3.3.4 Search Primitive Family ................................................................................................................................. 23 3.3.4.1 EXACT ........................................................................................................................................................ 23 3.3.4.1.1 Example Program ................................................................................................................................ 24 3.3.4.2 HAMMING ................................................................................................................................................. 25 3.3.4.2.1 Example Program ................................................................................................................................ 26 3.3.4.3 EDIT_DISTANCE ......................................................................................................................................... 27 3.3.4.3.1 Example Program ................................................................................................................................ 29 3.3.4.4 PCRE2 ........................................................................................................................................................ 30 3.3.4.4.1 Example Program ................................................................................................................................ 30 3.3.4.5 DATE .......................................................................................................................................................... 32 3.3.4.5.1 Example Program ................................................................................................................................ 33 3.3.4.6 TIME .......................................................................................................................................................... 34 3.3.4.6.1 Example Program ................................................................................................................................ 35 3.3.4.7 NUMBER .................................................................................................................................................... 37 3.3.4.7.1 Example Program ................................................................................................................................ 39 3.3.4.8 CURRENCY ................................................................................................................................................. 41 3.3.4.8.1 Example Program ................................................................................................................................ 43 3.3.4.9 IPV4 ........................................................................................................................................................... 44 3.3.4.9.1 Example Program ................................................................................................................................ 45 3.3.4.10 IPV6 ........................................................................................................................................................... 47 3.3.4.10.1 Example Program .............................................................................................................................. 48 3.3.4.11 EMPTY_FIELD ............................................................................................................................................ 49 3.3.4.11.1