Analysis with SiLK Analyst’s Handbook for SiLK Version 3.15.0 and Later AUGUST 2020 Paul Krystosek Nancy Ott Timothy Shimeall CERT® Situational Awareness Group Network Traffic Analysis with SiLK Analyst’s Handbook for SiLK Versions 3.15.0 and Later Paul Krystosek Nancy M. Ott Geoffrey Sanders Timothy Shimeall August 2020 CERT® Situational Awareness Group [DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. https://www.sei.cmu.edu Copyright 2020 Carnegie Mellon University. All Rights Reserved. This material is based upon work funded and supported by the Department of Defense under Contract No. FA8702-15-D-0002 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. The view, opinions, and/or findings contained in this material are those of the author(s) and should not be construed as an official Government position, policy, or decision, unless designated by other documentation. References herein to any specific commercial product, process, or service by trade name, trade mark, manu- facturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University or its Software Engineering Institute. This report was prepared for the SEI Administrative Agent AFLCMC/AZS 5 Eglin Street Hanscom AFB, MA 01731-2100 NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING IN- STITUTE MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MAT- TER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MER- CHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. [DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited dis- tribution. Please see Copyright notice for non-US Government use and distribution. Internal use:* Permission to reproduce this material and to prepare derivative works from this material for internal use is granted, provided the copyright and “No Warranty” statements are included with all reproductions and derivative works. External use:* This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other external and/or commercial use. Requests for permission should be directed to the Software Engineering Institute at [email protected]. * These restrictions do not apply to U.S. government entities. Carnegie Mellon®, CERT®, CERT Coordination Center® and FloCon® are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. DM20-0675 SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY ii [DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Adobe is a registered trademark of Adobe Systems Incorporated in the United States and/or other countries. Akamai is a registered trademark of Akamai Technologies, Inc. Apple and OS X are trademarks of Apple Inc., registered in the U.S. and other countries. Cisco Systems is a registered trademark of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. DOCSIS is a registered trademark of CableLabs. FreeBSD is a registered trademark of the FreeBSD Foundation. IEEE is a registered trademark of The Institute of Electrical and Electronics Engineers, Inc. JABBER is a registered trademark and its use is licensed through the XMPP Standards Foundation. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. MaxMind, GeoIP, GeoLite, and related trademarks are the trademarks of MaxMind, Inc. Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States and/or other countries. OpenVPN is a registered trademark of OpenVPN Technologies, Inc. Perl is a registered trademark of The Perl Foundation. Python is a registered trademark of the Python Software Foundation. SNORT is a registered trademark of Cisco and/or its affiliates. Solaris is a registered trademark of Oracle and/or its affiliates in the United States and other countries. UNIX is a registered trademark of The Open Group. VPNz is a registered trademark of Advanced Network Solutions, Inc. Wireshark is a registered trademark of the Wireshark Foundation. All other trademarks are the property of their respective owners. SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY iii [DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. This page intentionally left blank. SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY iv [DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Contents Contents v List of Figures xi List of Tables xiii List of Examples xv List of Hints xix Acknowledgements xxi Handbook Goals xxiii 1 Introduction to SiLK 1 1.1 What is SiLK? ............................................ 1 1.2 The SiLK Flow Repository ..................................... 2 1.2.1 What is Network Flow Data?................................ 2 1.2.2 Structure of a Flow Record................................. 3 1.2.3 Flow Generation and Collection............................... 3 1.2.4 Introduction to Flow Collection............................... 6 1.2.5 Where Network Flow Data Are Collected ......................... 6 1.2.6 Types of Network Traffic .................................. 7 1.2.7 The Collection System and Data Management ...................... 7 1.2.8 How Network Flow Data Are Organized.......................... 8 1.3 The SiLK Tool Suite......................................... 8 1.4 How to Use SiLK for Analysis.................................... 9 1.4.1 Single-path Analysis..................................... 9 1.4.2 Multi-path Analysis ..................................... 9 1.4.3 Exploratory Analysis..................................... 10 1.5 Workflow for SiLK Analysis..................................... 10 1.5.1 Formulate........................................... 10 1.5.2 Model............................................. 11 1.5.3 Test.............................................. 12 1.5.4 Analyze............................................ 12 1.5.5 Refine............................................. 12 1.6 Applying the SiLK Workflow .................................... 12 1.7 Avoiding Cognitive Biases...................................... 13 1.8 Dataset for Single-path, Multi-path, and Exploratory Analysis Examples............ 14 SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY v [DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. CONTENTS 2 Basic Single-path Analysis with SiLK: Profiling and Reacting 17 2.1 Single-path Analysis: Concepts................................... 17 2.1.1 Scoping Queries of Network Flow Data........................... 18 2.1.2 Excluding Unwanted Network Traffic............................ 19 2.1.3 Example Single-Path Analysis................................ 19 2.2 Single-path Analysis: Analytics................................... 19 2.2.1 Get a List of Sensors With rwsiteinfo .......................... 19 2.2.2 Choose Flow Records With rwfilter ........................... 23 2.2.3 View Flow Records With rwcut .............................. 27 2.2.4 Viewing File Information with rwfileinfo ........................ 29 2.2.5 Profile Flows With rwuniq and rwstats .......................... 31 2.2.6 Characterize Traffic by Time Period With rwcount .................... 39 2.2.7 Sort Flow Records With rwsort .............................. 41 2.2.8 Use IPsets to Gather IP Addresses............................. 43 2.2.9 Resolve IP Addresses to Domain Names With rwresolve . 47 2.3 Situational Awareness and Single-Path Analysis.......................... 51 2.3.1 Components of Situational Awareness ........................... 51 2.3.2 Single-Path Analysis for Desired Awareness: Validate Web and DNS Servers . 52 2.3.3 Single-Path Analysis for Actual Awareness: Examine Network Traffic.......... 54 2.3.4 Translate IDS Signatures into rwfilter Calls with rwidsquery . 57 2.4 Summary of SiLK Commands in Chapter 2 ............................ 58 3 Case Studies: Basic Single-path Analysis 59 3.1 Profile Traffic Around an Event................................... 59 3.1.1 Examining Shifts in Traffic ................................. 60 3.1.2 How to Profile Traffic .................................... 61 3.2 Generate Top N Lists ........................................ 63 3.2.1 Using rwstats to Create Top N Lists ........................... 63 3.2.2 Interpreting the Top-N Lists ................................ 66 4 Intermediate Multi-path Analysis with SiLK: Explaining and Investigating 69 4.1 Multi-path Analysis: Concepts ................................... 69 4.1.1 What Is Multi-path Analysis?................................ 69 4.1.2 Example of a Multi-path Analysis: Examining Web Service Traffic........... 71 4.1.3 Exploring Relationships and Behaviors With Multi-path Analysis............ 73 4.1.4 Integrating and Interpreting the Results of Multi-path Analysis............. 73 4.1.5 “Gotchas” for Multi-path Analysis............................. 74 4.2 Multi-path Analysis: Analytics