Immediate Insight User's Guide
Dec2015 Copyright Copyright 2015-2016 FireMon, LLC. All rights reserved. This product and related documentation are protected by copyright and distributed under licensing restricting their use, copying, distribution, and decompilation. No part of this product or related documentation may be reproduced in any form or by any means without the written authorization of FireMon, LLC. All right, title, and interest in the product shall remain with FireMon and its licensors. This product and related documentation are provided under a license agreement containing restric- tions on use and disclosure and are protected by intellectual property laws. This product and documentation may provide access to or information on content, products, and ser- vices from third parties. FireMon, LLC is not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. FireMon, LLC will not be respons- ible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services. The information in this document is subject to change without notice and is not warranted to be error- free. If you find any errors, please report them to us in writing.. FireMon is a registered trademark of FireMon, LLC. All other products or company names mentioned herein are trademarks or registered trademarks of their respective owners. Immediate Insight User's Guide
Contents
Copyright 2 Contents 3 About the Immediate Insight User's Guide 10 Document Structure 10 Document Typography 10 About Immediate Insight 11 Immediate Insight IT Data Discovery Methodology 11 Chapter 1: Installation 12 System Recommendations 13 Deploy the Virtual Appliance 14 Initial Setup Overview 14 Deploy on VMWare 14 Initial Configuration 15 Initial Setup 15 Configuration Notes 15 Install on VMWare Workstation 17 Step 1: Download Immediate Insight 17 Step 2: Import the OVA into VMWare: 17 Step 3: Adjust the settings 17 Step 4: Log on from the VMWare Console 17 Step 5: Log on from a browser 18 Step 6: Connect data sources 18 Installation - Non-clustered 19 Step 1: Download Immediate Insight OVA and License 19 Step 2: Import the OVA into VMWare 19 Step 3: Adjust the VM settings 19 Installation - Clustered 20 Sample Appliance Setup Session 22 Installation - Agent Only 25
Contents: Contents 3 Immediate Insight User's Guide
Before Beginning Agent Installation 25 Installing Agent Only on a Remote Machine 25 Log into Immediate Insight Server GUI, in order to use remote agent 26 Sign In to Immediate Insight 27 For Evaluation Users 27 About License Keys 28 Activate a License Key 28 View Current License Limits 28 Sign Out of Immediate Insight 28 Chapter 2: Administration 29 Key Administrative Tasks 30 How Immediate Insight Communicates 31 Open Ports 31 API Access 31 Security 32 Enabling Encryption 32 Certificates 32 About Users 33 User Types 33 Add a User 33 Data Administrator Accounts 34 CLI Admin Password Management 34 Data Repositories 34 Add a New Repository 34 Manage Data Flow 35 Data Collection 35 Getting Data into the System 35 Set Up Data Collectors 35 Standard Collectors 35 To permanently mount a remote server’s network share 36 To temporarily mount a network share (mount will be lost if the appliance reboots) 36
4 Contents: Contents Immediate Insight User's Guide
Add a New Data Collector 36 Additional notes about adding a data collector: 37 Immediate Insight Collector 38 Set Data Retention 39 Managing Data 39 Deleting Old Data 39 Work with Email 40 Inbound Email 40 Email Alerts 40 Setting up Email Alerts 40 Outbound Email 41 System Management 43 Command Reference 43 Configuration and Diagnostic Files 46 Config Files 46 Diag Files 46 About the Data Router 47 The Data Router: Processing Data as it Arrives 47 Quick Routes 47 Create a Data Route 48 Custom Feeds 50 Create a Custom Feed for a Data Route 50 Edit a Data Route 51 Delete a Data Route 51 Manage Routes 52 Remote Agents 53 Example of Commands using Streaming Format Structure 53 Examples of Commands using Request-Response Structure 54 NMAP 54 Iperf Bandwidth Check 54 OpenVAS 54
Contents: Contents 5 Immediate Insight User's Guide
Netstat 54 MTR 54 Traceroute 54 Request-response format structure for workflow-initiated commands 54 Example of Command using Request-Response Format Structure for Workflows 55 NMAP 55 Variables 55 Command Invocation Options 56 Output Options 57 Workflow Tracking 58 How it works 58 Overview 59 Workflow Linking and Interaction 59 Equipment Orders Workflow 59 Performance Monitor Workflow 60 Use a Workflow 64 Add a Workflow 64 Edit a Workflow 67 Delete a Workflow 67 Delete a State or Transition 67 Workflow Troubleshooting 68 Data Route Actions 68 Custom Action Scripts 70 Loading Custom Scripts 70 Writing Custom Scripts 70 Utility Functions for Custom Actions 71 Parsing 71 Example: 71 Chapter 3: Usage 73 The Home Page 74 Return to the Home Page 74
6 Contents: Contents Immediate Insight User's Guide
Search from the Home Page 75 Search Box 75 Time Selector 75 About Settings 76 Change Password 76 Activate License 76 About the DataFlow Page 77 Import 77 Drag & Drop Import 77 Scratch Pad 78 Import Settings 78 About the Firehose Page 79 About the Search Term Box 79 Apply Filter 79 Pause 79 Perform a search 79 Activity Chart 80 Activity Chart Options 80 Details 80 Live Event Count 80 Event Data 80 About the Search Page 81 Click Navigation 82 Search Page Sections 82 Details 83 Search Panels 84 Details 85 One Click Analytics: Action 85 Add Note 85 Spam 85 Action 86
Contents: Contents 7 Immediate Insight User's Guide
Auto-tag 86 Perform a Search 86 Search Terms 87 Key Concepts 87 Qualifiers 87 Grouping 88 Quotation Marks 88 Wildcards 88 Metadata 89 Time 89 About the Microscope Page 90 About the Reputation Page 91 Add Reputation Information 92 Add Bulk Reputation Information 92 Delete Reputation Entries 93 About Pinboard and Bookmarks 94 Pinboard 94 Bookmarks 94 Add a Pin or Bookmark 95 Manage Pinboard and Bookmarks 95 Share a Session 96 Quit Sharing 96 Shareboard 96 Chapter 4: API 97 APIs: Getting Data Out 98 HTTPS Query API 98 REST API Examples 99 API Credentials 100 REST API Certificates 101 Certificates 101 Command-Line Client 102
8 Contents: Contents Immediate Insight User's Guide
Installation 102 Command-Line Options 103 Command-Line Examples 103
Contents: Contents 9 Immediate Insight User's Guide
About the Immediate Insight User's Guide The purpose of this guide is to help you effectively use the Immediate Insight application. In this guide, you will find descriptions of the features and explanations of common processes you might perform in the applic- ation. Document Structure This guide contains the following chapters:
l Chapter 1: Installation—details how to deploy Immediate Insight to a virtual machine environment
l Chapter 2: Administration—details how to perform a variety of administrative tasks, such as set- ting up users and collecting data into Immediate Insight. Performing administrative related tasks will require a user with administrative privileges.
l Chapter 3: Usage—details how to search and analyze data in Immediate Insight, and get the most out of the application. User privileges also allows for ad-hoc import of data when needed.
l Chapter 4: API— is for system integrators who wish to create integrations between Immediate Insight and third-party systems.
Document Typography Buttons and graphical user interface (GUI) commands appear in bold. Example: Click Save. Commands are preceded by the verbs "click" and "right-click". These instructions assume default mouse settings on standard PC mouse devices, where the left mouse button is used for primary navigation ("click") and the right-mouse button is used for additional options and context menus ("right-click"). Code and command prompts appear in Courier New text. Variables appear in Courier New italics. In many cases, data that you enter or data that the system returns is specific to your system. For example, when you ping the Application Server, you will enter the IP address of your Application Server, not the sample data shown. Example: ping 192.125.27.25 Additional instructions may appear inside parentheses. Example: Host/IP: (for your Application Server).
A Note will appear in a green box to provide special information about a feature or process. A note may also stress the importance of the information that is provided.
Example code or example output will usually appear in a blue box.
A Caution! will appear to alert you to potential data corruptions that could occur if you proceed with an action.
A Prerequisite will appear to alert you to additional steps that must be taken before continuing with a pro- cedure.
: About the Immediate Insight User's Guide 10 Immediate Insight User's Guide
About Immediate Insight FireMon Immediate Insight collects and correlates all IT data to help analysts and operations staff increase visibility into the data and reduce the time and effort spent on correcting issues. Immediate Insight brings Google-like simplicity and speed to data analysis and discovery. It merges machine learning, correlation and natural language in a simple, workflow-centric interface to reveal rela- tionships in the data that users didn’t even know to look for. It transforms organizations from a ‘data as last resort’ mindset to the ‘data first’ practice necessary to enhance security, performance and operations. Immediate Insight’s real-time analysis across data silos provides the timely and detailed operational visibility necessary to:
l Identify and investigate the suspicious
l Search for indicators of breach and operational inefficiencies
l Get real-time analysis of security data
l Accelerate incident resolution and reduce escalations
l Automatically connect and correlate data silos
l Stage data for analysis by escalation teams
Immediate Insight IT Data Discovery Methodology
: About Immediate Insight 11 Chapter 1: Installation
System Recommendations 13 Deploy the Virtual Appliance 14 Initial Configuration 15 Install on VMWare Workstation 17 Installation - Non-clustered 19 Installation - Clustered 20 Sample Appliance Setup Session 22 Installation - Agent Only 25 Sign In to Immediate Insight 27 About License Keys 28 Sign Out of Immediate Insight 28
Immediate Insight User's Guide Immediate Insight User's Guide
System Recommendations Real-time search and analytics is a resource-intensive process and requires a fair amount of storage for index- ing and workspace. Select a storage system with decent write performance (IOPS). Since this type of data typically has a short lifespan and you likely have other copies of it, high-availability storage may not be an important concern. Storage requirements vary with data type – data is compressed on disk, but metadata gets added and auto- matic indexing requires working space. A reasonable rule of thumb is 1 GB of storage per 1 million active records. 500 GB is typically a good recommended size. Once storage is provisioned, extending it afterward is challenging, so it is preferable to provide as much disk space as possible at initial installation. The distributed architecture of Immediate Insight allows you to scale up by adding additional nodes and clus- tering them together, or by increasing physical resources of this single node. The following are the minimum recommended system requirements for Immediate Insight. RAM and memory recommendations:
l Recommend 32 GB RAM, 8 virtual CPUs for production
l Minimum 16 GB RAM, 4 virtual CPUs for demonstration
l 16 GB is typical for 250 million records
l 24 GB is ideal for 500 million active records
l 32 GB is ideal for 1 billion active records Non-clustered installations:
l 500 GB or higher disk space
l 32 GB of RAM
l 8 virtual CPUs Clustered installations have more robust hardware requirements than standalone installs, it is assumed each member meets the following minimum specifications.
l 16 or more cores
l 32 GB - 64 GB RAM, dedicated 1 GB+ NIC (bridged if VM)
l Enterprise quality RAID of SSDs or 7200+ RPM mechanical drives for the data. Raid 10 is optional. 1 TB of disk space. Use of single drives or consumer grade drives is not recommended. Assigning mul- tiple members to the same RAID is not recommended.
13 Chapter 1: Installation: System Recommendations Immediate Insight User's Guide
Deploy the Virtual Appliance Initial Setup Overview The following is an overview of the initial installation process. 1. Download the Immediate Insight file.
l It is packaged as an OVA file for VMWare ESXi5 or later.
Note: VMWare Workstation 8+ or VMWare Fusion Version 6+ are also supported, but these are primarily recommended for demonstration or testing purposes. See the steps for installing a Imme- diate Insight using VM Workstation.
2. Deploy the file in the hypervisor, assign data storage to it, and launch. 3. Log on to the Immediate Insight console, assign an IP address, initialize the data storage, and start the application. 4. Log on from a browser and connect data sources.
Note: Chrome is the supported browser for Immediate Insight.
Deploy on VMWare 1. Download the Immediate Insight OVA file provided by FireMon to your computer. The file is approx- imately 1 GB. 2. From the vSphere Management Client, select File > Deploy OVF Template... to import the down- loaded file. 3. Click Edit Settings to adjust the memory and storage, and to assign data storage. 4. Adjust the memory and processors as needed. 32GB of RAM and 8 virtual cores is typically a good size. 5. Click the Resources tab, and then select the Reserve all guest memory (All locked) check box. This will provide optimal application performance. 5. Click Add and follow the wizard to add a second virtual hard disk to hold your data. Use of SSD, or fast mechanical drives of 7200 RPM+ are strongly recommended for good performance. 500 GB or more is typically a good size.
Note: If you need to save space it is permissible to thin provision the drive, however thick pro- visioning provides the best performance and is recommended. Store the Virtual Disk as a single file.
6. Save changes, restart the VM and open a console to it.
Chapter 1: Installation: Deploy the Virtual Appliance 14 Immediate Insight User's Guide
Initial Configuration
These instructions assume you are running the VM in your own hosted environment, for cloud install- ations, please contact FireMon Immediate Insight Support.
Once the application is running in the hypervisor, log on from a console to set an IP address and initialize stor- age.
Console Administrative User: Insight
Default Password: WhatsHappeningNow
The appliance is basically just a stock installation of Ubuntu Server 14.04 LTS, so it has good connectivity and driver support, especially for virtual and cloud environments. The Immediate Insight software runs in user space and has very few dependencies, so you are free to make changes to the Linux environment as needed to integrate with your environment. Standard Linux utilities such as apt-get and editors such as nano, vi, and joe are available.
Note: Run all tasks as the Console Administrative User (CAU). If you need superuser privileges, preface the command with sudo. Do not attempt to makes changes as user "root" as this can cause conflicts within the system.
Caution! Do not attempt the change the default password for the console administrative user until the installation process is fully complete.
Initial Setup Initial setup is done with the install command. To verify that installation was successful, use the status command. It should show all 5 system processes as “running” and Data Storage should show 1% used. If you see “not running” or no number appears for pct used then there may be a problem. First try using the start command to resolve the issue, if this doesn’t work, contact FireMon Immediate Insight Support for assistance.
Note: After full reboot of the VM, if Immediate Insight processes don’t start automatically, you must type start from the console or SSH session.
At this point you can log on from a browser. You will only need the CLI if you want to add users, mount shares for data access, or do other privileged system administration tasks.
Configuration Notes
l IP Address: If you make a mistake on IP address, you can use the set-ip command to change it later. You can type “dhcp” in place of IP address for automatic address assignment You can also edit the /etc/network/interfaces file directly if you need to use more complex configuration options such as IPv6.
15 Chapter 1: Installation: Initial Configuration Immediate Insight User's Guide
l Time: Time is typically acquired from the network automatically, but you can use the date command to verify. set-time will force the system to try and update its time via NTP. You can use the set- timezone command to change timezone.
l Update: The software update is done as part of the initial install command, but if you wish to update in the future, just type update. The version command will tell you what version you are currently run- ning.
l Password: To change the system administrator (CLI) password, use the set-password command. This password is also the sudo password for privileged commands.
l SSD: Default assumes a mechanical hard drive, but if you are using a SSD you can change the set- ting to optimize Immediate Insight to make use of more efficient IO capabilities of SSD. To enable SSD, enter the command: set-ssd enable
l SSL: Default access is by HTTP. To change to HTTPS, use the command: set-ssl on and then apply the change by using the command: reload server.
The system uses self-signed certificates by default. You can install your own certificates by placing them in the app/config/certificates directory.
Note that although the Immediate Insight client runs in the browser, it is a web application rather than a web page, so the browser enforces greater security requirements. After log on, the client opens two separate websocket channels to the server, a control channel (port 3201) and a data channel (port 3801). If you are using self-signed certificates some browsers, such as Firefox, may require you to authorize access for each channel separately.
Note: Chrome is the supported browser for Immediate Insight.
Chapter 1: Installation: Configuration Notes 16 Immediate Insight User's Guide
Install on VMWare Workstation To install a non-clustered version of Immediate Insight for demo purposes, complete the following steps. Step 1: Download Immediate Insight 1. Download the Immediate Insight OVA file provided by FireMon to your computer. The file is approx- imately 1 GB. Step 2: Import the OVA into VMWare: 1. Click File > Open a Virtual Machine. 2. Open the OVA file you just downloaded. Step 3: Adjust the settings 1. Add a second virtual hard drive for data storage. We recommend 200 to 500 GB of disk space.
Note: Immediate Insight is write intensive. Choose a datastore with good performance.
2. Store the virtual disk as a single file. 3. Click Don’t Allocate All the Disk Space Now. 4. Set the RAM and CPUs. FireMon recommends 32 GB of RAM and 8 virtual CPUs.
Note: The minimum required is 16 GB of RAM and 4 virtual CPUs. Reserved RAM is also recom- mended.
5. In Network Adapter, select Bridged. 6. Start the VM. Step 4: Log on from the VMWare Console 1. Connect to the virtual machine through the console in the VMWare Workstation. 2. Log on to the Immediate Insight CLI. Enter the user name: insight and the password: What- sHappeningNow. 3. Type install and follow the prompts to configure your system. 4. The virtual machine prompts you to install a server or agent. Select server. 5. If you have a SSD drive, type Y and press Enter when prompted. 6. Enter the IP address. If it is dynamic, type dhcp. The virtual machine asks if you want to join a cluster. 7. Type N and press Enter. 8. Type sudo reboot.
Note: The password for sudo is also WhatsHappeningNow.
17 Chapter 1: Installation: Install on VMWare Workstation Immediate Insight User's Guide
9. Login and type update to verify you have the latest version of code. 10. To launch the software, type start. Step 5: Log on from a browser Chrome is the supported browser for Immediate Insight.
Note: If you don’t know your IP address, type ifconfig in the console to retrieve it.
1. In Chrome, go to http://your-ip-address:3201.
Note: To use HTTPS, type set-ssl at the CLI.
2. Log on as user: admin and password: admin. 3. When you are prompted for a license key, cut and paste in text from the .lic file provided by FireMon. Step 6: Connect data sources For more information, see the Immediate Insight User's Guide. 1. Go to Data Flow > Import. 2. Drag a grouping of LOG or PCAP files under Import as Lines or point log streams at the IP address of Immediate Insight using port 514 or 3000. 3. Go to the Search menu, and click Search. 4. Confirm that data appears.
Chapter 1: Installation: Step 5: Log on from a browser 18 Immediate Insight User's Guide
Installation - Non-clustered Installing Immediate Insight on VMware ESXi is the preferred, and typical, method in production Enterprise environments.
Note: For testing purposes, it is also possible to install using VM Workstation.
To install a non-clustered Immediate Insight, complete the following steps. Step 1: Download Immediate Insight OVA and License 1. From the FireMon User Center, download the Immediate Insight OVA file to your computer. The file is approximately 1 GB. 2. Download the Immediate Insight license file your computer. Step 2: Import the OVA into VMWare 1. Download and login to VMWare ESXi Console using vSphere Client. 2. Click File > Deploy OVF Template. 3. Specify the OVA file you downloaded from the FireMon User Center. 4. Select Thin or Thick provision, depending on your available disk space. Step 3: Adjust the VM settings 1. Add a second virtual hard drive for data storage. FireMon recommends 500 GB or higher disk space. 2. Create a new virtual disk. 3. Select Thin or Thick provision, depending on your available disk space.
Note: Immediate Insight is write intensive. Choose a datastore with good performance.
4. Store the Virtual Disk as a single file. 5. Set the RAM and CPUs. FireMon recommends 32 GB of RAM and 8 virtual CPUs.
Note: The minimum required is 16 GB of RAM and 4 virtual CPUs. Reserved RAM is also recom- mended.
6. Start the virtual machine.
19 Chapter 1: Installation: Installation - Non-clustered Immediate Insight User's Guide
Installation - Clustered Immediate Insight virtual appliances can optionally be clustered together, allowing for greater capacity, per- formance, and data collection rates. All the cluster member instance’s resources automatically become avail- able to a common event store, which can be searched and analyzed as if they were a single set of data. Once installed and configured, the common Cluster can be searched from any of the member instances. The following clustered installation instructions make the following assumptions:
l Each Immediate Insight member belonging to a cluster must have a static IP address
l Each Immediate Insight member belonging to a cluster must be on the same version
l Each Immediate Insight member belonging to a cluster must have the same password for the default admin & insight user accounts
l Each Immediate Insight member belonging to a cluster must have the same hostname (for typical vir- tual appliance install this is ‘insight’)
l This is a fresh install (not an upgrade or migration of an existing Immediate Insight system)
l Cluster members are connected with high capacity LAN connections (WAN connections are strongly discouraged)
l For best performance, all cluster member servers should be on the same hardware
Note: If your scenario deviates from any of the assumptions above, please contact the FireMon Immediate Insight support team for assistance before proceeding.
For a clustered installation, complete the following steps. 1. Install the first virtual appliance.
Complete the installation and configuration of your first Immediate Insight instance as a nor- mal standalone system. You must use a static IP. Make note of the IP address configured, you will need this in step 2.
2. Install the second virtual appliance. This time, your installation configuration will specify that the instance is part of a cluster. Joining the IP address of the first instance from step 1.
a. Enter Y to answer the question: Do you want this server to be part of a cluster [N]?
b. Enter the static IP address from Step 1: IP address of any server in the cluster to join:
3. Verify a successful installation
To verify that you installation was successful, use the status command. It should show information about the Cluster and its status, all 5 system processes as “running”, and Data Storage should show 1% used.
Chapter 1: Installation: Installation - Clustered 20 Immediate Insight User's Guide
4. Add additional Immediate Insight Virtual Appliances to the Cluster (if applicable). 5. Apply licenses.
The first time that you log on to the Immediate Insight web interface you will be prompted for a license key. Repeat the licensing process on each of the individual member units.
6. Configure the individual cluster members.
Although the cluster shares a common searchable event store, most items for cluster members still need to be individually configured. Depending on need the configurations can be common, or specific to each unit. However since you can Search the common event storage from any of the member units, you don’t necessarily need to repeat all configuration on each member unit – here are a couple of examples:
l Pinboards / Bookmarks – you could copy these to all the members, but since a Pinboard / Bookmark configured on any of the members has access to all the cluster’s data it isn’t necessary to do so.
l Collectors – you would usually set these up individually on each member for added input rate.
Almost everything in the GUI is configured individually. At the CLI, most things are configured individually. This includes users, agents, certificates, and all manual parameter changes to the server settings. Anything touching the event storage is global. That includes events added from the GUI or from collectors. Repositories, queries, event delete are global. CLI commands such as reset-data- store, new-repo, and delete-repo are global since they touch the event storage. CLI commands for managing the cluster are global: start, stop, update, revert, restart, reload, set-ip, install-tools. All other CLI commands are local. Status and diag are cluster aware because they report some information about the cluster, but focus on the local node.
21 Chapter 1: Installation: Installation - Clustered Immediate Insight User's Guide
Sample Appliance Setup Session A setup session should look something like the following example for a single node installation.
Ubuntu 14.04.1 LTS insight tty1
insight login: insight Password: WhatsHappeningNow
Welcome to the Immediate Insight Management Console
type 'start' to start the server
for a new installation:
'install' to set up the datastore and other con- figurations 'date' to check the time. 'set-timezone' to change timezone if necessary 'sudo reboot' for changes to take effect 'start' to start the server and then browse to http://this-server:3201
use 'set-agent' and 'set-ip' to change settings later 'status' checks the current setup
type 'stop' for a clean shutdown before issuing a shutdown command to the VM
insight@insight:~$ install
New Installation:
Configure this system as an agent or a server [server]? [sudo] password for insight: WhatsHappeningNow Is the data drive an SSD [N}: N (choose Y if using a SSD)
n n
storage volume initialized
IP Address: 192.168.3.103 (or for dynamic addressing type ‘dhcp’) Netmask: 255.255.255.0 Gateway: 192.168.3.1 DNS Server: 8.8.8.8
Applying new IP Address so we can download the proper updates
Chapter 1: Installation: Sample Appliance Setup Session 22 Immediate Insight User's Guide
If you lose connectivity from the IP change, you may need to login again
networking initialized application initialized datastore initialized
Do you want this server to be part of a cluster [N]? N
Installation Complete
Type 'sudo reboot' to reboot
This helps assure that the OS fully applies all IP & disk changes After the reboot, type 'start' to launch the application
insight@insight:~$ sudo reboot
make sure you have the latest version of code
insight login: insight Password: WhatsHappeningNow
insight@insight:~? update
You can now login via the Console or via SSH session:
insight login: insight Password: WhatsHappeningNow
insight@insight:~$ start
starting search - This takes several minutes for large indices...... starting app info: Forever processing file: data-marshal-server.js info: Forever processing file: data-marshal.js info: Forever processing file: data-marshal.js ..
When you use the status command to verify that the installation was successful, it should show all five sys- tem processes as “running” and Data Storage should show approximately 1% used.
insight@insight:~$ status ------System Status - Quick Check ------Immediate Insight version: app-2015-0-0
23 Chapter 1: Installation: Sample Appliance Setup Session Immediate Insight User's Guide
Search engine version: search-1. Personality: server IP Address: 192.168.3.103
Data Marshal: Running UI Marshal: Running Marshal Server: Running Agent: Running Search Engine: Running Search Memory: 16GB
Data Storage (pct used): 1% System Storage (pct used): 23% System Log Storage (pct used): 1%
Total System RAM: 32GB Free System RAM: 10GB Linux version: Ubuntu 14.04.1 LTS
Note: If you see “not running” or no number appears for pct used, this may indicate an error. To troubleshoot, type start. If this does not resolve the issue, contact FireMon Immediate Insight Support for assistance.
Chapter 1: Installation: Sample Appliance Setup Session 24 Immediate Insight User's Guide
Installation - Agent Only The optional Agent Only installation of Immediate Insight allows a slimmed-down version of Immediate Insight to be installed on a remote system such as a laptop or server. The purpose of the Agent is to run remote testing commands (for example, iPerf, port scan, or trace route) from a remote system, and to send the test results to the main Immediate Insight system to make available for searches. Before Beginning Agent Installation
l PuTTY to CLI main Immediate Insight Server that will be used to manage the Agent and receive its data.
l cd /app/config
l cat marshal-settings.conf
l Look for the agent secret line and copy the secret word that follows to notepad or something similar Installing Agent Only on a Remote Machine
Note: A license key is not required for the Agent Only version of Immediate Insight.
1. Deploy Immediate Insight OVA in VMPlayer or VMWorkstation (Open a Virtual Machine) 2. Edit Virtual Machine Settings
l Change VM settings to 1 GB RAM and 2 Cores
l Add another hard drive to the VM. 20 GB is sufficient because you aren’t really storing much here 3. Start the VM (Play Virtual Machine). 4. Log on to the Immediate Insight CLI. Enter the user name: insight and the password: What- sHappeningNow. 5. Type install at the prompt. 6. When prompted change from default install of server to agent. 7. When prompted for the IP address use dhcp (or static IP information if applicable). 8. Accept default NTP servers. 9. Specify the IP address of the main Immediate Insight server being used to manage this agent. 10. Enter the IP Address of the main server. 11. Accept default port 3205 12. Enter the secret word that you copied to notepad previously. It may or may not already be the same for both systems, if it’s the same you don’t need to do anything. If not, type the correct one.
Note: Because you are in the CLI, the copy and paste function will not work. You'll need to type in the secret word.
25 Chapter 1: Installation: Installation - Agent Only Immediate Insight User's Guide
13. Specify the name for the remote agent. 14. Type the command sudo reboot to reboot the system. 15. Log back into Agent. 16. Type the start command.
Note: If you receive an error message that the secret word is incorrect, then PuTTY to the agent, then use the command set-agent to correct. After correcting, type reload agent.
Log into Immediate Insight Server GUI, in order to use remote agent
Note: You will need an Immediate Insight Server account with admin privileges. Once installed and star- ted, there is no reason to log on to the remote agent CLI or GUI.
1. Go to DataFlow > Remotes. You should see your additional remote agent listed.
Note: You won’t see it unless the remote agent VM is online, and has IP connectivity to the Imme- diate Insight server, and then Immediate Insight processes are started on the remote agent using the start command.
2. Click the + icon to add commands. 3. Click the play button to run entered commands. Results will show in Search on the main Immediate Insight Server GUI.
Chapter 1: Installation: Log into Immediate Insight Server GUI, 26 in order to use remote agent Immediate Insight User's Guide
Sign In to Immediate Insight To log in to Immediate Insight, complete the following steps. 1. Open a browser.
Note: Google Chrome is the supported browser of Immediate Insight.
2. Connect to http://IP-address-of-server:3201 3. On the Immediate Insight Sign In page, enter the following information:
l User: admin
l Password: admin
Note: You can change this password from the Settings menu after you log in.
3. Click Sign In.
For Evaluation Users After logging in, you may be prompted to enter a license key. If you are prompted to enter a license key, complete the following step.
l In the Evaluation License Expired dialog box, click Enter License Key, paste the entire contents of the text from the .lic file provided by FireMon in the Activation Code field and then click OK.
27 Chapter 1: Installation: Sign In to Immediate Insight Immediate Insight User's Guide
About License Keys
Note: Only an admin user can activate a license.
Immediate Insight ships with a free base configuration which allows concurrent storage of 25 million events for active search. For this configuration you do not need to apply a license key. To purchase a higher volume event search license, please contact Immediate Insight Sales, iisales@fire- mon.com. If you have purchased a higher volume license, you will need to: 1. Download the license (file name ending in .lic) from your account in User Center (https://user- center.firemon.com ) to your computer. 2. Open the license in a text editor, such as Notepad. 3. Copy the entire text of the license key file to your clipboard. Activate a License Key To activate a license key, complete the following steps. 1. On the Immediate Insight toolbar, click the Settings icon, and then click Activate License. 2. In the Welcome to Immediate Insight dialog box, enter the activation code in the Activation Code field that you copied previously. 3. Click OK. View Current License Limits To view the event search volume limits that are currently licensed, complete the following steps. 1. Go to DataFlow > Monitor. 2. In the Search Engine > Data section, you will see License Limit in the Current Records field.
Sign Out of Immediate Insight
To sign out of Immediate Insight, click the Settings icon , and then click Logout.
Chapter 1: Installation: About License Keys 28 Chapter 2: Administration
Key Administrative Tasks 30 How Immediate Insight Communicates 31 Security 32 About Users 33 Data Repositories 34 Manage Data Flow 35 Data Collection 35 Work with Email 40 System Management 43 Configuration and Diagnostic Files 46 About the Data Router 47 Manage Routes 52 Remote Agents 53 Workflow Tracking 58 Custom Action Scripts 70
Immediate Insight User's Guide Key Administrative Tasks There are a few post install administrative tasks required in all Immediate Insight implementations.
l Ensure appropriate TCP/UCP ports are open on the network so that necessary traffic can pass to and from Immediate Insight. Reference How Immediate Insight Communicates.
l Add at least one account with user level permissions, assigning a private data repository to that user, so that the user can search their own data separately from other data if desired. Reference About Users.
l Configure Immediate Insight to collect data to be analyzed. Reference Data Collection.
l You should also familiarize yourself with common system administrations commands. Reference Sys- tem Management. Optional administrative tasks such as Data Routing, Remote Agents, Workflow, and Custom Action Scripts are also covered in the Administration chapter.
Immediate Insight User's Guide How Immediate Insight Communicates Open Ports By default, the appliance accepts connections on the following ports:
l For management: SSH traffic on port 22
l For client access: HTTPS traffic on port 3201 and 3801 Additionally, the application has configurable listeners to receive data that default to the following ports:
l Syslog: 514
l Netflow: 2055 (netflow)
l TCP / UDP: 3000
l Data over HTTP: 3001
l Data over HTTPS: 3002
l PCAP stream: 3003
l 3203 API Access REST API access uses the same HTTPS and port number settings as the user interface. The API is enabled by default, using an API key of “api-key”. You can change the API key, add more, or disable the API.
Note: For more information about using the API, please refer to the API chapter.
Immediate Insight User's Guide Security Enabling Encryption Immediate Insight streams data to the client by opening two websocket connections to the browser, a control channel and a data channel. By default, Immediate Insight is configured for HTTP. To activate encryption (HTTPS) on websockets, complete the following steps. 1. In the CLI, type set-ssl to enable encryption on browser sessions. 2. Type reload server to make changes take effect. 3. Exit the browser and re-login using HTTPS instead of HTTP (https://ip-address-of-server:3201)
Note: You may receive a certificate warning message, but you will be able to login after ignoring it.To prevent the warning message, refer to the steps to activate a CA certificate.
Certificates FireMon recommends the best practice use of matching CA certificates installed in your browser to reduce the possibility of man-in-the-middle attacks and provide a smoother user experience. During installation, a self-signed rootCA pair is generated automatically in app/config/certs.
Note: You can replace this pair with your own CA by overwriting the rootCA.key and rootCA.pem files, however the self-signed certificate provided is adequate.
To activate the certificate, complete the following steps. 1. In the CLI, type set-certs followed by reload server. 2. Copy the app/config/certs/rootCA.pem file from the Immediate Insight server to your com- puter using an SFTP client. 3. Load the certificate into your browser.
l In the Chrome browser, go to Settings > Show Advanced Settings > HTTPS/SSL > Manage Certificates > Trusted Root Certification Authorities > Import.
l Specify the rootCA.pem file. 4. Restart the browser.
Note: While the system has a reasonable set of security measures in place, the present release is designed to run in a secure and trusted environment. If you have a need to expose it directly to the Inter- net, please call FireMon Immediate Insight support to discuss additional hardening procedures.
Immediate Insight User's Guide Immediate Insight User's Guide
About Users Administrative tasks are divided such that security and system management tasks are handled from the CLI and data management is handled from the web interface. The admin account for web access is not used for the same tasks as the insight SSH administrative account. User Types In Immediate Insight, there are three types of users:
l System Administrator—is a CLI account and is only used for administrative tasks such as user man- agement. This account is only available from the CLI and does not work to access the web interface.
l Data Administrator—can perform data management administrative tasks in the web interface. This account does not have access to the CLI.
l Data User—cannot perform any administrative tasks. Only has access to the web interface. Additional user accounts can be created and deleted with the add-users tool within the CLI.
Add a User To add a user, complete the following steps. 1. At the command prompt type: add-user –n
-n or –u or –username username
-i or –index
-p or –password password
-g or –group group name [default = user] (user admin for administrator rights)
-r or –repo private/scratch repo [deault = user-shared]
-a or –acl data rights [default = *] (example: +logs, -priv*)
-c or –config location of the users.conf file [default=config/users.conf]
-l or –list list all users
--delete delete a user (example: user –u
Chapter 2: Administration: About Users 33 Immediate Insight User's Guide
Data Administrator Accounts To give a user Data Administrator privileges type: add-user -u
Replacing
CLI Admin Password Management The SSH password for the CLI administrator can be changed by typing passwd
Note: Data Administrator and Data User account passwords are changed from within the web interface in the Settings option.
Data Repositories Incoming data is stored in repositories. All users have access to the default repository: main-stream. Addi- tional repositories can be created or removed using the Immediate Insight SSH account. Add a New Repository To create a new repos, at the command prompt type: new-repo
l new-repo reponame
l delete-repo reponame
Note: Special characters are not allowed as part of the name.
Permission to access the newly created repository must be set for each user who will be working with it. First, get the list of repositories the user has access to (shown in the private repo field):
add-user -list
The default list is main-static,main-stream. Finally, add the new repository to the list shown in the out- put since the user may have access to other private repository.
l add-user -u username -r 'main-static,main-stream,reponame’ If you delete reponame, user permission needs to be adjusted by setting the list with the repo removed:
l add-user -u username -r 'main-static,main-stream'
34 Chapter 2: Administration: Data Administrator Accounts Immediate Insight User's Guide
Manage Data Flow There are four main systems for working with data-in-motion. Data flow is managed from the DataFlow page. Many of the features within these areas may require an account with administrator privileges.
l Collectorsare used to get data in.
l Data Routing is used for processing data as it arrives.
l Workflow is used for chain analysis of machine and team data.
l Reputation is used for learning from new data.
Data Collection Getting Data into the System There are two modes for adding data to the system streaming and interactive.
l Streaming data feeds, such as syslog and log files, are managed on the DataFlow > Collectors page. The system can get streaming data from following log files on a network share and by receiving data feeds over a network socket. Collectors can only be configured by an administrative level user.
l Interactive input, such drag & drop and cut & paste, is handled from the DataFlow > Import page. Imports can be done by either users with administrative or user level privileges. Hence this manual Import method is documented in the Usage Guide Chapter of this guide.
Set Up Data Collectors The default settings for the virtual appliance include several data collectors pre-configured. Data collectors can be viewed by clicking DataFlow > Collectors. Standard Collectors
l Socket Listener (TCP / UDP) is already running on syslog (port 514) and port 3000 and listening for streams on those port numbers. For example, if you point your device syslog destination to point to Immediate Insight’s IP address. You can also send in data ad hoc from remote machines using a tool like netcat (on the remote machine) to send log files over the network to Immediate Insight.
cat logfile | nc ip-of-collector 3000 or cat logfile > /dev/tcp/address-of-collector/3000
l Netflow Collector is already running on port 2055. It will auto-detect the traffic type and can handle Netflow 5, Netflow 9, Cisco ASA, IPFix, and Bluecoat Packeteer-2 Formats.
l A log file listener is already configured to watch files in the directories /media/logs, /media/logs1/ and /media/logs2. If you mount your logs at those directories or link to them from there, it should begin
Chapter 2: Administration: Manage Data Flow 35 Immediate Insight User's Guide
picking up automatically. To permanently mount a remote server’s network share 1. Edit Immediate Insight’s /etc/fstab and add a line similar to this:
//servername/sharename /media/logs cifs username=user,passwd=pwd 0 0
Note: You will need to know a valid username and password for the remote server share you are mounting.
2. Use the command sudo mount –a to activate it. To temporarily mount a network share (mount will be lost if the appliance reboots) 1. Use the command: sudo mount –t cifs //servername/sharename /media/logs -o user=username,uid=1000,gid=1000
2. You will be prompted to enter a password.
Note: CIFS support is enabled by default, but since the appliance is just standard Ubuntu 14.04 Linux, you can use sudo apt-get to install drivers for other file system types and use other mount methods if you prefer.
l If mount directory does not already exist on Immediate Insight, you will need to create it first and allow write permissions: sudo mkdir /media/logs sudo chmod a+rw /media/logs
l CIFS is the default mount type, but NFS is also supported. Since the appliance is just standard Ubuntu 14.04 Linux, you can use sudo apt-get to install drivers for other file system types and use other mount methods if you prefer. vi cheat sheet for editing system files: i to insert, esc when finished, :wq to save, :q! to quit w/o save
Add a New Data Collector In addition to handling syslog, you can also configure Immediate Insight TCP/UDP Listeners on different port numbers. Systems such as Firewalls, IDS, Network etc. can often be configured to stream data out to Imme- diate Insight’s IP address on specified port numbers. To add a new data collector, complete the following steps. 1. Scroll to bottom of DataFlow > Collectors (Inputs) and click the Add icon +. 2. A menu of collector choices will appear. 3. Select one to open a configuration dialog box (based on your selection) that will allow you to specify information or accept the default settings.
Chapter 2: Administration: To permanently mount a remote 36 server’s network share Immediate Insight User's Guide
l The collector Name is used as part of the source tag in the index, so a meaningful name can help with grouping data in searches.
l Repository allows the selection of private repos. The –stream suffix simply means it accepts data from collectors. The list of available repositories is refreshed only when the dialog is first opened.
l The Port used for access.
l Rate Limit helps protect the system from explosions if a source becomes misconfigured and emits excessive data. 0 means unlimited. 1000 would enforce a limit of 1000 records per second for this collector. This can also provide a type of sampling for excessively chatty and repetitive sources.
l Data TTL is the amount of time data for this collector is saved, 3d is the default – but longer time to live (TTL) is often possible depending on data input rates and system capacity.
l Multline is used in special circumstances only under direction of support (you can leave blank)
l The Allow Mirroring check box is used when you wish to allow a copy of the collected data to be retrieved by another Immediate Insight Collector. 4. Click Add.
Note: The Edit Dialog enables you to later customize the settings of the collector
Additional notes about adding a data collector:
l You can also add other types of collectors, for example; File Listeners for other directories, Mail col- lectors, Web collectors.
l As another example of Adding a Collector, if you need to collect Shares other than /media/logs, select File Listener.
l The file watcher is filename-aware, so if a log file is automatically rolled over, renamed or replaced by a new file with the same name, the file watcher will start following the new file of the same name.
Chapter 2: Administration: Additional notes about adding a data 37 collector: Immediate Insight User's Guide
l Use /* at the end of the path to watch an entire directory, such as /var/log/* The directory follower will automatically detect and follow new files as they are added to the directory.
l Wildcards are supported in file names, such as /media/logs/*.log
l The collector name is used as part of the source tag in the index, so a meaningful name can help with grouping data in searches. It is also used to help specify data routes.
l The Show Files button shows a sample of the files that this collector will follow, helping to confirm that the path is correct and active. The collector will actively follow files that have had recent activity (shown in black). It will also passively detect any new or newly re-activated files and then begin act- ively following them.
l Mode: the default is to ‘Follow’ file which is like a Unix tail, new lines are read into Immediate Insight as they are appended to the file. You can also choose the 'Consume’ mode which will read in new files in their entirely once.
Immediate Insight Collector The Immediate Insight Collector is a special type of Collector, used when you want to retrieve a copy of data collected on another Immediate Insight system (e.g. for a Hot Standby copy of the data). Example: On remote Immediate Insight system 192.168.1.117 you previously configured one or more it its Collectors with the ‘Allow Mirroring’ option enabled. Next on your other Immediate Insight below configure at follows. 1. DataFlow > Collectors 2. Scroll to the bottom of the Collectors Column, click the + icon. 3. Select Immediate Insight Collector. 4. Complete the configuration dialog box.
5. Click Add. The Immediate Insight Collector will pull a copy of collected data from the allowed collector (s) on the remote Immediate Insight system specified by the Server IP address. Data will be collected into the local Immediate Insight Collectors on the specified Server Port.
38 Chapter 2: Administration: Immediate Insight Collector Immediate Insight User's Guide
Set Data Retention Click on a collector in the collectors column and adjust the settings in its TTL field as needed. 3d (3 days) is the default. Use h or d to abbreviate hours or days. (24h, 30d, etc.) Data retention is set on a collector-by collector basis. Click on a collector from the DataFlow > Collectors > Input Column. Set Data TTL to an appropriate value. For example “3d” would set records that come in through this collector to be automatically deleted after 3 days (72 hours): You can see the volume of data currently being stored on the DataFlow > Collectors page in the Real-time Search Index box in the Outputs column. This will add an entry to the Data Routing table that you can subsequently manage or edit from the DataFlow > Data Routing menu.
Managing Data Immediate Insight is about working with real-time and current data. Real-time search data tends to decline in value rapidly, there is typically no need to keep it around in the search index for very long. To keep the index sizes under control, the system will automatically purge old records. You may separately have a log management or data warehouse application for audit and archiving purposes. Normally you set a records retention policy that expires data after a few days to keep the volume within the limits of the hardware it is running on. While storage and CPU capacity are important, RAM is usually the limiting factor for real-time search. For example, 32 GB of RAM is typically sufficient to handle up to 1 billion active records. You want to set a data retention policy that keeps the total volume of active records under the threshold of the current system. For example with this amount of memory, if the incoming volume of traffic is around 100 million records a day, then 7 days is a good expiration policy. If it is closer to 10 million records daily, then 60 days is probably a good expiration policy.
Note: In this release, the system still requires some manual tuning of expiration policies to keep volume under control. In the future we expect this process to be increasingly adaptive and automatic.
Deleting Old Data To bulk delete old records that are already in the system under a different TTL, complete the following steps. 1. On the Search page, click the time frame menu and select a Custom Time Period. 2. In the Custom Time Selection dialog box, enter a time range for From and To, and then click OK. 3. Click the Search Results menu and select Delete Result Records. 4. Confirm deletion of the records, click OK. 5. Click Search, then Delete Result Records from the results .
Chapter 2: Administration: Set Data Retention 39 Immediate Insight User's Guide
Work with Email The system offers two options for email:
l Inbound: listen to email accounts as a source of input data
l Outbound: send email when an alert condition is triggered Inbound Email To poll an email account, such as a notification mailing list, complete the following steps. 1. Click DataFlow > Collectors. 2. Scroll to the bottom of the Input column and click the Add icon +. 3. Select Mail Collector. 4. Configure it for your email system. Substitute server information relevant to the POP email account you wish to collect emails for.
5. Click Add.
Email Alerts Setting up Email Alerts To set up an email alert, complete the following steps. 1. Before you can setup email alert data route actions, the server settings for sending email must be spe- cified in the marshal configuration file. You can edit this file with a text editor, such as vi app/- config/marshal-settings.conf.
40 Chapter 2: Administration: Work with Email Immediate Insight User's Guide
app/config/marshal-settings.conf
2. Add the following lines, using the values for your email environment.
#
# maximum number of email alerts per address per hour
#
mail.maxPerHour = 3
#
# mail alert settings (account to send through)
#
mail.username = "[email protected]"
mail.password = "test"
mail.host = "smtp.gmail.com"
mail.port = 587
mail.subject = "New Alert"
Note: The default maximum setting is 3 messages per hour. You can boost this to a larger number as needed.
3. After saving the changes, type restart at the command prompt to make the changes take effect.
Outbound Email To create a new outbound email route, complete the following steps. 1. Click DataFlow > Data Routing. 2. Click the Add New Route icon +. 3. In the Action box, type the word mail. 4. Click the Edit Settings button to open a mail settings dialog box.
Chapter 2: Administration: Outbound Email 41 Immediate Insight User's Guide
5. Complete the fields. 6. Click Send Test Message to verify that the system was able to find a suitable mail server.
42 Chapter 2: Administration: Outbound Email Immediate Insight User's Guide
System Management Command Reference For most installations, start and stop may be the only command prompts you need after initial deployment, but the system does provide other commands for customization. The appliance is a stock installation Ubuntu 14.04 LTS Server. You can connect to it with SSH and perform standard Linux tasks to change passwords, mount data, etc. Immediate Insight adds the following system commands, which you can type at the Linux prompt after con- necting using SSH or the VMWare console.
Note: These commands only work when logged in as the user ‘insight’. They will not work if you change to 'root' or another user name.
Available Commands Command Function
start Starts both the search engine and the application
start-search Starts only the search engine areas of the application
start-app Starts only the data collector areas of the application
start-agent Starts only the remote controller area of the application
reload server Restarts the server daemons
reload agent Restarts the agent server
reload actions Reloads the actions from the app/config/actions directory
stop Stops both the search engine and the application
stop-search Stops only the search engine
stop-app Stops only the data collector
stop-agent Stops only the remote controller area of the application
install Complete initial setup of Immediate Insight
set-certs Load in new root CA. The change takes effect after the next restart.
set-memory Auto adjusts the amount of memory allocated to the search index
set-time Will force the system to try updating its time using NTP
set-timezone Change the system timezone
Used if Immediate Insight must access the internet via a proxy server. Use this com- set-updateproxy mand before running ‘update’
Chapter 2: Administration: System Management 43 Immediate Insight User's Guide
Available Commands Command Function date Change the time and date, if not set automatically set-ip Sets the IP address, will take effect after reboot set-ip dhcp Enables DHCP, useful in a configure remote environment
Activates an interface in promiscuous mode so it is ready for packet capture use set-capture from the agent
Installs and configures Samba to enable publishing a CIFS share that other hosts set-dropbox can mount and write files to. reload server Restart the application, equivalent to stop-app + start-app reset-settings Resets the collector and other setting to factory default reset-datastore Resets the add-user Create new users, change passwords for existing user (add-user-h) ii Searches from the comman prompt
? List of all CLI commands new-repo Creates a named repository for data collection and organization new-repo
Utility for installing/configuring/enabling (apt-get) Linus packages for use with agents add-tool add-tool
A convenience wrapper for apt-get update && apt-get upgrade which upgrades all add-tool update packages on the Linux system version Display version update download and install latest version of Immediate Insight code
revert : rollback to second newest version of code on box revert latest : roll forward from a revert back to the latest version revert revert show : list available versions revert app-09-23-2012 : move to specified version status Quick summary of the current system state
44 Chapter 2: Administration: Command Reference Immediate Insight User's Guide
Available Commands Command Function
Commands for system health checks show status: summary of system status show show issues: - list recent critical errors from the log files show settings: view of any custom settings in the config files. Makes JSON files human readable for better debugging.
firewall show: display current port blocks firewall clear: clears port restrictions, allowing firewall By default, the appliance blocks access to instances of immediate insight not run- ning on the local host. To expand from a single appliance to multiple appliances you will need to remove these limits with the firewall clear command, or put more spe- cific firewall rules for your environment.
Chapter 2: Administration: Command Reference 45 Immediate Insight User's Guide
Configuration and Diagnostic Files Config Files The app/config directory holds config files. There is no reason to change these for most installations, but if you need to customize behavior, they can be edited with a text editor. If settings become corrupted, you can restore them to default with the command:
reset-settings -a
This will reset the settings.conf and collectors.conf files; erasing your current collector settings. To make the changes take effect, type
reload
The previous settings files are stored in app/config with a .old extension in case you wish to recover data from them or discuss them with support.
l reset-settings –u will reset the users file, erasing all of your users and setting up the default admin:ad- min account.
l reset- will erase all of the data in the search engine and restore it to default. Diag Files A system diagnostics report can be downloaded from DataFlow > Collectors > Ouputs > Diagnostics > Download. This may help support with troubleshooting if you run into problems.
46 Chapter 2: Administration: Configuration and Diagnostic Files Immediate Insight User's Guide
About the Data Router The Data Router: Processing Data as it Arrives Each time a new event record arrives at the Immediate Insight server, it is immediately processed through the data routing table. Much like the packet routing table on a firewall or ACL list on a router, the Data Router lets you create sophisticated policies for how each event should be handled. Route Actions can do a variety of things including:
l Alert (using email or in the UI)
l Drop the record (spam)
l Auto-tag or rewrite the record (making it more searchable and readable)
l Learn specific facts from the event (such as mapping a user name or IP address to an email address)
l Decorate (add previously learned facts to this record as metadata)
l Feed the record back out (creating custom data feeds)
Note: See the route action list for a more detailed list. It is also possible to create custom actions for very specific behaviors or handling unique sets of data using a simple scripting language.
With data routes you can specify standard handling of incoming records, such as auto-tagging, deleting (spam), and triggering alerts. You can also work with metadata to learn or add additional insight to a new record. You can also create custom action scripts that do more complex combinations tasks. Data routes are found in DataFlow > Data Routing. A summary table is displayed showing the routes presently active. Routing only happens when a record first arrives. It is not later applied to search results.
Quick Routes Quick routes provide a way to match and group similar records. The quickest way to setup a new route is from the inline menu under an event in the search results.
To set a quick route from the inline menu, complete the following steps. 1. On the Search page, hover over an event to open the inline menu. 2. From the Action menu, select an action item:
l Spam
l Action
Chapter 2: Administration: About the Data Router 47 Immediate Insight User's Guide
l Auto-tag 3. An Actions dialog box will open.
l Spam: In the Block Unwanted Messages dialog box, a. The Action option is selected by default. b. Select a What option. c. Enter a text string to match or a comma-separated list of terms. d. Click OK.
l Actions: In the Take Action on Messages Like This dialog box, a. Select an Action option. b. Select a What option. c. Enter a text string to match or a comma-separated list of terms. d. Click OK.
l Auto-tag: In the Automatically Tag Messages Like This dialog box, a. In the Add these tags box, enter key words (include the hashtag symbol # before the word). b. Select a To What option. c. Enter a text string to match or a comma-separated list of terms. d. Click OK.
Note: You can add any additional qualifiers (AND or NOT) or just enter a term to match (instead of the more-like-this signature).
Create a Data Route To create a new data route, complete the following steps. 1. Click DataFlow > Data Routing. 2. Click the Add icon +. 3. Complete the Edit Data Route dialog box. The route editor will let you specify rules for what traffic to match and then specify an action to be taken on traffic that does match. a. Enter a Name for the data route. b. Stage defines the order of execution. Available stages are “arrival” (happens first thing after the record arrives, no metadata is available), “pre” (after the record has been tagged with source info, but before other metadata is added), “post” (after standard metadata has been added), and “final” (last thing, after the record has already been stored in the search index). Leave blank for automatic selection.
48 Chapter 2: Administration: Create a Data Route Immediate Insight User's Guide
c. Expires defines when the system should auto-delete this filter. Often you may need an alert or action for only a few days while you work on a specific project. Setting an expiration helps keep the routing table clean and svelte. You can use shorthand to specify an expiration time, such as “+3d” for three days from now or “sat” for next Saturday. d. Match is the condition that the new event has to match to invoke this route. For example, entering the string “error” would cause this route to match all events with the word error appear- ing anywhere in their text. e. Match Field is the field to match against. If you leave it blank, the default is “text”, which is the text of the original event. You can view other available fields by clicking on metadata in the inline menu of the search table. Some other interesting fields are “sourceName” and “tag”. “sig” is the fingerprint of the event, enabling you to match on similar records. f. Qualifiers are additional criteria to further narrow down a match. Similar to how you would con- struct a search query, you can use NOT or AND clauses to refine the match. Mouse over a field to see help with more details. g. Action is what to do with the event if it matches this route. See the action table below for avail- able options. Note that (unless a route instructs otherwise), each new event will be checked against all routes and can match more than one. Hit down arrow in this field to see a list of all the actions presently loaded and available in your system.
l Alert – generate alert within Immediate Insight (or use Quick Route)
l Feed – write search results out to specified file on drive mount (see Custom Feeds)
l FireEye – add additional FireEye metadata when FireEye logs present
l HL7 – add additional HL7 metadata when HL7 logs present
l Mail – send email out for specific search results (see Outbound Email)
l PaloAltoTraffic – add additional Palo Alto metadata when Palo Alto logs present
l RemoteAgent – call Remote Agent and execute specified Command, for example Packet Capture (see Remote Agents)
l Spam – mark search result as spam (or use Quick Route)
l Tag – add tagging metadata to specified searches (or initiate from Search Results)
l Vectra – add additional Vectra metadata when Vectra logs present 4. Click OK.
Note: It is also possible to create custom action scripts.
Chapter 2: Administration: Create a Data Route 49 Immediate Insight User's Guide
Custom Feeds The Data Router lets you re-stream selected records to disk or the network to create your own custom feed. Data routes act on new records immediately as they arrive. In this example, all records that have been tagged with #SMTP will be written to a file called myCus- tomStream.log on the network share that was mounted at /media/logs. You could use the command autoTagging to tag the different types of records you want to go into the feed. Instead of using the Match Field, another option would be to match using only the text of the record to directly select all records in the Match box, for example any records containing the word SMTP, leaving the Match Field box blank. You can have more than one route sending data to the same feed.
The feed action takes one property – the name of the file to write to. Use the path where you mounted the net- work share you wish to write to, such as feed(“/media/logs/myFilename.log”) . Also, streamUDP for Data Routing Action can be used to stream selected data over the network to third party servers or another Immediate Insight server. This feature requires Immediate Insight app-2015-10-07 or newer.
Create a Custom Feed for a Data Route To create a custom feed, complete the following steps. 1. Click DataFlow > Data Routing. 2. Select an existing route from the list or create a new data route. 3. In the Edit Date Route dialog box, click Edit Settings. a. Enter a Name for the data route. b. Enter the Match criteria of events that you want to stream out.
50 Chapter 2: Administration: Custom Feeds Immediate Insight User's Guide
c. Enter an Action, such as streamUDP. d. Click Edit Settings. e. Enter the IP address and port of the remote Immediate Insight or other system receiving the data. f. Click OK. g. Click OK. On the remote server, the selected events appear. For Immediate Insight you must have a UDP collector configured to match the specified port (if you use port 3000 it is there by default).
Note: Only UDP is supported to insure minimal impact on performance. You can collect data as TCP, then stream back out as UDP. Incorrect destinations have little impact.
Note: Events larger than the MTU will be not stream out due to a UDP standard datagram limitation.
Edit a Data Route To edit a data route, complete the following steps. 1. Click DataFlow > Data Routing, and select the route to edit from the list. 2. Make the needed changes. 3. Click OK.
Delete a Data Route To delete a data route, complete the following steps. 1. Click DataFlow > Data Routing, and select the route to delete from the list. 2. Click Delete.
Chapter 2: Administration: Edit a Data Route 51 Immediate Insight User's Guide
Manage Routes You will need to be logged on with an account that has administrator privileges for many route management tasks. The column on the right hand side of the routing table displays the number of hits. Clicking the Refresh but- ton (immediately above the word hits) will give you an idea if traffic is successfully hitting your route. In many cases you can generate test traffic to trigger the route simply by typing a few words or pasting a few lines of data into the Scratchpad. Using alert(admin) as the action will send all of the resulting hits to the alert feed on the Search page which will let you view which traffic is hitting your routing criteria. Once you are satisfied that you are capturing the traffic you want, you can simply edit the route and replace the alert action with another action.
52 Chapter 2: Administration: Manage Routes Immediate Insight User's Guide
Remote Agents Immediate Insight can initiate and capture the results of many common network and security analysis products. Any program that can be configured to run from the command prompt or shell script can be initiated by Immediate Insight, either manually or automatically. A copy of the output is streamed in and correlated with other infrastructure data.
There are two types of command structures, streaming and request-response. The output for a streaming command, tshark for example, is sent to a TCP port collector. A request-response command that outputs data in a blob, like nmap, is sent to an HTTP collector. There are default remote agent collectors configured for TCP port 3001 and HTTP port 3002. If you want to send the TCP or HTTP output to another port, a new TCP or HTTP port will need to be added for the required port(s). The following are examples of the command structures and commands for streaming and request-response programs. Example of Commands using Streaming Format Structure Packet Capture: (echo '@@sourceFile:tsharktag'; sudo tshark -b filesize:5000 -b files:5 -w /tmp/tsjunk -t ad -T fields -e frame.number -e col.Time -e col.Source -e col.Destination -e col.Protocol -e col.Length -e col.Info -E header=n -E separator=, -E quote=d '(host @@a- gentIP)') |nc @@serverIP:3001;sudo rm /tmp/tsjunk*
(echo ‘@@sourceFile:< tag >’; sudo < command with any options >) | nc@@serverIP: Chapter 2: Administration: Remote Agents 53 Immediate Insight User's Guide collector port > Examples of Commands using Request-Response Structure NMAP nmap
Chapter 2: Administration: Examples of Commands using 54 Request-Response Structure Immediate Insight User's Guide
Example of Command using Request-Response Format Structure for Workflows NMAP
nmap -vv @@workflowInstanceVal |tee /tmp/nmapjunk | curl -s -XPOST 'http://@@server- IP:3002?- workflow=@@workflowVal&workflowInstance=@@workflowInstanceVal&sourceFile=req_ ls' --data-binary @- >/dev/null;cat /tmp/nmapjunk;rm /tmp/nmapjunk Variables @@serverIP—the IP of the Immediate Insight virtual appliance managing this agent. @@agentIP—the IP of the machine running this agent. @@workflowInstandVal—the workflow instance (i.e. IP address). @@sourceFile—for tagging of command output data echo’@@sourceFile:
For example, the metadata value for "source" is passed as @@data_source in the command prompt. When executed, the shell command will see the value "Drag & Drop Uploads (Blobs)" in place of the variable name @@data_source.
Chapter 2: Administration: Example of Command using 55 Request-Response Format Structure for Workflows Immediate Insight User's Guide
Command Invocation Options Start command manually from Immediate Insight UI. Start command automatically from Workflow or Data Router.
56 Chapter 2: Administration: Command Invocation Options Immediate Insight User's Guide
Output Options For request-response programs, output can be sent to the UI by selecting stdin and stderr.
Note: Streaming output to the screen is not supported.
Additional configuration guidelines:
l Interactive commands are not supported and will hang execution when it waits for input. The stop but- ton will recover from this hang condition. All commands must be non-interactive so they can be launched in an automated fashion.
l All shell commands within a command prompt must respond to the TERM signal to gracefully exit.
l Standard shell script rules apply, and the order of commands determines order of execution within the command prompt.
l Only one non-returning command per command prompt is supported.
l Care must be taken to clean up any generated temp files at the end of the command prompt to avoid running out of space.
l Commands requiring sudo must be added to the sudo file.
l Run-time error checking and recovery is the responsibility of the one writing the command. No val- idation or sandboxing is performed at run time.
Chapter 2: Administration: Output Options 57 Immediate Insight User's Guide
Workflow Tracking Workflow tracking correlates team (people) or device (machines) driven business process states with the raw event data. This enables automatic real-time extraction of actionable information from the raw event streams that are captured by II. How it works
1. Raw events are collected from team and machine sources. 2. Data of interest is sent to the workflow instances for further analysis. New instances are created when the input is recognized as the first transition in a defined workflow. 3. Workflow instances generate events based on the raw data that are interesting to the organization. These events are collected just like any other events so they can be searched/audited/ acted on/ana- lyzed just like any other events. Workflows themselves are defined by the administrator to model the steps of business or machine process.
58 Chapter 2: Administration: Workflow Tracking Immediate Insight User's Guide
Overview The steps to enable workflow tracking are:
l Define a workflow to be tracked using the workflow editor.
l Add actions to consume the workflow generated events (optional). Email actions can be used to provide interactive participation within a workflow. The steps to use output from a workflow being tracked are:
l Admin communicates specific event keywords that should use when searching for tracked work- flows, alerts that were defined, and workflow generated emails to all II operators.
l Operators should take appropriate action to follow up on any received workflow events, alerts or email. Workflow tracking greatly reduces the noise that operators usually face when monitoring large amounts of data by allowing them to focus on the relevant key events affecting business impacting workflows.
Workflow Linking and Interaction It is possible to use the output of one workflow as input to another workflow. This advanced technique is use- ful for extracting aggregate or higher level information out of one or more workflows. In this example, an expense report process is being monitored by another process that tracks response time. In addition, the manager can remotely interact with the monitor workflow based on the current status. Equipment Orders Workflow This is the primary workflow process being tracked.
In this workflow, the Review Pending and 2nd Level Review states have times set.
Chapter 2: Administration: Overview 59 Immediate Insight User's Guide
In both states, the …active longer than event triggers an alert so the team knows they are running behind. If the …exited later than time passes, the Performance Monitor workflow is triggered.
Performance Monitor Workflow In this workflow, the manager wants to know about slow exit times and take action to resolve the issue.
60 Chapter 2: Administration: Performance Monitor Workflow Immediate Insight User's Guide
The transition for slow review is triggered by the output of the Equipment Order workflow.
In the advanced options (accessed by clicking More after the Match Field), there is an option to allow work- flow events as input.
Chapter 2: Administration: Performance Monitor Workflow 61 Immediate Insight User's Guide
In this case, the event from the Equipment Orders workflow is specified as the Match value.
To alert a manager of a slow equipment order, the monitor workflow is configured to generate an event when the Slow_Review_Seen state is entered.
A data route is configured to send email to a manager when the Slow Review Seen state is entered.
62 Chapter 2: Administration: Performance Monitor Workflow Immediate Insight User's Guide
The From field is the reply address account that Immediate Insight is polling for input. The original event must be included since it contains the unique ID used by the monitor workflow.
To allow the manager to reply back to Immediate Insight, the mail data collector has been configured to receive mail:
The performance monitor workflow has transitions that are configured to match keywords (warning, dis- missal) in the manager’s reply. The instance ID is picked up from the original event that was included in the email (using @@text).
Chapter 2: Administration: Performance Monitor Workflow 63 Immediate Insight User's Guide
Use a Workflow The most direct way to use the output of workflow tracking is to search for the events that were generated. The Instance ID for workflows is always enclosed in the number sign (#). In this example, the instance FRM:123 was moved to the form_submitted state. Metadata and a tag are added to the event automatically. Searching on the tag will return all events related to that particular instance.
Add a Workflow To add a new workflow, complete the following steps. 1. Go to the DataFlow > Workflow page.
2. Click the Add icon . 3. Complete the Workflow Properties dialog box. a. Workflow Name cannot be changed after adding. b. Workflow ID is automatically generated based on the workflow name entered. The ID is used in generated events. c. Description of the workflow, this is optional.
64 Chapter 2: Administration: Use a Workflow Immediate Insight User's Guide
d. Instance TTL specifies the length of time a workflow instance stays active. This is to prevent abandoned instances from consuming system resources. e. Maximum Number of Instances caps the number of instances that can be active at the same time. f. You can select or clear check boxes to best meet your needs. g. Click OK.
Note: All fields, except Description, are required .
4. In the Edit Workflow:
l Regular—intermediate step in the workflow that leads to another one.
l Stop—the end step in the logical sequence of steps. Transitioning to a stop state also causes the workflow instance to finish. More than one stop state is valid. It is recom- mended that all logical paths lead to a stop state. d. Select or clear check boxes to Generate these events. One or more events can be selected. The list of available events changes depending on the type and presence of other states. e. Click OK. Once the state is created, it will appear on the Edit Workflow:
Notes: It is best practice to have at least one stop state and provide transitions such that a path exists from every state to one or more stop states.
6. In the Edit Workflow:
Chapter 2: Administration: Add a Workflow 65 Immediate Insight User's Guide
e. Match is a regular expression that matches something in the event that identifies it as a type of this kind of input. Make the match expression as explicitly as possible to avoid detecting the wrong input for this transition. f. Match Field is the field that Match text will be checked against. Leave blank for the original event text. Other options are: Text, Source, SourceFile, and Tags. g. Click More to access additional parameters for the transition.
l Qualifier Field is the field that Qualifier terms will be checked against.
l AndAny Qualifier is a comma-separated list of additional terms. In addition to Match, at least one of these must also match for the route to be triggered.
l AndAll Qualifier is a comma-separated list of additional terms that all need to match for the route to be triggered.
l Not Qualifier is a comma-separated list of additional terms that all much not match. If any match, the route will not be triggered. h. Click OK. 6. Repeat step 6 to continue adding transitions to connect the states. 7. Click OK.
Here are some helpful tips to make it easier to correctly fill in the Instance and Match expressions.
l Gather samples of events generated by your devices and/or applications. In some cases, the devices and/or applications need to be configured to send events or add more details to events related to the workflow being tracked.
l Identify the common identifier across all the events that will be associated with a workflow. This can be a transaction ID, document name, ticket number, etc.
l For each event, determine the kind of input that it is and how it impacts the progress through the workflow. Add or modify states as needed to cover the steps of the process you want to track.
l There are numerous online resources for regular expression syntax.
l Test your new workflow by importing your sample data using scratchpad or sending it to the TCP col- lector and watching the firehose for your selected generated events.
66 Chapter 2: Administration: Add a Workflow Immediate Insight User's Guide
Edit a Workflow To edit an existing workflow, complete the following steps. 1. On the DataFlow > Workflow page, click the Name of a workflow. 2. You can edit the Properties, State or Transition.
l Click the Workflow Properties icon .
Note: A workflow's Name and Instance ID cannot be changed.
l Click the Edit icon in the State box.
l Click on the Transition line. 3. The Workflow Properties or Edit
Notes: Changing the type of state from regular to stop will delete all outbound transitions from that state because it is not valid to transition from a stop state.
4. Click OK.
Delete a Workflow To delete a workflow, click the name of the workflow and on the Edit Workflow:
Delete a State or Transition
Caution! Deleting a state will also delete all transitions connected to it.
1. On the DataFlow > Workflow page, click the Name of a workflow. 2. Click the Edit icon in the State box or click on the Transition line. 3. In the Edit
Chapter 2: Administration: Edit a Workflow 67 Immediate Insight User's Guide
Workflow Troubleshooting Always make sure the match expression is written as specifically as possible. If it is too general, the input may be sent to the wrong workflow. Always make sure the instance expression accurately matches the unique ID. It is okay that delimiters are part of the match. The file app/logs/data-marshal.log contains workflow tracking details that are useful for troubleshooting. Avoid making changes to a workflow during business hours since changes take effect immediately and could confuse the users who are working with instances of the workflow you changed.
Data Route Actions Creating an action based on a workflow event allows the system to respond to workflow activity. For example, to send an alert when a form submission is taking longer than it should. Enable the event of interest in the workflow state: form submitted
Add a data route action to generate an alert when that event is seen. But don’t enter the instance in the Match Field.
68 Chapter 2: Administration: Workflow Troubleshooting Immediate Insight User's Guide
The alert will now appear when the time has elapsed:
Chapter 2: Administration: Data Route Actions 69 Immediate Insight User's Guide
Custom Action Scripts To create more sophisticated actions, you can write your own scripts. Immediate Insight uses a JavaScript scripting environment, which enables quite a bit of power and flexibility. Keep in mind that if your action will be executed many times a second, performance matters. Try to keep scripts short and simple. Actions that are only hit once in a while are less of a performance concern. Loading Custom Scripts Place action scripts in the app/config/actions directory. You can upload scripts via SFTP or edit them dir- ectly from an SSH session using one of the command prompt editors (vi, pico, nano, joe). By convention, action scripts should have a .actions extension (example: myCustomAction.actions). You can have more than one action script in the same .action file and you can place more than one .action file in the actions dir- ectory. Once you have added or changed an .action file, tell the server to reload it by typing @@reload actions into the search box of the UI:
Alternatively, you can type restart from the SSH command prompt, which will restart the server process. After reloading the actions, your custom action script will appear as a choice in the Action menu, when editing or creating a Data Route. The following section provides some tips for creating custom actions scripts (for further assistance you can also contact [email protected] ) Writing Custom Scripts Code is standard JavaScript. A simple action file looks like this:
//------
// Sample Custom Action
//------
var actions = exports;
actions.myScript =
function(data){ var now = new
date() data.myDate = now
return data }
Lines starting with // are comments. All .actions files must start with the line:
70 Chapter 2: Administration: Custom Action Scripts Immediate Insight User's Guide
var actions = exports
After that you can have one or more action functions in the file. Action functions look like this:
actions.myScript = function(data){ … your code here … }
In this example, we’ve used the name myScript for the action. The rest of the line (actions. , function(data)) is required and always the same. Each time the action is called, it receives a data object which contains the text of the data record and whatever metadata information is available at that time. Typically the interesting values are:
l data.text: the text of the event
l data.source: the name of the collector
l data.sourceFile: the path or port number the collector got this record from
l data._ttl: 3d At this point in time, the record has not yet been stored and indexed, so you can modify it and add metadata to it. For example, this action would create a new metadata entry called xUppercase for the record that contains the original message converted to uppercase. You can view the metadata in the details panel of the search screen by clicking a panel display icon .
var actions = exports;
actions.myScript = function(data){ data.xUppercase = data.- text.toUpperCase() }
Utility Functions for Custom Actions Immediate Insight provides a number of utility functions that you can use to work with data. Parsing
utils.csv(text,delimiter)
l Takes a line of csv data and splits into an array.
l Handles escaping such as quoted fields containing commas or newlines and double-quotes like ""foo""
l By default, delimiter is comma or tab, but you can specify other values such as space (" ") or pipe ("|")
Example: For input data that looks like this: 12345,Jan-24-2013, "Smith, John","Supervisor". Extract the UserID (12345) and the name and add them to the record as metadata
Chapter 2: Administration: Utility Functions for Custom Actions 71 Immediate Insight User's Guide
actions.FindUserID = function(data){
var e = utils.csv(data.text)
if(e.length < 4){ return } //record too short, don’t
proceed data.userID = e[0] || "no id" data.userName = e[2]
|| "no name" data.userTitle = e[3] || "no title" }
The text following the || is the default to use if no value is available from the record
Note: The numbering is zero-based, so the first item is 0 and the third item is 2.
72 Chapter 2: Administration: Example: Chapter 3: Usage
The Home Page 74 About Settings 76 About the DataFlow Page 77 About the Firehose Page 79 About the Search Page 81 About the Microscope Page 90 About the Reputation Page 91 About Pinboard and Bookmarks 94 Share a Session 96
Immediate Insight User's Guide The Home Page After signing in to Immediate Insight, the Home page opens. From this page you navigate to other areas of Immediate Insight. Toolbar menu options include access to additional pages. To open, click the page name.
l Search—used to look through past data and expand or narrow scope. Real-time search capabilities.
l Firehose—used to follow data in real-time as it arrives.
l Microscope—used to examine a specific event in greater detail.
l Reputation—used to configure internal reputation. View internal and external reputation.
l DataFlow—used to manage data flow and connect to new data feeds. Additionally, from the toolbar you can access bookmarks, help menu and settings.
Toolbar Icons Icon Description
Click this icon to open a list of available Pinboards. Only available on the Home page or on the pinned search page. Not part of the toolbar.
Click this icon to return to the Home Page.
Click this icon to open Bookmarks. Also used to manage pinboards and book- marks, show history, share a session or share a configuration.
Click this icon to open the list of Help Menu items.
Click this icon to open account settings and to log out of the application.
Return to the Home Page You can return to the home page from anywhere within Immediate Insight. To return to the home page, com- plete the following step.
l Click the Immediate Insight icon in the upper left-hand corner.
Immediate Insight User's Guide Search from the Home Page Located in the middle of the page is a search box and time selector.
Search Box Enter a few words to start the search, For example, a fragment of an error message, a user name ot IP address, a file name or page name, a location or data source. You do not need to be specific because you can easily narrow the results on
l To search, enter the search criteria on the field and press Enter or click Search. The Search page opens. Time Selector Select the time range to search. Immediate Insight is about what's happening right now, so the last hour is usually a good place to start.
l To select a time frame, click the arrow to choose from a list of time-related options.
Immediate Insight User's Guide About Settings From the Settings option, you can manage your user password, activate a license and log out of the applic- ation. You can also view your logged in as user ID. Change Password All users can manage their own passwords. To change your password, complete the following steps. 1. On the toolbar, click the Settings icon, and then click Change Password. 2. Complete the User Preferences dialog box. a. Enter your current password in the Old Password field. b. Enter your new password in the New Password field. c. Enter your new password again in the New Password Again field. d. Click OK.
Activate License
Note: Only an admin user can activate a license.
To activate a product license, complete the following steps. 1. On the toolbar, click the Settings icon, and then click Activate License. 2. In the Welcome to Immediate Insight dialog box, enter the activation code in the Activation Code field, and then click OK.
Immediate Insight User's Guide About the DataFlow Page For user level access, the DataFlow page is where interactive data can be manually imported. For administrator level access, the DataFlow page also allows management of Collectors, Data Routing, system Monitoring, Remotes, and Workflow. There are six options to provide a different perspective of the data.
l Import page can be used for on-demand importing.
l Collectors page displays collector statistics. The most active collectors are shown at the top. Each collector will show separate statistics for each file it is actively following. Drops due do to rate con- trols, if you have applied any, will also be shown here. The display updates every few seconds.
l Data Routing gives you very granular control over the processing of each record as it arrives, much the way a network router or firewall gives you granular control over the processing of each packet passing through it.There are a few one-time configuration options available for data routing that are configured on the command prompt.
l Workflow tracking correlates team (people) or device (machines) driven business process states with the raw event data. This enables automatic real-time extraction of actionable information from the raw event streams that are captured by Immediate Insight.
l Monitor provides an overview of information coming in from various sources.
l Remotespage displays the available remote agents.
Import Interactive data can be manually imported from the DataFlow > Import menu. Options to select when importing include:
l Tags—you can append tag metadata to make an individual import easier to find later
l Repo—you can put your imports in a separate repository from main stuff streaming in
l TTL—how long the data is kept before discarding (default is 3 days)
l Multiline delimiter to help figure out where one line stops and the other begins (note: this is not nor- mally needed as Immediate Insight does a good job of this automatically, special cases only)
l Import Time—select Original if you want Immediate Insight to try using the time stamp present in the original data, otherwise select Now to normalize the data to the Immediate Insight clock.
Note: Search is time dependent, if you don’t see data check the time range
Drag & Drop Import You can select to import targets as lines or as blob. Import As Lines processes each line individually, such as a log file. Import As Blob processes the entire file as one item, such as with a configuration file. If in doubt as to which type to use, select Import as Lines. You can import a variety of file types: .rtf, .text (UTF-8, UTF-16), Pcap, Word, PowerPoint, Excel, .pdf, .pst, .zip, .rar, .tar, Gzip, Bzip2, .xz, JSON, and XML.
Immediate Insight User's Guide Note: Nested archives cannot be imported. Only text within files are imported, graphics are ignored.
Scratch Pad You can enter or copy notes directly into the scratchpad page area of the page that you want to add to a search. You can copy-and-paste information into the Scratchpad, such as lines from an SSH terminal ses- sion or an email. Import Settings You can select how imports are handled based on Import Time and Import Type. For Import Time specify if you wish to use the original source time stamp (where present), or the system time stamp when indexing the records into the system (system time stamp is the default).
Immediate Insight User's Guide Immediate Insight User's Guide
About the Firehose Page The Firehose page is used to view incoming data. It provides a quick real time view to data coming into Immediate Insight either from the collectors, and/or from drag & drop import function.You can use it for a high-level look at your data, but also to confirm that data collection is working. You can optionally filter the search on simple key works, and pause for easier viewing. Below is a list of icons that can be found on the Firehose page to help with navigation. Firehose Page Icons Icon Name Function
Show/Hide Click to show or hide the data stream section.
Show/Hide Click to show or hide the activity graph.
Show/Hide Click to show or hide the details section.
Search Click to search on the term. You will go to the Search page.
Click to erase the current search context. Click again to Reset restore the search.
The following are sections available on the Firehose page. About the Search Term Box The search term displays only lines matching this term. The text will turn red if the query term entered is invalid. Searches match metadata as well as the displayed event text, so a search term of "syslog" will match all records arriving from the syslog server.
Clicking the search button will open the Search page and do a historical search on the term. This is useful for terms you recently clicked on in the flow and if you want to look at past details. Apply Filter Clicking Filter will apply a new search term to limit what is displayed. The box will turn orange if changes to the search term have not yet been applied. Pause Clicking Pause will pause the feed so you can read what is there. Data continues to flow as long as you are on this page, it just isn't displayed. If you move off of this page, you will need to click Resume to tell the server to restart the feed. Perform a search To perform a search, complete the following steps.
1. On the toolbar, click Firehose.
Chapter 3: Usage: About the Firehose Page 79 Immediate Insight User's Guide
2. In the Search Query field type your search criteria. 3. Click Filter to limit the search to only the term. 4. Click Pause to visually pause the data stream. 5. Select to include All Data or Search Only. Activity Chart The activity chart displays a count of the number of matching events per second. This count is kept by the server and shows all matches, even if the record was not displayed on the screen due to rate limits. Activity Chart Options Click the Activity arrow to change the rate of update for the activity chart. Details Click on the line in the feed in the Details section to see its metadata. This is useful for viewing the name of the collection source. Live Event Count The live event count contains the line, time and event details of the record. To manage bandwidth and per- formance, as well as respect the limits of human visual processing, the live feed will not update faster than approximately 12 events per second. If more matches than that are available, the rest will not be sent from the server or displayed. A gap in the count indicates this. You can use the Search page if you need to go back and look in more detail at all records in a certain time period. Event Data Record data is colorized to highlight possibly interesting terms. Click on a term to set a filter and show only records containing that term. Click on the line number to see details of the record in the Details section.
80 Chapter 3: Usage: Activity Chart Immediate Insight User's Guide
About the Search Page On the Search page you can perform a search and then view a variety of search result information, as well as perform most of your detailed analysis. Navigation is divided into several mains sub-panes. It is helpful to understand these at a high level first.
Below is a list of icons that can be found on the Search page to help with navigation. Search Page Icons Icon Name Function
Panel Display Click to show events panel only.
Panel Display Click to hide top panel.
Panel Display Click to hide details panel.
Panel Display Click to show all panels.
Click to open the Add to Pinboard/Bookmark dialog box to Pinboard save the query.
Click to erase the current search context. Click again to Reset restore the search.
Click to toggle through IP addresses, locations, networks, Entity Quick Access tags, etc,
Additional Options Click to view: Details, Origin Info, Follow
Click to change to: Tools, Time Slices, Subsearch, Change View Options Details
Click to quickly move to a new view: Events, Asso- Quick Change Options ciations, Clusters, Comparisons, Tags/Notes
Reduce Click to eliminate most common results.
Colorize Click to highlight known entities or unique terms.
Short Form Click for the short form view.
Flatten Click to reduce multi-line messages to a single line.
Map View Click to view a map.
Chapter 3: Usage: About the Search Page 81 Immediate Insight User's Guide
Search Page Icons Icon Name Function
Zoom Click to zoom in or out of the map view.
Table View Click to view information in table form.
Details View Click to view a list of details.
Scroll Click to scroll through information.
Click Navigation Immediate Insight search attempts to highlight things in the data that might be interesting terms and calls them out with color and underlining. Click a term to pivot the search context to that term. You can use Shift+Alt+Ctrl or the Search Action bar in the lower left of the screen to modify what happens when you click each of the buttons. Click Navigation is available on all pages. Click Navigation Icons Button Name Function
Drill-down Adds clicked term to the search context to narrow results.
Reduce Clicking will eliminate the clicked term from search context.
Focus Clicking will change the current search context to the clicked term.
Click to view current activity for the clicked term in real-time (Fire- Follow hose).
Click to view the clicked term as a subset of the current search Subsearch context, without changing the search context.
Search Page Sections The Entity Quick Access button will show or hide additional
l Entities - Infrequent Entities
l Sources - Infrequent Sources
l Tags
l Locations - Location Country - Location Region - Infrequent Locations
l Networks - Infrequent Networks
82 Chapter 3: Usage: Click Navigation Immediate Insight User's Guide
l Users - Infrequent Users
l Apps - Infrequent Apps
The Change View options will open one of four sections.
l Tools offers four menus for quick navigation to the various search views.
l Time Slices shows counts of events. Click the arrow to select other metrics such as peaks, aver- ages, and other variables.
l Subsearch provides a way to easily view the activity of an individual entities within the current search context.
l Details provides metadata for currently selected event. Point to an event to show more options.
The Quick Change options will open one of five sections.
l Search Results provides access to One Click Analytics.
l Association Analytics summarizes data by counts.
l Event Clusters summarizes data by similarity.
l Activity & Change displays information changes in the data.
l Annotations provides summaries for notes, tags and alerts.
The Additional options will open one of three sections.
l Origin Info
l Follow
l Details provides an opportunity to look up reputation data.
Details To look up reputation data, complete the following steps.
1. Click . 2. Enter search text in the Entity field, and then click Lookup. To look-up more detailed reputation data, complete the following steps.
Chapter 3: Usage: Details 83 Immediate Insight User's Guide
1. Click . 2. Enter search text in the Entity, Field, Value and Expires fields. 3. Click Lookup.
Search Panels
Clicking one of the Change View options will open one of four panels.
l Tools offers four menus for quick navigation to the various search views.
l Time Slices shows counts of events.Click the arrow to select other metrics such as peaks, aver- ages, and other variables.
l Subsearch provides a way to easily view the activity of an individual entities within the current search context.
l Details provides metadata for currently selected event. Point to an event to show more options.
Clicking one of the Quick Change options will open one of five panels.
l Search Results provides access to One Click Analytics.
l Association Analytics automatically groups search results into Entities, Locations, Addresses, Users, Apps, Networks, and Sources. These are useful for narrowing search results.
o Click the toggle icon to switch from frequent to infrequent results.
o Entities are not only IP addresses. Click
o Click [more] to open a complete list, and then click the arrows to open more details.
l Event Clusters automatically summarizes and groups current Search Results based on similarity. Event clusters can be helpful when you don't know where to start looking.
o You can switch between Common and Unusual (least common) clusters.
o Click the colorize icon to highlight unique terms.
o Click the reduce icon to eliminate the most common results.
l Activity & Change compares what you are currently searching to an earlier time period (previous, an hour ago, yesterday, last week). This allows you to see if an event results are normal or deviating from baseline. You can also compare two subsets of the current data to narrow the baseline.
84 Chapter 3: Usage: Search Panels Immediate Insight User's Guide
o You can change the trend inspected by clicking the trend heading (Trending Up, Trending Down, Active, etc.).
o After selecting the initial time period, you can select another by clicking the currently selected period.
o An event cluster result isn't necessarily a problem, but if it is new or trending up then it may indicate a need for investigation.
l Annotations provides summaries for any notes, tags and alerts relevant to the current search res- ults.
Clicking one of the Additional options will open one of three panels.
l Origin Info
l Follow
l Details Details To look-up reputation data, complete the following steps.
1. Click . 2. Enter search text in the Entity field, and then click Lookup. To look-up more detailed reputation data, complete the following steps.
1. Click . 2. Enter search text in the Entity, Field, Value and Expires fields. 3. Click Lookup.
One Click Analytics: Action Point to an event in the Search Results section to access the Action menu options. Add Note To add a note, complete the following steps. 1. Click Add Note. 2. Type a note with or without hashtags. 3. Click Add Note. Spam To block unwanted message, complete the following steps.
Chapter 3: Usage: Details 85 Immediate Insight User's Guide
1. Click Spam. 2. Complete the Block Unwanted Messages dialog box. a. Select an Action to take. b. Select What to block. c. Enter matching terms in the box. d. Click OK. Action To take an action on similar messages, complete the following steps. 1. Click Action. 2. Complete the Actions dialog box. a. Select an Action to take. b. Select What to take the action on. c. Enter matching terms in the box. d. Click OK. Auto-tag To automatically tag similar messages, complete the following steps. 1. Click Auto-tag. 2. Complete the AutoTag dialog box. a. Enter hashtags in the box. b. Select To What to add the tags to. c. Enter matching terms in the box. d. Click OK.
Perform a Search To perform a search, complete the following steps. 1. On the toolbar, click Search. 2. In the Search Query field type your search criteria. 3. Select a Time from the list to further narrow the results. 4. Select a specific Repository or choose to show all data. 5. Click Search.
86 Chapter 3: Usage: Action Immediate Insight User's Guide
Search Terms Immediate Insight search works similarly to an Internet search engine. Enter a list of terms, separated by spaces, into the search box to find all the records that match those terms. Key Concepts
l An unstructured search works best when you start with as little specificity as possible. Enter a few words (fragments of an error message, names of a server or user, etc.) and see the results it returns. That will often catch more related information than if you start out with a more detailed query.
l The goal of a search is first to get in the ballpark and find things that *might* be what you are looking for or might be related, even if they aren’t exact. Because results come back quickly, you can then refine your search and have a better chance of not missing important or interesting related inform- ation.
l Use of quotation marks and wildcards are very useful. Terms are often tied together with unexpected use of spacing and punctuation. but if your search isn’t returning the results you were expecting, try adding or removing a wildcard or quotation marks to capture a different result.
l The value of streaming data tends to decline rapidly over time, so most of the focus of Immediate Insight is on what is happening right now. Searches tend to generate the most useful results when you start by looking at what has happened in the last hour or last 24 hours rather than trying to look at a report for yesterday or a week ago.
l To select a term that wasn’t pre-highlighted, you can copy and paste or double-click after selecting from the Search Action bar. Qualifiers By default, the search with several terms will look for records containing any of those terms. You can use AND, OR and NOT to create more complex and specific searches.Qualifiers before search strings can be used to tell Immediate Insight more specifically what metadata to search, for example text:string, source:string, tag:string.
Search Qualifiers Qualifier Example of Use
Use to search for records containing two terms. AND Fred AND login
Use to search for records containing only one term and not another. NOT Fred NOT login
OR A search for Mike Joe Mary, generates the same result as Mike OR Joe OR Mary.
Plus is equivalent to AND. Results should contain the term. + +vlan2
Minus means results NOT containing the term. - -vlan2
Chapter 3: Usage: Search Terms 87 Immediate Insight User's Guide
Grouping Use parenthesis to logically group things for easier understanding.
(Fred Mary Joe) AND login NOT Server32
Quotation Marks Use of quotation marks make a search term more specific. This “192.168.0.1” will yield a more specific result than this 192.168.0.1; which could return hits containing any of those four numbers. As a general rule, if your search is not returning the results you were expecting, try the search again using quo- tation marks.
Wildcards Wildcards are very useful for cases like object names or domain names. Wildcards are also useful for partials entries.
l Asterisk (*)—using an asterisk in place of a partial entry will return significantly more results than without the asterisk. For example, unstructured logs will frequently use multiple forms of a verb or term. For example, start* is helpful to find both phrases like “started service foo at 03:12:19” and “03:12:19 service foo starting up”.
l Tildes (~)—can be used to find similar words, one or two characters difference, such as a misspelling or pluralization.
l Regular Expression (/ /)—terminology can also be used. Place the regular expression inside slashes.
l IP address range—the range can be within a sub-net, or across contiguous sub-net ranges.
Useful Wildcards for Searches Character Examples of Use
192.168.0.* * Finds 192.168.0.1 and 192.168.0.254. It also finds 192.168.0.1:8080, 192.168.0.1/8080, and 192.168.0.1.8080
error~ ~ Finds “error” and “errors” (and also the misspelled version “eror”)
/Server[6-8]/ / / Finds “Server6”, “Server7”, or “Server8” but not “Server4”. Some regular expressions, such as lookarounds and greedy matches, can be quite slow or resource intensive.
IP address range ipv4:[10.1.1.1 TO 10.3.100.252]
88 Chapter 3: Usage: Grouping Immediate Insight User's Guide
Metadata By default, the search will look for matches across all data or metadata. Record metadata includes location, source, and other correlations. You can view metadata for an event record by hovering over it and selecting metadata from the inline menu. For example, to find all records from any syslog collector, just search for sys- log. To find all records from the file collector that you named Log Collector:
“Log Collector”
Metadata can be identified by field names, although because everything in an unstructured system is indexed together, there is often no benefit in specifying a field name. To use a field name, put the field name in front of your term, followed by a colon. This would find all records whose metadata source string contains the word “syslog” but would not capture records with the word syslog in the text of the message:
source:syslog
The following would capture records with the word “syslog” in the text of the message but would not capture records that are part of a syslog collector:
fulltext:syslog
Time The default time range for a search is the last 60 minutes. Keep in mind that this is sort of relative time spe- cification is a continuously moving window, so if you do the exact same search a few seconds later, the count of matches may not be identical. You can select a rough time specification from the time drop down next to the search box. You can enter a specific time range with the “custom” time selection, but with unstructured data, the most effective way to navigate time is usually to start from a larger period and then zoom in by clicking bars on the time slices chart. Start from last hour, last day, or last 7 days, and then zoom in to the precise time period you are interested in. Often you will see interesting anomalies as you go, such as patterns of previous occurrence of the same event, anomalous dropouts or spikes in activity, or timestamps that are slightly off from what you expected due to configuration issues.
Chapter 3: Usage: Metadata 89 Immediate Insight User's Guide
About the Microscope Page The Microscope page is used to view presentation of detailed data. It detects and displays common data in an easy-to-read format. It includes JSON and JSON fragments, XML, FireEye Security Appliances and Palo Alto Networks Security Appliance. The Microscope provides the greatest level of detail about a particular event. Depending on the source data and metadata enrichment, some events will show more details than others. You can link to the Microscope from a Search Result. There are three options to provide a different perspective of the data.
l Fulltext displays a complete view of the record.
l Metadata displays additional details about the record.
l Origin Info displays information about where the record came from and how it is stored.
To select a different perspective, click its name or use the View Selector buttons.
Click to move to the previous or next event in the list.
90 Chapter 3: Usage: About the Microscope Page Immediate Insight User's Guide
About the Reputation Page The Reputation page allows you to add what you have learned about a particular host(s). Such ‘human’ insights are automatically correlated and searchable alongside ‘machine’ data to provide a more complete picture. For an entity, you can create a custom key value pair. It becomes part of the reputation for the entity and is added to subsequent matching event and entity combinations. Example key value pairs include, Critical Infrastructure: Oracle; hasPriviledge:Twitter; isExec:CFO. There are four options to provide a different perspective of the data.
l Reputation Info
l Associations
l Activity
l Observations
To select a different perspective, click its name or use the View Selector buttons.
Chapter 3: Usage: About the Reputation Page 91 Immediate Insight User's Guide
Add Reputation Information To add reputation data, complete the following steps. 1. On the toolbar, click Reputation > Reputation Info. 2. Next to Reputation Info, click the Add icon. 3. Enter text in the Entity, Field, Value and Expires fields. 4. Click Save.
Note: In addition to the method above, bulk reputation info can be imported via CSV.
Add Bulk Reputation Information Adding bulk Reputation information can be imported into Immediate Insight using a .csv file. You can drag files ending with .iprep or .iprep.csv to DataFlow > Import > Import as Lines to populate the IP reputation fields. The first column must be IPMATCH and contain IPv4 match patterns. Fields are taken from the column head- ings and the values from each row. An event is generated for each entry so that other actions, workflows can be tied to changes. Overlapping reputations are allowed. Non-conflicting fields are merged in. Conflicting fields are overwritten to allow easy updating by reloading the IP reputation files again. To add bulk reputation information using a .csv file, complete the following steps. 1. Create the .csv file.
Note: IPMATCH must always be the first heading and contain IP addresses or subnets.
2. Save as a comma delimited CSV file. 3. Go to DataFlow > Import, and drag the .csv file into Import as Lines. 4. A dialog box will open, Import file in progress... notifying you of the import progress, click Dismiss to close. 5. You can go to Search to view the successful import. 6. Go to Reputation to view the imported information. a. Enter an IP address from the .csv file. b. Click Lookup. c. Click the Add icon to include more reputation information. d. Click Save.
92 Chapter 3: Usage: Add Reputation Information Immediate Insight User's Guide
Delete Reputation Entries Deleting reputation entries is performed by setting the first value to IPDELETE.
IPMATCH,Restricted
3.3.4.0/24,IPDELETE
2.2.2.20 to 2.2.2.21,IPDELETE
Events are generated for every delete:
Caution! A deletion will remove the entire reputation including fields that were merged in from other files.
Pay special attention when converting .xls files to .csv files as some data files may contain multiple delim- iters which must be removed in order to produce a clean csv file.
Chapter 3: Usage: Delete Reputation Entries 93 Immediate Insight User's Guide
About Pinboard and Bookmarks Pinboard Pinboard is another term for bookmark. It provides quick access to favorite, often-used searches and queries. Pinboards are useful starting points to monitor and investigate known issues or suspected scenarios. You access your pins from the Home page by clicking the Pinboard icon. Click a title to open its related page. To change the information that is displayed, click a pinboard icon. The available icons and their function are listed in the following table. Pinboard Icons Icon Name Description
Pinboard Click this icon to open a list of available Pinboards.
Displays event counts for a selected time period. Click the Counts arrow to choose a time period.
Defines the time frame for comparison. Click the arrow to Change choose a time period.
Concise view Click to display only counts.
Category view Click to display all category items on the same graph.
Click to display graphs and other details. One entry per Expanded view line.
Graph view Click to display only graphs.
Click to display a list of questions pertaining to the search Questions results.
Bookmarks You can access bookmarks from the toolbar at any time by clicking the Bookmark icon.
l Add to Pinboard or Bookmarks opens a dialog box used to add new pins or bookmarks.
l Manage Pinboard & Bookmarks opens a dialog box used to hide, share or delete bookmarks.
l Show History opens the Search History dialog box which displays a list of the most recent query searches.
l Share a Session allows you to join an existing session or begin a new one.
94 Chapter 3: Usage: About Pinboard and Bookmarks Immediate Insight User's Guide
l Share a Configurationopens the Shareboard, where you can view current items available to be exported and shared.
Add a Pin or Bookmark To add a new pin or bookmark, complete the following steps. 1. There are two ways to add a pin or bookmark.
a. On the toolbar, click Search. After entering your search criteria, click the pin icon next to the search box. b. Click the Bookmarks icon on the toolbar and select Add to Pinboard or Bookmarks. 2. Complete the Add to Pinboard or Bookmarks dialog box. a. The Search Terms field is populated with the actual query used for the search. b. The Display Name field is based on the search. You can leave as is or change it. c. The Display on Pinboard check box is selected by default. Clear the check box to remove it from the pinboard. d. Select a Section from the list or leave blank. e. Select a Page from the list or leave blank. f. Select the Display on Bookmarks Menu to add the search there as well. g. Select the Use as a shortcut in search (auto-replacement) check box if this query will be used often. h. Click Save or More Details to add additional details about the pin. More Details i. In the Sharing section, enter an optional Management Name and Description, j. In the Display section, select Panel Types from the lists. k. In the Colors section, select colors for various areas of the search. l. In the Questions section, type Questions and select Actions. m. Click Save.
Manage Pinboard and Bookmarks To manage existing pinboards and bookmarks, complete the following steps. 1. On the toolbar, click the Bookmarks icon and then click Manage Pinboards & Bookmarks. 2. From the list of current bookmarks, select the one to manage. 3. On the Add to Pinboards or Bookmarks dialog box, make your changes to any of the fields. 4. Click Save.
Chapter 3: Usage: Add a Pin or Bookmark 95 Immediate Insight User's Guide
Share a Session Share a Session allows you to join an existing session or begin a new one. To share a session, complete the following steps. 1. On the toolbar, click the Bookmarks icon and then click Share a Session. 2. In the Live Share dialog box, enter a tag to join an existing session or begin a new one. 3. Select the Follow this search activity stream or Publish my search activity for others to follow option. 4. Click OK.
Quit Sharing To quit sharing, clear the field and then click OK.
Shareboard The Shareboard downloads a copy of individual remote commands, workflows, and bookmarks for you to share with others.
Click the Shareboard icon or from the Manage Pinboards & Bookmarks dialog box, click Share. To share, click Export to create a data file of the included items. This data file can be imported into other sys- tems.
96 Chapter 3: Usage: Share a Session Chapter 4: API
APIs: Getting Data Out 98 REST API Certificates 101 Command-Line Client 102
Immediate Insight User's Guide Immediate Insight User's Guide
APIs: Getting Data Out Besides interactive search and the export button, there are a variety of programmatic ways to get data out of the system. These include:
l HTTPS Query
l Linux command line client, for use in scripting
l Custom Feeds in the data router HTTPS Query API The Immediate Insight REST API lets you query the search engine from scripts or other products by sending URLs. A basic query looks something like this. You can type it into your browser to try it and see the results:
https://ip-address:3201/search?k=api-key&q=query
Note: See the REST API Examples section below for more examples.
Note that:
l Queries are sent over HTTPS to port 3201 with a REST endpoint of /search .
l The API Key functions as a password.
l You can define multiple API Keys. API Parameters String Function
q search term (default = all records)
n number of search results to return (default = 10)
end time as either a JavaScript time integer or ii human readable form endtime (default = "now")
end time as either a JavaScript time integer or ii human readable form starttime (default = "1 hour ago")
length of time as either a JavaScript time integer or ii human readable time form (default = "1 hour") overrides starttime
API Key (default = none , note: use “api-key” to match the pre-con- k figured API Key in API Credentials)
f format text|json|html|csv (default = text)
d type of data info|events|summary|full|timeline (default = events)
the field to return summary or event info about (default = entities, full- type text)
98 Chapter 4: API: APIs: Getting Data Out Immediate Insight User's Guide
REST API Examples Get the 10 most recent records:
https://ip-address:3201/search?k=api-key
Get the 10 most recent records with the word root:
https://ip-address:3201/search?k=api-key&q=root
Get the 10 most recent records from the logfile "firewall.log":
https://ip-address:3201/search?k=api-key&q=sourceFile:firewall.log
Get the 50 most recent records from the logfile "firewall.log":
https://ip-address:3201/search?k=api-key&q=sourceFile:firewall.log&n=50
Display them in a browser-readable format:
https://ip-address:3201/search?k=api-key&q=firewall.log&n=50&f=html
Return them in a spreadsheet-readable format (csv):
https://ip-address:3201/search?k=api-key&q=firewall.log&n=50&f=csv
Return them with additional metadata in a software-friendly format (json):
https://ip-address:3201/search?k=api-key&q=firewall.log&n=50&f=json
Display all records matching the search "error" for the last 2 minutes (up to 250):
https://ip-address:3201/search?k=api-key&q=error&n=250&starttime=now- 2m&endtime=now
Display the last 10 records matching the search "error" over the last hour:
https://ip-address:3201/search?k=api-key&q=error&starttime=now-1h&end- time=now&d=events
Display just the count of records matching the search "error" over the last hour:
https://ip-address:3201/search?k=api-key&q=error&starttime=now-1h&end- time=now&d=info
Return the count of records matching the search "error" each minute over the last hour (suitable for a graph):
Chapter 4: API: REST API Examples 99 Immediate Insight User's Guide
https://ip-address:3201/search?k=api-key&q=error&starttime=now-1h&end- time=now&d=timeline
Return both the event text and the metadata for the last 10 records matching the search "error" over the last hour:
https://ip-address:3201/search?k=api-key&q=error&starttime=now-1h&end- time=now&n=10&d=full
Return a count for each of the top 100 entities in that search:
https://ip-address:3201/search?k=api-key&q=error&starttime=now-1h&end- time=now&n=100&d=summary
API Credentials The REST endpoint for API Queries is /search. The default port number is 3201 and protocol is HTTPS. HTTPS access must be enable in Immediate Insight CLI before API calls can be made.To enable HTTPS access, complete the following steps. 1. Type set-ssl command to enable encryption on browser sessions. 2. Type reload server to make changes take effect. 3. Close the browser and reopen using HTTPS instead of HTTP (https://ip-address-of-server:3201) Note that:
l The API key functions as a password.
l You can have multiple API keys active at once.
l The default API key is "api-key". API Keys are defined in app/config/marshal-settings.conf . You can edit this file to add or change them. Options include:
l Default—API.key = "api-key"
l No Key Required—API.key = none
l Specify an API key (quotes are optional)—API.key = my-secret-key
l Set several API keys—API.key = ["key1","key2","key3"]
l Disable the REST API entirely—API.enable = false (default value is true)
l Restart the marshal for changes to take effect by using the restart command—restart
100 Chapter 4: API: API Credentials Immediate Insight User's Guide
REST API Certificates Because REST API calls are over HTTPS, it is suggested to enable certificate keys. Certificates FireMon recommends the best practice use of matching CA certificates installed in user’s browsers to reduce the possibility of man-in-the-middle attacks and provide a smoother user experience. During installation, a self-signed rootCA pair is generated automatically in app/config/certs.
Note: You can replace this pair with your own CA by overwriting the rootCA.key and rootCA.pem files, however this is an advanced task. Most can use the self-signed certificates provided.
To set the certificate, complete the following steps. 1. Type set-certs followed by reload server to activate the certificate. 2. Copy the app/config/certs/rootCA.pem file from the Immediate Insight server to your computer using an SFTP client. 3. Load the Certificate into your browser. A. In the Chrome browser go to Settings > Show Advanced Settings > HTTPS/SSL > Manage Certificates > Trusted Root Certification Authorities > Import. B. Specify the rootCA.pem file. 4. Restart the Chrome browser. The next time you log into Immediate Insight you should not see a certificate warning.
Note: While the system has a reasonable set of security measures in place, the present release is designed to run in a secure and trusted environment. If you have a need to expose it directly to the Inter- net, please call us to discuss additional hardening procedures.
Chapter 4: API: REST API Certificates 101 Immediate Insight User's Guide
Command-Line Client The command-line client (CLI) lets you issue search queries from a Linux shell script or terminal session. You can run the script from the Immediate Insight console and also download it to remote systems. The CLI uses the REST API. See the API Credentials section for details on how to create API keys. The default API key is "api-key". Installation To put the command prompt client script on another system or expose the script on your own Immediate Insight system, you first download it from the appliance. 1. From the Linux shell, download either of the following scripts:
l wget --no-check-certificate https://ip-address:3201/tools/ii
l curl -o ii -k https://ip-address:3201/tools/ii
It is intentionally designed as a simple shell script so that you can customize or modify it for your needs. You will need to have the Curl HTTP client utility installed on your system. Most full Linux distributions include this by default and also in their repositories.
sudo yum install curl or sudo apt-get install curl
2. Mark the script as executable:
chmod +x ii
3. Edit the first lines of the script (using vi or nano) to set your server address and API key (if you wish). For convenience, the default server address and default API key are set in the first few lines of the file. You can edit it and modify these for your own environment to eliminate the need to specify them on every query.
l KEY—"api-key" (substitute other API key if you are using a different one)
l SERVER—"localhost" (substitute IP address of II server if accessing remotely)
102 Chapter 4: API: Command-Line Client Immediate Insight User's Guide
Command-Line Options
String Function
-h Print a list of options
-q The search query [default=*]
-n Number of results to return [default=10]
-t Time scope [default="now"]
-a Start time [default="now"]
-b End time [default="now"]
-f Format [text|json|html|csv] [default=”text”]
-d Data type [info|events|summary|full|timeline] [default=”events”]
events Returns just the original text of the event
Returns the event text and also the metadata such as source, arrival time, and cor- full relations
info Returns just a count of matching records
timeline Returns count-per-minute data over the time period
summary Returns a list of entities and their counts
-s Server address [default="localhost", or whatever you substituted when editing ii script]
-k API Key [default="api-key", or whatever you substituted when editing ii script]
Command-Line Examples Time examples
"now", "now-1h", "now-1d-4h-5m" "1361683874554"
Get the first 10 results matching the query from server 192.168.0.1 with credential “api-key”
ii -s 192.168.0.1 –k api-key -q "search terms" -n 10
If you already set the default address and api key for your server and placed the ii file in your executable path, this would shorten to:
ii -q "search terms" -n 10
Chapter 4: API: Command-Line Options 103 Immediate Insight User's Guide
You can pipe or direct the output in typical unix shell fashion. For example, this would get the most recent 100 lines from the source “fw.log”, split fields based on colon, select the value from the third field, and provide a count of each unique value:
ii -q "fw.log" -n 100 | cut -d ":" -f 3 | sort | uniq –c
Get just a count of the number of hits:
ii -q "search terms" -d info
Get just a count of the total number of hits for the last 5 minutes:
ii -q "search terms" -d info -a now-5m -b now
Get just a count of the number of hits for the last 5 minutes, on a per-minute basis
ii -q "search terms" -d timeline -a now-5m -b now
Get a list of the entities matching the search, and the number of hits for each:
ii -q "search terms" -d summary
Get the full data (message + entities + correlations + source, etc.) for the most recent 20 records matching the search:
ii -q "search terms" –n 20 -d full
104 Chapter 4: API: Command-Line Examples