DOCSLIB.ORG

Qflow: a Fast Customer-Oriented Netflow Database for Accounting and Data Retention

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Qflow: a Fast Customer-Oriented Netflow Database for Accounting and Data Retention qflow: a fast customer-oriented NetFlow database for accounting and data retention Hallgrímur H. Gunnarsson FacultyFaculty of of Industrial Industrial Engineering, Engineering, MechanicalMechanical Engineering Engineering and and ComputerComputer Science Science UniversityUniversity of of Iceland Iceland 20142014 QFLOW: A FAST CUSTOMER-ORIENTED NETFLOW DATABASE FOR ACCOUNTING AND DATA RETENTION Hallgrímur H. Gunnarsson 60 ECTS thesis submitted in partial fulfillment of a Magister Scientiarum degree in Computer Science Advisors Snorri Agnarsson Helmut Neukirchen Faculty Representative Jón Ingi Einarsson Faculty of Industrial Engineering, Mechanical Engineering and Computer Science School of Engineering and Natural Sciences University of Iceland Reykjavik, September 2014 qflow: a fast customer-oriented NetFlow database for accounting and data retention 60 ECTS thesis submitted in partial fulfillment of a M.Sc. degree in Computer Sci- ence Copyright c 2014 Hallgrímur H. Gunnarsson All rights reserved Faculty of Industrial Engineering, Mechanical Engineering and Computer Science School of Engineering and Natural Sciences University of Iceland Hjarðarhagi 2-6 107 Reykjavik, Reykjavik Iceland Telephone: 525 4000 Bibliographic information: Hallgrímur H. Gunnarsson, 2014, qflow: a fast customer-oriented NetFlow database for accounting and data retention, M.Sc. thesis, Faculty of Industrial Engineering, Mechan- ical Engineering and Computer Science, University of Iceland. Printing: Háskólaprent, Fálkagata 2, 107 Reykjavík Reykjavik, Iceland, September 2014 Abstract Internet service providers in Iceland must manage large databases of network flow data in order to charge customers and comply with data retention laws. The databases need to efficiently handle large volumes of data, often billions or tril- lions of records, and they must support fast queries of traffic volume per customer over time and extraction of raw flow data for given customers. Popular open-source tools for storing flow data, such as nfdump and flow-tools, are backed by flat binary files. They do not provide any type of indexing or summaries of customer traffic. As a result, flow queries for a given customer need to linearly scan through all the flow records in a given time period. We present a high-performance customer-oriented flow database that provides fast customer queries and compressed flow storage. The database is backed by indexed flow tablets that allow for fast extraction of customer flows and traffic volume per customer. Útdráttur Internetþjónustur á Íslandi þurfa að geyma mikið magn af netmælingargögnum til að gjaldfæra netnotkun og uppfylla lög um gagnageymd. Gagnagrunnskerfi fyrir net- mælingargögn þurfa að ráða við margar færslur, oft á tíðum milljarða eða billjarða, og þau verða að styðja hraðvirka uppflettingu á gagnamagni og hráum netmælin- garfærslum fyrir notendur. Vinsælar opnar lausnir, t.d. nfdump og flow-tools, geyma færslur í flötum skrám. Þau bjóða ekki upp á flýtivísa fyrir hraðvirka leit eða samantektir á gagnamagni notenda. Þar af leiðandi þarf að lesa allar færslurnar til að svara fyrirspurnum. Í þessu verkefni kynnum við til sögunar nýtt afkastamikið notendaskipt gagna- grunnskerfi fyrir netmælingargögn. Kerfið geymir gögnin á þjöppuðu sniði en býður samt upp á hraðvirkar fyrirspurnir um notendur. Gagnagrunnskerfið byggir ofan á safni af litlum töflusneiðum sem mynda eina heild. Hverri töflusneið fylgir flýtivísir sem gerir kleift að framkvæma hraðvirkar fyrirspurnir um færslur og gagnamagn notenda. v Contents List of Figures ix List of Tables xi 1. Introduction 1 1.1. Motivation . .1 1.2. Requirements . .2 1.3. Contribution . .4 1.4. Related work . .5 1.5. Structure of thesis . .6 2. Flow-based monitoring 7 2.1. Network monitoring . .7 2.2. Flow probes . .8 2.2.1. Overview . .8 2.2.2. Flow export . 10 2.2.3. Packet sampling . 11 2.3. Cisco NetFlow . 13 2.3.1. History . 13 2.3.2. Version 5 . 13 2.3.3. Version 9 . 16 2.3.4. Storage requirements . 19 2.4. Observation points . 19 2.4.1. Edge deployment . 19 2.4.2. Ingress/egress monitoring . 20 2.4.3. Deployment strategies . 22 2.4.4. Customer traffic . 22 3. Design and implementation 23 3.1. Architecture . 23 3.2. Collector . 24 3.2.1. Design . 24 3.2.2. Flow format . 25 3.2.3. Configuration . 27 3.2.4. Backend protocol . 30 vii Contents 3.3. Database . 31 3.3.1. Design . 31 3.3.2. Table queue . 32 3.3.3. Record format . 33 3.3.4. Indexer . 34 3.3.5. Tablets . 35 3.3.6. Materialized views . 36 3.4. Filtering . 38 3.4.1. Language . 38 3.4.2. Implementation . 39 3.5. Reports . 40 3.5.1. Flow extraction . 41 3.5.2. Flow summary . 41 3.5.3. Flow filter . 41 3.5.4. Time-based reports . 42 3.5.5. Customer reports . 43 4. Evaluation 45 4.1. Environment . 45 4.2. Collector . 45 4.2.1. Preparation . 46 4.2.2. Results . 48 4.3. Indexer . 51 4.4. Flow storage . 53 4.5. Flow extraction . 54 4.5.1. Preparation . 54 4.5.2. Results . 56 4.6. Materialized views . 57 5. Conclusions 61 5.1. Summary . 61 5.2. Future work . 61 Bibliography 63 A. Flow protobuf 65 B. Collector configuration protobuf 69 C. Grammar for the filter language 71 viii List of Figures 2.1. Flow probe internals . .9 2.2. Relative sampling error . 12 2.3. NetFlow v5 export packet . 14 2.4. Structure of NetFlow v9 export packet . 17 2.5. NetFlow v9 template flowset . 18 2.6. NetFlow v9 data flowset . 19 2.7. NetFlow edge deployment . 20 2.8. Example router with both ingress and egress monitoring enabled . 21 2.9. Provider network with both ingress and egress monitoring enabled . 21 3.1. An overview of the qflow system . 23 3.2. Collector overview . 25 3.3. Directory layout of the flow database . 31 3.4. Flow capture pipeline . 32 3.5. Structure of a block . 34 3.6. Directory layout for flow tablets . 35 3.7. Internal layout of a flow tablet . 36 3.8. View file format . 38 ix LIST OF FIGURES 3.9. Parse tree for example filter expression . 40 4.1. File size of materialized view . 57 4.2. Update time for materialized view . 58 4.3. Query time for materialized view . 58 4.4. Export time for materialized view . 59 x List of Tables 2.1. Format of NetFlow v5 header . 15 2.2. Format of NetFlow v5 record . 15 2.3. Format of NetFlow v9 header . 16 4.1. Collector performance results for NetFlow v5 . 49 4.2. Collector performance results for NetFlow v9 . 50 4.3. Indexer performance for 10M records . 51 4.4. Indexer performance for 20M records . 52 4.5. Indexer performance for 30M records . 52 4.6. Indexer performance for 40M records . 52 4.7. Storage efficiency of qflow vs. flow-tools . 53 4.8. Flow extraction performance for a single IP . 56 4.9. Flow extraction performance for a network . 56 xi 1. Introduction 1.1. Motivation Internet service providers (ISPs) in Iceland must store and query large volumes of network traffic monitoring data in order to charge customers and comply with data retention laws. A typical ISP might need to store and process up to 50K flow records per second, which translates into more than 8 GB per hour of monitoring data [9]. Data retention laws require the data to be stored for six months, and for 50K records per second it would take 34 TB to store the resulting 770 billion records. A number of commercial and open source solutions exist for storing and managing network flow data, e.g. Cisco NetFlow collector, pmacct, flow-tools and nfdump. They usually store flow records in either a relational database or raw binary files. In general, relational databases offer flexibility and a powerful query language, but they are known to be slower, especially in terms of insertion rate, and consume more disk space when compared to specialized tools that use raw binary files [6]. Furthermore, scaling relational databases to billions or trillions of records can be a real challenge. For example, the iiBench 1B row insert benchmark for MySQL shows that the insert rate is highly dependent on the table size. As the table grows larger and the data no longer fits in memory, the performance degrades dramatically. In the beginning of the benchmark, MySQL can sustain around 40,000 inserts per second, but after inserting 200M rows, it has fallen down to 5,000 inserts per second. Close to 1B rows, the rate is down to 876 inserts per second. With 50K records per second, the table would reach 200M rows in one hour, and 1B rows in less than six hours. In contrast, tools based on raw binary files can handle a high insertion rate, e.g. nfdump can store over 250K records per second on a dual-core machine [6]. The binary files are flat, without any sort of index, and the insertion rate does not degrade over time. The flow records are written in the order they are received and files are usually rotated regularly, e.g. every 5, 10 or 15 minutes. Although they support fast insertion, such tools present two.
Load more

Recommended publications
  • Ngenius Collector Appliance Scalable, High-Capacity Appliance for Collection of Cisco Netflow and Other Flow Data
    l DATA SHEET l nGenius Collector Appliance Scalable, High-Capacity Appliance for Collection of Cisco NetFlow and Other Flow Data Product Overview HIGHLIGHTS Deployed at key traffic aggregation locations, nGenius® Collectors extend the reach of the nGeniusONE® Service Assurance solution and are used primarily to generate flow‑based • Measure service responsiveness across statistics (metadata) in memory for specific traffic types. This NETSCOUT data source collects the network with up to 500 Cisco IP SLA metadata on IP SLA and IPPING protocols, flow data from NetFlow routers, link‑level statistics, synthetic transaction tests and utilization data from MIB‑II routers. • Scalable collection of up to 2 million Cisco NetFlow, IPFIX, Juniper J-Flow, Huawei® Listening passively on an Ethernet wire, nGenius Collectors examine specific traffic collected NetStream and sFlow flows per minute from flow‑enabled routers and switches (e.g., Cisco® NetFlow, Juniper® J-Flow, sFlow®, ® • Captures and stores Flow datagrams for NetStream ) and from IP SLA test results to generate a variety of statistics. In addition, Collectors historical deep-dive analysis can be configured to capture datagrams from Flow‑enabled routers and analyze them via datagram capture, which allows users to perform in‑depth capture and filtering. • Collects Flow data from up to 5,000 flow‑enabled router or switch interfaces Metrics from nGenius Collectors are retrieved through a managing nGenius for Flows Server per appliance for analysis, enabling display of utilization metrics, quality of service (QoS) breakdowns, and • Supports both IPv4 and IPv6 environments application breakdowns in nGenius for Flows and other tools in the nGeniusONE Service • Purpose-built hardware and virtual Assurance Solution.
    [Show full text]
  • Netflow Traffic Analyzer Real-Time Network Utilization and Bandwidth Monitoring
    DATASHEET NetFlow Traffic Analyzer Real-Time Network Utilization and Bandwidth Monitoring An add-on to Network Performance Monitor (NPM), SolarWinds® NetFlow DOWNLOAD FREE TRIAL Traffic Analyzer (NTA) is a multi-vendor flow analysis tool designed to proactively reduce network downtime. NTA delivers actionable insights Fully Functional to help IT pros troubleshoot and optimize spend on bandwidth by better for 30 Days understanding the who, what, and where of traffic consumption. Solve practical operational infrastructure problems with actionable insights and save money with informed network investments. WHY CHOOSE NETFLOW TRAFFIC ANALYZER? • NTA collects and analyzes flow data from multiple vendors, including NetFlow v5 and v9, Juniper® J-Flow™, sFlow®, Huawei® NetStream™, and IPFIX. • NTA alerts you to changes in application traffic or if a device stops sending flow data. • NTA supports advanced application recognition with Cisco® NBAR2. • NTA shows pre- and post-policy CBQoS class maps, so you can optimize your CBQoS policies. • NTA can help you identify malicious or malformed traffic with port 0 monitoring. • NTA includes WLC network traffic analysis so you can see what’s using your wireless bandwidth. • NTA supplements Network Performance Monitor by helping to identify the cause of high bandwidth. Built on the Orion® Platform, NTA provides the ability to purchase and fully integrate with additional network monitoring modules (config management, WAN management, VoIP, device tracking, IP address management), as well as systems, storage, and virtualization management in a single web console. page 1 DATASHEET: NETFLOW TRAFFIC ANALYZER FEATURES New! VMware vSphere Distributed Switch (VDS) Support Comprehensive support for the VMware VDS, providing visibility within the switch fabric to your east-west VM traffic to help IT pros avoid service impacts when moving workloads.
    [Show full text]
  • Netflow Optimizer™
    NetFlow Optimizer™ Installation and Administration Guide Version 2.4.7 (Build 2.4.7.0.23) January 2017 © Copyright 2013-2017 NetFlow Logic Corporation. All rights reserved. Patents both issued and pending. Contents Overview ....................................................................................................................................................................... 3 How NetFlow Optimizer Works .................................................................................................................................. 3 How NFO Updater Works .......................................................................................................................................... 3 NetFlow Optimizer Installation Guide ......................................................................................................................... 4 Before You Install NFO ................................................................................................................................................ 4 Pre-Installation Checklist ........................................................................................................................................... 4 Minimum Requirements ............................................................................................................................................. 4 Supported Platforms .............................................................................................................................................. 4 Virtual Hardware
    [Show full text]
  • Flow-Tools Tutorial
    Flow-tools Tutorial SANOG 6 Bhutan Agenda • Network flows • Cisco / Juniper implementation – NetFlow • Cisco / Juniper Configuration • flow-tools programs overview and examples from Abilene and Ohio- Gigapop Network Flows • Packets or frames that have a common attribute. • Creation and expiration policy – what conditions start and stop a flow. • Counters – packets,bytes,time. • Routing information – AS, network mask, interfaces. Network Flows • Unidirectional or bidirectional. • Bidirectional flows can contain other information such as round trip time, TCP behavior. • Application flows look past the headers to classify packets by their contents. • Aggregated flows – flows of flows. Unidirectional Flow with Source/Destination IP Key % telnet 10.0.0.2 10.0.0.1 login: 10.0.0.2 Active Flows Flow Source IP Destination IP 1 10.0.0.1 10.0.0.2 2 10.0.0.2 10.0.0.1 Unidirectional Flow with Source/Destination IP Key % telnet 10.0.0.2 % ping 10.0.0.2 login: 10.0.0.1 10.0.0.2 ICMP echo reply Active Flows Flow Source IP Destination IP 1 10.0.0.1 10.0.0.2 2 10.0.0.2 10.0.0.1 Unidirectional Flow with IP, Port,Protocol Key % telnet 10.0.0.2 % ping 10.0.0.2 login: 10.0.0.1 10.0.0.2 ICMP echo reply Active Flows Flow Source IP Destination IP prot srcPort dstPort 1 10.0.0.1 10.0.0.2 TCP 32000 23 2 10.0.0.2 10.0.0.1 TCP 23 32000 3 10.0.0.1 10.0.0.2 ICMP 0 0 4 10.0.0.2 10.0.0.1 ICMP 0 0 Bidirectional Flow with IP, Port,Protocol Key % telnet 10.0.0.2 % ping 10.0.0.2 login: 10.0.0.1 10.0.0.2 ICMP echo reply Active Flows Flow Source IP Destination IP prot srcPort
    [Show full text]
  • Brocade Vyatta Network OS Data Sheet
    DATA SHEET Brocade Vyatta Network OS HIGHLIGHTS A Network Operating System for the Way Forward • Offers a proven, modern network The Brocade® Vyatta® Network OS lays the foundation for a flexible, easy- operating system that accelerates the adoption of next-generation to-use, and high-performance network services architecture capable of architectures meeting current and future network demands. The operating system was • Creates an open, programmable built from the ground up to deliver robust network functionality that can environment to enhance be deployed virtually or as an appliance, and in concert with solutions differentiation, service quality, and from a large ecosystem of vendors, to address various Software-Defined competitiveness Networking (SDN) and Network Functions Virtualization (NFV) use cases. • Supports a broad ecosystem for With the Brocade Vyatta Network OS, organizations can bridge the gap optimal customization and service between traditional and new architectures, as well as leverage existing monetization investments and maximize operational efficiencies. Moreover, they can • Simplifies and automates network compose and deploy unique, new services that will drive differentiation functions to improve time to service, increase operational efficiency, and and strengthen competitiveness. reduce costs • Delivers breakthrough performance flexibility, performance, and operational and scale to meet the needs of any A Proven, Modern Operating efficiency, helping organizations create deployment System The Brocade Vyatta Network OS new service offerings and value. Since • Provides flexible deployment options separates the control and data planes in 2012, the benefits of this operating to support a wide variety of use cases software to fit seamlessly within modern system have been proven by the Brocade SDN and NFV environments.
    [Show full text]
  • Introduction to Netflow
    Introduction to Netflow Campus Network Design & Operations Workshop These materials are licensed under the Creative Commons Attribution-NonCommercial 4.0 International license (http://creativecommons.org/licenses/by-nc/4.0/) Last updated 14th December 2018 Agenda • Netflow – What it is and how it works – Uses and applications • Generating and exporting flow records • Nfdump and NfSen – Architecture – Usage • Lab What is a Network Flow • A set of related packets • Packets that belong to the same transport connection. e.g. – TCP, same src IP, src port, dst IP, dst port – UDP, same src IP, src port, dst IP, dst port – Some tools consider "bidirectional flows", i.e. A->B and B->A as part of the same flow http://en.wikipedia.org/wiki/Traffic_flow_(computer_networking) Simple flows = Packet belonging to flow X = Packet belonging to flow Y Cisco IOS Definition of a Flow • Unidirectional sequence of packets sharing: – Source IP address – Destination IP address – Source port for UDP or TCP, 0 for other protocols – Destination port for UDP or TCP, type and code for ICMP, or 0 for other protocols – IP protocol – Ingress interface (SNMP ifIndex) – IP Type of Service IOS: which of these six packets are in the same flows? Src IP Dst IP Protocol Src Port Dst Port A 1.2.3.4 5.6.7.8 6 (TCP) 4001 22 B 5.6.7.8 1.2.3.4 6 (TCP) 22 4001 C 1.2.3.4 5.6.7.8 6 (TCP) 4002 80 D 1.2.3.4 5.6.7.8 6 (TCP) 4001 80 E 1.2.3.4 8.8.8.8 17 (UDP) 65432 53 F 8.8.8.8 1.2.3.4 17 (UDP) 53 65432 IOS: which of these six packets are in the same flows? Src IP Dst IP Protocol Src Port Dst Port A 1.2.3.4 5.6.7.8 6 (TCP) 4001 22 B 5.6.7.8 1.2.3.4 6 (TCP) 22 4001 C 1.2.3.4 5.6.7.8 6 (TCP) 4002 80 D 1.2.3.4 5.6.7.8 6 (TCP) 4001 80 E 1.2.3.4 8.8.8.8 17 (UDP) 65432 53 F 8.8.8.8 1.2.3.4 17 (UDP) 53 65432 What about packets “C” and “D”? Flow Accounting • A summary of all the packets seen in a flow (so far): – Flow identification: protocol, src/dst IP/port..
    [Show full text]
  • Conntrack, Netfilter, Netflow and NAT Under Linux
    Xurble conntrack, Netfilter, NetFlow and NAT under Linux Oliver Gorwits 9th February 2010 Milton Keynes Perl Mongers 1 “Policy Compliance” • We have legal obligations • Avoiding the courts ✔ • Avoiding the newspapers ✔ 2 (alleged) Copyright Violations Subject: File-sharing of unauthorised content owned by Twentieth Century Fox From: [email protected] Dear Oxford University: Twentieth Century Fox Film Corporation, located in Los Angeles, and its affiliated companies (collectively, 'Fox') own intellectual property rights, including exclusive rights protected under copyright laws, in many motion pictures, television programs and other audio-visual works, including the motion picture AVATAR (collectively, the 'Fox Titles'). Fox conducted an online check by scanning public networks and discovered that your Oxford University internet account was used to access and distribute an unauthorised copy of AVATAR. By distributing Fox content without Fox's permission, you infringed Fox's copyright. Here is the information Fox obtained from the online check: Timestamp of report: 07 Feb 2010 23:12:44 GMT Title details: Avatar (2009) PROPER TS XviD-MAXSPEED IP address: 163.1.xxx.yyy Port ID: 30854 Protocol used: BitTorrent - L5 Please respond to Fox and identify what steps you have taken to resolve this matter by contacting Fox at [email protected] 3 The Process • So, given: ○Timestamp with Time Zone ○IP address ○TCP port number • We need: ○User’s identity • Usually via: ○Network log-in logs, and DHCP logs 4 Linux network subsystems Kernel conntrack netfilter iptables pietroizzo 5 Network Address/Port Translation O’Reilly 6 State Tracking User Firewall Internet 1 A ✔ 2 B pre-NAT post-NAT • Traditional loggers run two packet captures and correlate the timestamps.
    [Show full text]
  • Ipv6-Security Monitoring
    Security Monitoring ITU/APNIC/MICT IPv6 Security Workshop 23rd – 27th May 2016 Bangkok Last updated 22 July 2014 1 Managing and Monitoring IPv6 Networks p SNMP Monitoring p IPv6-Capable SNMP Management Tools p NetFlow Analysis p Syslog p Keeping accurate time p Intrusion Detection p Managing the Security Configuration 2 Using SNMP for Managing IPv6 Networks 3 What is SNMP? p SNMP – Simple Network Management Protocol p Industry standard, hundreds of tools exist to exploit it p Present on any decent network equipment p Query/response based: GET / SET p Monitoring generally uses GET p Object Identifiers (OIDs) p Keys to identify each piece of data p Concept of MIB (Management Information Base) p Defines a collection of OIDs What is SNMP? p Typical queries n Bytes In/Out on an interface, errors n CPU load n Uptime n Temperature or other vendor specific OIDs p For hosts (servers or workstations) n Disk space n Installed software n Running processes n ... p Windows and UNIX have SNMP agents What is SNMP? p UDP protocol, port 161 p Different versions n v1 (1988) – RFC1155, RFC1156, RFC1157 p Original specification n v2 – RFC1901 ... RFC1908 + RFC2578 p Extends v1, new data types, better retrieval methods (GETBULK) p Used is version v2c (simple security model) n v3 – RFC3411 ... RFC3418 (w/security) p Typically we use SNMPv2 (v2c) SNMP roles p Terminology: n Manager (the monitoring station) n Agent (running on the equipment/server) How does it work? p Basic commands n GET (manager → agent) p Query for a value n GET-NEXT (manager → agent) p Get next value (e.g.
    [Show full text]
  • White Paper Netflow Generation Security Value Proposition
    WHITE PAPER Leveraging NetFlow Generation for Maximum Security Value Overview This kind of sparse traffic record is enough to establish trends for network and application performance, but it’s a non-starter for Cisco introduced NetFlow in 1996 as a way to monitor packets security analytics. Advanced persistent threats (APTs) can operate as they enter and exit networking device interfaces. The aim is low and slow on the network, moving laterally and communicating to gain insight and resolve congestion. Typical information in a with command and control sites over long periods of time (days, NetFlow record reveals traffic source and destination, as well months or longer). The key to spotting anomalous traffic is looking as protocol or application, time stamps, and number of packets. for hard-to-spot patterns over the complete network activity Although NetFlow was initially not on a standards track, it has picture. Naturally, looking at a fraction of packet records severely been superseded by the Internet Protocol Flow Information hampers the breadth and accuracy of the analysis. eXport (IPFIX), which is based on the NetFlow Version 9 implementation, and is on the IETF standards track with RFC 5101, Here are the drawbacks of generating NetFlow records using RFC 5102. networking gear: As organizations refocus network security efforts on insider 1. NetFlow generation can degrade router and switch threats and detection of compromise, NetFlow provides rich and performance important contextual information about the traffic, augmenting 2. To manage the impact on performance, networking devices analysis in order to determine where compromise has occurred. may give sampled NetFlow or just drop packets This takes NetFlow out of the traditional realm of use for 3.
    [Show full text]
  • Netflow Overview
    Network Monitoring and Management NetFlow Overview These materials are licensed under the Creative Commons Attribution-Noncommercial 3.0 Unported license (http://creativecommons.org/licenses/by-nc/3.0/) Agenda 1. Netflow - What it is and how it works - Uses and applications 2. Generating and exporting flow records 3. Nfdump and Nfsen - Architecture - Usage 4. Lab What is a Network Flow? • A set of related packets • Packets that belong to the same transport connection. e.g. - TCP, same src IP, src port, dst IP, dst port - UDP, same src IP, src port, dst IP, dst port - Some tools consider "bidirectional flows", i.e. A->B and B->A as part of the same flow http://en.wikipedia.org/wiki/Traffic_flow_(computer_networking) Simple flows = Packet belonging to flow X = Packet belonging to flow Y Cisco IOS Definition of a Flow Unidirectional sequence of packets sharing: 1. Source IP address 2. Destination IP address 3. Source port for UDP or TCP, 0 for other protocols 4. Destination port for UDP or TCP, type and code for ICMP, or 0 for other protocols 5. IP protocol 6. Ingress interface (SNMP ifIndex) 7. IP Type of Service IOS: which of these six packets are in the same flows? Src IP Dst IP Protocol Src Port Dst Port A 1.2.3.4 5.6.7.8 6 (TCP) 4001 80 B 5.6.7.8 1.2.3.4 6 (TCP) 80 4001 C 1.2.3.4 5.6.7.8 6 (TCP) 4002 80 D 1.2.3.4 5.6.7.8 6 (TCP) 4001 80 E 1.2.3.4 8.8.8.8 17 (UDP) 65432 53 F 8.8.8.8 1.2.3.4 17 (UDP) 53 65432 IOS: which of these six packets are in the same flows? Src IP Dst IP Protocol Src Port Dst Port A 1.2.3.4 5.6.7.8 6 (TCP) 4001 80 B 5.6.7.8 1.2.3.4 6 (TCP) 80 4001 C 1.2.3.4 5.6.7.8 6 (TCP) 4002 80 D 1.2.3.4 5.6.7.8 6 (TCP) 4001 80 E 1.2.3.4 8.8.8.8 17 (UDP) 65432 53 F 8.8.8.8 1.2.3.4 17 (UDP) 53 65432 What about packets “C” and “D”? Flow Accounting A summary of all the packets seen in a flow (so far): - Flow identification: protocol, src/dst IP/port..
    [Show full text]
  • Configuring Netflow
    CHAPTER 63 Configuring NetFlow This chapter describes how to configure NetFlow statistics collection in Cisco IOS Release 12.2SX. Note For complete syntax and usage information for the commands used in this chapter, see these publications: http://www.cisco.com/en/US/docs/ios/netflow/command/reference/nf_book.html Tip For additional information about Cisco Catalyst 6500 Series Switches (including configuration examples and troubleshooting information), see the documents listed on this page: http://www.cisco.com/en/US/products/hw/switches/ps708/tsd_products_support_series_home.html Participate in the Technical Documentation Ideas forum This chapter contains the following sections: • Understanding NetFlow, page 63-1 • Default NetFlow Configuration, page 63-4 • NetFlow Restrictions, page 63-4 • Configuring NetFlow, page 63-9 Understanding NetFlow • NetFlow Overview, page 63-2 • NetFlow on the PFC, page 63-2 • NetFlow on the RP, page 63-3 • NetFlow Features, page 63-3 Cisco IOS Software Configuration Guide, Release 12.2SX OL-13013-06 63-1 Chapter 63 Configuring NetFlow Understanding NetFlow NetFlow Overview The NetFlow feature collects traffic statistics about the packets that flow through the switch and stores the statistics in the NetFlow table. The NetFlow table on the route processor (RP) captures statistics for flows routed in software and the NetFlow table on the PFC (and on each DFC) captures statistics for flows routed in hardware. Several features use the NetFlow table. Features such as network address translation (NAT) use NetFlow to modify the forwarding result; other features (such as QoS microflow policing) use the statistics from the NetFlow table to apply QoS policies. The NetFlow Data Export (NDE) feature provides the ability to export the statistics to an external device (called a NetFlow collector).
    [Show full text]
  • Network Monitoring Based on IP Data Flows Best Practice Document
    Network Monitoring Based on IP Data Flows Best Practice Document Produced by CESNET led working group on Network monitoring (CBPD131) Authors: Martin Žádník March 2010 © TERENA 2010. All rights reserved. Document No: GN3-NA3-T4-CBPD131 Version / date: 24.03.2010 Original language : Czech Original title: “NETWORK MONITORING BASED ON IP DATA FLOWS” Original version / date: 1.2 of 3.12. 2009 Contact: [email protected] CESNET bears responsibility for the content of this document. The work has been carried out by a CESNET led working group on Network monitoring as part of a joint-venture project within the HE sector in Czech Republic. Parts of the report may be freely copied, unaltered, provided that the original source is acknowledged and copyright preserved. The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 238875, relating to the project 'Multi-Gigabit European Research and Education Network and Associated Services (GN3)'. 2 Table of Contents Table of Contents 3 Executive Summary 4 1 Approaches Used for Network Monitoring 5 2 NetFlow Architecture 6 3 NetFlow Agent 7 3.1 Agent Parameters and Configuration 7 4 Collector 9 4.1 Collector Parameters and Configuration 9 5 NetFlow Protocol 11 6 NetFlow Data Analysis 12 7 Usage 14 8 Ethical Perspective of NetFlow 17 9 Available Solutions 18 10 Future Outlook 20 11 Conclusion 21 12 List of Figures 22 3 Executive Summary Do you monitor your network? Try to answer the following questions. Which users and which services use the most network bandwidth, and do they exceed authorised limits? Do users use only the permitted services, or do they occasionally "chat" with friends during work hours? Is my network scanned or assaulted by attackers? NetFlow will answer these and other questions.
    [Show full text]