CERN COMPUTER NEWSLETTERVolume 44, issue 3 July–September 2009 Contents CINBAD keeps an eye Editorial CINBAD keeps an eye on the CERN network 1 ETICS 2 offers guidance to software professionals 3 on the CERN network Announcements and news CERN welcomes 13 Intel ISEF pre-college winners 4 The CINBAD (CERN Investigation of Network network infrastructure misuse, violation of Computer team advises reviewing your Behaviour and Anomaly Detection) project a local network security policy and device security now and frequently 5 was launched in 2007 as a collaboration misconfiguration. In addition, the expected EGEE-III project is on track for EGI between CERN openlab, IT-CS and HP network behaviour never remains static transition 5 ProCurve Networking. The project’s aim because it can vary with the time of day, the Grid news is to understand the behaviour of large number of users connected and network Scientists demonstrate the role of CMS in computer networks in the context of services deployed. As a consequence, computing Grid 6 high-performance computing and campus anomalies are not easy to detect. Technical brief Indico’s new face goes live 7 installations such as those at CERN. The CERN updates Wi-Fi network 9 goals are to detect traffic anomalies in Network sniffing Conference and event reports such systems, perform trend analysis, To acquire knowledge about the network Prague hosts CHEP conference 10 automatically take counter measures and status and behaviour, CINBAD collects and Workshop identifies steps to reap provide post-mortem analysis facilities. analyses data from numerous sources. benefits from multicore and virtualization Alarms from different network monitoring technologies 11 CERN’s network systems, logs from network services like HEPiX event arrives in Sweden 12 Calendar 12 CERN’s campus network has more than Domain Name System (DNS), Dynamic 50 000 active user devices interconnected Host Configuration Protocol (DHCP), user by 10 000 km of cables and fibres, with feedback, etc – all of these constitute a more than 2500 switches and routers. solid base of information. A naive approach The potential 4.8 Tbps throughput might be to look at all of the packets flying within the network core and 140 Gbps over the CERN network. However, if we did connectivity to external networks offers this we would need to analyse even more countless possibilities to different network data than the LHC could generate. The LHC applications. The bandwidth of modern data are only a subset of the total data networks is growing much faster than crossing via these links. the performance of the latest processors. CINBAD overcomes this issue by applying This fact combined with the CERN specific statistical analysis and using sFlow, a configuration and topology makes network technology for monitoring high-speed behaviour analysis a very challenging and switched networks that provides randomly daunting task. sampled packets from the network traffic. The information that we collect is based CINBAD in a nutshell on the traffic from around 1000 switches The CINBAD project addresses many and routers and gives a representative Editor Natalie Pocock, CERN IT Department, 1211 aspects associated with the CERN network. sample of the CERN network traffic with Geneva 23, Switzerland. E-mail cnl.editor@cern. First, it provides facilities for a better more than 3 Terabytes of data per month. ch. Fax +41 (22) 766 8500. understanding and improved maintenance The multistage collection system was Web cerncourier.com/articles/cnl. of the CERN network infrastructure. designed and implemented in consultation Advisory board Frédéric Hemmer (head of IT Department), Alberto Pace (group leader, Data This includes analysing various network with experts from the LHC experiments and Management), Christine Sutton (CERN Courier statistics and trends, traffic flows and Oracle, to benefit from their data-analysis editor), Tim Smith (group leader, User and protocol distributions. Other factors and storage experience. The system has Document Services). that might have an impact on the current now been up and running for more than a Produced for CERN by IOP Publishing Dirac House, Temple Back, Bristol BS1 6BE, UK. network status or influence its evolution year (figure 1). Tel +44 (0)117 929 7481. E-mail jo.nicholas@iop. are also studied, such as connectivity, org. Fax +44 (0)117 930 0733. Web iop.org. bottleneck and performance issues. Network operation enhancements Published by CERN IT Department When we have learnt and understood The field of network monitoring and ©2009 CERN the network behaviour, CINBAD can help planning can greatly benefit from the The contents of this newsletter do not necessarily to identify various abnormalities and CINBAD activities. We provide tools and represent the views of CERN management. determine their causes. Because there are data that simplify the operation and many factors that can be used to describe problem-diagnosing process. In addition, the network status, anomaly definition our statistics help in understanding the is also very domain specific and includes network evolution and design. CERN Computer Newsletter • July–September 2009 1 Editorial A very basic piece of information that configurator configure rules is of interest for network operations is for analysis knowledge about the host’s activity. (live tcpdump, CINBAD is able to provide detailed configuration via data for adjusting fingerprints) statistics about the traffic sent and SNMP sFlow configuration received by a given host, it facilitates inference about the nature of the traffic on a given outlet/port and can thus identify collector the connected machine. This information level II raw storage could also be used to diagnose routing sFlow problems by looking at all of the packets datagrams unpacked aggregated outbound or inbound to a particular host. data data CINBAD is also able to provide CINBAD DB information about the traffic at CERN. The sampled data collected by the project are sufficient to obtain the switching/ routing/transport protocol information redundant collector as well as gaining information about the application data. This provides valuable network level I level I disk level II level III input for an understanding of the current devices processing storage processing processing network behaviour. Here the CINBAD team uses descriptive statistics. The potential Fig. 1. The CINBAD sFlow data collector receives and processes the CERN network traffic. set of metrics that we can provide to characterize the traffic at CERN is very (if no-one can get to it, no-one can would not scale. extensive and specific needs are currently harm it). Nowadays, we cannot avoid A second approach is to build various being discussed. For example, we can communicating with others and therefore network profiles by learning from the enumerate protocol-type distributions, we expose our machine to outside past. The selection of robust metrics that packet size distributions, etc. Depending on threats. Although CERN centrally managed are resistant to data randomness plays the requirements, these statistics can be desktops have up-to-date anti-virus an important role in characterizing the tailored even further. software and firewalls, this does not expected network behaviour. Once these Top n-list is another form of network guarantee that our machines and data are normal profiles are well established, the summary that might be of interest. Such shielded from attacks. These tools are statistical approach can detect new and lists would allow the identification of the usually designed to detect known patterns unknown anomalies. most popular application servers, either (signatures) and there are also other The CINBAD project combines inside or outside CERN. Although this machines (unmanaged desktops, PDAs, the statistical approach with the information might be available on each etc) connected to the CERN network that signature-based analysis to benefit from individual CERN server, CINBAD provides might be less protected. the synergy of the two techniques. While the possibility to collect these statistics Currently, detailed analysis is only the latter provides the detection system for all servers of a given type, whether performed at critical points on the with a fast and reliable detection rate, or not they are centrally managed by the network (firewall and gates between the former is used to detect the unknown IT Department. This information may be network domains). The CINBAD team has anomalies and to produce new signatures. of value to both network engineers and been investigating various data-analysis The CINBAD team constantly monitors application-server administrators. approaches that could overcome both the campus and internet traffic These statistics can also be useful for this limitation. These studies can be using this method. This has already led to network design and provisioning. The categorized into two main domains: the identification of various anomalies, CINBAD project can provide valuable statistical and signature-based analysis. e.g. DNS abuse, p2p applications, information about the nature of the The former depends on detecting rogue DHCP servers, worms, trojans, traffic on the links. These statistics can deviations from normal network behaviour unauthorized wireless base stations, etc. also be used to detect the trunks with while the latter uses existing problem Some of these findings have resulted in potential bottlenecks. This information signatures and matches them