Practical Network Tomography

Total Page:16

File Type:pdf, Size:1020Kb

Practical Network Tomography Practical Network Tomography THÈSE NO 5332 (2012) PRÉSENTÉE LE 27 AOÛT 2012 À LA FACULTÉ INFORMATIQUE ET COMMUNICATIONS LABORATOIRE POUR LES COMMUNICATIONS INFORMATIQUES ET LEURS APPLICATIONS 3 Laboratoire D'ARCHITECTURE DES RÉSEAUX PROGRAMME DOCTORAL EN INFORMATIQUE, COMMUNICATIONS ET INFORMATION ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE POUR L'OBTENTION DU GRADE DE DOCTEUR ÈS SCIENCES PAR Denisa Gabriela Ghiţă acceptée sur proposition du jury: Prof. E. Telatar, président du jury Prof. P. Thiran, Prof. A. Argyraki, directeurs de thèse Prof. A. Krishnamurthy, rapporteur Prof. J.-Y. Le Boudec, rapporteur Dr W. Willinger, rapporteur Suisse 2012 ii Bunicilor mei... iv Abstract In this thesis, we investigate methods for the practical and accurate localization of Internet performance problems. The methods we propose belong to the field of network loss tomography, that is, they infer the loss characteristics of links from end-to-end measurements. The existing versions of the problem of network loss tomography are ill-posed, hence, tomographic algorithms that attempt to solve them resort to making various assumptions, and as these assumptions do not usually hold in practice, the information provided by the algorithms might be inaccurate. We argue, therefore, for tomographic algorithms that work under weak, realistic assumptions. We first propose an algorithm that infers the loss rates of network links from end-to-end measurements. Inspired by previous work, we design an algorithm that gains initial information about the network by computing the variances of links’ loss rates and by using these variances as an indication of the congestion level of links, i.e., the more congested the link, the higher the variance of its loss rate. Its novelty lies in the way it uses this information—to identify and char- acterize the maximum set of links whose loss rates can be accurately inferred from end-to-end measurements. We show that our algorithm performs signifi- cantly better than the existing alternatives, and that this advantage increases with the number of congested links in the network. Furthermore, we validate its performance by using an “Internet tomographer” that runs on a real testbed. Second, we show that it is feasible to perform network loss tomography in the presence of “link correlations,” i.e., when the losses that occur on one link might depend on the losses that occur on other links in the network. More pre- cisely, we formally derive the necessary and sufficient condition under which the probability that each set of links is congested is statistically identifiable from end-to-end measurements even in the presence of link correlations. In doing so, we challenge one of the popular assumptions in network loss tomography, specifically, the assumption that all links are independent. The model we pro- v vi pose assumes we know which links are most likely to be correlated, but it does not assume any knowledge about the nature or the degree of their correlations. In practice, we consider that all links in the same local area network or the same administrative domain are potentially correlated, because they could be sharing physical links, network equipment, or even management processes. Finally, we design a practical algorithm that solves “Congestion Probability Inference” even in the presence of link correlations, i.e., it infers the probability that each set of links is congested even when the losses that occur on one link might depend on the losses that occur on other links in the network. We model Congestion Probability Inference as a system of linear equations where each equation corresponds to a set of paths. Because it is infeasible to consider an equation for each set of paths in the network, our algorithm finds the maximum number of linearly independent equations by selecting particular sets of paths based on our theoretical results. On the one hand, the information provided by our algorithm is less than that provided by the existing alternatives that infer either the loss rates or the congestion statuses of links, i.e., we only learn how often each set of links is congested, as opposed to how many packets were lost at each link, or to which particular links were congested when. On the other hand, this information is more useful in practice because our algorithm works under assumptions weaker than those required by the existing alternatives, and we experimentally show that it is accurate under challenging network conditions such as non-stationary network dynamics and sparse topologies. Keywords: Network Loss Tomography, Network Measurements, Network Monitoring, Link-loss Inference, Congestion Probability, Correlated Links Zusammenfassung In der vorliegenden Doktorarbeit untersuchen wir Methoden für die praxisnahe und genaue Feststellung von Leistungsproblemen in Internetverbindungen. Die vorgeschlagenen Methoden gehören dem Feld der Netzwerkverlusttomographie an, das heisst sie berechnen Verlusteigenschaften von Links mit Hilfe von Mes- sungen zwischen Endpunkten. Die existierenden Versionen von Netzwerkver- lusttomographie sind generell unterbestimmt, daher machen die tomographis- chen Algorithmen zu ihrer Lösung verschiedene Annahmen. Weil diese An- nahmen in der Praxis aber nicht immer erfüllt werden, sind die Informationen welche die Algorithmen berechnen teilweise ungenau. Demzufolge plädieren wir für tomographische Algorithmen die mit schwachen, realistischen Annahmen funktionieren. Zuerst stellen wir einen neuen Algorithmus zur Berechnung von Link-Verlusten vor, das heisst einen Algorithmus der Verlustraten von Netzwerklinks auf Grund von Messungen zwischen Endpunkten berechnet. Inspiriert von existierenden Arbeiten entwickeln wir einen Algorithmus welcher die Anfangsinformationen über das Netzwerk durch das Berechnen von Varianzen von Link-Verlustraten gewinnt: je mehr ein Link überlastet ist, desto höher ist die Varianz seiner Verlustrate. Seine Neuheit liegt in der Art wie diese Informationen verwen- det werden - zum Identifizieren und Beschreiben der grössten Menge von Links deren Verlustraten mit Endpunkt-Messungen genau berechnet werden können. Wir zeigen dass unser Algorithmus signifikant höhere Leistungen erbringt als existierende Alternativen, und dass diese Vorteile mit einer steigenden Anzahl von überlasteten Links im Netzwerk zunehmen. Ausserdem überprüfen wir seine Leistung mit einem “Internet-Tomographen” in einer echten Testumgebung. Zweitens zeigen wir dass Netzwerkverlusttomographie in der Anwesenheit von “Link-Korrelation” angewendet werden kann, das heiss wenn die Verlus- trate welche in einem Link auftritt von den Verlustraten von anderen Links im Netzwerk abhängen kann. Wir beweisen dass unter bestimmten wohldefinierten vii viii Bedingungen die Wahrscheinlichkeit, dass jede Menge von Links überlastet ist auf Grund von Endpunkt-Messungen statistisch berechnet werden kann, auch in der Gegenwart von korrelierten Links. Dadurch lösen wir uns von einer verbre- iteten Annahme in der Netzwerkverlusttomographie, nämlich von der Annahme dass alle Links im Netzwerk unabhängig sind. Unser Modell nimmt an, dass wir wissen, welche Links wahrscheinlich korreliert sind, aber es verlangt keine Annahme über die Art oder den Grad der Korrelation. In der Praxis behandeln wir alle Links in einem lokalen Netzwerk oder im selben Administrationsbereich als möglicherweise korreliert, denn diese Links können physische Verbindungen, Netzwerkanlagen oder auch Verwaltungsprozesse gemeinsam haben. Schlussendlich entwickeln wir einen praktikablen Algorithmus zur Berech- nung der Überlastungswahrscheinlichkeit welcher die Link-Korrelation berück- sichtigt, das heisst er berechnet die Wahrscheinlichkeit dass jede Menge von Links überlastet ist auch wenn die auftretenden Verluste auf einem Link von den Verlusten auf anderen Links im Netzwerk abhängen. Wir modellieren die Berechnung der Überlastungswahrscheinlichkeit als ein System von linearen Gle- ichungen, so dass jede Gleichung einer Menge von Pfaden entspricht. Weil es nicht praktikabel ist, eine Gleichung für jede Menge von Pfaden im Netzw- erk zu berücksichtigen findet unser Algorithmus die grösste Anzahl von linear unabhängigen Gleichungen durch das Auswählen von bestimmten Mengen von Pfaden, basierend auf unseren theoretischen Resultaten. Auf der einen Seite liefert die Berechnung der Überlastungswahrscheinlichkeit weniger Informatio- nen als herkömmliche Alternativen, welche entweder Verlustraten oder Überlas- tungszustände von Links berechnen, das heisst wir erfahren nur, wie oft eine Menge von Links überlastet ist, aber nicht wie viele Pakete auf einem Link verloren gingen, oder welche Links wann überlastet waren. Auf der anderen Seite ist diese Information in der Praxis nützlicher weil unser Algorithmus unter schwächeren und anspruchsvolleren Annahmen funktioniert als existierende Al- ternativen, und wir zeigen experimentell dass er bei anspruchsvollen Netzw- erkbedingungen wie nicht-stationären Netzwerken und dünnbesetzten Topolo- gien akkurat ist. Stichwörter: Netzwerkverlusttomographie, Netzwerkmessung, Netzwerküberwachung, Link- Verlustberechnung, Überlastungswahrscheinlichkeit, Korrelierte Links Acknowledgments It has been a great pleasure to have Professors Katerina Argyraki and Patrick Thiran as my thesis advisors. I am grateful for their guidance, trust, and kind- ness. Katerina’s dedication to research inspired me throughout this thesis, and although I was her first student, she proved to be a wonderful advisor right from the start. From Patrick, I have learned among
Recommended publications
  • Estimating Network Loss Rates Using Active Tomography
    Estimating Network Loss Rates Using Active Tomography Bowei XI, George MICHAILIDIS, and Vijayan N. NAIR Active network tomography refers to an interesting class of large-scale inverse problems that arise in estimating the quality of service parameters of computer and communications networks. This article focuses on estimation of loss rates of the internal links of a network using end-to-end measurements of nodes located on the periphery. A class of flexible experiments for actively probing the network is introduced, and conditions under which all of the link-level information is estimable are obtained. Maximum likelihood estimation using the EM algorithm, the structure of the algorithm, and the properties of the maximum likelihood estimators are investigated. This includes simulation studies using the ns (network simulator) to obtain realistic network traffic. The optimal design of probing experiments is also studied. Finally, application of the results to network monitoring is briefly illustrated. KEY WORDS: EM algorithm; Inference on graphs; Network modeling; Network monitoring; Network tomography; Probing experiments. 1. INTRODUCTION parameters requires access to the internal links and routers. But the lack of centralized control of modern networks means that The term “network tomography” was introduced by Vardi Internet service providers typically do not have access to all the (1996) to characterize a certain class of inverse problems nodes of interest, making collection of detailed QoS informa- in computer and communication networks. The goal here, tion at the individual router/link level difficult. Active tomogra- as in medical tomography problems, is to recover higher- phy provides an alternative approach through the use of active dimensional network information from lower-dimensional data.
    [Show full text]
  • Georgios B. Giannakis, DTC Director ECE, Mcknight Presidential Chair (Last Update on 09/08/2021)
    CV highlights - Georgios B. Giannakis, DTC Director ECE, McKnight Presidential Chair (last update on 09/08/2021) I. Leadership and administrative roles 1) Digital Technology Center (DTC) Director: College-wide, cross-disciplinary research center, University of Minnesota (2008-2021) a) Managed 12 administrative staff; space; seed funds; and endowed chairs; b) Spearheaded externally sponsored projects; facilitated resource allocation; and coordinated summer internships, industry partnerships, fellowships, and seminar series; c) Doubled DTC researchers (100 graduate students; 20+ postdoctoral fellows; 30+ research visitors; and 50 affiliated faculty) d) Increased by a factor of five publications, patents, proposals, and funding ($30M/10yrs) e) Broadened research spectrum to include Data Science, Network Science, Renewables, Grid, Environmental, and Health Informatics; f) Expanded cross-departmental/college partnerships to include the College of Liberal Arts; School of Public Affairs; Business School; Chemical Engineering, and Neuroscience; g) Enhanced community outreach (Robotics Tech Camp; Lab tours for middle and high school students; and Summer school on bioinformatics) 2) Major posts in professional society: Institute of Electrical and Electronic Engineers (IEEE) IEEE Signal Processing and Communication Societies (SPS and ComSoc) a) IEEE Fellow and IEEE Proceedings Committee Member b) Board of Governors member; and Editor-in-Chief (SPS) c) Chair of Steering and Technical Committees (SPS and ComSoc) d) General Conference Chair, including the IEEE Data Science Workshop, 2019 3) Multi-university projects and proposals a) Army Research Laboratory, Collaborative Technology Alliance; Technical Area Lead b) Medium- and large-size proposals to the National Science Foundation, NIH, DoD 4) Board of Regents elected member (University of Patras, Greece, 2014-2017) 5) Hellenic Quality Assurance and Accreditation Agency, Ministry of Education, Greece 6) Research group: 12 Ph.D.
    [Show full text]
  • Runtime Dynamic Path Identification for Preventing Ddos Attacks 1Shaik Zahanath Ali, 2Shobini.B and 3G.Shiva Krishna
    Science, Technology and Development ISSN : 0950-0707 Runtime Dynamic Path Identification for Preventing DDoS Attacks 1Shaik Zahanath Ali, 2Shobini.B and 3G.Shiva Krishna 1 Computer science and Engineering ,Swathi Institute of Technology & Sciences Near Ramoji Film City Beside Kothagudem 'X' Roads, Hyderabad, Telangana 501512 2 Computer science and Engineering, Swathi Institute of Technology & Sciences Near Ramoji Film City Beside Kothagudem 'X' Roads, Hyderabad, Telangana 501512 3Computer science and Engineering, Swathi Institute of Technology & Sciences Near Ramoji Film City Beside Kothagudem 'X' Roads, Hyderabad, Telangana 501512 ,[email protected] ,[email protected] ,[email protected] Abstract Cyber security is a biggest Challenge. Protecting our digital lives is an issue of paramount importance. DDOS attacks are launched by adversaries using botnet, an army of compromised nodes hidden in the network. Compromised nodes are a set of nodes controlled by a botnet.DDOS attack is a most popular threat and is categorized as volumetric attack where the target destination is overwhelmed with large number of requests leading to impossibility of serving any users. In DDOS attack large number of machines act cooperatively under the supervision of one or more bot masters. These bots may be malicious users by themselves or maybe preliminarily infected.In recent years, there are increasing interests in using path identifiers (PIDs) as inter-domain routing objects. However, the PIDs used in existing approaches are static, which makes it easy for attackers to launch distributed denial-of service (DDoS) flooding attacks. To address this issue, we present the design, implementation, and evaluation of dynamic path identification based approach or a framework that uses PIDs negotiated between neighboring domains as inter-domain routing objects.
    [Show full text]
  • Delay and Traffic Rate Estimation in Network Tomography
    DELAY AND TRAFFIC RATE ESTIMATION IN NETWORK TOMOGRAPHY by Neshat Etemadi Rad A Dissertation Submitted to the Graduate Faculty of George Mason University In Partial fulfillment of The Requirements for the Degree of Doctor of Philosophy Electrical and Computer Engineering Committee: Dr. Yariv Ephraim, Co-director Dr. Brian L. Mark, Co-director Dr. Jill K. Nelson, Committee Member Dr. James Gentle, Committee Member Dr. Monson H. Hayes, Department Chair Dr. Kenneth S. Ball, Dean, Volgenau School of Information Technology and Engineering Date: Fall Semester 2015 George Mason University Fairfax, VA Delay and Traffic Rate Estimation in Network Tomography A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at George Mason University By Neshat Etemadi Rad Master of Science Sharif University of Technology, Tehran, Iran, 2010 Bachelor of Science Amirkabir University of Technology, Tehran, Iran, 2008 Co-director: Dr. Yariv Ephraim, Professor Co-director: Dr. Brian L. Mark, Professor Department of Electrical and Computer Engineering Fall Semester 2015 George Mason University Fairfax, VA Copyright c 2015 by Neshat Etemadi Rad All Rights Reserved ii Dedication To my parents, Avisa and Hamid, for without their early inspiration and coaching, none of this would have happened. To the love of my life, Abbas, for without his support and enthusiasm, none of this would have been accomplished. iii Acknowledgments I would like to thank my dissertation advisors, Professor Yariv Ephraim and Professor Brian L. Mark, for being supportive and patient throughout my PhD research at George Ma- son University. I am very grateful to them for their in-depth technical knowledge, guidance and insightful discussions.
    [Show full text]