DOCSLIB.ORG

Bank of America Quartz Worldmap Infrastructure Monitoring

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Bank of America Quartz Worldmap Infrastructure Monitoring Bank of America Quartz WorldMap Infrastructure Monitoring A Major Qualifying Project submitted to the Faculty of the WORCESTER POLYTECHNIC INSTITUTE in partial fulfillment of the requirements for the Degree of Bachelor of Science by Jun Liang Zhaokun Xue October 14, 2013 Approved Professor Micha Hofri Major Advisor 1 Abstract Bank of America’s Quartz platform, which is used for developing all internal applications by the company, is made up of over 600 application and database servers deployed in over 25 data centers globally. There are many variables that can contribute to issues in the environment for example CPU/memory consumption, disk space, network latency, replication latency, etc. Our team was responsible for building a web based interactive world map, named “Quartz WorldMap Infrastructure Monitoring”, which could graphically show the environment and alert when key performance thresholds are breached. In this project, our team kept focused on building a reliable back-end server and designing a user- friendly front-end client. Our team built two back-end services to poll data from databases in real time, and used modern jQuery open source libraries to construct functionalities and user interface of the front- end. We proposed and implemented the well-tested two-way data binding method to automatically synchronize data between databases and front-end display. 2 Acknowledgments From Bank of America, we would like to acknowledge the guidance of Brian Petix, Marquis Rahman and Michael Covington. From Worcester Polytechnic Institute, we would like to thank our advisors Arthur Gerstenfeld and Micha Hofri for their help throughout the process of the project. 3 Executive Summary After the merger with Merrill Lynch, Bank of America has become the largest financial institution in the world. With a global workforce in more than 40 countries, Bank of America is all about providing people, companies and institutional investors the financial products and services they need to help achieve their goals at every stage of their financial lives. Our team worked on the bank’s Global Markets Technology Team which provides end-to-end technology solutions and operations support for the Global Markets businesses. In addition, the team is also responsible for establishing an Architecture and Strategy framework for consistency across the Global Markets platforms. After years working of the team, Bank of America Merrill Lynch Global Markets built up the next generation, cross-asset technology platform, Quartz. The Quartz platform is made up of over 600 application and database servers deployed in over 25 data centers globally. There are many variables that can contribute to issues in the environment for example CPU/memory consumption, disk space, network latency, replication latency, etc. In order to keep tracking these variables, we are responsible for building a web based interactive world map that would graphically show the environment and alert when key performance thresholds are breached. After interviewing with the bank’s Quartz Develop Team members, Marquis Rahman in Chicago and Michael Covington in Houston, specifically, we have decided on the objective of building a web based monitoring application to provide the ability for users tracking, searching and making comments on alerts reported by each data center. So far, on the Quartz Platform, web based monitoring applications just start at the very beginning. This project could help the Quartz Develop Team to fill this gap on the Quartz Platform. In order to achieve this objective, we divided the application into two parts, back-end programming and front-end design. For the application’s back-end server (more details in section 3.1.1 and section 4.1.1), we wrote all the code in Python on the company’s Quartz platform. 4 First, to get alerts reported by 25 data centers, our team built a service called “Valkyrie Alert Service” which is used to retrieve alerts from Valkyrie Server by calling the Valkyrie API. Valkyrie Server is an alert message center that collects all the issues from Bank of America internal services. In order to get the latest alerts in real-time, our team set the “Valkyrie Alert Service” call the Valkyrie API every 5 seconds. After polling data from Valkyrie, the service publishes data into the AMPS server “alerts_in_valkyrie” topic. Second, we do not only just want get details of alerts, but we also want to combine more information of the corresponding data center like, line of business (LOB), region, organization, etc. Therefore, our team built the “Alert Engine and Decorator”. This service polls additional information of data centers from the Hardware Diagram Caches, then combines them with new alerts from the “alerts_in_valkyrie” topic, and publishes alerts to the “reported_alerts” topic. As for the front-end part (more details in section 3.1.2 and 4.1.2), we built it in JavaScript, HTML code and Flask pattern. First, for the major part--the map, after online search and testing the source given by our managers, we recommended to use an external open source called JVectorMap JQuery Plugin which not only supports all the features required for our project, but is easy to implement. On the map, we use 25 markers to represent the locations of the 25 data centers globally. Red markers stand for data centers having alerts reported; green markers stand for data centers without alerts. Hovering with a mouse over markers, a label above the marker will show the data center’s name and the number of alerts that have been reported in this data center so far. To synchronize the number of alerts in the data center in real time, we recommended using JSON and Two-Way Data Binding methods. By using JSON and Two-Way Data Binding, we could just update part of the page without refreshing the whole page. Second, clicking on one of the red markers, a grid will show the details of alerts below the map. For the gird, we recommended building it on the PQgrid JQuery open library. For each of the alerts shown in the grid, users have the ability to disable it for a given period of time. Users also have the choices to view all the alerts collected in the database and all the disabled alerts. On the disabled grid, 5 users can reactivate disabled alerts. Disabled alerts could also be automatically reactivated after the disabling period expires For the page layout (more details in section 4.2), above the map, we display a set of world clocks and user login information. On the right side of the map, a scrolling panel shows the latest five PAPA tickets. The PAPA ticket is another error tracking tool used by the bank. Below the map, we use a scrolling bar to show all the news about alerts in real time. Below the news bar, a filter panel provides the feature to do multi-selection search for alerts based on specific properties of data centers and alerts. In addition, user could also do the historical search by giving a period of time. Our creation of a new alerts monitoring tool was aimed to give the Quartz Develop Team at Bank of America the ability to track and make comments on alerts reported by each data center. In this way, the Quartz Develop Team was given the means by which they can keep records of alerts reported by each data center. 6 Table of Contents Abstract ......................................................................................................................................................... 2 Acknowledgments ......................................................................................................................................... 3 Executive Summary ...................................................................................................................................... 4 Chapter 1 Introduction ................................................................................................................................ 10 Chapter 2 Background ................................................................................................................................ 11 2.1 Sponsor Description .......................................................................................................................... 11 2.1.1 Global Research of Bank of America Merrill Lynch ................................................................. 11 2.1.2 Global Markets Technology Team in Bank of America ............................................................ 12 2.2 Quartz Development Platform .......................................................................................................... 12 2.2.1 Introduction to Quartz Platform ................................................................................................. 12 2.2.2 Sandra and AMPS ...................................................................................................................... 13 2.2.3 QZDevelop ................................................................................................................................. 13 2.2.4 QSP – Web Apps on Quartz ...................................................................................................... 13 2.3 Programming Languages .................................................................................................................. 14 2.3.1 Python ........................................................................................................................................ 14 2.3.2 JavaScript ..................................................................................................................................
Load more

Recommended publications
  • Modern Web Application Frameworks
    MASARYKOVA UNIVERZITA FAKULTA INFORMATIKY Û¡¢£¤¥¦§¨ª«¬­Æ°±²³´µ·¸¹º»¼½¾¿Ý Modern Web Application Frameworks MASTER’S THESIS Bc. Jan Pater Brno, autumn 2015 Declaration Hereby I declare, that this paper is my original authorial work, which I have worked out by my own. All sources, references and literature used or ex- cerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Bc. Jan Pater Advisor: doc. RNDr. Petr Sojka, Ph.D. i Abstract The aim of this paper was the analysis of major web application frameworks and the design and implementation of applications for website content ma- nagement of Laboratory of Multimedia Electronic Applications and Film festival organized by Faculty of Informatics. The paper introduces readers into web application development problematic and focuses on characte- ristics and specifics of ten selected modern web application frameworks, which were described and compared on the basis of relevant criteria. Practi- cal part of the paper includes the selection of a suitable framework for im- plementation of both applications and describes their design, development process and deployment within the laboratory. ii Keywords Web application, Framework, PHP,Java, Ruby, Python, Laravel, Nette, Phal- con, Rails, Padrino, Django, Flask, Grails, Vaadin, Play, LEMMA, Film fes- tival iii Acknowledgement I would like to show my gratitude to my supervisor doc. RNDr. Petr So- jka, Ph.D. for his advice and comments on this thesis as well as to RNDr. Lukáš Hejtmánek, Ph.D. for his assistance with application deployment and server setup. Many thanks also go to OndˇrejTom for his valuable help and advice during application development.
    [Show full text]
  • 1 Introducing Symfony, Cakephp, and Zend Framework
    1 Introducing Symfony, CakePHP, and Zend Framework An invasion of armies can be resisted, but not an idea whose time has come. — Victor Hugo WHAT’S IN THIS CHAPTER? ‰ General discussion on frameworks. ‰ Introducing popular PHP frameworks. ‰ Design patterns. Everyone knows that all web applications have some things in common. They have users who can register, log in, and interact. Interaction is carried out mostly through validated and secured forms, and results are stored in various databases. The databases are then searched, data is processed, and data is presented back to the user, often according to his locale. If only you could extract these patterns as some kind of abstractions and transport them into further applications, the developmentCOPYRIGHTED process would be much MATERIAL faster. This task obviously can be done. Moreover, it can be done in many different ways and in almost any programming language. That’s why there are so many brilliant solutions that make web development faster and easier. In this book, we present three of them: Symfony, CakePHP, and Zend Framework. They do not only push the development process to the extremes in terms of rapidity but also provide massive amounts of advanced features that have become a must in the world of Web 2.0 applications. cc01.indd01.indd 1 11/24/2011/24/2011 55:45:10:45:10 PPMM 2 x CHAPTER 1 INTRODUCING SYMFONY, CAKEPHP, AND ZEND FRAMEWORK WHAT ARE WEB APPLICATION FRAMEWORKS AND HOW ARE THEY USED? A web application framework is a bunch of source code organized into a certain architecture that can be used for rapid development of web applications.
    [Show full text]
  • Development of a Blog System Using Codeigniter Framework
    Li Yicheng Development of a blog system using CodeIgniter framework Development of a blog system using CodeIgniter framework Li Yicheng Bachelor‟s thesis Spring 2011 Degree Programme in Business Information Technology Oulu University of Applied Sciences Abstract School: Oulu University of Applied Sciences Programme: Degree Programme in Business Information Technology Author: Li Yicheng Supervisor: Matti Viitala The title of thesis: Development of a blog system using CodeIgniter framework Year: 2011 Number of pages + number of appendices: 59 This thesis is project based thesis. It is a simple blog system. In the thesis will introduces the development blog background and discuss the design and development platform targets, set out the structure of the website and the function modules and introduce how to build this system step by step. The whole system can be divided into three parts: Information services, Album service, Message service. However, due to the large size of the project, It cannot be explained how to make it step and step. So in this thesis will be focus more on the main part. In the theory part, will be discussed the exactly advantages of those programming language which will be used in the project. In the tutorial part, will be introduced how to build this blog system. Use the programming codes and pictures which from the project to describe clearly. This purpose of this bachelor‟s thesis is to introduce the advantages and techniques of PHP framework (CodeIgniter) and jQuery. Another purpose is to build an example system which is using PHP framework and jQuery to student who want to learn more about programming.
    [Show full text]
  • Symfony2 Docs Documentation Release 2
    Symfony2 Docs Documentation Release 2 Sensio Labs January 10, 2016 Contents 1 Quick Tour 1 1.1 Quick Tour................................................1 2 Book 23 2.1 Book................................................... 23 3 Cookbook 263 3.1 Cookbook................................................ 263 4 Components 455 4.1 The Components............................................. 455 5 Reference Documents 491 5.1 Reference Documents.......................................... 491 6 Bundles 617 6.1 Symfony SE Bundles........................................... 617 7 Contributing 619 7.1 Contributing............................................... 619 i ii CHAPTER 1 Quick Tour Get started fast with the Symfony2 Quick Tour: 1.1 Quick Tour 1.1.1 The Big Picture Start using Symfony2 in 10 minutes! This chapter will walk you through some of the most important concepts behind Symfony2 and explain how you can get started quickly by showing you a simple project in action. If you’ve used a web framework before, you should feel right at home with Symfony2. If not, welcome to a whole new way of developing web applications! Tip: Want to learn why and when you need to use a framework? Read the “Symfony in 5 minutes” document. Downloading Symfony2 First, check that you have installed and configured a Web server (such as Apache) with PHP 5.3.2 or higher. Ready? Start by downloading the “Symfony2 Standard Edition”, a Symfony distribution that is preconfigured for the most common use cases and also contains some code that demonstrates how to use Symfony2 (get the archive with the vendors included to get started even faster). After unpacking the archive under your web server root directory, you should have a Symfony/ directory that looks like this: www/ <- your web root directory Symfony/ <- the unpacked archive app/ cache/ config/ logs/ Resources/ bin/ src/ Acme/ DemoBundle/ Controller/ Resources/ ..
    [Show full text]
  • Elgg Documentation Release Master
    Elgg Documentation Release master Various Sep 24, 2021 Contents 1 Features 3 2 Examples 5 3 Continue Reading 7 i ii Elgg Documentation, Release master Elgg( pronunciation) is an open source rapid development framework for socially aware web applications. It is a great fit for building any app where users log in and share information. Contents 1 Elgg Documentation, Release master 2 Contents CHAPTER 1 Features • Well-documented core API that allows developers to kick start their new project with a simple learning curve • Composer is the package manager of choice that greatly simplifes installation and maintenance of Elgg core and plugins • Flexible system of hooks and events that allows plugins to extend and modify most aspects of application’s functionality and behavior • Extendable system of views that allows plugins to collaborate on application’s presentation layer and built out complex custom themes • Cacheable system of static assets that allows themes and plugins to serve images, stylesheets, fonts and scripts bypassing the engine • User authentication is powered by pluggable auth modules, which allow applications to implement custom authentication protocols • Security is ensured by built-in anti CSRF validation, strict XSS filters, HMAC signatures, latest cryptographic approaches to password hashing • Client-side API powered by asynchronous JavaScript modules via RequireJS and a build-in Ajax service for easy communication with the server • Flexible entity system that allows applications to prototype new types of content and user interactions
    [Show full text]
  • Python Guide Documentation 0.0.1
    Python Guide Documentation 0.0.1 Kenneth Reitz 2015 11 07 Contents 1 3 1.1......................................................3 1.2 Python..................................................5 1.3 Mac OS XPython.............................................5 1.4 WindowsPython.............................................6 1.5 LinuxPython...............................................8 2 9 2.1......................................................9 2.2...................................................... 15 2.3...................................................... 24 2.4...................................................... 25 2.5...................................................... 27 2.6 Logging.................................................. 31 2.7...................................................... 34 2.8...................................................... 37 3 / 39 3.1...................................................... 39 3.2 Web................................................... 40 3.3 HTML.................................................. 47 3.4...................................................... 48 3.5 GUI.................................................... 49 3.6...................................................... 51 3.7...................................................... 52 3.8...................................................... 53 3.9...................................................... 58 3.10...................................................... 59 3.11...................................................... 62
    [Show full text]
  • Zend Framework : Bien Développer En
    __ g les Programmez intelligent Cahiers avec du les Pauli Cahiers Ponçon J. Programmeur du Programmeur G. Architecte certifié PHP et Zend Framework, Julien Pauli est responsable du pôle Zend Frame- Framework work/PHP chez Anaska (groupe Zend Alter Way). Contributeur de la pre- mière heure au framework en colla- En imposant des règles strictes de gestion de code et en offrant une très boration avec Zend Technologies, riche bibliothèque de composants prêts à l’emploi, le framework PHP 5 Zend conférencier et membre de l’AFUP, Framework guide le développeur web dans l’industrialisation de ses dévelop- il publie des articles sur PHP dans la pements, afin d’en garantir la fiabilité, l’évolutivité et la facilité de maintenance. presse. Fondateur et gérant de la société Cet ouvrage présente les meilleures pratiques de développement web avec OpenStates (partenaire Zend PHP 5 et le Zend Framework : design patterns, MVC, base de données, sécu- Technologies et Anaska), Bien développer en PHP rité, interopérabilité, tests unitaires, gestion des flux et des sessions, etc. Guillaume Ponçon intervient Non sans rappeler les prérequis techniques et théoriques à l’utilisation du fra- depuis plus de sept ans auprès de Julien Pauli mework, l’ouvrage aidera tant les développeurs débutants en PHP que les grands comptes sur de nom- chefs de projets ou architectes aguerris souhaitant l’utiliser en entreprise. breuses missions d’expertise, de Guillaume Ponçon conseil et de formation PHP. Ingé- nieur EPITA, expert certifié PHP et Zend Framework, il est aussi spé- cialiste des systèmes Unix/Linux et Framework Préface de Wil Sinclair pratique Java et C/C++.
    [Show full text]
  • 2Nd USENIX Conference on Web Application Development (Webapps ’11)
    conference proceedings Proceedings of the 2nd USENIX Conference Application on Web Development 2nd USENIX Conference on Web Application Development (WebApps ’11) Portland, OR, USA Portland, OR, USA June 15–16, 2011 Sponsored by June 15–16, 2011 © 2011 by The USENIX Association All Rights Reserved This volume is published as a collective work. Rights to individual papers remain with the author or the author’s employer. Permission is granted for the noncommercial reproduction of the complete work for educational or research purposes. Permission is granted to print, primarily for one person’s exclusive use, a single copy of these Proceedings. USENIX acknowledges all trademarks herein. ISBN 978-931971-86-7 USENIX Association Proceedings of the 2nd USENIX Conference on Web Application Development June 15–16, 2011 Portland, OR, USA Conference Organizers Program Chair Armando Fox, University of California, Berkeley Program Committee Adam Barth, Google Inc. Abdur Chowdhury, Twitter Jon Howell, Microsoft Research Collin Jackson, Carnegie Mellon University Bobby Johnson, Facebook Emre Kıcıman, Microsoft Research Michael E. Maximilien, IBM Research Owen O’Malley, Yahoo! Research John Ousterhout, Stanford University Swami Sivasubramanian, Amazon Web Services Geoffrey M. Voelker, University of California, San Diego Nickolai Zeldovich, Massachusetts Institute of Technology The USENIX Association Staff WebApps ’11: 2nd USENIX Conference on Web Application Development June 15–16, 2011 Portland, OR, USA Message from the Program Chair . v Wednesday, June 15 10:30–Noon GuardRails: A Data-Centric Web Application Security Framework . 1 Jonathan Burket, Patrick Mutchler, Michael Weaver, Muzzammil Zaveri, and David Evans, University of Virginia PHP Aspis: Using Partial Taint Tracking to Protect Against Injection Attacks .
    [Show full text]
  • Python Guide Documentation 0.0.1
    Python Guide Documentation 0.0.1 Kenneth Reitz 2015 09 13 Contents 1 Getting Started 3 1.1 Picking an Interpreter..........................................3 1.2 Installing Python on Mac OS X.....................................5 1.3 Installing Python on Windows......................................6 1.4 Installing Python on Linux........................................7 2 Writing Great Code 9 2.1 Structuring Your Project.........................................9 2.2 Code Style................................................ 15 2.3 Reading Great Code........................................... 24 2.4 Documentation.............................................. 24 2.5 Testing Your Code............................................ 26 2.6 Common Gotchas............................................ 30 2.7 Choosing a License............................................ 33 3 Scenario Guide 35 3.1 Network Applications.......................................... 35 3.2 Web Applications............................................ 36 3.3 HTML Scraping............................................. 41 3.4 Command Line Applications....................................... 42 3.5 GUI Applications............................................. 43 3.6 Databases................................................. 45 3.7 Networking................................................ 45 3.8 Systems Administration......................................... 46 3.9 Continuous Integration.......................................... 49 3.10 Speed..................................................
    [Show full text]
  • Configuring Laravel
    www.allitebooks.com Learning Laravel 4 Application Development Develop real-world web applications in Laravel 4 using its refined and expressive syntax Hardik Dangar BIRMINGHAM - MUMBAI www.allitebooks.com Learning Laravel 4 Application Development Copyright © 2013 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. First published: December 2013 Production Reference: 1171213 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78328-057-5 www.packtpub.com Cover Image by Aniket Sawant ([email protected]) www.allitebooks.com Credits Author Copy Editors Hardik Dangar Roshni Banerjee Brandt D'Mello Reviewers Janbal Dharmaraj
    [Show full text]
  • Zadání Bakalářské Práce
    ČESKÉ VYSOKÉ UČENÍ TECHNICKÉ V PRAZE FAKULTA INFORMAČNÍCH TECHNOLOGIÍ ZADÁNÍ BAKALÁŘSKÉ PRÁCE Název: BigCloud - uživatelské rozhraní pro služby SaaS a PaaS Student: Jindřich Máca Vedoucí: Ing. Jiří Chludil Studijní program: Informatika Studijní obor: Web a multimédia Katedra: Katedra softwarového inženýrství Platnost zadání: Do konce letního semestru 2016/17 Pokyny pro vypracování Na základě požadavků zadavatele (SIC, s.r.o.) navrhněte a implementujte minimálně 5 modulů k poskytování vybraných SaaS a PaaS služeb pro veřejný cloudový systém. 1. Analyzujte požadované SaaS a PaaS služby (OwnCloud, RemoteDesktop, Databázové úložiště, atd.) a jejich způsob zavedení do stávajícího systému. 2. Analyzujte stávající řešení (u IaaS služeb) komunikace s jádrem systému (správa virtuálních počítačů) a zvažte způsob jeho využití pro podporu SaaS a PaaS služeb. 3. Analyzujte stávající implementaci uživatelského rozhraní (MVP) pod prostředím NETTE a navrhněte jeho optimalizaci. 4. Na základě analýz navrhněte nové moduly (uživatelské rozhraní a jeho komunikaci s jádrem) k poskytování požadovaných SaaS a PaaS služeb. 5. Implementujte moduly (včetně navrhovaných optimalizací). 6. Podrobte implementované moduly vhodným testům (jednotkové, integrační a usability). Seznam odborné literatury Dodá vedoucí práce. L.S. Ing. Michal Valenta, Ph.D. prof. Ing. Pavel Tvrdík, CSc. vedoucí katedry děkan V Praze dne 24. prosince 2015 České vysoké učení technické v Praze Fakulta informačních technologií Katedra softwarového inženýrství Bakalářská práce BigCloud - uživatelské rozhraní pro služby SaaS a PaaS Jindřich Máca Vedoucí práce: Ing. Jiří Chludil 30. června 2016 Poděkování Především bych chtěl poděkovat vedoucímu práce, Jiřímu Chludilovi, za jeho podporu a cenné rady při psaní této práce. Velké díky patří také panu Petru Marešovi za četné konzultace, vstřícnost a celkovou spolupráci.
    [Show full text]
  • Webmethods Product Suite 9.8 Release Notes
    webMethods Product Suite 9.8 Release Notes Release 9.8 of the webMethods product suite represents a significant step towards the establishment of Software AG’s Digital Business Platform. In addition to expanding the webMethods cloud offerings, this major release improves developer productivity, supports key emerging standards, and makes business users more productive while also helping them make better decisions through improved analytics. Our goal with this release is to provide you with the agility to fully power your digital enterprise. Release Highlights Integration This release of the webMethods suite includes many integration enhancements. It includes advances in cloud integration, declarative template-based provisioning, ODATA support, hot deployment capabilities, advanced API management, and much more. The key features include: • webMethods Integration Cloud now enables the development of simple multi-step orchestration flows between any SaaS or on-premises applications, providing support for more complex integration patterns. • webMethods Integration Server now supports the Salesforce Lightning Connect standard, which is based on OData. This new standard enables applications hosted on Force.com to easily connect and access data from external sources using a simple point-and and-click model. • webMethods Integration Server offers native support for OData version 2.0. Native OData version 2.0 support enables a REST-based protocol for CRUD-style operations (Create, Read, Update and Delete) against resources exposed as data service. • Software AG Command Central allows provisioning of complex multi-server environments using declarative templates. Users can define environment-specific server mappings and configurations with the simplest possible set of configuration files. • Software AG Command Central encrypts passwords in templates using a secure shared- secret.
    [Show full text]