DOCSLIB.ORG

Infrastructure and Performance

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Infrastructure and Performance Infrastructure and Performance An Islandoracon Workshop Instructors: Gavin Morris & Luke Taylor Instructor: Luke Taylor DevOps Team Lead discoverygarden inc. 155 Queen St. Suite 101 Charlottetown, PE C1A 4B4 discoverygarden.ca [email protected] Instructors - Gavin Morris ● Team Lead & Dev Ops at Born Digital ○ http://born-digital.com/ ● Convener of the Islandora DevOps Interest Group ○ https://github.com/islandora-interest-groups/Islandora- DevOps-Interest-Group ● Led the Islandora DevOps Panel: Building Islandora at the inaugural Islandora Conference (Islandoracon) on Prince Edward Island, Canada ● Presented Automating Islandora Upgrations, Maintenance and Deploys at the Islandora Camp in Hartford, CT Overview ● Intro to the stack ● Security ○ Build Types ○ Split the stack ● ○ Operating Systems Best Practices ○ Packages ○ Services / Software ● The future of the stack ○ CLAW ● Provisioning ○ ISLE ○ Deploy & Config management tools ○ Pipeline ● Q&A ● Performance ● Resources ● Scaling Intro to the Stack - What is Islandora? Source: http://islandora.mnpals.net/pals/islandora/object/PALSrepository%3A412/datastream/OBJ/download/2016-08_Detailed_Islandora_Introduction.pdf Intro to the Stack - Build Types ALL All in One 2-3 servers 5-7 servers Intro to the Stack - Build Types: All in One Recommended Minimum ● 4-6 cores ● 16GB - 32GB RAM ● 100-200GB for OS, Temp files etc ● Volume large enough for repository data ● Additional space could be required for staging content Intro to the Stack - Build Types: 2-3 servers Web & Database Server (Minimum Requirements) ● 1-2 cores ● 4 - 16 GB RAM (*depends on platform type e.g. staging, dev) ● 150-250GB for OS, Temp files etc ● Additional space could be required for staging content Fedora Repository Server (Minimum Requirements) ● 2-4 cores ● 8 - 32 GB RAM (*depends on collection size) ● 150-250GB for OS, Temp files etc ● Additional space / volume for repository data e.g. 2 -20 TB Intro to the Stack - Build Types: 2-3 servers Intro to the Stack - Build Types: 5-7 servers Storage mount (e.g. NFS) Staff/Ingest Front End Public Front End Read-only fedora DB server Blazegraph Solr Read-write Fedora Intro to the Stack - Split the Stack ● Remote Solr ○ Use Gsearch 2.8+ ○ Edit fgsindex.indexBase in index.properties in Gsearch. ○ Still have to maintain a “dummy” index on the Gsearch server. ● Blazegraph ○ Used to replace Mulgara (Triplestore) for performance and stability gains ○ https://github.com/discoverygarden/trippi-sail ○ https://github.com/Smithsonian/trippi-sparql Intro to the Stack - Operating Systems Current Stable Recommendations ● Ubuntu 14.04 LTS ● RHEL/CentOS 6.9 Needing more definitive testing ● Ubuntu 16.04 TLS (w/PHP7) ● RHEL/CentOS 7 (challenges with temporary file system) Community Poll from Melissa Anez (Have your say!) ● Survey https://docs.google.com/forms/d/1E7NmS4944LD3E51A7SK_8MiNoOWCPnjgY8YWOUC7I-o ● Google Group topic https://groups.google.com/forum/?hl=en#!searchin/islandora/php$20testing|sort:relevance/islandora/WftNSPr7Xi0/vlh6eJU bAwAJ Intro to the Stack - Operating Systems packages (basic) man vim curl perl unzip automake subversion kernel-headers gcc zip dkms bzip2 openssh mercurial pkg-config build-essential git wget htop cmake libtool apt-utils kernel-devel libfreetype6-dev ntp yasm nasm rsync autoconf zlib1g-dev linux-headers Development tools Intro to the Stack - Services / Software ● Apache 2.2 - 2.4 Web server ○ Modules include but are not limited to ■ ssl, rewrite, deflate, headers, expires, xml2enc ■ reverse proxy for multi-systems: ● proxy, proxy_http, proxy_html, proxy_connect ● Databases ○ Mysql 5.5+ ○ Percona ○ Mariadb ○ Postgres ○ Recommend UTF-8 encoding Intro to the Stack - Services / Software ● Tomcat 7.0.52 + ○ Oracle Java JDK or OpenJDK 7/8 ○ SSL & port 8443 ■ Will need to compile own jks/P12/truststore (how to automate?) ○ see Gotcha section re versions above 7.0.72/8.0.39+ ● Apache Solr ○ versions 4.2, 4.6.1, 4.10 ○ Don’t use Gsearch Ant generated schema (not complete), missing catch_all entries etc. ○ Always helpful for starting out for schema & solrconfig .xml files https://github.com/discoverygarden/basic-solr-config Intro to the Stack - Services / Software ● PHP 5.3.x+ ○ Drupal 7.5.4 ■ Islandora 7.x / HEAD modules ■ Additional modules e.g. ctools, imagemagick, date, views etc. ○ Composer ■ Drush ● Fedora-Commons 3.8.1 ○ Triplestore (mulgara, Blazegraph) ● Fedoragsearch HEAD / 2.7.1 ○ DGI GSearch Extensions https://github.com/discoverygarden/dgi_gsearch_extensions ○ XSL Transforms for Gsearch https://github.com/discoverygarden/islandora_transforms Intro to the Stack - Services / Software ● Binaries, Derivative generation ○ Imagemagick ○ LAME (audio, mp3 etc) ○ FFMPEG (video) from source 3.3 ○ FITS ○ EXIF ○ XPDF ○ Ghostscript 9.05 (from source) ○ Tesseract (OCR) ○ Adore-djatoka 1.1 ■ On multi-system setups libraries should be additionally installed on web servers ■ Requires use of Oracle JDK 7/8 Provisioning - Deploy & Config management tools Puppet DSL / Ruby Free Puppet Enterprise https://puppet.com/ (up to 10 nodes) Chef DSL / Ruby Free Chef Automate / Hosted https://www.chef.io Ansible DSL / Python Free Tower https://www.ansible.com/ (Red Hat owned) (agentless) Saltstack DSL / Python Salt Open Salt Enterprise https://saltstack.com/ CFEngine DSL / C Community Edition CFEngine Enterprise https://cfengine.com/ https://www.gnu.org/softw Shell Scripts Bash / sh Free are/bash/ Packer DSL / JSON Free Builds Images https://www.packer.io/ Developer #1 Provisioning - Pipeline Production Web & Db server VM Web & DB server Fedora server VM Theming, solution packs, Fedora repo server modules, XSLTs, schemas, config etc. Code Up! Developer #2 Package & software updates, system Web & Db server VM configuration changes, data Fedora server VM migrations, re-indexing of triplestore etc. Data Down! Development Developer #3 Web & DB server Continuous Integration w/ Web & Db server VM Testing Suites for Code & Fedora repo server Fedora server VM Data Example Pipeline Performance ● Using Solr vs SPARQL/iTQL ○ Collection Solution Pack (Display Generation) ○ Islandora OAI (Query Backend) ○ Paged Content Module (Use Solr to derive pages and sequence numbers) ○ Breadcrumbs (Breadcrumb Generation) ● Breadcrumbs - Disable if not required or use Solr ● Enable Drupal caching options (Configuration - Development - Performance) ● Memcached / Varnish Performance “(XmlUsersFileModule) null” error ERROR 2017-03-10 08:56:54.796 [http-8080-21] (XmlUsersFileModule) null ERROR 2017-03-10 08:56:54.805 [http-8080-21] (AuthFilterJAAS) javax.security.auth.login.LoginException: Login Failure: all modules ignored Source: /usr/local/fedora/server/logs/fedora.log Reference: https://issues.apache.org/jira/browse/XERCESJ-211 https://jira.duraspace.org/browse/FCREPO-1230 Fix! https://github.com/discoverygarden/fcrepo3-security-jaas Performance ● Help too many multisites! ○ Islandora installations with Drupal multisites can cause unnecessary database connections. ● Multi-site optimization ○ https://github.com/discoverygarden/fcrepo3-security-jaas Performance ● Islandora Jobs ○ https://github.com/discoverygarden/islandora_job ○ Faster Ingests ○ Allows you to have multiple Gearman workers processing derivatives. ● Islandora Gsearcher ○ https://github.com/discoverygarden/islandora_gsearcher ○ Updates Solr index upon ingest completion vs waiting for ActiveMQ Security ● Directory permissions Tomcat/Drupal ● Run services using non-privileged users with no shell. ● Firewalls ○ Fail2ban (https://www.fail2ban.org) ○ Modsec (https://modsecurity.org/) ○ Ports / Rules ● Central logging ○ Syslog ○ Tripwire (https://www.tripwire.com/) (can be used for extended logging in addition to security) ○ ELK (ElasticSearch, Logstash & Kibana) https://logz.io/learn/complete-guide-elk-stack/ Best Practices, Gotchas, Tips ● Gsearch issues Tomcat 7.0.72/8.0.39+ ○ https://github.com/discoverygarden/gsearch.git ● Try the Islandora Deploy on Ubuntu guide https://github.com/islandora-interest-groups/Islandora-DevOps-Interest-Group/blob/master/Deployment %20Guides/Provisioning-Islandora-on-Ubuntu.md ● AWS S3 mounting as a file system ○ https://github.com/danilop/yas3fs ■ Debug mode first! ■ Make sure it re-mounts properly if system is restarted. ■ Gotcha: There may be an object size limit of 60 GB for ingested binaries e.g. video etc. ■ Mount the datastreamStore to S3 and leave objectStore on EBS for better performance ● Caution! Challenges with restoration! ○ Alternative https://bitbucket.org/nikratio/s3ql (same Gotchas apply!) The future of the stack - Islandora 7.2.x - CLAW https://github.com/Islandora-CLAW/CLAW/blob/master/docs/user-documentation/i ntro-to-claw.md https://github.com/Islandora-CLAW/CLAW/blob/master/docs/mvp/mvp_doc.md The future of the stack - ISLE Islandora + = Enterprise (ISLE) https://github.com/Islandora-Collaboration-Group https://islandora-collaboration-group.github.io/ https://islandora.ca/content/islandora-together-meet-islandora-consortial-group Q&A Resources ● Islandora http://islandora.ca ● Islandora sandbox https://sandbox.islandora.ca/ ● Vagrant up with Islandora Labs! https://github.com/Islandora-Labs/islandora_vagrant ● Please join the growing global community! http://islandora.ca/membership ● Perhaps jump on a call with one of the Islandora Interest groups? ○ https://github.com/islandora-interest-groups ○ https://github.com/islandora-interest-groups/Islandora-DevOps-Interest-Group ● One can learn so much from the Islandora Community on Google Groups! ○ https://groups.google.com/forum/?hl=en#!forum/islandora-dev ○ https://groups.google.com/forum/?hl=en#!forum/islandora Thank you!.
Load more

Recommended publications
  • Hidden Gears of Your Application
    Hidden gears of your application Sergej Kurakin Problem ● Need for quick response ● Need for many updates ● Need for different jobs done ● Need for task to be done as different user on server side ● Near real-time job start ● Load distribution Job Queue ● You put job to queue ● Worker takes the job and makes it done Job Queue using Crons ● Many different implementations ● Perfect for small scale ● Available on many systems/servers ● Crons are limited to running once per minute ● Harder to distribute load Gearman Job Server ● Job Queue ● http://gearman.org/ ● C/C++ ● Multi-language ● Scalable and Fault Tolerant ● Huge message size (up to 4 gig) Gearman Stack Gearman Job Types Normal Job Background Job ● Run Job ● Run Job in ● Return Result Background ● No Return of Result Gearman Parallel Tasks Gearman Supported Languages ● C ● Java ● Perl ● C#/.NET ● NodeJS ● Ruby ● PHP ● Go ● Python ● Lisp Job Priority ● Low ● Normal ● High Gearman Worker Example <?php // Reverse Worker Code $worker = new GearmanWorker(); $worker->addServer(); $worker->addFunction("reverse", function ($job) { return strrev($job->workload()); }); while ($worker->work()); Gearman Client Example <?php // Reverse Client Code $client = new GearmanClient(); $client->addServer(); print $client->do("reverse", "Hello World!"); Gearman Client Example <?php // Reverse Client Code $client = new GearmanClient(); $client->addServer(); $client->doBackground("reverse", "Hello World!"); Running Worker in Background ● CLI ● screen / tmux ● supervisord - http://supervisord.org/ ● daemontools
    [Show full text]
  • VSI's Open Source Strategy
    VSI's Open Source Strategy Plans and schemes for Open Source so9ware on OpenVMS Bre% Cameron / Camiel Vanderhoeven April 2016 AGENDA • Programming languages • Cloud • Integraon technologies • UNIX compability • Databases • Analy;cs • Web • Add-ons • Libraries/u;li;es • Other consideraons • SoDware development • Summary/conclusions tools • Quesons Programming languages • Scrip;ng languages – Lua – Perl (probably in reasonable shape) – Tcl – Python – Ruby – PHP – JavaScript (Node.js and friends) – Also need to consider tools and packages commonly used with these languages • Interpreted languages – Scala (JVM) – Clojure (JVM) – Erlang (poten;ally a good fit with OpenVMS; can get good support from ESL) – All the above are seeing increased adop;on 3 Programming languages • Compiled languages – Go (seeing rapid adop;on) – Rust (relavely new) – Apple Swi • Prerequisites (not all are required in all cases) – LLVM backend – Tweaks to OpenVMS C and C++ compilers – Support for latest language standards (C++) – Support for some GNU C/C++ extensions – Updates to OpenVMS C RTL and threads library 4 Programming languages 1. JavaScript 2. Java 3. PHP 4. Python 5. C# 6. C++ 7. Ruby 8. CSS 9. C 10. Objective-C 11. Perl 12. Shell 13. R 14. Scala 15. Go 16. Haskell 17. Matlab 18. Swift 19. Clojure 20. Groovy 21. Visual Basic 5 See h%p://redmonk.com/sogrady/2015/07/01/language-rankings-6-15/ Programming languages Growing programming languages, June 2015 Steve O’Grady published another edi;on of his great popularity study on programming languages: RedMonk Programming Language Rankings: June 2015. As usual, it is a very valuable piece. There are many take-away from this research.
    [Show full text]
  • Technical Update & Roadmap
    18/05/2018 VMS Software Inc. (VSI) Technical Update & Roadmap May 2018 Brett Cameron Agenda ▸ What we've done to date ▸ Product roadmap ▸ TCP/IP update ▸ Upcoming new products ▸ Support roadmap ▸ Storage ▸ x86 update, roadmap, and licensing ▸ x86 servers ▸ ISV programme ▸ Other stuff 1 18/05/2018 WhatDivider we’ve with done to date OpenVMS releases to date 3 OpenVMS I64 releases: Plus … Two OpenVMS Alpha Japanese version • V8.4-1H1 – Bolton releases: • DECforms V4.2 • – June 2015 V8.4-2L1 – February • DCPS V2.8 2017 • V8.4-2 - Maynard • FMS V2.6 • Standard OpenVMS – March 2016 • DECwindows Motif V1.7E release (Hudson) • V8.4-2L1 – Hudson • V8.4-2L2 - April 2017 – August 2016 • Performance build, EV6/EV7 (Felton) 2 18/05/2018 Products introduced from 2015 to date Technical achievements to date 65 Layered Product Releases 12 Open Source Releases OpenVMS Releases 4 VSI OpenVMS Statistics 515 VSI Defect Repairs 179 New Features Since V7.3-2 0 100 200 300 400 500 600 3 18/05/2018 Defect repairs 515 Total Defect Repairs 343 Source - Internal BZ VSI - Defect Repairs 118 Source - External BZ 54 Source - Quix 0 100 200 300 400 500 600 Testing hours Test Hours Per VSI OpenVMS Version TKSBRY IA64 V8.4-2L1 Test Hours V8.4-2 V8.4-1H1 0 20000 40000 60000 80000 100000 120000 140000 4 18/05/2018 ProductDivider roadmap with OpenVMS Integrity operating environment Released Planned BOE Components: • NOTARY V1.0 BOE Components: • • V8.4-2L1 operating system OpenSSL V1.02n • CSWS additional modules • • ANT V1.7-1B Perl V5.20-2A • PHP additional modules •
    [Show full text]
  • List of TCP and UDP Port Numbers from Wikipedia, the Free Encyclopedia
    List of TCP and UDP port numbers From Wikipedia, the free encyclopedia This is a list of Internet socket port numbers used by protocols of the transport layer of the Internet Protocol Suite for the establishment of host-to-host connectivity. Originally, port numbers were used by the Network Control Program (NCP) in the ARPANET for which two ports were required for half- duplex transmission. Later, the Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP) needed only one port for full- duplex, bidirectional traffic. The even-numbered ports were not used, and this resulted in some even numbers in the well-known port number /etc/services, a service name range being unassigned. The Stream Control Transmission Protocol database file on Unix-like operating (SCTP) and the Datagram Congestion Control Protocol (DCCP) also systems.[1][2][3][4] use port numbers. They usually use port numbers that match the services of the corresponding TCP or UDP implementation, if they exist. The Internet Assigned Numbers Authority (IANA) is responsible for maintaining the official assignments of port numbers for specific uses.[5] However, many unofficial uses of both well-known and registered port numbers occur in practice. Contents 1 Table legend 2 Well-known ports 3 Registered ports 4 Dynamic, private or ephemeral ports 5 See also 6 References 7 External links Table legend Official: Port is registered with IANA for the application.[5] Unofficial: Port is not registered with IANA for the application. Multiple use: Multiple applications are known to use this port. Well-known ports The port numbers in the range from 0 to 1023 are the well-known ports or system ports.[6] They are used by system processes that provide widely used types of network services.
    [Show full text]
  • Livejournal: Behind the Scenes Scaling Storytime
    LiveJournal: Behind The Scenes Scaling Storytime April 2007 Brad Fitzpatrick [email protected] danga.com / livejournal.com / sixapart.com This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/1.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA. http://danga.com/words/ 1 This Talk’s Gracious Sponsor Buy your servers from them Much love They didn’t even ask for me to put this slide in. :) http://danga.com/words/ 2 The plan... Refer to previous presentations for more details... http://danga.com/words/ Questions anytime! Yell. Interrupt. Part N: − show where talk will end up Part I: − What is LiveJournal? Quick history. − LJ’s scaling history Part II: − explain all our software, − explain all the moving parts http://danga.com/words/ 3 net. LiveJournal Backend: Today (Roughly.) BIG-IP perlbal (httpd/proxy) Global Database bigip1 mod_perl bigip2 proxy1 web1 master_a master_b proxy2 web2 proxy3 web3 Memcached slave1 slave2 ... slave5 djabberd proxy4 web4 mc1 djabberd proxy5 ... mc2 djabberd User DB Cluster 1 webN mc3 uc1a uc1b mc4 User DB Cluster 2 ... uc2a uc2b gearmand Mogile Storage Nodes gearmand1 mcN User DB Cluster 3 sto1 sto2 Mogile Trackers gearmandN uc3a uc3b ... sto8 tracker1 tracker3 User DB Cluster N ucNa ucNb MogileFS Database “workers” gearwrkN Job Queues (xN) mog_a mog_b theschwkN jqNa jqNb http:/slave1/danga.comslaveN/words/ 4 LiveJournal Overview college hobby project, Apr 1999 4-in-1: − blogging − forums − social-networking (“friends”) − aggregator: “friends page” + RSS/Atom 10M+ accounts Open Source! − server, − infrastructure, − original clients, − ..
    [Show full text]
  • Data Management Organization Charter
    Large Synoptic Survey Telescope (LSST) Database Design Jacek Becla, Daniel Wang, Serge Monkewitz, K-T Lim, Douglas Smith, Bill Chickering LDM-135 08/02/2013 LSST Database Design LDM-135 08/02/13 Change Record Version Date Description Revision Author 1.0 6/15/2009 Initial version Jacek Becla 2.0 7/12/2011 Most sections rewritten, added scalability test Jacek Becla section 2.1 8/12/2011 Refreshed future-plans and schedule of testing Jacek Becla, sections, added section about fault tolerance Daniel Wang 3.0 8/2/2013 Synchronized with latest changes to the Jacek Becla, requirements (LSE-163). Rewrote most of the Daniel Wang, “Implementation” chapter. Documented new Serge Monkewitz, tests, refreshed all other chapters. Kian-Tat Lim, Douglas Smith, Bill Chickering 2 LSST Database Design LDM-135 08/02/13 Table of Contents 1. Executive Summary.....................................................................................................................8 2. Introduction..................................................................................................................................9 3. Baseline Architecture.................................................................................................................10 3.1 Alert Production and Up-to-date Catalog..........................................................................10 3.2 Data Release Production....................................................................................................13 3.3 User Query Access.............................................................................................................13
    [Show full text]
  • Laith Shadeed [email protected] +31627249393 L3.Ai
    Laith Shadeed [email protected] +31627249393 l3.ai EXPERIENCE Senior Full-Stack Software Engineer AIMMS, Haarlem 2017 - Current Decrease page load 50th percentile from 20s to below 4s (Javascript, Node.js). Reduce GitLab end-2-end test pipeline execution time from 46 min to 3 min (Docker, Bash). Maintaining back-end REST APIs & TCP proxy in Node.js Stack: AWS, Docker, GitLab, Javascript, Node.js, Webpack and jQuery. Lead Software Engineer SuperSaaS, Amsterdam 2016 - 2017 Add credit card payments. 60% of new subscriptions start using it in the 1st week (Ruby, Javascript). Add automated bare-metal server provisioning, monitoring, real-time logging and alerting (Bash, Perl). Stack: Passenger, Capistrano, Apache httpd, MySQL, Monit, Munin, Nagios and Ubuntu. Back-end Software Engineer Booking.com, Amsterdam 2014 - 2016 Launch Apple Map integration with Booking.com Hotels for the first time for Mac users (Perl). Improve response time of Maps XHR requests by 50% (Perl, Javascript). On the hiring team for final face-face interviews. Stack: uWSGI, Ngnix, MySQL, Puppet, Redis, Hadoop, Graphite and Debian. Senior Back-end Software Engineer One.com, Dubai 2012 - 2014 In-house Photo Processing, Video Transcoding, Job Queuing serving +200,000 web domains (PHP, Node.js). Lead the design of v2 & v3 of the photo application (PHP, Javascript). Stack: Node.js, PostgreSQL, Apache httpd, Varnish, Gearman, Redis, Puppet and Debian. Back-end Software Engineer Yahoo!, Amman & Bangalore 2009 - 2012 Launch Y! News Mobile for Middle East market serving 17 countries (PHP, Javascript). Part of Y! Mail team launching the product for +70 million Arabic users (PHP, Javascript). Stack: Apache httpd, MySQL, Memcache, Apache Traffic Server and Redhat.
    [Show full text]
  • Ask Bjørn Hansen Develooper LLC [email protected]
    If this text is too small to read, move closer! Real World Web Scalability Slides at http://develooper.com/talks/ Ask Bjørn Hansen Develooper LLC [email protected] r8 Hello. • I’m Ask Bjørn Hansen • Tutorial in a box 59 minutes! • 53* brilliant° tips to make your website keep working past X requests/transactions per T time • Requiring minimal extra work! (or money) • Concepts applicable to ~all languages and platforms! * Estimate, your mileage may vary ° Well, a lot of them are pretty good Construction Ahead! • Conflicting advice ahead • Not everything here is applicable to everything • Ways to “think scalable” rather than end- all-be-all solutions Questions ... • Did anyone see this talk at OSCON or the MySQL UC before? • ... are using Perl? PHP? Python? Java? Ruby? • ... Oracle? • The first, last and only lesson: • Think Horizontal! • Everything in your architecture, not just the front end web servers • Micro optimizations and other implementation details –– Bzzzzt! Boring! (blah blah blah, we’ll get to the cool stuff in a moment!) Benchmarking techniques • Scalability isn't the same as processing time • Not “how fast” but “how many” • Test “force”, not speed. Think amps, not voltage • Test scalability, not just performance • Use a realistic load • Test with "slow clients" Vertical scaling • “Get a bigger server” • “Use faster CPUs” • Can only help so much (with bad scale/$ value) • A server twice as fast is more than twice as expensive • Super computers are horizontally scaled! Horizontal scaling • “Just add another box” (or another thousand or ...) • Good to great ... • Implementation, scale your system a few times • Architecture, scale dozens or hundreds of times • Get the big picture right first, do micro optimizations later Scalable Application Servers Don’t paint yourself into a corner from the start Run Many of Them • For your application..
    [Show full text]
  • What Powers Instagram: Hundreds of Instances, Dozens of Technologies Instagram Engineering
    What Powers Instagram: Hundreds of Instances, Dozens of Technologies ­ Instagram Engineering instagram­engineering.tumblr.com One of the questions we always get asked at meet­ups and conversations with other engineers is, “what’s your stack?” We thought it would be fun to give a sense of all the systems that power Instagram, at a high­level; you can look forward to more in­depth descriptions of some of these systems in the future. This is how our system has evolved in the just­over­1­year that we’ve been live, and while there are parts we’re always re­working, this is a glimpse of how a startup with a small engineering team can scale to our 14 million+ users in a little over a year. Our core principles when choosing a system are: Keep it very simple Don’t re­invent the wheel Go with proven and solid technologies when you can We’ll go from top to bottom: OS / Hosting We run Ubuntu Linux 11.04 (“Natty Narwhal”) on Amazon EC2. We’ve found previous versions of Ubuntu had all sorts of unpredictable freezing episodes on EC2 under high traffic, but Natty has been solid. We’ve only got 3 engineers, and our needs are still evolving, so self­hosting isn’t an option we’ve explored too deeply yet, though is something we may revisit in the future given the unparalleled growth in usage. Load Balancing Every request to Instagram servers goes through load balancing machines; we used to run 2 nginx machines and DNS Round­Robin between them.
    [Show full text]
  • Examples of Open Source Software on Openvms
    9/2/2014 Examples of Open Source software on OpenVMS What, where, why, how and when Brett Cameron September 2014 Abstract The speaker has done a considerable amount of work over the past decade porting various pieces of Open Source software to the OpenVMS platform and working with OpenVMS customers to craft solutions using this Open Source software. During this talk the speaker will describe some of software that has been ported and where and how this software might be useful, with particular emphasis on Open Source web services and message queuing technologies. Work in progress and future plans will also be discussed. Additionally, the talk will examine the general approach taken by the speaker to the porting process, along with some of the pitfalls that are commonly encountered and how these may be resolved. In addition to the porting of software to OpenVMS, work has been done to implement interfaces for some of the ported Open Source packages to simplify their use from OpenVMS high-level languages such as COBOL, Pascal, and FORTRAN. Examples of such interfaces will be presented and the approaches taken to implementing them will be briefly discussed. 2 9/2/2014 About me Brett Cameron currently works as a senior architect with HP's corporate Cloud Services group, focusing on the design and implementation of message queuing and related integration services for customers and for internal use. Brett lives in Christchurch, New Zealand, and has worked in the software industry for some 22 years. In that time he has gained experience in a wide range of technologies, many of which have long since been retired to the software scrapheap of dubious ideas.
    [Show full text]
  • Web Performance and Scalability with Mysql Ask Bjørn Hansen Develooper LLC
    Web Performance and Scalability with MySQL Ask Bjørn Hansen Develooper LLC 1 Real World Web Scalability MySQL Edition Ask Bjørn Hansen Develooper LLC 2 Hello. • I’m Ask Bjørn Hansen • Tutorial in a box 44 minutes! • 53* brilliant° tips to make your website keep working past X requests/transactions per T time • Requiring minimal extra work! (or money) • Concepts applicable to ~all languages and platforms! * Estimate, your mileage may vary ° Well, a lot of them are pretty good 3 Construction Ahead! • Conflicting advice ahead • Not everything is applicable to every situation • Ways to “think scalable” rather than end-all-be-all solutions 4 Questions ... • How many saw my talk last year? • ... Brian Akers replication talk earlier today? • ... Second Life talk a few hours ago? • How many are using Perl? PHP? Python? Java? Ruby? • ... Oracle? PostgreSQL? 5 • Lesson number 1 • Think Horizontal! • Everything in your architecture, not just the front end web servers • Micro optimizations and other implementation details –– Bzzzzt! Boring! 6 Benchmarking techniques • Scalability isn't the same as processing time • Not “how fast” but “how many” • Test “force”, not speed. Think amps, not voltage • Test scalability, not just performance • Use a realistic load • Test with “slow clients” 7 Vertical scaling • “Get a bigger server” • “Use faster CPUs” • Can only help so much (with bad scale/$ value) • A server twice as fast is more than twice as expensive • Super computers are horizontally scaled! 8 Horizontal scaling • “Just add another box” (or another thousand or ...) • Good to great ... • Implementation, scale your system a few times • Architecture, scale dozens or hundreds of times • Get the big picture right first, do micro optimizations later 9 Scalable Application Servers Don’t paint yourself into a corner from the start 10 Run Many of Them • For your application..
    [Show full text]
  • Design, Implementation and Practical Evaluation of an Iot Home Automation System for Fog Computing Applications Based on MQTT and Zigbee-Wifi Sensor Nodes
    sensors Article Design, Implementation and Practical Evaluation of an IoT Home Automation System for Fog Computing Applications Based on MQTT and ZigBee-WiFi Sensor Nodes Iván Froiz-Míguez, Tiago M. Fernández-Caramés * , Paula Fraga-Lamas * and Luis Castedo Department of Computer Engineering, Faculty of Computer Science, Universidade da Coruña, 15071 A Coruña, Spain; [email protected] (I.F.-M.); [email protected] (L.C.) * Correspondence: [email protected] (T.M.F.-C.); [email protected] (P.F.-L.); Tel.: +34-981-167-000 (ext. 6088) (T.M.F.-C.) Received: 6 July 2018; Accepted: 3 August 2018; Published: 13 August 2018 Abstract: In recent years, the improvement of wireless protocols, the development of cloud services and the lower cost of hardware have started a new era for smart homes. One such enabling technologies is fog computing, which extends cloud computing to the edge of a network allowing for developing novel Internet of Things (IoT) applications and services. Under the IoT fog computing paradigm, IoT gateways are usually utilized to exchange messages with IoT nodes and a cloud. WiFi and ZigBee stand out as preferred communication technologies for smart homes. WiFi has become very popular, but it has a limited application due to its high energy consumption and the lack of standard mesh networking capabilities for low-power devices. For such reasons, ZigBee was selected by many manufacturers for developing wireless home automation devices. As a consequence, these technologies may coexist in the 2.4 GHz band, which leads to collisions, lower speed rates and increased communications latencies.
    [Show full text]