Testing Code with Time for Net Game Review: Python’s pytest Giants to Pay Up Lamplight City Since 1994: The original magazine of the Linux community MONITORING Why Your Server Monitoring Sucks • The Evils of CloudWatch How-To: Resource-Friendly Monitoring Solution ISSUE 292 | NOVEMBER 2018 www.linuxjournal.com NOVEMBER 2018 CONTENTS ISSUE 292 84 DEEP DIVE: Monitoring 85 Why Your Server Monitoring (Still) Sucks by Mike Julian Five observations about why your server monitoring still stinks by a monitoring specialist-turned-consultant. 96 CloudWatch Is of the Devil, but I Must Use It by Corey Quinn Let’s talk about Amazon CloudWatch. 104 Bare-Bones Monitoring with Monit and RRDtool by Andy Carlson How to provide robust monitoring to low-end systems. 114 How-To: Implementing a Real-Time Syslog Shipper for Your Terminal by Fabien Wernli Ever wondered how to tail -F /var/log/messages from multiple servers at once? Read on. 130 Taking System Monitoring to the Next Level: an Interview with Scalyr CEO Steve Newman by Petros Koutoupis As computing ecosystems become more complex, monitoring and analyzing those often disconnected moving parts becomes increasingly challenging. 2 | November 2018 | http://www.linuxjournal.com CONTENTS 6 The Monitoring Issue by Bryan Lunduke 10 From the Editor—Doc Searls An Immodest Proposal for the Music Industry 19 Letters UPFRONT 27 What’s Your System’s Uptime by Ricardo Fraile 33 Patreon and Linux Journal 34 Getting Started with Scilab by Joey Bernard 43 FOSS Project Spotlight: BlueK8s by Tom Phelan 48 Lessons in Vendor Lock-in: Shaving by Kyle Rankin 51 Reality 2.0: a Linux Journal Podcast 52 News Briefs COLUMNS 55 Kyle Rankin’s Hack and / Schedule One-Time Commands with the UNIX at Tool 59 Reuven M. Lerner’s At the Forge Testing Your Code with Python’s pytest 68 Dave Taylor’s Work the Shell Roman Numerals and Bash 74 Zack Brown’s diff -u What’s New in Kernel Development 177 Glyn Moody’s Open Sauce Time for Net Giants to Pay Fairly for the Open Source on Which They Depend LINUX JOURNAL (ISSN 1075-3583) is published monthly by Linux Journal, LLC. Subscription-related correspondence may be sent to 9597 Jones Rd, #331, Houston, TX 77065 USA. Subscription rate is $34.50/year. Subscriptions start with the next issue. 3 | November 2018 | http://www.linuxjournal.com CONTENTS ARTICLES 138 Review: the Dell XPS 13 Developer Edition Laptop by Petros Koutoupis A look at Dell’s thin and sleek XPS 13 Developer Edition laptop that now ships with Ubuntu 18.04 LTS pre-installed. 151 Chrome OS Stable Channel Gets Linux Apps by Philip Raymond How to get started with Linux Apps for Chromebooks. 158 About ncurses Colors by Jim Hall Why does ncurses support only eight colors? 170 Game Review: Lamplight City by Patrick Whelan A well lit look into Grundislav Game’s latest release. AT YOUR SERVICE SUBSCRIPTIONS: Linux Journal is available as a digital SPONSORSHIP: We take digital privacy and digital magazine, in PDF, EPUB and MOBI formats. Renewing responsibility seriously. We've wiped off all old advertising your subscription, changing your email address for issue from Linux Journal and are starting with a clean slate. Ads delivery, paying your invoice, viewing your account details we feature will no longer be of the spying kind you find or other subscription inquiries can be done instantly on most sites, generally called "adtech". The one form of online: http://www.linuxjournal.com/subs. Email us at advertising we have brought back is sponsorship. That's where [email protected] or reach us via postal mail at Linux Journal, advertisers support Linux Journal because they like what we 9597 Jones Rd #331, Houston, TX 77065 USA. Please remember to do and want to reach our readers in general. At their best, include your complete name and address when contacting us. ads in a publication and on a site like Linux Journal provide useful information as well as financial support. ACCESSING THE DIGITAL ARCHIVE: Your monthly download There is symbiosis there. For further information, email: notifications will have links to the different formats and to the [email protected] or call +1-281-944-5188. digital archive. To access the digital archive at any time, log in at http://www.linuxjournal.com/digital. WRITING FOR US: We always are looking for contributed articles, tutorials and real-world stories for the magazine. An LETTERS TO THE EDITOR: We welcome your letters author’s guide, a list of topics and due dates can be found and encourage you to submit them at online: http://www.linuxjournal.com/author. http://www.linuxjournal.com/contact or mail them to Linux Journal, 9597 Jones Rd #331, Houston, TX 77065 USA. NEWSLETTERS: Receive late-breaking news, technical tips Letters may be edited for space and clarity. and tricks, an inside look at upcoming issues and links to in-depth stories featured on http://www.linuxjournal.com. Subscribe for free today: http://www.linuxjournal.com/ enewsletters. 4 | November 2018 | http://www.linuxjournal.com EDITOR IN CHIEF: Doc Searls, [email protected] EXECUTIVE EDITOR: Jill Franklin, [email protected] DEPUTY EDITOR: Bryan Lunduke, [email protected] TECH EDITOR: Kyle Rankin, [email protected] ASSOCIATE EDITOR: Shawn Powers, [email protected] EDITOR AT LARGE: Petros Koutoupis, [email protected] CONTRIBUTING EDITOR: Zack Brown, [email protected] SENIOR COLUMNIST: Reuven Lerner, [email protected] SENIOR COLUMNIST: Dave Taylor, [email protected] PUBLISHER: Carlie Fairchild, [email protected] Join a ASSOCIATE PUBLISHER: Mark Irgang, [email protected] DIRECTOR OF DIGITAL EXPERIENCE: Katherine Druckman, [email protected] community GRAPHIC DESIGNER: Garrick Antikajian, [email protected] ACCOUNTANT: Candy Beauchamp, [email protected] with a deep COMMUNITY ADVISORY BOARD appreciation John Abreau, Boston Linux & UNIX Group; John Alexander, Shropshire Linux User Group; Robert Belnap, Classic Hackers UGA Users Group; Aaron Chantrill, Bellingham Linux Users Group; Lawrence D’Oliveiro, Waikato Linux Users Group; Chris Ebenezer, Silicon Corridor Linux User Group; David Egts, Akron Linux Users Group; Michael Fox, Peterborough Linux User Group; for open-source Braddock Gaskill, San Gabriel Valley Linux Users’ Group; Roy Lindauer, Reno Linux Users Group; Scott Murphy, Ottawa Canada Linux Users Group; Andrew Pam, Linux Users of Victoria; Bob Proulx, Northern Colorado Linux User's Group; Ian Sacklow, Capital District Linux Users Group; Ron Singh, Kitchener-Waterloo Linux User Group; Jeff Smith, Kitchener-Waterloo Linux User Group; philosophies, Matt Smith, North Bay Linux Users’ Group; James Snyder, Kent Linux User Group; Paul Tansom, Portsmouth and South East Hampshire Linux User Group; Gary Turner, Dayton Linux Users Group; Sam Williams, Rock River Linux Users Group; digital Stephen Worley, Linux Users’ Group at North Carolina State University; Lukas Yoder, Linux Users Group at Georgia Tech freedoms Linux Journal is published by, and is a registered trade name of, Linux Journal, LLC. 4643 S. Ulster St. Ste 1120 Denver, CO 80237 and privacy. SUBSCRIPTIONS E-MAIL: [email protected] URL: www.linuxjournal.com/subscribe Mail: 9597 Jones Rd, #331, Houston, TX 77065 Subscribe to SPONSORSHIPS E-MAIL: [email protected] Linux Journal Contact: Publisher Carlie Fairchild Phone: +1-281-944-5188 Digital Edition LINUX is a registered trademark of Linus Torvalds. for only $2.88 an issue. Private Internet Access is a proud sponsor of Linux Journal. SUBSCRIBE TODAY! 5 | November 2018 | http://www.linuxjournal.com THE MONITORING ISSUE Bryan Lunduke is a former Software Tester, former Programmer, former VP of In 1935, Austrian physicist, Erwin Schrödinger, still flying Technology, former Linux high after his Nobel Prize win from two years earlier, created Marketing Guy (tm), former openSUSE Board Member...and a simple thought experiment. current Deputy Editor of Linux Journal as well as host of the It ran something like this: (aptly named) Lunduke Show. If you have a file server, you cannot know if that server is up or down...until you check on it. Thus, until you use it, a file server is—in a sense—both up and down. At the same time. This little brain teaser became known as Schrödinger’s File Server, and it’s regarded as the first known critical research on the intersection of Systems Administration and Quantum Superposition. (Though, why Erwin chose, specifically, to use a “file server” as an example remains a bit of a mystery—as the experiment works equally well with any type of server. It’s like, we get it, Erwin. You have a nice NAS. Get over it.) ... Okay, perhaps it didn’t go exactly like that. But I’m confident it would have...you know...had good old Erwin had a nice Network Attached Storage server instead of a cat. 6 | November 2018 | http://www.linuxjournal.com THE MONITORING ISSUE Regardless, the lessons from that experiment certainly hold true for servers. If you haven’t checked on your server recently, how can you be truly sure it’s running properly? Heck, it might not even be running at all! Monitoring a server—to be notified when problems occur or, even better, when problems look like they are about to occur—seems, at first blush, to be a simple task. Write a script to ping a server, then email me when the ping times out. Run that script every few minutes and, shazam, we’ve got a server monitoring solution! Easy-peasy, time for lunch! Whoah, there! Not so fast! That server monitoring solution right there? It stinks. It’s fragile. It gives you very little information (other than the results of a ping). Even for administering your own home server, that’s barely enough information and monitoring to keep things running smoothly. Even if you have a more robust solution in place, odds are there are significant shortcomings and problems with it.