INSIDE! Double-Sided DVD Double-Sided

Home , EGroupware, Libreboot, Mycroft (software), Packet Sender, Proprietary firmware, ROSA Linux

0 74820 58049 3 US$ 15.99 CAN$ 17.99 Issue 210 May 2018 and the the and

binary data Bash Tricks Manipulating Manipulating

liberation of firmware of liberation LIBREBOOT

Tutorial own Build your app GPS

MenuLibr e Flameshot Chomper • • • FOSSPicks

Keep your team in sync in team your Keep COLLABORATIVE SUITES: OFFICE

4 MAY 2018 MAY ool for teaching eate Android apps witheate Android

Raspberry Pi Slideshow Beam out your message

INSIDE! Double-Sided DVD Double-Sided

minimal coding guitarists classical notation Jasonette: Cr

Nootka: Cool learning t Easy cross-platform file exchange Easy cross-platform NitroShare:

WWW.LINUXPROMAGAZINE.COM meets the desktop Web programming programming Web Electron •

•

Libreboot and the liberation of firmware liberation and the Libreboot better than WordPress?

Is this free blogging tool Is this free blogging tool Ghost CMS

platform for non-programmers

Powerful data analysis data analysis Powerful KNIME

FREE BOOT! FREE

DVD FREE

LINUX PRO MAGAZINE ISSUE 210 MAY 2018 Free Boot! Collaboration Suites Electron BorgBackup Ghost KNIME Countdown Jasonette NitroShare Nootka

EDITORIAL Welcome HATS OFF TO LARRY

Dear Reader,

We all loved it when Microsoft finally embraced Linux after was already working on its cloud portfolio, and they have a history of attacking it. Some of those outrageous sound labored systematically since then to transform their busi- bites like “Linux is a cancer…” got played back in trium- ness around cloud services (just like everyone else is phant retrospective as we all marked the passing of an era doing). If you were an observer from another planet, and remarked on the how far we’d come. watching the behavior of Oracle the way Jane Goodall Should we be doing the same thing now with Oracle and the watched the behavior of wild chimpanzees, you would cloud? Oracle founder Larry Ellison made headlines back in never have guessed that these people who were artfully 2008 with some very theatrical denunciations of the cloud in- executing this methodical entry into the enterprise cloud dustry. Ellison called the cloud “nonsense,” and once said of market worked for a company that regarded cloud com- the cloud phenomenon, “Maybe I’m an idiot, but I have no puting as “complete gibberish.” idea what anyone is talking about. What is it? It’s complete What is my point then? Oh, just a lament about this busi- gibberish. It’s insane. When is this idiocy going to stop?” [1] ness I’m in, where you can interview these guys and they Part of his argument was that much of what was getting emphatically tell you one thing – with all the sincerity and called “cloud computing” was just a newfangled name for bravado of a tenor in the opera – but all the while they are what used to be called networking or web services – a fair out there doing something different. Every tweet, every point at the time. But his vociferous refusal to acknowledge speech, every photo moment, is all about strategy, and that anything was actually happening that resembled a new strategy never stays long in the house of truth. market or a paradigm shift seemed wildly controversial at a But I already knew that – and I guess you probably did to. time when other companies were all rushing to reinvent Hats off to you, Larry. Sounds like things are on course, and reorient. Many experts said that Ellison was just de- and your ship is sailing for another sunrise. So I guess nouncing the cloud because his company was so far behind maybe you DID understand the cloud after all. that it had no reasonable way to compete with cloud vendors other than to shame or frighten potential users away from using cloud services. (Actually, the same could be said about Microsoft’s attitude toward Linux.) Fast forward to March of this year, and we discover that Joe Casad, Oracle’s Q3 2018 report shows cloud revenues are up 32% Editor in Chief to US$1.6 billion. The press release includes a proud quote from Ellison, “The Oracle autonomous database is now fully available in the Oracle Cloud. And there are more autonomous cloud services to come. During this calendar year, we expect to deliver Autonomous Analytics, Autono- mous Mobility, Autonomous Application Development, and Autonomous Integration services. Oracle’s new suite of Autonomous PaaS services delivers an unprecedented level of automation and cost savings to our customers.” My they have come a long way! Or have they? At the time of Ellison’s anti-cloud tirade back in 2008, Oracle

Info [1] “Larry Ellison: Someone Explain to Me This Cloud Computing Thing”: http://www.businessinsider.com/2008/9/larry‑ellison‑ someone‑explain‑to‑me‑this‑cloud‑computing‑thing‑my‑ company‑is‑committing‑to‑orcl‑

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 3 LINUX MAGAZINE MAY 2018

WHAT'S INSIDE NEWS COVER STORIES

This month we tune into the 08 News 16 Free Firmware with Libreboot Libreboot project – an effort to • OpenStack Queens Released Libreboot is a project dedicated to making replace the proprietary firmware in • Kali Linux Comes to Windows firmware free as in freedom. laptop computers with an open • Ubuntu to Start Collecting Data with source equivalent. Other highlights Ubuntu 18.04 • CNCF Illuminates Serverless Vision this month include: • Red Hat Decision Manager 7 Released • Electron – discover how to • T opological Superconductor Could channel the cross-platform Lead to Quantum Computing with benefits of the web Weird Fermions programming paradigm when writing desktop applications. 12 Kernel News • KNIME – a graphical tool that • Improving a Hashing Function lets you perform complex data • Constant Values Adjustable at Boot analysis with custom Time • Dealing with Intel Design Flaw workflows. Also in these pages: check out MakerSpace for a tutorial on building a countdown timer with REVIEWS the Raspberry Pi, and jump to LinuxVoice to code your first Android app using Jasonette. 22 Collaborative Office Suites Several collaborative office suite providers are now competing for customers who are looking for the seamless exchange of SERVICE office documents within an enterprise.

3 Comment 6 DVD 96 Featured Events 97 Call for Papers 98 Preview

4 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM LINUX MAGAZINE MAY 2018

IN-DEPTH MAKERSPACE

30 Electron 56 Countdown The Electron project brings the benefits Counter of web programming to the realm of Build a countdown desktop applications. counter with a Raspberry Pi and 34 Command Line – BorgBackup the MAX7221 In Star Trek, the Borg adds individuals display driver. to its collective consciousness, an apt metaphor for any backup application 60 Open Hardware – that stores individual files in an archive. Keyboardio The Keyboardio Model 01 is a highly- 38 Programming Snapshot – customizable open hardware Stock Quote API keyboard. TWO TERRIFIC Financial wizard Mike Schilli is annoyed DISTROS that some web services have stopped 64 Raspberry Slideshow DOUBLE-SIDED serving up real-time stock market data. In Video sequences or slide shows DVD! order to keep an eye on his investment promote sales. With a Raspberry Pi dollars, Mike taps into a little known and the Raspberry Slideshow interface for stock prices. distribution, you can capture the SEE PAGE 6 FOR DETAILS attention you need. 42 Ghost CMS Blogging usually implies WordPress, but Ghost presents a serious challenge.

69 Welcome 80 Nootka Going cross-platform. With Nootka and your guitar, you can learn to recognize musical 70 Jasonette notes in classical notation. Build Android apps with a 45 Charly’s Column – sshuttle minimum of coding. When he doesn’t want to deal with OpenVPN version conflicts or congestion control problems during TCP tunneling, Charly catches a ride on sshuttle.

46 KNIME They say data is “the new oil,” but all that data you collect is only valuable if it leads to new insights. An open 84 FOSSPicks source analysis tool called KNIME lets Graham checks out VLC 3.0, you analyze data through graphical MenuLibre 2.1.5, Texttop, workflows – without the need for 75 Doghouse – Subutai Flameshot, Chomper, Godot 3.0, programming or complex spreadsheet Be a catalyst and help others break and much more! manipulation. the bonds of proprietary software. 90 Tutorial – GPSD 52 Binary Data in Bash 76 NitroShare Even though your computer does Bash is known for admin utilities and text This handy app simplifies the not come with its own GPS, you can manipulation tools, but the venerable exchange of files and directories hack one onto it using a mobile command shell included with most Linux between computers with Linux, phone – if you are ready for some systems also has some powerful Mac OS, Windows, and Android. fiddling. commands for manipulating binary data.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 5 DVD This Month’s DVD On the DVD

Kali Linux Kali puts the emphasis on penetration testing and security auditing. According to the developers, Kali comes with more than 300 security tools, including the Metasploit pen testing framework, as well as an assortment of utilities for port scanning, traffic analysis, password cracking, and more. The latest release comes with the Linux 4.14 kernel and includes increased memory limits and support for AMD secure memory encryption. KaOS This lean Linux is a showcase for the KDE desktop and the Qt development framework. The KaOS developers‘ goal is to create the most usable, stable, and sensible KDE environment with a limited size to keep the emphasis on quality rather than quantity. Development efforts are geared toward packaging and perfecting the desktop environment, rather than adding new tools and applications. TWO TERRIFIC DISTROS DOUBLE-SIDED DVD!

Additional Resources [1] Kali Linux: https://www.kali.org/ [2] Kali documentation: https://www.kali.org/ kali‑linux‑documentation/ [3] KaOS: https://kaosx.us/ [4] KaOS documentation: https://kaosx.us/docs/

Defective discs will be replaced. Please send an email to [email protected].

6 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM

NEWS Updates on technologies, trends, and tools

THIS MONTH’S NEWS

New OpenStack Cloud 08 Features • OpenStack Queens Released OpenStack Queens Released • Kali Linux Comes to The OpenStack Foundation has announced the 17th release of OpenStack, dubbed Windows Queens. OpenStack now powers some of the biggest public clouds in the world. Ac- cording to a 451 Research survey, 60 percent of enterprise workloads will run in the Linux Desktop Data 09 Collection cloud. The survey also reports that enterprise adoption of OpenStack is expanding • Ubuntu to Start Collecting Data with Ubuntu 18.04 in parallel, with enterprises in nearly all ver- ticals and regions now running mission- • CNCF Illuminates Serverless Vision critical workloads on OpenStack software. Queens is well equipped with many • More Online new features to handle such workloads. It Automating Business comes with Cinder multi-attach that en- 10 Decisions ables operators to attach the same Cinder • Red Hat Decision Manager volume to multiple virtual machines (VMs). 7 Released If one node goes down, the other takes

• Topological Superconductor over and has access to the volume. Ac- Image © satori, 123RF.com Could Lead to Quantum cording to the OpenStack Foundation, this Computing with Weird was one of the most-requested features in cloud environments. Fermions The Queens release also includes significant enhancements to support emerging use cases such as machine learning, network functions virtualization (NFV), container integration and edge computing. “Our developer community keeps building new tools that are helpful to the infrastructure operator. Eight years in, features that enhance manageability, resiliency, scalability and user experience are still paramount and still have sizzle factor,” said Jonathan Bryce, Executive Director of the OpenStack Foundation.

Kali Linux Comes to Windows Kali Linux, a penetration testing distro that you may have seen in Mr. Robot, is now available in the Microsoft Store. The Kali Linux team has been working with the Micro- soft Windows Subsytem for Linux (WSL) team to bring the distro to the platform that still dominates the PC landscape. By doing so, Kali has brought some of the best penetration testing tools to the biggest PC user base. “This is especially exciting news for penetration testers and security professionals who have limited toolsets due to enterprise compliance standards,” wrote Mati Aharoni, lead Kali developer, in a blog post. At the end of last year, Microsoft took WSL out of beta, making it available for every Windows 10 user for the latest Fall Creators Update. Users can simply go to the Micro- soft Store and install their preferred Linux distro just like any other app. Initially only Ubuntu, openSUSE Leap, and SUSE Linux Enterprise were available, but the WSL team is working with different distros to bring them to the store. Officially only command-line Linux utilities are available through WSL, as the target audience is developers and sys admins who want to write or deploy applications for

8 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM NEWS Linux News

Linux machines running on the cloud. That didn’t stop the Linux community from running GUI applications through WSL. In fact, even the Kali Linux team managed to run a MORE ONLINE full blown XFCE desktop. Running Linux distros under WSL has its own shortcomings. Your system is only as secure as Windows 10. However, the Kali Linux team sees some great possibilities Linux Magazine too, “While running Kali on Windows has a few drawbacks to running it natively (such www.linux-magazine.com as the lack of raw socket support), it does bring in some very interesting possibilities, such as extending your security toolkit to include a whole bunch of command-line tools ADMIN HPC that are present in Kali.” http://hpc.admin-magazine.com/ Being able to run Linux inside of Windows 10 gives developers native access to What to Do with System Data: Think Like a tools from both worlds. Vegan • Jeff Layton What do you do with all of the HPC data you harvested as a lumberjack? You think like a Ubuntu to Start Collecting Some Data Vegan. with Ubuntu 18.04 ADMIN Online Canonical, the parent company of Ubuntu, is planning to collect diagnostic data from its http://www.admin-magazine.com/ desktop operating system. In a message posted to the Ubuntu Developer mailing list, Identity Management from the Cloud Will Cooke, director of Ubuntu Desktop, explained the reason behind this move, “We Martin Kuppinger want to be able to focus our engineering efforts on the things that matter most to our Offers for identity management as a service users, and in order to do that, we need to get some more data about what sort of set- (IDaaS) are entering the market and promis- ups our users have and which software they are running on it.” ing simplicity. However, many lack functional- Ubuntu installer will have an opt-out checkbox for sending diagnostics information to ity, adaptability, and in-depth integration with help improve Ubuntu. existing systems. We look at how IT manag- Canonical has chosen to keep this feature opt-out, instead of opt-in, meaning un- ers should consider IDaaS in their strategy. less you uncheck the box, Canonical will collect diagnostic data. The Ubuntu privacy LibreOffice in the Workplace policy will be updated to reflect this change. In order to give users more control over Thomas Krumbein the features, there will be an opt-out option in the Gnome System Settings. How can LibreOffice be used in a corporate What kind of data will Canonical be collecting? Nothing invasive. They want to know environment and what difficulties have to be which flavor and version of Ubuntu you are running and whether you have network con- considered? In particular, you cannot under- nectivity (one may wonder how will they get the data if there is no network connectivity?). estimate acceptance by your employees. They will also collect data about the processor, Regular expressions and metacharacters in GPU, screen resolution, memory, storage, and PowerShell • Thomas Bär and Frank-Michael OEM manufacturer. Other data includes location Schlede (not IP address), installation duration, status of Almost all programming and scripting lan- auto login, and disk layout. guages allow the use of regular expressions, Cooke said that all of this data will be made but many professionals still believe regex is public. It could be a great way for Canonical to a relic from ancient times. With specific ex- start collecting stats about the Linux desktop. amples, we show how useful and meaningful There are no credible stats about who is using regex can be in PowerShell. the platform. Canonical’s move can be a step in

Image © Tatiana Venkova, 123RF.com Venkova, Image © Tatiana that direction. ADMIN DevOps Focus http://www.admin-magazine.com/DevOps CNCF Illuminates Serverless Vision Automated Jenkins CI • Nathan Bliss A Jenkins build pipeline frees develop- Serverless or Function as a Service is one of the hottest topics these days. But what ers from repetitive manual build tasks by is “serverless computing” and who is it for? Can it replace the existing models? facilitating automation and provides broad These are some of the many questions the Cloud Native Computing Foundation traceability with continuous logging and (CNFC) is attempting to answer in a paper drafted by feedback. the CNCF Serverless Working Group. “Serverless is a natural evolution of cloud-native computing. The CNCF is advancing serverless adoption through collaboration and community-driven ini- tiatives that will enable interoperability,” said Chris Aniszczyk, COO, CNCF. According to the whitepaper, “Serverless computing refers to the concept of building and running applications that do not require server management. It describes a finer-grained deployment Image © Galina Peshkova, 123RF.com

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 9 NEWS Linux News

model where applications, bundled as one or more functions, are uploaded to a platform and then executed, scaled, and billed in response to the exact demand needed at the moment.” Being a new technology, there is a lot of work to be done for the healthy growth of serverless ecosystem. The CNCF has recognized its role in the space and is attempting to address those needs. The CNCF will start a drive to encourage more serverless technology vendors and open source developers to join the CNCF. It will also look at ways to foster an open ecosystem by establishing interoperable APIs, ensuring interoperable implementations with vendor commitments and open source tools. CNCF is a Linux Foundation Collaborative project that was created to foster innovation in the cloud native space. Kubernetes was its anchor project. You can read the whitepaper on GitHub.

Red Hat Decision Manager 7 Released Red Hat has announced the release of Red Hat Decision Manager 7, an open source product that makes it easier for businesses to automate business decisions. If you want to book a flight, several variables might affect the cost. An airline or a travel agent platform needs rules that reflect those variables. Traditionally, those rules were created by business operators and implemented by software developers. Red Hat Decision Manager 7 eases that process by automating it. According to industry analyst firm IDC, non-traditional developers are expected to build 20 percent of business applications and 30 percent of new application features by 2021. Business processes play an important role in helping organizations improve efficiency and reduce workloads, and at the heart of every business process are business rules. Red Hat Decision Manager 7 is a low-code development tool that enables businesses to take a more active role in application development by creating an environment for collaboration between business and IT professionals. Red Hat Decision Manager 7 is built for both traditional and cloud-native applications.

Topological Superconductor Could Lead to Quantum Computing with Weird Fermions

With their natural insensitivity to decoherence, Majorana particles, which are also called Majorana fermions, have long been considered a promising ingredient for quantum computing, but the difficulty in obtaining and containing the illusive particles has limited their usefulness in quantum research. A team at Chalmers University of Technology in Sweden has announced that they have developed a means for corral- ling Majorana particles using topological superconductivity. According to the announcement at Physics.org, “Majorana fermions are highly original particles, quite unlike those that make up the materials around us. In highly simplified terms, they can be seen as half-electron. In a quantum computer, the idea is to encode information in a pair of Majorana fermions separated in the material, which should, in princi- ple, make the calculations immune to decoherence.” According to team leader and Chalmers professor Floriana Lombardi, “For practical reasons, the material is mainly of interest to those attempting to build a topological quantum computer. We want to explore the new physics hidden in topological superconductors—this is a new chapter in physics.”

10 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM

NEWS Kernel News Zack’s Kernel News

Improving a Hashing cluded, “the patch as-is doesn’t seem to Function buy anything, and only adds cost.” He Amir Goldstein posted a small fix to a also remarked, “we did have numbers at kernel function that would produce a one point. That’s what really matters: unique hash value for an input string. He how good the actual hashing ends up noticed that for 64-bit values, some bits being. So for me to take the patch, I were lost before calculating the hash – re- would need to see that it actually im- sulting in a slightly less good hash. For proves the hash bucket spreading 32-bit values, he said, no bits were lost, enough to be worth the cost.” and the code was left unchanged, but Amir replied that it was not his patch since the relevant stringhash.h file had that added the __hash_32() call at all – it no official maintainer, and Linus Torvalds was already there. had done some work on it in the past, To which Linus replied, “Oh, duh. Chronicler Zack Brown Amir asked Linus what he thought of the My bad. It was indeed there already. reports on the latest news, fix and for advice on how to test it. As Let me go back and look at the history Amir put it, “I wouldn’t even know of this thing.” views, dilemmas, and where to begin testing its affects, or how A few days later, he wrote back more developments within the to prove if there really is a problem.” fully: Linux kernel community. Linus dug back into his memory, saying he believed the lost bits were inten- After having looked more at it, I take By Zack Brown tional, though he acknowledged, “it’s a back all my complaints about the patch; long time ago, and we’ve changed some you were right, and I was misreading of the hashing since.” things or just being stupid. But Linus also said, “you’re wrong I also don’t worry too much about the that it’s a no-op on 32-bit. It’s a very ex- possible performance impact of this on pensive and pointless multiplication 64-bit, since most architectures that actually there too, even if the shift ends up being care about performance end up not using a no-op. The name hashing is pretty per- this very much (the dcache code is the most formance-sensitive.” performance-critical, but the word-at-a-time Amir corrected Linus, saying that he case uses its own hashing anyway). hadn’t said the patch would result in a So this ends up being mostly used for no-op (code that literally did nothing, filesystems that do their own degraded not even use time); he’d said that the hashing (usually because they want a patch left the code for the 32-bit case un- case-insensitive comparison function). changed, except for the bit shift Linus A _tiny_ worry remains, in that not ev- had mentioned. erybody uses DCACHE_WORD_ACCESS, and Linus pointed out that while the shift then this potentially makes things more might cost nothing, the __hash_32() used expensive on 64-bit architectures with nearby was very expensive. So, he con- slow or lacking multipliers even for the normal case. Author That said, realistically the only such The Linux kernel mailing list comprises the architecture I can think of is PA-RISC. No- core of Linux development activities. Traffic body really cares about performance on volumes are immense, often reaching that; it’s more of a “look ma, I’ve got 10,000 messages in a week, and keeping warts^W an odd machine” platform. up to date with the entire scope of So I think your patch is fine, and all development is a virtually impossible task my initial worries were just misplaced for one person. One of the few brave souls from not looking at this properly. to take on this task is Zack Brown. Sorry.

12 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM NEWS Kernel News

Linus also remarked, “I *would* love find a way to shave as many Assembly Peter replied, “I am currently working to get some kind of estimate on how this instructions off the code as possible. Re- on it much more comprehensive set of changes the hash distribution. Do you implementing the variable as a patch- patches for extremely this. I am already have anything like that? The way this able constant solved the problem. much further ahead and support most function is designed to be used, the However, Kirill acknowledged that it operations.” He added, “The patchset I upper bits should be used for the hash came with an unacceptable cost. He have is about 85% complete. It mostly bucket information, so doing some said, “This conversion makes GCC gen- needs cleanup, testing, and breaking bucket size distribution on (say) the erate worse code. Conversion __PHYSI- into reasonable chunks.” He also re- upper ~12 bits or something with some CAL_MASK to a patchable constant adds marked, “The main reason I haven’t sub- real string input would be really good.” about 5k in .text on defconfig and mitted it yet is that I got a bit overly am- Amir replied, “I’ve added bucket makes it slightly slower at runtime bitious and wanted to implement a counters and tracing number of en- (~0.2% on my box).” whole bunch of more complex subcases, tries/number of non-empty buckets So, the very effort to speed things up such as 64-bit shifts on a 32-bit kernel. and largest bucket to dcache (patch at- in one way, ended up slowing down the The win in that case is actually quite tached). Increased d_hash_shift to take code in another. Kirill pointed out that huge, but it requires data-dependent only 12 bits for hash table offset. The he was submitting this patch, not to be code patching and not just immediate results vary a bit per run, but below included in the kernel, but in the hopes patching, which requires augmentation are some outputs from sample runs of that someone might have a better idea of the alternative framework.” find/of 55K entries right after boot. for how to get the benefits of patchable The discussion ended at that point. The bottom line is that, at least w.r.t constants, without this unwanted cost. Because open source is so decentral- this simplistic bucket analysis and this Linus Torvalds replied, saying: ized, overlapping projects like this do specific workload, the patch does not sometimes happen, often resulting in a seem to move the needle. The bucket I actually wanted something very close to lot of wasted work or ongoing contro- distribution of before and after patch this, but I think your approach is much too versy over which approach is better. But are similar and also similar to the simplistic. You force all constants into a in this case, given that Kirill had posted word-at-a-time results.” register, which means that the resulting his patch specifically because he had not He included some numbers, and the code is always going to be very far from found a way to solve the underlying thread ended. non-optimal. You also force a big movabsq problems, it doesn’t look like there will instruction, which really is huge, and al- be much controversy, at least over which Constant Values Adjustable most never needed. Together with the “use project to accept into the kernel. There at Boot Time a register,” it just makes for big code. still may be some debate over the value Some parts of the kernel run so hot that of adding so much complex infrastruc- even the overhead of assigning a value He added, “What I wanted was some- ture to handle self-modifying constants. to a variable can result in a significant thing that can take things like a shift by It may turn out to be so bug-prone and speed difference. Kirill A. Shutemov re- a variable that is set once and turn it into unmaintainable that it defeats the value cently posted a patch to introduce a shift by a boot-time constant. Which of the speed increase. “patchable constants.” These are con- means that you literally end up patching stant values that can be set at boot time, the 8-bit immediate in the shift instruc- Dealing with Intel either as part of the kernel configuration tion itself.” Design Flaw itself or by the user at the bootloader He posted some assembly code, Recently, a serious design flaw with Intel command line. Once set, they thereafter showing how several lines could logi- microprocessors has forced various oper- behave as true constant values in the cally be boiled down to just one, which ating systems, including Linux, to come running kernel, rather than variables. he said would result in “much smaller up with some workarounds. Recently, As Kirill explained it, “Patchable con- code, and register %rcx isn’t used at all. KarimAllah Ahmed posted a patch to stants implemented by replacing a con- And no D$ miss on loading that constant control Indirect Branch Speculation stant with call to inline function that re- (that is a constant depending on boot- (IBS), essentially trying to manage the turns the constant value using inline as- time setup only).” He went on to say, hardware deficiencies of Intel chips, to sembler. In inline assembler, we also “It’s rather more complex, but it actu- activate them when needed and safe, write down into separate section loca- ally gives a much bigger win. The code and to lock them down when their flaws tion of the instruction that loads the con- itself will be much better, and smaller. might result in security issues. stant. This way we can find the location The *infrastructure* for the code gets Linus replied, “All of this is pure gar- later and adjust the value.” pretty hairy, though.” bage. Is Intel really planning on making The reason Kirill wanted to implement He pointed to a discussion from 18 this shit architectural? Has anybody this general feature was to solve the spe- months earlier, in which H. Peter Anvin talked to them and told them they are cific case of the __PHYSICAL_MASK variable. had done some work on a similar idea. f*cking insane? Please, any Intel engi- To do memory encryption, this had to be Peter had called it a “disgusting pseudo- neers here – talk to your managers.” dynamically settable. But because mem- self-modifying code idea,” and it hadn’t David Woodhouse said, “If the alterna- ory encryption has a speed impact on been completed because of all the extra tive was a two-decade product recall and absolutely everything, Kirill wanted to complexity it introduced. giving everyone free CPUs, I’m not sure

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 13 NEWS Kernel News

it was entirely insane.” He agreed that trying to protect the kernel.” We already you look at this KarimAllah’s patch was a “nasty hack,” have retpoline there, with less overhead. series, it very but added, “hey – the world was on fire, So somebody isn’t telling the truth much does that and in the end we didn’t have to just here. Somebody is pushing complete gar- kernel entry/ turn the datacentres off and go back to bage for unclear reasons. Sorry for hav- exit stuff. It was goat farming, so it’s not all bad.” ing to point that out. patch [number Linus replied, “You seem to have If this was about flushing the BTB at 10 of 10 in the bought into the cool-aid. Please add a actual context switches between different series]. In fact, healthy dose of critical thinking. Because users, I’d believe you. But that’s not at all the patch I was this isn’t the kind of cool-aid that makes what the patches do. replying to was for a fun trip with pretty pictures. This is As it is, the patches are COMPLETE explicitly setting the kind that melts your brain.” AND UTTER GARBAGE. that garbage up. He went on to say that the true situation They do literally insane things. They And I really don’t was much worse than a “nasty hack.” do things that do not make sense. That want to see these First, Linus acknowledged that, “The makes all your arguments questionable garbage patches speculation control cpuid stuff shows and suspicious. The patches do things just mindlessly that Intel actually seems to plan on that are not sane. sent around.” doing the right thing for Meltdown (the WHAT THE F*CK IS GOING ON? David replied main question being _when_). Which is And that’s actually ignoring the much with a very interest- not a huge surprise, since it should be _worse_ issue, namely that the whole ing and useful sum- easy to fix, and it’s a really honking big hardware interface is literally mis-de- mary of the whole situ- hole to drive through. Not doing the signed by morons. ation. He said: right thing for Meltdown would be com- It’s mis-designed for two major reasons: pletely unacceptable.” • the “the interface implies Intel will I think we’ve covered the technical part But he went on to say, “the IBRS gar- never fix it” reason. of this now, not that you like it – not that bage implies that Intel is _not_ planning See the difference between IBRS_ALL any of us *like* it. But since the peanut on doing the right thing for the indirect and RDCL_NO. One implies Intel will fix gallery is paying lots of attention, it’s branch speculation. Honestly, that’s something. The other does not. Do you probably worth explaining it a little more completely unacceptable too.” really think that is acceptable? for their benefit. He continued: • the “there is no performance indicator” This is all about Spectre variant 2, [reason] where the CPU can be tricked into mis- The whole IBRS_ALL feature to me very The whole point of having cpuid and predicting the target of an indirect clearly says “Intel is not serious about flags from the microarchitecture is that branch. And I’m specifically looking at this; we’ll have a ugly hack that will be we can use those to make decisions. But what we can do on *current* hardware, so expensive that we don’t want to en- since we already know that the IBRS over- where we’re limited to the hacks they can able it by default, because that would head is huge on existing hardware, all manage to add in the microcode. look bad in benchmarks.” those hardware capability bits are just The new microcode from Intel and So instead they try to push the garbage complete and utter garbage. Nobody sane AMD adds three new features. down to us. And they are doing it entirely will use them, since the cost is too damn One new feature (IBPB) is a complete wrong, even from a technical standpoint. high. So you end up having to look at barrier for branch prediction. After frob- I’m sure there is some lawyer there who “which CPU stepping is this” anyway. bing this, no branch targets learned earlier says “we’ll have to go through motions to I think we need something better than are going to be used. It’s kind of expensive protect against a lawsuit.” But legal rea- this garbage. (order of magnitude ~4,000 cycles). sons do not make for good technology, or The second (STIBP) protects a hyper- good patches that I should apply. David was actually in full agreement. thread sibling from following branch pre- Instead of Intel making a fix “opt-in,” he dictions which were learned on another Earlier, David had remarked, “We do said, they should just fix it, since failing sibling. You *might* want this when need the IBPB feature to complete the to opt in would always be unacceptable. running unrelated processes in user- protection that retpoline gives us – it’s But he felt that for the kernel, a short- space, for example. Or different VM that or rebuild all of userspace with ret- term fix that was horrendously ugly guests running on HT siblings. poline.” To which Linus said: would still be OK, given that it was just a The third feature (IBRS) is more com- short-term fix. plicated. It’s designed to be set when you BULLSHIT. He also pointed out that Linus had enter a more privileged execution mode Have you _looked_ at the patches you been looking slightly in the wrong place (i.e., the kernel). It prevents branch tar- are talking about? You should have – sev- when complaining about David’s gets learned in a less-privileged execution eral of them bear your name. patches – David pointed out that Linus mode, BEFORE IT WAS MOST RECENTLY The patches do things like add the gar- had been looking at IBRS rather than SET, from taking effect. But it’s not just a bage MSR writes to the kernel entry/exit IBPB. To which Linus replied, “Ehh. Odd “set-and-forget” feature, it also has bar- points. That’s insane. That says “we’re Intel naming detail.” And he went on, “If rier-like semantics and needs to be set on

14 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM NEWS Kernel News

*each* entry into the kernel (from user- tually lying awake at night fretting PERMITTED to talk directly to you can space or a VM guest). It’s *also* expen- about it, but nothing concrete that really at least get some clear feedback … to in- sive. And a vile hack, but for a while it says it’s OK. dicate *to them* that you fully grasp the was the only option we had. If you view retpoline as a performance technical situation … whilst at the same Even with IBRS, the CPU cannot tell the optimization, which is how it first ar- time not being permitted access to the difference between different userspace rived, then it’s rather unconventional to fucking microcode.” processes, and between different VM say “well, it only opens a *little* bit of a Meanwhile, David was joined by Ingo guests. So in addition to IBRS to protect security hole, but it does go nice and fast, Molnár and many others working on the the kernel, we need the full IBPB barrier so let’s do it.” technical side of generating new patches on context switch and vmexit. And But fine, I’m content with ditching the to solve the various issues. maybe STIBP while they’re running. use of IBRS to protect the kernel, and I’m There was clearly a lot of bitterness Then along came Paul with the cun- not even surprised. There’s a *reason* among the kernel developers – for exam- ning plan of “oh, indirect branches can we put it last in the series, as both the ple, as expressed by Pavel Machek at one be exploited? Screw it, let’s not have any most contentious and most dispensable point in the discussion, where he said, of *those* then,” which is retpoline. And part. I’d be *happier* with a coherent “someone at Intel put world on fire. And it’s a *lot* faster than frobbing IBRS on analysis showing Skylake is still OK, but then was selling faulty CPUs for half a every entry into the kernel. It’s a massive hey-ho, screw Skylake. year while world was on fire; they knew performance win. The early part of the series adds the they are faulty yet they sold them any- So now we *mostly* don’t need IBRS. new feature bits and detects when it can way. Then Intel talks about how great We build with retpoline, use IBPB on con- turn KPTI off on non-Meltdown-vulnera- they are and how security is important text switches/vmexit (which is in the first ble Intel CPUs and also supports the IBPB for them … . Intentionally confusing be- part of this patch series before IBRS is barrier that we need to make retpoline tween Meltdown and Spectre so they can added), and we’re safe. We even refac- complete. That much I think we defi- mask how badly they screwed. And tored the patch series to put retpoline first. nitely *do* want. There have been a without apologies.” But wait, why did I say “mostly”? bunch of us working on this behind the He went on to say, “and now Intel Well, not everyone has a retpoline com- scenes; one of us will probably post that wants to cheat at benchmarks, to put piler yet … but OK, screw them; they bit in the next day or so. companies that do right thing at disad- need to update. I think we also want to expose IBRS to vantage and thinks that that’s okay be- Then there’s Skylake, and that genera- VM guests, even if we don’t use it our- cause world was on fire? At this point, I tion of CPU cores. For complicated rea- selves. Because Windows guests (and believe that yes, product recall would sons they actually end up being vulnera- RHEL guests; yay!) do use it. be appropriate. If Intel is not willing to ble not just on indirect branches, but also If we can be done with the shouty part, do it on their own, well, perhaps courts on a “ret” in some circumstances (such I’d actually quite like to have a sensible can force them. Ouch and I would not as 16+ CALLs in a deep chain). discussion about when, if ever, we do mind some jail time for whoever is re- The IBRS solution, ugly though it is, IBPB on context switch (ptraceability and sponsible for selling known-faulty CPUs did address that. Retpoline doesn’t. There dumpable have both been suggested) and to the public.” are patches being floated to detect and when, if ever, we set STIPB in userspace. The discussion continued for quite prevent deep stacks and deal with some awhile, mostly on a technical level. of the other special cases that bite on Luke Kenneth Casson Leighton Aside from unhappiness toward SKL, but those are icky too. And in fact thanked David for this explanation, say- Intel, the issue is fascinating and helps IBRS performance isn’t anywhere near as ing, “there is actually a significant bene- illuminate some of the fundamental is- bad on this generation of CPUs as it is on fit to what you’re doing, not just peanut- sues confronting a universe in which earlier CPUs *anyway*, which makes it gallery-ing.” He said, “this is a cluster-f * open source has already essentially not quite so insane to *contemplate* where every single Intel (and AMD) en- successfully taken over the world. How using it as Intel proposed. gineer is prevented and prohibited from can the relationships and conflicting That’s why my initial idea, as imple- talking directly to you as they develop motivations between open source proj- mented in this RFC patchset, was to stick the microcode. … They’ve been ignored ects and for-profit corporations be with IBRS on Skylake and use retpoline and demoralized. It’s a lesson that I’m properly managed? How can private everywhere else. I’ll give you “garbage not sure their management are capable companies reveal crucial information patches,” but they weren’t being “just of learning, despite the head of the Intel about proprietary microcode and such mindlessly sent around.” If we’re going open source innovation centre has been things, while protecting their competi- to drop IBRS support and accept the cave- trying to get across to them for many tive secrets from other corporations? ats, then let’s do it as a conscious deci- years: OPEN UP THE FUCKING FIRM- And if, as Linus guessed in an early sion having seen what it would look like, WARE AND MICROCODE.” post, Intel intends to maintain a signif- not just drop it quietly because poor He went on, “unfortunately, the bur- icant flaw over the long term, how can Davey is too scared that Linus might den is on you, the members of the linux open source developers and users com- shout at him again. :) kernel team, to read between the lines, municate with, and perhaps put pres- I have seen *hand-wavy* analyses of express things clearly here on LKML so sure on, such companies to change the Skylake thing that mean I’m not ac- that the intel engineers who are NOT that kind of decision? nnn

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 15 COVER STORIES Free Firmware with Libreboot

Transparency in firmware with Libreboot Liberated Libreboot is a project dedicated to making firmware free as in freedom. By Erik Bärwaldt

or users who believe in free software, one of the biggest benefits is that, because you have access to the source code, you know what your computer is doing. Proprietary, closed-source Fsystems contain binary-only code that might be compromis- ing your security and privacy – and you wouldn’t even know it. Linux and other open source systems offer a level of transparency that simply isn’t possible with proprietary systems; however, even if you are using an all-free distro recommended by the Free Software Foundation (FSF) [1], your system might not be as free as you think. Most computers today come with proprietary firmware that boots the system and manages certain low-level tasks related to the hardware. Proprietary firmware has many of the same problems proprietary software has: You don’t really know what it is doing, and it could be putting your security and privacy at risk. BIOS, and its successor UEFI, are standards for the firmware systems preinstalled into most of the personal computers sold around the world, and they are generally not distributed under a free license. In addition, Intel’s management engine, which has been integrated into hardware since 2008, is a thorn in the side of the proponents of completely free systems, allowing remote access and opening up several possibilities for attack. Another problem with proprietary firmware for users with high security needs is microcode updates, which are supposed to improve the microcode of the CPU but add an extra layer of uncertainty and potential vulnerability. For all these reasons, resourceful developers have begun to replace the remaining non-free components in computer systems with free alternatives – including free firmware. Coreboot [2] is a community project dedicated to developing an open source firmware alternative. Coreboot, which was formerly known as the LinuxBIOS project, claims to support over 230 main boards. In addition to bringing transparency to the firmware, Coreboot also claims to improve boot time by removing some of the unnecessary features and code bloat associated with mainstream firmware systems. Players Several small international companies have emerged around free BIOS implementations. These vendors fill the niche of providing computers for users who want a system that is truly all free. They can also offer the benefit of faster boot time with a trimmed down system, and they are in a unique position to avoid the planned obso- lescence of the computer industry by maintaining support for older systems that are no longer relevant to the big hardware vendors. Nearly all of these companies primarily offer Lenovo laptops. The economies of the free firmware industry mean that the companies tend to work with large volumes of older computers: the most common models are series X200, T400, and

16 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM COVER STORIES Free Firmware with Libreboot

Leah Rowe” box). The company only sells computers that have been upgraded with Libreboot. Minifree contributes to the financing of the coreboot distribution, and it also pays for additional developers who port Libreboot to new hardware. Depending on availability of the necessary hardware, Minifree offers a freely configurable server and desktop system, notebooks, and accessories. The devices are mostly systems from older processor generations, because the great effort for the implementation of Libreboot allows only limited modifications, especially for laptops. In addition to the actual BIOS, the developers also adapt the embedded controllers, removing Intel’s management engine or the equivalent Secure Processor from AMD. In addition, they replace problematic network hardware with components that do not require proprietary firmware, such as Qualcomm/Athe- ros components. T500 After the upgrade, the computer systems are completely free Think- of closed source software, which means a distribution that Pads, does not use proprietary binary blobs and does not require sep- which the arate firmware for the operation of some hardware components original man- runs without any functional impairment. ufacturer pro- The Lenovo ThinkPad T400 notebooks, currently offered duced between by Minifree and certified by the FSF under the “Respects 2008 and mid-2010. Your Freedom” program, are therefore completely free sys- These systems are based on tems. These systems come with a two-year guarantee at the Core 2 duo processors of Intel’s prices of between just under EUR240 to EUR630 (~$296- Penryn architecture, which have two cores but $777), plus shipping. lack performance-enhancing features such as turboboost and hyper-threading. Operating System The performance of these devices is fine for office and Inter- Minifree preinstalls Trisquel Linux on its devices. Trisquel is an net applications, but demanding tasks such as CAD applications Ubuntu derivative from which the developers have removed or video transcoding push the hardware to its performance limit. any non-free components [5]. The system also differs visually Many modified Lenovo notebooks are certified by the FSF. from the baseline Ubuntu system: Instead of the fuzzy brown (The US provider Purism is an exception to this low-end- and purple tones in the original, Trisquel features fresh blue hardware strategy. See the box entitled “Purism: A Different and green, with application windows featuring 3D effects in Approach.”) light gray shades (Figure 1). Trisquel Linux, which originates in Spain, has the approval Libreboot of the FSF as a completely free operating system. Minifree pro- The Libreboot project [3] was founded in December 2013 in vides the complete source code for Trisquel, totaling almost the UK with the goal of replacing the BIOS and video BIOS 5GB, as well as the source code for Libreboot, which weighs in of common GPUs with free components by reverse engineer- at several hundred megabytes. ing. Libreboot, which is written in C and Assembler, is a derivative of coreboot, but the Libreboot developers re- Purism: A Different Approach move some remaining proprietary firmware blobs that US provider Purism is a laptop vendor that specializes in free still exist in the coreboot. Libreboot programmers do not try firmware, but Purism doesn’t depend on older systems. The to compete with coreboot but participate actively in its devel- company is itself a laptop manufacturer that uses state-of-art opment by contributing patches and simplified installation hardware. Purism equips the devices with coreboot and re- processes. moves other closed-source elements: For example, the com- Libreboot was simultaneously implemented on several pany’s Librem notebooks exclusively use WLAN cards with the server and desktop systems with standard components, as well free-firmware-friendly Atheros chipsets. In addition to the as on some notebooks. Since extensive adjustments for the coreboot BIOS, Purism uses the Debian-derived PureOS and chipset and motherboard were necessary, the software runs on Qubes OS operating systems. a limited number of hardware components. The worldwide The capitalization of the company demonstrates the value of this distribution of the devices is managed by a company called business idea: Purism received approximately US$2.5 million in Minifree [4]. The name Minifree is short for Ministry of Free- working capital as part of a crowdfounding scheme. This influx dom – a reference to the novel 1984. of capital enabled the company to modify its original concept of built-to-order and now sell their notebooks from stock – eliminat- Minifree was founded in December 2014 by 23-year-old ing months of delivery time. Leah Rowe (see the “Meet Libreboot and Minifree Founder

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 17 COVER STORIES Free Firmware with Libreboot

ware replaces the original Intel WLAN card. After the device switches on, the rapid speed is surprising. The Libre- boot BIOS, which does not display a ThinkPad logo, works significantly faster than the Lenovo BIOS. Tris- quel Linux also boots considerably faster than other systems on similar hardware. Unusual Unlike conventional BIOS or UEFI systems, Libreboot does not offer a way to call a dialog from the firmware at startup that lets you configure the boot order or other settings. Instead, a GRUB selection menu opens directly. You can choose the medium from Figure 1: The Trisquel Linux desktop looks fresh and friendly. which the computer will boot. It is possible to select the desired Test Device option using a hotkey. The most important functions are [O] to We ordered a Lenovo ThinkPad T400 from Minifree with Tris- start the installed Trisquel with an encrypted hard disk, [U] to quel and Libreboot preinstalled. Just three days after the order, boot from a storage medium connected via the USB bus, [D] to an unexpectedly large package arrived with a 14-inch notebook boot the system from an optical disc, and [S] to find a GRUB 2 (Figure 2). In addition to the ThinkPad and its power supply configuration on an external disk. You will also find a unit, the package included a patch cable and two power cords, Poweroff[p] option for shutting down the system and a Reboot[r] one with a plug for continental Europe and the other with the option for restarting the system. British three-pin connector and a built-in fuse. In our lab, manually burned CDs and DVDs from ISO images The ThinkPad T400 has a 14-inch display in the 16:10 form functioned as boot media without any problems. In other cases, factor with a resolution of 1280x800 pixels. The Core 2 duo the boot options failed: For example, distributions preinstalled processor of the “Penryn” generation includes energy-saving features, running at a maximum clock speed of 2.4GHz. The T400 comes with 4GB of RAM and a 160GB conventional SATA hard disk. The rechargeable battery supports mobile use for up to six hours. Although the test notebook was created in May 2009 and is therefore more than eight years old, it looks almost as good as new. The positive impression of the hardware was confirmed by reading out the parameters of the still- sealed hard disk: The test system was in use for just under 7100 hours of operation – an unusually low number for a computer of this age. The original keyboard, as well as the case itself, show no discernible signs of usage even if you look closely. Minifree also includes a two-page quick start guide that explains the authentication data provided in Trisquel Linux. In addition to the free BIOS and the VBIOS modification, the device has another special feature: An Atheros Figure 2: The T400 from Minifree arrived in our test lab in practically new WLAN card that supports free firm- condition.

18 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM COVER STORIES Free Firmware with Libreboot

The Libreboot project describes options for manually boot- Listing 1: Transferring with dd ing from external discs in its detailed documentation. The de- $ dd if=/image_name of=/dev/device_name bs=4M; sync velopers recommend transferring ISO images to data media such as USB memory using dd (Listing 1). Accordingly, we on different USB media would not start, although they started on prepared several USB sticks of different types with different similar Fujitsu-Siemens and Hewlett-Packard devices with a non- distributions. All manually created flash memory devices free BIOS. Linux Magazine DVDs generally wouldn’t boot. booted without any problems. Media created with tools like UNetbootin or the ROSA Image Writer did not work. When booting from removable media, the startup time was significantly shorter compared with conventional computers. We experienced no problems with hardware components during the test mode. Even the modified VBIOS of the ThinkPad T400 caused no problems (Figure 3). Weak Points As expected, the “liberated” ThinkPad T400, in combination with Trisquel Linux, displays weaknesses where developers have not yet replaced proprietary blobs and firmware with free components. This applies in particular to WWAN devices that could not be talked into cooperating, such as CardBus plugin cards. In view of the thousands of components and devices available on the market, it is hardly surprising that a small project like Libreboot is not able to equip all these components Figure 3: Even movies work on Trisquel Linux without any problems. with completely free firmware.

Figure 4: Thanks to Ubuntu’s software repositories, almost all users get their money’s worth.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 19 COVER STORIES Free Firmware with Libreboot

Non-free software codecs, which are completely miss- Meet Libreboot and Minifree Founder Leah Rowe ing in Trisquel Linux but which are necessary for playing LM: Minifree Ltd. currently only offers a laptop model, a modified certain multimedia content, also limit the possible applica- Lenovo ThinkPad T400. [Note: The ThinkPad X220 also became tions. Nevertheless, the complete system with more than available since the time of this interview.] Although technically 47,000 packages from the Ubuntu repositories is completely and visually in very good condition, these machines are around suitable for everyday use (Figure 4). eight years old. Lenovo has long been distributing more modern computers with considerably higher performance. Libreboot is not yet available for more modern devices with the Core i archi- Conclusions tecture. Is there any chance that this will change soon? Free software again demonstrates its value when it replaces Leah Rowe: We’re working on setting up Libreboot for Sandy hardware-specific proprietary firmware: Libreboot even gives Bridge systems. Specifically, we are porting to ThinkPad X220 older equipment a boost and has a positive influence on the laptops. After that, we’ll work on the ThinkPad T420. behavior of the system. The ThinkPad T400 systems from Mini- This work is still in progress because we still need the non-free free are ideal for tough everyday uses thanks to their careful firmware for Intel’s management engine to operate the devices. programming and configuration. Let’s hope that the develop- Without this firmware, the system performs a hard reset every 30 ment team led by Leah Rowe will receive more support for minutes – we’re trying to eliminate this. In the past, there was the ThinkPad X200 with Libreboot; we want to offer this option again porting Libreboot to new hardware. nnn in the future. In addition, we want to accelerate the sale of the T400 so that we have more resources to finance the X220 ports. Info LM: Is there any support from well-known hardware manufactur- [1] FSF Free GNU/Linux Distributions: ers for the Libreboot project? Have you tried to get support? https://www.gnu.org/distros/free‑distros.en.html LR: No. We do not receive any support from the manufacturers. We have tried to convince AMD to support us. At the beginning [2] coreboot: https://www.coreboot.org/ of 2017, we talked to AMD about cooperating with coreboot and [3] Libreboot: https://libreboot.org/ Libreboot, which would have led to the release of non-free PSP firmware [6]. I was just told that my request had been forwarded [4] Minifree: https://minifree.org to the responsible team. [5] Trisquel Linux website: https://trisquel.info/ I no longer believe that these manufacturers will ever cooperate with us. We are more oriented towards the RISC-V project [7], [6] AMD-PSP: https://www.amd.com/en/technologies/security which develops a completely free Instruction Set Architecture [7] RISC-V project: https://riscv.org for a CPU. RISC-V is not yet usable in laptops, workstations, or servers, but it is already suitable for developer boards or for a microcon- troller setup. With RISC-V, it would be possible in the future to offer large-scale OEM systems with completely free hardware, but that will take a few more years. LM: What support is Libreboot currently receiving from the free software community? And what support do you want – apart from programmers who can help develop free firmware? LR: Libreboot has recently appointed a new full-time maintainer for the project. On our website, we offer an overview for developers who wish to contribute. Recently, a number of people who have not been involved in the project have contributed to Libreboot. My main concern is to get more support outside programming. Anyone can participate in the Libreboot project – even without technical experience: For example, volunteers can provide help to users on the IRC channel or help improve the documentation. There is also a Librecore project that Libreboot is working with. Many highly qualified developers work for Librecore, setting up new hardware ports and making them available for Li- breboot. Libreboot is currently based on coreboot, but in the future, we want to use Li- brecore as an upstream for hardware initialization. Librecore is a fork of coreboot. Today, Libreboot is a coreBoot distribution; in the future, it will be a Librecore distribution. Primarily, we want to attract more supporters, so that their work will allow us to use more hardware and Libreboot’s list of supported hardware will continue to grow.

20 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM

REVIEWS Collaborative Office Suites

Collaborative workflows Shared Office Several collaborative office suite providers are now competing for customers who are looking for the seamless exchange of office documents within an enterprise. These cloud or on-premise solutions are designed for teamwork. By Erik Bärwaldt

oday, shifting documents from allow simultaneous use of the data. servers on site often proves to be compli- employee to employee by email Typical applications include diary, ad- cated and error-prone. Users leverage attachment is considered “so dress management, and project man- cloud as Software as a Service (SaaS) [2] 90s.” No wonder: Office suites agement software. to avoid the need to install a server vari- Tenable completely new forms of cooper- The other approach to collaborative ant onsite – operators completely man- ative work, often via cloud or on-site software involves web-based solutions age the services centrally. servers. This article takes a look at the storing data and software packages cen- rather confusing market that has devel- trally on a self-managed server. Users Uncharted Territory oped around collaborative office suites. access the service via a web browser. When it comes to editing texts, tables, On the upside, this concept offers lower and presentations, groupware-capable Concepts administrative overhead: You do not solutions do not yet play as important a There is no uniform approach to what have to install and maintain any appli- role as in traditional collaborative tasks: collaborative work looks like. Different cations on the workstations, and a web Project management, PIM, or CRM sys- methods reflect different visions of the browser is standard on modern com- tems have been focusing on cooperation modern work environment. However, one puter systems. The installation’s poten- for some time now – they no longer exist important distinction is between a client/ tially inferior reliability is the downside: as single-user versions. server-based collaborative working envi- If the server fails, the whole system Numerous Linux providers lag behind ronment and web-based collaboration. ceases to function. in terms of office teamwork solutions. Client/serv er structures store all data In addition, there are also purely They deliver office suites that are suit- processed by a team on a server. A cli- cloud-based programs in which both the able for use in workgroups. However, ent application uses the Message Appli- applications and the data storage reside there is a sticking point: If users work in

cation Programming Interface [1] to in the cloud. The advantage: Installing environments with a range of operating Lead Image © Surflifes, Fotolia.com

22 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM REVIEWS Collaborative Office Suites

Noteworthy the office suite locally on their desktop systems. In addition to traditional groupware office tion. It is available in a desktop version, However, in contrast to Collabora On- suites, which cover the entire spectrum of but effective use requires a commercial office work with texts, tables, and presen- account with the software developer. The line, CODE is rather experimental in tations, Linux supports several small proj- scripts can be stored in a cloud-based, en- character. The company, therefore, does ects that solve sub-tasks. crypted form on the manufacturer’s serv- not recommend it for production use. It is available as a Docker image for use in Particularly noteworthy are EtherPad ers. In this way, users can also make the the home office and is more likely to be Lite [3] and EtherCalc [4]. They are ded- scripts accessible to others. used for testing purposes [9]. If you icated to collaborative work with texts Prezi [6], online presentation software and tables. Both projects work across based on HTML5, is quite popular with want to use the two packages, you need platforms and are web-based. You users. It provides a virtual whiteboard an ownCloud, Nextcloud, Pydio, or Seaf- store the individual texts or tables in and allows access for up to 10 users. Prezi ile infrastructure as a prerequisite. “pads” on a web server. Every user is available for a subscription fee and As a third package, the company distrib- who knows the URL can contribute to does not offer native Linux applications utes the Collabora CloudSuite (see [10] the text. These programs color-high- for local editing or presentation playback. for a demo). CloudSuite is a combination light contributions by the different An alternative is Sozi [7], a free plugin for of an online and a desktop editor. It in- users. Thus, even if several users work Inkscape, which is functionally very simi- cludes a local installation of Collabora simultaneously on a text or a table, you lar to Prezi. However, unlike older Prezi Office as well as the cloud variant. The can still keep track. variants, Sozi does not require Adobe’s result is that CloudSuite makes docu- Celtx [5] allows collaborative work on Flash technology, but instead generates ments available at the workplace and si- screenplays, focusing on media produc- SVG graphics. multaneously pushes them onto smartphones or tablets. systems, they need to easily exchange Collabora Online benefits from long- Offline files without any problems; they don’t term support and is available across Collabora also offers a complete office want to worry about converting between platforms. The manufacturer provides suite based on LibreOffice for users who file formats. commercial support services via various don’t have a server. Compared to the service partners worldwide. Beyond free original, it primarily stands out with Collabora Online this, Collabora Online Development conversion filter improvements (i.e., it Collabora Online (Figure 1) might be fa- Edition (CODE) is another web-based converts documents into different for- miliar to users: The Collabora Office [8] teamwork solution derived from Collab- mats in a superior way). Currently, ver- enterprise desktop solution from the ora Office core. Ultimately, both vari- sion 5.1 is available in a 680MB Debian United Kingdom is a customized Libre- ants allow users to create and edit doc- package for 64-bit hardware. The Libre- Office that has been modified to work in uments on web browser clients. This Office version also includes additional the cloud on a server. works without requiring users to install language packs and help texts. Functional Overkill Compared to a locally installed office package, the online versions offer a significantly reduced range of functions in all areas. The editing options are similar to those of a LibreOffice, but you have to considerably restrict the formatting options. The online versions only accept DIN A4 and A5 paper formats, as well as the US letter and legal sizes. Even the formatting of inserted tables quickly reaches its limits. In terms of import and export formats, some of the less common file formats are missing. You can only download documents to your local system in four formats, the software distinguishes between text, table, and presentation formats. A positive feature is the comment function, which is important for col- Figure 1: Collabora Online is a functionally streamlined LibreOffice with laborative work. Users can access this teamwork components. in all documents via Paste | Comment.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 23 REVIEWS Collaborative Office Suites

Functions Google Docs provides all the features needed for collaborative work. This includes basic document editing, comments, and file sharing. If desired, two users can edit a document simultaneously. In this case, one participant can see almost in real time the changes the other participant is making based on the cursor movements. Another unique selling point is the modular concept, which means you can extend the applications with add-ons such as dictionaries and templates. One disadvantage of the add-on architecture is that Google tightly links Docs to its own web browser; many add-ons therefore only run in Chrome. Google Docs does not have the ability to save documents manually. Instead, the provider automatically saves them in Figure 2: In the right-hand pane, you can see all the important data on the Google Cloud. However, download- a document’s processing status. ing and converting the data is relatively flexible: You can download your files in In a separate window, you then enter a Forms. The Google suite is completely different formats to your local system, short comment, which appears to the cloud-based. Consequently, local desk- where you define the desired formats via right of the actual document after top computer clients don’t exist, and on- File | Download as. The individual op- clicking Save in the main window. The premise installation on local dedicated tions are configurable in a context-sensi- various team members’ comments all cloud servers is not possible. Google tive approach. appear together. This makes it easy to only offers apps for smartphones and You will also notice that older Micro- assign annotations to the document tablets; on stationary systems, the soft- soft formats are missing in the export text, similar to the comment function ware runs in the web browser. function. In addition, Google Docs offers in offline Office packages. To work with Google Docs (Figure 3), an unusual user interface: You do not Users can track changes in the docu- you need a Google account. Once you see an empty document at the begin- ments without difficulty. For each appli- have set up an account, you can log on ning; instead, to open a new file, you cation, you can switch a logging func- to the office packages separately. have to explicitly click on the red plus tion on and off via Edit | Track changes. This feature also allows you to scroll between the individual changes. This gives the user a quick overview of document versions. At the same time, Collabora Online presents these functions in the list view on the right-hand side of the screen; a mouse click opens an area. In this way, you can immediately see all the relevant data on the current document’s processing status (Figure 2) without having to click your way through complex menu hierarchies.

Google Docs Search engine giant Google inter- weaves its online office package Google Docs [11] tightly with the company’s many other services. The package itself consists of four components: Google Docs, Google Spreadsheets, Figure 3: Google Docs is completely web-based; clients are only avail- Google Presentations, and Google able for smartphones and tablets.

24 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM REVIEWS Collaborative Office Suites

also has a cloud connection to various services and proprietary cloud support. A CRM system including an accounting module, a project management application, an instant messenger, and a calendar with time tracking function are also available. The Enterprise Edition is also available as 30-day trial version. The Community Server is available for free download on the company’s website [14]. The Com- munity Edition is limited to a maximum of 20 users and lacks an integrated backup and restore solution. In addition, the Community Edition lacks automatic updates and a comment function for collaborative document editing.

System Requirements The on-premise installation of the On- lyOffice server has some hardware and Figure 4: When a new file is created, the user interface is initially empty. software requirements. The minimum hardware requirement specified by the symbol located bottom right in the view is lost because Google Docs does manufacturer is a server with a dual-core browser (Figure 4). not clearly distinguish between the con- CPU and 2GHz clock frequency, 2GB Like many local office applications, tributions from the individual users. RAM, a swapfile of 6GB, and 40GB of the presentation module provides a Also, the function of undoing individual available hard disk space. In addition, a number of templates, including various modifications is lost if several users col- Linux kernel version 3.10 or newer must transitions, designs, and formatting op- laborate on the same document. For this be running on the server. tions that allow you to create profes- reason, we recommend limiting the Depending on the installation method, sional presentations. In terms of font number of active users. there are additional software require- management, Docs additionally inte- ments. If you install the server manually, grates a large number of fonts, with at OnlyOffice it requires a MySQL database server, a least a rudimentary preview. However, Developed in Latvia by Ascensio System Nginx web server, the Mono framework, it lacks Asian fonts and an option for SIA, OnlyOffice [12] (formerly known and the Node.js runtime environment. If setting up Asian writing systems. as TeamLab) is a highly flexible, free of- you use Docker images, a Docker envi- fice suite under the GNU GPLv3. In ad- ronment including daemon is added. Compatibility dition to the locally installable desktop Fortunately, all installation methods Google claims that its office applications editors with the usual applications are well documented. In addition, On- are compatible with Microsoft Office. If (word processing, spreadsheet, and pre- lyOffice links to YouTube video tutorials the documents are simple texts, tables, sentations), OnlyOffice also offers two via the website. or presentations, documents stored in server-based variants for companies’ IT When installing the server variants, Microsoft formats in Google’s office ap- infrastructures. you should make sure that the required plications will look largely identical to The free community edition includes packages are installed in as up-to-date how they appear. However, for docu- three components: a mail and docu- versions as possible. Otherwise, the in- ments that embed macros in tables or ment server, as well as a groupware so- stallation may fail. use very complex design elements, there lution known as Community Server. For example, OnlyOffice under Ubuntu are deviations and sometimes incorrect There is also a commercial Enterprise 16.04 LTS does not need the packages formatting. Edition. provided by Ubuntu, but rather the offi- The variety of installation options is cial Node.js packages from the project Disadvantages striking: The Community Edition is side. The versions in the Ubuntu reposi- Because Google Docs is closely linked to available in the form of precompiled tories are obsolete, which prevents the Google services, it also has a number of packages in deb and RPM package for- OnlyOffice server from being installed. functional disadvantages: For example, mats, as well as a Docker image: the En- In short, you should strictly follow the all participants in a session must use a terprise Edition is also available as a Vir- instructions to avoid inconsistencies. Google account. The Google Docs user tualBox and VMware image, each about invites additional users by email. If more 4GB in size [13]. The Community Server Access than two users work on a document at lets you manage documents in collabora- Once the server is installed successfully, the same time, the modification over- tive environments and transfer files. It you can reach it from every workstation

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 25 REVIEWS Collaborative Office Suites

on the intranet. You just need to enter the IP address to access the portal page. Here, you first configure some authentication settings and log on to the server. This takes you to the work screen. Even a first glance reveals that OnlyOf- fice, unlike Google Docs or Collabora Of- fice, provides a complete environment for collaborative work. In addition to the document manager, you will find buttons for projects, a CRM system, an email client, symbols for user administration, and community activities on the start page. All these applications operate under a uniformly designed interface and provide an overview of the basic functions in a superimposed window (Figure 5) when clicked on.

Figure 5: The OnlyOffice interface and its services look pretty tidy. Office Suite In on-premise mode, OnlyOffice does not let you edit web-based documents. To edit web-based docs, you’ll need to install the document server. A conve- nient file manager makes it possible to share documents and work on them as a team. However, the actual editing functions of word processing, spreadsheets, or presentation programs are provided by the document server, which needs to be integrated separately. The on-premise mode also offers On- lyOffice Desktop Editors [15] locally on workstations. The software is licensed under the free AGPLv3 license and, unlike the web-based competitors, is a fully fledged office suite. Various deb and Figure 6: OnlyOffice Desktop Editors using the Community Server. RPM packages are available for Linux, but they only support some Linux derivatives. A portable version (around 217MB in size) that can be used across all distributions is also available. Note that OnlyOffice Desktop Editors is only available for 64-bit hardware. After downloading, install it locally on the workstations and integrate it into the server services. Once the desktop suite is installed, you can easily import the documents stored on the Community Server into the locally installed office suite and edit them. The advantage of this method is that even documents with very sophisticated formatting can be fully edited. Important: Other suites, such as Libre- Office, cannot be easily linked to the Figure 7: Even simple ODF documents can cause problems for OnlyOffice’s Community Server. It works, but in het- conversion filters. erogeneous environments with Microsoft

26 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM REVIEWS Collaborative Office Suites formats, this sometimes involves a con- ments in ODF for- siderable amount of post-processing. mat that required Due to the sometimes poor conversion post-processing filters, LibreOffice returns layout errors (Figure 7)? for current Microsoft formats even within Even though the Community Server’s simple sample you may not be al- documents. OnlyOffice’s filters are much lowed to edit all more mature in comparison and do not supported for- display these errors (Figure 6). mats, the func- Another drawback for collaborative tional scope of the environments: It is not possible to work individual editors on a document simultaneously. How- clearly outshines ever, the Community Server does offer that of other on- version history for individual docu- line solutions. On- ments. In this way, it is at least possible lyOffice Online to track which user has edited which does not let you document. edit a document simultaneously Online with two users, Ascensio System SIA provides the docu- but it offers a Figure 8: In a small dialog box, users release document server separately as free software comment function ments for other users and groups. for download. The company also offers a and an option to purely cloud-based solution. It is free of share documents. Other users can also These services are only provided to charge for personal use. You are simply work on mobile platforms (Figure 8). commercial users, for whom the com- required to register and activate your ac- pany offers a free 30-day trial version. count by email. After logging in, you end Pay Barrier This version has the full functionality of up directly in the document manager. Compared to the dedicated server, the on- the on-premise enterprise server and is The user interfaces of the on-premise line solution for educational and non- thus able to handle various backup and and online versions are identical. In con- commercial use lacks further modules for restore options, as well as the entire trast to the server version for intranets, collaborative work. Also missing are all spectrum of collaborative work, includ- OnlyOffice Online already provides edi- additional applications, including the ap- ing chats between individual users. tors for texts, tables, and presentations. pointment calendar, the CRM system, the At the click of a mouse, you can also inte- On request, you can also integrate cloud email server, and the project manager. grate many third-party services (Figure 9), services, such as Google, Dropbox, Box, or Yandex.Disk, into OnlyOffice. To integrate a cloud service, click on a start button with a major service provider and authenticate. Locally hosted cloud solutions, such as Nextcloud or ownCloud, implement the connection via the Web- DAV protocol. In heterogeneous environments, On- lyOffice Online also supports common formats. These formats include the Open Document Foundation file formats used in LibreOffice and OpenOffice, but also PDF, CSV, EPUB, or HTML formats. However, in some cases, the formats come with considerable restrictions. One serious disadvantage of OnlyOf- fice Online is that the cloud solution forcibly converts documents from a local instance to the current Microsoft Office Open XML format when they are uploaded. It then either stores the file alone or together with the original file in the cloud. However, during the conversion, what if there were already problems in the test with simple docu- Figure 9: OnlyOffice can be extended by apps.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 27 REVIEWS Collaborative Office Suites

including two-factor authentication. If so desired, you can also integrate your con- tacts from social networks into the CRM module.

EGroupware The collaborative solution EGroupware [16], developed in Kaiserslautern, Ger- many, provides a complete working environment much like OnlyOffice. The EGroupware environment is not limited to pure document processing with the help of an office suite. You can also test the package free of charge for 30 days by creating an account in the cloud. Just enter your email address to activate access. The provider sets up EGroupware, and the access data is delivered by email. The commercially distributed variants provide various cloud access models to users in Germany. The data are stored in Figure 10: At first glance, you might notice that the portal also supports German data centers and are thus also collaborative work on documents. subject to German data protection and security guidelines. up window. (You must either allow pop- quired, you can download documents Once you have installed an on-premise up windows in the web browser or grant to the local mass storage device. Use solution in either the Pro or Community an exception in the associated settings the File | Download as dialog, which version [17] or activated the cloud infra- dialog.) Alternatively, right-click on the also allows conversion to PDF format. structure, you can access a clearly laid file you created in the file manager and For other users, the owner of a file sets out portal that initially shows you a cal- choose Open. The Collabora editor then up shares for collaborative editing by endar after successfully logging in. The appears in a tab of the web browser. right-clicking the option from the con- left-hand sidebar contains the configura- The benefits: You can open multiple text menu. tion and service options required for files simultaneously, allowing fast access first-time users. As the administrator, to the file manager or other EGroupware On Premise you can set up new groups and users components without navigating through Just like OnlyOffice, EGroupware offers a with the appropriate rights (Figure 10). the web browser (Figure 11). server with the same name for local in- Unlike OnlyOffice, EGroupware’s of- The existing documents also store stallation. For local installs, you will find fice suite does not offer meaningful the cloud variants in the cloud. If re- a community version available under the icons on the portal page. The file manager in the left-hand sidebar in the lower area lets you access collaborative document processing. It displays the existing datasets in list form. You create documents using the hori- zontal buttons at the top. Clicking on the plus symbol opens a selection of the document types supported by the integrated Collabora server: In addition to text documents, these types include tables and presentations. Once you have selected a document type, file name, and file format, the Collab- ora Office component formats the files in the free ODF format. Alternatively, you can also choose proprietary Microsoft formats, which are limited to the two current variants. The software does not let you save a document as a TXT or PDF file. After entering the basic file information, the Collabora editor opens in a pop- Figure 11: EGroupware opens different tabs for each application.

28 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM REVIEWS Collaborative Office Suites

GNU GPLv2, which is available via Table 1: Overview of Collaborative Office Suites the manufacturer’s website for open- Collabora Google Docs OnlyOffice EGroupware SUSE, SLES, Fedora, RHEL, Centos, Online/Office Debian, and Ubuntu [17]. The server is Online server and apps Yes Yes Yes Yes also available in the repositories of some On-premise server Yes No Yes Yes distributions, such as ALT Linux, Open- Office suite used locally Yes No Yes No Mandriva, Mageia, and ROSA Linux. The installation automatically sets up Groupware functions No No Yes Yes the Apache web server and PHP as de- Import and export function Yes Yes Yes Yes pendent packages. In addition, you must ODF support Yes Yes Yes Yes manually maintain various add-on mod- Community version Yes No Yes Yes ules that you do not initially install into Enterprise version Yes Yes (integrated Yes Yes the suite. in G suite) The hardware requirements for operat- Integration of foreign cloud Yes No Yes No ing an on-premise server are 8GB RAM, services at least 50GB free mass storage, and a Fee-based support Yes No Yes Yes 64-bit operating system. Firefox, Chrome, or Safari are the recommended web browsers on clients. in-one solution. In addition, the appli- One of the drawbacks of all partici- For small businesses, a single server cations are more completely localized pants is the rather moderate quality of seems sufficient for all components, but in other languages, than those of the the conversion filters: In EGroupware, the developers advise medium and large duo EGroupware and Collabora. Collabora Online sometimes formats in- organizations to distribute the individ- EGroupware, on the other hand, offers correctly even simple ODF documents ual services across multiple servers. much better documentation in other lan- created with OpenOffice. OnlyOffice Email servers such as Dovecot, Postfix, guages and – together with Collabora – also sometimes fails on simple ODF or Cyrus, as well as the required web the best support in the test field. In the documents. servers Apache or Nginx and database case of cloud-based services, EGroup- All of the reviewed applications are servers (we recommend MariaDB or ware also stands out because the servers now functionally mature and fit for pro- MySQL) can be imported separately. run completely in German data centers ductive use (see Table 1). If you want to The locally installed community server and users benefit from strict German work cross-platform as part of a team, – as well as the paid enterprise version data protection guidelines. you will often find that they offer real – offers all groupware functions, but The purely cloud-based service from added value compared to local single- does not have a collaboration server by Google is also available for slower online user systems. Nevertheless, a practice- default. Collaborative document pro- access, but only provides basic func- oriented evaluation is recommended be- cessing is only possible if you integrate tions. In addition, it is firmly integrated fore introducing a solution. Otherwise, a cloud-based or locally installed Col- into the Google universe with non-trans- there may be an additional workload if labora server. parent data protection according to US you have to adapt infrastructure and standards. documents. nnn Documentation and Support The EGroupware suite, which has been Info around for several years, is well docu- [1] MAPI: https://en.wikipedia.org/wiki/MAPI [12] OnlyOffice: mented. The manufacturers offer tutori- [2] SaaS: https://en.wikipedia.org/wiki/ https://www.onlyoffice.com als and video tutorials on the Internet, Software_as_a_service [13] OnlyOffice Enterprise test version: but also hold regular webinars. Enter- [3] EtherPad Lite: https://yourpart.eu https://www.onlyoffice.com/ prise customers can use their own sup- enterprise‑edition‑free.aspx port platform in differently sized pack- [4] EtherCalc: https://ethercalc.net [14] OnlyOffice Community Edition: ages. You have to create your own user- [5] Celtx: https://www.celtx.com https://www.onlyoffice.com/download. name and password. In addition, the [6] Prezi: https://prezi.com aspx chargeable offers include personal sup- [7] Sozi: http://sozi.baierouge.fr port based on fixed time quotas. [15] OnlyOffice Desktop Editors: [8] Collabora: https://collaboraoffice.com https://www.onlyoffice.com/ Conclusions [9] Collabora version comparison: download‑desktop.aspx In practice, the two groupware offer- https://www.collaboraoffice.com/code/# [16] EGroupware: ings, EGroupware and OnlyOffice, were what_is_code https://egroupware.org particularly successful. Both succeed [10] Collabora CloudSuite: [17] EGroupware Community packages: in adding groupware functions to the https://www.collaboraoffice.com/ https://software.opensuse.org/ office suites almost seamlessly. On- solutions/ collabora‑cloudsuite/ download. html? lyOffice is ahead by a narrow margin: [11] Google Docs: project=server%3AeGroupWare& The suite looks like an ergonomic all- https://www.google.com/docs/about/ package=egroupware‑epl

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 29 IN-DEPTH Electron

Exploring the Electron application framework Universal Desktop

GitHub’s Electron project brings the benefits of web programming to the realm of desktop applications. By Andreas Möller

f the boss orders a cross-platform Bold Design run time, Electron avoids the possibility desktop app for internal use, An Electron app works like a web appli- of problems arising from dissimilar stressed IT staff often lack the cation but in the reduced browser envi- browsers or browser versions. Developers time to get familiar with C++/Qt, ronment outlined in Figure 1. The Chro- also benefit from the debugging tools that IJava, or Python and the associated ap- mium components execute the applica- come along with Chromium (Figure 2). plication packaging. The Electron proj- tion (top right in the picture). JavaScript The main process manages the ect [1] solves this problem by making libraries such as Bootstrap [4], React [5], browser window. The Node API facili- a desktop app more like a web app. Angular [6], or JQuery [7] contribute to tates the connection to the desktop oper- Electron combines Chromium [2] and the complete package. ating system. The developer can extend Node.js [3] into a single run time, thus Because the code within the applica- the Node environment by adding addi- allowing the developer to build the ap- tion always executes using the Chromium tional node packages. plication using the standard tools of web development, such as HTML, CSS, and JavaScript. The cross-platform nature of web development makes it quite easy to adapt your application to run on Windows, Mac OS, or Linux, and Electron even comes with tools that will package the app for the target OS. Electron applications even update themselves automatically. Updates are retrieved from known sources, such as a GitHub repository, via an update server. Electron began in 2013 as the framework for building GitHub’s Atom text editor. GitHub open sourced Atom and Electron in 2014. Electron has an MIT license. Version 1.0.0 was released in 2016, and as of this year, it is available in Windows and OS X app stores as well as Figure 1: The main process replaces the browser kernel as a freely

Linux repositories. programmable framework with a system connection. Lead Image © pino, 123rf.com

30 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM IN-DEPTH Electron

the development progress by switching to the app project directory and entering the forge init start command. Since a watch process like Angular is missing, a location.reload() reloads the app via the debugging console. Figure 2 shows the started app at run time. The debugging tool appears in the Chromium window on the right side of the screen by default. In the final version, the developer has to remember to disable the debugging tools in the main process code. The yarn.lock file from line 5 in List- ing 2 remembers the currently stored packets under node_modules. This mecha- nism makes it possible to reproduce node projects on other machines on a scale of 1 to 1. Figure 2: Electron developers can call upon the help of Chromium debugging tools. Porting You can port the boilerplate app from Installing a Framework projects internally use electron‑packer Figure 2 to other desktop systems Electron, the boilerplate project, and a to create the Electron application and using the forge package command. set of tools used with Electron, includ- electron‑builder to packetize platform- Without specifying options, the coming electron‑packer, electron‑builder, specific formats. electron‑forge, which mand (under Ubuntu 16.04) creates a or electron‑forge, all take the form of is installed in Listing 1, offers the best standalone version for Linux and the node packages. Listing 1 installs Elec- combination of both tools. It also comes existing x64 processor architecture in tron and all the tools a developer needs from the workshop of the creators of the out/app‑linux‑x64 subdirectory of to pack the Electron apps onto a electron‑packer and electron‑builder. the app project folder. In addition to the Ubuntu 16.04 node. Line 1 installs git The forge shell command makes use of approximately 80MB executable app in and curl via the package manager; line electron‑forge. ELF format, this directory also contains 3 installs the latest version of Node.js. The life cycle of an app starts when other binary files. You will find the web The necessary package list is created by the developer taps the forge init app application and the required node pack- a Bash script, which curl retrieves in command into the keyboard. This com- ages under resources/app. After switch- line 2, under /etc/apt/sources.list.d/ mand creates the app project directory ing to the directory, the ./app command nodesource.list, and then updates the from Listing 2 in the style of a boiler- starts the app in the shell, which looks list of available packages via apt-get. plate project. As with any node app, the exactly like Figure 2. After nodejs in line 3, the script inserts node_modules directory stores all pack- You can port the app to Windows the alternative node package manager, ages listed in the package.json configu- using the ‑‑platform command-line Yarn [8], one line later. With root privi- ration file – in the latest version. switch in the forge package ‑‑platform leges, Yarn then installs electron‑forge Finally, the src directory stores the win32 command. This command creates [9] (line 6). minimal web app, which the developer the app for 64-bit Windows in the out/ then converts step-by-step into the de- app‑win32‑x64 folder. In addition to the Empty Stage sired application. executable app.exe, the distribution con- Electron has a number of boilerplate Initially, the directory contains the tains a large number of Windows DLLs projects [10] that accelerate the develop- index.html HTML document and thus and the resources folder, which is more ment process by providing an minimal the entire web app, as well as the or less bit-identical to the matching executable app out of the box. These index.js file. The programmer checks folder for Linux. If you copy the folder from Linux to Listing 1: Installing a Node a Windows 10 system and start the

01 sudo apt‑get install git curl

02 sudo curl ‑sL https://deb.nodesource.com/setup_8.x | sudo ‑E bash ‑ Listing 2: Boilerplate App

03 sudo apt‑get install ‑y nodejs 01 |‑ app

04 npm install yarn 02 |‑ node_modules

05 sudo su 03 |‑ package.json

06 # it also works without Yarn with: sudo npm ‑g install electron‑forge 04 |‑ src

07 yarn add global electron‑forge 05 |‑ yarn.lock

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 31 IN-DEPTH Electron

Table 1: Electron Compatibility Listing 3: Configuration of forge

System Version IA-32 x64 ARMv7l ARM64 01 [...]

Ubuntu >=12.04 Yes Yes (tested) Yes Yes 02 "forge": { Debian 8 Yes Yes Yes Yes 03 "make_targets": { Fedora 21 Yes Yes Yes Yes 04 "win32": [ Windows >=7 Yes Yes (tested) Yes Yes 05 "squirrel" OS X >=10.9 No Yes No Yes 06 ], 07 "darwin": [

08 "zip"

Table 2: Possible Package Formats under forge 09 ],

Type System Description 10 "linux": [ zip All Zip archive 11 "deb", squirrel Windows Installer for Squirrel-Dot-Windows 12 "rpm" appx Windows Windows-Store packet 13 ] 14 }, dmg OS X Darwin DMG packet 15 "electronPackagerConfig": {

deb Linux Debian packet 16 "packageManager": "yarn"

rpm Linux RPM packet 17 }, flatpack Linux Flatpack packet 18 "electronWinstallerConfig": { 19 "name": "app"

20 }, app, you won’t see any notable differ- create a Windows installer with the 21 "electronInstallerDebian": {}, ences with Figure 2. Conversely, you forge make ‑‑platform win32 command. 22 "electronInstallerRedhat": {}, could install electron‑forge under In this case, .NET support was missing: 23 "github_repository": { Windows 10, and, in that case, the 24 "owner": "", Linux app created under Windows will sudo apt‑get install mono wine 25 "name": "" run under Linux. 26 }, You can assign a processor architec- Under out/make/squirell.Windows/x64, 27 "windowsStoreConfig": { ture with the arch command-line switch. you will find the app‑1.0.0 Setup.exe 28 "packageName": "", Table 1 shows the possible combinations installer file, which we were able to in- 29 "name": "app" of values for platform and arch. stall on Windows 10 without any prob- 30 } lems. 31 } Packing Listing 3 displays the configuration 32 [...] forge offers to create platform-specific object [11] from the package.json file as package formats or installers to facili- a portal. Lines 2 to 13 contain the pack- tate easy distribution of the app (Table age formats to be generated in the make_ you want to be on the safe side, simply 2). forge make generates a Debian pack- targets sub-object, listed by operating pack on the target system each time age for the Boilerplate app. However, system. and test the app immediately. nnn the script terminated in our lab while trying to create an RPM package, with Conclusions Info the error message from Figure 3. sudo Electron makes it easy to create desktop [1] Electron: https://electronjs.org fixed the problem. apps using popular web technologies apt‑get install rpm [2] Chromium: https://www.chromium.org Similar scenes occurred while trying to such as HTML, CSS, and JavaScript. At [3] Node.js: https://nodejs.org the same time, Electron offers an [4] Bootstrap: https://getbootstrap.com infrastructure that [5] React: https://reactjs.org can port apps to [6] Angular: https://angular.io other systems via [7] JQuery: https://jquery.com the command line. [8] Yarn: https://yarnpkg.com Compared to Cordova Phone [9] electron‑forge: https://github.com/ Gap [12], the in- electron‑userland/electron‑forge stallation is triv- [10] Boilerplate projects: ial, even if sup- https://github.com/sindresorhus/ port for mobile awesome‑electron# boilerplates operating systems [11] Configuration of electron‑forge: Figure 3: Starting rpmbuild as a separate process such as Android https://github.com/electron‑userland/ using spawn failed at first because rpm was not yet or iOS is cur- electron‑forge#config installed. rently missing. If [12] Phone Gap: https://phonegap.com

32 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM

IN-DEPTH Command Line – BorgBackup

Your files will be assimilated Borg Backup

In Star Trek, the Borg adds individuals to its collective consciousness, an apt metaphor for any backup application that stores individual files in an archive. BorgBackup creates folder repositories for multiple archives, making it an especially befitting description of assimilation. By Bruce Byfield

orgBackup [1] was originally a making them more visible and easier The Basics fork of Attic [2], begun in 2015 to manipulate. For example, because Like an increasing number of modern by developers who wanted an incremental backup is in a separate command-line applications, BorgBack- faster-paced development. Attic file from the original, it can be re- up’s command structure consists of isB no longer in development, although trieved by itself. In many ways, Borg- three parts: the basic command; a sub- stable versions remain available in Backup is a modernization of long- command, such as create, extract, or Debian and several other distributions. time Linux archivers such as tar, gzip, delete; and the options and paths to a Meanwhile, BorgBackup has lived up to and bzip2, offering increased efficiency file. Many options can be used with its mandate, not only enhancing the and speed with its choice of compres- more than one of the sub-commands Attic code, but also gaining a reputation sion formats and its own set of op- and can be placed either after the sub- for its innovative approach. Although tions. In addition, BorgBackup offers command or after the paths, although to other compression and backup utilities fine-tuned control of features, such as avoid any confusion, users should prob- have encryption, incremental, and remote the compression level and the details ably use one position consistently. options, what distinguishes BorgBackup stored in its logs, while using intelli- Before archiving, BorgBackup requires is how it implements such features, gent defaults that allow users to the creation of a repository in which to quickly get up and running. store backups. This repository can be pro- Author BorgBackup’s documentation [3] does tected by a passphrase, as well as en- Bruce Byfield is a computer journalist and warn that the application requires a rea- crypted. The arrangement has the advan- a freelance writer and editor specializing sonable amount of free space to operate. tage of reducing the number of pass- in free and open source software. In The exact amount, of course, depends phrases to remember, even though, techni- addition to his writing projects, he also on the size of the files it uses, but at cally, individual or additional passphrases teaches live and e-learning courses. In his least 2GB should be available. However, would be more secure in some cases. spare time, Bruce writes about Northwest to ensure trouble-free backups, reposito- To create a repository, enter: coast art. You can read more of his work at ries should be created on filesystems

http://brucebyfield.wordpress.com with plenty of free space. borg init PATH 123RF.com Kiselev, Lead Image © Andrey

34 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM IN-DEPTH Command Line – BorgBackup

Figure 1: To back up files, you must first create a repository. Viewing the new repository’s passphrase can help you remember it or encourage you to reconsider a weak password.

Figure 2: The list sub-command displays the contents of an archive.

This command automatically creates a In most cases, too, you may want to with HMAC-SHA256. The passphrase is repository, prompting for a passphrase include the ‑‑encryption=repokey option, stored in UTF-8 and should avoid any (Figure 1). BorgBackup does not prevent which stores the passphrase in the re- characters that have a specific meaning a weak passphrase. However, if you pository, or else the ‑‑encryption=keyfile at the command line, such as a forward choose the option to display the pass- option, which stores the passphrase in slash (/) or a regular expression like an word before continuing, it will give you ~/.config.borg/keys. The keyfile option asterisk (*). the chance to reconsider your choice. is most secure if your home directory is After a repository is created, you The path can be relative or absolute, as on a separate device from the repository. can add files to it with the create sub- well as local or remote and connected Later, the sub-command change‑pass‑ command: by SSH (ssh://PATH:2022/; USER@ phrase can be used to change the pass- HOST:PATHcs). If necessary, a remote or phrase, although if the repository has borg create REPO::ARCHIVE SOURCES external path can be activated using the been compromised, your data is more mount sub-command, most of whose op- secure if you create an entirely new one. Multiple sources can be specified in a tions are contained in the Filesystem in In either case, the passphrase uses space-separated list. You can create an Userspace (FUSE) man pages. 256-bit AES encryption and is verified incremental backup by specifying sources already archived, which will Table 1: Selected BorgBackup Sub-Commands substantially reduce the amount of time Sub-Command Description required to update an archive. Once a re- list Displays archives in a repository or an archive (Figure 2). pository and archives are created, you delete Removes an archive. can use other sub-commands for basic functions (see Table 1). extract: Restore archives. prune Sets a pattern for regularly removing archives and their con- Advanced Options for tents by specifying by prefix or setting what to keep with the options ‑‑keep‑daily, ‑‑keep‑weekly, and ‑keep‑monthly. Creating Archives Works well with automated operation (see the “Automating When using the create sub-command, the Command” section). one of the most important settings is rename Gives an existing archive a new name. compression. By default, BorgBackup diff Compares two archives. uses the LZ4 format, a fast choice, but one with a relatively low compression key change‑passphrase Changes a repository’s passphrase. rate. If you have the disk space, you may key export Creates a file to store the passkey. prefer to specify ‑‑compression none (‑C). key import Retrieves an exported passkey. Better control over compression can be export‑tar Creates a tar version of the archive. had by specifying another format, each config Edits the name and key value for the specified repository. of which has its own scale for the level

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 35 IN-DEPTH Command Line – BorgBackup

of compression: zstd (1-22), zlib (0-9), and lzma (0-9). In all of these scales, the lowest number gives the least compression but runs the quickest, and, if a compres- Figure 3: The check sub-command tests that an archive is usable. sion level is not specified, BorgBackup uses a medium the default 0077 permissions can be known unencrypted repository.” As you level. The format and compression level changed with: might expect, none of the available envi- can be set using the option: ronment variables will function if you ‑‑umask OCTAL‑PERMISSIONS use the sudo command to operate as an- ‑‑compression FORMAT LEVEL other user, since the change of user is The atime attribute, which records the generally accompanied by a change in The preferable format and compression last time a file was accessed, can be environment. level is a matter for expert debate, but omitted with ‑‑noatime; the ctime attri- You can also use Bash scripts to auto- keep an eye on CPU activity and free bute, which records the last time the mate backups and then use a crontab to disk space while running each format to file was changed, with ‑‑noctime; and run the script. BorgBackup’s Quick Start help decide which one to use in a partic- the user and group identifiers can be provides an example of such a script that ular circumstance. restricted to numeric identifiers with can be modified to suit your system [5]. Unless the ‑‑verbose (‑v) and ‑‑numeric‑owner. ‑‑show‑rc options are used, Borg- All these options can make the initially Going on from Here Backup gives minimal output to avoid simple borg command considerably more As you can tell by the number of sub- clutter. The output it gives is enough in complex, so, having set up a command, commands, there is much more to routine circumstances and often in- you might want to run: BorgBackup than I have presented cludes helpful suggestions. However, here. I have given an overview of basic at times you may need to look at the ‑‑dry‑run (‑n) considerations, plus enough additional logfile for more detailed information. material to use BorgBackup with some No logfile is written by default, but to test your carefully crafted command efficiency. If you want to learn more, I you can specify one as an environment before actually running it. Upon creating suggest not using the rather verbose setting (see “Automating the Com- an archive, you should also run the check man page, but going instead to the on- mand”), which may be advisable on a sub-command (Figure 3) to ensure that line documentation, which is divided network or when dealing with any im- the archive is reliable. After all, the very into a Quick Start and more advanced portant files. You can set what is worst time to find that an archive is un- instructions. logged by creating an archive with one usable is when you need to extract it. BorgBackup is just different enough of four options: from other archive tools that it takes a • ‑‑debug: Logs everything in detail. Automating the Command while to learn. However, once learned, • ‑‑warning: Logs warning, error, and Another way to make using a complex its structure is quick and efficient. Take critical-level messages. command easier is to add the most com- the time to understand it, and you might • ‑‑error: Logs error and critical-level monly used options as environment vari- just decide that BorgBackup is your messages. ables using the export command [4]. backup utility of choice – even if you are • ‑‑critical: Logs only critical-level The borg command will use the variables accustomed to a GUI. nnn messages defined as part of the environment un- While it might seem wisest to set the log less alternatives are specified. For exam- Info information to ‑‑debug, too much infor- ple, BORG_REPO=PATH sets the location of [1] BorgBackup: mation might make the log more difficult the default repository, while BORG_ https://www.borgbackup.org to read, so you should experiment to dis- PASSPHRASE=PASSPHRASE gives the pass- [2] Attic: https://en.wikipedia.org/wiki/ cover what level of information you ac- phrase for the default repository. Other Attic_(backup_software)#Borg tually need. possible environment variables store re- [3] Other options are also available. The sponses to warnings, such as BorgBackup documentation: https:// borgbackup. readthedocs.io/en/stable option ‑‑exclude PATH can be used to exclude items from an archive or else the BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESSU [4] BorgBackup environment variables: backup can be limited to one filesystem _IS_OK=yes https://www.mankier.com/1/borg with: [5] Backup script: https://borgbackup. which responds to the warning, “At- readthedocs.io/en/stable/quickstart. ‑‑one‑file‑system (‑x) tempting to access a previously un- html#automating‑backups

36 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM

IN-DEPTH Programming Snapshot – Stock Quote API

Determining assets across a shrinking number of financial APIs Bargain Hunter Financial wizard Mike Schilli is annoyed that some web services have discontinued serving up real-time stock market data. To keep an eye on his investment dollars, Mike taps into a little known interface for stock prices. By Mike Schilli

or users of stock-quote-serving real-time stock market data (Figure 1); hands of those who hope to find a re- APIs, the fruit is currently hang- then, Yahoo followed suit and pulled placement for the vanished free real-time ing higher. First, Google discon- the plug. With the end of their undocu- (or at least time-delayed) stock market tinued its web service [1] for mented but widely used share price in- price data source. F terface in CSV format [2], Yahoo, perhaps unknowingly, also pulled the Free Is Cheap Enough rug from under a number of open For a hefty monthly fee, of course, you source projects, such as the Perl can find dozens of business-level stock Yahoo::Finance CPAN module market data providers, such as Bloom- and the Python yahoo-finance berg, but only a few competitors offer package. A few open source free quotes, often only to registered portfolio-tracking applications users who are adding an issued API using these popular packages key to every request. It is probably were dragged into the abyss only a matter of time before the free along with them. source presented today also dries up, On the GitHub sites of these but I’m working on the assumption projects, the Issues tabs are now that, in the future, options for retriev- overflowing with coming the data free of charge will still be ments, there’s around. quite a bit of Developers should therefore be pre- wailing and pared for the fact that an application’s gnashing of data back end can change at short no- teeth from tice. Thanks to a design abstracting away users, and the share price source, this should be much easy to handle with a simple upgrade of wring- the quote library for applications; the ing of caravan of thirsty camels, after only find-

Author Mike Schilli works as a software engineer in the San Francisco Bay area, California. Each month in his column, which has been running since 1997, he researches practical applications of various programming languages. If you go to mschilli@perlmeister.com he will gladly answer any questions.

38 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM IN-DEPTH Programming Snapshot – Stock Quote API

of the share items via the API and multiplies the stock price by the number of shares in the deposit. To perform this calculation, Listing 4 (line 3) includes the yaml module, which you must have installed previously by typing:

pip install ‑‑user pyyaml

It provides the load() method used in line 8 to parse the account data from the portfolio file in Listing 2. Line 30 iterates across the accounts found; line 32 iterates across the items stored therein. In the current version, the script supports cash (cash) and shares Figure 1: Google’s financial site for portfolio tracking has seen better days. (stock) but could easily be extended to include bonds, options, or items in for- ing dried out water holes at the old The advantage of a home-grown script eign currencies, whose exchange rate oasis, will move on to more refreshing is that it adds all items across accounts the script then can retrieve in a similar destinations. or brokerages. For this purpose, the user fashion at run time. The cash_dump() Listing 1 [3] uses a still open Yahoo has to keep a YAML file with the items in function from line 18 determines the source to obtain the current market their portfolio up to date manually, as value of a cash item. There’s no magic prices of equities on the world’s major shown in Listing 2. However, this ap- being performed here; it only returns a stock exchanges. For example, the amzn proach preserves the investors’ privacy tuple with the name Cash and the ticker symbol in dollars reveals the daily more reliably than third-party applica- amount. value of the Amazon share on the New tions that have a monetary interest in York Stock Exchange. Those who want mining user data and using it for Listing 2: portfolio.yaml to explore Amazon’s share price on the marketing purposes or, worse, selling 01 ‑ account: Wells Fargo German Xetra Stock Exchange in euros it to the highest bidder. 02 holdings: enter amz.de instead. Listing 2 shows two fictitious ac- 03 ‑ type: stock To do this, the quote() function counts, one at Wells Fargo and one at awaits the ticker symbol and sends a Citibank, with 100 Amazon and 50 04 ticker: amzn request to the web API that returns a Google shares, the latter of which are 05 shares: 100 data structure encoded in JSON. Under now traded under the “Alphabet” name, 06 ‑ type: cash the regularMarketPrice key is the nu- although the company’s ticker symbol 07 amount: 5170.99 merical value for the price in the cur- on Nasdaq is still goog. Both accounts 08 rency customary on the market where also have a Cash item. The mymoney.py 09 ‑ account: Chase Bank it is traded. script in Listing 3 adds the cash at face value but determines the current value 10 holdings: Cash Check in 11 ‑ type: stock the Money Listing 1: myquote.py 12 ticker: goog

Bin 01 #!/usr/bin/env python 13 shares: 50 Banks and stock bro- 02 import requests 14 ‑ type: cash kerage houses offer 15 amount: 1234.56 portfolio management 03 import json functions and calcu- 04 late the net value of 05 def quote(symbol): Listing 3: mymoney.py Calculations the deposit for their 06 r = requests.get( $ ./mymoney.py clients. Google and 07 'https://query1.finance.yahoo.com'+ Wells Fargo Yahoo also offer port- 08 '/v7/finance/quote?symbols=%s' 100 x amzn: 148292.00 folios, although they 09 % symbol) Cash: 5170.99 are in the process of Chase Bank change [2]; both seem 10 data = json.loads(r.text) 50 x goog: 55567.00 to be looking for new 11 return (data['quoteResponse'] Cash: 1234.56 ways to make money 12 ['result'][0] off the service some- 13 ['regularMarketPrice']) how. Total: 210264.54

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 39 IN-DEPTH Programming Snapshot – Stock Quote API

Listing 4: mymoney.py

01 #!/usr/bin/env python 21 def pos_dump(pos):

02 import requests 22 if pos['type'] == 'stock':

03 import yaml 23 return stock_dump(pos['ticker'],

04 import myquote 24 pos['shares'])

05 25 elif pos['type'] == 'cash':

06 def pf_read(filename): 26 return cash_dump(pos['amount'])

07 with open(filename, 'r') as stream: 27

08 return yaml.load(stream) 28 total=0

09 29

10 def amount_fmt(amount): 30 for account in pf_read("portfolio.yaml"):

11 return "%10.2f" % amount 31 print(account['account'])

12 32 for pos in account['holdings']:

13 def stock_dump(ticker, shares): 33 (name, amount) = pos_dump(pos)

14 price=myquote.quote(ticker) 34 total += int(amount*100)

15 return("%d x %s" % (shares, ticker), 35 print("%15s: %s" %

16 price*shares) 36 (name, amount_fmt(amount)))

17 37

18 def cash_dump(amount): 38 print("\n%15s: %s" %

19 return "Cash", amount 39 ("Total", amount_fmt(total/100.0)))

To determine the value of a stock item, of the last two digits – Listing 4 does wrong, of course. Try it; it could be lu- on the other hand, Listing 4 uses the just that. crative, if it works! nnn myquote.py module from Listing 1 and calls its quote() function with the ticker Predicting the Future Info symbol as a parameter. It multiplies the With the relatively easily available [1] “Google Finance as you know it is returned market price by the number of daily price data of known stocks, you going away”: https://www.marketwatch. shares in the securities account and could fall into the trap of feeding an com/ story/ google‑finance‑as‑you‑know‑ feeds this back to the main program to- artificial intelligence network histori- it‑is‑going‑away‑here‑is‑what‑will‑ remain‑2017‑09‑27 gether with a table key, all wrapped into cal stock price data and trying to pre- a tuple. The main program only needs to dict future price jumps. Then, armed [2] “Has Yahoo finance web service disap- peared?”: https://stackoverflow.com/ add the line items across all accounts with this knowledge, you could make questions/38355075 and output the total net value for the a massive investment and sit back and user (Listing 3). wait for payday. Ha! However, the [3] Listings for this article: ftp://ftp.linux‑magazine.com/pub/ timeless arguments from the bestseller listings/linux‑magazine.com/210/ Rounding, but Correctly A Random Walk down Wall Street [5], [4] “Watchful Seller” by Michael Schilli, One thing to keep in mind is that com- which are no longer entirely new but Linux Magazine, issue 188, July 2016, puters are notoriously bad at adding all still valid, speak against this, stat- http://www.linux‑magazine.com/Issues/ floating-point numbers. If you add 1.23 ing that, in this case, the millions of 2016/188/Perl‑Monitoring‑Payments and 4.56 together, you may get a result market participants with similar ideas [5] Malkiel, Burton G. A Random Walk that deviates from the expected value of would quickly close any wormholes in down Wall Street: The Time-Tested 5.79, because the CPU does not repre- the time structure, if they actually ex- Strategy for Successful Investing. W. sent 1.23 exactly as 1 and 23 hun- isted, because they would have W. Norton & Company, 2016: dredths, but approximates it in binary known about possible price jumps at https://www.amazon.com/Random‑ format. In a past article, I discussed how the same time and would have reaped Walk‑down‑Wall‑Street/dp/0393352242 even eBay can’t add properly and that the profits way before the crusading [6] Random walk theory: their monthly bill regularly is off by a private investor. https://en.wikipedia.org/wiki/Random_ few cents [4]. There’s no easy money to be made walk When it comes to adding money at the exchanges, I’m afraid, or is amounts, it is smarter to compute in there? Are the exchanges just glorified cents rather than dollars. In other casinos and follow the “random walk” Find us on words, simply multiply the dollar theory [6], not because their develop- amount by 100 and add it as a ment is actually random, but because Facebook rounded integer; when displaying the prices are unpredictable? You’re free to http://www.facebook.com/linuxpromagazine result, inject a decimal point in front prove the random walker disciples

40 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM

IN-DEPTH Ghost CMS

Blogging with a state-of-the-art WordPress alternative Friendly Ghost Blogging usually implies WordPress, but Ghost presents a serious challenge. By Christoph Langner

ccording to an analysis by Ghostly now targeting. The project was given a W3Techs [1], neither Google nor In contrast to WordPress, the Ghost significant boost by a successful Kick- Amazon control the Internet – CMS does not rely on PHP, but on Ja- starter campaign. In just under a month, the WordPress content manage- vaScript and Node.js, which makes it a around £200,000 came together, which Ament system (CMS) does. They say the bit difficult to get started. However, the was far more than the £25,000 funding open source CMS now drives 30 percent modern web platform plays to its target [4]. of all websites, which translates to a strengths. The idea for Ghost came from The Ghost Foundation emerged from market share of just over 60 percent. Hannah Wolfe and John O’Nolan, who the accumulated capital and developed Thus, you are unlikely to go a day with- gained great experience in CMS as the and managed the code under the MIT li- out stumbling across a WordPress page former deputy head of design of the cense as a non-profit organization. The while browsing. WordPress team. O’Nolan no longer foundation currently employs eight peo- Even though most web developers are sees WordPress as the simple tool for ple, and the steadily growing community looking for one of the usual candidates bloggers that it once was, but as a plat- contributes time and code to the project. from the CMS trio of WordPress, form for all kinds of web projects [3]. As with WordPress, the Ghost CMS is Joomla, and Drupal, you should take a With a handful of plugins, you can set available for self-hosting, or Ghost(Pro) look at the CMS inhabiting the bottom up virtually any project on the web, [5] is offered as a hosted version for a of the W3Techs list. The still quite new from a bug tracker to a web shop to an fee starting at $19 a month. Ghost [2] has only secured a small online community. share of the market, but its modern ap- Its great flexibility is also the biggest Drafted proach and its active community make weakness of WordPress: The CMS has Ghost recommends Ubuntu 16.04 with the CMS interesting for open source- lost its focus on the original target group, 1GB of RAM, MySQL as the database minded bloggers. bloggers, which are exactly who Ghost is back end, and Nginx as the web server. Lead Image, toa-heftiba, Unsplash.com

42 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM IN-DEPTH Ghost CMS

Listing 1: Activating Web Server Nodes line assigns the user rights to the

$ sudo apt update Ghost developers recommend using the folder. Next, you switch to the newly 6.x LTS branch of Node.js. Ubuntu 16.04 created directory, which must be $ sudo apt full‑upgrade and newer is the easiest way to load it empty, and run the installation wiz- $ sudo apt install nginx mysql‑server via nodesource.com. The installation ard. The LANG=C changes the language $ sudo ufw allow 'Nginx Full' script inserts a package source into the settings to the default locale; other- system, from which it then installs the wise, the script runs into an error and As a JavaScript application based on application. With the node package terminates. Node. js [6], Ghost provides its own manager (npm), you can install the The installation routine first wants to web server. Node.js listens for requests ghost‑cli setup wizard: know the blog URL, the address of the on port 1024 or greater so that the ser- MySQL server, the root login, and the vice does not have to run with root $ wget U previously assigned MySQL password, privileges, and Nginx accepts requests https://deb.nodesource.com/setup_6.x as well as a name for the database. You on ports 80 and 443 (for SSL) as a re- $ chmod +x setup_6.x can enter the URL to access the blog verse proxy before passing them on to $ sudo ./setup_6.x later. If the blog and database are on the Ghost. In a test on Ubuntu Server $ sudo apt install nodejs same computer, the address for the 17.10, Ghost proved in our lab that it $ sudo npm i ‑g ghost‑cli MySQL server is localhost. The database can handle a state-of-the-art software does not have to exist: The setup creates stack. At first glance, this procedure might it, if necessary. As a prerequisite, we assume that the seem complicated, but it saves much The Ghost command-line interface server can be reached over an Internet time and effort in the long run. (CLI) handles most of the maintenance domain. In the case of a home server At this point, you have created the work and the installation. You can in- connected to the Internet via DSL or necessary prerequisites to put the blog stall the current version with ghost up‑ cable, you must ensure that the URL is system on the server: date. Alternatively, take a look at what’s accessible via DynDNS and forward going on behind the scenes with ghost ports 80 and 443 from the router to the $ sudo mkdir ‑p /var/www/ghost log. For information on all options of server. Refer to the Installation Guide $ sudo chown : U the Ghost client for the command line, [7] to update the server and install /var/www/ghost enter ghost help. Nginx and MySQL. The system requests $ cd /var/www/ghost a root password for the SQL server $ LANG=C ghost install Go to Blogging (which you must remember); then, you The script takes over configuration of the need to activate the web server in the The first line creates the /var/www/ Nginx web server. If necessary, you can firewall (Listing 1). ghost/ working directory; the second encrypt the connection using a Let’s En- crypt SSL certificate. In this case, in addition to port 80, port 443 must be forwarded to the home server. Finally, let systemd control Ghost and then start the service. If everything works fine, you can reach the blog in your browser under http://; if SSL is active, the system automatically redirects to https:// (Figure 1). You can load the front end for administration in http:///ghost. The first time you visit the site, create a user and give the blog a name; then, open Settings | General and set the local time under Timezone. The preinstalled content demonstrates the features of the blog. If necessary, delete all content with Labs | Delete all Content. The system sends messages (e.g., when resetting passwords) via Nodemailer [8]; you can check the function with Labs | Send a test email. Functions Figure 1: The Ghost CMS system is based on JavaScript and Node.js and When composing an article, Ghost delib- optimized for bloggers. erately avoids using a WYSIWYG editor

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 43 Figure 2: Ghost offers many of the standard CMS features.

and instead uses a simple Markdown [9] be expandable with plugins (known as editor. The syntax is easy to learn. For Ghost Apps) [14]. example, asterisks format *italic* and *bold* text. Lists, links, and code blocks Conclusions can be created just as easily. Buttons on All in all, Ghost offers many features found the lower edge and a side-by-side pre- in the CMS mainstream (Figure 2). How- view help with the first steps. ever, the newcomer still lacks a kind of Further support for writing articles can dashboard that provides information about be obtained in the desktop client [10] page access and page views, as well as and the smartphone app for Android other essential blog features such as a way [11]. If you don’t like the look and feel of to leave comments. However, these func- the blog, Ghost Marketplace [12] offers tions can be retrofitted, for example, with free and commercial themes. Pages like Google Analytics and Disqus or, if you pre- Ghost for Beginners [13] list more skins. fer open source tools, with Matomo (for- In the future, the blog system will also merly Piwik) [15] and Isso [16]. nnn

Info [1] CMSs for websites: [8] Nodemailer: https://github.com/ https://w3techs.com/technologies/ nodemailer/ nodemailer/tree/0.7 overview/ content_management/all [9] Ghost Markdown: https://help.ghost. [2] Ghost: https://ghost.org/ org/ article/4‑markdown‑guide [3] Project Ghost: [10] Desktop client: https://john.onolan.org/project‑ghost https://ghost.org/downloads/ [4] Ghost Kickstarter: [11] Android client: https://www.kickstarter.com/projects/ https://play.google.com/store/apps/ johnonolan/ghost‑just‑a‑blogging‑ details?id=org.ghost.android platform?ref=nav_search& [12] Marketplace: result=project&term=ghost%20blog https://marketplace.ghost.org [5] Ghost(Pro): https://ghost.org/pricing/ [13] Ghost for Beginners: https://www. ghostforbeginners.com/themes [6] Node.js: https://nodejs.org [14] Apps: https://apps.ghost.org [7] Installation instructions: https://docs.ghost.org/v1.0.0/docs/ [15] Matomo/Piwik: https://matomo.org install [16] Isso: https://posativ.org/isso

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM IN-DEPTH Charly’s Column – sshuttle

The sys admin’s daily grind: sshuttle Great Shuttle Service When he doesn’t want to deal with OpenVPN version conflicts or congestion control problems during TCP tunneling, Charly catches a ride on sshuttle. By Charly Kühnast

n untrustworthy networks, I let You can leave out the port number if it is vices, sshuttle works at session level. It OpenVPN tunnel my laptop. There the SSH standard port 22. The 0/0 means assembles the TCP stream locally, multi- are certainly alternatives, and I that Linux should direct all connections plexes it over the SSH connection, while would like to present a particu- into the tunnel. However, this means keeping the status, and splits it into Ilarly simple one: sshuttle [1]. As the that I cannot reach other devices in the packets again on the destination side. name suggests, the tool relies on SSH. local network. To keep the local LAN This avoids the TCP-over-TCP prob- The tunnel’s endpoint is a leased root still visible, I define it as an exception lem which plagues other tools such as server, just like with OpenVPN. Sshut- using the ‑x parameter: OpenVPN: TCP has an overload control tle is very frugal. It only needs SSH ac- (congestion control). The protocol de- cess with user privileges on the server; sudo sshuttle U fines a performance limit on the basis root privileges are not necessary. Addi- ‑r ‑‑dns @ 0/0 U of dropped packets. If you tunnel TCP tionally, Python must be installed on ‑x 192.168.2.0/24 over TCP, you lose congestion control the server – that’s it. for the inner connection, which can This is because sshuttle loads and ‑‑dns is included here. This means that lead to bizarre error patterns. Sshuttle executes the required Python code on DNS queries also run through the tunnel, is immune to the problem. the server after the SSH connection is which does not happen automatically. Verbose parameters can help if you established. It also avoids version con- This is sshuttle’s Achilles heel: It only do need to troubleshoot. Figure 1 flicts between server and client soft- transports TCP; ICMP and UDP do not shows a connection setup with ‑v. ware. The following command is all it pass through the tunnel, apart from DNS. With the verbose option, sshuttle is takes to set up the tunnel: very long-winded, so I recommend re- Congestion Alert directing the output to a file that can sudo sshuttle U Whereas other VPN technologies work be evaluated in peace. My conclusions: ‑r @: 0/0 at packet level and rely on TUN/TAP de- Sshuttle is an excellent and simple VPN for people who can do without UDP and ICMP. nnn

Info [1] sshuttle: https://github.com/apenwarr/sshuttle

Figure 1: Sshuttle builds a VPN for a server. Because of ‑v, the messages are more extensive than without.

Author Charly Kühnast manages Unix systems in the data center in the Lower Rhine region of Germany. His responsibilities include ensuring the security and availability of firewalls and the DMZ.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 45 IN-DEPTH KNIME

Workflow-based data analysis with KNIME Analyze This!

They say data is “the new oil,” but all that data you collect is only valuable if it leads to new insights. An open source analysis tool called KNIME lets you analyze data through graphical workflows – without the need for programming or complex spreadsheet manipulation. By Alexander Fillbrunn and Martin Horn

ata analysts like to use flexible the KNIME environment, a workflow is a terials, including blogs, videos, and sam- scripting languages such as R or graph with nodes showing a series of se- ple workflows. Python that come with large quential steps for processing and analyz- The KNIME Analytics Platform is open ecosystems of libraries and ex- ing the data. The user defines a pathway source; anyone can download and use it Dtensions. But many users don’t want to for the data by connecting the output of free of charge. Start the installer or un- have to write and debug their own cus- one node to the input of another node. A pack the archive, and KNIME is ready to tom programs just to analyze data. type system ensures that you can only go. However, you might wish to add some Visual workflows offer a different ap- connect compatible output and input. KNIME extensions, which are the source proach. You can use visual workflows to Real programming code is only necessary for many additional nodes. To add an ex- break down the analysis processes into if you want to integrate KNIME with R or tension, go to File | Install KNIME Exten- modular, sequential steps. Each step is Python – or if you want to develop your sions, check the list for the desired exten- symbolized by a graphic element called own modules. sion, and follow the instructions. a node. Each node performs an action, KNIME, which is pronounced “nime,” which might be a calculation, a format- was originally known as the Konstanz Looking Around ting function, or another step related to Information Miner; it began in 2006 at Figure 1 shows the KNIME user inter- data analysis and manipulation. By link- the Department of Bioinformatics and face. The KNIME Explorer in the top-left ing the nodes on the screen, users can Data Mining at the University of Kon- corner of the workspace provides an create workflows for complex investiga- stanz under the direction of Prof. Mi- overview of the workflows. Click the tions of the data – without producing chael Berthold. The basic idea was to View menu and select Workflow Coach any code. make data analysis easily accessible to access the Workflow Coach, which of- Visual workflows are the central ele- and affordable for users from different fers suggestions on building a workflow. ment of the KNIME Analytics Platform. In disciplines. The developers, therefore, The Node Repository (bottom left) lists turned their creation into an open the nodes of all installed extensions. The Authors source project, paying particular atten- The nodes are the building blocks of Alexander Fillbrunn is a PhD student at the tion to making the tool extensible and any KNIME workflow, and the vast li- Department of Bioinformatics and Informa- user-friendly. brary of available nodes gives KNIME its tion Mining at the University of Konstanz. KNIME Analytics Platform is written in versatility and power. KNIME nodes per- He is particularly interested in the develop- Java and is based on Eclipse and Open form tasks such as: ment of machine learning algorithms. Services Gateway Initiative (OSGI) tech- • data access Martin Horn is a postdoc at the Department nology. The latest version (version 3.5.1 • data manipulation of Bioinformatics and Information Mining at at the time this article was written) is • visualization the University of Konstanz, where he stud- available at the project website [1], where • analytics

ies data analysis – of course using KNIME. you will also find other introductory ma- • reporting Lead Image Photo by mari lezhava on Unsplash

46 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM IN-DEPTH KNIME

Table 1: Evaluation from a Sqlite Database Reader ID Article ID Evaluation Reader 1 Article 11 1 Reader 93 Article 31 3 Reader 45 Article 3 4 …

some data from a rating system that gives readers the opportunity to rate magazine articles with up to five stars. Each article is also classified in at least one of the five categories: Hardware, Software, Develop- ment, Security, and Internet. The editors hope to draw conclusions from the ratings on reader preferences and to identify groups of readers with common interests. They then want to suggest further articles to each reader that match their interests. Figure 1: The KNIME user interface, in which users compose workflows The starting points for the analysis are using drag & drop. a SQLite database with the article ratings (Table 1) and a CSV file with the classifi- cate what kind of data the cation of the categories for the articles. node needs or outputs at a par- Both data sources are therefore available ticular input or output (a black in a table format. triangle indicates a table). Loading the Data Sample Scenario The first step of any data analysis is load- The best way to get to know ing the required data (the red nodes in Fig- KNIME is to work through an ure 3). For KNIME, the data can come example workflow (Figure 3). from a variety of sources, such as text files, The editors of a fictitious on- documents, databases, or web services. Figure 2: Anatomy of a KNIME node. line magazine would like to Once loaded in KNIME, it doesn’t mat- know more about their read- ter where the data came from, because • flow control ers’ preferences. They have gathered the software always converts the data • scripting • big data The node description in the window on the right of the user interface gives the user the necessary documentation on the function and use of a node. In the middle is the workflow editor, where users string the nodes together to develop the actual workflow. During the course of developing a workflow, the user connects, config- ures, and executes the KNIME nodes individually and collectively (Figure 2). Once a node has been executed, which is indicated by a green traffic light symbol below the node, you can display the resulting data as a table, bar chart, or other format. If the traffic light symbol is red, the node is not yet configured. Yellow means the node is ready for execution. Nodes can have between zero and an arbitrary number of inputs and outputs Figure 3: The sample workflow, which analyzes reader preferences for (ports). The ports' shape and color indi- a publisher.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 47 IN-DEPTH KNIME

into an internal format. In this case, I reader’s preferences by calculating the into a workable format, you can begin to need to load data from an SQLite data- reader’s average rating for the articles in group readers according to their prefer- base and a CSV file. To help you access each section. ences. Readers with similar preferences databases, KNIME has several nodes to The result is a table in which each should end up in the same group. Clus- assist with data input, including the Data‑ reader becomes a row and each category ter analysis is a useful technique for this base Reader node, which delivers the re- becomes a column. The cells of the in- kind of problem, and KNIME has differ- sults of an SQL query as a KNIME table. tersection points display the reader’s av- ent nodes to support it, including k- The node works with different data- erage rating for articles in each category means, fuzzy c-means, hierarchical ag- base types and receives the connection (Figure 5). glomerative clustering, or k-medoids. information via its inbox. You can prefix This transformation is handled by the What all these cluster algorithms have an SQLite connector, which is configured pivot node. The pivot node dialog may in common is that users have to decide for with the connection information and for- prove intimidating at first glance, since it themselves how many groups they want to wards it to other nodes. requires information on the group, pivot, receive. They also all need a way to calcu- The easiest way for KNIME to load and aggregation columns in three sepa- late the similarity between two datapoints. text files is to use the File Reader. This rate tabs. The values in the group col- In the previous category, a table row was node automatically attempts to guess the umns determine which rows need to be created for each reader, containing five nu- file format, including column separators combined into a single row – that is, all meric values – one for each category. and row length, and displays a preview those that have the same values in these The workflow for the example uses the of the table in its configuration dialog. columns. For the sample workflow, the k-means node, which is quite fast and reader ID is a group column, the cate- generally produces good results. The al- Opening a Connection gory is the pivot column, and the aggre- gorithm works with different distance The next step is to relate the imported gation column contains the ratings that measurements – in KNIME it is imple- data to each other. If you are familiar an average function averages. mented with the Euclidean distance (see with databases, you know that the best If a reader has not rated a single article the “Computing the Distance” box). The way to do this is with a join. A join con- for a category, the cell in the pivotized k-means node in KNIME, which executes sists of one or more predicates that de- table does not contain a value. KNIME in- the algorithm, appends another column termine which rows belong together in dicates this with a red question mark. You to the input table that remembers the two tables. The rows that belong to- can replace missing values using the group in which the datapoint lies. gether are then combined in an output Missing Value node. In the example, it Cluster analysis reveals that readers table. makes sense to assume that the reader who give a high rating to articles in the This example needs to combine the does not like a category in which they categories Hardware and Security usu- two tables based on the article ID. In this have never rated an article. Instead of the ally lie in cluster 0, while readers from case, the join predicate determines that missing value, the table will use a zero. cluster 1 prefer the categories Internet ratings for an article must be linked to It is often the case that a node pro- and Software, and cluster 2 prefers De- the categories to which the same article duces a table with long or incompre- velopment and Software. Once you is assigned. The node that KNIME uses hensible column names. For example, know the results, it is possible to give for this is Joiner. Its output table is the pivot node from the previous cate- the clusters meaningful names. Due to shown in Figure 4. Like its database gory creates columns in the format their preferences, for example, readers in counterpart, Joiner supports various “Category + Rating.” The suffix, which Cluster 0 can be addressed as adminis- modes, including inner, outer, left, and is the same in each column, is not im- trators, those in Cluster 1 as power right joins. The user can also select portant. Two important KNIME nodes users, and those in Cluster 2 as develop- which columns are to be copied from assist with renaming columns. Column ers. These new names are added using which table into the output. Rename lets you rename each column the Cell Replacer node. name explicitly; Column Rename (Regex) Pivoting lets you change the column names Visualization: Histogram The next step is to create a kind of fin- using regular expressions. A histogram, provided by the histogram gerprint for each reader, describing the node, provides an overview of how many Cluster readers lie in which group. The histogram Analysis node can be connected directly to the Cell Now that the data Replacer from the previous step, but the is transformed whole thing looks a bit colorless. The Color

Figure 4: Output table for the Joiner node. Figure 5: Pivoting results.

48 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM IN-DEPTH KNIME

Computing the Distance which is where you will also find the Par‑ results. All this helps to gain interesting in- allel Coordinates visualization. Parallel sights into the raw data. In a further step, Let each reader be a five-component vector consisting of the ratings in the Coordinates represents the properties of it is now possible to identify for each five categories. The following formula the data with parallel y-axes (Figure 7). reader the articles that a reader has not yet then computes the distance between the D3.js, the JavaScript library on which read, but which could also be interesting two readers x and y: most KNIME JavaScript visualizations for the reader because of their interests. A are based, is one of the most widely web application can then suggest these ar- used libraries for creating interactive ticles to the reader. The k-means algorithm initiates the fol- data visualizations in the browser. How- lowing steps to find homogeneous ever, the KNIME user can only use some Keeping Track groups: of its possibilities. With each further step, the workflow 1. Initialize the cluster centers with ran- For all cases that KNIME does not yet threatens to become more complex and dom datapoints. cover, use the Generic Javascript View. confusing. To ensure that it remains Configuring this node means you can comprehensible, it is a good idea to en- 2. Compute the nearest cluster center for each datapoint. enter arbitrary JavaScript and CSS code capsulate individual parts in modules to to compute a colorful image from a conceal the complexity. This encapsula- 3. Compute the center for each group of datapoints assigned to a center in table. The code executed by the node tion is made possible using so-called Step 2. has access to the node table and the meta nodes. Meta nodes also let you browser Document Object Model (DOM) group the sections of the workflow using 4. Update the cluster centers with the newly found centers. and can generate HTML and SVG ele- meaningful names. Figure 9 shows a ments based on the data. possible restructuring of the workflow 5. If the centers remain unchanged, stop For example, you can use the Generic using meta nodes. the computations; otherwise go to step 2. Javascript View to create a Voronoi diagram (Figure 8), which visualizes the Recommended Reading clustered reader groups in 2D. To ensure The point of the exercise is to recommend Manager node lets you assign a color – that the data is in a format suitable for vi- articles that are closest to the reader’s based on the value of a column – to the sualization, you must first reduce the preferences. For example, if the reader is rows of a table. If the same column is number of dimensions. Up to now, the particularly interested in hardware and also used for the x-axis of the histo- example has used five dimensions per security, it would be a good idea to sug- gram, KNIME automatically takes over reader (one for each section), but I will gest articles from these categories – or the color for the bars (Figure 6). now break this down into two using the even articles that are in both categories at principal component analysis as calcu- the same time. The current workflow has Visualizing Multiple lated by the PCA (Principal Component already explored the extent to which a Dimensions Analysis) node. This type of transforma- certain reader has a preference for each Visualizations are available to display tion reduces the dimensionality, but at category, and the result is available in the data and also show whether the discov- the same time tries to keep the variance form of a vector (Figure 5). ered clusters have any significance. In in the data as complete as possible so that You can create a very similar vector for the past years, many visualization meth- as little information as possible is lost. each article, where the columns of those ods were implemented in KNIME with By this point, a proper data analysis categories contain a 1, to which the article the help of JavaScript and the D3.js workflow has already been created: from is assigned. Category columns without framework [2]. importing the data, through transforma- connection to the article, on the other These visualizations are available in tion and grouping, the KNIME Javascript Views extension, to visualizing the

Figure 6: Histogram of the groups of readers as Figure 7: A parallel coordinate chart of reader prefer- determined by cluster analysis. ences. Each color represents a group of readers.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 49 IN-DEPTH KNIME

Table 2: Categories Table Article ID Category Article 11 Hardware Article 11 Software Article 31 Development …

hand, contain a 0. The One‑to‑Many node makes this possible: The transformation of the article-heading assignments (Table 2) into a representation by a binary vector per article (Figure 10). Once the two vector types have been Figure 8: A Voronoi diagram of the readers created with the Generic created, the Similarity Search node can Javascript View. simply determine a distance between the vector of a reader (which represents its preferences) and the vector of an article (which indicates to which columns it is assigned). The smaller the distance, the more the reader’s preference corresponds to the categories in which the article is classified. From all articles that a certain reader has not yet read (see Row Reference Fil‑ ter node), it is now easy to determine the article that has the shortest distance to the reader’s preferences. This article is finally recommended for reading. A loop (consisting of the Chunk Loop Figure 9: The sample workflow from Figure 3 with parts of the work- Start and Loop End nodes) corresponds to flow summarized as meta nodes. a For loop across all rows of the table. The loop determines the smallest distance of the vectors for each reader. The overall result with one article rec- ommendation per reader (Figure 11) could then be written back to a database with the help of the Database Writer node. Conclusion This article only covers a fraction of the nearly 2,000 nodes available for KNIME. Other exciting KNIME features include flow vari- Figure 10: Output table of the one-to-many node. ables, the Workflow Coach, streaming, nodes for processing texts, and Deep Learning capabilities. You can get an idea of the KNIME An- alytics Platform by downloading and testing the software. Check out the documentation [3] and node guide [4] at the KNIME website for more on working with KNIME, or bring your questions to the KNIME Forum [5]. nnn

Info [1] KNIME: https://www.knime.com [2] D3.js framework: https://d3js.org [3] KNIME Documentation: https://www.knime.com/documentation [4] KNIME Node Guide: https://www.knime.com/nodeguide [5] KNIME Forum: Figure 11: The content of the meta node for article recommendations. https://www.knime.com/forum

50 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM

IN-DEPTH Binary Data in Bash

Manipulating Binary Data with Bash Bit Trip Bash is known for admin utilities and text manipulation tools, but the venerable command shell included with most Linux systems also has some powerful commands for manipulating binary data. By Andy Carlson

ne of the most versatile script- command> echo ‑n "hello" | xxd ‑p use plain hexdump style, representing ing environments available on output> 68656c6c6f each 8-bit byte as a two-digit hexadecimal Linux is the Bash shell. The number. To output data as a binary string core functionality of Bash in- outputs the binary values of the ASCII (containing 1s and 0s), you need to use cludesO many mechanisms for tasks such string “hello” as an ASCII string of hexa- different switches. The following function as string processing, mathematical com- decimal values (refer to Table 1). returns a binary string representation of putation, data I/O , and process manage- In the following command: ment. When you couple Bash with the Table 1: ASCII Lowercase Alphabet countless command-line utilities avail- command> echo ‑n "6162" | xxd ‑p ‑r Char Decimal Hexadecimal able for everything from image process- output> ab a 97 61 ing to virtual machine (VM) manage- b 98 62 ment, you have a very powerful script- The hexadecimal string “6162” is con- c 99 63 ing platform. verted into what the xxd man page calls a d 100 64 One thing that Bash is not generally “mail-safe ASCII representation” of the bi- e 101 65 known for is its ability to process data nary data. Because hexadecimal 61 and 62 f 102 66 at the bit level; however, the Bash shell correspond to ASCII characters “a” and g 103 67 contains several powerful commands “b,” respectively, the binary data is ex- h 104 68 that allow you to manipulate and edit pressed in the form of the string “ab.” i 105 69 binary data. This article describes Bash makes it easy to build these com- j 106 6A some of these binary commands and mands into functions. The following k 107 6B shows them at work in some practical functions, pack and unpack (named be- l 108 6C situations. cause of their similarity to the PHP pack m 109 6D function) use the preceding commands n 110 6E to convert a hexadecimal string to binary Viewing and Converting o 111 6F and convert binary to hex. Data p 112 70 Two tools you can use to represent data q 113 71 in hexadecimal and binary format are pack() { r 114 72 hexdump and xxd. The hexdump utility pro- echo ‑n "$1" | xxd ‑p ‑r s 115 73 vides many options for outputting hexa- } t 116 74 decimal data. Although xxd lacks some u 117 75 of the options available with hexdump, it unpack() { v 118 76 does have one key feature hexdump lacks: echo ‑n "$1" | xxd ‑p w 119 77 In addition to letting you output hexa- } x 120 78 decimal values, xxd also has the ability y 121 79 to convert a hexadecimal string into bi- All the examples so far have used the ‑p z 122 7A

nary data. The following command: switch to specify that the command will 123RF.com Lead Image © Yewkeo,

52 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM IN-DEPTH Binary Data in Bash binary data, along with a sample com- to a valid hexadecimal number in a sin- Listing 1: Bitwise Operations mand and output: gle command. The function pipes a 01 logicand() {

string of commands into bc. The first com- 02 echo $(($1&$2)) tobin() { mand, , sets the base in which obase=16 03 } echo ‑n "$1" | xxd ‑b ‑g0 | U data will be output (in this case, hexadeci- 04 awk '{ printf("%s",$2) }' mal or base 16). The second command, 05 logicor() { } ibase=A, sets the base used to read input 06 echo $(($1|$2)) data. The input base is set up A, which cor- 07 } command> tobin ab responds to base 10. The third and final 08 output> 0110000101100010 command is an addition statement com- 09 logicxor() { prised of an arithmetic expansion and the 10 echo $(($1^$2)) The two-character string “ab” is con- second function argument. The arithmetic 11 } verted into a binary string containing expansion uses the # operator to convert 12 two 8-bit values: 01100001 and 01100010 the number following it from the base 13 command> logicand 3 6 (97 and 98), corresponding to the base specified before the # to base 10. In the 14 output> 2 10 values for characters “a” and “b.” command, the hexadecimal value for “A” 15 command> logicor 3 6 is converted to decimal 10 using arithmetic 16 output> 7 Transforming Binary Data expansion, added to 2 using bc, and then 17 command> logicxor 3 6 Bash also offers some commands for converted back into hexadecimal using bc. 18 output> 5 transforming binary data using mathe- You can also perform bitwise opera- matical and logical operations. This tions on the data, including AND, OR, process is call arithmetic expansion. In XOR, and shift. The AND operation and for each bit space shifted to the the following example: (using the & operator) returns the bits right, the number is divided by 2. shared between the two numbers. The command> echo $(( 4*5 )) OR operation (using the | operator) re- Examples output> 20 turns all bits used by either of the two Switching from binary to hexadecimal, numbers. The XOR operation (using the and moving bits around to change an A the output of the echo command is the ^ operator) returns all bits that are into a C, is certainly interesting, but does value calculated by the statement en- unique to one of the numbers. this capability have any uses in the real closed in $(( and )). In Listing 1, the first command per- world? The following examples offer Another tool for performing mathe- forms a binary AND on 3 (or 0011 in bi- some hints for how you could use these matical operations is bc. bc is a com- nary) and 6 (or 0110 in binary). Because tools in practical ways. mand-line calculator that can read state- the only shared bit is the second bit, 2 (or ments from standard input. For example: 0010 in binary), the output for the com- ASCII to Unicode mand is 2. The second command per- In some situations, you might need to command> echo "4*5" | bc forms an OR operation against the same convert an ASCII character into Unicode. output> 20 two numbers: 3 and 6. Between the two ASCII is an 8-bit character set, whereas numbers, the lower three bits are used, as Unicode starts at a 16-bit length. Con- Note that the preceding operation is per- so 7 (or 0111 in binary) is returned. In the verting from ASCII to Unicode might formed in base 10. bc does not provide third example an XOR is performed seem complicated, but it is actually quite native functionality for arithematic oper- against the same two numbers. The sec- simple thanks to the backward compati- ations on hexadecimal numbers, so you ond and fourth bits are the same between bility built into the Unicode standard. To need to convert the numbers to base 10 the two numbers, however, the third and convert ASCII to Unicode, you just need and then perform the operation. For in- first bit are different, therefore 5 (or 0101 to prepend the value of 0 onto each stance, if you want to perform simple in binary), is returned. addition and subtraction operations The other bitwise operations is the Listing 2: Shifting Bits shift operation, which shifts the bits against hexadecimal numbers, use a 01 shiftr() { function like the following: in one direction or another (right or left). 02 echo $(($1>>$2)) Listing 2 shows functions for shifting the 03 } hexadd() { bits shifting right (using the >> operator) 04 echo "obase=16;ibase=A;U and left (using the << operator). 05 shiftl() { $((16#$1))+$2" | bc The first command in Listing 2 shifts 06 echo $(($1<<$2)) } the bits in 4 (0100) left by two spaces, re- 07 } turning a value of 16 (10000). The sec- 08 command> hexadd A 2 ond command shifts the bits in 16 09 command> shiftl 4 2 output> C (10000) to the right four spaces, return- 10 output> 16 ing a value of 1 (0001). As you might 11 command> shiftr 16 4 The hexadd function provides the ability to have noticed, for each bit space shifted 12 output> 1 add or subtract (add a negative number) to the left, the number is multiplied by 2,

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 53 IN-DEPTH Binary Data in Bash

ASCII character, thus making it a 16-bit line-by-line, and when the line contains a “%” by the sed command. When decod- character (see Listing 3). single character, it prints the character ing, all percent signs are stripped and In Listing 3, the output of the command prepended by the character value “0”. the output is piped into xxd to revert the appears to show no noticeable change. To hexadecimal string to ASCII. get a better view of the binary data behind URL Encoding and Decoding this text, pipe the output into xxd: Hexadecimal data is something you see Calculating IP Subnets every day, but it often goes unnoticed. On an IP network, the subnet mask command> ascii2unicode jello | xxd When data is passed as a query string in a specifies how many bits of the IP ad- output> 0000000: 006a 0065 006c U URL, it may be encoded using special for- dress will be dedicated to the network ID 006c 006f .j.e.l.l.o matting. This formatting consists of a per- and how many will be used for the host cent sign followed by the hexadecimal ID. The size of the host ID address space As you can see, the ASCII values have value of an ASCII character. For example, will tell you how many host IP addresses been prepended with “00,” which con- the URL encoded string of “%61%62%63,” are available. Listing 5 shows how to verts them to 16-bit Unicode characters. when decoded, becomes “abc.” Listing 4 convert the subnet mask to a binary Take a closer look at Listing 3 to see shows a function for performing URL en- string and determine the host ID count. what’s happening: The output of the echo coding and decoding. The output of the echo statement is fed statement is piped into the sed statement, The function in Listing 4 uses the stan- into the awk statement. This first awk com- which places each character of the output dard functionality of xxd. When encod- mand will generate the statement that is on a separate line. The awk command ing a string, the output of xxd is split into piped into the following bc command. reads the input from the sed command 1-byte chunks and prepended with a The statement will include ibase, obase, and each individual octet of the subnet Listing 3: ASCII to Unicode mask. Once bc evaluates the statements,

01 ascii2unicode() { it returns four lines: one line for each octet. The following sed statement finds 02 echo "$1" | sed 's/$.$/\1\n/g' | awk '/^.$/{ printf("%c%c",0,$0) }' lines containing only “0” and extends 03 } them to 8-bits of zeros. The sed statement 04 also puts each bit on a line by itself. This 05 command> ascii2unicode jello will be necessary to properly evaluate the 06 output> jello host bit length. The awk statement has three sections. The first section initializes Listing 4: URL Encoding and Decoding the ht and nt variables, which store the

01 urlencode() { host total bits and network total bits, re-

02 echo ‑n "$1" | xxd ‑p | tr ‑d '\n' | sed 's/$..$/%\1/g' spectively. The next section searches for lines containing 0 or 1. If the value is 1, 03 } the network total is incremented, and if 04 the value is 0, the host total is incre- 05 urldecode() { mented. The final section of the awk state- 06 tr ‑d '%' <<< "$1" | xxd ‑r ‑p ment prints the summary data for the net- 07 } work, including the host and network bit 08 counts, along with the host IP count. 09 command> urlencode name 10 output> %6e%61%6d%65 Conclusion 11 command> urldecode %64%6f%6e%65 The versatile Bash shell offers several 12 output> done utilities that allow you to output, transform, and manipu- Listing 5: Converting a Subnet Mask late binary values.

01 subnetcalc() { This article introduced you to the 02 echo ‑n "$1" | \ xxd and bc commands. 03 awk 'BEGIN { FS="." ; printf("obase=2;ibase=A;") } { printf("%s;%s;%s;%s;\n",$1,$2,$3,$4) }' | \ You also learned how 04 bc | sed 's/^0$/00000000/g;s/$.$/\1\n/g' | \ to add these binary 05 awk 'BEGIN { ht = 0; nt = 0; } commands to your 06 /[01]/ { if ($0=="1") nt++; if ($0=="0") ht++; } custom Bash func- 07 END { printf("Network bits: %s\nHost bits: %s\nHost IP Count: %d\n",nt,ht,2^ht); }' tions to build your 08 } own tools for per- command> subnetcalc 255.255.192.0 forming practical output> Network bits: 18 tasks like decoding output> Host bits: 14 URLs and calculating output> Host IP Count: 16384 subnet masks. nnn

54 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM

MAKERSPACE Countdown Counter MakerSpace A countdown counter with the MAX7221 and a seven-segment display 3, 2, 1 … Go!

Build a countdown counter with a Raspberry Pi and some electronics, and you can count down the time to any event. By Martin Mohr

ays until Christmas, time until seven-segment Kingbright SC08-11SRWA retirement: We count the days [1] display (20.32mm/0.8 inch high) is to many events in life. A count- used to display the remaining days to an down counter can help make event. The LEDs use a common cathode, the time to wait fly by and increase antici- which makes the display compatible pation.D In this article, I show you how to with a MAX7221 display driver. use the Raspberry Pi as a control unit to The combination of parts used here rebuild a chic countdown counter based on quires the use of a soldering iron. Com- an LED segment display. pletely assembled modules that match To ensure that the countdown can be the capabilities of the MAX7221 are seen from a distance easily, a large available, but they have considerably

Table 1: MAX7221: Register Assignment Register Function Description 0x00 No-op Functionless. In the case of several SPI blocks on one bus, it is used to address only one block. All others receive a no-op. 0x01 Digit0 Digit 0 of the display. 0x02 Digit1 Digit 1 of the display. 0x03 Digit2 Digit 2 of the display. 0x04 Digit3 Digit 3 of the display. 0x05 Digit4 Digit 4 of the display. 0x06 Digit5 Digit 5 of the display. 0x07 Digit6 Digit 6 of the display. 0x08 Digit7 Digit 7 of the display. 0x09 Decode Each bit corresponds to one digit of the display (bit 0 = digit 0, etc.). mode If it is set, the digit works as a segment display; the lower 4 bits of Author the digit registers are then interpreted as a BCD number. If it is de- Born in the era of magnetic core ring accu- leted, each segment of the digit can be controlled as a single LED. mulators and rotary dial phones, Martin 0x0A Intensity The lower 4 bits of the register control the brightness of the LEDs. Mohr developed a love for everything that 0x0B Scan limit The lower 3 bits of the register control which digits of the display flashes at an early age. After studying com- are used (0x00 = only digit 0; 0x01 = digits 0 and 1; etc.). puter science, the qualified electrician devel- 0x0C Shutdown Bit 0 = 1: Normal mode, all displays and functions active. Bit 0 = 0: Shutdown mode, all displays off, no response to commands. oped Java applications. His old passion for 0x0F Display test Lights all LEDs. electronics was revived by the Raspberry Pi. Lead Image © bluedarkat, 123RF.com

56 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM Countdown Counter MAKERSPACE

Figure 1: The Fritzing circuit diagram of the countdown project. smaller displays of (typically) eight dig- The setup is thus quite independent its. Four digits should be enough for a of the voltage source (e.g., a USB day countdown (i.e., 9,999 days or more power supply, car battery, etc.). Two than 27 years). If you really want to capacitors buffer fluctuations in the count down for a longer period of time, input voltage and counteract the oscil- simply add an additional segment to the setup. Table 2: Components No. Name Source Price/Unit (EUR) MAX7221 Display Driver 4 SC08-11SRWA 7-segment display reichelt.de 1.15 In the MAX7221 display driver, an external 1 MAX7221CNG 8-digit LED display driver reichelt.de 5.99 resistor controls the current for the LEDs. 1 PCB prototype 8x12 aliexpress.com ~1 I’ll shoot for 33kohm, which results in a Various small parts Craft kit 2 current of just under 20mA for one segment of the display. Otherwise, the Total cost ~EUR20 MAX7221 does not require external wiring. Internally, the display stores the information to be displayed in 8x8-bit RAM. Table 1 is an overview of the MAX7221 registers and their functions; a data sheet [2] provides further information. The MAX7221 is addressed through the SPI interface. Many microcontrollers are suitable for this purpose – I concentrate on the Raspberry Pi.

Circuit Diagram In the countdown counter circuit diagram (Figure 1), the MAX7221 is the central component. Connected to the four seven- segment displays, it only requires one external component: resistor R1. The complete circuit operates at 3.3V, which is sup- plied by a type LD1117v33 voltage regulator by STMicroelectronics (input voltage range = 4.3-15V). Figure 2: (Top) Countdown counter on a Euroboard; (bottom) Raspberry Pi.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 57 MAKERSPACE Countdown Counter

lation of the voltage regulator. A Program header plug joins the connectors of the The experimental setup is based on a SPI interface to the Raspberry Pi. Fig- Raspberry Pi 1 with a current Raspbian ure 2 shows the complete setup; all image, but the steps described here also components are listed in Table 2. work with any other Raspberry Pi model.

Cron The Unix cron service allows commands and scripts to be executed automatically at certain times. The crontab command-line tool is used for configuration. Use crontab ‑l to see the current settings for the logged in user. The ‑e switch puts Crontab in edit mode. An entry contains, separated by white space, the specifica- tions for minute, hour, day of the month, month, and weekday, as well as the command to be executed at the specified time. The first five parameters control execution time. Numbers stand for times or dates and an asterisk for any given minute, hour, or day. If necessary, you can also use a backslash to define multiple values (e.g., */2 for every second minute). As far as timing is concerned, it is important to ensure that one call can be processed before the next initiates; otherwise, the system will become overloaded over time. The following are some typical cron examples:

* * * * * /home/user/script # every minute

0 */2 * * * /home/user/script # at the start of every other hour

0 0 * * * /home/user/script # every day at midnight

30 18 * * * /home/user/script # every day at 6:30pm

* * 1 * * /home/user/script # every minute on the first day of the month

Further detailed information can be found on the cron man page. Useful pages on the web, such as at Corntab [3], can help you to compose cron lines. You need to make sure that cron executes scripts with the rights of its own user account but does not use environment variables. To do this, specify all paths in full and do not use aliases.

Listing 1: counter.c 01 #include 24 if (ioctl(handle, SPI_IOC_RD_MAX_SPEED_HZ, &speed)<0) {

02 #include 25 perror("SPI Speed Error");

03 #include 26 return 1;

04 #include 27 }

05 #include 28 // MAX7221‑Setup

06 #include 29 write_register(handle,0x0C,0x01); 30 write_register(handle,0x09,0xFF); 07 31 write_register(handle,0x0F,0x00); 08 static const char *device = "/dev/spidev0.0"; 32 write_register(handle,0x0B,0x03); 09 static uint32_t speed = 500000; 33 write_register(handle,0x0A,0x0F); 10 int handle, diff; 34 ts_now = time(NULL); 11 time_t ts_now, ts_target; 35 target.tm_mday=1; 12 struct tm target; 36 target.tm_mon=4; // Month 1 (Jan = 0) 13 37 target.tm_year=119; // Year ‑ 1900 14 void write_register(int handle, uint8_t regi, uint8_t value) { 38 target.tm_sec=59; 15 uint8_t tx[] = { regi, value }; 39 target.tm_min=59; 16 write(handle,tx,2); 40 target.tm_hour=23; 17 } 41 ts_target=mktime(&target); 18 42 diff=(ts_target‑ts_now)/(60*60*24); 19 int main() { 43 write_register(handle,0x04,(diff%10000)/1000);

20 if ((handle = open(device, O_RDWR)) < 0) { 44 write_register(handle,0x03,(diff%1000)/100);

21 perror("SPI Device Error"); 45 write_register(handle,0x02,(diff%100)/10);

22 return 1; 46 write_register(handle,0x01,(diff%10));

23 } 47 }

58 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM Countdown Counter MAKERSPACE

First, activate the SPI interface with current and target dates and write it to raspi‑config in 5 Interfacing Options | P4 the corresponding memory locations of SPI. After the next reboot, the /dev/spi‑ the MAX7221. By transferring the start dev0.0 Raspberry Pi device directory con- of the program to a cronjob (see the tains the SPI device, which can be used as “Cron” box), you can start the program a character-oriented device under Unix. once a day, calculate the current dis- Listing 1 shows the counter.c example play value, and transfer it to the dis- program, which loads all the libraries play driver. Save the program and com- used and defines the necessary variables. pile it with gcc (GNU C compiler): The write_register() function (lines 14- 17) encapsulates the write() C function $ gcc counter.c ‑o counter and arranges the parameters accordingly. $ ./counter The main() function (line 19 to end of code) establishes the connection to the The last line runs the program. SPI device and sets the speed of the SPI interface. It then initializes the MAX7221 Conclusions to use digits 0 to 3 in decoder mode and A close look at the MAX7221 reveals sets the brightness to the maximum. that the device is ideal for controlling The target date structure (line 12) gen- LEDs: It supports up to 64 individual erates the timestamp of the target date. diodes, the design does not require any Info This structure has some peculiar charac- series resistors, and the brightness of [1] SC08-11SRWA single-digit numeric dis- teristics that can cause a great deal of the LEDs can be adjusted through the play data sheet: trouble: Days count from 1 and months module. By the way, if the eagerly http://www.kingbrightusa.com/images/ from 0, and 1900 must be deducted from awaited event is very far into the fu- catalog/SPEC/SC08‑11SRWA.pdf the year. The values for time of day can ture, you should consider finding a [2] MAX7221 data sheet: be written one to one into the structure. more energy-efficient supplier of SPI https://datasheets.maximintegrated. The remaining lines of the program signals than the Raspberry Pi, such as com/en/ds/MAX7219‑MAX7221.pdf calculate the difference between the an ESP8266. nnn [3] Corntab: http://corntab.com MAKERSPACE Open Hardware – Keyboardio MakerSpace Reinventing the keyboard Custom Keyboard

The Keyboardio Model 01 is a highly-customizable open hardware keyboard. By Bruce Byfield

or a while, the Keyboardio [1] though shipping is still seriously back- Model 01 seemed destined to logged, the Model 01 (Figure 1) turns never ship. Although its 2015 out to be worth waiting for, both as a crowdfunding campaign raised high-end ergonomic keyboard and as over five times its goal of $120,000, one of the most customizable pieces of productionF was repeatedly thwarted by open hardware ever produced. problems with manufacturers and The Model 01 is a mechanical key- quality control – all of which is re- board built with Matias keyswitches [3] corded in painstaking detail on the that are not only longer-lasting and eas- company blog [2], and which should ier to repair than the usual cheap mem- become required reading for any would- brane keyboards, but require less pres- be hardware vendor. Yet despite every- sure to activate than standard key- thing, in November 2017, the Model 01 switches. The minimalist keyboard of keyboard finally began shipping. Al- 64 keys ships with three layers – that is, key bindings that can be activated while pressing a command key, just as the default lowercase keys produce uppercase letters when the Shift key is pressed. With the Model 01, pressing the Num key activates the number pad and arrow keys, while pressing the Fn key activates the 12 function keys, Delete, and square and curly brackets. The default layers are illustrated on a laminated sheet that ships with the Model 01. Keys are divided between two slabs of maple hardwood. Each of the keys is individually sculpted and shaped to help guide fingers from one key to the other. To reduce finger movement, rows of character keys are arranged in curving key banks and columns of keys on a straight diagonal, instead of having straight rows and columns staggered to the right, as on a standard keyboard. Command keys are to the left and right

Figure 1: The Model 01 straight out of the box. of the letter keys, or else arranged in an Lead Image © 3355m, 123RF.com

60 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM Open Hardware – Keyboardio MAKERSPACE arc of four at the bottom of the keyboard that are pressed by the thumbs. Besides the Fn key, the command keys include the replacement for a Windows key, the LED key for toggling through installed backlight settings, the Prog key for flashing the firmware, and the Any key, which originated in the old joke about procedure instructions to “press any key,” but became a blank key that can be programmed as desired. The keyboard ships without a Cap lock button, but, since the Shift key is accessed by the thumb, many users can continue to Figure 2: Depending on the type while holding it down. connecting cable, the Model 01 halves can placed together or The Advantages of widely separated. Ergonomics The arrangement and shapes of the This ergonomic advantage is no doubt Figure 3: The stands and center- Model 01’s keys, as well as the choice of due to the fact that the Model 01 began plate allow the keyboard to be posi- keyswitches, are designed to reduce the as a highly personal project of Keyboar- tioned in several different ways. typing stress. The result is that typing on dio’s technical partner Jesse Vincent. a Model 01 is a different experience from Those who regularly spend over eight other by an RJ45 cable – a short one for typing on most keyboards – especially hours a day at a keyboard can especially those who want to place the halves from the cheap membrane keyboards appreciate the advantages. Although it close together, or a longer one for users sold in most computer stores. As the almost seems beside the point, the ar- who prefer to separate them. The Quick Start Guide notes, learning to type rangement of keys on an arc and their halves can be joined by a center bar or at your accustomed speed on a Model 01 mounting on maple also makes the kept separate so that they can be posi- can take as long as several weeks (for Model 01 the most stylish keyboard since tioned individually on a flat surface. me, it was about five days). If my own those produced by custom keyboard Still another option is to place the two experience is any indication, a new user maker Datamancer [4] – although after halves flat or to mount each on its own can expect to start with a typing tech- the industry-standard ugliness, the aes- stand at a variety of angles (Figure 3). nique not far removed from hunt and thetics are a welcome change. For ergonomic purposes, the most effi- peck, then gain about half their usual cient arrangement is to tent each half, so speed until they become too tired to re- Hardware Customizations that the inner edge is higher than the member their new reflexes, and after When ordering a Model 01, you can outer (Figure 4). If the centerplate is also that to slowly and steadily regain their choose one programmed by default for used, the stands must be screwed into usual cruising speed. the standard QWERTY layout, or alter- position after the centerplate is added. This process can be trying, but it has native layouts such as Dvorak, Cole- Without any software editing, users the advantage of suggesting how you mak, Malt, or Workman. You also have can also change the keyboard backlight- might customize the keyboard layout to the choice of keycaps – including Runic ing by toggling the LED key (Figure 5). your individual advantage (see below). or blank – and can buy extra sets as A dozen different default options are For example, I find myself constantly well. In addition, you have the choice available, ranging from a half a dozen overreaching when I type an A or a Z and of the default quiet keyswitches or the backlight colors, the lighting up of a hitting the PgUp or PgDn keys beyond loud keyswitches that some typists pre- pressed key, the blinking on and off of them. Therefore, I decided to remap the fer for their audio two keys I was hitting by mistake so that feedback. they copy and paste instead. Once your order Moreover, contrary to what cynics arrives, you can might assume, in the case of Keyboardio’s experiment with Model 01, the word “ergonomic” is more several different than a marketing term (often used the way layouts. A Model that “organic” or “gluten-free” might be 01 is split into two used in the supermarket). My own experi- halves (Figure 2). ence is that, after a few days of stiffness as Each half contains finger muscles are used in unfamiliar a ATmega32u4 ways, the twinges and disablements of Arduino micro- repetitive stress injuries are reduced to controller and is short-lived ghosts of their former selves. connected to the Figure 4: The two sides positioned at an angle.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 61 MAKERSPACE Open Hardware – Keyboardio

the entire array of keys, or turning all key is named backlights off. If you have never used along with what backlit keys, you may be inclined to dis- happens when miss them as frivolous, but they may that key is help to make your typing more accurate pressed. The de- by making the keys more visible, espe- fault firmware de- cially in dim light. The Model 01 uses fines the default low-draw backlights, so using them layer for a should have minimal effects on the lon- QWERTY key lay- gevity of the hardware. out and separate Do-it-yourselfers can even open up layers for the key- the keyboard to adjust the microcon- bindings when the trollers. Unlike with other, proprietary, Fn and Num keys products, user exploration of the hard- are pressed. Users ware does not void the warranty. In can modify the fact, to emphasize Keyboardio’s dedica- default layers, en- Figure 5: Backlights make keys more visible. tion to openness, the Model 01 ships abling unused with a small multi-headed screwdriver keys, or adding new ones, such as tradi- Like keymaps, macros can also be suitable for opening it up. tional alternative layouts like Dvorak or added directly to keymap Model01‑Firm‑ Colemak [9]. Other popular modifica- ware.ino. For macros for a specific Keyboard Customization tions include rearranging the command piece of software, such as Vim, you can Unlike Input Club [5], another maker of keys such as Space or Esc to preferred add a new layer. Some users have also open source keyboards, Keyboardio positions. Some definitions for keys can created new LED effects as plugins, does not yet have a graphical interface be learned from the default file or from which are accessed via macros. Basic for customizing keybindings. An inter- the online reference [10]. instructions for writing a macro are face called Chrysalis [6] is in develop- To a certain extent, these changes can available online [11], and the existing ment, but it is not yet officially re- be reinforced by repositioning the easily macros can also be tweaked or com- leased. Nor does the Model 01 ship removable keycaps on the Model 01. mented out. A partial list of existing with any instructions on how to modify However, the individually-sculpted keys macros is available online [12]; an es- Kaleidoscope [7], the Model 01 firm- may limit the repositioning, which is pecially useful one is MacrosOnTheFly, ware – apparently on the assumption why additional sets of keycaps – espe- which can record keystrokes and play that those interested will know enough cially blank ones – are useful. them back as macros [13]. to find the online instructions. How- ever, even if they do, instructions and resources are scattered over several sites. Customizing the Model 01’s behavior means modifying the firmware and rein- stalling or flashing the result – the identical requirement for installing updated firmware from Keyboardio. The full set of instructions and their background are too lengthy to give here, but it requires the installation of an Arduino environment and Kaleidoscope on a computer to which the keyboard can be attached [8]. Once this prerequisite is completed, users can modify keymaps, macros, and plugins, either by themselves or by downloading and pasting keymaps created by other users. The simplest modification is editing the keymapping to change what happens when each individual key is pressed (Figure 6). Keymapping is set in the Model01‑Firmware.ino file, which is an example of what Arduino programming calls a Sketch. A keymapping is literally that – a text-based map in which each Figure 6: A default keymapping displayed in the Arduino IDE.

62 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM Open Hardware – Keyboardio MAKERSPACE

More complex modifications can be firmware should be protected as intellec- ality is there, and no other keyboard placed in plugins, which generally require tual property. However, Keyboardio re- offers anything near the Model 01’s some Arduino programming knowledge. mained determined to produce a quality combination of ergonomics, aesthetics, A description of core – that is, official open source product, and, despite the capacity for customization, and quality plugins – is available [14], as well as manufacturing delay and the chaotic hardware. descriptions of unofficial ones [15]. modification instructions, it has gener- In the Model 01, Keyboardio shows The resources listed above are reason- ally delivered. how much can still be done to improve a ably complete, but hackers should True, without shipped instructions piece of hardware that has changed little watch the Keyboardio community site and a graphical interface, the Model 01 for several decades. That is no mean ac- [16] for other examples and instruc- is less accessible than it should be. It is complishment and makes Keyboardio an tions. There has been some talk of a also true that other keyboards have outstanding example of what open hard- central repository and manuals, but, as some of its features. But the function- ware can accomplish. nnn I write, neither exists. All these modifications can be written Info in the Arduino IDE or in a text editor – [1] Keyboardio: https://shop.keyboard.io [10] Key definitions: https://github.com/ after, of course, a backup is created for [2] Keyboardio blog: keyboardio/ Kaleidoscope/blob/master/ recovering from disastrous results. http://blog.keyboard.io/ src/key_defs_keyboard.h When the modifications are complete, [3] Matias keyswitches: [11] Writing a macro: https://github.com/ save the revised Sketch in ~/Arduino/ http://matias.ca/switches/linear/ keyboardio/ Kaleidoscope‑Macros/blob/ Model01‑Firmware. From the same direc- master/README.md [4] Datamancer: https://datamancer.com/ tory, run the command make flash. [12] Partial list of macros: When the output signals that the com- [5] Input Club: https://input.club/ https://github.com/algernon/ mand is ready, hold down the Prog key [6] Chrysalis: Model01‑sketch# installation so that the command can bypass the https://community.keyboard.io/t/ [13] microcontrollers’ bootloaders before chrysalis‑layout‑keyboard‑settings‑ MacrosOnTheFly: https://github.com/cdisselkoen/ continuing. The Prog key will flash to editor/253/21 Kaleidoscope‑MacrosOnTheFly show that it is being used. When the [7] Kaleidoscope: https://github.com/ command is complete, unplug the key- keyboardio/ Kaleidoscope [14] Official plugins: https://github.com/keyboardio/ board, and plug it in again so that the [8] Installing Arduino IDE and Kaleido- Kaleidoscope/wiki/Core‑Plugins firmware can restart. scope: https://github.com/keyboardio/ Kaleidoscope/ wiki/ [15] Unofficial plugins: Rethinking the Keyboard Keyboardio‑Model‑01‑Introduction https://community.keyboard.io/c/ On the company blog, Keyboardio tells [9] Traditional alternative keymaps: programming/ Discuss‑Kaleidoscope‑ how at initial meetings, potential manu- https://github.com/keyboardio/ Plugins‑one‑thread‑per‑plugin facturers suggested that the keys should Kaleidoscope/ wiki/ [16] Keyboardio community site: be mounted on white plastic, or that the Common‑Alternate‑Layouts https://community.keyboard.io 64 SPACEMAKER MAY 2018 Raspberry Slideshow Raspberry

ISSUE 210

files canonlybe carriedoutwiththe manual modificationstoconfiguration SSH. Inthefree version, then,certain customizing theconfiguration over version doesnotinclude anoptionfor contrast totheDonors’ Build,thefree two functionallydifferent versions: In mixed ifnecessary. fade invideosequences, whichcanbe show toplay adigitalphotoalbumor the Raspbian-derivative Raspberry Slide- ing columnsandbillboards. You canuse Rasp Picaneasilysupplementadvertis- at avery low price. content tothescreen professionally and computer, you canbringmultimedia Raspberry Slideshow [1]andthetiny been usedforadvertising purposes. With vidual computers withdisplays have sets withVCRsor, inrecent times, indi- into play where previously expensive TV for industrialmachines. vironment, forexample asacontroller made itselfusefulinthecommercial en- the meantime, theminiature PChas of purely experimental applications. In can capture theattentionyouneed. a RaspberryPiandtheSlideshowdistribution,you T MakerSpace The operating system isavailable in Thanks tointelligentsoftware, the For thisarticle, theRaspberry Picomes long sincesurpassedthepoint ments, theRaspberry Pihas and incremental improve - hanks toitsactive community Video sequencesorslideshowspromoteVideo sales.With Advertising with the Raspberry Pi withtheRaspberry Advertising LINUX-MAGAZINE.COM |LINUXPROMAGAZINE.COM Show Talent the card. show thathasactuallymadeitonto then onlydisplay thedatainslide message. Asaresult, thesystem will cancels theprocess withoutanerror content totheSDcard, thesoftware vice. Ifspacerunsoutwhencopying even withoutanexternal memoryde- files canbeplayed backautomatically, from aUSBsticktothecard sothat tition (Figure 1). mended withanexpanded system par- higher capacitymemorycard isrecom- card atleast4GBinsize, althougha transfer theextracted imagetoanSD site. Afterunpackingthearchive, 580MB ZIParchive ontheSourceForge Raspberry Slideshow isavailable asa Getting Started EUR16.90. Build [2],whichisavailable for tions can onlybehandled by the Donors’ basis ofexisting Exifdata in presenta- automatic rotation oftheimages onthe on theInternet.Different server listsand content stored onaUSBmemory stickor sion onlyletsyou play backmultimedia the SDcard. edit thecorresponding files directly on help ofaLinuxPC,from which you can The software copiesthecontents Additionally, thefreely available ver - By ErikBärwaldt

Lead Image © 3355m, 123RF.com Raspberry Slideshow MAKERSPACE

for the Raspberry Pi 3 (RPi3) at least 2,500mA. Problems often arise when buying cheap power supplies from the Far East that come directly from the manufacturer: Often they do not comply with current EU standards and fail to deliver the promised performance.

Configuration Different form factors between photos and screens often result in black edges on the sides and occasionally on the upper and lower edges. A similar phenomenon often occurs when playing back videos that were produced with a 4:3 aspect ratio. To eliminate such unsightly edges as much as possible, expand the display area by mounting the SD memory card Figure 1: With GParted, you can increase the storage capacity for the with the boot partition on a Linux PC and Raspberry Slideshow system. editing the /boot/config.txt file. You need to enable the disable overscan=1 After installing the image on the SD Raspberry Slideshow uses the com- line, which is commented out, simply card, start your Rasp Pi with the Slide- mand-line program Fbi to display a by removing the preceding hashtag show system. The integration of external photo slide show (JPG, PNG, BMP, and (Figure 2). After a reboot, the system data sources into the network requires GIF) and OMXPlayer to play videos displays the videos and images in full- wired LAN access if an Raspberry Pi 2 (MP4 and AVI). screen mode, eliminating the annoying (RPi2) is used, because the system does If the system temporarily or continu- black borders when the image conforms not detect external wireless dongles in ously displays a yellow flash symbol at to the monitor. the standard version. Even the adapters the top right of the screen after power- Another problem arises when playing directly supported by Raspbian are not up, the power supply is not performing back large image collections: The Fbi integrated during startup. well enough, which could cause slide player uses only a relatively small buffer, The operating system plays a video shows to stop prematurely or the sys- causing jerky transitions when changing show for demonstration purposes: the tem to transfer video files incompletely images. To prevent this, edit the /etc/ northern Italian city of Verona at night. to the SD card, resulting in a black rs.conf file located in the root partition The calculator displays the show in full- screen. of the SD card. screen mode and does not allow you to Make sure that the power supply of This file controls the parameters for control it with the keyboard or mouse. the RPi2 outputs at least 1,200mA, and starting the two playback programs. For

Content To view content in the appropriate order, add an ext4 or (V)FAT filesystem to a USB stick. Create a single partition only, and then place the desired videos or pictures directly in the main directory. Raspberry Slideshow is not able to read content from a directory hierarchy. Connect the prepared data carrier to the Rasp Pi and reboot. After powering up, the content automatically appears on the screen, with the Rasp Pi’s HDMI port also transmitting audio signals. The restriction on sources in the freely available version of Raspberry Slideshow reduces the benefit somewhat: You are al- lowed to play a maximum of three files of mixed content with the standard version. This restriction is removed only in Figure 2: Modifying the /boot/config.txt file gives you superior the commercial version. image reproduction.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 65 MAKERSPACE Raspberry Slideshow

Another interesting feature is automatic image rotation, which is only available in the commercial version: It uses the Exif data of the photos, guaranteeing that all images appear in the correct orientation. This version also enables the use of the integrated wireless chip in the RPi3, which allows remote configuration (e.g., from a notebook over SSH). Content can also be transferred to the Rasp Pi in this way, making it easier to update displays. On a 3K monitor connected to the RPi2 (Figure 4), some jerkiness was ob- served in high-resolution photos. How- ever, this effect was eliminated by ad- justing the buffer memory.

Conclusions Raspberry Slideshow turns the single- board computer into a full-fledged adver- Figure 3: The /etc/rs.conf file manages the player configuration. tising system for trade fairs, exhibitions, and demonstrations. Flexible handling of Fbi, the command sequences contain the Hands On the sources, various possibilities for cus- cachemem parameter, which specifies the In our lab, Raspberry Slideshow was tomization, and simple configuration in maximum size of the buffer used. Here, pretty impressive. However, the restric- only two text files are all appealing. The you need to increase the specified value tions of the free version – the lack of SSH full range of functions is only available to avoid jerky transitions. support and a limitation to only three with the commercial option, but the price In this file, you can also modify the files – greatly reduce the utility value. On you invest is money well spent. nnn parameters for cross-fade effects and the up side, it is noticeable that the soft- display time. You can also rotate im- ware (and this is not stated in the docu- Info ages or play them in a random order mentation) works without connected [1] Download: https://sourceforge.net/ (Figure 3). Detailed instructions can input devices. projects/raspberrypictureframe/files/ be found in the software's documenta- Raspberry Slideshow makes it easy to [2] Purchase a license: tion [3]. show videos or picture collections in an http://www.binaryemotions.com/ endless loop, such as in shop windows raspberry‑slideshow‑download/ Off the Web or at an exhibition. Moreover, the Rasp [3] FAQ: http://www.binaryemotions.com/ If you want to load content from a Pi can be attached discreetly to the back raspberry‑slideshow‑faq/ web server instead of copying it to the of the display, which avoids unnecessary [4] Examples: http://www.binaryemotions. USB stick, instruct the system to re- cable clutter. com/ raspberry‑slideshow‑download/ trieve the data directly from the Inter- net or LAN. To do so, create a simple text file named remote‑urls.txt on the stick, in which you enter one URL per line that refers to the content. You can use both web and FTP servers as a source. Store this text file in the main directory of the disk and restart Raspberry Slideshow. It now retrieves the images and videos from the web and plays them back locally. You may mix local and remote content in one session. The commercial version of the software also lets you retrieve content for a slide show from a Samba server and play back individual content on a remote server. Examples and useful documentation can be found on the project page [4]. Figure 4: The Rasp Pi also looks good with 3K and 4K displays.

66 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM

INTRODUCTION LINUX VOICE

Many Linux users secretly hope Linux will one day be the only operating system in the world, but we all know that day hasn’t happened yet. Talking to other systems is part of the challenge of working with Linux. This month, we introduce you to NitroShare, an easy tool that lets you exchange files and directories across Linux, Windows, Mac OS, and Android systems. Jasonette 70 And speaking of Android…what is it? A whole different Dmitri Popov system, or just another kind of Linux? The beauty of open Build Android apps with a minimum of source software is that you can mold and customize it coding. anyway you like, but despite the similarities, coding for the Doghouse – Subutai 75 Android environment is not exactly like coding for Linux. Jon “maddog” Hall Be a catalyst and help others break the This month we show you how to build an Android app with bonds of proprietary software. minimal programming using Jasonette, a tool that lets you NitroShare 76 create a complete Android Ferdinand Thommes application with a single JSON file. This handy app simplifies the exchange of What about Android and Linux files and directories between computers working together, like Master with Linux, Mac OS, Windows, and Android. and Blaster? Have you ever Nootka 80 wished you could use your Hartmut Noack phone’s GPS capabilities to With Nootka and your guitar, you can learn to recognize musical notes in classical add location services to notation. your computer? We’ve FOSSPicks 84 got that too. Read on to Graham Morrison learn how to set up the Graham checks out VLC 3.0, MenuLibre Linux gpsd daemon to receive 2.1.5, Texttop, Flameshot, Chomper, Godot GPS data from an Android 3.0, and much more!

device. 123RF.com Moroz, Image © Olexandr Tutorials – GPSD 90 Paul Brown Even though your computer does not come with its own GPS, you can hack one onto it using a mobile phone – if you are ready for some fiddling.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 69 LINUX VOICE JASONETTE

Instant Android Apps

Jasonette makes it supremely easy to build simple and advanced Android apps with a minimum of coding.

BY DMITRI POPOV or someone who doesn’t write code for a openSUSE, installing Android Studio is a matter of living, creating even the simplest Android running the following commands: F app can be a daunting proposition. How- ever, for those of us with modest coding skills, sudo mv android‑studio‑xx‑xxx‑xxx.zip /opt there is Jasonette [1]. This ingenious solution cd /opt makes it possible to create a full-featured Android sudo unzip android‑studio‑xx‑xxx‑xxx.zip app using a single JSON file that defines the app’s sudo chown ‑R root:root android‑studio basic structure and behavior. A generic Jasonette sudo ln ‑s /opt/android‑studio/bin/studio.sh U Android app pulls this JSON file and renders it as /usr/local/bin/android‑studio native Android components. Creating a Jasonette-based app still requires You don’t need to use Android Studio to build an some basic coding skills, but writing a JSON file is app just yet, though. To preview your JSON files significantly easier than writing Java or Kotlin directly on your Android device and test your app’s code. Also, building the Jasonette app in the An- basic functionality without building it in Android droid Studio IDE requires only a few simple steps. Studio first, you can use the Jason browser [3]. Better still, because the JSON file controls the Instead of a regular text editor, you can use a app’s appearance and behavior, you can easily up- dedicated tool for JSON code, such as JSON Editor date and improve it by editing the JSON code in- Online [4], which runs directly in the browser, or the stead of rebuilding the entire app. Elegant JSON Editor [5] Firefox add-on (Figure 1).

Preparatory Work Creating a JSON File Figure 1: Elegant JSON Edi- Before you start, you need to do some preparatory To get a basic understanding of what makes Ja- tor adds JSON editing capa- work. First, download and install Android Stu- sonette tick and how JSON code works, take a bilities to Firefox. dio [2] on your machine. If you happen to use look at the simple Jasonette app in Listing 1.

70 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM JASONETTE LINUX VOICE

Listing 1: Simple Jasonette App

{ "href": {

"$jason": { "url": "https://unsplash.com",

"head": { "view": "web"

"title": "Random Monkey", },

"description": "Display a random monkey photo", "style": {

"actions": { "font": "Roboto",

"$pull": { "size": "19"

"type": "$reload" } } } }, }, "styles": { "sections": [ "caption": { { "font": "Roboto", "items": [ "size": "13", { "align": "center", "type": "image", "spacing": "15" "url": "https://source.unsplash.com/ }, random/800x600/?monkey", "image": { "class": "image" "width": "100%" }, } { } "type": "label", }, "text": "Here is a random monkey photo", "body": { "class": "caption" "header": { } "title": "Random Monkey", ] "style": { } "font": "Roboto", ] "size": "25" } }, } "menu": { } "text": "Unsplash",

This simple JSON code defines the basic struc- Listing 2: head Properties ture and behavior of the app that fetches and displays a random photo of a monkey. Similar to an "$jason": { HTML page, the JSON file’s head section (Listing 2) "head": { specifies general properties such as the app’s title, "title": "Random Monkey", description, and actions: "description": "Display a random monkey photo", The $pull action in the header enables the pull- "actions": { to-refresh action. The styles section is used to "$pull": { specify styling for various interface elements. In "type": "$reload" } this example, it is caption for image text, and image }, for the image element. The body section (Listing 3) "styles": { includes the three main elements of the app’s in- "caption": { terface: , , and . header menu sections "font": "Roboto", The header section in Listing 3 defines the "size": "13", app’s header (the fixed bar at the top of the "align": "center", screen). In this case, the header contains a title "spacing": "15" and a menu item that opens the specified URL in }, the default browser. Note that in addition to the "image": { global styles, it’s also possible to define inline "width": "100%" styles for individual elements. For example, an in- } line style right after the title element defines the } appearance of the header title. The sections part }, contains an array of items. In this example, the ... { array consists of two items: The first one is the

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 71 LINUX VOICE JASONETTE

Listing 3: body Properties

"body": {

"header": {

"title": "Random Monkey",

"style": {

"font": "Roboto Bold",

"size": "25"

"menu": {

"text": "Unsplash",

"href": {

"url": "https://unsplash.com",

"view": "web"

"style": {

"font": "Roboto",

"size": "19"

}

"sections": [

{

"items": [

{

"type": "image", Figure 2: Random Monkey Android app in action.

"url": "https://source.unsplash.com/random/800x600/?monkey", cated directory on the server for storing photos "class": "image" and the JSON file (e.g., photoapp). Before you }, copy photos into that directory, you should resize { and recompress them. This will make your app

"type": "label", run faster and reduce the amount of data that

"text": "Here is a monkey photo", needs to be transferred. The perfect tools for the job are jpeg‑recompress [7] and ImageMagick’s "class": "caption" mogrify. With these two tools installed on your } system, you can use the following command to recompress or resize all JPG files in the current image, and the second one is its text label. And directory: that’s all there is to it. To test your first app, you don’t have to host for file in *.jpg; do jpeg‑recompress U the JSON file on your own server. Instead, you ‑‑quality high $file $file; done can use the jasonbase [6] JSON hosting service. mogrify ‑resize "1600>" *.jpg Create a new file, give it a name, paste the code, and save the changes. Make note of the created With all the pieces in place, you are ready to start JSON file’s URL, launch the Jason app on your working on the JSON file. At first glance, all you Android device, enter the JSON file’s URL, and need to do is to change the appropriate proper- press Go. This adds a new entry, and you can ties and add image and label items for each launch the app by tapping on it (Figure 2). photo. This quick-and-dirty solution has serious drawbacks, though. All URLs are hard-wired into Building a Photo Portfolio App the JSON file, so every time you add a new photo, With a few simple tweaks, you can transform the you need to create another item in the JSON file simple example app into a portfolio for display- manually. ing photos stored on your server. For this app, One way to solve this problem is to replace you need a web server for hosting a JSON file the JSON file with a PHP script that generates and photos. To keep things tidy, create a dedi- JSON data on the fly. Listing the entire PHP

72 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM JASONETTE LINUX VOICE

script is not very practical, so instead, use the $caption = U command rtrim(pathinfo($file, PATHINFO_FILENAME)); }

[email protected]:dmpop/U jasonette‑tokyo‑taxi.git Finally, the obtained and assembled data is used to generate a complete JSON file. to grab the source code of the Tokyo Taxi Lights [8] If you have a working knowledge of PHP, you app. Then open the json.php script that powers can modify the script to fit your particular needs. the app. Otherwise, you can modify the user-defined val- The script is not particularly difficult to parse. ues only. Copy the resulting PHP script to the web It starts by reading all files with the specified file server and start building your first Android app in extension (by default it’s JPEG) into the $files Android studio. array: Use the command

$files = array(); git clone https://github.com/Jasonette/U

foreach (glob($ext) as $file) { JASONETTE‑Android.git

$files[] = $file; } to clone the JASONETTE-Android app’s source code. Launch Android Studio by running the To pick a specified number of random photos, the android‑studio command in the terminal and script uses the shuffle() function: import the JASONETTE-Android project. Next, open the app/res/values/strings.xml file, lo- shuffle($files); cate the Jasonette $files = array_slice($files, ‑$number); line, and replace Jasonette with the desired name (Figure 3). Then, replace the de- The script then checks whether each photo has fault URL in an accompanying .txt file. If the text file is de- tected, its content is used as a caption; otherwise, https://jasonette.github.io/U the file name is used as the caption: Jasonpedia/hello.json

$txt = pathinfo($file, PATHINFO_FILENAME).".txt"; with the actual URL to your JSON file. if (file_exists($txt)) { Now prepare an image file in the PNG format for $caption = rtrim(file_get_contents($txt)); use as an app icon. In Android Studio, right-click } else { on the app | res folder and choose New | Image

Figure 3: Editing default Jasonette values.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 73 LINUX VOICE JASONETTE

Figure 4: Creating an app icon.

Asset. Use the prepared PNG file to create and nize the changes. Now, choose Build | Build APK(s) add an app icon (Figure 4). to build the app. Open the Gradle Scripts | build.gradle(Module: To install the resulting app‑debug.apk package app) file, locate the applicationId "com.jason‑ on an Android device (Figure 5), switch to Set- ette." line, and replace com.jasonette with the de- tings | Security and enable the Unknown Sources sired app ID. Press the Sync Now link to synchro- option.

Conclusion Using Jasonette, you can build your first simple Android app in less than an hour, but despite its deceptive simplicity, Jasonette packs advanced features that make it possible to develop and build sophisticated apps. Make sure to check the Ja- sonette documentation [9] to get the most out of this excellent tool. nnn

Info [1] Jasonette: jasonette.com [2] Android Studio: developer.android.com/studio/ index. html [3] Jason: play.google.com/store/apps/details? id=com.jasonette.jason [4] JSON online editor: jsoneditoronline.org [5] Elegant JSON Editor: addons.mozilla.org/ en‑US/ firefox/ addon/elegant‑json‑editor [6] jasonbase: www.jasonbase.com [7] jpeg-recompress: github.com/danielgtaylor/jpeg‑archive [8] Tokyo Taxi Lights: gitlab.com/dmpop/jasonette‑tokyo‑taxi [9] Jasonette documentation: Figure 5: Tokyo Taxi Lights app. docs.jasonette.com

74 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM DOGHOUSE – SUBUTAI LINUX VOICE MADDOG’S

DOGHOUSE Jon “maddog” Hall is an author, educator, computer scientist, Be a catalyst and help others break the bonds of proprietary and free software pioneer who has been a passionate software. BY JON “MADDOG” HALL advocate for Linux since 1994 when he first met Linus Torvalds and facilitated the port of Linux to a 64-bit system. He serves as president of Linux Open source catalyst International®.

ver the past 50 years, I have worked for many compa- pay the price of an upgrade (forced by an incompatibility with a nies and was a consultant for many government agen- new version of the operating system). O cies and nonprofits. Since July 2017, I have been the Of course there is also the social effect – all of the person’s CEO of OptDyn, the creators of Subutai open source peer-to- friends use one make of software and that person wants to fit peer cloud software. I wrote about Subutai in the February in, to be able to show their friends what they learned or ask 2018 column and thought I would add an update. questions about problems in hope of getting an answer. Or While the cost of creating a business or even launching a they want to use software that is very expensive, to fit into a new product for an existing business has gone down over time, certain crowd. it is still amazing how much it costs and how long it takes. If you are reading this article, then you are probably someone In chemistry, there is a term “catalyst,” a substance that who is outside of the realm of inertia and who has found your speeds up a reaction often without actually taking part in the catalyst a long time ago in free software. reaction itself. Platinum is a catalyst that is often used in fuel Despite the use of the “cloud,” there are still a LOT of local cells to allow the hydrogen and oxygen to fuse together, releas- machines used, or should we say “not used.” Many machines ing electricity with a byproduct of water. that are in our homes, offices, and schools are not used as effi- Open source is another catalyst, allowing companies to start ciently as they might be, and therefore we use more commer- up without needing huge investments in cash to buy closed cial cloud resources than we might otherwise use. source development tools, library licenses, databases, and This is a good time to investigate the use of Subutai open other components to the overall solution. source peer-to-peer cloud software. Set it up on your own ma- Open data and open culture are both additional catalysts for chine and help others in your community (whether it be at business, lowering the costs and speeding up the ability of school, work, or home) to use the resources you already have fledgling companies and products to come to market. for CPU, memory, and storage. It does not cost you anything You have to be careful, however, of the difference between and could save you a lot of money. “free” as in beer and “free” as in freedom. Often the thing that You can download it from subutai.io. If you wish to buy or sell starts out “low cost” limits what you can do later or has hidden resources you need or have extra, you can do that at one of the costs that become visible as the use increases. Subutai Bazaars on the net. In physics, Newton’s laws of motion state that a body at rest Subutai uses a lightweight digital currency called GoodWill tends to stay at rest and a body in motion tends to stay in mo- to buy and sell resources. You can also earn GoodWill by tion unless they are acted on by some outside force. The short- doing a number of other activities, such as writing Subutai hand name for this is “inertia.” Blueprints. We have inertia in the software world also. The law seems Blueprints are files that tell Subutai how to run programs, and to be written: “People who use some particular software will allow people to run these programs by clicking on the Blueprint continue to use that particular software unless some out- name just as you might click on any other application that you side force makes them use other software.” People use a have on your system. Blueprints are not very hard or time-con- particular piece of software because they are familiar with it suming to create. A Blueprint for Mattermost (an open source and how it works, and even if there is another piece of soft- clone of Slack) took only three hours to create. ware that is demonstrably better, they still stick with “the There may be a good business in helping people use their software they know.” own machines more efficiently, or helping them move their ap- What does it take to make them break the mold and go to an- plication to a cloud model that allows them to use resources other piece of software? Perhaps the software company that from anywhere. Go to the Subutai website and learn how to made the software goes out of business or the person has to write a Blueprint. nnn

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 75 LINUX VOICE NITROSHARE In All Directions

This cool tool helps you exchange files and directories between computers with Linux, Mac OS, Windows, and Android. BY FERDINAND THOMMES

here are many tools for exchanging data the project’s website [1], and a DMG package is on heterogeneous local networks. Nitro- available for Mac OS from version 10.8. T Share promises to make this task as easy as possible. In addition to Linux, NitroShare also Immediately Ready for Operation supports Mac OS, Windows, and Android. After installation and an initial start from the re- In this article, my test included a network of spective menu, the program is installed in the sys- computers running Debian “Sid,” Windows 10, tem tray (Figure 1). From there, it offers the ex- Ubuntu, Ubuntu MATE, Fedora, and the Android change of files and directories with computers on app. Users with Debian Sid, Ubuntu, Fedora, the LAN on which the software is also installed. Gentoo, and openSUSE will find the software in NitroShare automatically detects the correspond- their respective repositories; for Arch Linux ing devices and offers them as transfer targets. users, check out the AUR user archive. If neces- The software is ready for use immediately after sary, compile the program for other distributions installation; no further configuration is required. In from the source code. the Settings menu, you have the option of configuring a number of settings depending on your dis- Qt5 as a Basis tribution and desktop (Figure 2 and Figure 3). Packages for Nautilus (Gnome), Nemo (Cinna- Checking Auto start after login causes the pro- mon), and Caja (MATE) are available for integra- gram to launch whenever the system is booted. Au- tion in file managers. Although NitroShare is tomatic detection of other devices can take up to based on the Qt5 framework, not all distributions five minutes the first time. At the end of this phase, have appropriate extensions. Packages for inte- the tool displays a message with the results. gration into Dolphin and Konqueror were only In the Settings dialog, you can also specify a available for openSUSE as dolphin-plugin-nitro- name other than the hostname for each device and share and for Fedora as nitroshare-kservice. Fe- a default folder for downloads in the General tab. dora also offers a plugin for use in the terminal. Under Advanced, you can change the port, buffer, The installation on Debian and Fedora is quickly time out, and interval, if necessary. done (Listing 1); the package itself is only 150KB with just one dependency on a small 40KB Qt web server. If you do not want to use the Personal Package Archive (PPA) in Ubuntu, use the same command as in Debian. For Windows, executa- bles for 32- and 64-bit systems are available on

Figure 1: NitroShare is located in the toolbar’s system area Figure 2: NitroShare offers more options in Settings under under Debian. Debian than anywhere else.

76 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM NITROSHARE LINUX VOICE

Problems Listing 1: NitroShare Installation on Debian and Fedora In my lab, I looked at # Debian: Linux-Linux and apt update Linux-Windows con- apt install nitroshare nections, and vice # Fedora versa, as well as An- sudo dnf install nitroshare droid-Linux and An- # Ubuntu‑PPA: droid-Windows con-

sudo apt‑add‑repository ppa:george‑edison55/nitroshare nections. NitroShare

sudo apt‑get update failed to cooperate

sudo apt‑get install nitroshare with a workstation on

sudo apt install nitroshare‑nautilus Debian Sid, with fail-

# Arch from the AUR: ures in both direc-

wget https://aur.archlinux.org/cgit/aur.git/snapshot/nitroshare.tar.gz tions and an error

tar xf nitroshare.tar.gz message regarding

cd nitroshare SSL that did not make

makepkg ‑sri sense. The problem resulted in a bug report on GitHub [3]. Encryption Is Possible The application worked perfectly in all combi- Although not normally necessary on your own nations and in all directions on Debian Sid, network, the software also offers the possibility to encrypt transfers via Transport Layer Security (TLS) in the Security tab. A description of the corresponding configuration (Figure 4) can be found on the Nitro- Share wiki [2]. The menu also offers the Send Files and Send Directory options, which allow you to select files or directories and then specify the desired transfer destination. During data transfer, the soft- Figure 3: Ubuntu MATE offers some options that aren’t present in Debian. ware displays its history. You can close this dialog without affecting the transfer. After the transfer has completed, you see a message to this effect. NitroShare does not limit the size of the data sent and can thus theoretically achieve high speeds on networks designed for this purpose. Behind the scenes, the program uses IPv4 broadcasts to send User Datagram Protocol (UDP) packets to other devices on the LAN. These contain information that allows other devices to initiate transmissions without having to worry about broadcast addresses and the like. By default, NitroShare sends all UDP packets on port 40816 to any active network interface with an IPv4 broadcast address. You can change the port individually in the settings, if Figure 4: NitroShare offers the possibility of encrypting necessary. data in transit using TLS.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 77 LINUX VOICE NITROSHARE

Ubuntu, and Windows notebooks and on an An- not being able to access the Debian Sid worksta- droid 8.1 smartphone. Controlling the application tion might not be the fault of the program itself. directly with its own tools could not have been Nevertheless, it would be practical to copy the (in easier. this case long) error message instead of typing Integration with the Nautilus file manager on it. The reason the integration in Nautilus and Caja Ubuntu (Figure 5) and Caja on Ubuntu MATE did did not work was still unclear after completing not work: The context menus did not show any the test. corresponding entries. Despite these few problems, NitroShare does I measured the speed of NitroShare with a what it promises: After the install, you can get 1.5GB file, which I sent wirelessly with Debian Sid started immediately without having to configure and Ubuntu three times in both directions be- any settings. Transfers take place quickly. tween the two notebooks. It reached transmission Android devices are limited to the sender role; rates between 90 and 110KBps. With many small the software does not offer mobile devices as tar- files totaling around 700MB, the data rate re- gets on other clients. To do this, some Android mained roughly the same. apps offer the option of sending files directly via NitroShare. During the test, I found corresponding Conclusions entries in Google Docs, Gallery, and Photos, but Work on NitroShare has been in progress for six there may be more of them. Nevertheless, as far years now, so it would appear that any teething as Android is concerned, KDE Connect – which is problems have now been eliminated. However, even available on Gtk desktops – seems to be the this was not always the case in the test, although more mature solution. nnn

Info [1] NitroShare: https:// nitroshare.net [2] TLS: https://github. com/ nitroshare/ nitroshare‑ desktop/ wiki/TLS [3] Bug report: https://github.com/ nitroshare/ nitroshare‑desktop/ issues/155 Figure 5: Ubuntu offers a PPA with up-to-date versions of NitroShare. However, embedding in Nautilus does not work.

LINUX VOICE NOOTKA String Theory – Fast guitar training for classical music

With Nootka and your guitar, you can learn to recognize musical notes in classical notation and train for precision and speed.

BY HARTMUT NOACK inux has many small tools that allow music musical notation, which is something JACK does enthusiasts to practice reading musical no- easily. If you select singing mode, it is also possible L tation. Nootka [1], by developer Tomasz Bo- to practice with a keyboard, for example, by con- jczuk, offers an extensive program that helps gui- necting a software synthesizer like Yoshimi or Calf tarists learn notation (Figure 1). to the Nootka JACK port. The program, which runs on many platforms (see the “Installation” box), trains basic musical Classical Notes knowledge and the ear. The software can only dis- Nootka covers the whole range of classical Euro- play and play whole notes and is thus not suitable pean music, as well as rock and pop, although the as a tool for composing music. Nootka does not restriction to simple keys in major and minor allow for odd metrics, rhythms, and subtleties might become boring in the long run. Additionally, such as triplets, nor does it support chords. In- the software lets you analyze and practice ecclesi- stead, it helps you concentrate fully on learning astical modes, 12-tone compositions, and other the difficult art of recognizing notes in classical special scales (i.e., scales containing notes whose notation. combined use in chords and melodies creates a Figure 1: With a variety of The application is integrated into the JACK pro- matching harmonic effect) by freely entering the exercises, playful interac- fessional sound system, which makes it easy to notes instead of the defaults. tions, and motivating evalu- connect musical instruments to the program. Some Localization of the program shows a love of de- ations, Nootka helps guitar- tasks require you to play or sing notes displayed in tail: You can adjust the conventions for naming ists learn classical notation. notes to various international languages, which the program then takes into account (Figure 2), and translates the user interface to an acceptable quality, although gram- matical errors sometimes slip through. The program teaches notation for all notes used in equal temperament [2] and pure intonation [3] tuning, although Nootka is mainly geared to standard guitar tuning. If you use a software synthesizer such as Yoshimi, you can use pure intonation, with the correct distinction between tones such as E-sharp and F, and learn and practice the correct notation for such tunings. In single-note mode, you can display enharmonic equivalents [4], wherein some notes have different names on different scales, although they sound the same in temperament tuning (as is customary with piano and guitar). Thus C-sharp on the

80 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM NOOTKA LINUX VOICE

Installation

Nootka relies on Qt5 and is available on many platforms, including Linux, Windows, BSD, and Mac OS. Also, you can install a Nootka version for Android via an APK, which you can find on the website and in the Google Play store. The package managers of common Linux distributions offer the software, and the current stable version is available as a DEB, RPM, and archive for Arch Linux. If you don’t want to use the package manager, you can download an AppImage of the current stable version from the website. Al- though it started under Ubuntu Studio 16.04 Figure 2: In the basic settings, you can decide whether without any problems, it didn’t run as stably Nootka should use AHC notation (e.g., as in Germany) or as the other packages. Crashes occurred the ABC standard. The Italian DoReMi nomenclature also when changing settings for the user interface can be configured. and audio port. To build a current version from source, you on your previous knowledge, you can reconfigure not only need build tools like Make and GCC, the tutorials as required (see the box titled “A but also the latest development packages for Question of Settings”). Qt5 and Cmake. Under Ubuntu 16.04, the Although easy levels are available from the beginning to test your knowledge, beginners should three-phrase cmake, make, and make install ran without problems for the current develop- first choose exercise rather than exam. Such tutorials do not set a time limit and simply ask one ment version used in the test (1.4.4.4). question after another on one subject. You can disable the somewhat inconvenient guitar is identical to B, but can actually be played way of asking the next question and confirming with a different frequency on a violin. the answer in the toolbar by pressing the Space- The supposedly simple concept of Nootka thus bar and Enter keys instead. After a short delay, turns out to be a simple training system for solid the program automatically asks new questions basic knowledge, which can be used for very de- and accepts the inserted or played note as the manding compositions (Figure 3). answer. You can quit an exercise at any time by pressing the Stop button in the upper left corner. Step by Step Clicking on Analyze shows an overview of the The tasks within the scope of training prove to be quality of answers given and the time required quite demanding: Even for the simplest level, (Figure 4). which is all about empty guitar strings, you need During exercises, an automatic function pro- to know the position of the six notes on the stave. poses to test the same topic with tougher specifi- The staff shows no preview of the note values in cations when the answers reach a certain quality Figure 3: Nootka can dis- exercise and exam modes. If a question asks you (Figure 5). If you answer most of the questions rel- play simple notation with to put a note on the fretboard at the bottom, a pre- atively quickly during testing, you will be shown a letters along with standard view would be useful for beginners cute little certificate. notation, if desired. If you don’t have any experience with notation, you should first use a function that is not necessarily obvious in the basic view of the program: Nootka displays detailed information about the notes played on a connected instru- ment, which allows beginners to explore the connections between notes played on a guitar and notation in free mode. The program recog- nizes each tone with remarkable accuracy and displays it as a note. The tutorials provided are all based on the assumption that you know the position of the notes in sheet music; it is all about training precision and speed, rather than learning the basics. Depending

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 81 LINUX VOICE NOOTKA

A Question of Settings The overall appearance and behavior of Nootka largely depends on the settings for the selected level and those in Appearance. For example, in a level that is set to single note, the software only displays one note in the stave, with the exact name in the box to the right. Levels set to Melody present long lines with multiple notes, all of which are whole notes, but can assume pitches available in classical notation. You can hide many elements of the interface to make room for others that expand accordingly. The sheet music and – in single note mode – the table of notes remain visible. In the level editor, you can select or complete training tasks of different difficulty levels and Figure 4: The analysis shows thematic focus. In this way, you can decide hints about the meaning of Alternatives whether a tutorial should contain exercises for the curves when you mouse If Nootka doesn’t quite meet your requirements, single notes only or for melodies. The soft- over an element. other projects offer additional possibilities. For ware explicitly points out contradictory selec- example, the Java program ScoreDate [5] offers tions and does not offer an automatic mecha- considerably less customization, but also allows nism to resolve conflicts, which means you rhythm training, including drum notes. Addition- have to resolve the problems yourself. ally, it does not limit its musical notation to whole notes. The last update of the program was in 2012, but the package from Ubuntu “Universe” Conclusions works fine. All in all, Nootka appears logical and easy to use Impro-Visor [6], which is not necessarily suit- after familiarization with its extensive dialogs. able for beginners, is aimed primarily at ambi- The simple, clear design and the friendly re- tious jazz musicians, offering a wide range of sponses offer motivation for long exercises. The practice possibilities. The program allows you free mode also lets you create notations for your to compile virtual backing bands for your own own melodies. nnn improvisations and comes with more than 400MB of musical material on board. Music ex- The Author amples by well-known jazz legends such as Dizzy Gillespie and Charlie Parker let you ana- Hartmut Noack works as a lecturer, author, and lyze the stylistic compositional methods of the musician in Celle and Hanover, Germany. He Figure 5: When your perfor- genre. Contrary to what might be expected from finds that free software and homemade music mance reaches a certain level the outdated website, the developers regularly go great together. You can find some Creative in exercises, Nootka offers to maintain Impro-Visor. The current version is Commons-licensed results of his work with administer an exam. from June 2017. free music software at http://lapoc.de.

Info [1] Nootka: http://nootka.sourceforge.net [2] Equal temperament tuning: https://en.wikipedia.org/wiki/Equal_ temperament [3] Pure intonation tuning: https://en.wikipedia.org/wiki/Just_intonation [4] Enharmonic equivalents: https://en.wikipedia.org/wiki/Enharmonic [5] ScoreDate: http://scoredate.sourceforge.net [6] Impro-Visor: https://www.cs.hmc.edu/ ~keller/ jazz/improvisor/

82 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM

LINUX VOICE FOSSPICKS

Sparkling gems and new releases from the world of FOSS Picks Free and Open Source Software Graham tears himself away from updating Arch Linux to search for the best new free software. BY GRAHAM MORRISON

Media player VLC 3.0 here can’t be many Linux links. These prominently list iOS, taken so long to get its now includes support for users who haven’t heard Android, Mac OS, Windows, and best new feature: the abil- the latest home-theatre T of or installed the vener- even Windows Phone, while ity to natively browse net- buzzwords, including the able VLC. For many years, it’s Linux users are challenged by work filesystems such as ability to play back high-dy- been an essential installation re- asking their favorite packager Samba, SFTP, NFS, and namic range video and am- quirement, even if you don’t play rather than providing a direct FTP. You’ve always been bisonics 3D audio, if you back movies on your Linux ma- download. There’s not a snap or able to paste an accessi- have the hardware configu- chine. It’s just as good at playing flatpak in sight, reflecting the ori- ble URL into the network ration to make sense of YouTube videos and for listen- gin of the majority of VLC users. file requester, but you these. ing to Internet radio, which is Fortunately, for most distribu- needed your desktop or This is where another why the release of version 3 is tions, the latest release can in- background operating great addition makes more such a milestone; it’s perhaps deed be installed with a simple system to have mounted sense. You can now send one of the biggest updates to sudo snap install vlc without a remote filesystem first video directly from your one of the most popular open even asking your packager. or delivered the data via desktop VLC to a Chrome- source projects out there. This It’s surprising that something its own transport mecha- cast plugged into your tele- is reflected in the download called the “VideoLAN client” has nisms. As most of us like vision. This is a fantastic to store our media on re- upgrade simply because mote devices, such as a Linux (and open source!) NAS, this was always a lacked an easy and reliable problem. Now VLC can ac- way of doing this. It’s a cess these files without great way of storing and any extra help, and it managing your movie col- makes a big difference, lection on the go. And if especially on Android. you do plug your Linux box While the VLC user inter- directly into the TV or am- face looks largely the same plifier, you can now choose as version 2 – complete to passthrough high-defini- with an even larger set of tion audio codecs, such as advanced settings – it can E-AC3, TrueHD, and DTS- play back far more media. HD. Because this is a ver- In particular, thanks to a sion that’s going to be 1 Browse networks: You can now access remote locations directly within VLC. considerable number of around for some time (it’s 2 Chromecast support: You can’t see it, but these menus hide the option to OpenGL acceleration addi- an LTS release), you can beam your video to your TV. 3 360-degree video: Play fully immersive video tions on Linux, you can build your system around without resorting to Chrome. 4 Playlist improvements: Stated simply, they now now play 4K and 8K videos Linux and VLC and know it work, unlike with version 2.5. 5 Adjust pitch: This new real-time audio effect is at 60 frames per second, will remain working perfect for learning Prince’s guitar solos. 6 HDR, 4K, and 8K playback: Thanks to as well as 360-degree vid- through various updates. lots of new OpenGL acceleration, Linux can now be part of the extreme bandwidth eos, such as those re- and storage party. 7 Cross platform: The same VLC will run on your Android device, corded for virtual reality Project Website your Linux box, and even any other computers you may have. headsets. Video playback https://www.videolan.org

84 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM FOSSPICKS LINUX VOICE

Menu editor MenuLibre 2.1.5

ven though many of us latter is important because it don’t use a launch menu means you can use the same tool E for every application to edit your launch menu regard- launch, they’re still useful to see less of desktop environment or what’s installed and for accessing distribution. forgotten things. This means that This is what MenuLibre does. It their associated menu editors has been tested with Budgie, aren’t the most exciting part of the Gnome, KDE (Plasma), LXDE, LXQt, desktop. In fact, there’s a good MATE, Pantheon, Unity, and Xfce, chance you’ve either never loaded and it lets you edit almost anything one up to play with or haven’t done about the applications you find in so for a very long time. But they do the launch menu equivalents for Get the same menu edi- perform an important function, es- those various desktops. You can tor regardless of which menu headings up or down pecially when you’re at the mercy change the icon for an application, desktop environment through the hierarchy or sort al- of your chosen distribution’s menu for example, or set whether it runs you use. phabetically. It’s a great tool that hierarchy, or worse, the whims of within a terminal or is hidden from does everything you need, on al- applications like Wine that seem menus. You can also set the com- most any desktop, which is a great to hook them into every sub-folder. mand to launch the application it- advantage if you like to make A menu editor needs to be quick, self and test the validity of the menus your own. powerful enough to accommo- desktop file generated by MenuLi- date your launch requirements, bre when you save your changes. Project Website and most importantly, Desktop General menu management is also https://bluesabre.org/2018/02/06/ Entry Specification compliant. The possible, with buttons to move menulibre‑2‑1‑5‑released/

Process monitor htop 2.1.0

here are few tools more most critical asset is that it’s still effective than the venera- being updated, as this release T ble top command for pro- shows. Version 2.1.0 is a great cess monitoring and control. upgrade on what was already an This is because it can be run excellent 2.0 release, adding pro- from almost anywhere that has a cess monitoring geekery such as Linux command line and gives delayed accounting metrics for you an immediate and real-time processes, DragonFly BSD sup- overview of what processes are port, improved meter styles for htop looks even better running; how much memory, the visual overview of CPU when you use split tmux that can be used to change the swap, and CPU they’re consum- usage, and better battery moni- sessions to monitor a sort mode, kill processes, and ing; and the ability to kill a pro- toring on Linux. network of machines! get help. There are lots of other cess by simply selecting it and is- Launched just like top, and shortcuts, too – such as press- suing a command. Except that with shortcuts and arguments ing c to see threaded processes top is perhaps a little too venera- that offer the same kind of func- or s for integrated strace. It’s ble at times, with its minimalist tions, most new users can sim- quick, powerful, and takes very and austere user interface writ- ply continue with this new tool few resources. If you haven’t al- ten for 1984-era Unix rather than just as they would with the old ready switched from top to htop, the cloud droplets of the 21st one. Firstly, they’ll get a much version 2.1.0 is the perfect time century. htop is a drop-in replace- more reactive curses-based in- to do so. ment for this fabulous com- terface and overview of what’s mand. While it adds lots of fea- running. Unlike top, there’s a Project Website tures lacking in the original, its function key-driven menu system http://hisham.hm/ htop

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 85 LINUX VOICE FOSSPICKS

Software modem Minimodem

ven before the Internet modems may be gone, but the was accessible outside of sounds they made can live again – E universities, thousands of all thanks to Minimodem. us wanted to connect our com- Minimodem can generate puters to each other. The best those exact modem sounds, di- solution was to use a modem rectly from the command line. But connected to your serial port to it’s far more than a piece of soft- connect to a privately run bulletin ware nostalgia, as it can even de- board system (BBS), using the code those sounds. This makes it Even if you don’t have a modem through its serial port. likes of FidoNet to communicate a pure software emulation of one modem with which to Simply connect an audio cable in across these federated portals, of those old hardware units and send and receive signals, lieu of a telephone line and con- download files, and share up- gives your machine the ability to Minimodem generated nect the modem at the other end, dates. The modem was at the communicate with any BBS you sound is almost its own and you’ll be able to launch your heart of this experience, turning can still find running, or perhaps music genre. own local BBS. Minimodem even the raw data sent from the serial that Amiga 1200 you left plugged lets you turn audio recordings of a port into an audio stream that into the phone line at your par- session into the data the session could be transmitted across a ent’s place. It’s even useful if you would have generated, which may telephone line. Like loading data have old hardware like the Amiga be a great way to reverse engi- from a tape recorder and com- hanging around, as you can use neer WOPR in War Games. pact cassette, the sounds emit- Minimodem to communicate ted by a modem as it negotiated with a computer that perhaps Project Website a connection became ingrained doesn’t have any networking ca- http://www.whence.com/ into our subconscious. Those pability but can still talk to a minimodem/

Terminal desktop Texttop

he screenshot might The difference with Texttop is look like it’s a screen that it’s interactive – it’s not T grab from a ZX Spec- simply broadcasting a passive trum emulator running a copy of stream. This is a big deal be- Hacker, but it’s actually way cause it fills in a missing gap on more advanced than that. It’s an the command line – the ability X session framebuffer being to run applications designed for rendered in a console session, graphical interaction. The web with the framebuffer itself show- browser is the perfect example, Stuck on a low band- ing Firefox with the Linux Voice because while command-line width connection and use at this stage, the proof of page open. This kind of thing alternatives like links2 are ex- need to access a web- concept does work, and the au- has been done before, with cellent for occasional use, such site? Try piping Firefox thor is working on a major revi- video or image data being piped as downloading new graphics through Texttop. sion. You can launch a remote through a graphical tool that will drivers after a kernel update browser within Docker and use turn the imagery into ASCII text. has destroyed your configura- Mosh to access it via SSH. It’s possible to do this quickly tion, they’re not every good with From there, you can view your enough that video input can be- most websites. For those, you remote session and interact come a recognizable video out- need a fully fledged browser with it using the mouse just as put, albeit rendered in text. And with proper JavaScript, such as you would locally. because it’s text, you can easily Firefox. Texttop lets you access view the stream from anywhere a remote Firefox session with remote terminal access, through a regular terminal. Project Website such as an SSH terminal. While it’s almost impossible to https://github.com/ tombh/texttop

86 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM FOSSPICKS LINUX VOICE

Storage analyzer ncdu 1.13

here are several great du, and it’s so effective that ways to check how you’re unlikely to want to resort T much storage space you to either a graphical or a com- have remaining and where stor- mand-line alternative. Its main age is being used. One of the feature is speed. Launch the best GUI-driven applications is command and within seconds Filelight as this gives a rapid vi- (storage medium permitting) it sual overview of your complete will populate a list of files and Celebrating more than 10 years of updates, the best feature in the filesystem organized by size. folders relative to your current new version is the inclusion of color! The bigger the file, the more it location with a hash-character consumes a pie-chart represen- bar graph used to visualize this is its speed, as you can tation of your directory struc- which take up the most space. quickly skip through the filesys- ture. On the command line, how- Pressing g switches the graph tem and instantly see where all ever, you’re typically left quite lit- between this mode and a per- the space is being used. It’s per- erally to your own devices. Com- centage indicator and then both. fect for a remotely connected mands like du and dh can be Pressing d will scarily delete the Raspberry Pi or any other re- used with a variety of argu- selected file or directory (after a mote server and is quicker than ments to present the same kind warning), and pressing b will trying to do the same thing from of data as Filelight, albeit with- spawn a shell in your new loca- the desktop. out the color. But there’s another tion. All these keyboard short- option, and that’s ncdu. cuts can be accessed by press- As its name implies, ncdu is an ing the ? key within the utility. Project Website ncurses-driven replacement for But the best thing about all of https://dev.yorhel.nl/ncdu

Screenshot taker Flameshot

t’s amazing that, for a simple Flameshot avoids these pitfalls tool, the humble screenshot and has even innovated over Hot- I utility hasn’t found perfection Shots. There’s a separate configu- and ascended to ubiquity years ration tool for setting things like ago. We could all agree that the the GUI colors, the buttons that copy-to-clipboard approach of appear, and exactly how screen- the PrntScr key on Windows shots are named when saved. You needed improvement, as could can then take screenshots with or the name of Scrot, but it’s been without the GUI appearing, but it’s harder to find innovation since the GUI that makes this such a Flameshot must be the Spectacle took over from KSnap- good tool. When Flameshot is most comprehensive can with HotShots, but apart from shot. But there’s a new con- launched and you select a screen screen tool to support that, it’s perfect. As it’s Qt-based tender, and it’s gone straight to area to grab, that area is immedi- Gnome Wayland and rather than KDE-based, it’s also the top of the pile. Flameshot ately highlighted and then aug- Plasma Wayland. desktop agnostic and even sup- gets nearly everything right, and mented by a series of buttons ports Gnome Wayland and more importantly, it does this (configured earlier) that let you an- Plasma Wayland alongside the without getting anything wrong. notate, sketch, heal, resize, draw usual suspects. If there is one The brilliant HotShots gets close, lines, and undo any changes you thing it’s missing, it’s a delay pe- for instance, with its ability to an- make. You can do all of this before riod. Otherwise, it’s perfect. notate and edit images before the image is automatically saved they’re saved. But saving is con- or uploaded. The only function it’s Project Website voluted enough to make it a pain really missing is the ability to cre- https://github.com/ lupoDharkael/ to use, despite the advantages. ate automatic annotations like you flameshot

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 87 LINUX VOICE FOSSPICKS

Text editor nedit-ng

his wouldn’t be FOSS- resulted in a crash. But the new Picks if we didn’t look at editor also updates the old visu- T least one text editor, and als without losing the Motif-like this month it’s nedit-ng’s turn. simplicity. It’s built using Qt 5, What makes nedit-ng special is which can look just like Motif if that it’s a complete and faithful you let it, but by default looks a port of the Nirvana Editor (NEdit) lot better. It will integrate with v5.6. This is an editor that’s been your desktop theme, for exam- around since the dawn of time, ple, and the colors and syntax or X windows, depending on highlighting don’t look out of which came first, and it still has place on a modern desktop. The Edit text like it’s 1992 syntax highlighting to make your a popular following because of new version even uses anti- with the complete code easier to read. In fact, the the austerity of its early-90s user aliased fonts. rewrite of the Nirvana syntax highlighting is the prod- interface and its C-like macro It’s what many people would Editor. uct of some carefully crafted language. The last update to the call a distraction-free editor, be- regular expressions, which original version was in 2014, cause it doesn’t waste space means you can edit them your- when version 5.6 was released, with modern additions like an self if you’ve got the skills. It’s and it’s this version for which object viewer or dependency just like moving from Vi to Vim, nedit-ng is built to be a drop-in manager. It’s clean and crisp, which must be exactly what the replacement. The main reason with split views, line numbering, authors planned. for this port was to fix some simple statistics, and tabs ap- long standing bugs in the origi- pearing if you need them. Project Website nal, some of which reportedly There’s also some excellent https://github.com/ eteran/nedit‑ng

Internet blocker Chomper

his sentence is being Chomper is basically a Python typed with the help of script that blocks access to a T two random Internet group of Internet sites taken searches, one YouTube video in- from a blacklist. It can also en- termission, and a brief chat with able access for only a specific a remote acquaintance. All amount of time, and mix up thanks to reasonably fast and times, blacklists, and whitelists cheap broadband. And this is according to the profile you want how most sentences are now to use for whatever length of Go on an Internet diet “hardcore mode.” This is really being written, and not just on time you want to use it. All of this and get your productive just good advice for running these pages. One side effect of is done through an easy to un- life back with Chomper. Chomper – set up an admin ac- pervasive Internet that most of derstand YAML configuration count, give it an impossibly long us could never have anticipated file. Create a social profile and password, lock this away some- is its ability to suck the life out of block Facebook, for example, or where on paper, and give this ac- attention spans – thirty seconds just enable access to your count permission to run the work, thirty seconds checking GitHub page. It works far better script. With that done, you won’t social networks. But it doesn’t than a dynamically updated be able to stop Chomper until the have to be this way, and like any hosts file, which often accom- time is up, and that’s when good food diet, cutting back on plishes little more than hiding the Chomper works best. fatty food or distractions is most cookies in a different drawer. The effective when there’s some project’s readme also offers ex- Project Website third-party oversight. Chomper is cellent advice for Linux users https://github.com/ aniketpanjwani/ that oversight. with root access for enabling chomper

88 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM FOSSPICKS LINUX VOICE

Game engine Godot 3.0

n recent years, Godot has realms, and Godot 3.0 is by far gone from being an inter- its biggest release yet. I esting side project for peo- Godot includes an editor, ple wanting to write their own both 2D and 3D games engines, open source games, to a seri- and animation tools, and it’s ous alternative to game en- completely open source and gines like Unity and Unreal. cross platform. This major new This is important because the release has taken 18 months to games industry, in particular, is get to this point. During this still driven by proprietary time, the community surround- games written on proprietary ing the project grew to such an platforms. While there’s abso- extent that its developers cre- lutely nothing wrong with this, ated a Software Freedom Con- Download open source/ This new version feels like a having a “free software” plat- servancy-ratified Patreon page, free software assets and completely different piece of form that attempts to imple- which has in turn funded the examples directly within software. Elements such as the ment a similar system is won- full-time employment of one of Godot 3.0. integrated asset download tool derful for all kinds of reasons. its core developers alongside and asset workflow (complete It means, for example, that ed- an additional developer to work with licence listing) and a fan- ucators can teach games de- on a new rendering back end tastic set of new documenta- velopment without the cost as- and the brilliant new GDNative tion help make this feel like sociated with platform licens- for this release. GDNative is a commercial software; there are ing, or even worse, being re- way for Godot to dynamically too many new features to list. stricted in how the games cre- use external libraries without Perhaps the strongest new fea- ated with education versions recompiling the engine. It ture is the new 3D renderer, can be used or sold. It means means, for example, you can which includes the full princi- students have no cost barrier dynamically support various pled BSDF shader. We covered for entry, and it means game audio and video codecs, and the details of this fantastic developers who want to release build games for virtual reality or shader when it was added to every line and asset of their Microsoft’s Kinect without hav- Blender, and the results are creation can do so without ing to build a special binary for equally stunning here. Along- complicating the situation by each use case. GDNative also side this comes global illumina- building atop a “free” platform. allows for the use of extra tion with a mode called GI that Godot has grown to fill this pur- scripting languages, such as probes for real-time lighting pose, much like Blender or Ar- the very popular (in gaming cir- and a classical lightmapping dour have in their respective cles) Mono/C#. workflow for slower machines. The end results are a major upgrade of Godot’s appearance and form a cornerstone of the new version being used for pro- fessional quality production. New tone mapping, GPU particles, and support for GLTF 2.0 (the latest standard of file formats for 3D scenes) close the deal. The end result: If you’ve been looking to get into Linux game development but thought there was nothing to compete with the likes of Unity, Godot is now worth your time and money.

There has never been a better time to get into game development, and you can do Project Website it for free with Godot and its huge community of supporters. https://godotengine.org

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 89 LINUX VOICE TUTORIALS – GPSD

Location, Location, Location

Even though your computer does not come with its own GPS, you can hack one onto it using a mobile phone. Although, it does take quite a bit of fiddling.

BY PAUL BROWN here is plenty of literature online on how to again the hackiness of the solution, due to pro- connect a GPS device to a computer. But cessing data coming from a GPS device not on T normally the devices covered are exclu- board or that is not connected over a serial or USB sively GPSs, that is, something you would proba- port, which is what gpsd usually expects. (See the bly have to go out of your way to buy. “NMEA” box.) What you don’t find so much information about gpsd is available in most, if not all, distributions. is how to use what you have at hand, namely, your On Debian and its derivatives, it is installed with: regular phone. However; it can be done. It is a bit hacky, but it works. apt‑get install gpsd

On Your Laptop If you are on Ubuntu, Raspbian, or something Linux comes with a daemon called gpsd [1] that similar, you may have to add sudo to the preced- can process the information delivered by a GPS ing instruction. device. Although gpsd is usually started and Be that as it may, this command will install stopped with systemd (and is usually launched at (and run) the GPS daemon and install several boot), you’ll be running it by hand. The reason is utilities such as gpsmon and gpspipe. The first thing you want to do is stop the sys- NMEA temd gpsd services. As explained previously, you want to run gpsd with your own parameters. You GPS devices deliver information in the shape can later work your special configuration into the of National Marine Electronics Association systemd version, but for now, run these two in- (NMEA) packages. These packages that con- structions to free up the gpsd service and the port form to the NMEA’s 0183 or 2000 standards on which it listens. contain all the data you need to position a GPS device on the Earth’s surface. systemctl stop gpsd

NMEA packages contain “sentences” that systemctl stop gpsd.socket look like this: If this doesn’t do the trick, check what systemd is GPRMC,092751.000,A,5321.6802,N,00630.3371,U running with: W,0.06,31.66,280511,,,A

An NMEA sentence can contain data on the systemctl | grep gpsd device’s altitude, latitude, longitude, and speed. It also indicates the time of the read- and stop whichever service shows up there. ing at the beginning of the sentence and a You may have to remove the /var/run/gpsd.sock checksum at the end. This helps identify bro- file as well (you will have to do that with sudo) if ken or partially received sentences. you are going to run gpsd as a regular user. Most programs and programming libraries On Your Phone used for positioning parse NMEA pack- On your phone, you need an app that shares your ages and sentences automatically. This GPS data over a network. There are some quite means you usually won’t need to write any fancy proprietary programs out there, but, unless code to turn the raw NMEA data into you go for the “premium” version, they are usually something usable. addled with ads and bloat.

90 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM TUTORIALS – GPSD LINUX VOICE

Fortunately, there is a unlikely that gpsd will poll very simple option created anything that isn’t the de- by Tiago Koji Castro Shi- vice you want it to poll. bata that you can get from Now move around with the Google Play app store your phone, preferably [2] or download and com- near a window, to try and pile yourself from the au- capture some coordi- thor’s GitHub page [3]. nates. Then open an- When you run it, it looks other terminal window like Figure 1. and run gpsmon. Configuring the app is The gpsmon utility does simplicity itself: Input your what it says on the box: laptop’s IP. Choose a port It monitors GPS input. to which to forward the in- Without any parame- formation. Press the Start ters, it monitors what button. The end. gpsd is receiving, but it can also monitor a Connecting Both stream of GPS data Back on your laptop, from other sources, in- check that your IP corre- cluding the Internet. Its sponds to what you input text-based interface on the mobile app and run shows information on the gpsd daemon from a the satellites in view terminal window: (the GSV section in Fig- ure 2), data which pro- gpsd ‑N upd://*: vides 3D location and Figure 1: A simple GPS client that allows you to share accuracy (the GGA sec- Figure 2: The gpsmon utility What you are doing here is data with your computer. tion), and so on. shows you GPS in action. polling information from a phone, any phone, on the port you established on the app. To all practical effects, your phone is acting like a server. For example, if you are connecting to an app configured as shown in Figure 1, you would run:

gpsd ‑N upd://*:29998

The ‑N parameter lets you run gpsd without it going into the background. Once you are happy that everything works as it should, you can run gpsd without it. As for why “any” phone and not your own spe- A healthy GPS connection will look like Figure 2. cific phone, that’s because, in theory you could se- If you see something different, wait a while to see lect a specific source on the network by passing if the stream starts, or move your phone to a loca- your phone’s IP to gpsd, like so: tion where it can “see” satellites, near or outside a window. If that doesn’t work, restart gpsd, gpsmon, gpsd ‑N udp://: the app on your phone, or all three. GPS is a fickle beast and breaks a lot. If you have followed the in- However this did not work for me. The only way structions above, at some point you will get a sig- gpsd managed to pick up the server on our phone nal, and the stream of data will begin. was by using the catch-all *. This shouldn’t be a big This is all very well if you are at home and run- problem: Pick an atypical and unique port, and it is ning stuff over the local network, but you would

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 91 LINUX VOICE TUTORIALS – GPSD

usually want to use your case of a run, you could also see the terrain’s pro- GPS-enabled system file and figure out how much you climbed, since when you are out and the NMEA sentences also include the altitude at about. What then? Use any given point in your route. your phone as a WiFi I could end the article right here. You have con- hotspot (Figure 3)! Just nected your phone; you can use its GPS on your because your phone is laptop or on a single board computer such as the the hotspot, it doesn’t Raspberry Pi; and you have learned how to store mean it cannot stream the data. With that alone, you can carry out a GPS data to your laptop. bunch of cool projects. None of which, useful as Activate Mobile Wi-Fi they may be, will be very visually appealing. hotspot from Settings | So let’s make a graphical map-tracking applica- Wireless & networks | tion instead. More | Tethering & mobile hotspot on your Mapping Out phone, connect to your The objective is to show a map of the GPS de- phone’s network from vice’s location, and, as it moves, have the map your laptop, and your move with it (see Figure 4). A circle indicates computer will be as- where the device is located on the map. signed an IP that is Your first reaction would probably be to base something like your app on some open source map app that al- 192.168.43.X (for the re- ready exists. Marble [4] seems like a good candi- cord, phones usually date, and if you run it like so adopt 192.168.43.1 as their IP when acting as a marble ‑‑latlon , ‑‑distance 2 U hotspot). Use that with ‑‑map "earth/openstreetmap/openstreetmap.dgml" your Android GPS client, and you will be able to where lat and lon are the latitude and longitude, take your GPS experi- Marble will open at the desired location, which is Figure 3: Activate the Mobile Wi-Fi hotspot feature on your ment out and about. already promising. phone and take your GPS rig on the road. A final gpsd tool to be Then, if you think that aware of is gpspipe. At its simplest, gpspipe takes a parameter that tells it gpspipe ‑w | sed ‑n s/.*lat:(.*),/1/p | U which format to use and just dumps the stream cut ‑d, ‑f1 from gpsd to the standard output: will give you the latitude of your phone, and gpspipe ‑r

gpspipe ‑w | sed ‑n s/.*lon:(.*),/1/p | U will stream raw NMEA sentences to the terminal, cut ‑d, ‑f1 while will give you the longitude, things definitely seem gpspipe ‑w to be shaping up. The preceding instructions will give you an on- sends native gpsd sentences (see Listing 1). going stream of latitudes and longitudes. It would gpspipe can also output to a file using the ‑o pa- be possible to pipe the output to a file and then rameter, be sent to the background as a daemon run through the rows and use them with Marble (‑d), or send data to a serial device (‑s). somehow. But this is an inelegant and convoluted You could use gpspipe to send all the informa- way of doing things. It is much smarter to break tion to a file and keep track of a trip or a run. In the out QML [5].

Listing 1: gpspipe ‑w output {"class":"TPV","device":"udp://*:29998","mode":3,"time":"2018‑03‑11T09:21:21.000Z","ept":0.005,"la t":36.736763683,"lon":‑4.381267583,"alt":149.000,"epv":20.700,"track":0.0000,"speed":0.000}

{"class":"SKY","device":"udp://*:29998","vdop":0.90,"hdop":1.50,"pdop":1.70,"satellites":[{"PRN":7 ,"el":26,"az":45,"ss":45,"used":false},{"PRN":13,"el":45,"az":298,"ss":24,"used":false},{"PRN":2 8,"el":35,"az":113,"ss":18,"used":false},{"PRN":30,"el":51,"az":46,"ss":38,"used":false},

...

]}

92 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM TUTORIALS – GPSD LINUX VOICE

QML App scribing it in text form, and then you do the same QML is a relatively simple declarative language with all the items contained in the layout. To exe- that uses the Qt libraries to build graphical appli- cute actions (say, when a button is pressed), you cations. How QML works is a bit like how you’d associate events with JavaScript modules to go about programming an interface using a vi- each object. sual IDE, but with text: You “design” the layout de- Getting back to the original idea, the Marble documentation says that it is easy to develop apps based on the Marble code in QML. It refers the would-be programmer to a tutorial and examples available at [6]. Unfortunately, the code examples and snippets are outdated, and it is quite hard to figure out how to do things. Ac- cording to one developer, the problem is that Marble’s code is in continuous flux, changing nearly daily. Instead, he suggests using QML’s own Location library. To get started, install all the bits and pieces you’ll need:

apt install qmlscene qml‑module‑qtpositioning U

qml‑module‑qtlocation

The modules will install positioning and location libraries for your app and drag in all the Qt QML Figure 4: Your app will show the position of your phone on a common files. The qmlscene utility is what you use map. As it moves, the red circle will follow it around. to run QML apps.

Listing 2: map.qml 01 import QtQuick 2.1 27 circle.border.width = 3;

02 import QtQuick.Window 2.0 28 myMap.addMapItem(circle);

03 import QtPositioning 5.5 29

04 import QtLocation 5.6 30 circle.center = pos.coordinate;

05 31 myMap.center = pos.coordinate;

06 Window { 32

07 id:page 33 //console.log("Coordinates: ", pos.coordinate.

08 width: 800 latitude, pos.coordinate.longitude);

09 height: 800 34 }

10 visible: true 35 }

11 36

12 Map { 37 Plugin {

13 id:myMap 38 id: mapPlugin

14 anchors.fill: parent 39 name: "osm"

15 plugin: mapPlugin 40 }

16 zoomLevel: 17 41

17 42 PositionSource {

18 property MapCircle circle 43 id: gpsPos

19 44 updateInterval: 500

20 function update(pos) { 45 active: true

21 removeMapItem(circle); 46 nmeaSource: "socket://localhost:29999"

22 47

23 circle = Qt.createQmlObject('import QtLocation 5.3; 48 onPositionChanged: { MapCircle {}', page); 49 myMap.update(position);

24 circle.radius = 30; 50 }

25 circle.color = "transparent"; 51 }

26 circle.border.color = "red" 52 }

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 93 LINUX VOICE TUTORIALS – GPSD

The code for your app might look like Listing 2. know what makes sockets super easy? Netcat [8] After loading in all the modules you need does. Netcat (nc) is preinstalled in nearly every (lines 1-4), you create an all-encompassing Linux installation on the planet, so it was simply a Window object (line 6). You give it a name (page), matter of doing this: set its size, and make it visible. Next you create a Map widget. Again, you give it a gpspipe ‑r | nc ‑l 29999 name (myMap), you set its size to automatically fit in Window (the parent object), and you set the plugin it Then, create a stream of GPS data on localhost will use. In this case, the plugin is declared on using port 29999, and get PositionSource to read lines 37 to 40, where you can see you will be using from it, as shown on line 46. the OpenStreetMap [7] data for your maps. Finally, Finally, you set up an event that fires when the you set the zoom level of the map, in this case, position you are polling changes (line 48). When it 17 meters up in the air. does fire, you visit the update(pos) function in the Map objects allow you to draw shapes on the Map object (lines 20-34) and redraw the circle and map. If you want a circle, you create a circle ob- the map with the new position data, pos.coordinate, ject on line 18. You then manipulate it using a Ja- that you pass in the function call. vaScript function (lines 20-34) and moving it (line To summarize, here’s what you do to make ev- 30) when the GPS position changes. You also erything work together: move the view of the map so that the circle is always in the center (line 31) . 1. Run gpsd udp://*.29998. Line 33 is for debugging and testing only. Un- 2. Connect your phone’s GPS client to your laptop comment it to see coordinates scroll by on the ter- using your laptop’s IP and port 29998. minal window as you move around. 3. Run gpsmon to check you are receiving GPS data As mentioned previously, QML is declarative correctly. and event-based and you can see how that works 4. Run gpspipe ‑r | nc ‑l 29999 to set up a data in the PositionsSource object (lines 42-49). This is stream through port 29999 on your computer. an object provided by the QtPositioning module, 5. Run your application with qmlscene map.qml. and it is in charge of retrieving and processing the GPS information. Conclusion As usual, you give your object a name (gpsPos) There you have it: A full GPS app on a laptop (or to refer to it later, you set how often it will poll Raspberry Pi) in 50 lines of code [9]. This is defi- the GPS source (once every 500 milliseconds), nitely a fun project, and it combines hardware and you set from where it will get the NMEA you probably already have hanging around at information. home in an interesting way. Expanding to build A slight detour here, because it turns out that, an actually useful application should be a piece unless you have GPS on the laptop or a GPS-only of cake. device connected via USB (that is, not a phone), Famous last words. nnn PositionSource does a poor job of determining where it should get its data. Info However, you can force it to look for data by set- [1] gpsd: http://www.catb.org/gpsd/ ting the nmeaSource attribute. You would usually nmeaSource to point to a file with static NMEA data [2] gpsd client on Google Play: to “simulate” a GPS positioning (nmeaSource is pri- https://play.google.com/store/apps/details? marily used when testing), but you can also point id=io.github.tiagoshibata.gpsdclient to a server. This got me thinking: First I tried to [3] A (free) gpsd client for Android: pipe the output from gpspipe to a file and read https://github.com/tiagoshibata/ from that: Android‑GPSd‑Client [4] Marble: https://marble.kde.org/ gpspipe ‑r > gps.dat [5] QML: Line 46 then looked like this: https://doc.qt.io/qt‑5.10/qtqml‑index. html [6] Marble for developers: nmeaSource: "gps.dat" https://marble.kde.org/dev‑intro.php [7] OpenStreetMap project: But then PositionSource was never updated – it https://www.openstreetmap.org/ just took the last value from the file and never went back to check if there were any more. [8] Netcat: http://nc110.sourceforge.net/ A bit more research revealed that Position‑ [9] Video of the app in action: Source can also be read from a socket URL. And https://youtu.be/ jcHOY07Rvpk

94 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM

SERVICE Events FEATURED EVENTS Users, developers, and vendors meet at Linux events around the world. We at Linux Magazine are proud to sponsor the Featured Events shown here. For other events near you, check our extensive events calendar online at http://linux-magazine.com/events. If you know of another Linux event you would like us to add to our calendar, please send a message with all the details to [email protected].

openSUSE Conference Texas Linux Fest ISC High Performance 2018 Date: June 8–9, 2018 2018

Date: May 25–27, 2018 Location: Austin, Texas Date: June 24–28, 2018

Location: Prague, Czech Republic Website: https://2018.texaslinuxfest.org/ Location: Frankfurt, Germany

Website: https://events.opensuse.org/ Texas Linux Fest is a community-run Website: https://www.isc-hpc.com/ conference/oSC18 conference for Linux and open source SC High Performance is dedicated to The annual openSUSE community software users and enthusiasts. Texas tackling HPC technological develop- event that brings people from around Linux Fest is a weekend event geared to- ment and its application in scientific the world together to meet and collab- ward individual users. Whether you use fields, as well as its adoption in com- orate. The organized talks, workshops, free software and Linux at home, in your mercial environments. This year the and BoF sessions provide a framework place of business, in your school or non- program team has selected 13 topics, around more casual meet ups and hack profit, or you are simply curious, Texas which will be addressed in a number of sessions. Linux Fest offers something for you. thought-provoking sessions.

Events heise Security Tour 2018 April 10, 12, 18, Numerous European cities https://www.heise-events.de/securitytour 24, and 26, 2018 International Javascript April 11–13, 2018 London, England https://javascript-conference.com/program/ Conference 2018 2018 HPC for Wall Street April 16, 2018 New York, New York http://www.flaggmgmt.com/linux/ Cloud Foundry Summit April 18–20, 2018 Boston, Massachusetts https://www.cloudfoundry.org/event/ North America nasummit2018/ Linux Presentation Day 2018.1 April 21, 2018 Europe-wide in many cities http://www.linux-presentation-day.org/ Linux Storage Filesystem and April 23–25, 2018 Park City, Utah http://events.linuxfoundation.org/events/ Memory Management Summit lsfmm-2018/ LinuxFest Northwest April 28–29, 2018 Bellingham, Washington https://linuxfestnorthwest.org/ conferences/lfnw18 KubeCon + CloudNativeCon May 2–4, 2018 Copenhagen, Denmark https://events.linuxfoundation.org/events/ kubecon-cloudnativecon-europe-2018/ openSUSE Conference 2018 May 25–27, 2018 Prague, Czech Republic https://events.opensuse.org/conference/ oSC18 Texas Linux Fest June 8–9, 2018 Austin, Texas https://2018.texaslinuxfest.org/ CeBIT 2018 June 11–15, 2018 Hannover, Germany http://www.cebit.de/en/ Open Source Data Center Conf. June 12–13, 2018 Berlin, Germany https://osdc.de/ ISC High Performance 2018 June 24–28, 2018 Frankfurt, Germany http://www.isc-hpc.com/ DebConf 18 July 29–August 5, 2018 Taipeh, Taiwan https://wiki.debconf.org/wiki/DebConf18 Akademy 2018 August 11–17, 2018 Vienna, Austria https://akademy.kde.org/2018 DevOpsDays Berlin September 12–13, 2018 Berlin, Germany https://www.devopsdays.org/events/ 2018-berlin/welcome/ Images © Alex White, 123RF.com

96 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM SERVICE Contact Info / Authors CALL FOR PAPERS We are always looking for good articles on Linux and the The technical level of the article should be consistent with tools of the Linux environment. Although we will consider what you normally read in Linux Magazine. Remember that any topic, the following themes are of special interest: Linux Magazine is read in many countries, and your article • System administration may be translated into one of our sister publications. There- fore, it is best to avoid using slang and idioms that might • Useful tips and tools not be understood by all readers. • Security, both news and techniques Be careful when referring to dates or events in the future. • Product reviews, especially from real-world experience Many weeks could pass between your manuscript submis- • Community news and projects sion and the final copy reaching the reader’s hands. When If you have an idea, send a proposal with an outline, an esti- submitting proposals or manuscripts, please use a subject mate of the length, a description of your background, and line in your email message that helps us identify your mes- contact information to edit@linux-magazine.com. sage as an article proposal. Screenshots and other support- ing materials are always welcome. Additional information is available at: http://www.linux-magazine.com/contact/write_for_us.

NOW PRINTED ON recycled paper from 100% post-consumer waste; no chlorine bleach is used in the production process.

Contact Info

Editor in Chief While every care has been taken in the content of Joe Casad, [email protected] the magazine, the publishers cannot be held respon- Managing Editor sible for the accuracy of the information contained Rita L Sooby, [email protected] within it or any consequences arising from the use of Localization & Translation it. The use of the disc provided with the magazine or Ian Travis any material provided on it is at your own risk. Authors News Editor Copyright and Trademarks © 2018 Linux New Media USA, LLC. Erik Bärwaldt 16, 22, 64 Swapnil Bhartiya Copy Editor No material may be reproduced in any form what- Swapnil Bhartiya 8 Amy Pettle soever in whole or in part without the written per- Paul Brown 90 Layout mission of the publishers. It is assumed that all cor- Dena Friesen, Lori White respondence sent, for example, letters, email, Zack Brown 12 faxes, photographs, articles, drawings, are sup- Cover Design plied for publication or license to third parties on Bruce Byfield 34, 60 Lori White a non-exclusive worldwide basis by Linux New Andy Carlson 52 Cover Image Media USA, LLC, unless otherwise stated in writing. © Vasily Pindyurin, 123RF.com Joe Casad 3 Linux is a trademark of Linus Torvalds. Advertising Mark Crutch 69 Brian Osborn, [email protected] All brand or product names are trademarks phone +49 89 3090 5128 of their respective owners. Contact us if we Alexander Fillbrunn 46 haven’t credited your copyright; we will always Marketing Communications correct any oversight. Jon “maddog” Hall 75 Gwen Clark, [email protected] Linux New Media USA, LLC Printed in Nuremberg, Germany by hofmann Martin Horn 46 616 Kentucky St. infocom GmbH on recycled paper from 100% Charly Kühnast 45 Lawrence, KS 66044 USA post-consumer waste; no chlorine bleach is Publisher used in the production process. Christoph Langner 42 Brian Osborn Distributed by Seymour Distribution Ltd, United Vincent Mealing 69 Customer Service / Subscription Kingdom Martin Mohr 56 For USA and Canada: LINUX PRO MAGAZINE (ISSN 1752-9050) is pub- Email: [email protected] lished monthly by Linux New Media USA, LLC, Andreas Möller 30 Phone: 1-866-247-2802 616 Kentucky St., Lawrence, KS, 66044, USA. Periodi- Graham Morrison 84 (Toll Free from the US and Canada) cals Postage paid at Lawrence, KS and additional Fax: 1-785-856-3084 mailing offices. Ride-Along Enclosed. POSTMASTER: Hartmut Noack 80 Please send address changes to Linux Pro Magazine, For all other countries: 616 Kentucky St., Lawrence, KS 66044, USA. Dmitri Popov 70 Email: [email protected] Published monthly in Europe as Linux Magazine Mike Schilli 38 www.linuxpromagazine.com – North America (ISSN 1471-5678) by: Sparkhaus Media GmbH, Ferdinand Thommes 76 www.linux-magazine.com – Worldwide Zieblandstr. 1, 80799 Munich, Germany.

LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 210 MAY 2018 97 NEXT MONTH Issue 211 Approximate May 05 UK / Europe Jun 01 USA / Canada Jul 02 Issue 211 / June 2018 Australia Mycroft On Sale Date

Shopping for a voice assistant is like so many other things in the high tech industry – a loss of privacy is often part of the purchase price. If you really want to explore the realm of voice technology, but you’re not interested in letting Google and Microsoft data-mine your home life, try Mycroft – the open source voice assistant.

Preview Newsletter The Linux Magazine Preview is a monthly email newsletter that gives you a sneak peek at the next issue, including links to articles posted online. Sign up at: www.linux-magazine.com/newsletter

98 MAY 2018 ISSUE 210 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM