CONTENTS
Dear Readers, Here is the November issue. We are happy that we didn’t make you wait for it as long as for October one. Thanks to contributors and supporters we are back and ready to give you some usefull piece of knowledge. We hope you will Editor in Chief: Patrycja Przybyłowicz enjoy it as much as we did by creating the magazine. [email protected] The opening text will tell you What’s New in BSD world. It’s a review of PC-BSD 9 by Mark VonFange. Good reading, Contributing: especially for PC-BSD users. Next in section Get Started you Mark VonFange, Toby Richards, Kris Moore, Lars R. Noldan, will �nd a great piece for novice – A Beginner’s Guide To PF Rob Somerville, Erwin Kooi, Paul McMath, Bill Harris, Jeroen van Nieuwenhuizen by Toby Richards. In Developers Corner Kris Moore will teach you how to set up and maintain your own repository on a Proofreaders: FreeBSD system. It’s a must read for eager learners. Tristan Karstens, Barry Grumbine, Zander Hill, The How To section in this issue is for those who enjoy Christopher J. Umina experimenting. Speed Daemons by Lars R Noldan is a very good and practical text. By reading it you can learn Special Thanks: how to build a highly available web application server Denise Ebery with advanced networking mechanisms in FreeBSD. The Art Director: following article is the �nal one of our GIS series. The author Ireneusz Pogroszewski will explain how to successfully manage and commission a DTP: complex GIS project. Text not only for GIS series followers! Ireneusz Pogroszewski We closed the section with article: Equip Your CA With a Senior Consultant/Publisher: HSM For <50 Euros by Erwin Kooi. You may take a closer Paweł Marciniak [email protected] look at the security of Certi�cate Authority together with the CEO: author. Ewa Dudzic Now, some stuff for Admins: Terminals Served Up BSD [email protected] Style by Toby Richards and OpenBSD Kernel Memory Production Director: Pools: Monitoring Usage With Systat by Paul McMath. The Andrzej Kuca �rst article presents two solutions for those who want to [email protected] establish a BSD terminal server. The second one is addressed Executive Ad Consultant: to more advanced users and explains how to understand Ewa Dudzic memory usage statistics for kernel memory pools as they [email protected] are displayed by the systat(1) command on OpenBSD. Advertising Sales: In Let’s Talk you will �nd a comparison of FreeBSD 8.2 Patrycja Przybyłowicz [email protected] and Ubuntu Server by Bill Harris. Worth reading to confront your own opinions or to gain some as well. We say farewell Publisher : Software Press Sp. z o.o. SK with EuroBSDcon Overview written from an organizers ul. Bokserska 1, 02-682 Warszawa perspective by Jeroen van Nieuwenhuizen. Check what you Poland have missed or see the event you participated in from a worldwide publishing tel: 1 917 338 36 31 different perspective. www.bsdmag.org
Software Press Sp z o.o. SK is looking for partners from all over Wish you a good read and hope for a feedback! the world. If you are interested in cooperation with us, please contact us via e-mail: [email protected] Patrycja Przybyłowicz All trade marks presented in the magazine were used only for & BSD Team informative purposes. All rights to trade marks presented in the magazine are reserved by the companies which own them.
Mathematical formulas created by Design Science MathType™.
4 11/2011 Contents
What’s New Admin PC-BSD 9 Turns a New Page Terminals Served Up BSD Style 06Mark VonFange 30Toby Richards Since 2005, PC-BSD has striven to make BSD accessible You may have your own reason for wanting a BSD to the desktop user. Offering a slew of improvements and terminal server. There are two solutions to this goal: added features, this release brings a more accessible FreeNX or XRDP. This article will show you how to use BSD operating system than ever. Read the review and both solutions. find out more about it. OpenBSD Kernel Memory Pools: Get Started 34 Monitoring Usage With Systat Paul McMath A Beginner’s Guide To PF This article explains how to understand memory usage 10 Toby Richards statistics for kernel memory pools as they are displayed OpenBSD, FreeBSD, and PC-BSD use a built-in firewall by the systat(1) command on OpenBSD. called “Packet Filter”. This article is intended for a PF beginner to get a beginner’s understanding of how to use PF in OpenBSD. Developers Corner Creating Your Own PBI Repository 12 Kris Moore In this article author describes the steps required for setting up and maintaining your own repository on a FreeBSD system. How To Speed Daemons Let’s Talk 14 Lars R. Noldan From this article you will learn how by using advanced FreeBSD 8.2 Against Ubuntu Server networking mechanisms in FreeBSD build a high 38 Bill Harris performance, highly available web application server. An Objective Comparison of Two Power House Open Source Server Platforms, BSD Unix and Linux. A GIS Strategy For Web-Enabled 20 Business EuroBSDcon Overview Rob Somerville In his final article of our GIS series, the author will show 42 EuroBSDcon 2011 From An Orgnizers you how to successfully manage and commission a Perspective complex GIS project. Jeroen van Nieuwenhuizen Have an inside look at the EuroBSDcon and get to 24 Equip Your CA With a HSM For <50 know about events and speeches that took place in the Euros beginning of October 2011 in Netherlands. Erwin Kooi The CA is used for identification and authentication of servers, clients and users. Together with author take a closer look at the security of Certificate Authority in his own network. WHAT’S NEW PC-BSD 9 Turns a New Page PC-BSD 9 Turns a New Page Since 2005, PC-BSD has striven to make BSD accessible to the desktop user. Offering a slew of improvements and added features, this release brings a more accessible BSD operating system than ever.
ontinuing on the successful features of prior versions, a small barrier to new users, and power users should have like the intuitive Install Graphical User Interface no issues navigating through any of the available options. C (GUI), its PBI system and its comprehensive For ease of use, all of these desktop managers are configuration menu, PC-BSD 9 brings added functionality available for installation by simply clicking check-boxes and greater choices for its users. during installation. No additional work is required for the user. The manager of choice is automatically installed and Greater Options For The Desktop runs on the first reboot. Another great feature is that these The most obvious and significant change from prior versions managers are also available after initial installation. All you is the addition of nine new desktop managers. Since it’s need to do is open up the PC-BSD Control Panel and Run inception, PC-BSD has utilized KDE as it’s default and only the System Manager Utility. This brings up a SuperUser fully supported desktop manager. While many consider password prompt. From there you can uninstall and reinstall KDE to be the most robust managers available, there are desktop managers at will from the System Packages tab substantial numbers of users that prefer others for various and just reboot to switch to the new manager. reasons. This is very limiting and one of the great things This gives the user a high degree of control over their about running an open source operating system (OS) is the system without requiring a great deal of technical knowledge. flexibility it provides. Also, a lot open source users generally New users will need to be careful when switching to the find their desktop manager of choice and stick with it due unsupported desktop managers, as not all features are to familiarity. Others like to experiment and change things available via the UI. Even if a user does switch to a manager around. Up until recently, if you wanted to run PC-BSD, that they aren’t comfortable with, simply running the pc-controlpanel meant you were forced to use KDE. Even then, you had to command in a terminal will bring up the PC-BSD Control Panel jump through hoops to get utilize any other options and you for them to switch to another. For those willing to persevere would lose some of the functionality built into PC-BSD. with those desktop managers, the PC-BSD Wiki page outlines That is all changing in version 9. The development the various steps needed to run unsupported utilities. team has separated the toolchain out from KDE and now everything is running in the Qt framework. The first four A Revamped Packaging System available desktop managers are GNOME, KDE, LXDE, One of the staples of PC-BSD is it’s Push Button Installer and XFCE. All of these are fully supported in PC-BSD, or PBI feature. With ease of use in mind the PBI system meaning all utilities have been integrated into the desktop makes installing applications a simple point-and-click environment itself. The other five available desktop operation. Similar to the .exe and .dmg file formats of managers are Awesome, Enlightenment, IceWM, ScrotWM Windows and OSX, PBI’s are self-contained applications and WindowMaker. While these are not fully supported that take the user through the install process via a GUI within the desktop environment, all the PC-BSD utilities are system and avoiding any need for utilizing the Command still available, though you may need to run commands via Line Interface (CLI). This greatly helps reduce the learning the Command Line in order to get them running. This is just curve for users to install their favorite applications.
6 11/2011 www.bsdmag.org 7 WHAT’S NEW PC-BSD 9 Turns a New Page
Another issue the PBI system is designed to solve is to the PBI installation system under version 9 is that users dependency conflict and breakage. Different applications can install most programs without a root password. This sometimes require different versions of the same will be beneficial for systems with multiple users. dependency. This means that installing one package would may break the functionality of another. In order to avoid More Intelligence, More Versatility this, users would have to use workarounds, which can be Beyond improvements to the installation system, the daunting to those who don’t know their way around the CLI. PBI system has been revamped is in regard to updating One problem this caused was that each individual PBI applications as well. Before, upgrading a PBI involved had to have all of it’s own libraries and dependencies downloading the entire new package. From PC-BSD 9 contained within itself. This caused redundancy and and beyond, when PBI’s are upgraded, only changed substantially increased the necessary size of the programs files will be downloaded. This means saved time in as well as runtime memory. PC-BSD 9 has revamped the downloads, which will especially help those on slower PBI system to utilize intelligent checks on the back-end of connections. In addition, the Upgrade Manager itself has pbi’s via a hash database in order to determine whether been redesigned. In order to facilitate kernel updating, the needed libraries and dependencies already exist. PC-BSD will now utilize the GENERIC FreeBSD kernel. This made it possible to avoid the redundancy issue and, This means that, in addition to the GUI, users will be
consequently, lightened up the programs a great deal. able to upgrade via built-in freebsd-update command. This These back-end checks go so far as to recognize when means that user accounts won’t need root privileges to a library is no longer needed due to an uninstall. This run upgrades. Furthermore, the Upgrade Manager will creates a lighter PC-BSD with less bloat for the user. enable users to update the FreeBSD world and kernel, as Not only does PC-BSD 9 solve the problems of well as packages, ports, PBI’s and security updates. redundancy and breakage, it also revamped the way As mentioned earlier, PC-BSD utilities now run PBI’s are downloaded and installed. This new method is a independently of the desktop. This has enabled a whole feature called AppCafe(TM). Prior to version 9, users had to new Control Panel that runs regardless of which manager go to pbidir.com in order to search for and download PBI’s. you are running at the time. From it, you can access all AppCafe simplifies this whole process by integrating the manner of system settings, even the utilities for desktop PBI library into the desktop environment itself for an easy managers you aren’t currently running. Along with this to use experience. This utility has added sanity checks new control panel comes a long overdue improvement to automatically detects your system architecture and installs display configuration. Up until version 9, changing display the proper PBI for you. Programs can be searched for by resolution has required rebooting your system. With the name via the search bar or you can browse through the new control panel, all you need to do is click on the Display conveniently categorized menu. Simply double click on icon in the hardware section of the menu. This will bring the desired program and it will populate to the installed tab a prompt indicating that the current session will be exited where you can view it’s download and installation status. If and you will be brought to a display configuration menu you want to search for more programs, just switch back to after asking for your root password. From there, you can the Browse tab and repeat the process. Another change adjust your display resolution, color depth and select the
Figure 1. PC-BSD Control Panel offers a GUI method of changing Figure 2. AppCafe(TM) brings improved ease of use to the PBI destop managers after installation installation system
6 11/2011 www.bsdmag.org 7 WHAT’S NEW
best driver for your video card. On the subject of display A Look Under The Hood of PC-BSD drivers, Nvidia drivers are now available for install via the While offering intuitive GUI-based menus for virtually all install GUI as well as the control panel. The advanced tab aspects of system management, PC-BSD is still offers the of the display menu allows users to set up options such as full benefits of FreeBSD as well. Starting at version 7, PC- refresh rates and dual-monitor configuration. In addition, BSD releases have been tied in with FreeBSD releases. It once GEM/KMS support for FreeBSD is finalized, PC- has many added features, but PC-BSD utilizes the current BSD will offer improved Intel / ATI graphics support. FreeBSD source tree as its basis. This means that when If those improvements were not already significant you see a new major release of FreeBSD, you know that enough, there are even more key additions to PC-BSD you are going to see many improvements and new features 9. The first of which is the ability to install directly to a along with it. One of the biggest new features in FreeBSD BootCamp partition. This should be a welcome addition for 9 is the addition of USB3 support, which means improved many mac users. Considering that OSX is largely based off performance of USB devices. Another advantage of FreeBSD of FreeBSD, this is a fitting addition to PC-BSD’s repertoire. is it’s ZFS support. The newest release has updated ZFS to The Life Preserver feature brings additional support for v28. This will likely be integrated into PC-BSD in version 9.1 system backups. While you can still simply add files to an or 9.2. ZFS is a robust open source file system with a high external via the file manager or install one of the backup degree of data integrity due to capabilities such as dynamic utilities available in AppCafe, Life Preserver provides an striping, variable block sizes and cache management, as integrated utility for both manual and automated system well as features such as encryption, deduplication and backups to any external device with SSH and rsync improved software RAID. FreeBSD 9 also offers multiple enabled. It provides the user with a comprehensive GUI improvements for multi-processor support including NUMA to handle things such as how often to back up, how many support and large scale SMP support as well as 4k sector backups to keep, configuring which files and directories to drives. These features bring enterprise capabilities to PC- back up, as well as the means to restore a backup. BSD unseen in most desktop oriented OS distributions. Several other additions are also available on PC-BSD 9. Other changes to FreeBSD include improved performance First off is a wireless configuration tool that lists available for virtual machines and laptops, better video support networks and allows for a quick connection. Second, Myth TV in linux compatibility mode, open source 3D hardware and XBMC are available during or after install for those who acceleration, and many others. These improvements all want to run media from their PC-BSD system. Touchscreen come on what is arguably the most stable and secure OS drivers are now available and the development team intends on the market, whether it’s open source or proprietary. for virtual keyboard and touchscreen capabilities from installation on at some point in the future. This means that A Stable, Secure System, With Ease of Use in Mind you may see PC-BSD running on tablets some day. Version All in all one can say that the PC-BSD development team has 9 also comes with Ipv6 enabled by default. Lastly, PC-BSD made many major improvements over previous versions. now includes a script to automatically detect USB devices for From additional features to improved system management, desktop managers that don’t do so on their own. There are PC-BSD 9 is an intelligent, versatile desktop operating other minor improvements, tweaks and bug-fixes of course, system that will bring BSD to the desktop for many new but those are the most noteworthy. users. With FreeBSD as a basis, one can rest assured that PC-BSD is a stable and secure operating system. With the intuitive features provided, users also gain many powerful tools for designing a desktop system that fits their needs. From new users to power users, home systems to enterprise systems PC-BSD 9 provides something for everyone.
MARK VONFANGE Mark VonFange is the Professional Services Manager at iXsystems, providing oversight and coordination of its FreeBSD, PC-BSD, and FreeNAS support and development services. The Professional Services Team provides services ranging from Figure 3. The Life Preserver Utility is opened by clicking the icon in the system tray, which launches a GUI based menu for adding new mission critical support to software and �rmware development backups.The picture shows the utility being opened in the XFCE to private consultation. Mark also develops internal and desktop manager external documentation for division sales and marketing.
8 11/2011
GET STARTED A Beginner’s Guide to PF A Beginner’s Guide to PF OpenBSD, FreeBSD, and PC-BSD use a built-in firewall called “Packet Filter” (PF for short). This article is intended for a PF beginner to get a beginner’s understanding of how to use PF in OpenBSD. Several important intermediate and advanced concepts are intentionally left out.
What you will learn… What you should know… • A bare bones understanding of the /etc/pf.conf �le for basic • The ability to navigate your BSD’s command line. security. • Some knowledge about TCP & UDP port numbers.
his guide may apply to other *BSD operating systems, /etc/pf.conf file. In most cases, you probably want to but I’ve not tested them all. I have tested PC-BSD, specify what is allowed, and then block everything else. T which means that these instructions will also work for To do this, we make liberal use of the quick keyword. For FreeBSD. More advanced topics may be OpenBSD specific. incoming traffic, let’s say that you only want to allow SSH & Neither NetBSD nor Dragonfly BSD use PF. HTTP. Add these rules: I find that the OpenBSD documentation often throws too much information at the beginner from the get-go. pass in quick on vic0 proto tcp to port 22 The PF User’s Guide is no exception. The following is a pass in quick on vic0 proto tcp to port 80 highly simplified subset of those instructions. The basic PF syntax that the beginner needs to know is this: Notice that I simply exclude those directives that I don’t need. In this case, I don’t need to specify which version action [direction] [quick] [on interface] [af] [proto protocol] \ of IP or where the traffic is coming from because I want to [from src_addr] [to dst_port] allow this traffic on both IP versions and from anywhere. Now, allowing pings is a little bit trickier. ICMP is a I have intentionally omitted more advanced/obscure options. different beast than TCP or UDP. We do have to specify the all keyword to mean from anywhere, to any port. We • action: This is either block or pass. Self explanatory. also have to specify the type of ICMP packet (echoreq): • direction: This is either in or out • quick: If a packet matches a rule specifying quick, pass in quick inet proto icmp all icmp-type echoreq then that rule is considered the last matching rule and the specified action is taken. Let’s also only allow certain outgoing traffic. This could • interface: Self explanatory. It’s vic0 on VMware. protect you in case a rogue user try to use an application • af: inet for IPv4 or inet6 for IPv6. that you don’t want to be used. For this example, we’ll let • proto: tcp, udp, icmp, or icmp6. our server use:
You can use the all directive in some places, but we’re not • FTP (for commands such as pkg _ add) going to talk about that other than for the final two rules in • SSH (remote access to other SSH servers)
10 11/2011 www.bsdmag.org 11 GET STARTED A Beginner’s Guide to PF
• SNMP (so that sendmail can send us notifications) • DNS (so that our server can resolve host names) References • HTTP (for using commands such as wget) • For the official list of TCP and UDP ports, we can check with the Internet Assigned Numbers Authority (IANA): http:// • Ping www.iana.org/assignments/service-names-port-numbers/ service-names-port-numbers.xml Here are the rules: • For more documentation on PF, consult your *BSD’s documentation: • FreeBSD & PC-BSD: http://www.freebsd.org/doc/en_US.ISO pass out quick on vic0 proto tcp to port 20 # Used by FTP in PASV mode 8859-1/books/handbook/�rewalls.html pass out quick on vic0 proto tcp to port 21 • OpenBSD: http://www.openbsd.org/faq/pf/index.html pass out quick on vic0 proto tcp to port 22 pass out quick on vic0 proto tcp to port 25 pass out quick on vic0 proto udp to port 53 # telnet bsdmag.org 22 pass out quick on vic0 proto tcp to port 80 Trying 79.125.23.174... pass out quick on vic0 proto tcp to port 443 pass out quick inet proto icmp all icmp-type echoreq Again, [CTRL+C] gets you back to a prompt. Now that we know what to expect on allowed ports, let’s try one To block all traffic that we haven’t allowed above, simply that isn’t allowed. Let’s try TFTP: add more lines to the end of the file: # telnet bsdmag.org 69 block out all Trying 79.125.23.174... block in all telnet: connect to address 79.125.23.174: No route to host
Now we see the importance of the quick directive. With it, We know that there’s a route to the host, so we must be the packet filter immediately applies the rule for matching blocking outbound TFTP connections with PF. That’s key. traffic (pass). Without it, the packet filter continues to When telnet tells you that there’s no route to a host that you
look for matches until the end of the rule set. Since block can otherwise connect to, then it’s time to look at /etc/pf.conf. out all & block in all match all traffic, we’d still be blocking What happens if you have a remote host, and you everything without quick. Could we have avoided all of accidentally lock yourself out of SSH? Well, just like my the business with quick by putting the block commands byline says, that’s one good reason to use bsdvm.com as at the top? Probably. I just find that using quick, and then a hosting provider. I can get to the VMware console to re- having the block commands at the bottom makes the enable SSH. Otherwise, I’d be up a certain creek which shall config file easier to read and understand. Now, simply do remain nameless. As a disclaimer: I am in no way affiliated
pfctl -f /etc/pf.conf to reload and test your rules. with bsdvm.com. I am just a fan. A big fan. Having console What happens when something goes wrong? If you think access is just so darned convenient for many, many tasks. that your firewall rules are causing problems, then comment In the upcoming 5.0 release of OpenBSD we can look
out your two block rules at the end of your /etc/pf.conf file. forward to using packet filter rules to prioritize certain Issue pfctl -f /etc/pf.conf to reload the rules, and see if that traffic. More on that later. For now, you ought to have a fixes things. The telnet command is also your friend. Notice grasp of the PF basics. Enjoy your new firewall! above that I have not allowed the telnet protocol (TCP port 23). The cool thing about the telnet utility is that it allows us to make a connection on any TCP port. We know that port TOBY RICHARDS 80 is HTTP, so we can test whether port 80 is working: Toby Richards has been a network administrator since 1997. He considers himself to be a jack of all operating systems, but a true # telnet bsdmag.org 80 master of none. He feels this to be a mastery in its own right since Trying 79.125.23.174... he understands principles that are common to all operating Connected to bsdmag.org. systems. His articles are the product of teaching himself to Escape character is '^]'. become better with OpenBSD and PC-BSD. He simply writes about what he has learned most recently. For a hosting provider, [CTRL+C] will get you back to a prompt. Now let’s try he highly recommends bsdvm.com. They give you access to your to connect to something that our firewall allows, but the VMware console so that you can re-install your OS at will, and remote host does not. We hang indefinitely at: with the settings of your own choosing.
10 11/2011 www.bsdmag.org 11 Creating Your Own PBI Repository With the major overhaul of the PBI system for the upcoming PC- BSD 9.0, it is now easier than ever to setup and run your own PBI repository, even on traditional FreeBSD systems.
his allows an individual or organization to maintain will be used to digitally sign your created PBI files,
and distribute their own set of software packages, and the pub.key will be included with the repository T in a secure and easy manner. In this article we configuration (.rpo) file. In order to begin creating your will take a look at the steps required for setting up and new repository you will need a FTP/HTTP/HTTPS maintaining your own repository on a FreeBSD system. location for hosting your repository’s meta files, as One of the first steps in building a PBI repository is well as a location(s) for clients to download PBI files. to first download and install the pbi-manager from the These can be a public internet URL, or private LAN FreeBSD ports tree: server, as long as it is accessible to your intended target audience. # cd /usr/ports/ports-mgmt/pbi-manager Once you have the location URL, the next step is # make install to create your repository .rpo file with the following command: After installing the pbi-manager, a variety of new command-line tools will become available for managing # pbi_makerepo --desc “My Example Repository” --key PBIs and repositories. Running a repository requires pub.key --mirror “ftp://ftp.example.org/pbi-files” --url that all the built PBIs be digitally signed with openssl, for “http://www.example.org/pbi-meta” /root security and identification purposes. The next step is to generate a public / private key pair with openssl using After running the command above, you will left with a the following command: pbi-repo.rpo file in the /root directory, or wherever you specified. # openssl genrsa -out privkey.pem 4056 This file is all that is needed for clients to register and # openssl rsa -in privkey.pem -pubout > pub.key begin using your new repository. On the client system, this would be installed with a single command: After running these commands, you will be left with two files, privkey.pem and pub.key. The privkey.pem # pbi_addrepo pbi-repo.rpo
12 11/2011 www.bsdmag.org 13 Creating Your Own PBI Repository
Once the repository is registered on the client system, the pbi daemon will automatically keep track of More Information on the Web downloading and updating both meta-files and PBIs from • PBI-Manager Homepage: http://wiki.pcbsd.org/index.php/ PBI_Manager the URL’s you included in the repository configuration • PBI Module Builder Guide: http://wiki.pcbsd.org/index.php/ file. Meanwhile, back on the server side, we need to start PBI_Module_Builder_Guide creating some meta-data and PBI files so the clients • PBI Developers Mailing List: http://lists.pcbsd.org/mailman/ have something to download and run. Luckily the pbi- listinfo/pbi-dev manager includes utilities which make the process quick and easy. The first file we can create is meta-data for the # pbi_makeport --sign privfile.key archivers/cabextract application(s) we wish to distribute via this repository. This meta-data is used to give the client information Once the build is finished, you will be left with a
about the applications available in your repository, such resulting cabextract-
The above commands will create your initial meta file, URL. In addition, we can also bzip2 the pbi-index-9, and which will need to be named pbi-meta-
application we are going to distribute. PBIs from your repository, using the pbi _ add command or the AppCafe GUI. # pbi_metatool add --app -n “cabextract” -c “Archivers” The pbi-manager includes a number of other tools and -a “Stuart Caie” -d “Utility for reading and extracting utilities for creating and managing PBI files. Repository .cab files.” -i “http://www.example.org/pbi-icons/cabextract.png” maintainers may also find the pbi_autobuild command -k “cab,archive,extract” -l “LGPL” -t “Text” -u “http: of interest, which monitors the ports tree, automatically //www.cabextract.org.uk” pbi-meta-9 rebuilding PBI’s when the respective port version is updated. This command can be used with optional
In this example we have added the Cabextract meta-data, module configurations, allowing additional make options, which includes the Author (-a), Keywords (-k), License desktop icons, helper scripts and other data to be used (-l), Type (-t) and others. For a more complete description in the building of your PBI files. Using this and other of all the available flags, you can run the command commands, managing a PBI repository becomes easy to
pbi _ metatool add. Now that we have some initial meta- initialize and maintain. All the supplied commands include data, we can compress the file with bzip2 and upload it man pages with additional tips and options for power to our meta location: http://www.example.org/pbi-meta/ users and additional information may be obtained via the pbi-meta-9.bz2. On the client system, they will be able to links below.
use the pbi _ browser command, or the AppCafe GUI (If on PC-BSD) to browse this meta-data. With the initial meta-data in place, we can now begin the process of building PBI files. In this example we will KRIS MOORE be building the cabextract package from the FreeBSD Kris Moore is the founder and lead developer of PC-BSD. He lives ports tree. This command will expect that you have the with his wife and four children in East Tennessee (USA), and FreeBSD ports tree already extracted and ready to be enjoys building custom PC’s and gaming in his (limited) spare used. time. [email protected]
12 11/2011 www.bsdmag.org 13 HOW TO Speed Daemons Speed Daemons
Using advanced networking mechanisms in FreeBSD to build a high performance, highly available web application server.
What you will learn… What you should know… • What CARP is and how to use it • How to use the ports system • Installing and con�guring ifstated • Basic understanding of the command line • Brief introduction to the ZFS �lesystem • Basic understanding of rc.conf • Understanding of TCP/IP networking
t’s no secret that web applications have recently been • System must support up to 50,000 registered users returning us full circle to a network computing model • System must integrate with other client-provided I where your computer is only a mechanism used to systems connect you to your applications. Gone are the days of a single i486 running hundreds of websites. Web application Design servers have grown more powerful and infinitely more The application used in this project is Plone, an enterprise complex. Reliability of these web applications is at least content management system capable of meeting the use as important, if not more so, than raw performance. The requirements of the client. In addition to Plone, an Apache following details a web application server cluster Six Feet
Up built for a Fortune 500 company specifically to address ���������� ���������� ���������� the issues of providing high performance, highly available web applications in a modern business setting. ������ ������ Requirements First, we need to review the requirements the application
needed to meet. �������� �������� ��������
• 99.9% application uptime ������������ ������������ • Pages load in under 10 seconds to users on 1.5Mb/s internet connections • User registration processing in under 20 seconds on ��������� ��������� ��������� 1.5Mb/s internet connections ��������� ��������� System Tools: carp(4), zfs(1M), nfsd(8), sftp(1), […] Major Ports: zope, solr, postgresql, nginx, varnish, haproxy, ifstated, […] ��������� ��������� ��������� External Packages: plone, […] Figure 1. Logical Layout of Hosting Stack
14 11/2011 www.bsdmag.org 15 HOW TO Speed Daemons
Solr full text search was integrated to implement excellent • Two Application Servers – This is where the Plone search capabilities throughout the application. Going into CMS and Apache Solr full text search applications the specifics of setting up the Plone application itself is are handled beyond the scope of this article. • Two PostgreSQL Database Servers – Plone operates Due to how Plone runs, we have a relatively non- with a PostgreSQL database back end, and maintains traditional web proxy chain installed using the nginx web a reports database for data processed by the server, varnish reverse proxy cache server, and haproxy application. load balancer. These services are in the ports tree. The specific installation and configuration of these services Proxy services are configured identically across our front are also beyond the scope of this article. end proxy servers. The proxy servers serve as a gateway There are six physical servers dedicated to the Plone to the application servers and don’t have very high loads application. (Figure 1: Logical Layout of Hosting Stack) In even during peak traffic hours. The CARP protocol is addition, there is a lower capacity server setup at another being used to provide redundancy in the event of a server datacenter for disaster recovery/business continuity. failure or in case of a service failure. CARP essentially Finally, there are a total of three ZFS based storage allows you to assign a single IP address across multiple servers providing data storage and shipping capabilities servers and provides an internal mechanism for failing to the application. The 6 physical servers used to host this over from one server to the next seamlessly. Setting up Plone application are paired up as follows: CARP requires recompiling the kernel. This section of the handbook: http://www.freebsd.org/doc/handbook/ • Two Proxy Servers – These provide web caching and kernelconfig.html covers how to recompile the kernel. The proxy services option to enable CARP is: device carp. After restarting
• Nginx web server – listening on ports: 80/443 with the custom kernel, edit /etc/rc.conf similar to these • Varnish caching reverse proxy server – listening examples: Listing 1.
on port: 3180 The interface em0 on each server is a unique private IP • Haproxy load balancing with session affinity – address. The em1 interface has a unique public IP address. listening on port: 3380 The carp0 interface is a shared private IP address between the two servers. (Figure 2: Proxy Server Logical Layout)
Listing 1. An example carp setup in rc.conf Finally the carp1 interface is a shared public IP address. The vhid needs to be unique per shared address. It is proxy1 /etc/rc.conf possible to share an address across two or more servers. hostname="proxy1.mysite.com" The server with the lowest advskew determines which is cloned_interfaces="carp0 carp1" the MASTER and is serving the address. See the carp ifconfig_em0="inet 10.0.0.1 netmask 255.255.255.0" manpage for more information on the specific values. ifconfig_em1="inet 12.12.12.1 netmask 255.255.255.0" In the event the MASTER proxy server fails, the BACKUP ifconfig_carp0="vhid 4 pass vhid4 advbase 3 advskew 50 with the lowest advskew will automatically promote itself 10.0.0.3/24" to the MASTER state and start serving traffic. This change ifconfig_carp1="vhid 10 pass vhid10 advbase 3 advskew over usually drops three or fewer packets making it nearly 50 12.12.12.3/24" seamless to any end user, especially on a stateless ifconfig_lo0="inet 127.0.0.1" protocol like http. By itself, CARP is only concerned with the network proxy2 /etc/rc.conf state of it’s peers, and does not detect the state of hostname="proxy2.mysite.com" cloned_interfaces="carp0 carp1" ���������� ���������� ���������� ifconfig_em0="inet 10.0.0.2 netmask 255.255.255.0" ifconfig_em1="inet 12.12.12.2 netmask 255.255.255.0"
ifconfig_carp0="vhid 4 pass vhid4 advbase 3 advskew 100 ������ ������ 10.0.0.3/24" ifconfig_carp1="vhid 10 pass vhid10 advbase 3 advskew 100 12.12.12.3/24" �������� �������� �������� ifconfig_lo0="inet 127.0.0.1" Figure 2. Proxy Server Logical Layout
14 11/2011 www.bsdmag.org 15 HOW TO Speed Daemons
Listing 2a. An ifstated con�guration on proxy servers
# Set the default state to proxy_up: init-state proxy_up # If we can't get through to the site using this server, or the # carp1 is the main site: # CARP master go to panic state: proxy_carp_up = "carp1.link.up" proxy_carp_init = "carp1.link.unknown" if ! $proxy_remote_ping { set-state panic # Ping our own proxy chain as well as the CARP master } every 15 seconds:
} proxy_ping = '("/usr/local/bin/checkproxy.sh" every 15)' # End of proxy_down state proxy_remote_ping = '("/usr/local/bin/check_remote_ proxy.sh" every 15)' # Define the default state, with the proxy chain # Define a state in which there is a problem with the responding normally. proxy chain
# on this machine. This state is set if the proxy_ping state proxy_up { command returns init { # unsuccsessful (non-zero) status: run "ifconfig carp0 advskew 50" run "ifconfig carp1 advskew 50" state proxy_down { run "echo 'date': Proxy chain up >> /tmp/ifstate.log" init { } run "sleep 10" # Set the state to proxy_down if the ping script returns # Log the problem with the proxy chain, and attempt to an unsuccessful bring # (non-zero) status: # the interface down if we're the CARP master:
if ! $proxy_ping { run "echo 'date': Proxy chain down >> /tmp/ set-state proxy_down ifstate.log" } if $proxy_carp_up { run "echo 'date': Proxy chain is down but I\'m # If the proxy chain is up on this machine, but it is CARP master. \ not the CARP master Attempting to become CARP slave by increasing # and the site is unresponsive attempt to take over by advskew \ decreasing advskew: >> /tmp/ifstate.log" run "ifconfig carp0 advskew 250" if ! $proxy_carp_up { run "ifconfig carp1 advskew 250" if ! $proxy_remote_ping { run "echo 'date': My proxy chain is up but I\'m } CARP slave and \ site isn\'t responding. Attempting to become CARP # Monitor the proxy chain on this machine, and set the master state to up by decreasing advskew. >> /tmp/ifstate.log" # if the ping script has a successful (0) exit status: run "ifconfig carp0 advskew 5" run "ifconfig carp1 advskew 5" if $proxy_ping { } set-state proxy_up } } }
16 11/2011 www.bsdmag.org 17 HOW TO Speed Daemons
services hosted on the individual proxy servers. There Quotation: A snip from man ifstated are a number of different tools that can be used to test The ifstated daemon runs commands in response to network state changes, the proxy chain and take actions based on event failures. which it determines by monitoring interface link state or running exter-
We chose to deploy ifstated (/usr/ports/net/ifstated) The nal tests. For example, it can be used with carp(4) to change running description of this tool in the man page reads as follows: services or to ensure that carp(4) interfaces stay in sync, or with pf(4) Quotation. to test server or link availability and modify translation or routing rules. Listing 2b. An ifstated con�guration on proxy servers Ifstated was built with CARP in mind, and as a result # End of proxy_up state is a lightweight tool that provides a lot of flexibility. The configuration on proxy1 and proxy2 is as follows: Listing 2. # Define the state in which the site is unreachable There are two scripts called by ifstated every 15 because there is either seconds. These are checkproxy.py and check_remote_ # a problem with the proxy chain on all proxy servers, proxy.py. Checkproxy.py connects to the local unique or there is a public IP address and makes an http connect to the # problem on the application servers. We've avoided application. If the application does not respond the script restarting services exits with a non zero exit code. Ifstated will detect this # for better debugging up until this point. In this and, depending on the current state, will take appropriate case since it is a total steps based on the configuration. # outage we will restart the proxy chain and set In simple tems, if proxy1 is the master, and checkproxy.py advskew low as a last attempt determines there is a problem with the local proxy chain, # to automatically correct an outage situation. In but the check_remote_proxy.py determines that the order to keep from remote proxy is functioning properly ifstated will raise the # constantly changing CARP state causing erratic local advskew of the carp interfaces to 250. This will usually behavior the only way to result in proxy1 being demoted to BACKUP status. At the # get out of this state is manual intervention. same time, proxy2 will have checked the local and remote proxy chain. If proxy2 is in state BACKUP, and gets a non- state panic { zero exit code when check_remote_proxy.py runs ifstated init { will lower the advskew on the local CARP interfaces to run "echo 'date': ALL proxy servers are unable 5. This will under most circumstances promote proxy2 to reach the site. \ to MASTER status. The panic state occurs when neither Entering panic state and attempting to restart proxy server has a functioning proxy chain. In this state the entire ifstated attempts to restart each component in the proxy proxy chain. MANUAL INTERVENTION IS REQUIRED AT chain, and then lower the advskew to 5. THIS POINT An external monitoring system is used to detect outages TO RESTORE AUTOMATIC FAILOVER >> /tmp/ and send alerts. This configuration of ifstated is not ifstate.log" generating any notifications to our systems administration staff. Ifstated is used only for attempting to keep the site run "/usr/local/etc/rc.d/nginx restart" alive in the event of service or server outages. This puts run "/usr/local/etc/rc.d/varnishd restart" notification tasks in a place that is independent of both run "/usr/local/etc/rc.d/haproxy restart" proxy servers and provides notifications in the case of run "ifconfig carp0 advskew 5" site-wide outages as well. run "ifconfig carp1 advskew 5" The application servers are using a single internal } CARP address for the Solr search process. } (Figure 3: Database Server Logical Layout) This provides a common address for each of the Zope application servers to connect to for internal search functions. Replication of the Solr data is managed by Solr itself via internal replication mechanisms. Ifstated is not running on the application servers as the advanced state
16 11/2011 www.bsdmag.org 17 HOW TO
������������ ������������ ��������� ���������
��������� ��������� ��������� ��������� ��������� ��������� Figure 3. Database Server Logical Layout Figure 4. Database Server Logical Layout
monitoring features aren’t necessary here. The Zope Ifstated is running on these servers with the following application servers are load-balanced via haproxy on the configuration: Listing 3. proxy servers further up the stack. Similar to the proxy servers, ifstated calls an external Moving further into the stack, we have a pair of database script every 10 seconds to check the state of the servers running PostgreSQL 9.0. Each of these servers PostgreSQL service. On these servers, if the database also has a unique private IP address, as well as a shared fails over we do not want to automatically attempt to CARP address. (Figure 4: Database Server Logical recover like we do with the proxy chain. In the case of Layout) PostgreSQL is handling replication internally a database failure a systems administrator’s attention is using Database2 as a read-only hotspare. desired to prevent potential data corruption or loss.
Listing 3. An ifstated con�guration on database servers
init-state psql_down ifstate.log" db_carp_up = "carp0.link.up" db_carp_init = "carp0.link.unknown" run "echo 'date': promoting Postgres to master STATUS = '("/usr/local/bin/psql-status.sh > /dev/null >> /tmp/ifstate.log" 2>&1" every 10)' run "/usr/bin/touch /var/db/pgsql/data/FAILOVER" } if ! $db_carp_up { state psql_down { set-state db_slave if ! $STATUS { } run "echo 'date': psql not running >> /tmp/ if ! $STATUS { ifstate.log" run "echo 'date': psql changed to non-running } state >> /tmp/ifstate.log" if $STATUS { set-state db_slave if ! $db_carp_up { } run "echo 'date': psql up and CARP is down, changing state to slave >> /tmp/ } ifstate.log" set-state db_slave state db_slave { } init { if $db_carp_up { run "echo 'date': CARP link DOWN >> /tmp/ run "echo 'date': psql up and CARP is up, ifstate.log" changing state to master >> /tmp/ run "/sbin/ifconfig carp0 advskew 250 >> /tmp/ ifstate.log" ifstate.log" set-state db_master } } if $db_carp_up } set-state db_master } } state db_master { init { run "echo 'date': CARP link UP >> /tmp/
18 11/2011 www.bsdmag.org HOW TO
The distributed cluster of servers as well as the Plone application and Apache Solr full text search require file system data be shared across multiple machines. One of the customer’s requirements was integrating the Plone application with some of their internal systems. To accomplish this, an sFTP server is setup on the proxy servers, allowing the customer to place a data file which will in turn to be processed by the application servers daily. In order to make the data available to the application servers, a ZFS filesystem, ftpdata, was configured on our ZFS-based storage server. The ftpdata filesystem is shared via NFS and is mounted on both proxy servers as well as both application servers. The Zope application also has data that needs to be available to all of the running application servers across both physical machines. This data is also stored on the ZFS storage servers using NFS mount, zopedata, which is mounted on both application servers. Finally we have a ZFS filesystem on the storage servers, solrdata, that is mounted on both application servers. Six Feet Up wrote a ZFS snapshot management package called The Fortville Snapshot Distribution, or FSD for short. FSD takes snapshots of the important filesystems and replicates them from our primary storage server onto our secondary storage server, as well as an off-site tertiary storage server. This provides redundancy of important data, as well as a way to roll back time to any five minute interval within the previous 24 hours, or any hour within the previous month without going to the backup system. As a side note, some of the work that went into FSD has been used in recent versions of FreeNAS 8. The redundancy in the hosting stack has allowed multiple FreeBSD system and software upgrades while the site is live by upgrading half of the stack, triggering a failover, and then upgrading the other half of the stack. Overall, the networking configuration and tools deployed have provided 99.995% uptime to date; which is significantly above the customer requirement we are required to maintain. The combination has allowed us to provide our client with the reliability and speed needed for a modern, high performance web application with a level of automation which keeps management overhead within reason.
LARS R. NOLDAN Lars has over 14 years of technical experience, including open source server administration. For the last three years Lars has worked as a Systems Administrator for Six Feet Up – managing approximately 100 servers and another 200 jails running FreeBSD. He has extensive experience with web hosting technologies, focusing primarily on complex deployments in higher education and Fortune 500 companies.
18 11/2011 www.bsdmag.org HOW TO A GIS Strategy For Web-Enabled Business A GIS Strategy For Web-Enabled Business
In this final article in our GIS series, we will look at how to successfully manage and commission a complex GIS project
What you will learn… What you should know… • How to plan and commission transformational technology • Previous FreeBSD GIS tutorials in this series
n the previous articles, we have looked at the core executives feel that IT is not delivering. Ironically, both technological nuts and bolts that comprise an groups ultimately want to achieve the same goals, I enterprise scale GIS. Rather than focusing on the but are looking at the problem from radically different technology, in this final article we will look at GIS from perspectives (albeit with entirely different motives a different angle – culturally – and how to roll-out a sometimes) and this can end in deadlock or a breakdown successful system that could potentially revolutionize a of trust. From senior managements perspective, if the business. While a substantial percentage of this article IT department cannot be trusted to handle the small will be GIS specific, a lot of the wisdom contained here matters (e.g. reliable servers, PC’s etc.) there will be will also be applicable to other major projects so I hope a lack of confidence in IT advising in the very sensitive there will be something here for everyone. area of corporate strategy, especially where technology will mean a major change in business practice. Likewise, Do You Have Buy-in? if management undermine IT (We want system X In these times of austerity the IT department is a critical from vendor Y regardless of the technological issues source of innovation and creativity especially in the area or long term cost) IT can get disillusioned from fire- of improving both the business model and generating fighting and vote with their goodwill or feet. Sometimes efficiencies. As a core business group, IT are often the perception is more valued than reality, but the first step first to sense the change of mood at ground floor, in the to a successful enterprise system is confidence and trust marketplace and at management level. Yet, in some from all sides. The same applies in the other direction, if organizations layers of management and bureaucracy senior management do not have buy-in from the users, prevent effective communication. Unfortunately, it the change in business practice that technology brings has been the authors experience in many businesses will be resisted and potentially in some cases even that there is a fundamental disconnect between board sabotaged. Again, this is where IT can offer an invaluable level (e.g. CEO) and IT unless the technologists have service to the CEO in helping to identify how technology a strong representation via a Chief Technical Officer can make life easier and better for both the customer or a very senior manager who can be their advocate and the employee alike. Once trust is developed across and argue their case. This ultimately ends in frustration the organization, the door is then open for everybody to for both groups, as IT feel nobody is listening and the contribute to innovation and moving the system forward.
20 11/2011 www.bsdmag.org 21 HOW TO A GIS Strategy For Web-Enabled Business
Realistically, technology is relatively straightforward, – money, time and innovation. Better still, don’t fall for the biggest hurdle is navigating through any cultural eloquent sales pitches, smoke and mirrors accounting resistance so that we ultimately end up at the position and read the contract and licence agreement, preferably of continuous improvement inside the organization and with a cynical lawyer who is on your side. That way you meeting our customers needs externally. will not bear the pain further down the road when your organization wants or needs to change. Integration The other issue that always raises it’s head is what (Both Human and Technological) is The Key happens when a key technological player falls under a Legacy systems, vendor lock in, proprietary API’s – all of bus? Rather than looking at this from the organizations these scenarios give IT departments a major headache. perspective (redundancy, risk assessment, having a Thankfully, the Open Systems model is gaining ground, very propriety system etc.) I tend to see this as a cultural not only with Open Source, but vendors are realizing barrier. I may be wrong, but in my mind the organization now that the era of the closed business model – where group-think is more aligned in this scenario to What if he the customer is the milch cow – is rapidly coming to leaves? This to me is rather sad, as good IT staff are hard an end. Where there will be some vendors that cling to come by, and really what the organization should be to the proprietary model and fight their corner through thinking is how can we use, retain, reward and expand the courts demanding justice for intellectual property our talent? Google manages to do this innovatively, rights, those with vision will be getting on with cutting along with a lot of blue-chips, and the concept of staff code, peer reviews, and realizing that the sum is being your greatest asset should be embedded at the always greater than its parts. The struggle from the very core of corporate culture. Inevitably, there will be technological perspective is when a vendor refuses to those who wish to progress their careers, sometimes take part in organizational change, and either makes at much cost to the organization, but in the authors this process financially prohibitive, or worse still refuses experience the easiest employers to work for are those point blank for commercial reasons citing a catalog that naturally and successfully manage to embed their of reasons. Without naming any names, the author employees into not only the corporate culture, but their was once asked by his manager to get the views of creative vision. Béla Hatvany (of Silver Platter fame) a well known IT vendor on a relatively minor systems ran a small entrepreneurial start up in the ‘80’s which I integration project which was of strategic importance. At was privileged to be associated with, and it was evident great cost to the company (they were blue-chip so they that a major contribution to the success of the venture could afford it!), 2 senior consultants were dispatched would be the employees, their synergy and personal from the vendor for an afternoon meeting to discuss commitment. When the business venture was heading how this could be achieved. After much coffee and towards hard times financially, all of the team offered discussion, both consultants shook their head and their without being asked what sacrifices they could make joint opinion was that a suitable product would not be to turn the operation around. Some were even willing available for 18 months at least. Needless to say, the to work for a subsistence salary. How many corporates author rolled out his own solution in 3 countries in 2 today could instill this level of loyalty and drive? The months, saving the organization on paper at least the difference here is that the synergy between team cost of his annual salary per year. It was fortunate that members and Béla was so strong that his problem was the author had a good relationship with his line manager, their problem. Human chemistry, dynamics, personality as it was understood that the invitation to the vendor and corporate culture all play their part here, but the was very much a “box ticking” exercise, and the supplier energy that is found around a cohesive team that is had no intention of providing us as the customer with allowed to flex their wings and get on with providing an a workable solution. It would have been easy to feel excellent solution is truly something to be experienced. undermined but once again it was a good illustration of Too often, micro-management, not empowering corporate politics getting in the way of progress. If the individuals, and an unwillingness to allow them to make level of trust had been greater, my employer would have their own mistakes prohibits creativity, and the net result saved the consultancy fee as well, and the potential is the delivery of the average or mediocre rather than pitfall of alerting the vendor that we were looking at other the superior. solutions. I am not advocating here that management needs to Ultimately, there are always solutions available to the take a totally hands off approach, rather that boundaries integration nightmare, the question is one of resources should be set and the right individuals chosen for the
20 11/2011 www.bsdmag.org 21 HOW TO A GIS Strategy For Web-Enabled Business
team and they are then allowed to get on with their Once persuaded that a new channel is important, then specialties. My best GIS functional specification to date two fundamental questions arise. What do you want to has been a few sketches on the back of some scrap achieve and why do you want to achieve it? I am a firm paper, this means more to me than all the Prince 2 believer that killer app 2 in the enterprise will be the project management software in the universe. Why? It combination of telephony, CRM, GIS and the Internet. Mr came from the customer, we both know what we want Jones telephones, CTI picks up his caller ID and pulls his to achieve and fortunately we have good synergy. service record and location from the CRM. Meanwhile, Sometimes a more formalized structure is necessary, GIS locates the nearest service representative, and at a but I have yet to see an inspired team and hungry click of a button the new service request is passed to the for success fail to rise to the challenge of solving the closest man in the field. Alternatively, CTI passes to GIS problem when confronted with a firm but respectful You and the call goes directly to the nearest representative. are not giving up until we have a solution. While this The possibilities are endless. Mobile applications on the meeting may be over coffee (or in the evening down at Iphone etc. can fire directly into a CRM / GIS combination the pub), it is the esprit de corps that delivers, not the for the reporting of issues in the field, allowing the paperwork or the ultimatum. company not only to compose a geographic picture of where the hot spots lie, but also develop a relationship What Does The Organization Want to Achieve? with the customer. The reverse scenario is also true, if People love maps. Google Earth and Google Maps are a you have multiple mobile devices located in your area, prime example of killer applications that are very hard to activate them automatically according to region. Region put down. Tie this in with a good user interface, and users A likes product B. Region B likes product A. You are a want to explore. This in turn raises the What if? scenario. marketing agency, company C wants to penetrate both Creativity begins to flow, and inevitably there will be a markets, so you beacon their sales representatives with clash of civilizations, between the flexible analogue of tailored point of sale and marketing information as they spatial data and the previous method of measurement. enter their different territories. Customers and requirement will be discovered in new What is more, all these platforms are available on *BSD. areas, different perspectives will be developed and SugarCRM, Geoserver and Asterisk with some glue code soon the previous ways of the organization will be found and additional hardware would not be a bad start. Coupled lacking. This is a dangerous time, as alongside the new with the true reliability, security and stability we all know, technology with all its successes and revelations will be this would be a dream development platform. the temptation to totally replace the current orthodoxy The key to this is to get together and ask What if? Blue with the newcomer and in the process throw the baby sky thinking and GIS make great team players. out with the bath water. Rinse, wash, repeat. This is the mantra the vendor doesn’t want you to hear – built in Data Quality, Gathering and Data Logging commercial obsolescence. Paradoxically, while an organization is greater than the The world will change, but provided core systems sum of its parts, no organization is greater than the quality are able to both speak and listen (send and receive of its’ data. Poor quality customer data is the bedrock of information to each other), the solution is simple – disasters, poor customer PR, financial loss, and ultimately snap more technology on the end. Mr Vendor will not a blame the system mentality. As we all know, garbage in = want his system to be this extrovert, so he will prefer garbage out, so as IT professionals we must continuously to provide you with his introverted version of GIS with be on our guard to validate any input to our systems. his Customer Relationship Management (CRM) system How can this be dealt with in legacy systems however? or whatever the latest understanding is on enterprise Data matching and cleansing is very processor and time technology. If you couple best of breed with best of intensive, even with an army of validators the scale and breed, you will end up with a thoroughbred, not a complexity can be daunting. Often, the solution is just mongrel. So, the first essential requirement is that your flush the database and start again. This goes against the GIS is truly open as indeed all your systems should be. grain for me, as there is bound to be something important The universe needs to expand – not shrink. If you want in there to the business. If your organization is serious your GIS department (or any others for that matter) about your data, I would recommend the former route – to live in a dark corner and not be a core part of your cleanse – but use this as an opportunity for some positive organization, a closed source solution is therefore ideal PR and create a feedback loop between yourself and the – for the vendor at least. customer. Using PR, and / or financial incentives, ask the
22 11/2011 www.bsdmag.org 23 HOW TO A GIS Strategy For Web-Enabled Business
References • Béla Hatvany 2000 Miles Conrad Memorial Lecture – http://www.nfais.org/page/46-bela-hatvany-2000 • NFAIS Annual Conference – How to prosper in the era of the Internet • Robin Gawlik – Inspire Case Study – http://www.lga.gov.uk/lga/aio/20246680 • Barrow Borough Council – GIS system – http://webgis1.barrowbc.gov.uk/webgis/bingis.html
customer to register with you for a discount or a free offer Thanks and Credits etc. Again, you have a win-win scenario, you know what The inspiration for this series of articles would not have customers are enthusiastic about dealing with you, and come about if it were not for Copeland Borough Council you will get accurate data (you hope!). Better still, get moving from a proprietary GIS to an Open Source customer services to phone the customer back just to platform. The catalyst for this transformation rests with make sure the details are correct. two individuals, Robin Gawlik of Barrow Borough Council Processes need to be in place not only to ensure and Julia Jackson at Copeland, both GIS Information Data Protection standards are met, but the quality of Officers. The ability to manipulate complex equations, the data is excellent. Edgar Codd, the father of RDMS, write great code, think spatially and maintain a good sense states in rule 1 – All information in the database is to be of humor are a rare combination of talents, especially as represented in only one way. An organization should be I am bereft of any sense of direction whatsoever. Without aiming for disparate systems with a common key (e.g. the developers, data trustees and the many others in the a unique property ID or unique customer ID) across GIS community Open Source GIS would not be a reality. boundaries, rather than replicating the lower level data Our collective thanks goes to you all – you know who you or functionality across systems. The problem is not the are. data that matches, it is the data that should match but The Barrow GIS system can be found at the link below, doesn’t. and the Copeland version will be live in the very near future, as it is currently being tested in-house. GIS is Not Cheap All opinions in this article are purely those of the author While Open Source is free, sadly infrastructure isn’t. The and do not necessarily represent either Copeland or one major lesson I have learned about GIS is that to do Barrow Borough Council. it properly you need decent hardware. Our largest tile- cache holds over 19 million files on one virtual guest, and ~ 10 million on the other, so JFS was the only practical option. I thought that 1TB of storage would be more than enough for my Virtual Machine images, but it would have been more prudent to have 5TB or a SAN. Geospatial data takes a lot of space and processing power, so specify your hardware with more than plenty of redundancy. Ironically, our Postgres database is actually very lean, the most CPU load is on the Geoserver processor (8 cores) and the physical size of the tile-cache. Interfaces to GIS can be accurate up to 5 centimeters for professional kit, but in certain environments (e.g. wooded areas) getting a good lock on a satellite can be a hit and miss affair. A ground station to provide a reference ROB SOMERVILLE signal may be required if pinpoint accuracy is essential. Rob Somerville has been passionate about technology since That aside, GIS location software is used for plotting his early teens. A keen advocate of open systems since the mid Formula 1 cars during the race. Decent GPS kit is not eighties, he has worked in many corporate sectors including cheap either, expect to pay > L10K for a highly accurate �nance, automotive, airlines, government and media in a ground station in a ruggedised case excluding hand held variety of roles from technical support, system administrator, units. Consumer grade units vary substantially, and your developer, systems integrator and IT manager. He has moved on accuracy will depend on which country you are in and from CP/M and nixie tubes but keeps a soldering iron handy just what satellites you have access to. in case.
22 11/2011 www.bsdmag.org 23 HOW TO Equip Your CA With HSM For <50 Euros Equip Your CA With HSM For <50 Euros With the recent breaches of Certificate Authorities (Comodo, Diginotar) I wanted to take a closer look at the security of my own Certificate Authority (CA). This CA is used for identification and authentication of servers, clients and users in my home network.
What you will learn… What you should know… • How to use smart cards to store private keys • Your way around a FreeBSD system • The use of OpenSSL with a different crypto provider • The basics of a PKI • Why you need a Certi�cate Authority
he keys for signing certificates are typically stored platform (i386 in my case, as I am working on putting it in text files on the file system. Access to the disk on an ALIX / NanoBSD installation). Attached is a Feitian T will reveal the keys. An attacker can make a copy SCR310 smart card reader (ftsafe-r310) and a Feitian PKI and start issuing certificates outside my control. The card (FTCOS / PK-01C). This reader is cheap, but any primary design requirement is therefore that private keys supported reader is fine (see http://pcsclite.alioth.debian. will never be accessible in plain text. org/ccid/section.html for the complete list). This card was primarily chosen for its price and availability. It has lots of Hardware Security Module storage memory available (64k), but a limitation of 2048 Enter HSM, hardware designed to keep the private key bits for RSA keysize. Please do your own research on the private. They come in various forms, from an appliance types of readers and cards to fit your requirements. / PCI card to a USB token or smart card. PCI cards are For my production system, I will move to a Gemalto typically encased in metal with features like automatic key USB TR reader. It has an adapter so it can be mounted in deletion upon physical tampering. They are basically a a 3.5” floppy bay. dedicated computer running HSM software and using the PCI bus for interfacing with the host computer. Appliances are typically these PCI cards in an enclosure providing networking and rudimentary user control. HSMs can also have the form of a single chip. These do not have their own power supply (for deleting keys), but are still a small computer running HSM software. They can be embedded on smart cards, in USB tokens or integrated in other systems like the TPM chip on your motherboard.
Design Choices For my HSM I decided to use smart cards, because they are cheap, readily available and easy to experiment with. The computer hardware is generic FreeBSD supported Figure 1. Cards
24 11/2011 www.bsdmag.org 25 HOW TO Equip Your CA With HSM For <50 Euros
The software is a clean FreeBSD 8.2 with the following Nr. Card Features Name ports: 0 Yes Feitian SCR310 00 00
• /usr/ports/devel/libccid – interface for USB and serial The reader is recognized by the driver. Let’s initialize the smart card readers card by formatting it with a PKCS15 structure. PKCS15
• /usr/ports/security/opensc – tools for smart card mana- is a cryptographic token information format standard gment (in PC/SC mode) originally designed by RSA. ISO 7816-15 now manages
• /usr/ports/security/engine-pkcs11 – engine for PKCS11 the card-related parts of this standard. support in OpenSSL • /usr/ports/security/openssl – tools for certificate mana- # pkcs15-init --erase-card gement (in this case) # pkcs15-init --create-pkcs15 \ --profile pkcs15+onepin \ Due to the many, many dependencies, it will take some --auth-id 01 \ time to install. All default settings for the ports are fine. --pin 0000 \ --puk 123456 \ There is a number of different CA designs possible. --label „ewak.net PKI” From very flat (the root CA issues all certificates directly) to very hierarchical (various sub and sub-sub CA’s issue This specific card supports only one user-PIN, so the certificates on behalf of the root CA). My design has one pkcs15+onepin profile is used. More advanced cards can root CA with a sub CA per functional area. This is done offer more users and role separation. for security reasons. First of all, the root CA only has The PIN for the user (auth-id 01) is set to 0000. If you to sign the sub CA’s. The use and exposure of the root want to reset the PIN, you will need the PUK, which is set CA’s private key is therefore very limited. Next to that, to 123456. The label is just a name to identify the card. the server sub CA never issues client certificates, so the The card now has a PKCS15 structure and is able to VPN concentrator has to trust only the clients sub CA store private keys and their certificates. when validating certificates. Client certificates issued by the (possibly hacked) server- or user sub CA are not Listing 1. Formatting the card accepted. # pkcs15-init --verbose \ Testing The Setup --store-private-key ewak.net_Sub_CA_servers.p12 \ After the installation of the software, it is time to plugin the --format pkcs12 \ reader and test the setup. Insert the card in the reader --auth-id 01 \ and run: --cert-label "ewak.net Sub CA servers"
# opensc-tool --list-readers Using reader with a card: Feitian SCR310 00 00 Connecting to card in reader Feitian SCR310 00 00... # Detected readers (pcsc) Using card driver entersafe. Found ewak.net PKI About to store private key. ������� Importing 1 certificates: 0: /C=NL /ST=GLD /L=T******e ������ ������ ������ ������� ������� ����� /O=ewak.net /OU=Certificate Services /CN=ewak.net Sub CA servers /[email protected] User PIN [User PIN] required. ����������� ����������� ��������� Please enter User PIN [User PIN]:
Figure 2. CA-design
24 11/2011 www.bsdmag.org 25 HOW TO Equip Your CA With HSM For <50 Euros
Importing Keys Listing 2. Contents of the card It is possible to have the card generate the private and public keys, so the private key will never be available in # pkcs15-tool –-dump plain text. This is the most secure option, but losing the card will also make the keys unrecoverable. Using reader with a card: Feitian SCR310 00 00 Because I already have a CA in place, I started with PKCS#15 Card [ewak.net PKI]: importing the existing keys. Generating the keys on Version : 0 the card will be discussed at the end of this article. Serial number : 3021303609260511 Importing the key material (in PEM or PKCS12 format) is Manufacturer ID: EnterSafe straightforward (Listing 1). Last update : 20110925214004Z The user PIN for auth-id 01 is defined during the Flags : EID compliant initialization of the card, 0000 in this article. It can also be
PIN [User PIN] Listing 3. Finding the card slot Object Flags : [0x3], private, modifiable ID : 01 # pkcs11-tool --module /usr/local/lib/opensc-pkcs11.so Flags : [0x32], local, initialized, --list-slots needs-padding Length : min_len:4, max_len:16, stored_ Available slots: len:16 Slot 0 (0xffffffff): Virtual hotplug slot Pad char : 0x00 (empty) Reference : 1 Slot 1 (0x1): Feitian SCR310 00 00 Type : ascii-numeric token label: ewak.net PKI (User PIN) Path : 3f005015 token manuf: EnterSafe token model: PKCS#15 Private RSA Key [Private Key] token flags: rng, login required, PIN Object Flags : [0x3], private, modifiable initialized, token initialized Usage : [0xC], sign, signRecover serial num : 3021303609260511 Access Flags : [0x0] Listing 4. Loading the engine ModLength : 2048 Key ref : 1 (0x1) Native : yes # openssl Path : 3f005015 OpenSSL> engine dynamic \ Auth ID : 01 -pre SO_PATH:/usr/local/lib/engines/engine_pkcs11.so ID : 53f70c3ea5be9aef27d959c134d8ebe \ f77322786 -pre ID:pkcs11 \ GUID : {53f70c3e-a5be-9aef-27d9- -pre LIST_ADD:1 \ 59c134d8ebef} -pre LOAD \ -pre MODULE_PATH:/usr/local/lib/opensc-pkcs11.so X.509 Certificate [ewak.net Sub CA servers] Object Flags : [0x2], modifiable (dynamic) Dynamic engine loading support Authority : no [Success]: SO_PATH:/usr/lib/engines/engine_pkcs11.so Path : 3f0050153100 [Success]: ID:pkcs11 ID : 53f70c3ea5be9aef27d959c134d8ebe [Success]: LIST_ADD:1 f77322786 [Success]: LOAD GUID : {53f70c3e-a5be-9aef-27d9- [Success]: MODULE_PATH:/usr/local/lib/opensc- 59c134d8ebef} pkcs11.so Encoded serial : 02 01 02 Loaded: (pkcs11) pkcs11 engine
26 11/2011 www.bsdmag.org 27 HOW TO Equip Your CA With HSM For <50 Euros
supplied to the pkcs15-init tool by adding the --pin 0000 the PKCS11 API, also known as Cryptoki. When using option. Now let’s see what is on the card (Listing 2). OpenSSL, we can specify keys on the smart card instead The import was successful. The info for the PIN, the of a keyfile on the disk. To do this, we need the card’s
private key and the certificate are displayed. Record slot and the key’s ID. The ID we found using the pkcs15- the ID of the private key, as we will need it later to tell tool –list-keys command. The slot can be found with the OpenSSL which signing key to use. A shorter command pkcs11-tool command (Listing 3).
for retrieving this ID is pkcs15-tool –-list-keys. More keys Slot 1 is our card reader with the correct smart card can be added to the card. My card has three keys for (ewak.net PKI) inserted. signing server, client and user certificates. Now start an OpenSSL prompt and load the PKCS11 engine: Listing 4. OpenSSL and PKCS11 The engine is loaded and can be used by specifying it in OpenSSL can be instructed to use an external crypto OpenSSL commands with the following parameters: provider for generating and storing key material using -engine pkcs11 \ Listing 5. openssl.cnf -keyform engine \ -key slot_1-key_53f70c3ea5be9aef27d959c134d8ebef77322786 openssl_conf = openssl_init [openssl_init] For example: engines = engine_section [engine_section] OpenSSL> req -new -engine pkcs11 -keyform engine \ -key slot_1-key_d893afddc82b28fb539e975b2a3e18efc2f3c474 \ [pkcs11_section] -out cert.csr –text engine_id = pkcs11 dynamic_path = /usr/local/lib/engines/engine_pkcs11.so (This will create a certificate signing request from the MODULE_PATH = /usr/local/lib/opensc-pkcs11.so public key with the id shown.) The use of PKCS11 can init = 1 be set in the OpenSSL configuration file by adding the following lines to your /path/to/openssl.cnf configuration Listing 6. Signing the certi�cate file (Listing 5). All OpenSSL commands can now be run using the PKCS11 engine by specifying: # openssl ca -config /etc/ssl/openssl.cnf \ -engine pkcs11 \ Listing 7. Contents of the card -keyform engine \ -keyfile 1:53f70c3ea5be9aef27d959c134d8ebef77322786 \ # pkcs15-tool --list-keys -cert ewak.net_Sub_CA_servers.crt \ Using reader with a card: Feitian SCR310 00 00 -in cert.csr -out cert.pem Private RSA Key [Private Key] Object Flags : [0x3], private, modifiable Using configuration from /etc/ssl/openssl.cnf Usage : [0x4], sign engine "pkcs11" set. Access Flags : [0x1D], sensitive, PKCS#11 token PIN: alwaysSensitive, neverExtract, Check that the request matches the signature local Signature ok ModLength : 1024 Certificate Details: Key ref : 1 (0x1) -snip- OpenSSL output omitted –snip- Native : yes Sign the certificate? [y/n]:y Path : 3f005015 Auth ID : 01 1 out of 1 certificate requests certified, commit? ID : d893afddc82b28fb539e975b2a3e18e [y/n]y fc2f3c474 Write out database with 1 new entries GUID : {d893afdd-c82b-28fb-539e- Data Base Updated 975b2a3e18ef}
26 11/2011 www.bsdmag.org 27 HOW TO
-config /path/to/openssl.cnf –engine pkcs11 \ Generating Keys –keyform engine –key slot_
Listing 8. Creating the csr Conclusion Smart cards can provide a low-cost and relative secure # openssl req -new \ storage for private key material. This article focused on -engine pkcs11 \ the use of smart cards in a CA environment. It did not -keyform engine \ focus on storing and using a user certificate for email -key slot_1-key_d893afddc82b28fb539e975b2a3e18efc2f signing, storing and using a client certificate for OpenVPN 3c474 \ or storing and using SSH keys. These are all interesting -out cert.csr –text options I will research further.
PKCS#11 token PIN: You are about to be asked to enter information that will be incorporated into your certificate request. -snip- OpenSSL output omitted –snip-
Listing 9. Storing the certi�cate
# pkcs15-init –-store-certificate cert.pem –-auth-id 01 \ --id d893afddc82b28fb539e975b2a3e18efc2f3c474
Using reader with a card: Feitian SCR310 00 00 User PIN [User PIN] required. Please enter User PIN [User PIN]: ERWIN KOOI Erwin Kooi is an Information Security Manager for a large grid operator. He started with FreeBSD 4.5 and is an avid fan ever since.
28 11/2011
ADMIN Terminals Served Up BSD Style Terminals Served Up BSD Style I believe there is a splash screen on the PC-BSD installer that says, “PC-BSD: The Desktop Served Up BSD Style.” It seems that the desktop BSD would be a good choice for a terminal server. My personal goal for this project is to provide a safe browsing environment to VIP’s at my organization.
hey refuse to let me filter out risky web sites. will serve multiple networked X sessions to multiple I figure if they do their browsing from a remote *nix users at the same time. T session, then I don’t have to worry about them • You should have used – at least once – some window getting malware on their Windows PC’s. You may have manager/desktop environment other than KDE or your own reason for wanting a BSD terminal server. GNOME (eg. Fluxbox, FVWM, IceWM, XFCE, xterm, There are two solutions to this goal: FreeNX etc). (a compression solution for the X protocol) or XRDP (a • You need to know the name of your network interface. product that wraps the VNC protocol inside Microsoft’s Mine is em0. Yours may vary, especially if your PC- Remote Desktop protocol). This article will show you how BSD is running in VMware. to use both solutions. • You need to know how to add users. I assume that I am admittedly better with OpenBSD than with PC-BSD you will add users as necessary to take advantage or FreeBSD. My understanding is that unlike OpenBSD, of your terminal server. Adding users will not be ports are preferred over packages under PC and Free. addressed in this article. This article will therefore use the ports system to obtain the necessary software. FreeNX In most cases, FreeNX is going to be the superior method Prerequisites for turning any *nix computer into a terminal server. It’s a faster protocol than VNC, and easier to get running on • You should understand that PC-BSD is simply PC-BSD. First, let’s install FreeNX: FreeBSD with KDE and an easy to use package manager installed by default. We can think of PC- # cd /usr/ports/net/freenx BSD as the desktop version of FreeBSD. # make install • You need to know how to use the command line interface. There will be some option screens. Simply accept all of • You need to have a basic understanding of TCP and the default answers. Next we need to have the NX server TCP ports. start at boot time. • You need to have the PC-BSD ports collection installed. There is an option for this during the PC- # /usr/local/NX/bin/nxsetup BSD install process. • It is assumed in this article that you are at the Answer “N” to both of the questions.
command line as root (indicated by the # prompt in Finally, let’s open up TCP port 22 so that we can the example commands). connect from a remote host. Edit your /etc/pf.conf file. • Understand that this is not a solution for screen Assuming you haven’t tweaked the default firewall rules sharing or simple remote administration. Our server with block directives, then you can simply add this line to
30 11/2011 www.bsdmag.org 31 ADMIN Terminals Served Up BSD Style
the bottom of the file (remember, my interface is em0; yours XRDP supposedly has a feature to serve actual RDP may be different): instead of VNC wrapped in RDP; however, you need something along the lines of xorg-x11-server-rdp. pass in on em0 proto tcp from any to (em0) port 22 keep state This doesn’t exist except as an RPM package for a very old version of OpenSUSE. Reload the firewall rules: It doesn’t work on current versions of OpenSUSE, and it certainly won’t work under PC-BSD’s Linux compatibility # pfctl -f /etc/pf.conf mode. Let’s get started. XRDP does not work well with KDE. Your FreeNX server is now ready. You can connect You’ll need to install an alternate environment (unless with the NX client for whatever operating system your you intend to use TWM or xterm, which are included by clients use. Download the client of your choice at http:// default with PC-BSD). www.nomachine.com/download.php. Unfortunately, you will If you know how to use ports, then install the environment find that there are no BSD NX clients. Your ports collection of your choice. If you don’t know how to use ports, or if you has one based on PC-BSD’s Linux compatibility feature, but want the easiest environment for your luddite customers, it doesn’t include all of the necessary Linux libraries. There then install GNOME from the software manager. Here is a are many threads to read about getting the net/linux-nx-client list of environments that I tried: Table 1. port to work, but I’m not going to address that in this article. Once you have the environment of your choice installed, we’ll install XRDP and its dependency, TightVNC: XRDP As I said before, XRDP is inferior to FreeNX in most ways, # cd /usr/ports/net/xrdp but there are reasons that you may want to use it anyway. # make install Here are some that I’ve encountered: # cd /usr/ports/net/tightvnc # make install • FreeNX uses lossy JPG compression. You will
sometimes notice blurriness in graphics. Let’s open up the RDP port on your firewall. Edit /etc/ • FreeNX can be slower than XRDP in certain pf.conf, and add the following line to the end of the file environments. I don’t know why, but I’ve speculated (remember to double-check the name of that network that one or all of these factors may play a role: interface): • FreeNX’s encryption. • The Cygwin layer of the Windows FreeNX client. pass in on em0 proto tcp from any to (em0) port 3389 keep state • QoS or other scheduling features on your routers. • RDP-specific features, such as being able to select Now we need to force XRDP to use the environment of
the color depth that suits you. our choice. Edit /usr/local/etc/xrdp/startwm.sh. You will • Lack of a FreeNX client for any BSD. find a line near the top that reads:
Table 1. Desktop Environments Environment Result Command to launch KDE Very messy & slow. Many artifacts. startkde GNOME Good gnome-session Enlightenment Fail enlightenment Fluxbox Good �uxbox IceWM Good, but you’ll have to create menus from scratch. icewm Afterstep Fail afterstep Windowmaker Good wmaker TWM Good, but difficult for novice twm XFCE Good, but difficult for novice xfce Openbox Good openbox Xterm Good, but difficult for novice xterm
30 11/2011 www.bsdmag.org 31 ADMIN
# /usr/local/sbin/xrdp && /usr/local/sbin/xrdp-sesman
Now this is important: Newer versions of Microsoft’s RDP client (mstsc) will cause XRDP to use 100% of your CPU until you reboot the server or kill the process like this:
# kill -KILL xrdp
The trick is to get a copy of rdesktop that has been precompiled for Windows. I found one here: http:// www.atomice.com/blog/?page_id=9. You can create batch files to launch rdesktop for your customers. The following would work fine (assuming that rdesktop is somewhere in your Windows PATH):
Figure 1. PC-BSD on a terminal rdesktop -g 1024x768 xrdpserver.mydomain.com
SESSIONS=”gnome-session blackbox startxfce4 startkde xterm” You’re Done! Whether you’ve used FreeNX or XRDP, you’re done. Edit that line to have ONLY the command for the We now hope that our customers enjoy using their new environment of your choice. Use the table above to find terminal server. the commands for some common environments. I’m using GNOME, so my line looks like this:
SESSIONS=”gnome-session”
Next we’re going to edit /usr/local/etc/xrdp/xrdp.ini to remove options that we don’t need (and might confuse our luddite customers). Delete the following line, and delete every line below it:
[xrdp2]
By default, XRDP wants us to create groups where users will be added to if we want them to have access to XRDP. The easiest thing to do is just give access to everybody. Edit /usr/local/etc/xrdp/sesman.ini, and find these lines:
TerminalServerUsers=tsusers TerminalServerAdmins=tsadmins TOBY RICHARDS Toby Richards has been a network administrator since 1997. He Just change tsusers to users, and change tsadmins to considers himself to be a jack of all operating systems, but a true wheel. The last thing to do is have XRDP start at boot time. master of none. He feels this to be a mastery in its own right since
Add the following to the bottom of your /etc/rc.conf file: he understands principles that are common to all operating systems. His articles are the product of teaching himself to /usr/local/sbin/xrdp become better with OpenBSD and PC-BSD. He simply writes /usr/local/sbin/xrdp-sesman about what he has learned most recently. For a hosting provider, he highly recommends bsdvm.com. They give you access to your
To start XRDP without rebooting so that /etc/rc.conf can VMware console so that you can re-install your OS at will, and do its job: with the settings of your own choosing.
32 11/2011
ADMIN OpenBSD Kernel Memory Pools: Monitoring Usage With Systat OpenBSD Kernel Memory Pools: Monitoring Usage With Systat This article explains how to understand memory usage statistics for kernel memory pools as they are displayed by the systat(1) command on OpenBSD.
What you will learn… What you should know… • Basic understanding of kernel memory pools • A page of memory – a �xed sized piece of memory (typically 4k in • Understand statistics regarding usage of kernel memory pools size) used by memory management system
memory pool, in its simplest form, is a pre-allocated allocations which use malloc(9). The vmstat view, which list of fixed sized pieces of memory. Memory pools is the default, is unique in that it displays an overall Acan ensure availability of memory resources, picture of system activity including statistics on interrupts, allow for more efficient memory allocation and reduce memory usage, process activity and filesystem activity. memory fragmentation. Both NetBSD and OpenBSD On OpenBSD, one can switch between views using the implement memory pools in their kernels. FreeBSD uses right and left arrow keys. a ‘slab allocator’ which is an extension of the functionality The systat utility first appeared in 4.3BSD and is provided by memory pools. NetBSD also implements available on FreeBSD, NetBSD and OpenBSD. The the extended functionality of slab allocation on top of its implementations are only slightly different – the set of memory pool implementation. views available are not exactly the same, and navigation The first section briefly describes the systat(1) utility. from one to the other is invoked differently. It is necessary The second section explains how memory pools are more to read the manpage on the particular system to learn efficient for certain types of dynamic memory allocations the specific details of the implementation. Many of the and reduce fragmentation. The final section looks at the statistics presented are available via other standard output of the systat(1) utility and explains the aspects of utilities from which the views take their names, e.g., the OpenBSD pool implementation which are relevant for netstat, iostat, vmstat. The particular systat view discussed understanding the statistics as reported by systat. in this article is the pool view and is only available on the OpenBSD implementation of systat. However, the same
Systat(1) Utility statistics can be seen using vmstat(8) with the -m option The sysat(1) utility displays a range of system statistics on NetBSD. On FreeBSD, vmstat -z will show statistics in a format similar to the familiar top(1) command, taking for zones, which are the equivalent of pools in the slab over the entire terminal and refreshing the values at allocator implementation. This article does not discuss specific intervals. Related statistics are grouped together aspects of slab allocation. and displayed on a single screen called a view. Different views display different statistics; for example, the iostat Memory Allocations view displays statistics related to I/O activity, and Memory allocations in the kernel occur frequently. It is the malloc view displays statistics on kernel memory therefore important that these allocations be fast and
34 11/2011 www.bsdmag.org 35 ADMIN OpenBSD Kernel Memory Pools: Monitoring Usage With Systat
efficient. Since the kernel manages the entire system, if implementation in OpenBSD as an example, memory for kernel performance is slow, it impacts the performance a given pool is allocated in 4kb pages. If a single page can of the system as a whole. Many requests for kernel hold 20 pool items, then a single memory allocation for a memory occur as user level programs make system pool creates 20 of these items which become immediately calls. If satisfying these memory requests is inefficient, available for use by the pool owner. The same benefits performance of user programs will degrade. exist for freeing memory. A single free operation releases Typically, the kernel’s general purpose memory allocator, multiple items with one function call. Also, because the
malloc(9), takes a request for an arbitrary piece of memory, items in a pool are of identical size, memory fragmentation rounds it up to a power of 2, and returns that amount of is reduced when the page is divided up. memory to the caller. For some types of allocations, this isn’t inefficient. However, for memory requests which are Output of the Systat(1) Utility large or of an odd size (i.e., not a power of 2), this can The remainder of this article looks at the interface and waste memory through fragmentation. implementation of memory pools in OpenBSD and The kernel often requests pieces of memory of an explains the pool usage statistics as they are displayed identical size and which are used for identical purposes. by the systat utility.
For instance, whenever a new process is created on The pool(9) manpage describes the interface through the system, the kernel must allocate memory for a which different parts of the kernel can initialize and destroy structure to hold information about the process such as pools, get and return pool items, and set other parameters the PID, information about signals and CPU scheduling, for pool management such as upper and lower bounds etc. Similarly, a vnode structure must be allocated for on the pool’s resources (pool items or pages of memory). vnode operations, socket structures for opening network These interface routines, as well as the backend functions
connections, and filedesc structure for open files. Due which implement them are in sys/kern/subr_pool.c. The to the various characteristics of these structures (in various pool related structures and other declarations are
particular, their size and the frequency in which they are in sys/sys/pool.h. allocated), using the general purpose memory allocator to A kernel memory pool is initialized with the pool_init() satisfy these requests is sub-optimal. function. pool_init() does not immediately allocate any A memory pool is an optimization whereby multiple fixed resources. This step is normally deferred until the first
sized pieces of memory are set aside for exclusive use request for an item in the pool. The pool_prime() function by a specific part of the kernel. These fixed size pieces of can be called to preallocate a specified number of items memory are called ‘pool items’ and the part of the kernel to the pool, but this function is not used in most cases. which initializes and uses the pool is the ‘pool owner’. A Although the GENERIC kernel creates many pools at pool can guarantee that a minimum amount of memory system boot time, it is not unusual for several of them to will be available to satisfy requests from the pool owner never receive a request for items during the uptime of the independent of the available memory allocated by the system. general purpose allocator. Each pool is dedicated to serving A pool dynamically manages its own resources based requests for memory for one type of data structure (vnode upon increased or decreased demand. Each pool tracks structure, process structure, etc). The size of each pool the number of items it has available for allocation. item is the size fo the data structure to be stored in the pool This number is decremented with each allocation and (e.g., on OpenBSD, each item in the socket pool is 424 incremented as items are returned to the pool. If a pool bytes, which is the size of the socket data structure). When is depleted (i.e., the number of available items is 0), the kernel needs one of these structures, it calls a routine additional memory will not be allocated until there is which retrieves an item from the appropriate pool. When the another request for a pool item. If, on the other hand, a structure is no longer needed (e.g., when the process exists, lower bound has been set and fulfilling a request will bring the socket or file closes), the item is returned to the pool the number of available items below the lower bound, where it can be used again the next time memory for that then additional memory will be allocated to the pool before particular type of structure is needed. the request is completed.
An important optimization (whose implementation Figure 1 is an example of output from the systat pool isn’t limited to memory pools) is the pre-allocation of command. The top line shows the number of users on the memory. This a achieved by making a single request for system, the load averages and system date/time. In the a large chunk of memory and carving up that chunk into middle of this line, in parentheses, is the number of lines smaller pieces which are ready for use. Taking the pool visible in the terminal window and the total lines output in
34 11/2011 www.bsdmag.org 35 ADMIN OpenBSD Kernel Memory Pools: Monitoring Usage With Systat
this view. On this machine, there are 123 pools, but only • HIWAT – the most pages concurrently allocated to the first 46 are visible. The up/down arrow keys or the the pool since initialization pageup/down keys will either scroll or page up/down to • MINPG – minimum number of pages to retain in the display the additional lines of pool statistics. The default pool is to refresh the screen every 5 seconds with new values. • MAXPG – maximum number of idle pages to keep in
The p command will pause the refresh operation. the pool (default value is 8) The output displays values pertaining to pool items • IDLE – number of idle pages, i.e., pages from which and pool pages. The values in the first four columns no items have been allocated after NAME apply to pool items; the last 5 columns of output pertain to pages of memory. The following briefly There are four ordering options which are invoked by describes the values in the columns: single key commands:
• NAME – the name of the pool • N – order alphabetically by NAME • SIZE – the size of the pool item in bytes • Q – order by value in the REQUEST column • REQUESTS – total number of requests for this pool • Z – order by total amount of memory allocated from a item since the pool was initialized given pool (SIZE * INUSE)
• FAIL – total number of failed requests • P – order by value in the NPAGES column • INUSE – total number of pool items currently
allocated and in use The r command will reverse the given sort order. The • PGREQ – total number of pages allocated to this pool o command will cycle through the orderings as listed since initialization above. • PGREL – total number of pages released from the The name of the pool and the size of the items are set
pool since initialization when the pool is initialized with the pool_init() function. • NPAGE – total number of pages currently allocated to REQUESTS is a running total of requests for items since the pool the pool’s initialization. INUSE is the total number of items which have been allocated from the pool and are actively being used in the system. This value will decrease as items are returned to the pool. Also, this value, when multiplied by the size of the pool item yields the amount of memory (in bytes) which has been allocated from this pool and is actively being used. The third ordering option mentioned above, invoked using
the Z command, orders the pools by this derived value (INUSE * SIZE). There is potential confusion
on this point, as the systat(1) man page refers to this as an ordering by size, which could lead one to expect an ordering by values in the SIZE column. Each pool item is associated with exactly one page of memory in the pool, and each page is kept on exactly one of three lists depending upon how many of its individual Figure 1. /usr/bin/systat displaying output for the ‘pool’ view pool items have been allocated:
36 11/2011 www.bsdmag.org 37 ADMIN OpenBSD Kernel Memory Pools: Monitoring Usage With Systat
one list for full pages – pages which have allocated all has been hit, increment a counter that tracks the number their items and cannot satisfy any more requests; another of failed requests and return immediately. The other case list for empty pages, which have none of their items occurs if a pool is depleted and has no more items to give
allocated; and the third list for partially full pages – pages out when a request is received. If the PR_WAITOK flag is not which have some items allocated but still have more set as part of the request and the attempt to allocate more available to satisfy additional requests. As pool items pages of memory from the system fails, then the request are allocated or returned to the pool, these pages will be will be unsuccessful and the number a failed requests will moved from one list to the other as their status changes be incremented. from being either full, partially-full or empty. Pages which The values in the columns PGREQ and PGREL are are ‘empty’ are also called idle pages and their count is simply running totals of the number of pages either displayed in the IDLE column of Figure 1. requested or released back to the system by the pool There are three functions available to explicitly set upper backend routines. The HIWAT value is the highest number and lower bounds on the resources (pool items or pages of pages ever allocated to the pool at one time since its
of memory) in a given pool. pool_sethardlimit() sets the initialization (N.B. – not the value set by set_hiwat()). maximum number of pool items which can be allocated Some of the pools created by the kernel are used for at one time. This value (which is not displayed by systat) internal operations. Others are for handling operations defaults to the maximum of an unsigned 32-bit integer. initiated from programs in the user space. Watching
The pool_setlowat() function sets the lower bound on the the statistics as they are refreshed, one can easily see number of items to keep in the pool for further allocations. the backend memory management activity at work, If an allocation will cause the number of remaining items requesting more pages to meet the demand for more pool in the pool to decrement below this low watermark, then items. To demonstrate quite obviously, one can execute
the backend routine, pool_catchup() will be called before the find(1) command over a directory with many files completing the allocation. pool_catchup() will request more (the /usr directory, for example). This causes a dramatic pages of memory from the system; these pages will be increase in demand for vnode structures as can be seen used to replenish the number of available pool items. by the spike in the number of requests for memory from Setting upper bounds establishes thresholds which the vnode pool. As the items are removed from the pool, define when memory should be returned to the system. If the backend pool allocator requests more pages of a spike in demand requires committing additional memory memory to keep up with the demand. This accordingly to a given pool, then this memory should be returned the increments the number of pages requested (PGREQ) and system once demand has subsided and the memory pages in use (NPAGE).
is no longer needed. The set_hiwat() function sets the maximum number of idle pages (i.e., empty pages) to Conclusion keep in the pool; this is the value in the MAXPG column of Monitoring kernel pool memory usage provides a glimpse Figure 1. If, as items are returned to the pool, the number into the activities of the running kernel. Pools provide an of idle pages exceeds the value set by this function, the efficient method for dynamic memory allocation and are idle pages will be taken out of the pool and returned to used for memory structures which are often allocated the system. By default, this value is set to 8. The value and are of odd sizes. Using pools can reduce memory in the IDLE column should never exceed the value in the fragmentation which might otherwise occur if memory MAXPG column. were allocated using the general purpose allocator. If
Both pool_setlowat() and pool_prime() will set the value in required, pools can be used to guarantee that parts of the column MINPG. This is the minimum number of pages the kernel will continue to have memory available for their to keep in the pool. The default value is zero. operation regardless of other memory constraints on the There are two scenarios in which a request for a pool system. The systat utility in OpenBSD (and the vmstat item can fail and increment the value in the FAIL column. utilityin NetBSD) return statistics about activity in the pool If a request is made for an item and the hard limit has subsystem. already been reached, the allocation routine checks
whether the requestor set the PR_WAITOK flag, in which case the routine will wait until the number of pool items in use PAUL MCMATH drops below the hard limit (i.e., an item is returned to the Paul McMath has worked as a Unix admin for 10+ years
pool) before fulfilling the request. If the PR_WAITOK flag is not in Europe and the United States. He has been using one set, then the routine will log a message that the hard limit BSD variant or another as his OS of choice since 2002.
36 11/2011 www.bsdmag.org 37 LET’S TALK FreeBSD 8.2 Against Ubuntu Server 11.10 FreeBSD 8.2 Against Ubuntu Server 11.10
An Objective Comparison of Two Power House Open Source Server Platforms, BSD Unix and Linux.
reeBSD is the dominate title holder in the BSD Although the FreeBSD appears to be a bit archaic, it is based OS market, but how does it compare to an up worth pointing out that FreeBSD’s simple CUI interface F and coming challenger if that perspective of Open allows the OS to be installed on devices that don’t have a Source server platforms is broadened? We’ll take a wide graphical display output, and only have a serial console. and ranging look at the strengths and weaknesses of each Examples would be the Alix SBC, and other server platform and let the reader decide. We’ll look specifically at platforms such as DEC Alpha’s and DecStations. Canonical’s latest release of Ubuntu Server, Oneric Ocelot, The typical steps of creating the partition table, allocating 11.10, and the latest release of FreeBSD, 8.2. slices, filesystems, follow and are pretty straight forward. Once the initial install is through, a myriad of other steps Installation Media are necessary, setting the root password, setting up FreeBSD is primarily distributed on CD iso’s either as a network interfaces, timezone, enabling ssh, etc, all of full size distribution weighing in close to 700 megabytes, which are post-installation. or also as a lightweight net-install if you have a good Ubuntu Server tends to focus primarily on the PC bandwidth to the Internet or local ftp source. Options enterprise server market though they have recently been are also available for USB thumb-drive installs as well. some interest in some of the embedded devices markets. Note, internet connectivity is not generally needed for As such, Ubuntu tends to explore the graphics modes standard CD distributions, either the 32 bit or 64 bit more freely, switching between various graphics and text versions. modes throughout the install process. It is possible to Ubuntu Server is distributed as well in CD iso format, install Ubuntu with a serial console, but it is by no means as well as a distribution that can be used on a thumb- as straight forward as FreeBSD is, involving changing drive. Both 32 bit and 64 bit versions are available as multiple system files, boot-loaders, etc. well. Internet connectivity is highly recommended as the installer downloads the latest packages from the repository during the install.
The Install Process FreeBSD’s installer hasn’t changed much in the 15 years that I’ve used it, since early 4.6 days... You have the quick, and expert modes, of course, and the most flexibility of disk layouts. The 8.x releases give more choices of filesystem types, including ZFS. One thing I have found annoying is the default sizes of /var are in most instances much too small for most configurations. Figure 1. FreeBSD Main Install Menu
38 11/2011 www.bsdmag.org 39 LET’S TALK FreeBSD 8.2 Against Ubuntu Server 11.10
Ubuntu’s boot screen launches the user into an menu Ubuntu Server doesn’t that allow a prompted install, and gives the user the chance to choose a guided disk layout or manual if • support kernel ZFS (Sun Zettabyte File System), but preferred. does offer BTRFS Ubuntu doesn’t by default enable the root account, and • limit RAM to 4 gig on 32bit hardware (default PAE in fact, requires the creation of a user account during install kernel included) that will be in the sudo group. That information is collected • use the Linux 2.6 kernel anymore (as of 11.10) after the disk layout is chosen, and the user account is • install software development tools by default (build- created during the install. Networking, timezone setup are essential) done during the initial installation process. One handy feature of the Ubuntu Installer is the Neither include a GUI by default. These are server Software Selection Box, or tasksel as it is called from operating systems after all. the command line. It lets you select several of the most common packages or configurations from one selection Package Management box. Both Operating Systems support remote package Typically, you might select to install SSH, or a full LAMP management and installation. They differ greatly and (Linux, Apache, Mysql, and PHP) stack. This is quite functionality. handy, as the installer not only install and configures Mysql, but let’s you set the mysql root password from FreeBSD has two primary methods for dialog boxes. No extra steps are necessary, and your up installing applications, Ports and packages and running with a LAMP server. One note, Ubuntu is still shipping Mysql 5.1 from it’s repositories, rather than the Ports is an organized directory structure of packages, more commonly available 5.5. Some differences each of comprised of pointers to remote locations of each the Installers: package’s remote location and dependencies. Executing FreeBSD doesn’t a make in a given package directory will retrieve not only that packages source, but all it’s dependencies and • require or expect Internet access compile them as necessary to complete the applications • initially configure network interface (post-install) build from source. • include a bash shell by default (available as a eg: package) • require a video display adapter $ cd /usr/ports/shells # [/usr/ports/shells]$ cd bash # [/usr/ports/shells/bash]$ make; make install //compiles and installs
Packages are compiled and complete compressed installation binary bundles of a given application and can be installed without compiling from source. An example would be to install the Bash shell after an initial install.
Figure 2. Ubuntu Server Main Install Menu Figure 3. Software Selectio
38 11/2011 www.bsdmag.org 39 LET’S TALK
This would be easily done in Packages by typing: On the ‘Net • Ubuntu Server – http://www.ubuntu.com/business/server/ # pkg_add -r bash // This will retrieve and overview install the binaries. • Debian KFreeBSD – http://wiki.debian.org/Debian_GNU/ kFreeBSD The FreeBSD package management system is both wonderful and a cause of frustration. It’s design is limited it’s handling of dependencies of local packages. This can be variety of high load areas. I use both systems in similar load a real irritation if you are installing a local package that has conditions, and each has it’s own distinct advantages. many dependencies. The dependencies have to be installed FreeBSD’s ZFS filesystem, has proven to outperform manually via pkg _ add -r some _ pkg, before the local package the Linux EXT4 filesystem on the same hardware, under will install. This is inherent in Pkg’s original design for the high i/o load. Note, ZFS requires 64bit hardware and a packages to be hosted by FreeBSD’s main package sites. minimum of 4 gigs of memory for successful operation. FreeBSD also seems to have an edge in network Ubuntu Server Uses The Package System That i/o under similar high load udp network load. If ZFS It’s Parent Os, Debian Uses, Called Advance filesystems are not used, then Ubuntu Server performs Packaging Tool, or APT much better with EXT4 against FreeBSD’s aging UFS2 filesystem. Ubuntu seems to better utilize the cache buffer It can also handle Redhat packages, using RPM, but than UFS does. APT is the primary method. APT uses a local cache Ubuntu’s has a pretty nice implementation of Logical of remote repositories for tracking and updating both Volume manager, which is tightly integrated in the the system and other package databases. installer, and can be enabled during the partitioning phase Once the server is up and on the network, executing of installation. It’s well designed, if software raid is desired. the following commands will update the database caches The FreeBSD kernel has other performance advantages locally, and allow searching for packages. over the typical Linux kernel, enough so that Debian released it’s distro with a FreeBSD kernel, called KFreeBSD. $ sudo apt-get update // Update caches from Ubuntu Server does virtualization very well, if you have repositories the hardware for it. The latest release 11.10 offers Xen, $ sudo apt-get upgrade // Upgrade installed packages and KVM. Other layer 2 hypervisors work just as well, $ sudo apt-cache search python // search for like VirtualBox, which is my favorite. The VM Host in my available package called python office is running Ubuntu server with 2 FreeBSD VM’s and another Ubuntu Server VM’s all happy in 4 gigs of ram. Additional packages can be added to the servers list of FreeBSD and Ubuntu Linux, though from completely sites, simply by the repository to the list of apt sources. A different lineages, share many common traits, and can very flexible system and works well. work well together.
Con�guration Files
FreeBSD uses a single system config file, /etc/rc.conf which is read by individual startup files in /etc/rc.d and it’s child directories... Network interface configuration, enabling the startup and options to various options are accomplished in /etc/rc.conf. Ubuntu Server uses /etc/network/interface for the primary interface configurations and /etc/init.d for startup of various system and other applications. Both OS’s use
/var/run and /var/log for obvious reasons. BILL HARRIS Bill Harris has 27 years experience in programming and Unix/ Performance, How Do They Stack Up? Linux Administration. His work experience includes data center This is one subjective issue, that has as many questions as environments as well as education IT. He hold a General Class answers... Ubuntu Server is the most similar to FreeBSD, Amateur Radio license, and enjoys working with the processing as far as the variants of Linux’s go. Each performs well, in a of realtime weather data.
40 11/2011
EUROBSDCON OVERVIEW EuroBSDcon 2011 From An Organizers Perspective EuroBSDcon 2011 From Organizers Perspective Although the idea of the EuroBSDcon 2011 was born during the EuroBSDcon 2009 in Cambridge, the preparations for the event actually started in 2010 in Karlsruhe, where some Dutch people offered to organize it.
he announcement was made that the 10th Our last challenge was to attract enough visitors. For EuroBSDcon would take place in the Netherlands that we needed a registration system online by the middle T from the 6th to the 9th of October 2011. Announcing of June. Unfortunately, it went live in the mid of August. exact dates without having done proper research was not How did that happen? Well, registration systems are a smart move as we learned along the way... more complex than we expected. They typically consist of Organizing a conference is simple. You just need a a website and a payment module. The payment module venue, a website, some hotel arrangements, speakers, communicates with a payment provider which talks with a registration system, a lot of visitors, some money and a bank. We faced difficulties with procedures and had you’re done. Right? contract issues. We were even rejected by one bank, As it turned out it was not that easy. Finding the right because we were in the Netherlands... location was the first challenge. Some of the places we Hotel arrangements were also a challenge. We looked into were not available during the conference learned that there was an electronics fair in Utrecht dates. Like Eindhoven for example, which was going to that same weekend, making it difficult to find affordable be closed on the 9th because of a marathon. This limited accommodations. As I mentioned before, we should have our choices. Apparently we should not have announced done some research before announcing the date of the a date up front before doing our research. Fortunately 10th EuroBSDcon. due to RedHat (yes, Linux) we stumbled upon Meeting Finally, after months of work and planning, Thursday Plaza in Maarssen. One of their big advantages is their the 6th of October 2011 came. At 8:00 we started to get pay per visit model, which makes conferences more everything ready in order to open the registration desk at 9: easily scalable. Instead of paying per room, a bit extra for 00. Around 8:30 the first enthusiastic visitors showed up. We using beamers and AV, some more extra for Wifi internet, quickly improvised a sign indicating that registration would and even more extra for coffee, tea, lunch, etc. Meeting open at 9:00 and that breakfast was served on the first floor. Plaza just charges a fixed price per person which includes At 9 o’clock we opened up the registration and everything everything what is needed. went quite well, until a few minutes passed 10 o’clock. Meeting Plaza’s scalability was exactly what we were A few minutes after the tutorials started, the first looking for. Despite having a lot of sponsors from the complaints arrived. There were not enough IP addresses beginning, our calculations showed that it would be nearly in the DHCP range. The internet came with the venue, so impossible to invite all the speakers that we wanted. By we contacted them and they started to call their network the end of August we were determined to make every provider, to get the problem solved. Unfortunately it could penny count. Then, unexpectedly two other companies not be fixed until the next day. contacted us and offered their support. We gained two On Thursday we had two tutorials, one by Kirk McKusick additional platinum sponsors this way, what gave us the which is always interesting if you want to know more financial room we needed to make the conference a real about the design behind FreeBSD. The other tutorial was 10th anniversary and to invite more speakers. the most visited tutorial of the conference: pfSense 2.0
42 11/2011 www.bsdmag.org 43 EUROBSDCON OVERVIEW EuroBSDcon 2011 From An Organizers Perspective
by Chris Buechler and Ermal Luçi. This tutorial covered PF this is a tutorial you should have attended. The third many of the features and changes in the 2.0 release from tutorial track on Friday consisted of two half day tutorials. the perspective of new and existing users. Unfortunately The first being a Dtrace tutorial by Tod McQuillin. Some of with so much information to cover, the tutorial went well the participants of the conference told me that they saw ���������� ����
beyond the allocated time and we had to cut it short great benefit from having Dtrace available for them. In the because the room was booked for another event in the afternoon Tod’s tutorial was followed by a NETGRAPH evening. tutorial by Adrian Steinmann, which gave great insights in On Friday Kirk continued with the second part of his what NETGRAPH can do. On Saturday, the conference An Introduction to the FreeBSD Open-Source Operating really started. System tutorial. The next tutorial track featured Peter Hans van der Looy had the honor of giving the first Hansteen, who gave his tutorial on PF, which provided keynote of the conference or we had the honor of having updates on the new PF syntax and features introduced Hans as a keynote speaker, depending on how you look at since OpenBSD 4.7. If you work or want to work with it. Hans took the audience on a tour through the DigiNotar
42 11/2011 www.bsdmag.org 43 EUROBSDCON OVERVIEW
incident and explored some alternatives for the trust issue On Sunday we started with our second keynote online. This resulted in a discussion between Hans and by Herbert Bos called: The eight-fold path to reliable Henning Brauer about viable alternative protocols. The operating systems. A very entertaining and interesting takeaway message was that it is hard to establish real keynote about the design choices made in Minix 3. trust online when everybody lies (sometimes). The most important one being that everything should The keynote was followed by talks about IPv6 on keep working despite parts of the system crashing and FreeBSD, Open vSwitch, Testing NetBSD automagically restarting. Hopefully this has provoked some thoughts and the 10th anniversary of PF. and ideas to make the BSD’s better. During the lunch break one of our sponsors treated us The first slot after the break featured a capsicum talk by to a dutch specialty (poffertjes – kind of small pancakes) Robert Watson. Capsicum being a practical approach to which were served on the ground floor, close to the fine grained access control. Another track in this slot was entrance. Having all previous lunches on the first floor, a ideal for who wanted to learn more about Minix after the lot of people stayed there. So it was my job to walk to the keynote. By following the Beastie Meets Raccoon: MINIX 3 first floor to point out the poffertjes to the visitors. On my as a BSD talk we could learn more about it very quickly. way to the lunch room I ran into some colleagues, who I think the Sunday program created a lot of hard choices asked if we had everything under control. Optimistically, for the visitors. Multiplicity, FreeNAS or learn more out I responded with yes, which proved to be a bad idea. how NetBSD started to use the fossil version management Pascale, one of the crew members, touched me on the system? Learning more about HAST and run the risk of back and uttered: We do have a problem. At the same missing out the latest developments in OpenSSH? Get time the smell of burned plastic reached my nostrils. Time introduced to ZFS from a system management point to turn around and take a look. As it turned out a power of view or get new insights of how to protect your data bar had burned through due to the difference in voltage in nowadays? And if you think these are difficult choices, Europe and the USA. Anyway, this smell of molten plastic remember that by following these you would have missed somehow solved the problem by driving our visitors the history of sendmail, the obsolescence of the OS, towards the poffertjes on the ground. insights into BSDcertficiation and the porting of OpenBSD After the lunch break we had to put one of our to Sun ultrasparc processors. contingency plans in place when we had a room overflow We are looking back at the EuroBSDcon 2011 as a for the NPF talk. Redirecting everyone to the plenary successful, entertaining and interesting conference, which session room solved this problem easily. The talk itself would not have been possible without the sponsors and of presented some really great ideas on handling traffic and course all speakers and visitors. It is very rewarding to processing the firewall rule set. Besides the NPF talk the see the enthusiasm and hear the positive feedback of other talks after the lunch break where about Webcamd, all who visited Maarssen. With visitors from 27 different nginx on FreeBSD and OpenBSD’s new suspend and countries it is safe to call the EuroBSDcon an international resume framework. After the tea break another talk about conference. PBI for FreeBSD and PC-BSD 9. For the people interested At the moment of writing it is still unclear where and in virtualization the Virtualization under *BSD: the case of when the EuroBSDcon will be held next year. Hopefully Xen talk provided insight into the Xen hypervisor. it will be known very soon. In the meantime we are busy The first conference day finished with the always trying to get the EuroBSDcon Foundation up and running. entertaining History of BSD by Kirk McKusick, after which The target of the EuroBSDcon Foundation will be to everyone was directed to the buses to get to the social provide support to the organizers of, share knowledge event at the Dutch Railway Museum in Utrecht. Everyone about and transfer resources to future EuroBSDcon got a tour around the Museum, in Dutch or English. In conferences. For example in helping with or providing the hindsight we should also have added a tour in German, registration system. but all the ingredients for a social event were present: interesting location, good food and the ability to socialize. JEROEN VAN NIEUWENHUIZEN We got a lot of positive feedback about part of conference. Jeroen van Nieuwenhuizen was the chair of the EuroBSDcon However it should be noted that it was organized by one 2011 organizing committee. In his daily life Jeroen works as a of our sponsors so we can’t claim the accolades. Without Unix Consultant for Snow B.V. He came in contact with Unix in Anja’s and Mona’s experience we would not have been 1997 and started to work with the BSD’s in 2002. His free time able to make the social event this good. So we are very activities include reading, recumbent cycling, speed skating and happy they organized it for us. herding his cats.
44 11/2011 The Passing of Steve Jobs: Not Just Apple’s Loss
I think it’s only appropriate that this magazine comment on the passing of Apple’s most charismatic co-founder. When I read Apple’s simple but tasteful words of memorial, I knew what I wanted to say. From http://apple.com/stevejobs:
Apple has lost a visionary and creative genius, and the world has lost an amazing human being. Those of us who have been fortunate enough to know and work with Steve have lost a dear friend and an inspiring mentor. Steve leaves behind a company that only he could have built, and his spirit will forever be the foundation of Apple.
Contrary to his company’s tribute to his life, the passing of Jobs is a loss to more than just Apple, his friends, and his family. Whether you love or hate Apple and Jobs, nobody can deny that his contributions to technology reach far beyond his own company’s products. Here is an excerpt from a February 26, 2004 post on the blog of outspoken FreeBSD community member, Grant Hayes (aka “Trollaxor”):
… only after Apple started modifying FreeBSD 4.x and submitting their modifications did FreeBSD progress to the 5.x branch. The advanced VM and SMP code that allows Mac OS X to run so efficiently is the very same code that finally put FreeBSD on the level with Linux. I run FreeBSD 5.2 on a four- way Xeon box at work and thank Apple every day. If it weren’t for the Mach micokernel from Apple we wouldn’t be able to do these nice things with FreeBSD now or probably ever.
The work and genius that Jobs put into Apple (especially after returning from NeXT) has had impacts on several major players. Android wouldn’t be where it is today if it weren’t for iOS. Even Microsoft had to follow Apple’s lead in order to create a Windows Mobile device that can compete. Mark Shuttleworth openly admits that OS X is the inspiration for his Ubuntu designs.
Jobs has affected so much more than just what Apple does. His vision sent ripples of positive innovation throughout the computing industry at large. The BSD community ought to be thankful for the life and work of Jobs every day.
Rest in peace, Steve.
By Toby Richards In the next issue:
- BSD Kernel Compiling - nsswitch, nscd and caching - OSPFv6 - and Other !
Next issue is coming in December!
������������������������������������ ��������������������������������
������������������������������������� ��������������������������
������������������������������������������������������������������������������������������������������� ��������������������������������������������������������������������������������������������������� ����������������������������������������������������������������������������������
���������������� ����������������������������������