conference reports
This issue’s report focusses on LISA LISA XVI only, so single failures aren’t fatal but only reduce capacity. XVI Sixteenth Systems Administration Conference To answer a query, the Web server (a custom package called gws) queries PHILADELPHIA, PENNSYLVANIA, index servers, document servers (cached USA pages), and ad servers, in parallel, and OUR THANKS TO THE SUMMARIZERS: NOVEMBER 3–8, 2002 keeps trying until it gets a response. KEYNOTE Each query may involve a dozen or more servers, using whichever reply comes in for LISA ’02 SCALING THE WEB: AN OVERVIEW OF the fastest (the average query time is .23 Josh Simon, who organized the collecting of GOOGLE (A LINUX CLUSTER FOR FUN seconds). Before the query reaches a the summaries AND PROFIT) Abiodun A. Alao Web server, however, it passes several Jim Reese, Chief Operations Engineer, Paul Anderson Google load balancers, both global and local, David Berg which use various methods (including Robert Beverly Summarized by J.D. Welch round-robin and least connections) to Kuzman Ganchev We all know, use, and love Google, but Jim Hickstein choose which servers to query. Rob Kolstad how do they make it work? In this Martin Krafft engaging talk, Jim Reese explained how “El Cheapo” PCs are used to maximize Renuka Nayak custom software, massive replication reliability through replication. Fault tol- James O’Kane and expendable, commodity hardware erance is kept very simple; timeouts are Will Partain have allowed Google to answer 150 mil- in the milliseconds, and machines are Peg Schafer restarted automatically and regularly J. D. Welch lion Web search queries a day. polled for their status. Racks of Steve Wormley The core technology that separates Garry Zacheiss machines are very dense, with 80 half- Google from other search services is the depth 1U boxes in each, along with PageRank system developed by founders paired switches, load balancers, and Larry Page and Sergey Brin while gradu- Gigabit uplinks to the routers. All disks ate students at Stanford University. This are local (100–120Gb/machine); large system aims to objectively rank Web fans are mounted atop the rack and heat content by popularity; according to is drawn from the space between the Reese, “a page’s importance is the sum of machines in the center of the rack. All the aggregate importance of the pages the machines run a “Googlized” distri- linking to it,”so a page linked to from bution of RedHat Linux as well as pro- the New York Times is given more weight prietary tools for serving content and than one linked to by a high-school system monitoring. newspaper. In addition to assessing pop- ularity, hypertext analysis is used to For comparison with the new, very quantify the importance of elements on organized racks, Reese showed photos of a page (e.g., larger text is probably more historical configurations, including a important). custom-built 1U machine with four motherboards, eight disks, eight NICs, To get a sense of scale of Google’s chal- and one power supply, which was con- lenge – there are 3.8 billion pages and figured with the disks mounted over the 256 million Web users, and 85% of them processors separated by a sheet of Plexi- use search services. Given this, any single glas (!). machine will always be too small for the task, so index and page data is divided up into pieces, called “shards,”which are distributed across many machines and multiple data centers. Thus, traffic is scalable by replication; the index is read-
64 Vol. 28, No. 1 ;login: INVITED TALKS deal about the specific component or first authenticating, and prevents the use SECURITY ON MACOS X service at work. of startup commands (which can make EPORTS
the machine act like a FireWire disk or R John Hurley, Apple OS X makes a point of separating be booted into UNIX-permissions-free authorization from authentication, a Summarized by J.D. Welch OS 9, for example). Hurley began by saying that Apple is in move designed for next-generation ONFERENCE an interesting position to deal with secu- applications, including smart card This talk was a little marketing-heavy C rity issues, as they manufacture the access, for which they are developing an and didn’t delve into technical details of hardware, firmware, operating system, SDK (called Smart Card Services) in col- the various systems implemented in OS and often the end application, so a great laboration with HP, Intel, and other ven- X beyond their GUI expression, but it degree of integration is possible in OS X dors. did provide a good introduction to the various services available to the OS X security features. Out of the box, OS X is reasonably well user or administrator. Since OS X is based on BSD, many of the locked down: Services like SSH, HTTP OS X security tools are ports of standard are off by default (but are easy to enable ETHICS FOR SYSTEM ADMINISTRATORS: UNIX tools, oftentimes GUIfied with a – from GUI or command line – if you DILEMMAS FOR LISA 2002 ATTENDEES Cocoa (native OS X Objective-C frame- know what you’re doing), no ports are Lee Damon and Rob Kolstad open, and the root account is disabled work) front end. For example, the Shar- Summarized by Steve Wormley ing and Firewall control panels are a (sudo is used for administrative access). Unlike the medical profession, which has front end to ipfw. OS X also offers Ker- OS X honors UNIX user/group/file per- had thousands of years to develop ethi- beros, OpenSSH, OpenSSL, and other missions and is designed to be a multi- cal standards, system administration familiar UNIX tools in its default instal- user OS. ethics are new. The mapping of conven- lation. Obviously, the use of familiar, The Keychain is a cornerstone applica- tional communications such as paper often open source, packages is a depar- tion, and was given much play in this mail and the telephone do not work in ture from (and significant improvement talk. Accessible to all Cocoa, Carbon, the realm of email and Instant Messag- over) OS 9. and UNIX applications running under ing. The quantity of sensitive data online A primary goal in designing the OS X OS X, the Keychain provides an and issues such as identity theft con- security architecture was to make it easy encrypted environment to store pass- tribute to awareness of the need for to use these important features. Addi- words for Web sites and file servers, ethics and privacy guidelines in new tionally, although many tools are pre- encrypted disk volumes, and the like. technology. sented plainly for users, they are con- Users “unlock” the Keychain with a mas- Since computer ethics is a new area, figurable beyond what most users would ter password, and applications can store novel situations and their attendant bother with – good news for longtime and read data from the Keychain. Addi- problems now happen at “Internet UNIX users and security types. Also, tionally, all Keychain items include an speed.”We as system administrators Software Update encourages users to access control list for fine-grained con- need to have knowledge of ethics, pri- keep up-to-date with patches, as it auto- trol. vacy, and security so that we can protect matically polls for and delivers updates Another highlighted technology was the rights and still get work done. directly to end users. ability of Disk Copy, a utility available One definition of a professional is a per- OS X implements a Common Data on all installations of OS X, to create son who conforms to the technical and Security Architecture API, which pro- encrypted disk images. Once the image ethical standards of a profession. For vides an expandable set of crypto algo- is created, it can be mounted (with suc- system administration to be regarded as rithms to various applications, including cessful authentication) read/write, a profession by the outside world, there- the Keychain (encrypted user informa- burned on a CD for transfer, etc. fore, ethical standards need to be tion store) and Disk Copy (which can The physical security of Apple hardware addressed. encrypt disk volumes). These “layered has also been considered. The XServe 1U services” include file signing and certifi- rack mount server, for example, sends A distinction should be made between cate management as well as APIs for messages to the console when physical ethics and policies. Since policies are adding plug-in modules for additional security of the rack is compromised; well defined and generally not open to services. With this modular architecture, other Mac models can be “locked” with interpretation, establishing a site policy developers can make use of security ser- the Open Firmware Password, which will often eliminate many ethical prob- vices without having to know a great prevents booting the machine without lems.
February 2003 ;login: LISA XVI 65 Ethics in the context of computer net- delivered facts and ideas that, if nothing lion dollars made available to NASA works pertain to all privileged users, else, were entertaining. every year and the 300 million the RIAA including anyone with access to others’ loses to “piracy” per year, this figure He isn’t a lawyer – he stressed this fact information, even if that access is acci- clearly indicates there’s something several times – so he approached his dental; even help desk personnel, for severely wrong. Advertising isn’t evil, it’s topic from a “common-sense” angle. example, need to be included. necessary. Rather, the lack of regulation Spam, or Monty Python’s breakfast deli- and control is what constitutes the prob- Lee and Rob went on to present five sce- cacy, is all those emails you never asked lem. Spam is the cheapest method of narios involving ethical dilemmas for for – commercial mails advertising get- advertising since it mostly raises costs system administrators: rich-quick schemes, mortgage loans, for the recipients. It is marketing with a advertisements for penis enlargement 1. A project you worked on at a previous bullhorn, as Dan put it. He wants to devices, and other breathtaking new client had a flaw which could kill peo- “take back the Net.” technology you wouldn’t lack before or ple if not corrected, but you only real- after the spam hit you. Dan started out Current anti-spam methods almost all ized the flaw while at your current by presenting a short history of his come in the form of a filter-and-block client, working on something similar, involvement with the Net and his expo- setup on the recipient side. As effective and could lose your job if you dis- sure to spam, and then proceeded to lay spam filters are becoming more and closed the flaw. out the numbers of an 80-day research more of a marketing technique of big 2. Your boss asked you to read the CTO’s period, in which he received one spam ISPs like AOL and gmx.net, the voices email to look for evidence of wrong- every 29 seconds. Even using a fairly around the “censorship” buzzword seem doing. You found a problem, reported restrictive set of anti-spam techniques, to be getting louder and louder. Censor- it to your boss, and nothing was done. he claimed the ratio of ham to spam he ship, in Dan’s view, is ubiquitous rather Now what do you do? receives is about the same as Earth’s than evil. “Abolish Censorship” may mass to Jupiter’s. But to place his figures sound good but it reveals how little is 3. The third scenario was the often- into relation to the real world, Dan known about the topic. People tolerate repeated case where the boss wants quotes hotmail.com as being burdened censorship more than they are willing to the root password but is not compe- by one billion spam messages per day. acknowledge, and yet scream at the idea tent on the system. How would you of having someone filter their mail. handle it? He attacks the problem from two sides, starting with the constitutional. Free- Dan sees little use in current methods, 4. In the course of routine administra- dom of speech seems to be commonly such as whitelists and confirmation sys- tion you discover that your boss is dis- misunderstood and extended to argue tems. He wants a legal solution, and if cussing doing something evil, such as for spam. Yet freedom of speech has not on a global level, then at least within going to a competitor with customer exceptions (e.g., screaming “fire” in a the United States as a starter. However, lists. Now what? public theater for no reason). You can he couldn’t lay out a strategy for how 5. You are providing network connectiv- say what you want, Dan pointed out, such a law would be enacted and con- ity to a neighbor with children, and “but I don’t have to listen to it, I can dis- trolled, for which he isn’t to blame — the children receive pornographic agree, and [most importantly] you can- anti-spam is a challenge to the entire email. What do you do? not make me pay for something I don’t infrastructure and requires a lot of want to hear.”Taken together with free- cooperation, from the MTA author to THE CONSTITUTIONAL AND FINANCIAL dom of the press (I can print or refuse to the ISP, from the government to the end ARGUMENTS AGAINST SPAM print whatever I want, and so can you), user. He wants a global opt-out mecha- Daniel V. Klein and the constitutional argument against nism rather than one focused on indi- Summarized by Martin Krafft spam is right there: you are forcing your vidual advertisers. spam to be printed on my press, and I A trip to Dan Klein’s home page (http:// Dan’s talk, albeit very amusing, did not have no choice but to receive it. What www.klein.com) reveals that he’s a geek really offer anything new. Some audi- Dan criticizes is that spammers seem to leaning toward the humorous. In his talk ence members came to the talk to be misinterpret freedom of speech as a on the constitutional and financial argu- comforted about their spam problems, guarantee of an audience, and freedom ment against spam, he used exactly that others to get an idea of what spam is of press as a free method to print. tack. “Spam steals my time” could be about. As such, Dan succeeded in reviv- seen as the motto as Dan proceeded to His financial argument against spam ing the subject and making it a promi- unroll his theories on preventing spam, claims that spam costs the American nent one, for a number of keeping his audience focused while he people in the vicinity of 165 billion (!) dollars per year. In contrast to the 15 bil-
66 Vol. 28, No. 1 ;login: anti-spam-related topics and discussions accident, the Persian Gulf war, the battle as $2,000 to build your own. Preston were evident throughout the remainder of Mogadishu. Yet several in the audi- recommends buying enough disk for EPORTS of the conference. ence, from outside the US, thought it two full backups and many incremental R was not just an American trait. backups. Then connect arrays to clients RISK-TAKING VS. MANAGEMENT or backup servers and make backups. Paul Evans MAKING BACKUPS EASIER WITH DISK
Finally, make duplicates (note that ONFERENCE W. Curtis Preston, The Storage Group duplicating is different from backing up) C Summarized by Jim Hickstein Paul gave a post-mortem of a dot-com Summarized by Renuka Nayak of what is on your disk to tape. One company, Webvan, extrapolating from The take-home message of W. Curtis might even want to place another disk that experience to the broader view that Preston’s talk was that system adminis- unit off-site and replicate to it. Except in social misperceptions of risk skewed trators should back up to inexpensive catastrophic disasters, one can easily business decisions and contributed to disks frequently while duplicating disk restore from disk. the dot-com bubble. He also looked at backups to tape. Doing anything else Preston then went on to say why using why our profession did not have enough might lead to situations that SUCK (a disk is better than using tape. Disk does credibility with management to influ- mantra that was chanted throughout the not require a constant stream of data ence those decisions. interactive talk). The presentation was and neither is there the need to multi- well-delivered and peppered with real- The essence of capitalism is putting plex, as is the case for some tape drives. life examples that Preston had encoun- assets at risk in the service of profit. Pro- He claims that if disk backups are multi- tered throughout his career. fessional financial managers get paid to plexed, then the tape copies can be easily balance the equation of assets, risk, and Preston first outlined some of the de-multiplexed without a performance profit. But managers are people, and advantages, disadvantages, and chal- penalty. Furthermore, since disk arrays people tend to underestimate familiar lenges associated with tape drives. Tapes can be protected via monitored RAID, risks and overestimate unfamiliar ones. and tape drives are high speed and low the loss of a single disk would be moni- cost, which makes them good archival tored and repaired. Making disk-to-tape Unfamiliar risk abounded in the dot- solutions. But tape backups take a long copies are easier than making tape-to- com world, but it was asymmetrically time, and newer, higher-speed drives are tape copies, and full backups can be per- distributed. Taking Webvan as an exam- becoming more expensive. Furthermore, formed less often, saving network and ple, the grocery business is pretty well it is difficult to make off-site tape copies CPU utilization. understood: Financial, operations, and with a stand-alone drive, which needs to even software development risks, were So, why should we even use tape at all? swap tapes. When trying to access the familiar. But risks in IT were unfamiliar Preston argues that tapes are still good tape in the drive, one might run into the to management. The “prevailing doc- for archiving purposes so that older problem of not having the desired tape trine of risk” changed: In 1999, it was backups can be available. Tapes are also in the drive. Challenges to using tape as about not appearing above the fold of much cheaper than disk, allowing for the only backup medium include the the Wall Street Journal; in 2000, the slo- multiple, stable copies to be stored “on time it takes to make tape-to-tape gan became “five nines,”whether mer- the shelf” or off-site. Furthermore, tapes copies, the rigors of regularly perfuming ited or not. are not susceptible to file-system cor- full backups, the limitations on writing ruption, as disks may be. The result was a business that overspent to a single tape drive from two shared on redundancy (the larger perceived servers, and the inability to know To find out more information, email risk), while making fatal errors about whether a tape is in good condition until Curtis Preston at curtis@thestorage- the fundamental business model. Some you actually need to use it. mountain.com. people like to shop online, from a list Using inexpensive disk arrays as a pri- “WHO ARE THESE PEOPLE?” INTERNET and with a two-day lead time, but many mary tool in backups in addition to GOVERNANCE, PEERING, AND LEGISLATION others have a different, opportunistic using tape is an excellent way to address Paul Vixie, Internet Software style that only works in an actual store. some of the challenges presented above, Consortium Bad acquisitions, over-aggressive growth Preston suggests. There are IDE/ATA- targets, and bad marketing decisions Summarized by Robert Beverly based disk arrays that are addressable via sank the company. Mr. Vixie, a self-professed “graybeard” Fibre Channel, SCSI, Firewire, NFS, and and “member of the loyal opposition,”is Paul gave several other examples of this CIFS and which can use RAID configu- a long-time programmer and main- misperception of risk, calling it peculiar rations. These units are as low as $5,000 tainer of BIND (a software implementa- to American society: the Challenger for off-shelf varieties, and it costs as low
February 2003 ;login: LISA XVI 67 tion of the Internet’s domain name ser- physically in the United States, England, would go to the government to get a vice). Mr. Vixie’s talk explored some of Japan, and Sweden. As one of the long- business license. Mr. Vixie concluded the the changing dynamics as the Internet term participants in the health of the talk by warning of the dangers of people metamorphoses from research network global DNS system, Mr. Vixie is very in decision-making positions who “don’t to commercial network to a component concerned with current politics that may understand the impact of those deci- of national security. The talk was timely circumvent the original “graybeard” sions.”He urged attendees to get given the recent denial-of-service attacks policies. Despite stating that everyone involved. on the root name servers. should be very concerned with recent policy directions, a general sense of pes- NOBODY NOTICES UNTIL IT’S BROKEN: Because of the academic nature of the simism emerged that those with “guns SELF-MARKETING FOR SYSADMINS early Internet, resources were given away and money” would eventually prevail. Moderator: Lee Damon, University of freely, as needed, by a loose collection of Washington individuals. Today many of these A second resource at stake is IP Panel: Karen Ken, Dan Klein, resources have become valuable com- addresses. IP address space, once abun- Strata Rose Chalup modities. Examples include IP address dant, is now a valuable resource. An space, domain names, top-level DNS organization may have either provider- Summarized by Abiodun A. Alao domains, autonomous system numbers, assigned space or provider-independent The session was devoted to why system and protocol numbers. These shifts have space. Smaller organizations requiring administrators are not very popular with produced a variety of stakeholders, all fewer addresses generally must obtain other staff and why their point of view is with different motives. The talk focused addresses from their providers. difficult to convey. They are generally repeatedly on ICANN (Internet Corpo- Provider-assigned space allows larger perceived as overpaid with unclear job ration for Assigned Names and Num- service providers to aggregate the rout- schedules. It was noted that sysadmins bers), a government-sponsored entity. ing announcements of their customers make the first error in introducing their An example of a current area of conflict into a single aggregate. Limiting the total role. How do you explain what you do to is ICANN’s control over the top-level number of routes in the global Internet someone who does not have any idea DNS domains. Many Internet users feel helps maintain its health and stability. what the term “sysadmin” stands for? that ICANN’s policies toward new top- The downside to provider-assigned Here are some responses: “I work with level domains is unjust. In fact, Mr. space is that if an organization wishes to computers”; “I make the Internet run”; Vixie’s contention was that because change providers, it must forfeit its cur- “I manage computers.” ICANN is a government-sponsored rent address space (which belongs to the It is essential that you are seen as a per- entity, it tries to be all things to all peo- provider) and obtain new space from its son, a member of the team and one ple and thus fails to serve anyone. new provider. Obtaining new space whose contributions are valuable in the requires renumbering the IP addresses “The Government is coming and they realization of the goals of the organiza- for all of the machines the organization want to take our toys.”The operation of tion. We must also make people under- owns. Therefore, there is a large disin- the root servers, so-called RSOs (Root stand what we do in concise terms and centive to switch providers, giving the Server Operators), is a clear example of a in ways that they see how we contribute existing large providers a distinct com- loosely organized resource that has to their ability to meet their tasks. petitive advantage. Currently, obtaining become part of the critical infrastruc- Avoiding all the techno jargons will go a provider-independent addresses requires ture. How one becomes an RSO is a long way toward making us understand- providing justification to a regional reg- question with no answer today. Until Dr. able and more acceptable. In the alterna- istry for a minimum-sized block. The Jon Postel’s death, he alone made the tive we may have to teach people to size of this minimum allocation is deter- determination. The original intent was speak our language (even at the risk of mined by the members of the registry to distribute the root name servers training them to take over from us). who themselves are often network oper- among commercial, research, and edu- Finally, we must also learn the language ators, creating an inherent conflict of cational entities in different countries, of business since we serve the business interest. such that there was enough natural dis- world. trust between operators to prevent a Mr. Vixie noted that while current Many of us complain we do not get any problem. No single entity should con- address-space policy prevents competi- respect from coworkers. Can we do a trol, or be able to control, the entire sys- tion in many respects, the worst-case better job at marketing ourselves? While tem. Specifically, no single government scenario is that government would take it may not always be possible to make should be able to take over the whole over the registries’ duties. One would others see things from our point of view system. Today the root name servers are then have to go to the government to or even understand our role, a sysadmin obtain IP address space, much as one
68 Vol. 28, No. 1 ;login: who takes time to explain what we do 2. Let management understand what “Make yourself a light. Be the illumina- and why will be doing a good job and you do, that you are not just “staring tor” (Karen Ken); “Whatever you do, EPORTS will help improve our image. If people at computers.”Take the manager own it” (Dan Klein); “You make things, R are not savvy, make use of pictures. around; draw analogies. Send in peri- that’s the goal. Work as a team with a Often we are seen as capable of provid- odic status reports of major accom- sense of duty” (Strata Rose Chalup); “Be
ing the silver bullet to all problems. plishments. Even if management does not ashamed, but be ye not arrogant ONFERENCE When the system fails to meet the not demand this, it’s a good idea. either” (Karen Ken). C “expectations” of the customer, we are 3. Get your users adequately informed seen as incapable. When people do not SYSADMIN, STORIES, AND SIGNING: by warning them about changes. Do understand or lack the capability to LEARNING FROM COMMUNICATION EXPERTS you let them know ahead of time effectively use the IT solutions provided, David Blank-Edelman, Northeastern when bringing the system down? Do their tendency is to blame the IT expert, University you warn them about the database especially the system administrator. Summarized by Jim Hickstein server? Send mass mailings; send Sysadmins have to talk to each other, Granted, no matter how hard you try, enough, not too much. Package your and to “other species,”in other fields, you can’t get everyone to cooperate or regular suggestions for users in “Use- especially when diagnosing system and appreciate what you do. For instance, ful Tips.”Give users a chance for feed- user problems. The speaker brought per- how do you deal with a marketing back. It is legitimate to occasionally spectives from two other disciplines: sto- department that has the attitude, “We’ve ask, “Is it useful?” rytelling and interpretation (specifically sold this product, you design it,”or with 4. Create opportunities and look for the in American Sign Language). your fear that the marketer is misrepre- next problem to be solved. Look for senting what you are developing? Storytelling has a long tradition in the opportunities to make yourself valu- sysadmin community, but it has an aca- Or what if your manager is technically able. That’s part of self-marketing! demic underpinning that most sysad- savvy and brought back a piece of IT Find a niche for yourself. When peo- mins aren’t aware of. Its mastery equipment from a trip. It is acceptable to ple know who you are and what you requires application, study, and practice. tell him, “I don’t try to do your job; you do, they will come to you. How else Yet in 20 minutes, the speaker gave a shouldn’t try to do mine.”Our response can you keep your reputation as a veritable short course in storytelling, to these and similar issues is careful edu- miracle worker? Do something differ- which anyone would do well to take. He cation so that our colleagues see the ent: for instance, publish an article. used the various methods along the way, error in trespassing on territory related 5. Keep a good problem-tracking system. repeating the part about repetition, to IT. Return phone calls and reply to using silence for effect in the part about Attitude is very important. Be coopera- emails. If a problem is not resolved silence. (The slides are good, but they tive and courteous. You can do it right quickly, acknowledge and give feed- don’t do justice to this performance.) and keep people around or do it wrong. back. If you cannot get a problem Stories are good for sequential or related Any unpleasant situation can be made solved, do not blame anyone; and if events; making diverse information even less pleasant by a negative attitude. you don’t know, say so. “I will research coherent; passing on lessons (either it” is a good response. Be honest about This is about marketing ourselves, so overtly or implicitly). They fulfill social your capabilities. here are some helpful hints: roles, as in establishing one’s member- 6. Take a vacation. Here is a cool sugges- ship in a community. Stories make expe- 1. Marketing is about educating. Educate tion: have some toys on your desk that rience reproducible and reusable, and people around you to help them could relieve tensions or put them in a they do so safely (i.e., with a happy end- understand how our jobs are interre- box with a label that reads, “Five- ing). Stories are good for constructing lated. Cultivating relationships helps minute stress relief box. Feel free to layers, which the listeners can then fol- in achieving this. Reveal yourself in use.”When you are able to get away low, especially in complex technical situ- ways other than technical; hobbies for vacation, put things in place that ations. and other general interests can help us make the system work while you’re connect to other people. In addition I won’t try to reproduce the whole gone. NO “I’m going for a week; let’s to books, mouse and toolkit, place course here, but the lesson was clear: If see how they cope” attitude. pictures of families, pets, etc. in your you learn to tell stories better, you will workspace. In closing, the panel members were in be a more effective sysadmin. agreement on the following: He told a story about a difficult network problem escalating through a front-line
February 2003 ;login: LISA XVI 69 technician via online chat. It was an oft- PERL 6: THE SCIENCE OF PERL, AKA STUDIES One new aspect of Perl is that variables repeated scene of a failure to communi- IN THE BALLISTIC ARTS will have properties, such as a compile- cate. But the technology was not really Larry Wall, Creator of Perl time property of “constant” as well as the problem: The parties seemed to Summarized by Steve Wormley runtime properties. These properties can speak different languages (though both Larry started off with a brief overview of also be accessed as methods. There are spoke English) and had different back- where Perl came from. Perl has roots in also new smart match and smart switch grounds and mindsets. linguistics, computer science, art, and statements. Explicit exception handlers will now exist (try, catch, throw). New What was needed? An interpreter! common sense. In addition he discussed how Perl draws from ecology, math, and OO support will include opaque data The speaker then went on to talk about golf, among other things. which must be accessed by methods, and interpretation, in general as well as how also the possibility for multimethod dis- it differs from translation. Interpretation Perl was described as initially a way to patch (the functions called depend on is live, and the interpreter can’t go over combine the “manipulexity” of C and the types used). the “source text” more than once. Gener- the “whipuptitude” of shell in one lan- The new pattern-matching support ally defined, interpretation creates in the guage. And Perl was designed to con- means that patterns are no longer inter- mind of the “target” the same idea that tinue evolving into both. One major polated as strings, the use of brackets is exists in the head of the “source.”It is feature of Perl is that it is designed to consistent, there are no postfix modifiers subtle and difficult, especially when no hide the fancy stuff. In addition Perl (all prefix or defaults). Other features direct translations exist: The interpreter behaves as a natural language. Some of will be new modifiers, meta-syntax, full must be able to move in two cultures these aspects of the language include: grammar support, easy parse-tree gener- and make the necessary mappings you can learn as you go, you can learn ation, and grammar inheritance. between them, accurately and in real something once and use it many times, time. there are many acceptable levels of com- Finally, Larry mentioned that Perl 6 will petence, and there are multiple ways to be able to use Perl 5 modules. And to In ASL, for instance, pronouns are spa- say something. create Perl 6 code there will be a Perl 5 to tial: One doesn’t say, “And then he said Perl 6 translator. X, and she said Y.”One creates people in Another important part of Perl is the culture. The Perl culture, like some oth- space, in front of the speaker, (Z is here HOW TO WRITE A BOOK WITH SOMEONE and T is over here), and then the “say- ers, accepts newcomers, is okay with YOU DON’T KNOW: INTERNET COLLABORA- ing” happens in that particular place. subtribes, encourages sharing, captures TION FOR THE TRULY GEEKY Another one of the many challenging knowledge, encourages cooperation, and Tom Limoncelli, Lumeta; Christine aspects is shown by the difference has fun. Perl 6 started by placing a Hogan, Independent Consultant request for comments for the new lan- between “leave the party” and “leave the Summarized by Kuzman Ganchev car at home.”Polysemous words, density guage. They received 361 comments. Tom and Chris started the presentation and context mismatches, preserving reg- The Perl 6 team decided to take the by comparing the process of writing a ister...the list goes on. Affect and intent Winnie-the-Pooh Approach: Think book to that of managing a system must be conveyed: The way something is Things Through Slowly. They wanted to administration project. To write their said is very important to its meaning. It keep everything good and throw out book, they used familiar tools such as is what the listener hears that matters. everything bad. The final goals for Perl 6 were simplification, power, better OO SSH, CVS, and make, and had to deal Mapping this to sysadmin communica- programming, better functional pro- with common system administration tion doesn’t take much imagination. The gramming, and better pattern matching. problems: security and data integrity. potential for misinterpretation is large; Their job was more difficult in that they SA has many “rich points.”One must use Some of the new features and changes lived two (and later, five) time zones feedback to detect a snag, then go back include: no more double parsing, com- apart and didn’t know each other. The and find the knot. ments work better in patterns, simpler book, The Practice of System and Net- precedence rules, removal of special work Administration, is divided into four He finished with a taxonomy of useless variables, no more parentheses on con- parts, 32 chapters, and three appendices. support email requests, one of which ditionals (now whitespace dependent), The presentation focused on how they read in its entirety: “Something is wrong and blocks are now closures. Full type had gone about writing their book. and I have know idea what.”(The signatures will exist, there is a new alias- speaker’s first reaction, before seeing the ing operator, and there will be vector First, they decided on a set of standards subject line, “Printer help,”was, “Yes, I operators. for formatting, tools they would use, and have days like that myself, sometimes.”) terminology (such as “customer” vs.
70 Vol. 28, No. 1 ;login: “user”). Then they used a top-down The product encourages a workflow: circuit was considered, but distance and approach to plan out the rest of the deploy on baseline first, tune it using other factors argued for using a packet- EPORTS book. existing tools, then capture. switched WAN. In this case, they wanted R to maximize the link utilization, for best They divided the work by splitting up The system embeds version control. cost-effectiveness. the chapters between them, and speci- Replacing a managed server can be done ONFERENCE
fied an explicit development cycle. They in minutes. Objects inside the system The basis of the system is point-in-time C used their scarce meetings to do high- include resources of various resource imaging (snapshot). You dribble a copy interaction brainstorming, used the types, each type having a resource han- of a recent snapshot to the remote site, phone for problem-solving sessions, and dler. The handlers are deployed on the constantly. The snapshot interval, and organized the logistics via email. They remote agents to do the capturing thus the data rate (given a fixed size) highly recommended automating as (pulling) or management (pushing) of depends on the question, “How much much as possible. For example, they applications and state. The system is (new) data can we afford to lose?” If it’s used CVS to automate synchronization, extensible by adding resource types and 24 hours, that’s the cycle; you need to Perl to generate the tables, and make for handlers. copy N GB per day, depending on the pretty much everything. They used open size of the data set for each application. The field is wide open for new research protocols, such as SSH and LaTeX, so in modeling, config generation, rollback, The copy uses traffic-shaping to limit that they could work from any platform. policy-based management, among other the transmission rate to a fixed upper The presentation ended with some com- areas. bound, and IP QoS to guarantee mini- ments about writing a book. They mum bandwidth equal to the maximum warned that the financial rewards are GEOGRAPHICALLY DISTRIBUTED SYSTEM FOR bit-rate, to ensure completion within the not likely to be great – minimum wage is CATASTROPHIC RECOVERY cycle. above average – and that it takes a lot of Kevin Adams, Naval Surface Warfare work; they devoted two years of their Center EMBRACING AND EXTENDING WINDOWS lives to the task. Finally, they gave some The speaker described a disaster-recov- 2000 advice for aspiring authors: Interview ery system that continuously copies data Jon Finke, Rensselaer Polytechnic Insti- your publisher as you would an to a backup data center over a wide area tute employer, negotiate hard on contracts, network, at a steady rate that is just fast The speaker described a meta-directory and retain a lawyer. enough to meet the requirement to not integration project that provides all stu- lose more than N hours/days of data. dents, faculty, and staff with a single REFEREED PAPERS Constant network utilization maximizes username/password for all computer the cost-effectiveness of using a switched system access. Rather than modifying all SERVICE, RISK, AND SCALE WAN rather than a private line. IP qual- authentication clients to use a central Summarized by Jim Hickstein ity-of-service (QoS) guarantees ade- server, the username and chosen pass- APPLICATION-AWARE MANAGEMENT OF quate total throughput. The minimum word are pushed out from a central sys- INTERNET DATA CENTER SOFTWARE and maximum data rates are nearly tem to several different client systems, Alain Mayer, CenterRun equal. including Active Directory (AD). The speaker described a new product When you can eliminate all small “sin- The institution needed AD for students, that can help manage large groups of gle” points of failure, the entire data cen- faculty, and staff; Exchange email for Web servers and their related applica- ter becomes the new single point. staff; and password and account syn- tions. The product guides the user to High-availability (HA) solutions like chronization across all platforms. Each “capture” the essence of an existing local, shared resources; disaster recovery person should have exactly one user- application (for instance IIS, all relevant (DR) wants things separated; HA eats name/password. Certain Web services content, ASPs, configuration files, etc.) bandwidth; DR wants distance – band- would tie into it. They also wanted this from a “baseline” server into a central width is a problem. mechanism to manage email addresses, repository. Then it can be pushed onto so the [email protected] alias could be They wanted to copy an HA system – new servers. directed to any of numerous internal migrate process, data, network identity, mail systems. The master server contains the reposi- “heartbeat” – but tried to minimize the tory and certain engines, remote agents bandwidth required. A relatively low- Existing administrative structures are on baseline, and “managed” servers. Any bandwidth pipe would also minimize not always along department or division server can be baseline and/or managed. the impact on the primary site. A private lines. Some groups go their own way.
February 2003 ;login: LISA XVI 71 But DNS is centralized; no delegation, gram. This appends data to a remote log The language they required needed to be ever. and log file status changes (such as cre- high level, be declarative, avoid duplica- ation, deletion, and truncation) to a sep- tion, support powerful validation and Currently, of 2000 employees total there arate file. distributed administration, and be are about 520 who use Exchange exclu- domain neutral. sively and 150 who are “casual” Stem configuration files contain the def- Exchange users. Windows 2000 authen- initions of cells, hubs, and portals Pan stores information in a tree struc- tication spans 400 public workstations through which hubs communicate with ture and supports template manipula- and a number of administrative Web each other. Stem is implemented in Perl tion and strong validation of data. It is applications; a ticket system; and so on. using an entirely peer-to-peer architec- licensed under the “European Data Grid ture, supports modules (that adminis- License,”an open license. It was The speaker presented a graph of the trators can write to create more complex designed to be portable but has not yet distribution of mail systems by division: cells), and allows encryption using SSL. been ported beyond its original platform Exchange exclusively, other mail – Linux on i386. Finally, the project is in (department), central system (POP, PAN: A HIGH-LEVEL CONFIGURATION its early stages; Pan is not yet being used [email protected]). Administrative depart- LANGUAGE in production. ments are mostly on Exchange; aca- Lionel Cons and Piotr Poznanski, CERN, demic departments mostly not, yet. European Organization for Nuclear WHY ORDER MATTERS: TURING He then showed a diagram explaining Research EQUIVALENCE IN AUTOMATED SYSTEM how systems are linked during a pass- Lionel Cons started his presentation by ADMINISTRATION word change. The user interacts via introducing the Large Hadron Collider Steve Traugott, TerraLuna; Lance HTTPS; that Web server encrypts the (LHC), what will become the world’s Brown, National Institute of Environmental Health Sciences password in a public key and stores it in largest particle accelerator, being built by a change queue in a database. The the European Organization for Nuclear Steve Traugott presented what he calls a encrypted password is shortly pulled Research. This facility will produce “theory paper.”He did not go into any from the queue, decrypted on the pass- enormous amounts of data. After on-site formalism in the presentation but word-change server using the private filtering, 10 petabytes will need to be instead focused more on the paper’s key, and distributed to the several stored to tape per year. The project will conclusions. authentication systems, including NT require 2 petabytes of disk storage, and Traugott argues that in many produc- domain servers. It does not happen over 100,000 processors. Pan is designed tion systems, there is a tendency for sys- instantly, but password propagation as part of an approach to solving the tem administrators to make changes by times were charted. Most were under 90 incredible system administration hand instead of using automation tools. seconds. requirements of a cluster-computing He calls the resulting system state “diver- project of that size. gent,”meaning that the difference PRACTICAL THEORY After this introduction, Lionel presented between the baseline machine state and Summarized by Kuzman Ganchev some principles of the system adminis- the current state is greater than expected, making rebuilds complicated, STEM: THE SYSTEM ADMINISTRATION tration project, such as automation, ENABLER abstraction, and the use of configuration or requiring backups for the entire oper- Uri Guttman, Stem Systems policies. The overall structure of the sys- ating system. A different situation that he calls “convergent” involves an auto- Uri presented Stem, a framework for tem would be a loop containing four mated tool synchronizing changes creating tools to automate system components: the cluster, a monitoring between hosts (hence making them administration. The Stem building block database, an “operator,”and a configura- closer to each other). He claims that this is called a cell. These are written in a tion database. The monitoring database is an ongoing procedure, since the mul- custom-made declarative configuration collects information about the cluster, tiple hosts are never quite identical. language, and are executed by a runtime which is then examined by the “opera- Finally, a “congruent” system is one daemon called a hub. Uri presented a tor” – probably a combination of auto- where all the hosts start out identical few example Stem programs. Of course, mated tasks and human administrators. and all changes are performed on them the first one was the obligatory “Hello This then modifies Pan source code, using a deterministic automated and world,”which in this case conducts a which is compiled into XML and stored repeatable process. conversation by replying with a greeting. in the configuration database, from He then went on to demonstrate more where the clients retrieve it. Traugott concludes that maintaining a complex but trivially written examples, congruent system is the least-cost including a remote log-monitoring pro-
72 Vol. 28, No. 1 ;login: method to guarantee that a host can PROCESS MONITOR: DETECTING EVENTS become involved in this, because they always be restored to its working state, THAT DIDN’T HAPPEN have the knowledge underlying it. The EPORTS especially if multiple identical hosts Jon Finke, Rensselaer Polytechnic speaker outlined the structure of the sys- R need to be kept (for example a Web- Institute tem, composed of SLA parameters, met- server farm). In particular in the long But what happens when a service doesn’t rics, and functions. Some are resource
run it pays to reinstall systems from run, and therefore nothing is logged? metrics, others composite metrics; for ONFERENCE scratch to bring them to an identical instance, a function might define the C Jon Finke addresses this problem with a state rather than to work with the dis- tool called Simon. With Simon, services l peak value of a metric over a given time parate systems. Traugott recommends a og when they last run into a database, period. Various services (measurement, tool he helped write called isconf to and when a service has not checked in evaluation) might be delegated to third deterministically automate changes to wit hin its configured window, a notifi- parties. The specification is flexible, different hosts. cation is sent to the administrators. For using a formal language. The software exa mple, if a service should run every package, WSTK 3.2 with SLA-compli- LOGGING AND MONITORING 24 hours and the last time it reported in ance monitor, can be downloaded. Summarized by James O’Kane wa s more than 25 hours ago, notifica- tion is sent. HOTSWAP – TRANSPARENT SERVER A NEW ARCHITECTURE FOR MANAGING FAIL-OVER FOR LINUX LOG DATA SERVICE AND NETWORK UPGRADES Noel Burton-Krahn, HotSwap Network Adam Sah, Addamark Technologies Solutions Summarized by Jim Hickstein When you have as much log data as Several techniques exist for adding fail- Yahoo! does, you need new methods to DEFINING AND MONITORING SERVICE LEVEL over capability to certain parts of a com- store and query it. That’s why Adam Sah AGREEMENTS FOR DYNAMIC E-BUSINESS puter system, with certain limitations. presented a Log Management System Alexander Keller and Heiko Ludwig, But most of them don’t address the (LMS) called Addamark. Some of the IBM T.J. Watson Research Center problem of a failing server which has a goals of this LMS were to handle 10’s Alexander Keller outlined a software sys- live application state and, especially, and sometimes 100’s of GB of data per tem that manages service level agree- open, long-lived TCP connections. day, parse and query arbitrary log for- ments, defined in such detail that a mats, be highly available and be able to computer can automatically evaluate The speaker presented a solution for keep the original files available in com- needs against offered services, and actual transparent fail-over of Linux servers, pressed format. Addamark achieves this performance against guarantees, all in which preserves internal state and con- by using a cluster of machines, and an aid of permitting e-business suppliers nections. It does this by running entire extensible SQL-like query l anguage. and consumers to find each other virtual servers on separate hosts, sharing dynamically. Dynamic e-business is cre- a virtual IP address, synchronized in MIELOG: INTERACTIVE VISUAL LOG ated and dissolved on demand. For near real time over a local network. A BROWSER FOR INSPECTING LOG instance, a Web site’s inventory, cart, and diagram showed a typical high-availabil- INFORMATION payment services might be distributed ity Web application, with network load Tetsuji Takada and Hideki Koike, Uni- among several providers. With a balancers distributing HTTP requests to versity of Electro-Communications dynamic system, the Web server could Web servers, and these talking to a back- When you are looking through logfile select, for example, payment providers end database. The HTTP connections after logfile, having a tool like MieLog, c on demand, based on a cost bid. are quickly over, but the database con- an be helpful. MieLog, presented by Tet- nections tend to be long-lived, and the suji Takada and Hideki Koike, gives an What do SLAs have to do with the daily database server itself becomes a single in teractive visual tool that can highlight chores of the sysadmin? In fact the point of failure. Commercial database key data. An administrator can see keyw sysadmin is constantly making such solutions exist to make this part fault- ords, periods of high log activity, or high evaluations: What is the cost to guaran- tolerant, but they tend to be expensive, word frequencies. Everything is color- tee a response time of less than one sec- and even the front ends will occasionally coded so you can see at a glance if there ond? How much should we bill a show a failure to a user, when, for exam- is a problem. customer for throughput of 1000 trans- ple, a Web browser times out. A fail-over actions per second? How much revenue system should never lose data; the clients is lost per hour of downtime? Can you should never be aware of a failure; no accommodate another customer and connections should be broken; the cost more workload? How would this impact SLAs with other customers? SAs will
February 2003 ;login: LISA XVI 73 should be low; and it should avoid forc- IP, and filtering it out entirely will break Using Non-Licensed Radios.”This ses- ing a rewrite of existing server processes. an IP network. Some ICMP types are sion presented a top-down view of wire- just more important than others. The less networking on a scale larger than Naturally, there are trade-offs. Cheap MTU (maximum transmission unit) is your average single-access-point LAN. backups mean long recovery times, the longest IP packet that will cross a Tim began with an overview of topolo- whereas full replication and quick recov- given network link. For best bulk-trans- gies, applications, and pros and cons of ery is expensive. The goal of this system fer performance, two IP hosts should 802.11. is to replicate a server on another box, send each other packets that are as large without a rewrite. It replicates the net- After laying out the basics, he moved as possible for the end-to-end network, work, TCP, and internal program state, into a more practical arena. Discussing but no larger. IP can fragment packets, if even memory, by knowing and duplicat- the design of networks, he mentioned they exceed the local MTU at any point ing all external stimuli coming in both “site surveying” and “engineering along the path. But the sender can set a through trappable system calls. This the link.”“Engineering the link” covered bit, called “don’t fragment” (DF), to say assumes the server processes are deter- signal loss/gain and attenuation – topics that a packet needing fragmentation ministic, which is often true, though that segued nicely into his comments on should instead be dropped, and an error OpenSSL had trouble until an uninitial- hardware. Tim presented various exam- returned to the sender. This error is ized memory bug was fixed. Timing- ples, including pictures, of classes of ICMP type 3 (unreachable) code 4 (frag- related code and direct hardware access antenna and access points. One of the mentation needed and DF set). may also break this assumption. Perfor- more fascinating access points was the mance may be an issue, of course, and Path MTU Discovery works by setting home-brew model, for which he, unfor- the network traffic between master and DF on all the packets in a connection; if tunately, didn’t provide instructions. slave may be large. But tests so far show the ICMP error comes back, the MTU is Pozar continued his speech with several a reasonably good result. The author’s reduced and a shorter packet sent. If no brief remarks on security, including the Master’s thesis was a demonstration of error comes back, the sender assumes forthcoming 802.11i standard. He con- the system serving streaming video. On the path MTU was large enough to cluded with a round-up of what we can HTTPS downloads, there was about 9% accommodate packets of this size, and it look forward to in the 802.11 family and degradation compared to a single server. proceeds. If a firewall blocks all ICMP a list of books and Web sites of particu- packets returning to the sending host, lar interest to the aspiring large-scale OVER-ZEALOUS SECURITY ADMINISTRATORS such a connection will not work: The wireless guru. ARE BREAKING THE INTERNET sender will time out and re-send the Richard van den Berg, Trust Factory; same, too large packet, and eventually SECURITY TRACK Phil Dibowitz, University of Southern give up. For a Web site, the user sees the California connection established, but nothing ever INTERNET SECURITY: BEYOND FIREWALLS, Path MTU Discovery (PMTUD) is used comes out. The users most affected are PASSWORDS, AND CRYPTO by many TCP implementations, usually those with a slightly constricted MTU, Peter H. Salus, Matrix NetSystems to good effect. But a growing number of typically because their Internet connec- Summarized by David Berg sites on the Internet have overly restric- tion requires a tunneling method such Salus’ presentation clued in the audience tive firewall rules that block certain criti- as GRE, PPTP, or PPPoE. Many con- on the myriad threats that the Internet cal ICMP packets, resulting in whole sumer-broadband users are in this faces and that lie beyond the control of classes of users who simply cannot see group. Their number is growing quickly. any local administrator. Peter presented these sites. They create self-inflicted The authors have started the MSS (max- the information using the analogy of a PMTUD “black holes.”More than a few imum segment size) Initiative, to try to medieval fortress under siege and a are security-related sites run by people educate those responsible for creating wealth of graphs depicting reachablility who ought to know better. The authors PMTUD black holes and to offer help in and packet loss on the entire Internet. are calling for better education on this fixing them. They also list their successes The discussion started with several slides issue and running a Web service that and failures. on the history of worldwide Internet users can check to see if a given site is in growth and the general state of the a known black hole. Certain ICMP NETWORKING TRACK Internet at present. packets have been an avenue for some LARGE-SCALE 802.11 attacks, so security administrators tend Emphasizing the siege theme, Peter pro- to decide that all ICMP packets are dan- Tim Pozar, Late Night Software ceeded to demonstrate the effect of gerous and none strictly necessary. They Summarized by David Berg some of the recent viruses (April Fool’s are wrong about that: ICMP is not an Tim actually titled his presentation Virus) and DDoS attacks on the overall optional extra. It is an essential part of “Long Distance Wireless Networking
74 Vol. 28, No. 1 ;login: flow of traffic across the matrix. He con- and related technologies. From user apa- cally, they attempt to compromise the tinued with other, perhaps less obvious, thy to developer incompetence, from security of a network and the hosts EPORTS threats to IP traffic, including the sever- politically influenced decisions to the within the network. Contrary to prevail- R ing of one of the oceanic fiber lines con- intractable problem of usability, cryp- ing opinion, the NSA only offers this necting China to the world, the 9/11 tography is experiencing a number of service (popularly referred to as “Red
terror attack, and the bankruptcy of problems as it tries to be accepted into Teaming” systems) to US government ONFERENCE WorldCom. He finished the slides with everyday use. As an important point to networks and the networks of govern- C the October 3, 2002, DDoS attack in back up his arguments, Len mentioned ment contractors. Further, the NSA will which the 13 root DNS servers were various fields in which cryptography has probe only with the explicit request of attacked. improved: where the user interface is the organization. In some cases, parts of simple, where there is a real need, and, the network that were considered critical The session ended with a discussion last but not least, where it’s actually were off-limits to the Red Team. For with the audience on the possible solu- used. example, the air traffic communications tions to these types of disruptions. Peter at an air base were not probed. A num- suggested that DDoS attacks might one But cryptography is suffering from the ber of people were skeptical, believing day be prevented with an IP “early warn- problem of weak links in a chain. Unless that this prepared the organization, in ing system.”Until that day, as Peter’s everybody uses it, it is not going to be effect, for a not very rigorous NSA answer to one participant’s query high- useful on a broad scale. A vast number “attack.”Nagle emphasized that they lighted, the solution is active monitor- of people don’t use cryptography were working together with the organi- ing. because it’s not standardized, not readily zation they probed, not against them. available, or simply too confusing. Len The team never tried to exploit social THE PROMISE OF PRIVACY questions whether it would help if the engineering to compromise systems. Len Sassaman, Consultant entire theory around encryption could Summarized by Martin Krafft be reduced to processes similar to seal- The NSA Red Team grew out of the 1987 Len Sassaman has been involved with ing a letter and sending it off. He points Computer Security Act that divided the PGP from the early days, which puts to various attempts at making crypto responsibility between NIST and the him in a role to analyze the position of easier, including PGP and TLS, as well as NSA. “Eligible Reviewer” was the code PGP and its relatives today. To sum up more high-level services like Hushmail, name for the summer 1987 DoD exer- his talk, everyone is screaming for pri- Zendit, and Lokmail. Most of these try cise to improve the war-readiness of vacy, and yet nobody uses the tools to reduce the user interface to the bare government computer systems. The available. Topics ranged from basic minimum, with TLS being “the best” exercise evaluated vulnerabilities of the crypto to why PGP and similar products because it is opportunistic and invisible. systems and scripted out what might have happened in the event of a mali- are failing. In conclusion, Len wants to see the tech- cious compromise. For instance, sending nology simplified for the user. He wants Privacy comes in various forms: finan- troops to the wrong location, disrupting friendly user interfaces, better integra- cial privacy, communication privacy, supply chains, and so on. privacy of stored data. The need for pri- tion, no room for individual error, and vacy in all these areas is high. Modern everything to be open-hooded. He wants Despite the prior warnings, the Red technology poses new risks in the form cryptography as a standard, with the Team invariably found security holes. In of credit card fraud, ID theft, and gen- proper usage being the only usage. You order to prove compromise, the team eral trust in the law to protect oneself. are not alone, Len. Who’s going to do generally left a file or some other evi- One of the answers to the general prob- something about it? dence of the security hole. Every key- lem of protecting privacy is cryptogra- stroke was logged to aid forensics and MY YEARS WITH THE NSA RED TEAM phy, which has seen great successes. PGP reproducibility. Often, the team was (“Pretty Good Privacy”) was released in Tim Nagle, TRW Systems required to prove what they did and did 1991, and technologies like SSL/TLS, S- Summarized by Robert Beverly not do. MIME, and anonymizers are also still in Nagle spoke to a capacity crowd, under- Much to the chagrin of at least a few widespread use. Consumers understand scoring the interest people have in one members of the audience, the talk did the threats, and the technologies are of the government’s most secret organi- not discuss any technical specifics of there, but privacy aspects of the current zations. The NSA Red Team is a group of how the Red Team compromised net- Internet are frightening. specialized individuals whose charter is works. However, Nagle provided inter- to protect information security, includ- The problems that Len isolates touch esting insight into the policies and ing voice, data, and encryption. Typi- on almost every aspect of cryptography procedures the NSA follows.
February 2003 ;login: LISA XVI 75 THE INTRUSION DETECTION TIMELINE cific requirements. Identify all the 5. Respond: Well into the attack or Paul Proctor, Network Flight Recorder source of risks and their costs in terms shortly after an attack, forensics, and Summarized by Abiodun A. Alao of potential damage to systems, loss of correlation will help determine what opportunities to do business, etc. Also has happened or what is currently We are slammed on all sides – viruses, estimate the value of each resource happening. Response must be timely rogue insiders, employee error, software and the implications of the breach of and appropriate; that is more than bugs, corporate spies, Web defacements, any of them for the organization. The enough to solve the problem and script kiddies, password crackers, net- most critical and vital resources deter further attacks. work vulnerability, worms, Trojans – the should get the best protection. Some list seems endless. The economic impact Check constantly the integrity of all files questions to consider include: of malicious codes has grown exponen- and fix problems as soon as they are tially to over $13 billion a year. What threats are most relevant to detected to minimize the cost of such your business? attacks. Review logs to reveal patterns of The number of attacks in the first three How critical is the data? likely attacks. Gather evidence and apply quarters of 2001 rose by over 60% com- Where does it reside? What is its trending and long-term analysis to pared with the entire year 2000, repre- value? determine further activity. This makes it senting a loss of almost $380 million by How do you define an attack? possible for firms to anticipate attacks. corporations, government agencies, What are the technology value Proactive firms are able to beat the financial institutions, medical institu- propositions? attackers. Finally, it is important to tions and universities! And it’s going to report and log all attacks and attempted get much much worse. 2. Anticipate: It is important to antici- attacks to ensure that the organization pate potential problems by creating The paper focuses on knowledge for has in place adequate data to plan with effective policies in the areas of secu- selecting and employing information and to use for prevention. rity, auditing, configuration, detec- security technologies that are appropri- tion, access, boundary, and Various technologies were examined, ate, meet organizational needs, are able application design. including system call trapping technol- to contain known risks and stated ogy (Intercept, OKENA, Trojan Trap), requirements, and pass a cost-benefit 3. Protect: Protect computers to reduce honeypots/decoy technologies, network analysis. the threat of compromise from the IDS, HIDS-Log analysis, and file inside or outside. The strength of a Most intrusions are the result of known integrity checkers. network or system is determined by vulnerabilities or configuration errors its weakest link. Therefore, ensuring Security is a process, not a destination; where countermeasures are available; adequate protection of all systems on use the right technology for the right 99% of intrusions could have been pre- the network is essential. The following problem. vented with patches, updated servers, specific steps should be taken: assess etc. A direct reaction to vulnerability Slides and other security resources are computers for vulnerabilities; install would be to close the window to expo- available at http://www.practicalsecurity.com. latest patches regularly; use best sures, but it is important to identify all industry practice; keep anti-virus soft- such windows as they emerge. Making GURU SESSIONS ware updated; disable Java, JavaScript everything secured stops business and in browsers; turn off macros in appli- NAS: NETWORK ATTACHED STORAGE drives administrative costs through the cations; and back up servers and W. Curtis Preston, The Storage Group roof. This returns us to the issues of the workstations. cost-benefit analysis of available solu- Summarized by Kuzman Ganchev tions. For instance some threats may not 4. Detect: Prevent attacks that are When I came in, the discussion had materialize or their effects may be muf- known, detect attackers probing for already started, and NetAppliance filers fled and not as significant as anticipated. weakness, and direct attackers into were being discussed. Essentially, the Investing huge sums to prevent such honeypots. This will make hacking problem with these is that you have to attacks may not be economical or effi- more difficult and less rewarding, and keep the NetApp filer around as long as cient. may reduce the incidence of attacks. you want your data. Curtis gave a few Detect network probes as attackers examples (without names) of companies How then can you defend your organi- search for vulnerability to exploit net- who still have to keep around archaic zation? There are six major steps: work scans, port scans, and systematic technology, because it’s the only thing 1. Analyze risk and classify resources: activities. This is usually accomplished that will read their old backups, which You have to set your enterprise-spe- with IDS technologies for log analysis. they still use from time to time.
76 Vol. 28, No. 1 ;login: The discussion then moved to non-tape know about Perl. Doesn’t get any better “Tired of chomping and putting \n at storage. Curtis mentioned a service at e- than that does it? Ah, but it did. A few the end of every print statement? Try EPORTS vault.com, for backing up a small unannounced guest gurus showed up: perl -l.” R amount of data over the Internet; this is Matthew Barr and Larry Wall. Where How do you top all that? Perhaps a Perl probably best for personal data – config- else but at LISA? script to generate unique pattern sets for
uration files and other compressible ONFERENCE
We were treated to a guided tour a quilt, then having your wife sew it. C information. For a small office, disk- through the coding of Larry’s own home And convincing her it’s a gift! Hmmm, based backups can be a better solution automation and monitoring setup, don’t try that at home. I guess that’s why than traditional tape. Curtis cited a accompanied by many fascinating side these guys are the gurus. backup failure of up to 40% in small trips into his life and personal interests: office environments, because of failure X10 problems, techniques, war stories, EMAIL/MTAS to insert the next day’s tape after a tape human-readable code. Are we asleep or Eric Allman, Sendmail is ejected. Though taking media off-site awake when the thing goes bump in the for disaster recovery is not possible with Summarized by Martin Krafft night? It does make a difference, at least a disk-based solution, at least the data is This year’s guru session on MTAs and in Larry’s home. I pity the poor mice. He being backed up. email was well attended, led by Eric All- didn’t touch on mousetraps; perhaps man, author of the infamous Sendmail Alacritus Software, a Livermore-based that’s a question for next year. and current CTO of Sendmail, Inc. It company writes storage software that Dan Klein could not be outdone, leading wasn’t a big surprise that the first ques- enables a disk-based system to act as one to displays of several such systems. tions were about spam. Eric talked about or more virtual tape libraries. They do Water-flow monitoring, and why you the simple anti-spam methods in Send- not actually provide out-of-the-box should care. A graphical display of fur- mail (which are still more advanced than solutions directly but have partnerships nace operation related to temperature most other MTAs), like per-host connec- with third-party vendors to do so. Curtis inside and outside the home. At the tion throttling, tweaking rule sets, mil- suggested backing up to a disk-based cabin on the lake, the water temperature ters (mail filters) and RBL, and he device and then periodically duplicating at the surface and underwater. Ways not referenced Spamassassin. The next ques- those to actual tape to be taken off-site to waterproof a temperature sensor, tion concerned remaking SMTP, clean- or stored in archives. This is better than complete with graphic descriptions of ing up its fundamental flaws as part of backing up the device to tape, since a failure modes, and at least one method the anti-spam war. Eric agreed, but he restore from tape only requires one that works. stressed the extensibility of SMTP and operation, as opposed to two in the case argued against a new protocol on a new of backing up the backup device. Mark-Jason Dominus happened to men- port – port 25 is the mail standard, he tion his Perl quiz of the week. Check it Discussion then moved to the Quantum argued, and changing standards is near out at http://perl.plover.com/qotw/.A DX30, which are disk arrays used as impossible: If you splinter the Net by new quiz every week followed later by backups for quick restore. According to trying to introduce a new standard, you sample solutions. No better way to learn Curtis, these are not quite as small as not only create chaos for email for some (except to get paid for it). This week’s originally intended due to unresolved period, but you also make it possible for entry: Find all the anagrams in a list of cooling issues. a company that would prefer that the words. And, they were off! Amazing how Net run on their proprietary standards PERL/SCRIPTING GURUS much can be done with a single line of to get a foothold. He also addressed the Daniel V. Klein, LoneWolf Systems; Perl. Oh, forgot to mention they were to single fax machine problem – either Mark-Jason Dominus, Plover Systems be sorted alphabetically . . . no problem. everyone employs the “new SMTP” or it Oh, and if there are more then two Summarized by Abiodun A. Alao is as useless as a single fax machine. The words . . . and this isn’t even the “expert” next question on spam dealt with a Larry Wall once said, “Most of the pro- quiz. gramming out there is not done by Perl buffer/moderation queue in Sendmail, experts...they learn by experience to do Some tidbits we grabbed out of the air: which would allow a postmaster to better over time, and eventually they intervene in case of a spam flood. “You can deal with unreliability in Finally, a couple of technical questions become experts.”We took a step in that automation ...a little bit.” direction with Perl scripting gurus Dan about MTAs and the RFCs yielded closer “How does one become a Perl guru? inspection of RFCs 2821 and 2142 about Klein and Mark-Jason Dominus. Just Volunteer a lot, try hard things, fail a type these guys’ names into Google and the type of email addresses one must lot, and learn.”
February 2003 ;login: LISA XVI 77
worked on with the Berkeley folks). 8.13 PERFORMANCE TUNING mance surfaced. First, one should deter- still has some problems with the latest Jeff Allen, Tellme Networks mine whether the problem is in fact due Linux implementation of flock(), which to an I/O-bound device. The iostat com- Led by Jeff Allen, author of the Cricket doesn’t behave as expected. Eric mand is ideal to observe disk and con- SNMP monitoring tool, the perfor- announced the “Bat Book” (O’Reilly’s troller performance. If the disks are in mance tuning session was loosely organ- Sendmail book) on version 8.12 for the fact the bottleneck, data should be ized and consisted of specific questions end of the year and said he will release stripped across as many disks as neces- as well as general problem-solving 8.13 before 2003 only over his dead body sary (often five or more). In this man- methodologies. – he wants the book to be current for at ner, a single datafile is divided so that a least a while. Allen emphasized that one should piece of the file exists on each disk. always understand data and statistics in Because disk read performance is the Performance and scaling comparisons context. As an example, WebTV engi- limiting factor, each disk can now read between various MTAs came up next. neers could not immediately offer an their portion of the file in parallel and Oracle’s new mail product (which uses explanation for a drastic dip in network fully utilize the controller bandwidth. Sendmail) is not their first attempt at usage for a particular day until they dis- Even though this will waste disk space, this market, but previous attempts were covered it coincided with the broadcast disks are relatively inexpensive today not commercial successes, which Eric of the Super Bowl. In general, one and this technique will yield much attributes to the inadequate speed of the should always form a scientific hypothe- higher performance. Oracle back end for a real-time mailer sis and test that hypothesis. When ana- application. Performance comparisons lyzing statistics, averages are generally of between the big UNIX mailers Send- little use since the most interesting mail, postfix, and qmail cannot really be events (and those that cause issues) are instituted. Eric believes that qmail does outliers. Many distributions have strong way too much sync-I/O. Following up modalities or heavy tails that negate the conclusions pure averages may find.
78 Vol. 28, No. 1 ;login: WORKSHOP SERIES language (more below); centralized vs. There was some discussion of SYSTEM CONFIGURATION WORKSHOP distributed control; and synchronous vs. whether or not a good model needs EPORTS Summarized by Will Partain and asynchronous operation. practical backing by a CPAN-like R Infrastructure Framework Library Paul Anderson Though the rest of the workshop talks (suggested by Mark Roth). The system configuration workshop, described particular system configura- with 22 participants herded by Paul ONFERENCE tion tools, the purpose of this workshop A surprising issue that emerged in the C
Anderson (University of Edinburgh), was to study tool-independent configu- discussions was usability: System config- built upon the cfengine workshop at ration principles. uration tools often fail to make headway LISA 2001 (http://www.cfengine.org/ because they are too hard to use. Possi- Much discussion arose from the notion Workshop/) and the Large-Scale System ble reasons for this: Configuration workshop in Scotland (raised by Luke Kanies) that existing (http://homepages.inf.ed.ac.uk/dcspaul/ tools comprise an unholy mix of model, Configuration tools are complex, publications/wshop/). language (to express an instance of the with a steep learning curve, espe- model), and implementation (of the cially for small sites. Better user Anderson led off with an introduction language). We will do better when these interfaces are needed (for both GUI to system configuration: Given a large concerns can be understood independ- tools and languages). computing infrastructure (dozens to ently. Points raised in this session A tool embeds its author’s notion of thousands of hosts), how can we included: sysadmin policy, which proves inap- describe its desired state in a humanly propriate at any other site. Ideally, the language should express tractable form? How can tools (better) A configuration tool is most useful what is true in a model of a config- use such a description to control the when it has complete control of the uration (a declarative approach), infrastructure? system. This is a big culture change not how to make it true (a proce- for many sysadmins. System configuration tasks span an dural approach). Existing tools are a diverse, frag- infrastructure’s whole life, including A model should be able to represent mentary bunch, each one covering pre-installation (e.g., BIOS configura- inter-machine relationships and be just a part of the problem; learning tion), operating system and software independent of implementation enough tools to do the whole job is install, configuration of that software, details. a daunting task. managing changes to the infrastructure The model needs to represent over time, and taking in feedback infor- dependencies between components, An idea that gained immediate accept- mation about the infrastructure and including runtime temporal ance by the group was that there must recovering from faults. dependencies. (“Service X must be be a strong connection between configu- Problems that arise in the design of sys- started before client Y tries to use ration, testing, and monitoring. Specific tem configuration tools include han- it.”) Temporal constraints are even points raised: more fun. (“Kernel upgrades can dling scale, diversity, and/or change; Monitoring and feedback of the only be deployed across lab supporting modular management so actual state are crucial. “We have to machines on Saturday nights, that different people can control indi- embrace failure.”(Andrew Hume) except in exam week.”) vidual aspects of an infrastructure sepa- What do we mean by “testing” a The “truths” expressed in a model rately; providing an explicit represen- configuration? How can we test and the “truth-checking” of a moni- tation of components (separate from the configurations before deploying toring system need to be closely components themselves); providing them? higher-level views of an infrastructure coupled (more below). (e.g., viewing a cluster as a single entity); The model and language must sup- Our sketch of this workshop should making possible the desired level of con- port devolved management. If more make clear that system configuration is sistency across systems; and security (of than one person is specifying con- an intellectual and practical challenge. course). figuration details, how do we know The conversation will continue at LISA the total infrastructure still “makes 2003 – interested configurationists take Solutions to these problems have to sense”? note! Until then, details about a configu- choose between static vs. dynamic con- The conversation continued about ration mailing list are at http:// figuration (e.g., JumpStart vs. cfengine); the importance (or not) of “order- homepages.inf.ed.ac.uk/group/lssconf/ getting to a “good” state by cloning vs. ing” in a model; see the Traugott/ config2002/, along with all of the materi- by scripting; declarative vs. procedural Brown LISA paper for one side of als (e.g., slides) from this workshop. the story.
February 2003 ;login: LISA XVI 79 EDUCATION AND BOOK OF KNOWLEDGE dardization is needed since people AFS WORKSHOP COMBINED WORKSHOP change jobs frequently (every 1.5 to 3 Coordinators: Esther Filderman, Coordinators: Geoff Halprin, Rob Kol- years), and it takes six months to adapt Pittsburgh Supercomputing Center; stad, SAGE; John Sechrest to a new job. We must understand the Derrick Brashear, Carnegie Mellon Summarized by Rob Kolstad nature of a problem space by breaking University This year, the Education and Book of the problem into its components and Summarized by Garry Zacheiss Knowledge (aka sysadmin taxonomy) understanding them. The workshop began with status reports groups merged their workshops in order Geoff covered several related programs from representatives of both Arla and to learn each other’s working style and and listed unique features of sysadmin. OpenAFS. The current released version interests. About 18 people attended, He also discussed professional develop- of Arla is 0.35.10, which supports all including organizers and individual con- ment and “key areas of responsibility.” *BSD UNIX variants, including MacOS tributors from both groups. The Educa- and Linux. An 0.36 release is expected to tion group included several people who The BoK seeks to define a sysadmin and branch before the end of the year. This were trying to run 10- to 18-week development maturity model. Several release will include Themis, their pack- courses and a fellow from NYU who is examples were given. The BoK is a refer- age utility replacement, which includes implementing a five-semester Master’s ence framework that supports best prac- features and extensions not found in the degree course in system administration tices, enables effective training, and feeds traditional AFS package utility. Themis (!). certification, education, and job descrip- should be a drop-in replacement for tions by listing the core skills, knowl- package . Improvements in Arla 0.36 Geoff Halprin presented his brilliant edge, and disciplines of the profession. include support for incremental open BoK history and motivation. We’re all and support for UUID-based callbacks solving similar problems, and we need to Rob Kolstad echoed many of these same sentiments and discussed the over 2000 (via the WhoAreYou RPC).Additionally, work and develop from the same foun- the afs3-callback port used by Arla will dation. Let’s address areas of personal elements now present in the current change from 7111/udp to 7001/udp, and ygrowth, organizational maturity, and a BoK matrix (of tasks/knowledge and the various factors that affect those tasks). XFS will be renamed to NNPFS. Win- framework upon which we can capture dows support will also be present in this “best practices.” Creating the document is the next step, given this list. release, along with a GUI ACL manager Sysadmin is about “intricacy,”the inter- for MacOS X that integrates with the play of components that come into play John Sechrest talked about the Educa- Finder. The MacOS X ACL manager will when dealing with complex environ- tion Committee’s work. “Last year, we also work with the OpenAFS MacOS X ments and a continuous stream of had a workshop, and I asked a lot of client. Future goals include implementa- microscopic changes. Thus, there is no demographic questions in an effort to tion of a cleaner and faster kernel/user- such thing as “a best practice.”There are learn how best to serve people and land interface, and the addition of IPv6 a number of best practices that we can enable sharing of teaching ideas.”He support for AFS. Work on integrating capture. showed his goals and discussed accredi- Kerberos 5 and GSSAPI into Rx contin- tation at his university. He gave a list of ues. System administrators ensure integrity about a dozen topics and sub-topics. of computing systems and assist users in OpenAFS recently celebrated its two- maximizing effectiveness of their com- Hours of lively discussion ensued, with year anniversary. Recent progress in puting environment. System administra- lots of time spent delving into topics like OpenAFS includes the addition of fake- tor roles include: troubleshooter, risk assessment and change manage- stat; with this feature enabled, the AFS walking encyclopedia/user manual, tool- ment. Teaching techniques and para- client will provide stat information for smith, researcher and student, tech digms were discussed, including the volume mountpoints not yet traversed writer, both strategist and tactician creation of virtual laboratories. A cur- without contacting remote file servers. (today and in the future), doctor, and riculum discussion group was formed This allows the use of graphical file counselor. for the purposes of creating a four-year managers to browse /afs without causing curriculum (from whence other curric- excessive hangs and timeouts. This Administrator tasks, challenges, and dif- ula will evolve). Attendees rated the day feature is present in OpenAFS 1.2.7; ficulty combine with availability and a general “thumbs up.” OpenAFS 1.2.8 will include a further hidden costs to present issues with many refinement to only present this behavior details. Professional development pro- for mountpoints to volumes in foreign ceeds along half-a-dozen paths. Stan- cells. Other recent features include ports
80 Vol. 28, No. 1 ;login: to MacOS X 10.2 and an experimental ADVANCED TOPICS WORKSHOP In order of presentation here are the port to FreeBSD, further Linux client Coordinators: Adam Moskowitz, submitters’ own descriptions. EPORTS tuning, and modifications to the file Consultant; Rob Kolstad, SAGE WHEN THE TROUBLE IS PEOPLE, R server to use Rx pings to determine if Summarized by Josh Simon (with help NOT TECHNOLOGY clients are reachable before allocating from Rob Kolstad) Chuck Pervo threads to them; this prevents asymmet- The Advanced Topics workshop was [email protected] ONFERENCE ric clients from consuming all available C once again ably hosted by Adam Sysadmins of the world unite! Are you file-server threads. Issues that OpenAFS Moskowitz. We first discussed what per- tired of being stepped on by others? is currently facing include recent RedHat centage of our time is spent on reactive Have you ever been in a situation where Linux kernels (which break the Ope- versus proactive tasks, which varied rela- there was a serious problem and you nAFS client by no longer exporting the tive to how close to the end user or cus- were out-shouted in the problem resolu- symbol sys_call_table), the minimal tomer our roles were. RedHat AFS client, and a forthcoming tion process by an unknowledgeable HP-UX 11 port. rxkad 2b, which will We next talked about the various barri- person? Or when the process was add Kerberos 5 support to Rx while still ers to fixing problems, including techni- directed by politics rather than solutions using fcrypt for encryption, will appear cal ones, economic problems, problems based on causality, data, or reason? If the in a future OpenAFS release, most likely of management not understanding, and answer to these or similar questions is OpenAFS 1.2.8. so on. Many of these issues are discussed YES, you are not alone! Alva Couch has in the forthcoming SAGE Short Topics encouraged me to do a paper on this Other discussions included: booklet on budgeting. topic, including case studies and a man- ual on formal problem resolution prac- Porting OpenAFS to HP-UX for the Our next discussion was on why we tices, which will include a Robert’s Itanium and to AIX 5.1 and later reinvent the wheel by recreating the Rules-style set of guidelines that should CERT’s transition from Transarc tools for the same task again and again. preempt such time-wasting, stressful AFS to OpenAFS/Kerberos 5 Reasons include ignorance of preexisting activity. Using AFS through a firewall tools, political factors influencing the Using AFS with Kerberos 5 and a decision (the “not invented here” syn- Kerberos migration kit TIVO & MACOS X drome), taste, where the tool falls in the Matthew Barr Performance benchmarks and issue of specific vs. general, and chang- [email protected] tuning ing needs. Common user errors – Backing up After being encouraged by some seem- After our discussions, we went around AFS cells ingly nameless party, I’ve been conned the room to list our favorite URLs that AFS on MacOS X 10.2 into doing this. So, you get to hear about might be unusual as information and IBM’s end-of-life announcement it. This WIP will focus on a MacOS X humor. We went through system admin- for their AFS implementation machine being the recipient of a copy of istration aphorisms — pithy sayings The workshop closed with a roundtable data from a Tivo. It includes informa- such as “Never send email in anger.” discussion on what AFS needs to do to tion on connecting a Tivo to a TCP/IP SAGE will be making a poster of these. gain more market share. Support for network, enabling external control of (Send your favorites to kolstad@sage. files larger than 2GB, byte-range file the Tivo via HTTP and Web browser, as org.) Finally, we talked about things we locking, better support for Windows well as how the heck to export data from learned in the past year and, as usual, clients, and more training opportunities a Tivo to a Mac/UNIX system. I am also made our annual predictions. and documentation were all cited as involved with a collaborator (who just being desirable for AFS to gain addi- happens to work at Apple :) on design- WORK-IN-PROGRESS REPORTS tional market share. ing a GUI system for all of this. Summarized by Peg Schafer The LISA ‘02 WIP session went very THE VMATRIX well. We had some interesting submis- Amr A. Awadallah sions! Amr Awadallah created a lot of [email protected] excitement with his vMatrix presenta- tion. However, the crowd gave the LISA The vMatrix is a network of virtual ‘02 WIP Whip to Jeremy Mates for his machine monitors allowing for fluid “Improving Productivity” (by reading server mobility between real machines. your daily cartoons) presentation. By building the servers inside of virtual
February 2003 ;login: LISA XVI 81 machines, we can easily move them to security and audit problems, to stan- ment remotely, collaborate with others, around. The applications that we are tar- dards conformance. We collect this data and mentor junior admins. (See Fine geting are dynamic content distribution, and save it for historical data collection and Romig, LISA IV Conference Pro- server switching, and warm standbys. (via CVS), as well as upload a significant ceedings, 97-100.) This is research work that I am doing portion to a database to do reporting Since then, many enhancements have with Prof. Mendel Rosenblum at Stan- across the company at various levels of been added. The current conserver.com ford. More info (papers, presentations) detail. We also combine this with our version (7.2.4) also includes basic SSL is at http://www.thevmatrix.com performance monitoring to identify the support so that, assuming you have a most over- and under-utilized systems. TAKING SYSNAV OPEN SOURCE network connection, you can securely Christian L Pearce SOFTWARE FOR OPTIMAL TIME TO PATCH interact with any of the equipment from home or wherever. The next version will [email protected] Adam Shostack have yet another slew of enhancements, sysnav.commnav.com [email protected] including complete SSL support and a SysNav started out as a closed source Following on research presented in the new config file format. In this WIP, I’ll project for managing servers via a portal refereed papers track, Adam has give you the scoop on the latest features infrastructure. It consists of storing con- founded a company to build decision and solicit you for additional cool ideas figuration information about machines support software for IT departments to for the code and a possible future paper. and what components they would like find the optimal time to install patches, managed. This information is held in maximizing their uptime and reliability. RCS.MGR LDAP and translated into cfengine files Adam is interested in talking to IT man- John Rowan Littell and configuration files by the middle agers who measure uptime and security. [email protected] layer. Then the back end takes these con- www.earlham.edu/~littejo/ figuration files and executes them via WHAT?? ANOTHER &%#!’ING BACKUP rcs.mgr is a basic, self-contained config- cfengine. This framework will install, PACKAGE? uration manager that wraps the RCS upgrade, and configure components James O’Kane process for textual configuration files automatically based on the information [email protected] and manages their installation, includ- stored in LDAP. SysNav is going through I’ll talk briefly about why I’m writing yet ing setting ownerships and permissions a transition. It is CommNav’s goal to another backup application and why and running any post-installation com- take the back end and the middle layer this one will be newer, better, different. mands necessary to activate the changes. and form an open source meta-project. So cool, that hopefully you’ll forget why The script has been in production for We, at CommNav, feel the community you thought digital watches were a 1.5 years. Future developments will will benefit from the project and other pretty neat idea. include better handling of unauthorized sub-projects that will be generated out changes and support for per-file editors, of taking SysNav open source. Collabo- RETURN OF THE SON OF THE BRIDE OF allowing the management of non-tex- ration has already begun internally and CONSERVER (AKA CONSERVER 8.0.0) tual files. will be released in 2003. Please see Bryan Stansell http://sysnav.commnav.com for more The conserver application was devel- INFINITE SCALABILITY DISTRIBUTION information. oped by Tom Fine in 1990 to allow mul- Doug Hughes tiple users to watch a serial console at [email protected] THE CONFIGURATION MONITORING AND the same time. Despite its indispensabil- REPORTING ENVIRONMENT I have a multicast distribution program ity, many sysadmins aren’t aware of it. Xev Gittler that has been “under development” for Conserver can log console output, [email protected] about two years now. It puts a sequence allows users to take write access of a number on each datagram and uses The Configuration Monitoring and console (one at a time), and has a variety selective retransmission from the Reporting Environment (CMRE) is of bells and whistles to accentuate that receiver to the sender to get missing designed to collect configuration data basic functionality. The idea is that con- sequence numbers. It also uses PGP sig- from all our systems and then correlate server will log all your serial traffic so natures on each whole “package” for and report on the information. This you can go back and review why some- authenticity and for integrity; this also allows us to understand exactly the state thing crashed, look at changes (if done allows building of a web of trust. The of our systems, from OS levels and hard- on the console), or tie the console logs “file” program is used to determine how ware, to software installed and patches, into a monitoring system. With multi- to process the received item on each user capabilities you can work on equip-
82 Vol. 28, No. 1 ;login: receiving host. Each “distrib item” is signed with PGP and multicasted to all EPORTS listening clients on a well-defined port. R Responses can be collated in many dif- ferent ways: syslog, mail, tcp socket, file,
etc. The software provides distribution ONFERENCE and an extensible framework upon C which to build. A distribution server can also be used as a generic request reposi- tory. A peer-to-peer network of senders and requestors can thus be built easily.
IMPROVING PRODUCTIVITY Jeremy Mates [email protected] http://www.sial.org/code/perl/modules/ Sial::Apache::ImageShow (1.2) The talk is available at http://www.sial. org/talks/productivity/ with pointers to the script. Peg’s Notes: Jeremy showed true WIP spirit by developing this presentation moments before he was to go on the stage! His HUGE contribution to pro- ductivity allows users to see all their favorite daily comics on ONE page!
ADMINISTERING SELF-SECURE DEVICES A. Chris Long [email protected] Suppose you had a host-based and a network-based IDS on every computer in your enterprise. How would you manage them? The “self-secure devices” we are developing are disk drives and NICs that include security measures, such as monitoring for changes to sys- tem files and virus traffic. I am in the early stages of designing the user inter- face for a system administrator to con- figure, monitor, and control self-secure devices.
February 2003 ;login: LISA XVI 83