DOCSLIB.ORG

Do Man-Pages Matter?

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Do Man-Pages Matter? Outline Do manual pages matter? Background Man pages: a counter-argument Michael Kerrisk Man pages matter for kernel developers Problems maintaining man pages linux.conf.au, Sydney, 17 January 2007 How to help www.kernel.org/pub/linux/docs/manpages [email protected] The man-pages project Contents of man-pages • Project started 1993 •As at man-pages-2.44: – ~800 man pages (== ~2000 printed pages) • Documents Linux kernel-userland API… – 2: syscalls • and (GNU) C library API – 3: library functions (glibc) • Sections 2, 3, 4, 5, and 7 of manual pages –4: devices Proportion of source lines (and • Target audience: userland programmers… – 5: file formats number of source files) • and kernel developers – 7: overviews, etc man2 (224) man3 (465) man4 (25) man5 (31) man7 (61) Background Man pages: a counter-argument “Documentation is fantasy: you have Man pages matter for kernel developers to read the source code to know Problems maintaining man pages the truth.” How to help 1 Time! Reading the source doesn’t cut it •The kernel is big: • Reading the source gives the “right” answer – 2.6.19 kernel source (*.[chS]) is 7.3M lines • but… too slow (and hard, especially for • and constantly changing: userland programmers) – Typical Linux 2.6.x diff –u patch > 1M lines • We just don’t have the time… We need summaries of the code • Understanding of code must be mediated by natural language summaries • Discussions – oral + email Man pages do matter! – Take place during development – but… not so useful later • Documentation – most useful form of summary for later Why man pages matter for kernel developers Background • Publicity Man pages: a counter-argument • Identifying bugs Man pages matter for kernel developers • Better testing (reducing # of released bugs) Problems maintaining man pages • Better interface design How to help • Better interface consistency 2 Identifying bugs Testing • Software is an implementation of an intention • Problem: too many bugs in released • bug == intention – implementation interfaces • Without documentation, how do we know • Why? Insufficient testing before release whether implementation matches intention? • And how can we test? Documentation and Testing Testing – example 1 • Documentation can help reduce bugs inotify • Evidence: the process can work in reverse… • File change notification API • Appeared in kernel 2.6.13 • 2.6.16-rc timeframe, I wrote inotify(7) •Testing: IN_ONESHOT had never worked • Bug reported; fixed for 2.6.16 Testing – example 2 Testing: conclusions splice() • Documentation goes hand in hand with • transfer data between file descriptors without testing going through user space • Documentation broadens range of testers • Appeared in kernel 2.6.17 • Testers can determine if implementation == • Simple test programs easily caused hangs intention • Bug reported; fixed for 2.6.18 • Good, early documentation Æ more & earlier testing Æ fewer released bugs 3 Interface design When interface design goes wrong • It’s hard to design a good programming dnotify (kernel 2.4; file change notification) interfaces • Many problems in interface design • Getting design wrong is painful… • Problems led to replacement by inotify – Using interface is difficult, and bug-prone • But… is the problem the developer(s)? – Difficult/impossible to change design • Or the process? Interface design: man pages help Interface consistency • Writing a man page (or other doc) can help • The problem: some new interfaces are with interface design inconsistent with existing similar interfaces • Writing documentation leads to self-review by • Man pages can be used as a reference when implementer(s) designing new interfaces • Documentation broadens audience who can understand and critique design Interface consistency: right Interface consistency: wrong (1) mbind(MPOL_MF_MOVE_ALL) • Various memory-related syscalls specify a • NUMA memory binding interface start address + a length • Requires privilege (CAP_xxx) • Some APIs (e.g., mprotect(start, length, ...)): • Initial (-rc) implementation used –Require start to be multiple of page size CAP_SYS_ADMIN – Round length up to next page boundary •Readingcapabilities(7) showed that existing • Some other APIs (e.g., mlock(start, length)): related APIs used CAP_SYS_NICE – Round start down to page size – Round length up to next page boundary • Final implementation used CAP_SYS_NICE – mlock(4000, 6000) affects bytes 0..12287 4 Interface consistency: wrong (2) remap_file_pages(start, length, ...): Background • Why settle just for inconsistent… Man pages: a counter-argument – Round start down to page boundary Man pages matter for kernel developers – Round length down to page boundary(!) Problems maintaining man pages • … when you can also have bizarre: How to help – What address range is affected by remap_file_pages(4000, 6000, ...) ? Problems maintaining man-pages • Much to do; too few people Background • Many man pages yet to be written Man pages: a counter-argument • Many existing man pages are stale Man pages are useful for kernel developers • Kernel developers have much valuable Problems maintaining man pages knowledge, but are largely absent How to help • How to know if an interface has changed? • How to know if a man page is broken? How to help Helping: anyone • Just about anyone can help •Read HOWTOHELP in man-pages tarball • Kernel developers would benefit by helping – List of missing pages • How companies could help – How to obtain list of FIXMEs – Tips on how to help in the most helpful way • Latest tarball at: http://www.kernel.org/pub/linux/docs/manpages 5 Helping: kernel developers Helping: kernel developers • Adding/changing an interface?… • System call man pages belong in man-pages, • Write/update the manual page! not separate tarballs • Can’t bear messing with groff? • Many virtues in a consolidated set of man – Submit plain text! pages: • Please provide test programs… – Formatting consistency – Single known address for man pages patches – Distributors know where to find manual page – Consistent interfaces… Helping: kernel developers A proposal for kernel developers “This [part of the] interface shouldn’t be documented, • Create and enforce a policy that requires because userland shouldn’t be using it [it’s only intended interface changes to be accompanied by for use in libraries].” documentation and test programs • Library developers are in same position as everyone else •“no documentation” doesn’t always mean “don’t use this” • Best approach: document interface with warning about usage Before saying no… Helping: companies/organisations • Consider that good documentation can help • Fund a man-pages maintainer prevent: – Write/update pages – Poorly designed/inconsistent interfaces –Vet patches – Bugs in new and changed interfaces – Test new interfaces • Look at long list of FIXMEs and missing pages – Track standards work (POSIX.1-200x/SUSv4 and beyond) • There are kernel coding standards; why not documentation (and testing) standards? – Write/choose a style guide – Maintain a website 6 [email protected] Michael Kerrisk Thanks! www.kernel.org/pub/linux/docs/manpages 7.
Load more

Recommended publications
  • Ivoyeur: Inotify
    COLUMNS iVoyeur inotify DAVE JOSEPHSEN Dave Josephsen is the he last time I changed jobs, the magnitude of the change didn’t really author of Building a sink in until the morning of my first day, when I took a different com- Monitoring Infrastructure bination of freeways to work. The difference was accentuated by the with Nagios (Prentice Hall T PTR, 2007) and is Senior fact that the new commute began the same as the old one, but on this morn- Systems Engineer at DBG, Inc., where he ing, at a particular interchange, I would zig where before I zagged. maintains a gaggle of geographically dispersed It was an unexpectedly emotional and profound metaphor for the change. My old place was server farms. He won LISA ‘04’s Best Paper off to the side, and down below, while my future was straight ahead, and literally under award for his co-authored work on spam construction. mitigation, and he donates his spare time to the SourceMage GNU Linux Project. The fact that it was under construction was poetic but not surprising. Most of the roads I [email protected] travel in the Dallas/Fort Worth area are under construction and have been for as long as anyone can remember. And I don’t mean a lane closed here or there. Our roads drift and wan- der like leaves in the water—here today and tomorrow over there. The exits and entrances, neither a part of this road or that, seem unable to anticipate the movements of their brethren, and are constantly forced to react.
    [Show full text]
  • Monitoring File Events
    MONITORING FILE EVENTS Some applications need to be able to monitor files or directories in order to deter- mine whether events have occurred for the monitored objects. For example, a graphical file manager needs to be able to determine when files are added or removed from the directory that is currently being displayed, or a daemon may want to monitor its configuration file in order to know if the file has been changed. Starting with kernel 2.6.13, Linux provides the inotify mechanism, which allows an application to monitor file events. This chapter describes the use of inotify. The inotify mechanism replaces an older mechanism, dnotify, which provided a subset of the functionality of inotify. We describe dnotify briefly at the end of this chapter, focusing on why inotify is better. The inotify and dnotify mechanisms are Linux-specific. (A few other systems provide similar mechanisms. For example, the BSDs provide the kqueue API.) A few libraries provide an API that is more abstract and portable than inotify and dnotify. The use of these libraries may be preferable for some applications. Some of these libraries employ inotify or dnotify, on systems where they are available. Two such libraries are FAM (File Alteration Monitor, http:// oss.sgi.com/projects/fam/) and Gamin (http://www.gnome.org/~veillard/gamin/). 19.1 Overview The key steps in the use of the inotify API are as follows: 1. The application uses inotify_init() to create an inotify instance. This system call returns a file descriptor that is used to refer to the inotify instance in later operations.
    [Show full text]
  • Kernel Korner
    Kernel Korner http://0-delivery.acm.org.innopac.lib.ryerson.ca/10.1145/1110000/11030... Kernel Korner Intro to inotify Robert Love Abstract Applications that watch thousands of files for changes, or that need to know when a storage device gets disconnected, need a clean, fast solution to the file change notification problem. Here it is. John McCutchan and I had been working on inotify for about a year when it was finally merged into Linus' kernel tree and released with kernel version 2.6.13. Although a long struggle, the effort culminated in success and was ultimately worth every rewrite, bug and debate. What Is inotify? inotify is a file change notification system—a kernel feature that allows applications to request the monitoring of a set of files against a list of events. When the event occurs, the application is notified. To be useful, such a feature must be simple to use, lightweight with little overhead and flexible. It should be easy to add new watches and painless to receive notification of events. To be sure, inotify is not the first of its kind. Every modern operating system provides some sort of file notification system; many network and desktop applications require such functionality—Linux too. For years, Linux has offered dnotify. The problem was, dnotify was not very good. In fact, it stank. dnotify, which ostensibly stands for directory notify, was never considered easy to use. Sporting a cumbersome interface and several painful features that made life arduous, dnotify failed to meet the demands of the modern desktop, where asynchronous notification of events and a free flow of information rapidly are becoming the norm.
    [Show full text]
  • Displaying and Watching Directories Using Lazarus
    Displaying and Watching directories using Lazarus Michaël Van Canneyt October 31, 2011 Abstract Using Lazarus, getting the contents of a directory can be done in 2 ways: a portable, and a unix-specific way. This article shows how to get the contents of a directory and show it in a window. Additionally, it shows how to get notifications of the Linux kernel if the contents of the directory changes. 1 Introduction Examining the contents of a directory is a common operation, both using command-line tools or a GUI file manager. Naturally, Free/Pascal and Lazarus offer an API to do this. In fact, there are 2 API’s to get the contents of a directory: one which is portable and will work on all platforms supported by Lazarus. The other is not portable, but resembles closely the POSIX API for dealing with files and directories. Each API has its advantages and disadvantages. Often, it is desirable to be notified if the contents of a directory changes: in a file manager, this can be used to update the display - showing new items or removing items as needed. This can also be done by scanning the contents of the directory at regular intervals, but it should be obvious that this is not as efficient. There are other scenarios when a notification of a change in a directory is interesting: for instance, in a FTP server, one may want to move incoming files to a location outside the FTP tree, or to a new location based on some rules (e.g. images to one directory, sound files to another).
    [Show full text]
  • Forensic Framework for Honeypot Analysis
    FORENSIC FRAMEWORK FOR HONEYPOT ANALYSIS A Thesis Presented to The Academic Faculty by Kevin D. Fairbanks In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the School of Electrical and Computer Engineering Georgia Institute of Technology May 2010 Copyright c 2010 by Kevin D. Fairbanks FORENSIC FRAMEWORK FOR HONEYPOT ANALYSIS Approved by: Professor Henry L. Owen, III, Advisor Professor Chuanyi Ji School of Electrical and Computer School of Electrical and Computer Engineering Engineering Georgia Institute of Technology Georgia Institute of Technology Professor John A. Copeland Professor Jonathon T. Giffin School of Electrical and Computer College of Computing Engineering Georgia Institute of Technology Georgia Institute of Technology Professor Raheem A. Beyah Date Approved: March 17th, 2010 School of Electrical and Computer Engineering Georgia Institute of Technology For my grandfather, Willie Lee Fairbanks. You are missed and remembered. iii ACKNOWLEDGEMENTS I have caught enough breaks and had too many of the right people injected in my life throughout the years such that I have garnered a great amount of suspicion about whether or not I have been hurtling through a series of coincidences. I therefore must thank God as I believe that I have been abundantly blessed and without the belief in a higher power, my journeys through the land of academia would have been more harsh and dark than they needed to be. Without the emotional and financial support of my mother and father, Sharron and Whayman Fairbanks Sr, my pursuit of an advanced degree would have either ended pre- maturely or have been extensively protracted. I must also thank my bother and sister, Whayman Jr and Veronica Fairbanks, for being my biggest cheerleaders throughout this process.
    [Show full text]
  • Zack's Kernel News
    Community Notebook Kernel News Zack’s Kernel News Chronicler Zack Simple Flash Filesystem about endianness issues in order to avoid an- Brown reports on Dan Luedtke recently announced LanyFS – a noying, difficult debugging issues further filesystem to use with any drive you might down the road. the latest news, carry on a lanyard or keychain – essentially, Theodore Ts’o came back to the issue of small flash drives. The goal was to make the whether LanyFS was needed at all. He said, views, dilemmas, filesystem so simple that it would work easily “What I would do if I needed to transfer such and developments on any operating system. In this case, the lack a [6MB] file, and I didn’t have access to high of features was itself a feature. speed networking, would be to use ext2, and within the Linux Richard Weinberger and Marco Stornelli then either use the ext2 FUSE driver with didn’t see the point of such minimalism. To FUSE for Windows or Macintosh – or, I would kernel community. them, it just seemed like reinventing the port the userspace e2tools package to the tar- wheel, because other filesystems already ex- get OS, and use that to access the ext2 file By Zack Brown isted with a larger set of features. And, Alan system. And I’d do that because the software Cox gave a link to an interesting article at is available today, right now, without having http:// lwn.net/ Articles/ 428584/ that discussed to figure out how to port LanyFS to the oper- the ins and outs of trying to code a flash-ori- ating system.” ented filesystem.
    [Show full text]
  • Oversubscribing Inotify on Embedded Platforms
    Oversubscribing inotify on Embedded Platforms By Donald Percivalle (CPE) and Scott Vanderlind (CSC) Senior Project California Polytechnic State University San Luis Obispo Dr. Zachary Peterson June 11, 2015 Abstract For most computers running the popular Linux operating system, the inte- grated kernel component inotify provides adequate functionality for monitor- ing changes to files present on the filesystem. However, for certain embedded platforms where resources are very limited and filesystems are very populated (like network attached storage (NAS) devices), inotify may not have enough resources to provide watchers for every file. This results in applications missing change notifications for files they have watched. This paper explores methods for using inotify most effectively on embedded systems by leveraging more la- tent storage. Benefits of this include a reduction in dropped notifications in favor of an introduced delay on notifications for files that are less frequently changed. Contents 1 Introduction 3 1.1 Application . .3 1.2 Problem . .3 1.3 Problem Statement . .4 2 Possible Solutions 5 2.1 Modification of inotify . .5 2.2 Wholesale replacement of inotify . .5 2.3 Development of user-space watch aggregator . .5 2.4 Chosen Approach . .6 2.5 Related Work . .6 3 Design 6 3.1 Constraints . .6 3.2 Implementation . .7 4 Takeaways 8 4.1 Solution Viability . .8 4.2 Future Work . .8 5 Reference Implementation 9 5.1 Driver Application . 10 5.2 Watch Aggregation Module . 12 5.3 Directory Analysis Module . 25 2 1 Introduction as possible. This goal requires a system to monitor the entire filesystem for new Western Digital produces lines of Net- files.
    [Show full text]
  • Monitor and Manage System and Application Configuration Files at Kernel Level in GNU/Linux
    2015-09-23 Monitor and manage system and application configuration files at kernel level in GNU/Linux Saša Stanković DEGREE PROJECT Computer Engineering Bachelor level G2E, 15 hec Department of Engineering Science, University West, Sweden DEGREE PROJECT Monitor and manage system and application configuration files at kernel level in GNU/Linux Summary The aim of this study is to investigate if there is a way a computer can accurately and automatically react on altered configuration file(s) with a minimum of resource utilization and by what means the developer(s) of an application can perform a check of the altered configuration file for their application. In a typical GNU/Linux installation the configuration files are literally counted by the thousands, monitoring these files is a task that for the most part exceeds any system administrator's abilities. Each file has its own syntax that needs to be known by the administrator. Either one of these two tasks could give any system administrator nightmares concerning the difficulty level especially when both tasks are combined. The system administrator could attempt to automate the monitoring tasks together with the syntax checking. There are some tools in the repositories of each distribution for monitoring files but none that lets one monitor and take (predefined or user defined) actions based on what the file monitor reports, the type of file and its contents. A complete tool is not presented in this study, merely a proof of concept that monitoring and taking actions especially with version 2.6.13 (or newer) kernel of GNU/Linux with plugins are quite possible with relatively small computer resource.
    [Show full text]
  • Around the Linux File System World in 45 Minutes
    Around the Linux File System World in 45 minutes Steve French IBM Samba Team [email protected] Abstract lines of code), most active (averaging 10 changesets a day!), and most important kernel subsystems. What makes the Linux file system interface unique? What has improved over the past year in this important part of the kernel? Why are there more than 50 Linux 2 The Linux Virtual File System Layer File Systems? Why might you choose ext4 or XFS, NFS or CIFS, or OCFS2 or GFS2? The differences are not al- The heart of the Linux file system, and what makes ways obvious. This paper will describe the new features Linux file systems unique, is the virtual file system in the Linux VFS, how various Linux file systems dif- layer, or VFS, which they connect to. fer in their use, and compare some of the key Linux file systems. 2.1 Virtual File System Structures and Relation- ships File systems are one of the largest and most active parts of the Linux kernel, but some key sections of the file sys- tem interface are little understood, and with more than The relationships between Linux file system compo- than 50 Linux file systems the differences between them nents is described in various papers [3] and is impor- can be confusing even to developers. tant to understand when carefully comparing file system implementations. 1 Introduction: What is a File System? 2.2 Comparison with other Operating Systems Linux has a rich file system interface, and a surprising number of file system implementations.
    [Show full text]
  • GNU Guix Reference Manual Using the GNU Guix Functional Package Manager
    GNU Guix Reference Manual Using the GNU Guix Functional Package Manager The GNU Guix Developers Edition 34cf1f4 29 September 2021 Copyright c 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Court`es Copyright c 2013, 2014, 2016 Andreas Enge Copyright c 2013 Nikita Karetnikov Copyright c 2014, 2015, 2016 Alex Kost Copyright c 2015, 2016 Mathieu Lirzin Copyright c 2014 Pierre-Antoine Rault Copyright c 2015 Taylan Ulrich Bayırlı/Kammer Copyright c 2015, 2016, 2017, 2019, 2020, 2021 Leo Famulari Copyright c 2015, 2016, 2017, 2018, 2019, 2020 Ricardo Wurmus Copyright c 2016 Ben Woodcroft Copyright c 2016, 2017, 2018, 2021 Chris Marusich Copyright c 2016, 2017, 2018, 2019, 2020, 2021 Efraim Flashner Copyright c 2016 John Darrington Copyright c 2016, 2017 Nikita Gillmann Copyright c 2016, 2017, 2018, 2019, 2020 Jan Nieuwenhuizen Copyright c 2016, 2017, 2018, 2019, 2020, 2021 Julien Lepiller Copyright c 2016 Alex ter Weele Copyright c 2016, 2017, 2018, 2019, 2020, 2021 Christopher Baines Copyright c 2017, 2018, 2019 Cl´ement Lassieur Copyright c 2017, 2018, 2020, 2021 Mathieu Othacehe Copyright c 2017 Federico Beffa Copyright c 2017, 2018 Carlo Zancanaro Copyright c 2017 Thomas Danckaert Copyright c 2017 humanitiesNerd Copyright c 2017, 2021 Christine Lemmer-Webber Copyright c 2017, 2018, 2019, 2020, 2021 Marius Bakke Copyright c 2017, 2019, 2020 Hartmut Goebel Copyright c 2017, 2019, 2020, 2021 Maxim Cournoyer Copyright c 2017, 2018, 2019, 2020, 2021 Tobias Geerinckx-Rice Copyright c 2017 George Clemmer Copyright c 2017 Andy Wingo Copyright c 2017, 2018, 2019, 2020 Arun Isaac Copyright c 2017 nee Copyright c 2018 Rutger Helling Copyright c 2018, 2021 Oleg Pykhalov Copyright c 2018 Mike Gerwitz Copyright c 2018 Pierre-Antoine Rouby Copyright c 2018, 2019 G´abor Boskovits Copyright c 2018, 2019, 2020 Florian Pelz Copyright c 2018 Laura Lazzati Copyright c 2018 Alex Vong Copyright c 2019 Josh Holland Copyright c 2019, 2020 Diego Nicola Barbato Copyright c 2019 Ivan Petkov Copyright c 2019 Jakob L.
    [Show full text]
  • The Sysfs Filesystem
    The sysfs Filesystem Patrick Mochel [email protected] Abstract sysfs is a core piece of kernel infrastructure, which means that it provides a relatively sim- ple interface to perform a simple task. Rarely sysfs is a feature of the Linux 2.6 kernel that al- is the code overly complicated, or the descrip- lows kernel code to export information to user tions obtuse. However, like many core pieces processes via an in-memory filesystem. The or- of infrastructure, it can get a bit too abstract ganization of the filesystem directory hierarchy and far removed to keep track of. To help al- is strict, and based the internal organization of leviate that, this paper takes a gradual approach kernel data structures. The files that are created to sysfs before getting to the nitty-gritty details. in the filesystem are (mostly) ASCII files with (usually) one value per file. These features en- First, a short but touching history describes its sure that the information exported is accurate origins. Then crucial information about mount- and easily accessible, making sysfs one of the ing and accessing sysfs is included. Next, most intuitive and useful features of the 2.6 ker- the directory organization and layout of sub- nel. systems in sysfs is described. This provides enough information for a user to understand the organization and content of the information that Introduction is exported through sysfs, though for reasons of time and space constraints, not every object and its attributes are described. sysfs is a mechanism for representing kernel objects, their attributes, and their relationships The primary goal of this paper is to pro- with each other.
    [Show full text]
  • Linux System Programming Focuses on Everything Above the Kernel, Where Applications Such As Apache, Bash, Cp, Vim, Emacs, Gcc, Gdb, Glibc, Ls, Mv, and X Exist
    Overview This book is about writing software that makes the most effective use of the system you're running on -- code that interfaces directly with the kernel and core system libraries, including the shell, text editor, compiler, debugger, core utilities, and system daemons. The majority of both Unix and Linux code is still written at the system level, and Linux System Programming focuses on everything above the kernel, where applications such as Apache, bash, cp, vim, Emacs, gcc, gdb, glibc, ls, mv, and X exist. Written primarily for engineers looking to program (better) at the low level, this book is an ideal teaching tool for any programmer. Even with the trend toward high-level development, either through web software (such as PHP) or managed code (C#), someone still has to write the PHP interpreter and the C# virtual machine. Linux System Programming gives you an understanding of core internals that makes for better code, no matter where it appears in the stack. Debugging high-level code often requires you to understand the system calls and kernel behavior of your operating system, too. Key topics include: • An overview of Linux, the kernel, the C library, and the C compiler • Reading from and writing to files, along with other basic file I/O operations, including how the Linux kernel implements and manages file I/O • Buffer size management, including the Standard I/O library • Advanced I/O interfaces, memory mappings, and optimization techniques • The family of system calls for basic process management • Advanced process management,
    [Show full text]