Working with the Community

A Guest Lecture for CS 378 class at the University of Texas at Austin

Khoa Huynh, Ph.D. IBM Austin, TX

Linux Technology Center TopicsTopics

Open source development Open source licences Common risks Working with the Linux community General advices & tips Submitting your work to community control Bug tracking Submitting your work to Linux distributors Linux debug tools

Linux Technology Center OpenOpen SourceSource SoftwareSoftware (OSS)(OSS)

Freely available software in source code format Often distributed via Internet or CD (e.g., ) Redistribution cannot be prohibited or royalty imposed Often developed in a collaborative community effort OSS community very sensitive about meaning & use of term "open source" For the current definition, see: http://www.opensource.org/osd/ Not "Public Domain" Copyright law gives author rights author determines appropriate uses of a work e.g. you can't reproduce or modify a work without author's permission author may grant permission via a license which can impose obligations

Linux Technology Center OpenOpen SourceSource LicensesLicenses

There are many open source licenses e.g. GPL, LGPL, CPL, IPL, BSD, Apache, etc. Some common features Governs use, modification and distribution of code Grant of rights (e.g. modification, redistribution, etc.) Obligations (e.g. attribution, redistribution, etc.) Different licenses have different objectives GPL = "General Public License" LGPL = "Lesser/Library Public License" CPL/IPL = "Common/IBM Public License" BSD = "Berkeley Software Delivery" Others Some licenses make source code available but should not be confused with OSS licenses (e.g. Sun

Linux Technology Community Source License) Center OpenOpen SourceSource LicensesLicenses (cont'd)(cont'd)

Many projects are under GPL (e.g., Linux kernel, GNU tools, Gnome) GPL often referred to as "" Author: Foundation Software available freely Free availability of all source code and modifications No restriction on what others may do with the code Common risks No warranty (provided "as is") Viral effect (or contamination) between proprietary and open source software Intermingling of objects Cutting & pasting source code Linking to libraries (static or dynamic) Possible relinquishing of patent rights More info on OS licenses

Linux Technology http://www.opensource.org/licenses/ Center WorkingWorking withwith LinuxLinux CommunityCommunity

Submit your work to the community Match the patch to the code Use multiple patches Submit early, submit often Stakeholders and Decision Makers

Linux Technology Center MatchMatch PatchPatch toto thethe CodeCode

If the patch is to existing code, make sure the patch matches the surrounding code From a style perspective match indentation styles match function/variable naming From a design perspective use a similar functional breakdown. your code should be at least as portable as the surrounding code make sure your changes don't interfere with the maintainer/author's long-term vision for the code

Linux Technology Center UseUse MultipleMultiple PatchesPatches

Small patches are accepted more easily than big patches Each patch should make a single functional change That way the maintainer can accept a subset of your patches, if he/she wishes Patches can be submitted as a set or separately Document what they do & pre-requisites

Linux Technology Center SubmitSubmit Early,Early, SubmitSubmit OftenOften

In general, there is no penalty if you submit a patch and it is rejected The rejection often contains good constructive criticism Even if the patch is not ready to be merged into the mainline, post the patch for comments Open source development Improvements suggested by the community Extra testing by the community

Linux Technology Center SubmissionSubmission CriteriaCriteria

As soon as minimal functionality is present Doesn't have to be feature-complete Doesn't have to be tuned for performance Passes the smoke test --- obvious bugs have been fixed Overall design completed (recommended)

Linux Technology Center StakeholdersStakeholders andand DecisionDecision MakersMakers

Identify who the stake holders and who the decision makers are They are not necessarily the same! Consult with the stake holders Satisfy the decision maker Avoid falling into the trap of spending too much time trying to achieve 100% approval from stake holders For the Linux kernel: Stake holders: LKML, individual developers, subsystem maintainers, etc. Decision maker:

Linux Technology Center SubmittingSubmitting toto thethe LinuxLinux KernelKernel Key considerations Source control: BitKeeper Bug tracking: Kernel Bug Tracker (new) Trivial Patch Monkey (for small patches) Types of kernel patches

Linux Technology Center SourceSource Control:Control: BitKeeperBitKeeper

Preferred form for large submissions to Linus Send e-mail to Linus with URL to BK tree, and description of the change Make sure the change sets (patches) will apply cleanly to the current BK tree BitKeeper http://www.bitkeeper.com Scalable through replication Change sets == Patches Powerful merging tools Repositories as branches allows controlled development Each person has his/her own personal repositories Repositories are compressed to reduce disk space and increase performance Multi-protocol (FS/RSH/SSH/HTTP/BKD/SMTP) connectivity

Linux Technology Center UsingUsing BitKeeperBitKeeper

File names revisioned and propagated just like contents Graphical interfaces provided for merging, browsing, and creating changes Change sets One or more changes to one or more files (i.e., patches) Built-in configuration management Creation of a change set saves the entire state of the repository (both what changed and what didn't) Open Logging Required for BitKeeper Open License (free use) Not required for paying customers Centralized change logs (everyone can see) Trading privacy loss for money

Linux Technology Center UsingUsing TrivialTrivial PatchPatch MonkeyMonkey

Preferred form for small patches Rusty Russell reviews & bundles small patches into a large submission to the mainline kernel source tree E-mail address of this patch system: [email protected] More details on how to submit trivial patches to this system http://www.uwsg.iu.edu/hypermail/linux/kernel/0205.3/1019.html.

Linux Technology Center TypesTypes ofof KernelKernel PatchesPatches

New subsystem Infrastructural patches Replacing / rewriting existing code

Linux Technology Center NewNew SubsystemSubsystem

Examples: New filesystem (JFS, CIFS) New subsystem (EVMS, SCTP) New device driver Most changes are in a self-contained file or directory Minimal changes needed to the rest of the kernel (config.in, Configure.help, makefiles) Small changes to kernel code are acceptable, but should be minimized

Linux Technology Center SubmittingSubmitting AA NewNew SubsystemSubsystem

Since it represents new functionality, Linus is much more forgiving Stylistic guidelines not as important If any infrastructure changes are required: Package them separately Submit at the same time as the new subsystem Create a Bitkeeper tree, and send e-mail to Linus with the Bitkeeper URL Include description/diffs of code changes to the core kernel in the e-mail

Linux Technology Center InfrastructureInfrastructure PatchesPatches

Example: Read-Copy-Update Patch should include at least one or two uses of the new interfaces Should demonstrate why new interface is needed API should be simple Underlying implementation can be complicated (as needed) Community support is much more important for infrastructure patches Because the community will be using the interface Submit early, submit often

Linux Technology Center ReplacingReplacing // RewritingRewriting ExistingExisting CodeCode

Acceptance from existing maintainer is critical Must provide noticeable improvements to community

Linux Technology Center BugBug Tracking:Tracking: KernelKernel BugBug TrackerTracker

Bug tracking system for Linux kernel community http://bugme.osdl.org Currently for 2.5 kernel bug tracking only (will likely extend for 2.6) Introduced by IBM Linux Technology Center (Khoa Huynh, Martin Bligh, Jon Tollefson) Why? Discussions, bugs, patches, announcements, etc., were all communicated via the Linux Kernel Mailing List (LKML) BitMover's Bug Manager is still under development Key characteristics Based on Mozilla.org's Bugzilla bug tracking system Web-based user interface Accessible via many popular network protocols (HTTP, SMTP, etc.) Automatic e-mail change notification MySQL database technology

Linux Technology File attachments (testcases, patches, etc.) Center LifeLife ofof AA BugBug inin KernelKernel BugBug TrackerTracker

Fix (patch) NOT OK

Bug Owner Assigned ASSIGNED

NEW Fix (patch) Available RESOLVED

Patch Already Available Patch Reviewed & Accepted By Kernel DEFERRED Community

Will Fix Later REJECTED VERIFIED Not A Valid Bug Duplicate Unreproducible Documented Bug Reopened Insufficient Data Patch Included In An Official Kernel Release Will Not Fix

CLOSED

Linux Technology Center SubmittingSubmitting CodeCode toto LinuxLinux DistributorsDistributors

Patches should be against current, official codebase of the project Patches should not lead to potential binary incompatible APIs with future official versions of the kernel and libraries Coding style should follow the coding style of the project Quality of patches should be in such a way that the officia maintainers could accept them Tests are always welcome Information on importance and relevance of the patches Is this a show stopper? Whom will the patch effect - positive and negative? Are there any binary incompatible issues ?

Linux Technology Center LinuxLinux DebugDebug ToolsTools

Linux RAS Package IBM's Linux Event Logging Opersys' Linux Trace Toolkit IBM's Dynamic Probes SGI's Linux Kernel Crash Dump (LKCD) SGI's Kernel Debugger (KDB) For Linux kernel, most useful (in my opinion): Kernel Crash Dump (LKCD) for system hangs (especially at customer sites) Kernel Debugger (KDB) for kernel oops (crashes), panics, etc.

Linux Technology Center LinuxLinux KernelKernel CrashCrash DumpsDumps (LKCD)(LKCD)

What is it? Set of utilities and kernel patches that allow a crash dump to be captured Tools that allow developers to analyze a crash dump When to install? Must be installed before the failure occurs A crash dump will be generated when A kernel oops occurs A kernel panic occurs A system administrator types Alt-SysRq- on the console Crash dump process Stage 1 Contents of system memory is copied to temporary disk location (dump device) System is rebooted Stage 2 Linux is booted Linux Technology Memory image is moved from dump device to /var/log/dump directory Center LKCDLKCD (cont'd)(cont'd)

Key files (in /var/log/dump) dump.n (crash dump file) kerntypes.n (info about kernel data structures, created during kernel build) map.n (info about kernel symbols, a copy of System.map file) lcrash.n (crash dump analyzer tool) Key options Non-disruptive dumps System is not rebooted after crash dump Dedicated dump device must be used Not available for kernel panics Location of dump device Default is swap partition Dedicated dump device is recommended (for faster recovery time) Must be at least as large as physical memory Location of /var/log/dump directory More info: http://lkcd.sourceforge.net Linux Technology Center LinuxLinux KernelKernel DebuggerDebugger (KDB)(KDB)

What is it? Set of kernel patches A way to examine kernel memory & data structures "live" When to install Must install patches and rebuild the kernel Key funtions Display essential system data structures given an identifier or address Single-step a processor Instruction disassembly Set breakpoints Upon execution of a specific instruction Upon access or modification of a specific virtual memory address Upon access to a register in the I/O address space Stack traceback for current active task or other tasks (via PID) More info: http://oss.sgi.com/projects/kdb/ Linux Technology Center SummarySummary

Open source software Beware of different licenses Avoid common risks (no warranty, contaminination, patent rights, etc.) Working with the Linux community Match your patch to the surrounding code Use multiple patches Submit early, submit often Consult stake holders, satisfy decision maker Working with the Linux kernel community Main communications medium: Kernel Mailing List (LKML) & sub-lists Source code control: BitKeeper Bug tracking: Kernel Bug Tracker Small patches --> Trivial Patch Monkey Submitting patches to Linux distributors

Linux Technology Center SummarySummary (continued)(continued)

Patches must be against current, official code base Avoid potential binary incompatibilities with APIs in future versions of the kernel and libraries Code quality must be high (must pass rigorous testing) Follow coding standards of affected projects Debug tools are introduced LKCD KDB

Linux Technology Center