Integration of Drivers in the Linux Kernel

THCHMUST04 Proceedings of ICALEPCS2011, Grenoble, France FREE AND OPEN SOURCE SOFTWARE AT CERN: INTEGRATION OF DRIVERS IN THE LINUX KERNEL Juan David González Cobas, Samuel Iglesias Gonsálvez, Julian Howard Lewis, Javier Serrano, Manohar Vanga, CERN, Geneva, Switzerland Emilio G. Cota, Columbia University, NY (formerly at CERN), U.S.A. Alessandro Rubini, Federico Vaga, University of Pavia, Italy Abstract Benefits Most device drivers written for accelerator control sys- There are many reasons that make the insertion of code tems suffer from a severe lack of portability due to the in the mainline kernel a desirable target. ad hoc nature of the code, often embodied with intimate knowledge of the particular machine it is deployed in. In • Smoother maintenance in the (frequent) event of ker- this paper we challenge this practice by arguing for the op- nel API changes. [3] posite approach: development in the open, which in our • Very strict process of peer review of the code by case translates into the integration of our code within the knowledgeable and specialised maintainers. Linux kernel. We make our case by describing the up- • Widespread distribution of the code base, which can stream merge effort of the tsi148 driver, a critical (and then be enhanced and get contributions by researchers complex) component of the control system. The encour- working on similar problems. aging results from this effort have then led us to follow • Input from the topmost experts in the field. the same approach with two more ambitious projects, cur- • Best practice and use of bleeding-edge tools selected rently in the works: Linux support for the upcoming FMC by experienced programmers, e.g. git [4], sparse [5] boards [1, 2] and a new I/O subsystem. and Coccinelle [6]. • Avoidance of suboptimal, ad hoc solutions in favour TSI148 DRIVER INTEGRATION IN THE of the best ones from the technical point of view. • KERNEL Contributing back in return to the many benefits the FOSS community gives us. Rationale • Being able to drive a critical hardware component with software in the vanilla kernel, with no local, id- The VME bus is a central component of the Controls iosyncratic modifications. System at CERN. We rely on 1140 FECs (Front End Com- puters), 710 of which are VME crates with SBC (Single It is worth noting that the importance of the last point Board Computers). A process of renovation is in course, only became apparent once the merge effort was well un- involving the migration from derway. In our control system, renovated FECs are diskless • CES RIO2/RIO3 SBCs with PowerPC CPUs runing machines that boot a custom rt kernel [11], carefully con- LynxOS (around 605 crates by August 2011), to figured and patched to match local needs. The upstream • MEN-A20 SBCs with Intel CPUs running real-time merge of the tsi148 driver allows us to deploy off-the-shelf Linux (around 105 by August 2011). kernels packaged by linux distributions, thus largely reduc- ing the kernel maintenance burden. The MEN-A20 SBC incorporates a TSI148 VME-to- PCI-X bridge chip. To support the functionalities re- quired, and for maximum backward compatibility with the Caveats legacy CES API, a device driver was developed by CERN’s Some of the caveats herewith enumerated will be elab- BE/CO group in spring 2009. Clearly, this is a critical com- orated further in the description of the integration process, ponent of the controls system: every VME device relies on but we summarize here the most important ones. the provided programming interface to the VME bus. The CERN-developed driver offers, therefore, a CES compati- • It is hard, frustrating at times. One should be ready for bility API to facilitate the transition during the renovation the peculiar culture of the Linux Kernel Mailing List. process and a new API more in line with what is common • Design, APIs and coding practice that are customary practice in the Linux kernel. or simply acceptable locally must be adapted or re- Work on the inclusion of the driver in the Linux kernel built to comply with the strict standards imposed by 2011 bystarted the respective authors — cc Creative Commons Attribution 3.0 (CC BY 3.0) in late 2010 and is currently nearing completion. the Linux kernel developers. The morale is that the c ○ This effort, along with its motivations and consequences, is end product will most certainly be different to (and documented in the remainder of this Section. better than) the original. Copyright 1248 Embedded + realtime software Proceedings of ICALEPCS2011, Grenoble, France THCHMUST04 • One must be prepared to compromise. The most prac- Wishbone Cores tical, short-lived solution might not be the technically Core Drivers .... perfect one kernel maintainers aim at. • Maintainers are occasionally hard to deal with. Wishbone Bus • The review process may be long; several iterations are usually necessary when merging significant changes. Carrier Firmware • Small, incremental changes are more likely to be ac- Driver cepted than big, hard-to-digest ones. The rule of thumb is to “Separate logical changes into a single MZ FPGA patch file.” [7] ID • Having a good history of prior contributions gives Application Bitstreams more respectability and ease of acceptance: the system is based on meritocracy. Figure 1: Wishbone bus driver architecture. The Process Submission of source code for acceptance in the kernel the overall quality of the code must improve signifi- is done by means of patches subject to a process of peer re- cantly. [8] • view before acceptance. The reviews are sometimes daunt- There are API incompatibilities with the driver cur- ing, and much editing and re-submissions can result. This rently used at CERN; this implies that a transient ker- process of strict code review is another good practice our nel module must adapt interfaces if driver code has to team adopted for its development process, a practice that remain untouched. has proved very beneficial. Our fist attempt of integration of the tsi148 driver was DRIVERS FOR THE FMC FAMILY initiated in late 2010 by one of the authors (Emilio G. Cota). A family of carrier/mezzanine modules compliant with By that time, a tsi148 driver had recently been admitted the ANSI VITA 57 FMC standard [2] is described in [1]. in the staging [8] area of the kernel, its maintainer being This kit of boards, developed by the BE/CO Hardware and Martyn Welch. This driver brought a tentative support for Timing section at CERN is another case in point for the the VME bus that covered two VME-to-PCI bridges: benefits of a Linux kernel-centric approach. The hardware concept and architecture are described in the aforemen- • The Tundra Universe chips, with work derived from tioned paper [1]; the key point is that the mezzanine pro- VMELinux by John Huggins and Michael Wyrick. vides basic circuitry, and the core of the application logic is • The Tundra TSI148 chip, inspired in the above. implemented in the FPGA of the carrier board. The cores inside this FPGA are interconnected via a Wishbone [9] The set of patches initially submitted provided improve- bus. ments in several areas, esp. in orthodox implementation of From the software point of view, a particular instance the Linux bus/device model concepts. Acceptance by the of this family behaves like a PCI-to-Wishbone or VME-to- maintainer was partial, and modifications related to the de- Wishbone bridge. The Wishbone bus interconnects a set of vice model implementation remained controversial and not cores providing functionalities that are either common to accepted. the whole family of mezzanines, or specific to the applica- In 2011, another of this paper’s authors, Manohar Vanga, tion board actually plugged in the FMC slot. We show in took over the submission effort. Some bugs in the staging figure 2 the block diagram of the FMC 100MS 14 bit ADC, driver were fixed, and the bulk of modifications concern- a typical example of an FPGA application comprising cores ing the device model were partially acknowledged. This for, among others led to several review iterations; at the time of this writ- • 2 ing, the definitive submission and acceptance of the last set Basic I C interfacing to the mezzanine board. • of patches concerning the core device model is in its fi- Wishbone mastering. • nal stage, and will be applied by the corresponding kernel DMA access to DDR3 memory in the carrier board. • maintainer in the next merge window. Mezzanine-specific control logic (e.g. ADC program- Although the whole set of patches concerning bug fixes ming/setup). • and device model of the tsi148 VME bridge driver is fi- Interrupt control. nally accepted, there is still a long way before reaching the final desideratum, i.e., driving our tsi148 devices with The consequence of this modular design of the applica- stock software from the mainline kernel tree. tion FPGA is that the device driver architecture reflects the structure of this set of cores (see figure 1). The driver for 2011 by the respective authors — cc Creative Commons Attribution 3.0 (CC BY 3.0) c • The outstanding goal should be now getting the VME the carrier board performs essentially the following basic ○ driver out of the ./staging/ tree. For this to happen, functions Embedded + realtime software 1249 Copyright THCHMUST04 Proceedings of ICALEPCS2011, Grenoble, France DDR RAM gree of adherence to standard practice among the Linux kernel developers, because of the LynxOS coding practice formerly prevalent. FPGA DDR CONTROLLER Two needs arise here clearly, in the light of our intention of having our whole set of drivers incorporated upstream: AQN AQN ADDR DMA UTC CVRT • Proceed to a more homogeneous, Linux-only code WISHBONE BUS 4cha base.

Load more