Integration of Drivers in the Linux Kernel

THCHMUST04 Proceedings of ICALEPCS2011, Grenoble, France

FREE AND OPEN SOURCE SOFTWARE AT CERN: INTEGRATION OF DRIVERS IN THE LINUX KERNEL

Juan David Gonz´alez Cobas, Samuel Iglesias Gons´alvez, Julian Howard Lewis, Javier Serrano, Manohar Vanga, CERN, Geneva, Switzerland Emilio G. Cota, Columbia University, NY (formerly at CERN), U.S.A. Alessandro Rubini, Federico Vaga, University of Pavia, Italy

Abstract Benefits Most device drivers written for accelerator control sys- There are many reasons that make the insertion of code tems suffer from a severe lack of portability due to the in the mainline kernel a desirable target. ad hoc nature of the code, often embodied with intimate knowledge of the particular machine it is deployed in. In • Smoother maintenance in the (frequent) event of ker- this paper we challenge this practice by arguing for the op- nel API changes. [3] posite approach: development in the open, which in our • Very strict process of peer review of the code by case translates into the integration of our code within the knowledgeable and specialised maintainers. Linux kernel. We make our case by describing the up- • Widespread distribution of the code base, which can stream merge effort of the tsi148 driver, a critical (and then be enhanced and get contributions by researchers complex) component of the control system. The encour- working on similar problems. aging results from this effort have then led us to follow • Input from the topmost experts in the field. the same approach with two more ambitious projects, cur- • Best practice and use of bleeding-edge tools selected rently in the works: Linux support for the upcoming FMC by experienced programmers, e.g. git [4], sparse [5] boards [1, 2] and a new I/O subsystem. and Coccinelle [6]. • Avoidance of suboptimal, ad hoc solutions in favour TSI148 DRIVER INTEGRATION IN THE of the best ones from the technical point of view. • KERNEL Contributing back in return to the many benefits the FOSS community gives us. Rationale • Being able to drive a critical hardware component with software in the vanilla kernel, with no local, id- The VME bus is a central component of the Controls iosyncratic modifications. System at CERN. We rely on 1140 FECs (Front End Com- puters), 710 of which are VME crates with SBC (Single It is worth noting that the importance of the last point Board Computers). A process of renovation is in course, only became apparent once the merge effort was well un- involving the migration from derway. In our control system, renovated FECs are diskless • CES RIO2/RIO3 SBCs with PowerPC CPUs runing machines that boot a custom rt kernel [11], carefully con- LynxOS (around 605 crates by August 2011), to figured and patched to match local needs. The upstream • MEN-A20 SBCs with Intel CPUs running real-time merge of the tsi148 driver allows us to deploy off-the-shelf Linux (around 105 by August 2011). kernels packaged by linux distributions, thus largely reduc- ing the kernel maintenance burden. The MEN-A20 SBC incorporates a TSI148 VME-to- PCI-X bridge chip. To support the functionalities re- quired, and for maximum backward compatibility with the Caveats legacy CES API, a device driver was developed by CERN’s Some of the caveats herewith enumerated will be elab- BE/CO group in spring 2009. Clearly, this is a critical com- orated further in the description of the integration process, ponent of the controls system: every VME device relies on but we summarize here the most important ones. the provided programming interface to the VME bus. The CERN-developed driver offers, therefore, a CES compati- • It is hard, frustrating at times. One should be ready for bility API to facilitate the transition during the renovation the peculiar culture of the Linux Kernel Mailing List. process and a new API more in line with what is common • Design, APIs and coding practice that are customary practice in the Linux kernel. or simply acceptable locally must be adapted or re- Work on the inclusion of the driver in the Linux kernel built to comply with the strict standards imposed by

2011 bystarted the respective authors — cc Creative Commons Attribution 3.0 (CC BY 3.0) in late 2010 and is currently nearing completion. the Linux kernel developers. The morale is that the c ○ This effort, along with its motivations and consequences, is end product will most certainly be different to (and documented in the remainder of this Section. better than) the original.

• One must be prepared to compromise. The most prac- Wishbone Cores tical, short-lived solution might not be the technically Core Drivers .... perfect one kernel maintainers aim at. • Maintainers are occasionally hard to deal with. Wishbone Bus • The review process may be long; several iterations are usually necessary when merging significant changes. Carrier Firmware • Small, incremental changes are more likely to be ac- Driver cepted than big, hard-to-digest ones. The rule of thumb is to “Separate logical changes into a single MZ FPGA patch file.” [7] ID • Having a good history of prior contributions gives Application Bitstreams more respectability and ease of acceptance: the system is based on meritocracy. Figure 1: Wishbone bus driver architecture. The Process Submission of source code for acceptance in the kernel the overall quality of the code must improve signifi- is done by means of patches subject to a process of peer re- cantly. [8] • view before acceptance. The reviews are sometimes daunt- There are API incompatibilities with the driver cur- ing, and much editing and re-submissions can result. This rently used at CERN; this implies that a transient ker- process of strict code review is another good practice our nel module must adapt interfaces if driver code has to team adopted for its development process, a practice that remain untouched. has proved very beneficial. Our fist attempt of integration of the tsi148 driver was DRIVERS FOR THE FMC FAMILY initiated in late 2010 by one of the authors (Emilio G. Cota). A family of carrier/mezzanine modules compliant with By that time, a tsi148 driver had recently been admitted the ANSI VITA 57 FMC standard [2] is described in [1]. in the staging [8] area of the kernel, its maintainer being This kit of boards, developed by the BE/CO Hardware and Martyn Welch. This driver brought a tentative support for Timing section at CERN is another case in point for the the VME bus that covered two VME-to-PCI bridges: benefits of a Linux kernel-centric approach. The hardware concept and architecture are described in the aforemen- • The Tundra Universe chips, with work derived from tioned paper [1]; the key point is that the mezzanine pro- VMELinux by John Huggins and Michael Wyrick. vides basic circuitry, and the core of the application logic is • The Tundra TSI148 chip, inspired in the above. implemented in the FPGA of the carrier board. The cores inside this FPGA are interconnected via a Wishbone [9] The set of patches initially submitted provided improve- bus. ments in several areas, esp. in orthodox implementation of From the software point of view, a particular instance the Linux bus/device model concepts. Acceptance by the of this family behaves like a PCI-to-Wishbone or VME-to- maintainer was partial, and modifications related to the de- Wishbone bridge. The Wishbone bus interconnects a set of vice model implementation remained controversial and not cores providing functionalities that are either common to accepted. the whole family of mezzanines, or specific to the applica- In 2011, another of this paper’s authors, Manohar Vanga, tion board actually plugged in the FMC slot. We show in took over the submission effort. Some bugs in the staging figure 2 the block diagram of the FMC 100MS 14 bit ADC, driver were fixed, and the bulk of modifications concern- a typical example of an FPGA application comprising cores ing the device model were partially acknowledged. This for, among others led to several review iterations; at the time of this writ- • 2 ing, the definitive submission and acceptance of the last set Basic I C interfacing to the mezzanine board. • of patches concerning the core device model is in its fi- Wishbone mastering. • nal stage, and will be applied by the corresponding kernel DMA access to DDR3 memory in the carrier board. • maintainer in the next merge window. Mezzanine-specific control logic (e.g. ADC program- Although the whole set of patches concerning bug fixes ming/setup). • and device model of the tsi148 VME bridge driver is fi- Interrupt control. nally accepted, there is still a long way before reaching the final desideratum, i.e., driving our tsi148 devices with The consequence of this modular design of the applica- stock software from the mainline kernel tree. tion FPGA is that the device driver architecture reflects the structure of this set of cores (see figure 1). The driver for 2011 by the respective authors — cc Creative Commons Attribution 3.0 (CC BY 3.0) c • The outstanding goal should be now getting the VME the carrier board performs essentially the following basic ○ driver out of the ./staging/ tree. For this to happen, functions

Embedded + realtime software 1249 Copyright THCHMUST04 Proceedings of ICALEPCS2011, Grenoble, France

DDR RAM gree of adherence to standard practice among the Linux kernel developers, because of the LynxOS coding practice formerly prevalent. FPGA DDR CONTROLLER Two needs arise here clearly, in the light of our intention of having our whole set of drivers incorporated upstream: AQN AQN ADDR DMA UTC CVRT • Proceed to a more homogeneous, Linux-only code WISHBONE BUS

4cha base. ADC CTRL CTRLR GN4124 100M 14b FMC ADC GN4124 GENNUM IRQ WB & DMA • Provide a general framework for data acquisition and CTRL control devices. BOARD CONTROL AND STATUS Concerning the latter, there are two candidate Linux kernel frameworks: Comedi [14] and IIO [15]. Both are in SYS CLK PWR ID & TEMP BOARD ID CTRL CTRL CHIP ./staging/, and careful analyses show that they prove insufficient for our needs. Therefore, the development of Figure 2: Block diagram of the FMC slow ADC applica- a more complete and acceptable framework for industrial tion. data acquisition and analog/digital I/O becomes part of our effort of integrating our supported device drivers in the Linux kernel. This effort is motivated both by the require- • Identify the carrier board and initialize it. ments of legacy drivers and by the requirements that the • Perform a basic identification of the mezzanine(s) in- FMC family will impose in driver design, esp. in the area stalled in the FMC slot(s), and their configured appli- of operating system interface. cations. Such a framework, codenamed zio, intends to cover all • Load the application firmware into the carrier FPGA. the relevant aspects of data acquisition devices, far beyond • Register a Wishbone bus with the kernel. the existing frameworks, such as • Enumerate the cores in that firmware (to wit, the inner blocks in figure 2). • Digital and analog input and output. • Register the devices those cores implement and install • One-shot and streaming (buffered) data acquisition or the drivers associated to them. waveform play. • Resolution. The process is directly implied by how the Linux kernel de- • Sampling rate. vice model is implemented as described in [10] or in chap- • Buffer management and timing for streaming conver- ter 14 of [12]. Conceptually, it amounts to the provision of sion. a bus driver for Wishbone, plus a PCI-to-Wishbone bridge • Support for DMA. driver for the carrier board. • Calibration, offset and gain. Modularity and reusability of cores and drivers are not • Bit grouping in digital I/O. the only rationale behind this design. Actually, the drivers • Timestamping. for the FMC family are the second family of drivers • Triggering of acquisition/output. planned to be integrated in the mainline kernel. Given that • Clean design conforming to Linux kernel practice. boards and their applications will be manufactured by ex- ternal companies and available to the general public and not At the time of this writing, prototype versions of the frame- only to CERN, having their drivers and the Wishbone bus work software are under development phase [16]. integrated upstream is the logical step to take. The Wish- bone enumeration is made possible through the definition FORTHCOMING STRATEGY of an FPGA configuration space developed in [13], that will be the basis for the integration of all drivers of this family The strategy for inclusion of our driver kit in the kernel in the kernel. relies now on a handful of key milestones 1. Initiate upstream integration of the Wishbone bus DATA ACQUISITION DRIVERS: KERNEL driver. FRAMEWORKS 2. Make the in-tree and our local API for tsi148 con- verge. The third family of drivers considered for integration up- 3. Get the VME driver out of staging. stream is less homogeneous than the FMC family. The 4. Develop the zio framework as a competitive alterna- BE/CO group supports a standard kit of hardware mod- tive to Comedi and IIO. ules for data acquisition and general analog and digital I/O, 5. Initiate integration of the sis33xx drivers into the some of whose characteristics can be compared in table 1. zio framework and the mainstream kernel. 2011 byLinux the respective authors — cc Creative Commons Attribution 3.0 (CC BYdrivers 3.0) for these modules were developed in differ- c ○ ent stages of the renovation process, leading from legacy Points 1, 2 and 4 are short-term priorities that should come LynxOS drivers to Linux; therefore, there is a varying de- out as quite straightforward.

Table 1: BE/CO Data Acquisition Modules Module Type Channels Resolution Max. Speed Bus VMOD-12E8/16 Analog input 8/16ch 12b 15us/sample VME/PCI VD80 Analog input 16ch 16b 200kS/s VME SIS3300 Analog input 8ch 12/14b 100MS/s VME SIS3302 Analog input 8ch 16b 100MS/s VME SIS3320 Analog input 8ch 12b 250MS/s VME “Fast” FMC ADC Analog input 4ch 14b 100Ms/s VME/PCIe (Wishbone) “Slow” FMC ADC Analog input 8ch 16b 100kS/s VME/PCIe (Wishbone) CVORB V4 Analog output 16ch 16b 5us/sample VME VMOD-12A2/4 Analog output 2ch 12b 10us/sample VME/PCI CVORG Analog output 2ch 14b 100 MS/s VME VMOD-TTL Digital I/O 20ch 1b n/a VME/PCI CVORA Digital I/O 32ch 1–32b 100Mhz VME

LESSONS LEARNED REFERENCES

The main lessons we got from this process, that we will [1] P. Alvarez, M. Cattin, J. H. Lewis, J. Serrano and T. Wlostowski, “FPGA Mezzanine Cards for CERNs Accel- have to take into account in the course of the forthcoming erator Control System”, in ICALEPCS’09, p. 376, 2009. integrations, are [2] VME International Trade Association, “FPGA Mezzanine Card (FMC) Standard”, http://www.vita.com/ • Being there first gives competitive advantage. How- [3] G. Kroah-Hartman, “The Linux Kernel Driver Interface”, ever technically sound a proposal might be, it is better see Linux sources under to get it accepted when it is new, and not contending Documentation/stable api nonsense.txt. with other more entrenched positions. • [4] L. Torvalds. “git: The Fast Version Control System”, see Getting input and enhancements from Linux peer de- http://git-scm.com/. velopers is tremendously beneficial to improve the [5] L. Torvalds. “Sparse - a Semantic Parser for C”, see quality of the code, of the design and of the working http://sparse.wiki.kernel.org/. environment (tools, policies and style being naturally [6] J.L. Lawall, J. Brunel, N. Palix, R.R. Hansen, H. Stuart, given by what the Linux kernel developers have al- G. Muller, “WYSIWIB: A declarative approach to find- ready tried and tested over many years). ing API protocols and bugs in Linux code,”, in The 39th • There is a change of thinking when it comes to devel- Annual IEEE/IFIP International Conference on Dependable opment after a while in the kernel community. The Systems and Networks, pp. 43-52, 2009. mindset changes from thinking of the short term goals [7] “Submitting Patches”, see to thinking of solutions that can scale to a larger set linux-src/Documentation/SubmittingPatches. of problems. One starts to think of more generic solu- [8] G. Kroah-Hartman, “linux-staging tree created”, announce- tions rather than quickly hacking together the fastest ment on the Linux Kernel Mailing List, June 2010. See solution to the problem at hand. https://lkml.org/lkml/2008/6/10/329 [9] OpenCORES, “Wishbone B4: WISHBONE System-on- Chip (SoC) Interconnection Architecture for Portable IP CONCLUSIONS Cores”, see http://opencores.org/opencores,wishbone Contrary to what conventional assumptions dictate, low- [10] “Linux Kernel Device Model”, see level software and development need not be intimately linux-src/Documentation/driver-model/. linked to, nor closely shaped after, the peculiarities of a [11] S. Rostedt and D.V. Hart, “Internals of the RT patch”. In single control system. Developing with a wider scope in Linux Symposium, 2007. mind is possible, as our experience proves. But not only [12] J. Corbet, A. Rubini, G. Kroah-Hartman, “Linux Device that: it results in technically superior, more scalable and Drivers”, 3rd edition. 2005. maintainable solutions. [13] FPGA config space Open Hardware Project, see http:// Last, but not least, peer review, advice and a bleeding- www.ohwr.org/projects/fpga-config-space/ edge set of tools created by top level programmers con- [14] See http://www.comedi.org/ tribute to the efficiency and quality of the development 2011 by the respective authors — cc Creative Commons Attribution 3.0 (CC BY 3.0)

[15] See article in http://lwn.net/Articles/339674/ c process; a priceless gift we also owe to the community of ○ [16] See git://gnudd.com/zio-beta.git Linux kernel developers.