INFORMATION TECHNOLOGIES 217

VISUALIZATION OF GRAPHIC INFORMATION IN THE GENERAL-PURPOSE OPERATING SYSTEMS Konstantin V. Pugin, Kirill A. Mamrosenko, Alexander M. Giatsintov Scientific Research Institute of System Analysis of the RAS, Center of Visualization and Satellite Information Technologies, https://niisi.ru Moscow 117218, Russian Federation [email protected], [email protected], [email protected] Abstract. Article describes solutions for developing programs for interaction between Linux operating system and display controller. Operating system architecture encourages creating a driver — component, which task is to perform interaction of hardware controller and OS kernel with the use of many protocols. The development of drivers for the open source OS is difficult due to continuous changes in the structure of the kernel, which breaks backward compatibility frequently. Several approaches to display controller driver development are provided in the article. Basic concepts of these drivers include Kernel Mode Setting (or KMS), meant to provide driver in the kernel and User Mode Setting, meant to provide driver in graphical server. Specific way is used for develop Linux kernel drivers - it is half-procedural, and half-object oriented. Sometimes it is called as “OOP in C”, because it is based on GNU C Language extensions usage to simulate object-oriented techniques. GNU C usage almost prevents using non-GNU compilers to build a driver. Newest concept of writing display controller driver for Linux kernel is based upon atomic mode setting concept, in modern Linux it is achieved via state concept - intermediate states of an object are stored and modified without side effects to other objects. In order to make this concept work, driver should conform to the principle “disable function undoes all effects of enable and nothing more”. Several approaches for display driver development are provided in this article, and most modern method – atomic KMS mode setting, - is described in detail. Keywords: drivers, embedded, KMS, kernel module, development UDC 004.454 Bibliography - 11 references Received 23.01.2019, accepted 05.02.2019 RENSIT, 2019, 11(2):217-224 DOI: 10.17725/rensit.2019.11.217 Contents 1. INTRODUCTION 1. Introduction (217) Currently, due to the increase in the share 2. Components of the Graphic System of various wearable and embedded systems, (218) it is a relevant objective to develop drivers 3. Linux Graphic Systems and Drivers for them. Many embedded systems use the (2182) Linux OS, which has the support of a large 4. Kernel Mode Setting and User Mode number of processor architectures, as well Setting. Historical Background (219) as the support of loadable modules, which 5. Object Model of the Linux Kernel allows the process of adding a driver to be (220) greatly simplified. 6. KMS Atomic Drovers. Basic Entities On many Linux-based embedded systems, and Interfaces Necessary for displaying information has its own specifics. Implementation (221) Often, a certain embedded system requires 7. Debugging Driver (222) the development of specialized drivers for 8. Conclusion (223) the applicable display controller. References (223)

RENSIT | 2019 | Vol. 11 | No.2 218 KONSTANTIN V. PUGIN, KIRILL A.MAMROSENKO, ALEXANDER M. GIATSINTOV INFORMATION TECHNOLOGIES

When developing a display controller interacting with video capture devices, driver based on the Linux operating system, and specialized random access memory it is necessary to take into account a number (Graphics Double Data Rate, GDDR) of aspects related to the kernel development for use by the graphic accelerator. style, development patterns, and rapid This list of components is not mandatory, changes in the kernel interaction interface since there are graphic subsystems in which (API). There are also few documents on one or more components are missing or not the development of open-source display involved. For example, in some NVIDIA controller drivers, and are mainly limited to Optimus solutions installed in laptops, the extracts from comments in the source code on-screen display controller is not involved, files of the Linux kernel, or schemes based the entire configuration of monitor modes on them. This article will look at a modern occurs through the controller of another approach to writing a display controller driver graphics card - Intel. Only a graphics for the Linux OS. accelerator is used from NVIDIA. You can also find systems with a different combination 2. COMPONENTS OF THE of selected components of the graphics GRAPHIC SYSTEM system. Due to the need for 3D rendering, high- intensity calculation of shaders, high- 3. LINUX GRAPHIC SYSTEMS AND resolution video processing and other DRIVERS requirements of a modern user of computer The graphic system in Linux is represented systems, the PC graphic subsystem has by a graphic stack and various libraries used become significantly more complicated in in it (Fig. 1) [2]. comparison with it in the early years, and If we consider the components of the most often a modern graphic system includes Linux graphical stack from top to bottom, [1]: we can say that the vast majority of them 1. A Graphic accelerator (Graphics are designed to interact with the GPU, in Processing Unit, GPU). It is its particular: most of the Wayland and libdrm, performance that is mainly measured by fully EGL and almost completely X. Only tests and various prototype software (for a small part of DDX. (Device Dependent example, by startup of 3D applications X) (Fig. 1.) and Wayland, as well as libgbm at maximum settings or specially created interacts with the on-screen display controller. scenes), it runs the shaders using a specialized multi-threaded architecture. 2. The display controller. This part of the graphics subsystem is used to determine the modes of graphics displaying, as well as to interact with monitors (obtaining available monitor resolutions, sending a finished frame, etc.). 3. You can also find specialized chips in the graphic system for video output, for Fig. 1. Graphic stack Linux.

No. 2 | Vol. 11 | 2019 | RENSIT VISUALIZATION OF GRAPHIC INFORMATION IN 219 INFORMATION TECHNOLOGIES GENERAL-PURPOSE OPERATING SYSTEMS

4.KERNEL MODE SETTING power management, and deleting the old AND USER MODE SETTING. code of the frame buffer drivers (in the era HISTORICAL BACKGROUND of tables and VBE (and for some drivers Initially, there was only user mode setting, even now), the kernel also contained a which was produced in the DDX drivers frame buffer management subsystem, which and enabled to set the mode for displaying allowed graphic output to be used without graphical information. [3] The detemination loading the graphic server for displaying was often executed through mode tables, console messages, or for drawing console and was not extensible, or it was done using programs). algorithms within a code that were unique to The first version of the KMS protocol [5] each chip. There was also a solution such as was built on the XRandR 1.2 model, which VBE (Vesa BIOS Extensions), an extension was good enough at that time to install modes for IBM PC BIOS, which allowed for some on ordinary workstations. But with the advent limited operations of installing modes with of mobile devices, the capabilities of the first its use (without writing a direct algorithm KMS specification began to be insufficient - for interacting with the graphics subsystem). in order to reduce energy consumption, the Gradually, most drivers for IBM PCs concept of unified planes was developed, switched to VBE. But VBE had some serious which was used to display video on a certain downsides: screen, to activate the hardware cursor (in the • Lack of support for multiple CRT first versions of KMS, the hardware cursor controllers (in most cases it is necessary interacted with the screen mode controller to support output from multiple graphic according to a separate protocol), and for accelerators to several displays or output other such cases. to TV). Then support for other KMS objects • Problems if there were several different was added, which allowed the gamma of the outputs (and they could be different on displayed output to be changed, the displayed the same chip series). image to be rotated directly in the driver, and • Problems of use of several different so on [6]. timers. There is one problem with all of this - for To solve these problems in 2008, an the interaction of these subsystems with the extension was accepted to the kernel [4], kernel, a large number of ioctl was required. which provided support for setting modes And when trying to interact, each device inside the kernel (kernel mode setting, or had to check every ioctl, which was very KMS). Along with the problems that this inconvenient. extension brought (such as some decrease in Due to these KMS problems on mobile the stability of the kernel due to the fact that devices, Google has developed its own the drivers are written not very properly, as interface to set modes, the Android Atomic well as the expansion of the kernel because Display Framework (ADF), for the Linux it includes many graphic drivers), there were kernel as part of the Android OS [7]. also some advantages, such as simplified Despite many advantages (atomic update debugging (because there is the entire mode of layers, ease of new drivers’ development), tuning code inside the kernel), simplified ADF had the following disadvantages:

RENSIT | 2019 | Vol. 11 | No.2 220 KONSTANTIN V. PUGIN, KIRILL A.MAMROSENKO, ALEXANDER M. GIATSINTOV INFORMATION TECHNOLOGIES

• There is only one update queue, and, This is how KMS introduced the concept therefore, the support of multi-screen of atomic updates. The modern driver of modes is very poor. If there were 2 the display controller must be written in the displays with different update rates, the kernel space using this concept. update rate was led to the same (most 5. OBJECT MODEL OF THE LINUX often less) one, which resulted in either KERNEL jerking at a faster display or slowing down To write a driver in kernel space, it is necessary at a slower one. to use its object model. [8] Despite the fact • ADF assumed that in user space there that the Linux kernel is written in C, there is a specific driver for the graphic chip, is an object model inside it that allows some which necessarily provides 2D or 3D object-oriented approaches to be applied acceleration, which did not satisfy the when writing kernel drivers. To build this developers of the main kernel, since object model the following methods are most it cut off all the "common" drivers often used: (designed for devices not yet supported • Establishment of a mutually one-to-one and providing minimal functionality) of correspondence between structures of the xf86-video-modesetting type. a special type and objects (within the • The ADF also had difficulties with meaning of the OOP). The structure of updating the chain of outputs. Since a special type contains: drivers with support for multiple outputs - pointers to functions or a pointer to a have a lot of shared resources that require vtable structure (where virtual methods a one-time update regardless of the are connected); number of outputs, whereas the ADF had - a pointer of the void* type, where a requirement for an atomic update of the the closed part of the structure data is entire chain of one output. For multiple connected (analogue of the closed type outputs, this could be implemented in a of inheritance); cycle, but this implementation did not - сontains the entire object of the base work in the case of multiple outputs and type (not in the form of a pointer) to shared resources. ensure the connectivity between the • ADF used the midlayer pattern that is not parent and the heir through special recommended in kernel development. compiler commands. This pattern is not used, because • Use of a special name and function despite the seeming usability and ease signature (object_type_name_operation_ of developing drivers for the midlayer, name (object_type* object, ...)), which there can always be a vendor that will be allows the type of object for which not satisfied with the midlayer - this is this function is intended to be precisely unacceptable for the kernel. defined. It is possible to establish a one- • ADF was new and there was no backward to-one correspondence between methods compatibility with old interfaces and in understanding the OOP and such APIs. functions. Therefore, it was decided not to save ADF, but to introduce its best features in KMS [5].

No. 2 | Vol. 11 | 2019 | RENSIT VISUALIZATION OF GRAPHIC INFORMATION IN 221 INFORMATION TECHNOLOGIES GENERAL-PURPOSE OPERATING SYSTEMS

• Vtable structures. C (unlike C++) graphic subsystem. Such a driver is most does not have such an entity as virtual often written for embedded systems, since functions, tables of the virtual function, components from different manufacturers and similar things. Therefore, to store (for example, a 3D chip from Mali, and on- virtual methods, the developers of the screen display controller from Kirin) can be kernel use structures consisting only used in them, which must be considered when of pointers to functions or storing in developing a driver for each component. addition to such pointers also variables The main components of the KMS driver that are similar to the variables of the (Fig. 2) [9]: protected type of inheritance in OOP. 1. Driver object (drm_driver). General These structures can be compared to a parameters (such as its version, creator, vtable from C++. and supported interfaces) are written • To build private fields in some structures in the driver structure body, and the there are void* variables in which a closed general issues of driver operation (work structure can be assigned (in particular, with GEM, interrupts, and device files) when initializing the KMS driver, it is are defined. As opposed to the general necessary to create a closed structure in practice of writing objects in the kernel, the drm_device structure) - container_of the driver object contains most of together with storing the structure of the pointers to virtual methods (only the "base class" in the "heir". To simplify file operations are added to the vtable the implementation of inheritance in structure). Initially, also work with vblank kernel structures, a method such as was determined there, but was transferred storing a complete basic structure within to CRTC (since in the case of multiple a derivative one is often used, and if it CRTC they may have different work with is necessary to obtain a derived structure vblank events). from the basic one, the container_of 2. Controller of CRT (CRTC). The state of macros is used, which internally uses the the mode displayed, objects of the main offsetof compiler command enabling plane and cursor, the atomic state of the to find the basic structure within the CRTC, and the activation bit are stored derivative one. in CRTC. Functions assigned by CRTC All of these things in general make writing code for the Linux kernel much easier and are used in most of the existing drivers. 6. KMS ATOMIC DROVERS. BASIC ENTITIES AND INTERFACES NECESSARY FOR IMPLEMENTATION This section will consider the issue of creating a Linux kernel module that works through DRM and KMS with a display controller and does not use other components of the Fig. 2. An example of the main components of the KMS driver.

RENSIT | 2019 | Vol. 11 | No.2 222 KONSTANTIN V. PUGIN, KIRILL A.MAMROSENKO, ALEXANDER M. GIATSINTOV INFORMATION TECHNOLOGIES

- working with vblank, with planes (main control functions and is linked to the plane, cursor and overlay planes) and connector. It sets the functions for with pages. obtaining permissible modes and timings 3. One or several planes. For each CRTC of the parent connector. there should be exactly one primary 7. DEBUGGING DRIVER plane, no more than one cursor plane, To debug the kernel space driver [10], it is and as many overlay planes as you want. possible to use: Planes are attached to the CRTC that 1. The kernel assembled in the Debug created them. mode (and the interfaces of such a 4. One or more encoders. The encoders kernel, output via debugfs). Inside the must convert the signal from the internal kernel assembled supported by dynamic DRM format to the output format and debugging, there is a debugfs virtual file vice versa if a non-trivial conversion system, in pseudo-files of which various is required. It adjusts the mode, and debugging information is output on user also has functions for managing the request. The disadvantages of this system power consumption of the information consist in the fact that it works only on output device. The encoder is not rigidly the loaded kernel and it is necessary to linked to the CRTC, but instead it sets read the files through the debugged PC, the bits possible to use by CRTC. The and this system also works only with the encoder may also check the set mode for areas of the kernel code considered fit by admission, but usually it is used only in the developers themselves. For arbitrary non-trivial cases. debugging it is necessary to make own 5. Connector. The connector defines the debugging changes on the kernel. output and connects the device directly 2. The KGDB debugger. The KGDB and, through it, with one or more CRTC debugger is GDB started up on a remote and encoders. Also, the connector has PC connected via a serial port interface a status and it is there that the driver to a debugged board. To do this, inside determines the mode check for whether the kernel, there is an opportunity to it is possible to set it in this output. assemble with support of waiting for a Most often the number of connectors connection from the GDB interface, with corresponds to the number of video a support of displaying error messages outputs. In many cases, the connector and stack tracing to the serial port. On itself converts data into the necessary the part of the remote user's PC, it is format, in this case it is linked to an empty necessary to install the GDB debugger, encoder. as well as a kernel sample assembled in 6. Bridge (drm_bridge). The bridge debug mode with debug information, connects several encoders if the signal only in this case KGDB will provide needs to be converted several times. In information about what is happening terms of functionality, it is similar to the inside the kernel. To separate the regular connector and encoder. output and the debugging output, special 7. Panel (drm_panel). The panel is a mixer programs working as an interlayer software representation of LCD panel

No. 2 | Vol. 11 | 2019 | RENSIT VISUALIZATION OF GRAPHIC INFORMATION IN 223 INFORMATION TECHNOLOGIES GENERAL-PURPOSE OPERATING SYSTEMS

between the debugger and the kernel are of such type can be used both with GPU used. drivers and with new developments in the 3. Specialized FPGA (Field-Programmable field of software rasterization (in particular, Gate Array), such as Palladium, which llvmpipe).

allow graphics subsystem to trace the Acknowledgment receipt of signals [11]. FPGA data has Publication is made as part of national assignment for access to signals and values in the registers fundamental scientific researches (GP 14) on the topic of devices that they emulate, which allows (project) No.0065-2019-0001. the value of kernel to be compared with the recorded values when debugging the REFERENCES 1. Madieu J. Linux Device Drivers Development. drivers. Birmingham, Packt Publishing, 2017, 586 p. 8. CONCLUSION 2. Efremov IA, Mamrosenko KA, When developing an on-screen display Reshetnikov VN. Metody razrabotki controller driver for embedded systems, it drajverov graficheskoj podsistemy is necessary to take into account a number [Methods of development graphics of aspects, such as debugging difficulties, subsystem drivers]. Programmnye a special object model of the Linux produkty i sistemy, 2018, 31(3):425-429 kernel, the specifics of interaction with (in Russ.). the hardware component. Since on-screen 3. Verhaegen L. X and Modesetting: Atrophy display controllers are available in almost illustrated. 2006, p. 7. each embedded system (except for those 4. Airlie Dave, Barnes Jesse, Bornecrantz that do not use on-screen display at all), the Jakob. DRM: add mode setting support problem of developing controller drivers for [Electronic resource]. URL: https:// them, approaches to their design, debugging git.kernel.org/pub/scm/linux/kernel/git/ and testing in accordance with the Linux torvalds/linux.git/commit/?id=f453ba046 kernel object model is an relevant objective. 0742ad027ae0c4c7d61e62817b3e7ef (last Currently, the development of these drivers access date: 04.09.2018). is carried out mainly in a closed way in 5. Vetter Daniel. Atomic mode setting design large companies that make only the final overview, part 1 [Electronic resource]. result available to the public, which has not URL: https://lwn.net/Articles/653071/ very positive impact on the development (last access date: 04.09.2018). of a single approach to the development 6. Pinchart L. Why and How to use KMS as of on-screen display controller drivers and Your Userspace Display API of Choice. approaches to the methods of screen display LinuxCon Japan Tokyo, 2013, 52 p. modes setting in general. 7. Syrjälä V, Clark R, Hackmann G. [RFC 0/4] This article considers the approaches to Atomic Display Framework [Electronic writing driver output controller modes used at resource]. URL: https://lists.freedesktop.org/ different times in Linux OS, as well as reviews archives/dri-devel/2013-August/044522.html the latest approach to the development of (last access date: 20.11.2018). data drivers for embedded systems in details. 8. Brown Neil. Object-oriented design The practical implementation of the driver patterns in the kernel, part 1 [Electronic

RENSIT | 2019 | Vol. 11 | No.2 224 KONSTANTIN V. PUGIN, KIRILL A.MAMROSENKO, ALEXANDER M. GIATSINTOV INFORMATION TECHNOLOGIES

resource]. URL: https://lwn.net/ Articles/444910/ (last access date: 04.09.2018). 9. The Linux Kernel Documentation [Electronic resource]. URL: https://www. kernel.org/doc/html/v4.11/gpu/drm-kms. html (last access date: 20.04.2018). 10. Bazhenov PS, Mamrosenko KA, Reshetnikov VN. Issledovanie i otladka komponentov dlya obrabotki trexmernoy grafiki operatsionnykh sistem [Research and debugging components for processing three dimensional graphics in operating systems]. Programmnye produkty, sistemy i algoritmy, 2018, 4:5 (in Russ.). 11. Bogdanov AYu. Opyt primeneniya platformy prototipirovaniya na PLIS "Protium" dlya verifikatsii mikroprotsessorov [Experience in applying the Protium SOC prototyping platform for verification of microprocessors]. Trudy NIISI RAN, 2017, 7(2):46-49 (in Russ.).

No. 2 | Vol. 11 | 2019 | RENSIT