Embedded systems 2/7
J.-M Friedt
Introduction
Virtual memory access Embedded systems 2/7 Kernel module basics
Using the kernel: timers J.-M Friedt Conclusion & lab session FEMTO-ST/d´epartement temps-fr´equence
slides at jmfriedt.free.fr
September 9, 2020
1 / 24 Embedded systems 2/7 J.-M Friedt Operating system: the need for Introduction drivers Virtual memory access
Kernel module basics
Using the kernel: timers • Hardware abstraction: hide low level functions so that the developer Conclusion & lab session can focus on the functionalities provided by the peripheral → a single entry point providing system calls (open, read, write, close) hiding access to hardware • Homogeneous interface to all peripherals (“Everything is a file”) • Only the kernel can access hardware resources (DMA, interrupts) • Share resources and make sure only one process can access a given hardware function • Add functionalities to the Linux kernel: modules
2 / 24 Embedded systems 2/7 J.-M Friedt Virtual memory/hardware memory
Introduction Hardware memory addressing
Virtual memory • hardware memory: a value on the address bus identifies which access peripheral is active Kernel module basics • each peripheral decodes the address bus to detect whether it is the Using the kernel: target of a message timers • Conclusion & lab only one peripheral must match a given physical address (otherwise, session conflict) Virtual memory addressing • each process has its own address space • memory organization independent of physical constraints • dynamic loading binaries and associated libraries • MMU: translates between hardware and virtual memory addresses Virtual organization (process) Physical organization (CPU)
4096 Page 1 4096 Page 1
Page 2 Page 2 page & 0xff..f000 page & 0xff..f000 Page 3 Page 3 page & 0x00..0fff page & 0x00..0fff
Page N Page N 3 / 24 Embedded systems 2/7 J.-M Friedt Hardware access from userspace Introduction Through /dev/mem Virtual memory access • advantage: not going through kernel abstraction layers (fast) Kernel module • drawback: not going through kernel abstraction layers (no handling basics
Using the kernel: of concurrent memory access) timers #include < f c n t l . h> Conclusion & lab #include session #defineMAP SIZE 4096UL // MMU page size #define MAP MASK ( MAP SIZE−1) // mask
i n t main(int argc, char ∗∗ a r g v ) { i n t fd; void ∗ map base , ∗ v i r t a d d r ; u n s i g n e d long read result , writeval; o f f t target=0x12345678; // physical@ f d = open ( "/dev/mem" ,O RDWR | O SYNC) ; // MMU access map base=mmap(0 , MAP SIZE , PROT READ | PROT WRITE , \ MAP SHARED, fd , t a r g e t & ˜MAP MASK) ; v i r t a d d r=map base+(target & MAP MASK) ; // virt.@ r e a d r e s u l t =∗((unsigned long ∗) v i r t a d d r ) ; // read mem p r i n t f ( "0x%X(%p):0x%X\n" ,target ,virt a d d r , r e a d r e s u l t ) ; // ∗ ((unsigned long ∗) virt a d d r)= writeval;// write munmap( map base , MAP SIZE); close(fd); return 0; }
4 / 24 Embedded systems 2/7 J.-M Friedt The devmem tool
Introduction
Virtual memory access
Kernel module basics
Using the kernel: timers • Fast prototyping when accessing processor registers Conclusion & lab 1 session • Available in busybox • Use: Read/write from physical address ADDRESS Address to act upon WIDTH Width (8/16/...) VALUE Data to be written
1$BUILDROOT/output/build/busybox-1.30.1/miscutils/devmem.c 5 / 24 Embedded systems 2/7 J.-M Friedt Hardware access from the kernel
Introduction Virtual memory • access Userspace: mmap on the /dev/mem pseudo-file Kernel module • Kernel space: ioremap function after requesting the address range basics
Using the kernel: used by the peripheral timers #include
i f (request m e m r e g i o n ( IO BASE,0x2e4 , "GPIO test" )==NULL) printk (KERN ALERT "mem request failed" ); j m f gpio=(u32)ioremap(IO BASE, 0x2e4); // UG585p.1349 w r i t e l (1<<9, j m f gpio+0x204) ; // dir. of MIO9
To free the resource: r e l e a s e m e m r e g i o n ( IO BASE, 0x2e4);
Check if the requested memory range is available (otherwise, kernel panic)
6 / 24 Embedded systems 2/7 J.-M Friedt Basic kernel module structure Introduction Kernel module = “plugin” requiring strarting and ending functions Virtual memory access #include
i n t hello s t a r t ( ) // init m o d u l e(void) { printk (KERN INFO "Hello\n" ); r e t u r n 0; }
v o i d hello e n d ( ) // cleanup m o d u l e(void) { printk (KERN INFO "Goodbye\n" ); }
m o d u l e i n i t ( h e l l o s t a r t ) ; m o d u l e e x i t ( h e l l o e n d ) ; No printf (write to /var/log/messages with printk accessible with dmesg), no floating point operation, no file operations.
7 / 24 Embedded systems 2/7 J.-M Friedt Kernel module compilation
Introduction
Virtual memory A kernel module must link to the running kernel: access • Kernel module requires kernel sources ($BR/output/build/linux*) or at least basics configured headers (linux-headers-X.Y.Z-amd64 on Using the kernel: timers Debian/GNU Linux) Conclusion & lab • dedicated Makefile calling the Linux kernel Makefile modules session method on PC:
obj−m += mymod . o a l l : make −C /lib/modules/$ (shell uname −r ) / b u i l d \ M=$ (PWD) modules
or for cross-compiling, linking to the Buildroot $BR
obj−m +=mymod . o a l l : make ARCH=arm CROSS COMPILE=arm−l i n u x − \ −C $ (BR)/output/build/linux −XXX \ M=$ (PWD) modules
8 / 24 Embedded systems 2/7 J.-M Friedt Communicating from userspace Introduction with the kernel Virtual memory access Kernel module Through /dev basics Using the kernel: • write, read, or control (ioctl) timers
Conclusion & lab • each kernel module implements its own answer to the system calls session • each peripheral is identified by its class (major number) and its index (minor number)
brw-rw---- 1 root disk 8, 0 Feb 28 06:21 sda brw-rw---- 1 root disk 8, 1 Feb 28 06:21 sda1 brw-rw---- 1 root disk 8, 2 Feb 28 06:21 sda2 brw-rw---- 1 root disk 8, 3 Feb 28 06:21 sda3 [...] crw-rw---- 1 root dialout 4, 64 Feb 28 07:21 ttyS0 crw-rw---- 1 root dialout 4, 65 Feb 28 07:21 ttyS1 crw-rw---- 1 root dialout 4, 66 Feb 28 07:21 ttyS2 crw-rw---- 1 root dialout 4, 67 Feb 28 07:21 ttyS3
In case an entry is missing (e.g. created by udev) in /dev : mknod
9 / 24 Embedded systems 2/7 2 J.-M Friedt Architecture of a POSIX Introduction compliant OS Virtual memory access
Kernel module basics Using the kernel: Userspace timers
Conclusion & lab applications (libc) session
/dev/ttyUSB /dev/partport0
Module 1 Module 2 Module 3 (USB) (GPIO) (fs) Monolithic kernel (scheduler, interrupts, timers ...)
system fonctions, networking, process scheduling and memory sharing, vfs
2/dev directory: pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap10.html 10 / 24 Embedded systems 2/7 J.-M Friedt The /dev directory Introduction • includes pseudo-files linking the kernel with the user space Virtual memory access • each device is defined with a major number and, within each class, Kernel module basics a minor number Using the kernel: • timers on the kernel side, a driver links to the same major number and
Conclusion & lab provides implementations of the system calls session s t a t i c struct file operations fops= • { . r e a d : qadc read , open() . open : qadc open , • close() . u n l o c k e d i o c t l : q a d c i o c t l , • write() .release: qadc r e l e a s e , • read() // no.write on an ADC! • ioctl() } ; s t a t i c int i n i t q a d c i n i t (void) { ... r e g i s t e r chrdev (qadc major , "ppp" → ,→ , &f o p s ) ; ... }
m o d u l e i n i t ( q a d c i n i t ) ; m o d u l e e x i t ( q a d c e x i t ) ;
11 / 24 Embedded systems 2/7 J.-M Friedt Input/Output Control (ioctl) Introduction • mechanism breaking the “Everything is a file” philosophy Virtual memory • access configures a peripheral by providing “what” index and as an
Kernel module argument the “value” basics • no standardized command: the header file common to the driver Using the kernel: timers and the userspace program provides the list of IOCTL calls 3 Conclusion & lab • example: OSS (Open Sound System ) in session include/uapi/linux/soundcard.h of the kernel source: /∗ Use ioctl(fd, OSS GETVERSION,&int) to get the v e r s i o n number of the currently active driver. ∗/ ... /∗ IOCTL commands for/dev/dsp and/dev/audio ∗/ #define SNDCTL DSP RESET SIO( ’P’ , 0) #define SNDCTL DSP SYNC SIO( ’P’ , 1) #define SNDCTL DSP SPEED SIOWR( ’P’ , 2 , int) #define SNDCTL DSP STEREO SIOWR( ’P’ , 3 , int) ... • Userspace: s a m p l e rate = 48000; i f ( i o c t l ( fd , SNDCTL DSP SPEED , &s a m p l e r a t e ) == −1) { p e r r o r ( "SNDCTL_DSP_SPEED" ) ; e x i t (−1) ; } 3http://www.linuxdevcenter.com/pub/a/linux/2007/08/02/ an-introduction-to-linux-audio.html 12 / 24 Embedded systems 2/7 J.-M Friedt IOCTL: kernel
Introduction s t a t i c long dev i o c t l (struct file ∗ ,unsigned int,unsigned →
Virtual memory ,→ l o n g); access
Kernel module s t a t i c struct file operations fops= basics { . r e a d=d e v r e a d , Using the kernel: . open=dev open , timers . u n l o c k e d i o c t l=d e v i o c t l , Conclusion & lab session . w r i t e=d e v w r i t e , . release=dev r l s , } ;
s t a t i c long dev i o c t l (struct file ∗ f , unsigned int cmd, → ,→ u n s i g n e d long arg) { s t a t i c char var[10]; // DON’T read arg which is located i n t dummy ; //... in userspace: copy f r o m u s e r printk (KERN ALERT "ioctlCMD%d" ,cmd) ; s w i t c h ( cmd ) // one of the known ioctl { c a s e 0 : dummy=c o p y f r o m u s e r ( var , (char ∗) arg , 5 ) ; printk (KERN ALERT "ioctl0:%s\n" ,(char ∗) v a r ) ; break; c a s e 1: printk(KERN ALERT "ioctl1:%s\n" ,(char ∗) v a r ) ; dummy=c o p y t o u s e r ( (char ∗)arg,var ,5); break; d e f a u l t : printk(KERN ALERT "unknown ioctl" );break; } r e t u r n 0; } 13 / 24 Embedded systems 2/7 J.-M Friedt IOCTL: userspace Introduction #include
Conclusion & lab c h a r msg[30] = "Message passed by ioctl\n" ; session f i l e d e s c = open ( "/dev/jmf" , 0) ; msg [ 4 ] = 0 ; r e t val = ioctl(file d e s c , IOCTL SET MSG , msg ) ; r e t val = ioctl(file d e s c , IOCTL GET MSG , msg ) ; } donne (dmesg)
[ 943.551282] Hello ioctl [ 946.385700] ioctl open [ 946.385757] ioctl CMD0 [ 946.385758] ioctl0: Mess [ 946.385762] ioctl CMD1 [ 946.385763] ioctl1: Mess [ 946.385766] bye
14 / 24 Embedded systems 2/7 J.-M Friedt Kernel timer: direct GPIO access
Introduction #define mio 0
Virtual memory access i n t hello s t a r t ( ) Kernel module { i n t delay = 1; basics u n s i g n e d int s; Using the kernel: timers i f (request m e m r e g i o n ( IO BASE1,0x12c , "GPIO cfg" )==NULL) Conclusion & lab session printk (KERN ALERT "mem request failed" ); j m f g p i o = (void iomem ∗)ioremap(IO BASE1, 0x4); s=readl(jmf gpio+0x12c) ; // PAS assignation(crash): mask s |=(1<<22) ; //...GPIO clock writel(s,jmf gpio+0x12c) ; r e l e a s e m e m r e g i o n ( IO BASE1, 0x12c);
i f (request m e m r e g i o n ( IO BASE2,0x2E4 , "GPIO test" )==NULL) printk (KERN ALERT "mem request failed" ); j m f g p i o = (void iomem ∗)ioremap(IO BASE2, 0x2E4); w r i t e l (1< User Guide : Appendix B (Register Details) p.785 → gpio @ 0xE000A000 → DATA 0 en 0x040, DIRM 0 en 0x204 et OEN 0 en 0x208. 15 / 24 Embedded systems 2/7 J.-M Friedt Kernel timer: configuration and Introduction callback function Virtual memory access Timer configuration and releasing resources when leaving the module Kernel module basics t i m e r s e t u p (& e x p t i m e r , do something ,0) ; // since 4.14 e x p timer.expires = jiffies + delay HZ; Using the kernel: ∗ timers // HZ specifies number of clock ticks generated per second Conclusion & lab a d d t i m e r (& e x p t i m e r ) ; session } v o i d hello e n d ( ) // cleanup m o d u l e(void) { r e l e a s e m e m r e g i o n ( IO BASE2, 0x2e4); d e l t i m e r (& e x p t i m e r ) ; } Callback function: s t a t i c void do something (unsigned long data) { [...] mod timer (& e x p timer, jiffies + HZ); // relaunch timer } Accessing kernel functions requires compliance with the GPL License 4: MODULE LICENSE( "GPL" ); 4J. Corbet, Unexporting kallsyms lookup name(), 28/02/2020 @ https://lwn.net/Articles/813350/ 16 / 24 Embedded systems 2/7 J.-M Friedt Kernel timer: gpiolib Introduction Virtual memory • Rather than accessing the low level registers, use existing kernel access functions Kernel module basics • GPIO access: gpiolib Using the kernel: timers • kernel configuration (make linux-menuconfig) Conclusion & lab CONFIG_GPIO_XILINX: session Say yes here to support the Xilinx FPGA GPIO device Symbol: GPIO_XILINX [=m] Type : tristate Prompt: Xilinx GPIO support Location: -> Device Drivers -> GPIO Support (GPIOLIB [=y]) -> Memory mapped GPIO drivers • we have selected not to compile static libraries but to use modules in order to keep the ability to deactivate the libraries by removing the module from the kernel (rmmod) • as was seen in userspace, MIOx is refered to by the index 906+x 17 / 24 Embedded systems 2/7 J.-M Friedt Kernel timer – gpiolib Introduction i n t hello s t a r t (void); Virtual memory v o i d hello e n d (void); access Kernel module basics i n t hello s t a r t ( ) Using the kernel: { i n t jmf gpio=906+0; //0= orange,7= red(heartbeat) timers e r r=g p i o i s v a l i d ( j m f g p i o ) ; Conclusion & lab e r r=g p i o r e q u e s t o n e ( j m f g p i o , GPIOF OUT INIT LOW, "→ session ,→jmf_gpio" ); // see linux −XXX/linux/gpio.h for available functions t i m e r s e t u p (& e x p t i m e r , do something ,0) ; // since 4.14 e x p timer.expires = jiffies + delay ∗ HZ; // HZ specifies number of clock ticks generated per second a d d t i m e r (& e x p t i m e r ) ; r e t u r n 0; } v o i d hello e n d ( ) // cleanup m o d u l e(void) { g p i o f r e e ( j m f g p i o ) ; d e l t i m e r (& e x p t i m e r ) ; } output pin handling: gpio set value(jmf gpio,jmf stat); Using gpiolib: the licence (GPL) of the module matters ! 18 / 24 Embedded systems 2/7 J.-M Friedt Summary Introduction Virtual memory • Two input and output functions: access module init (beginning function); and Kernel module basics module exit (final function); Using the kernel: timers • initialize the link between the module and userspace Conclusion & lab register chrdev (90 , "jmf" ,&fops ); session • print message in /var/log/syslog by printk (KERN ALERT "formated message"); (no coma !) • ability to dynamically the dev entry point insmod module creates /dev/jmf by s t r u c t miscdevice jmfdev; { jmfdev.name = "jmf" ; ///dev/jmf: major=misc class jmfdev.minor = MISC DYNAMIC MINOR ; jmfdev.fops = &fops; m i s c register(&jmfdev); // dyanmic creation/dev/jmf } which concludes with (rmmod module): m i s c deregister(&jmfdev); 19 / 24 Embedded systems 2/7 J.-M Friedt Lab session Introduction 1 compile a simple module (init, exit) on the PC – Makefile Virtual memory access linking to the kernel source, obj-m +=mon_module.o Kernel module basics all: Using the kernel: make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules timers since Conclusion & lab session $ ls -l /lib/modules/4.12.0-1-amd64/ build -> /usr/src/linux-headers-4.12.0-1-amd64 2 compile the same module for the Redpitaya by linking with the kernel sources found in buildroot – transfer and execute on the Zynq 3 add communication functionsopen, close – create the /dev entry point with mknod and test from shell and a C program 4 add IOCTL function and call from a C program using ioctl() Use conditional compilation x86 v.s ARM : #ifdef __ARMEL__5 5J.-M Friedt, Modification des appels syst`emesdu noyau Linux : manipulation de la m´emoiredu noyau, GNU/Linux Magazine Hors S´erie87 (Nov.-D´ec2016) 20 / 24 Embedded systems 2/7 J.-M Friedt D´etournementdes appels syst`eme Introduction $ strace mkdir toto Virtual memory access [...] Kernel module basics mkdir("toto", 0777) = 0 Using the kernel: 6 timers • Un appel syst`eme est une m´ethode fournie par le noyau pour Conclusion & lab acc´ederaux ressources g´er´eespar le syst`eme(norme POSIX : open, session close, read, write, mkdir ...) • un programme utilisateur fait appel `aun appel syst`eme... • ... qui correspond `aune fonction en espace noyau. • Une table des appels syst`emefait la correspondance entre les deux. ⇒ manipuler les appels syst`emesconsiste en 1 identifier l’emplacement en m´emoirede cette table de correspondance 2 modifier l’adresse de la fonction appel´eevers notre fonction ou ... 3 ... modifier le contenu de la m´emoirecontenant l’adresse de destination. 6liste des appels syst`emesPOSIX dans /usr/include/x86 64-linux-gnu/asm/unistd 64.h 21 / 24 Embedded systems 2/7 J.-M Friedt D´etournementdes appels syst`eme Introduction Virtual memory access Attaque ´evidente contre lesquelles les protections sont Kernel module • interdiction d’´ecriredans les pages m´emoirecontenant la table des basics appels syst`eme Using the kernel: timers /boot/System.map-4.6.0-1-amd64: ffffffff816001e0 R sys_call_table Conclusion & lab /proc/kallsyms : ffffffff816001e0 R sys_call_table session • ne pas exporter les symboles des appels syst`emes(port´eedes fonctions : static dans le fichier, EXPORT SYMBOL) Question : quelles sont les fonctions export´eespar le noyau vers les modules ? linux-4.4.2$ grep -r EXPORT_SYMBOL * | grep \(sys_ [...] fs/open.c:EXPORT_SYMBOL(sys_close); kernel/time/time.c:EXPORT_SYMBOL(sys_tz); Seul sys close() est export´e 22 / 24 Embedded systems 2/7 J.-M Friedt D´etournementdes appels syst`eme Introduction #include Virtual memory #include Kernel module s t a t i c unsigned long ∗ c h e r c h e t a b l e (void) basics { u n s i g n e d long int offset = PAGE OFFSET ; // debut kernel Using the kernel: u n s i g n e d long ∗ s c t ; timers printk (KERN INFO "PAGE_OFFSET=%lx\n" ,PAGE OFFSET) ; Conclusion & lab session w h i l e (offset < ULLONG MAX) // recherche toute la mem { s c t = (unsigned long ∗) o f f s e t ; i f( ∗ s c t == (unsigned long)&sys c l o s e ) //@ sys c l o s e r e t u r n sct; // ona trouve’l’@ de debut o f f s e t += sizeof(void ∗); } r e t u r n NULL ; } [...] [100266.370251] PAGE_OFFSET=ffff880000000000 [100266.433568] table: ffff8800016001f8 [100266.433570] NR close: 3 [100266.433571] NR mkdir: 83 => ffffffff812020d0 coh´erent(16001f8-24=16001e0 car sys close est appel 3) avec /boot/System.map-4.6.0-1-amd64: ffffffff816001e0 R sys_call_table /proc/kallsyms : ffffffff816001e0 R sys_call_table 23 / 24 Embedded systems 2/7 J.-M Friedt D´etournementdes appels syst`eme Introduction Pointeur de fonction Virtual memory #include Conclusion & lab session a s m l i n k a g e l o n g mon mkdir (const char u s e r ∗pnam , int mode) { printk (KERN INFO "intercept:%s:%x\n" , pnam,mode); r e t u r n 0; } s t a t i c int i n i t m o d u l e s t a r t (void) { a d d r table = cherche t a b l e ( ) ; r e f s y s m k d i r =(void ∗) a d d r t a b l e [ NR mkdir− NR c l o s e ] ; a d d r t a b l e [ NR mkdir− NR c l o s e ]=(unsigned long) → ,→mon mkdir ; r e t u r n(0); } $ mkdir toto [103669.045129] intercept: toto:1ff Ne fonctionne plus de puis le noyau 4.17.0 qui n’exporte plus sys close, cf https://github.com/f0rb1dd3n/Reptile 24 / 24