BEFN$?FN KSplice XeaXg\glebk#g_fkfZXj\%Zfd 8ggcp`e^lg[Xk\jkfXeXZk`m\b\ie\cn`k_Bjgc`Z\ G@<:<F=:8B< Uptime is often just as important as updates. But doesn’t a kernel patch which a developer has simply mis- counted a parameter such as an array require a reboot? Ksplice lets you have your cake and eat it too. boundary. Bugs of this type are very easy to resolve. For example, Listing 1 BY NILS MAGNUS shows a patch for the prctl() system call, which resolves the vulnerability listed as any administrators don’t relish Jeffrey Brian Arnold from the Massa- CVE-2006-2451. The problem, for which the thought of installing a new chusetts Institute of Technology estab- exploits exist, takes just one line of code Dkernel, so most distributions lished in a survey [1] that patches were to fix. come with package management and very simple in most cases. Eighty per- Arnold developed the Ksplice package installation tools that simplify the task. cent of them comprised fewer than 15 [2] with the idea of patching the active After you load and install the new kernel lines of code, and more than half were kernel directly, without the need for a re- and register it with the bootloader, the little more than one-liners. Problems are boot. The program just needs the source reboot should take just a couple of min- often caused by “off-by-one errors,” in code for the current kernel, the configu- utes. But a couple of minutes of down- time is too long for some applications. Listing 1: Patch for CVE-2006-2451 If you manage a system used for num- 01 diff --git a/kernel/sys.c b/kernel/sys.c ber crunching, such as in weather simu- 02 lations, or a system that has to manage 03 --- a/kernel/sys.c a large number of network connections, such as a phone server or online shop, 04 +++ b/kernel/sys.c you probably would prefer to avoid a re- 05 @@ -1991,7 +1991,7 @@ asmlinkage long sys_prctl(int option, unsigned boot. long arg2, unsigned long arg3, On the other hand, administrators are 06 case PR_SET_DUMPABLE: also responsible for the security of the 07 - if (arg2 < 0 || arg2 > 2) { systems they manage. Surveys have re- 08 + if (arg2 < 0 || arg2 > 1) { vealed that developers discover a Linux 09 error = -EINVAL; kernel bug every three weeks on aver- 10 break; age. The open development model means that patches appear often. 11 } 52 ISSUE 95 OCTOBER 2008 052-054_ksplice.indd 52 13.08.2008 16:14:40 Uhr KSplice BEFN$?FN JpdYfckXYc\ i\jfcm\jjkXikX[[i\jj Zfekifcj B\ie\cZfe]`^liXk`fe JfliZ\Zf[\f] FYa\ZkZf[\ Bjgc`Z\df[`]`\[ J\c\Zki\c\mXek XZk`m\b\ie\c gfjk :i\Xk\df[lc\j 9l`c[\em`ifed\ek :fdgXi`jfeXk ]leZk`fejXe[ `ek_\kXiXiZ_`m\ fYa\Zkc\m\c ZXcZlcXk\k_\`i GXkZ_n`k_ FYa\ZkZf[\ jkXikX[[i\jj\j j\Zli`kplg[Xk\ gi\ G_Xj\( G_Xj\) G_Xj\* G_Xj\+ =`^li\(1Kfgi\gXi\Z_Xe^\jkfk_\ilee`e^b\ie\c#Bjgc`Z\Yl`c[jknfb\ie\cki\\jG_Xj\( #XjZ\ikX`ej[`]]\i\eZ\jXkfYa\ZkZf[\c\m\cG_Xj\ ) #fgk`d`q\jk_\dG_Xj\* #Xe[Yle[c\jk_\dkfZi\Xk\Xe\ndf[lc\G_Xj\+ % ration data, and the symbol table. The thread is currently running one of the If the program were to use a different best thing is that you do not even need functions that needs to be replaced. To compiler to create the pre kernel, it could to prepare the running system to use avoid this, the program calls stop_ma- lead to incorrect assumptions about the Ksplice. The program can modify any chine_run() to stop the thread execution running system. Ksplice relies on sub- kernel as of version 2.6.8. because the function creates a high-pri- stantial logic to analyze the symbol ta- Linux distributions offer the kernel ority process for each CPU. The Ksplice bles, which many distributions store in source code, or, if you built your own module now checks to see whether the the /boot/Symbol.map file. On top of kernel, you will find it in /usr/src. Distri- change candidate contains threads. If it this, it builds the two kernels with a butions typically store the two files con- does, the module waits for a while and number of compiler options that assign fig and System.map in /boot. On top of then retries. This approach will not work a separate ELF text segment to each this, Ksplice obviously needs a patch file for some functions, such as the sched- function to make it easier to identify or one or multiple files with changes. uler, because the scheduler will always modified, relative jumps. The program builds two new kernels: pre be doing something. In that case, Ksplice Because it handles code like a black designates a version of the current sys- gives up; in all other cases, it installs the box, Ksplice can’t detect changes in data tem, and post designates the kernel after jump addresses. From now on, the Linux structures. For example, if a patch adds a the updates. kernel executes the patched version of new attribute to a data structure or the functions. changes its layout, unpredictable issues Jgfkk_\;`]]\i\eZ\ will occur. Trampoline jumps will typi- After building both kernels, Ksplice @ee\iMXcl\j cally pick up function pointers, but there looks for differences in the object code Ksplice has to find the right functions is no guarantee Ksplice will handle more (see Figure 1). If Ksplice were to analyze and vector addresses in relocatable code. complex pointer arithmetic or “creative” the source code directly, it would need The program is capable of detecting typecasting gracefully in each case. to emulate all of the compiler’s deci- changes in relative jump addresses sions, which is far too complicated for where the function itself is not affected F]]C`d`kj the task in hand. For this reason, Ksplice by a modification, thanks to the new In the documentation, the author em- uses the GNU BFD library [3] to search length of the patched function. The ker- phasizes that he mainly designed the for functions in the object code that have nel typically enters functions imple- tool for minor security patches and that changed in the new kernel. Ksplice then mented in C at the start only. In contrast, it is the system administrator’s own adds the new code to modules and in- the program has to search for the vector responsibility to read, understand, and serts trampoline jumps at the start of the in the case of assembler code. evaluate the patch before applying it. original functions that ]leZk`fe then point to the new ver- bjgc`Z\$Zi\Xk\ Gi`dXipdf[lc\ ?\cg\idf[lc\j sions. When the adminis- kiXdgfc`e\aldg Zi\Xk\j Jkfgb\ie\c :XcZlcXk\kiXdgfc`e\j k_i\X[j trator finally enables the nX`kXe[ i\kip fc[Zf[\ changes, Ksplice loads Gi`dXip ?\cg\i :f[\ df[lc\j df[lc\j p\j two kernel modules on jk`ccXZk`m\6 the running system that ef ]leZk`fe df[`]`\[ kXiXiZ_`m\ then perform the modifi- ]leZk`fej @ejkXcce\nZf[\Xe[kiXdgfc`e\j `ejkXccj cations (see Figure 2). e\nZf[\ cfX[\[Yp One critical issue is tim- C`elob\ie\c cfX[j cfX[j ing as to when Ksplice is bjgc`Z\$Xggcp allowed to install the trampoline jumps. Trou- =`^li\)1Kf\eXYc\k_\e\n]leZk`fej#Bjgc`Z\Xkk\dgkjkfni`k\kiXdgfc`e\aldgjXkk_\jkXikf]k_\fc[]leZ$ ble occurs if a kernel k`fej%Kf[fjf#k_\gif^iXd]`ijkjkfgjk_\gifZ\jj\jXe[dXb\jjli\k_Xkk_\Zf[\[f\jefklj\k_i\X[j% OCTOBER 2008 ISSUE 95 53 052-054_ksplice.indd 53 13.08.2008 16:14:42 Uhr BEFN$?FN KSplice ging the active kernel. If you just want to add a couple of printk() calls at various points to view data structures that are otherwise difficult to access, Ksplice gives you a simple approach to injecting them into a running system. However, this approach does not lend itself to more complex applications, for which dynamically loadable modules, Kprobes, or Systemtap are more useful. GXk\ek\[8ggifXZ_6 Developers have pointed out that Micro- soft posted a patent application with the =`^li\*1Kfgi\gXi\Bjgc`Z\#k_\X[d`e`jkiXkfikpg\jbjgc`Z\$Zi\Xk\`ek_\b\ie\cjfliZ\ US Patent Office (USPO) in December [`i\ZkfipXe[jg\Z`]`\jk_\gXkZ_kpg\%Bjgc`Z\k_\eYl`c[jk_\fc[Xe[e\nb\ie\cjXe[ 2002 titled “Patching of In-Use Functions Yle[c\jk_\Z_Xe^\j`ekfXelg[Xk\df[lc\% on a Running Computer System.” USPO had refused the application, and Micro- In other words, you need a great deal of ture is that the administrator can specify soft had appealed and posted a whole kernel expertise to apply the tool; other- a change in diff format with the --patch bunch of additional applications, includ- wise, the effect might be far more de- option or specify a file with the --diffext ing one for Efficient Patching (USPO ref- structive than a bit of reboot downtime. option in which the changes have al- erence 20050257208). Because Ksplice cannot make seman- ready been completed. On top of this, In response to this, half a dozen devel- tic changes to a running kernel, the ad- the program needs a ksplice subdirectory opers piped up in various forums point- ministrator’s pipe dream of measuring in the kernel tree, where the administra- ing out that this technology was public uptime in years is just that because most tor stores both the kernel configuration knowledge on various platforms from changes between one kernel release and and the symbol table (see Figure 3).