BEFN$?FN XeaXg\glebk#g_fkfZXj\%Zfd 8ggcp`e^lg[Xk\jkfXeXZk`m\b\ie\n`k_Bjgc`Z\ G@<:<F=:8B<

Uptime is often just as important as updates. But doesn’t a kernel which a developer has simply mis- counted a parameter such as an array require a reboot? Ksplice lets you have your cake and eat it too. boundary. Bugs of this type are very easy to resolve. For example, Listing 1 BY NILS MAGNUS shows a patch for the prctl() system call, which resolves the vulnerability listed as any administrators don’t relish Jeffrey Brian Arnold from the Massa- CVE-2006-2451. The problem, for which the thought of installing a new chusetts Institute of Technology estab- exploits exist, takes just one line of code Dkernel, so most distributions lished in a survey [1] that patches were to fix. come with package management and very simple in most cases. Eighty per- Arnold developed the Ksplice package installation tools that simplify the task. cent of them comprised fewer than 15 [2] with the idea of patching the active After you load and install the new kernel lines of code, and more than half were kernel directly, without the need for a re- and register it with the bootloader, the little more than one-liners. Problems are boot. The program just needs the source reboot should take just a couple of min- often caused by “off-by-one errors,” in code for the current kernel, the configu- utes. But a couple of minutes of down- time is too long for some applications. Listing 1: Patch for CVE-2006-2451 If you manage a system used for num- 01 --git a/kernel/sys.c b/kernel/sys.c ber crunching, such as in weather simu- 02 lations, or a system that has to manage 03 --- a/kernel/sys.c a large number of network connections, such as a phone server or online shop, 04 +++ b/kernel/sys.c you probably would prefer to avoid a re- 05 @@ -1991,7 +1991,7 @@ asmlinkage long sys_prctl(int option, unsigned boot. long arg2, unsigned long arg3, On the other hand, administrators are 06 case PR_SET_DUMPABLE: also responsible for the security of the 07 - if (arg2 < 0 || arg2 > 2) { systems they manage. Surveys have re- 08 + if (arg2 < 0 || arg2 > 1) { vealed that developers discover a 09 error = -EINVAL; kernel bug every three weeks on aver- 10 break; age. The development model means that patches appear often. 11 }

52 ISSUE 95 OCTOBER 2008

052-054_ksplice.indd 52 13.08.2008 16:14:40 Uhr KSplice BEFN$?FN

JpdYfckXYc\ i\jfcm\jjkXikX[[i\jj Zfekifcj B\ie\cZfe]`^liXk`fe

JfliZ\Zf[\f] FYa\ZkZf[\ Bjgc`Z\df[`]`\[ J\c\Zki\c\mXek XZk`m\b\ie\c gfjk :i\Xk\df[lc\j 9l`c[\em`ifed\ek :fdgXi`jfeXk ]leZk`fejXe[ `ek_\kXiXiZ_`m\ fYa\Zkc\m\c ZXcZlcXk\k_\`i GXkZ_n`k_ FYa\ZkZf[\ jkXikX[[i\jj\j j\Zli`kplg[Xk\ gi\

G_Xj\( G_Xj\) G_Xj\* G_Xj\+

=`^li\(1Kfgi\gXi\Z_Xe^\jkfk_\ilee`e^b\ie\c#Bjgc`Z\Yl`c[jknfb\ie\cki\\j G_Xj\( #XjZ\ikX`ej[`]]\i\eZ\jXkfYa\ZkZf[\c\m\c G_Xj\ ) #fgk`d`q\jk_\d G_Xj\* #Xe[Yle[c\jk_\dkfZi\Xk\Xe\ndf[lc\ G_Xj\+ %

ration data, and the symbol table. The is currently running one of the If the program were to use a different best thing is that you do not even need functions that needs to be replaced. To compiler to create the pre kernel, it could to prepare the running system to use avoid this, the program calls stop_ma- lead to incorrect assumptions about the Ksplice. The program can modify any chine_run() to stop the thread execution running system. Ksplice relies on sub- kernel as of version 2.6.8. because the function creates a high-pri- stantial logic to analyze the symbol ta- Linux distributions offer the kernel ority for each CPU. The Ksplice bles, which many distributions store in source code, or, if you built your own module now checks to see whether the the /boot/Symbol.map file. On top of kernel, you will find it in /usr/src. Distri- change candidate contains threads. If it this, it builds the two kernels with a butions typically store the two files con- does, the module waits for a while and number of compiler options that assign fig and System.map in /boot. On top of then retries. This approach will not work a separate ELF text segment to each this, Ksplice obviously needs a patch file for some functions, such as the sched- function to make it easier to identify or one or multiple files with changes. uler, because the scheduler will always modified, relative jumps. The program builds two new kernels: pre be doing something. In that case, Ksplice Because it handles code like a black designates a version of the current sys- gives up; in all other cases, it installs the box, Ksplice can’t detect changes in data tem, and post designates the kernel after jump addresses. From now on, the Linux structures. For example, if a patch adds a the updates. kernel executes the patched version of new attribute to a data structure or the functions. changes its layout, unpredictable issues Jgfkk_\;`]]\i\eZ\ will occur. Trampoline jumps will typi- After building both kernels, Ksplice @ee\iMXcl\j cally pick up function pointers, but there looks for differences in the object code Ksplice has to find the right functions is no guarantee Ksplice will handle more (see Figure 1). If Ksplice were to analyze and vector addresses in relocatable code. complex pointer arithmetic or “creative” the source code directly, it would need The program is capable of detecting typecasting gracefully in each case. to emulate all of the compiler’s deci- changes in relative jump addresses sions, which is far too complicated for where the function itself is not affected F]]C`d`kj the task in hand. For this reason, Ksplice by a modification, thanks to the new In the documentation, the author em- uses the GNU BFD library [3] to search length of the patched function. The ker- phasizes that he mainly designed the for functions in the object code that have nel typically enters functions imple- tool for minor security patches and that changed in the new kernel. Ksplice then mented in C at the start only. In contrast, it is the system administrator’s own adds the new code to modules and in- the program has to search for the vector responsibility to , understand, and serts trampoline jumps at the start of the in the case of assembler code. evaluate the patch before applying it. original functions that ]leZk`fe then point to the new ver- bjgc`Z\$Zi\Xk\ Gi`dXipdf[lc\ ?\cg\idf[lc\j sions. When the adminis- kiXdgfc`e\aldg Zi\Xk\j Jkfgb\ie\c :XcZlcXk\kiXdgfc`e\j k_i\X[j trator finally enables the nX`kXe[ i\kip fc[Zf[\ changes, Ksplice loads Gi`dXip ?\cg\i :f[\ df[lc\j df[lc\j p\j two kernel modules on jk`ccXZk`m\6

the running system that ef ]leZk`fe df[`]`\[ kXiXiZ_`m\ then perform the modifi- ]leZk`fej @ejkXcce\nZf[\Xe[kiXdgfc`e\j `ejkXccj cations (see Figure 2). e\nZf[\ cfX[\[Yp One critical issue is tim- C`elob\ie\c cfX[j cfX[j ing as to when Ksplice is bjgc`Z\$Xggcp allowed to install the trampoline jumps. Trou- =`^li\)1Kf\eXYc\k_\e\n]leZk`fej#Bjgc`Z\Xkk\dgkjkfni`k\kiXdgfc`e\aldgjXkk_\jkXikf]k_\fc[]leZ$ ble occurs if a kernel k`fej%Kf[fjf#k_\gif^iXd]`ijkjkfgjk_\gifZ\jj\jXe[dXb\jjli\k_Xkk_\Zf[\[f\jefklj\k_i\X[j%

OCTOBER 2008 ISSUE 95 53

052-054_ksplice.indd 53 13.08.2008 16:14:42 Uhr BEFN$?FN KSplice

ging the active kernel. If you just want to add a couple of printk() calls at various points to view data structures that are otherwise difficult to access, Ksplice gives you a simple approach to injecting them into a running system. However, this approach does not lend itself to more complex applications, for which dynamically loadable modules, Kprobes, or Systemtap are more useful. GXk\ek\[8ggifXZ_6 Developers have pointed out that Micro- soft posted a patent application with the =`^li\*1Kfgi\gXi\Bjgc`Z\#k_\X[d`e`jkiXkfikpg\jbjgc`Z\$Zi\Xk\`ek_\b\ie\cjfliZ\ US Patent Office (USPO) in December [`i\ZkfipXe[jg\Z`]`\jk_\gXkZ_kpg\%Bjgc`Z\k_\eYl`c[jk_\fc[Xe[e\nb\ie\cjXe[ 2002 titled “Patching of In-Use Functions Yle[c\jk_\Z_Xe^\j`ekfXelg[Xk\df[lc\% on a Running Computer System.” USPO had refused the application, and Micro- In other words, you need a great deal of ture is that the administrator can specify soft had appealed and posted a whole kernel expertise to apply the tool; other- a change in diff format with the --patch bunch of additional applications, includ- wise, the effect might be far more de- option or specify a file with the --diffext ing one for Efficient Patching (USPO ref- structive than a bit of reboot downtime. option in which the changes have al- erence 20050257208). Because Ksplice cannot make seman- ready been completed. On top of this, In response to this, half a dozen devel- tic changes to a running kernel, the ad- the program needs a ksplice subdirectory opers piped up in various forums point- ministrator’s pipe dream of measuring in the kernel tree, where the administra- ing out that this technology was public uptime in years is just that because most tor stores both the kernel configuration knowledge on various platforms from changes between one kernel release and and the symbol table (see Figure 3). PDP-11 through a state-of-the-art PC long the next add some new functionality. Depending on what kind of system before the software patent application you are using, the first phase can take was filed. Gif[lZk`feLj\ a while because Ksplice needs to build The current 0.8.6 version is available as two complete kernels: one in ksplice/pre :c\m\i?\cg\i]fiJfd\ a tarball with prebuilt binaries or as a and one in ksplice/post. After doing so, JZ\eXi`fj source code archive under the GPLv2. the program searches for differences and Ksplice includes clever mechanisms to Distribution packages do not exist as of merges the results to create two kernel support hot kernel updates at the binary this writing. If you build the tool from modules. level. Despite intelligent code pushing the source code, you will also need the By calling ksplice-apply, you can apply and vector analysis, administrators BFD library, which you can retrieve on the hotfix. The program first loads a should be aware that they do need to or a derivative such as module that takes care of trampoline check manually on a case-by-case basis with the following command: management, then waits for the right to determine whether the tool is useful. moment. When the moment occurs, Ksplice is useful for simple cases, but it sudo aptitude install 5 Ksplice loads the changes into the ker- is no replacement for a hardware failover binutils-dev nel, executes them, then removes itself solution in situations that require high to save memory. system availability. p Prominent developers, including Andi Kleen, have proposed adding Ksplice to GXkZ_DXeX^\d\ek INFO the official kernel as an upstream exten- Ksplice can also change a patched ker- [1] “Ksplice: An Automatic System for sion. Kleen hopes this step would mean nel. To do so, the patches from the first Rebootless Security permanent support for the project, phase must reside in the source code’s Updates” by Jeffrey Brian Arnold, which would, in the long term, lead to pre tree. ksplice-create and ksplice-apply Massachusetts Institute of Technol- an incremental compiler [4]. This would take the trampolines into consideration ogy, http:// web. mit. edu/ ksplice/ doc/ remove the need for developers to com- and modify them correspondingly. The ksplice. pdf pletely rebuild the kernel to test and same mechanism makes it possible to [2] Ksplice download and installation: modify patches. undo changes by calling ksplice-undo http:// web. mit. edu/ ksplice/ The program itself consists of four Perl because the system “remembers” the [3] GNU binutils and BFD library: scripts that call a number of tools writ- vector addresses. ksplice-view shows the http:// sourceware. org/ binutils/ ten in C to analyze the object code. changes performed by Ksplice. [4] Announcement and discussion on ksplice-create only needs a path to the On his website, Ksplice author Jeffrey the Linux Kernel mailing list: directory with the running kernel and Brian Arnold shows another potential http:// thread. gmane. org/ gmane. details of the patch. One practical fea- application scenario for the tool: debug- linux. kernel/ 669951

54 ISSUE 95 OCTOBER 2008

052-054_ksplice.indd 54 13.08.2008 16:14:43 Uhr