Buffer Overflow Attack and Prevention for Embedded Systems
Total Page:16
File Type:pdf, Size:1020Kb
Buffer Overflow Attack and Prevention for Embedded Systems A Thesis submitted to the Graduate School of The University of Cincinnati in partial fulfillment of the requirements for the degree of Master of Science in the Department of Electrical and Computer Engineering of the College of Engineering July 2011 by Amjad Basha Sikiligiri B.Tech, National Institute of Technology Karnataka, Surathkal 2007 Committee Chair: Dr. Carla Purdy ABSTRACT Embedded systems today play a significant role in all aspects of our lives ranging from critical medical applications to multi-purpose handheld devices to simple room temperature controls. Unfortunately, due to their ubiquity and characteristic features, embedded systems are prone to various security attacks. Software based security attacks, which target security loopholes in operating system and application software, are the most common security attacks because of their relatively easy and cost effective implementation. Hence it‟s important for embedded system designers and application developers to have knowledge about existing security attacks so as to avoid them in their design. We survey various embedded system security attacks and present a detailed description for a class of software based security attacks, buffer overflow attack. We demonstrate a stack based buffer overflow attack using the Altera Nios II softcore processor and the Micrium MicroC/OS II RTOS kernel. We also present a method to prevent such an attack for this specific system. This method can be modified to apply to a wide range of embedded systems products ii iii TABLE OF CONTENTS 1. INTRODUCTION...................................................................................................................1 1.1 Motivation............................................................................................................................1 1.2 Thesis Goals .........................................................................................................................2 1.3 Outline .................................................................................................................................2 2. BACKGROUND ....................................................................................................................3 2.1 Embedded systems security ................................................................................................3 2.2 Buffer overflow attack.........................................................................................................7 2.2.1 Function calling .......................................................................................................................... 7 2.2.2 Vulnerabilities in a C program................................................................................................... 11 2.2.3 The attack ................................................................................................................................. 13 2.2.4 Related attacks .......................................................................................................................... 14 2.2.5 Real world example: Apache htpasswd.c ................................................................................... 14 2.2.6 Countermeasures for buffer overflow attacks ............................................................................. 16 2.3 Altera Nios II processor .................................................................................................... 17 2.4 Micrium MicroC/OS II Kernel ......................................................................................... 18 2.5 System setup ...................................................................................................................... 18 3. PROCEDURES ..................................................................................................................... 20 3.1 Find an application with buffer overflow vulnerabilities ................................................ 20 3.2 Find the effective buffer length ......................................................................................... 21 3.3 Find the address of the buffer (on stack) .......................................................................... 21 3.4 Develop and inject the exploit string (code) ..................................................................... 25 3.4.1 Choose a different set of instructions and data types .................................................................. 28 3.4.2 Choose appropriate registers...................................................................................................... 28 3.4.3 Replace a single byte with a null byte during run time ............................................................... 29 3.4.4 Encode and decode the exploit code .......................................................................................... 30 iv 3.5 Code injection attack countermeasure ............................................................................. 34 3.5.1 Countermeasure implemented ................................................................................................... 35 3.5.2 Other approaches considered ..................................................................................................... 37 3.6 Buffer overflow in complex programs .............................................................................. 39 3.7 Conclusion ......................................................................................................................... 39 4. RESULTS ............................................................................................................................. 40 4.1 Initial stack address........................................................................................................... 40 4.2 The vulnerable program ................................................................................................... 42 4.3 The attack .......................................................................................................................... 44 4.4 The null byte problem ....................................................................................................... 50 4.5 Prevention .......................................................................................................................... 54 4.6 Summary ........................................................................................................................... 56 5. CONCLUSIONS AND FUTURE WORK ............................................................................. 57 5.1 Conclusions ........................................................................................................................ 57 5.2 Future Work ...................................................................................................................... 57 REFERENCES ......................................................................................................................... 59 APPENDIX A ........................................................................................................................... 67 Tutorial for Buffer Overflow Attack and Prevention in Embedded Systems ....................... 67 A.1 Nios II system.................................................................................................................... 67 A.2 Attack steps ....................................................................................................................... 72 A.2.1 Approximate address of the buffer ............................................................................................ 72 A.2.2 The vulnerable application ....................................................................................................... 74 A.2.3 Is the application buffer overflow vulnerable? .......................................................................... 74 A.2.4 Effective buffer length.............................................................................................................. 75 A.2.5 Develop and inject the exploit string ......................................................................................... 75 v A.3 Prevention ......................................................................................................................... 83 vi LIST OF FIGURES Figure 2.1: Function call in a C program using MicroC/OS II task …………….………………. 7 Figure 2.2: Disassembled C program ……………..…………………………………………….. 8 Figure 2.3: Prologue and epilogue for the function in Figure 2.1 [1] ………………………….. 10 Figure 2.4: Complete stack region after function call for the function in Figure 2.1 ………….. 11 Figure 2.5: Buffer overflow ..................................................................................................…... 13 Figure 2.6: Vulnerable part of code in htpasswd.c utility for Apache server [46, 5]……………15 Figure 3.1: Initial stack address ………………..………………………………………………. 22 Figure 3.2: Buffer overflow without NOP Sled …………………………………………………23 Figure 3.3: Buffer overflow with NOP Sled …………………………………………………… 24 Figure 3.4: C Program to spawn a new shell [2, 54]………….………………………………... 26 Figure 3.5: Nios II „jmpi 0x20004e‟ equivalent instructions without null bytes ……………… 28 Figure 3.6: Exploit string structure in the stack frame .……………………………………....... 29 Figure 3.7: Exploit string structure with instructions appended .……………………………… 30 Figure 3.8: Instructions that replace non-null byte