Paper ID #30573

CODE HARDENING: DEVELOPMENT OF A REVERSE EN- GINEERING PROJECT

Mr. Zachary Michael Steudel, Baylor University

Zachary Steudel is a Junior Computer Science student at Baylor University working as a Teaching Assis- tant under Ms. Cynthia C. Fry. As part of the Teaching Assistant role, Zachary designed and created the group project for the Computer Systems course. Zachary Steudel worked as a Software Developer Intern at Amazon in the Summer of 2019 and plans to join Microsoft as a Software Intern in the Summer of 2020. Ms. Cynthia C. Fry, Baylor University

CYNTHIA C. FRY is currently a Senior Lecturer of Computer Science at Baylor University. She worked at NASA’s Marshall Space Flight Center as a Senior Project Engineer, a Crew Training Manager, and the Science Operations Director for STS-46. She was an Engineering Duty Officer in the U.S. Navy (IRR), and worked with the Naval Maritime Intelligence Center as a Scientific/ Analyst. She was the owner and chief systems engineer for Services (SES), a computer systems , development, and consultation firm. She joined the faculty of the School of Engineering and Computer Science at Baylor University in 1997, where she teaches a variety of engineering and computer science classes, she is the Faculty Advisor for the Women in Computer Science (WiCS), the Director of the Computer Science Fellows program, and is a KEEN Fellow. She has authored and co- authored over fifty peer-reviewed papers.

c American Society for Engineering Education, 2020

Code Hardening: Development of a Reverse Project

Abstract

In CSI 2334, “Introduction to Computer Systems” (CompSys), at Baylor University, we introduce a group project to the students whose purpose is to simulate a team project on the job. Group projects are used very frequently to provide a similar learning environment which capitalizes on the benefits of peer-to-peer instruction, or cooperative learning. In this group project, students are presented with a challenge. A piece of executable code has been found on an older server, and the student teams must determine what the code is designed to do; and, in particular, whether the code is benign or malicious in nature.

In order to simulate this scenario a software system is developed and then hardened. The system developed is generally a game that has some malicious content, which is then obfuscated before the executable is presented to the student teams. The objective for the student teams is to research methods by which the binary file can be “read” without executing it, and to modify the behavior of the executable file, depending on the purpose of the code.

This paper will introduce methods of software hardening and obfuscation; and will discuss the design of the project, the implementation of the design, code obfuscation techniques used, and which obfuscation techniques were used to produce the mystery executable presented to the class as their class project.

Introduction

Group projects in engineering and computer science coursework are a critical part of the education . Not only do they enforce the concepts being taught, they also provide an environment in which essential professional skills (aka, soft skills) can be understood, culminating in a synergistic learning experience. The value of such group learning has been well documented in both engineering and computer science courses [1]-[5], and is a cornerstone to the education process according to ABET EAC and CAC [6], [7], the National Academies of Science, Engineering, and Medicine [8], and industry partners [9].

Before introducing the methods and challenges related to the obfuscation a software system for a classroom project, it is important to understand the depth of knowledge available in the field and the maturity of the discipline at this point.

Although one of the most important applications of binary-level reverse engineering is dealing with [10], it is certainly not the only application. It is often the case with older software systems that code is not adequately documented, and the original developers no longer available for consultation [11], in which case the binary executable would be reverse engineered for behavior and interface information.

Code obfuscation or hardening may also be employed intentionally. It might be that the digital rights of pieces of information [12] (e.g., cryptographic keys and protocols) must be protected. A company or governmental entity may need to protect ; perhaps they have developed an algorithm that is more efficient than others/competitors, and they need to secure it.

Code-level hardening changes can be defined as “changes in the in a way that prevents vulnerabilities without altering the design” [13]. Code hardening is a common practice in major software development firms, with many teams carving out sprints1 to harden new code by refactoring and removing unnecessary dependencies [14]. Code-level hardening is also employed to make it difficult for attackers from easily discovering vulnerabilities in the code, as well as to protect the program owner’s intellectual property [15].

Hardening can also refer to practices that make the original source code of an application more difficult to understand. This is generally defined as code obfuscation. The practice of code obfuscation in modern software engineering dates back to the 1980’s, with small competitions held to transform simple C code into confusing, abstract puzzles difficult for humans to parse and understand [16].

Since the 1980’s, literature on code-level obfuscation has been consistent but generally sparse. This field of research is relatively small, with no more than three or four papers published each year since the early 1990’s. With the growth of cloud-computing services that require paid subscriptions and API keys embedded in the code, the necessity of obfuscation techniques to halt reverse software engineering attempts has grown. Without sufficient code obfuscation, reverse engineers can easily disassemble or decompile code to gain access to API keys and compromise an application’s access to cloud databases or other cloud compute resources [17].

Introduction to the Challenge

Reversing software group projects have been a part of the CompSys course since the Fall 2012 semester. As a result, nearly 600 students have participated in one of the various group projects. The projects are different every semester, responding to changes in the available reversing software.

CompSys is a sophomore-level course in which the relationship between hardware and software is investigated at a low level2. As part of the course requirements, students in CompSys are paired up for the class group project a little more than halfway through the semester. Each pair of students is presented with a Windows executable file with no information other than that this file has unknown origins and unknown functionality. Given this, students are tasked with learning the effects and purpose of the executable and to quarantine any malicious elements while also further developing the functionality of the program.

1 A sprint is generally defined as a short 2-4 week iteration in a team’s software development process. Most agile development processes utilize sprints to develop software. 2 All materials used in this course in the Fall 2019 semester can be found here: https://classnotes.ecs.baylor.edu/wiki/CSI_2334_Fry_Fall_2019 (username: CSI2334F19, password: FrySteudelASEE2020). This includes the CSI 2334 Course syllabus, the CSI 2334 Course calendar, and the CSI 2334 Fall 2019 Project (among other items).

The CompSys course is primarily focused around low-level design, architecture, and code. This project plays into the goals of the course by mimicking the research and design process of small agile teams in the software engineering industry while also developing the low-level computing skills of the students. Given only an executable file, students must take advantage of open source software to gain access to the binary code and modify it. At the end of the semester, the student groups present their findings to the professor and the teaching assistant in a formal mock final review setting.

The groups are determined through a rigorous questionnaire administered through the Comprehensive Assessment of Team Member Effectiveness system (CATME.org)3. Time is spent in class discussing the importance of working together and learning to value each team member's contributions. The team members must assess each other's effectiveness with both a formative assessment conducted midway through the group project, as well as a summative assessment after the final projects are submitted. Students are given two assignments regarding team building and team dynamics before they begin the formative assessment.

These assessments are graded by evaluating the consideration used by each student in their assessment of their team members. This grade becomes an individual component in the student's overall course grade, and the individual adjustment factor is used as a multiplier in each student's group project grade.

The objectives of this group project are not only technically related, but also related to educational pedagogy: 1. To begin consideration of the project in a project-based manner, from the beginning of the term. 2. To develop and build a team, where what each team member brings to the project is known and valued. 3. To further develop team building skills, including communication between team members, team dynamics, conflict resolution, and presentation skills to various audiences. 4. To help students understand the importance of how code hardening is used to protect intellectual property. 5. To help students understand how the skills learned in the course can be directly applied in the detection of unknown code.

Determination of the Challenge

Before developing the course project for the students, it was first necessary to analyze the learning objectives of the project and potential outcomes. As outlined above, students are strongly encouraged to develop their collaboration and research skills throughout the extent of the challenge. Given this, it was important to design a program that included some layers of complexity that prompt students to begin researching. It was decided early in the design process to write the project’s program in the C programming language, adding one layer of complexity to the project, as students up to this point have mostly worked with the C++ language. Although these languages are

3 The Comprehensive Assessment of Team Member Effectiveness, https://catme.org/login/index

similar, students’ unfamiliarity with some of the lower-level conventions of C already provides one small layer of obfuscation to the executable.

After determining the language to write the program, it was critical to determine the functionality of the project executable. It was quickly decided to make the program into a simple game. Given that students are tasked to make changes to the executable, a game provides many chances to make clever edits. With a game, students can add cheat codes, new modes of play, and even change base- level game functionality in interesting and unique ways. After deliberating over the game to create, it was decided to make Snake in C. Snake is simple; players travel a game board with the arrow keys to eat food and grow larger. The game ends when the player hits a wall or hits their own body. Their goal is to eat as much food as possible before colliding with something, as food adds to the player’s score. Making the project Snake provided many opportunities for changes in the binary code, as students can change how the score is calculated, how losing works, how food is spawned, and much more.

With the core functionality of the project finalized as a Snake game written in C, it was important to add a few more layers of complexity to push students to stretch their research and collaboration skills further. It was decided to wrap the game with a malicious bomb to prevent students from simply running the executable upon retrieving the project. A fork bomb is categorized as a denial of service attack in which “the user repeatedly forks new processes,” absorbing computer resources “mak[ing] the system unusable for others” [19]. To create a fork bomb in a Windows executable created in C, the system() function was utilized in a for-loop.

Figure 1: Restricted fork() bomb As seen in the above code snippet from the program, with each call to the system() function, a command was sent to open ten terminals on the user’s computer. Within seconds of running this, there are hundreds of terminals open on the user’s computer each utilizing system resources and tremendously slowing down new inputs.

To avoid potentially damaging the computers of students, the fork bomb was set to be finite, ending after 1,000 calls to the system() function, opening a total of 10,000 terminals on the user’s computer. This code snippet runs when a user opens the Snake game and presses a key on their keyboard. Most students would find this malicious element if they ran the game and tried to move the snake on the game board. The purpose of this malicious layer in an otherwise normal game is to spur more research on tools and techniques to mitigate the issue. The fork bomb can potentially crash a user’s computer, so with this threat looming, students needed to discover virtual machine tools to run the executable.

At this point in the determination of the challenge, it has been decided that the project is to be a Snake game written in C that is wrapped in a malicious computer-crashing fork bomb. Multiple layers of complexity exist at this point to spur group research and collaboration, however with the rapid iteration and development of modern reverse engineering tools such as Ghidra and IDA, it was

necessary to add one more layer to the challenge. If the executable was to be given to students in its current state, it would only take minutes after loading the file into a disassembler such as IDA to locate the fork bomb’s code and quarantine it. This is where obfuscation techniques such as junk code insertion, rabbit-hole function calls, and stack pointer manipulation have been used. With these obfuscation techniques, students are required to do much more complex in the Assembly and hex code of the executable to determine what code is necessary.

Design, Testing, Obfuscation

With the general layout of the project program finalized, it was important to plan out the implementation process for the game. It was decided to first implement the core game functionality. To verify that core game functionality is correct, multiple exhaustive rounds of the new Snake game were to be played. After ensuring that the base functionality of the game was complete and correct, the hardening and obfuscation was to be included.

As previously mentioned, obfuscation techniques were to be junk code insertion, rabbit-hole function calls, and stack pointer manipulation. Junk code insertion is a common and relatively simple technique in basic code hardening and obfuscation. The goal of this technique is to muddy the source code of an application without changing the functionality of the application. To do this, random variables are allocated and never used and function calls that allocate memory are called upon, with none of these affecting the main gameplay. These insertions are placed randomly throughout the source code, with the goal of prompting students to analyze the source code more closely to decipher functionality of each section of code.

Similar to junk code insertion, rabbit-hole function calls do not affect the gameplay of the program but instead serve to confuse the reverse engineer by de-linearizing the execution flow. Mixed in with the necessary program code for the Snake game, there are rabbit-hole functions that are called that make further function calls that serve no purpose for the game itself. The structure of these rabbit- hole function calls is shown below.

Figure 2: Rabbit-hole Function Call Sequence Again, these useless functions are called amidst necessary program code in order to confuse the natural flow of the code written for the game. Students tasked with reverse engineering the executable file will find it challenging to determine which code is necessary game code and which

code has been placed there to divert their attention.

Finally, stack pointer manipulation is another obfuscation technique used to muddy the execution flow of an otherwise normal program. By using inline Assembly instructions within the C code, the stack pointer can be directly manipulated. This technique is key to challenge reverse engineers using a graph execution view of a program, as stack pointer manipulation will throw off the execution graph. This obfuscation technique further serves to force students to research other techniques and technologies in their reverse engineering journey, as they cannot simply rely on one method of reverse engineering for the whole project.

Agile Development of the Challenge

To develop multiple layers of complexity into the Snake game project, it was important to tackle the program with a plan. When developing the program, a test-driven process was used. With each feature developed within the program, time was spent testing and verifying the correctness of the program before moving on to the next feature. Before adding any hardening or obfuscation, the main Snake game was completed and tested for correctness.

Once it was verified that the base Snake game was correctly working, the hardening and obfuscation began. With each layer of program obfuscation, short manual tests were run to verify that the program functionality had not been changed. The first layer of obfuscation added was junk code insertion. Snippets of memory allocation and useless function calls were injected into the main game code and then tested to ensure no tampering with the game functionality had occurred. After junk code insertion, rabbit-hole function calls were similarly inserted into the main game code. Again, it was subsequently tested that the rabbit-hole function calls caused no effect within the game itself. Finally, all variable and function names were changed to arbitrary garbage names so as to add one final layer of obfuscation.

With the program obfuscation complete, the malicious element of the game was to be added. Within the main game control loop, a function call was added within the if statement that checks whether the user entered a key or not. The called function contains the code to open 10,000 terminals on the user’s computer. To test and verify this feature, the amount of terminals produced by the fork bomb was reduced to 50. Once verified that the fork bomb created the expected number of terminals upon triggering, the number was changed back to 10,000 and the program executable was exported.

Summary and Conclusions

Upon finishing the semester and listening to the final presentations of the students, a retrospective look was taken on the project. There were a few key takeaways from this retrospective. • Reverse software engineering tools have become advanced enough that just a few layers of program obfuscation do not provide a sufficient challenge. With the release of the open-source NSA software Ghidra in April of 2019, reverse engineering has been simplified. Ghidra packages all the typical reverse engineering tools such as disassemblers, , binary editors, and more, allowing many students to rely on Ghidra alone to finish the class project. To encourage the use of more tools and more research, further layers of obfuscation that take advantage of the weaknesses of Ghidra will need to be developed into future class projects.

• Malicious elements of future projects should be deeper and more complex. Using Ghidra and other modern reverse software engineering tools, students were easily able to find the fork bomb within the Snake game and quarantine it with simple binary code editing. In future iterations of the project, multiple malicious elements alongside obfuscation of those malicious elements will be necessary to provide a greater challenge to students.

Overall it was clear that this project provided a relevant outlet for students to strengthen their research and collaboration skills, as well as their understanding of the relationship between hardware and software. Given that most students entered the CompSys course with no knowledge of reverse software engineering, the project successfully simulated a short professional software engineering project that required stream-lined learning and strong communication, based on the students’ experience in the curriculum.

These conclusions apply to the nearly 600 students who have taken this course through Spring 2019. The modifications to how the project is designed and obfuscated, as a result of the Ghidra tool, was seen in the ~60 students who took the course in the Fall 2019 semester, as well as those registered for the Spring 2020 semester. Our conclusions from semester to semester have been much the same, with the students indicating (through their pre-test, formative assessment, and summative assessment) a much wider and deeper appreciation for the importance of understanding the difference between what is written and designed in a high-level language versus what the machine actually executes.

References

[1] C.L. Colbeck, S.E. Campbell, and S.A. Bjorklund, “Grouping in the Dark: What College Students Learn from Group Projects,” The Journal of Higher Education, volume 71, 2000, Issue 1, pp. 60-83, November 2016.

[2] I. Newman, M. Daniels, and X. Faulkner, “Open Ended Group Projects a ‘Tool’ for More Effective Teaching,” in ACE ’03: Proceedings of the fifth Australasian Conference on Computing Education, Volume 20, Australian Computer Society, Inc, Adelaide, Australia, 2003, pp. 95-103.

[3] M. Daniels, S. Cajander, A. Pears and T. Clear, “Engineering Education Research in Practice: Evolving Use of Open Ended Group Projects as a Pedagogical Strategy for Developing Skills in Global Collaboration,” International Journal of Engineering Education, vol. 26, No. 4, pp. 1-12, 2010.

[4] S.E. Scherling, “Designing and Fostering Effective Online Group Projects,” Adult Learning, vol. 22, issue 2, pp. 13-18, March 2011.

[5] L. Johns-Boast, G. Patch, “A Win-Win Situation: Benefits of Industry-Based Group Projects,” in Proceedings of the 2010 AaeE Conference, Sydney, Australia, January 2010, pp. 355-360.

[6] Accreditation Board for Engineering and Technology, Criteria for Accrediting Engineering Programs, 2017-2018, General Criterion 3: Student Outcomes. Available: https://www.abet.org/accreditation/accreditation-criteria/criteria-for-accrediting-engineering- programs-2017-2018/#GC3

[7] Accreditation Board for Engineering and Technology, Criteria for Accrediting Computing Programs, 2017-2018, General Criterion 3: Student Outcomes. Available: https://www.abet.org/accreditation/accreditation-criteria/criteria-for-accrediting-computing- programs-2017-2018/#GC3

[8] National Academy of Engineering Grand Challenges Scholars Program, Five Student competencies. Available: http://www.engineeringchallenges.org/GrandChallengeScholarsProgram.aspx

[9] D. González-Morales, L. M. Moreno de Antonio and J. L. Roda García, "Teaching “soft” skills in Software Engineering," 2011 IEEE Global Engineering Education Conference (EDUCON), Amman, 2011, pp. 630-637.

[10] S. Debray and J. Patel, “Reverse Engineering self-Modifying Code: Unpacker Extraction,” IEEE 17th Working Conference on Reverse Engineering (pp.131-140). October, 2010, Beverly, Massachusetts.

[11] E. Eilam, Reversing: Secrets of Reverse Engineering, Indianapolis, IN: Wiley Publishing, Inc., 2005.

[12] B. Dang, A. Gazet, and E. Bachaalany, Practical Reverse Engineering, Indianapolis, IN: Wiley Publishing, Inc., 2014.

[13] A. Mourad, M. Debbabi and M.A. Laverdière, “Security hardening of open source software,” International Conference on Privacy, Security and Trust: Bridge the Gap Between PST Technologies and Business Services (pp. 2-4). Ontario: ACM.

[14] B. Scarlett, “8 Reasons Why Hardening Sprints Are Truly Necessary.” Sagepath, July 6, 2017. Available: https://www.sagepath.com/blogs/strategy/articles/8-reasons-why-hardening-sprints-are- truly-necessary.

[15] S.K. Udupa, S.K. Debray and M. Madou, “Deobfuscation: Reverse Engineering Obfuscated Code”, IEEE 12th Working Conference on Reverse Engineering (WCRE), Pittsburgh, PA, Nov 7-11, 2005.

[16] H. Xu, Y. Zhou, Y. Kang, and M.R. Lyu. “On Secure and Usable Program Obfuscation: A Survey.” Computing Research Repository abs/1710.01139 (October 3, 2017): 1–2. Available: http://arxiv.org/abs/1710.01139.

[18] “Code Hardening (Obfuscation & ).” Guardsquare, December 6, 2019. Available: https://www.guardsquare.com/en/mobile-application-protection/code-hardening-obfuscation- encryption.

[19] M. Berlot and J. Sang. “Dealing with Process Overload Attacks in .” Information Security Journal: A Global Perspective 17, no. 1 (March 24, 2008): 1–1. Available: https://doi.org/10.1080/19393550801929547.