SECURITY SANDBOXING for PC2: WINDOWS VERSION a Project
Total Page:16
File Type:pdf, Size:1020Kb
SECURITY SANDBOXING FOR PC2: WINDOWS VERSION A Project Presented to the faculty of the Department of Computer Science California State University, Sacramento Submitted in partial satisfaction of the requirements for the degree of MASTER OF SCIENCE in Computer Science by Parita Shah SPRING 2017 SECURITY SANDBOXING FOR PC2: WINDOWS VERSION A Project by Parita Shah Approved by: __________________________________, Committee Chair Dr. John Clevenger __________________________________, Second Reader Dr. V. Scott Gordon ____________________________ Date ii Student: Parita Shah I certify that this student has met the requirements for format contained in the University format manual, and that this project is suitable for shelving in the Library and credit is to be awarded for the project. __________________________, Graduate Coordinator ___________________ Dr. Jinsong Ouyang Date Department of Computer Science iii Abstract of SECURITY SANDBOXING FOR PC2: WINDOWS VERSION by Parita Shah The purpose of a Sandbox software is to execute a program under restricted environment to protect a computer system from malicious activities. The Security Sandbox tool will be developed to be used for the Programming Contest Control System, called PC2 for short. PC2 software is used to support programming contests of ACM and its International Collegiate Programming Contests around the world. PC2 allows teams to submit their programs to the judges over the network. The judges can perform manual judging for the team’s submitted program. PC2 also supports automatic judging of these submitted programs. These are a few of the many functionalities of PC2. When teams submit their programs for judging, their code might perform malicious activities on the system that is running the judging module. Opening a network socket, accessing important or personal files, running a script to delete files, trying to hamper working of PC2, etcetera are a few examples of the illegal activities that the team’s programs could do to harm the judge’s system. Sandbox will keep a check on all the activities that the team’s program does during its execution and will stop the execution of iv the program as soon as it finds that it is performing an illegal activity. It will fulfill its task by inspecting all system calls made by the team’s program. The list of permissible or non- permissible system calls will be provided to the Sandbox by the judges ahead of time, in the form of a policy file. Apart from team’s submitted programs, Sandbox should also be able to run any executable file, as a target program provided to it by PC2 and return its result. One of the other major features of Sandbox will the Input/Output Forwarding of data from PC2 to this target program and vice versa. The deliverables of the project will be a Sandbox Software running on Windows and a report on the various details of this project. _______________________, Committee Chair Dr. John Clevenger _______________________ Date v ACKNOWLEDGEMENTS I would like to thank my professor, Dr. John Clevenger for his continuous support and encouragement during my research for this project. He helped me throughout the journey of the project development without which this project could not have reached its destination. I also would like to thank Dr. Scott Gordon for fulfilling his role as a second reader and helping me in the completion of this project. I would like to thank my husband, family and friends who have given me enormous strength, support and encouragement throughout my research and development for this project. vi TABLE OF CONTENTS Page Acknowledgements ............................................................................................................ vi List of Figures .................................................................................................................... ix Chapter 1. INTRODUCTION .......................................................................................................... 1 1.1 Programming Contest Control System ...................................................................... 1 1.2 Problem Description .................................................................................................. 2 1.3 Sandbox definition .................................................................................................... 2 2. EXISTING TECHNOLOGIES ....................................................................................... 4 2.1 Sandboxes available in market .................................................................................. 4 2.1.1 Virtual Machines .............................................................................................. 4 2.1.2 Cuckoo Sandbox ............................................................................................... 4 2.1.3 Sandboxie ......................................................................................................... 6 2.1.4 Chromium Sandbox .......................................................................................... 7 2.1.5 Security Sandbox for PC2: Linux Version ........................................................ 8 2.2 DLL Injection and API Hooking ............................................................................... 8 2.3 Conclusion ............................................................................................................... 11 vii 3. SANDBOX FOR PC2: SOFTWARE DESIGN AND APPROACH ............................ 13 3.1 Design Flow ............................................................................................................ 13 3.1.1 PC2 .................................................................................................................. 13 3.1.2 Sandbox.exe .................................................................................................... 14 3.1.3 Target program / Team Program / Child Process of Sandbox ........................ 14 3.1.4 Policy File ....................................................................................................... 15 3.1.5 Log File........................................................................................................... 15 3.2 Use Case Diagram ................................................................................................... 16 4. DLL INJECTION AND HOOKING IMPLEMENTATION ....................................... 18 4.1 Example: Blocking CreateFileA and DeleteFileA .................................................. 19 5. CHARACTERISTICS, LIMITATIONS AND FUTURE ENHANCEMENTS .......... 26 5.1 Characteristics ......................................................................................................... 26 5.2 Limitations .............................................................................................................. 27 5.3 Future Enhancements .............................................................................................. 28 6. CONCLUSION ............................................................................................................. 30 Appendix A Source Code ................................................................................................. 31 Bibliography ..................................................................................................................... 55 viii LIST OF FIGURES Figures Page 1. Cuckoo Sandbox Architecture ........................................................................................ 5 2. Steps of DLL Injection.................................................................................................... 9 3. Sandbox Data Flow Diagram ........................................................................................ 13 4. Use Case Diagram......................................................................................................... 16 5. Policy File Creation GUI .............................................................................................. 20 6. Policy File screenshot ................................................................................................... 20 7. Log File screenshot ....................................................................................................... 25 ix 1 1. INTRODUCTION 1.1 Programming Contest Control System PC2 is the abbreviation for Programming Contest Control system, pronounced as "P-C-squared" or "P-C-Two". This software was developed at California State University, Sacramento and it is designed to conduct programming contest operations on various computer environments. This software can support a programming contest occurring at multiple sites. It supports heterogeneous platform operations which includes Windows and Linux Operating systems. It is also dynamic and can send updates in real-time regarding the contest status to all the sites. [1] PC2 uses a client-server architecture for its operation [2]. When a contest is set up on a site, one PC2 server is configured per contest site which will administer the contest at that particular site. There are also multiple clients configured which communicate with that PC2 server. A client can perform one of several roles using different types of accounts such as, Administrator, Team, Scoreboard or Judge. These clients will communicate with their respective site servers only. The system takes the contestants’ (teams) programs as input and sends it to the contest judges over a network via TCP/IP. Once the team’s submitted program reaches the judges, they can judge it manually