DEGREE PROJECT IN TECHNOLOGY, FIRST CYCLE, 15 CREDITS STOCKHOLM, SWEDEN 2021

Automated secure code review for web­applications

SADEQ GHOLAMI ZEINEB AMRI

KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE Title

English: Automated secure code review for web­applications Svenska: Automatiserad kodgranskning för webbapplikationer

Authors

Sadeq Gholami Zeineb Amri School of Electrical Engineering and Computer Science KTH Royal Institute of Technology

Place for Project

Stockholm, Sweden

Examiner

Johan Montelius KTH Royal Institute of Technology

Supervisors

Fadil Galjic KTH Royal Institute of Technology Christoffer Jerkeby F­Secure

ii Abstract

Carefully scanning and analysing web­applications is important, in order to avoid potential security vulnerabilities, or at least reduce them. Traditional code reviewing methods, such as manual code reviews, have various drawbacks when performed on large code­bases. Therefore it is appropriate to explore automated code reviewing tools and study their performance and reliability.

The literature study helped identify various prerequisites, which facilitated the application of automated code reviewing tools. In a case study, two static analysis tools, CodeQL and Semgrep, were used to find security risks in three open source web­ applications with already known vulnerabilities. The result of the case study indicates that the automated code reviewing tools are much faster and more efficient than the manual reviewing, and they can detect security vulnerabilities to a certain acceptable degree. However there are vulnerabilities that do not follow a pattern and are difficult to be identified with these tools, and need human intelligence to be detected.

Keywords: automated code reviewing tools, CodeQL, Semgrep, code review, security vulner­ abilities, web­applications

iii Abstrakt

Det är viktigt att skanna och analysera webbapplikationer noggrant för att undvika potentiella säkerhetsproblem eller åtminstone minska dem. Traditionella kodgranskningsmetoder, såsom manuella kodgranskningar, har olika nackdelar när de utförs på stora kodbaser. Därför är det lämpligt att utforska automatiserade verktyg för kodgranskning och studera deras prestanda och tillförlitlighet.

Litteraturstudien hjälpte till att identifiera olika förutsättningar, som underlättade tillämpningen av automatiserade kodgranskningsverktyg. I en fallstudie användes två statiska analysverktyg, CodeQL och Semgrep, för att hitta säkerhetsrisker i tre open source­webbapplikationer med redan kända sårbarheter. Resultatet av fallstudien indikerar att de automatiska kodgranskningsverktygen är mycket snabbare och effektivare än de manualla kodgranskningar och att de kan upptäcka säkerhetsproblem i viss acceptabel grad. Det finns emellertid sårbarheter som inte följer ett mönster och som är svåra att identifiera med dessa verktyg, och behöver mänsklig intelligens för att upptäckas.

Nyckelord: automatiserade kodgranskningsverktyg, CodeQL, Semgrep, kodgranskning, säkerhet sårbarheter, webbapplikationer

iv Acknowledgements

Firstly, we would like to thank our advisors at KTH Royal Institute of Technology, Fadil Galjic and Johan Montelius, for providing us with their valuable insights and feedback throught the entirety of the degree project.

Secondly, we would like to give very special thanks to our supervisor at F­Secure, Christoffer Jerkeby, for his guidance and helping us to achieve this project.

v vi Contents

1 Introduction 1 1.1 Background ...... 2 1.2 Problem ...... 2 1.3 Purpose ...... 2 1.4 Goals ...... 3 1.5 Research Methodology ...... 3 1.6 Delimitations ...... 3 1.7 Structure of the thesis ...... 4

2 Theoretical Background 5 2.1 Data Security ...... 5 2.2 Security Risk Classes ...... 6 2.3 Security code reviews ...... 10 2.4 Automated code reviewing tools ...... 11 2.5 Related work ...... 13

3 Method 15 3.1 Research Methodologies ...... 15 3.2 Research method process ...... 17 3.3 Breaking down the research question into sub­questions ...... 19 3.4 Data Collection ...... 19 3.5 Development tools ...... 20 3.6 Documentation ...... 21

4 Prerequisites for Case Study 23 4.1 Prerequisites for the survey ...... 23 4.2 Prerequisites for CodeQL ...... 23

vii 4.3 Prerequisites for Semgrep ...... 26 4.4 CodeQL extension for Vscode ...... 28

5 Performing the Case study 31 5.1 Case Study: Part 1 ...... 31 5.2 Case Study: Part 2 ...... 33

6 Case study: Results 37 6.1 Case study: Part 1 ...... 37 6.2 Case study: Part 2 ...... 40

7 Discussion 43 7.1 Answering the research sub­questions ...... 43 7.2 Methods ...... 47 7.3 Limitations ...... 49 7.4 Summary ...... 50 7.5 Future Work ...... 50

Appendix A: Questionnaire 57

Appendix B: Comments about the questionnaire 69

viii Chapter 1

Introduction

In the modern digitalized world many aspects of one’s life are controlled or driven by digital tools. Cloud computing, smart devices, internet banking, personal information, social media, and many other technologies that involve individuals’ everyday life are common examples of digitalization. Furthermore, it has a great impact on industries, businesses, armies and governments as well [1]. Therefore protecting personal and important data stored in databases against unauthorized users, becomes one of the most discussed topics today.

Data security is about protecting these sensitive data from unwanted actions of malicious users. Its primary characteristic is to discover vulnerabilities in computer systems that might lead to unauthorized access of data or allow a variety of cyberattacks [19]. A cyberattack is the act of disruption of integrity or authenticity of private information. It’s main goal is to steal or alter sensitive information from organizations, government offices or even personal computers [2].

In recent years many new security vulnerabilities and common bugs that the developers mistakenly make have been detected. Furthermore, each developer develops more codes and applications, nowadays compared to 10 years ago [3]. Each new developed code needs to be reviewed, either manually or automatically, to identify security vulnerabilities and protect them against cyberattacks. Hence the increasing need for a more effective way to review the code arises.

1 1.1 Background

Security bugs which lead to exploitation occur repeatedly during the development stage of applications. The most common and critical security vulnerabilities are well classified and ranked by OWASP [4] and CWE [5]. In order to detect these security weaknesses and prevent potential vulnerabilities, there is a need to continuously review the newly developed code.

The code review can be done manually by reading the codes line by line and finding the security bugs. As most of the bugs follow a certain pattern, they can also be detected by automated code reviewing tools.

1.2 Problem

During manual code reviewing, developers who are familiar with all security weaknesses analyse the codes thoroughly, or ethical hackers try to hack the application and find security flaws in the application. This is a process that requires patience, experience and skills. One problem that may arise while manually reviewing these codes is the fact that some bugs can easily go undetected, because they are difficult to be identified by the human eyes. Another issue with manual code reviewing is that it can be time consuming and expensive. An alternative way to review the codes is to use automated vulnerability detection tools, which scans all the code and finds security vulnerabilities. Hence, the following research question:

RQ. What are the benefits and drawbacks of using automated code reviews in web applications?

1.3 Purpose

The purpose of the thesis is to contribute to a more effective and less time­consuming security code review by examining different automated code analysis tools and comparing them with manual code review. The aim is to present the concept of automated code reviewing tools to other developers and convince them to use them.

2 1.4 Goals

The main goal of the thesis is to present the benefits and drawbacks of secure automated code reviews using vulnerability detection tools against web applications. This is achieved by analysing and evaluating their effectiveness. Another goal is to find out if the tools have the potential to take over the manual code review.

1.5 Research Methodology

Primarily, a literature study is done in order to get essential knowledge to answer the research question. This study aims to gather relevant information about data security, different security vulnerabilities and automated code reviewing tools. This method is approached qualitatively, as only necessary information was obtained from different sources.

Secondly a quantitative experimental research method was done in the form of a case study. The experimental part of the study was conducted at the security company, F­ Secure. The case study goes as follows;

• Create a questionnaire with codes containing a certain number of common security bugs.

• Let a case study group (developers and security researchers from F­secure) review the code manually and detect as many bugs as possible from the questionnaire.

• Run the same code with the automated code reviewing tools.

• Compare the results.

1.6 Delimitations

One limitation of the thesis is that the case study only focuses on JavaScript and does not take other programming languages into account. Another limitation is that only two automated code reviewing tools are examined. One more limitation is the limited amount of code on which the experiment was performed. These can affect the result of the research, if they are not properly managed.

3 1.7 Structure of the thesis

The thesis has the following outline:

• Chapter 2 presents the theoretical background of the different bug classes and the automated code reviewing tools.

• Chapter 3 presents the methods and research approach for the thesis.

• Chapter 4 covers the requirements for creating automated static analysis code.

• Chapter 5 presents how the different stages of the case study went.

• Chapter 6 presents the results collected from the case study.

• Chapter 7 summarizes and presents a discussion of the drawn conclusions and thoughts about improvement for future research.

4 Chapter 2

Theoretical Background

The purpose of this chapter is to present the theoretical background of the thesis that provides helpful information to answer the research question. This chapter begins with an introduction of data security, followed by a more detailed description of the various security bug classes. Further comes an introduction to what code reviewing is, as well as an overall overview of what manual and automated code reviews are. Finally, the chapter ends with an introduction to the code reviewing tools.

2.1 Data Security

Many research are being held by private and governmental organizations to increase data security. Unfortunately, new security breaches can be found and developers might develop applications with bugs that are already known. Every piece of code written by developers, especially library codes that are used by other programs, is a potential way to a security breach. There are no indications that the security of new software releases are decreasing. Therefore it is important that every code that is developed should be tested and reviewed for potential bugs[6].

Because of the increasing usage of digital tools, software development has also become bigger, more important and more complex. Unfortunately this leads to a situation that in 2020 developers write a hundred times more code than that of 2010[3]. This can lead to an increase of security bugs and potential security vulnerabilities as every piece of code can be a potential threat to the security. It is also obvious that most developers are not familiar with the security bugs and by writing more code they can contribute

5 to more of these flaws. This means that the developers introduce more security holes than the security developers can detect. This means that the cost of finding security bugs also increases if they are found one by one[6].

Today most IT­companies try to upgrade their systems development life cycle (SDLC) to secure­SDLC. SDLC is a term used by system/software engineering to indicate a process for planning, creating, testing and deploying of software. It includes different models, such as the classic waterfall, iterative, agile and most recent DevOps. So due to the increasing security breaches, most IT­companies want to include security checks in the SDLC process so that every piece of code is tested against the security bugs before it is deployed[7].

2.2 Security Risk Classes

In order to review the code for security vulnerabilities, both manually and automatically, one has to understand the most critical security bugs as well as the most common ones among the developers. The following security risks are classified as most common and most critical by two open­source organizations, which work to improve the security of software. These organisations are The Open Web Application Security Project® (OWASP)[4] and the Common Weakness Enumeration (CWE™) [5]. OWASP focuses on listing the most serious web application security risks, whereas CWE lists the most common software and hardware weakness types.

Injection

An injection occurs when an attacker injects malicious data to an interpreter such as databases, shell commands or operating systems via user input. This type of vulnerability takes place when an application reads a user’s input data and inserts it to the interpreter without validating this data. The injection flaws are common in SQL, LDAP, XPath, NoSQL queries, OS commands, XML parsers, SMTP headers, expression languages and ORM queries. According to OWASP top ten this is the most dangerous type of security vulnerability, that can result in data corruption, data loss, loss of accountability, denial of access and sometimes even complete host takeover. This type of vulnerability is easy to detect in the code and therefore can be uncovered by automated tools [4][5].

6 Broken Authentication

This type of security flaw occurs when the functions or methods handling the authentication and session management are implemented poorly. Broken authentication security vulnerabilities allow attackers to compromise the usernames, passwords, session tokens and keys. This type of attack can occur when an application has the following flaws:

• When the application allows for weak and predictable credentials. When the credentials are stored in an insecure way e.g. without hashing.

• The application does not have protection for brute force attacks, credential stuffing, dictionary attacks or other automated attacks.

• The session tokens or IDs are exposed in the URL, they have long validation time, they are not rotated after each successful login or they are not invalidated after logout or inactivity.

• The application has only one or too few authentication factors.

• The credential recovery processes, e.g. when the user has forgotten the password, are not efficient.

These types of vulnerabilities are easy to detect in the code and can be reviewed by automated tools [4][5].

Sensitive Data Exposure

This class of security bug happens when the sensitive data such as personal information, health records, credentials, credit cards etc. are not well­protected. This attack occurs when the sensitive data is stored in a corrupted database, or databases with low security. This can also happen when the sensitive data is transmitted or stored with weak encryption or no encryption at all, or the data is transmitted over insecure protocols such as HTTP, FTP, SMTP with clear or weakly encrypted text [4][5].

XML External Entities (XXE)

When an application uses XML to transfer data from front­end (i.e. browser) to back­ end (i.e. server) and vice versa, it could be a victim of this type of attack. Many older XML processors allow specification or a reference to an external entity, while parsing

7 the XML­files. An external entity is a type of entity (storage unit of some type in XML), whose definition is located outside the DTD. DTD stands for document type definition, and it defines the structure of an XML document. So if the XML processor is not configured carefully to check the contents of the external entity, a hacker can send malicious code to the server via the reference. This vulnerability can lead to internal file shares, internal port scanning, denial of service attack or remote code execution that can do anything to the system [4][5].

Broken Access Control

This security risk class contains all vulnerabilities that give an unauthorized user access to privileged data or performing an action outside of one’s limit. These vulnerabilities can occur when an application misses or has poor authorization control functions. If an application has following flaws these types of vulnerabilities can occur.

• When the application does not have access control checks for different URLs or HTTP requests.

• CORS (Cross­Origin Resource Sharing) are not configured correctly.

• Generating simple primary keys for users and using them to give access.

• Misunderstanding the authentication as authorization [4][5].

Security Misconfiguration

This type of security weakness takes place when an application uses the insecure default configuration of a server or database that it is using. It can even happen when security settings of the application are not implemented. When developers use an external library, framework, or web service, they tend to forget to make sure to disable or delete the services that are not used. Many services have default security configurations that need to be overridden. This can be a very easy credential e.g. user: user, admin: password and etc. If the developers forget to override this configuration, the system will be a victim of security misconfiguration [4][5].

8 Cross­Site Scripting (XSS)

When an application reads input data from a user e.g. a HTML­form and passes it to the server or other users without validation, it can become a victim of XSS. The input can include malicious script that could be sent to other users or the web server. So an attacker can use XSS to send malicious scripts to other users. This script can be run in other end users’ browsers, which can lead to stealing sensitive data such as cookies, session tokens or even gaining privileged access within the application. There are three types of XSS:

• Reflected XSS: This type of XSS occurs when an application includes the unvalidated data in the response and sends it to other end users, e.g. a message to another user.

• Stored XSS: In this type of XSS the unvalidated data is stored in the database and later viewed to other users, e.g. a comment.

• DOM XSS: In this type of XSS the unvalidated data is written back to the DOM in an insecure way, e.g. by innerHTML method [4][5].

Insecure Deserialization

Insecure deserialization happens when a user­controllable data is deserialized by a website. Deserialization is the process of taking data from e.g. a file and rebuilding the data to an object. This vulnerability allows the attacker to transfer malicious code as a serialized object. If the code does not validate or sanitize the deserialized object, it can lead to remote code execution, injection attacks and even replay attacks. The best way to prevent this vulnerability from taking place is to not accept serialized objects from unsecure sources [4][5].

Using Components with Known Vulnerabilities

This security bug happens when frameworks, libraries or any other modules that are used within an application are executed with full privileges. When such vulnerable components are at risk of exploitation, an attack can occur, which might cause serious data losses. Some of the reasons why this bug might arise are:

• Software is outdated or not supported.

9 • Uncertainty of all the components’ versions.

• Not securing the configuration for the components.

Some simple ways to avoid this vulnerability from happening is to remove unused dependencies and to only obtain components from secure sources [4][5].

Insufficient Logging Monitoring

This type of vulnerability occurs when an application has poor logging, monitoring, detection as well as response. The shortage of these functionalities makes the appli­ cation easier for exploitation, which allows attackers to temper with and extract data without being detected. Studies showed that breaches take around 200 days to be identified, which is enough time to cause damage to the application. One way to prevent this security bug from happening is to make sure that the application has a good monitoring system by inspecting the logs following penetration tests [4][5].

2.3 Security code reviews

Code reviewing is an important process of software development that involves both manual and/or automated reviews. The primary aim of using code reviews is to evalu­ ate and increase the code efficiency, identify security matters and improve its quality by following best practices [8]. A secure code review does not recognize all issues and defects in the code, instead, it attempts to point out different security risks and threats within the code so that some mitigation measures can be invented. Although secure code reviews help protect applications, it is not 100% guaranteed that the code has no security weaknesses at all [9]. The next section will present manual code review, also known as peer review, followed by the automated code inspection.

Manual Code reviews

Peer code reviewing is a common practice where another team member, or any third party, inspects the code for opportunities of possible improvement, finding defects and even a long­lived maintainability [10]. Manual code reviewing means that the reviewer inspects the code line by line, attempting to spot possible vulnerabilities. Some case studies provide affirmation that code reviewing produces lower post­release defects in

10 comparison to poorly reviewed code [8]. The above mentioned benefits are heavily dependent on various factors, such as the social aspect and personal skills, making the code review costly, as it involves human intelligence, as well as time consuming since it is the longest process of the development phases [11].

Automated Code reviews

Automated code reviews involve the process of automatically analyzing the code using a set of standards to detect defects and bugs. It uses vulnerability detection tools to investigate the code for any possible risks or flaws that can lead to security vulnerabilities. It is widely recommended to use multiple code reviewing tools as they have different strengths as well as their own drawbacks [12]. Automated tools can review an application faster than with manual reviews. They have the advantage of inspecting large code bases that would otherwise not be pragmatic to analyze manually. Although these tools can find some bugs, they might entirely miss others and in other cases, even report false positives. Automated static analysis tools can be costly moneywise as well as time consuming, as it takes time for engineers and developers to learn how to use these tools [13]. Since the automated tools cannot understand the developer’s intentions and the business logic, they tend to struggle with identifying flaws that are linked to authentication, access control issues, insecure use of cryptography as well as data validation in general [14].

2.4 Automated code reviewing tools

There are many vulnerability detection tools that have been developed in order to review code automatically. They are also called static analysis tools or source code analyzers, as they examine a program’s code without the need to execute it [20]. Some examples of code reviewing tools are Rhodecode, Collaborator, Crucible, Veracode, Sonarqube, CodeQL as well as Semgrep.

CodeQL

CodeQL, which is a part of GitHub, is a semantic analysis engine used by developers to automate secure code reviews [15]. CodeQL handles code as if it is data where security bugs are modeled as queries. These queries are then executed against the database

11 taking out from the code. The database is a directory that contains data, source reference to display query results, the query results and lastly log files. CodeQL has support for multiple languages, such as C/C++, C, Go, Java, JavaScript, and Python. The most frequently used database model is a relational one, where the data is stored in tables [21].

QL is a declarative, object­oriented query language for codeQL databases. It uses arranged semantics and contains object­oriented classes, which improves modularity and permits code reuse. When a QL program is executed against a database, see Figure 2.1, it is compiled into a dialect of the logic programming language Datalog. The syntax for QL is similar to SQL, the only difference being that the QL semantics are based on Datalog. QL is used to retrieve data from relational databases. It is optimized for performance and later evaluated to yield a result. It is necessary that the result is always a finite set of values, otherwise it will lead to an error [22].

Figure 2.1: The analysis overview diagram of CodeQL shows the process of query compilation, extraction of code base as a relational database and the query evaluation on the code database [41].

Semgrep

Semgrep is a fast, semantic and open­source static analysis tool used to detect security vulnerabilities. Semgrep does not need the code to be compiled as it is static, so it can scan and review large amounts of code in a short time. It examines the code

12 and can be used to enforce secure coding best practices at editor, commit or even CI (Continuous Integration) time [16]. It runs locally in a build environment. It is based on Facebook’s pfff and Linux’s Coccinelle projects, which were also used as static analysis tools [23].

Semgrep has support for the following languages: Go, Java, JavaScript, JSX, JSON, Python, Ruby, TypeScript and TSX. There are many advantages to using Semgrep, the most important being, it has a registry containing rules and standards that cover many types of bugs. However, it still permits the user to have their own custom rules, without complicated queries, to find vulnerability weaknesses [17].

Semgrep combines the properties of grep and Abstract Syntax Trees (ASTs) to set up a model of the analysed code, unlike regexes that use strings [18]. Grep is a command line utility of Unix­like systems, that searches through plain texts and finds the lines that match a certain regular expression [25]. The ASTs are a tree representation of source code’s abstract syntactic structure, see Figure 2.2, in a programming language. The abstract part of ASTs means that it only captures the necessary structure of the input in a tree form, while excluding unessential syntactic details. They are mainly used in compilers during semantic analysis, to check for the correctness of the program, through a complete traverse of the tree. ASTs are generally created bottom­ up [26].

2.5 Related work

There are many articles and documentation, investigating the effectiveness and limitations of automated code reviews to find security related vulnerabilities.

Evaluating the cost reduction of static code analysis for software security

This study [29] examines the performance of the static analysing tool (SAT) Coverity Prevent on both mature telecom graded software and open­source projects. The results were later divided into three categories, namely false positives, security faults and functional faults. The source code used contained several known vulnerabilities that were reported by a third party. The study shows that the tool had a 30% success rate in identifying the known vulnerability risks while detecting 59 new vulnerabilities

13 throughout the examined products. It even proves that if the static analysis tool was used then a 17% cost savings would be achievable.

Figure 2.2: A representation of AST for Euclidean algorithm: while b ≠ 0 if a > b a := a − b else b := b − a return a [40].

14 Chapter 3

Method

The purpose of this chapter is to describe the research methods and approaches used in the degree project. The first section explains the research methodologies that were applied to the project. The following section presents the sub­questions used to answer the main research question. This is followed by an overview of the research process, which shows the different parts of the thesis project. Finally, the chapter ends with a discussion of the development tools and documentation.

3.1 Research Methodologies

This section describes the research methods and approaches that were applied in this project.

Qualitative and Quantitative Research Methods

Quantitative research method is used when the observations can be measured and tested by numerical values and mathematical calculations. In this type of research methodology data are collected as numbers by experiments and analyzed by mathematical calculations and statistical inferences. It is used to describe the relationship between the collected numerical data and the theory [30].

Qualitative research method is a descriptive method which is used for the data that can only be observed, but not measured. It is used for data that cannot be represented as numerical values, such as human feelings and emotions. This method is used to gain insights and knowledge about underlying reasons and motivations to a problem. It

15 can also be used to help come up with ideas and hypotheses for quantitative research [31].

Inductive and Deductive Research Approaches

Inductive approach, also known as inductive reasoning, presents theories and arguments at the end of the examination process, with clarifications from different observations. It uses a “bottom­up” approach, where empirical data is collected before developing a hypothesis. The data is collected and analysed, commonly with qualitative methods in order to reach conclusions [32] [33].

On the other hand, deductive approach, which is also known as deductive reasoning, tends to focus on verifying and testing existing theories. It uses a “top­down” approach, where already available information is applied when developing a hypothesis. Usually, a deductive reasoning tests the hypothesis and theories using quantitative methods with a large scale of data sets [32] [34].

Case study

A case study is an experimental research strategy, a guideline or a methodology that can be used for quantitative research, qualitative research, or a combination of both. It is used to experiment with ideas in a real­life context, where the borderlines between ideas and its context is not clear. It is used to gain a profound understanding of a phenomenon or an issue in its everyday environment, situation, or organization i.e., the case to be studied [32].

Applied research methods and approaches

In order to solve the thesis problem firstly a qualitative research method was used to gain deep insight and enough knowledge about the problem. This method was done by doing a literature study about data security, common security vulnerabilities in web applications, automated and manual code reviews to detect the vulnerabilities, the automated code review tools that are going to be used in the case study, which are CodeQL and Semgrep, and an overview of related works.

Secondly, a quantitative research method was applied using the case study approach. The conclusion should be drawn based on the data that are collected from the case

16 study, which is an inductive approach.

3.2 Research method process

This section describes and explains the research method process used in this project to answer the research question. An overview of the process is shown in Figure 3.1.

Figure 3.1: An overview of the research process used to answer the research question.

Identifying topic and problem

The first step of the research process was to identify the research question and divide it into sub­questions. This in turn breaks down the main problem into smaller problems that need to be unfolded in order to answer the sub­questions.

17 Literary study

Thereafter, a literary study was conducted to gain knowledge about different subjects. Many topics were addressed such as data security, security risk classes, security code reviews and automated code reviewing tools. The articles were found by mainly searching on KTH library and Google Scholar. These found articles were later reviewed, repeatedly, to make sure they were relevant for the thesis purpose and to see if they could potentially be used as the theoretical background.

Case study

The case study used in this thesis was a single activity consisting of two different parts with a common goal, which was to answer the research question. These parts are different ways to investigate code reviewing tools and get a clear picture of what they can do and how good they are. The questionnaire was necessary within the first part.

Case study: Part 1

The first part was performed as an attempt to compare the automated code reviewing tools with manual code review. This was achieved by creating a questionnaire that contained code­snippets with security vulnerabilities. This questionnaire was taken by a group of volunteers which consisted of last year computer science students, senior developers from different companies and security researchers. Their task was to find security bugs in the code snippets, if there were any and name the bug class according to OWASP top 10. The same test application was then scanned with both CodeQL and Semgrep in order to detect the vulnerabilities. The results were later compared in order to answer the research question.

Case study: Part 2

The second part of the case study was done by scanning large code­bases with codeQL and Semgrep. These include two open source test applications that contained all OWASP top ten bug classes for penetration testing. These two test applications were used because they contained intentional/reported security vulnerabilities. During this part, the performance of the two automated code reviewing tools were compared with each other.

18 Result

The final step of the research process was to analyze and evaluate the results, which in turn, facilitated drawing conclusions regarding the benefits and drawbacks of automated code reviews.

3.3 Breaking down the research question into sub­ questions

The research question of the thesis is ”What are the benefits and drawbacks of using automated code reviews in web applications?”. The question has been divided into open sub­questions in order to make it narrower and less complex. By answering these following sub­questions, it will make it easier to find the answer for the primary research question.

SRQ1. What are the prerequisites needed to use automated code­scanning tools? SRQ2. What is the accuracy of automated code reviews in comparison to manual code reviews? SRQ3. How well do code reviewing tools perform with large code­bases? SRQ4. What are the costs of using code reviewing tools?

3.4 Data Collection

This section introduces the methods implemented to gather data to answer the research question.

Literature study

As soon as the project started, a literature study was conducted to gain more knowledge and learn about the theoretical background of relevant topics. The literature includes books about computer security as well as scientific articles and documentation about different forms of code reviews. The articles were found on dl.acm.org and ieeexplore.ieee.org. Documentation about security concepts and vulnerabilities were found on cwe.mitre.org and owasp.org.

19 In order to have a better understanding of automated code reviewing tools, it was essential to know how they work. This included watching tutorial videos and reading software documentation about how to solve problems that could potentially occur. The documentation about automated code­scanning tools were found on codeql.github.com as well as semgrep.dev.

Logs

Issues and problems encountered during the development process were noted down in a Google document shared by the group members. These issues were thoroughly described, for instance, in addition to just introducing the problems, how they were solved and how long it took to solve them were also written down in the document. At times, it was helpful to revisit the literature and documentation to obtain a better understanding of the subject (matter) at hand.

3.5 Development tools

This section presents the tools such as the development environment, project model and programming language used in the project.

Development environment

The development environments used in the project were CodeQL CLI, Semgrep CLI and Visual Studio Code. CodeQL provides a command line interface (CLI) to the users, which is used to create relational databases from the code bases and execute the queries written in codeQL. Whereas in Semgrep there is no need of creating databases from codebase, instead the rules can be executed directly on the original code by Semgrep CLI. Visual Studio Code (Vscode) is an integrated development environment (IDE) developed by Microsoft and is used for editing code. It allows for installation of different extensions that add many extra functionalities and supports for the IDE. Both CodeQL and Semgrep have extensions that can be installed in Vscode. The OS used during the development was Windows 10, developed by Microsoft, since it was compatible with the tools used in the project.

20 Programming language

In this project, the reviewed code was written in JavaScript. JavaScript is a dynamic, client­side programming language and is one of the most popular web programming languages. It supports object­oriented programming with object prototypes as well as functional programming. Since JavaScript is primarily used in browsers, most modern browsers have a JavaScript engine, as it is important and essential for web applications [27][28].

The code scanning tool CodeQL uses QL, whereas Semgrep uses Python for analysing the code. Both CodeQL and Semgrep have support for JavaScript.

Project model

In this thesis, an iterative model was used. Since new tools and environments were explored, plans and decisions could be reconsidered during different parts of the research process. In order to manage the project in an effective way, a Gantt chart was used to plan and schedule different phases of the research process. It helped keep track of the deadlines and goals that needed to be accomplished by the end of the project.

The Moscow method was used to keep the research process under control and limit the research question. This method was used to categorize different priorities:

• Must have an answer to the main research question using scientific methods and methodologies.

• Should compare manual code review with automated code review tools as well as different tools with each other.

• Could write own queries in CodeQL and own rules in Semgrep.

• Would not use other scanning tools than CodeQL and Semgrep.

3.6 Documentation

For this thesis project, a Trello board was used for different tasks that needed to be done. Furthermore, a Lucidchart map, where UML was used to illustrate the outline for the research process. Additionally, a Google sheet was used to keep track of the time

21 and milestones for each iteration. In order to have a good structure for the report, the template provided by KTH for the thesis project was used. A Google form was used to create a questionnaire that was later sent out to the volunteers. Lastly, a spreadsheet on Excel was used for organizing the collected data sets and comparing them.

22 Chapter 4

Prerequisites for Case Study

This chapter presents an overview of the different automated code reviewing tools as well as the development tools used in this project. In order to work with automated code reviewing tools, it is important to learn how to set them up and learn how to use them. In addition to knowing how these tools work, some other requirements have to be fulfilled beforehand. These prerequisites are thoroughly described in this chapter.

4.1 Prerequisites for the survey

The results deducted from the literature study helped gain more knowledge on the different security vulnerabilities. These security vulnerabilities are classified into different categories based on their severity and common occurrence during development. OWASP top ten web­app vulnerabilities are the most serious security bugs that occur in web development. These vulnerability classes are also among top 25 in CWE, which lists the most common security vulnerabilities. After understanding the concepts and details of these bug classes, the next step was to prepare the questionnaire for the manual code review. To prepare the questionnaire, code­snippets with the OWASP top ten vulnerabilities were developed.

4.2 Prerequisites for CodeQL

In order to use CodeQL to review code, CodeQL CLI is needed. This section describes the prerequisites that were needed to successfully set up and use CodeQL to detect

23 vulnerabilities, following the information provided by the CodeQL page [35].

CodeQL CLI

It is a command­line tool that is used to run CodeQL queries provided by GitHub engineers on a locally open source software projects. It can be used to create CodeQL databases from the code­bases in order to run the CodeQl queries on. It can also be used to develop and test custom CodeQL queries.

The CLI can be installed on Windows, Linux and Mac operating systems. To install the CLI the following steps should be followed [35].

1. Download the CodeQL CLI binaries zip package for the required operating system from https://github.com/github/codeql­cli­binaries/releases.

2. Create a directory where the CLI, libraries and queries can be placed.

3. Clone the CodeQL repository from GitHub which contains queries and libraries required for analysing different programming languages supported by CodeQL. The link to repository is https://github.com/github/codeql. Change the folder­ name of cloned repository to codeql­repo as it is codeql by default.

4. Extract the CLI zip package to the folder created in step 2.

5. Launch codeql either by executing /codeql/codeql, or by adding /codeql in the PATH variable.

After installing and setting up the CLI, codeql commands can be run to create and analyse CodeQL databases. In order to run a command one can use:

codeql [command] [subcommand]

In order to run CodeQL queries on a code­base from CLI or via Vscode, a CodeQL database should be created. CodeQL database is relational data extracted from the code­base. Running the following command from the root of the code­base creates CodeQL database:

codeql database create --language=

In this command the is the path to the new database created and flag is for the identifier of the language for which the database is created. Table 6.1 shows the identifiers for supported languages in CodeQL.

24 Table 4.1: Languages supported by CodeQL and their identifier

Language Identifier

C/C++ cpp C# csharp Go go Java java JavaScript/TypeScript javascript Python python

CodeQL queries

A CodeQL query is similar to SQL query containing a from clause where the variables are declared, a where clause where logical formulas for the code­base analysis is written and a select clause where the CodeQL expressions are written. All CodeQL queries should be saved in files with .ql file extension. Figure 4.1 shows how a CodeQL query structure looks. It is possible to either create your own queries or use preexisting queries from LGTM.com and GitHub repository of CodeQL.

Figure 4.1: The Structure of a CodeQL query in .ql files

Once a CodeQL database is created from the code­base, the CodeQL queries can be run on this database either by the CLI or from the CodeQL extension in VS Code. The following command should be run for analyzing a CodeQL database with given CodeQL queries:

codeql database analyze --format= --output=

In this command the is the path to the CodeQL database that is analyzed.

25 In one or more query files can be specified, either by specifying a .ql file or a directory that will be searched recursively for the query files. The flag is used to specify the format of the results of the analysis. The format can be CSV where each line in the output corresponds to an alert, SARIF that is designed to represent the output of a broad range of static analysis tools, or graph format. The

is the path for the output of the results file generated during analysis.

4.3 Prerequisites for Semgrep

This section describes the prerequisites that were needed in order to use Semgrep. The Information provided by the Semgrep page [23], which was found from the literature study, was followed to successfully set up and use Semgrep to detect vulnerabilities.

Semgrep CLI

Semgrep CLI is an open source command line interface that is available for installation on Ubuntu, WSL, Linux and macOS, but not for Windows. There are several ways to locally install Semgrep CLI, or it can be run without installation via Docker, however in this project, pip was used. When downloading Semgrep CLI, it is required to already have Python3 and pip installed. The following are the Linux commands that were used to locally download Semgrep CLI:

sudo apt-get install python3

sudo apt-get install python-pip

sudo python3 -m pip install semgrep

This command line tool is used to run Semgrep rules and it operates directly on source code. Once the CLI is installed, Semgrep commands can be executed to analyse applications and code sources. A Semgrep command looks as follows:

semgrep [command]

In order to run a Semgrep ruleset on a code­base, run the following command:

semgrep --config

26 This command configures the YAML file or a directory of YAML files with .yml file extension. The user can add a URL of the configuration file or the Semgrep registry name containing various Semgrep rules.

Semgrep rules

After successfully installing Semgrep CLI, the user can write rules/rulesets to automatically detect security vulnerabilities. It’s possible for the users to either create their own rules or use preexisting rulesets from the Semgrep registry. Semgrep.dev [23] provides rules for most security vulnerabilities. When running different Semgrep rules, or checking for various security vulnerabilities simultaneously, all Semgrep rules should be saved in a YAML file. By running the following command from the directory where the source code is saved, all the rules written in the rules.yml file will then be executed.

semgrep -f /path/to/semgrep/rules.yml

What is most important in the YAML file are:

• patterns

• languages

The patterns operator consists of various different patterns that are to be scanned. The languages operator is the language that the code is written in and whose syntax is to be matched. For more information about the languages supported in Semgrep, see table 4.2.

Table 4.2: Languages supported by Semgrep as well as their extensions and tags

Language Extensions Tags Go .go go, golang Java .java java JavaScript .js, .jsx js, jsx, javascript Json .json json, JSON, Json Python .py, .pyi python, python2, python3, py TypeScript .ts, .tsx ts, tsx, typescript Ruby .rb ruby, rb JSX .js, .jsx js, jsx, javascript TSX .ts, .tsx ts, tsx, typescript

27 Figure 4.2 shows how a Semgrep rule can look. The pattern­either is used when code snippets can’t be matched by the same pattern, hence the need to write a pattern clause for each of these code snippets. Pattern­either consists of various pattern clauses that Semgrep will later have to match code for either one of these pattern clauses.

Figure 4.2: The Structure of a Semgrep rule in .yml file

In this project however, preexisting rules that were written by Return To Corporation, also known as r2c, and other contributors, were used. The two most known Semgrep rulesets could be executed using the following commands:

semgrep --config=p/ci semgrep --config=p/security-audit

The first is used to find logic bugs and security vulnerabilities, whereas the second is used to identify security audit points.

The user can choose which rule severity, ERROR, WARNING or INFO, they would want to report findings on. Note that Semgrep matches all the rules regardless of the their severity by default. In this project, all severities were reported.

4.4 CodeQL extension for Vscode

CodeQL extension can be installed from Visual Studio Code’s marketplace. This extension uses CodeQL CLI to compile and run the queries. If the CLI is installed and added to the PATH the extension will use this version, otherwise the extension will automatically manage access to the latest version of executable CLI. After installing the extension a CodeQL work­space should be set up, in order to access the standard CodeQL libraries. This would be done by following the following steps:

28 1. Clone the CodeQL’s starter work space from https://github.com/github/vscode­ codeql­starter/repository. This should be done by the git command git clone – recursive, to include all the sub­modules.

2. Open the work space by opening vscode­codeql­starter.code­workspace file from the cloned repository in Vscode.

After setting up the work space CodeQL databases can be added to extension either from a folder, that was created by the CodeQL CLI, or by downloading directly from the LGTM.com. If there are several databases in the extension, one can hover on the desired database and click on set current database to run queries on that database. In order to run a query, one has to open the .ql file of the desired query, right click anywhere in the file and click run query. The result will be shown as a list of alerts.

29 30 Chapter 5

Performing the Case study

The case study was designed to examine the functionality of the automated code reviewing tools. The tools were compared to each other, and against manual code reviewing.

5.1 Case Study: Part 1

The aim of this part was to compare the detection capabilities of automated and manual code reviews.

Creating a questionnaire on JavaScript vulnerabilities

To be able to compare code reviewing tools with manual reviewing, a suitable questionnaire was created. The main aim of the survey was for some volunteers, such as students, developers as well as security professionals to practice their detection capabilities. With the help of F­secure, it was possible to contact some organisations and ask various developers and security professionals to participate in this survey.

As a first step before creating the questionnaire, the code for the application to be reviewed was found. The test application implemented for the questionnaire was an active open source project, where the source code contained several already known security vulnerabilities. This application was deemed a good choice since it contained ”bad code” examples with reported vulnerabilities, as well as solutions on how to prevent them.

31 Then, some code examples were chosen and rewritten in order to make them more understandable to the reader. Since static analysis tools are prone to incorrectly report some warnings, which are also known as false positives, some code­snippets that did not contain security vulnerabilities were also included. Next, a Google form was used to create a survey on JavaScript vulnerabilities.

This questionnaire contained a total of 15 programming questions, with at least one question for each bug class. Another question was later added to gain insight on what the volunteers thought was the most difficult/easiest in this questionnaire.

In order to keep this chapter to the point, there will be no code review of the questionnaire. To have more information about the survey questions, see Appendix A.

Setting up the automated code reviewing tools

This section describes how Semgrep and CodeQL were used to find bugs. The produced results were later analysed and compared as an attempt to find answers to some of the research sub­questions.

Setting up CodeQL

The first step to running queries with CodeQL was to create a database of the test application that was used in the survey. The following CodeQL CLI’s command was used:

codeql database create database --language=javascript --source-root=app

Next was to import the newly created database to the Vscode’s extension for CodeQL. The queries that were executed against this database are already made and available at either CodeQL’s github page or at lgtm.com. When running the preexisting CodeQL queries for the first time using Vscode, the following error message was displayed: ”You tried to run 74 queries, but the maximum is 20. Try selecting fewer queries or changing the ’codeQL.runningQueries.maxQueries’ setting (codeQL.runQueries)”.

To resolve this error, the Vscode settings were visited. Then the CodeQL extension was clicked and the CodeQL settings were presented. Figure 5.1 shows how to fix the

32 error, where the number of Running Queries: Max Queries could be changed from 20 to any other number. After resolving the error like this, CodeQL was run against the vulnerable app database and the results were later displayed.

Figure 5.1: How to change the maximum number of queries when executing CodeQL queries

Setting up Semgrep

Because Semgrep operates directly on the source code, unlike CodeQL that needs a buildable environment, only Semgrep CLI was used. To be able to run Semgrep rules, the following Semgrep CLI command was executed:

time semgrep --config="https://semgrep.dev/r" > results.txt

The command was executed from the directory where the source code was located. The time command was used to keep track of how long it takes to run all of the preexisting rules in the Semgrep registry.

5.2 Case Study: Part 2

The aim of this part was to compare the detection capability of CodeQL and Semgrep, as well as to test the functionality of the automated code reviewing tools against larger code­bases.

33 Finding code­bases

In order to compare the detection capability of the named tools and test their functionality against large code­bases, some code­bases that contained known vulnerabilities were needed. This part of the case study was performed on two open­source projects, called Damn Vulnerable NodeJS Application (DVNA)[37] and NodeGoat [38], that contained all the OWASP top ten vulnerabilities. These two projects were considered good test application, as they were large enough to check the functionality of both tools.

Running CodeQL against databases

As a first step to run CodeQL against the two applications, the CodeQL databases needed to be created. The following command was executed to create CodeQL database for DVNA project:

codeql database create dvna-database --language=javascript --source-root=dvna

This command created a CodeQL database called dvna­databse from the folder called dvna in the current directory (–source­root flag was used for the path to the source­ code). The following command was run to create CodeQL database for NodeGoat project:

codeql database create node-goat-database --language=javascript --source- root=NodeGoat

This command created a CodeQL database called node­goat­database, from the folder called NodeGoat in the current directory.

After creating the CodeQL databases from the above two named projects, these databases were imported to the Vscode’s extension for CodeQL. A total of 74 preexisting CodeQL security queries for JavaScript were available in CodeQL’s official github repository[36]. These queries were written to detect different security vulnerabilities including OWASP top ten vulnerabilities. By right­clicking the folder containing all these queries in the Vscode, the following was loaded: ”CodeQL: Run the queries in the selected file”. By clicking this option, all the queries located in the folder were executed against the chosen CodeQL database. The history of all the queries that were run against the selected database were shown in the CodeQL extension under the

34 query history tab, see figure 5.2.The results were displayed by clicking on the queries. In the displayed result alerts, such as warnings, errors and informations were found, see figure 5.3. More information about each alert and where in the base­code this alert happened were also provided.

Figure 5.2: The history of all the queries that was run on the selected database

Figure 5.3: The result of the selected query chosen among query history

Running Semgrep against applications

In Semgrep the rules were directly executed against the code­base without the need to create a special database. The following command was used to execute all the 1174 public rules from https://semgrep.dev/r against both projects:

35 semgrep --config="https://semgrep.dev/r" > results.txt

This command was executed in the same directory where the project resided. The results were stored in a file called results.txt within the same directory. The operator > was used to redirect the output. The result contained the severity grade of the bug i.e. warning, info or error. It also gave relevant information about the detected vulnerabilities, a link to the rule that resulted to this alert and the location of the vulnerable code in the code­base (file name and row number).

36 Chapter 6

Case study: Results

This chapter presents the results gathered from the case study, with the purpose of answering the research question.

6.1 Case study: Part 1

The section presents the results of part 1 of the case study. The aim of this part was to compare the results of the questionnaire gained from manual and automated code reviews.

Manual code review result

There were a total of 20 volunteers who partook in this survey. Out of these 20, 8 were security professionals, 6 were developers and the last 6 were students. The result for the manual code review is given in figure 6.1. The upper chart shows the correct answer percentage of each question individually for the manual code review, the lower chart presents the result of the questionnaire per group.

Secure manual code reviewing are usually performed by someone knowledgeable, hence why the results obtained from the security professionals and developers (working with security) were considered the most important out of all the volunteers. The accuracy of these two groups were ranging around 46.66% and 48.33%, whereas the students’ was 28.88%. Together they had an accuracy rate of around 43%.

37 (a) Correct answers per question in the questionnaire

(b) Correct answers per group of volunteers

Figure 6.1: The percentage of correct answers

38 As the table 6.1 shows, three of the questions in the survey contained code­snippets with no security vulnerabilities. When observing the answers of these questions, many of the volunteers reported incorrect warnings. For the second question, 5 participants answered correctly, while the sixth and thirteenth question had a total of 11 and 6 correct answers respectively. These together added up to 22 correct answers out of 60, which corresponded to 38 warnings reported incorrectly.

Table 6.1: Solutions to the survey questions

Question Number Vulnerability Row Number

Q1 Broken Access Control 27/28 Q2 None — Q3 Broken Authentication/XSS 25 Q4 Injection 12 Q5 Cross Site Scripting (XSS) 64 Q6 None — Q7 Security Missconfiguration 41­45/44 Q8 Cross Site Scripting (XSS) 76 Q9 Injection 40­45 Q10 Deserialization 7 Q11 Sensitive Data Exposure 29 Q12 XML External Entities (XXE) 57­60 Q13 None — Q14 Sensitive Data Exposure 37 Q15 XML External Entities (XXE) 47­48

Automated code review results

The results from the tools output were divided into three groups.

• False positives

• Security risks

• Functional faults

The functional faults were not really important to take into account, as they were insignificant to the case study.

39 CodeQL result

The same application as the questionnaire taken by the volunteers was tested and analysed using CodeQL, as an attempt to detect these vulnerabilities. CodeQL found the majority of the security vulnerabilities, except for injection, which corresponded to questions number 4 and 9, see table 6.1.

It took approximately 36 min and 7 sec to run 76 queries. The results were checked and 1 false positive was reported.

Semgrep result

Semgrep had a very similar result to CodeQL. It managed to identify all of the security vulnerabilities in the application, with the exception of sensitive data exposure, which were questions number 11 and 14 in the table 6.1.

Semgrep was executed against the application using a total of 1168 public rules. This process took around 7 min and 21 sec and had the same output accuracy as CodeQL. After further inspection of the result, 2 false positives were reported.

6.2 Case study: Part 2

In this section the results of part 2 of the case study is presented. In part 2, CodeQL and Semgrep were run against two open source projects, called DVNA and NodeGoat. DVNA had 17 vulnerabilities and NodeGoat had 12 vulnerabilities in total.

CodeQL and Semgrep Results for DVNA Project

CodeQL found a total of 6 vulnerabilities out of 17 in DVNA project. Table 6.2 shows this result. The vulnerabilities found included injection, broken authentication, XML external entities (XXE), security misconfiguration and insecure deserialization. The number of queries that was executed against this code­base was 76 and it took approximately 9 min and 25 sec for CodeQL to execute them. CodeQL reported 5 false positives for this project.

Semgrep successfully detected 7 vulnerabilities out of 17 in DVNA project, see table 6.2. It took 6 min and 51 sec for Semgrep to detect these vulnerabilities, out of which 1 was false positive. The vulnerabilities detected by Semgrep included injection,

40 broken authentication, broken access control, cross site scripting (XSS) and insecure deserialization.

Table 6.2: CodeQL and Semgrep’s results for DVNA for OWASP top 10 2017

Vulnerability CodeQL Semgrep

A1: Injection Detected Detected A2: Broken Authentication Detected Detected A3: Sensitive Data Exposure Undetected Undetected A4: XML External Entities (XXE) Detected Undetected A5: Broken Access Control Undetected Detected A6: Security Missconfiguration Detected Undetected A7: Cross Site Scripting (XSS) Undetected Detected A8: Deserialization Detected Detected A9: Using Components with Known Vulnerabilities Undetected Undetected A10: Insufficient Logging and Monitoring Undetected Undetected

CodeQL and Semgrep’s Results for NodeGoat Project

CodeQL managed to detect a total of 8 vulnerabilities out of 12 in NodeGoat project. Three of the vulnerabilities that were detected were false positives. Table 6.3 shows this result. The amount of time taken by this code­scanning tool to execute 76 security queries, was approximately 8 min and 1 sec. The detected vulnerabilities by CodeQL on NodeGoat project included injection, broken authentication, security misconfiguration, sensitive data exposure, cross­site request forgery (CSRF) and unvalidated redirects and forwards.

Semgprep managed to identify 6 security risks out of 12 in NodeGoat project by executing 1174 rules in 12 min and 36 sec. The tool reported even 3 false positive answers. See table 6.3 for these results. The detected vulnerabilities included injection, broken authentication, cross site scripting (XSS), security misconfiguration, broken access control and unvalidated redirects and forwards.

41 Table 6.3: CodeQL and Semgrep’s results of NodeGoat project for OWASP top 10 2013

Vulnerability CodeQL Semgrep

A1: Injection Detected Detected A2: Broken Authentication Detected Detected A3: Cross Site Scripting (XSS) Detected Undetected A4: Security Missconfiguration Detected Detected A5: Sensitive Data Exposure Undetected Detected A6: Broken Access Control Detected Undetected A7: Insecure Direct Object Refrence: Undetected Undetected A8: Cross­site Request Forgery (CSRF) Undetected Detected A9: Using Known Vulnerable Components Undetected Undetected A10: Insecure Redirects Detected Detected

42 Chapter 7

Discussion

This chapter analyses the result of the research as well as the methods used. It then attempts to answer the research question of the thesis and suggests further research. Additionally, sustainability aspects associated to the thesis work are discussed.

7.1 Answering the research sub­questions

This section aims to answer the main research question by answering each research sub­question.

What are the prerequisites needed to use automated code­ scanning tools?

In order to scan any code with any code­scanning tool to detect security vulnerabilities, one has to be familiar with the concept of security vulnerabilities, how they occur and how to prevent them. This is because one should analyze the output from these code­ scanning tools and determine if they really are vulnerabilities or simply a false positive. One should even know how to fix these vulnerabilities, after finding them with the scanning tools. So knowledge about security is one of the fundamental prerequisites of using automated code scanning tools.

Based on the literature study, various prerequisites can be identified before using automated static analysis tools. When it comes to Semgrep, the prerequisites for just running the rules were to the point. This included getting familiar with Semegrep CLI

43 and installing it and executing the rules against the code­bases from the Command Line Interface. For writing rules, it was enough that one is familiar with functional programming like Python or Elixir, as it involves pattern matching, and analyzing data flow in different functions.

On the other hand prerequisites required for CodeQL were more complicated. As the CodeQL queries are similar to SQL queries, they are executed over CodeQL databases, which are relational databases extracted from code­base, see section 4.2. After learning all the commands in CodeQL and how they could be executed, the database had to be created from the code­base. Then the database had to be imported in Visual Studio Code’s extension for CodeQL. When all these steps are done, the codeQL queries can be executed. Furthermore, writing own CodeQL rules was also a very complicated process as CodeQL is an independent, object oriented query language, which not only includes rules for pattern matching, but also requires significant knowledge about data flow. For writing CodeQL queries it was required to understand advanced topics, such as predicates, SQL­like queries, types, modules, aliases, variables, expressions, formulas, annotations, recursion, lexical syntax and name resolution, which are very different from other normal object oriented programming languages.

What is the accuracy of automated code reviews in comparison to manual code reviews?

The answer to this question was reached through the conducted case study. The results gathered from the study indicated that the precision of automated code reviews is much higher than manual code reviews. To compare the performance accuracy of automated and manual code reviews, a questionnaire was created that consisted of questions with vulnerable code­snippets. This test application was tested by both automated code reviewing tools as well as a group of volunteers. Both of CodeQL and Semgrep had a high output accuracy at around 87% where CodeQL missed to report the injection warnings and Semgrep missed the sensitive data exposure warnings. Manual code review however only succeeded in reporting a total of 43% correct answers. CodeQL and Semgrep had a total of one and two false positive respectively, whereas manual code review incorrectly reported 38 warnings. Based on the test application’s results that were gathered, automated code reviews do in fact have higher accuracy in comparison to manual code review.

44 An additional question was added to the questionnaire where the authors asked the volunteers what they thought was the most easiest/difficult when they were performing the manual code review. Some complained about their limited knowledge about JavaScript, while others had difficulties knowing how various libraries work. Additionally, they thought it was difficult to manually review an application when they don’t know it’s purpose and behaviour. This is something that the automated code reviewing tools would not have a problem with. For more insights about the feedback from the volunteers, see Appendix B.

How well do code reviewing tools perform with large code­ bases?

The case study showed that the performance of both code reviewing tools were similar when run against large code­bases. The CodeQL analysing tool had a 35% success rate in detecting known vulnerabilities in the DVNA test applications and a 67% success rate when run against the NodeGoat test applications. However, the Semgrep tool detected around 41% in the DVNA application and 50% of the vulnerabilities when analysing NodeGoat application. This was deemed much worse than when they were tested against the questionnaire’s test application, which was considered a relatively small application. It took around the same time to run both tools against the two chosen test applications.

What are the costs of using code reviewing tools?

CodeQL is licensed under OSI (Open Source Initiative)­approved license. This license implies that it is free to use the tool for open­source projects, but it is not permitted to use this tool for commercial projects. So for commercial use one has to contact the sales team of the organizers i.e. github. It is free to use Semgrep for both open­source and commercial projects, as it is licensed under Lesser General Public License(LGPL) v2.1.

When it comes to understanding and getting started with these scanning tools, CodeQL cost more time than Semgrep, as it was a more advanced and complicated query language. Table 7.1 shows the time it took for the authors to get started with both vulnerability scanning tools. The reported time however does not include writing own queries in CodeQL and rules in Semgrep. As they are more complicated and they need

45 a deeper understanding of both tools, it may take even more time. Writing quesries and rules was not a requirement for this project, but nonetheless, it is a cost that the companies and organizations that are interested in using the tools should take into account.

Table 7.1: Time costs of getting started with each vulnerability detecting tool for authors

Tools Work hours spent in Work hours spent on research analysing test applications CodeQL 70 6 Semgrep 25 7

Result summary

Based on the results of the case study conducted in this thesis, the following conclusions can be drawn.

Benefits of Automated Code Reviewing Tools

• Automated code reviewing tools can detect vulnerabilities to a certain accepted extent.

• Automated code reviewing tools reported less false positive warnings in comparison with the manual code review.

• Automated code reviewing tools are fast and can scan large amount of code in a reasonable amount of time.

• Automated code reviewing tools examined in this thesis allows for writing and executing custom rules, which give them the flexibility of having great potential in finding more vulnerabilities in the future.

Drawbacks of Automated Code Reviewing Tools

• Automated code reviewing tools can not detect all the vulnerabilities, thus they are not 100% reliable, and manual reviews need to complement them.

• There are some vulnerabilities such as using components with known vulnerabilities and insufficient logging and monitoring, which can’t be detected

46 by automated code reviewing tools. This is because they do not follow any certain pattern, and thus it is hard for automated scanning tools to detect them. This kind of vulnerabilities need human intelligence to be detected, so they should be either detected by manual reviews or, the developers should be aware of them, and therefore should be more careful while developing the code.

• CodeQL and Semgrep can be used for analysing web applications. However CodeQL is not user friendly for beginners and requires time and experience to get familiar with it.

7.2 Methods

In this section, the project model and methods used during the project are discussed, followed by an evaluation of validity and reliability.

Project model

The project model that was used for the thesis compromised of an iterative approach as well as priorities that were set with the help of the MoSCoW method. The methodology used in this project happened to be suitable, since new tools and environments were investigated. Throughout the project’s process, some difficulties and problems were experienced that hindered the research procedure. The iterative approach facilitated keeping track of the different goals, such as adaptation of workload, deadlines, and tracing back to different parts of the research phases.

Furthermore, the MoSCoW method made it possible to prioritize different requirements and understand their significance to the project.

Validity

Validity is a term that refers to whether the results accurately measures whatever it was supposed to measure, which can later be used to answer the research question[39]. Concepts like security vulnerabilities and automated code reviewing tools were both novel and unknown. Therefore, a literature study was conducted in order to obtain more knowledge and a better understanding of the purpose as well as the problem. Moreover, the literature study contributed to facilitating the prioritization process of

47 the different requirements for the project. Additionally, it helped answering the first research sub­question with the identified prerequisites that were needed, in order to use automated code reviewing tools.

After obtaining as much knowledge as possible on the concepts and the problems, the case study iteratively provided results to find a solution, as well as one can. Since the main aim of the case study was to compare automated code reviewing tools to each other, and against manual code reviewing, it is fair to say that the method of research is considered to be valid.

Reliability

The term reliability refers to whether the same results would be achieved if the study was to be performed in a similar environment. This is to verify and ensure the resea­ rch method’s trustworthiness and applicability[39]. For the project’s case study, the automated code reviewing tools were tested by implementing three different open source test applications, as an attempt to answer the research question. The results gathered from this study are not subjective, since the authors did not develop any of these applications. Therefore there will be no problem repeating the study in a similar environment.

Sustainability

This thesis was aimed at computer science students, developers, security professionals or anyone interested and curious about automated secure code reviewing tools for web­applications. The project’s goal was to investigate the benefits and drawbacks of automated code reviewing tools and see if it’s possible for developers, as well as security professionals to easier detect security vulnerabilities as early in the development stage as possible. Within the scope of the thesis, there were no additional resources needed, such as hardware. Furthermore, there were no consumption of resources, hence why the project should not have any effect on the environment, and if it does, it should be a minimal impact.

Engineers need to be trained in order to be able to perform manual code reviewing. It takes up to a full year of learning and experience before they are considered ready to effectively perform manual secure code reviews. Code reviewing is a process that requires patience and skills. It is a task that takes a long time and therefore can

48 be very expensive[29]. Automated code review also requires training, which might as well take a full year. Nonetheless, the process of using them later is much faster than if the reviewings were performed manually, hence why it should be considered as economically sustainable.

7.3 Limitations

One of the limitations for this thesis was that the thesis was done only on web­ applications written in JavaScript and NodeJS. Because the scanning tools examined in this thesis have the capabilities to scan several programming languages, it is a good practice to examine them all. Because of the time limitation of the thesis, it was not possible for the authors to examine all the supported languages. This limitation could affect the result of the thesis, as the tools might be able to detect more or less security vulnerabilities in other programming languages.

Another limitation for the thesis was the limited amount of code scanned with the automated reviewing tools. As the authors had to find projects that included inten­ tional and known number of bugs in order to be able to present and analyze the result, they could not scan any code­base they could find.

One other limitation that could affect the result of the thesis is the number of people who participated in the survey, and the way they answered the survey questions. As they were volunteers, it could be possible that they were not taking the survey very serious. So if the authors had the resource to hire a large number of developers and security professionals to participate in the survey, the result of the manual code review could be somehow more reliable.

The last limitation that the authors had considered was the number of automated reviewing tools. There are several more secure automated reviewing tools out there that could perform better or worse than the examined tools. As mentioned before due to time limitation of the thesis work, the authors had to invest their time in limited resources in order to be able to analyze them well enough to draw an accurate conclusion.

49 7.4 Summary

The purpose of this thesis was to get a better understanding on what automated code reviewing tools are, and to investigate their benefits and drawbacks when used against web­applications. The literature study helped gain more awareness and insight on the different risks that could occur when developing web­applications. It further facilitated writing down a guide to the different prerequisites that need to be identified in order to use the automated code reviewing tools.

The case study aimed to test two automated code reviewing tools against three different open source test applications and was divided into two stages. The results gathered from the case study were later evaluated in the light of how the tools can be used against web­applications and their feasibility. Thereafter, the research sub­questions as well as the main research question were answered by discussing and analysing the results collected throughout the project’s process.

7.5 Future Work

According to the researches and experiments done on the automated code reviewing tools used in the thesis, it is concluded that these tools have great potential, since both CodeQL and Semgrep have the capabilities of pattern matching and analyzing data­flows. Furthermore both the tools provided the functionality of writing own custom rules and queries, specific to own project. These tools also supported other programming languages. So, the authors of the thesis encourage the readers and those who are interested in the subject of the degree project to continue the research by examining more specific rules and queries based on the projects they are scanning. They are also encouraged to test the tools on different languages and compare and analyze the outputed results.

50 References

[1] ”The ‘digital world’. What does it mean?”, Teaching in a digital world, 2015. [Online]. Available: https://donnakeenan.wordpress.com/2015/03/24/the-digit al-world-what-does-it-mean/ [Accessed: 28­ Jan­ 2021].

[2] M. Uma and G. Padmavathi, ”A Survey on Various Cyber Attacks and Their Classification”, International Journal of Network Security, Vol.15, No.5, PP.390­396, 2013. [Online]. Available: http://ijns.jalaxy.com.tw/contents/ijns-v15-n5/ij ns-2013-v15-n5-p390-396.pdf [Accessed: 28­ Jan­ 2021]

[3] J. Salter, ”Sourcegraph: Devs are managing 100x more code now than they did in 2010”, Ars Technica, 2021. [Online]. Available: https://arstechnica.com/gadgets/2020/10/sourcegraph-devs-are-man aging-100x-more-code-now-than-they-did-in-2010/ [Accessed: 08­ Jan­ 2021].

[4] ”OWASP Top Ten Web Application Security Risks”, Owasp.org, 2021. [On­ line]. Available: https://owasp.org/www-project-top-ten/ [Accessed: 10­ Jan­ 2021].

[5] ”CWE ­ Common Weakness Enumeration”, Cwe.mitre.org, 2021. [Online]. Available: https://cwe.mitre.org/ [Accessed: 10­ Jan­ 2021].

[6] N. Alexopoulos, S. Mahbub Habib, S. Schulz and M. Mühlhäuser, ”The Tip of the Iceberg: On the Merits of Finding Security Bugs”, ACM Trans­ actions on Privacy and Security: Vol 24, No 1, 2020. [Online]. Available: https://dl.acm.org/doi/abs/10.1145/3406112 [Accessed: 05­ Feb­ 2021].

51 [7] Synopsys Team, ”What is the secure software development life cy­ cle (SDLC)?”, Software Integrity Blog, 2020. [Online]. Available: https://www.synopsys.com/blogs/software-security/secure-sdlc/ [Ac­ cessed: 12­ Feb­ 2021].

[8] A. E. Hassan, S. McIntosh, Y. Kamei and B. Adams, ”The impact of code review coverage and code review participation on software quality: a case study of the Qt, VTK, and ITK projects”, Proceedings of the 11th Working Conference on Mining Software Repositories, 2014. [Online]. Available: https://dl.acm.org/doi/10.1145/2597073.2597076 [Accessed: 15­ Feb­ 2021].

[9] ”Sample Secure Code Review Report”, The MITRE Corporation, 2014. [On­ line]. Available: https://www.mitre.org/publications/all/sample-secure-code- review-report [Accessed: 22­ Feb­ 2021].

[10] A. Bacchelli, C. Bird, “Expectations, outcomes, and challenges of modern code review”, Proc. Int. Conf. on Soft. Eng.p, ICSE ’13, pages 712–721, 2013. [Online]. Available: https://sback.it/publications/icse2013.pdf [Accessed: 22­ Feb­ 2021].

[11] J. Czerwonka, M. Greiler and J. Tilford, ”Code reviews do not find bugs: how the current code review best practice slows us downs”, Proc. Int. Conf. on Soft. Eng.p, ICSE ’15, Volume 2, pages 28–29, 2015. [Online]. Available: https://dl.acm.org/doi/10.5555/2819009.2819015 [Accessed: 24­ Feb­ 2021].

[12] A. Buttner, R. Piazza, A. Summers and R. Purohit, ”A Secure Code Review Retrospective”, IEEE Secure Development (SecDev), pages 31­21, 2020. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9230284 [Accessed: 24­ Feb­ 2021].

[13] A. Buttner, ”Using Automated Static Analysis Tools for Code Re­ views”, The MITRE Corporation, 2013. [Online]. Available: h t t p s : //www.mitre.org/capabilities/cybersecurity/overview/cybersecurit

52 y-blog/using-automated-static-analysis-tools-for. [Accessed: 25­ Feb­ 2021]

[14] A. Buttner, ”The Importance of Manual Secure Code Review”, The MITRE Corporation, 2014. [Online]. Available: https://www.mitre.org/capabilities/cyb ersecurity/overview/cybersecurity-blog/the-importance-of-manual-secure-c ode-review [Accessed: 25­ Feb­ 2021].

[15] Microsoft Security Team, ”Microsoft open sources CodeQL queries used to hunt for Solorigate activity”, Microsoft Security, 2021. [Online]. Available: https://www.microsoft.com/security/blog/2021/02/25/microsoft-open-sour ces-codeql-queries-used-to-hunt-for-solorigate-activity/. [Accessed: 04­ Mar­ 2021].

[16] E. Woollacott, ”Semgrep: Static code analysis tool helps ‘eliminate entire classes of vulnerabilities’”, The Daily Swig Cybersecurity news and views, 2020. [Online]. Available: https://portswigger.net/daily-swig/semgrep-static- code-analysis-tool-helps-eliminate-entire-classes-of-vulnerabilities [Accessed: 04­ Mar­ 2021].

[17] ”Semgrep A Practical Introduction”, NotSoSecure part of claranet cyber security, 2020. [Online]. Available: https://notsosecure.com/semgrep-a-practi cal-introduction/ [Accessed: 04­ Mar­ 2021].

[18] ”semgrep”, PyPI, 2020. [Online]. Available: https://pypi.org/project /semgrep/0.15.0b1/ [Accessed: 05­ Mar­ 2021].

[19] M. Goodrich and R. Tamassia, Introduction to Computer Security: Pear­ son New International Edition. Harlow: Pearson Education Limited, chapter 1 page 2, 2014. [20] G. Mcgraw, ”Automated Code Review Tools for Security”, IEEE Computer, Volume: 41, Issue: 12, pages 108­111, 2008. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/4712512. [Accessed: 08­ Mar­ 2021].

[21] ”About CodeQL — CodeQL”, Codeql.github.com, 2021. [Online]. Avail­

53 able: https://codeql.github.com/docs/codeql-overview/about-codeql/. [Accessed: 08­ Mar­ 2021].

[22] ”QL language reference — CodeQL”, Codeql.github.com, 2021. [Online]. Available: https://codeql.github.com/docs/ql-language-reference/. [Accessed: 09­ Mar­ 2021].

[23] ”Home ­ Semgrep Docs”, Semgrep.dev, 2021. [Online]. Available: https://semgrep.dev/docs. [Accessed: 10­ Mar­ 2021].

[24] R. Kent, ”Introduction to variant analysis with QL and LGTM (part 2)”, Blog.semmle.com, 2019. [Online]. Available: https://blog.semmle.com/introduc tion-to-variant-analysis-part-2/ [Accessed: 12­ Mar­ 2021].

[25] V. Gite, ”How To Use grep Command In Linux / UNIX”, Cybercity.biz, 2021. [Online]. Available: https://www.cyberciti.biz/faq/howto-use-grep-com mand-in-linux-unix/ [Accessed: 12­ Mar­ 2021].

[26] J. Jones, “Abstract Syntax Tree Implementation Idioms,” Pattern Languages of Program Design, 2003. [Online]. Available: h t t p s : //hillside.net/plop/plop2003/Papers/Jones-ImplementingASTs.pdf. [Accessed: 12­ Mar­ 2021].

[27] MDN contributors, ”JavaScript”, Developer.mozilla.org, 2020. [Online]. Available: https://developer.mozilla.org/en-US/docs/Glossary/JavaScript. [Accessed: 21­ Mar­ 2021].

[28] MDN contributors, ”A re­introduction to JavaScript”, Developer.mozilla.org, 2021. [Online]. Available: https://developer.mozilla.org/en-US/docs/Web/JavaS cript/A_re-introduction_to_JavaScript. [Accessed: 21­ Mar­ 2021].

[29] D. Baca, B. Carlsson and L. Lundberg, ”Evaluating the cost reduction of static code analysis for software security”, Proceedings of the third ACM SIGPLAN workshop on Programming languages and analysis for security, pages 79­88, 2008.

54 [Online]. Available: https://dl.acm.org/doi/pdf/10.1145/1375696.1375707 [Accessed: 01­ Apr­ 2021].

[30] ”Quantitative Data Collection Methods”, Research­Methodology. [Online]. Available: https://research-methodology.net/research-methods/quantitative -research/ [Accessed: 01­ Apr­ 2021].

[31] ”Qualitative Data Collection Methods”, Research­Methodology. [Online]. Available: https://research-methodology.net/research-methods/qualitative- research/ [Accessed: 02­ Apr­ 2021].

[32] A. Håkansson, ‘Portal of Research Methods and Methodologies for Re­ search Projects and Degree Projects’, Proceedings of the International Conference on Frontiers in Education: Computer Science and Computer Engineering FECS’13, pp. 67–73, 2013. [Online]. Available: https://www.diva-portal.org/smash/get/diva2: 677684/FULLTEXT02.pdf [Accessed: 02­ Apr­ 2021].

[33] ”Inductive Approach (Inductive Reasoning)”, Research­Methodology. [Online]. Available: https://research-methodology.net/research-methodology/research -approach/inductive-approach-2/ [Accessed: 06­ Apr­ 2021].

[34] ”Deductive Approach (Deductive Reasoning)”, Research­Methodology. [Online]. Available: https://research-methodology.net/research-methodology/ research-approach/inductive-approach-2/ [Accessed: 06­ Apr­ 2021].

[35] ”Getting started with the CodeQL CLI”, Codeql.github.com, 2021. [On­ line]. Available: https://codeql.github.com/docs/codeql-cli/getting-started- with-the-codeql-cli/#getting-started-with-the-codeql-cli [Accessed: 22­ Apr­ 2021].

[36] ”github/codeql”, GitHub. [Online]. Available: https://github.com/git hub/codeql/tree/main/javascript/ql/src/Security [Accessed: 20­ Apr­ 2021].

[37] ”appsecco/dvna”, GitHub. [Online]. Available: https://github.com/app

55 secco/dvna [Accessed: 06­ Apr­ 2021].

[38] ”OWASP/NodeGoat”, GitHub. [Online]. Available: https://github.com /OWASP/NodeGoat [Accessed: 10­ Apr­ 2021].

[39] N. Golafshani, ”Understanding reliability and validity in qualitative re­ search”, The Qualitative Report, 8(4): pages 597–606, 2003. [Online]. Available: http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf [Accessed: 20­May­2021]

[40] ”Abstract syntax tree ­ Wikipedia”, En.wikipedia.org. [Online]. Available: https://en.wikipedia.org/wiki/Abstract_syntax_tree#/media/File: Abstract_syntax_tree_for_Euclidean_algorithm.svg [Accessed: 20­ Feb­ 2021].

[41] ”Introduction to variant analysis with QL and LGTM (part 2)”, Blog.semmle.com. [Online]. Available: https://blog.semmle.com/introduction-to-variant-analys is-part-2/ [Accessed: 20­ Feb­ 2021].

56 5/28/2021 Appendix A

Appendix A

Questionnaire The purpose of this questionnaire is to collect data. Your participation in this study is completely voluntary and your answers will be anonymous. Please do not use any automated code review tools or Google, in order to help us with an unbiased research.

Top 10 Web Application Security Risks according to OWASP

The following are the security vulnerability risks that may or may not be present in this questionnaire:

1. Injection

2. Broken Authentication

3. Sensitive Data Exposure

4. XML External Entities (XXE)

5. Broken Access Control

6. Security Misconfiguration

7. Cross Site Scripting (XSS)

8. Insecure Deserialization

9. Using Components with Known Vulnerabilities

10. Insufficient Logging & Monitoring

INSTRUCTIONS For every question, starting from question 3, make sure to write down the name of the vulnerability risk and state where in the code it occurs. Write the row number or the range.

An answer can look like this: (Insufficient Logging & Monitoring, row 105-107).

If you think that the code is OK and there are no security vulnerabilities, write "none" as an answer.

PS! This applies to all the following questions.

5/28/2021 Appendix A

1. Please choose one of the following:

Student Developer Security professional

2. Which organization do you belong to? (optional)

3. Question 1

5/28/2021 Appendix A

4. Question 2

5. Question 3

5/28/2021 Appendix A

6. Question 4

5/28/2021 Appendix A

7. Question 5

5/28/2021 Appendix A

8. Question 6

5/28/2021 Appendix A

9. Question 7

5/28/2021 Appendix A

10. Question 8

11. Question 9

5/28/2021 Appendix A

12. Question 10

13. Question 11

5/28/2021 Appendix A

14. Question 12

15. Question 13

5/28/2021 Appendix A

16. Question 14

17. Question 15

5/28/2021 Appendix A

18. What are your thoughts about these questions? Which bug was easiest/most difficult to detect?

This content is neither created nor endorsed by Google.

Forms

Appendix B

Comments about the questionnaire

Volunteer Occupation Thoughts about the questionnaire and which bug was Number easiest/most difficult to detect?

1 Security Professional Easiest: 4. Hardest: 6, 7. I’ve never used Node.JS so that makes it more difficult.

2 Security Professional Question 9 was easiest, and question 2, 6, 13 and 14 was most difficult.

3 Security Professional Don’t have deep knowledge about node, not using google to lookup the modules/methods means I don’t know the details of what they do...

4 Security Professional —

5 Security Professional Not using google to lookup the modules/methods means I don’t know the details of what they actually do

6 Developer sql injection was the easiest. in my opinion this might be because I feel sql injections is the most known vulnerability. I dont really know much about ”insufficient loggnig and monitoring” as none of these examples include neither logging nor monitoring so I guess they all have that problem :D

7 Developer If using new libraries or parameters, I always do research (googling) about how to use them. A large portion of the time developing is information gathering. Therefor it can be hard to answer questions about use cases not seen before. And I must admit that I did some searches during this questionnaire too.

8 Security Professionals Most of the bugs are situational and all depends on who can access the parameters. Also, not having the full code and not being able to look up suspected code/modules limits my accuracy

9 Student I think it would have been helpful with a short text explaining what the intended behavior of the code is. For example, the first code segment seems to allow anyone to read any file on the host using both row 27 and 28. But I can only assume that it was a vulnerability and not it’s intended function. All bugs were quite difficult to detect, but I’ll add that I am not very fluent in Javascript.

10 Security Professional Anything relating to nodejs­specific libraries is difficult for me as it’s not my area

11 Security professional Some bugs are too obvious to spot because the code snippets are very small. Bugs get harder to detect later on. SQLi was the easiest to spot (Q4).

12 Student It was pretty hard to find the vulnerabilities, a lot of the code was hard to understand when you don’t know the specific libraries.

13 Student The code was hard to understand when the whole picture was not available which led to harder understanding and detecting of the bugs.

14 Student Difficult questions, 4 was easiest.

15 Student As a student with limited experience in the web security, most bugs were difficult to detect. The easiest bugs to detect were those that exposed sensitive data such as questions 11. My general thoughts on the questions are that they are well thought­out, i.e the bugs weren’t super obvious and also varied in security risks.

16 Student A bit difficult

17 Developer a little bit difficult.

18 Developer Had some difficulties as I don’t actually know what some of these libraries do.

19 Developer Difficult since I don’t know what the code does.

20 Developer question 4 was the easiest, but difficult to know the functionality of the code.

69 70 71 TRITA-EECS-EX-2021:294

www.kth.se