Empirical Study of Vulnerability Scanning Tools for Javascript Work in Progress

Empirical Study of Vulnerability Scanning Tools for JavaScript Work In Progress

Tiago Brito, Nuno Santos, José Fragoso INESC-ID Lisbon 2020

Tiago Brito, GSD Meeting - 30/07/2020 Purpose of this WIP presentation

● Current work is to be submitted this year

● Goal: gather feedback on work so far

● Focus on presenting the approach and preliminary results

Tiago Brito, GSD Meeting - 30/07/2020 2 Motivation

● JavaScript is hugely popular for web development ○ For both client and server-side (NodeJS) development

● There are many critical vulnerabilities reported for software developed using NodeJS ○ Remote Code Executions (Staicu NDSS’18) ○ Denial of Service (Staicu Sec’18) ○ Small number of packages, big impact (Zimmermann Sec’19)

● Developers need tools to help them detect problems ○ They are pressured to focus on delivering features

Tiago Brito, GSD Meeting - 30/07/2020 3 Problem

Previous work focused on:

● Tools for vulnerability analysis in Java or PHP code (e.g. Alhuzali Sec’18)

● Studying very speciﬁc vulnerabilities in Server-side JavaScript ○ ReDos, Command Injections (Staicu NDSS’18 and Staicu Sec’18)

● Studying vulnerability reports on the NodeJS ecosystem (Zimmermann Sec’19)

So, it is still unknown which, and how many, of these tools can eﬀectively detect vulnerabilities in modern JavaScript.

Tiago Brito, GSD Meeting - 30/07/2020 4 Goal

Our goal is to assess the eﬀectiveness of state-of-the-art vulnerability detection tools for JavaScript code by performing a comprehensive empirical study.

Tiago Brito, GSD Meeting - 30/07/2020 5 Research Questions

1. [Tools] Which tools exist for JavaScript vulnerability detection?

2. [Approach] What’s the approach these tools use and their main challenges for detecting vulnerabilities?

3. [Eﬀectiveness] What is the eﬀectiveness of these tools in detecting vulnerabilities?

Tiago Brito, GSD Meeting - 30/07/2020 6 Expected Contributions

1. Qualitative evaluation of JS vulnerability analysis tools in full blown (known) vulnerable web applications (RQ2)

2. Qualitative evaluation of JS vulnerability analysis tools against real-world vulnerabilities in JavaScript packages (RQ3)

3. Annotated dataset of JavaScript code with known vulnerabilities (RQ3)

Tiago Brito, GSD Meeting - 30/07/2020 7 Empirical Study - 2 Steps

● [Study 1] - How do they do it? (Approach)

● [Study 2] - Do they work? (Eﬀectiveness)

Tiago Brito, GSD Meeting - 30/07/2020 8 Study 1 - Our approach

● Collect a set of analysis tools ○ Criteria: 1) Available, 2) CLI, 3) Code Analysis, 4) Vulnerability Detection ○ Academic tools, Open-source Popular tools, Commercial tools, etc.

● Collect a set of Known Vulnerable Applications ○ Web applications written in NodeJS that have known vulnerabilities ○ Purposely used to teach web security and used as a benchmark in some previous work

● Run all collected tools against all collected applications

Tiago Brito, GSD Meeting - 30/07/2020 9 Study 1 - Our approach

● Tools: ○ NodeJsScan/njsscan/SemGrep ○ Github’s CodeQL ○ Other tools exists, but we have not tested them yet

● Applications with known vulnerabilities ○ We collected 7 diﬀerent applications ○ Most popular: ■ Damn Vulnerable Node Application (DVNA) ■ OWASP NodeGoat ■ VulnerableNode ■ ...

Tiago Brito, GSD Meeting - 30/07/2020 10 Study 1 - How do they do it?

● OWASP Top 10: ○ A1 - Injection ○ A6 - Security Misconﬁguration ○ A2 - Broken Auth ○ A7 - XSS ○ A3 - Data Exposure ○ A8 - Deserialization ○ A4 - XXE ○ A9 - Known Vulnerable Component ○ A5 - Broken Access ○ A10 - No Logging

DVNA A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 Other Total s

# Vulns 2 2 2 1 2 2 3 1 1 NA 2 18

NodeJsScan 1 0 0 0 0 0 2 1 0 NA 1 5 (28%)

CodeQL 2 1 0 0 0 0 0 0 0 NA 2 5 (28%)

Tiago Brito, GSD Meeting - 30/07/2020 11 Study 1 - How do they do it? ● NodeJsScan is rule-based ● CodeQL models code into graphs and performs graph queries on it

● Rules/graph queries describe flow conditions seen in previous vulnerabilities ○ Matches new vulnerabilities with similar flow patterns from specific sources to specific sinks

There are 5 main approach takeaways: 1. Correctly implemented rules 2. Over speciﬁc (overﬁtting) rules 3. Unmodelled Sources, Sinks and Dependencies 4. Unmodelled Context 5. Unmodelled Languages/Interactions

Tiago Brito, GSD Meeting - 30/07/2020 12 Study 1 - Correctly implemented rules

var query = "SELECT name FROM Users WHERE login='" + req.body.login + "'"; db.sequelize.query(query,{ model: db.User }).then(user => { … });

rules: NodeJsScan SQL injection rule: - id: node_sqli_injection patterns: - pattern-either: - pattern: | $CON.query(<... $REQ.$QUERY.$VAR ...>, ...) - pattern: | $CON.query(<... $REQ.$QUERY ...>, ...) - pattern: | var $SQL = <... $REQ.$QUERY.$VAR ...>; ... $CON.query(<... $SQL ...>, ...); - pattern: | var $SQL = <... $REQ.$QUERY ...>; ... $CON.query(<... $SQL ...>, ...); (...)

Tiago Brito, GSD Meeting - 30/07/2020 13 Study 1 - Over speciﬁc rules (overﬁtting)

const exec = require('child_process').exec; exec('ping -c 2 '+ req.body.address,(err,stdout,stderr) => { … });

NodeJsScan Command injection rule: rules: - id: generic_os_command_exec const app = express(); patterns: - pattern-inside: | // Routing var $EXEC = require('child_process'); app.use(‘/ping’, function (req, res) { ... const execP = require(‘child_process’); - pattern-inside: | execP.exec( $APP.$METHOD(..., function $FUNC($REQ, $RES, ...){ ... }); ‘ping -c 2 ‘+ req.body.address, - pattern: | (err,stdout,stderr) => { … } $EXEC.exec(..., <... $REQ.$QUERY ...>, ...) ); }); (...)

Tiago Brito, GSD Meeting - 30/07/2020 14 Study 1 - Unmodelled Sources, Sinks and Dependencies

function (req,res){ if(req.files.products) { var products = serialize.unserialize(req.files.products.data.toString('utf8')) (...)

NodeJsScan Command injection rule: rules: - id: node_deserialize patterns: - pattern-inside: | require('node-serialize'); CodeQL only models the js-yaml ... - pattern: | package. Thus it misses this particular $X.unserialize(...) - id: yaml_deserialize patterns: vulnerable snippet - pattern-inside: | require('js-yaml'); ... - pattern: | $X.load(...)

Tiago Brito, GSD Meeting - 30/07/2020 15 Study 1 - Unmodelled Context db.User.findAll({}).then(users => { db.User.findAll({attributes: [ 'id' ,'name', 'email']},).then(users => { res.status(200).json({ res.status(200).json({ success: true, success: true, users: users users: users }); }); }); });

● Vulnerabilities exist even in ‘correct’ code; Tools miss them without proper context

● The users structure contains sensitive data accessible to everybody at this endpoint

● Deﬁnitely possible to detect these vulnerabilities using taint tracking

● Tools need to know which resources can be accessible and which data is sensitive

16 Tiago Brito, GSD Meeting - 30/07/2020 Study 1 - Main Takeaways

● OWASP Top 10: ○ A1 - Injection ○ A6 - Security Misconfiguration Failure category legend: ○ A2 - Broken Auth ○ A7 - XSS Over specific (overfitting) rules ○ A3 - Data Exposure ○ A8 - Deserialization Unmodeled Sources, Sinks and Dependencies ○ A4 - XXE ○ A9 - Known Vulnerable Component Unmodeled Context ○ A5 - Broken Access ○ A10 - No Logging Unmodelled Languages/Interactions

DVNA A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 Others Total

# Vulns 2 2 2 1 2 2 3 1 1 NA 2 18

NodeJsScan 1 0 0 0 0 0 2 1 0 NA 1 5 (28%)

CodeQL 2 1 0 0 0 0 0 0 0 NA 2 5 (28%)

Tiago Brito, GSD Meeting - 30/07/2020 17 Study 2 - Our approach to study effectiveness

● Build a curated dataset of NodeJS vulnerabilities ○ Collect all vulnerable versions of packages in npm security reports ○ Create a dataset of annotated vulnerabilities using the snippets

● Run all collected tools against each snippet ○ Check if the results include the reported vulnerability ○ Assess detection rates (TP/FP/FN)

Tiago Brito, GSD Meeting - 30/07/2020 18 Study 2 - Effectiveness (Curated Dataset)

● There are 1550+ advisories for npm ○ Of which 1350 have available code ○ There are other vulnerability DBs for npm we may look at, such as Snyk’s Vulnerability DB, NVD/CVE

● Challenges with looking at npm advisories: ○ Advisories lack information on the vulnerable code ○ External references do not follow a particular structure ○ Analysis has to be done manually

Tiago Brito, GSD Meeting - 30/07/2020 19 Study 2 - Effectiveness (Preliminary Results) CWE # Advisories # NodeJsScan # CodeQL Percentage (Max) CWE-506 - Embedded Malicious Code 405 0 0 0.0 %

CWE-22 - Path Traversal 156 0 109 69.9 %

CWE-79 - Cross-site Scripting 127 11 34 26.8 %

CWE-400 - Uncontrolled Resource Consumption 77 3 0 3.9 %

CWE-471 - Modification of Assumed-Immutable Data (MAID) 60 0 23 38.3 %

CWE-78 - OS Command Injection 43 1 33 76.7 %

CWE-94 - Code Injection 34 0 0 0.0 %

CWE-20 - Improper Input Validation 26 0 1 3.8 %

CWE-200 - Exposure of Sensitive Information to an Unauthorized Actor 22 0 0 0.0 %

CWE-89 - SQL Injection 20 15 0 75.0 %

Other CWEs 380 8 152 40.0 %

Total 1350 38 (2.8 %) 352 (26.1 %) 26.1 %

Tiago Brito, GSD Meeting - 30/07/2020 20 Study 2 - Effectiveness (Preliminary Results)

Using both: 28.1 %

Tiago Brito, GSD Meeting - 30/07/2020 21 Study 2 - Effectiveness (Preliminary Results)

NodeJsScan Top 5 detections by CWE (percentage of detections):

CWE # Advisories # NodeJsScan Percentage (Max)

CWE-89 - SQL Injection 20 15 75.0 %

CWE-185 - Incorrect Regular Expression 3 2 66.7 %

CWE-295 - Improper Certificate Validation 2 1 50.0 %

CWE-918 - Server-Side Request Forgery (SSRF) 2 1 50.0 %

CWE-943 - Improper Neutralization of Special Elements in Data Query Logic 5 2 40.0%

Total 32 (2.4 % total) 21 (65.6 %)

Tiago Brito, GSD Meeting - 30/07/2020 22 Study 2 - Effectiveness (Preliminary Results)

CodeQL Top 5 detections by CWE (percentage of detections):

CWE # Advisories # CodeQL Percentage (Max)

CWE-26 - Path Traversal: '/dir/../filename' 10 10 100.0 %

CWE-80 - Improper Neutralization of Script-Related HTML Tags in a Web Page (Basic XSS) 6 6 100.0 %

CWE-23 - Relative Path Traversal 6 5 83.3 %

CWE-25 - Path Traversal: '/../filedir' 13 10 76.9 %

CWE-78 - OS Command Injection 43 33 76.7 %

Total 78 (5.8 % total) 64 (82.1 %)

Tiago Brito, GSD Meeting - 30/07/2020 23 Study 2 - Main Takeaways

● Current JS vulnerability detection tools are ineﬀective ○ Low detection rates => Open research problem / lots of opportunities for further research

● Detected vulnerabilities in line with results from Study 1 ○ Injection, XSS, Path Traversals, etc. (classical taint tracking vulnerabilities) are detected ○ More complex vulnerabilities are not detected: Broken Access, Data Exposure, etc.

● CodeQL performs better than NodeJsScan ○ Modelling code into graph structures is better/more ﬂexible than rule/regex based matching

Tiago Brito, GSD Meeting - 30/07/2020 24 Conclusion

● What have we done here? ○ Explained the approach and deﬁciencies of two of the most popular tools ○ Showed preliminary results on the eﬀectiveness of selected tools

● So what? ○ Current tools are not adequate to detect a large number of vulnerabilities in JS code ○ There’s not suﬃcient previous work on JS vulnerability detection ○ There’s a lot of work to do in this area ■ Specially in detecting more complex vulnerabilities

Tiago Brito, GSD Meeting - 30/07/2020 25 What’s next?

● Finish this current work: ○ Study 1 - Do this analysis for more tools and applications - increase understanding of limitations ○ Study 2 ■ Finish manually analyzing/building the curated dataset ■ In depth analysis of tools’ results to assess eﬀectiveness ● Improve tools ○ Add contextual information to CodeQL’s taint tracking queries for speciﬁc system resources ○ Inspect npm packages and model their Node API calls to detect more dangerous sinks

Thank You! Questions? Tiago Brito, GSD Meeting - 30/07/2020 26 Study 1 - Unmodelled Sources, Sinks and Dependencies

if (req.body.eqn) { req.flash('result', mathjs.eval(req.body.eqn)); res.render('app/calc'); }

● Mathjs package contains a dangerous function that can lead to RCE

● Tools do not inspect dependency code, thus ignore possible transitive vulnerabilities

● Other tools detect dependency versions and compare them to vulnerability DBs (npm audit, github security advisory/dependabot, snyk, etc.)

● Ideally, a tool could also analyze dependency code (impractical / not scalable).

Tiago Brito, GSD Meeting - 30/07/2020 27 Study 1 - Unmodelled Languages/Interactions

<% if (output && output.searchTerm) { %>

Listing products with search query: <%- output.searchTerm %>

● Vulnerabilities often occur when diﬀerent components interact (PL, stacks, etc.)

● This EJS (JavaScript Templating ﬁle) contains a XSS vulnerability

● NodeJsScan detects it because it models EJS ﬁles, but CodeQL does not

● EJS is not pure JavaScript, but directly interacts with it

28 Tiago Brito, GSD Meeting - 30/07/2020 Study 2 - Effectiveness (Preliminary Results)

NodeJsScan Top 5 detections by CWE (absolute number of detections):

CWE # Advisories # NodeJsScan Percentage (Max)

CWE-89 - SQL Injection 20 15 75.0 %

CWE-79 - Cross-site Scripting 127 11 8.7 %

CWE-400 - Uncontrolled Resource Consumption 77 3 3.9 %

CWE-185 - Incorrect Regular Expression 3 2 66.7 %

CWE-943 - Improper Neutralization of Special Elements in Data Query Logic 5 2 40.0%

Total 232 (17.2 % total) 33 (14.2 %)

Tiago Brito, GSD Meeting - 30/07/2020 29 Study 2 - Effectiveness (Preliminary Results)

CodeQL Top 5 detections by CWE (absolute number of detections):

CWE # Advisories # CodeQL Percentage (Max)

CWE-22 - Path Traversal 156 109 69.9 %

CWE-79 - Cross-site Scripting 127 34 26.8 %

CWE-78 - OS Command Injection 43 33 76.7 %

CWE-471 - Modification of Assumed-Immutable Data (MAID) 60 23 38.3 %

CWE-25 - Path Traversal: '/../filedir' 13 10 76.9 %

Total 399 (29.6 % total) 209 (52.4 %)

Tiago Brito, GSD Meeting - 30/07/2020 30 Tools

Aether - https://github.com/codecombat/aether WALA - http://wala.sourceforge.net/wiki/index.php/Main_Page SemGrep - https://github.com/returntocorp/semgrep CodeQL - https://github.com/github/codeql NodeJsScan - https://github.com/ajinabraham/NodeJsScan Graudit - https://github.com/wireghoul/graudit/ EsLint Security Plugin - https://github.com/nodesecurity/eslint-plugin-security Yasca - https://github.com/scovetta/yasca EsLint Mozilla ScanJS - https://github.com/mozfreddyb/eslint-config-scanjs Google Closure Compiler - https://github.com/google/closure-compiler JsHint - https://github.com/jshint/jshint Sonarqube - https://github.com/SonarSource/sonarqube JsPrime - https://github.com/dpnishant/jsprime EsLint Security Scanner Configs - ApplicationInspector - https://github.com/microsoft/ApplicationInspector https://github.com/Greenwolf/eslint-security-scanner-configs Coala - https://github.com/coala/coala Synode (academic) - https://github.com/sola-da/Synode Codeburner - https://github.com/groupon/codeburner Insider - https://github.com/insidersec/insider PMD - https://github.com/pmd/pmd WALA - http://wala.sourceforge.net/wiki/index.php/Main_Page

Tiago Brito, GSD Meeting - 30/07/2020 31 Vulnerable Applications

DVNA - https://github.com/appsecco/dvna Snyk's Goof - https://github.com/snyk/goof NodeGoat - https://github.com/OWASP/NodeGoat VulnerableNode - https://github.com/cr0hn/vulnerable-node Juice Shop - https://github.com/bkimminich/juice-shop Appsec JavaScript Security - https://github.com/tlaskowsky/appsec_javascript_security Vulnerable Web Application - https://github.com/psmorrow/vulnerable-web-application

Tiago Brito, GSD Meeting - 30/07/2020 32