Exploring the Relationship Between Web Application Development Tools and Security Matthew Finifter David Wagner University of California, Berkeley University of California, Berkeley fi[email protected] [email protected] Abstract Security strategies of an organization often include de- veloping processes and choosing tools that reduce the How should software engineers choose which tools to number of vulnerabilities present in live web applica- use to develop secure web applications? Different devel- tions. These software security measures are generally opers have different opinions regarding which language, focused on some combination of (1) building secure soft- framework, or vulnerability-finding tool tends to yield ware, and (2) finding and fixing security vulnerabilities more secure software than another; some believe that in software after it has been built. there is no difference at all between such tools. This pa- How should managers and developers in charge of per adds quantitative data to the discussion and debate. these tasks decide which tools – languages, frameworks, We use manual source code review and an automated debuggers, etc. – to use to accomplish these goals? What black-box penetration testing tool to find security vul- basis of comparison do they have for choosing one tool nerabilities in 9 implementations of the same web ap- over another? Common considerations for choosing plication in 3 different programming languages. We ex- (e.g.,) one programming language over another include: plore the relationship between programming languages • How familiar staff developers are with the language. and number of vulnerabilities, and between framework support for security concerns and the number of vul- • If new developers are going to be hired, the current nerabilities. We also compare the vulnerabilities found state of the market for developers with knowledge by manual source code review and automated black-box of the language. penetration testing. • Interoperability with and re-usability of existing in- Our findings are: (1) we do not find a relationship house and externally-developed components. between choice of programming language and applica- • Perceptions of security, scalability, reliability, and tion security, (2) automatic framework protection mech- maintainability of applications developed using that anisms, such as for CSRF and session management, ap- language. pear to be effective at precluding vulnerabilities, while Similar considerations exist for deciding which web ap- manual protection mechanisms provide little value, and plication development framework to use and which pro- (3) manual source code review is more effective than au- cess to use for finding vulnerabilities. tomated black-box testing, but testing is complementary. This work begins an inquiry into how to improve one part of the last of these criteria: the basis for evaluating 1 Introduction a tool’s inclination (or disinclination) to contribute to ap- plication security. The web has become the dominant platform for new soft- Past research and experience reveal that different tools ware applications. As a result, new web applications can have different effects on application security. The are being developed all the time, causing the security software engineering and software development commu- of such applications to become increasingly important. nities have seen that an effective way to preclude buffer Web applications manage users’ personal, confidential, overflow vulnerabilities when developing a new appli- and financial data. Vulnerabilities in web applications cation is to simply use a language that offers automatic can prove costly for organizations; costs may include di- memory management. We have seen also that even if rect financial losses, increases in required technical sup- other requirements dictate that the C language must be port, and tarnished image and brand. used for development, using the safer strlcpy instead of strcpy can preclude the introduction of many buffer security of applications developed using that language. overflow vulnerabilities. For example, research has shown that type systems can This research is an exploratory study into the security statically find (and therefore preclude, by halting compi- properties of some of the tools and processes that orga- lation) certain types of vulnerabilities [21, 20]. In gen- nizations might choose to use during and after they build eral, static typing can find bugs (any of which could be their web applications. We seek to understand whether a vulnerability) that may not have been found until the the choice of language, web application development time of exploitation in a dynamically-typed language. framework, or vulnerability-finding process affects the Also, one language’s standard libraries might be more security of the applications built using these tools. usable, and therefore less prone to error, than another’s. We study the questions by analyzing 9 independent A modern exception handling mechanism might help de- implementations of the same web application. We col- velopers identify and recover from dangerous scenarios. lect data on (1) the number of vulnerabilities found in But programming languages differ in many ways be- these implementations using both a manual security re- yond the languages themselves. Each language has its view and an automatic black-box penetration testing tool, own community, and these often differ in their philoso- and (2) the level of security support offered by the frame- phies and values. For example, the Perl community val- works. We look in these data sets for patterns that might ues TMTOWTDI (“There’s more than one way to do indicate differences in application security between pro- it”) [4], but the Zen of Python [16] states, “[t]here should gramming languages, frameworks, or processes for find- be one – and preferably, only one – obvious way to do ing vulnerabilities. These patterns allow us to generate it.” Clear documentation could play a role as well. and test hypotheses regarding the security implications Therefore, we want to test whether the choice of lan- of the various tools we consider. guage measurably influences overall application security. This paper’s main contributions are as follows: If so, it would be useful to know whether one language • We develop a methodology for studying differences fares better than another for any specific class of vulner- in the effect on application security that differ- ability. If this is the case, developers could focus their ef- ent web application development tools may have. forts on classes for which their language is lacking good The tools we consider are programming languages, support, and not worry so much about those classes in web application development frameworks, and pro- which data show their language is strong. cesses for finding vulnerabilities. Web application development framework. Web ap- • We generate and test hypotheses regarding the dif- plication development frameworks provide a set of li- ferences in security implications of these tools. braries and tools for performing tasks common in web • We develop a taxonomy for framework-level de- application development. We want to evaluate the role fenses that ranges from always on framework sup- that they play in the development of secure software. port to no framework support. This can help developers make more informed decisions • We find evidence that automatic framework-level when choosing which technologies to use. defenses work well to protect web applications, but Recently, we have seen a trend of frameworks adding that even the best manual defenses will likely con- security features over time. Many modern frameworks tinue to fall short of their goals. take care of creating secure session identifiers (e.g., • We find evidence that manual source code analy- Zend, Ruby on Rails), and some have added support sis and automated black-box penetration testing are for automatically avoiding cross-site scripting (XSS) or complementary. cross-site request forgery (CSRF) vulnerabilities (e.g., Django, CodeIgniter). It is natural to wonder if frame- works that are pro-active in developing security features 2 Goals yield software with measurably better security, but up to this point we have no data showing whether this is so. Programming language. We want to measure the in- fluence that programming language choice has on the Vulnerability-finding tool. Many organizations man- security of the software developed using that language. age security risk by assessing the security of software If such an influence exists, software engineers (or their before they deploy or ship it. For web applications, two managers) could take it into account when planning prominent ways to do so are (1) black-box penetration which language to use for a given job. This informa- testing, using automated tools designed for this purpose, tion could help reduce risk and allocate resources more and (2) manual source code analysis by an analyst knowl- appropriately. edgeable about security risks and common vulnerabili- We have many reasons to believe that the features of ties. The former has the advantage of being mostly au- a programming language could cause differences in the tomated and being cheaper; the latter has a reputation as Team Language Frameworks used with metrics like performance, completeness, size, and Number usability, we re-analyze their data to evaluate the security 1 Perl DBIx::DataModel, Catalyst, Tem- 9 plate Toolkit properties of these programs.