University of Pennsylvania ScholarlyCommons Technical Reports (CIS) Department of Computer & Information Science January 2001 Sub-Operating Systems: A New Approach to Application Security Sotiris Ioannidis University of Pennsylvania Steven M. Bellovin AT&T Labs Follow this and additional works at: https://repository.upenn.edu/cis_reports Recommended Citation Sotiris Ioannidis and Steven M. Bellovin, "Sub-Operating Systems: A New Approach to Application Security", . January 2001. University of Pennsylvania Department of Computer and Information Science Technical Report No. MS-CIS-01-06. This paper is posted at ScholarlyCommons. https://repository.upenn.edu/cis_reports/149 For more information, please contact [email protected]. Sub-Operating Systems: A New Approach to Application Security Abstract In the current highly interconnected computing environments, users regularly use insecure software. Many popular applications, such as Netscape Navigator and Microsoft Word, are targeted by hostile applets or malicious documents, and might therefore compromise the integrity of the system. Current operating systems are unable to protect their users from these kinds of attacks, since the hostile software is running with the user's privileges and permissions. We introduce the notion of the SubOS, a process-specific protection mechanism. Under SubOS, any application that might deal with incoming, possibly malicious objects, behaves like an operating system. It views those objects the same way an operating system views users - it assigns sub-user id's - and restricts their accesses to the system resources. Keywords secure systems, capabilities, process-specific protection Comments University of Pennsylvania Department of Computer and Information Science Technical Report No. MS- CIS-01-06. This technical report is available at ScholarlyCommons: https://repository.upenn.edu/cis_reports/149 Sub-operating Systems: A New Approach to Application Security Sotiris Ioannidis Steven M. Bellovin [email protected] [email protected] University of Pennsylvania -4T&T Labs Research Abstract In the current highly interconnected computing environments, users regularly use insecure software. Many popular applications, such as Netscape Navigator and Microsoft Word, are targeted by hostile applets or mali- cious documents, and might therefore compromise the integrity of the system. Current operating systems are unable to protect their users from this kind of attacks, since the hostile software is running with the user's privileges and permissions. We introduce the notion of the SubOS, a process-specific protection mechanism. Under SubOS, any application that might deal with incom- ing, possibly malicious objects, behaves like an operating system. It views those objects the same way an operating system views users-it assigns sub-user id's-and restricts their accesses to the system resources. Keywords: Secure systems, capabilities, process-specific protection. 1 Introduction Many irslportant applications, such as mailers, m7eb browsers, word processors, etc., have rrlany of the characteristics of operating systenis. In particular, they accept requests fro111 a variety of mutually-suspicious sources. grant different permissions based on the source (or other attributes, such as a cryptographic token), a11d ~riediateaccess to assorted resources. But applicatioris are poorly suited to this task. For exarnple, they have to implenierit file access restrictions by ~natchingfile nanies against assorted patterns. History shows, however, that that approach is fraught with danger (i.e., CERT .Advisories CA:98-04 and CA:97-03). Real operating systems, which bind permissions to the protected objects, rarely have many problems like that. In this paper we irltroduce the riotiorl of a sub-operating systern (called SubOS hereafter). A SubOS is an applicatios~that might have to operate 011 u~itrustedobjects. By the term tintlusted object we refer to any ir~conlingfile to our systerri, such as a Word docurr~er~treceived i11 the mail, a postscript file dowrl-loaded from some ftp site, or a Java applet that a browser ~riightload fro111 a Web page. These applications use operating syste~riprotection ~rieclia~lis~ris to irnplerr~e~ittheir own. More precisely, applications that "touch" possibly malicious objects, like the ones listed above, will 110 lor~germaintain the users privileges, but will rather get restricted access riglits to the underlying resources. Figures 1 arld 2 display the differences of a regular, and a SubOS enabled operating system. -* - * $0 - 77s 23 Lan QM *, ..... g; ~pplications 2 i?, ~.ge E m.cC g 3 u 3 g.s i Unprotected Space Operating System Resources Protected Space (CPU, Memory, Disk, Network, etc.: Figure 1: User applicatio~isexecutirig 011 an operating system rnai11tai11tlie user privileges, allowing the111 al~riostfull access to tlie underlying operating syste111. The paper is orga~~izedas follows. 111 Section 2 we discuss the motivation behirid this work. 111 Sectio~l3 we present tlie desigri and irliplenientatiorl details of a SubOS-capable OpenBSD [Ill system. 111 Sectiori 4 we discuss work that is related to SubOS, arid finally we co~lcludei11 Section 5. 2 Motivation ,4 ~lu~rlberof trends in co~riputirigare fueli~~gthe need for a more flexible, yet stricter security rriodel in operating syste~ris. 2.1 Information Exchange With the growth of the Internet, excha~lgeof ir~for~natior~over wide-area net- works has become essential for both applicatio~isa~id users. Modern applica- tioris often fetch help files and other data over the World Wide Web. In extreme cases, like sorrie flavors of the BSD UNIX operating system, even wllole oper- ating systerns i~istalland upgrade themselves over the network. However, the ~riostcoInrnon case is electro~~ic~riail. Users regularly receive 111ai1from uriknown I -0 5: $3 Q w ~pplicationr 8 $ & ..... Unprotected Space Operating System p aV1 V, cn Resources Protected Space (CPU.Memory, Disk. Network, etc.: Figure 2: Under SubOS enabled operating systems user applicatiorls that "touch" possibly malicious objects no longer 111ai11tain the user access rights, and only get restricted access to the u~iderlyingsysterrl. sources wit11 a ~iu~rlberof possibly rrlalicious attacl~~nerlts.The attached docu- ments use vulnerabilities i11 the helper applicatio~lsthat are invoked to process them, which in turn could corrlpro~rlisesyste~rl security. The need for connec- tivity and exchange of i11formatio11eve11 at this rrlost basic level is therefore a major threat to security. It is also the case that see~ninglyinactive objects like Web pages or e-mail Irlessages are very ~nuchactive and potentially dangerous. One exa111ple is JavaScript prograIrls which are executed within the security co~~textof the page with which they were down-loaded, and they have restricted access to other resources withi11 the browser. Security flaws exist i11 certain Web browsers that per~r~itJavaScript prograrrls to rnonitor a user's browser activities beyo~ldtlie security context of the page with which the prograIri was dow~lloaded(CERT Advisory Ck97.20). It is obvious that such behavior autonlatically colnpro- ~nisestlie user's privacy. ilnother exa~npleis the use of Multipurpose 111terrlet Mail Exte~lsio~ls(MIME). The MIME for~natpermits e~nailto include erlhanced text, graphics, and au- dio in a standardiaed and i~~ter-operablemanIier. Metamail(1) is a package that i~rlple~rle~itsMIME. Using a configurable mailcap (4) file, metamail (1) deter~rlirieshow to treat blocks of electronic nail text based on the content as described by e~riailheaders. -4 conditiorl exists i~imetamail (1) in wl~iclithere is i~~sufficientvariable checking i11 soIrle support scripts. By carefully crafting appropriate Iliessage headers, a sender can cause the receiver of tlie Irlessage to execute a11 arbitrary co~rlrna~ldif the receive1 processes the Inessage usi~lgthe mailcap (4) package (CERT -4dvisory C.4:97.14) [lo]. 2.2 Application Complexity But the problem is deeper than obvious forms of mobile code. Given the in- creasingly complex enviro~imentpresented to many applicatiorls, we assert that these applications have many of the cliaracteristics of operating systems, and should be impler~ientedas such. Even siniple HTTP requests return a conlplex object, wherein the rernote side tells the local browser what to do, up to and including a request to run certai~~applications. Print spoolers have to check file access permissions. Elnail can be delivered directly to programs. Web servers have to run scripts, often via an interpreter, while denying direct access to the interpreter and perhaps ensuring that one script does not access or ~r~odifythe private data of another script. All of these applications sllould worry about resource co~isu~nption.And these, of course, are the characteristics of operating systems. 111 fact, arbitrating access to various objects is Inore or less the definition of what an operating systerrl does. However, re-in~plenientingan operati~~gsyste~r~ with each new applicatio~i would be extreme. Instead, our goal is to add sufficient functionality to an existing syster~~so that applicatiorls can rely on the base operating system to carry out its ow11 particular security policy. That security policy. in turn, can reflect its ow11 particular needs arid its degree of certainty as to the identity of user S. 2.3 Inadequate Operating System Support The lack of flexibility in modern operating syste~nsis one of the main reasons security is compromised.