TROGUARD: Context-Aware Protecon Against Web-Based Socially Engineered Trojans

Rui Han, Alejandro Mesa, University of Miami Mihai Christodorescu, QualComm Research Saman Zonouz, Rutgers University

Movaon • Waterfall screen saver Trojan

2 Mac OS threats Rank Name Percentage 1 Trojan.OSX.FakeCo.a 52% 2 Trojan-Downloader.OSX.Jahlav.d 8% 3 Trojan-Downloader.OSX.Flashfake.ai 7% 4 Trojan-Downloader.OSX.FavDonw.c 5% 5 Trojan-Downloader.OSX.FavDonw.a 2% 6 Trojan-Downloader.OSX.Flashfake.ab 2% 7 Trojan-FakeAV.OSX.Defma.gen 2% 8 Trojan-FakeAV.OSX.Defma.f 1% 9 Exploit.OSX.Smid.b 1% 10 Trojan-Downloader.OSX.Flashfake.af 1% McAfee anvirus soluon: hp:www.securelist.com

3 Example Malwares Malware Descripons Plaorm TrojanClicker.VB Trojan socially Windows engineered as adobe .395 flash update and Mac OS X Trojan or Adware socially Windows, Mac Faked An-Virus engineered as an-virus soware OS X, and Linux Malware socially Android Opfake Browser engineered as Opera Browser Legimate applicaons Mac OS X and WireLuker socially engineered with ad-wares and Trojan iOS

4 Contribuons • Answer the queson: “Is this program doing what I expected it to do?” • Bridge the semanc gap between funconality classes and low level behaviors • Built on 100 Linux app profiles • High detecon rate on 50 Trojan apps

5 TROGUARD Architecture

TROGUARD

Offline

Dynamic Application Functionality Application Functionality Functionality Class Profile Database Feature Tracing Generation Extraction

Application Functionality Profile Database

Online

Inference of Download Perceived Website Functionality Class Real-Time Classification Application Downloaded Functionality Application Tracing sandbox

Alert 6 Key Premise • TROGUARD detects Trojans based on the premise that applicaons with similar funconalies expose similar system-level behaviors • Applicaons with similar funconalies belong to a funconality class, they should exhibit common system level behaviors • Learn web-browser behavior of well know instances(e.g., Firefox and Chrome) • Compare the web-browser profile with the behaviors of unknown downloaded web-browser app

7 Funconality class • It represents both user’s understanding of soware category and the system’s observaon of a soware execuon behavior Funconality class sopedia.com download.cnet.com tucows.com Graphic Editor Arsc soware Graphic Design SW Design tools Game Games Games Games Browser Internet Browsers Internet Instant Messenger Communicaons Communicaons Media Player MP3/Audio Soware Audio Editor Mulmedia Audio/Video Video Editor Text Editor office Producvity soware Business IDE Programming Developer Tools Develop/Web Calculator Ulies Ulies Home/Educaon

8 Applicaons Class Studied Applicaons 1. Graphic Editor gimp, pinta, imagej, inkscape, kolourpaint, rawtherapee, mypaint, gpaint, gnome-paint, pencil 2. Games sol, wesnoth, glchess, neverball, kmahjongg, supertuxkart, hedge- wars, pingus, frozen-bubble, eboard 3. Browser chrome, firefox , opera, epiphany, midori, chromium, netsurf, arora, xxxterm, rekonq 4. Instant skype, kmess, emesene, kopete, pidgin, psi, gajim, empathy, amsn, qum Messenger 5. Media Player smplayer, vlc, audacious, quodli- bet, gmusicbrowser, qmmp, abraca, amarok, guayadeque, aqualung 6. Audio Editor audacity, avidemux, dvbcut, og- gconvert, kwave, wavbreaker, mp3splt-gtk, mhwaveedit, fillmore, soundconverter 7. Video Editor openshot, lives, iriverter, kino, pivi, videocut, winff, arista-gtk, kdenlive, curlew

8. Text Editor kile, geany, texmaker, calligra- words, soffice.bin, lyx, tea, jed, emacs, vi

9. IDE anjuta, codelite, codeblocks, net- beans, monodevelop, kdevelop, spyder, monkeystudio, drracket, idle

10. Calculator grpn, gcalctool, EdenMath, speed- crunch, kcalc, keurocalc, extcalc, gip, 9 galculator, gnome-genius Funconality Tracing • Manual tesng • Run 60 seconds for each applicaon • System call trace • User-space informaon User interacvity Resource consumpon IP addresses and port number

10 Feature Extracon • Processing tracing data • Four groups of feature file system Network resource usage user interacvity • Intermediate feature

11 Intermediate Features

Example: if( libssl3.so & fd = sys_socket(AF_INET, ..) &sys_write(fd, ..) & sys_read(fd, ..) ) HTTP = true 12 TROGUARD Architecture

TROGUARD

Offline

Dynamic Application Functionality Application Functionality Functionality Class Profile Database Feature Tracing Generation Extraction

Application Functionality Profile Database

Online

Inference of Download Perceived Website Functionality Class Real-Time Classification Application Downloaded Functionality Application Tracing

Alert 13 Web Page Analysis

• Give the explicit funconality class • Web page contents analysis • OCR to extract the texts in the images • Analysis based on keywords

14 User Interface • Browser extension for web page analysis

15 Sandboxing

• SELinux sandbox • One policy for each app class • Automacally generated by parsing all the logs from an app class

16 Classifier Evaluaon • 600 data points (10 second each) • 10 fold cross validaon • 5 classifiers with different feature group • Precision • Recall • Confusion Matrix

17 Precision

100% 90% 80% 70% 60% 50%

Precision 40% 30% 20% 10%

0 File Network CPU-Mem Interaction All Different Attributes Domains

Browser Game Graphic-Editor Calculator Office IDE Video-Editor Average IM Media-Player Audio-Editor

18 Recall

100% 90% 80% 70% 60% 50% Recall 40% 30% 20% 10%

0 File Network CPU-Mem Interaction All Different Attributes Domains

Browser Game Graphic-Editor Calculator Office IDE Video-Editor Average IM Media-Player Audio-Editor

19 Confusion Matrices File features Network features

Resource usage features User interacvity features

20 Confusion Matrices

21 Intermediate Feature Results 54 5 0 0 0 0 0 0 0 0 90% 56 0 0 1 0 0 3 0 0 0 93% 58 1 0 0 0 1 0 0 0 97% 0 58 0 0 0 2 0 0 97% 0 0 0 57 0 1 0 0 1 95% 0 0 1 0 42 8 0 0 0 0 0 5 5 0 70% 60 0 0 0 0 0 100% 0 0 0 0 57 0 0 3 0 0 0 0 0 0 95% 60 0 0 0 0 0 100% 0 0 0 0 54 0 0 0 90% 0 0 6 0 0 0

22 Web Page Analysis Accuracy • 100 Web page, 20 categories 100% Text analysis 90% OCR analysis 80% 70% 60% 50%

Accuracy 40% 30% 20% 10% 0 Anti-virusEbookEmailMediaThemes PlayerDownloaderDriverCalculatorGameOfficeBrowserVideoAudio EditorDatabase EditorIDE P2P AppIM GraphicsPDF ReaderNetworkEducation

23 Case Study

• 10 benign apps ×5 payload = 50 Trojans Funconality Class Applicaon Metasploit Payload 1. Graphic Editor gpaint 2. Games eboard 3. Browser xxxterm 4. Instant Messenger psi linux/x86/shell_bind_tcp 5. Media Player qmmp linux/x86/shell/reverse_tcp linux/x86/vncinject/bind_tcp 6. Audio Editor winff linux/x86/meterpreter/bind_tcp 7. Video Editor fillmore linux/x86/download_exec 8. Text Editor tea 9. IDE spyder 10. Calculator gnome-genius 24 Case Study • Predefined acceptance rate 0.8

Trojans" Benign"Apps"

1"

0.8"

0.6"

Rate" 0.4"

0.2"

0" True" False" Precision" Recall" F9Measure" Posi1ve" Posi1ve"

25 Symbolic Execuon • Tested Core Ulies (four funconality classes) Dirlist Filetype Userinfo Systeminfo • Features collected from symbolic execuon give us 52% precision • Features collected from user execuon give us 76% precision

26 User Execuon VS Symbolic Execuon

27 Performance Overhead • CPU usage: No-LTTng LTTng

28 Performance Overhead • Memory usage: No-LTTng LTTng

29 Performance Overhead • Disk throughput: No-LTTng LTTng

30 Performance Overhead • Network throughput: No-LTTng LTTng

31 Conclusions • TROGUARD detects Trojans based on the premise that applicaons with similar funconalies expose similar system-level behaviors • TROGUARD can detect Trojan applicaon download by bridging the gap between the user perceived funcons and genuine soware funcons

32 Rui Han r[email protected]

33 Conclusions • TROGUARD detects Trojans based on the premise that applicaons with similar funconalies expose similar system-level behaviors • TROGUARD can detect Trojan applicaon download by bridging the gap between the user perceived funcons and genuine soware funcons

34 Symbolic Execuon Code Coverage

35 SE Code Coverage Evoluon

36