Basics of Windows OS for Reverse Engineering Malware

Applications on Windows

• Object files and executables follow the PE (Portable Executable) file format • Full specification available online • http://www.microsoft.com/whdc/system/platform/ firmware/PECOFF.mspx • Best viewed with your hex editor (HT) or specialized PE viewer (PEBrowsePro ->)

• Filename extension hints to the executable type • EXE = executable application, anything from a DOS executable to 64-bit Windows applications • DLL = Dynamic link library, a set of callable routines compiled together as a loadable file • SYS = Driver • OBJ = Object file, input to the linker • Note that Windows does not really care much about the file extension • You can execute application.jpg just fine • All of these follow the PE/COFF specification • APPX is used for Windows 8 Windows Store apps • Not a PE, but a ZIP archive with all application contents inside

• Windows API (aka. Win32 API) is the interface to the operating system for applications • Exposed by a set of system libraries: kernel32.dll, user32.dll, … • Windows 7 refactored the system libraries (“MinWin”) so you will see e.g. kernelbase.dll • On 64-bit, because of 32-bit compatibilty, c:\windows\syswow64 contains another set of libraries • Several subcategories • Administration and management (WMI, …) • Diagnostics (event logging, …) • Networking • Security • System services (processes, threads, registry…) • MSDN is the reverse engineers best friend for Windows binaries • http://msdn2.microsoft.com/en-us/library/default.aspx

• Undocumented interface to OS functionality • One level below Windows API • Some low-level functionality only available through Native API • Examples of interesting functions • NtSetSystemInformation • NtQuerySystemInformation • NtQueryDirectoryFile • See “Windows NT/2000 Native API Reference” by Nebbett

Windows Architecture

System Support Service User Environment Processes Processes Applications Subsystems

Subsystem DLL’s

NTDLL.DLL

Executive Windowing & Graphics Kernel Device Drivers

HAL

System Mechanisms

0xFFFFFFFF

System-space (Ring 0)

Int 0x2e / Sysenter

User-space User-space (RingUser -3)space (Ring 3) (Ring 3) 0x00000000

Read the file Nt! Return to caller NtReadFile

Call NtReadFile Interrupt Nt! Dismiss interrupt KiSystemService

Sysenter Ntdll.dll! Return to caller NtReadFile

Call NtReadFile Kernel32.dll! Return to caller ReadFile

ReadFile(…) Application

Processes & Threads

• Abstraction of an executing program • A process has • A private virtual address space (user-mode address space) • A base image (the main executable) • A set of open handles to OS resources • An access token (“who is this process, what can it do”) • A process ID (PID) • One or more threads • Job = A group of processes that can be managed as a unit

• A thread is what gets scheduled for execution on the CPU • A thread has • A context (the state of execution) • A list of exception handlers • Two stacks, one for user-mode and one for kernel-mode • A thread ID (TID) • An access token • Thread-Local Storage (TLS), a per-thread storage area • Fiber = Manually scheduled unit of execution, shares the context of its owning thread (ConvertThreadToFiber, CreateFiber, SwitchToFiber)

18 February 18, 2013 Case Study: FU Rootkit

• The FU rootkit hides processes by unlinking them from the kernels process list • Why do those processes still get execution time?

• A technique to run code in the context of another process by forcing it to load a DLL • Motivation 1. Stealth: no extra processes in process list 2. Data capture using API hooks (e.g. banking trojans) 3. Making reverse engineering more difficult 4. Avoiding HIPS features (outbound network connectivity from trusted process) • Various ways to accomplish DLL injection on Windows • Registry: AppInit_DLLs • Using remote threads • SetWindowsHookEx • From the kernel using APC’s

• TEB = Thread Environment Block • Container for thread-specific things like the exception handler list, stack pointer, … • Windows uses the fs segment to store it (offset 0x18 has pointer to self) • mov eax, fs[18] • PEB = Process Environment Block • Container for process-specific things like the list of loaded modules • TEB has a pointer to PEB at offset 0x30 • Important when reversing code that • Enumerates loaded modules (Peb.Ldr) • Checks for an attached debugger (PEB.BeingDebugged) • Installs an exception handler (TEB.NtTib.ExceptionList)

; Call IsDebuggerPresent() call [IsDebuggerPresent] test eax, eax ; Do the same by checking PEB mov eax, large fs:18h ; Offset 18h has self-pointer to TEB mov eax, [eax+30h] ; Offset 30h has pointer to PEB movzx eax, byte ptr [eax+2] ; PEB.BeingDebugged test eax, eax

; Install a SEH exception handler push offset_my_handler ; pointer to our handler push fs:[0] ; pointer to old exception record mov fs:[0], esp ; update TEB.NtTib.ExceptionList

Memory Management

• Each process on Windows sees a large, continuous address space • The memory manager has two important tasks 1. Mapping access to virtual memory into physical memory 2. Paging contents of virtual memory to disk as physical memory runs out, and paging data back to memory when it’s needed

• Each process has private, virtual address space • Paging = the process of transferring memory contents to and from disk • Amount of virtual memory can exceed physical memory

• Total of 4 GB virtual memory • By default, only lower 2 GB accessible from user-mode • Upper 2 GB reserved for kernel-mode and shared between all processes

Management Mechanisms

• A directory for storing settings and configuration data for the Operating System and applications • Think of it as a huge .ini file • Basic concepts • Hive • Key • Value • Hives are just files, most under %SystemRoot%\System32\Config

• HKEY_LOCAL_MACHINE (HKLM) • System-wide information • HKEY_USERS (HKU) • User-specific settings for all accounts • HKEY_CURRENT_USER (HKCU) • A link to the current user under HKEY_USERS • HKEY_CLASSES_ROOT (HKCR) • File associations and COM registration, links to HKLM\Software\Classes • HKEY_PERFORMANCE_DATA • HKEY_CURRENT_CONFIG • Current HW profile, links to HKLM\System\CurrentControlSet\Hardware Profiles\Current

• Malware wants to survive a reboot • Registry is the most common way of starting after reboot • Hundreds of “launchpoints” • HKLM\Software\Microsoft\Windows\CurrentVersion\Run:MyApp • HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\explorer.exe:Debugger • Malware also wants to change (security) settings for other components • Windows Firewall, IE extensions and settings, Windows File Protection, … • The registry is also a great source for forensic data, for example: • HKEY_CURRENT_USER\Software\Microsoft\Windows\ShellNoRoam\MUICache • HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\ UserAssist

• Services are background processes that usually perform a specific task and require no user-interaction • For example, Automatic Updates • Controlled by the Service Control Manager (SCM), services.exe • Configuration data under HKLM\System\CurrentControlSet\Services • Different types of services • Kernel drivers • Separate process • Shared process (hosted by svchost.exe)

35 February 18, 2013

File Systems

• CDFS (cdfs.sys) • CD-ROM filesystem, considered legacy • UDF (udfs.sys) • Universal Disk Format, the standardized format for optical storage • FAT12, FAT16, FAT32 (fastfat.sys) • Legacy filesystem, mostly replaced by NTFS • exFAT (aka. FAT64) • Adds functionality on top of FAT (file size limit increase, ACL’s) • NTFS

• Native file system for Windows • Support for advanced features • ACL’s on files and directories • Alternate Data Streams • Disk quotas • Sparse files • Compression and encryption • Soft and hard links • Transactional semantics

39 February 18, 2013

Security Mechanisms

• Almost everything on Windows is an object • Process, thread, desktop, mutex, … • Basic concepts • Security Identifier (SID) is a unique ID for any actor • “S-1-5-21-525843606-2469437151-111719316-1006” = DOMAIN\user123 • A token identifies the security context of a process • “Member of Administrators group, can shut down OS” • Security Descriptor specifies who can do what to an object • Owner • Discretionary Access Control List (DACL) • Privileges

Windows kernel

ntdll.dll ntdll.dll ntdll.dll Filesystem

kernel32.dll kernel32.dll kernel32.dll

Registry

calc.exe

winword.exe services.exe

User User System

43 February 18, 2013 Sandboxing Applications

• Especially in later Windows versions (Vista, Windows 7), extensions to the security model can be used to isolate less trustworthy applications • Prevent exploited applications from changing the system • In practice, requires support from the application itself • Protected Mode Internet Explorer, Adobe Reader X, Google Chrome • A process is isolated from the rest of the system with additional mechanisms, e.g. • Restricted tokens • See CreateRestrictedToken() on MSDN • Job objects • CreateJobObject(), AssignToJobObject(), SetInformationJobObject() • Integrity Levels • Security descriptors and tokens are assigned an Integrity Level • A process with a lower IL cannot modify (sometimes not read) higher IL objects

44 February 18, 2013

Adobe Reader X

45 February 18, 2013 Adobe Reader X Broker

46 February 18, 2013 Windows 8 and Windows Store Apps