5.4 Winmemmanaginternals [Read-Only]
Total Page:16
File Type:pdf, Size:1020Kb
Unit OS5: Memory Management 5.4. Windows Memory Management Internals Windows Operating System Internals - by David A. Solomon and Mark E. Russinovich with Andreas Polze Copyright Notice © 2000-2005 David A. Solomon and Mark Russinovich These materials are part of the Windows Operating System Internals Curriculum Development Kit, developed by David A. Solomon and Mark E. Russinovich with Andreas Polze Microsoft has licensed these materials from David Solomon Expert Seminars, Inc. for distribution to academic organizations solely for use in academic environments (and not for commercial use) Windows Operating System Internals - by David A. Solomon and Mark E. Russinovich with Andreas Polze 2 1 Roadmap for Section 5.4. From working sets to paging dynamics Birth of a process working set Working set trimming, heuristics Paging, paging dynamics Hard vs. soft page faults Page files Windows Operating System Internals - by David A. Solomon and Mark E. Russinovich with Andreas Polze 3 All* Committed Virtual Address Space is Mapped to Files Ranges of virtual address space are mapped to ranges of blocks w ithin disk files Each such range is defined by a “section object” (visible in WinObj if named) These files are the “backing store” for virtual address space Commonly-used files are: The system paging file For writeable, nonshareable (copy -on-write) pages i.e. most writeable data For read-only application-defined code and for shareable data Executable program or DLL Can set up additional file/virtual address space relationships at run time (CreateFileMapping API) *almost - exceptions include nonpaged pool & system code mapped into each user process and/or physical memory allocated through new Windows 2000 extended addressing services 2 Application Startup Maps V.A.S. to Code on Disk 00000000 paging file .dll .exe 7FFFFFFF See link/dump/header, or QuickView for .exe’s and .dll’s CreateFileMapping, MapViewOfFile simply make the mechanism available to application-level code All of these files may simultaneously be mapped by other processes Working Set Working set: All the physical pages “owned” by a process Essentially, all the pages the process can reference without incurring a page fault Working set limit: The maximum pages the process can own When limit is reached, a page must be released for every page that’s brought in (“working set replacement”) Default upper limit on size for each process System-wide maximum calculated & stored in MmMaximumWorkingSetSize approximately RAM minus 512 pages (2 MB on x86) minus min size of system working set (1.5 MB on x86) Interesting to view (gives you an idea how much memory you’ve “lost” to the OS) True upper limit: 2 GB minus 64 MB 3 Working Set List newer pages older pages PerfMon Process “WorkingSet” A process always starts with an empty working set It then incurs page faults when referencing a page that isn’t in its working set Many page faults may be resolved from memory (to be described later) Birth of a Working Set Pages are brought into memory as a result of page faults Prior to XP, no pre-fetching at image startup But readahead is performed after a fault See MmCodeClusterSize, MmDataClusterSize, MmReadClusterSize If the page is not in memory, the appropriate block in the associated file is read in Physical page is allocated Block is read into the physical page Page table entry is filled in Exception is dismissed Processor re-executes the instruction that caused the page fault (and this time, it succeeds) The page has now been “faulted into” the process “working set” 4 Prefetch Mechanism First 10 seconds of file activity is traced and used to prefetch data the next time Also done at boot time (described in Startup/Shutdown section) Prefetch “trace file” stored in \Windows\Prefetch Name of .EXE-<hash of full path>.pf When application run again, system automatically Reads in directories referenced Reads in code and file data Reads are asynchronous But waits for all prefetch to complete Windows Operating System Internals - by David A. Solomon and Mark E. Russinovich with Andreas Polze 9 Prefetch Mechanism In addition, every 3 days, system automatically defrags files involved in each application startup Bottom line: Reduces disk head seeks This was seen to be the major factor in slow application/system startup Lab Run Filemon – set filter as Notepad.exe Make a temporary directory somewhere (e.g. \temp) Run “Notepad \temp\x.y” Exit Notepad Run Notepad again In Filemon log, find creation of .PF file after first run, then use of new .PF in 2nd run Windows Operating System Internals - by David A. Solomon and Mark E. Russinovich with Andreas Polze 10 5 Working Set Replacement PerfMon Process “WorkingSet” When working set max reached (or working set trim occurs), must give up pages to make room for new pages to standby Local page replacement policy (most Unix systems implement global replacement) or modified Means that a single process cannot take over all of physical memory unless other page list processes aren’t using it Page replacement algorithm is least recently accessed (pages are aged) On UP systems only in Windows 2000 – done on all systems in Windows XP/Server 2003 New VirtualAlloc flag in XP/Server 2003: MEM_WRITE_WATCH Soft vs. Hard Page Faults Types of “soft” page faults: Pages can be faulted back into a process from the standby and modified page lists A shared page that’s valid for one process can be faulted into other processes Some hard page faults unavoidable Process startup (loading of EXE and DLLs) Normal file I/O done via paging Cached files are faulted into system working set To determine paging vs. normal file I/Os: Monitor Memory->Page Reads/sec Not Memory->Page Faults/sec, as that includes soft page faults Subtract System->File Read Operations/sec from Page Reads/sec Or, use Filemon to determine what file(s) are having paging I/O (asterisk next to I/O function) Should not stay high for sustained period 6 Viewing the Working Set Working set size counts shared pages in each working set Vadump (Resource Kit) can dump the breakdown of private, shareable, and shared pages C:\> Vadump –o –p 3968 Module Working Set Contributions in pages Total Private Shareable Shared Module 14 3 11 0 NOTEPAD.EXE 46 3 0 43 ntdll.dll 36 1 0 35 kernel32.dll 7 2 0 5 comdlg32.dll 17 2 0 15 SHLWAPI.dll 44 4 0 40 msvcrt.dll Windows Operating System Internals - by David A. Solomon and Mark E. Russinovich with Andreas Polze 13 Working Set System Services Min/Max set on a per-process basis Can view with !process in Kernel Debugger System call below can adjust min/max, but has minimal effect prior to Server 2003 Limits are “soft” (many processes larger than max) Memory Manager decides when to grow/shink working sets New system call in Server 2003 (SetProcessWorkingSetSizeEx) allows setting hard min/max Can also self-initiate working set trimming Pass -1, -1 as min/max working set size (minimizing a window does this for you) Windows API: SetProcessWorkingSetSize( HANDLE hProcess, DWORD dwMinimumWorkingSetSize, DWORD dwMaximumWorkingSetSize) 7 Locking Pages Pages may be locked into the process working set Pages are guaranteed in physical memory (“resident”) when any thread in process is executing Windows API: status = VirtualLock(baseAddress, size); status = VirtualUnlock(baseAddress, size); Number of lockable pages is a fraction of the maximum working set size Changed by SetProcessWorkingSetSize Pages can be locked into physical memory (by kernel mode code only) Pages are then immune from “outswapping” as well as paging MmProbeAndLockPages Process Memory Information Task Manager 1 2 Processes tab 1u “Mem Usage” = physical memory used by process (working set size, not working set limit) u Note: shared pages are counted in each process 2u “VM Size” = private (not shared) committed virtual space in processes == process’s paging file allocation 3u “Mem Usage” in status bar is not total of “Mem Usage” column (see later slide) 3 Screen snapshot from: Task Manager | Processes tab 8 Process Memory Information PerfMon - Process Object 7 “Virtual Bytes” = committed + reserved virtual space, including shared pages 1 “Working Set” = working set size (not limit) (physical) 2 “Private Bytes” = private virtual space (same as “VM Size” from Task Manager Processes list) Also: In Threads object, look for threads in Transition state - evidence of swapping (usually caused by severe memory 2 pressure) 7 1 Screen snapshot from: Performance Monitor counters from Process object Balance Set Manager Nearest thing NT has to a “swapper” Balance set = sum of all inswapped working sets Balance Set Manager is a system thread Wakes up every second. If paging activity high or memory needed: trims working sets of processes if thread in a long user-mode wait, marks kernel stack pages as pageable if process has no nonpageable kernel stacks, “outswaps” process triggers a separate thread to do the “outswap” by gradually reducing target process’s working set limit to zero Evidence: Look for threads in “Transition” state in PerfMon Means that kernel stack has been paged out, and thread is waiting for memory to be allocated so it can be paged back in This thread also performs a scheduling-related function CPU starvation avoidance - already covered 9 Process Exploder Memory Information for a Process Virtual sizes of committed sections of image and DLLs or total of all (total selected) Virtual sizes of sections mapped after image startup (including DLLs loaded with LoadLibrary) Process-private committed virtual address space (i.e. paging file allocation) note, “writecopy” = “writeable, but not written to yet”. NT has yet to create process-private pages for these; they are still shared; they become “private commit” when written to Some, but not all, of this info is also shown by Process Viewer’s “memory detail” button Windows Operating System Internals - by David A.