Published by OpenTask, Republic of Ireland Copyright © 2016 by OpenTask Copyright © 2016 by Software Diagnostics Services Copyright © 2016 by Dmitry Vostokov All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior written permission of the publisher. You must not circulate this book in any other binding or cover, and you must impose the same condition on any acquirer. Product and company names mentioned in this book may be trademarks of their owners. OpenTask books and magazines are available through booksellers and distributors worldwide. For further information or comments send requests to [email protected]. This book is an exerprt from ISBN-l3: 978-1-908043-46-7 (Paperback) Revision 1.00 (July 2016) 2 Contents About the Author.............................................................................................................................................................. 5 Minidump Analysis ........................................................................................................................................................... 7 Scripts and WinDbg Commands ................................................................................................................................... 7 Component Identification .......................................................................................................................................... 10 Raw Stack Data Analysis ............................................................................................................................................. 15 Symbols and Images ................................................................................................................................................... 24 Wait Chain (Executive Resources) .................................................................................................................................. 27 3 4 About the Author 5 Dmitry Vostokov is an internationally recognized expert, speaker, educator, scientist and author. He is the founder of pattern-oriented software diagnostics, forensics and prognostics discipline and Software Diagnostics Institute (DA+TA: DumpAnalysis.org + TraceAnalysis.org). Vostokov has also authored more than 30 books on software diagnostics, forensics, problem-solving, memory dump analysis, debugging, software trace and log analysis, reverse engineering, and malware analysis. He has more than 20 years of experience in software architecture, design, development and maintenance in a variety of industries including leadership, technical and people management roles. Dmitry also founded DiaThings, Logtellect, OpenTask Iterative and Incremental Publishing (OpenTask.com), Software Diagnostics Services (former Memory Dump Analysis Services) PatternDiagnostics.com and Software Prognostics. In his spare time, he presents various topics on Debugging.TV and explores Software Narratology, an applied science of software stories that he pioneered, and its further development as Narratology of Things and Diagnostics of Things (DoT). 6 Minidump Analysis Reprinted from Memory Dump Analysis Anthology, Volume 1, pages 43 – 67 Scripts and WinDbg Commands Small Memory Dumps, also referred to as minidumps because they are stored in %SystemRoot% \ Minidump folder, contain only bugcheck information, kernel mode stack data, and the list of loaded drivers. They can be used to transmit system crash information to a vendor or a 3rd-party for an automated crash dump analysis. Another use is to keep system crash history. In this part, I discuss the scripting approach to extract information from all minidumps stored on a particular system. The script processes all minidump files and creates text log files containing the following information: 1. Crash dump name and type. 2. OS information, crash time and system uptime. 3. Processor context (r) and the verbose stack trace (kv) prior to applying !analyze -v. This is useful sometimes when WinDbg reconstructs a different stack trace after changing a processor context to the execution context at the time of a trap, exception or fault. 4. The output of !analyze -v command. 5. Processor context (r) and verbose stack trace (kv) after !analyze -v command. 6. Code disassembly for the current execution pointer (EIP or x64 RIP). This includes forward (u) and backward (ub) disassembly, and we also try to disassemble the whole function (uf) which should succeed if we have symbolic information. 7. Raw stack dump with symbol information (dps). 8. The same raw stack data but interpreted as pointers to Unicode zero-terminated strings (dpu). Some pointers on the stack might point to local string buffers located on the same stack. This can be a slow operation, and WinDbg may tem- porarily hang. 9. The same raw stack data but interpreted as pointers to ASCII zero-terminated strings (dpa). This can be a slow operation, and WinDbg may temporarily hang. 10. Verbose information about loaded drivers (lmv). 11. CPU, machine ID, machine-specific registers, and verbose SMBIOS information like motherboard and devices (!sysinfo). 7 Here is listing of our WinDbg script: $$ $$ MiniDmp2Txt: Dump information from minidump into log $$ .logopen /d /u .echo "command> ||" || .echo "command> vertarget" vertarget .echo "command> r (before analysis)" r .echo "command> kv (before analysis)" kv 100 .echo "command> !analyze -v" !analyze -v .echo "command> r" r .echo "command> kv" kv 100 .echo "command> ub eip" ub eip .echo "command> u eip" u eip .echo "command> uf eip" uf eip .echo "command> dps esp-3000 esp+3000" dps esp-3000 esp+3000 .echo "command> dpu esp-3000 esp+3000" dpu esp-3000 esp+3000 .echo "command> dpa esp-3000 esp+3000" dpa esp-3000 esp+3000 .echo "command> lmv" lmv .echo "command> !sysinfo cpuinfo" !sysinfo cpuinfo .echo "command> !sysinfo cpuspeed" !sysinfo cpuspeed .echo "command> !sysinfo cpumicrocode" !sysinfo cpumicrocode .echo "command> !sysinfo gbl" !sysinfo gbl .echo "command> !sysinfo machineid" !sysinfo machineid .echo "command> !sysinfo registers" !sysinfo registers .echo "command> !sysinfo smbios -v" !sysinfo smbios -v .logclose $$ $$ MiniDmp2Txt: End of File $$ 8 To run WinDbg automatically against each minidump file (.dmp) we can use the following VB script (we need to customize symbol search path after -y parameter to point to our own symbol folders): ' ' MiniDumps2Txt.vbs ' Set fso = CreateObject("Scripting.FileSystemObject") Set Folder = fso.GetFolder(".") Set Files = Folder.Files Set WshShell = CreateObject("WScript.Shell") For Each File In Files If Mid(File.Name,Len(File.Name)-3,4) = ".dmp" Then Set oExec = WshShell.Exec("C:\Program Files\Debugging Tools for Windows\WinDbg.exe -y ""srv*c:\ms*http://msdl.microsoft.com/download/symbols"" -z " + File.Name + " -c ""$$><c:\scripts\MiniDmp2Txt.txt;q"" -Q -QS -QY -QSY") Do While oExec.Status = 0 WScript.Sleep 1000 Loop End If Next ' ' MiniDumps2Txt.vbs: End of File ' We can also use kd.exe instead of WinDbg, but its window will be hidden if we use the same VB script. 9 Component Identification Now I will show how to find a 3rd-party driver that might have been responsible for a system crash and verify that WinDbg reports that driver correctly. For example, one of the log files shows the following output from !analyze -v WinDbg command: command> !analyze -v ************************************************************* * * * Bugcheck Analysis * * * ************************************************************* BAD_POOL_CALLER (c2) The current thread is making a bad pool request. Typically this is at a bad IRQL level or double freeing the same allocation, etc. Arguments: Arg1: 00000047, Attempt to free a non-allocated nonpaged pool address Arg2: 85083000, Starting address Arg3: 00005083, physical page frame Arg4: 000bfff9, highest physical page frame Debugging Details: ------------------ BUGCHECK_STR: 0xc2_47 CUSTOMER_CRASH_COUNT: 2 DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP PROCESS_NAME: 3rdPartyAntivirus.exe CURRENT_IRQL: 0 LAST_CONTROL_TRANSFER: from 8054d2eb to 805435b9 STACK_TEXT: b325db68 8054d2eb 000000c2 00000047 85083000 nt!KeBugCheckEx+0x19 b325db94 805689c2 85083000 00080000 85083000 nt!MmGetSizeOfBigPoolAllocation+0x1cb b325dbe4 f77c8098 00000000 00000000 00000004 nt!ExFreePoolWithTag+0x1d0 WARNING: Stack unwind information not available. Following frames may be wrong. b325dc24 f77c8234 8599d4e0 8599d4e0 88cecf38 3rdPartyAVDrv+0×1098 b325dc3c 804f04f3 871d2cf0 00004808 8743b6a0 3rdPartyAVDrv+0×1234 b325dc4c 80585208 8599d550 872c8028 8599d4e0 nt!IofCallDriver+0×3f b325dc60 80585fe6 871d2cf0 8599d4f0 872c8028 nt!IopSynchronousServiceTail+0×6f b325dd00 80586028 00000468 00000614 00000000 nt!IopXxxControlFile+0×607 b325dd34 804dfd24 00000468 00000614 00000000 nt!NtDeviceIoControlFile+0×28 b325dd34 7ffe0304 00000468 00000614 00000000 nt!KiSystemService+0xd0 0087b6c0 00000000 00000000 00000000 00000000 SharedUserData!SystemCallStub+0×4 STACK_COMMAND: kb 10 FOLLOWUP_IP: 3rdPartyAVDrv+1098 f77c8098 ?? ??? SYMBOL_STACK_INDEX: 3 SYMBOL_NAME: 3rdPartyAVDrv+1098 FOLLOWUP_NAME: MachineOwner MODULE_NAME: 3rdPartyAVDrv IMAGE_NAME: 3rdPartyAVDrv.sys DEBUG_FLR_IMAGE_TIMESTAMP: 410752c5 FAILURE_BUCKET_ID: 0xc2_47_3rdPartyAVDrv+1098 BUCKET_ID: 0xc2_47_3rdPartyAVDrv+1098 Followup: MachineOwner --------- MODULE_NAME and IMAGE_NAME fields report 3rd-party antivirus driver 3rdPartyAVDrv.sys as responsible for BSOD. However, the top lines from STACK_TEXT field report nt module,