Production Debugging for .NET Framework Applications

Total Page:16

File Type:pdf, Size:1020Kb

Production Debugging for .NET Framework Applications Production Debugging for .NET Framework Applications Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft, MS-DOS, Windows, Visual C#, Visual Basic, Visual C++, Visual Studio, and Win32 are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. © 2002 Microsoft Corporation. All rights reserved. Version 1.0 The names of actual companies and products mentioned herein may be the trademarks of their respective owners. Contents Chapter 1 Introduction to Production Debugging for .NET Framework Applications 1 Production Debugging vs. Development Debugging . 2 Environment . 2 Tools . 3 Actions . 3 Debugging Tools . 4 Discovery Phase . 4 Debugging Phase . 5 ASP.NET Process Model and Health Monitoring . 6 What Is the ASP.NET Process Model? . 7 IIS 5.x Process Model . 7 Health Monitoring in IIS 5.x . 9 IIS 6.0 Process Model in Windows .NET Server Release Candidate 1 . 9 IIS 6.0 Application Pools . 10 Orphaning Failed ASP.NET Worker Processes . 12 System Requirements . 12 Additional Software . 13 Documentation Conventions . 14 Summary . 15 Chapter 2 Debugging Memory Problems 17 .NET Memory Management and Garbage Collection . 17 Generations . 19 Roots . 19 Large Object Heap . 21 Scenario: Memory Consumption . 22 Breaking Down the Thought Processes . 22 Memory Consumption Walkthrough . 25 Debugging User-Mode Dump Files with WinDbg . 36 Analyzing the Dump: Summary . 41 Diagnosing Memory Leaks with the Allocation Profiler . 51 Conclusion . 55 iv Contents Chapter 3 Debugging Contention Problems 57 ASP.NET Thread Pools . 57 Managed Threads and AppDomains . 58 The .NET ThreadPool . 58 ASP.NET Thread Pool . 59 Scenario: Contention or Deadlock Symptoms . 60 Breaking Down the Thought Processes . 61 Contention Walkthrough . 64 Debugging a Dump File with WinDbg . 68 Debugging with Visual Studio .NET . 98 Conclusion . 102 Chapter 4 Debugging Unexpected Process Termination 103 Handling COM Threading in ASP.NET . 103 ASP.NET and COM Interop . 104 COM Components and ASP.NET . 104 Scenario: Unexpected Process Termination . 107 Breaking Down the Thought Processes . 107 Unexpected Process Termination Walkthrough . 109 Debugging a Dump File with WinDbg . 113 Examining the Dump File . 113 Conclusion . 146 Appendix Thread State Values . 148 Application Code . 149 Contention.aspx.cs . 149 Memory.aspx.cs . 151 Unexpected.aspx.cs . 155 Pmhealth.cs . 156 Debugging with CorDbg . 161 Using CorDbg to Display Call Stack Information . 162 Using a Script File . 164 The managed_threads.cmd Script . 170 Exploring Further . 171 1 Introduction to Production Debugging for .NET Framework Applications Every developer or support engineer debugs applications at some stage in his or her career, yet debugging is often viewed as an arcane and difficult topic. While it is often difficult to determine why an application is hanging, leaking memory, or crashing, there are techniques you can apply to make the debugging process more productive and efficient. The Production Debugging for .NET Framework Applications guide aims to equip you with the mindset, tools, and techniques that will help you successfully identify and resolve common application debugging. The techniques presented here are not simplified classroom exercises. They are based on the experience of both Microsoft customers and Microsoft internal devel- opment teams. These techniques represent proven, best-practice approaches to debugging a variety of issues. This guide presents walkthroughs of three scenarios that ASP.NET applications may encounter when running in production environments: memory consumption, contention (also known as “deadlock”), and unexpected server crashes. These scenarios concentrate on debugging Microsoft® .NET framework applications in a production environment, focusing on low-level details such as thread states and memory allocations. However, the thought processes and techniques discussed will help you understand debugging on a much broader scale. This document helps you learn how to think about debugging in a general sense, and equips you with the mindset to successfully approach unknown and unforeseen debugging situations in the future. 2 Production Debugging for .NET Framework Applications Although the example scenarios target ASP.NET and the Visual C#™ development tool, the thought processes and techniques discussed are common across all .NET technologies and languages, and problems encountered are common themes. The scenarios also introduce the tools that Microsoft provides for the Microsoft Win- dows® operating system and .NET debugging, including the use of WinDbg, Core Debugger (CDB), and debugger extension DLLs for displaying managed call stacks and object data. This chapter provides an introduction to the Production Debugging for .NET Framework Applications guide, defines the scope of the chapters, and provides background material. It also introduces the system requirements for the walkthroughs, and gives instructions for installing the required software. Production Debugging vs. Development Debugging The primary goal of debugging a system is to isolate and determine the root cause of a performance, configuration, or abnormal error condition that impedes normal system operation. Generally speaking, this goal holds true whether you are debug- ging in a development environment or in a production environment. However, there are crucial differences between the two environments. When debugging in a production environment, determining the root cause may be less important than simply getting the system into an operational state. Time constraints and business cases also differ between development and produc- tion environments. Debugging within a production environment is usually done within much tighter time constraints, given that the end user or customer may be losing revenue as a consequence of a non-operational system. The walkthroughs presented in this guide demonstrate many non-invasive tech- niques that can be used in a production debugging environment. Environment In a production environment, the system experiences live operational loads and timing situations, which may be difficult or impossible to reproduce in a develop- ment environment. In addition, scenarios can arise that have not been anticipated during system design. Timing-sensitive or load-sensitive scenarios that lead to abnormal behavior can be difficult to reproduce or debug successfully, even during live operation in a production environment. Because physical access to a production system is often restricted, offline or remote-access techniques, such as Terminal Services sessions and remote debugging sessions, are required. Chapter 1: Introduction to Production Debugging for .NET Framework Applications 3 Tools In a development environment, you usually have access to an integrated develop- ment environment (IDE), source code, and debug versions of libraries and symbols. In a production environment, the IDE and source.
Recommended publications
  • Deep Dive Into OS Internals with Windbg Malware and OS Internals
    2012 Deep Dive into OS Internals with Windbg Malware and OS Internals An approach towards reversing malwares, shellcodes and other malicious codes to understand the ways in which they use the OS Internals for their functionality. Sudeep Pal Singh Honeywell 1/26/2012 Page 1 Table of Contents Preface ............................................................................................................................................................................ 3 Reversing Windows Internals .......................................................................................................................................... 4 Portable Executable Anatomy ......................................................................................................................................... 5 Data Directories of Interest ............................................................................................................................................. 7 Import Directory .............................................................................................................................................................. 8 Import Address Table .................................................................................................................................................... 12 Export Directory ............................................................................................................................................................ 13 Manual Walkthrough of Export Directory ....................................................................................................................
    [Show full text]
  • Sample Content from Debugging Microsoft .NET 2.0 Applications
    Debugging Microsoft®.NET 2.0 Applications John Robbins (Wintellect) To learn more about this book, visit Microsoft Learning at http://www.microsoft.com/MSPress/books/8650.aspx 9780735622029 Publication Date: November 2006 Table of Contents Acknowledgments. .xv Introduction . .xvii Part I The Gestalt of Debugging 1 Bugs: Where They Come From and How You Solve Them . .3 Bugs and Debugging . 3 What Are Bugs? . 4 Process Bugs and Solutions . 8 Planning for Debugging . 17 Prerequisites to Debugging . 18 The Skill Set . 18 Learning the Skill Set . 20 The Debugging Process. 21 Step 1: Duplicate the Bug . 22 Step 2: Describe the Bug . 23 Step 3: Always Assume That the Bug Is Yours . 24 Step 4: Divide and Conquer . 24 Step 5: Think Creatively . 25 Step 6: Utilize Tools . 26 Step 7: Start Heavy Debugging . 27 Step 8: Verify That the Bug Is Fixed . 27 Step 9: Learn and Share. 29 Final Debugging Process Secret. 29 Summary . 30 2 Preparing for Debugging . 31 Track Changes Until You Throw Away the Project. 31 Version Control Systems . 32 Bug Tracking Systems . 36 Choosing the Right Systems for You . 37 Microsoft is interested in hearing your feedback about this publication so we can What do you think of this book? continually improve our books and learning resources for you. To participate in a brief We want to hear from you! online survey, please visit: www.microsoft.com/learning/booksurvey/ ix x Table of Contents Schedule Time for Building Debugging Systems . 38 Build All Builds with Debugging Symbols . 38 Treat Warnings as Errors . 41 Know Where Your Assemblies Load .
    [Show full text]
  • Practical-Malware-Analysis Index.Pdf
    INDEX Symbols and Numbers administrator privileges, for malware launchers, 254 ! (bang symbol), 305 Adobe Reader -- operation, 112 CVE-2010-0188 critical % operation, 112 vulnerability, 424 % symbol, 423 overflow in, 705 | (pipe symbol), in Snort, 304 ADS (Alternate Data Streams) ++ operation, 112 010 Editor, 468 feature, 139 32-bit applications, WOW64 and, 448 Advanced Encryption Standard 32-bit rotate-right-additive hash, 418 (AES), 618 64-bit malware, 441–449 decrypting, 625–626 clues to functionality, 448 advapi32.dll, 17 labs, 450–451 imports from, 20, 480, 481 solutions, 723–732 obtaining handle to, 237 advertisements, pop-up, 560–561 A AES (Advanced Encryption Standard), 618 A, at end of Windows function decrypting, 625–626 name, 17 Agobot, 376 absolute addresses, 443 air-gapped networks, 29 vs. relative addresses, in OllyDbg, _alloca_probe function, 522 184–185 alphabetic encoding, shellcode abstraction levels, in x86 disassembly, decoder with, 697 66–67 Alternate Data Streams (ADS) accept function, 143, 144, 454 feature, 139 access token, 246 ALU (arithmetic logic unit), 68 accuracy, vs. expediency, 304 AMD64 architecture, 441 active window, logging, 239 “Analysis of the Intel Pentium’s ADD encoding algorithm, 276 Ability to Support a Secure add instruction, 74, 349 Virtual Machine Monitor” AddCodeXref function (IDC), 342 (Robin and Irvine), 373 address space, loading executable AND logical operator, in x86 into another process’s, 595 architecture, 75 address space layout randomization anti-debugging, 351–366 (ASLR), 184 checks,
    [Show full text]
  • Windbg Superpowers for .NET Developers Sasha Goldshtein @Goldshtn CTO, Sela Group Github.Com/Goldshtn
    Software Design & Development #sddconf | @goldshtn https://s.sashag.net/sdd17-1 WinDbg Superpowers For .NET Developers Sasha Goldshtein @goldshtn CTO, Sela Group github.com/goldshtn 1 Software Design & Development #sddconf | @goldshtn https://s.sashag.net/sdd17-1 Agenda • Miscellaneous tips for making WinDbg friendlier • Mastering the superpower of scripts and breakpoints • Useful extensions to avoid tedious work • Startup and remote debugging sessions • Visual Studio can’t do 90% of what we’ll see today (or more) 2 Software Design & Development #sddconf | @goldshtn https://s.sashag.net/sdd17-1 Is Visual Studio A Powerful Debugger? • It’s a toy for people who like clicking things with the mouse • No macros • No scripts • No extensions* • Almost completely reliant on source code being present 3 Software Design & Development #sddconf | @goldshtn https://s.sashag.net/sdd17-1 Visual Studio WinDbg cdb ntsd debug.com Originally posted on Reddit 4 Software Design & Development #sddconf | @goldshtn https://s.sashag.net/sdd17-1 5 Software Design & Development #sddconf | @goldshtn https://s.sashag.net/sdd17-1 Why Is WinDbg So Scary? • Because I can’t remember all the obscure ugly commands • .cmdtree to the rescue! 0:000> .cmdtree cmdtree.txt 6 Software Design & Development #sddconf | @goldshtn https://s.sashag.net/sdd17-1 Why Is WinDbg So Scary? • Because commands are not discoverable, and require copy-pasting lots of hex numbers • .prefer_dml 1 (and a recent version of WinDbg) to the rescue! 7 Software Design & Development #sddconf | @goldshtn
    [Show full text]
  • Automating Problem Analysis and Triage Sasha Goldshtein @Goldshtn Production Debugging
    Automating Problem Analysis and Triage Sasha Goldshtein @goldshtn Production Debugging Requirements Limitations • Obtain actionable • Can’t install Visual information about Studio crashes and errors • Can’t suspend • Obtain accurate production servers performance • Can’t run intrusive information tools In the DevOps Process… Automatic build (CI) Automatic Automatic deployment remediation (CD) Automatic Automatic error triage monitoring and analysis Dump Files Dump Files • A user dump is a snapshot of a running process • A kernel dump is a snapshot of the entire system • Dump files are useful for post-mortem diagnostics and for production debugging • Anytime you can’t attach and start live debugging, a dump might help Limitations of Dump Files • A dump file is a static snapshot • You can’t debug a dump, just analyze it • Sometimes a repro is required (or more than one repro) • Sometimes several dumps must be compared Taxonomy of Dumps • Crash dumps are dumps generated when an application crashes • Hang dumps are dumps generated on-demand at a specific moment • These are just names; the contents of the dump files are the same! Generating a Hang Dump • Task Manager, right- click and choose “Create Dump File” • Creates a dump in %LOCALAPPDATA%\Te mp Procdump • Sysinternals utility for creating dumps • Examples: Procdump -ma app.exe app.dmp Procdump -ma -h app.exe hang.dmp Procdump -ma -e app.exe crash.dmp Procdump -ma -c 90 app.exe cpu.dmp Procdump -m 1000 -n 5 -s 600 -ma app.exe Windows Error Reporting • WER can create dumps automatically
    [Show full text]
  • Defcon DSE Bypass Workshop – Virtual Machine Setup Instruction
    Defcon DSE Bypass workshop – Virtual Machine setup instruction CSABA FITZL Table of Contents SETTING UP THE TESTING ENVIRONMENT ..................................................................................... 3 WINDOWS 10 X64 .................................................................................................................................... 3 WINDOWS 7 X64 AND 8.1 X64 (8.1 IS OPTIONAL) ......................................................................................... 4 TESTING INSTALLATION .............................................................................................................................. 4 Setting up the testing environment We will need two / three different virtual machines. You may use any virtualization software, but the instructor will use VMware. The software must have snapshot capabilities. You must be familiar using your own environment and have admin rights to do any changes if required. You can get a 30 day trial version of VMware from: https://my.vmware.com/web/vmware/downloads The VMs should be set up the following way: Windows 10 x64 Once installing a Windows 10 x64 version, we need to install the following software: 1. Windows 10 ISO can be downloaded from: https://www.microsoft.com/hu-hu/software- download/windows10ISO a. SHA1 hash: 08FBB24627FA768F869C09F44C5D6C1E53A57A6F, Filename: Win10_1803_English_x64.iso b. Also known as “en_windows_10_consumer_editions_version_1803_updated_march_2018_x64_dv d_12063379.iso“ 2. Visual Studio 2017 Community, available from: https://www.visualstudio.com/downloads/
    [Show full text]
  • Advanced .NET Debugging Kindle
    ADVANCED .NET DEBUGGING PDF, EPUB, EBOOK Mario Hewardt | 552 pages | 09 Dec 2009 | Pearson Education (US) | 9780321578891 | English | New Jersey, United States Advanced .NET Debugging PDF Book Thanks for this very interesting webinar! GetHashCode is to return an integer representing the reference id of the object in the. Video lessons Only. When I started this blog there was almost no knowledge online about advanced. Keep that in mind when you need to optimize the client load side of things. Check out a chapter from my book Practical Debugging for. However, these communications are not promotional in nature. Versus what version of. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes. Starting from. In particular, you can see how much time was spent on each method. As far as I could tell and its currently just an calculated guess , if you have a debugger installed and registered as a debugger,. NET Core There are numerous implementations of hashing functions built into. If you are not sure this is really your problem, its very easy figuring it out using WinDbg, by attaching to the process, loading the SOS. With this …. Brian Rasmussen Brian Rasmussen k 33 33 gold badges silver badges bronze badges. But where to start? In short, CDN can make your site faster by delivering static content i. Please be aware that we are not responsible for the privacy practices of such other sites. Topics: debugging , managed code , debugging with cdb. You can change the code while in break mode and VS knows how to deal with that.
    [Show full text]
  • Windbg. from a to Z!” Is a Quick Start and Introduction to Windbg
    WinDbg.WinDbg. FromFrom AA toto Z!Z! Everything you need to know about WinDbg. And nothing you don’t. By Robert Kuster December 2007. All rights reserved. www.windbg.info 1 WhyWhy WinDbg?WinDbg? Because WinDbg is: used by the Microsoft Windows product team to develop Windows much more powerful than the well-known Visual Studio debugger extensible through extension DLLs its debug engine is part of the Windows OS Up from Windows XP dgbeng.dll and dbghelp.dll are installed in “C:\Windows\System32”. 2 www.windbg.info WhyWhy ““WinDbgWinDbg .. FromFrom AA toto ZZ ”” ?? WinDbg's documentation is sub-optimal for people new to the topic Without good documentation and examples the learning curve for WinDbg is very steep In fact many people give up soon after the installation. “WinDbg. From A to Z!” is a quick start and introduction to WinDbg . After reading it you will have a good feeling about what WinDbg is and what it can do for you. While many parts of “WinDbg. From A to Z!” are based on user-mode examples, you will benefit from it even if you are doing kernel-mode development. Note that the same debugging engine is running behind the scenes, no matter if you debug user-mode or kernel-mode code. Essentially the only visible difference for kernel-mode debugging is that you will end up using another set of extension commands. 3 www.windbg.info TableTable ofof ContentsContents -- RoadmapRoadmap Behind the Scenes • Using WinDbg • Global Flags • Application Verifier • Process Dumps 4 www.windbg.info DebuggingDebugging ToolsTools forfor WindowsWindows XPXP • WinDbg.exe • ntsd.exe • cdb.exe • kd.exe dbgeng.dll • dbgsrv.exe • userdump.exe • drwtsn32.exe • livekd.exe • OlyDbg.exe dbghelp.dll • ProcessExplorer.exe • ..
    [Show full text]
  • Intro to Windows Debugger (Windbg) for .NET Developers and Concepts in C# Vijay Rodrigues (Vijayrod) Last Updated: 2011-11-30 I
    Intro to Windows Debugger (WinDBG) for .NET Developers And Concepts in C# Vijay Rodrigues (VijayRod) Last Updated: 2011-11-30 I’ve written this content as part of driving .Net Academy. Most of content has been written at home in my spare time with a real slow internet connection. Due to its very nature of publicly available symbols, .Net and its debugging information is publicly available on the internet. Hence this content can be freely redistributed for non-commercial use. A lot of acknowledgements to Ranjan Bhattacharjee for this idea, to Xavier S Raj for valuable mentorship and for the logo, and to Rishi Maini for his guidance whenever required. We’ve also discussed this with multiple resources and got ideas from them. They’ve made a difference - Kane Conway, Satya Madhuri Krovvidi, Dhiraj H N, Durai Murugan, Varun Dhawan, Balmukund Lakhani, Anmol Bhasin, Tejas Shah, Ajith Krishnan, Shasank Prasad, Chandresh Hegde. If you have any new inputs or suggestion that will help this effort please let me know. Table of Contents Section i: Getting Started 1. To give a start 2. Introduction and Concepts 3. Learning - videos (and books) 4. Learning - Advantages of C# over VB.NET and vice versa 5. Practical guide to .Net debugging 6. Useful websites Section ii: Introduction 7. Introduction 8. Intro to WinDBG for .NET Developers - part II 9. Intro to WinDBG for .NET Developers - part III (Examining code and stacks) 10. Intro to WinDBG for .NET Developers - part IV (Examining code and stacks contd.) 11. Intro to WinDBG for .NET Developers - part V (still more on Examining code and stacks) 12.
    [Show full text]
  • Accelerated Windows Memory Dump Analysis
    Published by OpenTask, Republic of Ireland Copyright © 2016 by OpenTask Copyright © 2016 by Software Diagnostics Services Copyright © 2016 by Dmitry Vostokov All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior written permission of the publisher. You must not circulate this book in any other binding or cover, and you must impose the same condition on any acquirer. Product and company names mentioned in this book may be trademarks of their owners. OpenTask books and magazines are available through booksellers and distributors worldwide. For further information or comments send requests to [email protected]. This book is an exerprt from ISBN-l3: 978-1-908043-46-7 (Paperback) Revision 1.00 (July 2016) 2 Contents About the Author.............................................................................................................................................................. 5 Minidump Analysis ........................................................................................................................................................... 7 Scripts and WinDbg Commands ................................................................................................................................... 7 Component Identification .......................................................................................................................................... 10 Raw Stack Data Analysis ............................................................................................................................................
    [Show full text]
  • Attacking the Windows Kernel
    Attacking the Windows Kernel Jonathan Lindsay ([email protected]) An NGSSoftware Insight Security Research (NISR) Publication ©2007 Next Generation Security Software Ltd http://www.ngssoftware.com Abstract Most modern processors provide a supervisor mode that is intended to run privileged operating system services that provide resource management transparently or otherwise to non-privileged code. Although a lot of research has been conducted into exploiting bugs in user mode code for privilege escalation within the operating system defined boundaries as well as what can be done if one has arbitrary supervisor access (typically related to modern root kit work), not a great deal of research has been done on the interface between supervisor and non-supervisor, and potential routes from one to the other. The biggest problem arises when trying to protect the kernel from itself - for example, under the IA32 architecture implementation of Windows, the distinction between user mode and kernel mode from the user mode perspective is easily enforced through hardware based protection. However, as the kernel is running as supervisor, how does the kernel make distinctions between what it should be accessing? This would be irrelevant if the supervisor was not exposed to interaction with supervisee; but that would defeat the purpose of having a kernel. This paper is focused on Windows and the Intel Architecture, and will briefly outline the current supervisor boundaries provided. Different attack vectors, along with relevant examples, will be provided to demonstrate how to attack the supervisor from the perspective of the supervised, as well as an outline of what possible architectures could be used to mitigate such attacks, such as the research operating system Singularity.
    [Show full text]
  • Codeview, the MS Debug Info Format, in LLVM
    CodeView, the MS debug info format, in LLVM Reid Kleckner Google (1 / 40) Talk overview 1. Why use CodeView? What are PDBs and CodeView? 2. CodeView format basics 3. CodeView deduplication techniques 4. Implementation status in LLVM 5. Lessons for LLVM (2 / 40) Terminology: What is CodeView? CodeView is stored inside object 㹁les and PDBs DWARF PDB is a container format written by linker and read by debugger dSYM/DWP CodeView :: DWARF PDB :: dSYM PDB :: DWP (3 / 40) Why add CodeView support to LLVM? Windows has a rich ecosystem built around PDBs: Visual Studio debugger WinDBG Stack dumpers built on dbghelp.dll Windows Performance Analyzer (WPA) pro㹁ler Symbol servers (4 / 40) Object file structure of CodeView Type information lives in one .debug$T section Symbol information (everything else) lives in .debug$S sections Symbol section broken down into subsections: Symbol records (most stuff), line table, string table, unwind info, etc struct Point { int x, y; }; int main() { Point p{1, 2}; printf("%d %d\n", p.x, p.y); } (5 / 40) Why split out type information? Type information is repeated in every TU Often dominates link input size Deduplicating type information: Reduces PDB output size Speeds up links (less IO) Speeds up debugging (more compact PDB) Splitting out type info makes this easier DWARF type units are similar (6 / 40) Type deduplication strategy Build a graph of type records Type records will be our graph nodes Type indices will be our graph edges Merge type graphs to deduplicate Problem: Graph isomorphism is slow! (7 / 40) Type
    [Show full text]