This is a repost of my article, originally published on CodeProject on 24 May 2016.
The Story of a Memory Dump
Memory dumps are a common way to diagnose various problems with our applications (such as memory leaks or application hangs). You may think of them as photos which allow you to have a look at the past and notice all the details you might have missed. There are different types of memory dumps which we may compare to different types of photos we take:
- minimal – focus is on one element (such as an exception) and the whole background is blurry, they take very little space on the hardrive (eg. 2MB)
- minidumps with thread and process data/heaps/stacks/exception data, etc. – depending on how many options we choose, they might be very detailed high-resolution pictures or very blurry ones, the range of space they take can vary from tens of MBs to several GBs
- full memory dumps – those can be compared to high-resolution pictures, they are as big as the whole process committed virtual memory
It’s often not an easy task to decide which memory dump we should take. When our application is failing under some specific conditions, a minimal memory dump should be enough (it’s actually the default format for Windows Error Reporting) as it will show you the exception data which might suffice to fix the error. Unfortunately, pretty often the problems we meet are much more complicated and there are many places to check in the memory before we can state the diagnosis. This is especially true for .NET processes which tend to consume much bigger amounts of memory compared to their native counterparts. You may ask: why is this happening? The answer is pretty straightforward. As you know, managed binaries do not contain code which is ready to be run directly on a CPU – there is the whole JIT (Just In Time compilation) layer which needs to convert IL (Intermediate Language) to the native form. We also have Garbage Collector code responsible for memory management and other components (such as assembly loader or thread scheduler). All those parts of the Common Language Runtime are native libraries which require memory to function properly. Finally, on top of that comes the managed memory composed of GC Heaps, where we store our precious objects. As you see a lot of memory to allocate, a lot of memory which is not relevant to our precious objects. So what can we do? We can run
procdump -ma and watch painfully the free space counter decrementing on our hard drive or …
But first some history! MiniDumper was brought to life by Sasha Goldstein in August 2015. In a subsequent post, Sasha describes in detail the idea and building blocks of the application. In simple words: MiniDumper dumps all the memory necessary to diagnose problems in managed code, thus skipping all the CLR native allocations – in a moment, I will show you some statistics. However, the initial version of MiniDumper was lacking some features, such as a way to respond to events happening inside an application (thrown exceptions, process exit, etc.) or a way to monitor an application from the very beginning. I am used to creating dumps with procdump – its command line is for me an unwritten standard for tools of this kind so I decided to port the procdump command line to minidumper. This is how MiniDumper 2.0 was born. You may find the description of my struggle on my blog. Unfortunately, not all features of procdump are yet implemented, so if you have some time, you know where to knock 🙂 Current stable version is 2.1.1 and you can download it from the release page.
You may now ask why I’m advertising MiniDumper? What’s so special about this tool? Let’s have a look at some statistics:
| Application type | Working Set | Procdump (-ma) | MiniDumper (-mh) |------------------|--------------|-----------------|------------------ | Console | 57MB | 60MB | 5MB | Windows Forms | 250MB | 263MB | 30MB | ASP.NET MVC | 362MB | 379MB | 98MB
As you can see, the dumps created with MiniDumper are much smaller. If I tell you that you may load them into WinDbg and netext commands will work, will you believe me? Well, have a look then:
0:000> .load netext netext version 220.127.116.1100 Jan 21 2016 License and usage can be seen here: !whelp license Check Latest version: !wupdate For help, type !whelp (or in WinDBG run: '.browse !whelp') Questions and Feedback: http://netext.codeplex.com/discussions Copyright (c) 2014-2015 Rodney Viana (http://blogs.msdn.com/b/rodneyviana) Type: !windex -tree or ~*e!wstack to get started 0:000> !windex Starting indexing at 19:18:20 1000000 objects... Indexing finished at 19:19:19 165,751,594 Bytes in 1,088,257 Objects Index took 00:00:58 0:000> !wfrom -type string ERROR: !wfrom: extension exception 0x80070057. "Missing required argument ''" 0:000> !windex -type System.String Index is up to date If you believe it is not, use !windex -flush to force reindex Address MT Size Heap Gen Type Name 0ffd14f8 5a50e918 38 0 1 System.String 0ffd1520 5a50e918 34 0 1 System.String 0ffd1754 5a50e918 112 0 1 System.String 0ffd1ae0 5a50e918 34 0 1 System.String 0ffd1b04 5a50e918 48 0 1 System.String ... 0:000> !wdo 0ffd1ae0 Address: 0ffd1ae0 Method Table/Token: 5a50e918/200006804 Class Name: System.String Size : 34 EEClass: 5a14f344 Instance Fields: 2 Static Fields: 1 Total Fields: 4 Heap/Generation: 0/1 Module: 5a0d0000 Assembly: 0725d200 Domain: 5c896670 Assembly Name: C:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_18.104.22.168__b77a5c561934e089\mscorlib.dll Inherits: System.Object (5A50ECB8) String: .appDomain 5a5107a0 System.Int32 +0000 m_stringLength a (0n10) 5a50f35c System.Char +0004 m_firstChar . 5a50e918 Static System.String +0040 Empty 00000000
Not always everything runs so smoothly. I noticed that SOS command, such as
DumpHeap might report problems for some dumps. Netext in such cases might be a better choice as under the hood it is using the same CLRMD library as
We are reaching the end of this article so let me show you some usage examples (the full help can be found on the project main page or using the
Print first-chance Exception Information with No Dump
PS x64> .\MiniDumper.exe -e1 -mh -f "NotExistingException" -x d:\temp .\Test.exe MiniDumper - writes .NET process dump files Copyright (C) 2015 Sasha Goldstein (@goldshtn) With contributions from Sebastian Solnica (@lowleveldesign) Process: Test (4912) Exception monitor: First Chance+Unhandled Exception filter: NotExistingException Dump folder: d:\temp Number of dumps: 1 Dump filename/mask: PROCESSNAME_YYMMDD_HHMMSS Terminal monitor: Disabled Debug output: Disabled Press Ctrl-C to end monitoring without terminating the process. Press key to throw an exception [19:31.29] Exception: E0434352.System.Exception ("test exception") Press any key to continue...
Make a Dump on a first-chance System.Exception Exception
PS x64> .\MiniDumper.exe -e1 -mh -f "System.Exception" -x d:\temp .\Test.exe MiniDumper - writes .NET process dump files Copyright (C) 2015 Sasha Goldstein (@goldshtn) With contributions from Sebastian Solnica (@lowleveldesign) Process: Test (3360) Exception monitor: First Chance+Unhandled Exception filter: System.Exception Dump folder: d:\temp Number of dumps: 1 Dump filename/mask: PROCESSNAME_YYMMDD_HHMMSS Terminal monitor: Disabled Debug output: Disabled Press Ctrl-C to end monitoring without terminating the process. Press key to throw an exception [19:33.23] Exception: E0434352.System.Exception ("test exception") [19:33.23] Dumping process memory to file: d:\temp\Test_160524_193323.dmp Number of dumps exceeded the specified limit - detaching. Press any key to continue...
Attach to an iisexpress Process and Make a Dump on first-chance Exception
PS x86> .\minidumper -mh -e1 iisexpress.exe MiniDumper - writes .NET process dump files Copyright (C) 2015 Sasha Goldstein (@goldshtn) With contributions from Sebastian Solnica (@lowleveldesign) Process: iisexpress.exe (3724) Exception monitor: First Chance+Unhandled Exception filter: * Dump folder: C:\Users\Sebastian\OneDrive\minidumper\x86 Number of dumps: 1 Dump filename/mask: PROCESSNAME_YYMMDD_HHMMSS Terminal monitor: Disabled Debug output: Disabled Press Ctrl-C to end monitoring without terminating the process. [19:36.08] Exception: E0434352.System.UnauthorizedAccessException ("Access to the path 'C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\~AspAccessCheck_29d98c541080.tmp' is denied.") [19:36.09] Dumping process memory to file: C:\Users\Sebastian\OneDrive\minidumper\x86\iisexpress.exe_160524_193609.dmp Number of dumps exceeded the specified limit - detaching.
As I stated previously, the current command line is quite limited (you may only make dumps when exception occurs or process exits), but I have plans to add support for performance counter values, which should cover many other diagnostics scenarios.
The latest binaries are published on the release page: https://github.com/goldshtn/minidumper/releases.
I hope you will find this tool useful and you will add it to your toolkit. Feel free to contact me if you have any problems running
MiniDumper or have questions how to run it.
- http://blogs.microsoft.co.il/sasha/2015/08/19/minidumper-smaller-dumps-net-applications/ – Original post by Sasha
- http://blogs.microsoft.co.il/sasha/2015/09/30/more-on-minidumper-getting-the-right-memory-pages-for-net-analysis/ – Details of the architecture by Sasha
- https://lowleveldesign.wordpress.com/2015/12/21/new-features-coming-to-minidumper/ – Changes in
MiniDumper2.0 and some internals
- Original repo: https://github.com/goldshtn/minidumper