MiniDumper is a diagnostic tool for collecting memory dumps of .NET applications. Dumps created by MiniDumper are significantly smaller than full-memory dumps collected by, for example, procdump. However, they contain enough information to diagnose most of the issues in the managed applications. MiniDumper was initially developed by Sasha Goldstein, and I made few contributions to its code base. You may learn more about this tool from Sasha’s or my blog posts.
Recently, one of MiniDumper users reported a memory leak in the application. The numbers looked scary as there was a 20MB leak on each memory dump. The issue stayed opened a few weeks before I finally found a moment to look at it. As it was quite a compelling case, I decided to share with you the diagnostics steps in the hope it proves useful in your investigations.
Every developer knows that unit testing improves the quality of the code. We also profit from static code analysis and use tools such as SonarQube in our build pipelines. Yet, I still find that many developers are not aware of a much older way of checking the validity of the code: assertions. In this post, I will present you the benefits of using assertions, as well as some configuration tips for .NET applications. We will also learn how .NET and Windows support them.
I recently spent some time analyzing OutputDebugString method. For my another experiment I needed a version of OutputDebugString which depends only on Native API. While implementing it, I discovered few interesting facts about OutputDebugString that maybe will interest you too. The title mentions System.Diagnostics.Trace. It is because the default trace configuration in .NET sends trace messages to an instance of the DefaultTraceListener class, which uses OutputDebugString. And if you do not remove it explicitly from the trace listener collection, your logs will always go through it. You will later see why sometimes it might not be a good idea.
One of my colleagues at work was struggling with a peculiar problem on his machine. Whenever he tried to access the address of his test project: http://my.project:8080 he was getting connection refused error (my.project points to 127.0.0.1 in the hosts file). The same error appeared when we opened the http://127.0.0.1:8080 address:
Recently, the idea of protected variables in TFS struck my attention and pushed me to do some more research on how exactly those variables are stored. I hope I will write a separate post on that subject, but today I would like to share with you a small trick I use whenever I need to work with managed application traces (and TFS is one of them).
On Windows, when I want to know how things work internally, I usually start with procmon. Seeing which paths and registry keys are accessed, combined with TCP/IP connections is often enough to get an idea where to put breakpoints in further analysis. My TFS investigation was no exception to this rule. I collected a trace while saving a protected build variable – this is how such a variable looks like (in case you are interested :)):
Process Monitor (or procmon) is a very powerful diagnostics tool for Windows. I remember when I first run it, a few years ago, I was overwhelmed by the number of events happening on my system each and every second. Thankfully, procmon has features to help you deal with this bulk of data, including filters, highlighters, and bookmarks. But sometimes even a filtered log is not enough to find the root cause of a problem. On such occasions, other log sources may contain the missing bits of information. We only need to correlate them with our procmon trace. And today, I would like to show you how I do that.
Application Insights is a performance monitoring service, created by Microsoft and available on Azure. It gives you space to store the performance metrics and logs of your application (1GB for free!), as well as functionalities to search and manage them. In this post I am not going to present you the whole platform – Microsoft already did it in the Azure documentation, but rather focus on an element of the log collection, named dependency calls tracking. I did some analysis on the Application Insights libraries, and decided to publish my findings, in the hope that the results might interest some of you too.
Dependency calls are requests, which your application makes to the external services (such as databases or REST services). When this telemetry type is enabled, all the dependent actions form a timeline within the scope of the parent action. Using this timeline we may easily verify whether the delay in our application is caused by an external service, or the application itself. Let’s analyze in detail how this data is collected.
This is how the story begins. On our build server we are using a JetBrains Resharper CLT to generate a code analysis report. In one of the projects build we started getting the following exception log:
Executing the powershell script: C:\install\TFS \1.0.691\resharp.ps1
JetBrains Inspect Code 2016.1.2
Running in 64-bit mode, .NET runtime 4.0.30319.42000 under Microsoft Windows NT 6.2.9200.0
'AutoMapper' already has a dependency defined for 'Microsoft.CSharp'.
--- EXCEPTION #1/2 [InvalidOperationException]
Message = "'AutoMapper' already has a dependency defined for 'Microsoft.CSharp'."
ExceptionPath = Root.InnerException
ClassName = System.InvalidOperationException
HResult = COR_E_INVALIDOPERATION=80131509
Source = NuGet.Core
StackTraceString = "
at NuGet.Manifest.ValidateDependencySets(IPackageMetadata metadata)
at NuGet.Manifest.ReadFrom(Stream stream, IPropertyProvider propertyProvider, Boolean validateSchema)
at NuGet.LocalPackage.ReadManifest(Stream manifestStream)
at NuGet.SharedPackageRepository.OpenPackage(String path)
at NuGet.LocalPackageRepository.GetPackage(Func`2 openPackage, String path)
at NuGet.LocalPackageRepository.<>c__DisplayClass13.<FindPackage>b__f(String path)
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
at NuGet.LocalPackageRepository.FindPackage(Func`2 openPackage, String packageId, SemanticVersion version)
at NuGet.SharedPackageRepository.FindPackage(String packageId, SemanticVersion version)
at JetBrains.ProjectModel.Packages.SharedPackageRepositoryInTemp.FindPackage(String packageId, SemanticVersion version)
at NuGet.PackageRepositoryExtensions.FindPackage(IPackageRepository repository, String packageId, SemanticVersion version, IPackageConstraintProvider constraintProvider, Boolean allowPrereleaseVersions, Boolean allowUnlisted)
at NuGet.PackageReferenceRepository.GetPackage(PackageReference reference)
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at JetBrains.Util.ILoggerEx.Catch[TValue](ILogger th?s, Func`1 F, ExceptionOrigin origin, LoggingLevel loggingLevel)
I am working on adding a support for ASP.NET performance counters into Musketeer. Compared to other .NET performance counters they have quite surprising instance names. ASP.NET developers decided that their performance counter instances will be identified by names derived from the AppDomain names (more information can be found here). This is probably due to a fact that one process may host multiple ASP.NET applications, thus one counter instance per process won’t be enough. Consequently, in order to match collected metrics with process ids we need to know which AppDomain belongs to which process. How can we do that?
In the last post I presented you my first WinDbg extension with a !injectdll command. Theoretically everything was correct, but after some more testing I noticed that the command is not always working as expected. Andrey Bazhan was pretty quick in pointing this out and advised me to use a remote thread, which, as you will see, is a much better approach. But let’s first have a look at the problems in lld 1.0.