In today’s short post I would like to present you a new tool in my diagnostics toolkit: wtrace, and an update to procgov (or Process Governor). Let’s start with wtrace.
On Linux, when I need to check what a given process is doing, I usually use strace. I was always missing such a tool for Windows. We have procmon (which is great), but it does not run in a console, and thus can’t be used in the command line scripts, or on a Nano server. This might change soon, as in one of the latest episodes of the Defrag Tools show, Mark Russinovich shared the plan of releasing the procmon version for Nano. Till then though we don’t have much choice when it comes to real-time tracing. You may think of xperf or wpr, but those tools only record ETW events for further analysis. However, we may use the same ETW events in a realtime session, and print information they provide to the console output. This is how the idea for wtrace was born in my head. Few weeks ago Sasha Goldshtein released another tool for ETW processing named etrace, which basically does something very similar and has many interesting options. I decided to publish wtrace nonetheless, as my point was to create a tool with an extremely simple interface. Wtrace is collecting only a small subset of events (FileIO, TcpIp, Process/Thread Start) from the kernel provider. It may either start a process, or trace one that is already running. At the end of the trace it also shows some statistics (unless you use the –nosummary switch). Trace session will end either when you press Ctrl+C, or when the traced process terminates. Events are printed in the console window. An example session might look as follows:
This is how the story begins. On our build server we are using a JetBrains Resharper CLT to generate a code analysis report. In one of the projects build we started getting the following exception log:
Executing the powershell script: C:\install\TFS \1.0.691\resharp.ps1
JetBrains Inspect Code 2016.1.2
Running in 64-bit mode, .NET runtime 4.0.30319.42000 under Microsoft Windows NT 6.2.9200.0
'AutoMapper' already has a dependency defined for 'Microsoft.CSharp'.
--- EXCEPTION #1/2 [InvalidOperationException]
Message = "'AutoMapper' already has a dependency defined for 'Microsoft.CSharp'."
ExceptionPath = Root.InnerException
ClassName = System.InvalidOperationException
HResult = COR_E_INVALIDOPERATION=80131509
Source = NuGet.Core
StackTraceString = "
at NuGet.Manifest.ValidateDependencySets(IPackageMetadata metadata)
at NuGet.Manifest.ReadFrom(Stream stream, IPropertyProvider propertyProvider, Boolean validateSchema)
at NuGet.LocalPackage.ReadManifest(Stream manifestStream)
at NuGet.SharedPackageRepository.OpenPackage(String path)
at NuGet.LocalPackageRepository.GetPackage(Func`2 openPackage, String path)
at NuGet.LocalPackageRepository.<>c__DisplayClass13.<FindPackage>b__f(String path)
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
at NuGet.LocalPackageRepository.FindPackage(Func`2 openPackage, String packageId, SemanticVersion version)
at NuGet.SharedPackageRepository.FindPackage(String packageId, SemanticVersion version)
at JetBrains.ProjectModel.Packages.SharedPackageRepositoryInTemp.FindPackage(String packageId, SemanticVersion version)
at NuGet.PackageRepositoryExtensions.FindPackage(IPackageRepository repository, String packageId, SemanticVersion version, IPackageConstraintProvider constraintProvider, Boolean allowPrereleaseVersions, Boolean allowUnlisted)
at NuGet.PackageReferenceRepository.GetPackage(PackageReference reference)
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at JetBrains.Util.ILoggerEx.Catch[TValue](ILogger th?s, Func`1 F, ExceptionOrigin origin, LoggingLevel loggingLevel)
I am working on adding a support for ASP.NET performance counters into Musketeer. Compared to other .NET performance counters they have quite surprising instance names. ASP.NET developers decided that their performance counter instances will be identified by names derived from the AppDomain names (more information can be found here). This is probably due to a fact that one process may host multiple ASP.NET applications, thus one counter instance per process won’t be enough. Consequently, in order to match collected metrics with process ids we need to know which AppDomain belongs to which process. How can we do that?
For some time I have been working on a monitoring solution for developers. Today I have a pleasure to announce its first official release. It is a set of tools which should help you better diagnose your applications. As there are many monitoring solutions on the market you may be using one of them (and that’s great). However, I’ve observed that it’s still uncommon for developers to collect application logs in one place. Therefore OPS monitor IIS logs and developers are checking application-specific targets. This is not the best approach as you can’t see at first sight if something is going wrong with your application. Few years ago I had an idea of an application board which will show statuses of applications on all the servers. This is one of the central part of the Diagnostics Kit and I named it the Diagnostics Castle. A sample board might look as follows:
Some time ago Sasha Goldstein wrote on his blog a post about a new open-source tool he created: minidumper. The idea is great and allows you to create more compact memory dumps for .NET processes. I completely agree with Sasha that most of the time when you are working with .NET processes memory dumps you are focused on .NET memory heap. All native elements are usually not necessary and take a lot of space in dumps. Having said that, I need to admit that there were times when those native parts were invaluable in my diagnosis, such as diagnosing native memory leaks (when using pinvoke) or handle leaks. But those are quite rare and in such situations we have no choice but to use the full memory dumps. For all other .NET problems you are welcome to use minidumper. If you are interested in minidumper internals there is another post on Sasha’s blog which explains them very well.
Today’s short post was inspired by a recent memory leak in Nancy. I thought it’s worth to describe it in detail as the reason why the memory was leaking was not so obvious and many of us could commit the same mistake as Nancy authors. The leak was present in the
NancyEngine class, which is the central point in the Nancy request handling process. In other words: each request served by the Nancy application must pass through this class instance.
NancyEngine processes the requests asynchronously and accepts as a parameter a
CancellationToken instance – thus making it possible to cancel the request from the “outside”. At the same time it uses an internal
CancellationTokenSource instance which cancels the current requests when the engine is getting disposed. As you see there are two cancellation tokens involved and the
HandleRequest method needs to respect their statuses. For such an occasion there is a method in the
CancellationTokenSource class in .NET Framework which creates for you a special “linked”
CancellationTokenSource that will depend on values in the related tokens. From now on you don’t need to worry about the other tokens as whenever they get cancelled your linked token will become cancelled too. With this introduction the prologue of the
HandleRequest becomes clear:
I have been playing recently with quite a new windbg extension (released by Rodney Viana from Microsoft) called NetExt. Rodney Viana published an introductory post about it, which you may find on his blog. In this post I would like to show you my usage samples as well as encourage you to start using it by yourself. Netext documentation is thorough and nicely organized which is good because at the beginning you probably will spend a lot of time on this page 🙂 In paragraphs that follow I will focus mainly on dump debugging, but most of the techniques presented here should work as well in live debugging sessions.
This post is the second and final one dedicated to debugging .NET Windows services (you may read the first one here). The inwrap tool (presented in the first part) is not very friendly to use and I myself haven’t used it since then 🙂 It’s not the best advertisement of someone’s own work, but it did motivate me to write another tool which maybe will gain your interest. The winsvcdiag is a simple application that allows you to debug a start of a Windows service from Visual Studio (or any other debugger – even the remote one).
I recently had an interesting issue in one of our applications. The SMS router, responsible for sending and receiving SMSes, hanged – there was no CPU usage and we haven’t observed any activity in the application logs. I collected a full memory dump and restarted the service, which seemed to come back to its normal state. Curious what happened I opened the dump in WinDbg, loaded PDE and SOS and started an investigation
In one of our applications I recently observed timeouts in code performing HTTP requests to the REST service. While investigating this issue I discovered few interesting facts about System.Net namespace and would like to share them with you. We were using objects of type System.Net.HttpWebRequest in our code, but some of the information presented in this post will also apply to the newer System.Net.HttpClient implementation.