Debug notes

Process Monitor (or procmon) is a very powerful diagnostics tool for Windows. I remember when I first run it, a few years ago, I was overwhelmed by the number of events happening on my system each and every second. Thankfully, procmon has features to help you deal with this bulk of data, including filters, highlighters, and bookmarks. But sometimes even a filtered log is not enough to find the root cause of a problem. On such occasions, other log sources may contain the missing bits of information. We only need to correlate them with our procmon trace. And today, I would like to show you how I do that.

Log Files

The application log file is usually the first place we check after encountering a problem in the application. If they contain timestamps, finding the procmon events corresponding to the error message is quite straightforward – just set a filter on the Time of Day column. Things get more complicated if our logs are missing this information. In such a case, we may open the log file in a binary editor, locate our error message, and write down its offset in the file (binary editors usually have this information in the status bar), e.g.

In the next step, we open the trace file in procmon and search the Details column for the offset value. We may also add a filter on the Path to make the search faster. Notice that procmon prints numbers with thousand separators, so you need to add them, if necessary. Example result might look as follows:

We found a moment when the application recorded the error. We may safely exclude all later events and scroll up in search for the actual issue.

Exceptions

When diagnosing a faulty application, it is worth to check the exceptions it throws, both first and second chance. And the tool I use for this purpose is, not surprisingly, procdump. With an empty exception filter, procdump only outputs exception information, without taking any dumps. Additionally, if procmon is running at the same time, this information will appear in the recorded trace, e.g.

You may find more details in my older post describing this feature.

Custom events

The interaction with procmon event stream is not reserved to procdump only. By using a device installed by procmon in the system (.\Global\ProcmonDebugLogger), we may send custom profiling events to it. They will later appear in the trace the same way procdump output does, e.g.

Logging to procmon from.NET applications is fairly easy, thanks to the following libraries:

• NlogProcmon (Michał Grzegorzewski), implementing Nlog target
• ProcMonDebugOutput (by John Robbins), implementing System.Diagnostics.TraceListener and log4net appender

I also developed a command line tool: send2procmon, which will send any text it gets on the input (pipeline is supported) to procmon.

Network

When you need to diagnose networking issues in an application, will you consider using procmon? Until very recently, my answer to this question was no. I used Wireshark to record and analyze network traces. I am still a fan of Wireshark (especially with npcap as the capture driver), but more and more often procmon is also running in the background. Procmon shows only TCP and UDP activity, including source and destination addresses in the Path column, and message length in the Details column, e.g.

The information I am missing in Wireshark is the ID of a process that generated a given network event. And I can easily find it in procmon by using as filters IP addresses, port numbers and the time, when the event occurred. Additionally, if it turns out that the network trace does not contain the answer to my problem, I have many other events waiting for me in procmon to analyze.