Recently, I was looking into the internals of the Visual Studio debugger for the .NET Diagnostics Expert course. I was especially interested in how the Docker debugging works. For those of you who haven’t tried it yet, let me provide a concise description.
In Visual Studio 2019, when we work on the ASP.NET Core project, it is possible to create a launch profile that points to a Docker container, for example:
And that’s fantastic as we can launch the container directly from Visual Studio. And what’s even better, we can debug it! To make this all work, Visual Studio requires a Dockerfile in the root project folder. The default Dockerfile (which you can create in the ASP.NET Core application wizard) looks as follows:
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
COPY ["WebApplication1.csproj", ""]
RUN dotnet restore "./WebApplication1.csproj"
COPY . .
RUN dotnet build "WebApplication1.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "WebApplication1.csproj" -c Release -o /app/publish
FROM base AS final
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "WebApplication1.dll"]
And that’s it. If we press F5, we land inside an application container, and we can step through our application’s code. It all looks like magic, but as usual, there are protocols and lines of code that run this machinery behind the magical facade. And in this post, we will take a sneak peek at them 😊.
While working on a new version of wtrace, I am analyzing the PerfView source code to learn how its various features work internally. One of such features is the call stack resolution for ETW events. This post will show you how to use the TraceEvent library to decode call stacks, and, as an exercise, we will write a sampling process profiler. Before we start, remember to set DisablePagingExecutive to 1. That is a requirement to make call stacks work for ETW sessions.
❗ ❗ ❗ Visit wtrace.net to receive updates on wtrace and my other troubleshooting tools. ❗❗❗
If you are working on Windows, you know that the registry is a crucial component of this system. It contains lots of system and application configuration data. Apps use the registry to access some of the in-memory OS data. Therefore, monitoring the registry activity is one of the essential parts of the troubleshooting process. Fortunately, we have several tools to help us with this task, Process Monitor being probably the most popular one. In this post, though, I am going to prove that we could use ETW for this purpose as well.
This month marks ten years since I started this blog 🥂🥂🥂. On this occasion, I would like to thank you for being my reader! Let’s celebrate with a new post on ETW 🙂
Empty paths issue in the wtrace output has been bugging me for quite some time. As I started working on a new wtrace release (coming soon!), there came the right moment to fix it. I’ve seen other people struggling with this problem too, so I thought that maybe it’s worth a blog post 🙂 Wtrace uses the TraceEvent library to interact with the ETW API, and in this post, I will use this library as well. Note that this issue affects only the real-time ETW sessions.
While analyzing the PerfView source code, I stumbled upon an interesting README file in the src/OSExtensions folder:
// The OSExtensions.DLL is a DLL that contains a small number of extensions
// to the operating system that allow it to do certain ETW operations.
// However this DLL is implemented using private OS APIs, and as such should
// really be considered part of the operating system (until such time as
// the OS provide the functionality in public APIs).
// To discourage taking dependencies on these internal details we do not
// provide the source code for this DLL in the open source repo.
// IF YOU SIMPLY WANT TO BUILD PERFIVEW YOU DO NOT NEED TO BUILD OSExtensions!
// A binary copy of this DLL is included in the TraceEvent\OSExtensions.
// However we don't want this source code to be lost. So we check it in
// with the rest of the code but in an encrypted form for only those few
// OS developers who may need to update this interface. These people
// should have access to the password needed to unencrpt the file.
// As part of the build process for OSExtension.dll, we run the command 'syncEncrypted.exe'.
// This command keeps a encrypted and unencrypted version of a a file in sync.
// Currently it is run on this pair
// OSExtensions.cs <--> OSExtesions.cs.crypt
// Using a password file 'password.txt'
// Thus if the password.txt exists and OSExtesions.cs.crypt exist, it will
// unencrypt it to OSExtesions.cs. If OSExtesions.cs is newer, it will
// be reencrypted to OSExtesions.cs.crypt.
Hmm, private OS APIs seem pretty exciting, right? A simple way to check these APIs would be to disassemble the OSExtensions.dll (for example, with dnSpy). But this method would not show us comments. And for internal APIs, they might contain valuable information. So let’s see if we can do better.
Recently, I hit an interesting error during a deployment orchestrated by Ansible. One of the deployment steps was to execute a custom .NET application. Unfortunately, the application was failing on each run with an ACCESS DENIED error. After collecting the stack trace, I found that the failing code was ProtectedData.Protect(messageBytes, null, DataProtectionScope.CurrentUser), so a call to the Data Protection API. To pinpoint a problem I created a simple playbook:
Interestingly, the original playbook succeeds if the testu user has signed in to the remote system interactively (for example, by opening an RDP session) and encrypted something with DPAPI before running the script.
It only made me even more curious about what is happening here. I hope it made you too 🙂
If you are developing, testing, or supporting web applications, you probably encounter situations when you need to record or modify HTTP traffic. Quite often, the browser request viewer might be enough, but what if you need to modify the traffic on the fly? Another challenging task is testing how your application behaves when put behind a load balancer or an edge server. There are many great HTTP proxies available in the market, including mitmproxy, Burp Suite, or Fiddler and they may be perfect in diagnosing/testing your applications. In this post, however, I am encouraging you to write small tools for your specific needs. There are many reasons why you may want to do so, such as the need for complex requests modifications, better control over the request processing, or customizations of the certificate creation. Of course, implementing the HTTP protocol could be demanding so, don’t worry; we won’t do that 🙂 Instead, we will use the open-source Titanium Web Proxy. The code samples in this post are meant to be run in LINQPad, which is my favorite tool for writing and running .NET code snippets, but you should have no difficulties in porting the samples to a C# script or a console application.
Recently at work, I needed to trace several syscalls to understand what SQL Server was doing. My usual tool for this purpose on Windows was API Monitor, but, unfortunately, it hasn’t been updated for a few years already and became unstable for me. Thus, I decided to switch back to WinDbg. In the past, my biggest problem with tracing the system API in WinDbg was the missing symbols for the internal NT objects. Moreover, I discovered some messy ways to work around it. Fortunately, with synthetic types in WinDbg Preview it’s no longer a problem. In this post, I will show you how to create a breakpoint that nicely prints the arguments to a sample NtOpenFile syscall.
MiniDumper is a diagnostic tool for collecting memory dumps of .NET applications. Dumps created by MiniDumper are significantly smaller than full-memory dumps collected by, for example, procdump. However, they contain enough information to diagnose most of the issues in the managed applications. MiniDumper was initially developed by Sasha Goldstein, and I made few contributions to its code base. You may learn more about this tool from Sasha’s or my blog posts.
Recently, one of MiniDumper users reported a memory leak in the application. The numbers looked scary as there was a 20MB leak on each memory dump. The issue stayed opened a few weeks before I finally found a moment to look at it. As it was quite a compelling case, I decided to share with you the diagnostics steps in the hope it proves useful in your investigations.
Developing system applications in C# requires a lot of PInvoking. Although there are many great PInvoke Nuget libraries, for smaller projects I still prefer to import only the definitions I use. The pinvoke.net site is an excellent source of stub definitions. However, it happens that the online definition does not contain all the needed constants or lacks something. In such a case you have to look into the Windows headers (which btw. contain not only definitions but also a lot of interesting comments). I used to search through those files using Total Commander “Find Files” dialog, but it was slow and inefficient. So I switched to Sublime Text and created a project for the Windows headers folder (C:\Program Files (x86)\Windows Kits\10\Include\10.0.x.x). Once the folder index is cached, Sublime becomes a great tool for analyzing the source code (not only for C++!). However, when you read a lot of code and switch between various projects, Sublime replaces the old cached projects with the new ones to keep the cache at a reasonable size. That triggers the cache rebuilt when you open the “old” project again, which takes time and makes your search inefficient again.
I then started looking for a way to build a permanent index on the folders I regularly scan (such as the Windows headers directory). At first, I was thinking about running a local instance of Elasticsearch or Apache Solr server, but that seemed like overkill. I was looking for something simpler, some kind of a wrapper over the Apache Lucene library, which is the core engine for the servers mentioned above. Then I stumbled upon the Lee Holmes article about Scour, a PowerShell module that wraps the Lucene.Net library and provides cmdlets to create full-text indexes for your folders. After using it for some time, I am happy with the results so I decided to share my simple setup with you.