Enumerating AppDomains in a remote process

I am working on adding a support for ASP.NET performance counters into Musketeer. Compared to other .NET performance counters they have quite surprising instance names. ASP.NET developers decided that their performance counter instances will be identified by names derived from the AppDomain names (more information can be found here). This is probably due to a fact that one process may host multiple ASP.NET applications, thus one counter instance per process won’t be enough. Consequently, in order to match collected metrics with process ids we need to know which AppDomain belongs to which process. How can we do that?

ClrMD

If you have any experience with ClrMD https://github.com/microsoft/clrmd you probably know the ClrRuntime class, which has an AppDomains property. In the main repository there is even this sample code:

ClrRuntime runtime = ...;
foreach (ClrAppDomain domain in runtime.AppDomains)
{
    Console.WriteLine("ID:      {0}", domain.Id);
    Console.WriteLine("Name:    {0}", domain.Name);
    Console.WriteLine("Address: {0}", domain.Address);
}

That was easy, right? There is one caveat though. ClrMD library is using CLR COM API under the hood, and allows us to enumerate only certain types of processes: processes which have the same bitness and .NET version as our application. So, for instance, it’s not possible to enumerate AppDomains in a 32-bit remote process from a 64-bit application. This restriction made the library useless for me. I’m wondering if it would be possible to use some COM remoting to overcome it – please leave a comment if you have any ideas.

Raw memory regions (Process Hacker approach)

I knew Process Hacker lists app domains under the .NET Performance tab:

By examining the source code I found out that this tab is rendered by a DotNetTools plugin, which code can be found (unsurprisingly :)) under the plugins/DotNetTools folder. The interesting methods for us are QueryDotNetAppDomainsForPid_V4 and QueryDotNetAppDomainsForPid_V2. I won’t paste the code here as it’s pretty complex. Enough to say that it’s a slightly modified portion of the Core CLR. To make the enumeration safe they first need to acquire some internal CLR mutex, which secures the memory region holding CLR metadata. Then, they extract the AppDomain information from the raw memory. It’s working really fast, but I see two problems with this approach:

• it’s quite complicated – I can only imagine all the pesky pinvoke problems when porting this solution to Musketeer
• it’s using internal CLR structures – if CLR developers decide to change the AppDomain memory layout we are screwed

ETW rundown events

Yes, Event Tracing. You might be surprised with this solution, but the CLR Rundown Provider (A669021C-C450-4609-A035-5AF59AF4DF18) provides an easy way to enumerate CLR metadata in all the running .NET processes. With a help from the Microsoft.Diagnostics.Tracing.TraceEvent library the logic can be implemented in just few lines:

public void CollectAppDomainInfo()
{
    Debug.Assert(!completed);
    using (session = new TraceEventSession("MusketeerEtwSession")) {
 
        session.Source.Dynamic.All += ProcessTraceEvent;
        session.EnableProvider("Microsoft-Windows-DotNETRuntimeRundown", TraceEventLevel.Verbose,
            0x40L | // StartRundownKeyword
            0x8L    // LoaderRundownKeyword 
        );
 
        ThreadPool.QueueUserWorkItem(WatchDog);
 
        lastTimeEventWasReceivedUtc = DateTime.UtcNow;
        session.Source.Process();
    }
    completed = true;
}

void WatchDog(object o)
{
    while (true) {
        Thread.Sleep(TimeSpan.FromSeconds(1));
        if (session.IsActive && DateTime.UtcNow.Subtract(
            lastTimeEventWasReceivedUtc) > rundownTimeout) {
            // rundown should be finished by now
            session.Stop();
            break;
        }
    }
}

void ProcessTraceEvent(TraceEvent traceEvent)
{
    lastTimeEventWasReceivedUtc = DateTime.UtcNow;
 
    if (traceEvent.ProcessID == currentProcessId) {
        return;
    }
 
    if ((ushort)traceEvent.ID == DCStartInitEventId) {
        Debug.Assert(!processAppDomainsMap.ContainsKey(traceEvent.ProcessID));
        processAppDomainsMap.Add(traceEvent.ProcessID, new List());
    } else if ((ushort)traceEvent.ID == AppDomainDCStartEventId) {
        Debug.Assert(processAppDomainsMap.ContainsKey(traceEvent.ProcessID));
        processAppDomainsMap[traceEvent.ProcessID].Add(new AppDomainInfo() {
            Id = (long)traceEvent.PayloadByName("AppDomainID"),
            Name = (string)traceEvent.PayloadByName("AppDomainName")
        });
    }
}

The whole code snippet can be found on my gist. As usual, there are some problems with this approach too :):

• it works only for .NET applications using .NET Framework version 4.0 or above
• it is not possible (at least I’m not aware of a way how to do that) to know when the enumeration is finished. That’s why I have this watchdog thread, checking the time which passed from the last received event.

Sometimes the .NET rundown provider is not found in the system (you may check this by running the command: logman /query providers | findstr /i dotnetruntimerundown). In such a case you need to reinstall .NET ETW providers by running: wevtutil im CLR-ETW.man in the .NET installation folder.

For Musketeer I’ve chosen the ETW approach. I won’t support .NET 2.0 web applications, but I wrote the code pretty fast and it’s easily maintainable. I hope you will also find it useful if you ever had to enumerate AppDomains in a remote process 🙂

Advertisement