How to decode managed stack frames in procmon traces

Recently, the idea of protected variables in TFS struck my attention and pushed me to do some more research on how exactly those variables are stored. I hope I will write a separate post on that subject, but today I would like to share with you a small trick I use whenever I need to work with managed application traces (and TFS is one of them).

On Windows, when I want to know how things work internally, I usually start with procmon. Seeing which paths and registry keys are accessed, combined with TCP/IP connections is often enough to get an idea where to put breakpoints in further analysis. My TFS investigation was no exception to this rule. I collected a trace while saving a protected build variable – this is how such a variable looks like (in case you are interested :)):

When I saved the build definition, I stopped the trace and started examining the events. TFS server is just another ASP.NET application hosted in IIS. There were not many w3wp.exe events in the trace, so it did not take me long to find the ones that seemed interesting (having ‘crypto’ string in the path was one of the criteria):

Our next task is to find the TFS module where the logic is implemented. Not surprisingly, we will start by opening the call stack window for the selected event:

Remember to set a valid path to the dbghelp.dll (should be the one distributed with Debugging Tools for Windows) and a symbol path (I use: SRV*C:\symbols\dbg*https://msdl.microsoft.com/download/symbols). As you can see in the image, procmon correctly decodes native stack frames, but managed frames are just hex numbers (frame 27). Now, it’s time for the trick 🙂

When you know that the application you are tracing is a managed one, collect a full memory dump during the analysis (better later than sooner, so all the interesting modules are already loaded). You may use procdump -ma process name|PID for this purpose. Now, open the dump in windbg, load SOS (.loadby sos clr) and resolve the address from procmon:

0:000> !ip2md 0x7fe9677d598
MethodDesc:   000007fe92611f68
Method Name:  Microsoft.TeamFoundation.Framework.Server.TeamFoundationStrongBoxService.AddStream(Microsoft.TeamFoundation.Framework.Server.IVssRequestContext, System.Collections.Generic.IEnumerable`1<System.Tuple`2<Microsoft.TeamFoundation.Framework.Server.StrongBoxItemInfo,System.IO.Stream>>)
Class:        000007fe926022c0
MethodTable:  000007fe92612050
mdToken:      0000000006004a5c
Module:       000007fe90fc7768
IsJitted:     yes
CodeAddr:     000007fe9677d110
Transparency: Critical

Later, you may extract the module to some local file and perform analysis with your favorite decompiler (in my case I will definitely check TeamFoundationStrongBoxService.AddStream):

0:000> !SaveModule 000007fe90fc7768 c:\temp\Microsoft.TeamFoundation.Framework.Server.dll
3 sections in file
section 0 - VA=2000, VASize=606b64, FileAddr=200, FileSize=606c00
section 1 - VA=60a000, VASize=52858, FileAddr=606e00, FileSize=52a00
section 2 - VA=65e000, VASize=c, FileAddr=659800, FileSize=200

I know that a full memory dump might seem as an overkill for decoding stacks, especially when you need to copy the trace and the dump to some other machine. Unfortunately, I haven’t found a more reliable way to do that. You may eventually consider using minidumper to make the dumps lighter (I think we could even write a microdumper for this :)). It should be also possible to write a tool based on ETW (Perfview uses only ETW events to decode the stacks), but I haven’t yet had time to check this out.