Diagnosing ASP.NET views compilation with FrontMan

In today’s post I will show you how we fought a pesky compilation problem with Razor views in our ASP.NET MVC application. One of the parts of our application is an email-sending engine which uses Razor templates to create message bodies. After deploying this app on IIS we started receiving the following errors (unimportant parts are stripped):

...
Line: 0\\t Col: 0\\t Error: Metadata file \u0027ev-server/dev2/appname/bin/EmailSystem.Client.DLL\u0027 could not be found

//------------------------------------------------------------------------------
// \u003cauto-generated\u003e
//     This code was generated by a tool.
//     Runtime Version:4.0.30319.235
//
//     Changes to this file may cause incorrect behavior and will be lost if
//     the code is regenerated.
// \u003c/auto-generated\u003e
//------------------------------------------------------------------------------
...

The first line suggests that it’s a compilation problem, but we didn’t really know exactly which part of the application was throwing it. I thought having a full compiler command line might help us. Unfortunately gathering it from ASP.NET application is not so easy.

How ASP.NET compilation works?

By default ASP.NET framework uses csc.exe to compile .cs files in order to execute logic implemented in them. You may now wonder what .cs files I am referring to as probably you don’t have any of them in your IIS web application path (unless you’re using the App_Code folder). So now have a look at all the files in your Views folder and imagine that each one of them eventually becomes a .cs file with some auto-generated name. Every time an ASP.NET application renders a page it first checks whether the given page has been compiled and thus can be executed. If not ASP.NET framework calls an interpreter (such as RazorTemplateEngine) to create a .cs file out of .cshtml (or .aspx, .asmx etc.) and then compiler to generate the .dll file. You may easily observe that by using Process Monitor:

Let’s have a look at an example csc.exe command line call from ASP.NET:

"C:\WINDOWS\Microsoft.NET\Framework64\v4.0.30319\csc.exe" /noconfig /fullpaths @"C:\WINDOWS\TEMP\p0ulcxtl.cmdline"

As you can see all compilation arguments (probably because of their length) are passed to the compiler using a temporary file. This file is then removed after csc.exe process finishes. Now imagine that something goes wrong here (as in the case presented at the beginning of this post). You probably would like to know what arguments were exactly passed to the compiler and why it failed. And this is how FrontMan was born:)

FrontMan to the rescue

FrontMan is a simple application that can start other processes, a bit like the start.exe command. What’s special about it is that it additionally logs the starting process arguments and if any of the arguments contains a path to a file, a copy of this file will be created in the FrontMan output folder. Let’s have a look how it works with csc.exe and simple Test.cs C# file:

PS temp> .\FrontMan.exe csc /debug+ .\Test.cs

ProcDump v5.13 - Writes process dump files
Copyright (C) 2009-2013 Mark Russinovich
Sysinternals - www.sysinternals.com
With contributions from Andrew Richards

Process:               csc.exe (2388)
CPU threshold:         n/a
Performance counter:   n/a
Commit threshold:      n/a
Threshold seconds:     n/a
Number of dumps:       1
Hung window check:     Disabled
Exception monitor:     Unhandled
Exception filter:      Disabled
Terminate monitor:     Disabled
Dump file:             c:\Users\admin\AppData\Local\Temp\frontman\20130128_195337.547_csc\csc_YYMMDD_HHMMSS.dmp


Press Ctrl-C to end monitoring without terminating the proMiccreossofst .(R
) V
isual C# Compiler version 4.0.30319.17929
for Microsoft (R) .NET Framework 4.5
Copyright (C) Microsoft Corporation. All rights reserved.

The process has exited.

If you now open c:\Users\<your-login>\AppData\Local\Temp\frontman\20130128_195337.547_csc\ you should see two new files: arg_2_Test.cs and callargs.txt. The first one is a copy of Test.cs at the time when it was compiled and the second one contains our command line: "csc" "/debug+" ".\Test.cs". You probably noticed procdump messages in the above output. That’s another FrontMan’s feature – it does not start the requested process directly, but runs it under procdump. By default FrontMan configures procdump to create a full memory dump on an unhandled exception, but this behavior might be easily changed to any other suported by procdump. Having procdump attached to a process opens a possibility for FrontMan to be a system-wide process sniffer. This can be achieved through Image File Execution Options. By adding a following key to the registry:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\csc.exe]
"Debugger"="c:\\tools\\Frontman.exe"

we configure FrontMan to log any csc.exe calls in the system, including ASP.NET compilations. Fortunately you don’t need to manually edit the registry. Running FrontMan -install csc.exe from the elevated command prompt will create the needed keys for you. On the contrary, running FrontMan -uninstall csc.exe will remove them from the registry.

Let’s see how it works in our ASP.NET example. If you have installed FrontMan for csc.exe, in your process tree you should see w3wp.exe that calls FrontMan which then runs procdump, which finally runs csc.exe. In your temporary path there should be a new folder created named frontman with subfolders for each csc.exe call. Find the interesting one and open it. In our case the callargs.txt contained:

C:\WINDOWS\Microsoft.NET\Framework64\v4.0.30319\csc.exe /noconfig /fullpaths @C:\WINDOWS\TEMP\pbjs1i2y.cmdline

and arg_3_pbjs1i2y.cmdline:

/t:library /utf8output /R:"\\dev-server\dev2\appname\bin\EmailSystem.Contracts.dll" /R:"\\dev-server\dev2\appname\bin\EmailSystem.Client.dll" /R:"System.dll" /R:"System.Core.dll" /R:"System.Web.Extensions.dll" /R:"Microsoft.CSharp.dll" /R:"ev-server/dev2/appname/bin/EmailSystem.Client.DLL" /out:"C:\WINDOWS\TEMP\pbjs1i2y.dll" /debug- /optimize+  "C:\WINDOWS\TEMP\pbjs1i2y.0.cs"

One of the EmailSystem.Client.dll references seemed to be broken and unnecessary. After seeing this command line I easily found a place in code where the compiler was incorrectly called.

To conclude, I hope that after having read this post you will find a place for FrontMan in your diagnostics toolkit. For instance it might provide some help when diagnosing Windows Service start failures (that will be probably a subject of another post:)). As usual this tool is available for download at my .NET Diagnostics Toolkit project page. Before using it make sure that you fill its configuration file with valid settings (like procdump path) – each setting has a comment so you should easily see what it does. Any comments and ideas how FrontMan might be improved or where else it can be used are highly appreciated 🙂