Wednesday, April 20, 2011

MSTest and 64bit

This post is about running MSTest for applications that target mixed platforms.

If you are lucky enough to be able to write your applications in pure .NET, then you may never encounter 32bit/64bit platform issues. However, if any dependent library or plug-in is compiled for a specific architecture, then your whole application must be run in that mode. This is why the default Window's Internet Explorer is still 32bit despite the 64bit version shipping since Vista: it has to be the same architecture as any legacy plug-ins. By contrast, Notepad doesn't have any plug-ins, so it can get away with being 64bit only.

My companies applications rely on many native libraries, which are obviously compiled for specific architectures (x86 and x64). Deploying an application for multiple target processors is a complex subject in itself that can be solved with a range of strategies from dynamic library linking to processor-specific installers, but however you deploy, your application will behave differently in these two different modes so they must both be tested.

For better or for worse, we use MSTest to control application quality. Since the release of Visual Studio 2010 this has been able to run in 64-bit mode as well as 32-bit mode, but there are certain subtleties that complicate the practical aspects of administering your tests.

To understand the problem, consider the way MSTest works: Testing is done using two programs MSTest.exe and QTAgent32.exe. MSTest is told what assemblies to load and it scans those assemblies (using reflection) to find any classes and methods annotated as tests using the various test attributes. To do this it must be able to load the assembly and all its dependent assemblies and because MSTest is a 32bit process, none of these assemblies can be exclusively 64bit. Once loaded, MSTest instructs QTAgent32 to run these tests, which means QTAgent32 must load the assemblies itself and execute the test methods, but because it is also a 32bit process it cannot load 64bit assemblies either.

In Visual Studio 2010 a new version of QTAgent32 was added called QTAgent.exe, which can run 64bit assemblies. This means that even though MSTest is still 32bit, QTAgent can execute in full 64bit mode so that pure .NET assemblies can now be tested in 32bit and 64bit mode. However, it still doesn't easily allow applications with mixed-mode assemblies to be tested in 64bit mode because they cannot be loaded by MSTest in the first place.

One interesting solution to this is to force MSTest.exe to be a 64bit application. This implies that MSTest is actually pure .NET code anyway, but has been forced to run in x86 mode. If you are going down this road, note that MSTest relies on various registry entries (HKLM\SOFTWARE\Microsoft\VisualStudio\10.0\EnterpriseTools\QualityTools\TestType and HKLM\SOFTWARE\Microsoft\VisualStudio\10.0\Licenses) to decide which extensions it can handle and what features are licensed for use, and that these are installed by default to the WOW6432Node registry "shadow" branch. To run in 64bit mode you must copy some of the registry entries over as well as editing the binaries themselves.

There is an alternative approach that doesn't involve editing executable files and local machine registry settings (which can be a pain across a large development team). Our application builds for two distinct platform targets x86 and x64 (note however that most assemblies are compiled as "Any CPU" except the ones that contain native code) and test projects are only built in the x86 solution configuration. This ensures that the code in their bin folder is 32bit compatible, and they can therefore be loaded into MSTest. Also, the tests are configured to run against the binaries in the actual application deployment folder rather than running in their binary folder using the new root folder feature of Visual Studio 2010:

In this case we have used an environment variable that signifies where to find the deployed binaries. Whether you do this or not, it seems to always want a full path for one reason or another, which can make supporting multiple development environments a challenge.

We make one of these test configurations for each target platform, remembering to change the Hosts section that controls 32bit and 64bit execution. Then we can run both configurations from the command line like this:

mstest /testsettings:WorkStation32.testrunconfig /testmetadata:SOLUTION.vsmdi
mstest /testsettings:WorkStation64.testrunconfig /testmetadata:SOLUTION.vsmdi

The trick here is that in both cases MSTest will load the 32bit binaries to decide what tests to run, but the different configuration files will control if QTAgent32 or QTAgent is used. Note that this cannot work with the /noisolation switch, because MSTest cannot host the 64bit binaries.

The disadvantage of running your unit tests against the deployment folder is that your tests are less "clean" and test failures could take longer to diagnose. The advantage is that the tests are being run on code as it will appear in the wild, which can include complex deployment features such as assembly obfuscation.

This system will work on development desktops and build servers. It may give the Visual Studio IDE pause for thought occasionally, but it is fundamentally compatible, which is one of the only real advantages of MSTest in the first place.


Anonymous said...

Thanks for the info.
I do have a question though, as I was unable to run a test in 64 bit.
I created a new Console app, with a new Test project. In the test project I have a single TestMethod with a single line of code:

bool is64 = Environment.Is64BitProcess;

I changed the Test Settings to run the test in 64bit, but it always returns false.
In the output window I see the reason: QTAgent32.exe and not QTAgent.exe. Why? (Needless to say, both the console project and then test project are set to AnyCPU and I'm running this on a 64-bit Win7).

Rupert said...

@Anonymous Make sure that Test Settings > Hosts is set to run in 64-bit mode. This should tell it to use the 64-bit version of QTAgent. I realize now that this step is not fully explained in the blog post, but that is the difference between WorkStation32.testrunconfig and WorkStation64.testrunconfig.

David said...

I seem to have hit on a scenario where when I run mstest on an AnyCPU assembly (e.g. AnyCPUTestingx64Production.dll) which references an x64 assembly (e.g. x64Production.dll), I get a BadImageFormatException.

The issue occurs when an interface in x64Production.dll is implemented (even if unused) by the AnyCPUTestingx64Production.dll test assembly:
Unable to load the test container 'D:\AnyCPUTestingx64Production.dll' or one of its dependencies. error details: System.BadImageFormatException: Could not load file or assembly 'x64Production, Version=, Culture=neutral, PublicKeyToken=null' or one of its dependencies. An attempt was made to load a program with an incorrect format.

The testsettings file specifies hostProcessPlatform="MSIL", peverify and corflags reveal nothing interesting and I can readily reproduce this in a toy solution. All this makes me think I'm doing something obviously wrong, otherwise someone else would surely have hit on this same problem.

I can't believe this is unresolvable, as it would effectively rule out test doubles.

In your adventures, have you ascertained whether or not this is simply unsupported in the VS2010 mstest?

Anonymous said...

@Rupert - Obviously the host was set to 64 bit (and the settings in the file was set to MSIL).

@David - what I ended up doing for now was ugly but working - I renamed QTAgent32.exe to QTAgent32.exe.old, and renamed QTAgent.exe to QTAgent32.exe, so that the 64bit QTAgent will run. This worked around the problem, but naturally it's not the desired solution.

Having said all that - I retried today and may have found a more adequate solution. I noticed the if the test was run first on the default 32 bit settings, and then set to 64 bit, it usually continues to run the QTAgent32.exe. However, if I use Clean Solution and then re-run the test, it runs QTAgent.exe as expected.

Rupert said...

@Anonymous Thanks for the update. The "Clean Solution" rings a bell for me; perhaps some of the metadata is cached during the test process, but not checked for architectural changes?

@David There is nothing obviously wrong with what you are doing but it does appear to be running the 32-bit version of the test harness. You can confirm that through Process Explorer, which will report which QTAgent is running. At the same time it is worth checking which DLLs are being picked up by QTAgent, sometimes stale DLLs for the wrong architecture can be picked up erroneously.

If you have a short reproduce I would recommend submitting it to StackOverflow. I would have a look myself, but I have moved to Eclipse (for Android development) and so no longer maintain a working Visual Studio Environment.

David said...

@Rupert and @Anonymous - thanks for the replies!

I'll try and catch the launching of QTAgent32/QTAgent in the act (but, as it fails immediately, I suppose I'll have to resort to fusion log or process monitor to witness it).

So, as suggested, my first stackoverflow question is:
"BadImageFormatException when AnyCPU test assembly implements interface from x64 production assembly".

Fingers-crossed, once I get into the office, I can answer it myself with the help of your replies.

David said...

@Rupert - it was my failure to interpret the guts of your blog entry that led to my issue.

Thanks to you, I finally got the 64-bit tests running!

Rupert said...

@David Awesome! Now all we have to do is get it working on ARM processors in time for Windows 8 ;-)

Anonymous said...

I had to get native mixed tests working too. I could not add the reg settings to our build environment. Instead I created an tool uses reflection emit to generate proxy assembly for mixed mode dlls. Proxy assembly has stubs for tests and uses reflection load actual test assembly and invoke tests.

MaLKaV_eS said...

After a two day struggle, made this work. Using /testsettings makes this work perfectly. Thanks!

Anonymous said...

In Visual Studio 2013 (at least, didn't try in 2012) I didn't have to do anything but choose Test->Test Settings->Default Processor Architecture->x64. Can also use test settings to achieve the same result. None of those old kluges necessary.

I'll date my post as Jan 7, 2014, since this blog only seems to show the time of the post.