Care when unit testing via Resharper and MsTest

by Geert 13. January 2012 12:54

I like unit tests. So much that I wrote 1003 for Catel to make sure we don’t introduce bugs when changing code. We also love automation, so we use build scripts to create a new (beta) release, create a nuget package and upload the beta versions including symbols.

Anyway, enough history, let’s get to the problem. This week, I fixed an issue that as soon as a ReflectionTypeLoadException occurred, the successfully loaded types were not used by Catel either. That is wrong, so we fixed it. We always run the unit tests using Resharper and dotCover and all succeeded. However, as soon as we ran the unit tests in msbuild (which uses mstest), 3 unit tests failed. It was about this particular code:

   1: public static Type[] GetAllTypesSafely(Assembly assembly, bool logLoaderExceptions)
   2: {
   3:     Argument.IsNotNull("assembly", assembly);
   4:  
   5:     Type[] foundAssemblyTypes;
   6:  
   7:     try
   8:     {
   9:         foundAssemblyTypes = assembly.GetTypes();
  10:     }
  11:     catch (ReflectionTypeLoadException typeLoadException)
  12:     {
  13:         foundAssemblyTypes = typeLoadException.Types;
  14:  
  15:         Log.Warning("A ReflectionTypeLoadException occured, adding all {0} types that were loaded correctly", foundAssemblyTypes.Length);
  16:  
  17:         if (logLoaderExceptions)
  18:         {
  19:             Log.Warning("The following loading exceptions occurred:");
  20:             foreach (var error in typeLoadException.LoaderExceptions)
  21:             {
  22:                 Log.Warning("  " + error.Message);
  23:             }
  24:         }
  25:     }
  26:  
  27:     return foundAssemblyTypes;
  28: }

According to the documentation, the ReflectionTypeLoaderException.Types contains all the successfully loaded types. However, when running the unit tests (which loads more assemblies into the current AppDomain), the array contained 3 instances of null.

WTF? A successfully loaded type that is null? /*sarcasme on*/ I bet the Microsoft engineers had a good reason for this /*sarcasm off*/. Anyway, I fixed the issue by filtering out the null values. Below is the fixed code (notice the LINQ expression):

   1: public static Type[] GetAllTypesSafely(Assembly assembly, bool logLoaderExceptions)
   2: {
   3:     Argument.IsNotNull("assembly", assembly);
   4:  
   5:     Type[] foundAssemblyTypes;
   6:  
   7:     try
   8:     {
   9:         foundAssemblyTypes = assembly.GetTypes();
  10:     }
  11:     catch (ReflectionTypeLoadException typeLoadException)
  12:     {
  13:         foundAssemblyTypes = (from type in typeLoadException.Types
  14:                                 where type != null
  15:                                 select type).ToArray();
  16:  
  17:         Log.Warning("A ReflectionTypeLoadException occured, adding all {0} types that were loaded correctly", foundAssemblyTypes.Length);
  18:  
  19:         if (logLoaderExceptions)
  20:         {
  21:             Log.Warning("The following loading exceptions occurred:");
  22:             foreach (var error in typeLoadException.LoaderExceptions)
  23:             {
  24:                 Log.Warning("  " + error.Message);
  25:             }
  26:         }
  27:     }
  28:  
  29:     return foundAssemblyTypes;
  30: }

So, what have we learned today? Even unit tests cannot be trusted…

Tags: , ,

C#

Comments (1) -

Marcin Floryan
Marcin Floryan United Kingdom
1/13/2012 1:39:42 PM #

Interesting case. It would be good to understand why exactly the two test-runners behaved differently. Perhaps it's not so much the unit tests that cannot be trusted but the environment they're executed in.

Maybe you just found a bug in MSTest?

Comments are closed

About the Author

Geert van Horrik is a independent freelance software developer since January 1st, 2007. Since then he was been working on several projects from C++ to C# (WPF, ASP.NET, etc). Currently he loves to write his software using WPF (or Silverlight if WPF isn't an option).

Lately, Geert is spending a lot of time on Catel, a free open-source MVVM Framework for WPF and Silverlight. Actually, it's more than "just" an MVVM Framework, it's a complete application library!