Moles: Tool-Assisted Environment Isolation with Closures Jonathan de Halleux and Nikolai Tillmann Microsoft Research One Microsoft Way, Redmond WA 98052, USA {jhalleux,nikolait}@microsoft.com Abstract. Isolating test cases from environment dependencies is often desirable, as it increases test reliability and reduces test execution time. However, code that calls non-virtual methods or consumes sealed classes is often impossible to test in isolation. Moles is a new lightweight framework which addresses this problem. For any .NET method, Moles allows test-code to provide alternative implementations, given as .NET delegates, for which C# provides very concise syntax while capturing local variables in a closure object. Using code instrumen- tation, the Moles framework will redirect calls to provided delegates instead of the original methods. The Moles framework is designed to work together with the dynamic symbolic execution tool Pex to enable automated test generation. In a case study, testing code programmed against the Microsoft SharePoint Founda- tion API, we achieved full code coverage while running tests in isolation with- out an actual SharePoint server. The Moles framework integrates with .NET and Visual Studio. 1 Introduction In software testing, it is often desirable to test individual components in isolation. It makes testing more robust and scalable. Especially in the context of unit testing, where the intention is to test a single unit of functionality at a time, all irrelevant environment dependencies should be mocked, or simulated, so that the unit tests run quickly and give deterministic results. (This is in contrast to integration testing, where the goal is to test an entire software system, including all environment dependencies, at the same time. Integration tests are usually neither quick, nor entirely reliable.) However, many programming languages and runtime systems with static type check- ing and binding have language constructs that enforce the use of particular implementa- tions. For example, in .NET, the code-under-test may call static or non-virtual methods, or it may consume sealed classes. In those cases, the standard .NET runtime does not provide any way to redirect calls for testing purposes. As a result, isolating the code for testing purposes becomes impossible. In theory, the best solution to the problem is to refactor the code [3], introducing ex- plicit interface boundaries and allowing different interface implementations. However, it is often not possible in practice to refactor existing legacy code, especially when it is provided by a third party. J. Vitek (Ed.): TOOLS 2010, LNCS 6141, pp. 253–270, 2010. c Springer-Verlag Berlin Heidelberg 2010 254 J. de Halleux and N. Tillmann We have developed a new lightweight framework called Moles to address this prob- lem. The Moles framework allows the test-code to provide alternative implementations for non-abstract methods of any .NET type. Using code instrumentation, the Moles framework will redirect calls to an alternative implementation. An alternative imple- mentation is given as a .NET delegate instance, which consists of a method together with a receiver object. This enables the direct use of closures in languages such as C# 2.0. The Moles framework generates mole types which provide explicit type-safe at- tachment points to register delegates as alternative implementations for every method of the code-under-test. As a welcome side effect of explicit type-safe attachment points, the developer is helped by the Visual Studio editor via automatic code completion when writing delegate attachments for any particular attachment point. The Moles framework has been designed to work together with the dynamic sym- bolic execution [4] tool Pex [18,14] to enable automated test generation. Pex relies on tracing control- and dataflow at runtime, analyzing the test-code together with the code- under-test. Pex can only analyze .NET code. When an environment-facing component is invoked that is not implemented in .NET itself, then Pex cannot infer the constraints that reflect the behavior of that component, and as a consequence Pex cannot gener- ate test cases that achieve high code coverage. Moles allow the tester to replace any environment-facing component at test time with .NET code that simulates its behavior, which in turn enables test case generation with Pex. The test cases generated by Pex do not depend on Pex anymore, but they can reproduce their results in isolation, using only Moles. We have performed a case study, testing code programmed against the SharePoint Foundation [10] API using Moles and Pex, and we were able to achieve full code cov- erage while running the code in isolation without the need for an actual SharePoint server. Moles supports .NET 2.0 and higher; Moles integrates with Visual Studio 2008 and higher. Moles is part of Pex, an incubation project for Visual Studio developed at Mi- crosoft Research. Pex is available for academic use and for commercial evaluation. The main contributions of this paper are: – The description of a low-level, instrumentation-based detour approach. – The description of a code generation framework that provides type-safe attachment points for detours in the form of delegates. – A formalization of the semantics of attaching, detaching, and invoking detours, which enables the analysis of detours, in particular with symbolic execution frame- works. – An evaluation of the feasibility of our approach in the context of a real-world ap- plication. We will explain the basic idea behind the Moles framework along a simple example in Section 2. Moles leverages several features of .NET and C#, notably delegates, anony- mous methods, lambda-expressions, and closures, which we will discuss in Section 3. We will describe in detail the implementation of Moles, and give a formal description of its semantics in Section 4. We describe a case study in which we applied Moles and Pex on a SharePoint application in Section 5. We discuss related work in Section 6, and conclude in Section 7. Moles: Tool-Assisted Environment Isolation with Closures 255 2 Motivating Example Consider a C# method that throws an exception on January 1st of 2000. public static class Y2KChecker { public static void Check() { if (DateTime.Now == new DateTime(2000,1,1)) throw new ApplicationException("y2k bug!"); } } (Note: Now is a property of the DateTime type which is part of the .NET framework. When referencing this property as shown in the code above, a call to a method get_Now is generated by the C# compiler.) Testing this method is particularly problematic because the program depends on DateTime.Now, a property that in turn depends on the computer clock. In other words, DateTime.Now is environment-dependentand non-deterministic. Moreover, the Date- Time.Now property is static, so it is not possible to use virtual method overriding to supply a different implementation at test time. This problem is symptomatic of the isolation issue in unit testing: programs that directly call into the database APIs, com- municate with web services, etc., are hard to unit test because their logic depends on the environment. If .NET allowed us to redefine the meaning of static methods at runtime, we could easily solve this testing problem by replacing the computed value of DateTime.Now with something else. // does not compile: DateTime.Now cannot be set DateTime.Now = new DateTime(2000, 1, 1); Moles enables this scenario. Mole types provide a mechanism to detour any .NET method. When a method is detoured, all invocations of that method get forwarded to an alternative implementation, and the original method is never actually invoked. To this end, the Moles code generator creates a type MDateTime which has a settable property NowGet. This property can be set to an instance of a delegate which doesn’t take parameters, and returns a DateTime value. C# 3.0 allows one to write a function inline within another method. // attach delegate to the mole property to redirect DateTime.Now // to return January 1st of 2000 MDateTime.NowGet = () => new DateTime(2000, 1, 1); Y2KChecker.Check(); After setting MDateTime.NowGet, all calls to the property getter of DateTime.Now are detoured to the user-supplied delegate. This is realized by code instrumentation. To enable the code instrumentation when using the Visual Studio Unit Test frame- work, the attribute [HostType("Moles")] must be added to instruct the Visual Stu- dio Unit Test framework to run the test under the control of the Moles instrumentation framework. [TestMethod] [HostType("Moles")] // run with code instrumentation 256 J. de Halleux and N. Tillmann public void Y2kCheckerTest() { ... } The [TestMethod] annotation identified the above method as a unit test. Pex, an automated white-box test generation tool for .NET, enables Parameterized Unit Testing [19]. The following method is identified as a parameterized unit test by the [PexMethod] attribute. It states that for all possible parameter values, the test should succeed (and not terminate with an uncaught exception). [PexMethod] public void Y2kCheckerTest(DateTime time) { // hook to the method to redirect DateTime.Now MDateTime.NowGet = () => time; Y2KChecker.Check(); } By symbolically tracing the test inputs through the C# compiler-generated closure, the delegate, and the detoured invocation of DateTime.Now, Pex discovers the path condi- tion time == new DateTime(2000,1,1) where time is the test input. As a result, Pex generates two traditional unit tests such as the two unit tests shown below, one where the path condition is fulfilled, and one where it is not. [TestMethod] [HostType("Moles")] public void Y2kCheckerTest01() { Y2kCheckerTest(new DateTime(1, 1, 1)); } [TestMethod] [HostType("Moles")] public void Y2kCheckerTest02() { Y2kCheckerTest(new DateTime(2000, 1, 1)); } Without Moles, Pex would not have been able to construct test cases that cover all cases in the code, as the actual implementation of DateTime.Now relies on system calls to query the current machine time, which is outside of the semantics of .NET, and therefore outside of the scope which Pex can analyze.