Coverage with TeamCity and dotCover with MSTest, NUnit or MSpec

As some of you know, we recently shipped TeamCity 6 which includes, out of the box, a bundled version of dotCover. What this means is that you can now get coverage reports for your code easily, and of course, for free if you’re using the Professional version of TeamCity.  The setup is quite easy if you are using MSTest and NUnit. For MSpec, you need to take a few additional steps.

Using MSTest / NUnit Runners

Normally build files consist of a series of tasks that involve compilation and running of tests. With TeamCity, you can separate some of these steps out into individual TeamCity Build steps, which is what we will be doing in this case (everything that we see here applies to both MSTest and NUnit).

Here is our build.xml (MSBuild) file:

image

As we can see, other than compiling a solution, which in this case consists of the actual application and the test assemblies, not much else going on.

[Note: this could have been done using the SLN as the Runner Type under TeamCity since this example build script does not do much else. In real scenarios however, build scripts do more than just call a solution (in fact normally you’d call projects not solutions)].

In TeamCity, we create a new build project and setup the VCS root. We then add a new Build Step which calls this MSBuild file:

image

Notice that all we are doing here is calling our MSBuild script. No coverage settings yet.

Next thing is to add an additional build step in TeamCity. This time, we are going to call MSTest as opposed to MSBuild:

SNAGHTML1e386e79

SNAGHTML1e3d642d

(the sections cut out are blank).

In the .NET Coverage tool section we select JetBrains dotCover and then  add the assemblies we want coverage for (just the name of the assembly) prefixing them with +: and filtering out those we do not want coverage for with –:.

That’s all there is to it. Once we run the Build, we should now see a new tab with Coverage Reports as well as a new Artifact which contains the Coverage files zipped up.

image

The Code Coverage tab goes into more detail:

image

We can even drill down into individual classes and examine the code coverage:

image

If we are using NUnit instead of MSTest, the only difference is there test runner we select when adding a new Build Step in TeamCity. Instead of MSTest we choose NUnit along with the version:

image

What about MSpec or my Specific Test Runner?

If we are using MSpec or a different test runner that is not supported directly by TeamCity, we can still get coverage reports; we just need to do a little bit of additional configuration. TeamCity has an API which allows us to send it messages when we want to interact with it (this is actually quite a powerful feature but out of scope for this post so please leave a comment if you’d like me to cover it in more detail). We can leverage this API to tell it when to start coverage and where to get the results from.

Here is the build script for MSpec:

image

We have created two targets. The second one (TeamCity) is the one we are interested in. This does a couple of things:

1. The first <Exec> runs all MSpec tests so that we can see the test results inside TeamCity. This is not strictly necessary for Code Coverage but usually build processes do display these results. The —teamcity option we are passing in to MSpec is for it to generate the system messages that are then fed to TeamCity (see point 3)

2. The second <Exec> is the one that runs dotCover. This uses a configuration file called dotCover.xml which we will examine further down. We pass in the c (or coverage) option when calling dotCover.

3. This is a message we send to TeamCity to tell it that we have run coverage. We indicate the tool we are using (dotcover in this case) and where the results are located. TeamCity uses this information to then display the results in the UI. This is one of the API messages mentioned earlier.

Finally we need to define the dotcover.xml file with out configuration for running dotCover (for detailed information on creating dotCover configuration files, see here and here):

image

In terms of TeamCity, we then just define our build step that calls out to the build script:

image

Notice how we do not specify .NET Coverage options explicitly. And if all goes well, we can see the coverage output just as before:

image

Summary

We can see that running code coverage is now pretty straightforward when using MSTest, NUnit or even a custom test runner. Most of what we have covered for MSpec will work with pretty much any test runner in terms of coverage (feel free to vote here for MSpec support).

With any build process, there are numerous ways of doing the same thing. I’m going to show you one of them. Based on your setup and needs you might want to do things differently. Fortunately TeamCity is flexible enough to allow for many scenarios.

One thing to be aware of is that dotCover creates some temporary files for the XmlSerailizer in the Temp profile folder. This normally is not a problem unless the folder does not exist. If you are running TeamCity under the SYSTEM account, make sure that the folder C:\Windows\system32\config\systemprofile\AppData\Local\Temp exists. This will probably change in future versions so to avoid any possible issues.

Enjoy!

11 thoughts on “Coverage with TeamCity and dotCover with MSTest, NUnit or MSpec

  1. Lars

    I just upgraded our Teamcity installation to version 6 and tried the MSpec integration. Unfortunately under “Code Coverage” I am presented with the sentence “This is an autogenerated index file (there was no index.html found in the generated report).” so I looked at the log files and found this:

    –snip–

    [17:31:16]: [CallTarget] Teamcity
    [17:31:16]: [Teamcity] Exec
    [17:31:16]: [Exec] C:\TeamCity\buildAgent\plugins\dotCover\bin\dotcover c dotcover.xml
    [17:31:16]: [Exec] JetBrains dotCover Console Runner v1.0.121.4. Copyright (c) 2009-2010 JetBrains s.r.o. All rights reserved.
    [17:31:18]: [Exec] Coverage session started [22.12.2010 17:31:18]
    [17:31:26]: [Exec] Coverage session finished [22.12.2010 17:31:26]
    [17:31:26]: [Teamcity] Message
    [17:31:26]: [Message] ##teamcity[importData type=’dotNetCoverage’ tool=’dotcover’ path=’Source\coverage.xml’]
    [17:31:26]: Importing data from ‘Source\coverage.xml’ with ‘dotNetCoverage’ processor
    [17:31:26]: Process exited with code 0
    [17:31:26]: Waiting for 1 service processes to complete
    [17:31:26]: Processing 1 coverage report(s)
    [17:31:26]: Generating coverage report by dotcover for files: [C:\TeamCity\buildAgent\work\90f20b2d65a18ec4\Source\coverage.xml]
    [17:31:26]: Generate dotCover report
    [17:31:27]: [Generate dotCover report] Started dotCover: C:\TeamCity\buildAgent\plugins\dotCover\bin\dotCover.exe report C:\TeamCity\buildAgent\temp\buildTmp\dotcover8993701003855557597.xml
    [17:31:27]: [Generate dotCover report] Output: JetBrains dotCover Console Runner v1.0.121.4. Copyright (c) 2009-2010 JetBrains s.r.o. All rights reserved.

    Failed to read source file ‘C:\TeamCity\buildAgent\temp\buildTmp\dotcover8993701003855557597.xml’. Could not find a part of the path ‘C:\Windows\system32\config\systemprofile\AppData\Local\Temp\tbqs-1r6.tmp’.

    [17:31:27]: [Generate dotCover report] dotCover exited with code: -1
    [17:31:27]: [Generate dotCover report] dotCover returned non-zero exit code.
    [17:31:27]: Remove dotCover snapshot files
    [17:31:27]: [Remove dotCover snapshot files] Started dotCover: C:\TeamCity\buildAgent\plugins\dotCover\bin\dotCover.exe delete C:\TeamCity\buildAgent\temp\buildTmp\dotcover3661148334430340130.xml
    [17:31:27]: [Remove dotCover snapshot files] Output: JetBrains dotCover Console Runner v1.0.121.4. Copyright (c) 2009-2010 JetBrains s.r.o. All rights reserved.

    Failed to read source file ‘C:\TeamCity\buildAgent\temp\buildTmp\dotcover3661148334430340130.xml’. Could not find a part of the path ‘C:\Windows\system32\config\systemprofile\AppData\Local\Temp\xylkmy48.tmp’.

    [17:31:27]: [Remove dotCover snapshot files] dotCover exited with code: -1
    [17:31:27]: [Remove dotCover snapshot files] dotCover returned non-zero exit code.
    [17:31:27]: Generate dotCover HTML report
    [17:31:27]: [Generate dotCover HTML report] Loading dotCover report file…
    [17:31:27]: [Loading dotCover report file…] Failed to read dotCover report from: C:\TeamCity\buildAgent\temp\buildTmp\dotCover5950237266608218863Report. Premature end of file.
    [17:31:27]: Failed to compute .NET Coverage statistics for dotCover report generator’ Premature end of file.

    — snap–

    Any ideas what is wrong here? The coverage.xml gets created and can be read but after that things seem to fail…

    Thanks,

    Lars

    Reply
  2. hhariri Post author

    @Lars,

    The issue is that the Temp folder it refers to does not exist. If you can create it, it should work. I pointed this out in the Summary of the post:

    “…If you are running TeamCity under the SYSTEM account, make sure that the folder C:\Windows\system32\config\systemprofile\AppData\Local\Temp exists. This will probably change in future versions so to avoid any possible issues.”

    Let me know if that works for you.

    Thanks.

    Reply
  3. Lars

    Hi,

    actually I tried that, but it didn’t help. I also checked the Security settings for that folder just in case and the System Account does have all rights in this folder. Do you have any other advice?

    Maybe it helps if I create another Account that the teamcity services can run under?

    Thanks,

    Lars

    Reply
  4. Anton

    Hi, I try to repeat the same but with xUnit.net and all works fine locally, but when I try to run build on TeamCity I always get error: “The command “dotCover.exe c dotcover.xml” exited with code 9009″. What can be the cause of this error? Temp folder exists.

    Reply
      1. Anton

        Thank you. Indeed it was a cause of the problem. I thought since dotCover support is built-in into TeamCity, that it’s not necessary to install dotCover onto CI server separately.

  5. hhariri Post author

    @Anton,

    It should pick it up if you use dotcover, but if you’re specifying the executable then it needs it in the path. You don’t need to install a new copy.

    Reply
  6. Pingback: dotCover 1.0 console runner and relative paths | IDisposable Thoughts

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s