Jenkins CI in MVC.NET Part 2: Tests and Coverage

We have already talked about how to install Jenkins, Build a project and push it to another branch here. But Jenkins is capable of much more. In this topic we will talk about running tests and code coverage tools. For this purpose I have written an simple application that just open a web page with a Login and Home Page.

The application can be downloaded here.

The Application

The application do not have any database or business logic. Its purpose is to contains several unit tests and to measure code coverage of this tests. There are three type of tests in the application and we will demonstrate how to run all of them in Jenkins.

Functional .NET Tests

This are general c# tests that test a units or components and are written with Visual Studio Quality Tools. We can run this with MSTest.

Acceptance Tests

This are test that use the browser to test actual behavior. I have written them with SpecFlow wich can be run again with MSTest. SpecFlow uses Selenium Browser Driver to start and control Chrome (or any other browser you prefer). We do not need to run this test every time we make a build because they are slow and require fully loaded and working environment. We can run them prior of deploying to tests, stage or production environment.

Javascript Tests

Test that run on the client side and verify the behavior of the client scripts. We can run this tests and code coverage for them with the Chutzpah tool.

Jenkins Setup

Build Variables

Some build variables are set for the current job and are as fallow:

• BUILDMODE – Release or Debug.
• UNITTESTPROJECT – The name of the project and assembly with the unit tests.
• ACCEPTANCETESTPROJECT – The name of the project that hold the acceptance tests.
• WEBPROJECT – The name of the Web application project.
• PORT – The port that acceptance tests run against.
• ASSEMBLIES – The prefix of the Assemblies. If this is ‘App’, all your project assemblies must start with App.
  Example: App.Core, App.Services, App.Web, etc.

Build

First thing to do is download the project from the repository, clean and build it. You can fallow the steps in the previous article if you have problem setting this step.

Run Functional Tests With MSTest

Next you need to acquire MSTest. You can do this several ways:

• Install Visual Studio on the Jenkins machine. I do not prefer this method, because you do not actually need it there, except if you are using this machine for developing purposes.
• Install “Agents for Microsoft Visual Studio” from the Microsoft download page. They include MSTest.
• Follow the steps in this article. With this option you can put all files at “C:ToolsMSTest”. Note that you have to play with the registers in order to make it run.

To add the unit tests you can use the MSTest plugin, but I have found problems when using it so I am using the batch scripts.

Also it is a good idea to set the MSTest path as Environment variables in the Jenkins configuration page.

Another problem is that Jenkins is working with JUnit report files and MSTest is generating its own report files. We can transform with the help of MSXSL and a XSL transform file. We can download MSXSL from the Microsoft download site here.

You may download the transformation xsl from here. We can add the both files in single directory like C:ToolsMSTestToJUnit.

This is the final script.

REM 'Clean Unit Tests Reports'
del /q "%UNITTESTPROJECT%MSTestReport.trx"
del /q "%UNITTESTPROJECT%MSTestJUnitReport.xml"

REM 'Run the .NET Unit Tests.'
"%MSTEST_PATH%MSTest.exe" /nologo /testcontainer:%UNITTESTPROJECT%bin%BUILDMODE%%UNITTESTPROJECT%.dll /resultsfile:"%UNITTESTPROJECT%MSTestReport.trx"
"C:ToolsMSTestToJUnitmsxsl.exe" %UNITTESTPROJECT%MSTestReport.trx C:ToolsMSTestToJUnitmstest-to-junit.xsl -o %UNITTESTPROJECT%MSTestJUnitReport.xml

This script delete the MSTest and the JUnit report files if they exists. Run MSTest on the tests binary and after that transform it into JUnit report.

Run Acceptance Tests With MSTest

The acceptance test are written with SpecFlow. But the SpecFlow allow the tests to be run by MSTest so the acceptance test are run the same way as the unit test we just point to the acceptance test dlls.

The acceptance tests are generally much slower then the unit test and we shouldn’t run them on every build. We can run them when doing a delivery to environment.

In order to run the acceptance test we need an actual running site. We have 2 options:

1. Deploy to a development environment before you run the tests (environment with IIS and web site visible by the Jenkins CI server). After that run the acceptance test against that site.
2. Run the site on the spot with IIS Express. You can start IIS express and run the test. For this approach you must download and install IIS Express from the Microsoft download site here.
   Than you can run it and stop it like this:

   REM 'Start IIS Express'
   "%IISEXPRESS%iisexpress" /path:%WORKSPACE%Web /port:59725 /systray:false
   
   // Run the acceptance tests against http://localhost:59725
   
   REM 'Stop IIS Express'
   taskkill /IM iisexpress.exe
   

Run Client Tests With Chutzpah

In order to run the JavaScript tests we need the Chutzpah console application. We can download it from the chutzpah site here.

You need to download the full chutzpah archive and extract it. The .pdb and the chutzpah.console.vshost files are not generally needed. It is good idea again to extract it in some location with easy access like “C:ToolsChutzpah”.

This is the batch script that run the JS tests.

REM "Clean JavaScript Unit Tests Reports"
del /q "TestResultsJasmineJUnitReport.xml"

REM "Run the JavaScript Unit Tests"
"C:ToolsChutzpahchutzpah.console.exe" /silent /path %UNITTESTPROJECT%WebJSapp /junit TestResultsJasmineJUnitReport.xml

Functional Tests Coverage

In order to run code coverage we need a code coverage analyze tool. I will use OpenCover, but other tools are also available. This are the tools we will use to run our code covarage.

1. OpenCover – being used as the coverage checker.
2. OpenCoverToCoberturaConverter – a tool to convert the XML output from OpenCover into a format recognized by Cobertura (for which there is a Jenkins plugin).
3. Cobertura plugin – used to give a nice progress chart showing (hopefully) increases in coverage over time.
4. ReportGenerator – a tool to convert the XML output from OpenCover into clickable HTML showing line by line usage – used to show the current build’s usage in an informative manner
5. HTML Publisher plugin – used to publish the HTML output from ReportGenerator into a way that can be accessed via the Jenkins dashboard.

When you include code coverage for unit test you can run the tests and the coverage at the same time.
Here is a script that do this.

"C:ToolsOpenCoverOpenCover.Console.exe" -target:"%MSTEST_PATH%MSTest.exe" -targetargs:"/nologo /noisolation /testcontainer:%UNITTESTPROJECT%bin%BUILDMODE%%UNITTESTPROJECT%.dll /resultsfile:TestResultsUnitTestReport.trx" -filter:"+[%ASSEMPLIES%*]*" -register -mergebyhash -hideskipped:Filter -output:"TestResultsOpenCoverCodeCoverage.xml"

We run MSTest through open cover with the targeted .dll and the output file. /noisolation attribute tell MSTest not to copy the assemblies in another folder when testing. This is needed because PDBs may not be found and the code coverage may not run successful.
After that we will need to convert the OpenCover report to Cobertura report with the OpenCoverToCoberturaConverter tool.

REM 'Convert OpenConver report to Cobertura report'
"C:ToolsOpenCoverToCoberturaOpenCoverToCoberturaConverter.exe" -input:TestResultsOpenCoverCodeCoverage.xml -output:TestResultsCoberturaCodeCoverage.xml -sources:%WORKSPACE%

Client Tests Coverage

The client tests coverage are run again with Chutzpah just by adding coverage option. It creates 2 files json report and a html report. You can also add the options /coverageIncludes and/or /coverageExcludes options to add extra files in the coverage report or exclude files. You can also exclude files not needing coverage with chutzpah-exclude=”true” attribute in the reference tags.
There is 2 problems when using this approach.

1. Chutzpah will create the 2 files on the root folder, not the one where /junit option point to.
2. You can not visualize the json report and also have to convert it to Cobertura.

So for the first problem we may move the files by executing move command. Chitzpah create 2 files “_Chutzpah.coverage.html” and “_Chutzpah.coverage.json”, so we just do.

move _Chutzpah.coverage.json TestResults
move _Chutzpah.coverage.html TestResultsHTMLJSReportChutzpahIndex.html

For the second part we will need to convert the chutzpah to cobertura report. We can use this tool.
We download it and extract the files in the “C:ToolsChutzpahToCobertura” folder of the Jenkins CI server.

"C:ToolsChutzpahToCoberturaChutzpahToCobertura.exe" TestResults_Chutzpah.coverage.json TestResultsJSCoberturaCodeCoverage.xml

Visualization

In order to show the results we need to do several things.

• Add the “Post-build action” -> “Publish JUnit test result report”.
  For the filter you can apply something like TestResults*UnitReport.xml.
• Add the “Post-build action” -> “Publish HTML reports”.
  Add the OpenCover and the chutzpah report files.
• Add the “Post-build action” -> “Publish Cobertura Coverege Report”.
  Add the report files path (ex: “TestResults*CoberturaCodeCoverage.xml”).

Executed Command

In the end the full batch script should look something like this:

REM 'Clean Unit Tests Reports'
rd /S /Q TestResults
md TestResultsHTMLJSReport

REM 'Run the .NET Unit Tests with Coverage.'
"C:ToolsOpenCoverOpenCover.Console.exe" -target:"%MSTEST_PATH%MSTest.exe" -targetargs:"/nologo /noisolation /testcontainer:%UNITTESTPROJECT%bin%BUILDMODE%%UNITTESTPROJECT%.dll /resultsfile:TestResultsUnitTestReport.trx" -filter:"+[%ASSEMBLIES%*]*" -register -mergebyhash -hideskipped:Filter -output:"TestResultsOpenCoverCodeCoverage.xml"

REM 'Start IIS Express'
start "IIS" "%IIS_PATH%iisexpress.exe" /path:"%WORKSPACE%%WEBPROJECT%" /port:%PORT% /systray:false

REM 'Run the .NET Acceptance Tests.'
"%MSTEST_PATH%MSTest.exe" /nologo /noisolation /testcontainer:%ACCEPTANCETESTPROJECT%bin%BUILDMODE%%ACCEPTANCETESTPROJECT%.dll /resultsfile:TestResultsAcceptanceTestReport.trx

REM 'Stop IIS Express'
taskkill /IM iisexpress.exe

REM 'Convert MSTest report to JUnit report'
"C:ToolsMSTestToJUnitmsxsl.exe" TestResultsUnitTestReport.trx C:ToolsMSTestToJUnitmstest-to-junit.xsl -o TestResultsUnitTestsJUnitReport.xml
"C:ToolsMSTestToJUnitmsxsl.exe" TestResultsAcceptanceTestReport.trx C:ToolsMSTestToJUnitmstest-to-junit.xsl -o TestResultsAcceptanceTestsJUnitReport.xml

REM 'Convert OpenConver report to Cobertura report'
"C:ToolsOpenCoverToCoberturaOpenCoverToCoberturaConverter.exe" -input:TestResultsOpenCoverCodeCoverage.xml -output:TestResultsCoberturaCodeCoverage.xml -sources:%WORKSPACE%

REM 'Create HTML Report from OpenCover report.'
"C:ToolsReportGeneratorReportGenerator.exe" -reports:TestResultsCoberturaCodeCoverage.xml -targetDir:TestResultsHTMLCSReport

REM 'Run the JavaScript Unit Tests with Coverage.'
"C:ToolsChutzpahchutzpah.console.exe" /silent /coverage /path %UNITTESTPROJECT%WebJSApp /junit TestResultsJasmineJUnitReport.xml

REM 'Move the reports to TestResults'
move _Chutzpah.coverage.json TestResults
move _Chutzpah.coverage.html TestResultsHTMLJSReportChutzpahIndex.html

REM 'Convert Chutzpah report to Cobertura report'
"C:Tools\ChutzpahToCobertura\ChutzpahToCobertura.exe" TestResults_Chutzpah.coverage.json TestResultsJSCoberturaCodeCoverage.xml

Resources

• http://jane.dallaway.com/jenkins-code-coverage-net/
• Writing Automated Acceptance Tests with Spec Flow

Gallery