Skip to content
  • Services
    Our Approach
    Personalized, in-depth technical guidance on a global scale that helps organizations achieve their digital transformation goals.
    Learn more
    • Our Approach
    • Development
    • Design
    • Digital Experience Platform
    • Data & Analytics
    • Cloud & DevOps
    • Support
  • Work
    Our Work
    Through our expertise in strategy, design, and engineering, we help clients deliver digital transformation at scale.
    Learn more
    • Our Work
    • Healthcare
    • Finance
    • Manufacturing
    • Agriculture
    • Education
  • About
    About us
    For over 20 years, we’ve partnered with companies of all sizes and industries to solve their most complex business problems.
    Learn more
    • About us
    • Leadership
    • Locations
    • Events
    • News
  • Careers
    Join our Team
    Take your career to the next level. We offer exciting opportunities across every stage of the software development life cycle.
    Learn more
    • Join our Team
    • Open Positions
    • Application Process
    • Benefits
    • Learning & Development
  • Insights
    Our Insights
    Read our latest blogs, watch our recent videos, and browse our library of e-books — all full of insights from our experts.
    Learn more
    • Our Insights
    • Blog
    • Videos
    • Downloads
  • Contact
Menu

Jenkins CI in MVC.NET Part 2: Tests and Coverage

While we covered the installation of Jenkins, here we talk about running tests and its code coverage tools.

Annika Hey

Annika Hey

Design Principal

Atanas Atanasov

Atanas Atanasov

Software Development Manager – Agile Frameworks

Björn Stansvik

Founder & Chief Executive Officer

Daniela Nazim

Daniela Nazim

MentorMate Alumni

Dimitar Dobrev

Dimitar Dobrev

MentorMate Alumni

Craig Knighton

Chief Operating Officer

Eleonora Georgieva

Global VP, Delivery

Georgi Dormishev

George Dormishev

System Administration Manager

Ivaylo Kostadinov

Director, Software Engineering - .NET

Jamie Bolseth profile picture

Jamie Bolseth

MentorMate Alumni

Jay Miller

President

Jeni Kyuchukova

Director, Quality Engineering

Jessica Anderson

VP of Finance and Administration

Liz Spolyar

Global Director, Continuation Engineering

Nick Curran

Nick Curran

Technical Architect

Nikolay Lyubchev

Global Director, Talent Acquisition, MentorMate

Stefan Tsvyatkov

Director, Software Engineering - Mobile

Stefan Tzanev

Chief Financial Officer

Vesselin Dobrev

Vesselin Dobrev

General Manager (Bulgaria)

Sylvia Vassileva

Sylvia Vassileva

Software Development Manager - Spok

Filip Gajtanovski

Software Development Manager - Storyworks

Krasimir K. Nikolov

VP of Technology

Katherine Kelly

Director of Operations (USA)

Carrie Siewert

Strategic Account Manager

Brady Swanson

Global Director, Marketing

Eve Poeschl

MentorMate Alumni

Ryan Peña

MentorMate Alumni

Vassil Vassilev

Software Development Manager - .NET

Pavel Petrov

Director, Software Engineering - LAMP&FE

Ivan Peev

Senior Technology Manager

Bob Reuss

MentorMate Alumni

Vera Kasapova

QA Manager

Greta Yamacheva

QA Manager

Robert Samuelsson

General Manager (Sweden)

Kyle Simmons

Solutions Architect

Robin Thomas

Solutions Architect

Nataliya Naydenova

MentorMate Alumni

Adam Malone

Alexander Dimitrov

Enterprise Architect

Andrea Kates

CEO, LaunchPad Central

Andrew Eklund

CEO, Ciceron

Andrew Marinov

Angel Nikolov

MentorMate Alumni

Anurag Shukla

Aron Wolde

MentorMate Alumni

Ashley Goodridge

Office Assistant

Benjamin Gramlich

MentorMate Alumni

Chris Black

MentorMate Alumni

Christa Haeg

MentorMate Alumni

Colin Lee

MentorMate Alumni

Deyan Stoynov

MentorMate Alumni

Dimitar Danailov

MentorMate Alumni

Dobrinka Tabakova

Doug Leatherman

Emily Genco

MentorMate Alumni

Fanka Vassileva

Gabriela Zagarova

MentorMate Alumni

Gary Conkright

CEO, physIQ

Gary Fingerhut

Executive Director, Cleveland Clinic Innovations

Gavin Finden

MentorMate Alumni

Georgi

Graham Klang

Hyusein Hyuseinov

Senior Automation QA

Ian Good

Global VP, Operations

Iva

Jack Cosentino

James Williams

John Byrne

Kaloyan Stoilkov

MentorMate Alumni

Kosta Hristov

Krasimir Gatev

Senior Android Developer

Lazar Petrakiev

Lyubomir Dobrev

Senior .NET Developer

Lubomir Velkov

Marin Yotovski

Mark Smith

MentorMate Alumni

Martin Dimitrov

MentorMate Alumni

Martin Kalyonski

Mike Hagan

MentorMate Alumni

Nikolay Andonov

Nikolay Arhangelov

Riley Panko

Guest Contributor

Roger Ferguson

MentorMate Alumni

Ryan Sysko

Chairman, WellDoc

Ryan Blake

MentorMate Alumnus

Sarah Rockholt

MentorMate Alumni

Sean McDevitt

CEO, Sensei

Siyana Slavova

Stanislas Walden

MentorMate Alumni

Stanislav Atanasov

Stanislava Bogdanova

MentorMate Alumni

Stefanie Trimble

MentorMate Alumnus

Stephen Fluin

Stoyan Stoyanov

MentorMate Alumnus

Tessa Cacek

Staffing Manager

Tom Clemens

MentorMate Alumnus

V8 JavaScript Engine

Viktor Mitev

Yolanda Petkova

Marketing Design Lead

Pete Anderson

Lead Product Owner, Target

MentorMate Software Development Lead Vasil Nonchev

Vasil Nonchev

Java Software Development Manager

Dilyana Totseva

QA Manager

Stanimir Nikolov profile picture

Stanimir Nikolov

Software Development Lead - iOS, MentorMate

Rosen Kolev

Technology Principal

Dimitar Mihaylov

MentorMate Alumni

Nikola Genov

Software Architect - .NET

Neli Todorova

Software Development Manager - LAMP

Yavor Dimitrov

MentorMate Alumni

Georgi Karanedyalkov

Software Development Lead - Android, MentorMate

Denislav Ganchev

Technology Principal

Stefan Shopov

QA Manager

Konstantin Rusev

Java Developer

Borislav Dimitrov profile picture

Borislav Dimitrov

Senior Android Developer, MentorMate

Tsvetelina Lazarova

MentorMate Alumni

Dimitar Gadzhev

Developer

Plamen Stoev

Software Development Manager - Front-end

Jake Nelsen

Senior Experience Designer

Zlati Pehlivanov

Zlati Pehlivanov

Senior Software Engineer II

Kate Tolmie, MentorMate Senior UX Designer

Kate Tolmie

Senior Experience Designer

Martin Angelov

Director, Software Engineering - LAMP&FE, MentorMate

Dimitar Zhelev

Senior .NET Developer

Joel Swenson, MentorMate Content Writer

Joel Swenson

Content Manager

Kiril Ivanov

Quality Assurance Analyst

Viktor Hristoskov profile picture

Viktor Hristoskov

Software Development Lead - iOS, MentorMate

Violeta Nikolcheva

Database Developer

Biliana Kadakevlieva

Senior Quality Assurance Analyst

Chris McLeod

Senior Solutions Consultant

Antonii Georgiev

Junior .NET Developer

Alexander Rusev

Front-End Developer

Matt Erickson, MentorMate PR and Social Media Manager

Matt Erickson

MentorMate Alumni

Brian Buchkosky

Global Director, PMO

David Tran, MentorMate VP of Solutions

David Tran

MentorMate Alumni

Kristin Krueger

MentorMate Alumni

Magdalena Chervenkova

Business Analyst

Denny Royal

Chief Design Officer

MentorMate Technical Account Strategist Joe Bodell

Joe Bodell

MentorMate Alumni

Viktoriya Chuchumisheva - MentorMate HR Manager

Viktoria Chuchumisheva

HR Manager

Kalina Tekelieva Headshot

Kalina Tekelieva

Senior Content Marketing Artist

Daniel Rankov profile picture

Daniel Rankov

MentorMate Alumni

MentorMate Senior Business Analyst Alexander Alexandrov

Alexander Alexandrov

BA Lead

MentorMate

Clint Rowles

VP, Business Development

Nikola Donev - SysAdmin

Nikola Donev

SysOps & DevOps Lead

Tseko Tsolov

Frontend Developer

Denislav Lefterov

Automation QA Analyst

MentorMate Content Writer Dilyana Kodjamanova

Dilyana Kodjamanova

MentorMate Alumni

MentorMate Project Manager Emma Jorstad

Emma Jorstad

Project Manager, Lead

Georgi Georgiev profile picture

Georgi Georgiev

Software Development Lead - LAMP, MentorMate

Martin Panayotov profile picture

Martin Panayotov

Senior iOS Developer, MentorMate

John Blake

John Blake

Senior Account Manager

Tyler Compton

Tyler Compton

Solutions Architect

MentorMate Software Developer Nikola Peevsk

Nikola Peevski

Software Developer — Lamp & Front-End

Aaron Whitney

Director of Client Strategy

MentorMate Senior Cloud Engineer Veliko Ivanov

Veliko Ivanov

Senior Cloud Engineer

MentorMate Senior Project Manager Suzanne O'Brien

Suzanne O’Brien

Senior Project Manager

Svetlin Stanchev profile picture

Svetlin Stanchev

Software Development Lead - Front-end, MentorMate

MentorMate Senior Cloud Engineer Todor Todorov

Todor Todorov

Senior Cloud Engineer

MentorMate Senior QA Analyst Kate Stamatova

Kate Stamatova

Senior QA Analyst

Frank Anselmo profile pic

Frank Anselmo

Global Director, Project Management

Gyuner Zeki Headshot

Gyuner Zeki

Solutions Architect

Galin Stanchev

QA Analyst

Sarah Hoops

Business Development Manager

Brenden Diehl

Business Development Manager

Anna Krivova profile picture

Anna Krivova

Software Development Lead - Front-end, MentorMate

Ivelina Kavalova profile picture

Ivelina Kavalova

Senior Business Analyst, MentorMate

Paul Sanders

MentorMate Alumni

Jim Cikanek

Senior Client Strategist

Samuil Yanovski profile picture

Samuil Yanovski

Software Development Manager - Android, MentorMate

Krasimir Gatev profile picture

Krasimir Gatev

Senior Android Developer, MentorMate

Kristina Goryalova headshot

Kristina Goryalova

Talent Acquisition Manager

Elena Petrova Headshot

Elena Petrova

HR Specialist

Jay Matre

Senior Business Architect, MentorMate

Lilyana Dimitrova

QA Specialist

Josh Marquart

Chief Strategy Officer

Mario Gorki

Senior Mobile Developer

Simeon Zhekov Headshot

Simeon Zhekov

Cloud Engineer

Hristo Stoyanov Headshot

Hristo Stoyanov

Cloud & DevOps Lead

Ben Wallace

Enterprise Architect

Boyan Stoyanov

Data & Dota Specialist

MentorMate Director of Software Engineering Petya Ivanova

Petya Ivanova

Director, Software Engineering - Java

Sebastian Ortiz-Chamorro

VP of Engineering, Latin America

Consuelo Merino profile pic

Consuelo Merino

Director of Operations

We have already talked about how to install Jenkins, Build a project and push it to another branch here. But Jenkins is capable of much more. In this topic we will talk about running tests and code coverage tools. For this purpose I have written an simple application that just open a web page with a Login and Home Page.

The application can be downloaded here.

The Application

The application do not have any database or business logic. Its purpose is to contains several unit tests and to measure code coverage of this tests. There are three type of tests in the application and we will demonstrate how to run all of them in Jenkins.

Functional .NET Tests

This are general c# tests that test a units or components and are written with Visual Studio Quality Tools. We can run this with MSTest.

Acceptance Tests

This are test that use the browser to test actual behavior. I have written them with SpecFlow wich can be run again with MSTest. SpecFlow uses Selenium Browser Driver to start and control Chrome (or any other browser you prefer). We do not need to run this test every time we make a build because they are slow and require fully loaded and working environment. We can run them prior of deploying to tests, stage or production environment.

Javascript Tests

Test that run on the client side and verify the behavior of the client scripts. We can run this tests and code coverage for them with the Chutzpah tool.

Jenkins Setup

Build Variables

Some build variables are set for the current job and are as fallow:

  • BUILDMODE – Release or Debug.
  • UNITTESTPROJECT – The name of the project and assembly with the unit tests.
  • ACCEPTANCETESTPROJECT – The name of the project that hold the acceptance tests.
  • WEBPROJECT – The name of the Web application project.
  • PORT – The port that acceptance tests run against.
  • ASSEMBLIES – The prefix of the Assemblies. If this is ‘App’, all your project assemblies must start with App.
    Example: App.Core, App.Services, App.Web, etc.
Build

First thing to do is download the project from the repository, clean and build it. You can fallow the steps in the previous article if you have problem setting this step.

Run Functional Tests With MSTest

Next you need to acquire MSTest. You can do this several ways:

  • Install Visual Studio on the Jenkins machine. I do not prefer this method, because you do not actually need it there, except if you are using this machine for developing purposes.
  • Install “Agents for Microsoft Visual Studio” from the Microsoft download page. They include MSTest.
  • Follow the steps in this article. With this option you can put all files at “C:ToolsMSTest”. Note that you have to play with the registers in order to make it run.

To add the unit tests you can use the MSTest plugin, but I have found problems when using it so I am using the batch scripts.

Also it is a good idea to set the MSTest path as Environment variables in the Jenkins configuration page.

Another problem is that Jenkins is working with JUnit report files and MSTest is generating its own report files. We can transform with the help of MSXSL and a XSL transform file. We can download MSXSL from the Microsoft download site here.

You may download the transformation xsl from here. We can add the both files in single directory like C:ToolsMSTestToJUnit.

This is the final script.

REM 'Clean Unit Tests Reports'
del /q "%UNITTESTPROJECT%MSTestReport.trx"
del /q "%UNITTESTPROJECT%MSTestJUnitReport.xml"

REM 'Run the .NET Unit Tests.'
"%MSTEST_PATH%MSTest.exe" /nologo /testcontainer:%UNITTESTPROJECT%bin%BUILDMODE%%UNITTESTPROJECT%.dll /resultsfile:"%UNITTESTPROJECT%MSTestReport.trx"
"C:ToolsMSTestToJUnitmsxsl.exe" %UNITTESTPROJECT%MSTestReport.trx C:ToolsMSTestToJUnitmstest-to-junit.xsl -o %UNITTESTPROJECT%MSTestJUnitReport.xml

This script delete the MSTest and the JUnit report files if they exists. Run MSTest on the tests binary and after that transform it into JUnit report.

Run Acceptance Tests With MSTest

The acceptance test are written with SpecFlow. But the SpecFlow allow the tests to be run by MSTest so the acceptance test are run the same way as the unit test we just point to the acceptance test dlls.

The acceptance tests are generally much slower then the unit test and we shouldn’t run them on every build. We can run them when doing a delivery to environment.

In order to run the acceptance test we need an actual running site. We have 2 options:

  1. Deploy to a development environment before you run the tests (environment with IIS and web site visible by the Jenkins CI server). After that run the acceptance test against that site.
  2. Run the site on the spot with IIS Express. You can start IIS express and run the test. For this approach you must download and install IIS Express from the Microsoft download site here.
    Than you can run it and stop it like this:
    REM 'Start IIS Express'
    "%IISEXPRESS%iisexpress" /path:%WORKSPACE%Web /port:59725 /systray:false
    
    // Run the acceptance tests against http://localhost:59725
    
    REM 'Stop IIS Express'
    taskkill /IM iisexpress.exe
    
Run Client Tests With Chutzpah

In order to run the JavaScript tests we need the Chutzpah console application. We can download it from the chutzpah site here.

You need to download the full chutzpah archive and extract it. The .pdb and the chutzpah.console.vshost files are not generally needed. It is good idea again to extract it in some location with easy access like “C:ToolsChutzpah”.

This is the batch script that run the JS tests.

REM "Clean JavaScript Unit Tests Reports"
del /q "TestResultsJasmineJUnitReport.xml"

REM "Run the JavaScript Unit Tests"
"C:ToolsChutzpahchutzpah.console.exe" /silent /path %UNITTESTPROJECT%WebJSapp /junit TestResultsJasmineJUnitReport.xml

Functional Tests Coverage

In order to run code coverage we need a code coverage analyze tool. I will use OpenCover, but other tools are also available. This are the tools we will use to run our code covarage.

  1. OpenCover – being used as the coverage checker.
  2. OpenCoverToCoberturaConverter – a tool to convert the XML output from OpenCover into a format recognized by Cobertura (for which there is a Jenkins plugin).
  3. Cobertura plugin – used to give a nice progress chart showing (hopefully) increases in coverage over time.
  4. ReportGenerator – a tool to convert the XML output from OpenCover into clickable HTML showing line by line usage – used to show the current build’s usage in an informative manner
  5. HTML Publisher plugin – used to publish the HTML output from ReportGenerator into a way that can be accessed via the Jenkins dashboard.

When you include code coverage for unit test you can run the tests and the coverage at the same time.
Here is a script that do this.

"C:ToolsOpenCoverOpenCover.Console.exe" -target:"%MSTEST_PATH%MSTest.exe" -targetargs:"/nologo /noisolation /testcontainer:%UNITTESTPROJECT%bin%BUILDMODE%%UNITTESTPROJECT%.dll /resultsfile:TestResultsUnitTestReport.trx" -filter:"+[%ASSEMPLIES%*]*" -register -mergebyhash -hideskipped:Filter -output:"TestResultsOpenCoverCodeCoverage.xml"

We run MSTest through open cover with the targeted .dll and the output file. /noisolation attribute tell MSTest not to copy the assemblies in another folder when testing. This is needed because PDBs may not be found and the code coverage may not run successful.
After that we will need to convert the OpenCover report to Cobertura report with the OpenCoverToCoberturaConverter tool.
REM 'Convert OpenConver report to Cobertura report'
"C:ToolsOpenCoverToCoberturaOpenCoverToCoberturaConverter.exe" -input:TestResultsOpenCoverCodeCoverage.xml -output:TestResultsCoberturaCodeCoverage.xml -sources:%WORKSPACE%

Client Tests Coverage

The client tests coverage are run again with Chutzpah just by adding coverage option. It creates 2 files json report and a html report. You can also add the options /coverageIncludes and/or /coverageExcludes options to add extra files in the coverage report or exclude files. You can also exclude files not needing coverage with chutzpah-exclude=”true” attribute in the reference tags.
There is 2 problems when using this approach.

  1. Chutzpah will create the 2 files on the root folder, not the one where /junit option point to.
  2. You can not visualize the json report and also have to convert it to Cobertura.

So for the first problem we may move the files by executing move command. Chitzpah create 2 files “_Chutzpah.coverage.html” and “_Chutzpah.coverage.json”, so we just do.

move _Chutzpah.coverage.json TestResults
move _Chutzpah.coverage.html TestResultsHTMLJSReportChutzpahIndex.html

For the second part we will need to convert the chutzpah to cobertura report. We can use this tool.
We download it and extract the files in the “C:ToolsChutzpahToCobertura” folder of the Jenkins CI server.
"C:ToolsChutzpahToCoberturaChutzpahToCobertura.exe" TestResults_Chutzpah.coverage.json TestResultsJSCoberturaCodeCoverage.xml

Visualization

In order to show the results we need to do several things.

  • Add the “Post-build action” -> “Publish JUnit test result report”.
    For the filter you can apply something like TestResults*UnitReport.xml.
  • Add the “Post-build action” -> “Publish HTML reports”.
    Add the OpenCover and the chutzpah report files.
  • Add the “Post-build action” -> “Publish Cobertura Coverege Report”.
    Add the report files path (ex: “TestResults*CoberturaCodeCoverage.xml”).

Executed Command

In the end the full batch script should look something like this:

REM 'Clean Unit Tests Reports'
rd /S /Q TestResults
md TestResultsHTMLJSReport

REM 'Run the .NET Unit Tests with Coverage.'
"C:ToolsOpenCoverOpenCover.Console.exe" -target:"%MSTEST_PATH%MSTest.exe" -targetargs:"/nologo /noisolation /testcontainer:%UNITTESTPROJECT%bin%BUILDMODE%%UNITTESTPROJECT%.dll /resultsfile:TestResultsUnitTestReport.trx" -filter:"+[%ASSEMBLIES%*]*" -register -mergebyhash -hideskipped:Filter -output:"TestResultsOpenCoverCodeCoverage.xml"

REM 'Start IIS Express'
start "IIS" "%IIS_PATH%iisexpress.exe" /path:"%WORKSPACE%%WEBPROJECT%" /port:%PORT% /systray:false

REM 'Run the .NET Acceptance Tests.'
"%MSTEST_PATH%MSTest.exe" /nologo /noisolation /testcontainer:%ACCEPTANCETESTPROJECT%bin%BUILDMODE%%ACCEPTANCETESTPROJECT%.dll /resultsfile:TestResultsAcceptanceTestReport.trx

REM 'Stop IIS Express'
taskkill /IM iisexpress.exe

REM 'Convert MSTest report to JUnit report'
"C:ToolsMSTestToJUnitmsxsl.exe" TestResultsUnitTestReport.trx C:ToolsMSTestToJUnitmstest-to-junit.xsl -o TestResultsUnitTestsJUnitReport.xml
"C:ToolsMSTestToJUnitmsxsl.exe" TestResultsAcceptanceTestReport.trx C:ToolsMSTestToJUnitmstest-to-junit.xsl -o TestResultsAcceptanceTestsJUnitReport.xml

REM 'Convert OpenConver report to Cobertura report'
"C:ToolsOpenCoverToCoberturaOpenCoverToCoberturaConverter.exe" -input:TestResultsOpenCoverCodeCoverage.xml -output:TestResultsCoberturaCodeCoverage.xml -sources:%WORKSPACE%

REM 'Create HTML Report from OpenCover report.'
"C:ToolsReportGeneratorReportGenerator.exe" -reports:TestResultsCoberturaCodeCoverage.xml -targetDir:TestResultsHTMLCSReport

REM 'Run the JavaScript Unit Tests with Coverage.'
"C:ToolsChutzpahchutzpah.console.exe" /silent /coverage /path %UNITTESTPROJECT%WebJSApp /junit TestResultsJasmineJUnitReport.xml

REM 'Move the reports to TestResults'
move _Chutzpah.coverage.json TestResults
move _Chutzpah.coverage.html TestResultsHTMLJSReportChutzpahIndex.html

REM 'Convert Chutzpah report to Cobertura report'
"C:Tools\ChutzpahToCobertura\ChutzpahToCobertura.exe" TestResults_Chutzpah.coverage.json TestResultsJSCoberturaCodeCoverage.xml

Resources

  • http://jane.dallaway.com/jenkins-code-coverage-net/
  • Writing Automated Acceptance Tests with Spec Flow
Gallery

MSBuild Post-Build. Report

Tags
  • Development
Share
  • Share on Facebook
  • Share on LinkedIn
  • Share on Twitter
Share
  • Share on Facebook
  • Share on LinkedIn
  • Share on Twitter
Sign up for our monthly newsletter.
Sign up for our monthly newsletter.

Read what's next.

Blog

How to Write Effective Technical Tasks and User Stories

Blog

Measuring the Business Value of Technology Investments

  • Twitter
  • LinkedIn
  • Instagram
  • Facebook
United States
MentorMate1350 Lagoon Ave, Suite 800
Minneapolis
, MN 55408

+1 612 823 4000
Bulgaria
67 Prof. Tsvetan Lazarov Blvd.
Sofia 1592, Bulgaria,
+359 2 862 2632
Sweden
Drottninggatan 29
411 14 Göteborg

+46 3 199 0180
Paraguay
Carlos M. Gimenez 4855
Asunción, Paraguay

+595 21 327 9463

Copyright © 2023 MentorMate, Inc.

  • Cookies
  • Privacy
  • Terms
  • Continuity Policy
This site is registered on wpml.org as a development site.