Jenkins CI in MVC.NET Part 2: Tests and Coverage While we covered the installation of Jenkins, here we talk about running tests and its code coverage tools. Annika Hey Design Principal Atanas Atanasov Software Development Manager – Agile Frameworks Björn Stansvik Founder & Chief Executive Officer Daniela Nazim MentorMate Alumni Dimitar Dobrev MentorMate Alumni Craig Knighton Chief Operating Officer Eleonora Georgieva Global VP, Delivery George Dormishev System Administration Manager Ivaylo Kostadinov Director, Software Engineering - .NET Jamie Bolseth MentorMate Alumni Jay Miller President Jeni Kyuchukova Director, Quality Engineering Jessica Anderson VP of Finance and Administration Liz Spolyar Global Director, Continuation Engineering Nick Curran Technical Architect Nikolay Lyubchev Global Director, Talent Acquisition, MentorMate Stefan Tsvyatkov Director, Software Engineering - Mobile Stefan Tzanev Chief Financial Officer Vesselin Dobrev General Manager (Bulgaria) Sylvia Vassileva Software Development Manager - Spok Filip Gajtanovski Software Development Manager - Storyworks Krasimir K. Nikolov VP of Technology Katherine Kelly Director of Operations (USA) Carrie Siewert Strategic Account Manager Brady Swanson Global Director, Marketing Eve Poeschl MentorMate Alumni Ryan Peña MentorMate Alumni Vassil Vassilev Software Development Manager - .NET Pavel Petrov Director, Software Engineering - LAMP&FE Ivan Peev Senior Technology Manager Bob Reuss MentorMate Alumni Vera Kasapova QA Manager Greta Yamacheva QA Manager Robert Samuelsson General Manager (Sweden) Kyle Simmons Solutions Architect Robin Thomas Solutions Architect Nataliya Naydenova MentorMate Alumni Adam Malone Alexander Dimitrov Enterprise Architect Andrea Kates CEO, LaunchPad Central Andrew Eklund CEO, Ciceron Andrew Marinov Angel Nikolov MentorMate Alumni Anurag Shukla Aron Wolde MentorMate Alumni Ashley Goodridge Office Assistant Benjamin Gramlich MentorMate Alumni Chris Black MentorMate Alumni Christa Haeg MentorMate Alumni Colin Lee MentorMate Alumni Deyan Stoynov MentorMate Alumni Dimitar Danailov MentorMate Alumni Dobrinka Tabakova Doug Leatherman Emily Genco MentorMate Alumni Fanka Vassileva Gabriela Zagarova MentorMate Alumni Gary Conkright CEO, physIQ Gary Fingerhut Executive Director, Cleveland Clinic Innovations Gavin Finden MentorMate Alumni Georgi Graham Klang Hyusein Hyuseinov Senior Automation QA Ian Good Global VP, Operations Iva Jack Cosentino James Williams John Byrne Kaloyan Stoilkov MentorMate Alumni Kosta Hristov Krasimir Gatev Senior Android Developer Lazar Petrakiev Lyubomir Dobrev Senior .NET Developer Lubomir Velkov Marin Yotovski Mark Smith MentorMate Alumni Martin Dimitrov MentorMate Alumni Martin Kalyonski Mike Hagan MentorMate Alumni Nikolay Andonov Nikolay Arhangelov Riley Panko Guest Contributor Roger Ferguson MentorMate Alumni Ryan Sysko Chairman, WellDoc Ryan Blake MentorMate Alumnus Sarah Rockholt MentorMate Alumni Sean McDevitt CEO, Sensei Siyana Slavova Stanislas Walden MentorMate Alumni Stanislav Atanasov Stanislava Bogdanova MentorMate Alumni Stefanie Trimble MentorMate Alumnus Stephen Fluin Stoyan Stoyanov MentorMate Alumnus Tessa Cacek Staffing Manager Tom Clemens MentorMate Alumnus V8 JavaScript Engine Viktor Mitev Yolanda Petkova Marketing Design Lead Pete Anderson Lead Product Owner, Target Vasil Nonchev Java Software Development Manager Dilyana Totseva QA Manager Stanimir Nikolov Software Development Lead - iOS, MentorMate Rosen Kolev Technology Principal Dimitar Mihaylov MentorMate Alumni Nikola Genov Software Architect - .NET Neli Todorova Software Development Manager - LAMP Yavor Dimitrov MentorMate Alumni Georgi Karanedyalkov Software Development Lead - Android, MentorMate Denislav Ganchev Technology Principal Stefan Shopov QA Manager Konstantin Rusev Java Developer Borislav Dimitrov Senior Android Developer, MentorMate Tsvetelina Lazarova MentorMate Alumni Dimitar Gadzhev Developer Plamen Stoev Software Development Manager - Front-end Jake Nelsen Senior Experience Designer Zlati Pehlivanov Senior Software Engineer II Kate Tolmie Senior Experience Designer Martin Angelov Director, Software Engineering - LAMP&FE, MentorMate Dimitar Zhelev Senior .NET Developer Joel Swenson Content Manager Kiril Ivanov Quality Assurance Analyst Viktor Hristoskov Software Development Lead - iOS, MentorMate Violeta Nikolcheva Database Developer Biliana Kadakevlieva Senior Quality Assurance Analyst Chris McLeod Senior Solutions Consultant Antonii Georgiev Junior .NET Developer Alexander Rusev Front-End Developer Matt Erickson MentorMate Alumni Brian Buchkosky Global Director, PMO David Tran MentorMate Alumni Kristin Krueger MentorMate Alumni Magdalena Chervenkova Business Analyst Denny Royal Chief Design Officer Joe Bodell MentorMate Alumni Viktoria Chuchumisheva HR Manager Kalina Tekelieva Senior Content Marketing Artist Daniel Rankov MentorMate Alumni Alexander Alexandrov BA Lead MentorMate Clint Rowles VP, Business Development Nikola Donev SysOps & DevOps Lead Tseko Tsolov Frontend Developer Denislav Lefterov Automation QA Analyst Dilyana Kodjamanova MentorMate Alumni Emma Jorstad Project Manager, Lead Georgi Georgiev Software Development Lead - LAMP, MentorMate Martin Panayotov Senior iOS Developer, MentorMate John Blake Senior Account Manager Tyler Compton Solutions Architect Nikola Peevski Software Developer — Lamp & Front-End Aaron Whitney Director of Client Strategy Veliko Ivanov Senior Cloud Engineer Suzanne O’Brien Senior Project Manager Svetlin Stanchev Software Development Lead - Front-end, MentorMate Todor Todorov Senior Cloud Engineer Kate Stamatova Senior QA Analyst Frank Anselmo Global Director, Project Management Gyuner Zeki Solutions Architect Galin Stanchev QA Analyst Sarah Hoops Business Development Manager Brenden Diehl Business Development Manager Anna Krivova Software Development Lead - Front-end, MentorMate Ivelina Kavalova Senior Business Analyst, MentorMate Paul Sanders MentorMate Alumni Jim Cikanek Senior Client Strategist Samuil Yanovski Software Development Manager - Android, MentorMate Krasimir Gatev Senior Android Developer, MentorMate Kristina Goryalova Talent Acquisition Manager Elena Petrova HR Specialist Jay Matre Senior Business Architect, MentorMate Lilyana Dimitrova QA Specialist Josh Marquart Chief Strategy Officer Mario Gorki Senior Mobile Developer Simeon Zhekov Cloud Engineer Hristo Stoyanov Cloud & DevOps Lead Ben Wallace Enterprise Architect Boyan Stoyanov Data & Dota Specialist Petya Ivanova Director, Software Engineering - Java Sebastian Ortiz-Chamorro VP of Engineering, Latin America Consuelo Merino Director of Operations We have already talked about how to install Jenkins, Build a project and push it to another branch here. But Jenkins is capable of much more. In this topic we will talk about running tests and code coverage tools. For this purpose I have written an simple application that just open a web page with a Login and Home Page. The application can be downloaded here. The Application The application do not have any database or business logic. Its purpose is to contains several unit tests and to measure code coverage of this tests. There are three type of tests in the application and we will demonstrate how to run all of them in Jenkins. Functional .NET Tests This are general c# tests that test a units or components and are written with Visual Studio Quality Tools. We can run this with MSTest. Acceptance Tests This are test that use the browser to test actual behavior. I have written them with SpecFlow wich can be run again with MSTest. SpecFlow uses Selenium Browser Driver to start and control Chrome (or any other browser you prefer). We do not need to run this test every time we make a build because they are slow and require fully loaded and working environment. We can run them prior of deploying to tests, stage or production environment. Javascript Tests Test that run on the client side and verify the behavior of the client scripts. We can run this tests and code coverage for them with the Chutzpah tool. Jenkins Setup Build Variables Some build variables are set for the current job and are as fallow: BUILDMODE – Release or Debug. UNITTESTPROJECT – The name of the project and assembly with the unit tests. ACCEPTANCETESTPROJECT – The name of the project that hold the acceptance tests. WEBPROJECT – The name of the Web application project. PORT – The port that acceptance tests run against. ASSEMBLIES – The prefix of the Assemblies. If this is ‘App’, all your project assemblies must start with App. Example: App.Core, App.Services, App.Web, etc. Build First thing to do is download the project from the repository, clean and build it. You can fallow the steps in the previous article if you have problem setting this step. Run Functional Tests With MSTest Next you need to acquire MSTest. You can do this several ways: Install Visual Studio on the Jenkins machine. I do not prefer this method, because you do not actually need it there, except if you are using this machine for developing purposes. Install “Agents for Microsoft Visual Studio” from the Microsoft download page. They include MSTest. Follow the steps in this article. With this option you can put all files at “C:ToolsMSTest”. Note that you have to play with the registers in order to make it run. To add the unit tests you can use the MSTest plugin, but I have found problems when using it so I am using the batch scripts. Also it is a good idea to set the MSTest path as Environment variables in the Jenkins configuration page. Another problem is that Jenkins is working with JUnit report files and MSTest is generating its own report files. We can transform with the help of MSXSL and a XSL transform file. We can download MSXSL from the Microsoft download site here. You may download the transformation xsl from here. We can add the both files in single directory like C:ToolsMSTestToJUnit. This is the final script. REM 'Clean Unit Tests Reports' del /q "%UNITTESTPROJECT%MSTestReport.trx" del /q "%UNITTESTPROJECT%MSTestJUnitReport.xml" REM 'Run the .NET Unit Tests.' "%MSTEST_PATH%MSTest.exe" /nologo /testcontainer:%UNITTESTPROJECT%bin%BUILDMODE%%UNITTESTPROJECT%.dll /resultsfile:"%UNITTESTPROJECT%MSTestReport.trx" "C:ToolsMSTestToJUnitmsxsl.exe" %UNITTESTPROJECT%MSTestReport.trx C:ToolsMSTestToJUnitmstest-to-junit.xsl -o %UNITTESTPROJECT%MSTestJUnitReport.xml This script delete the MSTest and the JUnit report files if they exists. Run MSTest on the tests binary and after that transform it into JUnit report. Run Acceptance Tests With MSTest The acceptance test are written with SpecFlow. But the SpecFlow allow the tests to be run by MSTest so the acceptance test are run the same way as the unit test we just point to the acceptance test dlls. The acceptance tests are generally much slower then the unit test and we shouldn’t run them on every build. We can run them when doing a delivery to environment. In order to run the acceptance test we need an actual running site. We have 2 options: Deploy to a development environment before you run the tests (environment with IIS and web site visible by the Jenkins CI server). After that run the acceptance test against that site. Run the site on the spot with IIS Express. You can start IIS express and run the test. For this approach you must download and install IIS Express from the Microsoft download site here. Than you can run it and stop it like this: REM 'Start IIS Express' "%IISEXPRESS%iisexpress" /path:%WORKSPACE%Web /port:59725 /systray:false // Run the acceptance tests against http://localhost:59725 REM 'Stop IIS Express' taskkill /IM iisexpress.exe Run Client Tests With Chutzpah In order to run the JavaScript tests we need the Chutzpah console application. We can download it from the chutzpah site here. You need to download the full chutzpah archive and extract it. The .pdb and the chutzpah.console.vshost files are not generally needed. It is good idea again to extract it in some location with easy access like “C:ToolsChutzpah”. This is the batch script that run the JS tests. REM "Clean JavaScript Unit Tests Reports" del /q "TestResultsJasmineJUnitReport.xml" REM "Run the JavaScript Unit Tests" "C:ToolsChutzpahchutzpah.console.exe" /silent /path %UNITTESTPROJECT%WebJSapp /junit TestResultsJasmineJUnitReport.xml Functional Tests Coverage In order to run code coverage we need a code coverage analyze tool. I will use OpenCover, but other tools are also available. This are the tools we will use to run our code covarage. OpenCover – being used as the coverage checker. OpenCoverToCoberturaConverter – a tool to convert the XML output from OpenCover into a format recognized by Cobertura (for which there is a Jenkins plugin). Cobertura plugin – used to give a nice progress chart showing (hopefully) increases in coverage over time. ReportGenerator – a tool to convert the XML output from OpenCover into clickable HTML showing line by line usage – used to show the current build’s usage in an informative manner HTML Publisher plugin – used to publish the HTML output from ReportGenerator into a way that can be accessed via the Jenkins dashboard. When you include code coverage for unit test you can run the tests and the coverage at the same time. Here is a script that do this. "C:ToolsOpenCoverOpenCover.Console.exe" -target:"%MSTEST_PATH%MSTest.exe" -targetargs:"/nologo /noisolation /testcontainer:%UNITTESTPROJECT%bin%BUILDMODE%%UNITTESTPROJECT%.dll /resultsfile:TestResultsUnitTestReport.trx" -filter:"+[%ASSEMPLIES%*]*" -register -mergebyhash -hideskipped:Filter -output:"TestResultsOpenCoverCodeCoverage.xml" We run MSTest through open cover with the targeted .dll and the output file. /noisolation attribute tell MSTest not to copy the assemblies in another folder when testing. This is needed because PDBs may not be found and the code coverage may not run successful. After that we will need to convert the OpenCover report to Cobertura report with the OpenCoverToCoberturaConverter tool. REM 'Convert OpenConver report to Cobertura report' "C:ToolsOpenCoverToCoberturaOpenCoverToCoberturaConverter.exe" -input:TestResultsOpenCoverCodeCoverage.xml -output:TestResultsCoberturaCodeCoverage.xml -sources:%WORKSPACE% Client Tests Coverage The client tests coverage are run again with Chutzpah just by adding coverage option. It creates 2 files json report and a html report. You can also add the options /coverageIncludes and/or /coverageExcludes options to add extra files in the coverage report or exclude files. You can also exclude files not needing coverage with chutzpah-exclude=”true” attribute in the reference tags. There is 2 problems when using this approach. Chutzpah will create the 2 files on the root folder, not the one where /junit option point to. You can not visualize the json report and also have to convert it to Cobertura. So for the first problem we may move the files by executing move command. Chitzpah create 2 files “_Chutzpah.coverage.html” and “_Chutzpah.coverage.json”, so we just do. move _Chutzpah.coverage.json TestResults move _Chutzpah.coverage.html TestResultsHTMLJSReportChutzpahIndex.html For the second part we will need to convert the chutzpah to cobertura report. We can use this tool. We download it and extract the files in the “C:ToolsChutzpahToCobertura” folder of the Jenkins CI server. "C:ToolsChutzpahToCoberturaChutzpahToCobertura.exe" TestResults_Chutzpah.coverage.json TestResultsJSCoberturaCodeCoverage.xml Visualization In order to show the results we need to do several things. Add the “Post-build action” -> “Publish JUnit test result report”. For the filter you can apply something like TestResults*UnitReport.xml. Add the “Post-build action” -> “Publish HTML reports”. Add the OpenCover and the chutzpah report files. Add the “Post-build action” -> “Publish Cobertura Coverege Report”. Add the report files path (ex: “TestResults*CoberturaCodeCoverage.xml”). Executed Command In the end the full batch script should look something like this: REM 'Clean Unit Tests Reports' rd /S /Q TestResults md TestResultsHTMLJSReport REM 'Run the .NET Unit Tests with Coverage.' "C:ToolsOpenCoverOpenCover.Console.exe" -target:"%MSTEST_PATH%MSTest.exe" -targetargs:"/nologo /noisolation /testcontainer:%UNITTESTPROJECT%bin%BUILDMODE%%UNITTESTPROJECT%.dll /resultsfile:TestResultsUnitTestReport.trx" -filter:"+[%ASSEMBLIES%*]*" -register -mergebyhash -hideskipped:Filter -output:"TestResultsOpenCoverCodeCoverage.xml" REM 'Start IIS Express' start "IIS" "%IIS_PATH%iisexpress.exe" /path:"%WORKSPACE%%WEBPROJECT%" /port:%PORT% /systray:false REM 'Run the .NET Acceptance Tests.' "%MSTEST_PATH%MSTest.exe" /nologo /noisolation /testcontainer:%ACCEPTANCETESTPROJECT%bin%BUILDMODE%%ACCEPTANCETESTPROJECT%.dll /resultsfile:TestResultsAcceptanceTestReport.trx REM 'Stop IIS Express' taskkill /IM iisexpress.exe REM 'Convert MSTest report to JUnit report' "C:ToolsMSTestToJUnitmsxsl.exe" TestResultsUnitTestReport.trx C:ToolsMSTestToJUnitmstest-to-junit.xsl -o TestResultsUnitTestsJUnitReport.xml "C:ToolsMSTestToJUnitmsxsl.exe" TestResultsAcceptanceTestReport.trx C:ToolsMSTestToJUnitmstest-to-junit.xsl -o TestResultsAcceptanceTestsJUnitReport.xml REM 'Convert OpenConver report to Cobertura report' "C:ToolsOpenCoverToCoberturaOpenCoverToCoberturaConverter.exe" -input:TestResultsOpenCoverCodeCoverage.xml -output:TestResultsCoberturaCodeCoverage.xml -sources:%WORKSPACE% REM 'Create HTML Report from OpenCover report.' "C:ToolsReportGeneratorReportGenerator.exe" -reports:TestResultsCoberturaCodeCoverage.xml -targetDir:TestResultsHTMLCSReport REM 'Run the JavaScript Unit Tests with Coverage.' "C:ToolsChutzpahchutzpah.console.exe" /silent /coverage /path %UNITTESTPROJECT%WebJSApp /junit TestResultsJasmineJUnitReport.xml REM 'Move the reports to TestResults' move _Chutzpah.coverage.json TestResults move _Chutzpah.coverage.html TestResultsHTMLJSReportChutzpahIndex.html REM 'Convert Chutzpah report to Cobertura report' "C:Tools\ChutzpahToCobertura\ChutzpahToCobertura.exe" TestResults_Chutzpah.coverage.json TestResultsJSCoberturaCodeCoverage.xml Resources http://jane.dallaway.com/jenkins-code-coverage-net/ Writing Automated Acceptance Tests with Spec Flow Gallery Tags Development Share Share on Facebook Share on LinkedIn Share on Twitter Share Share on Facebook Share on LinkedIn Share on Twitter Sign up for our monthly newsletter. Sign up for our monthly newsletter.