Ensuring “dotnet test” TRX & Coverage files end up in SonarQube

Do feel free to provide any comments/feedback to @TheRichCarey on Twitter

I have written before about using SonarQube to do static analysis, but one issue I never came back to was ensuring that code coverage files generated via a build pipeline end up being picked up by the Sonar Scanner to assess code coverage.

Note that the following I am actually using the ‘dotnet test’ build step, rather than the ‘Vs Test’ one. Do let me know if you find a nice work around for the VS Test variant, as I couldn’t get it to drop coverage files!

The issue

The issue is that:

  • When using VSTest, TRX files are deleted automatically if using version 2+ of the VS Test task as per this stack overflow post.
  • When I switched back to ‘dotnet test’ the same thing appeared to be happening.
  • .coverage files are not output by default
  • TRX and Coverage files are placed in a temporary folder of the build agent rather than the executing agents working directory.
  • Even though SonarQube could detect the tests, it would still register as 0.0% code coverage!

Getting ‘dotnet test’ to collect coverage

The first step was to get the ‘dotnet test’ build step to collect the code coverage, and not just dump TRX files.

To do this, go to the “Arguments” field of the dotnet test build step and append --collect "Code Coverage", as well as ensuring that “Publish test results and code coverage” is enabled.

Ensure generated files are copied to the working directory

As the coverage files will end up in the /tmp folder of the build agent, SonarQube will not be able to scan them.

We will need to add a new build step of “copy files” with the correct filter set to get the .trxand .coverage files from the default temporary directory on the build agent, to the test results folder of the workspace. To do this we need to add the “Copy Files” task into the build and place it after the “VS Test” task. The source folder for the copy will be $(Agent.HomeDirectory)\_work\_temp and the target folder will be $(Common.TestResultsDirectory) – The contents can remain as ** but feel free to filter if required. Example below.

If we run a build now, we should now see files in the TestResults folder of the build agent’s working directory.

I didn’t have to make any changes to the configuration within SonarQube as it should just pick up the coverage files. If I follow the above I get the following (Lets just ignore the fact the number is low ๐Ÿ˜‰)

CSPROJ Changes to test projects

One thing I did notice in the console when attempting to fix this code coverage issue was that I got a lot of warnings like:

SonarQube.Integration.targets: warning : The project does not have a valid ProjectGuid. Analysis results for this project will not be uploaded to SonarQube. 

As all my projects were .net core or .net standard the CSPROJ files do not contain a <ProjectGuid> tag by default. As also suggested in this stack overflow answer , I added a GUID to my test project file. I am not 100% if this is required, but it stopped warnings appearing in my console and does no harm.

Bonus

If you have multiple builds to update and you are using Azure Devops, you can take advantage of “Task Groups”. This allows you to create a single build step which in turn executes a series of other build steps. Using the steps above, you can create a new Task Group to create a single build step to run the test script and make sure the files are copied to the correct location for analysis. For example I have the single build step below:

Which means I can then just call this single build step in all my builds

Debugging ES6 Mocha unit tests using VS Code

The world of Mocha, VS Code and Node is still fairly new to me. Typically in the past all my JS unit tests have been debuggable in-browser using DevTools, but with Mocha this is not the case (As I am not deploying my spec files). I got Mocha to load via a launch config, but it would not originally work due to using ES6 directly.

If you do not have a launch.json, start here. Otherwise skip to the next section. Add a new Debug Configuration by selecting “Debug”, and then “Add Configuration”. Selecting “Node.js” automatically creates a “launch.json” under a root folder named .vscode. If you already had debug set up, this step would be irrelevant.

Add Mocha configuration to launch.json

In the launch.json, much like the surprisingly helpful comments suggest, you can simply type “Mocha” then [ctrl]+[space] to bring up the intellisense for a Mocha configuration!

Which will insert the appropriate snippet.

Now, in theory it is as simple as clicking the play icon in debug, with “Mocha Tests” selected.

Supporting ES6.

For me however, this didn’t work.

The issue here is that I get a lot of unexpected token errors as my tests are using ES6 and I suspect that by default it wants to use ES5. The issue of using ES6 for unit tests was resolved in another post .

Much like my previous post, I can update the launch arguments to use require to pull in the same 2 Babel modules, and will also specify a wild card file name of my tests so it doesn’t pick up any other code.

 {
            "type": "node",
            "request": "launch",
            "name": "Mocha Tests",
            "program": "${workspaceFolder}/node_modules/mocha/bin/_mocha",
            "args": [
                "./test/**/*.spec.js",
                "--require", "@babel/polyfill",
                "--require", "@babel/register",
                "-u",
                "tdd",
                "--timeout",
                "999999",
                "--colors",           
            ],
            "internalConsoleOptions": "openOnSessionStart"
        }

Now for me, this also didn’t work as I am using Chai for my BDD test syntax.

For this I had to change “tdd” to “bdd” under the args.

Now I can attach and debug, providing a breakpoint is set!

Attempting to use Mocha & Chai to unit test ES6.

In this post I will cover using Mocha (JS test framework) and Chai (For BDD syntax) to unit test ES6 Javascript in VS Code.

I started working on a small side project, for no reason other than to play with ES6+. It’s a(nother) relatively simple toast library written in as much vanilla JS as possible to avoid reliance on libraries & packages.

I got the code working, but I couldn’t prove that the functions worked. I used qUnit in the past to test JavaScript but if I am completely honest my JavaScript testing knowledge is a bit lacking.

My aim is to get some unit tests for one of my main classes where I can test directly against ES6 and not against the compiled ES5 code. I want the tests to be clear to what they are doing. What I am doing is not new at all, nor is the library! I just wanted to keep notes of how I achieved this first time around.

Disclaimer: This is by no means a comprehensive guide or walkthrough, just the results of me messing about to see if I can get the outcome I wanted whilst learning something new!

Enter, Mocha

I decided to useMocha to do my unit testing, which was chosen purely as it seemed to work well with ES6 code (using Babel). Later I will go into how I also used Chai along side to provide much nicer, fluid assertions using BDD-style syntax.

First of all, I had to install Mocha.

> npm install --save-dev mocha

Then under a new root folder of “test” I created a bread.spec.js – where “bread” here is the name of the class I am testing.

At this point it is fairly easy to create a simple test, like so.

import {Bread} from "../src/bread";
var assert = require('assert');
describe('Fluent methods', function() {
  describe('Title set is not called', function() {
    it('should set the title correctly (null)', function() {
        let options = [ ... code to get options ... ]     
        let b = new Bread(0,"Foo", options);       
      assert.equal(b.Title, null);
    });
  });
});

I then added the appropriate script to package.json to allow us to run the tests.

 "test": "mocha --require @babel/polyfill --require @babel/register './test/**/*.spec.js'"

Which is ran with:

npm run-script test
VS code window with output of script above. Shows a single completed unit test.
Output of running above command.

This script states that it will run Mocha, on all files under the test directory where the JS file ends with “.spec.js”. I then had to add the 2 requires which enable Mocha to call the ES6 directly and not have to use the transpiled version. Failing to do provide these requires will mean Mocha will not run as it cannot parse ES6.

Using Chai for BDD syntax

In the above, I import my class then create a “test set”. In this test set I then have a single test which is checking if the title gets automatically set. It’s fairly easy to attain what the test does, but it could be clearer. This is where I decided to use Chai. Chai will allow me to have a BDD-style test written which is closer to plain english. Mocha does support some of this (at time of writing) but Chai is much closer to BDD-style syntax I was used to.

To use Chai I need to install the package:

npm install --save-dev chai

Then import the “expect” module from the framework, and refactor the method so it looks a little like this:

import { expect } from "chai";
import {Bread} from "../src/bread";
describe("Fluent methods", () => {
    describe("Title set is not called", () => {
        it("should set the title correctly (null).", () => {
            var options = getValidOptions();            
            let b = new Bread(0,"Foo", options);
            expect(b.Title).to.equal(null);
        });
    });  
)};

Running the tests will yield the same result as before, but now its a lot more readable (In my opinion!)

Not a lot more to add really. Mocha and Chai both have great documentation to read through. The only difficulty I had was getting Mocha to run ES6 directly, as a lot of the information online for this was out of date (that I found…)

Update: I have also posted about debugging using ES6 Mocha tests here