Test Code Coverage with .NET Core and TFS

There are at least two ways to collect and publish test code coverage in a Team Foundation Server/Azure DevOps Server build.

  1. Using a Visual Studio Test task.
  2. Using the cross-platform tool Coverlet.

To use solution 1, you replace the .NET Core test task, with a Visual Studio Test task and check the Code coverage enabled check-box (in the Execution options section). That should be it! Unfortunately, this didn’t work for me. I got “Cannot find CodeCoverage.exe” in the log, and didn’t care to troubleshoot this.

That brings us to the second sulution, which ironically uses a Java code coverage file format called Cobertura. This solution requires a few steps:

  1. Add the coverlet.msbuild Nuget package to each of your test projects.
  2. Add a few extra arguments to your .NET Core test task(s). In my case, I had two test tasks, one executing unit tests and one executing integration tests, and I wanted results from these to be merged. Therefore, I used the following arguments in the first task:
--no-restore --no-build --configuration $(BuildConfiguration) --logger trx --verbosity normal /p:CollectCoverage=true /p:CoverletOutput=$(Common.TestResultsDirectory)

With these arguments, a coverage.json file is written to the TestResults folder in the work folder. The following is logged:

2018-11-29T11:27:39.9598640Z Calculating coverage result...
2018-11-29T11:27:39.9906878Z   Generating report 'E:\BuildAgents\DeployAgentTest\_work\4\TestResults\coverage.json'
2018-11-29T11:27:40.1140999Z 
2018-11-29T11:27:40.1270710Z +-----------+--------+--------+--------+
2018-11-29T11:27:40.1270948Z | Module    | Line   | Branch | Method |
2018-11-29T11:27:40.1271345Z +-----------+--------+--------+--------+
2018-11-29T11:27:40.1271595Z | Xxxxx.Web | 37%    | 37%    | 33,1%  |
2018-11-29T11:27:40.1271742Z +-----------+--------+--------+--------+
2018-11-29T11:27:40.1271861Z 
2018-11-29T11:27:40.1271988Z Total Line: 37%
2018-11-29T11:27:40.1272136Z Total Branch: 37%
2018-11-29T11:27:40.1272263Z Total Method: 33,1%

(Fairly poor coverage, I know.)

For the second task, I used the following arguments:

--no-restore --no-build --configuration $(BuildConfiguration) --logger trx --verbosity normal /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Common.TestResultsDirectory)\ /p:MergeWith=$(Common.TestResultsDirectory)\coverage.json

Here, I’m telling Coverlet to merge with the previous result and write the result to a Cobertura file (coverage.cobertura.xml). From the log:

2018-11-29T11:31:16.7685746Z Calculating coverage result...
2018-11-29T11:31:16.8020389Z   Generating report 'E:\BuildAgents\DeployAgentTest\_work\4\TestResults\coverage.cobertura.xml'
2018-11-29T11:31:16.8928999Z 
2018-11-29T11:31:16.9045908Z +-----------+--------+--------+--------+
2018-11-29T11:31:16.9046719Z | Module    | Line   | Branch | Method |
2018-11-29T11:31:16.9047145Z +-----------+--------+--------+--------+
2018-11-29T11:31:16.9047518Z | Xxxxx.Web | 88,1%  | 87,5%  | 83,5%  |
2018-11-29T11:31:16.9047735Z +-----------+--------+--------+--------+
2018-11-29T11:31:16.9047862Z 
2018-11-29T11:31:16.9049202Z Total Line: 88,1%
2018-11-29T11:31:16.9049685Z Total Branch: 87,5%
2018-11-29T11:31:16.9050083Z Total Method: 83,5%

(Better coverage with this test project.)

    1. Add a Publish code coverage results task with the following settings:
      Code coverage tool: Cobertura
      Summary file: $(Common.TestResultsDirectory)\coverage.cobertura.xml

If the build succeeds, you should have a nice Code Coverage section under the Test Results section in your build summary.

 

Advertisements

Entity Framework Update Database without Source Code

If you, as part of your Team Foundation Server (TFS) relase, wish to perform automatic migration of your database using Entity Framework Core, there is a problem. If you try the standard PowerShell command Update-Database or the CLI equivalent dotnet ef database update you will receive the error message No project was found. Change the current working directory or use the –project option. This is because you’re published application consists of binaries only. You got two options:

  1. Package the source code as part of your release (and hope it won’t get installed on the production server).
  2. Use the following hack.

It turns out you can get away with the undocumented dotnet exec command like the following example (assuming the web application is called WebApp):

set rootDir=$(System.DefaultWorkingDirectory)\WebApp\drop\WebApp.Web
set efPath=C:\Program Files\dotnet\sdk\NuGetFallbackFolder\microsoft.entityframeworkcore.tools\2.1.1\tools\netcoreapp2.0\any\ef.dll
dotnet exec --depsfile "%rootDir%\WebApp.deps.json" --additionalprobingpath %USERPROFILE%\.nuget\packages --additionalprobingpath "C:\Program Files\dotnet\sdk\NuGetFallbackFolder" --runtimeconfig "%rootDir%\WebApp.runtimeconfig.json" "%efpath%" database update --verbose --prefix-output --assembly "%rootDir%\AssemblyContainingDbContext.dll" --startup-assembly "%rootDir%\AssemblyContainingStartup.dll" --working-dir "%rootDir%"

Note that the Working Directory (hidden under Advanced) of this run command line task must be set to where the artifacts are (rootDir above).

Testing .NET Core with Team Foundation Server

Testing .NET Core projects as part of a build in Team Foundation Server is relatively easy – just add a Test .NET Core task – but collecting and publishing the results so that they can be viewed in the UI is a bit trickier. Since I’m using XUnit as testing framework, I tried using their dotnet xunit command, but it didn’t work. After some searching and experimenting, I came up with the following solution.

  1. Add the following arguments to the Test .NET Core task:
    –no-restore –no-build –configuration $(BuildConfiguration) –logger trx
    The important part here is the logger parameter.
  2. Add a Publish Test Results task with the following options:
    Test Result Format: VSTest
    Test Result Files: **/*.trx

That’s it!

TeamCity and Octopus Deploy Tips and Tricks

Setting Version

It is nice to have TeamCity set build number. I tend to use major.minor.build.revision for AssemblyVersion and major.minor.revision.build for AssemblyInformationalVersion (product version). So in AssemblyInfo.cs we have for example:

[assembly: AssemblyVersion("2.2.*")] // AssemblyFileVersionAttribute is not supplied, so the AssemblyVersionAttribute is used for the Win32 file version that is displayed on the Version tab of the Windows file properties dialog.
[assembly: AssemblyInformationalVersion("2.2.1")] // A.k.a. product version

In order to have the build number inserted, create two file content replacer build features with the following configurations:

Path pattern: “**/AssemblyInfoGlobal*.cs”
File encoding: <Auto-detect>
Search for: (^\s*\[\s*assembly\s*:\s*((System\s*\.)?\s*Reflection\s*\.)?\s*AssemblyVersion(Attribute)?\s*\(\s*@?\”)(([0-9\*]+\.)+)[0-9\*]+(\”\s*\)\s*\])
Match case: true
Replace with: $1$5\%build.number%.*$7

Path pattern: “**/AssemblyInfoGlobal*.cs”
File encoding: <Auto-detect>
Search for: (^\s*\[\s*assembly\s*:\s*((System\s*\.)?\s*Reflection\s*\.)?\s*AssemblyInformationalVersion(Attribute)?\s*\(\s*@?\”)(([0-9\*]+\.?)+)(\”\s*\)\s*\])
Match case: true
Replace with: $1$5.\%build.number%$7

Build and Publish Multiple Branches

I usually want some flexibility in what branch to build, and I have found that the following settings work quite well.

To make it possible to build and publish multiple branches, you can have wildcards in the VCS Root Branch specification, e.g.:

+:refs/heads/release/*
+:refs/heads/develop
+:refs/heads/feature/*

The default branch in this case would probably be set to

refs/heads/develop

Now, you probably only want automatic build and publish for certain branches, not all. To have automatic build for just develop, go to the build configuration Triggers setting and set e.g. the following branch filter:

+:<default>

Now, you can click the ellipses next to Run, go to the Changes tab and select the desired  branch to build.

Build Pull Requests

You probably also want to have automatic build (but not publish) of pull requests. This is accomplished by having a separate build configuration for that, with the following VCS Root Branch specification:

+:refs/pull/(*/merge)
-:develop
-:master

This means pull requests are built but not the develop and master branches. (For example, pull request #1 have branch name refs/pull/1/merge.

Deploy a Specific Version

The default behavior of the OctopusDeploy: Create release step is to create a release of the latest version. If you want to build and deploy another version, probably from a release branch, you can do like this:

  1. Create a new VCS Root with default branch set to e.g. refs/heads/release/2.2 and use this in your build configuration.
  2. In General Settings, set Build Number Format to e.g. 2.2.1.%build.counter%.
  3. In your Deploy/create release step, set Release number to %build.number%, and Additional command line arguments to –packageversion=%build.number%. This will make octo.exe use this version as default for every package. You can override that with the package parameter, e.g. –packageversion=%build.number% –package=EntityFramework:1.6.2.

Note: It would be better to read the version from AssemblyInfo.cs rather than hard-configuring it, but I haven’t tried that out yet. It would require some scripting.

NuGet Publish

In a TeamCity NuGet Publish step, instead of specifying packages one by one, you can use:

**\obj\octopacked\*.nupkg

Integration Tests

I prefer to run integration tests as part of deployment rather than build, for two reasons:

  • It takes quite some time to run them, and I don’t like really long builds.
  • To a large extent, integration tests test configuration, so it make sense to run them on the target environment rather than on the build server.

Here are the steps I have used to facilitate integration testing as part of deployment. In the integration test project:

  • Add the OctoPack NuGet package.
  • Add app.config transforms. They must be called <project>.IntegrationTests.dll.<environment>.config, build action should be None and copy to output directory should be Copy if newer.
  • Add a PostDeploy.ps1 script. This could look like the example below. Make sure it has the same properties as the ones in previous step.

In TeamCity:

  • Add the integration project to NuGet publish step, or use wildcard as described above.

In Octopus Deploy:

  • Add a new Deploy a NuGet package step and choose the integration test package.

Example PostDeploy.ps1

# Clean-up
Remove-Item Project.IntegrationTests.dll.*.config

# Run integration tests
choco upgrade visualstudio2015testagents -y

$exePath = "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\MSTest.exe"
$testBinariesFolder = "."
$testBinariesFilter = "*.IntegrationTests.dll"
$scheme = $OctopusParameters['scheme']
$hostname = $OctopusParameters['hostname']
$webAppPath = $scheme + "://" + $hostname 
$webAppPhysicalPath = $OctopusParameters['Octopus.Action[Deploy Web].Output.Package.InstallationDirectoryPath']

# Search for integration test DLLs
$testDlls = ""
(Get-ChildItem -Path $testBinariesFolder -Filter $testBinariesFilter).FullName | ForEach-Object { $testDlls += "/testcontainer:""$_"" " }

# Exclude some categories.
$environment = $OctopusParameters['Octopus.Environment.Name']
if ($environment -match "xxx") { $categories = '/category:"CategoryX"' }
if ($environment -match "yyy") { $categories = '/category:"CategoryY"' }
if ($environment -match "zzz") { $categories = '/category:"CategoryZ"' }
if ($categories -eq $null -or $categories -eq "") {
    Write-Error "Unknown environment ""$environment"". Integration tests will not be run." -ErrorAction Continue
    return
}

# Start the test
Write-Output "& ""$exePath"" $testDlls $categories"
Invoke-Expression "& ""$exePath"" $testDlls $categories"

# Check results
if ($LASTEXITCODE -eq 0) {
    Write-Output "All integration tests passed."
} else {
    Write-Error "One or more integration tests failed." -ErrorAction Continue
}

# Upload result file
$testResultFiles = Get-ChildItem -Path .\TestResults -Filter *.trx -ErrorAction SilentlyContinue
if ($testResultFiles -ne $null)
{
	$resultMessage = "Test results available at "
	$testResultFiles | ForEach-Object {
		Move-Item $_.FullName $webAppPhysicalPath
		$resultMessage += "$webAppPath/$($_.Name) "
	}
	Write-Output $resultMessage

	# Allow download of the result
	if ((Get-WebConfiguration -Filter "//staticContent/mimeMap[@fileExtension='.trx']" -PSPath IIS:\) -eq $null)
	{
		Add-WebConfiguration -Filter "//staticContent" -PSPath IIS:\ -Value @{fileExtension=".trx";mimeType="application/x-test"}
	}
} else {
	Write-Error "Found no test results." -ErrorAction Continue
}

# Always return 0 because we don't want to fail the deployment until integration tests are stable.
exit 0

There are several things to note here.

  • To do…