Troubleshooting missing tests in Atlassian Bamboo

Our team at work uses Bamboo for our continuous integration tasks for our sites. We have an AngularJS 1.5 app that I’m primarily responsible for, and, as such, have been working to build out our suite of unit tests for the application. My local development setup includes a terminal window that runs these tests every time I save a JS file anywhere in the application.

I have the Karma config set up to use the ‘jUnit’ reporter to output a jUnit-compatible test report after the tests complete. The Bamboo build job is configured to read the resulting XML file and include the results of the test run in the build job result. That’s been working smoothly for several months.

Recently, however, I noticed a discrepancy between the number of tests reported being run in my terminal window and the number of tests that Bamboo reported in the build job result. The number Bamboo reported was consistently 6 fewer than my terminal reported. Project deadlines being what they are, I didn’t pay this too much mind for a few weeks after I noticed it, but last week I just couldn’t ignore it any more so I began digging.

In importing the jUnit XML file, Bamboo lists gives you the ability to list out all the tests that it runs in a very easy to read format. Our individual test specs aren’t that large so it was a fairly easy task to look at my userSpec.js file and count the number of tests and compare that against the Bamboo report. It didn’t take long to find the specs that had more tests in it than Bamboo was reporting.

In ‘BDD-style’ unit tests, your tests look something like this:

describe( 'new', function(){
	it('should return a generic user object when an invalid type is requested', function() {
		var user ='someBogusUserType', {});

What I found was that I had some “copy/paste” issues. I’d copied an ‘it()’ function block and changed out the innards, but had failed to change the descriptor text (the ‘should return…’ text above). This didn’t matter to the ‘progress’ reporter that is also configured in the Karma config as it just listed the raw number of tests run and any failures.

However, evidently that descriptor text is the only unique thing that Bamboo has available to determine the number of tests run. The second instance of the same descriptor text was replacing the previous instance from the jUnit report. Once I’d made sure that the descriptor text for every ‘it()’ function was unique, Bamboo started reporting the same numbers of tests run/passed as my local terminal window reported while developing.

Feel free to insert your own anecdote about the evils of copy/paste here…