Build-Deploy-Test. continuous integration

How we arrived to continuity

Our company has launched a new big retail-project on .NET which involves various trendy technologies, SOA. The development started from scratch, and components were gradually integrated. Development and testing of components were carried out on different continents.

Our task was to create the core and service components. In addition, we had to test performance and components installation-removal procedures. Also, the project requirement was to ensure 100% code coverage by unit tests. This is what we got: auto-tests, unit tests, 4 test environments, partially implemented components, failing assemblies and bugs (of course!), and one tester for 14 developers. But we wanted everything to be SELF: -assembling, -installing, -testing, and -removing. And, of course, we wanted to present the most stable and high-quality result to main test team.

We had only 15 members in our team, so testing 4 systems by one person was difficult, to put it mildly. So we decided to try approach taken from Microsoft — Build-Deploy-Test; to help the tester to keep up with the development we used these tools: TFS 2012, MSTest Manager 2012, Visual Studio 2012 Ultimate (switched to 2013 with time).

We decided to build 2 CI servers: Jenkins and TFS server. At the Jenkins all components were self-assembled once per hour in debug mode. After that there was an installation on the server, running of smoke tests (tests to show that the application can be started), and then deinstallation started. Sonar started up all unit and integration tests once a day, during the night. This enabled to get rid of a problem related to inconsistence of assembly and early warn the developers about problems. On the second CI server (TFS build server) the assembly was done in debug mode too. Then there was installation on a test machine and running of functionality autotests. Upon completion of all tests (successfully or not) deinstallation started.

To separate the developers’ tests (unit and integration) from the testers’ functionality tests we decided to build two CI servers. Assemblies on Jenkins were performed much more frequently so that to respond immediately to failing tests, while assemblies on TFS were done by tester only if they were necessary and every day during the night, since the full cycle of functional testing is time-consuming. In addition, to make everything built in the Release configuration and to make the code at least always compilable, we introduced Gated Check-in assembly. With this, when the programmer or tester introduces modifications into the TFS, at first the whole system is assembled in the Release configuration, and modifications pass to the TFS only if the assembly was successful. In case of problems the modifications are not applied, and the assembly in TFS remains operating.

We started automation of functional tests with the selection of their type. Microsoft offers the following:

• Unit tests — written by programmers.
• Coded Ui Test — tests designed to automate functionality tests and user interface testing.
• Web Performance Test — functional testing of web applications under the Load Testing.
• Load Test — tests for load testing of web applications.
• Generic Test — a special test which allows running external test applications.
• Ordered Test — allows arranging the running of written automated tests in a certain order.

As a result, after thinking for a while, we decided that generic and ordered tests would suit us better.

For the generic test the necessary settings and *.exe-file must be specified. Each test-case in TFS was associated with the generic test. For autotests we had one project per each component, and with the start up of the autotest from TM the generic test was called up; it transmits the current autotest’s number to a certain *.exe file, and then the required method is performed.

More details about the Build-Deploy-Test process can be found on Habrahabr.