CyberFlex - Jira, TeamCity and Plastic integration
CyberFlex Software is an Australian software company that has been developing software for the Heavy Glass, Automotive and Transport industries since 2001.
We've been using Plastic SCM for around 10 years now and have always been happy with the approach Plastic SCM takes to source control and customer service.
Right now, we use Plastic SCM in a fully distributed configuration leveraging the Plastic Cloud offering by using it as our central repository with each developer having their own local repository.
In this blogpost, I want to show you how this configuration works well for our team who work across multiple offices and allows for flexible working from home if required.
The Problem
Upon doubling the size of our small team of developers in early 2018 we quickly noticed that something would have to change with our current process.
Our original process required that a single merge master would monitor the status of Jira tasks. It involved merging into a /main/integration branch and having the merge master decide when to create a new label as a baseline for new branches. Once a baseline had been properly tested, the merge master would merge the branch into /main, ready to perform a release.
It became apparent that the following issues existed with this approach:
- Builds were being run for every checkin. In some cases, this would cause a large number of queued builds on our build server.
- Merges were being performed manually and the merge master would have to set aside time to perform them.
- As more and more branches were merged without creating a new baseline label, the higher the chances were that there would be merge conflicts, taking up the precious time of the merge master who also fills a developer role.
- Tasks that were dependency blocked by other tasks that had been completed but were waiting to be merged were delayed.
- The process of creating a child branch from the latest baseline label instead of the head of the branch was not intuitive for new developers.
- It did not support our view that tasks should be able to be developed, reviewed and tested independently.
- Continuous Delivery would be difficult with this approach.
The Solution
The solution was to remove road blocks and bottle necks where possible, remove the role of the merge master and share code reviews among more team members. To achieve this, we integrated our source control, bug tracking and continuous integration/build server to automate as much as possible.
Step 1 - Jira to TeamCity
The first step in solving the issue was to get Jira integrated with TeamCity. The goal was to automate the triggering of builds in TeamCity based on status changes on Jira tasks.
I should mention at this point that we've also get Plastic integrated with Jira using Plastic's excellent issue tracker integration. This ensures that developers create a branch per task with the branch name as an exact match to the task key in Jira (e.g PROJ-1234).
To achieve the TeamCity integration, we enlisted the help of the Automation for Jira plugin. The plugin allows us to set up an outgoing web-hook whenever it detects that a task is moved to a particular status. We've created a Ready for Merge status in our Jira configuration so all that's required is that the task is transitioned to this status and a build will automatically be triggered in TeamCity by the outgoing web-hook performing a HTTP POST to the TeamCity REST API. Since the automation has access to information about the task, we pass a number of build parameters over to TeamCity including the task key and the name of the branch to build which is always in the format of /main/{JiraTaskKey}.
Step 2 - TeamCity and Plastic
Plastic already provides a great plugin for TeamCity which is used to add plastic repositories as VCS Roots inside of TeamCity. The plugin will allow you to configure automatic merging but we had a number of other requirements that wouldn't be addressed by the functionality of the plugin alone.
Build Speed
The speed of our builds is a major concern for us. We have a large number of integration tests that can take a while to complete as well as needing to compile our Angular front-end which can take a notoriously long time. Couple this with the fact that we're pulling code from the Cloud which increases the time it takes to download a repository and our builds were taking far too long to complete.
To combat this problem, we've used several advanced features of TeamCity:
- The Swabra build feature - If you're like us and like to have a clean checkout directory for each build to ensure artifacts from previous builds dot not interfere, then this feature may help. It monitors the directory and file structure of the checkout directory after the source code has been downloaded from the repository and at the end of the build, cleans up any changes that have been made to the directory structure. In most cases this allows us to keep the source code from the previous build, allowing Plastic SCM to only download missing/changed files instead of the entire source code. When using this feature, the checkout process can, in some cases, take only 1/10th of the time it would take to download the entire source code.
- Parallel Builds - TeamCity allows you to chain several builds together. They call it 'Build Chaining'. If configured correctly, this can allow you to run multiple builds on different agents simultaneously. This has a dramatic impact on build times since they are no longer sequential. In our case, we run our DotNet and Angular builds in parallel saving up to 8 minutes on our build times.
- Single Source Code Checkout - As I mentioned in the last point, we utilise build chaining in TeamCity to take advantage of parallel builds. We also use the chaining feature to ensure we only need to download the source code once. The first step of all our builds is to download a copy of the source code and create a zip file with the source code contents. Since all of our build agents are on the same network, we've disabled compression when creating the zip file which speeds up the performance. The zip file is then transferred to the subsequent builds as an artifact dependencies in TeamCity. Several zip files are created, mainly there is an angular-src.zip and a dotnet-src.zip. The Angular and DotNet builds then each extract the relevant zip file and continue with the compile and test steps of their build configurations.
An overview of our build chain showing the first step of obtaining the source code, the Angular and DotNet builds running in parallel and all build artifacts being amalgamated at a final stage.
Automated Merging
The most important part to get right about our TeamCity/Plastic SCM integration was getting automated merges to work. This would solve the problem of needing a merge master and would shift the responsibility of fixing conflicts automatically back to the developer. It would also mean less merge conflicts since merges happens more frequently.
To achieve automated merges, we had to ensure that the code being compiled and tested was not the code from the branch, but the source code as it would appear after it has been merged to /main. We needed to obtain a copy of this merged code without actually checking in the merge in case there were conflicts or build errors. Only after a successful build should the branch be merged into /main.
Using the Single Source Code Checkout method, I mentioned above is also a big part of obtaining the correct source code. In this step we use Powershell scripts to interface with the command line version of Plastic SCM and perform a merge. If conflicts are detected here, the build fails and the developer is notified with a Slack notification to prompt them to resolve the issue. Providing there were no conflicts with the merge, the source is zipped and provided to the subsequent builds in the chain.
The last step in the chain (Branch Build) will perform the merge. If we've reached this step in the chain the build was a success, it compiled correctly and all tests have passed. Using more Powershell scripts interacting with cm.exe we can now leverage the recently introduced 'Merge-To' feature of Plastic SCM to merge from the branch into /main. Using the Merge-To feature saves us from having to swap to the /main branch.
And we're done. There is a new changeset on /main and after a short time the developer can start a new task which may otherwise have been blocked waiting for the lazy merge master to merge in this task.
0 comentarios: