DevOps with TeamCity

DevOps with TeamCity

Monday, May 21, 2018 Pablo Santos DevOps 0 Comments

Previously, I explained how to implement a fully automated branch per task cycle with Bamboo to merge branches when certain conditions were met. Now, I will explain how to implement the same solution with TeamCity.

What I'm going to explain now is how to configure TeamCity to monitor Plastic SCM branches and directly merge them to main when certain conditions are met (when a certain value is assigned to an attribute or even certain status in the associated issue tracker).

In the example, I'm going to keep using Jira as issue tracker, but nothing prevents you from connect to any other issue tracking system.

DevOps vision and initiative

In case you didn't read the previous blogposts on DevOps, it is worthwhile that you know we are in the middle of a big DevOps initiative.

We were always adamant that "branch per task" was the way to go, and we tried to make it shine in Plastic. But, we realized recently, to make it really great, a lot of automation was required.

What I mean is that just monitoring the main branch for changes is no longer enough like it was when it was proposed 20 years ago, back in the SVN days. Today, you need to test every single task-branch before you merge it.

This was pretty natural for us, because we have our own custom CI system, but we realized many Plastic users had to tweak their CIs to achieve the same thing. So, out of the box, everything we loved about task branches was simply out of sight for many teams.

We decided we had to make something.

Improve the Plastic integration CI systems

We develop plugins for the major CI systems, but they are not so much into testing every branch like we would like to. Yes, it is possible to do it nowadays because they have the logic to monitor new branches, but always in a very Git-like mindset where as soon as a branch shows up it gets built and tested. This doesn't make sense for Plastic.

The reason: if you are working on a central server, you can checkin multiple times to the branch before it is finished. Merging each checkin to main wouldn't make any sense.

If you are working distributed, you are not limited to do regular pushes to central to backup your work. Again, getting the branch merged before it is ready wouldn't make sense.

What we released for Bamboo first, and now for TeamCity is the ability to control when a branch is ready to be built and merged.

The common practice is to only build and test and merge the branch when it has a certain attribute set and/or when its associated task (in Jira or whatever) has a given status.

These changes are basically what we implemented in TeamCity, and the rest of the post explains how to use it.

Configure TeamCity

The first thing is to configure the VCS Root (TeamCity jargon) as shown in the Edit Build Configuration:

There are a few things to notice, 2 quite basic and 2 important:

1. Selecting Plastic SCM as the version control provider. Quite obvious.
2. Choosing the right server and repository. You have to type both correctly.



Now come the two important ones:

3. The branches TeamCity will monitor: Use the expression +:(/main/*) to monitor everything under main.
4. The new branch filtering attributes: In the example above, I'm specifying that a branch must have the status attribute set to resolved before it will be merged. The section also includes the attributes to set if the build fails, succeeds and when the merge was done.

5. Go to "Build Features" and configure the "Automatic Merge" which is the key to the entire process:
   1. Set the "Branch filter" to "+:*" so that TeamCity merges all branches that match the filter.
   2. Leave the other fields with the default values (the branches will be merged and checked-in to the default tracked branch /main).
6. Now go to "Triggers" to add a new "VCS Trigger". All you have to do is to set the set "Branch filter" to "+:*".

Your TeamCity should now be able to start monitoring Plastic and building and merging branches accordingly.

What if you need to merge to a different branch instead of main? All you have to do is to change the "branch" property in step 2.

Read how to integrate Plastic SCM with TeamCity CI server

A step by step branch per task example with auto merging

Now I'm going to show you what a development cycle looks like with TeamCity merging branches to main automatically when a certain attribute is set.

First, I create a new branch named DTC-14 to fix task 14 in the associated issue tracker (Jira in my case, although I'll be covering further integration with Jira later on).

Next, I work on the branch and make a few changes to get the task fixed. I'm not adding a screenshot for that, but well, this is just the day-to-day work of any developer :-)

Once you are done, you mark the branch as resolved setting a status attribute to it.

TeamCity is continuously monitoring the repo, so DTC-14 is detected and since it matches the filter conditions, it is selected to be built and merged.

Everything went fine for task branch DTC-14, and the TeamCity status reflects it:

Once the build is ready, the Plastic SCM plugin for TeamCity triggers the merge to main, as I configured above, so the final status in the repo is as follows:

As you can see, DTC-14 is now merged to main, and its status attribute was updated to merged. All this has been performed by TeamCity.

The outcome – a bot working for the team

The main outcome of this DevOps setup is that now instead of having someone in the team performing the merges, or even each developer taking care of it, TeamCity now acts as a mergebot, taking an active role in the development and automating the merge process.

When does TeamCity actually build and merge?

Here is what happens behind the scenes:

TeamCity picks the branch if it matches the conditions, then builds it and runs tests. If everything goes fine, the branch is then merged to main. If the merge succeeds, a new build is triggered to ensure the merged code works.

Warning – possible broken builds: There is a caveat with this cycle due to how TeamCity works, and it is that you can end up with a broken build in main. This is because TeamCity doesn't test the code before checking in the merge (like Bamboo does, and this is something we prefer). While not perfect, it is still workable and when a build is broken a new task can be merged to fix the broken build status.

Advanced custom branch filtering

So far, I described basic filtering. Only Plastic branches with a certain attribute are considered for build and merge. But, what if you need to ensure the associated task in the issue tracker has a specific status too? Or maybe any other more advanced condition?

Well, since we considered this to be a possibility, we introduced a way to implement custom branch filtering by running an external program and implementing a very simple communication protocol.

First, you need to go back to VCS Root configuration in TeamCity and fill in the "enable custom branch filtering" area as follows:

You can specify any program you want and then pass arguments to it in key/value pairs.

The program receives a JSON list of branches to filter from the stdin and returns a list with the actual branches that you want TeamCity to process in the stdout, also in JSON format.

The following is an example of a possible input and output if the external program just wants TeamCity to process DTC-14:

[
   {
      "BranchName":"/main/DTC-15",
      "BranchHeadChangeset":"21651",
      "BranchComments":"Change comment size",
      "BranchCreationDate":"2018-03-19 17:30:00",
      "BranchOwner":"John",
      "Repository":"devops_tc2",
      "Server":"blackmore:7070",
   }


   {
      "BranchName":"/main/DTC-14",
      "BranchHeadChangeset":"29726",
      "BranchComments":"Fix Jira 14 – null in head",
      "BranchCreationDate":"2018-03-20 18:27:28",
      "BranchOwner":"Tom",
      "Repository":"devops_tc2",
      "Server":"blackmore:7070",
   }


   {
      "BranchName":"/main/DTC-16",
      "BranchHeadChangeset":"29999",
      "BranchComments":"Fix broken unit test boo",
      "BranchCreationDate":"2018-03-21 8:19:18",
      "BranchOwner":"Bill",
      "Repository":"devops_tc2",
      "Server":"blackmore:7070",
   }

]

And the output:

[
   {
      "BranchName":"/main/DTC-14",
      "BranchHeadChangeset":"29726",
      "BranchComments":"Fix Jira 14 – null in head",
      "BranchCreationDate":"2018-03-20 18:27:28",
      "BranchOwner":"Tom",
      "Repository":"devops_tc2",
      "Server":"blackmore:7070",
   }
]

This way we basically enable users to create the custom filters they need, connect to their issue trackers, project management tools and the like.

We have published Java code implementing a custom filter here.

A custom filter example

I'm going to modify the example I introduced above, where I worked on the DTC-14 task, to show what can be achieved when implementing the integration with Jira through a custom filter.

This is the simple task workflow configured in Jira:

And, the configuration will filter the tasks to be merged if the branch has an attribute status set to resolved AND if the associated Jira task has a done status.

The cycle of the branch DTC-14 would be the same as I described, but once the task is detected by TeamCity, and after it checked the status attribute is correctly set to resolved, it will invoke the external filter. The filter program will check the status of the associated Jira task.

Since the task is marked as done, the filter will return DTC-14 and TeamCity will build it and merge it.

Note: How can your program know that a given branch is associated to a certain task? Very simply, by using a naming convention. In this case, I use a prefix "DTC-" and the number taken from the Jira issue.

FAQ

What happens when a branch can't be automatically merged?

Simply, the developer will have to "merge down" from main to resolve the conflict, then submit it again to TeamCity by changing the "status" attribute again.

Is there a way to totally avoid broken builds with TeamCity?

Yes, and we are working on it. Our goal is to have an alternative where Plastic does the merge and simply asks the CI system to perform the build of a temporary change before it gets merged. This way, Plastic will ensure builds are never broken following the workflow described here. We call it MergeBot and we are working on it right now at full speed.

Availability

The improved TeamCity support was first released in 7.0.16.2175 back on April 26th 2018.

Pablo Santos

I'm the CTO and Founder at Códice.
I've been leading Plastic SCM since 2005. My passion is helping teams work better through version control.
I had the opportunity to see teams from many different industries at work while I helped them improving their version control practices.
I really enjoy teaching (I've been a University professor for 6+ years) and sharing my experience in talks and articles.
And I love simple code. You can reach me at @psluaces.