Who we are

We are the developers of Plastic SCM, a full version control stack (not a Git variant). We work on the strongest branching and merging you can find, and a core that doesn't cringe with huge binaries and repos. We also develop the GUIs, mergetools and everything needed to give you the full version control stack.

If you want to give it a try, download it from here.

We also code SemanticMerge, and the gmaster Git client.

Plastic SCM DevOps: Custom plugs

Wednesday, October 03, 2018 Miguel 0 Comments

Our DevOps ecosystem was designed with versatility in mind. All functionality is provided by separate pieces of software, each one with a specific purpose: plugs to perform actions, mergebots to orchestrate the process. In this guide we'll be looking at the plugs and their general structure so you can implement your own!

What is a plug?

We define a plug as an independent program that connects to a Plastic SCM Server through WebSockets, where it registers itself and provides a generic interface to perform actions in systems external to Plastic SCM. Those include Issue Tracking systems, Continuous Integration systems and Messaging systems.

Real life examples of plug actions range from setting an issue as Done in your issue tracker or triggering a build in a CI server to sending an email to the QA team when a task is ready to be tested.

Besides the executable itself, each plug has to provide a plug type definition (typically yourplug.definition.conf) and a plug configuration template (typically yourplug.config.template). The Plastic SCM Server can then combine that information to enable administrators to create their own plug configurations. Each one of them runs the plug app in a separate process with the specific set of parameters that the administrator defined.

You can find a list of the plugs and mergebots we ship with a regular Plastic SCM server installation in our DevOps repository in GitHub.

Plug types

Note: We'll use ${DEVOPS_DIR} from now on as a reference to %PROGRAMFILES%\PlasticSCM5\server\devops in Windows or /var/lib/plasticscm/devops in macOS or Linux.

The plug type definition is a simple JSON document like this one:

{
  "name": "homegrown",
  "type": "ci",
  "displayName": "Our CI system",
  "description": "This plug provides connectivity with our in-house CI system.",
  "command": "/path/to/my/homegrownplug.exe --my-non-generic-param",
  "template": "/path/to/my/homegrownplug.config.template"
}

It contains the plug description in a human-readable way, the plug executable file path and the plug template path.

This is how it would look like in WebAdmin:

Let's see what these parameters mean.

  • name is the internal identifier of the plug inside the Plastic SCM Devops system. Make sure it's unique!
  • type specifies what kind of interface the plug implements. It can be either issuetracker, ci or notifier. You can see actions are provided by each interface here.
  • displayName is the readable name for your plug to be displayed in the Plug Types page in the WebAdmin.
  • description provides additional information about your plugin to administrators browsing the available plug types list.
  • command defines the program to start and any additional command line arguments that it requires. All plug programs must support a particular set of parameters (more on that here), but any additional one required by your custom plugin can be saved here.
  • template sets the path where the plug configuration template can be found.

If you don't want to manually create the plug type file yourself, you can also add new custom plug types using the WebAdmin, at http://your.plasticscm.server.com:7178/devops/plugs/types/custom/new (replace the host with your actual instance hostname). Otherwise, make sure you store your brand new plug type file inside ${DEVOPS_DIR}/config/plugs/available. We name these file like ${type}-${name}.definition.conf to keep it organized, but any other name will do. We would have named the type definition file of homegrown as ci-homegrown.definition.conf.

When it's ready, it will be displayed in the available plug types list:

Plug configuration templates

Your custom plug might not need any extra configuration because it has everything hardcoded. But chances are that as a developer you want the Admin to configure certain params like the URL of your custom issue tracker.

If that's the case, then it's very useful to let the Admin enter those params using a UI instead of having to type a config file.

That's exactly what the configuration templates are for: a way to define the params your plug needs so that WebAdmin knows how to render a UI to let the Admin enter the right config.

The WebAdmin will use the configuration template to render a UI, but also to generate a JSON configuration file that will be sent to your custom plug on start up so it can process it accordingly.

The template itself is a JSON document whose root is an array of parameter objects. Take a look at this example:

[[
  {
    "name": "url",
    "type": "string",
    "displayName": "Base URL",
    "description": "Root path of the CI system"
  },
  {
    "name": "user",
    "type": "email",
    "displayName": "User",
    "description": "User name to connect to the CI system."
  },
  {
    "name": "password",
    "type": "password",
    "displayName": "Password",
    "description": "Password to connect to the CI system."
  },
  {
    "name": "max-concurrent-builds",
    "type": "int",
    "displayName": "Max concurrent builds",
    "description": "Set the maximum number of builds that should be running at a given time."
  },
  {
    "name": "use-privileged-access",
    "type": "bool",
    "displayName": "Use privileged access",
    "description": "Select the type of access to the CI system"
  }
]

Each parameter has these attributes available:

  • name is the key of the current parameter. It should be unique to prevent collisions.
  • type can be one of these:
    • bool for yes/no values
    • int for numeric values
    • email for text in email format
    • string for text
    • password for sensitive information
  • displayName contains the readable name of the parameter.
  • description adds detailed information about the parameter to be displayed in the configuration form.

The WebAdmin will create an actual configuration file (JSON) based on the template you defined plus the actual values entered by the administrator. Check the following section to see the result config file.

We recommend to include these config template files in the directory where the plug executable is, using the format ${name}plug.config.template. In our example, that would be /path/to/my/homegrownplug.config.template.

Creating plug configurations

The Plastic SCM Server allows administrators to configure individual instances of a given plug, based on the plug type definitions we discussed above. They are JSON files with fields that set concrete values of the generic type properties. They can be created through the WebAdmin (http://your.plasticscm.server.com:7178/devops/plugs/new) or manually, placing them in ${DEVOPS_DIR}/config/plugs/start.

Here's an example of a plug configuration based on the template introduced above:

{
  "name": "Homegrown-Central",
  "plug": "homegrown",
  "apikey": "CC75A440E4FEDCF6A71D1FC5F84E0A36AEE2C80DEB7476D7422F6508D416EC6B",
  "start": "yes",
  "type": "IssueTracker",
  "configuration":{
    "url": "https://ci.server.net/homegrown",
    "user": "admin@mydomain.com",
    "password": "1234",
    "max-concurrent-builds": 5,
    "use-privileged-access": "no"
  }
}

Where:

  • name is the key of the configuration. Avoid duplicates in your plug configurations!
  • plug defines the plug type on which this configuration is based.
  • apikey sets the autentication key for the Plastic SCM Server API to be used. See more info here.
  • start is a toggle value ("yes", "no", true or false) indicating whether this plug configuration should run on server startup.
  • type refers to the plug type category: IssueTracker, ContinuousIntegration or Notifier.
  • configuration contains the configuration values in an object as specified by the plug type configuration template.

Before a plug configuration is run, the Plastic SCM Server extracts the value of the configuration property and writes it to a temporary file. Its path is then passed as a command line argument.

This is the temporary file generated by the Plastic SCM Server from the plug configuration above:

{
  "url": "https://ci.server.net/homegrown",
  "user": "admin@mydomain.com",
  "password": "1234",
  "max-concurrent-builds": 5,
  "use-privileged-access": "no"
}

And this is how it would look like in WebAdmin:

API key

API keys are automatically generated when you create a new plug configuration through the WebAdmin. They're just random strings of hexadecimal numbers that are used in the server to authenticate valid incoming plug registration requests. They're stored at ${DEVOPS_DIR}/config/apikeys/conn.

You can generate your own, bypassing the server. Their file names should be ${key}.key and their JSON contents are like this:

{
  "key": "CC75A440E4FEDCF6A71D1FC5F84E0A36AEE2C80DEB7476D7422F6508D416EC6B"
}

Plug implementation

Now it's time to tackle the actual plug implementation! The most important thing to have in mind is that plugs can be implemented in any programming language, so choose your favorite one! Just make sure you have an appropriate library to provide you websocket connectivity.

Command line arguments

Every plug is expected to admit four arguments. These contain information stored in the server when a plug configuration needs to start. This is a sample execution of a plug configuration, started from the Plastic SCM Server:

/path/to/my/homegrownplug.exe \
    --my-non-generic-param
    --server wss://plasticscm.server.net:7111/plug
    --config homegrown-central.conf
    --apikey CC75A440E4FEDCF6A71D1FC5F84E0A36AEE2C80DEB7476D7422F6508D416EC6C
    --name homegrown-central
  • --server is the websockets endpoint in the Plastic SCM Server. The plug must connect to it and register it self later on.
  • --config sets the path where the configuration file should be found. It contains the plug configuration in the format defined by the plug template.
  • --apikey is the string used to authenticate the requests sent by the plug in the registration stage.
  • --name is the name of the configuration as set by the administrator that created it.

As we mentioned before, if your plug needs some other parameters beyond these, you can set those in the command property of the plug type JSON file. In this case, we used --my-non-generic-param to illustrate that.

It's also advisable to allow a -h or --help parameter in case you'd like to manually launch it and you don't have the time to read this guide.

Plug logging

If your custom plug is generating log information (and it should!), you can output it in output their log files to ${DEVOPS_DIR}/logs/{plug-type-name}.{plug-configuration-name}.log.

The Plastic SCM Server will also store the PID of the running plug at ${DEVOPS_DIR}/run/${plug-configuration-name}.pid

Registering the plugin

The first thing your custom plug should do is check that the received command line arguments and the configuration found at the specified file (passed with --config) are all valid.

Then, move on to the websocket connection. Use the URI passed as a command line argument and connect to it. Plastic SCM Server expects a Login message followed by a RegisterPlug message when a new websocket connection is established. The message contents are JSON objects in both cases.

The login message includes the API key to authenticate:

{
  "action": "login",
  "key": "CC75A440E4FEDCF6A71D1FC5F84E0A36AEE2C80DEB7476D7422F6508D416EC6B"
}

The RegisterPlug message lets the Plastic SCM server know the name and type of the plug, enabling it to receive requests:

{
  "action": "register",
  "type": "ciPlug",
  "name": "homegrown-central"
}
The type property shouldn't be configuration dependent, and it's related to the type property in the plug type definition. It can be one of issueTrackerPlug, ciPlug or notifierPlug. The name property contains the value of the --name command line argument.

At this point, your plug is successfully registered in the Plastic SCM Server and it should be standing by for requests!

Reconnection

You might consider adding some internal logic to your plug so it can automatically reconnect to the server if the connection is broken at some point!

Handling requests

All incoming messages from the server to a plug will arrive through the websocket connection and they'll contain a JSON payload like this one:

{
  // global properties for *all* requests
  "requestId": "924b0a7d-6b76-4aef-b4b0-d8253d018c38",
  "action": "getfieldvalue",
  // operation parameters, different for each request type
  "projectKey": "PRJ",
  "taskNumber": "3670",
  "fieldName": "status"
}

Websocket requests are asynchronous in nature, so you can take your time to process the request. When you've performed the required actions and you have a response to send back to the server, just send back an object including the expected return data and the requestId value that triggered that response:

{
  // mandatory field to identify the response in the Plastic SCM Server
  "requestId": "924b0a7d-6b76-4aef-b4b0-d8253d018c38",
  // response parameters, different for each response type
  "value": "To Do"
}

There are some actions that don't return anything. Even in that case you need to send the response object with the appropriate requestId back to the server to let it know that the operation finished.

If something went wrong, include the error description in the response. This applies to all messages!

{
  "requestId": "924b0a7d-6b76-4aef-b4b0-d8253d018c38",
  "error": "Unable to contact with the remote server."
}

Your plug will have to support the messages associated with its interface. Let's break those down in the following categories!

Issue tracker requests

Issue tracker plugs should know how to generate the URLs of valid issues, retrieve the values of certain issue fields and be able to alter them.

GetIssueUrl

Build a valid URL to open the specified issue in a web browser.

Request

{
  "requestId": "70a7d65a-2cdc-4504-9b85-0e8a76dcf3f3",
  "action": "getIssueUrl",
  "projectKey": "PRJ",
  "taskNumber": "3670"
}

Response

{
  "requestId": "70a7d65a-2cdc-4504-9b85-0e8a76dcf3f3",
  "value": "https://my.secured.issue.tracker.net/projects/PRJ/issues/PRJ-3670"
}
GetIssueFieldValue

Retrieve the value of the specified field for a given issue.

Request

{
  "requestId": "977e09ee-3495-47cc-8145-442491828c96",
  "action": "getFieldValue",
  "projectKey": "PRJ",
  "taskNumber": "3670",
  "fieldName": "status"
}

Response

{
  "requestId": "977e09ee-3495-47cc-8145-442491828c96",
  "value": "inprogress"
}
SetIssueFieldValue

For a given issue, set the value of the specified field to the supplied value.

Request

{
  "requestId": "977e09ee-3495-47cc-8145-442491828c96",
  "action": "setFieldValue",
  "projectKey": "PRJ",
  "taskNumber": "3670",
  "fieldName": "status",
  "newValue": "done"
}

Response

{
  "requestId": "977e09ee-3495-47cc-8145-442491828c96"
}

Continuous integration requests

Continuous integration plugs should be able to start new builds in the configured plans and also retrieve their status on demand.

LaunchPlan

Start a build in the specified plan. It should return the build ID of the started build.

Request

{
  "requestId": "70d5a7bd-dca0-488b-8749-a167e30cee16",
  "action": "launchPlan",
  "planName": "trunk-plan",
  "objectSpec": "sh:3520@codename-kamchatka@plasticscm.server.net:8087",
  "comment": "Merge task branch /trunk/task3670 (cs:19383) to /trunk",
  "properties":{
    "pipeline": false,
    "releaseName": "Okhotsk",
    "buildNumber": 3151
  }
}

Response

{
  "requestId": "70d5a7bd-dca0-488b-8749-a167e30cee16",
  "value": "577"
}
GetStatus

Retrieve the status of the specified build.

Request

{
  "requestId": "c528c28d-859d-4eb3-ab04-f08937dc8b2b",
  "action": "getStatus",
  "planName": "trunk-plan",
  "executionId": "577"
}

Response

{
  "requestId": "c528c28d-859d-4eb3-ab04-f08937dc8b2b",
  "isFinished": true,
  "succeeded": false,
  "explanation": "3/2009 Tests failed. Build aborted."
}

Notifier requests

Notifier plugs are just a layer of abstraction to send messages from a MergeBot to affected Plastic SCM users or groups.

SendMessage

This action should send a message to a list of recipients. It doesn't include the action property because this is the only action that a Notifier plug can accept at the moment.

Request

{
  "requestId": "62d05633-3d18-4036-8a7a-3c9ff7a9e519",
  "message": "Task PRJ-3670 failed. Reason: 3/2009 Tests failed.",
  "recipients":[
    "development@our.company.com",
    "devops@our.company.com",
    "joan.jett@our.company.com"
  ]
}

Response

{
  "requestId": "62d05633-3d18-4036-8a7a-3c9ff7a9e519"
}

Wrapping up

We've gone through all the required steps to build a custom plug. They unlock endless possibilities for devops engineers and enable them to tune their processes according to their needs. We'll publish the code of our built-in plugs, take them as references to develop your own!

Besides that, remember that you can contact us anytime in our forum or at support@codicesoftware.com. Don't get stuck if you bump into any issues or you're unsure about how to proceed at some point! Let us help!

And remember: release early, release often!

Miguel González
Prior to become a Plastic hard-core developer, I worked in a consulting firm in France where I also finished his Computing Engineering master's degree.
I'm a Linux enthusiast (I was the one developing the Plastic SCM linux packages), heavy-metal guitar player on a band, LP collector, youtube expert and talented Plastic hacker.
You can find me at @TheRealMig_El.

Miguel

0 comentarios:

Subscribe to: Post Comments (Atom)