Fully automated Plastic SCM setup on a Raspberry Pi

Fully automated Plastic SCM setup on a Raspberry Pi

Tuesday, January 22, 2019 Sergio L. 0 Comments

Today, I bring to you a blogpost that is long overdue. If you are a DIY enthusiast like me, you probably have a bunch of Raspberry Pi's lying around. Who knows, maybe Santa (or the Three Wise Men, or your local traditional gifter) brought you one these past holidays. If you want to put one to good use, and you already have your very own CandyVault up and running, maybe this software-only project is a good one to take over the next weekend!

By the end of this post, we will have a Plastic SCM server and client installed on our Raspberry Pi, we will have automatic upgrades, cloud-based repository backups, and the cool thing is, we will get notified on our smartphone every time one of these events occur through IFTTT)!

Ingredients

For this DIY project we will need the following:

• A Raspberry Pi. Even the basic models such as the Zero should work OK, but I recommend the beefiest one available by the time you read this.
• An external hard disk drive. This is optional, depending on how large your repositories are (and the usage you are going to make of them I/O wise); it is better to store them apart from the SD card.
• An account on IFTTT, the web service that lets you automate your digital life.
• An account on an online storage service that has a CLI client; OneDrive, Google Drive, Mega, Dropbox or Amazon S3 all have at least one command line client (official or unofficial) available for GNU/Linux.
• Some free time and patience!

1.- Install the dependencies on the Raspberry Pi

For this tutorial, I'm going to assume that you are using: Raspbian, or any other Debian-based GNU/Linux distribution for you Raspberry. This won't work with other RPi-supported Operating Systems such as Windows for the IoT.

Installing Mono on a Raspberry Pi nowadays is a breeze. I personally recommend you to go directly to the source (the Mono Project), and install Mono by following the official instructions. The versions of Mono that you can find on your distro's repositories are usually outdated.

For completion sake, I'll copy-paste the instructions for Raspbian 9 Stretch here, but you have them available at their downloads page.

$ sudo apt install apt-transport-https dirmngr

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF

$ echo "deb https://download.mono-project.com/repo/debian stable-raspbianstretch main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list

$ sudo apt update

$ sudo apt install mono-complete

We will also need 7za to pack and compress the Jet files. 7za is only distributed in the full package of p7zip, so install it:

$ sudo apt install p7zip-full

2.- Install the Plastic SCM server and client

Unfortunately, we don't have an official repository for Raspbian to install Plastic SCM from the command line using apt. But that doesn't mean it is going to be difficult! We'll just download the zipped binaries, unzip them wherever we want, and ensure to export both locations (client and server) to our $PATH.

You can download the zips for the current Plastic SCM release here:

• Client binaries zip.
• Server binaries zip.

You can copy those zips to a clean Raspbian install using scp or sftp. I trust you'll know how to do that on your own! Once you have the zips on your $HOME, execute the following (change client.zip and server.zip to the actual names of the zips you just copied):

$ unzip client.zip
$ unzip server.zip
$ sudo mkdir -p /opt/plasticscm7/client
$ sudo mkdir /opt/plasticscm7/server
$ sudo chown --recursive pi:pi /opt/plasticscm7
$ cp --recursive client/* /opt/plasticscm7/client
$ cp --recursive server/* /opt/plasticscm7/server
$ cd /opt/plasticscm7/client/scripts
$ chmod u+x genscripts.sh
$ ./genscripts.sh
$ cd /opt/plasticscm7/server/scripts
$ chmod u+x genscripts.sh
$ ./genscripts.sh
$ export PATH=/opt/plasticscm7/client:/opt/plasticscm7/server:$PATH

To make the $PATH change permanent, you should export it on your $HOME/.bashrc file (if you use bash), or your $HOME/.zshrc file (if you use zsh, like me).

Once everything is in place, you can check that the plasticd and cm binaries (well, they are just scripts that end up executing the actual binaries) are in place the following way:

$ which plasticd
$ which cm

Additionally, it would be nice to check that Plastic SCM runs OK on top of Mono. Let's just check the Plastic SCM version we just downloaded and installed:

$ cm version
7.0.16.2858

3.- Configure the Plastic SCM server

When you install the Plastic SCM server using the PKG installer on macOS, the EXE installer on Windows, or the packages in our repositories on other GNU/Linux distros, the server has a minimal setup that allows it to be launched right away. Unfortunately, that's not the case when you use the server zip; right now, we need to generate a client.conf file and get a valid license file.

To generate the client.conf file, you just need to run clconfigureserver and follow the wizard:

$ clconfigureserver

Now, we need to configure the storage for our Jet backend files. By default, the server will look for them at /var/lib/plasticscm, which just doesn't exist yet. You can override this by writing a jet.conf file along the server binaries at /opt/plasticscm7/server or create a symbolic link at /var/lib/plasticscm pointing to the actual directory where you want these files. I chose the latter, becasue I want to store the files in an external HDD, but I don't want to override the default search path for the Jet backend files.

$ sudo ln -s /var/lib/plasticscm /media/hdd/plasticscm

If you configured your server to work in UPWorkingMode (User + Password), you need to create the users first. You can do it with the umtool:

$ umtool createuser sergio mysecretpassword

Finally, we just need to copy our license to server binaries directory. You can download one of your licenses from the plasticscm.com's private area). If you don't have any, you can request a personal or community license through the linked forms.

It is time to launch the server!

$ plasticd --console

4.- Configure the Plastic SCM client

Configuring the client is easier than the server. We also have a CLI wizard to do so, so execute the following and follow the steps:

$ clconfigureclient

The wizard will also autodetect server's authentication mode, so if you set it to UPWorkingMode and added a user using the umtool, enter those credentials.

Now, we can check everything is in place using the license and whoami commands.

$ cm li
(...license information...)

$ cm whoami
sergio

As you can see, the license information matches what I can see in plasticscm.com's private area, and the credentials I set up using umtool are OK.

5.- Create a bunch of applets in IFTTT

Now that we have the basic setup up and running, it's automation time! My goals are:

• Upgrading both the server and the client as soon as a new Plastic SCM version is released.
• Making encrypted, cloud-based backups at least three times a week.
• Getting notified on my smartphone as soon as some of these actions start executing (and know if they ended up OK or failed, so I can act accordingly).

It's easy to figure out how I'm going to address bullet points 1 and 2: bash scripts periodically executed through crontab. But how about the third one? That's where IFTTT comes in handy; it allows you to create workflows of the style "if this happens, then do that". For us, the trigger will be a POST request against their Webhooks endpoint, and the action displays a notification through the IFTTT app (iOS and Android). But you can customize the action in any way you want. Maybe adding a row to a Google Sheets document? Making your Phillips Hue lights blink in a red tone? There are hundreds of possibilities.

I'm not going to cover in detail how to create these applets (if-this-then-that flows), as it is straightforward, and it might have changed by the time you read this. What you need to achieve is creating at least six applets with these names (if you change the name, you will have to modify the code of the scripts I'll provide later):

• plastic_upgrade_start
• plastic_upgrade_success
• plastic_upgrade_fail
• plastic_backup_start
• plastic_backup_success
• plastic_backup_fail

The trigger will be the Webhook, and the action will be sending a rich notification through the IFTTT app.

In the action, we can use the variables we receive through the Webhook trigger. We have up to three variables available (Value1 through 3), so let me show you the texts of my rich notification actions:

• plastic_upgrade_start
  • Title: Plastic SCM Server upgrade started
  • Text: Started upgrading {{Value1}}'s Plastic SCM server to version {{Value2}}

• plastic_upgrade_success
  • Title: Plastic SCM Server upgraded!
  • Text: Plastic SCM server upgraded on {{Value1}} to version {{Value2}}.
• plastic_upgrade_fail
  • Title: Plastic SCM Server upgrade failed
  • Text: Plastic SCM server upgrade failed on {{Value1}}. Reason: {{Value2}}
• plastic_backup_start
  • Title: Plastic SCM's Jet backup started
  • Text: Started Jet backend backup on {{Value1}}.
• plastic_backup_success
  • Title: Plastic SCM's Jet backup complete!
  • Text: Jet backend backup finished on {{Value1}}.
• plastic_backup_fail
  • Title: lastic SCM's Jet backup failed
  • Text: Jet backend backup failed on {{Value1}}. Reason: {{Value2}}

Once you create the applet, you can manually test it with IFTTT. Go here while logged-in, and by clicking in documentation, it will lead you to an online test tool.

We'll be using it through CURL on our bash scripts, so it won't hurt to test this again from your Raspberry Pi command line:

$ curl -X POST -H "Content-Type: application/json" -d '{"value1":"fobos","value2":"7.0.16.2858"}' https://maker.ifttt.com/trigger/plastic_upgrade_success/with/key/ccH-redacted

If everything goes OK, you should receive a response like "Congratulations! You have fired the plastic_upgrade_success event!". And, there should be a notification like this waiting for you on your phone:

6.- Install the command line for your cloud storage of choice

As I said at the beginning of the post, you've got a handful of choices to choose from for your cloud storage, including others that are not listed. The scripts I'm going to post here are prepared for OneDrive; I have an Office 365 subscription, so with 1 TB of online storage, it is perfect for me. But that's me; you can use the one that better suits your needs. I found some CLI clients for the most popular services for you:

• OneDrive: onedrivecmd (unofficial, based on Python).
• Google Drive: gdrive) (unofficial, based on Go, no longer actively maintained. Tread lightly).
• Dropbox: dbxcli (official, based on Go).
• MEGA: megacmd (official, based on C++).
• Amazon S3: AWS CLI (official, and supports every AWS service, not just only S3).

You will find instructions on how to setup and use each one at the repositories / webpages linked above. Install the one you need, configure it, and play with it a little bit to become familiar with it. Then, test if you can upload files from your Raspberry Pi. Once you are done, continue reading!

7.- Install the bash scripts

As Linus Torvalds would say, "talk is cheap, show me the code"! And I've been talking for a while now, so here are the long-promised bash scripts that will do the automation for us:

• SergioLuis/backupdatabases.sh: A bash script to automatically backup Plastic SCM's backend files to OneDrive.
• SergioLuis/updateplastic.sh: A bash script to automatically autodetect and upgrade Plastic SCM server and client to the latest available release.

Once you grab them, you must tweak them to suit your needs. The code is straightforward. First, change the $TEMP, $JETFILES and $REMOTEPATH variables to match the paths on your Raspberry.

• $TEMP is the path to store the temporary files that are generated by the backup process. Mine differs from /tmp because I just don't have enough space on my SD card to use it.
• $JETFILES is where the Plastic SCM's Jet files are stored. By default, it is /var/lib/plasticscm/jet, but you will need to change it if you use a custom jet.conf file overriding the path setting.
• $REMOTEPATH is the path in OneDrive where I want to upload the backup to.

Then, modify the $IFTTTKEY variable to match your IFTTT key to access the Maker API discussed before.

If you chose OneDrive to store your backups, you are ready to go. Otherwise, look for the onedrivecmd call at line 89 and change it to the equivalent in your tool of choice.

Now, you must install the scripts using crontab. You can edit your crontab file executing the following:

$ crontab -e

If you have troubles with the crontab format, I found this useful tool: Crontab Generator. It will help you to generate crontab lines by just filling some fields.

I run the backupdatabases.sh script on Mondays, Wednesdays, and Fridays, at 4 in the morning. The updateplastic.sh script runs every day at 6 in the morning.

Be careful: the scripts use the plasticd and cm scripts, which are NOT exported in the $PATH when cron runs. And, neither is onedrivecmd, which is available at /home/pi/.local/bin if you installed it using python-pip. We need to ensure that they are on the $PATH before invoking the scripts, either making symbolic links at /usr/local/bin, or by exporting the $PATH just before invoking the script.

So, my crontab looks as follows:

0 4 * * 2,4,6 PATH=/home/pi/.local/bin:/opt/plasticscm7/client:$PATH /home/pi/scripts/backupdatabases.sh "mypassword"
0 6 * * * PATH=/home/pi/.local/bin:/opt/plasticscm7/client:/opt/plasticscm7/server:$PATH /home/pi/scripts/updateplastic.sh

Wrapping up

That's all! We now have a Plastic SCM setup on a Raspberry Pi that upgrades automatically, encrypts and uploads our repositories to the cloud, and notifies us on our smartphone about all of that. It is pretty satisfying to have repetitive and boring tasks like this automated.

I'd recommend you verify that the first backup works, and to verify them occasionally. SysAdmin 101 is: a backup that is not verified is as good as no backup at all.

Now you can tweak your NAT and install a service such as no-ip to access your Plastic SCM server wherever you are, but that is out of the scope of this guide. If you want to use ngrok for the same, you would do it as follows:

$ ngrok tcp 8087

I hope everything went OK. If not, don't hesitate to leave a comment here or ping us on Twitter at @plasticscm. And, as always, any additional input is always welcome!

Sergio Luis

After an intense internship at Codice during spring and part of summer 2015, I joined the ranks of Plastic SCM as junior developer.
I already contributed code to the Plastic REST API, the HAL Slack bot that controls our CI system, migrated our internal main server to "new" hardware, coded an Android repo browser and hacked wifi-direct for the upcoming Plastic version.
You can reach me at @_sluis_.