GUI testing Plastic SCMEach time I’ve the chance to run the GUI test suite we’ve developed for Plastic SCM I find it amazing. The amount of work done by the team is simply overwhelming. It hasn’t been a weekend-long project, that’s pretty clear, but a continuous effort during the last years of development.
This video shows one of the tests checking the code review functionality under Windows XP (it's recorded at HD, so turn to full screen if you want).
Overall, the automated test suite can take almost 24 hours to run (as of release BL187.3). It doesn’t mean it is perfect, but it gives us a lot of advantages. The ones I consider most important are:
As a side note: the test suite doesn’t really take 24 hours to run because we split it in chunks and run it in parallel on different machines, so the real time is just a fraction of that, but if we run it on a single machine it would take very close to one day to finish.
Different testing layers
How do we exactly test? I think I already covered it in previous blog posts but basically we’ve three different testing tiers: unit tests, smoke tests and gui tests as the following picture shows:
The unit tests are regular nunit. We’ve tests for both the client and the server and it is, like the other suites, always growing. It is smaller than it should, though. It covers core functionality at method level.
The gui test suite, the one I’m focusing on today, starts up the GUI client and a server and performs different actions on the GUI. Every test focuses on different functionalities and new ones are added when new features are developed or to cover specific bugs. The suite is thicker than it should: first it takes a long time to run, which is far from perfect, and second it probably overlaps too much with the other suites. We’re working on making it thinner and more focused, in order to make it faster. The great thing is that the basic suite (which every developer runs once a task is finished together with the nunit and smoke, to check the task is ok) is run, during release testing, on all the supported Windows flavors: W2K, W2K3, XP, Vista, W7 and using different backends (SQL Server, MySql and so on), service packs, .NET frameworks and so on. At the end it needs a lot of CPU time to finish (too much is not good) but it is able to give us a very good view of where potential issues are, if any.
Finally the smoke tests are probably the most extensive ones. They’re similar in concept to the GUI ones but they take advantage of PNUnit to automate the Plastic command line. They focus on the CLI instead of the GUI. They’re also thicker than they should.
At the end the testing pyramid should look like this:
But in our case it is a little bit inverted: smoke and GUI are much bigger than unit tests in coverage and required run time.
The automated test suite gives us a lot of security and avoids lots of regressions and we’re constantly working on making it better, so I expect it to look like the former pyramid sooner than later…