Time to turn the Test Pyramid on its head?

Most people involved in testing will be familiar with the Test Pyramid depicted below.

The Test Pyramid was first described by Mike Cohen in 2009 in his book ‘Succeeding with Agile’. The arguments he makes for the shape of the triangle were certainly logical and relevant at the time, especially for large business applications built in that era built from the ground up by large teams of software developers.

Over a decade later, a number of things have changed that perhaps should cause us to re-think the pyramid – perhaps even turned it on its head!

The shape of the pyramid was derived from the desirable quantity of automated tests that should be created for each type of test. More automated Unit Tests should be created than Integration Tests, and very few automated UI tests should be created.

The reasons given for why lots of Automated Unit Tests should be created are as follows:

• Quick and easy to create
• Quick to execute
• Cheap to maintain
• Deterministic – i.e. the results are reliable
• Defects caught at the earliest time, thereby reducing the cost of the defect

The reasons given for why only a few Automated UI tests should be created were as follows:

• Requires specialist test automation knowledge like Selenium developer skills
• Slow to execute – long delays between test steps are needed in case the app renders slowly. Automated UI tests are executed sequentially and often on a users browser.
• Costly to maintain i.e. needs the same Selenium developers to maintain the tests.
• Non-deterministic – i.e. the results are un-reliable. Automated tests are brittle. They have a tendency to fail because of changes to UI of the system under test.
• Defects are trapped late in the development process, thereby making them costly to fix.

Firstly, most organisations do not build business applications from the ground up. Most organisations use at least one Platform as a Service solution like ServiceNow, Salesforce.com etc.

PaaS solutions no longer require thousands of lines of code, instead they are:

• most functions are provided out of the box and configured
• workflows are assembled using pre-defined UI components
• pre-defined UI components are integrated together
• security rules are configured
• business rules and process workflows are configured with very minimal code written

Some code is written sometimes, but this is normally kept to a minimum to avoid customising the out of the box configuration too much.

In this world, where the focus is on configuration rather than development, unit testing developed code units is not as relevant. The ‘units’ are provided by the platform vendor and therefore expected to work.

What is of paramount importance when testing these types of systems, is testing of user interface and the end-to-end business processes i.e. the business workflow.

Logic follows that the majority of tests should therefore be of the UI and the testing of the business processes. If the business processes work, then how this is done in the code at the unit level is of little or no relevance.

In the PaaS world, these ‘units’ are provided by the vendor and have been thoroughly tested, if not thoroughly by the vendor pre-release, then very thoroughly by virtue of a very large user community using them everyday in live operation.

So is it time the Test Pyramid was turned on its head for the testing of PaaS solutions like ServiceNow?

It would seem so.

But wait. What about the list of UI automation issues that meant it was not desirable to do very many UI automated tests?

Well, technology has moved on since 2009 and there are now automated testing platforms, that remove all of these UI test limitations. One such platform is AutomatePro designed to overcome these traditional automated testing obstacles.

• Does not require specialist test automation knowledge like Selenium developer skills. Automated tests can be built by non-technical and even business users.
• Execute quickly with intelligent interaction with the PaaS UI. No fixed waits or delays.
• Scalable test automation mean that thousands of tests can be run concurrently on VM server architecture without needing to manage the infrastructure.
• Tests are built using re-usable model blocks, so maintenance effort is vastly reduced and does not require Selenium developer skills.
• Results are deterministic. AutomatePro guarantees that if the test fails it is because of an issue in the process rather than because of a change to the UI of the system under test.
• UI process based tests can be run by developers, so Defects can be trapped early on in the development process.

In addition, platforms like AutomatePro provide additional benefits of building many UI based process tests. For example, test results can be viewed by UAT testers, dramatically reducing the amount of time and resource needed for UAT. User guides and process documentation can automatically be created from the UI test results, reducing the overall cost of the effort to create the UI tests.