Why Automating High Level Software Testing?

At wega we have a lot of activities around software validation in order to get formal proof of the proper functioning of applications used in regulated pharmaceutical environments.

One of these activities is manually executing test scripts and thereby following precisely defined test steps in the system to be validated. These scripts are pages of instructions that have to be executed very precisely, step by step.

• When some of us did this for the first time, there were some mixed feelings and thoughts:
• Whoa, this is really taking all my concentration.
• Oh no, I just skipped step 9!
• I'm bored. How can that be? It's challenging, needs full concentration, and it's exciting to discover this application after all. But still, this task is really tiring. I feel like a robot...

And that was the magic word: “Robot”! Let's automate!

Thinking about what can be automated and what not, and where machines are better than humans revealed a new and fascinating way of testing to us. We also began to understand the difference between testing and checking as described for example on tutorialspoint.

Testing in contrast to checking involves exploring the system to evaluate if it behaves as expected; if the users feel comfortable with it and are supported in their tasks by the system, if potential human errors in handling the system are acceptable, if the quality of the system is as expected, if the look and feel is appealing, and ultimately to provide a holistic feedback on the system under test. Testing is a creative and exploratory process that requires human judgment, experience, and skill. This cannot be automated.

On the other hand, checking is verifying that a system or component meets a set of predefined requirements or specifications; in our case, high level requirements. Scripts are used to compare the actual results of a test with the expected results to determine whether the system under test is performing correctly. Checking is a rather structured and procedural process that can be executed using a test automation tool.

But even though a precise step-by-step description in a test script can be automated according to this definition – does it really make sense?  

As often, the answer is: It depends – and that is true, it depends on the goals you want to achieve by using test automation.

So far, we've experienced three different goals that could be achieved with test automation:

1. Reducing testing effort and costs while increasing reliability when you have a use case where the same test script needs to be executed repeatedly, for example when performing regression tests.
2. Reducing the test execution time is also valuable. Let’s say you have a new software feature that was just released and needs to be tested quickly, having test automation scripts available to be executed allows you to greatly reduce the test execution time compared to manual testing.
3. Increasing reliability and user satisfaction by implementing system monitoring. Life Science digitalization is leading to the creation of applications’ ecosystems that need to communicate with each other in more or less real time. But when dealing with applications sold as SaaS, it becomes more difficult to manage releases and their impact on the entire ecosystem. If we include the fact that there may still be IT activities, such as patching and maintenance, taking place outside office hours, this adds up to a lot of factors that are out of our control, potentially bringing crucial day-to-day business processes to a complete standstill. In such a case, the added value of test automation seems enormous to us: there is certainly a greater initial investment to develop the test scripts that will enable you to cover all the crucial processes, but once this is in place, there's nothing to stop you from scheduling the run of all the tests every weekend, or even every night! You can program your tool to send you a notification in case of defect detection, enabling you to act upstream; maybe you can solve the issue before the users would even realize there was a problem, or in the worst-case scenario you can at least warn them that the problem is known and being dealt with, avoiding at the same time a lot of support requests.

To sum up, we'd like to highlight a few of the experiences we made:

• Already while writing automated test scripts, you can test the system very precisely. The script needs to be defined in its last detail, and you cannot describe it in a ‘more or less’ manner as it could be done in scripts for manual execution.
• Errors in the test script are immediately detected when you write automated test scripts: as the test execution is very fast, you can execute it repeatedly to check the quality of your script. In contrast, errors in the test script for manual execution are only checked during the dry runs performed by a tester clicking through each step and verifying that it is written in an understandable and unambiguous way for humans.
• And last but not least: Writing automated tests is fun! Unlike manually executing test scripts, we can now observe the mouse moving and clicking in the application under test all on its own. After spending hours doing this manually, this is a very special satisfaction.