Balancing testing speed and release quality is one of the biggest challenges in software engineering. Even if you can keep up with testing the new functionality of a release, the continuous needs of testing the ever-growing regression scope can often seem unattainable.
Regression testing is performed when changes are made to the existing functionality of the software or if there is a bug fix in the software. It is the mechanism by which software is validated to ensure existing product features are still in working order after code changes are applied. As the function of a software application grows with each succeeding release, so do the regression testing needs. Managing these while keeping to shortened delivery timelines can be difficult.
One option is Automated Intelligent Regression Testing (AIRT), which intelligently runs automated regression tests relevant to the changes in code. It reduces the overall regression test cycle by identifying and executing only the set of cases that must be executed to satisfy the test scope, optimally managing risk related to meeting the demands of delivering quality software and the pressures of meeting time-to-market expectations.
While the value of regression testing is widely acknowledged within technology departments, it is often viewed by the rest of the business as costly and time-consuming. Executives want to see the product move forward, and making a considerable time investment in regression testing to ensure existing functionality is working, can be a hard sell.
To address this challenge, more and more companies are adopting regression test automation tools, using software to control the execution of test scripts and comparing the actual outcomes to expected outcomes. This reduces the overall regression test cycle and allows for repeatable, labourious tests to be offloaded from the manual test execution function.
These can be carried out in full, executing the entire regression suite for each release, or partially, blindly selecting and executing a sub-set of the regression suite.
Running a full set of automated regression scripts for each release is costly and often deemed wasteful because it results in testing code and functionality that are seemingly unaffected, and prolongs the testing cycles without showing any added value. However, partial regression cycles carry significant risk because the process of selecting the sub-set of scripts is blind, speculative, and open to interpretation. As a result, it can feel ad-hoc and chaotic. By contrast, the AIRT approach introduces the concept of intelligence into the automated cycle, selecting and executing specific regression scripts based on the code changes that were implemented. Both the value-add and time-to-market objectives are realised because only test cases that provide test coverage for the impacted features are executed, ensuring test coverage is not compromised. Non-impacted features are deemed out of scope, therefore eliminating wasted time and effort.
Another issue is knowing when to start the regression test cycle, whether it’s in parallel with functional testing, in between the first and final functional QA cycles, or only after the functional and system integration tests are signed off. Each option carries a risk. Conducting it in parallel with functional testing can potentially mean running a regression test on a moving target, in between functional testing cycles can also mean testing a moving target – though this could make more sense as it is presumed that the majority of the defects should have already been identified in early test cycles making the release more stable, whilst running the regression cycle after functional and system integration tests are complete runs a risk of defects being found late in the cycle and therefore increasing the time and cost of any fixes.
AIRT allows automated regression testing to be applied in iterations, delivering high-quality software in multiple successive drops whilst conforming to short delivery cycles. It also frees up the quality assurance team to focus on testing new functionality rather than running endless regression cycles.
AIRT is not purely about balancing the demands of delivering quality software and time-to-market constraints, it also contributes to the organisation’s cost reduction objectives by addressing one of the primary reasons for the increased overall cost of quality – the high costs of detecting and correcting defects late in the cycle.
AIRT is integrated with the development code so that each check-in by the developer is automatically verified, and any new defects are detected and fixed early, and within the development phase. AIRT acts as a gatekeeper that helps keep a tab on code quality, delivery timelines, and overall costs.
Organisations typically bound by traditional lengthy regression-test strategies can now apply Brickendon’s bespoke AIRT solution to address both the value-add and time-to-market considerations and reliably deliver small, safe and faster releases.