You’ve probably heard someone wondering if they really need a manual tester on a project, especially if they’re looking to reduce costs or streamline processes. In this article, we’ll share our experience working without manual QAs so you can see if it’s really worth it.
This article is based on the experience of Exadel managers who worked on projects without manual QAs. We’ll discuss how the team self-managed these assignments and found a balance between costs and QA resources.
Introduction
Each project owner faces the same key questions when it comes to quality assurance (QA):
- How can I reduce project costs?
- Do I need a separate manual software tester?
- What are the consequences of bringing a manual tester from the Scrum Team to the development team?
- Why do I need automated QA in addition to/instead of manual, and how will automation reduce project costs?
It’s critical that you view your QA processes holistically so you can see how manual and automated QA work together - automation to reduce costs and manual to respond quickly to changes in product features. We will also consider how the agile model in manual software testing works.
Why is QA Necessary?
The key purpose of QA specialists is to check whether your new increment meets business requirements and to measure its quality.
As a project manager or product owner, you’ll know that quality both directly and indirectly affects project costs. Every new bug discovered before the release will push the release back, and a delayed release is delayed profit. A critical bug can stop development in its tracks, and developers will continue to be paid by the hour, even when they can’t work until the bug is fixed.
Management needs to pay attention to bugs at every stage, even in the production environment. We’d like to believe that product quality will be high enough by the release of the production environment and that each and every bug will have been resolved by then. But if test coverage was insufficient, bugs can appear during the trial, and some of them can even block the use of the product. This can lead to unpredictable losses, both financial and reputational.
A budget owner (a manager, product owner, or stakeholder) has to understand that testing also carries a cost, and in some situations, those costs outweigh the financial impact of bugs. For example, does it make sense to test deliverables from different sides for a simple website, where a system failure won’t have a major impact on the owner? I’d argue that complex testing isn’t justified in such a case, especially if the business use of the site is simple.
Exadel’s Case: How We Succeed Without a Manual Tester
Sometimes a customer says that a manual tester is no longer required. The customer owns the budget and can decide how fast they need a project implemented, so we have to decide what to do next with this new direction. Should we save money and reduce the team? Skip manual testing entirely? The answer is to have developers and automation QAs (AQAs) take over manual testing.
AQAs and developers can work together in this situation. AQAs understand test cases in greater depth, so they’re responsible for writing them. A developer takes test cases and manually checks the product increment after development but before transferring the task to testing. Next, the AQA writes auto-tests and manually tests again. Although AQAs will have more work on their hands, it’s still feasible.
We did this on a challenging project! We’d finished developing the story, but we hadn’t done testing because of issues with capacity, pitfalls, and estimation. So when tickets came raining down on our AQA on the last day of an already difficult Sprint, they needed to ask for help, and they did. The whole Scrum Team is responsible for the result of each increment, so we were all ready to help. Plus, since the increment had made it to testing, our developer had free time. This “ask for help” approach works both ways; if the developer had too many tickets to cover, we would have all jumped in there, too.
This shows the way that a Scrum team can come together to cover tasks, even if they’re slightly outside of each group’s normal duties.
How Does Your Team Work?
According to David Guest’s article,
- A T-shaped colleague is an expert in one area, but understands many
- An I-shaped colleague is an expert and understands only one area
- A multi-shaped colleague understands many fields but is an expert in none
T-shaped team members are the most valuable assets, as they do a great job in their area and can support other team activities. We try to cultivate T-shaped people on our teams, and having developers and AQAs help with manual testing is a great way to do this. When a developer has enough experience and the need arises, they can take over manual testing - saving the customers money without compromising quality.
Automation vs Manual Software Testing
We’ve already discussed how developers can help with manual testing, but let’s take an in-depth look at how manual and automation testers relate to one another.
There are frequent disputes about what testing methodology to apply. Is it enough to just do manual testing? Is it possible to only do automation testing? You should consider this question from a variety of angles. While automation QA can function without manual testing, a manual tester can cover more cases, since they know about how similar products behave. So while we’ve made it work without manual QA, there are trade-offs.
To help you determine if those trade-offs are worth it, you should first get an estimate for automating testing. Calculate the costs and the payback period. It makes no sense to invest in automating a typical two-month project except where very high quality is needed. However, if we are talking about a two-year project, with each iteration, more and more time will be spent on pre-release build checks. Investing in automation, in this case, is vital, otherwise, multi-week regressions involving the entire team will block the development of the project.
What are the Benefits of Manual Testing?
A manual tester can flexibly and quickly respond to changes and achieve your desired testing quality. We all know that an agile product is very dynamic and subject to change, even though the core rarely changes. A manual tester can move features to production more quickly.
At the same time, it’s ideal for the core to be covered by autotests. By the core, we mean a platform or some kind of permanent product functionality that is not subject to significant or frequent change. This will help save on regression. When a product core passes all of the autotests, we know that the critical/main functionality of the product works as expected.
What Testing Practices are Used at Exadel?
In order to decide what types of testers to hire, you also need to consider which types of tests you need. We’ll focus on two major areas here: regression and unit/integration tests.
Regression testing
Regression testing is conducted on previously tested products to ensure that the functionality remains unchanged. The underlying purpose of the regression test is to find and correct regression errors. Regression errors are the same bugs, but they do not appear when writing the program. Rather, they’re incorporated when adding a new part of the program to an existing build or when fixing other bugs, causing new defects in a previously tested product.
Another purpose of regression testing is to make sure that some bugs have been corrected and that the upgrade of the build has not created new defects in the code already tested.
Regression is a typical problem for large projects. Sometimes developers make changes in one service, and as a result, the functionality in the other services doesn’t work as expected. This is where a QA comes in. A developer can take anywhere from two to three times as long as a QA to carry out a regression, which is a major vote in favor of a QA. Over time, developers won’t be able to cope with regression on a large project, and they should be focusing on developing new features anyway.
Unit & Integration Tests
Unit tests and integration tests are integral parts of the code. Absolutely all engineers on all projects are required to accompany features with automatic tests. Continuous Integration is a system in which automated tests are run whenever the code changes, and we highly recommend this approach.
We also provide a number of Agile-certified leaders that can run Agile delivery services across a range of disciplines. Our Agile delivery experts specialize in software development, test engineering, and Agile DevOps, as well as Scrum and program management.
How to Know that You’ve Maintained Product Quality
Metrics are the most popular tool for understanding product quality.
The following QA metrics in software testing are the most significant values that can be used to infer other derivative metrics:
- Total number of test cases
- Passed test cases
- Failed test cases
- Blocked test cases
- Identified bugs
- Accepted bugs
- Rejected bugs
- Deferred bugs
- Critical bugs
- Determined test hours
- Actual test hours
- Bugs detected after release
Metrics will help you see progress and determine if your project quality is increasing or decreasing.
How can you know that you’ve maintained quality without a manual tester? Is one autotest per feature enough to say that feature is covered? If not, then what is a threshold? Manual QAs create test cases, and test cases define a level (and depth) of testing. If you get rid of this approach, then you should define how to keep and maintain at least the same level (and depth) of testing.
For automation tests, we rely on requirements. We use acceptance criteria of each user story to create test cases and, based on test cases, AQA creates automation tests to cover each of them. In simple words, an automation engineer is creating a script to automate manual work but the test coverage should be the same as with manual testing.
Conclusion
Based on my experience, a manual tester is important for small, simple, and/or typical projects because their work allows other team members to focus on what they are best at. if you do something complex, then you have to use agile and automation and maybe even remove manual testers in the interest of investing more resources into these other areas
While those are ideal circumstances, as you can see from this article, it’s possible to either do simple or complex projects without a manual tester. If you need to reduce costs, going without a manual tester is an option, as long as you closely monitor your metrics. In the event that the metrics are deteriorating, it makes sense to think about returning the manual tester to the team.
Want to understand how your company can become more Agile? Let's get in touch!