My CTO: “We’re really impressed with your testing solution, you’ve gone above and beyond!”
(Immediate feeling of guilt washes over me)
This happened a couple months ago after my CTO heard about a testing solution I put together for our web applications. My solution was to auto-generate the QA Selenium code instead of manually coding it by recording/configuring the tests. (Shameless plug: it’s available for free over at snaptest.io) . This tool has dramatically increased coverage - but why did I still feel guilty about accepting this compliment?
It’s because I unreasonably assume testing should be exhaustive.
Exhaustive testing: refers to running a test on every possible combination of actions/data.
The guilt comes when I say things like: “I’ve tested the new feature”, or “the tests are passing”. In the back of my mind, I know there are hundreds of thousands of combinations I didn’t test for.
We must give ourselves a break and remember that exhaustive testing is usually not possible. One must reconcile the cost/reward balance sheet of testing… Especially when you start thinking about negative tests.
Negative tests/Destructive tests/error path testing — refers to checking how gracefully a system handles errors or unexpected situations. This is when exhaustive testing gets exhausting… Don’t expect to get them all.
Here are my recommendations for handling negative test cases:
- Pick off the easy negative tests like form validation.
- Defer negative tests that will take a lot of work to create and maintain. Such as slow network connections on mobile applications. It may be more economical just to manually test those occasionally.
- Spend time making negative tests on the most commonly experienced errors. Ask yourselves, which errors are users most likely to see? Cover those ones.
- Don’t worry if you don’t cover every state of the app. Just remember, any test is helpful!