How We Handle Software QA at Blue Label Labs

Many businesses have a process they undergo before the products are shipped off to stores across the country or globe – it shouldn’t come as much of a surprise, but software QA is much the same. When you build a piece of software, whether a mobile app or otherwise, the first goal is to get design elements in place then from there, you ensure that the app functions as intended, meaning it both “looks right” and “behaves correctly” much like when building a physical product.

Because coding software is a complex process, there are often several little elements that programmers and designers don’t catch during an initial review which later surface over time or under specific conditions.

Unlike the QA process that physical products endure, software QA is a multifaceted endeavor that requires more than one set of eyes to properly assess digital products from multiple angles. Every software development company is a little different but the end goal is the same. So here, we’re going to provide an overview of how we approach the software QA process used at Blue Label Labs.

QA isn’t a single function or job at Blue Label: it encompasses several activities done by various team members throughout the life of a project. However, on every development project, there is always at least one person whose sole job is to ensure the quality of the delivered product meets our client’s needs, namely the QA engineer.

At Blue Label Labs, we hire completely independent QA engineers who act separately from our development team. Their main purpose is to ensure functionality is implemented properly, no regressions are introduced, and that the overall state of quality is maintained in the software. 

We feel it’s really important to keep QA teams separate from development teams as well as to incentivize their performance based on their ability to find defects. In general, you don't want software QA being handled by your development teams as I’ve found that developers have an incentive to ship rather than fix. Further, just because you are good at creating software doesn’t make you good at testing it.

Modern purists often say you must automate everything, including QA tests, but really, when you are building MVPs or first versions of a product, automated testing doesn’t do much good. In fact, it introduces a significant engineering cost. Writing automation code increases your engineering burden greatly, meaning every time you change functionality you not only need to change the code of the app but also go change the automation code. 

Based on years of experience, we understand that you should only automate once your app or product has reached a certain level of stability. Until then, we leverage good, old fashioned hard work in manually testing our products.

After developers complete a new, major build of an app or website, what happens next is a series of QA steps designed to balance the needs of getting the build in the hands of users quickly while at the same time making sure that no major bugs slip through the cracks.  The goal is to ensure that all bugs are found, logged, and ideally fixed before the product is published to a marketplace or packaged for distribution. 

The heart of the development process is our ticketing system JIRA – every feature or bug worked on by the development team exists in JIRA as either a Story, Task, or Bug ticket. JIRA serves as the main control point to understand what the development team is working on, what is being released, and where the QA stands based on the tickets within the system.

Our software QA process can be broken down into several directives. The following is the baseline for how we approach this function at Blue Label Labs.

Before our development team transfers a build to our QA teams for verification, developers perform a series of unit tests against their components to ensure basic validation of input and outputs. Generally, we have our developers perform manual unit tests of previously outlined scenarios that are designed to serve as the first line of defense. Once the development team has completed their manual unit testing, they move all the JIRA tickets which were fixed or resolved in this release to the status of “Ready For Testing” then update our internal release note documents, explicitly listing all JIRA tickets that are ready for verification.

Once the developers have released a build into the testing environment (which we refer to as sandbox or staging), our QA team beings their work.

Phase 1: smoke testing. Right after the development team has released a build but before leaving for the night, the QA teams perform a series of end-to-end basic validation tests to ensure the build itself is not fundamentally broken and that the QA team can proceed without further work. If a build is broken and thus, rejected by the QA team, then the developers are in for a long night as they have to resolve whatever issues are found before resubmitting a build.

Phase 2: feature testing and validation. Once a build clears smoke testing, the QA team begins work by running through a full set of manual test cases to test each user story as well as every bug returned in the release. Our QA team will, ahead of time, organize their QA plan in what is called a "Test Plan" in JIRA through the use of a plugin called Zephyr.  As the QA engineer proceeds through their test plan, they mark the associated Zephyr test case complete which is then reflected by the original story ticket in JIRA.

Once our QA teams have reviewed and verified each of the tickets in a release, they move the associated JIRA ticket to the "Verified (By QA)" status in JIRA at which point the tickets are moved back to the PM's plate. For issues that do not pass QA validation, they return to the development team with a status of "QA Fail". 

The PM serves as the last line of defense in terms of quality control before builds get released to clients. A PM will take the build for a spin by performing a series of basic validations based on the intended functionality.

This is a very important step as the PM is ultimately responsible for ensuring that the overall spirit of what was meant to be built has been accomplished. Here, they also identify “finish issues” by finding elements like improper spelling, grammar, and other minor visual discrepancies that might have slipped through the QA net.

Once the tickets have passed PM validation, they are moved to the "Done" state in JIRA, effectively completing the lifecycle of the ticket.

Finally, builds are released to the client, generally either through the iTunes External Beta program for iOS apps or through the Google Play Beta testing channel for Android devices. 

During the UAT portion of software QA, the PM works with the client to identify and log any issues found by the client. Any problems found are plugged into JIRA then triaged alongside other outstanding tickets by the development team. We like to use plugins like Bugsee which makes it very easy for end-users to file bugs from within the app by using a shaking gesture to trigger a new bug report screen.

At Blue Label, we want to minimize the number of issues found in UAT as we’d much rather catch our mistakes than have our clients find them. Each JIRA ticket opened as a result of UAT is logged as such and retained so we can revisit them at a later time. We periodically review this data to ensure we minimize instances of bugs slipping through our QA controls where they’re found by the client.

At the end of the day, our software QA process is what allows us to stay in business. If we were constantly fumbling through redesigns and tweaking faulty code, our clients wouldn’t want to work with us.

Through big-picture analysis of the market and focused testing, our solutions-oriented teams of designers and engineers develop prototypes and ultimately, finished products for today’s digital marketplaces.