Unit testing, if done the right way, will become your best buddy. The time you invest on writing tests will come handy in the most unexpected ways, once the product moves to a maintenance level. Unit tests will stay by your side as your pet doggy and bark at the silly mistakes you make during a feature addition, a refactoring or even a bug fix. Unit tests together with a static code analyser can prevent a majority of easy to make mistakes.
Unit testing is your best buddy
Pytest is undoubtedly the most popular and stable testing solution for python
This article will share my experience on pytest, specially about unit testing and the stuff that I learned the hard way; or rather, about some stuff that most online articles or forums don’t talk about.
Anatomy of a test
Before diving into pytest, let me cover the basics of a unit test.
As a practice, I mentally divide every unit test I write into 3 or 4 sections!!
As a summary, first there will be few statements for setting up the test. Then there will be some statements for executing the actual methods followed by few other statements for doing assertions. Occasionally as a fourth step, we may also need to wrap up the test. Let’s go through these sections in a bit more detail.
The first section is important to “make the test a unit-test”. It is vital to mock all the side effects of the function or method that is being tested. Otherwise, the test you write may pass at the first time, but not all the time. For instance, if the method performs an actual database operation, the test may pass when a specific value is present in the database, and will fail otherwise. This concept is similar to pure functions in functional programming, which return the same output to the given inputs no matter how many times you call the function. In fact, a unit test function can be thought of as a pure function itself. Thus the first part of a test method can be used to perform all the ground work for running a test properly.The python mock library, which is now a part of the python standard library, will help us in this aspect when writing unit tests for a python project.
However in general, multiple test cases in a module require the same initialisation steps. To avoid rework and inconsistencies in such circumstances, we can use the setup & teardown methods to instruct the test runner to run the desired step before and after each test respectively. In some languages like Javascript, setup & teardown functions are known as before and after hooks.
The second part of the test can be used to call the actual method and perform any other required operations. Although this is not the place to write assertions, there are some instances where a test may yield the result here itself. For example, if you’ve ever worked with ReactJs, it is enough in some cases just to render a component. The test will not pass if the vital attributes of the component are not written correctly. This theory will apply for other non-react scenarios as well. However, it is always better to write some more assertions afterwards to validate the behaviour of the tested method or function. As per terminology, a test which does not do any assertions, but only checks for the execution without unexpected halts, is known as a smoke test.
The third part, which is the most important part of a test, is for validating the behaviour of a function or a method. The assertions should not be too strict nor too flexible. Too much flexibility can hide the mistakes you might do during a code refactoring. Assertions being too strict can cause the tests to over-fit and fail even if the changes don’t cause issues. This will result in having to update the tests whenever you do even a minor code change, making the tests brittle, unstable and a nightmare to maintain. Deciding the ideal amount of assertions is an art which should be expertised with time. When it comes to python, assertions can be done with respect to the return values as well as through mocks if any.
On a separate note, it is quite easy to abuse this part to show increased test coverage. This is why the concept of mutation testing came into action. I will explain this concept in another article.
The final and fourth part of a test is for wrapping up the test execution. For instance, if you mocked some method of a module at the start of a test, you might need to restore the original functionality. Otherwise, it can yield unexpected behaviour in preceding tests that use the same method. As mentioned above, this is a place where a teardown function can come in handy. However you might not notice this pattern when using mocks in python. That is because, when you use the patch decorator or context manager, the mocks will be automatically restored when the test ends. Thanks to this pattern, you rarely will need to use the setup and teardown methods in pytest.
Pytest Basics
Although Pytest is a great tool, I found their documentation quite descriptive yet hard to read. Therefore here, I will summarise the most important concepts from the original docs.
Installation & Running
You can install using pip install -U pytest and then simply run pytest from the root directory to run all tests.
Test Directory & File Names
Although there are many possibilities, the best place to define tests is, alongside the source file, in a directory called “test”. AKA “Tests as part of application code”
The test files can be named either as <source_name>_test.py or test_<source_name>.py But please make sure you stick to one of these patterns. Here is an example from the original docs.
setup.pymypkg/__init__.pyapp.pyview.pytest/__init__.pytest_app.pytest_view.py...
Integrating with Setuptools
Most python projects are based on setuptools. If your python project is also using setuptools, you should integrate pytest with the setup.py.
The pytest documentation clearly explains how to integrate pytest with setuptools using pytest-runner. Yet it does not explain why you should do that.
When I had that question, I had no idea why I should do the said integration, when I could simply execute pytest from the terminal. So I did the obvious; I skipped the integration.
Later on, at a time when I had just forgotten about this integration, I faced a problem. The coverage report generated using pytest-cov is completely wrong!! The source files show a coverage of 0% and all the test files are included in the report. I did not see that coming eh.
After putting too much effort, I learnt that, unless you integrate pytest with the setup.py file, the tests will not run against the source as expected. Apart from that, there are few other reasons why you should integrate pytest with the setup.py.
- You can specify dependencies in the
tests_requireoption insetup.py. This will ensure all required helper packages are installed before actually running pytest. - You can alias pytest in the setup configuration file. For example, you can define your setup.cfg as follows and by doing so, pytest will run evertime before you create a source distribution using the
sdistcommand.
**[aliases]**test=pytest
clean_all=clean --allrotate --match=.tar.gz --keep=0
build=clean_alltestinstallsdist
**[tool:pytest]**addopts =--cov-config .coveragerc--cov-report html--cov=my_pkg--pylint--pylint-rcfile=.pylintrc--pylint-error-types=EF
Also note, although not directly related to the setuptools script, how easy it is to define options to pytest using the setup.cfg file.
Mocks
As explained earlier mocking plays an essential role when writing unit tests in any language. The easiest way to mock methods in python is to use the patch decorator provided by the mock library. The main advantage of using patch is the automatic mock restore behaviour as explained above. On the other hand, a test written by making use of the patch decorator looks cleaner and easier to understand compared to the direct use of Mock or MagicMock. However, there can be rare cases where you need more flexibility in the mock objects. Just be cautious to reset the mock objects if you happen to use the core class directly.
Every mock object created by the library have methods to do the required assertions directly. You can check whether a particular method was called and with what arguments just by calling these methods.
You can directly specify a return value to a mock method. But you can also specify a side_effect method, which will be called instead of the original method. This concept is useful when you want to return some value based on the arguments passed when calling the method.
Special Return Values
When writing tests to methods that use external libraries, the return values may be special objects. For example, the return value of an actual call to the request library is a special Response object.
In such scenarios, I made my way out by defining fake classes. As in above example, I configured the return value of calls to the request library by defining a fake Response class. Given below is a very basic example class like that.
Thus now I can define the return value of a class as follows,
mock_req.return_value = Response(status_code=200, data="abc123")
Although there are many useful features provided by the pytest and mock libraries, the basic features will be enough for most cases. Nevertheless, the time you invest to read the documentation of a library will never go in vain. You will surely discover features which can do the same stuff that you already do in a better way with less code. Thus I strongly recommend you to at least glance through the pytest and mock documentation once you have grasped a some understanding about the basics.
If you found this story interesting, encourage me to write more by giving some claps to this story. 👏👏👏 And I would love to hear your feedback through comments.