AAtsushi's Blog
Python

Trying Out Pytest

→ 日本語版を読む

Overview of Pytest

pytest is a software framework for Python. It discovers written tests by name and runs them automatically. Combined with CI tools like Circle CI, it enables test automation.

Install pytest with pip install pytest.

Suppose there is a Python file called is_odd.py. Write the corresponding tests in test_is_odd.py. pytest runs simply by executing the pytest command. If you name files test_*.py or *_test.py, pytest will find and run them. Test method and function names should follow the test_* format. Test class names should follow the Test* format. You can write clear tests using assert statements. If an error is expected to be raised, you can use pytest.raises() to confirm the error occurs as expected.

is_odd.py

def is_odd(x):
  if x <= 0:
    raise TypeError
  elif type(x) != int:
    raise TypeError
  elif x%2 == 1:
    return True
  else:
    return False

test_is_odd.py

import pytest
from is_odd import is_odd

def test_is_odd():
    with pytest.raises(TypeError):
        is_odd(-1)
    with pytest.raises(TypeError):
        is_odd(0)
    with pytest.raises(TypeError):
        is_odd(0.1)
    assert is_odd(2) == False
    assert is_odd(3) == True

The pytest execution result is as follows. A period . indicates a passing test.

> pytest
============ test session starts ===========
platform win32 -- Python 3.10.7, pytest-7.2.0, pluggy-1.0.0
rootdir: C:\Users\bluen\Documents\Python\pytest_test
plugins: anyio-4.0.0, cov-4.0.0
collected 1 item

test_is_odd.py .                                                                                                                                                                                 [100%]

============ 1 passed in 0.02s =============

If you intentionally change the assert statement to make it fail, the result looks like this. F means the test failed, and it also shows the specific location in the test where the error occurred.

> pytest
================================= test session starts ===============================
platform win32 -- Python 3.10.7, pytest-7.2.0, pluggy-1.0.0
rootdir: C:\Users\bluen\Documents\Python\pytest_test
plugins: anyio-4.0.0, cov-4.0.0
collected 1 item

test_is_odd.py F                                                                                                                                                                                                     [100%]

======================FAILURES ===============================
_____________________ test_is_odd ____________________________

    def test_is_odd():
        with pytest.raises(TypeError):
            is_odd(-1)
        with pytest.raises(TypeError):
            is_odd(0)
        with pytest.raises(TypeError):
            is_odd(0.1)
>       assert is_odd(2) == True
E       assert False == True
E        +  where False = is_odd(2)

test_is_odd.py:12: AssertionError
===================== short test summary info =================
FAILED test_is_odd.py::test_is_odd - assert False == True
===================== 1 failed in 0.15s =======================
PS C:\Users\bluen\Documents\Python\pytest_test>

If you don't need detailed error information, you can add the --tb=no option to turn off tracebacks.

> pytest --tb=no
========================== test session starts ==========================
platform win32 -- Python 3.10.7, pytest-7.2.0, pluggy-1.0.0
rootdir: C:\Users\bluen\Documents\Python\pytest_test
plugins: anyio-4.0.0, cov-4.0.0
collected 1 item

test_is_odd.py F                                                                                                                                                                                                     [100%]

========================== short test summary info ==========================
FAILED test_is_odd.py::test_is_odd - assert False == True
========================== 1 failed in 0.04s ==========================

That covers the very basics of pytest usage. Using fixtures, parameterization, markers, and mocks described later allows you to write tests even more easily.

Test Discovery

When running pytest without specifying a file or directory, Pytest searches the current directory and subdirectories.

It searches for files named test_*.py or *_test.py.

Test method and function names should follow the test_* format.

Test class names should follow the Test* format.

If there are many files, functions, or classes that don't follow the naming rules, you can also change the test discovery settings.

If you want to run only specific tests, you can use <filename>::test_ to run only a specific function within a test file.

pytest test_foo.py::test_func

Test Results

Test results are as follows. The most commonly seen are the period . and F.

  • PASSED(.)
  • FAILED(F)
  • SKIPPED(s)
  • XFAIL(x): Failed as expected
  • XPASS(X): Passed unexpectedly
  • ERROR(E): An error occurred outside the test itself

Comprehensive Testing

Just because you use pytest doesn't mean your tests are comprehensive. Writing tests that consider the following cases is important for thorough testing:

  • Edge cases
  • Error cases
  • Data structure corruption
  • Negative numbers
  • Very large strings

Structuring Tests

Structuring tests makes them easier to understand.

Structure them to follow Given -> When -> Then (given data => behavior test => result).

Grouping Tests into Classes

By grouping test functions into classes as shown below, you can run them together. However, classes should basically be used for the purpose of grouping multiple tests. And it is important to use them sparingly without doing complex things like inheritance.

class TestEvenOdd():
  def test_is_odd(self):
      with pytest.raises(TypeError):
          is_odd(-1)
      with pytest.raises(TypeError):
          is_odd(0)
      with pytest.raises(TypeError):
          is_odd(0.1)
      assert is_odd(2) == False
      assert is_odd(3) == True

  def test_is_even(self):
      with pytest.raises(TypeError):
          is_even(-1)
      with pytest.raises(TypeError):
          is_even(0)
      with pytest.raises(TypeError):
          is_even(0.1)
      assert is_even(2) == True
      assert is_even(3) == False

Fixtures

  • Can separate test pre-processing and post-processing from test functions
  • Used to retrieve data for tests or set up the initial state for testing
  • Declare a fixture with @pytest.fixture()
  • What comes before yield is pre-processing (setup), and what comes after yield is post-processing (teardown). The test runs at the yield point.
@pytest.fixture()
def api_client():
  client = ApiClient()
  yield api_client
  client.close()

def test_api_client(api_client):
  assert api_client.timeout == 240

You can display the setup/teardown order with the --setup-show option.

> pytest test_api_client.py --no-header --setup-show
======== test session starts ==============
collected 1 item

test_api_client.py
        SETUP    F api_client
        test_api_client.py::test_api_client (fixtures used: api_client).
        TEARDOWN F api_client

========== 1 passed in 0.03s ===============

Fixture Scope

By setting a scope for a fixture, you can run the fixture only once per class or module (file), preventing the fixture from running repeatedly.

@pytest.fixture(scope="class")
  • @pytest.fixture(scope='function)
  • @pytest.fixture(scope='class)
  • @pytest.fixture(scope='module)
    • Run once per module (.py file)
  • @pytest.fixture(scope='package)
    • Run once per package (a collection of modules)
  • @pytest.fixture(scope='session)
    • One execution of the pytest command is one session
    • The fixture runs once per session

When sharing a fixture across multiple files, i.e., when the scope is package or larger, you need to create a conftest.py and write the fixture there.

conftest.py

import pytest

@pytest.fixture(scope='package')
def api_client():
  client = ApiClient()
  yield api_client
  client.close()

conftest.py is automatically loaded by Pytest, so no import is needed.

Fixtures can exist in the test module itself or in conftest.py files in directories up to the root of the tests, so it can sometimes be hard to know where they are.

In such cases, you can display fixture locations with pytest --fixtures or pytest --fixtures-per-test.

--fixtures-per-test shows which fixtures each test uses, so it's easier to read than --fixtures.

You can see that the fixture api_client used in test test_api_client exists in conftest.py.

> pytest --fixtures-per-test
======================== test session starts ========================
platform win32 -- Python 3.10.7, pytest-7.2.0, pluggy-1.0.0
rootdir: C:\Users\bluen\Documents\Python\pytest_test
plugins: anyio-4.0.0, cov-4.0.0
collected 2 items

------------------------- fixtures used by test_api_client -------------------------
------------------------- (test_api_client.py:7) -------------------------
api_client -- conftest.py:5
    no docstring available

======================== no tests ran in 0.03s ========================

You can rename a fixture using the name parameter. However, it is preferable not to rename fixtures.

@pytest.fixture(name ="tableau_api_client")
def api_client():

Creating Temporary Directories

Using the built-in fixture tmp_path, a temporary directory is prepared. The directory remains after the test ends, so you can check files (by default, the last 3 test runs are retained).

You can see that a file has been placed in the temporary directory at C:/Users/xxxxx/AppData/Local/Temp/pytest-of-xxxx/pytest-62/test_tmp_file_path0/tmp_test_file.txt.

def test_tmp_file_path(tmp_path):
    file = tmp_path / "tmp_test_file.txt"
    print('File type is', type(file)) # `tmp_path` returns a pathlib.Path object
    print('File location is ', file.as_posix())
    file.write_text('Mr.Bean went to Paris')
    assert file.read_text() == 'Mr.Bean went to Paris'


> pytest -s --no-header
======================== test session starts ========================
collected 3 items

test_api_client.py Hello World!!
.
test_is_odd.py .
test_tmp.py File type is <class 'pathlib.WindowsPath'>
File location is  C:/Users/xxxxx/AppData/Local/Temp/pytest-of-xxxx/pytest-62/test_tmp_file_path0/tmp_test_file.txt
.

======================== 3 passed in 0.04s ========================

A similar built-in fixture is tmp_path_factory. While tmp_path is a function-scoped fixture, tmp_path_factory is session-scoped, so when you need to use the same temporary directory across functions, use tmp_path_factory.

Testing Standard Output

There are cases where you want to test an application's standard output and standard error output. With the built-in fixture capsys, you can easily capture standard output and standard error output.

from api import ApiClient
def test_api_version(capsys):
    api_client = ApiClient()
    api_client.version()
    stdout = capsys.readouterr().out.rstrip()
    stderr = capsys.readouterr().err.rstrip()
    assert stdout == '1.0'
    assert stderr == ''

Changing Attributes with Monkeypatch

Using the built-in fixture monkeypatch, you can easily change attribute values within a test. In the example below, the timeout value is changed from the default within the test.

from api import ApiClient

def test_api_timeout(monkeypatch):
    api_client = ApiClient()
    print('Timeout value is ', api_client.timeout) # Displays the default timeout value "240"
    monkeypatch.setattr(api_client, 'timeout', 100) # Changes the timeout attribute value to "100"
    print('Timeout value is ', api_client.timeout)　# Displays the changed timeout value "100"
    assert api_client.timeout == 100

monkeypatch can also delete attributes, set/delete dictionary entries, set/delete environment variables, and change the working directory using provided methods beyond the setattr() method for attribute setting.

That's it for now. In future posts, I'll cover parameterization, markers, mocks, coverage, and pytest configuration files.

← Back to articles