Use Fixtures Effectively in PyTest

Fixtures are a great feature of PyTest but using them can be tricky. I found them to be very useful when all important environment logic resides in them and tests use a consistent interface without regard to how the environment is actually set up.

Initially when my team started using fixtures we had tests using a couple at most. We had a bunch of class instances for various connections, for example SSH, DB, etc. As the environment under test evolved it was hard to keep up with changes. We put in a lot of effort refactoring tests.

An example of how we were initially using fixtures:

tests/
    conftest.py
        <Empty>

    test_1.py
        import pytest
        from our_module import SSH, DB

        @pytest.fixture(scope="module")
        def fixture1():
            return fix1

        @pytest.fixture(scope="module")
        def fixture2():
            return fix2

        @pytest.fixture(scope="module")
        def setup_module(fixture1, fixture2):
            fixture1.setup()
            fixture2.setup()

        @pytest.fixture(scope="module")
        def teardown_module(fixture1, fixture2):
            fixture1.teardown()
            fixture2.teardown()

        def test_1(fixture1, fixture2):
            ssh = SSH(creds)
            ssh.command("hostname")
            db = DB(creds)
            db.query("")

We later realized that we needed to abstract away connections into fixtures. The primary reasons included:

• Not having to carry login creds in the tests
• Not needing to create and teardown connections for each test

The first round of changes included creating connection fixtures once and re-using them in tests:

tests/
    conftest.py
        import pytest
        from our_module import SSH, DB

        @pytest.fixture(scope="session")
        def ssh():
            return SSH(creds)

        @pytest.fixture(scope="session")
        def db():
            return DB(creds)

    test_1.py
        import pytest

        @pytest.fixture(scope="module")
        def fixture1():
            return fix1

        @pytest.fixture(scope="module")
        def fixture2():
            return fix2

        @pytest.fixture(scope="module")
        def setup_module(fixture1, fixture2):
            fixture1.setup()
            fixture2.setup()

        @pytest.fixture(scope="module")
        def teardown_module(fixture1, fixture2):
            fixture1.teardown()
            fixture2.teardown()

        def test_1(fixture1, fixture2, ssh, db):
            ssh.command("hostname")
            db.query("")

The SSH and DB objects had logic in them for reconnecting as needed. This way these fixtures could be re-used in various tests during the test run session.

Another problem we had was that our environments under test were created with the same major components but deployed in various ways. For example, the same packages were deployed directly on the same VM or in Docker containers on one VM. This meant that we had to support two kinds of connections to the same components and discover what environment was under test.

For example, SSH port of the VM was 22 but each Docker container had its own SSH port mapped from the host. Sure this is not how Docker should be used; each container must run one service only and docker exec must be used instead of running an SSH server in each container. Our problem domain unfortunately did not fit into Docker best practices. Therefore, we were stuck with treating containers as mini-VMs running atop a VM.

Within our fixtures we needed to support both scenarios. DB could be connected through the IP:port pair where both IP and port were of the host VM and DB. Or the pair could be IP of host VM and port would be the Docker mapped port. Fixtures allowed us to abstract these differences away from the test.

Now we had a new problem. Say we had 10 containers on a VM host to test. Each container was running a service, say a DB or a REST API or something else. Each container was also running an SSH server. Therefore, for each container we needed a class for using the service and an SSH connection. Passing 20 fixtures to each test was getting very ugly.

The beauty of fixtures in PyTest is that they can use other fixtures. This is where we got around the ugliness of passing in large numbers of fixtures.

Essentially, we had two VMs running multiple containers each. Or we could have multiple VMs running the same things as the two VMs. We grouped our "mega" fixtures into two basic groups: group1 and group2. These would be passed to tests:

tests/
    conftest.py
        from collection import namedtuple
        import pytest
        from our_module import SSH, DB
        from rest_module import REST1, REST2

        @pytest.fixture(scope="session")
        def rest1():
            r = namedtuple("r", "rest ssh")
            return r(REST1(), SSH(creds))

        @pytest.fixture(scope="session")
        def db1():
            return DB(creds)

        @pytest.fixture(scope="session")
        def rest2():
            r = namedtuple("r", "rest ssh")
            return r(REST2(), SSH(creds))

        @pytest.fixture(scope="module")
        def fixture1():
            return fix1

        @pytest.fixture(scope="module")
        def fixture2():
            return fix2

        @pytest.fixture(scope="session")
        def group1(rest1, db, fixture1):
            g = namedtuple("g", "rest1 db1 fixture1")
            return g(rest1, db, fixture1)

        @pytest.fixture(scope="session")
        def group2(rest2, db, fixture2):
            g = namedtuple("g", "rest2 db2 fixture2")
            return g(rest2, db, fixture2)

    test_1.py
        import pytest

        @pytest.fixture(scope="module")
        def setup_module(group1, group2):
            group1.fixture1.setup()
            group2.fixture2.setup()

        @pytest.fixture(scope="module")
        def teardown_module(group1, group2):
            group1.fixture1.teardown()
            group2.fixture2.teardown()

        def test_1(group1, group2):
            group1.rest1.ssh.command("hostname")
            group1.db1.query("")
            group2.rest2.ssh.command("hostname")
            group2.db2.query("")

With this scheme tests always need few fixtures passed to them, use fixtures consistently (group.service.method()), and do not care at all about the environment under test. All they need are class instances on which they can execute methods.

Minimize - if you can't eliminate - the use of fixtures set to autouse. They slow down the time it takes to run tests if those tests do not need those fixtures. Since they are set to autouse they get created whether the current test set uses them or not.

Create simple fixtures that do one thing only. Treat them as blocks that can be combined for higher abstraction. But try to minimize the levels of such abstractions.

Tests should not import anything outside of the standard library. Whatever they need to use from within your code base should be provided as a fixture. Instead of importing classes you provide a class instance as a fixture. This helps when refactoring because you only need to refactor reusable fixtures rather than refactoring multiple tests.