Tests using live_server fixture removing data from data migrations

[ekiro]

I've created a simple test case to reproduce this behavior https://github.com/ekiro/case_pytest/blob/master/app/tests.py which fails after second test using live_server fixture.
MyModel objects are created in migration, using RunPython. It seems like after any test with live_server, every row from the database is truncated. Both, postgresql and sqlite3 was tested.

EDIT:
Tests

"""
MyModel objects are created in migration
Test results:
    app/tests.py::test_no_live_server PASSED
    app/tests.py::test_live_server PASSED
    app/tests.py::test_live_server2 FAILED
    app/tests.py::test_no_live_server_after_live_server FAILED
"""

import pytest

from .models import MyModel


@pytest.mark.django_db()
def test_no_live_server():
    """Passed"""
    assert MyModel.objects.count() == 10


@pytest.mark.django_db()
def test_live_server(live_server):
    """Passed"""
    assert MyModel.objects.count() == 10


@pytest.mark.django_db()
def test_live_server2(live_server):
    """Failed, because count() returns 0"""
    assert MyModel.objects.count() == 10


@pytest.mark.django_db()
def test_no_live_server_after_live_server():
    """Failed, because count() returns 0"""
    assert MyModel.objects.count() == 10

[blueyed]

That's because of the transactional_db fixture being used automatically by live_server (there is no need to mark it with @pytest.mark.django_db). There are several issues / discussions in this regard, but the last time I looked closer at this, there was no easy solution to this problem that comes with using data migrations.
A workaround is to use pytest fixtures to wrap your data migrations / data fixtures.
(btw: better put the tests inline in the issue rather than into an external resource that might change over time.)

[aaugustin]

I just hit this as well and it took me a long time to track it down, even though I'm reasonably familiar with Django's test framework and migrations.

This is actually a surprising performance tradeoff of Django's TestCase. It's documented here:
https://docs.djangoproject.com/en/1.9/topics/testing/overview/#rollback-emulation

In a regular Django test suite, the workaround consists in setting a serialized_rollback = True class attribute on the test case.

I don't know how to achieve the same effect with pytest-django's dynamically generated test classes.

[aaugustin]

The following change "solves" the issue at the expense of unconditionally selecting the least efficient behavior.

--- pytest_django/fixtures.py.orig  2016-04-27 17:12:25.000000000 +0200
+++ pytest_django/fixtures.py   2016-04-27 17:21:50.000000000 +0200
@@ -103,6 +103,7 @@

     if django_case:
         case = django_case(methodName='__init__')
+        case.serialized_rollback = True
         case._pre_setup()
         request.addfinalizer(case._post_teardown)

[aaugustin]

The following technique works, but I can't recommend it for rather obvious reasons...

import pytest
from django.core.management import call_command
from pytest_django.fixtures import transactional_db as _transactional_db


def _reload_fixture_data():
    fixture_names = [
        # Create fixtures for the data created by data migrations
        # and list them here.
    ]
    call_command('loaddata', *fixture_names)


@pytest.fixture(scope='function')
def transactional_db(request, _django_db_setup, _django_cursor_wrapper):
    """
    Override a pytest-django fixture to restore the contents of the database.

    This works around https://github.com/pytest-dev/pytest-django/issues/329 by
    restoring data created by data migrations. We know what data matters and we
    maintain it in (Django) fixtures. We don't read it from the database. This
    causes some repetition but keeps this (pytest) fixture (almost) simple.

    """
    try:
        return _transactional_db(request, _django_db_setup, _django_cursor_wrapper)
    finally:
        # /!\ Epically shameful hack /!\ _transactional_db adds two finalizers:
        # _django_cursor_wrapper.disable() and case._post_teardown(). Note that
        # finalizers run in the opposite order of that in which they are added.
        # We want to run after case._post_teardown() which flushes the database
        # but before _django_cursor_wrapper.disable() which prevents further
        # database queries. Hence, open heart surgery in pytest internals...
        finalizers = request._fixturedef._finalizer
        assert len(finalizers) == 2
        assert finalizers[0].__qualname__ == 'CursorManager.disable'
        assert finalizers[1].__qualname__ == 'TransactionTestCase._post_teardown'
        finalizers.insert(1, _reload_fixture_data)

[pelme]

Would an option to conditionally enable serialized_rollback be a good solution?

I.e. something like

@pytest.mark.django_db(transaction=True, serialized_rollback=True)
def test_foo():
    ...

[aaugustin]

I think that would be good.

In my use case:

• the transactional_db fixture is triggered by live_server, but I wouldn't mind adding a django_db fixture to specify serialized rollback
• unrelated to pytest-django -- case.serialized_rollback = True still fails because Django attemps to deserialized objects in an order that doesn't respect FK constraints, I think that's a bug in Django

[pelme]

Ok, perfect :)

I don't have time to work on a fix right now, but adding such an option to the django_db marker should and setting case.serialized_rollback = True (like you did above) should be relatively straightforward.

[aaugustin]

Ticket for the Django bug I mentioned above: https://code.djangoproject.com/ticket/26552

[aaugustin]

@pelme -- does this pull request implement the "relatively straightforward" approach you're envisionning?

[pelme]

Thanks a lot for the PR. The implementation looks correct and what I imagined. I did not think of having a serialized_rollback fixture but it is indeed useful to be able to force this behaviour from other fixtures.

We would need a test for this too, but the implementation looks correct!

[aaugustin]

I'll try to find time to polish the patch (I haven't even run it yet). I need to get the aformentionned bug in Django fixed as well before I can use this.