In late 2019 I modernized a large Django 1.11/Python 2 application to use Django 2.2/Python 3. The application included hundreds of database migrations, many of which depended on legacy packages and deprecated functionality that blocked upgrading Django and Python.

This post documents how I cleaned up the legacy migrations to unblock upgrading to modern versions of Django and Python.

Attempt #1: Squash the migrations

The standard way to reduce the number of migrations in Django is called squashing. Squashing generates new migrations that co-exist with the old migrations, which simplifies deploying the changes.

Unfortunately, the application’s complex relationships between models, RunSQL, and RunPython operations prevented the squashmigrations command from running cleanly without manually editing and possibly splitting the existing migration files. Specifically, problems when attempting to squash included:

  • CircularDependencyError exceptions, which require manual resolution.
  • RunPython calls with their code or reverse_code arguments specified as lambda functions, which cannot be serialized. Most of the reverse_code problems could be fixed by replacing the lambda with RunPython.noop since the reverse logic wasn’t implemented.
  • Unserializable model parameters, such as schema-based validators that use the Voluptuous data validation library.

Because of these problems, an alternative to squashing is desirable.

Attempt #2: “–fake” it

An alternative is to create new migrations from the current models and mark them as applied by through the --fake argument to the migrate command.

The advantage of this approach is that all the old migrations can immediately be safely deleted along with their dependencies and associated functions.

The disadvantage of this approach is that is requires special handling of Django’s internal django_migrations table. Therefore, deployment must be coordinated with prior and subsequent model changes.

Because this application’s deployment scripts automate applying migrations, the scripts must be temporarily modified to install the new migrations, as described in the following steps.

Steps

  1. Write a script that deletes the contents of the django_migrations table.
  2. Delete the old migrations, dependencies, and associated functions.
  3. Run python manage.py makemigrations to create new migrations from the current models.
  4. Temporarily replace the startup logic that applies migrations to:
    • Run the script that deletes the contents of the django_migrations table.
    • Run python manage.py migrate --fake. This marks the new migrations as being applied, but doesn’t actually perform database operations—the database is already in the proper state.
  5. After deploying these changes, restore the original startup logic.

Deployment

Because this procedure resets Django’s internal tracking of applied migrations, all prior migrations must be deployed in each environment before these changes are merged in. Similarly, these changes must be applied in each environment before subsequent migrations are merged in.

Specifically, releases to each environment must take place in the following distinct phases:

  1. Apply any unapplied migrations, merge, and deploy.
  2. Clean up migrations following the steps in the previous section. Merge and deploy.
  3. Restore the original deployment scripts, merge, and deploy.

Implementation notes

This section contains some of the specific commands used in this process.

Deleting the old migration files

  • Run the clean_pyc command to delete compiled Python bytecode.
  • Run find . -path "*/migrations/*.py" -not -name "__init__.py" -exec git rm {} \; to mark the old migration files for deletion.
  • Delete unused functions in **/migrations/__init__.py and other modules imported by the old migrations.

Clean the django_migrations table

The django-extensions runscript command runs a script in the context of a Django application. A script that deletes the contents of the django_migrations table could look like:

from django.db import connection

def run():
    with connection.cursor() as cursor:
        cursor.execute("DELETE FROM django_migrations")

References