Cleaning up Django migrations

17. mar 2016 10:48 UTC

Background

Djangos database migrations system greatly simplifies the task of keeping multiple databases in sync (the schema, not the data) as the schema evolves over time. Formerly known as South, Djangos migrations system is also a good example of how Django embraces and includes 3rd party packages when it makes sense.

Over the last year I (and colleagues) have been developing a Django-based provisioning system for an ISP. The system has grown rather large with a lot of models (160ish plus abstract models at the moment) and plenty of foreignkey constraints connecting them. Migrations have helped us keep everything in sync between local instances on three developers laptops, staging, and production. There is simply no way we could have have done this, in the time we've done it, without the migrations system.

The following was all done on Django 1.8.5 but the approach should work fine on other recent Django versions.

The problem

Running makemigrations and migrate takes longer and longer to run. We currently have 450 migration files across 23 apps, the largest app having 61 migrations. This might seem excessive, but it is simply a result of an iterative development process where we have started using a very early version of the system in production, and slowly over many months the system has grown to where we are today.

Running migrate on production takes way too long:

[tsr@mrx /usr/local/www/mrx]$ time python manage.py migrate
Operations to perform:
  Synchronize unmigrated apps: simple_history, panther, staticfiles, wkhtmltopdf, messages, allauth, django_countries, humanize, haystack, bootstrap3
  Apply all migrations: radius, cpeservice, network, ipsystem, workers, sessions, sms, sites, contenttypes, mrxcore, booking, infrastructure, partners, auth, tvservice, customer, account, admin, notes, zabbix, phoneservice, tftp, internetservice
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying phoneservice.0043_auto_20160317_1014... OK

real    7m29.112s
user    7m17.795s
sys     0m7.442s
[tsr@mrx /usr/local/www/mrx]$

Depending on what part of the system the migration touches this could mean seven and a half minutes of production downtime, which is obviously unacceptable. I would have addressed it sooner, but instead I've just mostly been running migrations in production in off-hours.

squashmigrations

Django offers a solution to this issue called squashmigrations which allows you to squash a bunch of migrations for a given app together, optimizing them as much as possible. It has a few shortcomings though, which has made me choose the "clean slate" approach described in this blogpost instead:

I have a bunch of runPython migrations (which can't be optimized) that I no longer need to keep around.
It requires manual editing of the migration files after running squashmigrations, which I don't want to do because I'm lazy and it is errorprone.

A clean cut

I decided to to a "clean cut" where I:

Make sure all databases are at the same migration level
Delete all files except __init__.py from all apps migrations/ folders, commit to Git

Delete all rows from django_migrations table:

mrxdb=# delete from django_migrations ;
DELETE 389
mrxdb=#

Run manage.py makemigrations (and commit the resulting migration files to Git)

{ mrx } master » python manage.py makemigrations                 /cygdrive/c/tr/devel/mrx/src/mrx
Migrations for 'customer':
  0001_initial.py:
    - Create model ContactInfo
    - Create model ContactInfoOverride
    - Create model Customer
    - Create model HistoricalContactInfo
    - Create model HistoricalContactInfoOverride
    - Create model HistoricalCustomer
    - Create model HistoricalMigrationStatus
    - Create model HistoricalOwnerPermissionLetter
    - Create model HistoricalSolution
    - Create model HistoricalVacationHousePeriod
    - Create model MigrationStatus
    - Create model OwnerPermissionLetter
    - Create model Solution
    - Create model SolutionType
    - Create model VacationHousePeriod
    - Create model WebsiteOrder
    - Add field type to solution
    - Add field solution to historicalvacationhouseperiod
    - Add field type to historicalsolution
  0002_auto_20160317_1224.py:
    - Add field delivery to historicalcontactinfooverride
    - Add field history_user to historicalcontactinfooverride
    - Add field history_user to historicalcontactinfo
    - Add field contactinfo to customer
    - Add field streetaddress to customer
    - Add field contactinfo to contactinfooverride
    - Add field delivery to contactinfooverride
    - Alter unique_together for customer (4 constraint(s))
    - Alter unique_together for contactinfooverride (1 constraint(s))
Migrations for 'cpeservice':
  0001_initial.py:
    - Create model CPE
    - Create model CPEConfig
    - Create model CPEManagementService
    - Create model CPEModel
    - Create model CPEStatus
    - Create model CPEWifiConfig
    - Create model HistoricalCPE
    - Create model HistoricalCPEConfig
    - Create model HistoricalCPEManagementService
    - Create model HistoricalCPEModel
    - Create model HistoricalCPEStatus
    - Create model HistoricalCPEWifiConfig
  0002_auto_20160317_1224.py:
    - Add field atp to historicalcpemanagementservice
    - Add field cpe to historicalcpemanagementservice
    - Add field cpeconfig to historicalcpemanagementservice
    - Add field delivery to historicalcpemanagementservice
    - Add field history_user to historicalcpemanagementservice
    - Add field ipaddress to historicalcpemanagementservice
    - Add field servicetype to historicalcpemanagementservice
    - Add field wificonfig to historicalcpemanagementservice
    - Add field history_user to historicalcpeconfig
    - Add field model to historicalcpeconfig
    - Add field history_user to historicalcpe
    - Add field model to historicalcpe
    - Add field status to historicalcpe
    - Alter unique_together for cpestatus (1 constraint(s))
    - Alter unique_together for cpemodel (1 constraint(s))
    - Add field atp to cpemanagementservice
    - Add field cpe to cpemanagementservice
    - Add field cpeconfig to cpemanagementservice
    - Add field delivery to cpemanagementservice
    - Add field ipaddress to cpemanagementservice
    - Add field servicetype to cpemanagementservice
    - Add field wificonfig to cpemanagementservice
    - Add field model to cpeconfig
    - Add field model to cpe
    - Add field status to cpe
    - Alter unique_together for cpemanagementservice (3 constraint(s))
    - Alter unique_together for cpe (1 constraint(s))
Migrations for 'infrastructure':
  0001_initial.py:
    - Create model HistoricalRack
    - Create model HistoricalRoom
    - Create model HistoricalSite
    - Create model HistoricalStreetAddress
    - Create model HistoricalStreetRack
    - Create model HistoricalZone
    - Create model HistoricalZoneStatus
    - Create model Rack
    - Create model Room
    - Create model Site
    - Create model StreetAddress
    - Create model StreetRack
    - Create model Zone
    - Create model ZoneStatus
    - Add field status to zone
    - Add field zone to streetrack
    - Alter unique_together for streetaddress (1 constraint(s))
    - Add field address to site
    - Add field site to room
    - Add field room to rack
    - Add field parentzone to historicalzone
    - Add field site to historicalzone
    - Add field status to historicalzone
    - Add field address to historicalstreetrack
    - Add field history_user to historicalstreetrack
    - Add field parent to historicalstreetrack
    - Add field zone to historicalstreetrack
    - Add field address to historicalsite
    - Add field history_user to historicalsite
    - Add field site to historicalroom
    - Add field room to historicalrack
    - Alter unique_together for zone (1 constraint(s))
    - Alter unique_together for streetrack (1 constraint(s))
    - Alter unique_together for site (1 constraint(s))
Migrations for 'network':
  0001_initial.py:
    - Create model ATP
    - Create model AtpHint
    - Create model Circuit
    - Create model CircuitStatus
    - Create model Delivery
    - Create model Equipment
    - Create model EquipmentModel
    - Create model HistoricalATP
    - Create model HistoricalAtpHint
    - Create model HistoricalCircuit
    - Create model HistoricalCircuitStatus
    - Create model HistoricalDelivery
    - Create model HistoricalEquipment
    - Create model HistoricalEquipmentModel
    - Create model HistoricalPort
    - Create model HistoricalPortTemplate
    - Create model HistoricalServiceType
    - Create model HistoricalSTP
    - Create model HistoricalStpHint
    - Create model Port
    - Create model PortTemplate
    - Create model ServiceType
    - Create model STP
    - Create model StpHint
    - Alter unique_together for servicetype (1 constraint(s))
    - Add field template to port
    - Add field servicetype to historicalstphint
    - Add field stp to historicalstphint
    - Add field port to historicalstp
    - Add field template to historicalport
    - Add field port to historicalcircuit
    - Add field status to historicalcircuit
    - Add field streetrack to historicalcircuit
    - Add field servicetype to historicalatphint
    - Add field port to historicalatp
    - Add field stp to historicalatp
    - Alter unique_together for equipmentmodel (1 constraint(s))
    - Add field model to equipment
    - Add field rack to equipment
    - Alter unique_together for circuitstatus (1 constraint(s))
    - Add field port to circuit
    - Add field status to circuit
    - Add field streetrack to circuit
    - Add field equipment to atphint
    - Add field servicetype to atphint
    - Add field port to atp
    - Add field stp to atp
    - Alter unique_together for stphint (1 constraint(s))
    - Alter unique_together for equipment (2 constraint(s))
    - Alter unique_together for delivery (1 constraint(s))
    - Alter unique_together for circuit (1 constraint(s))
    - Alter unique_together for atphint (1 constraint(s))
Migrations for 'ipsystem':
  0001_initial.py:
    - Create model HistoricalIpAllocation
    - Create model HistoricalIpPool
    - Create model HistoricalIpPoolHint
    - Create model HistoricalIpPoolType
    - Create model IpAllocation
    - Create model IpPool
    - Create model IpPoolHint
    - Create model IpPoolType
    - Add field type to ippool
    - Add field ippool to ipallocation
    - Add field ippool to historicalippoolhint
    - Add field stp to historicalippoolhint
    - Add field type to historicalippool
    - Add field ippool to historicalipallocation
    - Alter unique_together for ippoolhint (1 constraint(s))
    - Alter unique_together for ipallocation (1 constraint(s))
Migrations for 'partners':
  0001_initial.py:
    - Create model WaooInternetProduct
    - Create model WaooInternetService
    - Create model WaooIPTVChannel
    - Create model WaooIPTVPackage
    - Create model WaooIPTVProduct
    - Create model WaooSettopBox
    - Create model WaooSoapRequest
    - Create model WaooSoapResponse
    - Create model WaooVoipCompany
    - Create model WaooVoipProduct
    - Create model WaooVoipService
    - Alter unique_together for waoovoipproduct (2 constraint(s))
    - Alter unique_together for waoovoipcompany (2 constraint(s))
  0002_auto_20160317_1224.py:
    - Add field settopbox_model to waoosettopbox
    - Add field includes to waooiptvproduct
    - Add field package to waooiptvproduct
    - Add field channels to waooiptvpackage
    - Alter unique_together for waoointernetservice (2 constraint(s))
    - Alter unique_together for waoointernetproduct (2 constraint(s))
    - Alter unique_together for waoovoipservice (2 constraint(s))
    - Alter unique_together for waoosettopbox (2 constraint(s))
    - Alter unique_together for waooiptvproduct (1 constraint(s))
    - Alter unique_together for waooiptvpackage (2 constraint(s))
Migrations for 'notes':
  0001_initial.py:
    - Create model CustomerNote
    - Create model DeliveryNote
Migrations for 'sms':
  0001_initial.py:
    - Create model HistoricalIncomingSMS
    - Create model HistoricalOutgoingSMS
    - Create model HistoricalOutgoingSMSJob
    - Create model IncomingSMS
    - Create model OutgoingSMS
    - Create model OutgoingSMSJob
    - Create model SMSGateway
    - Alter unique_together for smsgateway (1 constraint(s))
    - Add field gateway to outgoingsms
    - Add field smsjob to outgoingsms
    - Add field gateway to incomingsms
    - Add field gateway to historicaloutgoingsms
    - Add field history_user to historicaloutgoingsms
    - Add field smsjob to historicaloutgoingsms
    - Add field gateway to historicalincomingsms
    - Add field history_user to historicalincomingsms
    - Alter unique_together for outgoingsms (1 constraint(s))
Migrations for 'mrxcore':
  0001_initial.py:
    - Create model EventLog
    - Create model HistoricalMrxPrice
    - Create model HistoricalMrxProduct
    - Create model HistoricalMrxProductType
    - Create model HistoricalProfile
    - Create model MrxPrice
    - Create model MrxPriceList
    - Create model MrxProduct
    - Create model MrxProductType
    - Create model Profile
    - Add field mrxproducttype to mrxproduct
  0002_auto_20160317_1224.py:
    - Add field waoo_internetproduct to mrxproduct
    - Add field waoo_internetservice to mrxproduct
    - Add field waoo_iptvproduct to mrxproduct
    - Add field waoo_settopbox to mrxproduct
    - Add field waoo_voipproduct to mrxproduct
    - Add field waoo_voipservicebundle to mrxproduct
    - Add field mrxproduct to mrxprice
    - Add field history_user to historicalprofile
    - Add field user to historicalprofile
    - Add field history_user to historicalmrxproducttype
    - Add field servicetype to historicalmrxproducttype
    - Add field history_user to historicalmrxproduct
    - Add field mrxproducttype to historicalmrxproduct
    - Add field waoo_internetproduct to historicalmrxproduct
    - Add field waoo_internetservice to historicalmrxproduct
    - Add field waoo_iptvproduct to historicalmrxproduct
    - Add field waoo_settopbox to historicalmrxproduct
    - Add field waoo_voipproduct to historicalmrxproduct
    - Add field waoo_voipservicebundle to historicalmrxproduct
    - Add field history_user to historicalmrxprice
    - Add field mrxproduct to historicalmrxprice
    - Add field user to eventlog
    - Alter unique_together for profile (1 constraint(s))
    - Alter unique_together for mrxproduct (1 constraint(s))
Migrations for 'zabbix':
  0001_initial.py:
    - Create model ZabbixDirtyDeliveries
Migrations for 'radius':
  0001_initial.py:
    - Create model HistoricalNas
    - Create model HistoricalRadacct
    - Create model HistoricalRadcheck
    - Create model HistoricalRadgroupcheck
    - Create model HistoricalRadgroupreply
    - Create model HistoricalRadpostauth
    - Create model HistoricalRadreply
    - Create model HistoricalRadusergroup
    - Create model Nas
    - Create model Radacct
    - Create model Radcheck
    - Create model Radgroupcheck
    - Create model Radgroupreply
    - Create model Radpostauth
    - Create model Radreply
    - Create model Radusergroup
Migrations for 'tvservice':
  0001_initial.py:
    - Create model HistoricalSettopBox
    - Create model HistoricalSettopProduct
    - Create model HistoricalTvService
    - Create model SettopBox
    - Create model SettopBoxModel
    - Create model SettopProduct
    - Create model TvService
    - Create model VODREntry
    - Create model VODRFile
    - Alter unique_together for vodrfile (1 constraint(s))
    - Add field vodrfile to vodrentry
    - Add field tvservice to settopproduct
    - Add field model to settopbox
    - Add field settopbox to historicalsettopproduct
    - Add field tvservice to historicalsettopproduct
    - Add field model to historicalsettopbox
    - Alter unique_together for vodrentry (1 constraint(s))
    - Alter unique_together for settopproduct (1 constraint(s))
    - Alter unique_together for settopbox (2 constraint(s))
Migrations for 'phoneservice':
  0001_initial.py:
    - Create model CDREntry
    - Create model CDRFile
    - Create model HistoricalPhoneService
    - Create model PhoneNumberRange
    - Create model PhoneService
    - Create model SipServer
    - Create model VoipServiceBundle
    - Add field sip_server to phoneservice
    - Add field voipservices to phoneservice
    - Add field sip_server to historicalphoneservice
    - Alter unique_together for cdrfile (1 constraint(s))
    - Add field cdrfile to cdrentry
    - Alter unique_together for phoneservice (1 constraint(s))
    - Alter unique_together for cdrentry (1 constraint(s))
Migrations for 'tftp':
  0001_initial.py:
    - Create model TftpLog
Migrations for 'internetservice':
  0001_initial.py:
    - Create model HistoricalInternetService
    - Create model HistoricalInternetSpeed
    - Create model InternetService
    - Create model InternetSpeed
  0002_auto_20160317_1224.py:
    - Add field mrxproduct to internetspeed
    - Add field atp to internetservice
    - Add field delivery to internetservice
    - Add field internetservices to internetservice
    - Add field ipaddress to internetservice
    - Add field servicetype to internetservice
    - Add field speed to internetservice
    - Add field history_user to historicalinternetspeed
    - Add field mrxproduct to historicalinternetspeed
    - Add field atp to historicalinternetservice
    - Add field delivery to historicalinternetservice
    - Add field history_user to historicalinternetservice
    - Add field ipaddress to historicalinternetservice
    - Add field servicetype to historicalinternetservice
    - Add field speed to historicalinternetservice
    - Alter unique_together for internetspeed (1 constraint(s))
    - Alter unique_together for internetservice (1 constraint(s))
Migrations for 'workers':
  0001_initial.py:
    - Create model HistoricalPortProvisioningJob
    - Create model PortProvisioningJob
Migrations for 'booking':
  0001_initial.py:
    - Create model HistoricalTechBooking
    - Create model TechBooking
  0002_auto_20160317_1224.py:
    - Add field delivery to techbooking
    - Add field technician to techbooking
    - Add field booked_by to historicaltechbooking
    - Add field delivery to historicaltechbooking
    - Add field history_user to historicaltechbooking
    - Add field technician to historicaltechbooking
{ mrx } master »

Run manage.py migrate --fake

[tsr@mrx /usr/local/www/mrx]$ time python manage.py migrate --fake
Operations to perform:
Synchronize unmigrated apps: simple_history, panther, staticfiles, wkhtmltopdf, messages, allauth, django_countries, humanize, haystack, bootstrap3
Apply all migrations: radius, cpeservice, network, ipsystem, workers, sessions, sms, sites, contenttypes, mrxcore, booking, infrastructure, partners, auth, tvservice, customer, account, admin, notes, zabbix, phoneservice, tftp, internetservice
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... FAKED
Applying auth.0001_initial... FAKED
Applying account.0001_initial... FAKED
Applying account.0002_email_max_length... FAKED
Applying admin.0001_initial... FAKED
Applying contenttypes.0002_remove_content_type_name... FAKED
Applying auth.0002_alter_permission_name_max_length... FAKED
Applying auth.0003_alter_user_email_max_length... FAKED
Applying auth.0004_alter_user_username_opts... FAKED
Applying auth.0005_alter_user_last_login_null... FAKED
Applying auth.0006_require_contenttypes_0002... FAKED
Applying infrastructure.0001_initial... FAKED
Applying customer.0001_initial... FAKED
Applying network.0001_initial... FAKED
Applying booking.0001_initial... FAKED
Applying booking.0002_auto_20160317_1224... FAKED
Applying ipsystem.0001_initial... FAKED
Applying cpeservice.0001_initial... FAKED
Applying cpeservice.0002_auto_20160317_1224... FAKED
Applying customer.0002_auto_20160317_1224... FAKED
Applying mrxcore.0001_initial... FAKED
Applying internetservice.0001_initial... FAKED
Applying internetservice.0002_auto_20160317_1224... FAKED
Applying partners.0001_initial... FAKED
Applying phoneservice.0001_initial... FAKED
Applying mrxcore.0002_auto_20160317_1224... FAKED
Applying notes.0001_initial... FAKED
Applying tvservice.0001_initial... FAKED
Applying partners.0002_auto_20160317_1224... FAKED
Applying radius.0001_initial... FAKED
Applying sessions.0001_initial... FAKED
Applying sites.0001_initial... FAKED
Applying sms.0001_initial... FAKED
Applying tftp.0001_initial... FAKED
Applying workers.0001_initial... FAKED
Applying zabbix.0001_initial... FAKED

real 2m4.277s
user 2m0.303s
sys 0m2.360s
[tsr@mrx /usr/local/www/mrx]$

After this I just repeat point 3) and 5) on all other instances (including production). This change means that makemigrations and migrate both take seconds instead of minutes. Very nice.

Final thoughts

It would have been nice if Djangos squashmigrations system could have handled all this. On the other hand, if you (like me) don't need to keep old stuff (we have no need to unapply migrations for example), or if you have runPython data migrations which are no longer needed, the clean slate approach described here offers a nice fast way to reset your migrations. I might need to do the same thing again in another year when we've made a bunch of new changes - fortunately I can just go find this blogpost by then :)

Update

Just to let everyone know, we've done this exercise a few more times, how often depends on the amount of database changes we do, but on average I'd say about every year, give or take. Git log tells me that we did again on October 24th 2016, which is only around 6 months after this blogpost was written, and again in now in January 2018. YMMV.

Search this blog

Tags for this blogpost

django migrations