Modify

Ticket #16 (closed enhancement: fixed)

Opened 6 years ago

Last modified 12 months ago

No way to provide initial data for models

Reported by: craig.kimerer@… Owned by: andrew
Priority: minor Milestone:
Component: migrations Version:
Keywords: Cc:

Description

There is currently no way to import custom SQL like you can with django's default syncdb command.

I currently have:

myapp/sql/MyModel.sql

And I want to set up myapp to use south, but I can't figure out a way other than creating a migration that calls db.execute() per line.

It would be useful to not only be able to import when the database is first created (and the data is migrated as the table is altered (ie defaults go in the columns, etc), or to be able to say the data should be inserted after any(all) of the migrations.

I am more than willing to write this patch, I just want to make sure its a feature that fits with south, and to see if anyone has any suggestions on the correct way to implement this in south (ie is it a new db call... db.importdata('MyModel?.sql'), or does it just automagically look in the sql folder and if something exists it gets imported

Attachments

Change History

comment:1 Changed 6 years ago by winhamwr@…

To work around the explicit support, I use things like the following:

from django.core.management import call_command

...snip
#some table creation

db.commit_transaction()
db.start_transaction()

call_command('loaddata', 'some_fixture')

comment:2 Changed 6 years ago by grail@…

I have this problem when trying to do BDD a la Django. In order to test the application the test suite needs to load fixtures which contain data which it uses to ensure that the application behaves as expected in a number of situations. The test suite can't use models since anything in models.py is not stable in time (and thus we have the schema migration system in the first place).

The loaddata option means fixtures must be available as JSON or YAML - which is no great burden since my Rails apps use YAML already. The catch here is that JSON or YAML fixtures are imported through the models, which may not match the structure of the database at the point in time that the fixture was created.

The way Django currently handles SQL is that you store model-specific initialisation in the <app>/sql/<modelname>[.<backend>].sql files. Then when syncdb is run, those SQL files will be run when the tables for those models are created.

Perhaps there is room to reuse the per-application SQL directory to provide raw SQL to be run by the migration?

comment:3 Changed 5 years ago by anonymous

When you run syncdb, it will create any new tables as needed, then whatever happens will load the initial_data for all the apps

I think that when you run migrate, it should do its migrates as needed. At this point, if all migrations are applied, it should load the initial_data for all the apps.

(If south isn't at the latest migration for everything, it should probably hold off the initial_data load, since the initial_data file might not match the state of the database as it is then)

Then, if people have very rapidly changing stuff, they can load it in via migrations as needed. If they have a few bits that almost never change, but always want to be in there (and the definitions of it might be being tweaked), they can go for initial_data and have things behave as they do with syncdb

comment:4 Changed 5 years ago by nick@…

The patch below will implement the scheme I described above

diff -r e51933dbc18f website/onzo_pss/lib/south/management/commands/migrate.py
--- lib/south/management/commands/migrate.py Fri Oct 31 12:37:03 2008 +0000
+++ lib/south/management/commands/migrate.py Fri Oct 31 13:49:04 2008 +0000
@@ -46,10 +46,31 @@

apps = [migration.get_app(app)]

else:

apps = migration.get_migrated_apps()

+
+ mig_app_list = []

for app in apps:

+ mig_app_list.append(app)

migration.migrate_app(

app,
resolve_mode = resolve_mode,
target_name = target,
fake = fake,

)

+
+ # If they migrated up to current, load any initial data for these apps
+ if not target and not skip and not merge and not only and not backwards and not fake:
+ app_list = []
+ for mig_app in mig_app_list:
+ real_app_name = ".".join( mig_app.name.split(".")[0:-1] )
+ app_list.append( import(real_app_name) )
+
+ print
+ print "Applying initial data fixtures"
+
+ # Hack it so that the "full apps list" is actually just the
+ # list of applications we're currently migrating
+ import django.db.models
+ django.db.models.get_apps = lambda: app_list
+
+ from django.core.management import call_command
+ call_command('loaddata', 'initial_data', verbosity=1)

comment:5 Changed 5 years ago by andrew

  • Status changed from new to assigned

OK, I've committed a possible fix for this in [87] - it will run initial_data fixtures on a per-app basis if you migrate up to the current version of an app.

Does it work/help?

comment:6 Changed 5 years ago by winhamwr@…

That definitely helps for initial data.

comment:7 Changed 5 years ago by anonymous

since app is retrieved from get_app which appends 'migrations' it tries to pull from (app)/migrations/fixtures/initial_data.[xml,json,yaml]
not (app)/fixtures/initial_data.[xml,json,yaml]

Is this intentional?

comment:8 Changed 5 years ago by andrew

  • Status changed from assigned to closed
  • Resolution set to fixed

No, that wasn't intentional. It's fixed in [102].

Since that looks like it loads initial_data nicely (it does here), I'm closing this.

comment:9 Changed 5 years ago by anonymous

It's kind of a bummer that the initial_data is applied when the last fixture is reached since initial data has to be modified and migrate with the migrations.

comment:10 follow-up: ↓ 11 Changed 5 years ago by andrew

Indeed, but you're more than welcome to load fixtures as you go inside migrations; just make sure the migrations only do data loading or database altering, not both, or you'll get in a muddle.

However, your initial_data fixture will at least be in a VCS, so it's not a total muddle. The whole data loading/migrations thing is somewhat of a headache, really...

comment:11 in reply to: ↑ 10 Changed 5 years ago by anonymous

Replying to andrew:

Indeed, but you're more than welcome to load fixtures as you go inside migrations; just make sure the migrations only do data loading or database altering, not both, or you'll get in a muddle.

In my experience, any kind of data loading that's not done as the current last step migration gets totally muddled. You'll always be loading data from the latest schema in to a database that's potentially many schema changes behind the latest (unless you write raw SQL I guess to keep things in sync). My "solution" to the problem has been to rip out any data alteration stuff as I create more migrations and move it to the current last migration as I go and by ensuring that any data alteration I do can be performed twice without negative effects (so using get_or_create instead of just create() and things like that). It's a pretty crummy workaround, but the better ways of doing it seem very difficult.

comment:12 Changed 12 months ago by rhoerbe

Workaround: start your project without south:

  • remove myapp from INSTALLED_APPS
  • prune myapp/migrations
  • run ./manage.py convert_to_south myapp to have your initial SQL applies


After that, enable south and run
./manage.py convert_to_south myapp

View

Add a comment

Modify Ticket

Action
as closed
The resolution will be deleted. Next status will be 'reopened'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.