New split repo - manageiq-schema

We’ve just completed another repo split, and this time, it’s a big one. This time we’ve extracted the ManageIQ database migrations from the db/migrate directory into the new repo. Coming along is the entirety of the migration test suite, a part of the replication tests that were specific to migrations, the Reserve and ReservedMixin classes, and the part of EvmDatabase that was used for schema checking.

From this point forward, changes to the database will need to be done as a separate PR to the manageiq-schema repo. If you have any existing pull requests to ManageIQ that include database changes, they will need to be transferred over. One of the easier ways to pull them over is to do the following:

cd manageiq
git checkout my_feature_branch
git format-patch -1 db/migrate spec/migrations # Change the -N here to the number of commits if you have more than 1

cd ../manageiq-schema
git checkout -b my_feature_branch
git am ../manageiq/00*

git format-patch creates a set of patch files from your commits, and then git am applies those commits in the new repo.

If you have any questions, please ask them here.

1 Like

@Fryguy For those who might be unaware of the rational for this split (I think I fall in that category… but not sure…), mind explaining why we are splitting out the migrations?


On ManageIQ master, you will likely have a new untracked file (or files) at spec/replication/util/data/*_migrations. Those are safe to delete (in fact you can delete the entire spec/replication/util/data directory).

When going back in time to older versions, it’s already tricky to recreate matching versions of all repos.
I’m afraid trying to run old manageiq against master of manageiq-schema will be especially painful :fearful:

Have we considered versioning the schema and explicitly bumping the dependency from manageiq?
(or committing Gemfile.lock, or using submodules, or any other way to keep a history of versions that work together)

  • As a first line of defense, perhaps bin/update could warn if core date is far from schema date?

This repo is not structured like other repos: It is common to create a sym link in spec/ to the core ManageIQ repo and get the settings, Gemfiles and db definitions. But here there is a spec/dummy that is meant to do the same but is already populated.
Why is it like that and will you consider changing it to be like the other repositories?

Thanks Nick. I tried searching, and it seems I never put an announcement out about the rationale for this, and for that I apologize…I forget what we have and haven’t talked about in talk vs other forums.

So, the rationale behind this change is partially technical and partially non-technical.

  • The new repo runs tests in an empty dummy app.
    • Ensures that migrations don’t need anything from the main application. No models accidentally picked up in migrations.
    • Removes need for good_migrations gem
  • Moves running of the migration tests on each PR from main repo to this repo, saving testing time overall.
    • Removing a test suite from the main repo allows us to set up coverage numbers easier since multiple tests suites won’t be involved.
  • We can limit the number of people as mergers on the new repo.
    • Allows those that are “experts” in our particular schema to have the final decision.
    • Allows us to enforce the various schema rules better.
    • Makes it impossible to have schema changes buried in other PRs. We have accidentally merged schema changes in the past because they were buried in large PRs. This also keeps PRs smaller leading to focused discussions and hopefully a faster time to merge.
  • Makes it easier to discuss/schedule the schema freeze at release time, as the freeze date on this particular repo.
1 Like

That is an interesting dilemma, though I think it’s only a problem if you go back in time AND have to recreate the database. Put another way, even before the repo split, if you migrated your database to a certain point, then went back in time in the code base, but didn’t “unmigrate” your database, it is technically out of sync, but devs didn’t seem to have a problem with that. However, I do agree that if you have to go back to a point in time and create a new database at that time, it could be difficult. I’ll have to think on that bit.


@enoodle Unlike the other repos, this repo runs the specs against a dummy app, and thus should not be run against ManageIQ. This ensures that the migrations use nothing from the ManageIQ main repo.


1 Like

Cool, I think I mostly agree with what is being done here, and my main gripe is that this will make things much more convoluted to those who aren’t used to this workflow.

That said, I think that can mostly be addressed by doing a few different things:

  1. Include the above in some modified form in the README of that project, so that when someone stubbles upon that project, they aren’t totally lost (currently it is the box standard README you get from bundler basically). Maybe include some links to a guide or something as well.
  2. Put some errors in place when someone opens up a PR with a migration in the main repo, or even hijack the rails g migration to bark when someone is trying to create a migration within the manageiq project, and point them to instructions on proper migration process.

I think the above honestly won’t be run into too often by outsiders, but it will happen enough me-thinks that having a process outlined at a minimum will reduce question churn by people who are unaware of the new process.

@Fryguy was developer setup updated?
from a brief look, it won’t work for newcomers in the way it’s written now because they won’t fork and fetch the schema repo.

@abonas Good point. Do you think the majority of developers will be making pull requests on this new repo? We also don’t have the ui-classic repo listed. Do you have suggestions on how to list all of the various repos without making it really confusing and complicated for newcomers to manageiq? Pull requests welcome!


I believe it is very relevant for everyone working on a provider.
Whoever works on a provider often needs to add new entities/new capabilities for existing entities = db migrations

Also, although it’s great that PRs are welcome, my expectation is that keeping the documentation/guides intact following code/arch changes, should be the responsibility of the community members who are doing those changes.

I don’t work on providers so I’m not often making schema changes. If it’s relevant to you or others, then feel free to correct any documentation or ask someone who’s made code changes to update the specific areas of the doc that are now stale or wrong. I still contend that information should probably be linked to from the getting started guide.