How to run tests on ManageIQ


#1

I’ve made various changes to the testing framework recently for some specific reasons, and this changes how specs are run. It was confusing to others in the past, so I’m putting this together to clear it up.


tl;dr: To run the vmdb tests you just cd vmdb then rake or rake test:vmdb . Much more obvious. Other test suites are run similarly.


Much of the old infrastructure was originally designed to support our internal CruiseControl.rb CI systems. However, with Travis we have a lot more features, including automatic PR monitoring. Everything can still technically be done with CruiseControl.rb, it’s just now the commands are a little more complicated than they used to be, but that doesn’t concern me.

The goals of the changes were

  • Leverage as much of Travis as possible
    • Advanced bundler usage such as auto-retry and parallel gem installation
    • Collapsing blocks of work so they don’t clutter up the output
    • Color!
  • Make it simpler for developers to run the specs
    • Support default rake tasks
    • Simplify setup for the complicated test suites.

So, in order to accomplish this, I’ve done the following:

  • Loosened the restrictions on rake and bundler.
    • We can now use whatever bundler and rake are given to us by Travis. On your development system, you can upgrade bundler if you like. There are some nice improvements especially in the performance area. However, if you don’t want to upgrade bundler that’s fine too.
  • Removed the monolithic rake tasks at the root.
    • These rake tasks recreated the database from scratch, rebundled if necessary, and ran all of the specs for a particular suite. Because of its monolithic nature, the auto-collapsing on Travis could not be leveraged, we had to run bundle commands ourselves, and it ate all the color output. Now, one has to cd into vmdb or lib and run the rake tasks from there. The .travis.yml is still at the root, but it knows which subdirectory to change into.
  • Removed the various complicated namespaces.
    • They have been simplified it to test:$TEST_SUITE Previously you had a mix of namespaces, and this makes it obvious.
  • Split out the “setup” for a test suite from its execution.
    • The latter now assumes the database is set up correctly, so for developers there is no need to go through blowing away the database every time. This also allows Travis to “collapse” the setup, since it’s not really what we’re after when viewing the build.
  • Added default rake tasks to both lib and vmdb.
    • A developer can just do rake in vmdb or lib and it will run test:vmdb or test:lib, respectively.

So what does this mean for you?

To run lib tests

cd lib
bundle exec rake test:lib

To run vmdb tests

cd vmdb
bundle exec rake test:vmdb
bundle exec rake test:migrations
bundle exec rake test:replication
bundle exec rake test:automation
bundle exec rake test:javascript

Individual spec files can still be run independently.

bundle exec rspec spec/models/vm_or_template_spec.rb:46

Every test suite has a corresponding setup task, such as test:vmdb:setup, test:migrations:setup, etc. These will destroy the test database and recreate them to prepare for execution of the tests. In the case of replication it will create both master and slave databases. In the case of lib, it will do nothing and is just a stub for consistency.

There is a slightly different set of options when the tests are run on Travis, and those are found in the .rspec and .rspec_ci files. If you want to run like on Travis you can set CI=true before running the task. We can debate the usefulness of this separation…I’ve been thinking of making it all just progress format, or perhaps make it so full suite runs use the “progress” format and individual test runs use “documentation” format.

You may have noticed the new test:javascript suite. Thanks @eclarizio for getting that together so we can run javascript tests with jasmine!


How (not) to run specs in the lib directory
#2

This needs to be a blog post - can you write is up, copying one of the blog posts from github.com/manageiq/manageiq.org/tree/master/source/blog/ ?

Issue a pull request, and I’ll edit and publish.

Thanks!
JM


#3

You know, I almost made this a blog post to begin with, but thought it was too technical or detailed. Do you think the blog post should just be a copy of this, or should it be more of a summary paragraph linking to this talk article?