How (not) to run specs in the lib directory


#1

When running the specs in the lib directory, I will sometimes see:

NoMethodError:
  undefined method `fetch_path' for {}:Hash

Note: the fetch_path method comes from the more_core_extensions gem (I believe).

Today, I believe I’ve finally managed to track down how to reliably reproduce this error and how to work around it. I haven’t yet figured out what’s actually causing the error.

How to reproduce the error

To produce the error, I did the following:

manageiq/lib$ bundle exec rspec spec/openstack

This command will attempt to run only the specs under the spec/openstack directory. For me, this produces four failures, all occurring in the OpenstackEventMonitor#cache_result method.

How to workaround the error

To workaround the error, I did the following:

Run *all* of the tests

manageiq/lib$ bundle exec rspec

The overhead of running all of the tests in the lib directory is fairly low. So, to me this is an acceptable solution.

Questions

1. What is the lib directory?

There are two lib directories. So, this moniker can be a bit confusing. The lib directory I’m talking about is a sibling to the vmdb directory. Not the child of the vmdb directory. So, in a fresh clone of the repository from the root dir, you should see:

build/
ca/
CONTRIBUTING.MD
host/
lib/             <-- This directory
LICENSE.AL
LICENSE.GPL
LICENSE.txt
Rakefile
README.md
system/
vmdb/

2. Is there a deeper reason for the failure?

Quite possibly. But, I haven’t been able to track it down yet. It could be that the specs in question simply need to require the more_core_extensions gem (which is where I believe the fetch_path method comes from).


#2

I don’t see that directory.

But anyway, it looks like fetch_path is provided my more_core_extensions and we have that set to :require => false.

git grep "more_core"
host/Gemfile:gem "more_core_extensions", "~>1.1.0", :require => false
lib/Gemfile:gem "more_core_extensions", "~>1.2.0",   :require => false
lib/cfme_client/Gemfile:gem "more_core_extensions", "~>1.2.0"
lib/cfme_client/lib/cfme_client.rb:require 'more_core_extensions/all'
lib/util/extensions/miq-array.rb:require 'more_core_extensions/core_ext/array'
lib/util/extensions/miq-hash.rb:require 'more_core_extensions/core_ext/hash'
lib/util/extensions/miq-string.rb:require 'more_core_extensions/core_ext/string'
lib/util/miq-extensions.rb:require 'more_core_extensions/all'
vmdb/lockfiles/RHEL6.4/Gemfile.lock:      more_core_extensions (~> 1.1)
vmdb/lockfiles/RHEL6.4/Gemfile.lock:    more_core_extensions (1.2.0)
vmdb/lockfiles/RHEL6.4/Gemfile.lock:  more_core_extensions (~> 1.2.0)

Note, miq-extensions is requiring more_core_extensions/all.

@bdunne suggested we can remove :require => false so more_core_extensions would be auto required.
Then, as long as you are running through bundler, fetch_path would be available.

If that’s NOT possible, then perhaps miq-extensions or more_core_extensions could be required in our spec_helper?

Note, other specs seems to be ok as log as they run through the miq-logger and vmdb_helper lines BEFORE they try to use fetch_path, see below:

git grep miq-extensions
lib/HyperV/MiqHypervInventoryParser.rb:require 'miq-extensions'
lib/HyperV/MiqHypervService.rb:require 'miq-extensions'
lib/RedHatEnterpriseVirtualizationManagerAPI/rhevm_api.rb:require 'util/miq-extensions'
lib/RedHatEnterpriseVirtualizationManagerAPI/rhevm_inventory.rb:require 'util/miq-extensions'
lib/RedHatEnterpriseVirtualizationManagerAPI/rhevm_object.rb:require 'util/miq-extensions'
lib/RedHatEnterpriseVirtualizationManagerAPI/rhevm_service.rb:require 'util/miq-extensions'
lib/VMwareWebService/MiqVimDataStore.rb:require 'miq-extensions'  # Required patch to open-uri for get_file_content
lib/Verbs/implementations/VmwareWinCom.rb:require 'miq-extensions'
lib/libvirt/MiqLibvirt.rb:require 'miq-extensions'
lib/metadata/VmConfig/VmConfig.rb:require 'miq-extensions'
lib/util/miq-ipmi.rb:require 'miq-extensions'
lib/util/miq-logger.rb:require 'miq-extensions'
vmdb/lib/vmdb_helper.rb:require 'miq-extensions'

#3

@jrafanie I tried changing the :require => false to :require => 'more_core_extensions/all' or :require => ['active_support/all', 'more_core_extensions/all'] both of which should load all of more_core_extensions, but it doesn’t appear to work. Here is the change if you’re interested: https://github.com/ManageIQ/manageiq/pull/363/files

The interesting thing is that if I add require 'more_core_extensions/all' to spec_helper the tests pass, the only thing I can think of right now is that it’s an issue with require in Bundler. (Upgrading to the latest Bundler didn’t help either)

btw… the RHEVM API specs fail in the same way.


#4

What command was run and what was the error? Tests can be invoked from ruby, rake, rspec, bundle exec ruby, bundle exec rake, bundle exec rspec, etc. Does it fail because something else is not expecting the extensions to load?


#5

So, @Fryguy told me I was doing this wrong all along.

The way to run tests is the following:

code/manageiq $> vmdb/bin/rake test:lib
code/manageiq $> vmdb/bin/rake test:vmdb

With these commands you can pass specific tests with the SPEC argument:

code/manageiq $> vmdb/bin/rake test:lib SPEC=spec/openstack
code/manageiq $> vmdb/bin/rake test:vmdb SPEC=spec/models/ems_refresh

Or, even multiple values for SPEC

code/manageiq $> vmdb/bin/rake test:vmdb SPEC="spec/models/ems_refresh spec/models/vm_openstack_spec"

Notice a couple of things about these commands:

  1. It’s from the git root directory. Not the vmdb or lib directories.
  2. The values for the SPEC argument are passed as if you’re in the respective subdirectory. So, for vmdb/bin/rake test:vmdb, the arguments for SPEC should be relative to the $git_clone/vmdb directory.

hope that helps


#6

I freely admit that this is terrible, but I haven’t heard a good alternative yet. I keep hearing "I just want to cd vmdb; rspec", but that ignores the fact that many specs can’t run together.

  • vmdb:automation - uses before(:all) to import the “real” automation datastore for testing automate methods.
  • vmdb:replication - sets up (and tears down) two databases, a master and a slave, to verify replication works between them
  • vmdb:migrations - does a db:migrate from 0 to the latest running “up” specs along the way. When it hits the end it migrates back down to 0 running “down” specs along the way.

Because of the randomness of rspec (intentionally chosen), if we ran everything, it could, for example, run a migration spec which puts the database at an old revision, then runs some spec in vmdb which needs the latest db, and would just fail.


#7

I honestly don’t think it’s terrible.

Simply knowing how and why are important. And, look here! That’s what we’ve just done. This might even be important enough to put on the repo’s readme.


#8

I never cd to root. I have an alias that runs rake with a db reset and the backend specs.

rake is expected to run the tests. Can :default just point to the “safe tests”, i.e.: backend only? It seems like the 95% use case.

If you want to run larger integration specs, then you can look for specific rake task.
Automate and replication in my mind is mostly integration. Thought @gmccullough has a better opinion than I do.

At a later time, maybe we could tease apart the unit tests in automate from the ones that say “load all data please”. The ones that don’t need the before blocks, can go in with the units.

Also note that rspec spec/models/user_spec.rb currently works great.


#9

I thought of that, but I’m concerned that it would give the illusion of “everything works” when it might not. Maybe that’s an unfounded concern?

That’s already what’s there. The specs in the automation directory (which is what rake:automation) runs, uses the before(:all) to load all the data up front. The unit tests in spec/models don’t do need the shared setup

Totally forgot to mention that so thanks! So, to run all the specs you can do the rake task, but you can also just do what @kbrock mentioned, for a single spec, or even rspec spec/models to do the entire models directory.


#10

Of note on this thread, @Fryguy is modifying the specs so they are easier to run on travis and for developers as well.


#11

I’ll have a follow up talk thread when I finish it…another PR incoming.


#12

See


#13