VM.destroy is slow


#1

Hey Guys,

had a new issue due to delete an Archived VM in your

tools/purge_archived_vms.rb

First of all, your script has a bug:

[root@otto01 vmdb]# bin/rails r tools/purge_archived_vms.rb
DEPRECATION WARNING: `config.serve_static_files` is deprecated and will be removed in Rails 5.1.
Please use `config.public_file_server.enabled = false` instead.
 (called from serve_static_files= at /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/application/configuration.rb:81)
[----] I, [2017-02-24T16:44:14.069354 #8893:699984]  INFO -- : Searching for archived VMs older than 2017-01-24 15:44:14 UTC UTC.
[----] W, [2017-02-24T16:44:14.069454 #8893:699984]  WARN -- : Will delete any matching records.
/opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/bundler/gems/rails-6c2301d7fd22/activerecord/lib/active_record/relation/batches.rb:111:in `find_in_batches': unknown keywords: conditions, include (ArgumentError)
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/bundler/gems/rails-6c2301d7fd22/activerecord/lib/active_record/querying.rb:9:in `find_in_batches'
	from tools/purge_archived_vms.BAK:19:in `<top (required)>'
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/commands/runner.rb:60:in `load'
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/commands/runner.rb:60:in `<top (required)>'
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/commands/commands_tasks.rb:138:in `require'
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/commands/commands_tasks.rb:138:in `require_command!'
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/commands/commands_tasks.rb:104:in `runner'
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/commands/commands_tasks.rb:49:in `run_command!'
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:4:in `require'
	from bin/rails:4:in `<main>'

Fixed that with:

Vm.where("vms.updated_on IS NULL OR vms.updated_on < '#{ARCHIVE_CUTOFF}'").each do |vm|

But this is not the issue.

My problem: during this script is that a deletion of 1 VM takes more than 30 seconds. In some Environments we getting more VMs during this period than we’re able to delete…

Is there another possibility to delete VMs ? Or do you have a hint which component we have to debug to tune up the deletion a bit?

Thanks for your help !


#2

@schmandforke Thanks for finding the bug. Suggest you make a pull request at https://github.com/ManageIQ/manageiq/pulls to get your fix in. Thanks, Oleg


#3

@schmandforke thanks for the PR to fix that bug.

Delete vms shouldn’t take that long, perhaps it’s being run remotely from the database? Maybe you can try it on the appliance with the postgresql database or from an appliance with the smallest db ping(latency) as possible.

Try using vmdb; bin/rails runner tools/db_ping.rb to get an idea of the latency from different locations.

If it’s still slow, you can run the script in development mode to see the queries in the log/development.log.

To do this:

  • make a copy of your config/database.yml
  • configure your development database in config/database.yml to match what’s in production
  • run the tools/purge_archived_vms.rb script again and look in the development.log for very repetitive queries (hundreds to thousands of the same query in a row)
  • Make sure you put back your original config/database.yml

Joe


#4

Hello @schmandforke

I was looking at purged_archived_vms.rb

We currently bring back all vms and detect if they are archived or not.
Seems it would be more efficient to just bring back the records to be archived.

Try the following. note the addition of all_archived. I removed the OR because it was ambiguous if all_archived was inside or outside the parens for the where:

 query = Vm.all_archived
           .where("updated_on IS NULL OR updated_on < ?", ARCHIVE_CUTOFF)
           .includes(:ext_management_system, :storage)

I will put together a PR fixing this script

Thank you

Keenan

UPDATE: tweaked the query


#5

Hey @kbrock,

thanks for your help, tried your Query and got the following:

irb(main):010:0* Vm.all_archived.where("updated_on IS NULL OR updated_on < ?", ARCHIVE_CUTOFF).includes(:ext_management_system, :storage)
NoMethodError: undefined method `where' for #<Array:0x0000001d14af38>
	from (irb):10
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/commands/console.rb:65:in `start'
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/commands/console_helper.rb:9:in `start'
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/commands/commands_tasks.rb:78:in `console'
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/commands/commands_tasks.rb:49:in `run_command!'
	from /opt/rubies/ruby-2.2.5/lib/ruby/gems/2.2.0/gems/railties-5.0.0.rc2/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:4:in `require'
	from bin/rails:4:in `<main>'

#6

Hello @schmandforke

Aah, PR 10961 removed the to_a from all_archived.

You can manually build the where clause locally for now:

query = Vm.where(Vm::ARCHIVED_CONDITIONS)
.where("updated_on IS NULL OR updated_on < ?", ARCHIVE_CUTOFF)
.includes(:ext_management_system, :storage)

I have made these changes in PR 14176