Our CPU problem rised again today, when 6 vmware windows servers were ordered in 10 minutes span. 2 were completed ok (r1792, r1795), 4 failed.
I managed to track the automation flow to this code ( in our / Infrastructure / VM / Provisioning / StateMachines / Methods / vmware_PostProvision):
unless disks.empty?
disks.each do |dsk|
$evm.log(:info, "Adding #{dsk['size_gb']}GB disk to VM #{vm.name}")
vm.add_disk( "[#{vm.storage_name}]", dsk['size_mb'], {:sync => true, :thin_provisioned => true, :dependent => true})
end
end
$evm.log(:info, "Additional disks: #{disks}")
(User can request 1 to 3 optional disks in the service catalog form)
if I grep these messages, results are these:
miq server1:
[----] I, [2020-05-04T10:54:24.719293 #30713:d6f5c04] INFO – : Q-task_id([r1793_miq_provision_4342]) Adding 30GB disk to VM SAPLAB691
[----] I, [2020-05-04T10:54:25.074380 #2336:bedfca0] INFO – : Q-task_id([r1794_miq_provision_4333]) Adding 30GB disk to VM SAPLAB692
miq server2:
[----] I, [2020-05-04T10:54:19.868512 #28140:120311c0] INFO – : Q-task_id([r1791_miq_provision_4328]) Adding 40GB disk to VM SAPLAB689
[----] I, [2020-05-04T10:54:21.774805 #25223:ef797a8] INFO – : Q-task_id([r1796_miq_provision_4340]) Adding 20GB disk to VM SAPLAB688
No " Additional disks: " message in output here. So vm.add_disk was last called method in every one of that hunged requests.
for me is suspicious that all vm.add_disk methods were called almost simultaneously. Maybe some concurrency problem?
2 other vmware windows servers (with 1 optional disk each as well) were requested in that time but both completed successfully (much slower then usually due to CPU load). And log from that:
[----] I, [2020-05-04T11:42:37.359249 #18649:8171940] INFO – : Q-task_id([r1792_miq_provision_4330]) Adding 40GB disk to VM SAPLAB690
[----] I, [2020-05-04T11:44:12.487391 #20901:cdf9948] INFO – : Q-task_id([r1795_miq_provision_4336]) Adding 30GB disk to VM SAPLAB693
[----] I, [2020-05-04T11:47:43.720788 #18649:8171940] INFO – : Q-task_id([r1792_miq_provision_4330]) Additional disks: [{“drive_letter”=>“D”, “size_gb”=>40, “size_kb”=>41943040, “size_mb”=>40960, “drive_label”=>“DATA”, “id”=>0}]
[----] I, [2020-05-04T11:48:24.706817 #20901:cdf9948] INFO – : Q-task_id([r1795_miq_provision_4336]) Additional disks: [{“drive_letter”=>“D”, “size_gb”=>30, “size_kb”=>31457280, “size_mb”=>30720, “drive_label”=>“DATA”, “id”=>0}]
This was working fine in hammer-4
Do you have any suggestion?
Thank you
