SmartState Analysis fails on VMware vms with independent - persistent disks

Hello everyone.

My ManageIQ environment consists of one oVirt and two VMWare infrastructure providers and I have a problem with the SmartState Analysis on the VMWare providers.

Most of our Servers have a dependent disk as their main system volume and also a second independent - persistent disk attached.

Perfoming a SmarteState Analysis on VMs no matter which OS (CentOS or Windows) with no independent - persistent disk works without any problem, while doing a Analysis with a independent - persistent disk fails after the job timed out.
If the analysis fails it doesn’t delete the snapshot and leaves lock on the vmdk-File which results in a failed snapshot consolidation and only can be resolved by migrating the existing machine onto another datastore.

If i change the independent - persistent disk to a independent - non-persistent or dependent disk, there are no problems.

What I have found in the evm.log:
[----] I, [2019-07-24T12:07:20.906463 #6081:814f5c] INFO – : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#sync_stashed_metadata) from ManageIQ::Providers::Vmware::InfraManager::Vm
[----] W, [2019-07-24T12:07:20.931361 #6081:814f5c] WARN – : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#sync_stashed_metadata) Synchronize: No data found for [vmconfig]. Items:Total[0] Selected[0] TaskId:[b91e6c06-1f59-4a45-9f26-a8bca52dd55b] VM:[VM001]
[----] W, [2019-07-24T12:07:20.937388 #6081:814f5c] WARN – : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#sync_stashed_metadata) Synchronize: No data found for [accounts]. Items:Total[0] Selected[0] TaskId:[b91e6c06-1f59-4a45-9f26-a8bca52dd55b] VM:[VM001]
[----] W, [2019-07-24T12:07:20.943166 #6081:814f5c] WARN – : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#sync_stashed_metadata) Synchronize: No data found for [software]. Items:Total[0] Selected[0] TaskId:[b91e6c06-1f59-4a45-9f26-a8bca52dd55b] VM:[VM001]
[----] W, [2019-07-24T12:07:20.949151 #6081:814f5c] WARN – : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#sync_stashed_metadata) Synchronize: No data found for [services]. Items:Total[0] Selected[0] TaskId:[b91e6c06-1f59-4a45-9f26-a8bca52dd55b] VM:[VM001]
[----] W, [2019-07-24T12:07:20.954884 #6081:814f5c] WARN – : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#sync_stashed_metadata) Synchronize: No data found for [system]. Items:Total[0] Selected[0] TaskId:[b91e6c06-1f59-4a45-9f26-a8bca52dd55b] VM:[VM001]
[----] I, [2019-07-24T12:07:20.954968 #6081:814f5c] INFO – : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#sync_stashed_metadata) Starting: Sending target summary to server. TaskId:[b91e6c06-1f59-4a45-9f26-a8bca52dd55b] target:[VM001]
[----] D, [2019-07-24T12:07:20.954996 #6081:814f5c] DEBUG – : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#sync_stashed_metadata) xml_summary2 = REXML::Document
[----] I, [2019-07-24T12:07:20.955308 #6081:814f5c] INFO – : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#save_metadata_op) target [eeb3a5ce-6248-47d4-a0f2-b383c01d2ca1], job [b91e6c06-1f59-4a45-9f26-a8bca52dd55b] enter
[----] I, [2019-07-24T12:07:20.955380 #6081:814f5c] INFO – : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#save_metadata_op) target [eeb3a5ce-6248-47d4-a0f2-b383c01d2ca1] found target object id [634], job [b91e6c06-1f59-4a45-9f26-a8bca52dd55b]
[----] I, [2019-07-24T12:07:20.969147 #6081:814f5c] INFO – : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#save_metadata_op) target [eeb3a5ce-6248-47d4-a0f2-b383c01d2ca1] data put on queue, job [b91e6c06-1f59-4a45-9f26-a8bca52dd55b]
[----] I, [2019-07-24T12:07:20.969194 #6081:814f5c] INFO – : Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#sync_stashed_metadata) Completed: Sending target summary to server. TaskId:[b91e6c06-1f59-4a45-9f26-a8bca52dd55b] target:[VM001]
[----] I, [2019-07-24T12:07:24.696830 #5517:814f5c] INFO – : Q-task_id([job_dispatcher]) JOB([b91e6c06-1f59-4a45-9f26-a8bca52dd55b] Message update: [Synchronization in progress]
[----] I, [2019-07-24T12:07:24.727251 #5517:814f5c] INFO – : Q-task_id([job_dispatcher]) MIQ(VmOrTemplate.save_metadata) TaskId = [b91e6c06-1f59-4a45-9f26-a8bca52dd55b]
[----] I, [2019-07-24T12:07:24.735842 #5517:814f5c] INFO – : Q-task_id([job_dispatcher]) MIQ(VmScan#process_data) starting…
[----] I, [2019-07-24T12:07:24.749396 #5517:814f5c] INFO – : Q-task_id([job_dispatcher]) MIQ(VmScan#process_data) Document=summary
[----] I, [2019-07-24T12:07:24.749794 #5517:814f5c] INFO – : Q-task_id([job_dispatcher]) MIQ(VmScan#process_data) Summary XML []
[----] E, [2019-07-24T12:07:24.750169 #5517:814f5c] ERROR – : Q-task_id([job_dispatcher]) MIQ(VmScan#process_data) scan operation yielded no data. aborting
[----] E, [2019-07-24T12:07:24.785208 #5517:814f5c] ERROR – : Q-task_id([job_dispatcher]) MIQ(VmScan#process_abort) job aborting, scan operation yielded no data. aborting

And something from the vim.log
I, [2019-07-24T11:47:15.787280 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/etc/vmware/vsphereFeatures/vsphereFeatures.cfg”: No such file or directory.
I, [2019-07-24T11:47:15.787350 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/etc/vmware/vsphereFeatures/vsphereFeatures.cfg”: No such file or directory.
I, [2019-07-24T11:47:15.787368 #9293] INFO – : VMware(VixDiskLib): FeatureStateLib: Error while loading feature config file: /etc/vmware/vsphereFeatures/vsphereFeatures.cfg, using default feature state values.
I, [2019-07-24T11:47:15.787428 #9293] INFO – : VMware(VixDiskLib): OBJLIB-LIB: Objlib initialized.
I, [2019-07-24T11:47:15.787456 #9293] INFO – : VMware(VixDiskLib): VixDiskLib: Advanced transport module not loaded.
I, [2019-07-24T11:47:15.787472 #9293] INFO – : VMware(VixDiskLib): VixDiskLib: Using transport modes from DiskLib: file:nbdssl:nbd.
I, [2019-07-24T11:47:15.787485 #9293] INFO – : VMware(VixDiskLib): VMware VixDiskLib (6.7) Release build-8173251
I, [2019-07-24T11:47:15.787530 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/etc/vmware/config”: No such file or directory.
I, [2019-07-24T11:47:15.787548 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/etc/vmware/config”: No such file or directory.
I, [2019-07-24T11:47:15.787562 #9293] INFO – : VMware(VixDiskLib): PREF Optional preferences file not found at /etc/vmware/config. Using default values.
I, [2019-07-24T11:47:15.787606 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/usr/lib/vmware/settings”: No such file or directory.
I, [2019-07-24T11:47:15.787624 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/usr/lib/vmware/settings”: No such file or directory.
I, [2019-07-24T11:47:15.787637 #9293] INFO – : VMware(VixDiskLib): PREF Optional preferences file not found at /usr/lib/vmware/settings. Using default values.
I, [2019-07-24T11:47:15.787655 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/usr/lib/vmware/config”: No such file or directory.
I, [2019-07-24T11:47:15.787669 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/usr/lib/vmware/config”: No such file or directory.
I, [2019-07-24T11:47:15.787683 #9293] INFO – : VMware(VixDiskLib): PREF Optional preferences file not found at /usr/lib/vmware/config. Using default values.
I, [2019-07-24T11:47:15.787701 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/root/.vmware/config”: No such file or directory.
I, [2019-07-24T11:47:15.787717 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/root/.vmware/config”: No such file or directory.
I, [2019-07-24T11:47:15.787729 #9293] INFO – : VMware(VixDiskLib): PREF Optional preferences file not found at /root/.vmware/config. Using default values.
I, [2019-07-24T11:47:15.787745 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/root/.vmware/preferences”: No such file or directory.
I, [2019-07-24T11:47:15.787769 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/root/.vmware/preferences”: No such file or directory.
I, [2019-07-24T11:47:15.787784 #9293] INFO – : VMware(VixDiskLib): PREF Optional preferences file not found at /root/.vmware/preferences. Using default values.
I, [2019-07-24T11:47:15.789591 #9293] INFO – : VMware(VixDiskLib): lib/ssl: OpenSSL using FIPS_drbg for RAND
I, [2019-07-24T11:47:15.789721 #9293] INFO – : VMware(VixDiskLib): lib/ssl: protocol list tls1.2
I, [2019-07-24T11:47:15.789742 #9293] INFO – : VMware(VixDiskLib): lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)
I, [2019-07-24T11:47:15.789774 #9293] INFO – : VMware(VixDiskLib): lib/ssl: cipher list !aNULL:kECDH+AESGCM:ECDH+AESGCM:RSA+AESGCM:kECDH+AES:ECDH+AES:RSA+AES
I, [2019-07-24T11:47:16.026728 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/etc/vmware/vsphereFeatures/vsphereFeatures.cfg”: No such file or directory.
I, [2019-07-24T11:47:16.026837 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/etc/vmware/vsphereFeatures/vsphereFeatures.cfg”: No such file or directory.
I, [2019-07-24T11:47:16.026856 #9293] INFO – : VMware(VixDiskLib): FeatureStateLib: Error while loading feature config file: /etc/vmware/vsphereFeatures/vsphereFeatures.cfg, using default feature state values.
I, [2019-07-24T11:47:16.026872 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_Init: Initialization is completed.
I, [2019-07-24T11:47:16.027289 #9293] INFO – : VdlWrapper.connect:
I, [2019-07-24T11:47:16.027316 #9293] INFO – : VdlConnection.initialize:
I, [2019-07-24T11:47:16.027392 #9293] INFO – : VMware(VixDiskLib): VixDiskLib: VixDiskLib_Connect: Establish connection.
W, [2019-07-24T11:47:16.027419 #9293] WARN – : VMware(VixDiskLib): VixDiskLib: VixDiskLibIsLegacyConnParams: the instance of VixDiskLibConnectParams is NOT allocated by VixDiskLib_AllocateConnectParams. The new features in 6.7 or later are not supported.
I, [2019-07-24T11:47:16.027455 #9293] INFO – : VMware(VixDiskLib): VixDiskLib: Resolve host.
D, [2019-07-24T11:47:16.044263 #9293] DEBUG – : VdlDisk.new <16438020>: opening [DISK_0004] VM001/VM001.vmdk
I, [2019-07-24T11:47:16.044322 #9293] INFO – : VMware(VixDiskLib): VixDiskLib: VixDiskLib_OpenEx: Open a disk.
I, [2019-07-24T11:47:16.044345 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_GetFileName: Get file name for [DISK_0004] VM001/VM001.vmdk.
I, [2019-07-24T11:47:16.044361 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_GetNfcTicket: Get NFC ticket for [DISK_0004] VM001/VM001.vmdk.
I, [2019-07-24T11:47:16.044392 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/etc/vmware/config”: No such file or directory.
I, [2019-07-24T11:47:16.044410 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/etc/vmware/config”: No such file or directory.
I, [2019-07-24T11:47:16.044421 #9293] INFO – : VMware(VixDiskLib): PREF Optional preferences file not found at /etc/vmware/config. Using default values.
I, [2019-07-24T11:47:16.044461 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/usr/lib/vmware/settings”: No such file or directory.
I, [2019-07-24T11:47:16.044475 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/usr/lib/vmware/settings”: No such file or directory.
I, [2019-07-24T11:47:16.044485 #9293] INFO – : VMware(VixDiskLib): PREF Optional preferences file not found at /usr/lib/vmware/settings. Using default values.
I, [2019-07-24T11:47:16.044499 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/usr/lib/vmware/config”: No such file or directory.
I, [2019-07-24T11:47:16.044523 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/usr/lib/vmware/config”: No such file or directory.
I, [2019-07-24T11:47:16.044535 #9293] INFO – : VMware(VixDiskLib): PREF Optional preferences file not found at /usr/lib/vmware/config. Using default values.
I, [2019-07-24T11:47:16.044548 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/root/.vmware/config”: No such file or directory.
I, [2019-07-24T11:47:16.044567 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/root/.vmware/config”: No such file or directory.
I, [2019-07-24T11:47:16.044577 #9293] INFO – : VMware(VixDiskLib): PREF Optional preferences file not found at /root/.vmware/config. Using default values.
I, [2019-07-24T11:47:16.044590 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/root/.vmware/preferences”: No such file or directory.
I, [2019-07-24T11:47:16.044602 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/root/.vmware/preferences”: No such file or directory.
I, [2019-07-24T11:47:16.044612 #9293] INFO – : VMware(VixDiskLib): PREF Optional preferences file not found at /root/.vmware/preferences. Using default values.
I, [2019-07-24T11:47:16.718284 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: Request RandomAccessRO diskKey = 2000, readOnly = 1, openSnapshot = 0.
I, [2019-07-24T11:47:16.938820 #9293] INFO – : startup has happened, shutdown flag is
I, [2019-07-24T11:47:16.938902 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVimFreeInternalVCAboutInfo: Free Internal VC AboutInfo.
I, [2019-07-24T11:47:16.939576 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_HandleDiskCryptoKey: handle the key of disk [DISK_0004] VM001/VM001.vmdk.
I, [2019-07-24T11:47:16.939607 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_FreeNfcTicket: Free NFC ticket.
I, [2019-07-24T11:47:16.939626 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_FreeFileName: Free file name.
I, [2019-07-24T11:47:16.939692 #9293] INFO – : VMware(VixDiskLib): NBD_ClientOpen: attempting to create connection to ha-nfc://[DISK_0004] VM001/VM001.vmdk@10.1.1.26:902
I, [2019-07-24T11:47:17.078427 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/etc/vmware/vsphereFeatures/vsphereFeatures.cfg”: No such file or directory.
I, [2019-07-24T11:47:17.078523 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/etc/vmware/vsphereFeatures/vsphereFeatures.cfg”: No such file or directory.
I, [2019-07-24T11:47:17.078551 #9293] INFO – : VMware(VixDiskLib): FeatureStateLib: Error while loading feature config file: /etc/vmware/vsphereFeatures/vsphereFeatures.cfg, using default feature state values.
I, [2019-07-24T11:47:17.079189 #9293] INFO – : VMware(VixDiskLib): NFC Async IO session is established with log level 1.
I, [2019-07-24T11:47:17.079222 #9293] INFO – : VMware(VixDiskLib): Opening file [DISK_0004] VM001/VM001.vmdk (ha-nfc://[DISK_0004] VM001/VM001.vmdk@10.1.1.26:902)
I, [2019-07-24T11:47:17.084312 #9293] INFO – : VMware(VixDiskLib): DISKLIB-LINK : Opened ‘ha-nfc://[DISK_0004] VM001/VM001.vmdk@10.1.1.26:902’ (0xe): custom, 41943040 sectors / 20 GB.
I, [2019-07-24T11:47:17.090401 #9293] INFO – : VMware(VixDiskLib): DISKLIB-LIB : Opened “ha-nfc://[DISK_0004] VM001/VM001.vmdk@10.1.1.26:902” (flags 0xe, type custom).
I, [2019-07-24T11:47:17.090472 #9293] INFO – : VMware(VixDiskLib): VixDiskLib: VixDiskLib_GetInfo: Retrieve disk info.
I, [2019-07-24T11:47:17.098706 #9293] INFO – : VdlConnection.getDisk: open disks = 1
D, [2019-07-24T11:47:17.140241 #9293] DEBUG – : VdlDisk.new <16404900>: opening [DISK_P002] VM001_Neu/VM001_Neu_1.vmdk
I, [2019-07-24T11:47:17.140310 #9293] INFO – : VMware(VixDiskLib): VixDiskLib: VixDiskLib_OpenEx: Open a disk.
I, [2019-07-24T11:47:17.140363 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_GetFileName: Get file name for [DISK_P002] VM001_Neu/VM001_Neu_1.vmdk.
I, [2019-07-24T11:47:17.140387 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_GetNfcTicket: Get NFC ticket for [DISK_P002] VM001_Neu/VM001_Neu_1.vmdk.
I, [2019-07-24T11:47:17.788289 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: Request RandomAccessRO diskKey = 2001, readOnly = 1, openSnapshot = 0.
I, [2019-07-24T11:47:18.016296 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVimFreeInternalVCAboutInfo: Free Internal VC AboutInfo.
I, [2019-07-24T11:47:18.016429 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_HandleDiskCryptoKey: handle the key of disk [DISK_P002] VM001_Neu/VM001_Neu_1.vmdk.
I, [2019-07-24T11:47:18.016464 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_FreeNfcTicket: Free NFC ticket.
I, [2019-07-24T11:47:18.016484 #9293] INFO – : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_FreeFileName: Free file name.
I, [2019-07-24T11:47:18.016530 #9293] INFO – : VMware(VixDiskLib): NBD_ClientOpen: attempting to create connection to ha-nfc://[DISK_P002] VM001_Neu/VM001_Neu_1.vmdk@10.1.1.26:902
I, [2019-07-24T11:47:18.148099 #9293] INFO – : VMware(VixDiskLib): DictionaryLoad: Cannot open file “/etc/vmware/vsphereFeatures/vsphereFeatures.cfg”: No such file or directory.
I, [2019-07-24T11:47:18.148184 #9293] INFO – : VMware(VixDiskLib): [msg.dictionary.load.openFailed] Cannot open file “/etc/vmware/vsphereFeatures/vsphereFeatures.cfg”: No such file or directory.
I, [2019-07-24T11:47:18.148209 #9293] INFO – : VMware(VixDiskLib): FeatureStateLib: Error while loading feature config file: /etc/vmware/vsphereFeatures/vsphereFeatures.cfg, using default feature state values.
I, [2019-07-24T11:47:18.148830 #9293] INFO – : VMware(VixDiskLib): NFC Async IO session is established with log level 1.

  • Version: Hammer 7
  • Esxi Version: 6.7
  • Vddk Version: 6.7

I tried different datastores and changed the scan_via_host variable but I always get the same failed result.

Does anyone have some hints on how to solve this?

Thanks
Gregor