SSA on VMware fails with non-root user with Admin Priviliges


#1

Hi,

I’m having an SSA failure, did anybody came across a similar issue?

I’ve created a local user with Administrator privileges on one ESX server, and entered this users details through GUI via:

Infrastructure --> Hosts --> <ESX_Host> --> Configuration --> Edit this host --> Credentials --> Default

Validate button displays “Credential validation was successful”, and I can save this info successfully. But when I try to perform SmartState Analysis for one of the VM’s on this ESX host, task fails with the following error:

Unable to mount filesystem. Reason:[VixDiskLibApi#open (errcode=14009 - VIX_E_HOST_NETWORK_CONN_REFUSED): NBD_ERR_NETWORK_CONNECT - [lera_vmware2_DS] elrond/elrond.vmdk

Reverting credentials to built-in ESX root user enables SSA to complete successfully, but regarding the security policy, we prefer not to use the root account.

Please find the system details and relevant vim.log below:

Vcenter - 5.5
VDDK - Tested with 5.5.2 and 5.5.4
ESX - Tested ssh login with the new user, telnet to port 902, verified vddk connection with vix-disklib-sample - all seems fine

/var/www/miq/vmdb/log/vim.log:

/var/www/miq/lib/VixDiskLib/VixDiskLibServer.rb:14: warning: already initialized constant MIQ_ROOT
/var/www/miq/lib/VixDiskLib/VixDiskLibServer.rb:15: warning: already initialized constant LOG_DIR
/var/www/miq/lib/VixDiskLib/VixDiskLibServer.rb:16: warning: already initialized constant LOG_FILE
[----] I, [2015-05-12T13:09:02.133621 #25114:bc7014]  INFO -- : Started DRb service on URI druby://cfme3.linuxera.hq:39498
[----] I, [2015-05-12T13:09:02.133828 #25114:bc7014]  INFO -- : calling watchdog for startup
[----] W, [2015-05-12T13:09:03.898303 #25114:d7c9cc]  WARN -- : VMware(VixDiskLib): VixDiskLib: Invalid configuration file parameter.  Failed to read configuration file.
[----] I, [2015-05-12T13:09:03.904700 #25114:d7c9cc]  INFO -- : VMware(VixDiskLib): OBJLIB-LIB: Objlib initialized.
[----] I, [2015-05-12T13:09:03.904862 #25114:d7c9cc]  INFO -- : VMware(VixDiskLib): VixDiskLib: Advanced transport module not loaded.
[----] I, [2015-05-12T13:09:03.905008 #25114:d7c9cc]  INFO -- : VMware(VixDiskLib): VixDiskLib: Using transport modes from DiskLib: file:nbdssl:nbd.
[----] I, [2015-05-12T13:09:03.905142 #25114:d7c9cc]  INFO -- : VMware(VixDiskLib): VMware VixDiskLib (5.5) Release build-1890828
[----] I, [2015-05-12T13:09:03.936864 #25114:d7c9cc]  INFO -- : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_Init: Initialization is completed.
[----] I, [2015-05-12T13:09:03.937952 #25114:d7c9cc]  INFO -- : VixDiskLibServer.connect: 
[----] I, [2015-05-12T13:09:03.938089 #25114:d7c9cc]  INFO -- : VdlConnection.initialize: 
[----] I, [2015-05-12T13:09:03.938352 #25114:d7c9cc]  INFO -- : VMware(VixDiskLib): VixDiskLib: VixDiskLib_Connect: Establish connection.
[----] I, [2015-05-12T13:09:04.035649 #25114:d85478]  INFO -- : VMware(VixDiskLib): VixDiskLib: VixDiskLib_OpenEx: Open a disk.
[----] I, [2015-05-12T13:09:04.035816 #25114:d85478]  INFO -- : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_GetNfcTicket: Get NFC ticket for /vmfs/volumes/53da497b-fd0362aa-aa9c-28924a30b1f8/elrond/elrond.vmdk.
[----] I, [2015-05-12T13:09:04.035993 #25114:d85478]  INFO -- : VMware(VixDiskLib): VThreadBase detected multiple threads.
[----] I, [2015-05-12T13:09:04.811433 #25114:d85478]  INFO -- : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_FreeNfcTicket: Free NFC ticket.
[----] I, [2015-05-12T13:09:04.813429 #25114:d85478]  INFO -- : VMware(VixDiskLib): NBD_ClientOpen: attempting to create connection to ha-nfc:///vmfs/volumes/53da497b-fd0362aa-aa9c-28924a30b1f8/elrond/elrond.vmdk@192.168.3.102:902
[----] W, [2015-05-12T13:09:04.829887 #25114:d85478]  WARN -- : VMware(VixDiskLib): SSL_VerifyCbHelper: Certificate verification is disabled, so connection will proceed despite the error
[----] I, [2015-05-12T13:09:04.830111 #25114:bc7014]  INFO -- : startup has happened, shutdown flag is 
[----] W, [2015-05-12T13:09:04.830338 #25114:d85478]  WARN -- : VMware(VixDiskLib): SSL_VerifyCbHelper: Certificate verification is disabled, so connection will proceed despite the error
[----] W, [2015-05-12T13:09:04.830710 #25114:d85478]  WARN -- : VMware(VixDiskLib): SSL_VerifyCbHelper: Certificate verification is disabled, so connection will proceed despite the error
[----] I, [2015-05-12T13:09:04.923328 #25114:d85478]  INFO -- : VMware(VixDiskLib): CnxConnectAuthd: Returning false because CnxAuthdProtoConnect failed
[----] I, [2015-05-12T13:09:04.923583 #25114:d85478]  INFO -- : VMware(VixDiskLib): Cnx_Connect: Returning false because CnxConnectAuthd failed
[----] I, [2015-05-12T13:09:04.923840 #25114:d85478]  INFO -- : VMware(VixDiskLib): Cnx_Connect: Error message: 550 User not authorized for host agent contact
[----] W, [2015-05-12T13:09:04.924027 #25114:d85478]  WARN -- : VMware(VixDiskLib): [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: 550 User not authorized for host agent contact
[----] I, [2015-05-12T13:09:04.924405 #25114:d85478]  INFO -- : VMware(VixDiskLib): NBD_ClientOpen: Couldn't connect to 192.168.3.102:902 550 User not authorized for host agent contact
[----] I, [2015-05-12T13:09:04.924659 #25114:d85478]  INFO -- : VMware(VixDiskLib): DISKLIB-DSCPTR: : "ha-nfc:///vmfs/volumes/53da497b-fd0362aa-aa9c-28924a30b1f8/elrond/elrond.vmdk@192.168.3.102:902" : Failed to open NBD extent.
[----] I, [2015-05-12T13:09:04.924864 #25114:d85478]  INFO -- : VMware(VixDiskLib): DISKLIB-LINK  : "ha-nfc:///vmfs/volumes/53da497b-fd0362aa-aa9c-28924a30b1f8/elrond/elrond.vmdk@192.168.3.102:902" : failed to open (NBD_ERR_NETWORK_CONNECT).  
[----] I, [2015-05-12T13:09:04.925036 #25114:d85478]  INFO -- : VMware(VixDiskLib): DISKLIB-CHAIN : "ha-nfc:///vmfs/volumes/53da497b-fd0362aa-aa9c-28924a30b1f8/elrond/elrond.vmdk@192.168.3.102:902" : failed to open (NBD_ERR_NETWORK_CONNECT).
[----] I, [2015-05-12T13:09:04.925370 #25114:d85478]  INFO -- : VMware(VixDiskLib): DISKLIB-LIB   : Failed to open 'ha-nfc:///vmfs/volumes/53da497b-fd0362aa-aa9c-28924a30b1f8/elrond/elrond.vmdk@192.168.3.102:902' with flags 0x1e NBD_ERR_NETWORK_CONNECT (2338).
[----] I, [2015-05-12T13:09:04.925553 #25114:d85478]  INFO -- : VMware(VixDiskLib): VixDiskLib: Detected DiskLib error 2338 (NBD_ERR_NETWORK_CONNECT).
#<VixDiskLibError: VixDiskLibApi#open (errcode=14009 - VIX_E_HOST_NETWORK_CONN_REFUSED): NBD_ERR_NETWORK_CONNECT>
/var/www/miq/lib/VixDiskLib/vixdisklib_api.rb:440:in `check_error'
/var/www/miq/lib/VixDiskLib/vixdisklib_api.rb:266:in `open'
/var/www/miq/lib/VixDiskLib/vixdisklib_server.rb:203:in `initialize'
/var/www/miq/lib/VixDiskLib/vixdisklib_server.rb:150:in `new'
/var/www/miq/lib/VixDiskLib/vixdisklib_server.rb:150:in `block in getDisk'
/opt/rh/ruby193/root/usr/share/ruby/sync.rb:227:in `sync_synchronize'
/var/www/miq/lib/VixDiskLib/vixdisklib_server.rb:147:in `getDisk'
/opt/rh/ruby193/root/usr/share/ruby/drb/drb.rb:1548:in `perform_without_block'
/opt/rh/ruby193/root/usr/share/ruby/drb/drb.rb:1508:in `perform'
/opt/rh/ruby193/root/usr/share/ruby/drb/drb.rb:1586:in `block (2 levels) in main_loop'
/opt/rh/ruby193/root/usr/share/ruby/drb/drb.rb:1582:in `loop'
/opt/rh/ruby193/root/usr/share/ruby/drb/drb.rb:1582:in `block in main_loop'
[----] I, [2015-05-12T13:09:04.934348 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): VixDiskLib: VixDiskLib_OpenEx: Open a disk.
[----] I, [2015-05-12T13:09:04.934504 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_GetNfcTicket: Get NFC ticket for /vmfs/volumes/53da497b-fd0362aa-aa9c-28924a30b1f8/elrond/elrond_1.vmdk.
[----] I, [2015-05-12T13:09:05.646738 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_FreeNfcTicket: Free NFC ticket.
[----] I, [2015-05-12T13:09:05.659004 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): NBD_ClientOpen: attempting to create connection to ha-nfc:///vmfs/volumes/53da497b-fd0362aa-aa9c-28924a30b1f8/elrond/elrond_1.vmdk@192.168.3.102:902
[----] W, [2015-05-12T13:09:05.671704 #25114:e6ff64]  WARN -- : VMware(VixDiskLib): SSL_VerifyCbHelper: Certificate verification is disabled, so connection will proceed despite the error
[----] W, [2015-05-12T13:09:05.671951 #25114:e6ff64]  WARN -- : VMware(VixDiskLib): SSL_VerifyCbHelper: Certificate verification is disabled, so connection will proceed despite the error
[----] W, [2015-05-12T13:09:05.672255 #25114:e6ff64]  WARN -- : VMware(VixDiskLib): SSL_VerifyCbHelper: Certificate verification is disabled, so connection will proceed despite the error
[----] I, [2015-05-12T13:09:05.767653 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): CnxConnectAuthd: Returning false because CnxAuthdProtoConnect failed
[----] I, [2015-05-12T13:09:05.767826 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): Cnx_Connect: Returning false because CnxConnectAuthd failed
[----] I, [2015-05-12T13:09:05.767984 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): Cnx_Connect: Error message: 550 User not authorized for host agent contact
[----] W, [2015-05-12T13:09:05.768126 #25114:e6ff64]  WARN -- : VMware(VixDiskLib): [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: 550 User not authorized for host agent contact
[----] I, [2015-05-12T13:09:05.768302 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): NBD_ClientOpen: Couldn't connect to 192.168.3.102:902 550 User not authorized for host agent contact
[----] I, [2015-05-12T13:09:05.768538 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): DISKLIB-DSCPTR: : "ha-nfc:///vmfs/volumes/53da497b-fd0362aa-aa9c-28924a30b1f8/elrond/elrond_1.vmdk@192.168.3.102:902" : Failed to open NBD extent.
[----] I, [2015-05-12T13:09:05.768709 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): DISKLIB-LINK  : "ha-nfc:///vmfs/volumes/53da497b-fd0362aa-aa9c-28924a30b1f8/elrond/elrond_1.vmdk@192.168.3.102:902" : failed to open (NBD_ERR_NETWORK_CONNECT).  
[----] I, [2015-05-12T13:09:05.768867 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): DISKLIB-CHAIN : "ha-nfc:///vmfs/volumes/53da497b-fd0362aa-aa9c-28924a30b1f8/elrond/elrond_1.vmdk@192.168.3.102:902" : failed to open (NBD_ERR_NETWORK_CONNECT).
[----] I, [2015-05-12T13:09:05.769089 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): DISKLIB-LIB   : Failed to open 'ha-nfc:///vmfs/volumes/53da497b-fd0362aa-aa9c-28924a30b1f8/elrond/elrond_1.vmdk@192.168.3.102:902' with flags 0x1e NBD_ERR_NETWORK_CONNECT (2338).
[----] I, [2015-05-12T13:09:05.769253 #25114:e6ff64]  INFO -- : VMware(VixDiskLib): VixDiskLib: Detected DiskLib error 2338 (NBD_ERR_NETWORK_CONNECT).
#<VixDiskLibError: VixDiskLibApi#open (errcode=14009 - VIX_E_HOST_NETWORK_CONN_REFUSED): NBD_ERR_NETWORK_CONNECT>
/var/www/miq/lib/VixDiskLib/vixdisklib_api.rb:440:in `check_error'
/var/www/miq/lib/VixDiskLib/vixdisklib_api.rb:266:in `open'
/var/www/miq/lib/VixDiskLib/vixdisklib_server.rb:203:in `initialize'
/var/www/miq/lib/VixDiskLib/vixdisklib_server.rb:150:in `new'
/var/www/miq/lib/VixDiskLib/vixdisklib_server.rb:150:in `block in getDisk'
/opt/rh/ruby193/root/usr/share/ruby/sync.rb:227:in `sync_synchronize'
/var/www/miq/lib/VixDiskLib/vixdisklib_server.rb:147:in `getDisk'
/opt/rh/ruby193/root/usr/share/ruby/drb/drb.rb:1548:in `perform_without_block'
/opt/rh/ruby193/root/usr/share/ruby/drb/drb.rb:1508:in `perform'
/opt/rh/ruby193/root/usr/share/ruby/drb/drb.rb:1586:in `block (2 levels) in main_loop'
/opt/rh/ruby193/root/usr/share/ruby/drb/drb.rb:1582:in `loop'
/opt/rh/ruby193/root/usr/share/ruby/drb/drb.rb:1582:in `block in main_loop'
[----] I, [2015-05-12T13:09:05.781113 #25114:dfd914]  INFO -- : VdlConnection.disconnect: 
[----] I, [2015-05-12T13:09:05.781506 #25114:dfd914]  INFO -- : VixDiskLibServer.__disconnect__: 
[----] I, [2015-05-12T13:09:05.781736 #25114:dfd914]  INFO -- : VMware(VixDiskLib): VixDiskLib: VixDiskLib_Disconnect: Disconnect.
[----] I, [2015-05-12T13:09:05.781979 #25114:bc7014]  INFO -- : Shutting Down VixDiskLibServer
[----] I, [2015-05-12T13:09:05.782083 #25114:bc7014]  INFO -- : VixDiskLib has exited cleanly
[----] I, [2015-05-12T13:09:05.782152 #25114:bc7014]  INFO -- : VixDiskLibServer.__exit__ finished
[----] I, [2015-05-12T13:09:05.782387 #25114:bc7014]  INFO -- : Finished shutting down DRb
[----] I, [2015-05-12T13:09:05.782460 #25114:bc7014]  INFO -- : Service has stopped

Best Regards,

Ekin.


Unable to run a SSA on VMware environment
#2

Google shows up many results with the similar problems.
It seems like a common vddk issue where the user should have the same permissions as root but doesn’t.

https://communities.vmware.com/thread/447562
http://www.thinware.net/Community/Forums/tabid/70/forumid/16/postid/537/scope/posts/Default.aspx

@rpo Any ideas?


#3

is it possible that the following article might be of use?
https://access.redhat.com/articles/312063


#4

Hi,

Thanks for the pointers, but I’ve already reviewed it and had checked against it beforehand. The document is for ESX 3.x/4.x, but basic principles apply to ESX 5.5 - see below for details:

  • I’m using a non-root user with administrator role, so all permissions mentioned are granted,
  • In ESX 5.5, we don’t have the adm, bin, daemon, sys and wheel groups - no groups are listed in the vsphere client as well. I went on and tried to add the user to the root group (the only group in esx5) by hand, but that did not help either…

While investigating, I collected hostd logs from ESX server. One apparent diff between root and non-root log are the following lines - it seems Nfcsvc plugin does not start when we log-in with non-root user:

2015-05-13T19:47:35.043Z [31481B70 info 'Nfcsvc'] Successfully initialized nfc callback for a  write to the socket to be invoked on a separate thread
2015-05-13T19:47:35.043Z [31481B70 info 'Nfcsvc'] Plugin started
2015-05-13T19:47:35.074Z [31481B70 info 'Libs'] NfcServerProcessClientMsg: NFC Client authenticity check skipped!

Please let me know if you need different logs or tests, I have an available test system which I can reproduce and debug this issue.


#5

Hi again,

The logs I mentioned in the previous post are here:

http://www.linuxera.com/files/hostd-cfmeuser.log : Verbose level hostd.log for non-root user (SSA failed)
http://www.linuxera.com/files/hostd-cfmeuser-trivia.log.tar.bz2 : Trivia level hostd.log for non-root user (SSA failed)
http://www.linuxera.com/files/hostd-root.log : Verbose hostd.log for root user (SSA succeded)


#6

Hi,

Just to note, I went on and tried VDDK 5.1.3 and 5.1.4, but they produce the same result - i.e. SSA fails with errcode=14009…


#7

Hey there everyone!
Has there been any movement on this issue since May? I’m finding the same thing exactly in our CloudForms Proof-of-Concept environment. Is there a solution for this, or is “root” the only workaround at present? That would be a potential show-stopper, given some of our tenants have stringent security requirements, and using root would be a problem.


#8

We have SSA running in VMware vCenter 5.5/ESXi 5.5, but it is set up to run through vCenter instead of individual hosts.

This is the post I followed for the setup: