Cloud-init on VMware provider

vmi · March 1, 2016, 9:57am

Hello,

I am looking for best approach how to provision linux (RHEL 7, CentOS 7) virtual machine on VMware (vSphere 6) infrastructure provider, including some customization action.

So I am wondering if it is possible to use cloud-init.

Thanks in advance for any advice.
Vaclav

jprause · March 1, 2016, 2:23pm

@gmccullough can you review this question by @vmi and forward to a SME if necessary.

gmccullough · March 1, 2016, 11:44pm

Hi @vmi,

ManageIQ supports VMware’s Customization Specifications for linux. That would be the best place to start. It does not support Cloud-Init on VMware, but does for other providers.

vmi · March 2, 2016, 7:03am

Hi @gmccullough,

Through VMware customization specification I am able to define networking basics as IP address, hostname and DNS records…

But how can I modify other stuff as setting passwords and keys, adding sw packages including basic configuration of deployed sw? PXE is not suitable in my case.

Kind regards,
Vaclav

gmccullough · March 2, 2016, 4:34pm

@ramrexx @bascar Do you have experience or guidance here you can share?

geertj · March 2, 2016, 8:43pm

Out of curiosity, what would be required to get CloudInit to work on VMware? Could be a nice community project for someone.

Having CloudInit on VMware would be cool as it would allow you to employ the same way of customization across VMware, RHEV, OpenStack and AWS.

gmccullough · March 2, 2016, 11:02pm

I agree that having CloudInit would be very useful here. To support it we would need to determine how to pass the CloudInit script to VMware as part of the clone API calls. From there the effort would be to include CloudInit as an option on the Customize tab in the provision dialogs.

ramrexx · March 4, 2016, 3:32am

This is typically where configuration management comes into play. After a VM is provisioned you may choose to use Chef, Puppet and ofcourse Ansible to finish the guest configuration. If those options are not available, you are in luck as the MIQ appliance ships with an SSH ruby gem that makes this a breeze.

See the ruby sample method below:

#!/usr/bin/env ruby
require ‘net/ssh’
new_size = 20
new_disk = “/dev/sdd”
volume_group = “centos”
volume_group_path = “/dev/#{volume_group}/root”

ssh = Net::SSH.start( ‘10.10.10.10’, ‘root’, :password => ‘evm123’ )
ssh.exec!(“for i in ls /sys/class/scsi_host/|awk {'print $1'}; do echo ‘- - -’ > /sys/class/scsi_host/$i/scan; done”)
ssh.exec!(“ssm add -p #{volume_group} #{new_disk}”)
ssh.exec!(“ssm resize -s+#{new_size}GB #{volume_group_path}”)
ssh.close

ramrexx · March 4, 2016, 3:35am

@gmccullough - If you guys are serious here is a sample method that someone wrote that writes an ISO per VM and mounts it to the guest. there is quite a bit of orchestration here but it is possible.

perl cloudforms-mountiso.pl --url https://192.168.252.24:443/sdk/webservice --username Administrator --password mypassword --operation mount --vmname DHCP --datastore NFS-ISO --filename e1000.iso

perl cloudforms-mountiso.pl --url https://192.168.252.24:443/sdk/webservice --username Administrator --password mypassword --operation umount --vmname DHCP

###################################
begin
@method = ‘Reconfigure_CDROM’
$evm.log(“info”, “#{@method} - EVM Automate Method Started”)

Turn of debugging

@debug = true

###################################

Method: reconfigure_cdrom

###################################
def reconfigure_cdrom( vm, operation, datastore, filename )
# Build the Perl command using the VMDB information
cmd = “perl /usr/lib/vmware-vcli/apps/vm/vmISOManagement.pl”
cmd += " --server #{vm.ext_management_system.ipaddress}"
cmd += " --username "#{vm.ext_management_system.authentication_userid}""
cmd += " --password "#{vm.ext_management_system.authentication_password}""
cmd += " --vmname "#{vm.name}""
cmd += " --operation "#{operation}""
if operation == ‘mount’
cmd += " --datastore "#{datastore}""
cmd += " --filename "#{filename}""
end
$evm.log(“info”, “Running: #{cmd}”)
results = system(cmd)
return results
end

###################################

Method: boolean

###################################
def boolean(string)
return true if string == true || string =~ (/(true|t|yes|y|1)$/i)
return false if string == false || string.nil? || string =~ (/(false|f|no|n|0)$/i)
end

$evm.log(“info”, “#{@method} - EVM Automate Method Started”)

Get vm from root object is 3 layers down from the service template provision task

service_template_provision_task = $evm.root[‘service_template_provision_task’]
service_template_provision_task.miq_request_tasks.each do |child_miq_request_task|
child_miq_request_task.miq_request_tasks.each do |grandchild_miq_request_task|
vm = grandchild_miq_request_task.destination
raise “#{@method} - VM object not found” if vm.nil?

# Get root attributes passed in from the service dialog
#operation = $evm.root['operation'] || 'umount'
operation = 'mount'
#datastore = $evm.root['datastore'] || nil
datastore = 'cloudinit-nfs'
#filename  = $evm.root['filename']  || nil
filename = 'ipxe/customization/cloud-init.iso'

$evm.log("info", "#{@method} - Detected Operation:<#{operation}> for VM:<#{vm.name}> Datastore:<#{datastore}> Filename:<#{filename}>")

results = reconfigure_cdrom( vm, operation, datastore, filename )
if results
  $evm.log("info", "#{@method} - VM Reconfigure of CD/DVD Successful:<#{results.inspect}>")
  if operation == 'mount'
    vm.custom_set(:DVDROM, filename)
    vm.start
  else
    vm.custom_set(:DVDROM, nil)
  end
else
  raise "#{@method} - VM Reconfigure of CD/DVD Failed:<#{results.inspect}>"
end

end
end

Exit method

$evm.log(“info”, “#{@method} - EVM Automate Method Ended”)
exit MIQ_OK

hyclak · March 8, 2016, 7:57pm

We are using the VIX API to pass scripts/calls through to the VM via VMware Tools.

There can either be a script on the VMware Template that is called by VIX with the appropriate arguments or VIX can be used to inject a script that is executed. We have a simple script for setting the password:

#!/bin/sh
#
# Usage: set_local_password.sh username password

if [ $# -ne 2 ]; then
  echo "Usage: $0 username password"
  exit 1
fi

echo -e "$2\n$2" | sudo passwd $1

The automate code is a bit convoluted to get it loaded up into the running VM via the tools, but if that would be helpful I can try and pare it down. After the script above is injected, we run the following:

  @logger.log(:info, "guest_os: #{guest_os}")
  # Determine the username we want to set
  case
  when guest_os.match(/ubuntu/)
    username = 'admin'
  else
    username = 'root'
  end

  # Get the password from the provision dialog
  password = vm.miq_provision.options[:root_password]

  # Determine command to run in guest
  guest_program_path, guest_args =
    get_guest_command(platform, guest_file, username, password)
  @logger.log(:info, "Attempting to run: '#{guest_program_path} " \
                     "#{guest_args}' as '#{@guest_auth[:username]}'")

  # Run the command in the guest
  pids = get_program_pids(guest_program_path, guest_args, guest_directory)

  # Wait for the command to complete
  wait_for_command(pids)

  # Cleanup and process results
  process_command_results(pids)
  @logger.log(:info, "Deleting '#{guest_file}' " \
                     "from '#{vm.name}'")
  @gom.fileManager.DeleteFileInGuest(vm: @vim_vm, auth: @guest_auth,
                                     filePath: guest_file)

  @logger.log(:info, 'set_root_password Method Ended')
  exit MIQ_OK

Hope that helps!

Matt

opsourcetrail · March 9, 2016, 2:09pm

Hi Matt,
this really seems interesting. Would it be possible to share how the get_program_pids method looks like? It would be helpful to see how to work with the VIX API.

Thanks a lot for the tip.

All the best,
Jan

hyclak · March 9, 2016, 3:05pm

It’s not pretty, but here it is. I think this is where the “methods should be 10 lines or less” rule makes things a little more convoluted than it could be.

def construct_guest_auth
  username = @gom_config['username']
  password = @gom_config.decrypt('password')

  fail 'guest username & password are required!' unless username && password

  guest_auth = RbVmomi::VIM::NamePasswordAuthentication(
    interactiveSession: false,
    username: username,
    password: password
  )

  guest_auth
end

def get_guest_temp_file(filename)
  temp_file_suffix = File.extname(filename)
  temp_file_prefix = File.basename(filename, temp_file_suffix) + '_'

  guest_file = @gom.fileManager.CreateTemporaryFileInGuest(
    vm: @vim_vm,
    auth: @guest_auth,
    prefix: temp_file_prefix,
    suffix: temp_file_suffix
  )

  fail 'unable to obtain a temp guest file' if guest_file.nil?

  guest_file
end

def use_sudo
  if @guest_auth[:username] == 'root'
    false
  elsif $evm.object['use_sudo'].nil?
    @gom_config['use_sudo']
  else
    $evm.object['use_sudo']
  end
end

def linux_program_path(interpreter)
  sudo_path = $evm.object['sudo_path']

  use_sudo ? sudo_path : interpreter
end

def get_guest_program_path(platform, interpreter)
  guest_program_path =
    if platform == 'linux'
      linux_program_path(interpreter)
    elsif platform == 'windows'
      interpreter
    else
      fail 'only linux & windows are supported'
    end

  guest_program_path
end

def get_linux_program_args(guest_file, interpreter, interpreter_args, username, password)
  guest_args =
    if use_sudo
      "#{interpreter} #{interpreter_args} #{guest_file} #{username} #{password}"
    else
      "#{interpreter_args} #{guest_file} #{username} #{password}"
    end

  guest_args
end

def get_guest_args(platform, guest_file, interpreter, interpreter_args, username, password)
  guest_args =
    if platform == 'linux'
      get_linux_program_args(guest_file, interpreter, interpreter_args, username, password)
    else
      fail 'only linux supported'
    end

  guest_args
end

def get_guest_command(platform, guest_file, username, password)
  interpreter = $evm.object["#{platform}_interpreter"]
  interpreter_args = $evm.object["#{platform}_interpreter_args"]

  guest_program_path = get_guest_program_path(platform, interpreter)
  guest_args = get_guest_args(platform, guest_file, interpreter,
                              interpreter_args, username, password)

  [guest_program_path, guest_args]
end

def get_file_attrs(platform)
  # file will be created with 0644 permissions
  file_attrs =
    if platform == 'linux'
      RbVmomi::VIM::GuestPosixFileAttributes()
    elsif platform == 'windows'
      RbVmomi::VIM::GuestWindowsFileAttributes()
    else
      fail 'only linux & windows are supported'
    end

  file_attrs
end

def get_put_path(guest_file, platform, file_size)
  file_attrs = get_file_attrs(platform)
  put_path = @gom.fileManager.InitiateFileTransferToGuest(
    vm: @vim_vm, auth: @guest_auth, guestFilePath: guest_file,
    fileAttributes: file_attrs, fileSize: file_size, overwrite: true
  )

  put_path
end

def construct_request(request_uri, host_file, file_size)
  request = Net::HTTP::Put.new(request_uri)
  request.body_stream = File.open(host_file)
  request['Content-Type'] = 'multipart/form-data'
  request.add_field('Content-Length', file_size)

  request
end

def put_file(host_file, guest_file, platform)
  file_size = File.size(host_file)

  put_path = get_put_path(guest_file, platform, file_size)
  uri = URI.parse(put_path)
  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = (uri.scheme == 'https')
  http.verify_mode = OpenSSL::SSL::VERIFY_NONE

  request = construct_request(uri.request_uri, host_file, file_size)
  http_response = http.request(request)

  http_response
end

def process_http_response(http_response)
  if http_response.code == '200'
    @logger.log(:info, http_response.body)
  else
    fail "The transfer of '#{host_file}' to '#{guest_file}' FAILED!"
  end
end

def get_program_pids(guest_program_path, guest_args, guest_directory)
  prog_spec = RbVmomi::VIM::GuestProgramSpec(
    programPath: guest_program_path,
    arguments: guest_args,
    workingDirectory: guest_directory
  )
  pids = []
  pids << @gom.processManager.StartProgramInGuest(
    vm: @vim_vm, auth: @guest_auth, spec: prog_spec
  )

  pids
end

def log_process(process, sleep_time)
  if process.endTime.nil?
    @logger.log(:info, "'#{process.cmdLine}' is still running, sleeping " \
                       "for #{sleep_time} seconds")
    return true
  else
    @logger.log(:info, "'#{process.cmdLine}' has finished running")
    return false
  end
end

def report_proccess(processes)
  need_to_sleep = false
  sleep_time = 15.seconds
  processes.each do |process|
    need_to_sleep ||= log_process(process, sleep_time)
  end
  sleep sleep_time if need_to_sleep
end

def wait_for_command(pids)
  processes = @gom.processManager.ListProcessesInGuest(
    vm: @vim_vm, auth: @guest_auth, pids: pids
  )

  # wait for completion
  while processes.any? { |process| process.endTime.nil? }
    report_proccess(processes)
    processes = @gom.processManager.ListProcessesInGuest(
      vm: @vim_vm, auth: @guest_auth, pids: pids
    )
  end
  report_proccess(processes)
end

def process_command_success(processes)
  return unless processes.all? { |process| process.exitCode == 0 }
  @logger.log(:info, 'All processes completed successfully!')
end

def process_command_failure(processes)
  return unless processes.any? { |process| process.exitCode != 0 }
  processes.each do |process|
    @logger.log(:info, "'#{process.cmdLine}' finished with exitCode " \
                       "'#{process.exitCode}'")
  end
  fail 'One or more processes did not complete sucessfully.'
end

def process_command_results(pids)
  processes = @gom.processManager.ListProcessesInGuest(
    vm: @vim_vm, auth: @guest_auth, pids: pids
  )

  process_command_success(processes)
  process_command_failure(processes)
end

And to set it up:

platform = vm.platform
# Guest Operations Manager Setup
@gom = vim.serviceContent.guestOperationsManager
@gom_config = $evm.instantiate("/Configuration/GuestCredentials/#{platform}" \
                                                     "/#{cust_id}")
@guest_auth = construct_guest_auth

I hope that’s enough to get you going. We keep the to-be-inserted files on an NFS share that is mounted to all of the appliances for ease of management.

Matt

opsourcetrail · March 10, 2016, 7:59am

Hi Matt,
this is exactly what we need. Thanks a lot for help.

All the best,
Jan

ramrexx · March 10, 2016, 4:26pm

I have been giving this a lot of thought and wanted to share what i came up with.

If only we were so lucky to have VMware tools support cloud-init user-data injection! There is a way to accomplish this today but it is “wonky”. You basically have to (at a high-level):

Source: Datasources — cloud-init 22.3 documentation

install and configure cloud-init on your template
create a cloud-init script it MUST be called ‘user-data.txt’
create an ISO with the created ‘user-data.txt’
after cloning the vm, ensure that the ISO is mounted
in the guest on bootup cloud-init will look at the mounted cdrom ISO for the user-data.txt script and begin customization.

Here is what i propose:

If we could get the ability to mount and unmount ISO’s built into the product with simple and reliable vm methods (i.e. ‘vm.mount_cdrom(’', ‘path-to-datastore-iso’ and ‘vm.unmount_cdrom(’) to mount and unmount ISO’s ).
Add the mounting and unmounting of ISO’s to the VmReconfigureRequest state machine so that it follows miq_request approvals, etc… and the task.
For provisioning this approach seems the most logical and is probably the correct way to do it. Leverage the “Infrastructure / PXE / ISO Datastores” to catalog VMware datastores and be able to identify which ISO’s are associated with a system image type. This way an ISO could be loaded when the VM first boots very much like MIQ/CFME already does with RHEV/oVIRT. Then we can drive provisioning based on dialogs to choose which ISO to mount (not boot from, just mount) during VM provisioning to load the correct user-dat.txt file. Bottom line: Most of the backend logic and UI is already there. It would just need to be extended to support VMware mounting of ISO’s during provisioning.

james_ · April 26, 2016, 1:15pm

I am trying to do just this. We have been told that RedHat do not support cloud init on vmware yet as this shows. Instead I have been trying to run an arbitrary command on boot up and Have got a little way down the rabbit hole.

From and irb session I get:

irb(main):043:0> f=Fog::Compute::Vsphere.new( :vsphere_username => vmware.authentication_userid  , :vsphere_password=> vmware.authentication_password, :vsphere_server => vmware.hostname)
LoadError: cannot load such file -- rbvmomi
    from /opt/rh/cfme-gemset/gems/activesupport-4.2.5.1/lib/active_support/dependencies.rb:274:in `require'

From the above I can see that you are using:

prog_spec = RbVmomi::VIM::GuestProgramSpec

Should I log a ticket about RbVmomi to RedHat ?

james_ · April 28, 2016, 9:30am

Replying to myself to log Redhat’s recommendation which for running arbitrary commands is to use vmrun.

I have installed Vix from Home - VMware {code} ( note that SDK 1.12, Released 24 SEP 2012 is the last release to support vcenter). After that /usr/lib/vmware-vix/Workstation-9.0.0-and-vSphere-5.1.0/64bit needs to be linked to /usr/lib/vmware.

furlongm · December 22, 2016, 5:30am

Has any tried this → GitHub - xing/cloudinit-vmware-guestinfo