Hello,
I am looking for best approach how to provision linux (RHEL 7, CentOS 7) virtual machine on VMware (vSphere 6) infrastructure provider, including some customization action.
So I am wondering if it is possible to use cloud-init.
Thanks in advance for any advice.
Vaclav
Hi @vmi,
ManageIQ supports VMware’s Customization Specifications for linux. That would be the best place to start. It does not support Cloud-Init on VMware, but does for other providers.
Hi @gmccullough,
Through VMware customization specification I am able to define networking basics as IP address, hostname and DNS records…
But how can I modify other stuff as setting passwords and keys, adding sw packages including basic configuration of deployed sw? PXE is not suitable in my case.
Kind regards,
Vaclav
Out of curiosity, what would be required to get CloudInit to work on VMware? Could be a nice community project for someone.
Having CloudInit on VMware would be cool as it would allow you to employ the same way of customization across VMware, RHEV, OpenStack and AWS.
I agree that having CloudInit would be very useful here. To support it we would need to determine how to pass the CloudInit script to VMware as part of the clone API calls. From there the effort would be to include CloudInit as an option on the Customize tab in the provision dialogs.
This is typically where configuration management comes into play. After a VM is provisioned you may choose to use Chef, Puppet and ofcourse Ansible to finish the guest configuration. If those options are not available, you are in luck as the MIQ appliance ships with an SSH ruby gem that makes this a breeze.
See the ruby sample method below:
#!/usr/bin/env ruby
require 'net/ssh’
new_size = 20
new_disk = "/dev/sdd"
volume_group = "centos"
volume_group_path = “/dev/#{volume_group}/root”
ssh = Net::SSH.start( ‘10.10.10.10’, ‘root’, :password => ‘evm123’ )
ssh.exec!(“for i in ls /sys/class/scsi_host/|awk {'print $1'}; do echo ‘- - -’ > /sys/class/scsi_host/$i/scan; done”)
ssh.exec!(“ssm add -p #{volume_group} #{new_disk}”)
ssh.exec!(“ssm resize -s+#{new_size}GB #{volume_group_path}”)
ssh.close
@gmccullough - If you guys are serious here is a sample method that someone wrote that writes an ISO per VM and mounts it to the guest. there is quite a bit of orchestration here but it is possible.
###################################
begin
@method = ‘Reconfigure_CDROM’
$evm.log(“info”, “#{@method} - EVM Automate Method Started”)
@debug = true
###################################
###################################
def reconfigure_cdrom( vm, operation, datastore, filename )
# Build the Perl command using the VMDB information
cmd = "perl /usr/lib/vmware-vcli/apps/vm/vmISOManagement.pl"
cmd += " --server #{vm.ext_management_system.ipaddress}"
cmd += " --username “#{vm.ext_management_system.authentication_userid}”"
cmd += " --password “#{vm.ext_management_system.authentication_password}”"
cmd += " --vmname “#{vm.name}”"
cmd += " --operation “#{operation}”"
if operation == 'mount’
cmd += " --datastore “#{datastore}”"
cmd += " --filename “#{filename}”"
end
$evm.log(“info”, “Running: #{cmd}”)
results = system(cmd)
return results
end
###################################
###################################
def boolean(string)
return true if string == true || string =~ (/(true|t|yes|y|1)/i)
return false if string == false || string.nil? || string =~ (/(false|f|no|n|0)/i)
end
$evm.log(“info”, “#{@method} - EVM Automate Method Started”)
service_template_provision_task = $evm.root[‘service_template_provision_task’]
service_template_provision_task.miq_request_tasks.each do |child_miq_request_task|
child_miq_request_task.miq_request_tasks.each do |grandchild_miq_request_task|
vm = grandchild_miq_request_task.destination
raise “#{@method} - VM object not found” if vm.nil?
# Get root attributes passed in from the service dialog
#operation = $evm.root['operation'] || 'umount'
operation = 'mount'
#datastore = $evm.root['datastore'] || nil
datastore = 'cloudinit-nfs'
#filename = $evm.root['filename'] || nil
filename = 'ipxe/customization/cloud-init.iso'
$evm.log("info", "#{@method} - Detected Operation:<#{operation}> for VM:<#{vm.name}> Datastore:<#{datastore}> Filename:<#{filename}>")
results = reconfigure_cdrom( vm, operation, datastore, filename )
if results
$evm.log("info", "#{@method} - VM Reconfigure of CD/DVD Successful:<#{results.inspect}>")
if operation == 'mount'
vm.custom_set(:DVDROM, filename)
vm.start
else
vm.custom_set(:DVDROM, nil)
end
else
raise "#{@method} - VM Reconfigure of CD/DVD Failed:<#{results.inspect}>"
end
end
end
$evm.log(“info”, “#{@method} - EVM Automate Method Ended”)
exit MIQ_OK
We are using the VIX API to pass scripts/calls through to the VM via VMware Tools.
There can either be a script on the VMware Template that is called by VIX with the appropriate arguments or VIX can be used to inject a script that is executed. We have a simple script for setting the password:
#!/bin/sh
#
# Usage: set_local_password.sh username password
if [ $# -ne 2 ]; then
echo "Usage: $0 username password"
exit 1
fi
echo -e "$2\n$2" | sudo passwd $1
The automate code is a bit convoluted to get it loaded up into the running VM via the tools, but if that would be helpful I can try and pare it down. After the script above is injected, we run the following:
@logger.log(:info, "guest_os: #{guest_os}")
# Determine the username we want to set
case
when guest_os.match(/ubuntu/)
username = 'admin'
else
username = 'root'
end
# Get the password from the provision dialog
password = vm.miq_provision.options[:root_password]
# Determine command to run in guest
guest_program_path, guest_args =
get_guest_command(platform, guest_file, username, password)
@logger.log(:info, "Attempting to run: '#{guest_program_path} " \
"#{guest_args}' as '#{@guest_auth[:username]}'")
# Run the command in the guest
pids = get_program_pids(guest_program_path, guest_args, guest_directory)
# Wait for the command to complete
wait_for_command(pids)
# Cleanup and process results
process_command_results(pids)
@logger.log(:info, "Deleting '#{guest_file}' " \
"from '#{vm.name}'")
@gom.fileManager.DeleteFileInGuest(vm: @vim_vm, auth: @guest_auth,
filePath: guest_file)
@logger.log(:info, 'set_root_password Method Ended')
exit MIQ_OK
Hope that helps!
Matt
Hi Matt,
this really seems interesting. Would it be possible to share how the get_program_pids method looks like? It would be helpful to see how to work with the VIX API.
Thanks a lot for the tip.
All the best,
Jan
It’s not pretty, but here it is. I think this is where the “methods should be 10 lines or less” rule makes things a little more convoluted than it could be.
def construct_guest_auth
username = @gom_config['username']
password = @gom_config.decrypt('password')
fail 'guest username & password are required!' unless username && password
guest_auth = RbVmomi::VIM::NamePasswordAuthentication(
interactiveSession: false,
username: username,
password: password
)
guest_auth
end
def get_guest_temp_file(filename)
temp_file_suffix = File.extname(filename)
temp_file_prefix = File.basename(filename, temp_file_suffix) + '_'
guest_file = @gom.fileManager.CreateTemporaryFileInGuest(
vm: @vim_vm,
auth: @guest_auth,
prefix: temp_file_prefix,
suffix: temp_file_suffix
)
fail 'unable to obtain a temp guest file' if guest_file.nil?
guest_file
end
def use_sudo
if @guest_auth[:username] == 'root'
false
elsif $evm.object['use_sudo'].nil?
@gom_config['use_sudo']
else
$evm.object['use_sudo']
end
end
def linux_program_path(interpreter)
sudo_path = $evm.object['sudo_path']
use_sudo ? sudo_path : interpreter
end
def get_guest_program_path(platform, interpreter)
guest_program_path =
if platform == 'linux'
linux_program_path(interpreter)
elsif platform == 'windows'
interpreter
else
fail 'only linux & windows are supported'
end
guest_program_path
end
def get_linux_program_args(guest_file, interpreter, interpreter_args, username, password)
guest_args =
if use_sudo
"#{interpreter} #{interpreter_args} #{guest_file} #{username} #{password}"
else
"#{interpreter_args} #{guest_file} #{username} #{password}"
end
guest_args
end
def get_guest_args(platform, guest_file, interpreter, interpreter_args, username, password)
guest_args =
if platform == 'linux'
get_linux_program_args(guest_file, interpreter, interpreter_args, username, password)
else
fail 'only linux supported'
end
guest_args
end
def get_guest_command(platform, guest_file, username, password)
interpreter = $evm.object["#{platform}_interpreter"]
interpreter_args = $evm.object["#{platform}_interpreter_args"]
guest_program_path = get_guest_program_path(platform, interpreter)
guest_args = get_guest_args(platform, guest_file, interpreter,
interpreter_args, username, password)
[guest_program_path, guest_args]
end
def get_file_attrs(platform)
# file will be created with 0644 permissions
file_attrs =
if platform == 'linux'
RbVmomi::VIM::GuestPosixFileAttributes()
elsif platform == 'windows'
RbVmomi::VIM::GuestWindowsFileAttributes()
else
fail 'only linux & windows are supported'
end
file_attrs
end
def get_put_path(guest_file, platform, file_size)
file_attrs = get_file_attrs(platform)
put_path = @gom.fileManager.InitiateFileTransferToGuest(
vm: @vim_vm, auth: @guest_auth, guestFilePath: guest_file,
fileAttributes: file_attrs, fileSize: file_size, overwrite: true
)
put_path
end
def construct_request(request_uri, host_file, file_size)
request = Net::HTTP::Put.new(request_uri)
request.body_stream = File.open(host_file)
request['Content-Type'] = 'multipart/form-data'
request.add_field('Content-Length', file_size)
request
end
def put_file(host_file, guest_file, platform)
file_size = File.size(host_file)
put_path = get_put_path(guest_file, platform, file_size)
uri = URI.parse(put_path)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = (uri.scheme == 'https')
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = construct_request(uri.request_uri, host_file, file_size)
http_response = http.request(request)
http_response
end
def process_http_response(http_response)
if http_response.code == '200'
@logger.log(:info, http_response.body)
else
fail "The transfer of '#{host_file}' to '#{guest_file}' FAILED!"
end
end
def get_program_pids(guest_program_path, guest_args, guest_directory)
prog_spec = RbVmomi::VIM::GuestProgramSpec(
programPath: guest_program_path,
arguments: guest_args,
workingDirectory: guest_directory
)
pids = []
pids << @gom.processManager.StartProgramInGuest(
vm: @vim_vm, auth: @guest_auth, spec: prog_spec
)
pids
end
def log_process(process, sleep_time)
if process.endTime.nil?
@logger.log(:info, "'#{process.cmdLine}' is still running, sleeping " \
"for #{sleep_time} seconds")
return true
else
@logger.log(:info, "'#{process.cmdLine}' has finished running")
return false
end
end
def report_proccess(processes)
need_to_sleep = false
sleep_time = 15.seconds
processes.each do |process|
need_to_sleep ||= log_process(process, sleep_time)
end
sleep sleep_time if need_to_sleep
end
def wait_for_command(pids)
processes = @gom.processManager.ListProcessesInGuest(
vm: @vim_vm, auth: @guest_auth, pids: pids
)
# wait for completion
while processes.any? { |process| process.endTime.nil? }
report_proccess(processes)
processes = @gom.processManager.ListProcessesInGuest(
vm: @vim_vm, auth: @guest_auth, pids: pids
)
end
report_proccess(processes)
end
def process_command_success(processes)
return unless processes.all? { |process| process.exitCode == 0 }
@logger.log(:info, 'All processes completed successfully!')
end
def process_command_failure(processes)
return unless processes.any? { |process| process.exitCode != 0 }
processes.each do |process|
@logger.log(:info, "'#{process.cmdLine}' finished with exitCode " \
"'#{process.exitCode}'")
end
fail 'One or more processes did not complete sucessfully.'
end
def process_command_results(pids)
processes = @gom.processManager.ListProcessesInGuest(
vm: @vim_vm, auth: @guest_auth, pids: pids
)
process_command_success(processes)
process_command_failure(processes)
end
And to set it up:
platform = vm.platform
# Guest Operations Manager Setup
@gom = vim.serviceContent.guestOperationsManager
@gom_config = $evm.instantiate("/Configuration/GuestCredentials/#{platform}" \
"/#{cust_id}")
@guest_auth = construct_guest_auth
I hope that’s enough to get you going. We keep the to-be-inserted files on an NFS share that is mounted to all of the appliances for ease of management.
Matt
Hi Matt,
this is exactly what we need. Thanks a lot for help.
All the best,
Jan
I have been giving this a lot of thought and wanted to share what i came up with.
If only we were so lucky to have VMware tools support cloud-init user-data injection! There is a way to accomplish this today but it is “wonky”. You basically have to (at a high-level):
Source: http://cloudinit.readthedocs.org/en/latest/topics/datasources.html
Here is what i propose:
If we could get the ability to mount and unmount ISO’s built into the product with simple and reliable vm methods (i.e. ‘vm.mount_cdrom(’’, ‘path-to-datastore-iso’ and ‘vm.unmount_cdrom(’) to mount and unmount ISO’s ).
Add the mounting and unmounting of ISO’s to the VmReconfigureRequest state machine so that it follows miq_request approvals, etc… and the task.
For provisioning this approach seems the most logical and is probably the correct way to do it. Leverage the “Infrastructure / PXE / ISO Datastores” to catalog VMware datastores and be able to identify which ISO’s are associated with a system image type. This way an ISO could be loaded when the VM first boots very much like MIQ/CFME already does with RHEV/oVIRT. Then we can drive provisioning based on dialogs to choose which ISO to mount (not boot from, just mount) during VM provisioning to load the correct user-dat.txt file. Bottom line: Most of the backend logic and UI is already there. It would just need to be extended to support VMware mounting of ISO’s during provisioning.
I am trying to do just this. We have been told that RedHat do not support cloud init on vmware yet as this shows. Instead I have been trying to run an arbitrary command on boot up and Have got a little way down the rabbit hole.
From and irb session I get:
irb(main):043:0> f=Fog::Compute::Vsphere.new( :vsphere_username => vmware.authentication_userid , :vsphere_password=> vmware.authentication_password, :vsphere_server => vmware.hostname)
LoadError: cannot load such file -- rbvmomi
from /opt/rh/cfme-gemset/gems/activesupport-4.2.5.1/lib/active_support/dependencies.rb:274:in `require'
From the above I can see that you are using:
prog_spec = RbVmomi::VIM::GuestProgramSpec
Should I log a ticket about RbVmomi to RedHat ?
Replying to myself to log Redhat’s recommendation which for running arbitrary commands is to use vmrun.
I have installed Vix from https://www.vmware.com/support/developer/vix-api/ ( note that SDK 1.12, Released 24 SEP 2012 is the last release to support vcenter). After that /usr/lib/vmware-vix/Workstation-9.0.0-and-vSphere-5.1.0/64bit needs to be linked to /usr/lib/vmware.