VM provisioning failing - Mac Address Pool Empty?

Support
#1

Help

I am having trouble provisioning VMs it fails at the acquire MAC address part of the provisioning either manual or catalog item initiated.

I can provision VMs manually going straight to Vcenter, but cloudforms seems to think that there are no more MAC addresses.

I have

  1. Cleaned up any orphan VMs in the infrastructure
  2. Refreshed Inventory and power states from Providers
  3. Tried provisioning both methods of direct infrastructure pro

I have to admit I have been provisioning tons of VMs, but the fact that Vcenter can still provision VMs directly, leads me to some value being set on the provider that needs to be refreshed.

Heres my log output:

[----] I, [2019-09-26T08:11:47.616712 #9246:3a8f54] INFO – : MIQ(MiqReport.sync_from_file) Report: [VMs by MAC Address] file exists, synchronizing with model
[----] I, [2019-09-26T08:12:41.029254 #9246:3a8f54] INFO – : MIQ(MiqServer.start) Server MAC Address: 00:50:56:b0:be:65
- MAC_ADDRESS_IS_IN_USE
- MAC_POOL_EMPTY

#2

I think those log entries are possibly misleading. If you open evm.log and search for ‘MAC’ then you’ll see that

  • MAC_ADDRESS_IS_IN_USE
    and
  • MAC_POOL_EMPTY

are listed as 2 possible status types that are defined in the VMDB. Similarly the “VMs by MAC Address” is one of the built-in reports. These aren’t errors.

AcquireMACAddress is normally an empty state (ie does nothing) in the out-of-the-box VM Provision state machine. Have you defined your own method to run here? Otherwise what indications do you have that this is the state that’s failing?

Have a search for ‘ERROR’ in automation.log and see if you can narrow down where the problem might be.

Hope this helps,
pemcg

#3

Thank you. I’ll look into it

#4

I believe I found the issue. It has do with the way I am calling upon my custom Method in the VM Provisioning schema. I think I need some direction on how to properly call a custom method . It seems that even though I am putting the Path to the Instance - it is looking for it in the namespace that is in use when it is going through the states.

In this example, I simply trying to call the Instance /System/Request/InspectMe which lives in the in the ManageIQ domain. You can see it is looking for it in the log output.

The error spawned from me calling Instances to my custom methods. When I did this I would get a failure on the previous state. Which in this case was the aquire mac address.

image
image.png2293×778 94.9 KB

image
image.png2253×65 9.85 KB

I think copying the method and instance in my custom datastore and provisioning namespace is counter-intuitive in that I would have redundant copies of the same method. Which leads me to think that I am probably calling it incorrectly. Rather than copy a bunch of methods I would rather see what the recommended way to do this should be.

Tried it using “state” as well:
I tried it another way using a “State” type of call instead of a method call. When I change it from a method call to a “state” call in the schema for my functions, it seems to repeat over and over because I dont think its getting the proper return code saying everything executed correctly. I do see one of my methods (An ansible playbook) execute over and over because the state that is called from keeps trying again until all the retries are exhausted. Even though the method worked like it was supposed to.

Long story short. I think I need some guidance on how to properly call a custom method in a schema such as the Infrastructure VM provisioning Schema. If you can point me to the proper documentation or give me some tips here I would greatly appreciate it!

Thanks in advance!

#5

Hmm…

Regarding calling my methods using the “State” calls.
I found this "State Retries"reference https://manageiq.gitbook.io/mastering-cloudforms-automation-addendum/embedded_ansible/chapter-6#state-machine-retries

which states that in a state machine an inline ansible playbook will keep retrying. I’m guessing I have to call the on_exit method somehow to stop it from retrying? How exactly do you call this “on_exit” method inside the playbook ? This might be the missing piece I need.

Many thanks!

#6

ok, there are several points to mention here. The first is that (as you discovered), all of the schema entries in a state machine should be of type “State”. This is so that if any state in the state machine retries, after the delay the state machine will jump straight to the retrying state, bypassing all previous states. I suspect this is why your method call is failing.

The error notification popup that you’re seeing is from the ‘checkprovisioned’ state in the service provision state machine, as the failure in the VM provision state machine has been propagated up to this state machine.

To correctly call your own method from a state machine you should use a State type of entry, and call your own instance, which itself calls your method (you can directly call the method from the state machine, but using an instance gives you more flexibility).

Regarding calling Ansible playbooks from a state machine; yes the playbook’s state is put into an immediate retry condition, but when the playbook completes the state machine continues.

There’s a description of how to edit the VM Provision state machine to add your own customised stages here if it helps.

Hope this helps,
pemcg

#7

So, I changed all in the schema to a “State” type and I am calling the Instance which calls on my playbook. This playbook is designed to run on a remote windows which run a powershell script.

However, what happens is the playbook retries the playbook until all retries are spent and then basically terminates the state machine saying it failed. However, the playbook successfully ran.

Should I consider trying to use a ruby method to then call the playbook inside of it to prevent the retry from occuring?

#8

Found my issue.

I had defined my method and instance inside of the same namespace, class and schema as the state machine. I think my instance being in there must have caused some kind of conflict where it was expecting some return code that never happened.

After making my own custom namespace, class and instance for the playbook- it ran just fine.

-Ben