Enhancements to the MIQ internal IPAM system


#1

Cox Automotive (AKA: AutoTrader) has implemented many new features in the ManageIQ internal IP Address Management (IPAM) system. These updates are on the CloudForms v3.0 code base but could included in the Botvinnik release. A robust IPAM system is a core component of push-button automation. The current ManageIQ IPAM code has been extended to support many common business needs and is currently being used in Cox Automotive production workflows.

State of current development:
Cox Automotive (AKA: AutoTrader) personnel have completed the work in their local production environment for CloudForms v3.0. Integration into the ManageIQ codebase has not been done.

Who is doing it?:
JD Calder of Cox Automotive is the primary developer

Target release and timeline:
Could be implemented in the Botvinnik release

Description:
We have extended the current IPAM feature set to include:
• Updated schema/code to include support for dual NICs
• Implemented a reservation token to avoid the race situation of issuing same IP to multiple requests
• Use of tags to acquire and release IPs from correct IPAM
• Code reuse by having all automation workflows call a single method to acquire or release an IP
• Release of IP if automation workflow fails after it has reserved an IP
• Email and log warning messages if IPAM is getting low on IPs
• Email and log error message if IPAM has run out of IPs
• Write IP reservations counts to the automation log which is reportable via Splunk/ELK/Log Insight
• Created a tool set to convert IPAM automation XML to a CSV and another script to convert the CSV to a properly formatted MIQ Automation XML file

More details on the Reservation Token
We have created serveral push button automation workflows where the user can request multiple VMs with a single request. The challenge we faced was in a distributed application such as MIQ, multiple workers were attempting to acquire and IP/Hostname at the same time (or a few miliseconds different). Just flipping the “inuse” flag was not sufficient to ensure each request received a unique IP.

The normal flow is

  1. Worker calls to acquire an IP address
  2. IPAM_Acquire_from_MIQ method calls to get the IPAM DB Class and all the instances
  3. Method sorts the instances and deletes any that have the inuse flag set to true
  4. Method gets the first available instance, sets its inuse flag to true
  5. Method returns the IP instance to the worker

The solution was to create a reserve token field in the IPAM schema. When a worker requests an IP, it uses the MIQ request ID as a unique token and sets this in the reserve_token field. The reserve_token is validated prior to returning the IP to the worker. So if multiple requests acquire the same IP instance, the last one to write to the IPAM DB will have a validated reserve_token and wins the IP.

The Reserve Token flow is

  1. Worker calls to acquire an IP address
  2. IPAM_Acquire_from_MIQ method calls to get the IPAM DB Class and all the instances
  3. Method sorts the instances and deletes any that have the inuse flag set to true
  4. Method gets the first available instance, sets its inuse flag to true and sets the reserve_token field with the MIQ request ID
  5. Method calls again to the IPAM DB Class to get the instance it just reserved and confirms it’s request ID matches the reserve_token field
    a. If they don’t match, return to step #2 and repeat the process to get a IP reserved
    b. If they do match, method returns the IP instance to the worker

Core feature or independent project?:
This could be treated as an independent project as we have not altered the current IPAM methods.

Dependencies:
There are no dependancies, no new libraries or changes to MIQ core code


#2

There a couple of points to consider for the MIQ projects starting with Anand release

  • Domains have been added as a parent to Namespaces.

  • The import/export format has been changed from XML to YAML. So the XML file would have to be converted to YAML before we import the data. During the conversion and import/export the domain has to be specified.

  • The automate IPAM method would to be aware of the domain name when it searches for instances in the Automate DB.

  • There are certain domains which are not writable so the IPAM instances would have to be in an editable domain.


#3

I’ve implemented this with Cloudforms 4.1, but ran into a permissions issue, possibly just something I don’t understand yet. What I’ve found is that members of a group that are in a Child Tenant OR a Project under the main Tenant do not have the ability to modify the instances and set the reservation flags, etc. The code domain is locked for anyone except members of the main Tenant only. Is this by design? Am I missing any sort of elevated rights mechanism or domain security?