Openstack Instances zero

Using CloudForms 5.0 with OpenStack 13.
The Cloud Manager is using:

  • Keystone v3
  • tenant mapping enabled
  • default zone
  • The endpoint credentials validate successfully
  • admin user is used and a member of all projects.
  • The infrastructure provider is specified.

What would cause a total timeout of the Cloud Provider?

The CloudManager Refresh Workers are timing out at 7200 seconds.
There is a firewall in between but ports seem open.
A curl command to get a token and then GET /v2.1/servers/detail returns an empty set.
{servers:}

Later we realize that the all_tenant API option was not specified.
But the response shows the firewall isn’t stopping the response.
curl -k -g -i -X GET https://OPENSTACK_DOMAIN:13774/v2.1/servers/detail -H “Accept: application/json” -H “X-Auth-Token: MYTOKEN” > Server-details.txt

Anything else to test or check?

Update:

  • There is no firewall rules blocking between CloudForms and OSP.
  • The curl command for token request doesn’t specify scope. But that isn’t how the OSP plugin creates API calls.

Update:

  • There is no other error in the fog.log (debug level).
  • Could this be a plain timeout?
    The OSP environment has over 2,000 Instances
    How can the refresh timeout be increased from 7200?

Hey @bkinney

How can the refresh timeout be increased from 7200?

That is controlled by the queue_timeout setting in the ems_refresh_worker section. Look for the ems_refresh_worker_openstack section and add:
Screenshot from 2021-12-10 09-41-33

That’s still a very long time even for 2000 instances, I wonder if API calls are timing out and retrying. During a refresh can you tell from the fog.log if the API calls are succeeding but the number of pages of instances is just enormous, or are they not returning at all.

The nova-api.log has successful entries, 200, for /server/details? request from CloudForms.
The only error is the timeout in the evm.log.
No errors in fog.log file that ran for more than 3 hours in debug mode.

This leads me to think that the Projects/Tenants in OpenStack aren’t allowed to use Nova API? The policy on OpenStack must have been modified?

There is a Load Balancer in between CloudForms and OpenStack.
Customer is unclear why the session affinity needs to be enabled.
Is this still required in version 5?