Quotas does not properly checked for Service Provision and not multiplied by number_of_vms for cloud instances provisioning

euwe

#1

Hi people.
The requested.rb method have poor quality.
The dialog_values function always return zero because dialog for service does not have any values that does not prefixed by “dialog_”. In this way any value that passed to provision workflow such as vm_memory, number_of_cpus etc. (with names dialog_vm_memory and dialog_number_of_cpus) does not checked by quotas workflow.

Moreover this method does not check resources for all provisioned vms. Memory, storage and cpus does not multiplied by number_of_vms value.

For cloud instances this method does not check requested cloud volumes in storage array, only flavor root disk.

All these drawbacks lead to the fact that the quota is only checked for the service template. Users can exceed the quota. And quota report can have this values :joy:


#2

Hi igortiunov,

I’m sorry that quota is not working properly for you.

We’ve had some challenges keeping up with all of the different Service and VM provisioning types.

There are 11 different types of Service Catalog items and VM provisioning requests for multiple cloud and infrastructure providers.

Answering the question as to what are the requested resources for a provisioning item for a given provider is challenging in the current implementation.

Our goal is to change the way we’re calculating the requested values for quota in the following ways:

  1. Create model methods for each provider to get the requested resources for VMs and Service provisioning requests.

  2. Rewrite the requested method to use the provider methods,

  3. Enhance quota to support in flight provisioning requests.

Our current implementation for Services is that all Service dialog values are prefixed with “dialog_”. The dialog parsing and Service state machine rely heavily on this convention. In keeping with this, the quota requested method looks for the “dialog_” prefixed items.

The number of vms is considered for calculating the requested_memory, number_of_cpus, and requested storage. The vaue for each requested item is multiplied by the number_of_vms in the request. I’d be happy to look into it if this isn’t working properly for you.

Thanks for bringing to our attention that cloud volumes is missing in quota calculations. I’ve opened a ticket on the issue: https://bugzilla.redhat.com/show_bug.cgi?id=1455349

Please let me know if there is anything I can help you with.

Regards,

Tina


#3

Hello @tinaafitz !
Thank you for your reply.

First of all for service provisioning options still not properly checked. You can analyze that “dialog_” prefixed options are collected to related options_hash:

https://github.com/ManageIQ/manageiq/blob/euwe/db/fixtures/ae_datastore/ManageIQ/System/CommonMethods/QuotaMethods.class/methods/requested.rb#L230

but values of this hash never accessed by its “dialog_” prefixed names:

https://github.com/ManageIQ/manageiq/blob/euwe/db/fixtures/ae_datastore/ManageIQ/System/CommonMethods/QuotaMethods.class/methods/requested.rb#L153

Second unresolved drawback is for number_of_vms for cloud instances provisioning. As you can see from my logs:

<AutomationEngine> <AEMethod requested> Request: Provision from [Red Hat Enterprise Linux 7] to [server###] id: 148 
<AutomationEngine> <AEMethod requested> Retrieving flavor t1.2medium memory => 4294967296
<AutomationEngine> <AEMethod requested> Retrieving cloud flavor t1.2medium cpus => 4

<AEMethod validate_quota> Request: Provision from [Red Hat Enterprise Linux 7] to [server###] id: 148 
<AEMethod validate_quota> quota_warning: {:cpu=>0, :memory=>0, :storage=>0, :vms=>0}
<AEMethod validate_quota> quota_limits: {:memory=>85899345920, :vms=>20, :storage=>3324304687104, :cpu=>46}
<AEMethod validate_quota> Item: storage Used: (1715839434752) Requested: (68719476736) Max: (3324304687104) Warn: (0)
<AEMethod validate_quota> Item: vms Used: (6) Requested: (2) Max: (20) Warn: (0)
<AEMethod validate_quota> Item: cpu Used: (32) Requested: (4) Max: (46) Warn: (0)
<AEMethod validate_quota> Item: memory Used: (51136954368) Requested: (4294967296) Max: (85899345920) Warn: (0)

As you can see, I requested two instances with flavor t1.2medium (t1.2medium cpus => 4, t1.2medium memory => 4294967296 and 60Gb root disk). But quota calculated only for one instance.

You can found that for Infrastructure Providers there is correct calculation for number of vms:

for memory:
https://github.com/ManageIQ/manageiq/blob/euwe/db/fixtures/ae_datastore/ManageIQ/System/CommonMethods/QuotaMethods.class/methods/requested.rb#L106

and for cpus:
https://github.com/ManageIQ/manageiq/blob/euwe/db/fixtures/ae_datastore/ManageIQ/System/CommonMethods/QuotaMethods.class/methods/requested.rb#L114

But for cloud instances there is no multiplying by the number of provisioned instances. There is my override to workaround all these issues:

There is another major architectural issue that I would like to draw special attention - Requested resources does not allocated before provisioning and quota can be exeeded during simultaneous requests.
Quotas are fixed only after provisioning of resources. In this case a large number of simultaneous requests can significantly exceed the allocated quotas. A relatively small number of users (i.e. one hundred) simultaneously or with some short difference in time can execute a request to provision one small virtual machine. The quota validation workflow checks each request individually and each small request passes a quota check, although all requests in aggregate can significantly exceed the limits.

I can say that to reproduce this problem you can use two users. Each user executes the provisioning request. Second user should send a request before virtual machines of first user are provisioned.


#4

Hi @tinaafitz Please note my comments.


#5

For service provisioning request rised https://bugzilla.redhat.com/show_bug.cgi?id=1455844 through RH Support.


#6

Hi igortiunov,

Thank you for bringing these issues to our attention and for including your code showing a workaround.
Our current quota implementation doesn’t consider simultaneous provisioning requests and quota limits can be significantly exceeded as a result. We realize this is an issue and I hope to start working on the changes very soon.

I’m on vacation until May 30th, and will be working on quota next week.

Regards,
Tina