Requested resources does not allocated before provisioning and quota can be exeeded during simultaneous (or with some short difference in time) requests

automate

#1

There is another major architectural issue that I would like to draw special attention - Requested resources does not allocated before provisioning and quota can be exeeded during requests with some short difference in time.

Quotas are fixed only after provisioning of resources. In this case a large number of simultaneous requests can significantly exceed the allocated quotas. A relatively small number of users (i.e. one hundred) simultaneously or with some short difference in time can execute a request to provision one small virtual machine.

The quota validation workflow checks each request individually and each small request passes a quota check, although all requests in aggregate can significantly exceed the limits.

How to reproduce:

I can say that to reproduce this problem you can use two users. Each user executes the provisioning request. Second user should send a request before virtual machines of first user are provisioned.


#2

@tinaafitz
There is BZ for this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1456819

and screencast that describe it:

From this screencast second user sends a request before virtual machines of first user are provisioned.


#3

Hi @igortiunov,

Thanks for the screencast showing the quota issue.
I started working on the issue yesterday.

Regards,
Tina


#4

@tinaafitz

Hi Tina

Thank you for your work for issues about quotas calculation but the problem has not been completely solved yet. There is race condition between Provider Inventory and quota check. EMS Refresh process can take 20-60 minutes to parse provider inventory but cloud instance creation process takes 30-60 seconds. In this way user can create too many instances on the same time but CFME will not known about these instances until EMS Refresh worker end. During this time period the quota can be significantly exceeded.