AWS API optimization


#1

Hi,

We have 18 AWS providers configured and about 800 EC2 instances and about 140 load balancers.
For some days we are getting Aws::EC2::Errors::RequestLimitExceeded: Request limit exceeded error for refreshing instances, or when doing an action, etc.

As AWS API request limit is 10000/sec, and we have some other applications doing requests, when manageiq is doing refresh+ some actions, we are reaching this limit. Is there a way to optimize AWS API call? like instead of refreshing each instance, doing a refresh by batch?

Thanks,


#2

@LorkScorguar what version of ManageIQ do you have?


#3

I have ManageIQ Gaprindashvili and Cloudforms 5.8.3.4 (so based on fine)


#4

@LorkScorguar So in Gaprindashvili, we are doing targeted refresh of specific entities based on events. So we fetch it from the API, only if those entities changed. And if more entities changed, we fetch them all at once (e.g. if 5 Vms changed, we fetch all 5 Vms by 1 filtered API query)

For Fine release, on each event we refresh the whole inventory. So fetching everything from the API every-time.

Can you say which Managers are causing the API bloat? Or what API queries are the most called?


For Gaprindashvili, the managers are doing full refresh every 2 hours, which is driven by option restart_interval: 2.hours, in advanced settings. Targeted refresh should be able to fetch all needed info, so you can bump that up to 24.hours, or something like that.


Also, do all 18 providers share the limit? Is it under 1 region? Do all providers receive the same events?


#5

@Ladas thanks for all those informations, I will increase full refresh interval for Gaprindashvili and will wait for us to upgrade our cloudforms to the latest version to get the same.
On our 18 providers, we have 5 differents accounts, I think we only reach the limit on our main account on main AWS region, nearly all resources are here.
I think Gaprindashvili changes will be enough for us.