Service provisioning does not initiate quota validation and hangs at "pending"

I’ve run into an issue that occurs once in a while where provision requests stay in Pending mode after being approved. The only way to clear the issue is to remove the Automate “QuotaStateMachine” class and instance/methods and remove the pending requests. Re-add quotastatemachine and Quotas seem to be happy again. Anyone run into this issue?

<AEMethod [/MYCOMPANY/System/CommonMethods/ Quota Methods/validate_ quota ]> Starting

Make sure that you have at least one appliance in each zone with the ‘Provider Operations’ server role enabled. Requests staying in a ‘pending’ state is a known ‘feature’ if this role is missing in any zone.

pemcg

Hello, Thank you for responding. “provider operations” is indeed enabled. The only way to resolve it is to delete all “pending” and disable “quota” validation. Issue happened again today and tracked these errors on a worker node that might shed some light

From automate.log

ERROR – : Q-task_id([r 1000000002699 service_template_provision_request 1000000002699 ]) The following error occurred during instance method <check_quota> for AR object <#<ServiceTemplateProvisionRequest id: 1000000002699

miq_ae_service_miq_provision_mixin.rb:42:in `check_quota’

<AEMethod [/comp/System/CommonMethods/QuotaMethods/used]> Ending

State= running raised exception: <Method exited with rc=Unknown RC: [1]>

Error in State=[used]

[----] E, [2019-08-19T14:17:29.802829 #17283:99cce40] ERROR – : Q-task_id([r1000000002699_service_template_provision_request_1000000002699]) Method STDERR: from /comp/System/CommonMethods/QuotaMethods/used:70:in `active_provision_counts’
[----] E, [2019-08-19T14:17:29.803053 #17283:99cce40] ERROR – : Q-task_id([r1000000002699_service_template_provision_request_1000000002699]) Method STDERR: from /comp/System/CommonMethods/QuotaMethods/used:31:in `used’
[----] E, [2019-08-19T14:17:29.803287 #17283:99cce40] ERROR – : Q-task_id([r1000000002699_service_template_provision_request_1000000002699]) Method STDERR: from /comp/System/CommonMethods/QuotaMethods/used:18:in `main’
[----] E, [2019-08-19T14:17:29.803531 #17283:99cce40] ERROR – : Q-task_id([r1000000002699_service_template_provision_request_1000000002699]) Method STDERR: from /comp/System/CommonMethods/QuotaMethods/used:89:in `’

From evm.log

MIQ(MiqAeEngine.deliver) Error delivering {:event_type=>“request_starting”, “EventStream::event_stream”=