Sporadic azure provider refresh issues

Question
#1

We have our miq running on an aks cluster. Everything has been fine for a time, however we are starting to refresh issues in some of our providers. For instance our non prod subscription in ncus has refresh errors and even when i attempt to manually start relationship and power state refresh nothing happens.
However in another region in the same subscription it runs fine. I have searched the evm log but do not see specific error popping out.

The only thing in the console that shows is |Success - About 6 Hours Ago|
|| Any help or suggestions would be greatly appreciated. Pulling my hair out with this intermittent issue.

Thank you

Jason

#2

For the details about refreshes grep the log/evm.log for ERR and paste here any refresh related errors you’ll see.

(the log from an appliance where the refresh worker is running for the affected regions)

#3

@djberg96 @agrare ?

#4

Sorry for dragging my feet work has been crazy.
[----] E, [2019-10-04T09:30:33.618830 #17:50ef60] ERROR – : /usr/local/lib/ruby/2.4.0/net/http.rb:906:in rescue in block in connect' /usr/local/lib/ruby/2.4.0/net/http.rb:903:inblock in connect’
/usr/local/lib/ruby/2.4.0/timeout.rb:93:in block in timeout' /usr/local/lib/ruby/2.4.0/timeout.rb:103:intimeout’
/usr/local/lib/ruby/2.4.0/net/http.rb:902:in connect' /usr/local/lib/ruby/2.4.0/net/http.rb:887:indo_start’
/usr/local/lib/ruby/2.4.0/net/http.rb:876:in start' /usr/local/lib/ruby/gems/2.4.0/gems/rest-client-2.0.2/lib/restclient/request.rb:715:intransmit’
/usr/local/lib/ruby/gems/2.4.0/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in execute' /usr/local/lib/ruby/gems/2.4.0/gems/rest-client-2.0.2/lib/restclient/request.rb:52:inexecute’
/usr/local/lib/ruby/gems/2.4.0/gems/azure-armrest-0.9.13/lib/azure/armrest/armrest_service.rb:212:in rest_execute' /usr/local/lib/ruby/gems/2.4.0/gems/azure-armrest-0.9.13/lib/azure/armrest/armrest_service.rb:301:inrest_execute’
/usr/local/lib/ruby/gems/2.4.0/gems/azure-armrest-0.9.13/lib/azure/armrest/armrest_service.rb:305:in rest_get' /usr/local/lib/ruby/gems/2.4.0/gems/azure-armrest-0.9.13/lib/azure/armrest/resource_provider_service.rb:70:inget’
/usr/local/lib/ruby/gems/2.4.0/gems/memoist-0.15.0/lib/memoist.rb:213:in get' /usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-46d1cb3a2f14/app/models/manageiq/providers/azure/cloud_manager.rb:45:inblock in insights?’
/var/www/miq/vmdb/app/models/ext_management_system.rb:413:in with_provider_connection' /usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-46d1cb3a2f14/app/models/manageiq/providers/azure/cloud_manager.rb:43:ininsights?’
/usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-46d1cb3a2f14/app/models/manageiq/providers/azure/cloud_manager.rb:30:in block in <class:CloudManager>' /var/www/miq/vmdb/app/models/mixins/supports_feature_mixin.rb:263:ininstance_eval’
/var/www/miq/vmdb/app/models/mixins/supports_feature_mixin.rb:263:in block in define_supports_feature_methods' /usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-46d1cb3a2f14/app/models/manageiq/providers/azure/cloud_manager/event_catcher.rb:9:inblock in all_valid_ems_in_zone’
/usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-46d1cb3a2f14/app/models/manageiq/providers/azure/cloud_manager/event_catcher.rb:8:in select' /usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-46d1cb3a2f14/app/models/manageiq/providers/azure/cloud_manager/event_catcher.rb:8:inall_valid_ems_in_zone’
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:32:in desired_queue_names' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:38:insync_workers’
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in block in sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:ineach’
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:inmonitor_workers’
/var/www/miq/vmdb/app/models/miq_server.rb:339:in block in monitor' /usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-4ccebc6545f4/lib/gems/pending/util/extensions/miq-benchmark.rb:11:inrealtime_store’
/usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-4ccebc6545f4/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in realtime_block' /var/www/miq/vmdb/app/models/miq_server.rb:339:inmonitor’
/var/www/miq/vmdb/app/models/miq_server.rb:380:in block (2 levels) in monitor_loop' /usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-4ccebc6545f4/lib/gems/pending/util/extensions/miq-benchmark.rb:11:inrealtime_store’
/usr/local/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-4ccebc6545f4/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in realtime_block' /var/www/miq/vmdb/app/models/miq_server.rb:380:inblock in monitor_loop’
/var/www/miq/vmdb/app/models/miq_server.rb:379:in loop' /var/www/miq/vmdb/app/models/miq_server.rb:379:inmonitor_loop’
/var/www/miq/vmdb/app/models/miq_server.rb:241:in start' /var/www/miq/vmdb/lib/workers/evm_server.rb:27:instart’
/var/www/miq/vmdb/lib/workers/evm_server.rb:48:in start' /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in’
[----] I, [2019-10-04T09:30:33.619237 #17:50ef60] INFO – : MIQ(ManageIQ::Providers::Azure::CloudManager::EventCatcher.all_valid_ems_in_zone) Internal Error: Failed to open TCP connection to management.azure.com:443 (getaddrinfo: Name or service not known)

#5

This looks to me like an issue on the Azure side. Our requests are simply timing out. Just poking around the intertubes, you are not the only one experiencing intermittent failures, e.g. https://github.com/Azure/azure-libraries-for-java/issues/799