Ansible inside task stuck


#1

Hello Team,

how can I find out why my embedded Ansible tasks are stuck: queued but never run? This is CF4.5


#2

I tried to reset the setup according to RH KB: “Cannot create ansible repository, process does not seem to be running” but did not help:

  • disable Ansible role
  • rm /etc/tower/SECRET_KEY
  • vmdb; bin/rails c
    MiqDatabase.first.ansible_secret_key = nil
    MiqDatabase.first.save
  • enable Ansible role

#3

Tower seems to be up and running:

# supervisorctl status
exit-event-listener                       RUNNING   pid 3586, uptime 0:17:18
tower-processes:awx-callback-receiver     RUNNING   pid 3591, uptime 0:17:18
tower-processes:awx-celeryd               RUNNING   pid 3593, uptime 0:17:18
tower-processes:awx-celeryd-beat          RUNNING   pid 3592, uptime 0:17:18
tower-processes:awx-channels-worker       RUNNING   pid 3587, uptime 0:17:18
tower-processes:awx-daphne                RUNNING   pid 3590, uptime 0:17:18
tower-processes:awx-fact-cache-receiver   RUNNING   pid 3589, uptime 0:17:18
tower-processes:awx-uwsgi                 RUNNING   pid 3588, uptime 0:17:18

Embedded Ansible also looks good:

# vmdb; bin/rake evm:status | grep Ansible
EmbeddedAnsibleWorker                                                 | started | 13858 | 2521 | 3001 |         2 |                       | 2017-11-12T19:11:46Z | 2017-11-12T19:33:45Z
ManageIQ::Providers::EmbeddedAnsible::AutomationManager::EventCatcher | started | 13880 | 4054 | 3150 |         2 | ems_10                | 2017-11-12T19:13:00Z | 2017-11-12T19:33:42Z

I tried to create an Ansible credential “testansible”. The task is created but never completes:

# grep testansible /var/www/miq/vmdb/log/evm.log
[----] I, [2017-11-12T14:23:47.277548 #3041:1b72aa4]  INFO -- : MIQ(MiqQueue.put) Message id: [387617],  id: [], Zone: [UI Zone], Role: [ems_operations], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [ManageIQ::Providers::EmbeddedAnsible::AutomationManager::MachineCredential.create_in_provider], Timeout: [600], Priority: [20], State: [ready], Deliver On: [], Data: [], Args: [10, {:id=>nil, :name=>"testansible", :type=>"ManageIQ::Providers::EmbeddedAnsible::AutomationManager::MachineCredential", :password=>"********", :userid=>"root"}]
[----] I, [2017-11-12T14:23:47.277774 #3041:1b72aa4]  INFO -- : MIQ(MiqTask.generic_action_with_callback) Task: [1022] Queued the action: [Creating Ansible Tower Credential (name=testansible)] being run for user: [system]

#4

I’ve downloaded a newer version of CF4.5
Old: 5.8.0.17.20170525183055_6317a22
New: 5.8.2.3.20171016155816_aaec796

I enabled Embedded Ansible in the latter successfully. I tried to compare the difference and found that the new version has an extra worker EmbeddedAnsible::AutomationManager::RefreshWorker

# vmdb; bin/rake evm:status | grep Ansible
EmbeddedAnsibleWorker                                                  | started | 17 |       | 1029  |         1 |                         |     2017-11-13T20:22:57Z | 2017-11-13T21:38:00Z
ManageIQ::Providers::EmbeddedAnsible::AutomationManager::EventCatcher  | started | 19 |  7238 | 7260  |         1 | ems_2                   | 2017-11-13T20:35:25Z | 2017-11-13T21:37:51Z
ManageIQ::Providers::EmbeddedAnsible::AutomationManager::RefreshWorker | started | 18 |  7229 | 7257  |         1 | ems_2                   | 2017-11-13T20:35:24Z | 2017-11-13T21:38:07Z

In Configuration/Diagnostics/Servers by Roles the Embedded Ansible role is green for both appliances. I tried to search for this Refreshworker in logs but only the new server has it, there are no traces on the old one of this worker.

Still would like to fix the old one if possible instead migrating…


#5

On the old one can you confirm ems_operations server role on the appliance has ansible role enabled


#6

Provider Operations ( = ems_operations if I understand correctly) is enabled.

I redeployed the appliance with the old version and now Embedded Ansible works after enabling it, I could not reproduce the issue.

Thanks.