Using cloud providers (Amazon, OpenStack)

Support
#1

I have a problem with adding and using cloud providers. Applying the manageiq version from this week already solved the problems with credential validation, which did not work in manageiq version from previous week.

Today I have successfully added an Amazon and two OpenStack accounts (one Grizzly and one devstack), but none of them has any images. “Refresh relationships and power states” action finishes successfully, but only the Amazon provider obtains some flavors, availability zones and security groups.

Besides, if I create new user in manageiq, he cannot log in due to an error: Logins not allowed, no providers are being managed yet. Please contact the administrator.

#2

What errors are you showing in the /var/www/miq/vmdb/log/evm.log for openstack? The user required for Openstack should be a admin level for all of the tenants its a member of.

#3

@cankarm This part seems like a bug. You have an Amazon provider, but it’s not detecting it properly. Can you open an issue on the github issues page?

@dclarizio Even if this is fixed I have a bigger question. Even if I don’t add any managenent systems, why can’t users log in anyway? Why is this restriction even in the application? cc @jhardy

#4

Here are copy-paste parts from the logs (only what seems to be relevant):

Database error:

 ERROR -- : MIQ(abstract_adapter) Name: [SCHEMA], Message: [PGError: ERROR:  relation "rr0_pending_changes" does not exist
LINE 5:              WHERE a.attrelid = '"rr0_pending_changes"'::reg...
                                        ^
...:             SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
              FROM pg_attribute a LEFT JOIN pg_attrdef d
                ON a.attrelid = d.adrelid AND a.attnum = d.adnum
             WHERE a.attrelid = '"rr0_pending_changes"'::regclass
               AND a.attnum > 0 AND NOT a.attisdropped
             ORDER BY a.attnum
...]

Openstack refresher & FOG:

    INFO -- : MIQ(EmsRefresh::Refreshers::OpenstackRefresher.refresh) Refreshing all targets...
[----] I, [2015-01-14T06:09:46.474548 #6236:4a7ea0]  INFO -- : MIQ(EmsRefresh::Refreshers::OpenstackRefresher.refresh) EMS: [vagrant], id: [4] Refreshing targets for EMS: [vagrant], id: [4]...
[----] I, [2015-01-14T06:09:46.474622 #6236:4a7ea0]  INFO -- : MIQ(EmsRefresh::Refreshers::OpenstackRefresher.refresh) EMS: [vagrant], id: [4]   EmsOpenstack [vagrant] id [4]
[fog][DEPRECATION] Fog::XML::Connection is deprecated use Fog::Core::Connection instead (/var/www/miq/lib/openstack/openstack_handle/identity_delegate.rb:19:in `visible_tenants')
[----] I, [2015-01-14T06:09:46.646219 #22383:bf3eac]  INFO -- : MIQ(ReplicationWorker) Replicate Process gone. Restarting...
[----] E, [2015-01-14T06:09:46.647879 #22383:bf3eac] ERROR -- : MIQ(ReplicationWorker) Replication configuration is invalid.
[----] I, [2015-01-14T06:09:47.648916 #22383:bf3eac]  INFO -- : MIQ(ReplicationWorker) Replicate Process gone. Restarting...
[----] E, [2015-01-14T06:09:47.650631 #22383:bf3eac] ERROR -- : MIQ(ReplicationWorker) Replication configuration is invalid.
[----] E, [2015-01-14T06:09:48.005115 #6236:4a7ea0] ERROR -- : <Fog> excon.error     #<Excon::Errors::SocketError: No route to host - connect(2) (Errno::EHOSTUNREACH)>

[----] E, [2015-01-14T06:09:48.005636 #6236:4a7ea0] ERROR -- : MIQ(MiqQueue.deliver)    Message id: [23341], Error: [No route to host - connect(2) (Errno::EHOSTUNREACH)]
[----] E, [2015-01-14T06:09:48.005778 #6236:4a7ea0] ERROR -- : [Excon::Errors::SocketError]: No route to host - connect(2) (Errno::EHOSTUNREACH)  Method:[rescue in deliver]
[----] E, [2015-01-14T06:09:48.005873 #6236:4a7ea0] ERROR -- : /opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/socket.rb:203:in `connect_nonblock'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/socket.rb:203:in `rescue in block in connect'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/socket.rb:172:in `block in connect'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/socket.rb:168:in `each'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/socket.rb:168:in `connect'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/socket.rb:28:in `initialize'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/connection.rb:382:in `new'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/connection.rb:382:in `socket'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/connection.rb:105:in `request_call'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/middlewares/mock.rb:47:in `request_call'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/middlewares/instrumentor.rb:19:in `block in request_call'
/var/www/miq/vmdb/lib/vmdb/logging/fog_logger.rb:22:in `instrument'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/middlewares/instrumentor.rb:18:in `request_call'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/middlewares/base.rb:15:in `request_call'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/middlewares/base.rb:15:in `request_call'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/middlewares/base.rb:15:in `request_call'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/connection.rb:232:in `request'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/fog-core-1.25.0/lib/fog/core/connection.rb:63:in `request'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/fog-1.24.0/lib/fog/openstack/core.rb:215:in `get_supported_version'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/fog-1.24.0/lib/fog/openstack/image.rb:206:in `authenticate'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/fog-1.24.0/lib/fog/openstack/image.rb:122:in `initialize'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/fog-core-1.25.0/lib/fog/core/service.rb:115:in `new'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/fog-core-1.25.0/lib/fog/core/service.rb:115:in `new'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/fog-core-1.25.0/lib/fog/image.rb:12:in `new'
/var/www/miq/lib/openstack/openstack_handle/handle.rb:55:in `raw_connect'
/var/www/miq/lib/openstack/openstack_handle/handle.rb:30:in `raw_connect_try_ssl'
/var/www/miq/lib/openstack/openstack_handle/handle.rb:103:in `connect'
/var/www/miq/lib/openstack/openstack_handle/handle.rb:178:in `detect_service'
/var/www/miq/lib/openstack/openstack_handle/handle.rb:144:in `detect_image_service'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/openstack.rb:26:in `initialize'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/openstack.rb:11:in `new'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/openstack.rb:11:in `ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/openstack_refresher.rb:18:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/openstack_refresher.rb:7:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/openstack_refresher.rb:7:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/base_refresher.rb:8:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:73:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:72:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:72:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:360:in `block in deliver'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/2.0.0/timeout.rb:66:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:356:in `deliver'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work'
/var/www/miq/vmdb/lib/workers/worker_base.rb:317:in `block in do_work_loop'
/var/www/miq/vmdb/lib/workers/worker_base.rb:314:in `loop'
/var/www/miq/vmdb/lib/workers/worker_base.rb:314:in `do_work_loop'
/var/www/miq/vmdb/lib/workers/worker_base.rb:135:in `run'
/var/www/miq/vmdb/lib/workers/worker_base.rb:123:in `start'
/var/www/miq/vmdb/lib/workers/worker_base.rb:23:in `start_worker'
/var/www/miq/vmdb/lib/workers/bin/worker.rb:5:in `<top (required)>'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/bundler/gems/rails-cab310aeee22/railties/lib/rails/commands/runner.rb:52:in `eval'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/bundler/gems/rails-cab310aeee22/railties/lib/rails/commands/runner.rb:52:in `<top (required)>'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/bundler/gems/rails-cab310aeee22/railties/lib/rails/commands.rb:64:in `require'
/opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/bundler/gems/rails-cab310aeee22/railties/lib/rails/commands.rb:64:in `<top (required)>'
script/rails:6:in `require'
script/rails:6:in `<main>'

Frequently I see error:

ERROR -- : MIQ(ReplicationWorker) Replication configuration is invalid.
INFO -- : MIQ(ReplicationWorker) Replicate Process gone. Restarting...

From the logs I can see that some data are fetched from the clouds, but I cannot access them from UI, for example (similar for all three clouds):

 INFO -- : Q-task_id([log_status]) MIQ(MiqWorker.log_status) MiqEventCatcherOpenstack: [Event Monitor for Cloud/Infrastructure Provider: vagrant] Worker ID [18], PID [9666], GUID [ea121bcc-9a68-11e4-9ea4-fa163e85aba2], Last Heartbeat [2015-01-14 06:07:09 UTC], Process Info: Memory Usage [151298048], Memory Size [425041920], Memory % [1.83], CPU Time [3345.0], CPU % [0.02], Priority [21]
#5

@Fryguy This was originally implemented to prevent non admin users from accessing the UI until the administrator had at least defined/discovered one provider, as there was really nothing to see in the UI at that point. There is at least one enhancement request to remove this restriction, but this would require making sure the UI functions properly in this situation and I’m not sure how much work it would be to verify this. If there is a Cloud provider in the system already, we should certainly look into why users are not being allowed to login.

#6

Hi @cankarm

I’ll add some detail to what you posted.

These are all errors with the replication worker. It looks like you turned on the database synchronization role without setting it up perhaps? If you don’t need replication and/or don’t have it setup, I would turn that role off.

This is indicating that pid 6236 (One of your Openstack refresh workers) can not connect. I would verify what service it is trying to connect to and that it is reachable from where you deployed ManageIQ.

Your appliance is thrashing that worker. Basically it attempts to restart it every time it fails so right now it is constantly restarting it. This is bad for system resources :frowning: !

This is actually the status of that specific worker process on the appliance. It is reporting the Memory and CPU usage from ManageIQ’s viewpoint of the worker.

For your Openstack refresh issues, I’d recommend checking on what network your endpoint services are deployed and if those ip addresses are reachable from your ManageIQ appliance. You can always enable logging for level_fog to debug if you want a greater level of detail in the fog.log in the advanced configuration of ManageIQ.

Hope that helps!

#7

Hi @akrzos,

thanks for the explanation.

I realized this before and manage to find where to turn it of. Now I have only network/connection.

This same error “No route to host” occured in logs activating the authetication validation proces when I used the ManagaIQ version “anand-1.20150109083936_ea4c62a” previous week. In version “master.20150112084306_356237b” the validation is ok, but this error occurs when the “Refresh Cloud Providers” action is initiated. The network connections are OK, as the ManageIQ can connect to the cloud, validate authetication and in the logs I can see the data about the workers. Also I tested network visibility from the shell to be sure that servers can see each other. Could this be related with the deprecad call of the method? Did you noticed the lines which occur just before the error:

[----] I, [2015-01-15T06:59:29.324937 #31907:127dea8]  INFO -- : MIQ(EmsRefresh::Refreshers::OpenstackRefresher.refresh) EMS: [vagrant], id: [4]   EmsOpenstack [vagrant] id [4]
[fog][DEPRECATION] Fog::XML::Connection is deprecated use Fog::Core::Connection instead (/var/www/miq/lib/openstack/openstack_handle/identity_delegate.rb:19:in `visible_tenants')
[----] E, [2015-01-15T06:59:30.053009 #31907:127dea8] ERROR -- : <Fog> excon.error     #<Excon::Errors::SocketError: No route to host - connect(2) (Errno::EHOSTUNREACH)>

[----] E, [2015-01-15T06:59:30.053381 #31907:127dea8] ERROR -- : MIQ(MiqQueue.deliver)    Message id: [31746], Error: [No route to host - connect(2) (Errno::EHOSTUNREACH)]
[----] E, [2015-01-15T06:59:30.053565 #31907:127dea8] ERROR -- : [Excon::Errors::SocketError]: No route to host - connect(2) (Errno::EHOSTUNREACH)  Method:[rescue in deliver]
[----] E, [2015-01-15T06:59:30.053663 #31907:127dea8] ERROR -- : /opt/rubies/ruby-2.0.0-p598/lib/ruby/gems/2.0.0/gems/excon-0.42.0/lib/excon/socket.rb:203:in `connect_nonblock'

Now I’m running ManageIQ with the debug logging mode for fog, nothing new for now.

#8

Hi,

I solved this problem. The adminurl in OpenStack services was wrong. It would be good if the “Validate” actions in provider settings (Provider/AMQP) would fail, if it is not possible control or obtain data from providers.

Thanks to all for your help!

#9

Weird I thought it actually did that already…Maybe there’s a bug or something in there. Or maybe I’m mistaken.

#10

I did not digg into the code to see how this is implemented, but when I was testing my OpenStack (OS) settings manually, I found out that “curl” to the AMQP ip and port successfully returns some data (info about service) even when no credentials are used. This test is not enough for controlling the OS. To test the control ability, the MiQ needs to test if the credentials (user-name/password) of the AMQP user are sufficient.

Maybe this helps …

[Solved] Excon::Errors::Timeout
#11

We only use generic catalog services with no providers. Can you tell me where the enhancement request is posted? We would like to be able to disable this feature.