Cannot see information after adding Openstack provider

providers

#1

Hi everyone,

I have downloaded latest image of ManageIQ for Openstack and then did everything according to an official manual to lunch the instance. When I added Openstack provider, I had no errors of validations. Moreover, I see green check mark that I suppose indicates everything is ok with my credentials. However, I still don’t see anything in “Relationship” except the number of security groups. Also, I see an error in Status:

Error - 18 Minutes Ago
Expected(200) <=> Actual(404 Not Found) excon.error.response :body => "{\"message\": \"The resource could ...

I have tried to find any logs into the image OS and found this one in /var/www/miq/vmdb/log/fog.log :

[----] E, [2017-11-07T16:16:35.453470 #7432:b2ee6f8] ERROR -- : excon.error     #<Excon::Error::NotFound: Expected([200, 204, 300]) <=> Actual(404 Not Found)
excon.error.response
  :body          => "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL / was not found on this server.</p>\n<hr>\n<address>Apache/2.4.6 (Red Hat Enterprise Linux) Server at 10.189.128.95 Port 8777</address>\n</body></html>\n"
  :cookies       => [
  ]
  :headers       => {
    "Content-Length" => "296"
    "Content-Type"   => "text/html; charset=iso-8859-1"
    "Date"           => "Tue, 07 Nov 2017 16:19:11 GMT"
    "Server"         => "Apache/2.4.6 (Red Hat Enterprise Linux)"
  }
  :host          => "10.189.128.95"
  :local_address => "10.0.100.4"
  :local_port    => 42722
  :path          => "/"
  :port          => 8777
  :reason_phrase => "Not Found"
  :remote_ip     => "10.189.128.95"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

[----] W, [2017-11-07T16:21:37.198108 #7432:165df14]  WARN -- : MIQ(Class#raw_connect) Service Event not available for openstack provider http://10.189.128.95:5000/v3/auth/tokens
[----] W, [2017-11-07T16:21:37.198292 #7432:165df14]  WARN -- : Could not find service event. Have alarming, application-catalog, cloudformation, compute, dns, identity, image, metering, metric, network, orchestration, placement, share, sharev2, volume, volumev2, volumev3
[----] W, [2017-11-07T16:21:37.218079 #7442:1656304]  WARN -- : MIQ(Class#raw_connect) Service Event not available for openstack provider http://10.189.128.95:5000/v3/auth/tokens
[----] W, [2017-11-07T16:21:37.218258 #7442:1656304]  WARN -- : Could not find service event. Have alarming, application-catalog, cloudformation, compute, dns, identity, image, metering, metric, network, orchestration, placement, share, sharev2, volume, volumev2, volumev3
[----] E, [2017-11-07T16:21:37.871744 #7432:165df14] ERROR -- : excon.error     #<Excon::Error::NotFound: Expected([200, 204, 300]) <=> Actual(404 Not Found)
excon.error.response
  :body          => "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL / was not found on this server.</p>\n<hr>\n<address>Apache/2.4.6 (Red Hat Enterprise Linux) Server at 10.189.128.95 Port 8777</address>\n</body></html>\n"
  :cookies       => [
  ]
  :headers       => {
    "Content-Length" => "296"
    "Content-Type"   => "text/html; charset=iso-8859-1"
    "Date"           => "Tue, 07 Nov 2017 16:24:14 GMT"
    "Server"         => "Apache/2.4.6 (Red Hat Enterprise Linux)"
  }
  :host          => "10.189.128.95"
  :local_address => "10.0.100.4"
  :local_port    => 42926
  :path          => "/"
  :port          => 8777
  :reason_phrase => "Not Found"
  :remote_ip     => "10.189.128.95"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

[----] E, [2017-11-07T16:21:38.014464 #7442:1656304] ERROR -- : excon.error     #<Excon::Error::NotFound: Expected([200, 204, 300]) <=> Actual(404 Not Found)
excon.error.response
  :body          => "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL / was not found on this server.</p>\n<hr>\n<address>Apache/2.4.6 (Red Hat Enterprise Linux) Server at 10.189.128.95 Port 8777</address>\n</body></html>\n"
  :cookies       => [
  ]
  :headers       => {
    "Content-Length" => "296"
    "Content-Type"   => "text/html; charset=iso-8859-1"
    "Date"           => "Tue, 07 Nov 2017 16:24:14 GMT"
    "Server"         => "Apache/2.4.6 (Red Hat Enterprise Linux)"
  }
  :host          => "10.189.128.95"
  :local_address => "10.0.100.4"
  :local_port    => 42928
  :path          => "/"
  :port          => 8777
  :reason_phrase => "Not Found"
  :remote_ip     => "10.189.128.95"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

Regards
Oleg


#2

@Oleg_Kucherov

Did you enable ceilometer events on Provider configuration? If you do not have Ceilometer endpoint on your OS installations you can switch these settings to AMQP events.


#3

@igortiunov

Thanks for quick response. Actually, yes the problem obviously in ceilometer,
I have ceilometer/aodh in my RDO build of Openstack (packstack installation), but for some reasons it doesn’t work for me.

I have switched to AMQP events and it still doesn’t work. For now I see already other error messages in the same log file.

[----] W, [2017-11-07T16:41:55.645454 #29610:aad138]  WARN -- : MIQ(OpenstackHandle::Handle.accessor_for_accessible_tenants) HTTP 404 Error during OpenStack request. Skipping inventory item Compute #<Proc:0x00000014efa2b8@/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-gems-pending-5dcbe6b97ad8/lib/gems/pending/openstack/openstack_handle/multi_tenancy/loop.rb:7 (lambda)>
Expected([200, 203]) <=> Actual(404 Not Found)
excon.error.response
  :body          => "{\"itemNotFound\": {\"message\": \"Flavor detail.json could not be found.\", \"code\": 404}}"
  :cookies       => [
  ]
  :headers       => {
    "Content-Length"               => "84"
    "Content-Type"                 => "application/json; charset=UTF-8"
    "Date"                         => "Tue, 07 Nov 2017 16:44:31 GMT"
    "Openstack-Api-Version"        => "compute 2.1"
    "Vary"                         => "OpenStack-API-Version, X-OpenStack-Nova-API-Version"
    "X-Compute-Request-Id"         => "req-35730570-ccfb-48bb-bf30-2ef7cd60fd2d"
    "X-Openstack-Nova-Api-Version" => "2.1"
    "X-Openstack-Request-Id"       => "req-35730570-ccfb-48bb-bf30-2ef7cd60fd2d"
  }
  :host          => "10.189.128.95"
  :local_address => "10.0.100.4"
  :local_port    => 57946
  :path          => "/v2.1/caddac01927f44f394474fa27c0bac9d/flavors/detail.json"
  :port          => 8774
  :reason_phrase => "Not Found"
  :remote_ip     => "10.189.128.95"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"

[----] E, [2017-11-07T16:41:56.891675 #29610:aad138] ERROR -- : excon.error     #<Excon::Error::NotFound: Expected([200, 203]) <=> Actual(404 Not Found)
excon.error.response
  :body          => "{\"itemNotFound\": {\"message\": \"Flavor detail.json could not be found.\", \"code\": 404}}"
  :cookies       => [
  ]
  :headers       => {
    "Content-Length"               => "84"
    "Content-Type"                 => "application/json; charset=UTF-8"
    "Date"                         => "Tue, 07 Nov 2017 16:44:33 GMT"
    "Openstack-Api-Version"        => "compute 2.1"
    "Vary"                         => "OpenStack-API-Version, X-OpenStack-Nova-API-Version"
    "X-Compute-Request-Id"         => "req-26e7840a-cbcb-4fbe-8705-d0b74f865b5b"
    "X-Openstack-Nova-Api-Version" => "2.1"
    "X-Openstack-Request-Id"       => "req-26e7840a-cbcb-4fbe-8705-d0b74f865b5b"
  }
  :host          => "10.189.128.95"
  :local_address => "10.0.100.4"
  :local_port    => 57948
  :path          => "/v2.1/fd907ede4ded4422a5ff68e3c225b1f7/flavors/detail.json"
  :port          => 8774
  :reason_phrase => "Not Found"
  :remote_ip     => "10.189.128.95"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

[----] W, [2017-11-07T16:41:56.891885 #29610:aad138]  WARN -- : MIQ(OpenstackHandle::Handle.accessor_for_accessible_tenants) HTTP 404 Error during OpenStack request. Skipping inventory item Compute #<Proc:0x00000014efa2b8@/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-gems-pending-5dcbe6b97ad8/lib/gems/pending/openstack/openstack_handle/multi_tenancy/loop.rb:7 (lambda)>
Expected([200, 203]) <=> Actual(404 Not Found)
excon.error.response
  :body          => "{\"itemNotFound\": {\"message\": \"Flavor detail.json could not be found.\", \"code\": 404}}"
  :cookies       => [
  ]
  :headers       => {
    "Content-Length"               => "84"
    "Content-Type"                 => "application/json; charset=UTF-8"
    "Date"                         => "Tue, 07 Nov 2017 16:44:33 GMT"
    "Openstack-Api-Version"        => "compute 2.1"
    "Vary"                         => "OpenStack-API-Version, X-OpenStack-Nova-API-Version"
    "X-Compute-Request-Id"         => "req-26e7840a-cbcb-4fbe-8705-d0b74f865b5b"
    "X-Openstack-Nova-Api-Version" => "2.1"
    "X-Openstack-Request-Id"       => "req-26e7840a-cbcb-4fbe-8705-d0b74f865b5b"
  }
  :host          => "10.189.128.95"
  :local_address => "10.0.100.4"
  :local_port    => 57948
  :path          => "/v2.1/fd907ede4ded4422a5ff68e3c225b1f7/flavors/detail.json"
  :port          => 8774
  :reason_phrase => "Not Found"
  :remote_ip     => "10.189.128.95"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"

[----] E, [2017-11-07T16:41:57.171205 #29610:aad138] ERROR -- : excon.error     #<Excon::Error::NotFound: Expected(200) <=> Actual(404 Not Found)
excon.error.response
  :body          => "{\"message\": \"The resource could not be found.<br /><br />\\n\\n\\n\", \"code\": \"404 Not Found\", \"title\": \"Not Found\"}"
  :cookies       => [
  ]
  :headers       => {
    "Content-Length"         => "112"
    "Content-Type"           => "application/json"
    "Date"                   => "Tue, 07 Nov 2017 16:44:33 GMT"
    "X-Compute-Request-Id"   => "req-d930893c-53a9-489b-84ed-2248f9912a53"
    "X-Openstack-Request-Id" => "req-d930893c-53a9-489b-84ed-2248f9912a53"
  }
  :host          => "10.189.128.95"
  :local_address => "10.0.100.4"
  :local_port    => 57950
  :path          => "/v2.1/135c0e0642cf42c5b7fce6ca012f2932/os-availability-zone.json"
  :port          => 8774
  :reason_phrase => "Not Found"
  :remote_ip     => "10.189.128.95"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

Regards
Oleg


#4

@Oleg_Kucherov Oh, I see. Did you use Pike version ? As I know this version currently does not supported by fog-openstack library which is used by MIQ:


#5

Yes, I use exactly latest Pike.
Seems like the best solution is configuration of events for Ceilometer :-\


#6

@Oleg_Kucherov You will get the errors about another endpoints anyway because of API-version mismatch. Only way is to wait for related PR to be released.


#7

Hi Guys,

I debugged this further, see below. In fact the issue is CloudForms dynamically uses most current API for a service endpoint in OpenStack. Starting with pike this is v2.1. Unfortunately in v2.1 the URIs CloudForms is using appear to be depricated or changed so they dont work. How can we force CloudForms to use v2.0 of Nova API? Would it not be better if we are hardcoding the URI used in API call to also hardcode the coresponding API version known to work or at least allow user to change it easily?

Here we see that for Nova in the fog log that Nova v2.1 is chosen. Nova 2.0 is also available however.

]
:headers => {
“Content-Length” => "84"
“Content-Type” => "application/json; charset=UTF-8"
“Date” => “Wed, 07 Feb 2018 17:39:19 GMT”
“Openstack-Api-Version” => "compute 2.1"
“Vary” => “OpenStack-API-Version, X-OpenStack-Nova-API-Version”
“X-Compute-Request-Id” => "req-f086c992-435d-4ddb-b1e8-c67d46ba8bf0"
“X-Openstack-Nova-Api-Version” => "2.1"
“X-Openstack-Request-Id” => “req-f086c992-435d-4ddb-b1e8-c67d46ba8bf0”
}

Next CloudForms tried to get information on availability zones. Note the path and URI.

:host => “144.76.111.119”
:local_address => “192.168.1.4”
:local_port => 46482
:path => “/v2.1/d5f8a8531bb34ac3a0d3781348bcfc32/flavors/detail.json”
:port => 8774
:reason_phrase => “Not Found”
:remote_ip => “144.76.111.119”
:status => 404
:status_line => “HTTP/1.1 404 Not Found\r\n”

[----] D, [2018-02-07T18:39:18.377002 #5363:e25130] DEBUG – : excon.request
{:uri=>
http://144.76.5111.119:8774/v2.1/d5f8a8531bb34ac3a0d3781348bcfc32/os-availability-zone.json”,
:query=>{},
:method=>“GET”,
:headers=>
{“User-Agent”=>“fog-core/1.44.3”,
“Content-Type”=>“application/json”,
“Accept”=>“application/json”,
“X-Auth-Token”=>
"MY_AUTH_TOKEN,
“Host”=>“144.76.111.119:8774”},
:body=>nil}

As you can see CloudForms gets back a 404 error.

Again the URI being used by CloudForms is http://144.76.111.119:8774/v2.1/d5f8a8531bb34ac3a0d3781348bcfc32/os-availability-zone.json

Next I ran same test from curl directly against OpenStack Nova v2.1 API using my auth token

curl -s -H “X-Auth-Token: $MY_AUTH_TOKEN” “http://144.76.111.119:8774/v2.1/d5f8a8531bb34ac3a0d3781348bcfc32/os-availability-zone.json” | python -mjson.tool

No JSON object could be decoded (hence why we get 404)

Clearly the URI CloudForms us using no longer works in Nova 2.1.

In fact the working URI for Nova 2.1 API to get availability zones would be: http://144.76.111.119:8774/v2.1/d5f8a8531bb34ac3a0d3781348bcfc32/os-availability-zone/detail

Now see what happens when I change from Nova v2.1 to v2.0 using same call CloudForms is using.

[root@bigred ~(keystone_admin)]# curl -s -H “X-Auth-Token: $MY_AUTH_TOKEN” “http://144.76.111.119:8774/v2.0/d5f8a8531bb34ac3a0d3781348bcfc32/os-availability-zone.json” | python -mjson.tool
{
“choices”: [
{
“id”: “v2.0”,
“links”: [
{
“href”: “http://144.76.111.119:8774/v2/v2.0/d5f8a8531bb34ac3a0d3781348bcfc32/os-availability-zone.json”,
“rel”: “self”
}
],
“media-types”: [
{
“base”: “application/json”,
“type”: “application/vnd.openstack.compute+json;version=2”
}
],
“status”: “SUPPORTED”
},
{
“id”: “v2.1”,
“links”: [
{
“href”: “http://144.76.111.119:8774/v2.1/v2.0/d5f8a8531bb34ac3a0d3781348bcfc32/os-availability-zone.json”,
“rel”: “self”
}
],
“media-types”: [
{
“base”: “application/json”,
“type”: “application/vnd.openstack.compute+json;version=2.1”
}
],
“status”: “CURRENT”
}
]
}

It works.