SSL failing on OpenStack provider

Support
#1

Hi,

I am using ManageIQ to manage different providers. Running on our own dev environment I have one vSphere and one OpenStack – neither of them is using SSL connection.
Now I need to manage a different OpenStack provider as well, but this one external to our dev environment (and outside our control as well), and all the OpenStack APIs are being exposed using SSL; and it is failing to authenticate.

Using ssldump, I was able to verify that ManageIQ is indeed trying to connect using SSL as well, but failing on server key exchange [ServerKeyExchange Not enough data. Found 258 bytes (expecting 32767)].
Using curl I am able to acquire a valid token from OpenStack with no errors.

Have anyone ingrated MiQ with OpenStack over SSL?

PS: the OpenStack API is not being exposed on the default port (i.e. 5000), but on 7001.

Thanks,
Gis

#2

Hi,

Just to add more information:
I have changed the log level of Fog to Debug, and was able to verify that MiQ is indeed changing to SSL connections: it tries first with non-SSL and fails with a EOF error; then MiQ tries again with SSL – and is seems to work with 200 codes.

There is no other information in the logs that seems useful at this point. The evm.log has three errors - the initial fog error (before failing over) and then the error showing up on the ManageIQ frontend (“Unexpected response returned from system, see log for details”). They seem to occur at the same time, which makes me wonder whether the first fog error (before failing over to SSL) is causing ManageIQ to think the connection failed. Anyway, this is about as much as we can see right now. If you have any further suggestions that would be most appreciated.

evm.log ERROR entries:

[----] E, [2015-03-12T09:53:45.349166 #2262:131de94] ERROR – : excon.error #<Excon::Errors::SocketError: end of file reached (EOFError)>

[fog][DEPRECATION] Fog::Connection is deprecated use Fog::Core::Connection instead (/opt/manageiq/lib/openstack/openstack_handle/identity_delegate.rb:19:in `visible_tenants’)
[----] E, [2015-03-12T09:53:45.556861 #2262:131de94] ERROR – : MIQ(EmsOpenstack.verify_api_credentials) Error Class=Excon::Errors::SocketError, Message=end of file reached (EOFError)
[----] E, [2015-03-12T09:53:45.557076 #2262:131de94] ERROR – : MIQ(ems_cloud_controller-update): Unexpected response returned from system, see log for details

I am adding the fog.log content as well below, in case it can be useful:

[----] D, [2015-03-12T09:53:44.736641 #2262:131de94] DEBUG – : excon.request
{:uri=>“http://os-a1.xyzprovider.com:7001/v2.0/tokens”,
:method=>“POST”,
:headers=>
{“User-Agent”=>“fog/1.28.0 fog-core/1.29.0”,
“Content-Type”=>“application/json”,
“Host”=>“os-a1.xyzprovider.com:7001”},
:body=>
{“auth”=>
{“passwordCredentials”=>{“username”=>"", “password”=>"********"},
“tenantName”=>""}}}

[----] E, [2015-03-12T09:53:45.349278 #2262:131de94] ERROR – : excon.error #<Excon::Errors::SocketError: end of file reached (EOFError)>

[----] D, [2015-03-12T09:53:45.351542 #2262:131de94] DEBUG – : excon.request
{:uri=>“https://os-a1.xyzprovider.com:7001/v2.0/tokens”,
:method=>“POST”,
:headers=>
{“User-Agent”=>“fog/1.28.0 fog-core/1.29.0”,
“Content-Type”=>“application/json”,
“Host”=>“os-a1.xyzprovider.com:7001”},
:body=>
{“auth”=>
{“passwordCredentials”=>{“username”=>"", “password”=>"********"},
“tenantName”=>""}}}

[----] D, [2015-03-12T09:53:45.428597 #2262:131de94] DEBUG – : excon.response
{:status=>200,
:headers=>
{“Vary”=>“X-Auth-Token”,
“X-Distribution”=>“Ubuntu”,
“Content-Type”=>“application/json”,
“Content-Length”=>“376”,
“Date”=>“Thu, 12 Mar 2015 09:53:41 GMT”},
:body=>
{“access”=>
{“token”=>
{“issued_at”=>“2015-03-12T09:53:41.274672”,
“expires”=>“2015-03-12T10:53:41Z”,
“id”=>“808d91bceec046f58c85d73686b18acc”,
“audit_ids”=>[“ETt_Feg4TkOVccxa_NOQOw”]},
“serviceCatalog”=>[],
“user”=>
{“username”=>"",
“roles_links”=>[],
“id”=>“c8fab46239b8439ba041ecfa10c4058e”,
“roles”=>[],
“name”=>""},
“metadata”=>{“is_admin”=>0, “roles”=>[]}}}}

[----] D, [2015-03-12T09:53:45.430179 #2262:131de94] DEBUG – : excon.request
{:uri=>“https://os-a1.xyzprovider.com:7001/v2.0/tenants”,
:method=>“GET”,
:headers=>
{“User-Agent”=>“fog/1.28.0 fog-core/1.29.0”,
“Content-Type”=>“application/json”,
“Accept”=>“application/json”,
“X-Auth-Token”=>“808d91bceec046f58c85d73686b18acc”,
“Host”=>“os-a1.xyzprovider.com:7001”},
:body=>nil}

[----] D, [2015-03-12T09:53:45.470266 #2262:131de94] DEBUG – : excon.response
{:status=>200,
:headers=>
{“Vary”=>“X-Auth-Token”,
“X-Distribution”=>“Ubuntu”,
“Content-Type”=>“application/json”,
“Content-Length”=>“149”,
“Date”=>“Thu, 12 Mar 2015 09:53:41 GMT”},
:body=>
{“tenants_links”=>[],
“tenants”=>
[{“description”=>“AlphaAccount001”,
“enabled”=>true,
“id”=>“52354e72e8c94647ae746cff76b7e0be”,
“name”=>""}]}}

[----] D, [2015-03-12T09:53:45.472115 #2262:131de94] DEBUG – : excon.request
{:uri=>“https://os-a1.xyzprovider.com:7001/v2.0/tokens”,
:method=>“POST”,
:headers=>
{“User-Agent”=>“fog/1.28.0 fog-core/1.29.0”,
“Content-Type”=>“application/json”,
“Host”=>“os-a1.xyzprovider.com:7001”},
:body=>
{“auth”=>
{“token”=>{“id”=>“808d91bceec046f58c85d73686b18acc”},
“tenantName”=>""}}}

[----] D, [2015-03-12T09:53:45.545013 #2262:131de94] DEBUG – : excon.response
{:status=>200,
:headers=>
{“Vary”=>“X-Auth-Token”,
“X-Distribution”=>“Ubuntu”,
“Content-Type”=>“application/json”,
“Content-Length”=>“2989”,
“Date”=>“Thu, 12 Mar 2015 09:53:41 GMT”},
:body=>
{“access”=>
{“token”=>
{“issued_at”=>“2015-03-12T09:53:41.384309”,
“expires”=>“2015-03-12T10:53:41Z”,
“id”=>“69d86783035b4910a5e0c5aa2b20012b”,
“tenant”=>
{“description”=>“AlphaAccount001”,
“enabled”=>true,
“id”=>“52354e72e8c94647ae746cff76b7e0be”,
“name”=>""},
“audit_ids”=>[“g0UfLsUYSymsIscj03eBTQ”, “ETt_Feg4TkOVccxa_NOQOw”]},
“serviceCatalog”=>
[{“endpoints”=>
[{“adminURL”=>
“http://ctrl-os-a1-nova-comp-api:8774/v2/52354e72e8c94647ae746cff76b7e0be”,
“region”=>“Dunsfold-Alpha-1”,
“internalURL”=>
“http://ctrl-os-a1-nova-comp-api:8774/v2/52354e72e8c94647ae746cff76b7e0be”,
“id”=>“5a9d39435fd4469781e628b05e4fd3e9”,
“publicURL”=>
“https://os-a1.xyzprovider.com:7020/v2/52354e72e8c94647ae746cff76b7e0be”}],
“endpoints_links”=>[],
“type”=>“compute”,
“name”=>“nova”},
{“endpoints”=>
[{“adminURL”=>“http://ctrl-os-a1-neutron:9696”,
“region”=>“Dunsfold-Alpha-1”,
“internalURL”=>“http://ctrl-os-a1-neutron:9696”,
“id”=>“167c9692f01445c6b0d4b7dd2a6486a3”,
“publicURL”=>“https://os-a1.xyzprovider.com:7030”}],
“endpoints_links”=>[],
“type”=>“network”,
“name”=>“neutron”},
{“endpoints”=>
[{“adminURL”=>
“http://ctrl-os-a1:8776/v2/52354e72e8c94647ae746cff76b7e0be”,
“region”=>“Dunsfold-Alpha-1”,
“internalURL”=>
“http://ctrl-os-a1:8776/v2/52354e72e8c94647ae746cff76b7e0be”,
“id”=>“0efb0d983d5843b186a27fdf6d81a609”,
“publicURL”=>
“https://os-a1.xyzprovider.com:7040/v2/52354e72e8c94647ae746cff76b7e0be”}],
“endpoints_links”=>[],
“type”=>“volumev2”,
“name”=>“cinderv2”},
{“endpoints”=>
[{“adminURL”=>“http://ctrl-os-a1-glance:9292”,
“region”=>“Dunsfold-Alpha-1”,
“internalURL”=>“http://ctrl-os-a1-glance:9292”,
“id”=>“00b5ee7ea9554d3ab6c05e988336e964”,
“publicURL”=>“https://os-a1.xyzprovider.com:7010”}],
“endpoints_links”=>[],
“type”=>“image”,
“name”=>“glance”},
{“endpoints”=>
[{“adminURL”=>“http://ctrl-os-a1:8777”,
“region”=>“Dunsfold-Alpha-1”,
“internalURL”=>“http://ctrl-os-a1:8777”,
“id”=>“75445c7dc92642e281e67c4d2dad95f7”,
“publicURL”=>“https://os-a1.xyzprovider.com:7060”}],
“endpoints_links”=>[],
“type”=>“metering”,
“name”=>“ceilometer”},
{“endpoints”=>
[{“adminURL”=>
“http://ctrl-os-a1:8776/v1/52354e72e8c94647ae746cff76b7e0be”,
“region”=>“Dunsfold-Alpha-1”,
“internalURL”=>
“http://ctrl-os-a1:8776/v1/52354e72e8c94647ae746cff76b7e0be”,
“id”=>“2bc804a222ee45f295be7666aa13ed81”,
“publicURL”=>
“https://os-a1.xyzprovider.com:7040/v1/52354e72e8c94647ae746cff76b7e0be”}],
“endpoints_links”=>[],
“type”=>“volume”,
“name”=>“cinder”},
{“endpoints”=>
[{“adminURL”=>“http://ctrl-os-a1-keystone:35357/v2.0”,
“region”=>“Dunsfold-Alpha-1”,
“internalURL”=>“http://ctrl-os-a1-keystone:5000/v2.0”,
“id”=>“8af2e9bde92941d79b3a12c9e0fe61a0”,
“publicURL”=>“https://os-a1.xyzprovider.com:7001/v2.0”}],
“endpoints_links”=>[],
“type”=>“identity”,
“name”=>“keystone”}],
“user”=>
{“username”=>"",
“roles_links”=>[],
“id”=>“c8fab46239b8439ba041ecfa10c4058e”,
“roles”=>[{“name”=>“member”}, {“name”=>“admin”}],
“name”=>""},
“metadata”=>
{“is_admin”=>0,
“roles”=>
[“3058088dc0bc45d5a097bfb8bd491c7a”,
“9b3b3065f6934c4face9f62e35dd74a3”]}}}}

Thanks,
Gis

#3

Hi there Gis,

thanks for taking a look at ManageIQ!

We have tested ManageIQ and OpenStack over SSL. There are a few peculiarities to our implementation, but before I dig into those, I’ll take a look at your logs and see if anything jumps out at me.

Greg Blomquist

#4

Gis,

can you give me some idea of what you actually see failing? Or, do you just see the errors in the logs?

For instance, do you see that the inventory collection from OpenStack is not working? Or, maybe when you try to provision to OpenStack it fails?

Also, I don’t see the ServerKeyExchange error in the logs you posted. Do you see that anywhere in your logs? That’s an error I haven’t seen before in this scenario (or, anywhere for that matter).

From what I can tell from the logs, it looks like it’s working as expected (and precisely as you described):

  • first ManageIQ tries to connect over non-SSL
  • if non-SSL fails, ManageIQ attempts an SSL connection

To me, it looks like these things are happening, and then it’s collecting data as expected.

For instance, after failing over to SSL, here’s the request for tokens (shown below, somewhat truncated):

… and the response:

#5

Sorry, I missed the bit about

#6

Hi Greg,

Thanks for looking at this issue.

The ServerKeyExchange error appears on the SSL dump logs only.

The authentication of MiQ onto OpenStack is failing when adding a new cloud provider (.i.e. the validation of the input information of the provider). Even if I ignore this error and save the provider, it shows the “Default credentials” with error and it can’t read the information from OpenStack (.i.e the refresh is never made successfully) and because of this it is not really possible even to launch a VM using MiQ.

As a side note, when the provider added an HAProxy some more logs were shown, about failing to handshake:

Mar 13 14:36:50 ctrl-os-a1 haproxy[2566]:YYY.YYY.YYY.YYY:37052 [13/Mar/2015:14:36:50.265] keystone-public/1: SSL handshake failure

Are you able to tell me which cyphers are supported by MiQ (or does it just use all available ones on the OS)?

Thanks,
Gis

#7

Gis,

can you give me an idea of what you’re environment looks like?

For instance, where does SSL terminate? At the services themselves, or are you using a proxy for termination? Are all of the services behind SSL, or just Keystone?

I’d like to get an environment setup similarly here to try to reproduce what you’re seeing.

I mentioned that we have some peculiarities with our SSL implementation, but all that amounts to is what you’ve already seen: We try non-SSL first then fail over to SSL. The main reasons behind this is that we (1) don’t have a good way to let the user specify whether the connection is SSL (yet), and (2) there are no standardized ports for the OpenStack services if they’re SSL protected (like http has 80 and https has 443).

#8

It should be able to use whatever is available. If you’re able to curl from the same appliance, then I’d expect that the application would be able to do the same…

#9

Hi Gis,

I have an environment here with SSL enabled for OpenStack, and I’m able to reproduce your error. Looks like we have a regression bug… :frowning:

Our environment uses HA Proxy to terminate SSL before reaching the services. But, I’m still seeing the same thing. I also know that we had tested this same scenario not too long ago against this same environment. So, it must be very recent that it failed.

I’ll dig into it on our side and figure out a fix asap.

Greg

#12

Hi Gis,

Sorry I didn’t get back to you on this! :-/

I believe this is the fix for the problem you were seeing: https://github.com/ManageIQ/manageiq/pull/2178

You can get a new appliance image from http://releases.manageiq.org/ (search for botvinnik-1-rc3). Or, you can patch your appliance directly with the pull request.

Hope that helps!

Greg

#13

How to enable SSL endpoints for public API?

Introduction

There are at least two solution to do this. The first one is to enable SSL for all OpenStack services. The other solution is to use a reverse proxy with SSL termination. We will use the second approach because we see several issues to the first one. Enabling SSL for all services that expose API on public network is not the best solution because:

  • API services need to respond quickly and SSL handshake can add a significant overhead.
  • Python services are not good for SSL management

The other solution is to use a web server like Apache, Nginx or another one to handle incoming SSL connections, decrypting the SSL and passing on the unencrypted request to the corresponding service. With this solution, communication between the reverse proxy and the endpoints will not be encrypted. As said, several proxies are availables. We will use HAProxy because this proxy is already deployed in the HA solution. That means that we have manifests already there even if SSL is not enabled. That means that we will use it for non HA if the user enables SSL. Apache and Nginx are already used for the OpenStack dashboard and for the FUEL dashboard. So we prefer to keep configuration separated and use HAProxy for HA and SSL endpoints.

Implementation of the reverse proxy solution

We need to install HAProxy. SSL is supported since 1.5.0 that is not available in ubuntu 12.04 nor 14.04. So we need to install it from PPA.
We need to create a CA to be used for that purpose. The first approach is to create it but the second approach could be to allow a user to submit its own CA.
We need to configure API public endpoints as private. do you mean binding services on “private networks” only (localhost, managment and admin networks) ? I mean binding them on a private local IP so the public endpoints will not be available from internet and then will be available only through the HA proxy.
We need register the public endpoint of the haproxy in keystone
… next step … in progress

#14

@gisnalbert Hi, I have one question, how to change the log level of Fog to Debug? thanks

#15

That’s a very helpful info provided Yasmin.

I had recently setup manageIQ env in centos from devlopement setup giude from http://manageiq.org/docs/guides/developer_setup

Now as you explained above method for ssl. So do you have some doccumentation oe steps regarding Apache HAProxy setup for manageIQ.

I am unable to do that as my apache not abl to proxy to localhost:3000