ManageIQ behind load balancer


#1

Hi Guys. I try to switch manageiq frontend to http instead of https for load balancing purposes. I uncomment the appropriate settings in the manageiq-http.conf file. But now I cannot login to administrative interface. Any time I enter login/password of admin user I redirect to ?timeout=false url. If I login to https everything work fine. There is a log from production.log file: http://paste.openstack.org/show/604265/


#2

Hi!

What version of ManageIQ do you have?

I have been trying the operation behind a proxy recently and needed 2 fixes: https://github.com/ManageIQ/manageiq-ui-classic/pull/448 and https://github.com/ManageIQ/manageiq-ui-classic/pull/583

In the 2nd PR, you can see the config that we tested with David:

# Be more permissive with SSL certificates
SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off

# Make sure that X_FORWARDED is set
RequestHeader set X_FORWARDED_PROTO 'https'
RequestHeader set X-Forwarded-Ssl on
AllowEncodedSlashes NoDecode

# Load balancer for HTTP(S) requests
<Proxy balancer://manageiq>
  BalancerMember https://localhost:8443
</Proxy>

# Separate load balancer for WS(S) requests
<Proxy balancer://websocket>
  BalancerMember wss://localhost:8443
</Proxy>

# These two lines should always go BEFORE the rules for /
ProxyPass /ws/ balancer://websocket/ws/
ProxyPassReverse /ws/ balancer://websocket/ws/

ProxyPass / balancer://manageiq/
ProxyPassReverse / balancer://manageiq/

#3

And please, share where you got. Seems more people are trying various load balancer configurations.


#4

Hi @martinpovolny

I use euwe and I want to use haproxy. I use it for openstack deployment and I want to use one access point for users to openstack dashboard and manageiq self_service portal:
http://paste.openstack.org/show/604272/
In this config I use http for access to manageiq backend and for self_service it work fine, but I cannot login to admin portal. If I access manageiq directly to http://miq01.example.com I have the same issue - browser just redirected to ?timeout=true url.

If I configure haproxy backends through https like this:

backend selfservice
    server miq0 miq01.example.com:443 check ssl verify none

then I have a working frontend.


#5

Now I found that my problem is related to cookie storage. If I disable secure option for session_store:

grep secure config/initializers/session_store.rb    
    session_options[:secure]   = false

than I can connect to MIQ. But this option decreases the security because of MITM between haproxy and MIQ apache. May be I need to configure haproxy backend directly to puma server. But I dont know how to set puma listener to 0.0.0.0.


#6

You cannot have HTTPS only cookies if you are going through HTTP.

I need more info on what goes wrong in your setup. But I have a guess. You wrote that you are using Euwe. That means that you don’t have the patches from PR number one above. Then I guess you cannot get pass the login screen because the login form is being sumbited to a wrong URL (internal appliance instead of the haproxy). You can confirm this using the Network tab of your browser debug tool (CTRL-SHIFT-J in Chrome and FF).

I don’t have a haproxy instance at hand but I can try that and check where I get. Can you give me your haproxy setup here? (I am not familiar with haproxy).


#7

I apologize for my English.
First I want to enable access to ManageIQ via http. For this, I uncomment appropriate configuration in /etc/httpd/conf.d/manageiq-http.conf. If I try to login to administrative portal via http I get the error. The error is related to cookie store and if I disable this option (session_options[:secure] = false) I can access to manageiq via http.

Second I want to configure haproxy to access to manageiq and there is my configuration for haproxy.cfg: http://paste.openstack.org/show/604272/

But may be I chose not the right way and I need to configure haproxy to balance to puma servers directly instead of apache on the manageiq.


#8

Currently there is my final configuration for haproxy and http virtual host on the apache for manageiq.
First I need to enable http access to manageiq appliance for smart load balancing through haproxy:

RewriteEngine On
Options SymLinksIfOwnerMatch
<VirtualHost *:80>
  DocumentRoot /var/www/miq/vmdb/public
  Include conf.d/manageiq-redirects-ui
  Include conf.d/manageiq-redirects-ws
  Include conf.d/manageiq-redirects-websocket
  ProxyPreserveHost on
  RequestHeader set X_FORWARDED_PROTO 'https'
  <Location /assets/>
     Header unset ETag
     FileETag None
     ExpiresActive On
     ExpiresDefault "access plus 1 year"
  </Location>
  <Location /proxy_pages/>
     ErrorDocument 403 /error/noindex.html
     ErrorDocument 404 /error/noindex.html
  </Location>
</VirtualHost>

I need to enable this setting: RequestHeader set X_FORWARDED_PROTO ‘https’ because of ‘secure’ configuration of cookies for rails application (config/initializers/session_store.rb: session_options[:secure] = true).

My haproxy uses https listeners and tls termination work on them:

frontend cloud
   bind cloud.example.com:443 ssl crt /etc/haproxy/ssl/cert.pem no-sslv3 no-tls-tickets ciphers AES128+EECDH:AES128+EDH:AES256+EECDH:AES256+EDH
   mode http
   reqadd X-Forwarded-Proto:\ https
   timeout  client 3h
   timeout  server 3h
   acl has_dashboard_uri url_beg /dashboard
   acl is_os_domain hdr(host) -i cloud.example.com
   use_backend horizon if has_dashboard_uri is_os_domain
   acl has_self_uri url_beg /self_service
   use_backend selfservice if has_self_uri
   default_backend selfservice

backend horizon
   mode http
   opion  forwardfor
   option  httpchk
   option  httpclose
   option  httplog
   stick  on src
   stick-table  type ip size 200k expire 30m
   cookie SERVERID insert indirect nocache
   server node-0 ctl01.net.example.com:80 check inter 5000 rise 2 fall 3 cookie node-0
   server node-1 ctl02.net.example.com:80 check inter 5000 rise 2 fall 3 cookie node-1
   server node-2 ctl03.net.example.com:80 check inter 5000 rise 2 fall 3 cookie node-2

backend selfservice
   mode http
   option  forwardfor
   option  httpclose
   option  httplog
   balance  source
   stick  on src
   stick-table  type ip size 200k expire 30m
   cookie SERVERID insert indirect nocache
   acl is_root path -i /
   acl is_domain hdr(host) -i cloud.example.com
   redirect code 301 location https://cloud.example.com/self_service/ if is_domain is_root { ssl_fc }
   server miq0 miq01.net.example.com:80 check inter 5000 rise 2 fall 3 cookie miq0
   server miq1 miq02.net.example.com:80 check inter 5000 rise 2 fall 3 cookie miq1

Now I can use cookie for balancing and without decrease security (I does not disable secure cookie store).
It seems that I cannot use haproxy for balancing directly to puma servervs because of static content on the manageiq server. But maybe I’ll find solution for this problem. I found the ability to configure the listening address for puma server via an environment variable:

 [vmdb]# grep BIND /etc/default/evm
  export BINDING_ADDRESS=0.0.0.0

#9

A small remark for my configuration. If SSO is configured through the following documentation: http://manageiq.org/docs/reference/latest/auth/active_directory, then the keytab file you should to produce for the external name of the load balancer. For example, if you have multiple applicances miq01/miq02, etc., and the external name of balancer is cloud.domain.local then in the Active Directory domain.local must be a “cloud” account with the configured SPN for HOST/miq[01/02]


#10

@igortiunov: I have taken a recent “Fine” nightly appliance and have started testing it behind a Apache proxy. It seems to work fine. Have not yet tested the websocket stuf, will do that too.

Now I am trying the haproxy. I’ll let you know how far I get.

Using Fine instead of Euwe is crucial. For the proxy to work you need the PRs I referenced earlier in this thread.


#11

So trivial haproxy:

defaults
    mode                    http
    .....

frontend main
    bind *:5000
    default_backend             app

backend app
    balance     roundrobin
    server  app1 192.168.122.81:80 check

seems to work for me (have not tested the websockets yet)


#12

I got both HAproxy and the Apache proxy balancer configuration running including websockets.

The configuration is available here:

I want to so testing with more servers (so that it’s actually balancing or doing some HA) and I also want to test AWS proxy…


#13

Hi @martinpovolny thank you for your explanation. Can you please explain network scheme for your environment ? The backend in haproxy config is just MIQ Appliance ?


#14

I have updated the docs in the wiki above. Most important is that to get the websocket based stuff running, you have to setup a shared memcache server. W/o shared memcached the API and the notification and console cannot work properly. I have created a ticket: https://github.com/ManageIQ/manageiq/issues/14882

As for my environment: right now I have 2 appliances and one front-end server. The appliances are nightly Fine builds. For the front end (running haproxy and apache) I have used latest Fedora (just to be sure that I have Apache and HAproxy in version where wss:// proxying works).

I am running everything in KVM environment on my laptop and I am hitting the performance limit of my W541 Lenovo :wink: So next I am moving my testing to a RHEV or Vsphere and recreating the environment.


#15

After reading Peter McGowan Reference Architecture I found that more intelligence way is to use /ping url for MIQ health checking. So for haproxy there should be the following config:

option httpchk HEAD /ping HTTP/1.0

https://access.redhat.com/documentation/en-us/reference_architectures/2017/html/deploying_cloudforms_at_scale/web-user-interface#load_balancers


#16

I did not have time to work on this since the least note.

But since https://github.com/ManageIQ/manageiq/pull/14947 is merged the shared memcache is not needed any more as SQL database can be used as the session store.