ManageIQ behind load balancer

Hi Guys. I try to switch manageiq frontend to http instead of https for load balancing purposes. I uncomment the appropriate settings in the manageiq-http.conf file. But now I cannot login to administrative interface. Any time I enter login/password of admin user I redirect to ?timeout=false url. If I login to https everything work fine. There is a log from production.log file: http://paste.openstack.org/show/604265/

Hi!

What version of ManageIQ do you have?

I have been trying the operation behind a proxy recently and needed 2 fixes: https://github.com/ManageIQ/manageiq-ui-classic/pull/448 and https://github.com/ManageIQ/manageiq-ui-classic/pull/583

In the 2nd PR, you can see the config that we tested with David:

# Be more permissive with SSL certificates
SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off

# Make sure that X_FORWARDED is set
RequestHeader set X_FORWARDED_PROTO 'https'
RequestHeader set X-Forwarded-Ssl on
AllowEncodedSlashes NoDecode

# Load balancer for HTTP(S) requests
<Proxy balancer://manageiq>
  BalancerMember https://localhost:8443
</Proxy>

# Separate load balancer for WS(S) requests
<Proxy balancer://websocket>
  BalancerMember wss://localhost:8443
</Proxy>

# These two lines should always go BEFORE the rules for /
ProxyPass /ws/ balancer://websocket/ws/
ProxyPassReverse /ws/ balancer://websocket/ws/

ProxyPass / balancer://manageiq/
ProxyPassReverse / balancer://manageiq/

And please, share where you got. Seems more people are trying various load balancer configurations.

Hi @martinpovolny

I use euwe and I want to use haproxy. I use it for openstack deployment and I want to use one access point for users to openstack dashboard and manageiq self_service portal:
http://paste.openstack.org/show/604272/
In this config I use http for access to manageiq backend and for self_service it work fine, but I cannot login to admin portal. If I access manageiq directly to http://miq01.example.com I have the same issue - browser just redirected to ?timeout=true url.

If I configure haproxy backends through https like this:

backend selfservice
    server miq0 miq01.example.com:443 check ssl verify none

then I have a working frontend.

Now I found that my problem is related to cookie storage. If I disable secure option for session_store:

grep secure config/initializers/session_store.rb    
    session_options[:secure]   = false

than I can connect to MIQ. But this option decreases the security because of MITM between haproxy and MIQ apache. May be I need to configure haproxy backend directly to puma server. But I dont know how to set puma listener to 0.0.0.0.

You cannot have HTTPS only cookies if you are going through HTTP.

I need more info on what goes wrong in your setup. But I have a guess. You wrote that you are using Euwe. That means that you don’t have the patches from PR number one above. Then I guess you cannot get pass the login screen because the login form is being sumbited to a wrong URL (internal appliance instead of the haproxy). You can confirm this using the Network tab of your browser debug tool (CTRL-SHIFT-J in Chrome and FF).

I don’t have a haproxy instance at hand but I can try that and check where I get. Can you give me your haproxy setup here? (I am not familiar with haproxy).

I apologize for my English.
First I want to enable access to ManageIQ via http. For this, I uncomment appropriate configuration in /etc/httpd/conf.d/manageiq-http.conf. If I try to login to administrative portal via http I get the error. The error is related to cookie store and if I disable this option (session_options[:secure] = false) I can access to manageiq via http.

Second I want to configure haproxy to access to manageiq and there is my configuration for haproxy.cfg: http://paste.openstack.org/show/604272/

But may be I chose not the right way and I need to configure haproxy to balance to puma servers directly instead of apache on the manageiq.

Currently there is my final configuration for haproxy and http virtual host on the apache for manageiq.
First I need to enable http access to manageiq appliance for smart load balancing through haproxy:

RewriteEngine On
Options SymLinksIfOwnerMatch
<VirtualHost *:80>
  DocumentRoot /var/www/miq/vmdb/public
  Include conf.d/manageiq-redirects-ui
  Include conf.d/manageiq-redirects-ws
  Include conf.d/manageiq-redirects-websocket
  ProxyPreserveHost on
  RequestHeader set X_FORWARDED_PROTO 'https'
  <Location /assets/>
     Header unset ETag
     FileETag None
     ExpiresActive On
     ExpiresDefault "access plus 1 year"
  </Location>
  <Location /proxy_pages/>
     ErrorDocument 403 /error/noindex.html
     ErrorDocument 404 /error/noindex.html
  </Location>
</VirtualHost>

I need to enable this setting: RequestHeader set X_FORWARDED_PROTO ‘https’ because of ‘secure’ configuration of cookies for rails application (config/initializers/session_store.rb: session_options[:secure] = true).

My haproxy uses https listeners and tls termination work on them:

frontend cloud
   bind cloud.example.com:443 ssl crt /etc/haproxy/ssl/cert.pem no-sslv3 no-tls-tickets ciphers AES128+EECDH:AES128+EDH:AES256+EECDH:AES256+EDH
   mode http
   reqadd X-Forwarded-Proto:\ https
   timeout  client 3h
   timeout  server 3h
   acl has_dashboard_uri url_beg /dashboard
   acl is_os_domain hdr(host) -i cloud.example.com
   use_backend horizon if has_dashboard_uri is_os_domain
   acl has_self_uri url_beg /self_service
   use_backend selfservice if has_self_uri
   default_backend selfservice

backend horizon
   mode http
   opion  forwardfor
   option  httpchk
   option  httpclose
   option  httplog
   stick  on src
   stick-table  type ip size 200k expire 30m
   cookie SERVERID insert indirect nocache
   server node-0 ctl01.net.example.com:80 check inter 5000 rise 2 fall 3 cookie node-0
   server node-1 ctl02.net.example.com:80 check inter 5000 rise 2 fall 3 cookie node-1
   server node-2 ctl03.net.example.com:80 check inter 5000 rise 2 fall 3 cookie node-2

backend selfservice
   mode http
   option  forwardfor
   option  httpclose
   option  httplog
   balance  source
   stick  on src
   stick-table  type ip size 200k expire 30m
   cookie SERVERID insert indirect nocache
   acl is_root path -i /
   acl is_domain hdr(host) -i cloud.example.com
   redirect code 301 location https://cloud.example.com/self_service/ if is_domain is_root { ssl_fc }
   server miq0 miq01.net.example.com:80 check inter 5000 rise 2 fall 3 cookie miq0
   server miq1 miq02.net.example.com:80 check inter 5000 rise 2 fall 3 cookie miq1

Now I can use cookie for balancing and without decrease security (I does not disable secure cookie store).
It seems that I cannot use haproxy for balancing directly to puma servervs because of static content on the manageiq server. But maybe I’ll find solution for this problem. I found the ability to configure the listening address for puma server via an environment variable:

 [vmdb]# grep BIND /etc/default/evm
  export BINDING_ADDRESS=0.0.0.0

A small remark for my configuration. If SSO is configured through the following documentation: http://manageiq.org/docs/reference/latest/auth/active_directory, then the keytab file you should to produce for the external name of the load balancer. For example, if you have multiple applicances miq01/miq02, etc., and the external name of balancer is cloud.domain.local then in the Active Directory domain.local must be a “cloud” account with the configured SPN for HOST/miq[01/02]

@igortiunov: I have taken a recent “Fine” nightly appliance and have started testing it behind a Apache proxy. It seems to work fine. Have not yet tested the websocket stuf, will do that too.

Now I am trying the haproxy. I’ll let you know how far I get.

Using Fine instead of Euwe is crucial. For the proxy to work you need the PRs I referenced earlier in this thread.

So trivial haproxy:

defaults
    mode                    http
    .....

frontend main
    bind *:5000
    default_backend             app

backend app
    balance     roundrobin
    server  app1 192.168.122.81:80 check

seems to work for me (have not tested the websockets yet)

I got both HAproxy and the Apache proxy balancer configuration running including websockets.

The configuration is available here:

I want to so testing with more servers (so that it’s actually balancing or doing some HA) and I also want to test AWS proxy…

Hi @martinpovolny thank you for your explanation. Can you please explain network scheme for your environment ? The backend in haproxy config is just MIQ Appliance ?

I have updated the docs in the wiki above. Most important is that to get the websocket based stuff running, you have to setup a shared memcache server. W/o shared memcached the API and the notification and console cannot work properly. I have created a ticket: https://github.com/ManageIQ/manageiq/issues/14882

As for my environment: right now I have 2 appliances and one front-end server. The appliances are nightly Fine builds. For the front end (running haproxy and apache) I have used latest Fedora (just to be sure that I have Apache and HAproxy in version where wss:// proxying works).

I am running everything in KVM environment on my laptop and I am hitting the performance limit of my W541 Lenovo :wink: So next I am moving my testing to a RHEV or Vsphere and recreating the environment.

After reading Peter McGowan Reference Architecture I found that more intelligence way is to use /ping url for MIQ health checking. So for haproxy there should be the following config:

option httpchk HEAD /ping HTTP/1.0

https://access.redhat.com/documentation/en-us/reference_architectures/2017/html/deploying_cloudforms_at_scale/web-user-interface#load_balancers

I did not have time to work on this since the least note.

But since https://github.com/ManageIQ/manageiq/pull/14947 is merged the shared memcache is not needed any more as SQL database can be used as the session store.