General Information

Creating a highly available pair of load balancers with HAProxy and Keepalived.

Checklist

• Number of systems
  • 2 servers to be load balancers
  • 2 servers for web servers (in the example)
• Distro(s): Enterprise Linux 7

Network configuration used in the examples below.

Load Balancers

• Server “lb01” → 10.1.2.1 (eth0)
• Server “lb02” → 10.1.2.2 (eth0)
• “lbvip” → 10.1.2.3 (load balancer virtual IP - floats between servers)

Web Servers (used in haproxy example config)

• web01 → 10.1.2.50
• web02 → 10.1.2.51

Install the required packages on the load balancer servers

• KeepAliveD (high availability)

  yum install keepalived

• HA-Proxy (load balancing)

  yum install haproxy

Configuring keepalived and haproxy.

Keepalived utlizes a Linux kernel implementation of VRRP. (Virtual Router Redundancy Protocol)

Official Site: http://www.keepalived.org/

• Configure all nodes with these keepalive settings (/etc/keepalived/keepalived.conf). Example:

  ! Configuration File for keepalived
   
  vrrp_script check_haproxy {
    script "killall -0 haproxy"  # check the haproxy process
    timeout 1
    interval 2  # every 2 seconds
    weight 2  # add 2 points if OK
  }
   
  vrrp_instance VI_1 {
      state BACKUP  # All instances 'BACKUP' to prevent VIP flapping
      interface eth0
      virtual_router_id 51
      priority 100  # All instances same priority to prevent VIP flapping
      advert_int 1
   
      authentication {
        auth_type PASS
        auth_pass PASSWORDHERE
      }
   
      virtual_ipaddress {
        10.1.2.3
      }
   
    track_script {
      check_haproxy
    }
  }

HAProxy is a TCP/HTTP load balancer.

Official Site: http://www.haproxy.org/

• Configure HA-Proxy (/etc/haproxy/haproxy.cfg)
  • Remove all example frontend and backend config sections (leave default section)
  • Add a section for the HAProxy Stats page

    #---------------------------------------------------------------------
    # HAProxy Stats
    #---------------------------------------------------------------------
    listen stats
      # SSL Mode and Cert
      bind *:9000 ssl crt /etc/pki/tls/mycertfiles.pem
      mode http
     
      # Enable Stats and Hide Version
      stats enable
      stats hide-version
     
      # Authentication realm. This can be set to anything. Escape space characters with a backslash.
      stats realm HAProxy\ Statistics
     
      # The virtual URL to access the stats page
      stats uri /haproxy_stats
     
      # The user/pass you want to use. Change this password!
      stats auth admin:adminpassword

• The pem certificate file is a concatenation of the SSL key, cert, and certificate authority. Example

  cat mykey.key mycert.crt myCAs.crt >> mycertfiles.pem

• Create new directory to hold frontend/backend config files

  mkdir /etc/haproxy/config.d

• Create new frontend/backend config files (Example: /etc/haproxy/config.d/http.cfg)
  • Add New frontend/backend sections Example:

    #---------------------------------------------------------------------
    # fe_http frontend which proxys to the backends
    #---------------------------------------------------------------------
    frontend  fe_http *:80
        # Log format
        option httplog
     
        # Timeout Settings
        #no option http-server-close
        #timeout client 1m  #default: 50s
     
        #-- ACLs - Match HTTP Requests --#
        acl url_web       path_beg    -i /mywebsite
     
        #-- Backend Selection based on ACLs --#
        use_backend be_web_pool1    if url_web
     
        # If not using ACLs for backend selection or to have a fall back selection
        #default_backend be_web_pool1
     
    #---------------------------------------------------------------------
    # Backend Configuration
    #---------------------------------------------------------------------
    backend be_web_pool1
        # Replace "/mywebsite/" with "/" at the beginning of the request
        reqirep ^([^\ ]*\ /)mywebsite[/]?(.*)  \1\2
     
        # Backend Protocol
        mode http
     
        #-- Timeout Settings --#
        #timeout connect 1m  #default: 5s
        #timeout server 2m  #default: 50s
     
        #-- Health check options --#
        # Use http layer 7 check instead of default layer 4 port check
        option httpchk HEAD /
        # inter: How often to execute a health check (default: 2s)
        # rise: Number of consecutive checks before server is UP (default: 2)
        # fall: Number of consecutive checks before server is DOWN (default: 3)
        default-server inter 5s rise 2 fall 3
        # timeout check: Fail health check after x seconds of no response (default: 10s)
        timeout check 12s
     
        #-- Balancing --#
        balance  leastconn
        # fullconn: does nothing since we are not using minconn (just makes the dashboard less confusing)
        fullconn 1000
        server  web01 10.1.2.50:80 check maxconn 500
        server  web02 10.1.2.51:80 check maxconn 500

• Ensure each additional config file in config.d/ is setup in haproxy's environment options(/etc/sysconfig/haproxy)

  # Config files specifying frontend/backends
  OPTIONS="-f /etc/haproxy/config.d/http.cfg"

• Multiple config files example:

  OPTIONS="-f /etc/haproxy/config.d/http.cfg -f /etc/haproxy/config.d/otherfrontend.cfg"

Session Persistence

• Cookies: Application layer persistence (app needs to support cookies)

      #-- Balancing --#
      balance  leastconn
      # Use Cookie for Session Persistence
      cookie SERVERID insert indirect nocache
      # fullconn: does nothing since we are not using minconn (just makes the dashboard less confusing)
      fullconn 1000
      server  web01 10.1.2.50:80 check cookie web01 maxconn 500
      server  web02 10.1.2.51:80 check cookie web02 maxconn 500

• Source IP: Affinity based on source IP hash (app doesn't need to know about it)

      #-- Balancing --#
      balance  source
      # fullconn: does nothing since we are not using minconn (just makes the dashboard less confusing)
      fullconn 1000
      server  web01 10.1.2.50:80 check maxconn 500
      server  web02 10.1.2.51:80 check maxconn 500

Setup logging for HAProxy.

• Create a Rsyslog drop in file for HA-Proxy (/etc/rsyslog.d/haproxy.conf)

  ## HA-Proxy Rsyslog Config ##
   
  # Load UDP Modules
  $ModLoad imudp
   
  # Run UDP server
  $UDPServerRun 514
   
  # Allow only localhost
  $AllowedSender UDP, 127.0.0.1
   
  # Send local2 haproxy logs to /var/log/haproxy.log
  local2.none  /var/log/messages
  local2.*     /var/log/haproxy.log

• Restart rsyslog

  systemctl restart rsyslog

Operating the load balancers.

Start and enable the services on each node.

• HA-Proxy

  systemctl start haproxy
  systemctl enable haproxy

• Keepalived

  systemctl start keepalived
  systemctl enable keepalived

Reboot procedure and dependencies.

• Load Balancers (lb01, lb02) can be rebooted 1 at a time to avoid service interruption.
• Determine the inactive system (the system that does NOT have the virtual IP as a secondary address

  ip addr sh

  • Reboot the inactive system

    reboot

  • Once the inactive system is up, verify keepalived and haproxy are running

    systemctl status keepalived haproxy

• Stop keepalived on the active system in order to force a fail over

  systemctl stop keepalived

  • Verify connections to the frontend listeners go away

    netstat -anpt | grep haproxy | grep -v 9000

  • Reboot the system with keepalived stopped and no more client connections

    reboot