====== Load Balancing with HAProxy And Keepalived ======

**General Information**

Creating a highly available pair of load balancers with HAProxy and Keepalived. 

**Checklist**
  * Number of systems
    * 2 servers to be load balancers
    * 2 servers for web servers (in the example)
  * Distro(s): Enterprise Linux 7

----

====== Network Addressing Setup ======

Network configuration used in the examples below.

Load Balancers
  * Server "lb01" -> 10.1.2.1 (eth0)
  * Server "lb02" -> 10.1.2.2 (eth0)
  * "lbvip" -> 10.1.2.3 (load balancer virtual IP - floats between servers)

Web Servers (used in haproxy example config)
  * web01 -> 10.1.2.50
  * web02 -> 10.1.2.51

\\
{{ haproxy_keepalived_example.jpg |}}
\\

----

====== Install ======

Install the required packages on the load balancer servers
  * KeepAliveD (high availability)<code bash>yum install keepalived</code>
  * HA-Proxy (load balancing)<code bash>yum install haproxy</code>

----

====== Configure ======

Configuring keepalived and haproxy.

----

===== Keepalived =====

Keepalived utlizes a Linux kernel implementation of VRRP. (Virtual Router Redundancy Protocol)

Official Site: http://www.keepalived.org/

  * Configure all nodes with these keepalive settings (/etc/keepalived/keepalived.conf). **Example**:<code bash>! Configuration File for keepalived

vrrp_script check_haproxy {
  script "killall -0 haproxy"  # check the haproxy process
  timeout 1
  interval 2  # every 2 seconds
  weight 2  # add 2 points if OK
}

vrrp_instance VI_1 {
    state BACKUP  # All instances 'BACKUP' to prevent VIP flapping
    interface eth0
    virtual_router_id 51
    priority 100  # All instances same priority to prevent VIP flapping
    advert_int 1

    authentication {
      auth_type PASS
      auth_pass PASSWORDHERE
    }

    virtual_ipaddress {
      10.1.2.3
    }

  track_script {
    check_haproxy
  }
}</code>

----

===== HA-Proxy =====

HAProxy is a TCP/HTTP load balancer.

Official Site: http://www.haproxy.org/

==== Main Config ====

  * Configure HA-Proxy (/etc/haproxy/haproxy.cfg)
    * Remove all example frontend and backend config sections (leave default section)
    * Add a section for the HAProxy Stats page<code bash>#---------------------------------------------------------------------
# HAProxy Stats
#---------------------------------------------------------------------
listen stats
  # SSL Mode and Cert
  bind *:9000 ssl crt /etc/pki/tls/mycertfiles.pem
  mode http

  # Enable Stats and Hide Version
  stats enable
  stats hide-version

  # Authentication realm. This can be set to anything. Escape space characters with a backslash.
  stats realm HAProxy\ Statistics

  # The virtual URL to access the stats page
  stats uri /haproxy_stats

  # The user/pass you want to use. Change this password!
  stats auth admin:adminpassword</code>
  * The pem certificate file is a concatenation of the SSL key, cert, and certificate authority. Example<code bash>cat mykey.key mycert.crt myCAs.crt >> mycertfiles.pem</code>

==== Frontend/Backend Configs ====

    * Create new directory to hold frontend/backend config files<code bash>mkdir /etc/haproxy/config.d</code>
    * Create new frontend/backend config files (Example: /etc/haproxy/config.d/http.cfg)
      * Add New frontend/backend sections **Example**:<code bash>#---------------------------------------------------------------------
# fe_http frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  fe_http *:80
    # Log format
    option httplog

    # Timeout Settings
    #no option http-server-close
    #timeout client 1m  #default: 50s
    
    #-- ACLs - Match HTTP Requests --#
    acl url_web       path_beg    -i /mywebsite

    #-- Backend Selection based on ACLs --#
    use_backend be_web_pool1    if url_web

    # If not using ACLs for backend selection or to have a fall back selection
    #default_backend be_web_pool1
    
#---------------------------------------------------------------------
# Backend Configuration
#---------------------------------------------------------------------
backend be_web_pool1
    # Replace "/mywebsite/" with "/" at the beginning of the request
    reqirep ^([^\ ]*\ /)mywebsite[/]?(.*)  \1\2

    # Backend Protocol
    mode http

    #-- Timeout Settings --#
    #timeout connect 1m  #default: 5s
    #timeout server 2m  #default: 50s
    
    #-- Health check options --#
    # Use http layer 7 check instead of default layer 4 port check
    option httpchk HEAD /
    # inter: How often to execute a health check (default: 2s)
    # rise: Number of consecutive checks before server is UP (default: 2)
    # fall: Number of consecutive checks before server is DOWN (default: 3)
    default-server inter 5s rise 2 fall 3
    # timeout check: Fail health check after x seconds of no response (default: 10s)
    timeout check 12s
    
    #-- Balancing --#
    balance  leastconn
    # fullconn: does nothing since we are not using minconn (just makes the dashboard less confusing)
    fullconn 1000
    server  web01 10.1.2.50:80 check maxconn 500
    server  web02 10.1.2.51:80 check maxconn 500</code>
  * Ensure each additional config file in config.d/ is setup in haproxy's environment options(/etc/sysconfig/haproxy)<code bash># Config files specifying frontend/backends
OPTIONS="-f /etc/haproxy/config.d/http.cfg"</code>
    * Multiple config files example:<code bash>OPTIONS="-f /etc/haproxy/config.d/http.cfg -f /etc/haproxy/config.d/otherfrontend.cfg"</code>

==== Additional Config Examples ====

**Session Persistence**
  * Cookies: Application layer persistence (app needs to support cookies)<code bash>    #-- Balancing --#
    balance  leastconn
    # Use Cookie for Session Persistence
    cookie SERVERID insert indirect nocache
    # fullconn: does nothing since we are not using minconn (just makes the dashboard less confusing)
    fullconn 1000
    server  web01 10.1.2.50:80 check cookie web01 maxconn 500
    server  web02 10.1.2.51:80 check cookie web02 maxconn 500</code>
  * Source IP: Affinity based on source IP hash (app doesn't need to know about it)<code bash>    #-- Balancing --#
    balance  source
    # fullconn: does nothing since we are not using minconn (just makes the dashboard less confusing)
    fullconn 1000
    server  web01 10.1.2.50:80 check maxconn 500
    server  web02 10.1.2.51:80 check maxconn 500</code>

----

===== Logging =====

Setup logging for HAProxy.

  * Create a Rsyslog drop in file for HA-Proxy (/etc/rsyslog.d/haproxy.conf)<code bash>## HA-Proxy Rsyslog Config ##

# Load UDP Modules
$ModLoad imudp

# Run UDP server
$UDPServerRun 514

# Allow only localhost
$AllowedSender UDP, 127.0.0.1

# Send local2 haproxy logs to /var/log/haproxy.log
local2.none  /var/log/messages
local2.*     /var/log/haproxy.log</code>
  * Restart rsyslog<code bash>systemctl restart rsyslog</code>

----

====== Operate ======

Operating the load balancers.

----

===== Services =====

Start and enable the services on each node.

  * HA-Proxy<code bash>systemctl start haproxy
systemctl enable haproxy</code>
  * Keepalived<code bash>systemctl start keepalived
systemctl enable keepalived</code>

----

===== Reboots =====

Reboot procedure and dependencies.

  * Load Balancers (lb01, lb02) can be rebooted 1 at a time to avoid service interruption.
  * Determine the **inactive system** (the system that does NOT have the virtual IP as a secondary address<code bash>ip addr sh</code>
    * Reboot the **inactive system**<code bash>reboot</code>
    * Once the inactive system is up, verify keepalived and haproxy are running<code bash>systemctl status keepalived haproxy</code>
  * Stop keepalived on the active system in order to force a fail over<code bash>systemctl stop keepalived</code>
    * Verify connections to the frontend listeners go away<code bash>netstat -anpt | grep haproxy | grep -v 9000</code>
    * Reboot the system with keepalived stopped and no more client connections<code bash>reboot</code>

----