linux_wiki:load_balancing_haproxy_and_keepalived

Load Balancing with HAProxy And Keepalived

General Information

Creating a highly available pair of load balancers with HAProxy and Keepalived.

Checklist

  • Number of systems
    • 2 servers to be load balancers
    • 2 servers for web servers (in the example)
  • Distro(s): Enterprise Linux 7

Network Addressing Setup

Network configuration used in the examples below.

Load Balancers

  • Server “lb01” → 10.1.2.1 (eth0)
  • Server “lb02” → 10.1.2.2 (eth0)
  • “lbvip” → 10.1.2.3 (load balancer virtual IP - floats between servers)

Web Servers (used in haproxy example config)

  • web01 → 10.1.2.50
  • web02 → 10.1.2.51




Install

Install the required packages on the load balancer servers

  • KeepAliveD (high availability)
    yum install keepalived
  • HA-Proxy (load balancing)
    yum install haproxy

Configure

Configuring keepalived and haproxy.


Keepalived utlizes a Linux kernel implementation of VRRP. (Virtual Router Redundancy Protocol)

Official Site: http://www.keepalived.org/

  • Configure all nodes with these keepalive settings (/etc/keepalived/keepalived.conf). Example:
    ! Configuration File for keepalived
     
    vrrp_script check_haproxy {
      script "killall -0 haproxy"  # check the haproxy process
      timeout 1
      interval 2  # every 2 seconds
      weight 2  # add 2 points if OK
    }
     
    vrrp_instance VI_1 {
        state BACKUP  # All instances 'BACKUP' to prevent VIP flapping
        interface eth0
        virtual_router_id 51
        priority 100  # All instances same priority to prevent VIP flapping
        advert_int 1
     
        authentication {
          auth_type PASS
          auth_pass PASSWORDHERE
        }
     
        virtual_ipaddress {
          10.1.2.3
        }
     
      track_script {
        check_haproxy
      }
    }

HAProxy is a TCP/HTTP load balancer.

Official Site: http://www.haproxy.org/

  • Configure HA-Proxy (/etc/haproxy/haproxy.cfg)
    • Remove all example frontend and backend config sections (leave default section)
    • Add a section for the HAProxy Stats page
      #---------------------------------------------------------------------
      # HAProxy Stats
      #---------------------------------------------------------------------
      listen stats
        # SSL Mode and Cert
        bind *:9000 ssl crt /etc/pki/tls/mycertfiles.pem
        mode http
       
        # Enable Stats and Hide Version
        stats enable
        stats hide-version
       
        # Authentication realm. This can be set to anything. Escape space characters with a backslash.
        stats realm HAProxy\ Statistics
       
        # The virtual URL to access the stats page
        stats uri /haproxy_stats
       
        # The user/pass you want to use. Change this password!
        stats auth admin:adminpassword
  • The pem certificate file is a concatenation of the SSL key, cert, and certificate authority. Example
    cat mykey.key mycert.crt myCAs.crt >> mycertfiles.pem
  • Create new directory to hold frontend/backend config files
    mkdir /etc/haproxy/config.d
  • Create new frontend/backend config files (Example: /etc/haproxy/config.d/http.cfg)
    • Add New frontend/backend sections Example:
      #---------------------------------------------------------------------
      # fe_http frontend which proxys to the backends
      #---------------------------------------------------------------------
      frontend  fe_http *:80
          # Log format
          option httplog
       
          # Timeout Settings
          #no option http-server-close
          #timeout client 1m  #default: 50s
       
          #-- ACLs - Match HTTP Requests --#
          acl url_web       path_beg    -i /mywebsite
       
          #-- Backend Selection based on ACLs --#
          use_backend be_web_pool1    if url_web
       
          # If not using ACLs for backend selection or to have a fall back selection
          #default_backend be_web_pool1
       
      #---------------------------------------------------------------------
      # Backend Configuration
      #---------------------------------------------------------------------
      backend be_web_pool1
          # Replace "/mywebsite/" with "/" at the beginning of the request
          reqirep ^([^\ ]*\ /)mywebsite[/]?(.*)  \1\2
       
          # Backend Protocol
          mode http
       
          #-- Timeout Settings --#
          #timeout connect 1m  #default: 5s
          #timeout server 2m  #default: 50s
       
          #-- Health check options --#
          # Use http layer 7 check instead of default layer 4 port check
          option httpchk HEAD /
          # inter: How often to execute a health check (default: 2s)
          # rise: Number of consecutive checks before server is UP (default: 2)
          # fall: Number of consecutive checks before server is DOWN (default: 3)
          default-server inter 5s rise 2 fall 3
          # timeout check: Fail health check after x seconds of no response (default: 10s)
          timeout check 12s
       
          #-- Balancing --#
          balance  leastconn
          # fullconn: does nothing since we are not using minconn (just makes the dashboard less confusing)
          fullconn 1000
          server  web01 10.1.2.50:80 check maxconn 500
          server  web02 10.1.2.51:80 check maxconn 500
  • Ensure each additional config file in config.d/ is setup in haproxy's environment options(/etc/sysconfig/haproxy)
    # Config files specifying frontend/backends
    OPTIONS="-f /etc/haproxy/config.d/http.cfg"
  • Multiple config files example:
    OPTIONS="-f /etc/haproxy/config.d/http.cfg -f /etc/haproxy/config.d/otherfrontend.cfg"

Session Persistence

  • Cookies: Application layer persistence (app needs to support cookies)
        #-- Balancing --#
        balance  leastconn
        # Use Cookie for Session Persistence
        cookie SERVERID insert indirect nocache
        # fullconn: does nothing since we are not using minconn (just makes the dashboard less confusing)
        fullconn 1000
        server  web01 10.1.2.50:80 check cookie web01 maxconn 500
        server  web02 10.1.2.51:80 check cookie web02 maxconn 500
  • Source IP: Affinity based on source IP hash (app doesn't need to know about it)
        #-- Balancing --#
        balance  source
        # fullconn: does nothing since we are not using minconn (just makes the dashboard less confusing)
        fullconn 1000
        server  web01 10.1.2.50:80 check maxconn 500
        server  web02 10.1.2.51:80 check maxconn 500

Setup logging for HAProxy.

  • Create a Rsyslog drop in file for HA-Proxy (/etc/rsyslog.d/haproxy.conf)
    ## HA-Proxy Rsyslog Config ##
     
    # Load UDP Modules
    $ModLoad imudp
     
    # Run UDP server
    $UDPServerRun 514
     
    # Allow only localhost
    $AllowedSender UDP, 127.0.0.1
     
    # Send local2 haproxy logs to /var/log/haproxy.log
    local2.none  /var/log/messages
    local2.*     /var/log/haproxy.log
  • Restart rsyslog
    systemctl restart rsyslog

Operate

Operating the load balancers.


Start and enable the services on each node.

  • HA-Proxy
    systemctl start haproxy
    systemctl enable haproxy
  • Keepalived
    systemctl start keepalived
    systemctl enable keepalived

Reboot procedure and dependencies.

  • Load Balancers (lb01, lb02) can be rebooted 1 at a time to avoid service interruption.
  • Determine the inactive system (the system that does NOT have the virtual IP as a secondary address
    ip addr sh
    • Reboot the inactive system
      reboot
    • Once the inactive system is up, verify keepalived and haproxy are running
      systemctl status keepalived haproxy
  • Stop keepalived on the active system in order to force a fail over
    systemctl stop keepalived
    • Verify connections to the frontend listeners go away
      netstat -anpt | grep haproxy | grep -v 9000
    • Reboot the system with keepalived stopped and no more client connections
      reboot

  • linux_wiki/load_balancing_haproxy_and_keepalived.txt
  • Last modified: 2019/05/25 23:50
  • (external edit)