HAProxy automatic failover

HAProxy automatic failover

HAProxy is a TCP load balancing tool with some useful features, including ACLs and SSL termination support. I’ve been using it for a while now on a number of load-balanced sites where scalability is key. What I haven’t yet looked at, however, is automated failover capabilities.

My aims were simple:

• If a server fails, stop using it.
• If said server starts working again (i.e. because the problem is fixed) start using it again.
• If all servers in the load balancer pool fail, serve a temporary static page from another location.

But first: Some basic HAProxy concepts
HAProxy as a load balancer is fairly simple, and works on the basis of defined frontends and backends. A frontend is simply an IP and port declaration that you want the load balancer to listen on. A backend is the set of servers that requests to a frontend are sent to. Your HAProxy configuration lists the frontends and their respective backends, as well as the load balancing algorithm you wish to use to distribute traffic amongst the backend nodes. Here’s an example- one frontend proxies to two nodes at the backend.

 frontend my_tls_frontend
          mode tcp
         bind *:443 # port/IP to bind on
         bind ipv6@:443
          default_backend my_tls_backend # Backend to proxy to

 ...

 backend my_tls_backend
    mode tcp
     balance     roundrobin # Balancing algorithm
     server node1 10.0.0.1:443 # Server 1
     server node2 10.0.0.2:443 # Server 2

The ‘balance’ statement here defines a load balancing algorithm to use:
roundrobin: This is the default option, and simply picks between each node in turn. It can also be weighted to prioritise some nodes above others.
leastconn: Picks the node with the least number of active connections (i.e. the least busy server).
source: Hashes the visitor IP to determine which node to send a user to. This ensures that, assuming nothing changes in the pool, a visitor will always hit the same backend node.
I was interested in HAProxy’s state-checking abilities:

Introducing: HAProxy health checking

HAProxy has some fairly broad health-checking features which allow it to check the state of any server in a backend pool on a set interval.
At it’s most basic, you simply add the ‘check’ statement to each backend node:

  server node1 10.0.0.1:443 check # Server 1
  server node2 10.0.0.2:443 check # Server 2

This instructs HAProxy to perform basic healthchecks on the servers, by opening a TCP connection and verifying if it gets a response. If it doesn’t, then it marks the offending server as “Down” and stops directing traffic to it.

That’s all well and good, but what if we need a more detailed check? Fortunately, HAProxy has the ability to check services in more depth. In this case, we want to send a HTTP request to each backend and check whether we get a good HTTP response back (i.e. a 2XX or 3XX code).

This is done by defining a ‘httpchk’ option statement in your backend configuration, which tells HAProxy to request each server over HTTP. In your backend definition, add:

  option httpchk GET / HTTP/1.1\r\nHost:\ blog.jcdev.org

(I know HAProxy’s method is a little hacky- having the whole request on one line).
The above is a GET request / for each backend, defining a Host header as appropriate to ensure the backend servers handle the request correctly.

Next, we need to define the health checking interval, and the conditions for marking a node as ‘Down’ and then (hopefully) ‘Up’ again. This is done in the backend with the ‘inter’ statement, which looks like this:

 default-server inter 60s fall 2 rise 5

What this says is:

• Inter: This is the interval at which to check each node.
• Fall: The number of requests that have to fail before you mark the server as ‘Down’.
• Rise: The number of requests that have to succeed before you mark the server as ‘Up’ again.

The above settings are quite trigger-happy, so feel free to adjust them to your needs.

Backup nodes

Finally- in the event of a failure of all the nodes we’re using, we want to define a ‘backup’ server to direct requests to. Although HAProxy has custom error support to display a holding page in the event all nodes in a backend aren’t available, I wanted something a little more flexible, so I spun up an instance of Nginx on port 8443 to serve a simple message. Adding this to our load balancer configuration is as simple as adding the node with the ‘backup’ keyword:

server backup 127.0.0.1:8443 backup

This ensures that it isn’t part of the normal pool for serving load-balanced traffic.
Our completed backend configuration would look something like this:

 backend my_tls_backend
     mode tcp
      balance roundrobin
      default-server inter 60s fall 1 rise 2
      option httpchk GET / HTTP/1.1\r\nHost:\ blog.jcdev.org
      server node1 10.0.0.1:443 check check-ssl
      server node2 10.0.0.2:443 check check-ssl
      server backup 127.0.0.1:8443 backup

Note that I’ve used the ‘check-ssl’ parameter for the ‘check’ keyword, which ensures the health check connects over TLS. If you don’t do this, HAProxy sends a plain HTTP request to port 443, which your backend servers won’t like.
With the above configuration, we now have a working load balancer which will fail over in the event of a problem, and fail back automatically when the issue is fixed.

Bonus: Email Alert notifications.

Although it’s a good idea to monitor the health of each node separately (using a monitoring tool such as Nagios), HAProxy 1.6 is also able to send email notifications when a server changes state.
More information on this feature can be found here.