From PhoenixWing

Note:

This was taken from documentation written up by a coworker for a project they worked on. I felt I didn't want to lose such great material, and copied it. It has been changed to protect work related information & confidential information. By using any of this information, you do so at your own risk. I provide no guarantee of the effects you may or may not experience from using this documentation, and cannot be held responsible for you using it. By using it, you agree to this fact.

General

We have a few different load balancing solutions in use. Probably the most 'normal' solution is provided by an Alteon 180e load balancer. They generally work well, but their copper ports are only 100mbps (fiber ports are gigabit, however). This combined with a strange problem with them on a new customer's cluster led us to the deployment of a Linux-based load balancer using LVS/IPVS. This page deals with the configuration and implementation of such a load-balanced cluster.

Note: the configuration described here does not break ProxyPass, logging, or anything related to the remote IP. You can still do null-routes or log processing or anything else you'd normally do in a single-server environment, or in a normal Alteon environment. You do need to be a bit careful, given that there is frontend-side configuration, however.

Setup

Your network layout should look something like this. Note that this is for the 'Direct Routing' method of IPVS load balancing. For NAT or Tunneling, refer to http://www.linuxvirtualserver.org/ for descriptions.

switch port   servername   IP
1             web01        64.38.208.4
2             web02        64.38.208.5
3             web03        64.38.208.6
4             web04        64.38.208.7
5             lb01         64.38.208.3
24            <uplink>
(backend links omitted)

Start by configuring each server as if it were standalone (Apache, etc). All servers should be using the core router default gateway... the load balancer should not be the default gateway for anything.

Load Balancer

This will install the main tools you'll need to make the load balancer do it's job. Yes, it pulls in quite a few dependencies... mostly Perl libs.

yum install --disablerepo=centosplus --disablerepo=rpmforge heartbeat heartbeat-ldirectord ipvsadm
chkconfig --level 345 ldirectord on

Create the file /etc/ha.d/ldirectord.cf with content like this. Replace the virtual IP and real IP's with the correct ones.

logfile = "local0"
checkinterval = 5
autoreload = yes

virtual = 64.38.208.13:80
        real = 64.38.208.4:80 gate 5
        real = 64.38.208.5:80 gate 5
        real = 64.38.208.6:80 gate 5
        real = 64.38.208.7:80 gate 5
        scheduler = wlc
        protocol = tcp
        checktype = 4
        request = "/server-status"
        receive = "Apache Server Status"
        negotiatetimeout = 10

This config will check the URL "http://<real-ip>/server-status" for each real server. If it doesn't respond with a page containing "Apache Server Status", it concludes that the server is down and removes it from the pool. It continues checking it, however, and will add it back as soon as the server starts responding correctly again.

'gate' refers to 'gatewaying' mode, also known as Direct Routing. In this mode, incoming traffic to the VIP hits the load balancer, which chooses a real server and forwards the connection to it. The real server responds directly to the remote IP, not through the load balancer.

You also need to assign the VIP on the load balancer. Something like this is sufficient:

[root@wickedmedia-lb01 ha.d]# cat /etc/sysconfig/network-scripts/ifcfg-eth4:1
# This is the VIP interface
DEVICE=eth4:1
IPADDR=64.38.208.13
NETMASK=255.255.255.255
ONBOOT=yes

At this point, you should be able to bring up the load balancer:

service ldirectord start

Check /var/log/messages for errors. Check 'ipvsadm -L -n' for status. As we haven't configured the frontends yet, they will probably all show with a weight of zero (meaning "this node is down").

Frontends

For this configuration to work, the frontends must be aware of what the VIP is (this is unlike the Alteon-style cluster, where the frontends are oblivious and route through the Alteon, which cleans up after them). This is done by binding the VIP to each frontend. HOWEVER, the VIP is actually in use by the load balancer, so we need to configure the servers to not respond to ARP requests for the VIP. This is done by binding the VIP as an alias of the loopback, and configuring Linux to only respond to ARP requests on the interface to which the appropriate IP is bound. That is, we tell it "don't respond to arp for eth0's IP if the request comes in on eth1".

Add these lines to the file /etc/sysctl.conf

# For load balancing
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

Then run 'sysctl -p'

Now that we've "fixed" ARP, we can bind the VIP. This is sufficient (substitute the correct VIP):

[root@wickedmedia-web01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-lo:1
DEVICE=lo:1
IPADDR=64.38.208.13
NETMASK=255.255.255.255
ONBOOT=yes

Now, go edit your httpd.conf files on each frontend to include the VIP in each vhost container. Don't forget to add a 'NameVirtualHost <VIP>' line!

Lastly, make sure that whatever resource the LB is configured to check for is actually available to the LB. Otherwise, it will think the server is down and won't send traffic to it! (hint: in this example, you need to turn on the server-status page, and grant the LB's IP access to it)

Status

Load balancer status is easily monitored with any of these commands:

watch ipvsadm -L -n
watch ipvsadm -L -n --stats
watch ipvsadm -L -n --rate

Additional Improvements

One common and fairly simple improvement to make would be the addition of a second load balancer, for High Availability. The basic changes are the configuration of the full 'heartbeat' system rather than just the 'ldirectord' portion, as well as a crossover link between the load balancers (for simplicity... other configurations can be used). The secondary LB determines whether the main is down, and takes control of the VIP.

See the links below for more details.