Clustering ColdFusion or JRun servers running in Distributed Mode behind hardware load-balancing devices

The designation "hardware load balancing device" (HLD) is a broad generalization. There are many HLD's offering many different features. On the high-end of the HLD spectrum, names come to mind such as Cisco, F-5 and Alteon, to name but a few. This article provides configuration details that guide you through one basic way and two optional ways of setting up ClusterCATS on servers running in distributed mode behind a HLD.

By Frank DeRienzo
Principal Technical Support Engineer
Macromedia Inc.

This article first appeared in the ColdFusion Developer Center. It has been condensed for the TechNote to include only the necessary technical information. You may find the full version here.

The designation, hardware load balancing device (HLD) is a broad generalization. There are many HLD's offering many different features. On the high-end of the HLD spectrum, names come to mind such as Cisco, F-5 and Alteon, to name but a few. This article provides configuration details that guide you through one basic way and two optional ways of setting up ClusterCATS (CC) on servers running in distributed mode (DM) behind a HLD.

This article is broken into three sections:

• The first section of this article will offer configuration details for setting up the HLD to perform all load-balancing and failover functions. ClusterCATS will provide web server recovery in the event of a hang or crash, automatic e-mail alerts for server problems and daily server status reports. This first option I will call passive CC HLD integration; it has become the preferred HLD/CC integration option because most of the newer versions/revisions of HLD's and their commensurate firmware and software have substantially improved methods of load-balancing and failover.
• To cover the increasingly rare instances where your HLB may look and run like a chain toilet, Macromedia offers another ClusterCATS integration option. An example of a limiting characteristic of an older chain-toilet load balancing device, may be the blind channeling of client load/sessions to the web-server farm with incredible pressure through an old pipe in a manner similar to (but one hopes more scalable than) DNS Round-Robin. In these increasingly rare cases you will want to use the second configuration option in this article: active CC HLD integration and push secondary corrective load-leveling decisions onto CC.
• Lastly, in section three, this article discusses a third possible option for running a Solaris or Linux-based ColdFusion cluster - this time behind a midrange HLD, one that is neither antique, homemade or cutting edge, but runs with some limited functionality.

Note: This article does not apply to the custom integration of CF/CC with Cisco LocalDirector (CLD).

I. Option 1. Passive CC HLD integration in DM: The HLD will actively distribute load to the front-end distributed web servers based on packet flow. CC will provide web server recovery in the event of a hang or crash, automatic e-mail alerts for server problems and daily server status reports.

Note about option one: If you are running ColdFusion 5 (or are running ColdFusion 4.5.x and are able to upgrade to ColdFusion 5), instead of clustering, you may wish to use the Server Monitoring option with your high-end HLD. ColdFusion 5 Server Monitoring provides all the server level capabilities of CC without the overhead of a cluster.

To integrate CC with an HLD:

A. Configure the load balancing device or software product as recommended by the manufacturer. You may also wish to peruse the ColdFusion or JRun Support Centers to see if there is a recommended procedure for your specific HLD: www.macromedia.com/support.

B. Behind an HLD in DM, you should place the two front-end web servers into a cluster. You must use static web site IP addresses and CC failover (high-availability) must be turned off. During ColdFusion installation, select no server failover. If you are running NT 4.0, do not set up your Web servers with dynamic IP addresses. Your HLD will be providing failover services. Dynamic IP addressing is only used with CC's implementation of failover. If you are adding an HLD to a CF/JR and CC cluster that is already set up with dynamic Web site IP addresses on NT 4.0, you must switch to static Web site IP addresses and disable CC failover.

C. Open the CC Explorer and select a cluster: Start > Programs > ColdFusion server 4.5 > ClusterCATS Explorer > Right-click on a cluster and select > Properties > Load-balance > Load-balancing Product > Other.

If you are clustering Solaris or Linux servers, HLD integration with CC does not require the Windows-based CC explorer, but if you have Windows loaded on a workstation or laptop, you will want to use the Windows-based CC Explorer, it is a great interface and it is an available download from ftp://ftp.allaire.com/kbftp/clustercats/ccwin2k.exe. You also may use it to remotely administer an NT cluster.

When installing ccwin2k.exe to use as an explorer, choose only the explorer option. Install it on a Windows platform (NT4.0 Workstation or Server, Win98, Win95). The CC explorer is an excellent GUI; it will manage your Solaris/Linux cluster using port 9123; if the explorer is on the opposite side of a firewall from the servers, you will need to open ports 9123 and 9129 on the firewall. If you choose not to download the CC Explorer, you may use the CC browser-based explorer to configure your Cluster.

D. Configure the HLD to have CF/JR application awareness. As previously mentioned, do not set up a standard CC application Probe (CFProbe or JRunprobe) in this mode. Remember, with passive CC HLD integration in DM, we want the HLD to do everything it can. (In the second and third sections of this article I will cover what you can do with CC probes in the case that your HLD is not up to the job). There are various redirection features available on most HLD's that you will want to enable to probe for points of failure.

Take the following familiar example of a cluster in a DM environment. Imagine four server-class systems, two running backend CF/JR engines and two running on the front end as visible Web servers. Engine-one runs the CF/JR engine for Webserver-one, and engine-two runs the CF/JR server for Webserver-two; Each Web server points to its respective DM backend CF/JR engine. Webserver-one.qa.nashua.allaire.com and Webserver-two.qa.nashua.allaire.com are configured in a passive-mode cluster.

All of the high-end HLD's have features roughly similar to F5 Networks BIG-IP Extended Content Verification (ECV). When properly configured using cfprobe.cfm or jrunprobe.jsp, ECV, will detect a failure on one of the distributed CF/JR engines and redirect sessions to a working distributed pair of web & application servers.

In this DM configuration, without ECV or a similar feature properly configured, the HLD would never notice a failure of either engine-one or engine-two. Because on the front-end of the DM cluster, webserver-one and webserver-two remain up and running, the HLD will happily send sessions into a distributed black hole. The following HLD example will work for BIG-IP, but also serves as a guide for configuring other HLD's.

In order to configure BIG IP ECV service checking, you must first configure your BIG-IP according to the basic tasks outlined in the F5 BIG-IP Administrator Guide. ECV can then be configured to retrieve CF/JR generated content from distributed webservers one& two; if either backend engine-one or two go down, the content retrieved will not match the content expected, and BIG-IP will mark the webserver down and redirect around it.

Following our example, use ECV-normal to look for the receive expression "Hello" using the page found at the URL //Webserver-one.qa.nashua.allaire.com/btauxdir/cfprobe.cfm. When BIG-IP reads "Hello" on the Cfprobe.cfm page, it will know that the backend application server is running:

D. This concludes section one of this article. The end-state if you chose one of the variations on this option (passive HLD integration) is an HLD conducting load balancing, failover and redirection while CC monitors CF/JR load and availability. CC will provide web server recovery in the event of a hang or crash, automatic e-mail alerts for web server problems and daily server status reports. In this configuration, you must be able to set up the HLD to detect a failure of a backend CF/JR engine.

II. Option 2. Active CC HLD integration: This option assumes that your HLD is sub-optimal (as previously labeled ; a chain toilet). It configures CC in a manner similar to setting up behind DNS RR, but with some critical variations. The HLD will actively distribute load to the Web servers based on packet flow while CC monitors CF load and availability. CC can be configured to detect when the web server is becoming overloaded. If set up this way, CC will supersede the load balancing device and redirect traffic accordingly to the least loaded server. To integrate CC with a third-party load balancing device:

A. Configure the load balancing device or software product as recommended by the manufacturer.

B. As with option one you should use static web-site IP addresses and CC failover should usually be turned off. There is not any HLD that I know of that will not do failover when the IP stack on a given web server crashes, but if the HLD cannot do failover, then enable CC failover. Otherwise, during ColdFusion installation, select no server failover. If you are running NT 4.0, do not set up your web servers with dynamic IP addresses. LD will be providing failover services. Dynamic addressing is only used with CC's implementation of failover. If you are adding an HLD to a CF/CC cluster that is already set up with dynamic web-site IP addresses on NT 4.0, you must switch to static web-site IP addresses and disable CC failover.

III. Optional CC Probe Configuration for Linux or Solaris servers running ColdFusion in DM behind an HLD: Section one of this article assumes that you are using a robust HLD; section two assumes that you are using something only marginally better than DNS RR. What if you have an HLD that will not parse ColdFusion code to detect the failure of a backend ColdFusion engine, but will detect a dead web server? Let's say also that since your HLD is reasonably efficient, you do not want to put CC into the active mode to compensate by handling restrictions or subsequent load leveling. In this case you have another clustering option. You may cluster the backend engines of your Solaris or Linux-based DM cluster and customize the CC application probe to stop the web server prior to restarting the application server.

F. To set up a CFprobe, right-click on a server in CC Explorer, choose new monitor, choose OK, click on the blue arrow over the label probe-type, click in the startup parameters field and arrow to the right without deleting any text, change the word NORESTART to RESTART, arrow back to the left and put in the path to the cf or jrprobe.cfm on the front-end web server, click on register, and close. You now have a monitor (CFProbe) that is looking for the word "hello" on a ColdFusion page on the front-end web server. If the page becomes unavailable, this customized probe will stop the front-end webserver then restart the ColdFusion server. CC's built-in wsprobe will then restart the dead front-end server. Be sure to edit the probed cfprobe.cfm page on the front-end web server according to the previous procedure to present a ColdFusion generated string.

G. When clustering the backend to enable probes, you may also use backend failover. Upon enabling backend failover, you will allow both front-end web servers to remain functional in the case of a disk crash or major problem on one of the two backend engines. When the IP address from the crashed engine moves over to the remaining engine, the front-end Web server corresponding to the dead engine will start using the remaining engine while the HLB continues to see both web servers as functional.

Conclusion: If your CF/JR enterprise web site has outgrown the capabilities of software load-balancing, then it is time to find a hardware solution, but you may still incorporate CC into the structure of your site. Compare the features of the HLD with the features of CC and see if CC either covers points of failure ignored by the HLD or provides additional functionality.