Looking At ColdFusion Memory Usage

When trying to judge expected behavior from potentially problematic behavior, looking at memory behavior often is a place ColdFusion Administrators will start.

Even though the standard amount of RAM continues to increase, a close examination of an application's use of system memory remains a critical part of the Q&A process.

Fortunately for the developer, a variety of memory tracking abilities exist to troubleshoot an application's use of system memory. Unfortunately for the developer, the multitude of metrics to choose from increases the chances of making the wrong choice and tracking the incorrect data set.

This article will describe some of the available metrics for tracking memory usage in Microsoft's Windows NT/Windows 2000, Sun Microsystems' Solaris, and Linux.

When monitoring ColdFusion memory usage in Windows NT or 2000, many developers use the "Mem Usage" statistic in the Task Manager. This statistic measures the working set size. Working set size roughly translates into how much of the ColdFusion process is being kept in RAM. Keep in mind, "Mem Usage" is NOT a measure of overall memory usage.

If overall memory usage needs to be checked periodically, use the "VM Size" column in the Task Manager. To view the "VM Size" column, make sure "Processes" is selected. Click on the "View" menu and choose "Select Columns". A dialog box will appear in which "VM Size" will be an option.

To monitor memory usage and log it over time, Performance Monitor will need to be used. In order to enable Performance Monitor to log memory usage statistics properly, "Pdlcnfig.exe" will need to be executed. Assuming that the system has Microsoft's Resource Kit installed, "Pdlcnfig.exe" will be located in the \WinNT\system32 directory.

This installs the performance logging service and launches the configuration window. To use the counter for ColdFusion memory logging, follow these steps:

• Select "Process" in the "Performance Object" drop-down box.
• Select "cfserver" from the "Instances" list.
• Select "Private Bytes" from the "Performance Counters" list.
• Click on "Add."

Set the log file name, and select the "Comma Separated Value" (CSV) file format if Excel charting is desired with the data. (CSVs can be opened directly in Excel.) Set the sampling interval (i.e., every 10 minutes) and start the performance logging service by clicking "start" or starting it via the control panel. If you get an error trying to start the service, reboot and try again.

If Windows is doing its job correctly, expect to see Mem Usage rise as ColdFusion runs. This just means more of ColdFusion is being stored in RAM and less is left out on disk while virtual memory is waiting to be paged in.

Remember, "Private Bytes" is the total virtual memory size of the cfserver process and equates to the "VM Size" metric inside Task Manage.

Neither of the statistics provided by the Task Manager or Performance Monitor will show memory being returned. That's because ColdFusion 4.5 incorporates the latest version of SmartHeap, which no longer returns unused memory to NT. Instead, memory is held, assuming ColdFusion may reuse it in the near future. While this method is faster and more efficient, SmartHeap can lead to confusion when looking at memory statistics on NT.

The "VSZ" (Virtual Memory Size) column is the most applicable for memory usage evaluation and equates with Windows "VM Size." The "RSS" column simply how much virtual memory is currently in physical RAM. Both "VSZ" and "RSS" can be obtained with the ps command.

To use, copy and paste these lines into an editor window, save the file as "cfmemstat," change the file permission, and execute:

The # ./cfmemstat 600 will log to a file "cfmemstat.log" in the same directory every 10 minutes and spawn the process in the background.

All of these commands assume that ColdFusion was installed in /opt/coldfusion.

When using these utilities, remember that the "size" parameter differs from Linux to Solaris. Therefore, "VSZ" statistic is what you want to monitor. On Solaris, top "size" may equal ps "VSZ," but on Linux, it appears to be the "RSS."

The release and reacquisition of the memory cycle places a significant drain on server resources. That was the main reason for the change from CFAS4.0 to CFAS4.5. If your application needs a certain amount of memory at one point in time, it will never need less. This makes it appear that memory usage is always rising and giving the impression of a leak.

Depending on the application and the complexity of the Web site, a well-behaved site's memory usage will grow but eventually plateau. On a number of very complex sites, Allaire closely examined reports of memory usage growing without bound. In many of the cases, the problems were traced to a documented leaky DB driver or gargantuan amount of cached queries.

In the case of the cached query problem, this was due to a slight difference in each query. In another case, the memory plateau was higher than expected. Once enough swap space was configured the memory plateau was established. No ColdFusion application independent memory leak was identified in any of the cases reported.

Here is a checklist for possible causes for increased memory usage:

• Check which MDAC version is being used. In many of the reported cases, the MDAC version 2.1.3711.11 contained a documented memory leak.
• Examine ColdFusion query caching to make sure no differences appear.
• Make sure that growing files, logs, or record sets are not being brought into memory, even if only for the duration of the request.
• Check accumulating states in persistent scopes, including server, application, and session.
• Operations on collections (e.g., CFSearch and CFIndex) can cause growth over time.
• Keep in mind that operations needing large contiguous blocks of memory like CFFile or CFHTTP can take longer to plateau.
• Homegrown CFX tags may be allocating memory and not properly freeing it up.
• When running tests for memory usage, check for the existence of any application.cfm files located in the directory path that enable session, application, or server variables. These variables may add processing cycles or additional demand on memory, which may not be immediately apparent.
• One of the most common causes are improperly locked access to session, application, and server variables. These variables are stored in memory and need to be enclosed in a CFLOCK tag with the appropriate SCOPE attribute specified (i.e. session, application, or server) when referenced.
• When accessing client variables stored in an Oracle DB with the CFAS native driver for Oracle, see KB article 15885.

An application reads a log file into an application variable every day, and every day that log file gets 10MB bigger. Therefore, memory usage is going to go up by 10MB every day. This is intended behavior; ColdFusion was told to put the log file into memory and it does.

Two solutions exist for this problem. A sufficient amount of swap space (paging file on Windows NT and 2000) must exist to accommodate the maximum size of the log file. To check this setting and make changes, go to:

Start > Settings > ControlPanel >System > Performance > VirtualMemory > Change > Paging File Size for Selected Drive >Initial/Maximum Size

Alternately, the log file can be kept to the same size by removing older data on a regular basis. For this remedy, it's very important to control how much memory ColdFusion uses for an application. ColdFusion allows a developer to write an application as memory-intensive (bring in the whole database) or memory sparse (bring in a single column from a single record) as needed.

Here's a hint on memory usage and stability: Swap space should be at least 12MB greater than your RAM (according to Windows NT and 2000 documentation). In fact, a larger increment is probably safer. If sufficient disk space is available, doubling the swap space will probably work best.

In short, ColdFusion Server machines should be equipped with enough memory to handle the sizes being requested, or the size of the original data should be shrunk.