In addition to the performance tips in this section, you can also optimize the underlying Apache web server. For more information, see Apache Performance Tuning.
Note:
You can’t send aggregate messages and combine audio samples in the same application.
Large Scale Broadcast (scale is more important than latency)
In this use case, you want to reach as many people as possible and are willing to allow some latency. You may also want to keep costs down while still reaching as many people as possible. Send large aggregate messages and configure large stream chunk sizes.
Large Scale Broadcast (low latency is more important than scale)
In this use case, you want to keep latency as low as possible while reaching as many people as possible. You are willing to trade some scale for decreased latency. Send smaller aggregate messages. Or, combine audio samples and do not send aggregate messages.
Interactive (lowest latency)
Interactive applications require very low latency. Do not send aggregate messages. For the lowest latency, do not combine audio samples.
If data latency is your concern, combine audio samples. Combining audio samples introduces audio latency but doesn't affect the rest of the stream. Aggregating messages introduces latency to the whole stream.
Interactive applications are the most difficult to scale and require more hardware than the large scale broadcast use cases.
64-bit servers can address more physical memory than 32-bit servers. When delivering on-demand video on 64-bit servers, increase the size of the recorded media cache size. Only the amount of RAM available on the computer limits the recorded file cache size.
Note:
The system warns that a process is close to the memory limit when the process memory is more than 90% of the system memory.
When a stream is requested from the server, segments of the stream are stored in a cache on the server. As long as the cache has not reached capacity, the server places segments in the cache. Each time a stream attempts to access a segment, the server checks the cache. If the segment is available, the server gives the stream a reference to the segment for playback. If the segment is not available, the server retrieves the segment from its source, inserts it into the cache, and returns a reference to that segment to the stream.
When the cache is full, the server removes unused segments, starting with the least recently used. After removing all unused segments, if there still isn’t enough room for a new segment, the server notifies the client that the stream is not available and makes an entry in the core log file.
If you have cache-full events in the core log file, increase the size of the cache or limit the number of streams playing. For more information about the core log file, see Monitoring and Managing Log Files.
-
Edit the SERVER.FLVCACHE_MAXSIZE parameter.
This is the maximum size of the cache, in megabytes. The default value is 500. The file cache shares process address space with the core process. Each core process has a separate file cache. For 32-bit processes, it has a limit of 2 GB in Windows and 3 GB in Linux. For 64-bit installations, the limit is greater. For more information, see Configure a 64-bit server.
The size of the cache limits the number of unique streams the server can publish. To increase the probability that a requested stream will be located in the recorded media cache, increase the value of SERVER.FLVCACHE_MAXSIZE. To decrease the amount of memory the server process uses, decrease the value of SERVER.FLVCACHE_MAXSIZE. While a large cache size is useful, Adobe recommends that you ensure that your total system memory usage does not exceed the process limit of your OS. Consider memory limits and desired memory and stream performance when utilizing the memory cache.
Note:
Cache settings have no effect on live streams, as live streams do not need or utilize the cache.
There is no exact way to calculate the value of the cache size because it varies depending on the amount of RAM available and the number of other processes that are running. However, you can follow some general guidelines to approximate the recommended size of the cache.
Out of the total system RAM, some amount will be used by the OS and other non-Adobe Media Server processes:Amount of RAM available for AMS (R) = RAM size - RAM used by the OS and non-AMS processes
The RAM available for Adobe Media Server is divided among the number of cores started:Amount of RAM per core (Rc) = Amount of RAM available for AMS (R) / Number of Cores
Out of this, some amount of RAM is used up by the core process, and is not available to the file cache:Recommended File Cache Size = Amount of RAM per core (Rc) - RAM used by each core process
RAM used by each Adobe Media Server process varies based on load, and there is no exact figure. You can use 512 MB as a ballpark figure.
The following example attempts to determine a reasonable file cache size:RAM size = 8 GB RAM used by OS and non-AMS processes = 1.2 GB (approximately) Number of AMS core processes allowed = 3 Amount of RAM available for AMS (R) = 8 - 1.2 = 6.8 Amount of RAM per core (Rc) = 6.8 / 3 = 2.27 GB Recommended file cache size = 2.27 - 0.5 = 1.76 GB
Amount of RAM available for AMS (R) = RAM size - RAM used by the OS and non-AMS processes
Amount of RAM per core (Rc) = Amount of RAM available for AMS (R) / Number of Cores
Recommended File Cache Size = Amount of RAM per core (Rc) - RAM used by each core process
RAM size = 8 GB RAM used by OS and non-AMS processes = 1.2 GB (approximately) Number of AMS core processes allowed = 3 Amount of RAM available for AMS (R) = 8 - 1.2 = 6.8 Amount of RAM per core (Rc) = 6.8 / 3 = 2.27 GB Recommended file cache size = 2.27 - 0.5 = 1.76 GB
Streams break into chunks as they are written to the network. You can specify the size of a chunk. Large values reduce CPU usage because there are fewer writes. However, large values can delay other content on lower bandwidth connections. The larger the content size and the higher the bandwidth of the receiving connection, the more benefit is gained from larger chunk sizes.
-
In the Client element, set the OutChunkSize element to a value between 128 and 65536 bytes. The default value is 4096 bytes.
For more information, see Application.xml file.
Note:
Do not send aggregate messages and combine audio samples in the same application.
An aggregate message is a single message that contains a list of submessages. Sending aggregate messages reduces CPU usage and increases server capacity. You can configure applications to deliver aggregate messages to clients running on Flash Player 9.0.60.0 and above. When this setting is disabled, the server breaks up aggregate messages into individual messages before delivering them to clients.
Aggregate messages can be used with live and recorded streams.
An origin server can deliver aggregate messages to an edge server, and an edge server can deliver aggregate messages from the FLV data that is has cached on the disk. An edge server can deliver aggregate messages even if the origin server did not.
To ensure that aggregate messaging is enabled, be sure the following settings are enabled:
EnableAggMsgs in the Server.xml file
AggregateMessages in the Application.xml file
AggregateMessages in the Vhost.xml file (if applicable)
The following table describes these settings, plus other configuration settings that are related to using aggregate messages:
Element |
Configuration file |
Description |
---|---|---|
Streams/EnableAggMsgs |
Server.xml |
Enables the creation of aggregate messages in the FLV module. When this setting is “true”, the FLV module returns aggregate messages when loading segment data. When this setting is “false”, the FLV module returns regular audio, video, and data messages. The default value is “true”. |
Streams/MaxAggMsgSize |
Server.xml |
Determines the maximum size (in bytes) of aggregate messages returned from the FLV module (when they are enabled). The actual message size returned by the module might be slightly larger because the module will not fragment individual messages, but instead includes whole messages until this size is exceeded. The default value is 65536. |
Client/AggregateMessages |
Application.xml |
Enables the delivery of aggregate messages to clients that support them. When this setting is disabled, the server breaks up aggregate messages into individual messages before delivering them to clients. The default value is “true”. |
StreamManager/Live/Queue |
Application.xml |
Queues incoming messages that are published to live streams. This is so that the server can create aggregate messages before transmission to subscribing clients. If queueing is not enabled, aggregate messages are not created. The default value is “true”. |
StreamManager/Live/Queue/MaxQueueSize |
Application.xml |
Defines the maximum size (in bytes) that the live queue can grow to before the messages it contains are transmitted. Increasing the size of the queue allows for larger aggregates to be created, which increases the efficiency of the server, but introduces latency in the transmission. Decreasing the queue size reduces latency but is less efficient. |
StreamManager/Live/Queue/MaxQueueDelay |
Application.xml |
Defines the maximum time (in milliseconds) that the live queue delays messages before the messages are transmitted. This setting can be used to ensure that unacceptable amounts of latency are not introduced into the publishing stream. Increasing the delay allows for larger aggregates to be created, which increases the efficiency of the server. Decreasing the delay reduces latency but is less efficient. |
StreamManager/Live/Queue/AggregateMessages |
Application.xml |
Determines whether or not aggregate messages are generated when the queue is flushed. When this setting is enabled, the messages in the queue are grouped together as aggregates to improve the efficiency of transmission. It is possible to enable queing and disable the generation of aggregate messages, but this does not result in performance benefits. The default value is “true”. |
StreamManager/Live/Queue/AggregateMessages/MaxAggMsgSize |
Application.xml |
Defines the maximum size of an aggregate message that the server creates. The server can create smaller aggregates, as necessary. The server cannot create messages that are larger than the MaxQueueSize, so this number is generally set to something less than or equal to the configured size of the queue. |
Proxy/AggregateMessages |
Vhost.xml |
Determines whether aggregate messages are delivered from the edge cache when a vhost is configured as an edge proxy. The default value is "false". If the edge server receives aggregate messages from the origin when this setting is disabled, the messages are broken up before being cached. |
Proxy/AggregateMessages/MaxAggMsgSize |
Vhost.xml |
Determines the size (in bytes) of aggregate messages returned from the edge cache when aggregate messages are enabled. This setting only applies to messages retrieved from the disk cache. Aggregate messages received directy from the origin server are returned as-is, and therefore their size is determined by the origin server's settings for aggregate message size. |
Even when you configure the server and application to use aggregate messages, the following circumstances will prevent them from being delivered:
The client’s Flash Player version is earlier than 9.0.60.0.
The stream is set to filter audio, filter video, or control FPS.
The stream requires transcoding.
If any of these conditions are present, Adobe Media Server breaks the aggregated message into its submessages, and the messages are delivered individually.
After you confirm that aggregate messaging is enabled, you can use the settings in the Server.xml file to configure the size of the messages. You can also enable or disable the messages based on the type of stream. For more information, see Streams.
You can also configure message queuing in the Application.xml file. For more information, see Queue.
Note:
Do not send aggregate messages and combine audio samples in the same application.
-
Locate the StreamManager/Audio/CombineSamples section of the file and set values the following elements:
Element
Description
Subscribers
If there are more than this number of subscribers to a stream, audio samples are combined. The default value is 8. To increase live streaming capacity, set this value to 1.
LoCPU
If the CPU is lower than this value, audio samples are not combined. The default value is 60. To increase live streaming capacity, set this value to 1.
HiCPU
If the CPU is higher than this value, audio samples are not combined. The default value is 80. To increase live streaming capacity, set this value to 1.
MaxSamples
Combine this many samples into one message. The default value is 4. To increase live streaming capacity, set this value to 8.
A high connection rate to the server can negatively impact the experience of users already connected to the server.
-
<Root> <Server> ... <Protocol> <RTMP> <Edge> <MaxConnectionRate>100</MaxConnectionRate>
Element
Description
Impact
MaxConnectionRate
The maximum number of incoming connections per second the server accepts, per listener. Listeners are defined in the HostPort element in the Adaptor.xml file. Each port the server is configured to listen on represents a listener. You can set a fractional maximum connection rate, such as 12.5. A value of 0 or -1 disables this feature.
The value of this element is a global setting for all listeners. If the element is set to 10 connections per second, each listener has a limit of 10 connections per second. If there are three listeners and the MaxConnectionRate is set to 10, the server imposes a maximum total combined rate of 30 connections per second.
Connections requested at a rate above this value remain in the TCP/IP socket queue and are silently discarded by the operating system whenever the queue becomes too long.
Sometimes clients have left an application without the server-end of the socket knowing. This behavior can happen when a router is unplugged or crashes without sending TCP close messages for the sockets it is managing, when a laptop is undocked from a docking station, and so on. To reclaim these resources for new and active clients, the server can close the idle clients.
Adobe Media Server sends a small ping message over sockets that have been quiet for awhile. If the client at the far end has gone away, the server's TCP eventually stops retransmitting that data. When that happens, the socket is shut down and the server can clean up any associated client object, and so on.
Elements in the Server.xml, Vhost.xml, and Application.xml configuration files specify how often the server should check for idle clients. When a client has been idle longer than the maximum idle time (10 minutes, by default), the server sends a status message to the NetConnection object (the client) with the code property set to NetConnection.Connect.Idle followed by NetConnection.Connect.Closed. The server closes the client connection to the server and writes an x-status code of 432 in the access log. The server also writes a message such as “Client x has been idle for y seconds” in the core and event logs.
To close idle connections, enable the feature in the Server.xml file. Once you enable the feature in the Server.xml file, you can disable the feature for individual virtual hosts or individual applications in the Vhost.xml files and Application.xml files. The values defined in the Server.xml configuration file apply to all clients connected to the server, unless the values are defined in the Vhost.xml file (the Vhost.xml values override the Server.xml values). The values defined in the Application.xml file override the values defined in the Vhost.xml file.
-
Element
Description
Impact
AutoCloseIdleClients
Set the enable attribute to true to close idle clients. If the enable attribute is omitted or not set to true, the feature is disabled. The default value is false.
CheckInterval
Specifies the interval, in seconds, at which the server checks for active client connections. The default interval is 60 seconds.
A client is disconnected the first time the server checks for idle connections if the client has exceeded the MaxIdleTime value. A shorter interval results in more reliable disconnection times.
MaxIdleTime
Specifies the maximum idle time allowed, in seconds, before a client is disconnected. If this element is 0 or less, the default idle time is used. The default idle time is 600 seconds (10 minutes).
A low value may cause unneeded disconnections. When you configure this element, consider network latency. An idle time that is less than a typical round trip between the server and the client can result in a disconnection.
You can disable this feature for a virtual host or specify a different maximum idle time for a virtual host in the Vhost.xml file.
-
Element
Description
AutoCloseIdleClients
Disable this feature for an individual virtual host by setting the enable attribute to false. If this element is disabled in Server.xml, the feature is disabled for all virtual hosts, even if you specify true in the Vhost.xml file.
MaxIdleTime
Specifies the maximum idle time allowed, in seconds, before a client is disconnected. The default idle time is 600 seconds (10 minutes). You can set a different value for each virtual host.
If no value is set for this element, the server uses the value set in the Server.xml. file.
The value of the MaxIdleTime element in the Vhost.xml file overrides the value of the MaxIdleTime element in the Server.xml file.
You can disable this feature for an application or specify a different maximum idle time for an application in the Application.xml file.
-
Element
Description
AutoCloseIdleClients
Disable this feature for an individual application by setting the enable attribute to false. If this element is disabled in Server.xml, the feature is disabled for all applications, even if you specify true in the Application.xml file.
MaxIdleTime
Specifies the maximum idle time allowed, in seconds, before a client is disconnected. The default idle time is 600 seconds (10 minutes). You can set a different value for each application.
If no value is set for this element, the server uses the value set in the Vhost.xml. file. If no value is set for this element in the Vhost.xml file, the server uses the value in the Server.xml file.
The value of the MaxIdleTime element in the Vhost.xml file overrides the value of the MaxIdleTime element in the Server.xml file.
Note:
This section does not apply to Adobe Media Streaming Server because it doesn’t support multiple processes.
In some scenarios, you might want to change how applications are assigned to server processes. When you start the server, you are starting a process called AMSMaster.exe (Windows) or amsmaster (Linux). Application instances run in processes called AMSCore.exe (Windows) and amscore (Linux). The master process is a monitor that starts core processes when necessary. There can be only one master process running at a time, but there can be many core processes running at a time.
Note:
The number of core processes you can run is limited by system memory. Do not run more than 100, and you probably won’t need more than 20. If you are configuring the number of core processes and using the reloadApp() Administration API or HTTP command, see Tech Note kb403044.
You can configure how applications are assigned to server processes in the Process section of the Application.xml configuration file. Settings in an Application.xml file in a virtual host folder (for example, rootinstall/conf/_defaultRoot_/_defaultVHost_/Application.xml) apply to all the applications running in that virtual host. Settings in an Application.xml file in an application’s folder (for example, rootinstall/applications/myApp/Application.xml) apply only to that application. The following is the XML structure:
<Application> <Process> <Scope>vhost</Scope> <Distribute numprocs="3"></Distribute> <LifeTime> <RollOver></RollOver> <MaxCores></MaxCores> </LifeTime> <MaxFailures>2</MaxFailures> <RecoveryTime>300</RecoveryTime> </Process> ... </Application>
-
The Scope tag specifies at which level application instances are assigned to core processes. An application instance can run by itself in a process or it can run in a process with other instances. Enter one of the following values for the Scope element.
Value
Description
adaptor
All application instances in an adaptor run together in a process.
vhost
All application instances in a virtual host run together in a process. This is the default value.
app
All instances of a single application run together in a process.
inst
Each application instance runs in its own process. This provides the best application isolation and uses the most system resources.
Running every instance in its own process can create many processes. You can set the Distribute numprocs attribute to a value greater than 1 to distribute instances across that number of processes.
The four process scopes don’t provide a good distribution for all scenarios. For example, if you have one application and want to run 25 instances of that application, you could either distribute those instances to 1 core process (<Scope>app</Scope>) or to 25 core processes (<Scope>inst</Scope>). In this scenario, you could set Scope to app and Distribute numprocs to 3 to distribute the application instances among three core processes.
Note:
There is no limit to the value of the numprocs attribute, but you should never need more than 40. Depending on available RAM, a number between 3 and 11 is realistic for most cases. Adobe recommends using prime number values because they result in a more even distribution of connections to processes.
Scopes have an enclosing relationship with a strict ordering: adaptors contain virtual hosts, which contain applications, which contain instances, which contain clients. The value of the Distribute tag must be a scope that is lower in order than the value in the Scope tag. In other words, if the value of Scope is adaptor, the value of Distribute can be vhosts, apps, insts, or clients. If the value of Scope is app, the value of Distribute can be insts or clients. By default, the server uses the value immediately lower than the one specified in the Scope tag.
-
Value
Description
vhost or vhosts
All instances of applications in a virtual host run together in a process.
app or apps
All instances of an application run together in a process.
inst or insts
Each application instance runs in its own process. This is the default value. If you choose this value, you must also set the Distribute numprocs attribute to a value greater than 1.
client or clients
Each client connection runs in its own process.
Use this value for stateless applications only. Stateless applications don’t require clients to interact with other clients and don’t have clients accessing live streams. Most vod (video on demand) applications are stateless because each client plays content independently of all other clients. Chat and gaming applications are not stateless because all clients share the application state. For example, if a shared chat application were set to client, the messages wouldn't reach everyone in the chat because they’d be split into separate processes.
-
Specify the number of core processes in the MaxCores tag (the maximum number of core processes that can exist concurrently) and the number of seconds that a core process can run in the RollOver tag. When a core process reaches the limit, any new connections roll over to a new core process.
The following diagram depicts the way in which the server rolls over processes. In the XML, the rollover time is set to 3600 seconds (1 hour), indicating that every hour a new process should start, and the maximum core processes value is set to 3, indicating that the maximum number of processes at any given time is 3:
<Process> <Scope>app</Scope> <LifeTime> <RollOver>3600</RollOver> <MaxCores>3</MaxCores> </LifeTime> ...
When each process starts, it accepts new connections until the next process starts: that is, when process 1 starts, it accepts new client connections to the application until process 2 starts; process 2 then accepts new client connections until process 3 starts; and so on.
Note that the duration of process 1 might or might not be the full duration specified by the rollover value, because rollover values are calibrated to the real clock time. The duration of process 1 is partially determined by the current time when process 1 starts. For example, as shown in the diagram, when process 1 starts, the current time is 12:20, so the duration of process 1 is only 40 minutes (because it is 40 minutes until the beginning of the hour in real time). The duration of the first process is determined by the clock time; subsequent processes have a duration equal to the specified rollover time.
To disable this feature, set RollOver to 0. This feature is disabled by default.
Note:
If you have multiple VHosts with Process/Scope set to adaptor, you must set an identical RollOver value for each VHost.
In stateless applications, such as vod applications, old core processes continue to serve earlier connections. In this case, you can specify a value in the MaxCores tag to limit the maximum number of core processes that can run simultaneously. If the application is not stateless, the server ignores any value you assign to MaxCores and sets it to 1. This ensures that an application instance is not split across multiple processes, but clients are disconnected periodically. To disable this feature, set MaxCores to 0. This feature is disabled by default.
Note:
An application is considered stateless if you configure it to distribute clients over multiple processes. To do this, set the Distribute numprocs attribute to a value greater than 1, then set the Distribute tag to clients or set the Scope tag to inst.
Use this feature to guard against a behavior in which a faulty core process can use the entire CPU by being repeatedly launched very quickly.
Note:
Applications that are loaded using the Administration API (including applications loaded using the Administration Console) are not checked for process failures.
The RAW (Record and Watch) file format records live media into configurable chunks that stream to any version of Flash Player. Use the RAW file format to serve long-length, multi-bitrate DVR streams without running into performance issues. The RAW file format records and plays back all streams that Adobe Media Server supports, including H.264 video, data-only, audio-only, and so on.
-
<Server> <Streams> <StreamLogLevel>warning</StreamLogLevel> <Raw> <EnableAggMsgs>true</EnableAggMsgs> <MaxAggMsgSize>65536</MaxAggMsgSize> </Raw> ...
Element
Default value
Description
StreamLogLevel
warning
Controls log levels for all stream adaptors (FLV, MP4, and RAW). The default value is warning. Possible values are verbose, warning, and error.
Raw
Container element.
Contains elements that control the RAW adaptor.
EnableAggMsgs
true
Specifies whether the RAW adaptor generates aggregate messages (true) or not (false). The default value is true. Aggregating messages improves server performance.
MaxAggMsgSize
65536
The maximum size of an aggregate message, in bytes. The default value is 65536. You can use any positive integer.
Note:
By default, RTMPE is enabled in the server’s Adaptor.xml file. For maximum security, it’s best to leave RTMPE enabled. For more information, see the Adobe Media Server Hardening Guide.
In some scenarios, however, you might want to disable RTMPE (encrypted Real-Time Messaging Protocol). Because RTMPE uses encrypted channels, there is a minor impact on performance; RTMPE requires about 15% more processing power than RTMP. If you don’t control the applications that connect to Adobe Media Server and you don’t want them to use RTMPE, you can disable RTMPE at the adaptor level.
To request an encrypted or encrypted tunnelling channel, applications specify rtmpe or rtmpte, respectively, in the NetConnection.connect() URL, for example, nc.connect("rtmpe://www.example.com/myMediaApplication"). If an application specifies RTMPE without explicitly specifying a port, Flash Player scans ports just like it does with RTMP, in the following order: 1935 (RTMPE), 443 (RTMPE), 80 (RTMPE), 80 (RTMPTE).
Note:
RTMPE cannot currently be used between servers or from edge to origin. In these cases, RTMPS can be used instead.