Dynamic streaming

Adobe Media Server can receive commands to switch between versions of a content stream that are encoded at different bit rates. This feature lets your media application adapt to changing network conditions. It also lets your application adapt to clients with different capabilities, such as mobile devices with lower processing power and smaller screens. For example, suppose the server is streaming high-definition video to a client application but encounters poor network conditions. The server can switch to a standard-definition stream at a lower bit rate. If network conditions improve, the server can switch back to HD video. The transitions occur seamlessly in the client. Although network conditions have changed, the video streaming to the client is uninterrupted.

For optimal user experience, dynamic streaming requires the following:

• The different versions or pieces of content are synchronized: the video timelines must match.

• Audio or other data in each content stream is synchronized with the video data in that stream.

The server implements a transition between two pieces of recorded content in three possible ways, depending on the type of content that is being streamed:

• Video-only streams. Transitions occur at the nearest keyframe in the target timeline.

• Video and audio streams. Transitions occur at the audio sample that immediately precedes the nearest keyframe in the timeline of the target stream. The audio timelines of the initial and target streams must match, or an audio artifact results.

• Audio-only streams. Transitions occur at the nearest possible sample.

Implementing transitions between live video content is slightly more complicated. The streams must have timestamps that are synchronized closely enough, within 3-5 milliseconds, so that the server can select accurate transition points.

Using ActionScript to switch streams

Stream transitions occur on the server, but the command to switch streams and the determination to do so comes from the client application. The application developer includes logic to monitor download and playback statistics and to switch from the old stream to the new stream when appropriate.

Use the ActionScript 3.0 NetStream.info property and the NetStreamInfo class to monitor download and playback statistics. Use the NetStream.play2() method and the associated NetStreamPlayOptions class to change streams in mid-play.

Use the NetStreamPlayOptions.offset property to perform fast switching between streams. Use the default value of NetStreamPlayOptions.offset which is -1. The default value of -1 indicates a fast switch time of netstream.time + 3. You can also specify a greater value, such as netstream.offset + 4, but do not use a lower value.

The application developer must also ensure that the client application has a playback buffer that is large enough to absorb any delay caused by the transition. Two factors that can cause a delay are the keyframe interval of the live stream and if the two streams being switched are not synchronized. For example, a 2-second buffer cannot accommodate a 3-second transition delay.

Adobe developed a new class called DynamicStream that extends the NetStream class. The DynamicStream class contains event listeners that monitor bandwidth, buffer usage, and dropped frames. The class switches streams based on that information. You can use the DynamicStream class to implement dynamic streaming, or you can use the DynamicStream class as a reference to write your own dynamic streaming algorithms. If you’re migrating legacy code, it’s a good idea to use the DynamicStream class.

Download the ActionScript class files and documentation for these classes from www.adobe.com/go/fms_dynamicstreaming.

Determining when to use dynamic streaming

Adobe recommends that you use dynamic streaming for content that meets some or all of the following criteria:

• Video with long duration

• Video with large file size

• HD video

• Video with larger dimensions, such as full screen video

• Content distributed to users who are more susceptible to bandwidth issues, such as home users, rather than corporate users

Encoding recommendations

To provide users with the best experience, when you encode the content, follow the recommendations in this DevNet article and in Will Law's presentation at Adobe MAX 2008.

The following is a summary of the recommendations:

• For the smoothest switching, encode audio with AAC.

• For streams with MP3 audio, keep the audio bit rates the same.

• Use the same audio sample rate when possible.

• To encode streams for lower bandwidth usage, encode the audio with a single channel (mono).

• Ensure that the video timelines of the streams are related and compatible.

• Use the same codecs and audio bitrates in all streams. If you don’t, you may hear small audio pops when the streams switch.

• While not required, it is helpful if the keyframe interval (keyframe frequency) and frame rate (fps) are consistent across the different versions of content. A shorter keyframe interval lets the server switch streams more quickly, which means that the client can have a smaller playback buffer.

The following table shows various bit rates that you could use to encode a single piece of content:

Determining when to switch streams

You can consider many factors when determining when to switch streams, such as the buffer length, number of bytes downloaded, and number of frames dropped. The DynamicStream and DynamicStreamItem classes, which you can download from adobe.com, are built with these factors in mind and contain the logic required for a dynamic streaming application.

If you prefer to develop your own application logic, it may be helpful to use the following strategy for streaming video:

Check client bandwidth

Monitor the client bandwidth to help determine when switching streams is desirable. When client bandwidth is good, the client application can request the server to switch to a higher video bit rate. When client bandwidth is low, the client application can request the server to switch to a lower bit rate.

To measure bandwidth, use the NetStream.info property. A call to NetStream.info returns a NetStreamInfo object with properties that reflect the rate of incoming audio, video, and data bytes of the stream. With information about the incoming data rate, you can deduce the quality of the bandwidth.

Specifically, use the *byteCount and *bytesPerSecond properties in the NetStreamInfo class (or, in ActionScript 2.0, the object returned by NetStream.getInfo()). For details on these properties, see the ActionScript Language References.

One way to measure the client bandwidth is to measure the NetStreamInfo.byteCount property over a period of time to get the value of bytes per second and when a NetStream.Buffer.Full status event is received. This value approximates the maximum bandwidth available. Then compare the available bandwidth to the available bit rates and implement transitions as needed.

Check for dropped frames

In addition to monitoring the buffer, check for dropped frames. Switch to a stream with a lower bit rate if too many frames are being dropped. Use the NetStreamInfo.droppedFrames property. This read-only property is a number and returns the number of video frames dropped in the current NetStream playback session.

One strategy to determine the rate of dropped frames is as follows: using a timer, calculate the difference between the current value of dropped frames and a previous value. Store that difference in a variable, droppedFPS. Monitor the current number of incoming frames per second in another variable, currentFPS. If droppedFPS exceeds 20% of the value of currentFPS, switch to a lower bit rate.

Switch streams

To request a transition between streams with the same content encoded at different bit rates, the client application uses the NetStream.play2() method. This method takes as a parameter a NetStreamPlayOptions object, which indicates how the server switches streams.

The NetStreamPlayOptions object contains the following properties:

The following code example uses the SWITCH option to tell the server to switch to a higher bit rate stream. The example does not pass a value for oldStreamName, which tells the server to switch to the new stream at the next logical keyframe. This technique provides the smoothest viewing experience. (When using playlists, pass a value for oldStreamName; see Swap streams in a playlist.) In most dynamic stream-switching cases with a recorded video+audio stream, you can keep the default values of start and len, as in the example.

When the client requests a transition, the server sends a NetStatusEvent.NET_STATUS event with the code NetStream.Play.Transition. (In ActionScript 2.0, it sends an onStatus event with the same code.) The server sends this event to the client almost immediately, which indicates that the operation has succeeded. When the first frames of the new stream render, the server sends an onPlayStatus message with the code NetStream.Play.TransitionComplete. This event lets the client know exactly when the new stream has started to render.

If a client seeks after Flash Player sends a NetStream.Play.Transition message, the streams switch successfully, but Flash Player does not send a NetStream.Play.TransitionComplete message. The player doesn’t send the message because after the seek it enters a new state and cannot send status events about the old state. The player behaves the same way for other callback methods such as onMetaData().

The following example handles a stream transition:

The NetStream.Play.Transition status event contains a reason field. Use this field to get additional information about the state of a transition request. The reason field usually contains the code "NetStream.Transition.Success", meaning the transition request succeeded and was processed normally. When switching between live streams, it is possible that the server could not find a synchronization point between the two streams. In this case, the server forces a transition to occur at an arbitrary frame and the reason field contains the code "NetStream.Transition.Forced". This situation can occur under the following circumstances:

• The two streams being switched don't have the same timeline, and therefore the server cannot select a time to perform the switch.

• The keyframe interval of the new stream is longer than the client's playback buffer, which is the maximum amount of time the server will wait for a transition. Since the server did not see a keyframe, it cannot select a frame for the switch.

• The live queue delay for the live streams is longer than the client's playback buffer, which creates a delay similar to a long keyframe interval.

Handling metadata during stream transitions

Adobe Media Server sends a NetStream.Play.TransitionComplete status event when the bits of a stream transition render to the client. When switching to a new stream, the onMetadata message for the new stream is sent immediately following the NetStream.Play.TransitionComplete status event. Listen for the TransitionComplete event before capturing the metadata. If the stream is live, all data keyframes associated with the stream are transmitted.

Setting the client buffer

Buffering manages fluctuations in bandwidth while a video is playing.

To create a good experience for users, set the buffer to a small value initially. A smaller value lets the stream begin playing relatively quickly. Once playback begins, increase the buffer to a larger value. A larger value ensures that the stream plays more smoothly regardless of noise or interruptions on the network.

To create the best experience for users, monitor the progress of a video and manage buffering as the video downloads. Consider setting different buffer sizes for different users, to ensure the best playback experience. One choice is to measure client bandwidth and set an initial buffer size based on it.

Monitor the buffer size in the client to determine when to switch streams. When client bandwidth is good, the amount of data in the buffer grows or the buffer is full. The client can request the server to switch to a higher video bit rate. When client bandwidth is low, the amount of data in the buffer shrinks or the buffer empties. The client can request the server to switch to a lower bit rate.

Call NetStream.info() to get a NetStreamInfoObject with properties that reflect the current statistics of the stream. The properties that deal with the buffer are the BufferLength and BufferByteLength properties. For details on these properties, see the ActionScript 3.0 Reference.

While the stream is playing, you can also detect and handle netStatus events. For example, when the buffer is full, the netStatus event returns an info.code value of NetStream.Buffer.Full. When the buffer is empty, another event fires with a code value of NetStream.Buffer.Empty. When the data has finished streaming, the NetStream.Buffer.Flush event is dispatched. You can listen for these events and set the buffer size smaller when empty and larger when full.

Set buffer time

To change the buffer time, in seconds, set the NetStream.bufferTime property:

The right size for the buffer varies depending on user bandwidth, and the following values are only suggestions. 5-10 seconds is a good initial buffer size for fast connections; 10 seconds is a good initial buffer size for slow connections. After playback starts, 30-60 seconds is a good buffer size.

Handle buffer events

This example shows how to detect buffer events and adjust the buffer time dynamically, as events occur. Highlights of the code are shown here; to see the complete sample, see the Buffer.as sample file.

1. In the constructor function of your main client class, create a NetConnection object and connect to the server (see Buffer.as in the documentation/samples/Buffer directory in the Adobe Media Server root install directory):

     nc = new NetConnection();   nc.addEventListener(NetStatusEvent.NET_STATUS, netStatusHandler);   nc.connect("rtmp://localhost/Buffer");

2. Write a netStatus event handler, checking for success, failure, and full buffer and empty buffer events and changing buffer size accordingly:

     private function netStatusHandler(event:NetStatusEvent):void {   switch (event.info.code) {   case "NetConnection.Connect.Success":   trace("The connection was successful");   connectStream(nc);   break;   case "NetConnection.Connect.Failed":   trace("The connection failed");   break;   case "NetConnection.Connect.Rejected":   trace("The connection was rejected");   break;   case "NetStream.Buffer.Full":   ns.bufferTime = 30;   trace("Expanded buffer to 30");   break;   case "NetStream.Buffer.Empty":   ns.bufferTime = 8;   trace("Reduced buffer to 8");   break;   }   }

3. Write a custom method to play a stream. In the method, set an initial buffer time, for example, 2 seconds:

   private function connectStream(nc:NetConnection):void { ns:NetStream = new NetStream(nc); ns.addEventListener(NetStatusEvent.NET_STATUS, netStatusHandler); ns.client = new CustomClient(); video = new Video(); video.attachNetStream(ns); ns.play( "bikes", 0 ); ns.bufferTime = 8; trace("Set an initial buffer time of 8 seconds"); addChild(video); }

4. Create the onMetaData() and onPlayStatus() event handlers:

    public function onMetaData(info:Object):void {   trace("Metadata: duration=" + info.duration + " width=" + info.width + " height=" + info.height + " framerate=" + info.framerate);  }    public function onPlayStatus(info:Object):void {   switch (info.code) {   case "NetStream.Play.Complete":   trace("The stream has completed");   break;   }  }  

   These event handlers are needed when you call NetStream.play().

Recognizing transitions in log files

You can track stream events through access logs. Use the logs to differentiate a single stream play with transitions from multiple plays of a stream or different streams. When a transition for a single content stream occurs, the server tracks the status of the stream as a transition. The server logs a stop event for the original stream and a play event for the new stream. Normal stop and play events (that is, a stop or play without a transition) have a status code of 200. Stream transitions have a status code of 210. The access logs provide the following additional information:

If you use a log processor, ensure that it looks at both the status codes and the x-sid values. Look at both values to recognize that a transition occurred on a single logical stream. As with normal streams, stream transitions occur in play/stop pairs. Your log processor can track stream transitions by recognizing stop events that have a 210 status code followed by play events with a 210 status code on the same stream. By looking at the status code, the log processor can also differentiate a stream transition from a play or stop event without a transition.