Configure HTTP Streaming failover

In a real-world clustered HTTP Streaming scenario, you want your site to offer robust playback, even in the face of server-side streaming problems. HTTP Streaming failover, new in FMS 4.5.2, lets you control how AMS handles certain streaming problems.

HTTP Streaming failover features primarily address the issues of liveness and dropout.

The following figure illustrates a simple two-server cluster of packagers with a Reverse Proxy controlling access. The explanations that follow use this figure as a reference.

• Liveness is a server-side situation in which a packager advertises a stale bootstrap (that is, a stale view of a live stream). Although there are other potential scenarios, the following describes one liveness situation:

  1. The stream has been running normally for some time on the server side.

  2. Packager 2's connection to the encoder fails. The connection between Packager 2 and the proxy continues running. Suppose that at the time of this failure, the last successfully written fragment on Packager 2 was still #700 with current media time as X.

  3. A short amount of time elapses, in this case, 10 fragment intervals.

  4. The server-side now suffers from liveness. Packager 2 is serving stale bootstrap information. Packager 2 indicates that there has been no update in the bootstrap after time X, even though more recent fragments are available on Packager 1 (that is, the current media time is greater than X.)

  5. Playback stalls at the player end.

• Dropout - is a server-side situation in which a packager has gaps in its bootstrap (that is, gaps in its fragment list). The following scenario describes a dropout situation:

  1. The stream has been running normally for some time on the server side.

  2. Packager 2's connection to the encoder fails. The connection between Packager 2 and the proxy continues running. Suppose that at the time of this failure, the last successfully written fragment on Packager 2 was still #700.

  3. A short amount of time elapses, in this case, 10 fragment intervals.

  4. Packager 2's encoder connection is restored and remains restored for a long time.

  5. The server-side now suffers from dropout. Packager 2's bootstrap is missing entries for #701-710, even though those fragments are still available on Packager 1.

  6. Playback stalls at the player end.

HTTP Streaming failover provides a multi-faceted solution:

• Best-effort fetch

• Server-side configuration

• Control plane management

Best-effort fetch enables the OSMF and iOS video players to continue playback as normally as possible in the presence of short-term liveness and dropout problems on the server-side.

OSMF

The OSMF 2.0 player includes HDS functionality that integrates with server-side configuration and run-time settings. For more information on OSMF 2.0 client side functionality, see the OSMF documentation on SourceForge.

HLS

FMS 4.5.2 includes server-side settings and functionality that enable best-effort fetch.

You configure server settings for HTTP Streaming failover in the httpd.conf and manifest.xml files.

Configure HDS and HLS settings in httpd.conf. For more information on server-side configuration in the httpd.conf file, see Configure live and on-demand HTTP Streaming at the server level (httpd.conf). Configure the bestEffortFetchInfo element in the manifest.xml file, as described in Manifest.xml.

To implement HTTP Streaming failover, you write a client application that manages the state of events and streams by using a set of REST-based control plane APIs. Control plane is a router term and in effect, that is what your client application does through these APIs.

The basic workflow is as follows:

• Begin repeatedly checking stream status at a set interval. The exact interval depends on your configuration.

• Disable any stream with a bad status code.

• You could also compare the age of the bootstrap file and disable the stream for which the bootstrap is older than some predetermined interval.

AMS control plane REST interface request path

The AMS control plane REST interface request path differs, depending on whether you are using HDS, HLS with adaptive bit-rate (ABR), or HLS with a single bit-rate stream:

• HDS: http://server/ctrlplane/application-name/instance-name/event-name.f4m/

• HLS with ABR: http://server/ctrlplane/application-name/instance-name/event-name.m3u8/

• HLS with single bit-rate: http://server/ctrlplane/application-name/instance-name/event-name/stream-name.m3u8/

For example:

• HDS: http://www.example-server.com/ctrlplane/livepkgr/_definst_/liveevent.f4m/

• HLS with ABR: http://www.example-server.com/ctrlplane/livepkgr/_definst_/liveevent.m3u8/

• HLS with single bit-rate: http://www.example-server.com/ctrlplane/livepkgr/_definst_/liveevent/livestream.m3u8/

You use HTTP GET or POST, depending on the endpoint and parameters passed.

Description

Returns stream status.

Endpoint

status

Result

If the .disabled file exists in the streams directory for the respective associated stream, the control plane module returns "disabled" in the status element for the particular stream. This endpoint returns stream age for each stream (multiple streams in case of MBR) calculated as current time in minutes - last modified time for the bootstrap file.The up element indicates whether the stream is up. It is set to false if the bootstrap has not been updated in specified number of seconds (configurable as MaxBootstrapAge). The global up element is set to true if all bootstrap up elements are true, otherwise it is set to false. The endpoint sets the upper level status to disabled if all streams are disabled, otherwise it is set to enabled.

The done element indicates whether the stream is finished.

Sample request

[GET] http://www.example-server.com:8134/ctrlplane/livepkgr/_definst_/livestream.f4m/status

Sample response

This is an example response for an MBR stream in which all streams are disabled:

Description

Disables all streams for an event (works for both HDS and HLS streams). You send this as an HTTP POST request with no body.

Endpoint

disable

Result

Upon receiving this request, the control plane module creates a .disabled file in all streams directories (multiple streams directories for an MBR stream) and returns an XML formatted response indicating success or failure for each stream.

If a client subsequently requests streams for the event, the Apache module (mod_f4fhttp or mod_hlshttp) checks for the existence of a .disabled file before processing the request. If a .disabled file exists, the module returns a 503 error (configurable in httpd.conf).

Sample request

[POST] http://www.example-server.com:8134/ctrlplane/livepkgr/_definst_/livestream.f4m/disable

Sample response

Description

Disables a stream (works for both HDS and HLS streams). You send this as an HTTP POST request with the stream name specified in the body.

Disable a stream if you detect a liveness problems or to take a faulty packager out of production for some other reason. As mentioned below, the Apache module returns a 503 error, making the faulty packager a non-entity.

Endpoint

disable

Result

Upon receiving this request, the control plane module creates a .disabled file in the streams directory and returns an XML formatted response indicating success or failure.

If a client subsequently requests the stream, the Apache module (mod_f4fhttp or mod_hlshttp) checks for the existence of a .disabled file before processing the request. If a .disabled file exists, the module returns a 503 error (configurable in httpd.conf).

Sample request

[POST] http://www.example-server.com:8134/ctrlplane/livepkgr/_definst_/livestream.f4m/disable

Sample response

Description

Enables all streams for an event (works for both HDS and HLS streams). You send this as an HTTP POST request with no body.

Endpoint

enable

Result

Upon receiving this request, the control plane module deletes any existing .disabled files in all streams directories (multiple streams directories for an MBR stream) and returns an XML formatted response indicating success or failure for each stream.

If a client subsequently requests streams for the event, the Apache module (mod_f4fhttp or mod_hlshttp) processes the request.

Sample request

[POST] http://www.example-server.com:8134/ctrlplane/livepkgr/_definst_/livestream.f4m/enable

Sample response

Description

Enables a stream (works for both HDS and HLS streams). You send this as an HTTP POST request with the stream name specified in the body. This request enables a stream and removes the .disabled file.

Endpoint

enable

Result

Upon receiving this request, the control plane module deletes any existing .disabled file in the streams directories and returns an XML formatted response indicating success or failure.

If a client subsequently requests the stream, the Apache module (mod_f4fhttp or mod_hlshttp) processes the request.

Sample request

[POST] http://www.example-server.com:8134/ctrlplane/livepkgr/_definst_/livestream.f4m/enable

Sample response

Description

Signals that an event is complete. You send this as an HTTP POST request with no body.

You signal an event as done so that the client is notified when content ends. This achieves two things: the control plane does not activate failover, and client applications can take appropriate action to signal the end of content to the user.

Endpoint

done

Result

Upon receiving this request, the control plane module creates a .done file in all streams directories (multiple streams directories for an MBR stream) and returns an XML formatted response indicating success or failure for each stream.

Sample request

[POST] http://www.example-server.com:8134/ctrlplane/livepkgr/_definst_/livestream.f4m/done

Sample response

Description

Signals that a stream complete. You send this as an HTTP POST request with the stream name specified in the body.

You signal a stream as done so that the client is notified when content ends. This achieves two things: the control plane does not activate failover, and client applications can take appropriate action to signal the end of content to the user.

Endpoint

done

Result

Upon receiving this request, the control plane module creates a .done file in the associated streams directory and returns an XML formatted response indicating success or failure.

Sample request

[POST] http://www.example-server.com:8134/ctrlplane/livepkgr/_definst_/livestream.f4m/done

Sample response

Description

Signals that an event is still in progress. You send this as an HTTP POST request with no body. This request undoes a /done request.

Endpoint

inProgress

Result

Upon receiving this request, the control plane module deletes any existing .done files in all streams directories (multiple streams directories for an MBR stream) and returns an XML formatted response indicating success or failure for each stream.

Sample request

[POST] http://www.example-server.com:8134/ctrlplane/livepkgr/_definst_/livestream.f4m/inProgress

Sample response

Description

Signals that a stream is still in progress. You send this as an HTTP POST request with the stream name specified in the body. This request undoes a /done request.

Endpoint

inProgress

Result

Upon receiving this request, the control plane module deletes any existing .done file in the associated streams directory and returns an XML formatted response indicating success or failure for each stream.

Sample request

[POST] http://www.example-server.com:8134/ctrlplane/livepkgr/_definst_/livestream.f4m/inProgress

Sample response