Distribute peer introductions across servers

Search
[adoberuntime]Flash Media Server 4.5[/adoberuntime]

Workflow for distributing introductions

Note:

You can distribute introductions only among origin servers. You cannot distribute introductions among edge servers. Also, you cannot use the Authorization Plug-in to distribute introductions.

Adobe Media Server introduces RTMFP clients to each other so the clients can connect to each other directly and in a group. When Adobe Media Server plays this role, it’s called an introducer. Flash Media Server 4.0 can introduce clients to each other and help them join a group only if the clients are connected to a single server. Use Server-Side ActionScript APIs added in Flash Media Server 4.5 to introduces clients to each other and let them join a group even if the clients are connected to separate servers. Distributing introductions across servers allows you to scale peer-assisted networking applications.

These steps describe the workflow for distributing peer introductions:

  1. Configure the server to dispatch peer lookup events to the server-side script engine. See Configure the server to dispatch events to the script engine.

  2. Write a server-side script that creates a robust Group of servers to act as introducers. See Deploy fault-tolerant server-only groups.

  3. Write a server-side script that distributes peer introductions across servers. See Distributed introductions API.

How peer introductions work

Note:

For information about port and IP configuration and NAT traversal, see Configure ports and IP addresses for peer-assisted networking.

Peer introduction flow on a single server

By default, the server introduces clients to each other if they connect to the same server.

The introduction flow for clients connected to a single RTMFP introducer.

Client 1 and Client 2 have established connections through their local NATs/firewalls to the same Adobe Media Server. Client 1 has discovered Client 2 either manually or through automatic peer discovery. Client 1 wants to connect directly to Client 2. The following steps describe the introduction flow:

  1. Client 1 sends a peer lookup request to the introducer (Adobe Media Server). The peer lookup request contains the NetConnection.nearID of Client 2. The nearID is a globally unique fingerprint for the client and its active connection to an introducer.

  2. The introducer does the following:

    1. Replies to Client 1 with a redirect message that contains the IP address of Client 2.

    2. Sends a forward message to Client 2. The message tells Client 2 that Client 1 wants to connect.

  3. The clients do the following:

    1. Client 1 resends its peer lookup request to the addresses it received in the redirect message.

    2. Client 2 sends a responder hello message to the addresses of Client 1.

Because of the common introducer, the clients establish a direct connection even though they're behind NATs/firewalls.

Peer introduction application flow on multiple servers

Use Server-side ActionScript to distribute client introductions across multiple Adobe Media Server introducers.

The introduction flow for clients connected to different RTMFP introducers.

In this scenario, Client 1 is connected to Server A and Client 2 is connected to Server B. Client 1 knows the peer ID for Client 2 and wants to establish a direct connection. However, Client 2 is not connected to the same introducer as Client 1. Use Server-Side ActionScript to allow the servers to introduce these clients to each other. The following steps describe the introduction flow:

  1. Client 1 sends a peer lookup request to Introducer 1.

    The peer lookup request calls the application.onPeerLookup() event in a server-side script. The peer passes the IP address of Client 1, the peer ID of Client 2, a tag that identifies the request, and an ID for the RTMFP interface on which the request was received.

  2. In the application.onPeerLookup() callback, Introducer 1 calls application.sendPeerRedirect() to reply to Client 1 with a redirect message. The redirect message includes the known addresses for Client 2 and the public address for Introducer 2.

    Note:

    To prevent a client from establishing a peer connection, Introducer 1 does not reply to the peer lookup request. See Filter introduction requests.

  3. Client 1 resends a peer lookup request to the address for Introducer 2.

  4. Introducer 2 does the following to handle the peer lookup request:

    1. Replies to Client 1 with a redirect message containing the IP addresses for Client 2.

    2. Sends a forward message to Client 2 informing it that Client 1 wants to connect.

  5. The clients do the following:

    1. Client 1 resends its peer lookup request to the addresses for Client 2 that it received from Introducer 2.

    2. The server-script calls Client.introducePeer() on the Client 2 object to send a responder hello message to the addresses for Client 1.

Configure the server to dispatch events to the script engine

By default, the server handles peer lookup events internally. To write a script that distributes introductions across multiple servers, configure the server to dispatch peer lookup events to the script engine. In the Application.xml file, set the mode attribute of PeerLookupEvents to "All".

By default, the server handles a peer joining and leaving a group internally. To write a script that uses the server channel to bootstrap peers within a group, configure the server. Set the mode attribute of JoinLeaveEvents to "All"to dispatch these events to the script engine.

<Application> 
... 
<RTMFP> 
    <PeerLookupEvents mode="All"/> 
    <GroupControl> 
        <JoinLeaveEvents mode="All"/> 
    </GroupControl> 
</RTMFP> 
... 
</Application>

The following are possible values of the mode attributes:

Value

Element

Description

"None"

PeerLookupEvents and JoinLeaveEvents

The default value. The server handles all peer lookup events. You cannot use Server-Side ActionScript to distribute peer introductions or filter peer introductions.

"Partial"

PeerLookupEvents

The server handles peer lookup events for clients connected to the same fmscore process. Handle other peer lookup events with Server-Side ActionScript.

"All"

PeerLookupEvents and JoinLeaveEvents

Handle all lookup events with Server-Side ActionScript.

Deploy servers in a robust server-only group

To deploy servers to act as distributed RTMFP introducers, use the following techniques:

  • Use a serverless RTMFP NetConnection and a NetGroup to join a server-only group (in this case, the groupspec is private and not shared with clients).

  • Add permanent group members at known addresses.

  • Use the server channel to distribute peer bootstrapping across servers in the group.

Use a serverless RTMFP NetConnection to create a NetGroup

To distribute introductions, the servers must share state. All the servers must know about each other and they must know the peer ID and near and far IP addresses of all connected clients. The easiest way to share state is to connect the servers in a NetGroup (also called a group).

To distribute introductions, the server group should be robust. When the servers are joined in a NetGroup, if an introducer fails, all the peers connected to that introducer fail. To create a robust group, use serverless mode to create the RTMFP NetConnections between servers. In serverless mode, if any of the peer servers fail, the other servers stay connected to each other and are informed about the failed server leaving the group.

Use a serverless connection between servers so that the server-side “peer” cannot loose connectivity and its peer ID never changes. The group can use GroupSpecifier.ipMemberUpdatesEnabled to discover neighbors on the same subnet.

The following code creates a connection in serverless mode:

var nc:NetConnection = new NetConnection; 
nc.connect("rtmfp:");

Add permanent members to a server-only group

Use this technique to join groups of peers across different subnets. Choose servers to act as permanent group members. For example, you can identify a known address for one server on each subnet, or for one server at each data center. Distribute the known address to the other servers using a configuration file. The server at the known address is a member of the group and opens peer-to-peer connections to other nodes that contact it over this known address. Once connected, this “bootstrap node” automatically informs its neighbors of other peers in the group. The built-in gossip functionality within groups creates a server-only group with full transitive connectivity.

Choose a server in each data center to act as a permanent group member. Assign these servers endpoint names and IP addresses. On the other servers, write application logic that reads the endpoint names and IP addresses from a configuration file. Use this information to add members directly.

Red servers are bootstrap nodes in server-only groups

In the previous diagram, red servers are bootstrap nodes. Solid lines are connections to permanent members. Dotted lines are peer connections created through the automatic gossip functionality of a group.

Use the following Server-Side ActionScript API to add permanent members to a server-only group:

  • NetConnection.rtmfpBindAddresses

    An Array of Strings representing the specific address or addresses that the NetConnection binds locally when it opens its RTMFP protocol stack.

  • NetConnection.rtmfpEndpointName

    The endpoint name for the local RTMFP protocol stack.

  • NetGroup.addPermanentNeighborByName()

    Manually adds a neighbor by RTMFP endpoint name, instead of by peer ID. If a peer is disconnected from this neighbor, it automatically attempts to reconnect.

  • NetGroup.removePermanentNeighborByName()

    Manually removes the “permanent” status for a neighbor by RTMFP endpoint name. Calling this method does not drop a connection. However, if either end chooses to drop the connection at a future time, the drop is allowed.

The following image describes the application flow for bootstrapping a server-only group:

Note:

The calls in bold are the new Server-Side ActionScript APIs.

This type of bootstrapping is unique because Client 1 initiates the interaction with Client 2 but doesn't know its peer ID. Client 1 knows only the IP address and the rtmfpEndpointName.

When Client 1 calls addPermanentNeighborByName() and passes the rtmfpEndpointName of Client 2, Client 2 retrieves the peer ID of Client 1. Client 2 uses that peer ID to add Client 1 as a neighbor in the reverse direction.

Once these two peers become neighbors, they both receive NetGroup.Neighbor.Connect events in Server-Side ActionScript. Their respective RTMFP stacks gossip to each other about other peers, which eventually results in a connected Group.

There isn't a security risk because the initiator must use the proper groupspec value to add a neighbor. You can distribute this value securely and make it unguessable. If Client 1 does not know the groupspec, the server ignores the request to add Client 2 as a permanent neighbor.

Unlike NetGroup.addNeighbor() or NetGroup.addMemberHint(), the NetGroup.addPermanentNeighbor() method causes the local RTMFP stack to automatically add this neighbor if it is removed. Permanent neighbors allow a group to maintain full connectedness across groups that have member nodes in separate subnets.

To determine whether the local node is a member of a connected Group, call NetGroup.post() from the bootstrap node on a set interval. Other nodes can listen for these messages to detect whether they are connected to the slice of the Group containing the bootstrap node. If they aren’t receiving the messages, they could notify an admin or shutdown.

For detailed information about these APIs, see Server-Side ActionScript Language Reference.

Use the server channel to distribute peer bootstrapping across servers

Server-Side ActionScript provides the following techniques to bootstrap peers to a group:

  • GroupSpecifier.addBootstrapPeer(peerID)

    Adds a peer ID to the group specifier before the peer joins the group.

  • NetGroup.addNeighbor(peerID) and NetGroup.addMemberHint(peerID)

    Manually add a peer to a group. These methods require an external framework to distribute peer IDs.

  • GroupSpecifier.ipMulticastMemberUpdatesEnabled

    Allows peers to find each other on a LAN.

  • Use the server channel.

    This method does not require a LAN and does not require knowledge of peer IDs.

Use the following APIs to bootstrap a server to a group over the server channel:

  • Client.onGroupJoin(groupcontrol)

    Called when a peer joins a group. By default, the server handles the join and leave events internally. To handle these events in a server-side script, configure the Application.xml file. See Configure the server to dispatch events to the script engine.

  • Client.onGroupLeave(groupspecDigest)

    Called when a peer leaves a group.

  • GroupControl.addMemberHint(peerID)

    Manually adds a record specifying that peerID is a member of the group. An immediate connection to this peer is attempted only if needed for the topology.

  • GroupControl.addNeighbor(peerID)

    Manually adds a neighbor by immediately connecting directly to the specified peerID, which must already be in this group.

  • GroupControl.groupspecDigest

    A digest of the canonical groupspec, which securely identifies the group the client has joined.

Use the GroupControl object to tell a peer to add a neighbor. In your application code, as you receive the onGroupJoin() and onGroupLeave() events, maintain a table of group names and peers in the group. Send a message to peers in the group telling them to add the new peer as a neighbor.

When a client joins a group with the GroupSpecifier.serverChannelEnabled flag set to true, a Client.onGroupJoin() callback event is sent to Server-Side ActionScript. The parameter to this method is a GroupControl object, which contains a digest of the canonical GroupSpecifier String for the group. In a script, this groupspecDigest String may be used as a key into a table that stores the list of connected Clients and the GroupControl object representing their membership in the group. When a Client joins a group, search in this table for other connected Clients who have joined the same group. If the search is successful, call the groupControl.addNeighbor() or groupControl.addMemberHint() methods to bootstrap the new client with peer connections to other neighbors within the group.

To maintain a mapping of groupspecDigests to a specific NetGroup object that joined a group, call the server-side GroupSpecifier.encodeGroupspecDigest() method. If you have a source GroupSpecifier, this method generates the groupspecDigest. Otherwise, the server must treat the groupspecDigest property in GroupControl objects as opaque strings. You cannot go from a digest to the starting groupspec String value.

The following server-side script uses the server channel bootstrapping API:

var groups = {}; 
Client.prototype.onGroupJoin = function(groupControl) 
{ 
    groupControl["client"] = this; // Remember the associated Client. 
    var groupControlArray = groups[groupControl.groupspecDigest]; 
    if (groupControlArray) 
    { 
        trace("Register Client in existing Group (by groupspec digest): " + 
              groupControl.groupspecDigest + 
              ", current Group size is: " + 
              groupControlArray.length); 
 
        // find a random member to bootstrap with 
        r = Math.random(); 
        index = Math.floor(r * groupControlArray.length); 
 
        var peerGroupControl = groupControlArray[index]; 
        groupControl.addNeighbor(peerGroupControl["client"].farID); 
        groupControlArray.push(groupControl); 
    } 
    else 
    { 
        trace("Track client joining new Group (by groupspec digest): " + groupControl.groupspecDigest); 
 
        groupControlArray = []; 
        groupControlArray.push(groupControl); 
        groups[groupControl.groupspecDigest] = groupControlArray; 
    } 
}

For detailed information about these APIs, see Server-Side ActionScript Language Reference.

Distributed Introductions API

Use the distributed introductions Server-Side ActionScript API to control the peer introduction process and to create a shared peer registry.

Control the peer introduction process

  • Application.onPeerLookup()

    Called when a client initiates a peer look-up. This event receives an object with the following properties:

    Property

    Data type

    Description

    targetPeerID

    String

    The peer ID of the target peer to which the initiating peer wants to connect.

    initiatorAddress

    String

    The IP address of the peer initiating the request to connect. Send redirect information to this address.

    tag

    ByteArray

    A value that uniquely identifies this lookup request.

    interfaceID

    Number

    Identifies the RTMFP interface on which the request was received.

  • Application.sendPeerRedirect()

    Call this method from the application.onPeerLookup() callback function to send the initiating peer an Array of addresses for the target peer.

  • Client.introducePeer()

    Call on a target peer to open a connection with the peer that initiated the request to connect.

    For detailed information about these APIs, see Server-Side ActionScript Language Reference.

Create a peer registry

Use these APIs to create a peer registry. The peer registry lists the peer ID and IP addresses of every client that connects to an introducer. Every server acting as an introducer in a distributed environment shares this peer registry and uses it to locate peers when they receive a peer lookup. Write the application logic to create and share the peer registry.

  • Client.farAddress

    The IP address and port number from which the server sees the client connection originate.

  • Client.nearAddress

    The public address on the server to which the client connected.

  • Client.potentialNearAddresses

    The public interfaces available for communication with the server.

  • Client.reportedAddresses

    The addresses at which a client can receive RTMFP traffic. The client can update this value multiple times over the lifetime of its RTMFP connection to the server. This value can contain IP addresses and ports that the client has opened behind its NAT. If so, use these addresses and ports for interacting with other peers behind a common NAT.

  • Client.onFarAddressChange()

    Called when a client's farAddress has changed. For example, an address changes when a client transitions from a LAN to a wireless connection.

  • Client.onReportedAddressChange()

    Called when a client reports new addresses.

For detailed information about these APIs, see Server-Side ActionScript Language Reference.

Example: Distribute introductions across servers

This example explains at a high level how to distribute introductions across servers.

To distribute introductions across a group of servers, every server must know the peer ID and IP addresses for every connected client. Track these values in a peer registry. In the following example, assume the following functions exist to control the peer registry:

  • addClient(peerId, clientAddresses)

    When a client connects to the server, add the peer ID and address to the peer registry.

  • removeClient(peerId)

    When a client disconnects from the server, delete it from the peer registry.

  • updateClient(peerId, clientAddresses)

    When a far address in the reported addresses changes, update the reported addresses in the peer registry.

  • getClientIfLocal(peerId)

    Returns the Client object if the client with the given peerId is connected to the current server, otherwise returns null.

  • getRemoteClientAddrs(peerId)

    Returns the client addresses from the peer registry for the given peer ID. If the client is not found, returns null.

To create the peer registry of client addresses, create a helper function that returns all the client addresses:

function getClientRedirectAddrs(client) 
{ 
    var redirectAddresses = client.reportedAddresses.slice(0); 
    redirectAddresses.push(client.farAddress); 
    redirectAddresses.concat(client.potentialNearAddresses); 
    return redirectAddresses; 
}

When a client connects over RTMFP, add it to the peer registry:

application.onConnect = function(client) { 
    if (client.protocol == "rtmfp") 
    { 
        addClient(client.farID, getClientRedirectAddrs(client)); 
    client.onFarAddressChange = function() { 
        updateClient(this.farId, getClientRedirectAddrs(client)); 
    }; 
    client.onReportedAddressesChange = function() { 
        updateClient(this.farId, getClientRedirectAddrs(client)); 
    }; 
    } 
} 
 
application.onDisconnect = function(client) { 
    removeClient(client.farId); 
}

Send redirect messages and responder hello messages in the application.onPeerLookup() callback function:

application.onPeerLookup = function(event) { 
    // Check whether the target peer is local 
    // (connected to the same server as the initiating peer). 
    var targetPeer = getClientIfLocal(event.targetPeerID); 
    if (targetPeer) 
    { 
    // Send the addresses of the target peer to the initiating peer. 
     application.sendPeerRedirect(getClientRedirectAddrs(targetPeer), event); 
    // Note: getClientIfLocal() must return the actual client object, 
    // because that allows us to call introducePeer() 
    // The target peer sends a responder hello to the initiating peer. 
    targetPeer.introducePeer(event.initiatorAddress, event.tag); 
    } 
    else 
    { 
    // Client is not connected locally. 
    targetPeerAddrs = getRemoteClientAddrs(event.targetPeerID); 
    if (!targetPeerAddrs) 
    { 
        // Returning without calling application.sendPeerRedirect() 
        // prevents the peer-to-peer connection. 
        return; 
    } 
    // Send the addresses of the target peer to the initiating peer. 
    application.sendPeerRedirect(targetPeerAddrs, event); 
    } 
}

Consider the following scenario in which the previous code is used on two Adobe Media Server introducers: Server A and Server B. Client 1 is connected to Server A and Client 2 is connected to Server B. Client 1 wants to connect to Client 2.

  • Client 1 sends a peer lookup request to Server A.

  • The peer lookup request calls the callback function application.onPeerLookup()in the server-side script.

  • Because Client 2 is not connected to Server A, the else branch of code runs that does not have the local Client object. Server A calls getRemoteClientAddrs() to get the target addresses out of the shared store. These addresses include the addresses of Server B (to which Client 2 is directly connected). The script calls application.sendPeerRedirect()to pass the target addresses to Client 1.

  • When the redirect reaches Client 1, Flash Player sends a peer lookup request to all the target addresses, including the address of Server B.

  • When the request arrives at Server B, the if branch of the code runs that has the local target Client object. The code calls sendPeerRedirect() again to send information to Client 1. But now that the script has the Client object representing Client 2 at Server B, it can also call introducePeer() on Client 2. This call tells Client 2 to start sending packets to Client 1. (Client 1 has been trying to send packets toward Client 2 since it got the first sendPeerRedirect() address set from Server A).

  • When both clients are sending traffic at each other, the necessary NAT hole-punching has happened. Both clients can see each other's traffic and establish a peer-to-peer connection. (Assuming that both clients are behind well-behaved NATs.)

Filter introduction requests

Flash Media Server 4.5

[adoberuntime]Flash Media Server 4.5[/adoberuntime]

You can also use the distributed introduction API to filter introduction requests. The following sample script creates a map of peer IDs and client addresses. When the server receives a peer lookup request, the application.onPeerLookup() callback checks to see whether the target peer is in the map. If it isn’t, the server denies the lookup request.

// A registry mapping peer ID values to Client objects 
// for this application instance. 
var peerIDToClientMap = {}; 
 
// On client connect, add RTMFP clients to the peer ID registry. 
application.onConnect = function(client) { 
    if (client.protocol == "rtmfp") 
        peerIDToClientMap[client.farID] = client; 
} 
 
// 
application.onPeerLookup = function(event) { 
    var targetPeer = peerIDToClientMap[event.targetPeerID]; 
    if(!targetPeer){ 
        // Returning without calling application.sendPeerRedirect() 
        // prevents the peer-to-peer connection. 
        return; 
    } 
}

For detailed information about these APIs, see Server-Side ActionScript Language Reference.

Use the Administration API to monitor distributed introductions

Use the following properties returned in a call to the getServerStats()Administration API method to monitor distributed introductions:

  • rtmfp_forwards

  • rtmfp_lookups

  • rtmfp_lookups_denied

  • rtmfp_redirects

There are many ways to use these statistics. The number of lookups minus the number of redirects yields the number of lookup queries that were ignored or denied. The number of redirects minus the number of forwards yields the number of lookups in which the initiating client was redirected to a different Adobe Media Server node to connect with the target peer. The rtmfp_lookups_denied statistic is a counter that tracks the number of lookups that were explicitly denied (for example, because they were invalid). It is incremented when application.denyPeerLookup() is called. If there are too many denials, it could indicate a DOS attack.

Use the following properties returned in a call to the getServerStats()Administration API method to monitor peers joining and leaving a group:

  • group_join

  • group_leave

Get help faster and easier

New user?

Quick links

View all your plans Manage your plans
View quick links
Hide quick links

Legal Notices    |    Online Privacy Policy