Adobe Connect 12.12 LiveKit installation (Enhanced Audio-Video setup)

Search

Introduction

Adobe Connect uses LiveKit, an open-source WebRTC-based media server, to provide enhanced Audio and Video capabilities. This LiveKit-based Enhanced Audio-Video setup typically runs on multiple Linux® nodes with specific roles: signalling nodes, LiveKit media/SIP nodes, and recording and ASR (captioning) nodes. This setup also uses PostgreSQL, Redis, and a NATS message-bus cluster (required by LiveKit's media and SIP servers), which can be installed on one or separate machines, depending on the usage.

Each node’s setup is done by copying the installer zip file to it, editing the config files, running the dependency installation script, and finally, the main installation script. Below are the topics that are covered in the following:

These steps are described in the Installation section.

Pre-requisites and system requirements

Estimate the size for Enhanced Audio/Video servers/nodes

The file, session_participants_count.sql, can be run to estimate the size of the enhanced audio/video servers. The output from the SQL queries is the input to the calculator, which is an Excel file, Additional Enhanced A/V hardware Livekit-estimator.xlsx.

The calculator helps in estimating the number of VMs needed based on your past usage of Adobe Connect. The calculator needs the following set of inputs.

  • The number of server CPU cores and RAM.
  • The attached SQL queries are used to determine the peak number of concurrent sessions done in the last 12 months and the average number of attendees.
  • The estimated number of publishers in each session. A publisher is a meeting attendee (host, presenter, or participant) connecting their microphone in the meeting room (both muted or unmuted) and connecting their webcam in the meeting room (both live and paused).
Note

Both the session_participants_count.sql and Additional Enhanced A/V hardware Livekit-estimator.xlsx files are included in the Installer package and are not available elsewhere.

Port opening requirements

Source Destination Port range Protocol Use
WebRTC LB Signalling Node 18443 TCP WebRTC Signalling
0.0.0.0 WebRTC LB 443 TCP WebRTC Signalling
Connect Pro Server WebRTC LB 443 TCP  
Self Signalling Node 6379 TCP Redis
Media Server Node Signalling Node 6379 TCP Redis
Recording Node Signalling Node 6379 TCP Redis
Media Server Node Signalling Node 4222 - 4224 TCP NATS
Self Signalling Node 5432 TCP Postgres
Self Signalling Node 6222 TCP NATS
Self Signalling Node 7890 TCP LiveKit LB
Signalling Node Connect Pro Server LB 443 TCP  
Auto Captioning/ASR Node Signalling Node 7890 TCP LiveKit LB
Auto Captioning/ASR Node WebRTC LB 443 TCP WebRTC Signalling
Media Server Node Signalling Node 7890 TCP LiveKit LB
Self Auto Captioning/ASR Node 6000 TCP Captioner Service
Signalling Node Recording Node 5000-5100 TCP Recording Request
Connect Pro Server Recording Node 80 TCP Download Recording
Signalling Node Media Server Node 7880 TCP WebRTC Signalling
Self Media Server Node 6883 TCP Cluster Connection
Self Media Server Node 6884 UDP Cluster Connection
0.0.0.0 Media Server Node 50000 - 60000 UDP SRTP (real-time media flow)
Self Media Server Node 30000 - 40000 UDP TURN Relay
0.0.0.0 Media Server Node 3478 UDP TURN over UDP
0.0.0.0 Media Server Node 443 TCP TURN over TLS
0.0.0.0 Media Server Node 5060 UDP SIP
0.0.0.0 Media Server Node 5060 TCP SIP
0.0.0.0 Media Server Node 10000-20000 UDP SIP
Auto Captioning/ASR Node Media Server Node 50000 - 60000 UDP SRTP (real-time media flow)
Auto Captioning/ASR Node Media Server Node 3478 UDP TURN over UDP
Recording Node Media Server Node 50000 - 60000 UDP SRTP (real-time media flow)
Recording Node Media Server Node 3478 UDP TURN over UDP
Recording Node WebRTC LB 443 TCP WebRTC Signalling

0.0.0.0 = Users/Client/Internet

Setup load balancing for on-premise setup of Livekit

The load balancing is typically done via an external Application Load Balancer with the below configuration.

  • HTTPS port 443: Signalling nodes HTTP port 18443 - CPS and End-users connect to this listener, and that is the entry point to the new WebRTC cluster.

View Set up load balancing for on-premise setup of Enhanced Audio/Video (WebRTC) for more information.

Note

For the LiveKit engine, load balancing is only needed for port 18443 – there is no separate admin panel port (9090) to load-balance, unlike the LiveSwitch engine.

Prepare the environment

A LiveKit setup would need at least four Red Hat servers: one each for Signalling, Media, Recording, and ASR nodes. The estimator, based on meeting load, could suggest multiple Media Servers, Recording nodes, ASR nodes, and Signalling nodes. An Application Load Balancer would be used for SSL offloading and for communication between CPS and end-user clients with Signalling Nodes.

Here, we’ll describe the four most common server setups in detail.

  • Provision 4 Red Hat servers. Kindly refer to the Hardware estimator for configuration.
  • The installation requires a non-root user with sudo access.
    • After a fresh install of Red Hat, create a new user-id.
    • To enable sudo for the new user ID on RHEL, add the ID to the wheel group:
      • Become root by running su.
      • Run usermod -aG wheel your_user_id.
      • Log out and back in again using the new ID.

Copy the Installer zip file

  1. Copy the Patch_ACS_12_12_0_EnhancedAV.zip to the home directory of all the nodes. For example, scp Patch_ACS_12_12_0_EnhancedAV.zip -i ssh_key.pem my-user@webrtc.corp.example.com:/home/my-user/

  2. Optionally verify the downloaded signed zip using Jarsigner. Jarsigner is installed as part of JAVA.

    1. Verify if JAVA is installed by using command [ java -version ]. If Java is installed, then you get the JAVA version as an output.
    2. If Java is not present on the machine then, install JAVA.

      sudo yum install java-1.8.0-openjdk-devel

    3. Now, copy the below command in the Terminal window and click Enter.

      jarsigner -verify -verbose Patch_ACS_12_12_0_EnhancedAV.zip

    4. The output of the verification contains:
      • list of files inside the zip
      • certificate information from Adobe for authentication
      • the successful output message "jar is verified" or the unsuccessful "jar is not verified"
    5. If the certificate information is valid and if the successful verification output message is printed, the user can use the zip contents and then proceed to installation else the user needs to contact Adobe Support.

  3. Extract the ZIP. Make sure that the files have appropriate permissions. Use the commands: Do not run any command with root/sudo access unless clearly specified.

    unzip Patch_ACS_12_12_0_EnhancedAV.zip
    cd ~/ncc-onprem-installer/
    sudo chmod +x ExternalDependencies/install.sh
    sudo chmod +x MainInstall.sh
    sudo chmod +x check-connectivity.sh
    sudo chmod +x uninstall.sh

    Note

    When running the setup on an environment with no internet access or a locked-down environment, run the below command:

  4. Execute the dependency installation script.

    1. When running the setup on an environment with no internet access or a locked-down environment, run the below command to install external dependencies. If you have internet access, continue with the following step 2.

      Traverse to the Installer parent directory. For Example, ensure you are in ~/ncc-onprem-installer/ directory.
      Execute bash ExternalDependencies/package-util.sh --install --rhel8 on RHEL8.
      Execute bash ExternalDependencies/package-util.sh --install --rhel9 on RHEL9.
      This installs all required external dependencies on the box.
      This is needed once per node.
    2. Traverse to the Installer parent directory. For Example, ensure you are in ~/ncc-onprem-installer/ directory.
      Execute bash ExternalDependencies/install.sh. This installs all required external dependencies on the box. This is needed once per node.

Installation process

The following steps below walk through installing the LiveKit environment (as an example) on 4 separate Red Hat instances. A Signalling node, a Recording node, a Media-Server node, and an ASR node. Pay close attention to the services that are configured in the containers.conf for each node’s instructions. You need to configure each node with a specific set of services/containers in the configuration files.

** On a lab system, you can install all these ‘nodes’ (Signalling, Recording, Media, and ASR Server) on one Linux instance. In that case, in the containers.conf you would set ‘count=1’ for all servers/containers needed for your environment.

Signalling node

On the Signalling nodes, you would typically run the following services. Each service runs as a docker container.

  • config (Configuration Service)
  • cas (New Connect API Service)
  • apigw (API Gateway/Router)
  • nats (3-node NATS cluster – message bus required by the LiveKit media/SIP servers)
  • redis
  • postgres

Signalling node(s) typically be in the private subnet accessible to Connect client via an External Load balancer.

Procedure

  1. Edit the Hosts file. This hosts’ file entry must be added on every node in the deployment – Signalling, Media, Recording, and ASR – with <private-ip> always set to the Signalling node's private IP. You could use a text editor like nano or vi.

    1. Open the /etc/hosts file using nano or vi editor e.g. sudo vi /etc/hosts.
    2. Add the following line at the end. Replace the <private-ip> with the private IP of the Signalling node. Kindly note the space between each word below.

    <private-ip> controller.livekit.svc.cluster.local nats.livekit.svc.cluster.local redis.ncc.svc.cluster.local postgres.ncc.svc.cluster.local cas.ncc.svc.cluster.local config.ncc.svc.cluster.local pi.ncc.svc.cluster.local auth.ncc.svc.cluster.local

    192.168.1.100 controller.livekit.svc.cluster.local nats.livekit.svc.cluster.local redis.ncc.svc.cluster.local postgres.ncc.svc.cluster.local cas.ncc.svc.cluster.local config.ncc.svc.cluster.local pi.ncc.svc.cluster.local auth.ncc.svc.cluster.local

  2. Edit the configuration files

    Edit the configuration file, present in, ncc-onprem-installer/Config/config.conf. The instructions to edit the file are added as comments in the config file. This file needs to be edited on the Signalling node; other nodes copy this same file rather than editing it independently (see their respective sections below).

    1. Update the following keys under the hostnames block:

    Key Single box Multi box
    fqdn NCC_FQDN (e.g. https://onpremautomation.rtc.dev.adobeconnect.com) same
    signalingNode instance private IP Signalling node private IP
    configService instance private IP Signalling node private IP
    recordingNode instance private IP HCR node private IP
    mediaNode instance private IP Media node private IP
    sipNode instance private IP Media node private IP

    2. Update the following keys under the livekit block (identical for single box and multi box):

    Key Value
    enabled "true"
    mediaServers ["<media node private IP>"]

    3. Now, edit ncc-onprem-installer/Config/containers.conf. The instructions to edit the file are added as comments in the file. This file needs to be edited for each host separately. Depending on the services to be installed and the number of containers to be deployed.

    4. On a typical Signalling Node, you would install:

    • casServer
    • configService
    • apiGateway
    • nats (set count for a 3-replica NATS cluster)
    • redis
    • postgres
    • lb

    5. So, set count=1 for all the services above and 0 for all others.

    Important

    Kindly set restart=1 for configService, after making any change to Config.conf.

  3. Execute the main installer script. Switch to the Main installer directory cd ~/ncc-onprem-installer/ Execute the main installer script, bash MainInstall.sh. Wait for the confirmation message. Success Message: 2023-01-31 18:21:34,033 : INFO : Main : 55 : Installation successful. Failure Message: 2023-01-31 20:04:44,849 : ERROR : Main : 59 : Installation failed. Check the installer.log for further information.

  4. Verify Installation

    1. Health Check API For the health check of the node, you can browse to the URL: http://<private_ip>:18443/health. A healthy response must be 200 OK {"apigw":"ok"}. To verify from the RedHat machine you can use the CURL command For example, curl -v http://172.31.56.203:18443/health
    2. Verify Container Status In the Terminal window, run the command docker ps. The output should look like the below, and the STATUS should not be restarted for any of the containers. Confirm the nats1/nats2/nats3 containers are all healthy before proceeding to the Media node.


    CONTAINER ID   IMAGE                                                          COMMAND                  CREATED        STATUS               PORTS                                                                 NAMES
aec6e82c8a6d   docker-connect-release.dr-uw2.adobeitc.com/ncc-onprem/redis:12.4     redis-server /red...     2 hours ago    Up 2 hours           0.0.0.0:6379->6379/tcp                                              redis
5add9e041cc0   docker-connect-release.dr-uw2.adobeitc.com/ncc-onprem/postgres:12.4  postgres                 2 hours ago    Up 2 hours           0.0.0.0:5432->5432/tcp                                              postgres
991a1a6e9145   docker-connect-release.dr-uw2.adobeitc.com/ncc-onprem/lb:12.4        nginx -g daemon o...     2 hours ago    Up 2 hours           0.0.0.0:7890->7890/tcp, 80/tcp, 443/tcp, 5349/tcp                    lb
246cdf8b2fdd   docker-connect-release.dr-uw2.adobeitc.com/ncc/config:12.12.0        /bin/bash /run.sh        2 hours ago    Up 2 hours           0.0.0.0:8089->8089/tcp                                              config
c310c28f04af   docker-connect-release.dr-uw2.adobeitc.com/ncc/cas:12.12.0          /bin/bash /run.sh        2 hours ago    Up About an hour     0.0.0.0:8090->8090/tcp                                              cas
4e8fddab5aba   docker-connect-release.dr-uw2.adobeitc.com/ncc/apigw:12.12.0        /bin/bash /run.sh        2 hours ago    Up 2 hours           0.0.0.0:18080->8080/tcp, 0.0.0.0:18443->8443/tcp, 80/tcp             apigw
dd88436359bc   docker-connect-release.dr-uw2.adobeitc.com/nats:2.11.6-alpine       -cluster nats://n...     2 hours ago    Up 2 hours           0.0.0.0:4222->4222/tcp, 0.0.0.0:6222->6222/tcp, 0.0.0.0:8222->8222/tcp   nats1
d08343e01019   docker-connect-release.dr-uw2.adobeitc.com/nats:2.11.6-alpine       -cluster nats://n...     2 hours ago    Up 2 hours           0.0.0.0:4223->4223/tcp, 0.0.0.0:6223->6222/tcp, 0.0.0.0:8223->8222/tcp   nats2
a578607fb4cc   docker-connect-release.dr-uw2.adobeitc.com/nats:2.11.6-alpine       -cluster nats://n...     2 hours ago    Up 2 hours           0.0.0.0:4224->4224/tcp, 0.0.0.0:6224->6222/tcp, 0.0.0.0:8224->8222/tcp   nats3

Media Server Node

On the Media node(s), you would run the following services:

  • livekit-server (Media Server): For enhanced AudioVideo
  • livekit-sip (SIP service): For using SIP, installed only when livekit.enabled=true

Media nodes should be in the public subnets (or DMZ) and assigned a public IP or mapped to a public IP via 1:1 NAT. Clients connects to the Media node’s Public IPs directly.

Procedure

  1. Edit the Hosts file Add the same /etc/hosts line documented in the Signalling node's procedure above (<private-ip> controller.livekit.svc.cluster.local nats.livekit.svc.cluster.local redis.ncc.svc.cluster.local postgres.ncc.svc.cluster.local cas.ncc.svc.cluster.local config.ncc.svc.cluster.local pi.ncc.svc.cluster.local auth.ncc.svc.cluster.local) to this node's /etc/hosts as well, with <private-ip> set to the Signalling node's private IP.

  2. Edit the Configuration files.

    • Copy config.conf from the Signalling node, ~/ncc-onprem-installer/Config/config.conf – refer to the Signalling node section above for what needs to be updated there.
    • In the livekit block of config.conf, add this node's private IP to the mediaServers array. Also update hostnames.mediaNode in config.conf to this node's private IP.
    • If this node also runs the SIP service, set hostnames.sipNode in config.conf to this node's private IP.
    • Edit ~/ncc-onprem-installer/Config/containers.conf and set livekitServer.count=1 (and livekitSip.count=1 if SIP/Telephony is needed). Also update livekitServer.mediaNodeIP (and livekitSip.sipNodeIP, if applicable) to this node's IP.

  3. Execute the main installer script. Switch to the Main installer directory cd ~/ncc-onprem-installer/ Execute the main installer script, bash MainInstall.sh. Wait for the confirmation message. Success Message 2023-01-31 18:21:34,033 : INFO : Main : 55 : Installation successful. Failure Message 2023-01-31 20:04:44,849 : ERROR : Main : 59 : Installation failed. Check the installer.log for further information.

  4. Verify Installation In the Terminal window, run the command docker ps and ensure that the livekit-server container (and livekit-sip container, if enabled) is running.

Configure SIP

Unlike the LiveSwitch engine, the LiveKit engine does not use the FM admin web panel for SIP configuration. Instead, on the Signalling node, configure the SIP trunk directly in Config/config.conf's livekit block:

livekit {
...
sipTrunkName = "<your trunk name>"
sipTrunkAddress = "<your SIP trunk address>"
sipTrunkNumber = "<your SIP trunk number>"
sipTrunkAuthUser = "<SIP trunk auth username>"
sipTrunkAuthPassword = "<SIP trunk auth password>"
}

After editing these values on the Signalling node, re-run bash MainInstall.sh on the Signalling node to apply the change, then restart the cas container on the Signalling node:

docker restart cas

Recording Node

Recording nodes should run in a private network. On recording nodes, you can run one or more instances of:

  • hcr (Recording Container. # of hcr containers decide simultaneous recordings that can run)
  • recordingserver (WebServer to serve recordings files to CPS. 1 per Recording Node)

Recording nodes should be reachable on:

  • TCP 80 from the local network, so that CPS can download the recordings.
  • TCP 5000-5100 from the local network, individual recording containers would be bound to host ports in that range.
  • TCP 8090 from the local network.

Recording nodes should be able to reach the Media nodes on the public IP on ports listed in the Media nodes section and to CPS on port 443 to make a successful recording.

Procedure

  1. Edit the Hosts file Add the same /etc/hosts line documented in the Signalling node's procedure above (<private-ip> controller.livekit.svc.cluster.local nats.livekit.svc.cluster.local redis.ncc.svc.cluster.local postgres.ncc.svc.cluster.local cas.ncc.svc.cluster.local config.ncc.svc.cluster.local pi.ncc.svc.cluster.local auth.ncc.svc.cluster.local) to this node's /etc/hosts as well, with <private-ip> set to the Signalling node's private IP.

  2. Edit the Configuration files

    • Copy config.conf from the Signalling node, ~/ncc-onprem-installer/Config/config.conf – refer to the Signalling node section above for what needs to be updated there; the hostnames.recordingNode key should already point to this HCR node's private IP.
    • Now, edit ~/ncc-onprem-installer/Config/containers.conf. The instructions to edit the file are added as comments in the file.
    • On a Recording Server Node, you would install:
      • recordingContainer
      • recordingserver
    • So, set count >= 1 for recordingContainer and 0 for all the others. recordingServer will be automatically installed.
    Note

    You can have one recording container per 8 CPUs. For example, on a 16-CPU machine, you should have a maximum of 2 recording containers.

  3. Execute the main installer script Switch to the Main installer directory cd ~/ncc-onprem-installer/ Execute the main installer script, bash MainInstall.sh. Wait for the confirmation message. Success Message 2023-01-31 18:21:34,033 : INFO : Main : 55 : Installation successful. Failure Message 2023-01-31 20:04:44,849 : ERROR : Main : 59 : Installation failed. Check the installer.log for further information.

  4. Verify Installation In the Terminal window, run the command docker ps and ensure that the hcr and recordingserver containers are running.

ASR Node

ASR or Closed Captioning nodes should run in a private network. On ASR nodes, you run instances of:

  • asrCaptionerCpu
  • asrClient (LiveKit Agents dispatcher that routes captioning jobs to asrCaptionerCpu)

Both are installed only when livekit.enabled=true.

  1. Edit the Hosts file Add the same /etc/hosts line documented in the Signalling node's procedure above (<private-ip> controller.livekit.svc.cluster.local nats.livekit.svc.cluster.local redis.ncc.svc.cluster.local postgres.ncc.svc.cluster.local cas.ncc.svc.cluster.local config.ncc.svc.cluster.local pi.ncc.svc.cluster.local auth.ncc.svc.cluster.local) to this node's /etc/hosts as well, with <private-ip> set to the Signalling node's private IP.

  2. Edit the Configuration files.

    • Copy config.conf from the Signalling node, ~/ncc-onprem-installer/Config/config.conf – refer to the Signalling node section above for what needs to be updated there.
    • Now, edit ~/ncc-onprem-installer/Config/containers.conf.
    • On a ASR Server Node, you would install: asrCaptionerCpu, asrClient So, set count=1 for asrCaptionerCpu and asrClient

  3. Execute the main installer script. Switch to the Main installer directory cd ~/ncc-onprem-installer/ Execute the main installer script, bash MainInstall.sh. Wait for the confirmation message. Success Message 2023-01-31 18:21:34,033 : INFO : Main : 55 : Installation successful. Failure Message 2023-01-31 20:04:44,849 : ERROR : Main : 59 : Installation failed. Check the installer.log for further information.

Configure Livekit in Adobe Connect Service (CPS)

The following steps need to be performed on all Adobe Connect 12 (CPS) server(s):

  1. Add the below configs in custom.ini of all the Connect server(s) present at <Installation_Dir>/Connect/custom.ini.

    # comma separated list of CAS discovery URLs
# Possible values
WEBRTC_CAS_DISCOVERY_URLS=http://<Signalling Node IP >:18443/api/cps/ingest
WEBRTC_CAS_DISCOVERY_URLS=http://<Load Balancer URL>/api/cps/ingest
# Shared secret for CAS, used to sign requests to CAS. Enter the one that you set under the hmac section in the config.conf file on the Signalling node.
WEBRTC_CAS_SHARED_SECRET - # Shared secret for JWT. Enter the one that you set under the jwtZonalSecrets section in the config.conf file on the Signalling node. CAS_JWT_SHARED_SECRET=<CorrectHorseBatteryStaple>
# Enable the LiveKit engine for Enhanced Audio-Video
ENABLE_LIVEKIT=true

  2. Save the file and restart connectpro service.

  3. Call the following API:

    https://server_domain/api/xml?action=acl-field-update&field-id=1267&value=true&acl-id=<account-id>

Post installation process

  1. Verify Network Connectivity. Run the Connectivity Test workflow to verify network connectivity between different nodes:

    • Login to the Signalling Node.
    • Traverse to the Installer parent directory, for example, cd ~/ncc-onprem-installer/ncc-onprem-installer/
    • Run the command bash check-connectivity.sh.
    • The connectivity tool uses the provided values in the Config.conf file to run basic network tests like:
      • HTTP requests between different services
      • Network connectivity tests on the required TCP and UDP ports
      • Name resolution
      • Database connectivity, including the NATS cluster
    • This test checks the required connectivity between the nodes and services and share the results to help diagnose network connectivity issues.

  2. Test Adobe Connect Meeting with LiveKit-based Enhanced Audio-Video

    • Create a new meeting and ensure that Enhanced Audio/Video option is selected.
    • Join the meeting room and turn on the Mic and the Camera.
    • If possible also join the room from a different PC or Mobile device and verify the audio-video exchange.
    • Try Screen Sharing.
    • Start Recording. Wait for 30s and Stop. Now, verify if the recording is accessible.

Configure the TURNS binding

This is needed so that we can have secure TURN communication.

Unlike the LiveSwitch engine, there is no certificate upload step in an admin panel. TURNS is configured directly via dedicated fields in config.conf's livekit block:

  • Set livekit.turnsDomain to the FQDN clients will use to reach TURNS (TLS relay over port 443). This is typically only needed when clients connect from outside your network or from other companies with restrictive firewall policies.
  • Create the certs directory on the Media Server node: cd ~ && mkdir -p connect/turns
  • Copy the SSL certificate for the TURNS domain to ~/connect/turns/turns.crt, and its private key to ~/connect/turns/turns.key (or update livekit.certFileName / livekit.keyFileName in config.conf if you use different filenames).
  • Run bash MainInstall.sh to apply the change.

If livekit.turnsDomain is left blank, TURNS is not configured and clients fall back to STUN/TURN over UDP/TCP 3478 or the SRTP media range.

Uninstallation

  • Change to the installer root directory(Root directory contains uninstall.sh), cd ~/ncc_onprem_installer.
  • Execute, bash uninstall.sh.
  • To remove Postgres and Redis databases data directory from the Signalling node, run the below command.

    sudo rm -rf ~/connect

Note

This removes all NCC components and images from the system. But the external dependencies(Python, PIP, Virtualenv, and Python packages) are not be deleted.

Adobe, Inc.

Get help faster and easier

New user?

Quick links

View all your plans Manage your plans
View quick links
Hide quick links

Legal Notices    |    Online Privacy Policy