JSDoc: Tutorial: Twilio Video Integration

tutorial-twiliovideo

Overview

The Twilio Video API does not directly expose the WebRTC objects required to use MDPS, therefore we provide API functions that provide access to those objects via the associated Twilio objects. This tutorial introduces these functions and their usage within the context of the Twilio Video API.

Required Objects

The following table shows the required WebRTC objects and the corresponding Twilio Video objects.

WebRTC	Twilio Video
RTCPeerConnection	Room
Local MediaStream	LocalAudioTrack
Remote MediaStream	RemoteAudioTrack
Local RTCDataChannel	LocalDataTrack
Remote RTCDataChannel	RemoteDataTrack

API

The following MDPS API is provided for accessing the Twilio Video objects:

/**
 * Extract the RTCPeerConnection object from the given Twilio Room.
 * 
 * @memberOf TwilioVideo
 * @param {Room} room the Twilio Room object 
 * @returns {RTCPeerConnection} the associated RTCPeerConnection object
 */
twilioVideoExtractPeerConnectionFromRoom(room)

/**
 * Extract the MediaStream object from the given Twilio MediaTrack.
 * 
 * @memberOf TwilioVideo
 * @param {MediaTrack} track the Twilio MediaTrack object 
 * @returns {MediaStream} the associated MediaStream object
 */
twilioVideoExtractMediaStreamFromTrack(track)

/**
 * Extract the RTCDataChannel object from the given Twilio LocalDataTrack.
 * 
 * @memberOf TwilioVideo
 * @param {LocalDataTrack} track the Twilio LocalDataTrack object 
 * @returns {RTCDataChannel} the associated RTCDataChannel object
 */
twilioVideoExtractDataChannelFromTrack(track)

/**
 * Setup data channel message monitoring for the given Twilio RemoteDataTrack.
 * 
 * @memberOf TwilioVideo
 * @param {RemoteDataTrack} track the Twilio RemoteDataTrack object
 */
twilioSetupRemoteDataChannelHandlerForTrack(track)

Usage

Local Audio

The setMicStream API requires the local MediaStream object that corresponds to the user's mic as a parameter, and returns the MediaStream object that represents the MDPS-processed stream, which needs to be added to the created RTCPeerConnection. Fortunately, the Twilio Video Video.connect API allows for passing in an optional WebRTC MediaStreamTrack (for which MediaStream is the containing object). Therefore, the MDPS mic processing can be setup via the following code:

        var constraints = {
            video: true,
            audio: true,
        };
        if (navigator.mediaDevices.getUserMedia) {
            navigator.mediaDevices.getUserMedia(constraints).then(function(mediaStream) {
                localStream = mdps.setMicStream(mediaStream);
                // Join the Room with the token from the server and the
                // LocalParticipant's Tracks.
                connectOptions.tracks = localStream.getTracks();
                Video.connect(data.token, connectOptions).then(roomJoined, function(error) {
                    log('Could not connect to Twilio: ' + error.message);
                });
            });
        }

where we get the local MediaStream via getUserMedia, get the MDPS processed stream, and add its track to the options for Video.connect.

Remote Audio

The setSpeakerStream API requires the remote MediaStream object. After the Twilio Room is connected and roomJoined from above is called, the remote MediaStream can be obtained from the RemoteAudioTrack either via the existing Room participants:

     function setupSpeakerStream(track, element) {
        if ("RemoteAudioTrack" == track.constructor.name) {
            var mediaStream = mdps.twilioVideoExtractMediaStreamFromTrack(track);
            mdps.setSpeakerStream(mediaStream);
            // need to mute the remote video element to allow
            // for MDPS processing
            el.onloadedmetadata = function(e) {
                el.play();
                el.muted = true;
            };
         }
     }
     
    room.participants.forEach(function(participant) {
        var tracks = Array.from(participant.tracks.values());
        tracks.forEach(function(track) {
           setupSpeakerStream(track, el);
        }
    });

or by listening to the Room's trackAdded event for new participants:

     room.on('trackAdded', function(track, participant) {
        setupSpeakerStream(track, el);
     });

where element is the video HTML element.

Local Data Channel

Unlike RTCDataChannels, which are bi-directional, the Twilio data tracks are uni-directional. Therefore, each end of the connection has to setup its own data track, which the remote end will listen to. This can also be performed in the roomJoined function after the Room is connected, like so:

    // get the RTCPeerConnection from the Room
    var conn = mdps.twilioVideoExtractPeerConnectionFromRoom(room);
    // we are the controller if we're the 1st in the room
    var isController = 0 == room.participants.size;
    // create a new LocalDataTrack to be published
    var localDataTrack = new Video.LocalDataTrack();
    // publish a data track
    room.localParticipant.publishTrack(localDataTrack).then(function(publication) {
        var dataChannel = mdps.twilioVideoExtractDataChannelFromTrack(localDataTrack);
        mdps.setupDataChannel(conn, isController, dataChannel);
    });

Remote Data Track

The remote end needs to listen for the data track publication from the other end and register for the data channel's message events. This can be done in response to a local participant's RemoteDataTrack. As with the RemoteAudioTrack, this can be triggered from existing or new room participants:

     function setupDataChannel(track) {
        if ("RemoteDataTrack" == track.constructor.name) {
            // listen for data channel messages from the remote data track
            mdps.twilioSetupRemoteDataChannelHandlerForTrack(track);
            return;
        }
     }
     
    room.participants.forEach(function(participant) {
        var tracks = Array.from(participant.tracks.values());
        tracks.forEach(function(track) {
           setupDataChannel(track);
        }
    });

    room.on('trackAdded', function(track, participant) {
       setupDataChannel(track, el);
    });

Conclusion

Once the above is taken into consideration, the rest of the MDPS API can be used as normal.

A working example using the above methods (taken from the Twilio Video QuickStart example) can be found in the /thirdparty/twilio-video folder of the Zero Footprint SDK. There are setup instructions at that Twilio link, but briefly, from the twilio-video directory:

$ cp .env.template .env
$ vi .env       # add your Twilio credentials
$ npm install
$ npm start

You can then go to https://localhost:3000 in your browser to use the example. MDPS info is logged at the top of the page. Note that this will currently only work for 2 participants in the room.