LiveData RTVT

This guide is provided by LiveData. Agora is planning a documentation upgrade program for all extensions on the marketplace. Please stay tuned.

LiveData RTVT extension allows you to embed real-time voice transcription and translation into your mobile application providing automated speech recognition(ASR) without any upfront data training requirements.You can find the integration examples below.

Understand the tech

The LiveData RTVT extension encapsulates the core APIs of the LiveData RTVT SDK. By calling the setExtensionProperty or setExtensionPropertyWithVendor method of the Agora Video SDK v4.x and passing in the corresponding key and value, you can quickly integrate capabilities of LiveData RTVT. For details, see the key-value overview.

Prerequisites

The development environment has to meet the following requirements:

• Android
  • Android Studio 4.1 or later
  • A physical device (not an emulator) running Android 5.0 or later
• iOS
  • Xcode 9.0 or later.
  • A physical device (not an emulator) running iOS 11.0 or later.

Project Setup

The LiveData RTVT extension works together with the Video SDK v4.x. Refer to the following doc to integrate the SDK and start a basic voice call:

• SDK quickstart
  • Android
  • iOS

To receive a appKey and a appSecret from LiveData

• please buy and activate the extension project on Agora Console, then click View in the Secret column.
• or contact us via Agora.

Integrate the extension

Android

1. Download the Android package of LiveData RTVT from the Extensions Marketplace.
2. Unzip the package, and save all .aar files to the /app/libs path of your project folder.
3. In the app/build.gradle file, add the following line in dependencies:

implementation fileTree(dir: "libs", include: ["*.jar", "*.aar"])

iOS

1. Download the iOS package of LiveData RTVT from the Extensions Marketplace.
2. Unzip the package, and save all .framework files to the <ProjectName> path.
3. Ensure that you select Embed & Sign instead of Embed.

You can save .framework files under your project folder, as follows:

.
├── <ProjectName>
├── <ProjectName>.xcodeproj

Call Sequence

This section describes the call sequence you implement to use LiveData RTVT features in your app.

1. Enable the extension

Android

When you initialize AgoraRtcEngine :

• first call addExtension to load the extension
• then call enableExtension to enable the extension
  

Tips:If you are sure that the so library exists, and the call to addExtension returns -3, please try to use System.loadLibrary to load the dynamic library.

RtcEngineConfig config = new RtcEngineConfig();
    config.addExtension("agora-iLiveData-filter-pre"); 
    config.addExtension("agora-iLiveData-filter-post");

    engine = RtcEngine.create(config);
    engine.enableExtension("iLiveDataPre", "RTVT_PRE", true); 
    engine.enableExtension("iLiveDataPost", "RTVT_POST", true);

iOS

When you initialize AgoraRtcEngine, call enableExtensionWithVendor to enable the extension.

AgoraRtcEngineConfig *config = [AgoraRtcEngineConfig new];

// set config.eventDelegate = self to accept the `onEvent` callback
// enable RTVT extension
[_agoraKit enableExtensionWithVendor:[iLiveDataSimpleFilterManager companyName]
                          extension:[iLiveDataSimpleFilterManager plugName]
                            enabled:YES]；

2. Start using the extension

Android

Call setExtensionProperty to specify key as startAudioTranslation_post, startAudioTranslation_pre and pass appkey, appsecret and other parameters in value in json format. appsecret` parameters in value.

    JSONObject jsonObject = new JSONObject();
    jsonObject.put("appKey", "");
    jsonObject.put("appSecret", "");
    jsonObject.put("srcLang", "zh");
    jsonObject.put("dstLang", "en");
    jsonObject.addProperty("asrResult", true);
    jsonObject.addProperty("transResult", true);
    jsonObject.addProperty("tempResult", false);

    ExtensionInfo extensionInfo = new ExtensionInfo();
    extensionInfo.localUid = ;
    extensionInfo.channelId = ;
    extensionInfo.remoteUid = ;

-Post

    engine.setExtensionProperty("iLiveDataPost", "RTVT_POST", extensionInfo, "startAudioTranslation_post", jsonObject.toString());

-Pre

    engine.setExtensionProperty("iLiveDataPre", "RTVT_PRE", "startAudioTranslation_pre", jsonObject.toString());

iOS

When you are ready to start using RTVT, call setExtensionPropertyWithVendor and pass in the corresponding keys and values:

• set key asstartAudioTranslation
• set value as appkey, appsecret,srclang dstLang in JSON

NSDictionary * translateDic = @{
                                @"appKey":@"appKey",
                                @"appSecret":@"appSecret",
                                @"srcLanguage":@"zh",
                                @"destLanguage":@"en"
                              };

NSData * translateDicJsonData = [NSJSONSerialization dataWithJSONObject:translateDic options:NSJSONWritingPrettyPrinted error:nil];
NSString * translateDicJsonString = [[NSString alloc] initWithData:translateDicJsonData encoding:NSUTF8StringEncoding];

[_agoraKit setExtensionPropertyWithVendor:[iLiveDataSimpleFilterManager companyName]
                                extension:[iLiveDataSimpleFilterManager rtvt_plugName])
                                      key:@"startAudioTranslation"
                                    value:translateDicJsonString];

3. Stop using the extension

Android

When stop using RTVT, call setExtensionProperty and pass in the corresponding keys and values:

    engine.setExtensionProperty("iLiveDataPost", "RTVT_POST", "closeAudioTranslation_post", "{}");

• 前处理

    engine.setExtensionProperty("iLiveDataPre", "RTVT_PRE", "closeAudioTranslation_pre", "{}");

iOS

When stop using RTVT, call setExtensionPropertyWithVendor and pass in the corresponding keys and values:

• set key ascloseAudioTranslation
• set value as end

[_agoraKit setExtensionPropertyWithVendor:[iLiveDataSimpleFilterManager companyName]
                                extension:[iLiveDataSimpleFilterManager rtvt_plugName]
                                      key:"closeAudioTranslation"
                                    value:"end"];

4. Result callback

LiveData RTVT extension provides result of voice transcription and translation, you can receive all return of results or either of them, using onEvent callback of the Agora SDK.

Android

After enable LiveData RTVT extension， you can receive the results of voice transcription and translation via onEvent callback. The description of onEvent callback keys can be seen below.

• Post

@Override
public void onEvent(String vendor, String extension, String key, String value) {
  vendor: “iLiveDataPost”
    key: “recognizeResult ‘RecognizeResult identifier ’translateResult ‘TranslateResult ’TranslateResult identifier ‘recognizedTempResult ’Temporary RecognizeResult identifier ‘translatedTempResult ’TemporaryTranslateResult identifier 
    extension: “RTVT_POST”
      value: the corresponding keys are recognized result and translated result and temporary recognized result and temporary translated result, respectively.
}

• Pre

@Override
public void onEvent(String vendor, String extension, String key, String value) {
  vendor: “iLiveDataPre”
    key: “recognizeResult ‘RecognizeResult identifier ’translateResult ‘TranslateResult ’TranslateResult identifier ‘recognizedTempResult ’TemporaryResult identifier ‘translatedTempResult ’TemporaryTranslateResult identifier 
    extension: “RTVT_PRE”
      value: the corresponding keys are recognized result and translated result and temporary recognized result and temporary translated result, respectively.
}

iOS

After enable LiveData RTVT extension， you can receive the results of voice transcription and translation via onEvent callback. The description of onEvent callback keys can be seen below.

-(void)onEvent:(NSString *)provider extension:(NSString *)extension key:(NSString *)key value:(NSString *)value{

       provider:"iLiveData"
      extension:"RTVT"
            key: "recognizeResult"  "translateResult"
          value:
              {
                ""result", 
                "startTs", 
                "endTs",
              }
}

Sample Project

The complete sample code and project is provided on GitHub:

Platform	Language	Sample Project
Android	Java	Sample Project
iOS	Objective-C	Sample Project

Run the project

Android

1. git clone:

git clone https://github.com/highras/rtvt-agora-marketplace.git

2. Open the Sample project in Android Studio.
3. Gradle sync with the project.
4. Open app/src/main/res/values/string_configs.xml file:
   • change agora_app_id and agora_access_token to your own Agora project information.
   • change livedata_translate_pid and livedata_translate_key to your own LiveData RTVT extension information。
5. Using a physical Android device (not an emulator) to run the project.

iOS

1. git clone:

git clone https://github.com/highras/rtvt-agora-marketplace.git

2. Open the project in Xcode. Access to the iOS/SW_Test path, and run the CocoaPods command as below:

pod install

3. Open the sample project SW_Test.xcworkspace in Xcode.
4. Open iOS/SW_Test/SW_Test/ViewController.mm file:
   • Fill in you own project information appId and Token from Agora console.
   • Fill in your own LiveData RTVT extension information appKeyRTVT and appSecretRTVT from Agora console。
5. Using a physical iOS device (not an emulator) to run the project.

Result

After running the project successfully, LiveData RTVT sample project will be installed on you Android or iOS device.

1. Start the sample, fill in the channel number in the input box and click Join. The button Join will be change to Leave after join the channel successfully.
2. Click the button start translation, speak to the device microphone, and you will see the real-time transcription and translation on the screen.
3. Click the button stop translation to stop transcription and translation.
4. Click the button stopto stop running LiveData RTVT sample project.

API reference

This section lists the APIs related to using extensions with the Agora SDK.

Android

• addExtension in the RtcEngineConfig class
• enableExtension in the RtcEngine class
• setExtensionProperty in the RtcEngine class
• getExtensionProperty in the RtcEngine class
• onEvent in the IMediaExtensionObserver class

iOS

• enableExtensionWithVendor in the AgoraRtcEngineKit class
• setExtensionPropertyWithVendor in the AgoraRtcEngineKit class
• getExtensionPropertyWithVendor in the AgoraRtcEngineKit class
• onEvent in the AgoraMediaFilterEventDelegate class

Key description

To implement the LiveData RTVT extension in app, you need to pass in the corresponding key-value pair。

setExtensionProperty or setExtensionPropertyWithVendor

When calling setExtensionProperty or setExtensionPropertyWithVendor, you can pass keys:

Key	Description
startAudioTranslation	start real-time voice transcription and translation
closeAudioTranslation	stop real-time voice transcription and translation

onEvent

The Agora SDK onEvent contain keys:

Key	Description
recognizeResult	result of real-time voice transcription
translateResult	result of real-time voice translation
start	error message when calling startAudioTranslation

Key-value description

startAudioTranslation

Value	Description
appKey	buy and activate the extension project on Agora Console, then click View in the Secret column
appSecret	buy and activate the extension project on Agora Console, then click View in the Secret column.
srcLanguage	source language, see language supported list
destLanguage	destination language, see language supported list

closeAudioTranslation

Value	Description
end	stop real-time voice transcription and translation

recognizeResult

Value	Description
{"result", "startTs", "endTs"}	result of real-time voice transcription

translateResult

Value	Description
{"result", "startTs", "endTs"}	result of real-time voice translation

start

Value	Description
start	error message when calling startAudioTranslation