ONNX extensions & Whisper support for Linux

TODO.md

@@ -1,6 +1,8 @@
 # TODO
 - [ ] Windows CI can't see dylib
-- [ ] Set up Linux CI for running models
-- [ ] ONNX Runtime Extensions for Linux/Windows/macOS Intel/iOS/Android
+- [x] Set up Linux CI for running models
+- [x] ONNX Runtime Extensions for Linux
+- [x] Whisper for Linux
+- [ ] ONNX Runtime Extensions for Windows/macOS Intel/iOS/Android
 - [ ] Whisper iOS
 - [ ] Whisper Android

example/lib/whisper_widget.dart

@@ -30,7 +30,7 @@ class _WhisperWidgetState extends State<WhisperWidget> {
           style: Theme.of(context).textTheme.headlineSmall,
         ),
         Text(
-          'macOS ARM only',
+          'macOS ARM and Linux 64 only, currently',
           style: Theme.of(context).textTheme.bodySmall,
         ),
         heightPadding,

lib/onnx/ort.dart

@@ -600,7 +600,13 @@ String get ortDylibPath {
 String get ortExtensionsDylibPath {
   final isTesting = !kIsWeb && Platform.environment['FLUTTER_TEST'] == 'true';
   if (isTesting) {
-    return 'macos/onnx_runtime/osx/libortextensions.0.9.0.dylib';
+    if (Platform.isMacOS) {
+      return 'macos/onnx_runtime/osx/libortextensions.0.9.0.dylib';
+    } else if (Platform.isLinux) {
+      return 'linux/onnx_runtime/libortextensions.so.0.9.0';
+    } else {
+      throw 'Unsure how to load ORT during testing for this platform (${Platform.operatingSystem})';
+    }
   }
   switch (defaultTargetPlatform) {
     case TargetPlatform.android:
@@ -610,7 +616,7 @@ String get ortExtensionsDylibPath {
     case TargetPlatform.iOS:
       throw UnimplementedError();
     case TargetPlatform.linux:
-      throw UnimplementedError();
+      return 'libortextensions.so.0.9.0';
     case TargetPlatform.macOS:
       return 'libortextensions.0.9.0.dylib';
     case TargetPlatform.windows:

linux/CMakeLists.txt

@@ -46,6 +46,7 @@ target_link_libraries(${PLUGIN_NAME} PRIVATE PkgConfig::GTK)
 # external build triggered from this build file.
 set(fonnx_bundled_libraries
   "${CMAKE_CURRENT_SOURCE_DIR}/onnx_runtime/libonnxruntime.so.1.16.1"
+  "${CMAKE_CURRENT_SOURCE_DIR}/onnx_runtime/libortextensions.so.0.9.0"
   PARENT_SCOPE
 )
 

onnx_runtime/updating.md

@@ -1,4 +1,4 @@
-# Updating
+# Updating ONNX Runtime
 1. 
 For macOS/Linux/Windows:
 - Find latest release on [Github](https://github.com/microsoft/onnxruntime/releases). 
@@ -14,23 +14,33 @@ a. Setup LLVM, etc. See "Using this package" at https://pub.dev/packages/ffigen
 b. `dart run ffigen --config onnx_runtime/ffigen_config.yaml`
 c. Done!
 
+## Verifying
 
-# Verifying
-
-## Test
+### Test
 Launching the Example app and press Test Mini-lm-l6-v2. 
 Then, the word "match" should display below the button.
 
-## macOS
+### macOS
 The binary must support both Apple Silicon and Intel. See [here](https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary) for detailed info from Apple. 
 
 TL;DR:
 1. Build on Apple Silicon in profile mode.
 2. You can then find the .app in the build/ directory. The easiest way to locate it is to right click on the running app, hover Options, then press Show in Finder. 
 3. Right click on the .app and choose Get Info. At the bottom of the General section, there is a checkbox that says Open using Rosetta. Launch the app again, and verify it works.
 
-## Windows
+### Windows
 Windows x64 is all we need to support currently.
 Flutter is not quite stable for Windows arm64 ([Github issue](https://github.com/flutter/flutter/issues/62597)).
 Neither arm nor arm64 devices are particularly numerous.
-As of October 2023, at most ~0.35% of Windows devices could plausibly be Win32. [source](https://www.pcbenchmarks.net/os-marketshare.html)
+As of October 2023, at most ~0.35% of Windows devices could plausibly be Win32. [source](https://www.pcbenchmarks.net/os-marketshare.html)
+
+# Updating ONNX Runtime Extensions
+ONNX Runtime Extensions is a supplementary library that includes everything from audio decoding to a BPE tokenizer: this enables Olive, a more intense ONNX optimizer, to do things such as export a Whisper model that combines the encoder and decoder model, takes audio bytes as input, and provides strings as output: this is _much_ more convenient and presumably faster.
+
+It is also _significantly_ harder to land. 
+
+For desktop platforms, you need to checkout [the Github repo](https://github.com/microsoft/onnxruntime-extensions) and build on the platform you're producing. For example, on Linux, I ran `./build.sh` in the checkout root, then copied `/GitHub/onnxruntime-extensions/out/Linux/RelWithDebInfo/lib/libortextensions.so.0.10.0` to `/Github/fonnx/linux/onnx_runtime/libortextensions.so.0.9.0`.
+
+The version naming difference is because the repo labels with the next version, but it makes more sense to couple it to the current release, so it's easier to understand if iOS/Android are on the same version.
+
+iOS and Android require importing a package, see [here on onnxruntime.ai](https://onnxruntime.ai/docs/extensions/).

test/models/whisper_test.dart

@@ -11,8 +11,8 @@ void main() {
   /// worth the accuracy loss, it's too frustrating to use.
   const modelPath = 'example/assets/models/whisper/whisper_tiny.onnx';
   final whisper = WhisperNative(modelPath);
-  final shouldSkip = !Platform.isMacOS;
-  final skipReason = shouldSkip ? 'Whisper only works on ARM64 Mac currently' : null;
+  final shouldSkip = !Platform.isMacOS && !Platform.isLinux;
+  final skipReason = shouldSkip ? 'Whisper only works on ARM64 Mac and X64 Linux currently' : null;
 
   test('Whisper works', skip: skipReason, () async {
     String testFilePath = 'test/data/rain_in_spain.wav';