#2355 Handle multiple audio tracks on a single audio element

[kayakr]

GitHub Issue: (link)
#2355 Handle multiple audio tracks

• Other Relevant Links (Google Groups discussion, related pull requests,
  Release pull requests, etc.)

What does this Pull Request do?

Reworks islandora audio js to add a eventListener to detect a track change, to allow the user to switch between multiple webvtt tracks for an audio element (e.g. different webvtt files for different languages).

What's new?

I don't expect this to work for multiple audio elements on the same page. The event listener assumes only one track has the mode='showing'.

It wraps the previous logic in a new function bindVttSource() so we can invoke it on different track files.

How should this be tested?

1. Create an Audio repository item that includes a multiple WebVTT tracks on the Audio media.
2. Play the audio and observe that if webvtt captions are available, it is not possible to switch between them.
3. Apply the PR
4. Clear caches
5. Play the audio again and observe the captions appear, and it is possible to switch between the captions while playing.

Browser should see markup like this:

<audio controls="controls" data-once="islandora_audio_captions">
    <source src="/_flysystem/fedora/2025-02/myinterview.mp3" type="audio/mpeg">
    <track src="/_flysystem/fedora/2025-02/transcript-french.vtt" srclang="fr" label="(fr)Transcript" kind="captions" default="default">
    <track src="/_flysystem/fedora/2025-02/transcript-english.vtt" srclang="en" label="(en)Transcript" kind="captions">
    <track src="/_flysystem/fedora/2025-02/transcript-punjabi.vtt" srclang="pa" label="(pa)Transcript" kind="captions">
</audio>

Documentation Status

• Does this change existing behaviour that's currently documented? Unsure
• Does this change require new pages or sections of documentation? Probably.
• Who does this need to be documented for? Users
• Associated documentation pull request(s): ___ or documentation issue ___

Interested parties

@Islandora/committers @seth-shaw-asu

[kayakr]

Turning captions off in HTML5 player doesn't remove captions from display. This appears to be an issue with the current code e.g. https://sandbox.islandora.ca/islandora/interview-diane-hill

[seth-shaw-asu]

Ah, I didn't even realize this was a thing because I've been using Firefox which doesn't give a captions select/on-off option for audio. Looks like I should have been using Chrome or Edge when working on the audio captions.

[seth-shaw-asu]

@kayakr, where can I get a copy of the audio with multiple webVTTs so that I can test this out?

[seth-shaw-asu]

As for removing the captions when they are turned off; we could modify https://github.com/kayakr/islandora/blob/2.x/modules/islandora_audio/js/audio.js#L53-L63 so that we return when we find a showing text track and then, if none are showing, reset the captions box. E.g.

        element.textTracks.addEventListener('change', function(e)  {
          // Which track is showing?
          for (let i = 0; i < this.length; i++) {
            if (this[i].mode === 'showing') {
              // Build cues for this track, now we know the index.
              // @todo: Make this work with multiple audio elements.
              var src = document.querySelectorAll("track[kind='captions'], track[kind='subtitles']")[i].getAttribute('src');
              bindVttSource(src);
              return;
            }
          }
          document.querySelector('div.audioTrack').innerHTML = "";
        });

Granted, this would reset the captions for all audio players on the page (if there are more than one). Unfortunately, I don't see a good way to fix this.

[kayakr]

@seth-shaw-asu The sample files I am working with are part of a project that is not yet live. I grabbed the following files from https://ableplayer.github.io/ableplayer/demos/desc2.html
2035-webvtt-sample-files.zip

• ableplayer-wwa.mp3
• wwa_captions_en.vtt
• wwa_captions_fr.vtt

The markup I have in Islandora is:

<audio controls="controls" data-once="islandora_audio_captions">
  <source src="/_flysystem/fedora/2025-02/ableplayer-wwa.mp3" type="audio/mpeg">
  <track src="/_flysystem/fedora/2025-02/wwa_captions_fr.vtt" srclang="fr" label="(fr)Transcript" kind="subtitles">
  <track src="/_flysystem/fedora/2025-02/wwa_captions_en.vtt" srclang="en" label="(en)Transcript" kind="subtitles" default="default">
</audio>

[kayakr]

As for removing the captions when they are turned off; we could modify https://github.com/kayakr/islandora/blob/2.x/modules/islandora_audio/js/audio.js#L53-L63 so that we return when we find a showing text track and then, if none are showing, reset the captions box.

I just tried this code and it didn't work for me as the event listeners are already in place for cues and will fire after the captions box is cleared. We need to tear down or block the cues. I'm not so familiar with the cue code so will need to dig into it more...

[kayakr]

@seth-shaw-asu It looks like we can't removeEventListener() to stop the timeupdate events because it's an anonymous function...

[kayakr]

@seth-shaw-asu I've added a data attribute as a semaphore to indicate whether a track is showing or not. It works, but it's a bit hacky as the cue events are still firing but are gated by checking the data-showing attribute on the audio element. It works for me, but I suspect the entire approach could be implemented in a more elegant way if my javascript-foo was better...

[rosiel]

I'm excited about this! I documented the current one-track behaviour, and would be happy to [edit the docs] if this goes through. If I understand:

• "Follow the same instructions as for video" -- this stands; you have to go to the service file (or whatever media is playing) and add track files in the Track field, selecting options like "kind" and "label"
• "For audio only upload one Track file (others will not be displayed) - For audio files, you can add multiple tracks and will be given the option to choose from them in the video player[?]
• "For audio, ensure the track that one track is marked "Default track" (otherwise it will not be displayed) so that it will be displayed automatically
• For audio, the selected "kind" of media track does not have any effect. [is this still true?]
• [updated screenshot of the audio player with captions and switcher]
• While you can add subtitles in different languages, you may only choose from the site's installed languages. --- still true?

[seth-shaw-asu]

It works, but it's a bit hacky ... but I suspect the entire approach could be implemented in a more elegant way if my javascript-foo was better...

Same here. My JS-foo is weak, but we do what we can to get the job done.

It works, but I have one nit-pick that it is throwing an error at the beginning of the track play:

Uncaught TypeError: Cannot read properties of null (reading 'start')
    at audio.js?srzpne:99:40
    at Array.find (<anonymous>)
    at HTMLAudioElement.<anonymous> (audio.js?srzpne:97:36)
    at HTMLAudioElement.<anonymous> (audio.js?srzpne:84:24)
(anonymous)	@	audio.js?srzpne:99
(anonymous)	@	audio.js?srzpne:97
(anonymous)	@	audio.js?srzpne:84

If we can catch/avoid that then I'm ready to green-light it.

[seth-shaw-asu]

                let cue = e.detail.find(
                  (cue) =>
                    cue !== null &&
                    currentTime >= cue.start &&
                    currentTime <= cue.end
                );

[kayakr]

@seth-shaw-asu I added that line to suppress null cue warning.
Audio player on Chrome showing selection between en and fr captions, then turning captions off.

There's more hackiness in that showing or hiding captions triggers when the next cue event happens, that may not coincide with the timing of making the change in the UI. Again, a refactor could fix this kind of thing. But this is better than what we had before...

Screen.Recording.2025-02-21.at.9.49.09.AM.mov

@rosiel See screen recording above. The 'kind' needs to be captions of subtitles. Other kinds like chapters are used differently. The media form track field language selector is based on enabled languages for the Drupal site.

[seth-shaw-asu]

There's more hackiness in that showing or hiding captions triggers when the next cue event happens, that may not coincide with the timing of making the change in the UI. Again, a refactor could fix this kind of thing. But this is better than what we had before...

Amen. If someone with more JS skill wants to come and refactor it, they are welcome to do so. In the mean-time we'll run with what we have.