Merging audio blob with video blob (not an issue, just a question!)

[Faksprod]

Hello @Vanilagy and thanks for your work 🙏
I tried to achieve what I need to do with your lib but I think I'm missing something.

I'm trying to merge an audio blob with a video blob to get a final MP4 file.

• I have the audio blob (which is a dynamic merge of severals mp3 file).
• I have the video blob (which is a blob got from a canvas animation with MediaRecorder).

Does your lib can achieve that or did I misunderstood your examples?

I read your docs, tried a lot of mechanisms, but I can't figured out how I should do for this specific case.
Could you please give me some advices so we could add it to your demo folder (it could be helpfull for other devs)?

I did my test by including your lib directly in the HTML and by loading a sample.mp3 and a sample.webm

<script src="build/mp4-muxer.js"></script>

window.onload = go;

const audioUrl = 'sample.mp3';
const videoUrl = 'sample.webm';

async function loadFileAsBlob(url) {
    const response = await fetch(url);
    const arrayBuffer = await response.arrayBuffer();
    return new Blob([arrayBuffer]);
}

async function go() {
    try {
        const audioBlob = await loadFileAsBlob(audioUrl);
        const videoBlob = await loadFileAsBlob(videoUrl);

        await mergeAudioVideo(audioBlob, videoBlob);// <-- what mechanism should have this function?

    } catch (error) {
        console.error('Error loading files:', error);
    }
}

Thanks for your help and time!

[Vanilagy]

Hi @Faksprod. Doing this is non-trivial actually. My library only takes care of muxing, not demuxing which is required for reading media files. That means you'll need to combine several web APIs to make this work. You'll also need to reencode the media chunks.

For the audio, your best bet is to decode it using an AudioContext and then to create an AudioData from it, which you can pass into an AudioDecoder.

For the video it's more tricky as there's no neat API for extracting frames from a video. There are a few possibilities here but all are hacky and suboptimal or slow. Your best bet would be not to use MediaRecorder but directly pipe the canvas into a VideoFrame, and then into a VideoEncoder, and then into the muxer. Is that possible in your case?