Media Overview

The following is a brief overview of support for streaming media in Platform for Situated Intelligence.

Basic Components

There are three main components exposed by the \psi Media Library. All of these components are part of the Microsoft.Psi.Media namespace.

• MediaCapture: Enables capturing of video from a camera.

  • Note for Windows: Only Media Foundation devices are supported (physical and virtual cameras). DirectShow is not supported.
  • Note for Linux: The webcam must be accessible via a virtual device node such as /dev/video0. Audio is not emitted from this component, but an AudioCapture component can be used for that purpose.

• MediaSource: Eables playback of video from an external file/URL.

  • Note: Only available on Windows.

• Mpeg4Writer: Enables writing of \psi Images to an external MPEG4 file.

  • Note: Only available on Windows.

Common Patterns of Usage

Capturing video from a web camera.

The following code snippet demonstrates how to capture audio and video from a web camera. The video is then converted into a stream of JPG images.

Note: Currently, the MediaCapture component requires you to specify a image resolution and frame rate that the hardware supports. If you specify an unsupported resolution or frame rate \psi will throw an ArgumentException error.

using (var pipeline = Pipeline.Create())
{
    var webcam = new Microsoft.Psi.Media.MediaCapture(pipeline, 1920, 1080, 30);
    var encodedImages = webcam.Out.EncodeJpeg(90, Microsoft.Psi.DeliveryPolicy.LatestMessage);
    encodedImages.Out.Do(
	(img, e) =>
	{
		// Do something with the JPG image
	});
    var audioConfig = new Microsoft.Psi.Audio.AudioCaptureConfiguration()
	{
            OutputFormat = Microsoft.Psi.Audio.WaveFormat.Create16kHz1Channel16BitPcm()
	});
    var audioInput = new Microsoft.Psi.Audio.AudioCapture(pipeline, audioConfig);
    audioInput.Out.Do(
	(audioBuffer, e) =>
	{
		// Do something with the audio buffer
	});
    pipeline.Run();
}

Playing video from a .mp4 file

This next snippet of code demonstrates how to instatiate a MediaSource component to use for playing back an MPEG file.

using (var pipeline = Pipeline.Create())
{
    var player = new Microsoft.Psi.Media.MediaSource(pipeline, "test.mp4");
    player.Image.Do(
	(image, e) =>
	{
            // Do something with the video frame
	});
    var convertedAudio = player.Audio.Resample(WaveFormat.Create16kHz1Channel16BitPcm());
    convertedAudio.Do(
	(audio, e) =>
	{
            // Do something with the audio block
	});
    pipeline.Run();
}

Writing images to an .mp4 file

This next snippet of code demonstrates how to instatiate a Mpeg4Writer component to generate a .mp4 file from a \Psi pipeline. We read images from the webcam and write them out to output.mp4.

using (var pipeline = Pipeline.Create())
{
    var webcam = new MediaCapture(pipeline, 1920, 1080, 30.0);

    var audioConfig = new Microsoft.Psi.Audio.AudioCaptureConfiguration();
    audioConfig.Format = WaveFormat.Create16BitPcm(48000, 2);

    var audioCapture = new Microsoft.Psi.Audio.AudioCapture(pipeline, audioConfig);

    var writer = new Mpeg4Writer(pipeline, "output.mp4", 1920, 1080, Microsoft.Psi.Imaging.PixelFormat.BGR_24bpp);
    audioCapture.Out.PipeTo(writer.AudioIn);
    webcam.Out.PipeTo(writer.ImageIn);
    pipeline.RunAsync();
    pipeline.WaitAll(TimeSpan.FromSeconds(30));
}