Implement VirtIO sound device

[Cuda-Chen]

Currently, semu lacks of sound playing feature.

To implement, we can use VirtIO sound with ALSA architecture.

[jserv]

we can use VirtIO sound with ALSA architecture.

Can you illustrate the progress and the potential integration considerations?

[Cuda-Chen]

Hi @jserv ,

For the progress:

• The system can boot up and the ALSA driver will be registered (using make check but not sudo make check as it will complains incorrect parameter related errors).
• The host system can see the ALSA driver in system settings (named ALSA plug-in [semu] in Volume Levels).

For potential integration considerations:

• We should install the ALSA utilities such as alsa-utils for testing the sound device.
• TinyALSA can be used if semu is aiming for a lightweight emulator.

[Cuda-Chen]

Hi @jserv ,

For the supporting operations mentioned in #53, to let semu plays sound I consider it requires to support more operations (and the operations are mentioned in VirtIO official document), should we investigate then list the operations that have to be implemented to support the common sound operation (e.g., playing sound, querying sound device information, etc.)?

[jserv]

should we investigate then list the operations that have to be implemented to support the common sound operation (e.g., playing sound, querying sound device information, etc.)?

Yes, go ahead.

[Cuda-Chen]

For this issue, I am going to implement VirtIO sound device supporting these operations:

• querying the device information
  • VIRTIO_SND_R_PCM_INFO
  • VIRTIO_SND_R_CHMAP_INFO
  • VIRTIO_SND_R_JACK_INFO
• playing sound (PCM, specifically)
  • VIRTIO_SND_R_PCM_SET_PARAMS
  • VIRTIO_SND_R_PCM_PREPARE
  • VIRTIO_SND_R_PCM_RELEASE
  • VIRTIO_SND_R_PCM_START
  • VIRTIO_SND_R_PCM_STOP

[Cuda-Chen]

Update:

• Sound Card
  • After investigation, the VirtIO SoundCard at platform/f4400000.virtio/virtio2 is actually attached to the kernel.
  • However, I can only see controlCX (X stands for the card number of VirtIO SoundCard) in /dev/snd/.
  • Compared to LoopBack, which creates the following files in /dev/snd/:
    • controlCX
    • pcmCXDYc
    • pcmCXDYp
• PCM
  • As the VirtIO SoundCard does not expose any endpoint for PCM playback, there is no way to play sound via VirtIO SoundCard.

I am going to solve the sound card endpoint issue first.

[Cuda-Chen]

Update:

• Sound Card
  • The VirtIO Sound Card can be brought up after initialization.
  • I can see the pcmCXDYc appears in /dev/snd/, and aplay -l can list the VirtIO Sound Card.
• PCM
  • speaker-test and aplay exit with non-zero return value because an unknown type (0x0) is sent from driver to virtio-snd after pcm_prepare state.

[Cuda-Chen]

Statue update:

• PCM
  • After investigation, semu VirtIO device receive the data from TX queue after VIRTIO_SND_R_PCM_PREPARE state.
  • I am going to implement to receive the PCM frames from TX queue to buffer so that we can play the sound.

[jserv]

• I am going to implement to receive the PCM frames from TX queue to buffer so that we can play the sound.

Do you think whether if single-threaded queue manipulation is enough. I am not sure that such TX queue can be operated without extra threads.

[Cuda-Chen]

Do you think whether if single-threaded queue manipulation is enough. I am not sure that such TX queue can be operated without extra threads.

For my current findings, qemu and rust-vmm do not use any extra threads to operate TX queue.
However, we may consider using extra threads as it seems we have to notify the device to complete transmission once it gets PCM frame from TX queue.

[Cuda-Chen]

Statue update:

• PCM
  • Findings: the driver sends arbitrary number of PCM frames when transferring data.
  • I will implement a data structure which holds these arbitrary number of frames (e.g., queue or scatter list).

[Cuda-Chen]

Statue update:

• PCM
  • The queue holding arbitrary number of PCM frames is implemented.
  • The driver (guest Linux OS) still sends the PCM frames after the device receives the PCM frames. It seems that we have to send some signal to inform the driver that the device has received the PCM frames and it should send pcm_start to play the sound.

[Cuda-Chen]

Update:

• PCM
  • Create a dedicated thread to handle TX.
  • The pcm_release signal is sent asynchronously.

Actions:

• PCM
  • Create a lock in pcm_start/pcm_stop state.
  • Check the possible implementation to handle pcm_release state.

[jserv]

• PCM
  • Create a lock in pcm_start/pcm_stop state.
  • Check the possible implementation to handle pcm_release state.

@idoleat, can you comment the recent work of #53 ?

[Cuda-Chen]

• PCM

  • Create a lock in pcm_start/pcm_stop state.
  • Check the possible implementation to handle pcm_release state.

@idoleat, can you comment the recent work of #53 ?

Hi @jserv ,

I thought you may tag the wrong person.
So should I conclude the recent work of #53?

[jserv]

I thought you may tag the wrong person.

No, I was intentionally referring to him.

should I conclude the recent work of #53?

Go ahead.

[Cuda-Chen]

should I conclude the recent work of #53?

Go ahead.

Hi @jserv , I am going to summarize the recent work in this comment.

Past

• Initialize VirtIO sound device.
• Implement a thread for receiving PCM frames sent from driver.

Ongoing

• Implement pcm_release state.
• Implement a ring buffer to store the PCM frames for playback.

[idoleat]

For my clarity, I would like to confirm the workflow works as:

• The driver (guest Linux OS) prepares PCM frame data in memory as a descriptor
• The driver (guest Linux OS) signals the TX thread to take the descriptor and tell audio server (PulseAudio/Pipewire) where the PCM frame data is as a audio client
• TX thread interrupts the driver (guest Linux OS) that it has instructed audio server to play the PCM frame data so it can be cleaned up/overwritten in memory

I have some concern on the third step since I see TX thread send interrupt right after finishing step 2 (correct me if I'm wrong). Typically, sound card sends interrupt every period and ALSA driver polls that interrupt to update hardware pointer for the buffer. If the interrupt is sent right after step 2, audio server may have nothing to play because the driver (guest Linux OS) thinks it has already been played. The driver may think audio under-run is happening and may address it.

From my understanding, the length of period normally depends on the capability of sound card hardware and buffer size. So we may use the length that reflects the real hardware. Or we may adjust it for other reasons like batch/fine-grained submission, as long as it won't cause under/over-run or long latency. For sending interrupt every period we may use SIGALRM? It could be too tedious to hook into real hardware interrupts. I've read comments above saying that the driver may prepare arbitrary size of PCM frame data. So to introduce period, we may need to tidy PCM frame data as chunks per period.

Edit: fix typo

[Cuda-Chen]

Hi @idoleat , I would like to make some statements of the workflows you are concerning for, all based on my understanding of VirtIO standard. So if you think there are still some rooms of uncertainty, just let me know. I will make my observations for interacting with VirtIO standard and VirtIO sound driver (namely, guest Linux OS with Linux Kernel version 6.1) in each list and some statements you have mentioned.

Before we begin, let's look about the Virtqueues that a VirtIO sound device will use for communicating with VirtIO sound driver (from Virtqueue section in sound device):

1. controlq: sending control messages from driver to device.
2. eventq: sending notifications from device to driver.
3. txq: sending PCM frames for output streams (i.e., playback).
4. rxq: sending PCM frames for input streams (i.e., capture).

Then, let me reply the list you have mentioned.

• The driver (guest Linux OS) prepares PCM frame data in memory as a descriptor

By the VirtIO standard, the driver prepares PCM frames in memory, then sends these PCM frames via txq to the device.

• The driver (guest Linux OS) signals the TX thread to take the descriptor and tell audio server (PulseAudio/Pipewire) where the PCM frame data is as a audio client

This is not mentioned in the VirtIO standard. For my implementation, as we are using MMIO, I use a TX thread to receive PCM frames sent from the driver so that the main thread of device won't just hang for merely receiving PCM frames.
For my finding, other publicly available implementations such as qemu and rust-vmm do not use a TX thread for receiving PCM frames, and I guess the reason is that they use PCI interrupt mechanism.
What's more, as we can select the PCM features according to 5.14.6.6.2 VIRTIO_SND_R_PCM_INFO, though it seems I receive PCM frames with values within the range of short datatype, I guess the reason why the host playback plays nothing is because it maybe just send the memory area (using VIRTIO_SND_PCM_F_SHMEM_*).

• TX thread interrupts the driver (guest Linux OS) that it has instructed audio server to play the PCM frame data so it can be cleaned up/overwritten in memory

For my understanding, there are two ways to interrupt the driver:

1. PCM I/O messages: in 5.14.6.8 PCM I/O Messages, the standard mentions: the completion of such an I/O request can be considered an elapsed period notification.
2. PCM notifications: using eventq, as mentioned in 5.14.6.7 PCM Notifications.

I've read comments above saying that the driver may prepare arbitrary size of PCM frame data.

Sorry for my dumb writings, it should be as follows: the driver sends arbitrary number of virtqueue descriptors (for instance of receiving PCM frames from the driver, it may send the virtqueue descriptors like one request, then follow two payloads consisting of PCM frames--the summation size of these payloads will be the size of buffer_bytes set by VIRTIO_SND_R_PCM_SET_PARAMS--and at last, one response.