admin-cmds-device-parts.tex

\subsubsection{Device parts}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts}

In some systems, there is a need to capture the state of all or part of
a device and subsequently restore either the same device or a
different one to this captured state. A group owner device can support
administration commands to facilitate these get and set operations for
the group member devices.

For example, a hypervisor can use the administration commands to
capture parts of the device state and save the result as part of
a VM snapshot. Later, the hypervisor can retrieve the snapshot and use the
administration commands to restore parts of a device to resume
VM operation.

As another example, these commands can be used to facilitate VM migration by the
hypervisor: one (source) hypervisor can get parts of a device and send
the results to another (destination) hypervisor, which will in turn
set (restore) parts of (another) device to resume the VM operation on
the destination.

The device comprises many device parts which the driver can get and set.
Administration commands are provided to either get and set all the device
parts at once, or to get the device parts metadata that indicates which
device parts are present, and later to get and set specific device parts.
To get and set the device parts or their metadata, the driver first creates a
device parts resource object, indicating whether the object should
handle get or set operations but not both simultaneously. The device and the
driver indicate the device parts resource objects' limit using the capability
VIRTIO_DEV_PARTS_CAP.

The device can be stopped to prevent device parts from changing.
When the device is stopped, it does not initiate any transport requests.
For instance, the device refrains from sending any configuration or
virtqueue notifications and does not access any virtqueues or the driver's
buffer memory. While the driver may remain active and continue to send
notifications to the device, potentially updating some device parts,
the device itself will not initiate any transport requests.

\paragraph{VIRTIO_DEV_PARTS_CAP}
\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / VIRTIO-DEV-PARTS-CAP}

The capability VIRTIO_DEV_PARTS_CAP indicates the device parts resource objects limit.
\field{cap_specific_data} is in the format \field{struct virtio_dev_parts_cap}.

\begin{lstlisting}
struct virtio_dev_parts_cap {
        u8 get_parts_resource_objects_limit;
        u8 set_parts_resource_objects_limit;
};
\end{lstlisting}

\field{get_parts_resource_objects_limit} indicates the supported device parts
resource objects for retrieving the device parts.
\field{set_parts_resource_objects_limit} indicates the supported device parts
resource objects for restoring the device parts.

\paragraph{VIRTIO_RESOURCE_OBJ_DEV_PARTS}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / VIRTIO-RESOURCE-OBJ-DEV-PARTS}

A device parts resource object is used to either get or set the device parts.
Before performing any get or set operation for the device parts, the driver
creates the device parts resource object
VIRTIO_RESOURCE_OBJ_DEV_PARTS using the administration command
\nameref{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device resource objects / VIRTIO-ADMIN-CMD-RESOURCE-OBJ-CREATE}.
The driver indicates the intended purpose (get or set) at the time of creating the
device parts resource object.
For the device parts resource object, both \field{resource_obj_specific_data} and
\field{resource_obj_specific_result} are in the format
\field{struct virtio_resource_obj_dev_parts}.

\begin{lstlisting}
struct virtio_resource_obj_dev_parts {
        u8 type;
#define VIRTIO_RESOURCE_OBJ_DEV_PARTS_TYPE_GET 0
#define VIRTIO_RESOURCE_OBJ_DEV_PARTS_TYPE_SET 1
        u8 reserved[7];
};
\end{lstlisting}

When \field{type} is set to VIRTIO_RESOURCE_OBJ_DEV_PARTS_TYPE_GET,
the driver can use the object to capture the device parts and the metadata of
these device parts. When \field{type} is set to
VIRTIO_RESOURCE_OBJ_DEV_PARTS_TYPE_SET, the driver can use the
object to restore the device parts.

\paragraph{Device parts handling commands}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands}

The owner driver uses the following resource object handling administration
commands. These commands are only used for the device parts resource
object after the driver creates the VIRTIO_RESOURCE_OBJ_DEV_PARTS object.
These commands are currently only defined for the SR-IOV group type:

\begin{enumerate}
\item VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET
\item VIRTIO_ADMIN_CMD_DEV_PARTS_GET
\item VIRTIO_ADMIN_CMD_DEV_PARTS_SET
\end{enumerate}

\subparagraph{VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET}
\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands / VIRTIO-ADMIN-CMD-DEV-PARTS-METADATA-GET}

This command obtains the metadata of the device parts. This metadata includes
the maximum size of the device parts, the count of device parts, and a list of
the device part headers.

For the command VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET, \field{opcode} is set
to 0xe. The \field{command_specific_data} is in the format
\field{struct virtio_admin_cmd_dev_parts_metadata_data}.
\field{group_member_id} refers to the member device to be accessed.
The resource object \field{type} in the \field{hdr} is set to
VIRTIO_RESOURCE_OBJ_DEV_PARTS and \field{id} is set to the ID of the
device parts resource object.

\begin{lstlisting}
struct virtio_admin_cmd_dev_parts_metadata_data {
        struct virtio_admin_cmd_resource_obj_cmd_hdr hdr;
        u8 type;
        u8 reserved[7];
};

#define VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_SIZE 0
#define VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_COUNT 1
#define VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_LIST  2

struct virtio_admin_cmd_dev_parts_metadata_result {
        union {
                struct {
                        le32 size;
                        le32 reserved;
                } parts_size;
                struct {
                        le32 count;
                        le32 reserved;
                } hdr_list_count;
                struct {
                        le32 count;
                        le32 reserved;
                        struct virtio_dev_part_hdr hdrs[];
                } hdr_list;
        };
};
\end{lstlisting}

When the command completes successfully, the
\field{command_specific_result} is in the format
\field{struct virtio_admin_cmd_dev_parts_metadata_result}.

When \field{type} is set to VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_SIZE,
the device responds with \field{parts_size}. \field{parts_size.size} indicates
the maximum size in bytes for all the device parts.

When \field{type} is set to VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_COUNT, the
device responds with \field{hdr_list_count.count}. The
\field{hdr_list_count.count} indicates an count of
\field{struct virtio_dev_part_hdr} metadata entries that the device can
provide when the \field{type} is set to VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_LIST
in a subsequent VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET command.

When \field{type} is set to VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_LIST,
the device responds with \field{hdr_list}. \field{hdr_list}
indicates the device parts metadata.

\field{reserved} is reserved and set to 0.

The command responds with the \field{status} VIRTIO_ADMIN_STATUS_ENOMEM
when the size of \field{command_specific_result} is not sufficient enough
for the response.

\subparagraph{VIRTIO_ADMIN_CMD_DEV_PARTS_GET}
\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts /  Device parts handling commands / VIRTIO-ADMIN-CMD-DEV-PARTS-GET}

This command captures the device parts. For the command
VIRTIO_ADMIN_CMD_DEV_PARTS_GET, \field{opcode} is set to 0xf.
The \field{command_specific_data} is in the format
\field{struct virtio_admin_cmd_dev_parts_get_data}.
\field{group_member_id} refers to the member device to be accessed.
The resource object \field{type} in the \field{hdr} is set to
VIRTIO_RESOURCE_OBJ_DEV_PARTS and \field{id} is set to the ID of the
device parts resource object.

\begin{lstlisting}
struct virtio_admin_cmd_dev_parts_get_data {
        struct virtio_admin_cmd_resource_obj_cmd_hdr hdr;
        u8 type;
        u8 reserved[7];
        struct virtio_dev_part_hdr hdr_list[];
};

#define VIRTIO_ADMIN_CMD_DEV_PARTS_GET_TYPE_SELECTED 0
#define VIRTIO_ADMIN_CMD_DEV_PARTS_GET_TYPE_ALL 1

struct virtio_admin_cmd_dev_parts_get_result {
        struct virtio_dev_part parts[];
};

\end{lstlisting}

When the driver wants to capture specific device parts, \field{type} is set to
VIRTIO_ADMIN_CMD_DEV_PARTS_GET_TYPE_SELECTED and \field{hdr_list} is set to the
device parts of interest.

When the driver wants to retrieve all the device parts, \field{type} is set to
VIRTIO_ADMIN_CMD_DEV_PARTS_GET_TYPE_ALL, and \field{hdr_list} is empty.

\field{reserved} is reserved and set to 0.

When the command completes successfully, the \field{command_specific_result} is
in the format \field{struct virtio_admin_cmd_dev_parts_get_result}, containing
either the selected device parts or all the device parts.

If the requested device part does not exist, the device skips the device part
without any error.

\subparagraph{VIRTIO_ADMIN_CMD_DEV_PARTS_SET}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands / VIRTIO-ADMIN-CMD-DEV-PARTS-SET}

This command sets one or multiple device parts. For the command
VIRTIO_ADMIN_CMD_DEV_PARTS_SET, \field{opcode} is set to 0x10.
The \field{group_member_id} refers to the member device to be accessed.
The resource object \field{type} in the \field{hdr} is set to
VIRTIO_RESOURCE_OBJ_DEV_PARTS and \field{id} is set to the ID of the
device parts resource object.

\begin{lstlisting}
struct virtio_admin_cmd_dev_parts_set_data {
        struct virtio_admin_cmd_resource_obj_cmd_hdr hdr;
        struct virtio_dev_part parts[];
};
\end{lstlisting}

The \field{command_specific_data} is in the format
\field{struct virtio_admin_cmd_dev_parts_set_data}.

This command has no command specific result.

The driver stops the device before setting any device parts.

When the command completes successfully, the device has updated device
parts to the value supplied in \field{virtio_admin_cmd_dev_parts_set_data}.

The device parts set by this command take effect when the device is resumed
using the VIRTIO_ADMIN_CMD_DEV_MODE_SET command.

When the command fails with a status other than VIRTIO_ADMIN_STATUS_OK, the
device does not have any side effects.

\subparagraph{VIRTIO_ADMIN_CMD_DEV_MODE_SET}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands / VIRTIO-ADMIN-CMD-DEV-MODE-SET}

This command either stops the device from initiating any transport requests or
resumes the device operation. For the command VIRTIO_ADMIN_CMD_DEV_MODE_SET,
\field{opcode} is set to 0x11. \field{group_member_id} indicates the member
device to be accessed.

The \field{command_specific_data} is in the format
\field{struct virtio_admin_cmd_dev_mode_set_data}.

\begin{lstlisting}
struct virtio_admin_cmd_dev_mode_set_data {
        u8 flags;
};

#define VIRTIO_ADMIN_CMD_DEV_MODE_F_STOPPED 0
\end{lstlisting}

This command has no command specific result.

When the command completes successfully and if the \field{flags} field is set
to VIRTIO_ADMIN_CMD_DEV_MODE_F_STOPPED (bit 0), the device is stopped.
When the device is stopped, the device stops initiating all transport
communications, which includes:

\begin{enumerate}
\item stopping configuration change notifications
\item stopping all virtqueue notifications
\item stops accessing all virtqueues and the driver buffer memory
\end{enumerate}

After the device is stopped, the device parts remain unchanged unless
the driver initiates any transport requests.

When the device is stopped, it writes back any associated descriptors for all
observed buffers to prevent out-of-order processing if the device is resumed.

When the command completes successfully and if the \field{flags} field
is set to zero, the device resumes its operation. If the command completes
with an error, it does not produce any side effects on the device.

\paragraph{Device parts order}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts order}

Device parts are usually captured and restored using get and set administration
commands respectively; when multiple device parts are captured or restored,
they are arranged in the specific order listed:

Some of the device parts do not need to be written to the device when restored, such
device parts are listed as \field{O}. When a such an optional device part is
exchanged using \field{struct virtio_dev_part}, it is marked as optional by
setting VIRTIO_DEV_PART_F_OPTIONAL(bit 0) in the \field{flags}.

\begin{table}[H]
\caption{Device parts order}
\label{table:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts order/ Device parts order}
\begin{tabularx}{\textwidth}{ |l|l|X| }
\hline
Part name & Optional & Mandatory preceding parts \\
\hline \hline
\hline
VIRTIO_DEV_PART_DEV_FEATURES & O & Nil \\
\hline
VIRTIO_DEV_PART_DRV_FEATURES & - & Nil \\
\hline
VIRTIO_DEV_PART_PCI_COMMON_CFG & - & VIRTIO_DEV_PART_DEV_FEATURES, VIRTIO_DEV_PART_DRV_FEATURES \\
\hline
VIRTIO_DEV_PART_DEVICE_STATUS & - & VIRTIO_DEV_PART_DEV_FEATURES, VIRTIO_DEV_PART_DRV_FEATURES, VIRTIO_DEV_PART_PCI_COMMON_CFG \\
\hline
VIRTIO_DEV_PART_VQ_CFG & - & VIRTIO_DEV_PART_DEV_FEATURES, VIRTIO_DEV_PART_DRV_FEATURES, VIRTIO_DEV_PART_PCI_COMMON_CFG,
                             VIRTIO_DEV_PART_DEVICE_STATUS \\
\hline
VIRTIO_DEV_PART_VQ_NOTIFY_CFG & - & VIRTIO_DEV_PART_DEV_FEATURES, VIRTIO_DEV_PART_DRV_FEATURES, VIRTIO_DEV_PART_PCI_COMMON_CFG,
                             VIRTIO_DEV_PART_DEVICE_STATUS, VIRTIO_DEV_PART_VQ_CFG \\
\hline
\hline
\end{tabularx}
\end{table}

\devicenormative{\subparagraph}{Device parts}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts}

A device MUST either support all of, or none of
VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET,
VIRTIO_ADMIN_CMD_DEV_PARTS_GET, VIRTIO_ADMIN_CMD_DEV_PARTS_SET,
VIRTIO_ADMIN_CMD_RESOURCE_OBJ_CREATE,
VIRTIO_ADMIN_CMD_RESOURCE_OBJ_DESTROY, VIRTIO_ADMIN_CMD_RESOURCE_OBJ_MODIFY
VIRTIO_ADMIN_CMD_RESOURCE_OBJ_QUERY, and
VIRTIO_ADMIN_CMD_DEV_MODE_SET commands, where resource commands apply to
the resource object VIRTIO_RESOURCE_OBJ_DEV_PARTS.

The device MUST support getting the device parts multiple times
with the command VIRTIO_ADMIN_CMD_DEV_PARTS_GET.

When there are multiple device parts in the command
VIRTIO_ADMIN_CMD_DEV_PARTS_GET, the device MUST respond the device parts in the
same order as listed in the table
\nameref{table:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts order/ Device parts order}.

The device SHOULD respond with an error status for the command
VIRTIO_ADMIN_CMD_DEV_PARTS_SET if the device is not stopped.

The device MUST support the command VIRTIO_ADMIN_CMD_DEV_PARTS_SET,
allowing the same or different device parts to be set multiple times.

The device MUST respond with an error for the command
VIRTIO_ADMIN_CMD_DEV_PARTS_SET, if there is a mismatch between the
device part length supplied in the VIRTIO_ADMIN_CMD_DEV_PARTS_SET
and the device part length in the device.

The device MUST NOT set the device part VIRTIO_DEV_PART_DEV_FEATURES in
the command VIRTIO_ADMIN_CMD_DEV_PARTS_SET; instead,
it must verify that the device features supplied in
VIRTIO_DEV_PART_DEV_FEATURES match those the device has.

The device may ignore the setting of a device part that has the
VIRTIO_DEV_PART_F_OPTIONAL bit set.

For the SR-IOV group type, when the device is stopped using the command
VIRTIO_ADMIN_CMD_DEV_MODE_SET,
\begin{itemize}
\item the device MUST not initiate any PCI transaction,
\item the device MUST finish all the outstanding PCI transactions before completing
      the command VIRTIO_ADMIN_CMD_DEV_MODE_SET,
\item the device MUST write any associated descriptors to the driver memory for
      all the observed buffers,
\item the device MUST accept driver notifications and the device MAY update any
      device parts,
\item the device MUST respond with valid values for PCI read requests,
\item the device MUST operate in the same way for the PCI architected interfaces
      regardless of the device mode.
\item the device MUST not generate any PCI PME.
\end{itemize}

When the device is stopped,
\begin{itemize}
\item the device MUST not access any virtqueue memory or any memory referred
      by the virtqueue.
\item the device MUST not generate any configuration change notification
      or any virtqueue notification.
\end{itemize}

For the SR-IOV group type,
\begin{itemize}
\item the device MUST respond to the commands
VIRTIO_ADMIN_CMD_DEV_MODE_SET, VIRTIO_ADMIN_CMD_DEV_PARTS_SET
after the member device completes FLR, if the FLR is in progress on the device
when the device receives any of these commands.

\item the member device MUST respond to the commands
VIRTIO_ADMIN_CMD_DEV_MODE_SET and VIRTIO_ADMIN_CMD_DEV_PARTS_SET
after the device reset completes in the device, if the
device reset is in progress when the device receives any of these commands.

\item the member device MUST respond to commands
VIRTIO_ADMIN_CMD_DEV_MODE_SET and VIRTIO_ADMIN_CMD_DEV_PARTS_SET
after the device power management state
transition completes on the device, if the power management state transition
is in progress when the device receives any of these commands.
\end{itemize}

When the \field{flags} is set to VIRTIO_ADMIN_CMD_DEV_MODE_FLAGS_STOPPED
in the command VIRTIO_ADMIN_CMD_DEV_MODE_SET, and if the device is already
stopped before, the device MUST complete the command successfully.

When the VIRTIO_ADMIN_CMD_DEV_MODE_FLAGS_STOPPED \field{flags} clear,
in the command VIRTIO_ADMIN_CMD_DEV_MODE_SET, and if the device is
not stopped before, the device MUST complete the command successfully.

For the SR-IOV group type, the device MUST clear all the device parts to
the default value when the member device is reset or undergo an PCI FLR.

The device MAY NOT respond to the selected device part in \field{hdr_list}
in the command VIRTIO_ADMIN_CMD_DEV_PARTS_GET if the device part is invalid
in the device.

For the commands VIRTIO_ADMIN_CMD_DEV_PARTS_GET and
VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET, when the device responds with:
\begin{itemize}
\item
VIRTIO_DEV_PART_DRV_FEATURES or VIRTIO_DEV_PART_PCI_COMMON_CFG, it MUST be
preceded by VIRTIO_DEV_PART_DEV_FEATURES.

\item VIRTIO_DEV_PART_PCI_COMMON_CFG, it MUST be preceded by
VIRTIO_DEV_PART_DEV_FEATURES.

\item VIRTIO_DEV_PART_PCI_COMMON_CFG, it MUST be preceded by
VIRTIO_DEV_PART_DEV_FEATURES and VIRTIO_DEV_PART_DRV_FEATURES.

\item VIRTIO_DEV_PART_DEV_CFG, it MUST be preceded by VIRTIO_DEV_PART_DEV_FEATURES.

\item VIRTIO_DEV_PART_DRV_CFG, it be preceded by VIRTIO_DEV_PART_DEV_FEATURES,
VIRTIO_DEV_PART_DRV_FEATURES and VIRTIO_DEV_PART_DEV_CFG.

\item VIRTIO_DEV_PART_DEVICE_STAtUS, it is preceded by VIRTIO_DEV_PART_DEV_FEATURES,
VIRTIO_DEV_PART_DRV_FEATURES, and VIRTIO_DEV_PART_DEV_CFG.
\end{itemize}

When the device receives a VIRTIO_ADMIN_CMD_DEV_PARTS_SET command containing the
parts VIRTIO_DEV_PART_DEV_FEATURES, VIRTIO_DEV_PART_PCI_COMMON_CFG and
VIRTIO_DEV_PART_DEV_CFG, the device SHOULD only verify that the provided configuration is
correct but SHOULD NOT apply it, especially for the fields that are designated
as read-only and invariant. This ensures that the device respects the
immutability of certain configuration aspects while still performing necessary
validation checks.

\drivernormative{\subparagraph}{Device parts}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts}

The driver MUST set the mode to VIRTIO_ADMIN_CMD_DEV_MODE_F_STOPPED in
the command VIRTIO_ADMIN_CMD_DEV_MODE_SET before setting parts using the command
VIRTIO_ADMIN_CMD_DEV_PARTS_SET.

When there are multiple device parts in the command
VIRTIO_ADMIN_CMD_DEV_PARTS_SET, the driver MUST set the device parts in the same
order as listed in the table
\nameref{table:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts order/ Device parts order}.

For the SR-IOV group type, the driver SHOULD NOT access the device configuration
space described in section
\ref{sec:Basic Facilities of a Virtio Device / Device Configuration Space}
when the device is stopped.

The driver SHOULD allocate sufficient response buffer to receive all the device
parts metadata in the command VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET.

The driver SHOULD allocate sufficient response buffer to receive all the device
parts in the command VIRTIO_ADMIN_CMD_DEV_PARTS_GET.