Virtio 1.0 cs02 doesn't include a modern balloon device. At some point we'll likely define an incompatible interface with a different ID and different semantics. But for now, it's not a big effort to support a transitional balloon device: this has the advantage of supporting existing drivers, transparently, as well as transports that don't allow mixing virtio 0 and virtio 1 devices. And balloon is an easy device to test, so it's also useful for people to test virtio core handling of transitional devices. Three issues with legacy hypervisors have been identified: 1. Actual value is actually used, and is necessary for management
to work. Luckily 4 byte config space writes are now atomic.
When using old guests, hypervisors can detect access to the last byte.
When using old hypervisors, drivers can use atomic 4-byte accesses. 2. Hypervisors actually didn't ignore the stats from the first
buffer supplied. This means the values there would be
incorrect until hypervisor resends the request.
Add a note suggesting hypervisors ignore the 1st buffer. 3. QEMU simply over-writes stats from each buffer it gets.
Thus if driver supplies a different subset of stats
on each request, stale values will be there.
Require drivers to supply the same subset on each
request. This also gives us a simple way to figure out
which stats are supported. Additionally, this fixes a typo: - the driver SHOULD omit unsupported statistics. should be: + devices omit unsupported statistics. Note that we have a normative statement with this requirement already, this is just general high-level description. Signed-off-by: Michael S. Tsirkin <> --- Changes from v2: address comments by Cornelia and Rusty since devices can use actual field for debugging, make the requirement not to use it while driver updates it a SHOULD NOT and not a MUST NOT.
conformance.tex
26 ++++++++- content.tex
173 +++++++++++++++++++++++++++++++++++++++++++++++
---------
2 files changed, 169 insertions(+), 30 deletions(-) diff --git a/conformance.tex b/conformance.tex index 80b333f..ac4feab 100644 --- a/conformance.tex +++ b/conformance.tex @@ -15,13 +15,13 @@ Conformance targets:
egin{itemize}
item Clause
ef{sec:Conformance / Driver Conformance},
item One of clauses
ef{sec:Conformance / Driver Conformance / PCI Driver Conformance},
ef{sec:Conformance / Driver Conformance / MMIO Driver Conformance} or
ef{sec:Conformance / Driver Conformance / Channel I/O Driver Conformance}. -
item One of clauses
ef{sec:Conformance / Driver Conformance / Network Driver Conformance},
ef{sec:Conformance / Driver Conformance / Block Driver Conformance},
ef{sec:Conformance / Driver Conformance / Console Driver Conformance},
ef{sec:Conformance / Driver Conformance / Entropy Driver Conformance} or
ef{sec:Conformance / Driver Conformance / SCSI Host Driver Conformance}. +
item One of clauses
ef{sec:Conformance / Driver Conformance / Network Driver Conformance},
ef{sec:Conformance / Driver Conformance / Block Driver Conformance},
ef{sec:Conformance / Driver Conformance / Console Driver Conformance},
ef{sec:Conformance / Driver Conformance / Entropy Driver Conformance},
ef{sec:Conformance / Driver Conformance / Traditional Memory Balloon Driver Conformance} or
ef{sec:Conformance / Driver Conformance / SCSI Host Driver Conformance}.
end{itemize} item[Device] A device MUST conform to three conformance clauses:
egin{itemize}
item Clause
ef{sec:Conformance / Device Conformance},
item One of clauses
ef{sec:Conformance / Device Conformance / PCI Device Conformance},
ef{sec:Conformance / Device Conformance / MMIO Device Conformance} or
ef{sec:Conformance / Device Conformance / Channel I/O Device Conformance}. -
item One of clauses
ef{sec:Conformance / Device Conformance / Network Device Conformance},
ef{sec:Conformance / Device Conformance / Block Device Conformance},
ef{sec:Conformance / Device Conformance / Console Device Conformance},
ef{sec:Conformance / Device Conformance / Entropy Device Conformance} or
ef{sec:Conformance / Device Conformance / SCSI Host Device Conformance}. +
item One of clauses
ef{sec:Conformance / Device Conformance / Network Device Conformance},
ef{sec:Conformance / Device Conformance / Block Device Conformance},
ef{sec:Conformance / Device Conformance / Console Device Conformance},
ef{sec:Conformance / Device Conformance / Entropy Device Conformance},
ef{{sec:Conformance / Device Conformance / Traditional Memory Balloon Device Conformance} or
ef{sec:Conformance / Device Conformance / SCSI Host Device Conformance}.
end{itemize} end{description}
@@ -123,6 +123,16 @@ An entropy driver MUST conform to the following normative statements: item
ef{drivernormative:Device Types / Entropy Device / Device Operation} end{itemize}
+subsection{Traditional Memory Balloon Driver Conformance}label{sec:Conformance / Driver Conformance / Traditional Memory Balloon Driver Conformance} + +A traditional memory balloon driver MUST conform to the following normative statements: + +egin{itemize} +item
ef{drivernormative:Device Types / Memory Balloon Device / Feature bits} +item
ef{drivernormative:Device Types / Memory Balloon Device / Device Operation} +item
ef{drivernormative:Device Types / Memory Balloon Device / Device Operation / Memory Statistics} +end{itemize} + subsection{SCSI Host Driver Conformance}label{sec:Conformance / Driver Conformance / SCSI Host Driver Conformance}
An SCSI host driver MUST conform to the following normative statements: @@ -230,6 +240,16 @@ An entropy device MUST conform to the following normative statements: item
ef{devicenormative:Device Types / Entropy Device / Device Operation} end{itemize}
+subsection{Traditional Memory Balloon Device Conformance}label{sec:Conformance / Device Conformance / Traditional Memory Balloon Device Conformance} + +A traditional memory balloon device MUST conform to the following normative statements: + +egin{itemize} +item
ef{devicenormative:Device Types / Memory Balloon Device / Feature bits} +item
ef{devicenormative:Device Types / Memory Balloon Device / Device Operation} +item
ef{devicenormative:Device Types / Memory Balloon Device / Device Operation / Memory Statistics} +end{itemize} + subsection{SCSI Host Device Conformance}label{sec:Conformance / Device Conformance / SCSI Host Device Conformance}
An SCSI host device MUST conform to the following normative statements: @@ -292,7 +312,7 @@ Feature Bits / Legacy Interface: A Note on Feature Bits} item Section
ef{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation} item Section
ef{sec:Device Types / Console Device / Device configuration layout / Legacy Interface: Device configuration layout} item Section
ef{sec:Device Types / Console Device / Device Operation / Legacy Interface: Device Operation} -item Section
ef{drivernormative:Device Types / Memory Balloon Device / Device Operation} +item Section
ef{sec:Device Types / Memory Balloon Device / Feature bits / Legacy Interface: Feature bits} item Section
ef{sec:Device Types / Memory Balloon Device / Device Operation / Legacy Interface: Device Operation} item Section
ef{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics / Legacy Interface: Memory Statistics} item Section
ef{sec:Device Types / SCSI Host Device / Device configuration layout / Legacy Interface: Device configuration layout} diff --git a/content.tex b/content.tex index 11015a5..036f38c 100644 --- a/content.tex +++ b/content.tex @@ -1,4 +1,4 @@ -chapter{Basic Facilities of a Virtio Device}label{sec:Basic Facilities of a Virtio Device} +{Basic Facilities of a Virtio Device}label{sec:Basic Facilities of a Virtio Device}
A virtio device is discovered and identified by a bus-specific method (see the bus specific sections:
ef{sec:Virtio Transport Options / Virtio Over PCI Bus}~
ameref{sec:Virtio Transport Options / Virtio Over PCI Bus}, @@ -1046,7 +1046,7 @@ Transitional PCI Device ID & Virtio Device
\ hline 0x1001
&
block device
\ hline -0x1002
& memory ballooning (legacy) \ +0x1002
& memory ballooning (traditional) \ hline 0x1003
&
console
\ hline @@ -2966,7 +2966,7 @@ Device ID & Virtio Device
\ hline 4
& entropy source
\ hline -5
& memory ballooning (legacy) \ +5
& memory ballooning (traditional) \ hline 6
&
ioMemory
\ hline @@ -4357,14 +4357,13 @@ how many random bytes were received. The device MUST place one or more random bytes into the buffer, but it MAY use less than the entire buffer length.
-section{Legacy Interface: Memory Balloon Device}label{sec:Device Types / Memory Balloon Device} +section{Traditional Memory Balloon Device}label{sec:Device Types / Memory Balloon Device}
-This device is deprecated, and thus only exists as a legacy device -illustrated here for reference. The device number 13 is reserved for -a new memory balloon interface which is expected in a future version -of the standard. +This is the traditional balloon device. The device number 13 is +reserved for a new memory balloon interface, with different +semantics, which is expected in a future version of the standard.
-The virtio memory balloon device is a primitive device for +The traditional virtio memory balloon device is a primitive device for managing guest memory: the device asks for a certain amount of memory, and the driver supplies it (or withdraws it, if the device has more than it asks for). This allows the guest to adapt to @@ -4393,6 +4392,25 @@ guest memory statistics to the host.
memory statistics is present. end{description}
+drivernormative{subsubsection}{Feature bits}{Device Types / Memory Balloon Device / Feature bits} +The driver SHOULD accept the VIRTIO_BALLOON_F_MUST_TELL_HOST +feature if offered by the device. + +devicenormative{subsubsection}{Feature bits}{Device Types / Memory Balloon Device / Feature bits} +If the device offers the VIRTIO_BALLOON_F_MUST_TELL_HOST feature +bit, and if the driver did not accept this feature bit, the +device MAY signal failure by failing to set FEATURES_OK +field{device status} bit when the driver writes it. +subparagraph{Legacy Interface: Feature bits}label{sec:Device +Types / Memory Balloon Device / Feature bits / Legacy Interface: +Feature bits} +As the legacy interface does not have a way to gracefully report feature +negotiation failure, when using the legacy interface, +transitional devices MUST support guests which do not negotiate +VIRTIO_BALLOON_F_MUST_TELL_HOST feature, and SHOULD +allow guest to use memory before notifying host if +VIRTIO_BALLOON_F_MUST_TELL_HOST is not negotiated. + subsection{Device configuration layout}label{sec:Device Types / Memory Balloon Device / Device configuration layout}
Both fields of this configuration
are always available. @@ -4404,25 +4422,31 @@ struct virtio_balloon_config { }; end{lstlisting}
-Note that these fields are always little endian, despite convention -that legacy device fields are guest endian. +subparagraph{Legacy Interface: Device configuration layout}label{sec:Device Types / Memory Balloon Device / Device +configuration layout / Legacy Interface: Device configuration layout} +When using the legacy interface, transitional devices and drivers +MUST format the fields in struct virtio_balloon_config +according to the little-endian format. +egin{note} +This is unlike the usual convention that legacy device fields are guest endian. +end{note}
subsection{Device Initialization}label{sec:Device Types / Memory Balloon Device / Device Initialization}
+The device initialization process is outlined below: + egin{enumerate} item The inflate and deflate virtqueues are identified.
item If the VIRTIO_BALLOON_F_STATS_VQ feature bit is negotiated:
egin{enumerate}
item Identify the stats virtqueue. - - item Add one empty buffer to the stats virtqueue and notify the -
device. + item Add one empty buffer to the stats virtqueue. + item DRIVER_OK is set: device operation begins. + item Notify the device about the stats virtqueue buffer.
end{enumerate} end{enumerate}
-Device operation begins immediately. - subsection{Device Operation}label{sec:Device Types / Memory Balloon Device / Device Operation}
The device is driven by the receipt of a @@ -4451,16 +4475,16 @@ configuration change interrupt.
item If the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is negotiated, the
guest informs the device of pages before it uses them.
- item Otherwise, the guest MAY begin to re-use pages previously + item Otherwise, the guest is allowed to re-use pages previously
given to the balloon before the device has acknowledged their
withdrawalfootnote{In this case, deflation advice is merely a courtesy. }.
end{enumerate}
-item In either case, once the device has completed the inflation or - deflation, the driver updates field{actual} to reflect the new number of pages in the balloonfootnote{As updates to device-specific configuration space are not atomic, this field -isn't particularly reliable, but can be used to diagnose buggy guests. -}. +item In either case, the device acknowledges inflate and deflate +requests by using the descriptor. +item Once the device has acknowledged the inflation or + deflation, the driver updates field{actual} to reflect the new number of pages in the balloon. end{enumerate}
drivernormative{subsubsection}{Device Operation}{Device Types / Memory Balloon Device / Device Operation} @@ -4474,12 +4498,38 @@ The driver MUST use the deflateq to inform the device of pages that it wants to use from the balloon.
If the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is negotiated, the -driver MUST wait until the device has used the deflateq descriptor -before using the pages. +driver MUST NOT use pages from the balloon until +the device has acknowledged the deflate request. + +Otherwise, if the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is not +negotiated, the driver MAY begin to re-use pages previously +given to the balloon before the device has acknowledged the +deflate request. + +In any case, the driver MUST NOT use pages from the balloon +after adding the pages to the balloon, but before the device has +acknowledged the inflate request. + +The driver MUST NOT request deflation of pages in +the balloon before the device has acknowledged the inflate +request.
The driver MUST update field{actual} after changing the number of pages in the balloon.
+devicenormative{subsubsection}{Device Operation}{Device Types / Memory Balloon Device / Device Operation} + +The device MAY modify the contents of a page in the balloon +after detecting its physical number in an inflate request +and before acknowledging the inflate request by using the inflateq +descriptor. + +If the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is negotiated, the +device MAY modify the contents of a page in the balloon +after detecting its physical number in an inflate request +and before detecting its physical number in a deflate request +and acknowledging the deflate request. + paragraph{Legacy Interface: Device Operation}label{sec:Device Types / Memory Balloon Device / Device Operation / Legacy Interface: Device Operation} @@ -4488,7 +4538,25 @@ When using the legacy interface, the driver SHOULD ignore the field{len} value Historically, some devices put the total descriptor length there, even though no data was actually written. end{note} - +When using the legacy interface, the driver MUST write out all +4 bytes each time it updates the field{actual} value in the +configuration space, using a single atomic operation. + +When using the legacy interface, the device SHOULD NOT use the +field{actual} value written by the driver in the configuration +space, until the last, most-significant byte of the value has been +written. +egin{note} +Historically, devices used the field{actual} value, even though +when using Virtio Over PCI Bus the device-specific configuration +space was not guaranteed to be atomic. Using intermediate +values during update by driver is best avoided, except for +debugging. + +Historically, drivers using Virtio Over PCI Bus wrote the +field{actual} value by using multiple single-byte writes in +order, from the least-significant to the most-significant value. +end{note} subsubsection{Memory Statistics}label{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics}
The stats virtqueue is atypical because communication is driven @@ -4513,10 +4581,11 @@ as follows:
subsequent request) and consumes the statistics. end{enumerate}
+ Within the buffer, statistics are an array of 6-byte entries.
Each statistic consists of a 16 bit
tag and a 64 bit value. All statistics are optional and the
driver chooses which ones to supply. To guarantee backwards - compatibility, the driver SHOULD omit unsupported statistics. + compatibility, devices omit unsupported statistics.
egin{lstlisting} struct virtio_balloon_stat { @@ -4526,17 +4595,67 @@ struct virtio_balloon_stat { #define VIRTIO_BALLOON_S_MINFLT
3 #define VIRTIO_BALLOON_S_MEMFREE 4 #define VIRTIO_BALLOON_S_MEMTOT
5 -
u16 tag; -
u64 val; +
le16 tag; +
le64 val; } __attribute__((packed)); end{lstlisting}
+drivernormative{paragraph}{Memory Statistics}{Device Types / Memory Balloon Device / Device Operation / Memory Statistics} +Normative statements in this section apply if and only if the +VIRTIO_BALLOON_F_STATS_VQ feature has been negotiated. + +The driver MUST make at most one buffer available to the device +in the statsq, at all times. + +After initializing the device, the driver MUST make an output +buffer available in the statsq. + +Upon detecting that device has used a buffer in the statsq, the +driver MUST make an output buffer available in the statsq. + +Before making an output buffer available in the statsq, the +driver MUST initialize it, including one struct +virtio_balloon_stat entry for each statistic that it supports. + +Driver MUST use an output buffer size which is a multiple of 6 +bytes for all buffers submitted to the statsq. + +Driver MAY supply struct virtio_balloon_stat entries in the +output buffer submitted to the statsq in any order, without +regard to field{tag} values. + +Driver MAY supply a subset of all statistics in the output buffer +submitted to the statsq. + +Driver MUST supply the same subset of statistics in all buffers +submitted to the statsq. + +devicenormative{paragraph}{Memory Statistics}{Device Types / Memory Balloon Device / Device Operation / Memory Statistics} +Normative statements in this section apply if and only if the +VIRTIO_BALLOON_F_STATS_VQ feature has been negotiated. + +Within an output buffer submitted to the statsq, +the device MUST ignore entries with field{tag} values that it does not recognize. + +Within an output buffer submitted to the statsq, +the device MUST accept struct virtio_balloon_stat entries in any +order without regard to field{tag} values. + paragraph{Legacy Interface: Memory Statistics}label{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics / Legacy Interface: Memory Statistics} + When using the legacy interface, transitional devices and drivers MUST format the fields in struct virtio_balloon_stat according to the native endian of the guest rather than (necessarily when not using the legacy interface) little-endian.
+When using the legacy interface, +the device SHOULD ignore all values in the first buffer in the +statsq supplied by the driver after device initialization. +egin{note} +Historically, drivers supplied an uninitialized buffer in the +first buffer. +end{note} + subsubsection{Memory Statistics Tags}label{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics Tags}
egin{description}
--
MST
