RE: [EXT] [virtio] [PATCH requirements 2/7] net-features: Add low latency transmit queue requirements

[Satananda Burla <sburla@marvell.com>]

Hi Parav

> -----Original Message-----
> From: virtio@lists.oasis-open.org <virtio@lists.oasis-open.org> On
> Behalf Of Parav Pandit
> Sent: Sunday, July 23, 2023 8:34 PM
> To: virtio-comment@lists.oasis-open.org
> Cc: shahafs@nvidia.com; hengqi@linux.alibaba.com; virtio@lists.oasis-
> open.org; Parav Pandit <parav@nvidia.com>
> Subject: [EXT] [virtio] [PATCH requirements 2/7] net-features: Add low
> latency transmit queue requirements
> 
> External Email
> 
> ----------------------------------------------------------------------
> Add requirements for the low latency transmit queue.
> 
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> ---
> chagelog:
> v1->v2:
> - added generic requirement to inline the request content
>   along with the descriptor for non virtio-net devices
> - added requirement to inline the request content along
>   with the descriptor for virtio flow filter queue as two
>   features are similar
> v0->v1:
> - added design goals for which requirements are added
> ---
>  net-workstream/features-1.4.md | 88 ++++++++++++++++++++++++++++++++++
>  1 file changed, 88 insertions(+)
> 
> diff --git a/net-workstream/features-1.4.md b/net-workstream/features-
> 1.4.md
> index 4c3797b..eb95592 100644
> --- a/net-workstream/features-1.4.md
> +++ b/net-workstream/features-1.4.md
> @@ -7,6 +7,7 @@ together is desired while updating the virtio net
> interface.
> 
>  # 2. Summary
>  1. Device counters visible to the driver
> +2. Low latency tx virtqueue for PCI transport
> 
>  # 3. Requirements
>  ## 3.1 Device counters
> @@ -33,3 +34,90 @@ together is desired while updating the virtio net
> interface.
>  ### 3.1.2 Per transmit queue counters
>  1. le64 tx_gso_pkts: Packets send as transmit GSO sequence
>  2. le64 tx_pkts: Total packets send by the device
> +
> +## 3.2 Low PCI latency virtqueues
> +### 3.2.1 Low PCI latency tx virtqueue
> +0. Design goal
> +   a. Reduce PCI access latency in packet transmit flow
> +   b. Avoid O(N) descriptor parser to detect a packet stream to
> simplify device
> +      logic
> +   c. Reduce number of PCI transmit completion transactions and have
> unified
> +      completion flow with/without transmit timestamping
> +   d. Avoid partial cache line writes on transmit completions
> +
> +1. Packet transmit descriptor should contain data descriptors count
> without any
> +   indirection and without any O(N) search to find the end of a packet
> stream.
> +   For example, a packet transmit descriptor (called vnet_tx_hdr_desc
> +   subsequently) to contain a field num_next_desc for the packet stream
> +   indicating that a packet is located in N data descriptors.
> +
> +2. Packet transmit descriptor should contain segmentation offload-
> related fields
> +   without any indirection. For example, packet transmit descriptor to
> contain
> +   gso_type, gso_size/mss, header length, csum placement byte offset,
> and
> +   csum start.
> +
> +3. Packet transmit descriptor should be able to place a small size
> packet that
> +   does not have any L4 data after the vnet_tx_hdr_desc in the
> virtqueue memory.
> +   For example a TCP ack only packet can fit in a descriptor memory
> which
> +   otherwise consume more than 25% of metadata to describe the packet.
> +
> +4. Packet transmit descriptor should be able to place a full GSO header
> (L2 to
> +   L4) after header descriptor and before data descriptors. For
> example, the
> +   GSO header is placed after struct vnet_tx_hdr_desc in the virtqueue
> memory.
> +   When such a GSO header is positioned adjacent to the packet transmit
> +   descriptor, and when the GSO header is not aligned to 16B, the
> following
> +   data descriptor to start on the 8B aligned boundary.
> +
> +5. An example of the above requirements at high level is:
> +
> +```
> +struct vitio_packed_q_desc {
> +   /* current desc for reference */
> +   u64 address;
> +   u32 len;
> +   u16 id;
> +   u16 flags;
> +};
> +
> +/* Constant size header descriptor for tx packets */
> +struct vnet_tx_hdr_desc {
> +   u16 flags; /* indicate how to parse next fields */
> +   u16 id; /* desc id to come back in completion */
> +   u8 num_next_desc; /* indicates the number of the next 16B data desc
> for this
> +		      * buffer.
> +		      */
> +   u8 gso_type;
> +   le16 gso_hdr_len;
> +   le16 gso_size;
> +   le16 csum_start;
> +   le16 csum_offset;
> +   u8 inline_pkt_len; /* indicates the length of the inline packet
> after this
> +		       * desc
> +		       */
> +   u8 reserved;
> +   u8 padding[];
> +};
> +
> +/* Example of a short packet or GSO header placed in the desc section
> of the vq
> + */
> +struct vnet_tx_small_pkt_desc {
> +   u8 raw_pkt[128];
> +};
> +
> +/* Example of header followed by data descriptor */
> +struct vnet_tx_hdr_desc hdr_desc;
> +struct vnet_data_desc desc[2];
> +
> +```
> +
> +6. Ability to zero pad the transmit completion when the transmit
> completion is
> +   shorter than the CPU cache line size.
Did you mean that the last completion in a cache line will be padded ?
For ex if a transmit completion is 16 byte, you could fit 4 of them into
a cacheline. But if device has got only 3 completions to post, it would post
3 and pad the last 16 bytes. I hope you did not mean that every transmit
completion is padded to cacheline size. 
> +
> +7. Ability to place all transmit completion together with it per packet
> stream
> +   transmit timestamp using single PCIe transcation.
> +
> +8. A generic feature of the virtqueue, to contain such header data
> inline for virtio
> +   devices other than virtio-net.
> +
> +9. A flow filter virtqueue also similarly need the ability to inline
> the short flow
> +   command header.
> --
> 2.26.2
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe from this mail list, you must leave the OASIS TC that
> generates this mail.  Follow this link to all your TCs in OASIS at:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oasis-
> 2Dopen.org_apps_org_workgroup_portal_my-
> 5Fworkgroups.php&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=NHDPsfcAYlN2z-
> NXHHG4WB09qqS0voo_nf6_kGS625A&m=tvyHiYG_HE5F75ytZ2r2VlJuL4SVMmHF1elkPvWb
> byeYrTpljeSms0FoFoRXZR4g&s=NadfoNGGIuFl6YXYFu0njgTXjGISJgOdW-
> AVVvvK2Ms&e=