Re: [virtio-comment] Re: [virtio] [PATCH requirements 6/7] net-features: Add packet timestamp requirements

[Jason Wang <jasowang@redhat.com>]

On Fri, Aug 11, 2023 at 5:28âAM Willem de Bruijn
<willemdebruijn.kernel@gmail.com> wrote:
>
> On Thu, Aug 10, 2023 at 2:56âAM Jason Wang <jasowang@redhat.com> wrote:
> >
> > On Wed, Aug 9, 2023 at 4:47âPM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >
> > > On Mon, 24 Jul 2023 06:34:20 +0300, Parav Pandit <parav@nvidia.com> wrote:
> > > > Add tx and rx packet timestamp requirements.
> > > >
> > > > Signed-off-by: Parav Pandit <parav@nvidia.com>
> > > > ---
> > > >  net-workstream/features-1.4.md | 26 ++++++++++++++++++++++++++
> > > >  1 file changed, 26 insertions(+)
> > > >
> > > > diff --git a/net-workstream/features-1.4.md b/net-workstream/features-1.4.md
> > > > index d228462..37820b6 100644
> > > > --- a/net-workstream/features-1.4.md
> > > > +++ b/net-workstream/features-1.4.md
> > > > @@ -10,6 +10,7 @@ together is desired while updating the virtio net interface.
> > > >  2. Low latency tx and rx virtqueues for PCI transport
> > > >  3. Virtqueue notification coalescing re-arming support
> > > >  4  Virtqueue receive flow filters (RFF)
> > > > +5. Device timestamp for tx and rx packets
> > > >
> > > >  # 3. Requirements
> > > >  ## 3.1 Device counters
> > > > @@ -280,3 +281,28 @@ struct virtio_net_rff_delete {
> > > >       u8 padding[2];
> > > >       le32 flow_id;
> > > >  };
> > > > +
> > > > +## 3.5 Packet timestamp
> > > > +1. Device should provide transmit timestamp and receive timestamp of the packets
> > > > +   at per packet level when the device is enabled.
> > > > +2. Device should provide the current free running clock in the least latency
> > > > +   possible using an MMIO register read of 64-bit to have the least jitter.

Btw, register is not offered via all transports.

> > > > +3. Device should provide the current frequency and the frequency unit for the
> > > > +   software to synchronize the reference point of software and the device using
> > > > +   a control vq command.
> > > > +
> > > > +### 3.5.1 Transmit timestamp
> > > > +1. Transmit completion must contain a packet transmission timestamp when the
> > > > +   device is enabled for it.
> > > > +2. The device should record the packet transmit timestamp in the completion at
> > > > +   the farthest egress point towards the network.
> > > > +3. The device must provide a transmit packet timestamp in a single DMA
> > > > +   transaction along with the rest of the transmit completion fields.
> > > > +
> > > > +### 3.5.2 Receive timestamp
> > > > +1. Receive completion must contain a packet reception timestamp when the device
> > > > +   is enabled for it.
> > > > +2. The device should record the received packet timestamp at the closet ingress
> > > > +   point of reception from the network.
> > > > +3. The device should provide a receive packet timestamp in a single DMA
> > > > +   transaction along with the rest of the receive completion fields.
> > >
> > >
> > > According to the last discuss, the feature will depend on the new desc
> > > structure.
> > >
> > > I would to know, can we introduce this to the current spec with a simple change?
> > >
> > >  struct vring_used_elem {
> > >         /* Index of start of used descriptor chain. */
> > >         __virtio32 id;
> > >         /* Total length of the descriptor chain which was used (written to) */
> > >         __virtio32 len;
> > >
> > > +       __virtio64 timestamp;
> > >  };
> >
> > I think this could be one way and another proposal from Willem is:
> >
> > https://lists.linuxfoundation.org/pipermail/virtualization/2021-February/052422.html
> >
> > which might be tricky for TX but it's more flexible since it allows
> > timestamps to be done per buffer.
>
> Thanks for looping me in, Jason.
>
> I would separate timestamping data from clock synchronization and
> timestamping control path.
>
>
> For synchronization, a single 64-bit MMIO read is not necessarily the
> fastest when traversing real hardware boundaries, or sufficient. PCIe
> has Precision Time Measurement (PTM) hardware logic to capture
> clock measurement with less uncertainty and possibly latency, for instance.
> Indeed, uncertainty is more important than raw latency.
>
> FWIW, we're also trying to capture similar requires for physical devices
> through the Open Compute Platform NIC Core Offloads effort:
> https://www.opencompute.org/w/index.php?title=Core_Offloads#Timestamping

This is interesting and the whole idea of "NIC Core Offloads" looks
great as well.

>
> Virtio defines both a virtual interface and one that can be supported
> by hardware, so it's not a 1:1 match, but many subtle points probably
> apply to both.

Yeah, I think virito-net may try to align with that.

>
>
> For timestamping, the difficult part is the transmit completion stamp,
> asynchronous with data flow. If this is taken on the device before or
> at the time of queuing the completion to the host, then it can be stored
> in\ the completion descriptor.

Yes, this is the proposal from Xuan actually.

> But if taken at the PHY, say, it is not
> uncommon for this to happen after the completion is written.

This seems the requirement of the this patch which says:

"
 The device should record the packet transmit timestamp in the completion at
 the farthest egress point towards the network.
"

It seems to imply the PHY but with a DMA interface:

"
The device must provide a transmit packet timestamp in a single DMA
transaction along with the rest of the transmit completion fields.
"

Parav, can you clarify? And

> And
> instead a block of dedicated registers is used and the host must poll
> a register. There are examples of these in the Linux device driver
> directory. For virtio, it is probably sufficient to only support the
> first kind.

RIght, for the second, it could be tricky for a virtual environemnt.

>
>
> A third aspect is the control path: querying and configuring hardware
> timestamps (siocgstamp/siosgstamp). If hardware is capable, easiest is
> to just timestamp every packet. Much hardware out there targets low
> rate PTP messages. But a virtualized virtio device could timestamp
> all. In which case this control API can maybe be punted on.
>

That could be one way.

Thanks