Re: [virtio-comment] Re: [virtio] [PATCH requirements 6/7] net-features: Add packet timestamp requirements

On Fri, Aug 11, 2023 at 5:28 AM Willem de Bruijn <> wrote: >
> On Thu, Aug 10, 2023 at 2:56 AM Jason Wang <> wrote: > > >
> On Wed, Aug 9, 2023 at 4:47 PM Xuan Zhuo <> wrote: > > > > >
> On Mon, 24 Jul 2023 06:34:20 +0300, Parav Pandit <> wrote: > > >
> Add tx and rx packet timestamp requirements. > > > > > > >
> Signed-off-by: Parav Pandit <> > > > > --- > > >
>  net-workstream/features-1.4.md
26 ++++++++++++++++++++++++++ > > >
>  1 file changed, 26 insertions(+) > > > > > > >
> diff --git a/net-workstream/features-1.4.md b/net-workstream/features-1.4.md > > >
> index d228462..37820b6 100644 > > > > --- a/net-workstream/features-1.4.md > > > > +++ b/net-workstream/features-1.4.md > > > > @@ -10,6 +10,7 @@ together is desired while updating the virtio net interface. > > >
>  2. Low latency tx and rx virtqueues for PCI transport > > >
>  3. Virtqueue notification coalescing re-arming support > > >
>  4  Virtqueue receive flow filters (RFF) > > > > +5. Device timestamp for tx and rx packets > > > > > > > >  # 3. Requirements > > > >  ## 3.1 Device counters > > > > @@ -280,3 +281,28 @@ struct virtio_net_rff_delete { > > >
>
u8 padding[2]; > > >
>
le32 flow_id; > > > >  }; > > > > + > > > > +## 3.5 Packet timestamp > > > > +1. Device should provide transmit timestamp and receive timestamp of the packets > > > > +
at per packet level when the device is enabled. > > > > +2. Device should provide the current free running clock in the least latency > > > > +
possible using an MMIO register read of 64-bit to have the least jitter.  Btw, register is not offered via all transports.  > > > > +3. Device should provide the current frequency and the frequency unit for the > > > > +
software to synchronize the reference point of software and the device using > > > > +
a control vq command. > > > > + > > > > +### 3.5.1 Transmit timestamp > > > > +1. Transmit completion must contain a packet transmission timestamp when the > > > > +
device is enabled for it. > > > > +2. The device should record the packet transmit timestamp in the completion at > > > > +
the farthest egress point towards the network. > > > > +3. The device must provide a transmit packet timestamp in a single DMA > > > > +
transaction along with the rest of the transmit completion fields. > > > > + > > > > +### 3.5.2 Receive timestamp > > > > +1. Receive completion must contain a packet reception timestamp when the device > > > > +
is enabled for it. > > > > +2. The device should record the received packet timestamp at the closet ingress > > > > +
point of reception from the network. > > > > +3. The device should provide a receive packet timestamp in a single DMA > > > > +
transaction along with the rest of the receive completion fields. > > > > > > > >
> According to the last discuss, the feature will depend on the new desc > >
> structure. > > > > >
> I would to know, can we introduce this to the current spec with a simple change? > > > > >
>  struct vring_used_elem { > > >
/* Index of start of used descriptor chain. */ > >
>
__virtio32 id; > > >
/* Total length of the descriptor chain which was used (written to) */ > >
>
__virtio32 len; > > > > > > +
__virtio64 timestamp; > > >  }; > > >
> I think this could be one way and another proposal from Willem is: > > >
>  https://lists.linuxfoundation.org/pipermail/virtualization/2021-February/052422.html  > > >
> which might be tricky for TX but it's more flexible since it allows >
> timestamps to be done per buffer. >
> Thanks for looping me in, Jason. >
> I would separate timestamping data from clock synchronization and
> timestamping control path. > >
> For synchronization, a single 64-bit MMIO read is not necessarily the
> fastest when traversing real hardware boundaries, or sufficient. PCIe
> has Precision Time Measurement (PTM) hardware logic to capture
> clock measurement with less uncertainty and possibly latency, for instance.
> Indeed, uncertainty is more important than raw latency. >
> FWIW, we're also trying to capture similar requires for physical devices
> through the Open Compute Platform NIC Core Offloads effort:
>  https://www.opencompute.org/w/index.php?title=Core_Offloads#Timestamping
This is interesting and the whole idea of "NIC Core Offloads" looks great as well.  >
> Virtio defines both a virtual interface and one that can be supported
> by hardware, so it's not a 1:1 match, but many subtle points probably
> apply to both.  Yeah, I think virito-net may try to align with that.  > >
> For timestamping, the difficult part is the transmit completion stamp,
> asynchronous with data flow. If this is taken on the device before or
> at the time of queuing the completion to the host, then it can be stored
> in the completion descriptor.  Yes, this is the proposal from Xuan actually.
> But if taken at the PHY, say, it is not
> uncommon for this to happen after the completion is written.  This seems the requirement of the this patch which says:  "  The device should record the packet transmit timestamp in the completion at  the farthest egress point towards the network. "  It seems to imply the PHY but with a DMA interface:  " The device must provide a transmit packet timestamp in a single DMA transaction along with the rest of the transmit completion fields. "  Parav, can you clarify? And
> And
> instead a block of dedicated registers is used and the host must poll
> a register. There are examples of these in the Linux device driver
> directory. For virtio, it is probably sufficient to only support the
> first kind.  RIght, for the second, it could be tricky for a virtual environemnt.  > >
> A third aspect is the control path: querying and configuring hardware
> timestamps (siocgstamp/siosgstamp). If hardware is capable, easiest is
> to just timestamp every packet. Much hardware out there targets low
> rate PTP messages. But a virtualized virtio device could timestamp
> all. In which case this control API can maybe be punted on.
>  That could be one way.  Thanks