Alleviating Performance Interference Through Intra-Queue I/O Isolation for NVMe-over-Fabrics

EasyChair Preprint 9380

Abstract

The NVMe-over-Fabrics (NVMeoF) protocol enables high-performance Protocol Data Units (PDUs) exchanges between hosts and remote NVMe controllers. The performance benefits of NVMeoF are mainly derived from the multiple deep queue pairs for parallel PDUs transfers. NVMeoF has significantly facilitated NVMe SSD disaggregation from compute nodes for better resource utilization and scaling independence. However, as the performance of NVMe SSD and network infrastructure increases, the near-perfect performance delivery of NVMeoF is harder to achieve. The primary reason is the increased CPU interrupts and performance interference originated from the I/O requests served by the same NVMeoF queue pair.

In this paper, we investigate how intra-queue requests are mutually affected, and propose PINoF, a Performance Isolated remote storage access mechanism for NVMe-over-Fabrics. PINoF separates CMD and Data PDUs resources in each NVMeoF queue pair to achieve intra-queue I/O isolation, transfers PDUs in batch along with read or write specific I/O path to achieve isolated interrupt-coalescing, and introduces differentiated PDU reordering schemes to achieve isolated scheduling. Our experimental results demonstrate that compared with state-of-the-art NVMeoF implementations, PINoF achieves 23.92% lower latency, increases bandwidth by up to 19.59%, and improves IOPS by 12.41% on average.

Keyphrases: I/O Isolation, NVMeoF, performance interference

@booklet{EasyChair:9380,
  author    = {Wenhao Gu and Xuchao Xie and Dezun Dong},
  title     = {Alleviating Performance Interference Through Intra-Queue I/O Isolation for NVMe-over-Fabrics},
  howpublished = {EasyChair Preprint 9380},
  year      = {EasyChair, 2022}}