NVMe® over RDMA Transport: Improving Network-Based Storage

By Curtis Ballard, Distinguished Technologist, Hewlett Packard Enterprise

The NVM Express® (NVMe®) over Remote Direct Memory Access (RDMA) transport was recently broken out into its own specification with the release of the NVMe 2.0 specifications. The NVMe over RDMA transport specification defines an interface for host software to communicate with NVM subsystems over a variety of memory-based transports and message-based transports. The RDMA API, or Application Programming Interface, very closely matches the models that are used for communicating over the PCIe® bus. This makes RDMA a good fit for mapping NVMe/PCIe onto a fabric technology. Below I’ll go into more detail about the benefits of using the NVMe/RDMA transport, along with key use cases and future outlooks.

Amplify Storage Performance with the NVMe/RDMA Transport

The NVMe/RDMA transport provides a method of transporting NVMe traffic over a network that supports RDMA, such as Ethernet or InfiniBand, using a minimal amount of protocol translations between the two layers. In addition, RDMA enables applications to bypass the software stack when processing network traffic, which produces lower latency and higher performance speeds. While implementing an RDMA network setup can be difficult, most networking infrastructure vendors that support RDMA have tools for simplifying the configuration and settings, which removes much of the complexity. At HPE, we utilize the NVMe/RDMA transport for our products to obtain high performance with the lowest cost CPU, target CPU requirements and predictable latency.

Supporting Disaggregated Storage

An ongoing theme in the storage industry is the concept of composable architectures such as disaggregated storage. With RDMA, this means there is less dedicated storage inside of a server and, with storage outside of the server enclosure, that outside storage is attached over a high-performance network such as an ethernet running RDMA. With the disaggregated storage model, there is a much more general-purpose data center structure with composers, or workflow orchestration devices, that can recompose your infrastructure as your business needs evolve.

The NVMe/RDMA transport can help support the emerging space of disaggregated storage. By utilizing the NVMe/RDMA transport in a disaggregated storage model, it is possible to get close to the performance speeds of direct attached storage. This model will be particularly useful for hyperscalers and within cloud computing environments. Plus, there is ample opportunity for NVMe/RDMA technology to have value in composable architectures.

The NVMe/RDMA Transport Moving Forward

NVM Express is continuously maturing the NVMe over Fabrics (NVMe-oF™) ecosystem. In the next revision of the NVMe library of specifications, one of the major focus areas is enabling the growth of fabrics support. We can expect changes in how we can utilize the NVMe/RDMA transport. The industry is also working to refine RDMA technology specifically, with plenty of opportunity for companies to innovate and make infrastructure management more simple and seamless.

There has been chatter in the industry about future storage architectures and where we go from here. Some of these, like the previously mentioned disaggregated storage and the further development of the fabrics ecosystem, will factor into the NVMe/RDMA transport journey going forward.

Download the RDMA transport specification

You can learn more about the NVMe/RDMA transport on the NVM Express website and by watching my recent video interview. The RDMA transport specification is publicly available for download here.