NVMe® over TCP Transport: Enabling NVMe Adoption for the Masses

Sagi Grimberg, Co-Founder and CTO, LightBits Labs

With the release of the NVM Express® over Fabrics (NVMe-oF™) 1.0 specification in 2016, the NVM Express organization made it possible for storage environments to extend the benefits of NVMe® technology across a networked fabric. In late 2018, NVM Express took this a step further by announcing added support for the NVMe over TCP transport and released the ratified technical proposal for public use. This advancement was key for enabling NVMe-oF technology adoption because it allowed enterprises to layer NVMe technology over existing software-based Transport Control Protocol (TCP) transport implementations.

NVMe/TCP Transport: The Basics

At a high level, TCP/IP is a wire protocol that provides the ability for two computers to talk across the network, making TCP one of the most common transports. The NVMe/TCP transport was created to define the mapping of NVMe queues and data delivery over TCP. Now, with the NVMe/TCP specification, implementers can know how to map NVMe queues to TCP/IP connections, what the wire protocol and message format is and what exactly they need to be compatible with a controller or entity that is complying to NVMe technology.

With the NVMe/TCP transport, all the attributes that make NVMe technology perform and operate more efficiently are inherited. So, just as NVMe technology is a massive improvement in terms of IOPS (input/output operations per second), latency and throughput, NVMe/TCP protocol implementations will see the same improvements.

Deploy NVMe Technology Anywhere

Most data centers utilize TCP for their data-transfer applications. Given this, the added support for the NVMe/TCP transport opened the possibility for widespread NVMe technology adoption. With the new NVMe/TCP transport, almost every environment under the sun can now benefit from the performance benefits of NVMe-oF technology. Since many data centers and enterprises have already implemented TCP in their applications, existing hardware, etc., they can utilize their existing infrastructure and conduct a simple software update to take advantage of NVMe technology today.

Future NVMe/TCP Transport Innovations

This past year, the NVMe/TCP transport was released as its own specification. This was part of the NVMe 2.0 specifications refactoring, which separated the NVMe-oF transports into individual specifications. While this release didn’t have any large changes to the NVMe/TCP protocol, there were some small changes and clarifications with ECN (electronic communication network), which are now implemented in the latest specification version here. For a high-level overview of what was discussed in this blog, check out the recent interview I did for the NVM Express YouTube channel.