NVM Express® (NVMe®) Technology Support for CXL®: Unleashing Computational Workloads

By: Bill Martin, Principal Engineer SSD I/O Standards and NVM Express Board Member, Samsung Electronics, and Jason Molgaard, Principal Storage Solutions Architect, Solidigm

As the data center ecosystem continues to evolve to enable AI and HPC workloads, NVM Express® (NVMe®) technology delivers key updates to boost the performance, scalability and efficiency of computational workloads in modern data centers. One important next step that we are working on in the Technical Working Group is adding support for Compute Express Link® (CXL®) in NVMe devices. We believe this will help facilitate high-speed, low-latency access to memory and storage resources, critical for quickly completing computational processes.

In this blog, we’ll showcase how NVM Express Support for CXL can help enable new storage architectures to boost efficiency for computational processes, vital for AI and HPC workloads in enterprise and hyperscale data centers.

Essential Benefits of NVM Express Support for CXL for Modern Computational Processes

NVMe devices are increasingly providing host-accessible memory in the form of Subsystem Local Memory (SLM), which provides NVMe command-based access to the memory. Using NVM Express Support for CXL, memory can be accessed through a memory protocol while simultaneously remaining coherent. This support also allows for peer-to-peer communication between NVMe devices, bypassing the need to communicate through the host.

You may be wondering, what are the benefits of providing support for CXL in NVMe devices? The short answer is that the CXL memory access protocol provides several necessary capabilities that can enhance the existing benefits of NVMe technology, including memory coherency and low latency for small transfers.

Memory coherency helps data centers using NVMe devices experience significant improvements over the current memory access commands. With memory coherency, the host and the NVMe device perceive the same view of shared memory or shared data and copies that grow stale are reduced or eliminated. Currently, NVMe Controller Memory Buffer (CMB) and Persistent Memory Region (PMR) lack coherency, and therefore, use uncached memory spaces. Additionally, memory coherency enables NVMe devices to perceive the same view of shared memory or shared data with the host. Data can be easily read or written with load/store commands and processed in the drive with NVMe Computational Programs.

NVM Express Support for CXL Use Cases in the Data Center

NVM Express Support for CXL can unlock new possibilities for your data center with NVMe devices by providing low latency, high-efficiency computational storage processes.

One use case occurs in data post-processing before the data is written to storage. In this use case, we would move data to SLM in a Computational Storage Drive using CXL.mem, and then process the data with a program invoked by an NVMe Execute Program command. The processed data is then written to an output buffer and transferred to persistent memory. The key advantages here are that we avoid copying data using Direct Memory Access (DMA) from the Host Memory and we also receive the benefits of lower latency from CXL-based Load/Store access.

Another key use case involves bypassing data movement through host memory. In this use case, data is written to host addressable SLM in a Computational Storage NVMe drive using CXL.mem, similar to the data post-processing use case. The data is processed using a Computational Program, and the result is written to an output buffer also in host addressable SLM. Since the data is in host addressable memory, the data may be transferred to a traditional NVMe SSD without first moving the data into host memory.  In other words, this use case enables an NVMe Computational Storage Drive to process data and ultimately store the data on a traditional NVMe drive and not burden the host for the processing or the data movement.

Dive Into Resources: NVM Express Support for CXL

For additional information on the benefits of NVM Express Support for CXL, watch our presentation on this topic from FMS 2024 available on the NVM Express YouTube channel.

We also recently participated in a video interview, available on our YouTube channel, for a brief overview of the benefits of this technology. Finally, visit the specifications page of the NVM Express website to find out more about NVM Express Support for CXL.