NVMe VIP Architecture: Host Features
In my last post, I covered a basic NVMe VIP test-case including some basic setup, sending a command and receiving a completion. Here, we’ll look at a few more NVMe commands, touching on some of the features and capabilities of the VIP.
Here’s where you can learn more about Synopsys VC Verification IP for NVMe and for PCIe.
A (Reminder) View of the VIP
We overviewed this briefly last time. This time we’ll go into a bit more depth, so we will continue to refer to this diagram:
The NVMe VIP provides a set of feature to assist in testing. These include randomizations, feature snooping, simplified PRP and data buffer handling, memory fencing and built-in score-boarding. We’ll look at each of these in turn with another example.
Continuing our Test Case…
Following up on our “trivial test-case” from the last post (again, we are not showing some of the task arguments or checking errors), let’s take a look at a few more commands to get our NVMe test case rolling.
Just a reminder: the tasks that start with the word Script are NVMe commands. The others (that don’t start with Script), are a VIP status/control/configuration task.
// We will assume that the PCIe stack is setup and running bit [63:0] base_addr = 32’h0001_0000; // Ctlr BAR base addr dword_t num_q_entries, ctlr_id; // Tell the host where the controller has its base address AllocateControllerID(base_addr, ctlr_id, status); num_q_entries = 2; // Create the Admin Completion and Submission Queues ScriptCreateAdminCplQ(ctlr_id, num_q_entries, status); ScriptCreateAdminSubQ(ctlr_id, num_q_entries, status); // Send an “Identify Controller” Command data_buf_t #(dword_t) identify_buffer; // identify data identify_buffer = new(1024); ScriptIdentify(ctlr_id, 0, identify_buffer, 0, status);
We ended our last sample with a call to Identify Controller. Now, continuing at that point, we read bytes 519:516 to get the number of valid namespace IDs. We hand that to the host VIP with the SetNumNamespaces() call. Note that we had to byte-swap the (little-endian) data returned in the Identify Controller buffer.
int num_ns, nsid, blk_size_pow2, blk_size_in_bytes; bit [63:0] ns_size_in_blks; feature_identifier_t feature_id; nvme_feature_t set_feature; // We’ll grab the Number of valid namespaces (NN) from the // identify buffer. Note index converted from bytes to dword. num_ns = ByteSwap32(identify_buffer[516 >> 2]); // bytes 519:516 // Tell the VIP how many active NSIDs the controller has SetNumNamespaces(ctlr_id, num_ns, status);
Next we read the information for one of the namespaces (Namespace ID=1). Note that we “cheated” a bit here, as we should have walked all the valid namespaces. For the example we’ll just assume we have only NSID=1. Although the Identify calls don’t take a PRP list, their host memory buffer can have an offset. If this is desired, select the argument “use_offset=1”. The actual offset is randomized via the constraints MIN/MAX_PRP_DWORD_OFFSET_VAR.
// Now send an “Identify Namespace” command for nsid=1
nsid = 1;
use_offset = 1; // Randomize buffer offset
ScriptIdentify(ctlr_id, nsid, identify_buffer,
use_offset, status);
// Pull information from format[0]
blk_size_pow2 = ByteSwap32(identify_buffer.GetData(128 >> 2)));
blk_size_pow2 = (blk_size_pow2 >> 16) & 32’hff; // dword[23:16]
blk_size_in_bytes = 1 << blk_size_pow2; // Convert
ns_size_in_blks = ByteSwap64({identify_buffer.GetData(8 >> 2),
identify_buffer.GetData(12 >> 2)});
// Before we create queues, we need to configure the num queues
// on the controller.
feature_id = FEATURE_NUM_QUEUES;
set_feature = new(feature_id);
Once the Identify Namespace returns, we now have both the block size and the namespace size. We set the number of requested queues with Set Features. Via the VIP’s feature snooping, this will (transparently) set the VIP with the current number of supported submission and completion queues (for later checking and error injection support.)
The next steps format our namespace (using Format 0 in the Identify Namespace data structure). We then update the VIP view of the namespace information. The VIP needs this namespace information to keep a per-namespace scoreboard.
set_features.SetNumCplQ(2); // Request number of sub &
set_features.SetNumSubQ(3); // cpl queues
// Call Set Features command to set the queues on the ctlr
ScriptSetFeatures(ctlr_id, set_features, …, status);
// Note that Set Features Number of Queues command need not
// return the same amount of queues that were requested. We can
// check by examining set_features.GetNumCplQ() and
// GetNumSubQ(), but in this case we’ll just trust it…
// Format the Namespace
sec_erase = 0; // Don’t use secure erase
pi_md_settings = 0; // Don’t use prot info or metadata
format_number = 0; // From Identify NS data structure
ScriptFormatNVM(ctlr_id, nsid, sec_erase, pi_md_settings,
format_number, …, status);
// Tell the VIP about this NS
SetNamespaceInfo(ctlr_id, nsid, blk_size_in_bytes,
ns_size_in_blks, md_bytes_per_blk,
pi_md_settings, 0, status);
We next create a pair of I/O queues. Since the submission queue requires its companion completion queue to be passed along with it, we create the completion queue first. Note that queue creation routines take an argument contig. If contig is set, the queue will be placed in contiguous memory, otherwise a PRP list will be created for that queue. In addition to creating the actual queue, the VIP creates a fence around the queue to verify memory accesses to the queues. Attempts from the controller to (for example) read from a completion queue will be flagged as an invalid access attempt. The actual queue IDs are randomized (within both legal and user-configurable constraints).
// Create the I/O Queues num_q_entries = 10; contig = 1; // Contiguous queue ScriptCreateIOCplQ(ctlr_id, num_q_entries, contig, …, cplq_id, …, status); contig = 0; // PRP-based queue ScriptCreateIOSubQ(ctlr_id, num_q_entries, contig, cplq_id …, subq_id, …, status);
Once we have I/O queues created, we can start doing I/O. Using the ScriptWrite() and ScriptRead() calls, we send data to the controller and immediately retrieve that same data back. The underlying data structure of the data (in host memory) is built automatically by the VIP. Note the use_offset argument (as with our queue creation tasks) to control whether we generate PRP and PRP List offsets (controlled by MIN/MAX_PRP_DWORD_OFFSET_VAR and MIN/MAX_PRP_LIST_DWORD_OFFSET respectively). Due to our built in score-boarding, we don’t have to compare the data read from that written, the VIP is checking data returned against its shadow copy that is tracking successful VIP writes to the controller.
// Do our I/O write then read with a random LBA/length
data_buf_t #(dword_t) wbuf, rbuf; // Write/Read Data buffers
num_blks = RandBetween(1, ns_size_in_blks);
lba = RandBetween(0, ns_size_in_blks – num_blks);
num_dwords = (blk_size_in_bytes / 4) * num_blks;
wbuf = new(num_dwords);
for (int idx = 0 ; idx < num_dwords ; idx++) // Fill the buffer
wbuf.SetData(idx, { 16’hdada, idx[15:0] } );
ScriptWrite(ctlr_id, subq_id, lba, nsid, wbuf, num_blks,
use_offset, …, status);
// We’ll read the same LBA since we know it’s been written
ScriptRead(ctlr_id, subq_id, lba, nsid, rbuf, num_blks,
use_offset, …, status);
// Do what you’d like with the rbuf (that’s the data we just read).
We’re Done!
Hopefully that’s gotten us through most of the basics. You should have a good feel for the operation of the VIP. Again, many of these tasks have more arguments allowing more control and error injection, but our goal is to get through without dealing with the more esoteric features. If you have the VIP handy, feel free to walk through the examples: they should look quite familiar.
In my next post, we will look into actually testing a controller, especially going into features like error injection.
As always, thanks for joining us. See you again soon.
Authored by Eric Peterson
Here’s where you can learn more about Synopsys VC Verification IP for NVMe and for PCIe.
