Developers Guide

Attention: This guide describes the 2.0 release.

This guide explains the process for developing new designs. Developing a new design consists of writing the necessary Verilog/C/Perl code, defining registers, and writing simulation and regression tests.

Show Contents...Hide Contents...

Overview

Designs consist of the code (Verilog/C/Perl/etc) that implements the design and a set of tests that demonstrate the operation of the design. As explained in #Regression Tests , the purpose of the tests is to specify the features of the design and demonstrate the correctness of that design.

Most designs consist of one or more modules that are inserted into the user data path. Modules can be reused across designs.

Frequently modules have a set of registers associated with them. The NetFPGA system uses an XML-based system%STARTENDNOTE%The XML-based register system was introduced in the 2.0 community release%STOPENDNOTE% for declaring registers and automatically allocating addresses.

Project Directory Structure

The directory structure for contributed projects follow the structure of the reference projects.

  NF2                    {base directory}
    projects             {project directory}
           {contributed project}
        doc              {documentation}
        include          {project.xml, project specific module XML}
        lib              {perl and C headers}
        src              {non library verilog}
        synth            {all .xco files}
        regress          {regression tests}
        verif            {simulation tests}
        sw               {project software}

Regression Tests

All contributed designs are specified by the set of tests that they pass. In other words, the developer of a reusable design is expected to clearly document the set of tests their design passes (e.g. "IPv4 packets from 64-1500 bytes long"), and supply the set of tests that can be repeated by users of that design. This creates unambiguous specifications of a reusable design, sets clear expectations of how complete (or not) the design is, and instills greater confidence for whoever is reusing the code.

Every function of a module or project must have a regression test associated with it. Each test is defined by the overall goal and the process used to test the function. The NetFPGA package provides Perl modules that allow users to send/receive live packets and read/write registers of the NetFPGA. As an example, the Packet Generator has four regression tests written using the Perl libraries. These tests run live network traffic to verify the ability of the hardware to send PCAP data, utilize the rate limiter, perform iterations, and capture packets. Below is the description of the Packet Generator iteration test.

• Name
  • test_iterations
• Description
  • Load and send two PCAP files from nf2c0 and nf2c1 with a specified number of iterations for each. Verify the packet sent counters.

• Process
  1. Initialize NetFPGA hardware
  2. Load two PCAP files for nf2c0 and nf2c1
  3. Set 10 iterations for nf2c0 and 100 iterations for nf2c1
  4. Run Packet Generator (send traffic)\ 1. Check counters to verify the number of packets sent

Within the Reference Router project there are a total of 16 tests that vary from testing ARP misses to longest prefix match hits. All of the regression tests are documented on the Wiki page associated with each project. Examples of the Reference Router and Packet Generator regression tests can be found here: Reference Router Regression Tests, Packet Generator Regression Tests

Register System

Project XML

The project XML file is a key component to any project. It is required by the NetFPGA backend scripts. It should be located in the project's include directory (${NF2_ROOT}/project//include). In addition to naming and describing the project it belongs to, the project file has two main purposes: identify library modules used, and define the register layout.

Identifying NetFPGA Library Modules

All NetFPGA library modules used by the project are listed in the nf:modules XML tag. This allows the backend scripts to identify and load only the source Verilog and module XML files needed by the project. Below is an example of the including: generic_top, reference_core, rr_input_arbiter, and io_queues.

  <nf:use_modules>
    nf2/generic_top
    nf2/reference_core
    io_queues/cpu_dma_queue
    io_queues/ethernet_mac
    input_arbiter/rr_input_arbiter
  </nf:use_modules>

Define Register Layout

Each project must define the register layout. The register layout is wrapped in the XML tag nf:memalloc. This tag uses an attribute called layout to specify the placement of the registers. At the present time the only layout defined is called 'reference'.

Within the reference layout there exist two groups called core and udp. The core groups are 4 MB sections of the address space. The reference layout contains 4 core groups (core0-core3). The first core group (core0) is reserved for the CPCI registers and should never be used. The second core group (core1) contains the register addresses for the device id, dma, mdio, and also the MAC and CPU queues. Core groups two and three are unused by the reference implementations. The udp group is a 32 MB section of the register space. The udp (user data path) group is contains the main reference pipeline. When modules are added to a project or a new project is created, it is recommended to insert modules into the user data path. Most users will be adding there modules into the udp group.

Group	Size	Address Range
core 0	4MB	0x0000000 – 0x03FFFFF	reserved (don't use)
core 1	4MB	0x0400000 – 0x07FFFFF
core 2	4MB	0x0800000 – 0x0BFFFFF
core 3	4MB	0x0C00000 – 0x0FFFFFF
sram	16MB	0x1000000 – 0x1FFFFFF	reserved
udp	32MB	0x2000000 – 0x3FFFFFF
dram	64MB	0x4000000 – 0x7FFFFFF	reserved

Modules are added to groups by using the nf:instance tag and a name attribute. The name attribute is the name of the module used in the module XML (module XML files are explained later). In addition to the name, the attributes base and count can be used. The base attribute tells the register system backend what base address the module's registers should be placed at. The register backend will not guarantee placement at the specified base address, but it will do it's best. Usually when the system fails to place the registers at the specified base address it is due to other module register blocks overlapping the base address. The count attribute is very useful. It allows a modules registers to be replicated by the number equal to count. For instance in the reference router the MAC queues (nf2_mac_grp) is replicated four times (once per physical port). Below is the example of register definition section of the reference router.

   <nf:memalloc layout="reference">
     <nf:group name="core1">
       <nf:instance name="device_id" />
       <nf:instance name="dma" base="0x0500000"/>
       <nf:instance name="mdio" />
       <nf:instance name="nf2_mac_grp" count="4" />
       <nf:instance name="cpu_dma_queue" count="4" />
     </nf:group>
     <nf:group name="udp">
       <nf:instance name="in_arb" />
       <nf:instance name="router_op_lut" />
       <nf:instance name="strip_headers" />
       <nf:instance name="output_queues" />
     </nf:group>
   </nf:memalloc>

Full Example

Putting together the library module definitions and the register layout we get the full project XML file. Below is the full project XML for the reference router.

  <?xml version="1.0" encoding="UTF-8"?>
  <nf:project xmlns:nf="http://www.NetFPGA.org/NF2_register_system" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.NetFPGA.org/NF2_register_system NF2_register_system.xsd ">
    <nf:name>Reference router</nf:name>
    <nf:description>Reference IPv4 router</nf:description>
    <nf:use_modules>
      io_queues/cpu_dma_queue
      io_queues/ethernet_mac
      input_arbiter/rr_input_arbiter
      nf2/generic_top
      nf2/reference_core
      output_port_lookup/cam_router
      output_queues/sram_rr_output_queues
      sram_arbiter/sram_weighted_rr
      user_data_path/reference_user_data_path
      io/mdio
      cpci_bus
      dma
      user_data_path/udp_reg_master
      io_queues/add_rm_hdr
      strip_headers/keep_length
      utils
      utils/generic_regs
    </nf:use_modules>
    <nf:local_modules>
    </nf:local_modules>
    <nf:memalloc layout="reference">
      <nf:group name="core1">
        <nf:instance name="device_id" />
        <nf:instance name="dma" base="0x0500000"/>
        <nf:instance name="mdio" />
        <nf:instance name="nf2_mac_grp" count="4" />
        <nf:instance name="cpu_dma_queue" count="4" />
      </nf:group>
      <nf:group name="udp">
        <nf:instance name="in_arb" />
        <nf:instance name="router_op_lut" />
        <nf:instance name="strip_headers" />
        <nf:instance name="output_queues" />
      </nf:group>
    </nf:memalloc>
  </nf:project>

Module XML

Each Verilog NetFPGA module that contains registers must have a module XML file. This file enumerates and defines the registers and allows the register system backend scripts to automatically assign register addresses.

Module XML Header

Each module XML file starts by defining the name, prefix, description, location, and blocksize for the modules registers. The name must contain no spaces and is used in the project XML as the instance name.

The prefix defines a prefix that is added to all the registers in the module. This ensures an easy correlation between the register names and the module that contains the registers.

The locations specifies whether the modules registers should be located in the user data path (udp) or the core register groups.

Within the user data path (udp) group we can specify a register blocksize of an arbitrary size. Below is an example of the round robin input arbiter a that specifies a blocksize of 256. Within a core group the blocksize is restricted to 256k (anything else will cause problems).

  <nf:name>in_arb</nf:name>
  <nf:prefix>in_arb</nf:prefix>
  <nf:location>udp</nf:location>
  <nf:description>Round-robin input arbiter</nf:description>
  <nf:blocksize>256</nf:blocksize>

Individual Registers

The first way to define registers is to define individual registers in a module. Each register definition contains a name, description, and type. Below is an example register from the round robin input arbiter. The type can be one of the predetermined types, or a user defined type in the module XML. The type field is discussed later in the guide.

  <nf:register>
    <nf:name>num_pkts_sent</nf:name>
    <nf:description>Number of packets sent</nf:description>
    <nf:type>counter32</nf:type>
  </nf:register>

Register Groups

There are times when a module will require register groups. This usually occurs when a module has a set of registers that are repeated. For instance, the output queues module has a group of registers for each output queue. These registers are same across each of the output queues.

Register groups are specified using the tag nf:register_group followed by the name and instance tags. The name is appended to the module prefix in addition to the instance number when the register names are created. As an example, the output queue module has a prefix of 'oq'. When the register group below is added to the module XML the register names created are the following: OQ_QUEUE_0_CTRL_REG, OQ_QUEUE_0_NUM_PKT_BYTES_STORED_REG. The number will increase from zero to the number of instances specified. In the case of the output queues the number of instances are set to eight.

 <nf:register_group>
   <nf:name>queue</nf:name>
   <nf:instances>:NUM_OUTPUT_QUEUES</nf:instances>
     <nf:register>
       <nf:name>ctrl</nf:name>
       <nf:description>Control register</nf:description>
       <nf:type>oq_control</nf:type>
     </nf:register>
     <nf:register>
       <nf:name>num_pkt_bytes_stored</nf:name>
       <nf:description>Number of packet bytes stored</nf:description>
       <nf:type>sram_byte_cnt</nf:type>
     </nf:register>
     ....
 </nf:register_group>

Types

Standard Types

There are a total of nine standard types defined for users. These types are shown in the table below.

Type	Width	Description
ethernet_addr	48	ethernet address
ip_addr	32	IP address
counter32	32	32 bit counter
software32	32	software 32-bit register
generic_counter32	32	generic 32-bit counter
generic_hardware32	32	generic 32-bit hardware register
generic_software32	32	generic 32-bit software register
dataword	64 (DATA_WIDTH)	Data word in the data path
ctrlword	8 (CTRL_WIDTH)	Control word in the data path

Constants

Each module can have constants defined. The constants can be for offsets within a register or parameters such as the number of output queues. Below is a code segment from the Output Queues module XML. The number of output queues is set to 8. Notice the colon in front of the name. This means the register will be in the global name space and can be used by other modules. The enable_send_bit is defined as bit zero of a control register. The initialize_oq_bit_num is defined as the second bit of a control register.

  <nf:constants>
    <nf:constant>
      <nf:name>:NUM_OUTPUT_QUEUES</nf:name>
      <nf:value>8</nf:value>
    </nf:constant>
    <nf:constant>
      <nf:name>ENABLE_SEND_BIT_NUM</nf:name>
      <nf:value>0</nf:value>
    </nf:constant>
    <nf:constant>
      <nf:name>INITIALIZE_OQ_BIT_NUM</nf:name>
      <nf:value>1</nf:value>
    </nf:constant>
  </nf:constants>

Full Examples

Input Arbiter

  <?xml version="1.0" encoding="UTF-8"?>
  <nf:module xmlns:nf="http://www.NetFPGA.org/NF2_register_system" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.NetFPGA.org/NF2_register_system NF2_register_system.xsd ">
    <nf:name>in_arb</nf:name>
    <nf:prefix>in_arb</nf:prefix>
    <nf:location>udp</nf:location>
    <nf:description>Round-robin input arbiter</nf:description>
    <nf:blocksize>256</nf:blocksize>
    <nf:registers>
      <nf:register>
        <nf:name>num_pkts_sent</nf:name>
        <nf:description>Number of packets sent</nf:description>
        <nf:type>counter32</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>last_pkt_word_0</nf:name>
        <nf:description>Data word 0 of the last packet to pass through</nf:description>
        <nf:type>dataword</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>last_pkt_ctrl_0</nf:name>
        <nf:description>Control word 0 of the last packet to pass through</nf:description>
        <nf:type>ctrlword</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>last_pkt_word_1</nf:name>
        <nf:description>Data word 1 of the last packet to pass through</nf:description>
        <nf:type>dataword</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>last_pkt_ctrl_1</nf:name>
        <nf:description>Control word 1 of the last packet to pass through</nf:description>
        <nf:type>ctrlword</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>state</nf:name>
        <nf:description>State of the internal state machine</nf:description>
      </nf:register>
    </nf:registers>
  </nf:module>

Output Queues

  <?xml version="1.0" encoding="UTF-8"?>
  <nf:module xmlns:nf="http://www.NetFPGA.org/NF2_register_system" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.NetFPGA.org/NF2_register_system NF2_register_system.xsd ">
  <nf:name>output_queues</nf:name>
  <nf:prefix>oq</nf:prefix>
  <nf:location>udp</nf:location>
  <nf:description>SRAM-based output queue using round-robin removal</nf:description>
  <nf:blocksize>4k</nf:blocksize>
  <nf:registers>
    <nf:register_group>
      <nf:name>queue</nf:name>
      <nf:instances>:NUM_OUTPUT_QUEUES</nf:instances>
      <nf:register>
        <nf:name>ctrl</nf:name>
        <nf:description>Control register</nf:description>
        <nf:type>oq_control</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>num_pkt_bytes_stored</nf:name>
        <nf:description>Number of packet bytes stored</nf:description>
        <nf:type>sram_byte_cnt</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>num_overhead_bytes_stored</nf:name>
        <nf:description>Number of overhead (control) bytes stored</nf:description>
        <nf:type>sram_byte_cnt</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>num_pkt_bytes_removed</nf:name>
        <nf:description>Number of packet bytes removed</nf:description>
        <nf:type>sram_byte_cnt</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>num_overhead_bytes_removed</nf:name>
        <nf:description>Number of overhead (control) bytes removed</nf:description>
        <nf:type>sram_byte_cnt</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>num_pkts_stored</nf:name>
        <nf:description>Number of packets stored</nf:description>
        <nf:type>sram_pkt_cnt</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>num_pkts_dropped</nf:name>
        <nf:description>Number of packets dropped</nf:description>
        <nf:type>sram_pkt_cnt</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>num_pkts_removed</nf:name>
        <nf:description>Number of packets removed</nf:description>
        <nf:type>sram_pkt_cnt</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>addr_lo</nf:name>
        <nf:description>Queue low address</nf:description>
        <nf:type>sram_addr</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>addr_hi</nf:name>
        <nf:description>Queue high address</nf:description>
        <nf:type>sram_addr</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>rd_addr</nf:name>
        <nf:description>Queue read address</nf:description>
        <nf:type>sram_addr</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>wr_addr</nf:name>
        <nf:description>Queue write address</nf:description>
        <nf:type>sram_addr</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>num_pkts_in_q</nf:name>
        <nf:description>Number of packets in the queue</nf:description>
        <nf:type>sram_pkt_cnt</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>max_pkts_in_q</nf:name>
        <nf:description>Maximum number of packets allowed in queue</nf:description>
        <nf:type>sram_pkt_cnt</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>num_words_in_q</nf:name>
        <nf:description>Number of words in the queue</nf:description>
        <nf:type>sram_word_cnt</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>num_words_left</nf:name>
        <nf:description>Number of words of space left</nf:description>
        <nf:type>sram_word_cnt</nf:type>
      </nf:register>
      <nf:register>
        <nf:name>full_thresh</nf:name>
        <nf:description>Full threshold (minimum number of words)</nf:description>
        <nf:type>sram_word_cnt</nf:type>
      </nf:register>
    </nf:register_group>
  </nf:registers>
  <nf:constants>
    <nf:constant>
      <nf:name>:NUM_OUTPUT_QUEUES</nf:name>
      <nf:value>8</nf:value>
    </nf:constant>
    <nf:constant>
      <nf:name>DEFAULT_MAX_PKTS</nf:name>
      <nf:width>SRAM_PKT_CNT_WIDTH</nf:width>
      <nf:value>0xffffffff</nf:value>
    </nf:constant>
    <nf:constant>
      <nf:name>SRAM_PKT_CNT_WIDTH</nf:name>
      <nf:value>:SRAM_ADDR_WIDTH</nf:value>
    </nf:constant>
    <nf:constant>
      <nf:name>SRAM_WORD_CNT_WIDTH</nf:name>
      <nf:value>:SRAM_ADDR_WIDTH</nf:value>
    </nf:constant>
    <nf:constant>
      <nf:name>SRAM_BYTE_CNT_WIDTH</nf:name>
      <nf:value>:SRAM_ADDR_WIDTH</nf:value>
    </nf:constant>
    <nf:constant>
      <nf:name>ENABLE_SEND_BIT_NUM</nf:name>
      <nf:value>0</nf:value>
    </nf:constant>
    <nf:constant>
      <nf:name>INITIALIZE_OQ_BIT_NUM</nf:name>
      <nf:value>1</nf:value>
    </nf:constant>
  </nf:constants>
  <nf:types>
    <nf:type xsi:type="nf:SimpleType">
      <nf:name>sram_pkt_cnt</nf:name>
      <nf:width>SRAM_PKT_CNT_WIDTH</nf:width>
    </nf:type>
    <nf:type xsi:type="nf:SimpleType">
      <nf:name>sram_word_cnt</nf:name>
      <nf:width>SRAM_WORD_CNT_WIDTH</nf:width
    </nf:type>
    <nf:type xsi:type="nf:SimpleType">
      <nf:name>sram_byte_cnt</nf:name>
      <nf:width>SRAM_BYTE_CNT_WIDTH</nf:width>
    </nf:type>
      <nf:type xsi:type="nf:SimpleType">
      <nf:name>sram_addr</nf:name>
      <nf:width>:SRAM_ADDR_WIDTH</nf:width>
    </nf:type>
    <nf:type xsi:type="nf:SimpleType">
      <nf:name>oq_control</nf:name>
      <nf:width>2</nf:width>
      <nf:bitmask>
        <nf:name>enable_send</nf:name>
        <nf:pos>0</nf:pos>
      </nf:bitmask>
      <nf:bitmask>
        <nf:name>initialize_oq</nf:name>
        <nf:pos>1</nf:pos>
      </nf:bitmask>
    </nf:type>
  </nf:types>
  </nf:module>

Back End Scripts

The NetFPGA platform contains various backend scripts and Makefiles that simplify the process of creating, simulating, synthesizing, and running regression tests for a project. The three main backend scripts that developers should know about are discussed below.

Register Generation

Creating registers has been a complicated process in the past. To simplify the creation of registers and allow modules to be easily inserted into projects an XML based register backend was created. This backend utilizes module XML files (one file for each module that contains registers) and a project XML file (one project file for each project). Once these files are in place users can use the nf2_register_gen.pl script to generate the Verilog, C headers, and Perl headers needed for a project. The register generation script is automatically called from the simulation and synthesis Makefiles; however, a user can use the script manually to create the necessary register files. Below is an example of running the script for the reference router project. To run the script on a different project, just change the project parameter.

  nf2_register_gen.pl --project reference_router

Simulation

The Perl library that contains the simulation functions is called SimLib.pm and can be found under NF2/lib/Perl5.

Functions

• enable_interrupts
• prepare_DMA
• resetDevice
• cpu_rxfifo_rd_pkt
  • Description: Have the CPU read specified packet from the FPU RxFIFO
  • Parameters: $src_port, $length, $pkt_string, $delay
• PCI_sendpkt
  • Description: Send packet via the PCI bus (CPU TxFIFO)
  • Parameters: $port, $packet
• PCU_create_and_send_pkt
  • Description: Create and send a packet on the port specified
  • Parameters: $port, $length
• make_ethernet_pkt
  • Description: Make an ethernet packet
  • Parameters: $length, $dst_addr, $src_addr, $type
• make_IP_pkt
  • Description: Make an IP packet
  • parameters: $length, $dst_addr, $src_addr, $ttl, $dst_ip, $src_ip
• make_IP_IP_pkt
  • Description: Make an IP packet encapsulated in an IP packet
  • Parameters: $length, $dst_addr, $src_addr, $ttl, $dst_ip_tun, $src_ip_tun, $dst_ip, $src_ip
• make_RCP_pkt
  • Description: Create an RCP packet
  • Parameters: $length, $dst_addr, $src_addr, $ttl, $dst_ip, $src_ip, @RCP
  • @RCP = $fwd, $rev, $rtt, $proto

Regression Tests

References