Contents
|
The NetFPGA is a low-cost platform, primarily designed as a tool for teaching networking hardware and router design. It has also proved to be a useful tool for networking researchers. Through partnerships and donations from sponsor of the project, the NetFPGA is widely available to students, teachers, researchers, and anyone else interested in experimenting with new ideas in high-speed networking hardware.
At a high level, the board contains four 1 Gigabit/second Ethernet (GigE) interfaces, a user programmable Field Programmable Gate Array (FPGA), and four banks of locally-attached Static and Dynamic Random Access Memory (SRAM and DRAM). It has a standard PCI interface allowing it to be connected to a desktop PC or server. A reference design can be downloaded from the http://NetFPGA.org website that contains a hardware-accelerated Network Interface Card (NIC) or an Internet Protocol Version 4 (IPv4) router that can be readily configured into the NetFPGA hardware. The router kit allows the NetFPGA to interoperate with other IPv4 routers.
The NetFPGA offloads processing from a host processor. The host's CPU has access to main memory and can DMA to read and write registers and memories on the NetFPGA. Unlike other open-source projects, the NetFPGA provides a hardware-accelerated hardware datapath. The NetFPGA provides a direct hardware interface connected to four GigE ports and multiple banks of local memory installed on the card.
NetFPGA packages (NFPs) are available that contains source code (both for hardware and software) that implement networking functions. Using the reference router as an example, there are three main ways that a developer can use the NFP. In the first usage model, the default router hardware can be configured into the FPGA and the software can be modified to implement a custom protocol.
Another way to modify the NetFPGA is to start with the reference router and extend the design with a custom user module. Finally, it is also possible to implement a completely new design where the user can place their own logic and data processing functions directly in the FPGA.
A block diagram that shows the major components of NetFPGA platform is shown below.
The NetFPGA platform contains one large Xilinx Virtex2-Pro 50 FPGA which is programmed with user-defined logic and has a core clock that runs at 125MHz. The NetFPGA platform also contains one small Xilinx Spartan II FPGA holding the logic that implements the control logic for the PCI interface to the host processor.
Two 18 MBit external Cypress SRAMs are arranged in a configuration of 512k words by 36 bits (4.5 Mbytes total) and operate synchronously with the FPGA logic at 125 MHz. One bank of external Micron DDR2 SDRAM is arranged in a configuration of 16M words by 32 bits (64 MBytes total). Using both edges of a separate 200 MHz clock, the memory has a bandwidth of 400 MWords/second (1,600 MBytes/s = 12,800 Mbits/s).
The Broadcom Gigabit/second external physical-layer transceiver (PHY) sends packets over standard category 5, 5e, or 6 twisted-pair cables. The quad PHY interfaces with four Gigabit Ethernet Media Access Controllers (MACs) instantiated as a soft core on the FPGA. The NetFPGA also includes two interfaces with Serial ATA (SATA) connectors that enable multiple NetFPGA boards in a system to exchange traffic directly without use of the PCI bus.
Depending on your goals, you may find certain chapters of this guide more relevant than others.
If your task is to set up machines, start by reading the steps to obtain hardware and software, follow the steps to install software, then verify the software and hardware.
If you already have NetFPGA systems up and running in your laboratory and want to understand how it works, read the walkthroughs of the Reference Designs to understand the operation of the reference NIC, the software component of the router (SCONE), the router kit, the reference router hardware, and the buffer monitoring system.
We encourage feedback and discussion about the progress and problems with the NetFPGA. An bug-tracking system called Bugzilla is available to read about and post bugs. A email discussion group is available to discuss progress on the project.
We track and maintain bugs using BugZilla
We maintain a mailing list for discussion about NetFPGA topics. One list is used to communicate with the developers, while the other is used to communicate with the entire Beta user community.
We do not guarantee that any or all of the NetFPGA components will work for you. FPGAs allow for an enormous range of freedom in the implementation of circuits. We do not guarantee that anything you get from us will not damage the hardware on the NetFPGA, the software on the PC, or anything else. And finally we do not guarantee that you will get support. However, we do guarantee that we did/will do our best. Hence, the license.
The first thing to be done is putting the board in the the box and making sure it runs. To get started, you'll need to perform the following steps:
The NetFPGA boards can be obtained from a third-party company, Digilent Inc. The cards are sold for a discounted price when used for Educational purposes. They are also available for commercial use, but pricing is higher. Stanford University provided the open reference design to Digilent Inc., but is not involved in the sale of the hardware.
Complete NetFPGA systems can also be ordered on-line that include the NetFPGA hardware pre-installed in a host PC.
The easiest way to purchase hardware is to order on-line from Digilent
Academic institutions can order the hardware with a discount by placing a purchase order.
NetFPGAs host systems can be built from commodity, off-the-shelf (COTS) parts. The NetFPGA card fits into a standard PCI slot in a desktop or server-class PC. We have only tested the NetFPGA in a few of a few widely-available systems. The NetFPGA may work with other PCs as well, but we do not support such configurations.
There are currently multiple ways to obtain a NetFPGA host system:
To install a NetFPGA, you will need to open the case to your computer. To minimize the chance that you damage your computer or the NetFPGA module, we suggest that you wear an anti-static wrist strap when handing the hardware.
The most cost-effective way to build a high-performance NetFPGA host system is to purchase the components from on-line vendors and assemble your own machine. This effort is not for the faint of heart, however, as you will need to place multiple orders for components and have the time to assemble and test the PC. We assembled all of the nf-test machines at Stanford University. You can use the Bill of Materials (BOM) below to do the same.
Stanford constructed 11 PCs for North American tutorials; more info is available in the NF-TEST machines page. These PCs were built with the following components:
The total cost to build a single machine is:
The total cost to equip a laboratory with Qnty=11 PCs, cables (not including NetFPGAs)
Note that tax and shipping are not included in the above figures.
A pre-configured Dell 2950 2U Rackmount PC can be purchased from Dell. We have verified that the NetFPGA works in the PCI-X slot of the Dell 2950 2U Rack-mount server. The cost for a pre-built Dell server typically ranges from $3,000 to $5,000 depending on the configuration you select. Running the selftest requires purchasing a SATA cable and two Ethernet cables.
Note: When installing the NetFPGA in a system, it is important that the card is securely fastened to the chassis. In addition to locking in the faceplate at the front of the system, the card should also be locked in at the rear of the card using a mounting bracket.
Retainers to secure the back of the NetFPGA to a chassis are available from Gompf as: http://www.bracket.com/downloads/brackets/pdf/91060000AFG.pdf
The actual bracket required depends on the size of the chassis. http://www.bracket.com/retainerslist.asp
During shipment, the printed circuit board can vibrate or shake within the chasis causing mechanical damage. Systems should not be shipped with the NetFPGA card pre-installed.
A third-party vendor has just started building complete system with the NetFPGA hardware and software pre-installed. The complete turn-key system, including the NetFPGA card, are available from Accent Technolgy Inc.
http://www.accenttechnologyinc.com/product_details.php?category_id=0&item_id=1
During shipment, the printed circuit board can vibrate or shake within the chasis causing mechanical damage. Systems should not be shipped with the NetFPGA card pre-installed. Cards should be shipped separately from the chassis and installed on site to avoid damage.
The Beta release of the NetFPGA Package (NFP) contains the source code for gateware, system software, and regression tests. The NFP includes an IPv4 Router, a four-port NIC, an IPv4 Router with Output Queues Monitoring System, the PW-OSPF software that interacts with the IPv4 Router (SCONE), and the Router Kit which is a daemon that reflects the routing table and ARP cache from the Linux host to the IPv4 router on NetFPGA.
The instructions in this section have been superseded by the instructions in the Install Software 1.2 section below.
To download the Beta NFP:
The NFP currently comprises two tarballs:
Download the tarballs from http://NetFPGA.org/beta/distributions. Later, you will unpack them in the same directory.
An extended package that includes source code to the reference router is available to instructors of courses like Stanford's CS344 and researchers that need full source code to the router. The full source distribution is NOT available to the general public because course instructors that use the NetFPGA assign projects to implement components of the router.
For users that join the BetaPlus group, they must promise not to redistribute the source code. For the sake of educating future generations of networking students, it is critical that teachers and researchers that download this package do not redistribute it.
To qualify for access to the additional source code:
We support use of the popular Linux Distribution CentOS as the operating system for the Host PC. CentOS is a free variation of the popular RedHat distribution.
We have tested the NetFPGA with the 32-bit version of CentOS 4.4, CentOS 4.5, and CentOS 5.1 operating systems.
You can create your own bootable CentOS DVD by downloading an ISO [1]
Burn the image onto a DVD:
Install CentOS http://netfpga.org/CentOS_Install.pdf
Use of other operating systems is possible, but we do not support them.
For archival purposes the install instructions for the NetFPGA Package 1.0 can be found at Install_Software_1.0. Use the instructions below to install newest version NetFPGA package.
java -version
chmod +x jdk-6u6-linux-i586-rpm.bin
./jdk-6u6-linux-i586-rpm.bin
rpm --import http://jpackage.org/jpackage.asc
cd /etc/yum.repos.d wget http://www.jpackage.org/jpackage17.repo
yum -y --enablerepo=jpackage-generic-nonfree install java-1.6.0-sun-compat.i586
Dependencies Resolved ============================================================================= Package Arch Version Repository Size ============================================================================= Installing: java-1.6.0-sun-compat i586 1.6.0.06-1jpp jpackage-generic-nonfree 54 k Transaction Summary ============================================================================= Install 1 Package(s) Update 0 Package(s) Remove 0 Package(s) Total download size: 54 k Is this ok [y/N]: y Downloading Packages: (1/1): java-1.6.0-sun-com 100% |=========================| 54 kB 00:00 Running Transaction Test Finished Transaction Test Running Transaction Installing : java-1.6.0-sun-compat ######################### [1/1] Installed: java-1.6.0-sun-compat.i586 0:1.6.0.06-1jpp Complete!
/usr/sbin/alternatives --config java
There are 2 programs which provide 'java'.
Selection Command
-----------------------------------------------
1 /usr/lib/jvm/jre-1.4.2-gcj/bin/java
*+ 2 /usr/lib/jvm/jre-1.6.0-sun/bin/java
Enter to keep the current selection[+], or type selection number:
cat /etc/redhat-release
rpm -Uhv http://netfpga.org/yum/el4/RPMS/noarch/netfpga-repo-1-1_CentOS4.noarch.rpm
rpm -Uhv http://netfpga.org/yum/el5/RPMS/noarch/netfpga-repo-1-1_CentOS5.noarch.rpm
yum install netfpga-base
Run the following script to copy the entire NF2 directory into your account (typically: /root/NF2). WARNING: Running this command WILL overwrite any existing NF2 directory or files in your user account! If you have files that you want to preserve, 'mv' your NF2 directory to another location, such as NF2_backup.
To copy the NetFPGA directory and set the environment variables run the following command
/usr/local/NF2/lib/scripts/user_account_setup/user_account_setup.pl
It also adds the following environment variables to your .bashrc file.
Reboot your machine in order to finalize the installation.
We provide the Verilog source code the modules so that users can compile, simulate, and synthesize gateware for the NetFPGA. We have tested simulation and synthesis using a specific version of the Xilinx tools (as described below). Use of other versions of the tools (older or newer) is not supported. If you do not plan to rebuild the hardware circuits, you can skip installation of CAD tools.
cd ~/NF2/ make
make -C C make[1]: Entering directory `/home/gac1/temp/NF2/lib/C' make -C kernel make[2]: Entering directory `/home/gac1/temp/NF2/lib/C/kernel' make -C /lib/modules/2.6.9-55.0.9.ELsmp/build M=/home/gac1/temp/NF2/lib/C/kernel LDDINC=/home/gac1/temp/NF2/lib/C/kernel/../include modules make[3]: Entering directory `/usr/src/kernels/2.6.9-55.0.9.EL-smp-i686' Building modules, stage 2. MODPOST make[3]: Leaving directory `/usr/src/kernels/2.6.9-55.0.9.EL-smp-i686' make[2]: Leaving directory `/home/gac1/temp/NF2/lib/C/kernel' make -C download make[2]: Entering directory `/home/gac1/temp/NF2/lib/C/download' make -C ../common make[3]: Entering directory `/home/gac1/temp/NF2/lib/C/common' make[3]: Nothing to be done for `all'. make[3]: Leaving directory `/home/gac1/temp/NF2/lib/C/common' make[2]: Leaving directory `/home/gac1/temp/NF2/lib/C/download' make -C reg_access make[2]: Entering directory `/home/gac1/temp/NF2/lib/C/reg_access' make -C ../common make[3]: Entering directory `/home/gac1/temp/NF2/lib/C/common' make[3]: Nothing to be done for `all'. make[3]: Leaving directory `/home/gac1/temp/NF2/lib/C/common' make[2]: Leaving directory `/home/gac1/temp/NF2/lib/C/reg_access' make -C router make[2]: Entering directory `/home/gac1/temp/NF2/lib/C/router' gcc -lncurses cli.o ../common/nf2util.o ../common/util.o ../common/reg_defines.h -o cli gcc -lncurses regdump.o ../common/nf2util.o ../common/reg_defines.h -o regdump gcc -lncurses show_stats.o ../common/nf2util.o ../common/util.o ../common/reg_defines.h -o show_stats make[2]: Leaving directory `/home/gac1/temp/NF2/lib/C/router' make[1]: Leaving directory `/home/gac1/temp/NF2/lib/C' make -C scripts make[1]: Entering directory `/home/gac1/temp/NF2/lib/scripts' make -C cpci_reprogram make[2]: Entering directory `/home/gac1/temp/NF2/lib/scripts/cpci_reprogram' make[2]: Nothing to be done for `all'. make[2]: Leaving directory `/home/gac1/temp/NF2/lib/scripts/cpci_reprogram' make -C cpci_config_reg_access make[2]: Entering directory `/home/gac1/temp/NF2/lib/scripts/cpci_config_reg_access' make[2]: Nothing to be done for `all'. make[2]: Leaving directory `/home/gac1/temp/NF2/lib/scripts/cpci_config_reg_access' make[1]: Leaving directory `/home/gac1/temp/NF2/lib/scripts'
make install
for dir in lib bitfiles projects/scone/base projects/selftest/sw ; do \
make -C $dir install; \
done
make[1]: Entering directory `/home/gac1/temp/NF2/lib'
for dir in C scripts java/gui ; do \
make -C $dir install; \
done
make[2]: Entering directory `/home/gac1/temp/NF2/lib/C'
for dir in kernel download reg_access router ; do \
make -C $dir install; \
done
make[3]: Entering directory `/home/gac1/temp/NF2/lib/C/kernel'
make -C /lib/modules/2.6.9-55.0.9.ELsmp/build M=/home/gac1/temp/NF2/lib/C/kernel LDDINC=/home/gac1/temp/NF2/lib/C/kernel/../include modules
make[4]: Entering directory `/usr/src/kernels/2.6.9-55.0.9.EL-smp-i686'
Building modules, stage 2.
MODPOST
make[4]: Leaving directory `/usr/src/kernels/2.6.9-55.0.9.EL-smp-i686'
install -m 644 nf2.ko /lib/modules/`uname -r`/kernel/drivers/nf2.ko /sbin/depmod -a
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/C/kernel'
make[3]: Entering directory `/home/gac1/temp/NF2/lib/C/download'
install nf2_download /usr/local/bin
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/C/download'
make[3]: Entering directory `/home/gac1/temp/NF2/lib/C/reg_access'
install regread /usr/local/bin
install regwrite /usr/local/bin
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/C/reg_access'
make[3]: Entering directory `/home/gac1/temp/NF2/lib/C/router'
make[3]: Nothing to be done for `install'.
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/C/router'
make[2]: Leaving directory `/home/gac1/temp/NF2/lib/C'
make[2]: Entering directory `/home/gac1/temp/NF2/lib/scripts'
for dir in cpci_reprogram cpci_config_reg_access ; do \
make -C $dir install; \
done
make[3]: Entering directory `/home/gac1/temp/NF2/lib/scripts/cpci_reprogram'
install cpci_reprogram.pl /usr/local/sbin
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/scripts/cpci_reprogram'
make[3]: Entering directory `/home/gac1/temp/NF2/lib/scripts/cpci_config_reg_access'
install dumpregs.sh /usr/local/sbin
install loadregs.sh /usr/local/sbin
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/scripts/cpci_config_reg_access'
make[2]: Leaving directory `/home/gac1/temp/NF2/lib/scripts'
make[2]: Entering directory `/home/gac1/temp/NF2/lib/java/gui'
make[2]: Nothing to be done for `install'.
make[2]: Leaving directory `/home/gac1/temp/NF2/lib/java/gui'
make[1]: Leaving directory `/home/gac1/temp/NF2/lib'
make[1]: Entering directory `/home/gac1/temp/NF2/bitfiles'
for bitfile in CPCI_2.1.bit cpci_reprogrammer.bit ; do \
install -D -m 0644 $bitfile /usr/local/NF2/bitfiles/$bitfile ; \
done
make[1]: Leaving directory `/home/gac1/temp/NF2/bitfiles'
make[1]: Entering directory `/home/gac1/temp/NF2/projects/scone/base'
make[1]: Nothing to be done for `install'.
make[1]: Leaving directory `/home/gac1/temp/NF2/projects/scone/base'
make[1]: Entering directory `/home/gac1/temp/NF2/projects/selftest/sw'
make[1]: Nothing to be done for `install'.
make[1]: Leaving directory `/home/gac1/temp/NF2/projects/selftest/sw'
reboot
lsmod | grep nf2
nf2 28428 0
ifconfig -a | grep nf2
nf2c0 Link encap:Ethernet HWaddr 00:4E:46:32:43:00 nf2c1 Link encap:Ethernet HWaddr 00:4E:46:32:43:01 nf2c2 Link encap:Ethernet HWaddr 00:4E:46:32:43:02 nf2c3 Link encap:Ethernet HWaddr 00:4E:46:32:43:03
/usr/local/sbin/cpci_reprogram.pl --all (to reprogram all NetFPGAs in a system)
Loading the CPCI Reprogrammer on NetFPGA 0 Loading the CPCI on NetFPGA 0 CPCI on NetFPGA 0 has been successfully reprogrammed
The NetFPGA self-test is an FPGA bitfile and software that ensures that all of the components on your platform are fully functional. The self-test consists of both an FPGA bitfile that contains logic and interfaces to external components as well the software that displays the results. The self-test excercises all of the hardware in parallel. The test continues to run repeatedly until terminated by the user. The self-test was run at the factory just after the cards were manufactured. Cards are not distributed unless they completely pass all functions of the self-test process.
The self-test bitfile performs rigorous testing of the SRAM and DDR2 DRAM to ensure that all memory lines can be properly written to and read back with the same data. Multiple data patterns are used to ensure that no address or data lines have faults. The network test sends bursts of packets on the Ethernet interfaces and the loopback cables are put in place to that packets can be read and compared to the data that was transmitted. The SATA loopback test transmits data using the Multi-Gigabit I/O lines (MGIOs) to ensure that data can be reliably transmitted on the high-speed I/O interfaces. The DMA test exercises the PCI Controller (CPCI), the VirtexII, and the PCI bus to ensure that large blocks of data can be sent between the NetFPGA the host computer's memory. The selftest bitfile runs all of the tests above in parallel and continously runs until it is terminated. The self-test software displays the results of testing on a console.
We provide the self-test bitfile and the software to end-users so that the self-test can be run when the hardware is delivered. When you receive a NetFPGA card, we suggest that you run the self-test to ensure that the card is still fully functional and that the card works properly in your environment. Before running the self-test, be sure that you have connected the loopback cables as shown in the directions on how to set up a NetFPGA in your system. If the loopback cables are not connected, the self-test will correctly report that an interface appears non-functional.
The following instructions assume that you have successfully installed a NetFPGA card with CentOS 4.4. The selftest is an enhanced version of the test run on every NetFPGA at the factory by Digilent to verify proper hardware operation.
Install a SATA cable to loopback the board-to-board high-speed serial I/O.
Note: To minimize the chance that you damage your computer or the NetFPGA module, wear an anti-static wrist strap when handing the hardware.
Install two Ethernet cables as shown:
nf2_download ~/NF2/bitfiles/selftest.bit
~/NF2/projects/selftest/sw/selftest
~/NF2/projects/selftest/sw/selftest -n
Found net device: nf2c0 NetFPGA selftest 1.00 alpha Running..... PASSED
The regression test suite is a set of tests that exercise the functionality of the released gateware and software. On a fast machine, this test should take approximately 10 minutes.
The features exercised by regression test suite are the only features we will try to provide support for. Additional features might be available and functional in the released gateware, but they are not supported.
For more information on the features we support, as defined by tests, see the following:
Please make sure that you have successfully completed the selftest before proceeding to the regression tests.
nf2_download ~/NF2/bitfiles/reference_router.bit
Found net device: nf2c0 Bit file built from: nf2_top_par.ncd Part: 2vp50ff1152 Date: 2007/10/ 9 Time: 22: 3: 4 Error Registers: 1000000 Good, after resetting programming interface the FIFO is empty Download completed - 2377668 bytes. (expected 2377668). DONE went high - chip has been successfully programmed.
~/NF2/bin/nf21_regress_test.pl
Running tests on project 'driver'... Running test 'driver_compile'... PASS Running test 'driver_install'... PASS Running test 'verify_mtu'... PASS Running global teardown... PASS Running tests on project 'reference_nic'... Running test 'download_nic'... PASS Running test 'test_loopback_random'... PASS Running test 'test_loopback_minsize'... PASS Running test 'test_loopback_maxsize'... PASS Running test 'test_loopback_drop'... PASS Running test 'test_ip_interface'... PASS Running global teardown... PASS Running tests on project 'reference_router'... Running global setup... PASS Running test 'test_router_cpusend/run.pl'... PASS Running test 'test_wrong_dest_mac'... PASS Running test 'test_nonip_packet'... PASS Running test 'test_nonipv4_packet'... PASS Running test 'test_invalidttl_packet'... PASS Running test 'test_lpm_misses'... PASS Running test 'test_arp_misses'... PASS Running test 'test_badipchecksum_packet'... PASS Running test 'test_ipdest_filter_hit'... PASS Running test 'test_packet_forwarding'... PASS Running test 'test_lpm'... PASS Running test 'test_lpm_next_hop'... PASS Running test 'test_queue_overflow'... PASS Running test 'test_oq_limit'... PASS Running test 'test_ipdest_filter'... PASS Running test 'test_oq_sram_sz_cpu'... PASS Running test 'test_oq_sram_sz_mac'... PASS Running test 'test_router_table/run.pl'... PASS Running test 'test_send_rec/run.pl'... PASS Running test 'test_lut_forward'... PASS Running global teardown... PASS Running tests on project 'scone'... Running global setup... PASS Running test 'test_build'... PASS Running test 'test_mac_set'... PASS Running test 'test_ip_set'... PASS Running test 'test_rtable_set'... PASS Running test 'test_disabled_interfaces/run.pl'... PASS Running test 'test_noniparp_ethtype'... PASS Running test 'test_arp_rpl/run.pl'... PASS Running test 'test_arp_norpl/run.pl'... PASS Running test 'test_arp_quepkt/run.pl'... PASS Running test 'test_ip_error/run.pl'... PASS Running test 'test_ip_rtblmiss/run.pl'... PASS Running test 'test_ip_intfc/run.pl'... PASS Running test 'test_ip_checksum/run.pl'... PASS Running test 'test_ttl_expired/run.pl'... PASS Running test 'test_send_receive/run.pl'... PASS Running test 'test_arp_req/run.pl'... PASS Running test 'test_tcp_port/run.pl'... PASS Running test 'test_udp_packet/run.pl'... PASS Running test 'test_icmp_echo/run.pl'... PASS Running test 'test_icmp_notecho/run.pl'... PASS Running global teardown... PASS Running tests on project 'gui_scone'... Running global setup... PASS Running test 'test_main_frame'... PASS Running test 'test_routing_table'... PASS Running test 'test_arp_table'... PASS Running test 'test_port_config_table'... PASS Running global teardown... PASS Running tests on project 'router_kit'... Running global setup... PASS Running test 'test_00_make/run.sh'... PASS Running test 'test_01_ip_dst_filter/run.pl'... PASS Running test 'test_02_route_table/run.pl'... PASS Running test 'test_03_arp_table/run.pl'... PASS Running test 'test_04_ip_packets/run.pl'... PASS Running global teardown... PASS Running tests on project 'router_buffer_sizing'... Running global setup... PASS Running test 'test_time_stamp/run'... PASS Running test 'test_store_event/run'... PASS Running global teardown... PASS
If you installed the CAD tools, you should run this test to verify that you can build a new circuit. Skip this step if you do not plan to modify hardware.
Note: This step will take about 45-60 mins. This can be used to verify the setup of the machine for synthesis. You will need to have the NetFPGA Beta Plus package. The Beta (not Plus) package does not include the sources for this step.
cd ~/NF2/projects/reference_router/synth time make
ls | grep nf2_top_par.bit
nf2_top_par.bit
nf2_download nf2_top_par.bit
~/NF2/bin/nf21_regress_test.pl
Running tests on project 'driver'... Running test 'driver_compile'... PASS Running test 'driver_install'... PASS Running test 'verify_mtu'... PASS Running global teardown... PASS Running tests on project 'reference_nic'... Running test 'download_nic'... PASS Running test 'test_loopback_random'... PASS Running test 'test_loopback_minsize'... PASS Running test 'test_loopback_maxsize'... PASS Running test 'test_loopback_drop'... PASS Running test 'test_ip_interface'... PASS Running global teardown... PASS Running tests on project 'reference_router'... Running global setup... PASS Running test 'test_router_cpusend/run.pl'... PASS Running test 'test_wrong_dest_mac'... PASS Running test 'test_nonip_packet'... PASS Running test 'test_nonipv4_packet'... PASS Running test 'test_invalidttl_packet'... PASS Running test 'test_lpm_misses'... PASS Running test 'test_arp_misses'... PASS Running test 'test_badipchecksum_packet'... PASS Running test 'test_ipdest_filter_hit'... PASS Running test 'test_packet_forwarding'... PASS Running test 'test_lpm'... PASS Running test 'test_lpm_next_hop'... PASS Running test 'test_queue_overflow'... PASS Running test 'test_oq_limit'... PASS Running test 'test_ipdest_filter'... PASS Running test 'test_oq_sram_sz_cpu'... PASS Running test 'test_oq_sram_sz_mac'... PASS Running test 'test_router_table/run.pl'... PASS Running test 'test_send_rec/run.pl'... PASS Running test 'test_lut_forward'... PASS Running global teardown... PASS Running tests on project 'scone'... Running global setup... PASS Running test 'test_build'... PASS Running test 'test_mac_set'... PASS Running test 'test_ip_set'... PASS Running test 'test_rtable_set'... PASS Running test 'test_disabled_interfaces/run.pl'... PASS Running test 'test_noniparp_ethtype'... PASS Running test 'test_arp_rpl/run.pl'... PASS Running test 'test_arp_norpl/run.pl'... PASS Running test 'test_arp_quepkt/run.pl'... PASS Running test 'test_ip_error/run.pl'... PASS Running test 'test_ip_rtblmiss/run.pl'... PASS Running test 'test_ip_intfc/run.pl'... PASS Running test 'test_ip_checksum/run.pl'... PASS Running test 'test_ttl_expired/run.pl'... PASS Running test 'test_send_receive/run.pl'... PASS Running test 'test_arp_req/run.pl'... PASS Running test 'test_tcp_port/run.pl'... PASS Running test 'test_udp_packet/run.pl'... PASS Running test 'test_icmp_echo/run.pl'... PASS Running test 'test_icmp_notecho/run.pl'... PASS Running global teardown... PASS Running tests on project 'gui_scone'... Running global setup... PASS Running test 'test_main_frame'... PASS Running test 'test_routing_table'... PASS Running test 'test_arp_table'... PASS Running test 'test_port_config_table'... PASS Running global teardown... PASS Running tests on project 'router_kit'... Running global setup... PASS Running test 'test_00_make/run.sh'... PASS Running test 'test_01_ip_dst_filter/run.pl'... PASS Running test 'test_02_route_table/run.pl'... PASS Running test 'test_03_arp_table/run.pl'... PASS Running test 'test_04_ip_packets/run.pl'... PASS Running global teardown... PASS Running tests on project 'router_buffer_sizing'... Running global setup... PASS Running test 'test_time_stamp/run'... PASS Running test 'test_store_event/run'... PASS Running global teardown... PASS
This section describes some walkthroughs to help in using the distributed source code. Each walkthrough will deal with a different aspect of using the reference designs. Some sections will be marked as optional reading. These sections explain what is going on behind the scenes to put things together and are not within the main focus of the walkthrough. The walkthroughs are the following:
The reference NIC walkthrough will go through an example of using some of the tools that are distributed with the release, and an example of how to write a simple C program to interface to the hardware.
Assuming the regression tests have been completed successfully and the NFP is installed correctly, we can now proceed to use one of the distributed projects: the NIC. In the rest of the exercises, we assume that NFP is installed in the user's home directory. Replace the '~' with the full path to the installation location if not. To run the tools, IP addresses must be assigned to the nf2cX interfaces. To do that, run the following command as root after replacing all the x's:
/sbin/ifconfig nf2cX x.x.x.x
Next, we need to download the NIC bitfile onto the NetFPGA. As root, run the following:
~/NF2/lib/C/download/nf2_download ~/NF2/bitfiles/reference_nic.bit
You should see output similar to the following:
Found net device: nf2c0 Bit file built from: nf2_top_par.ncd Part: 2vp50ff1152 Date: 2007/11/21 Time: 11: 0: 3 Error Registers: 1000000 Good, after resetting programming interface the FIFO is empty Download completed - 2377668 bytes. (expected 2377668). DONE went high - chip has been successfully programmed.
To compile the utilities type:
cd ~/NF2/lib/C/nic make
One of the built tools is called counterdump. This simple tool reads several hardware counters and dumps the counts to the terminal. To use it, type:
./counterdump
You should see an output similar to this:
Found net device: nf2c0 Num pkts received on port 0: 0 Num pkts dropped (rx queue 0 full): 0 Num pkts dropped (bad fcs q 0): 0 Num bytes received on port 0: 0 Num pkts sent from port 0: 0 Num bytes sent from port 0: 0 Num pkts received on port 1: 0 Num pkts dropped (rx queue 1 full): 0 Num pkts dropped (bad fcs q 1): 0 Num bytes received on port 1: 0 Num pkts sent from port 1: 0 Num bytes sent from port 1: 0 Num pkts received on port 2: 0 Num pkts dropped (rx queue 2 full): 0 Num pkts dropped (bad fcs q 2): 0 Num bytes received on port 2: 0 Num pkts sent from port 2: 0 Num bytes sent from port 2: 0 Num pkts received on port 3: 0 Num pkts dropped (rx queue 3 full): 0 Num pkts dropped (bad fcs q 3): 0 Num bytes received on port 3: 0 Num pkts sent from port 3: 0 Num bytes sent from port 3: 0
Now let's try to send and receive some packets and check the counter outputs again. One of the tools that are included in the NFP is a tool called send_pkts. This tool can send arbitrary Ethernet packets from any given port, but you will need root access to use it. To use this tool:
cd ~/NF2/lib/C/tools/send_pkts make
If everything goes correctly, you should see an output similar to this:
gcc `libnet-config --defines --cflags` -O2 -o send_pkts send_pkts.c `libnet-config --libs` -L/usr/lib-lnet -lpcap --static
For the next part, we will test sending a few packets from one of the ports. To send packets, issue the following commands:
cd ~/NF2/lib/C/tools/send_pkts sudo ./send_pkts -i nf2c0 -s 10 -l 100
The last command sends 10 100-byte packets out of port 0 on the NetFPGA (port 0 is the port closest to the PCI connector on the NetFPGA). If you have a machine connected to the same LAN section as that port, you should be able to capture the packets using Wireshark.
Now check the counters again:
~/NF2/lib/C/nic/counterdump
You should see an output similar to this:
Found net device: nf2c0 Num pkts received on port 0: 0 Num pkts dropped (rx queue 0 full): 0 Num pkts dropped (bad fcs q 0): 0 Num bytes received on port 0: 0 Num pkts sent from port 0: 10 Num bytes sent from port 0: 1000 Num pkts received on port 1: 0 Num pkts dropped (rx queue 1 full): 0 Num pkts dropped (bad fcs q 1): 0 Num bytes received on port 1: 0 Num pkts sent from port 1: 0 Num bytes sent from port 1: 0 Num pkts received on port 2: 0 Num pkts dropped (rx queue 2 full): 0 Num pkts dropped (bad fcs q 2): 0 Num bytes received on port 2: 0 Num pkts sent from port 2: 0 Num bytes sent from port 2: 0 Num pkts received on port 3: 0 Num pkts dropped (rx queue 3 full): 0 Num pkts dropped (bad fcs q 3): 0 Num bytes received on port 3: 0 Num pkts sent from port 3: 0 Num bytes sent from port 3: 0
If your NetFPGA ports are not connected to a quiet network, then you'll probably see different results.
The counters that have been dumped using counterdump are actually memory-mapped I/O registers. These are counters that exist in the NetFPGA FPGA hardware and are exported via the PCI interface. The software uses ioctl calls to do reads and writes into these registers. The ioctl calls are wrapped in two simple functions readReg and writeReg. In this section, we will open up counterdump.c and understand the software/hardware interface. counterdump.c is shown below with line numbers for reference.
1 /****************************************************************************
2 * vim:set shiftwidth=2 softtabstop=2 expandtab:
3 * $Id$
4 *
5 * Module: counterdump.c
6 * Project: NetFPGA NIC
7 * Description: dumps the MAC Rx/Tx counters to stdout
8 * Author: Jad Naous
9 *
10 * Change history:
11 *
12 */
13
14 #include <stdio.h>
15 #include <stdlib.h>
16 #include <unistd.h>
17
18 #include <net/if.h>
19
20 #include "../common/reg_defines.h"
21 #include "../common/nf2.h"
22 #include "../common/nf2util.h"
23
24 #define PATHLEN 80
25
26 #define DEFAULT_IFACE "nf2c0"
27
28 /* Global vars */
29 static struct nf2device nf2;
30
31 /* Function declarations */
32 void dumpCounts();
33 void processArgs (int , char **);
34 void usage (void);
35
36 int main(int argc, char *argv[])
37 {
38 nf2.device_name = DEFAULT_IFACE;
39
40 processArgs(argc, argv);
41
42 // Open the interface if possible
43 if (check_iface(&nf2))
44 {
45 exit(1);
46 }
47 if (openDescriptor(&nf2))
48 {
49 exit(1);
50 }
51
52 dumpCounts();
53
54 closeDescriptor(&nf2);
55
56 return 0;
57 }
58
59 void dumpCounts()
60 {
61 unsigned val;
62
63 readReg(&nf2, RX_QUEUE_0_NUM_PKTS_STORED_REG, &val);
64 printf("Num pkts received on port 0: %u\n", val);
65 readReg(&nf2, RX_QUEUE_0_NUM_PKTS_DROPPED_FULL_REG, &val);
66 printf("Num pkts dropped (rx queue 0 full): %u\n", val);
67 readReg(&nf2, RX_QUEUE_0_NUM_PKTS_DROPPED_BAD_REG, &val);
68 printf("Num pkts dropped (bad fcs q 0): %u\n", val);
69 readReg(&nf2, RX_QUEUE_0_NUM_BYTES_PUSHED_REG, &val);
70 printf("Num bytes received on port 0: %u\n", val);
71 readReg(&nf2, TX_QUEUE_0_NUM_PKTS_SENT_REG, &val);
72 printf("Num pkts sent from port 0: %u\n", val);
73 readReg(&nf2, TX_QUEUE_0_NUM_BYTES_PUSHED_REG, &val);
74 printf("Num bytes sent from port 0: %u\n\n", val);
75
76 readReg(&nf2, RX_QUEUE_1_NUM_PKTS_STORED_REG, &val);
77 printf("Num pkts received on port 1: %u\n", val);
78 readReg(&nf2, RX_QUEUE_1_NUM_PKTS_DROPPED_FULL_REG, &val);
79 printf("Num pkts dropped (rx queue 1 full): %u\n", val);
80 readReg(&nf2, RX_QUEUE_1_NUM_PKTS_DROPPED_BAD_REG, &val);
81 printf("Num pkts dropped (bad fcs q 1): %u\n", val);
82 readReg(&nf2, RX_QUEUE_1_NUM_BYTES_PUSHED_REG, &val);
83 printf("Num bytes received on port 1: %u\n", val);
84 readReg(&nf2, TX_QUEUE_1_NUM_PKTS_SENT_REG, &val);
85 printf("Num pkts sent from port 1: %u\n", val);
86 readReg(&nf2, TX_QUEUE_1_NUM_BYTES_PUSHED_REG, &val);
87 printf("Num bytes sent from port 1: %u\n\n", val);
88
89 readReg(&nf2, RX_QUEUE_2_NUM_PKTS_STORED_REG, &val);
90 printf("Num pkts received on port 2: %u\n", val);
91 readReg(&nf2, RX_QUEUE_2_NUM_PKTS_DROPPED_FULL_REG, &val);
92 printf("Num pkts dropped (rx queue 2 full): %u\n", val);
93 readReg(&nf2, RX_QUEUE_2_NUM_PKTS_DROPPED_BAD_REG, &val);
94 printf("Num pkts dropped (bad fcs q 2): %u\n", val);
95 readReg(&nf2, RX_QUEUE_2_NUM_BYTES_PUSHED_REG, &val);
96 printf("Num bytes received on port 2: %u\n", val);
97 readReg(&nf2, TX_QUEUE_2_NUM_PKTS_SENT_REG, &val);
98 printf("Num pkts sent from port 2: %u\n", val);
99 readReg(&nf2, TX_QUEUE_2_NUM_BYTES_PUSHED_REG, &val);
100 printf("Num bytes sent from port 2: %u\n\n", val);
101
102 readReg(&nf2, RX_QUEUE_3_NUM_PKTS_STORED_REG, &val);
103 printf("Num pkts received on port 3: %u\n", val);
104 readReg(&nf2, RX_QUEUE_3_NUM_PKTS_DROPPED_FULL_REG, &val);
105 printf("Num pkts dropped (rx queue 3 full): %u\n", val);
106 readReg(&nf2, RX_QUEUE_3_NUM_PKTS_DROPPED_BAD_REG, &val);
107 printf("Num pkts dropped (bad fcs q 3): %u\n", val);
108 readReg(&nf2, RX_QUEUE_3_NUM_BYTES_PUSHED_REG, &val);
109 printf("Num bytes received on port 3: %u\n", val);
110 readReg(&nf2, TX_QUEUE_3_NUM_PKTS_SENT_REG, &val);
111 printf("Num pkts sent from port 3: %u\n", val);
112 readReg(&nf2, TX_QUEUE_3_NUM_BYTES_PUSHED_REG, &val);
113 printf("Num bytes sent from port 3: %u\n\n", val);
114 }
115
116 /*
117 * Process the arguments.
118 */
119 void processArgs (int argc, char **argv )
120 {
121 char c;
122
123 /* don't want getopt to moan - I can do that just fine thanks! */
124 opterr = 0;
125
126 while ((c = getopt (argc, argv, "i:h")) != -1)
127 {
128 switch (c)
129 {
130 case 'i': /* interface name */
131 nf2.device_name = optarg;
132 break;
133 case '?':
134 if (isprint (optopt))
135 fprintf (stderr, "Unknown option `-%c'.\n", optopt);
136 else
137 fprintf (stderr,
138 "Unknown option character `\\x%x'.\n",
139 optopt);
140 case 'h':
141 default:
142 usage();
143 exit(1);
144 }
145 }
146 }
147
148
149 /*
150 * Describe usage of this program.
151 */
152 void usage (void)
153 {
154 printf("Usage: ./counterdump <options> \n\n");
155 printf("Options: -i <iface> : interface name (default nf2c0)\n");
156 printf(" -h : Print this message and exit.\n");
157 }
Let's go through the code.
20 #include "../common/reg_defines.h" 21 #include "../common/nf2.h" 22 #include "../common/nf2util.h"
Line 20 includes the header file that contains all the register addresses on the NetFPGA. This is needed to refer to register addresses as constant names rather than numeric addresses. Lines 21 and 22 include macros to access registers. These functions are used later in the code.
29 static struct nf2device nf2;
The nf2 struct will hold information about the device we are trying to access.
Now let's go into our main function.
38 nf2.device_name = DEFAULT_IFACE;
Set a default name for the device we are trying to access. This is useful so that the user of this program doesn't have to specify the name if she is using the default device (which is true in most cases).
40 processArgs(argc, argv);
Parses the command line options. In this simple program, the only command line option is to change the name of the interface we are trying to access.
43 if (check_iface(&nf2))
Checks that the interface exists and can be reached.
47 if (openDescriptor(&nf2))
Tries to open the interface for reading/writing using ioctl calls. The interface has to be up and assigned an IP address for a non-root user to be able to access it using ioctl calls.
52 dumpCounts();
calls the function to dump all the counts.
54 closeDescriptor(&nf2);
Closes the interface after we are done using it to be polite.
Reading and writing registers uses two functions:
int readReg(nf2device *nf2, unsigned int addr, unsigned int *val)
addr from device nf2 and writes the value in *val. Returns 1 on fail, 0 on success.
int writeReg(nf2device *nf2, unsigned int addr, unsigned int val)
val into the register at address addr from device nf2 . Returns 1 on fail, 0 on success.
As an example we will look at two lines in the dumpCounts() function. The rest of the lines are similar:
63 readReg(&nf2, RX_QUEUE_0_NUM_PKTS_STORED_REG, &val);
64 printf("Num pkts received on port 0: %u\n", val);
Line 63 reads the number of packets received into Rx Queue 0 into val, and line 64 prints out the result.
The registers available for access are documented in the Register Map. Unfortunately, registers keep getting added and removed as the design evolves, so the most current list of registers is in the reg_defines.h file that defines the register addresses. This file is generated automatically from the Verilog source code when it is simulated so it always has the most recent list of registers and their addresses. The registers in the Register Map are divided into groups corresponding to the modules described in the Verilog. The next section will go into more details on these modules.
Next, we will modify this file to dump the device ID which is assigned at implementation time. To do that, open file NF2/lib/C/common/reg_defines.h file and copy the macro that defines the device ID to replace the XXXX in the following lines, and copy the lines into the start of dumpCounts() function after the declaration of val.
readReg(&nf2, XXXX, &val);
printf("Device ID: %u\n\n", val);
Then after saving the file, type make in the directory containing counterdump.c and run the program again. You should now see an output similar to the following:
Found net device: nf2c0 Device ID: 1
Num pkts received on port 0: 0 Num pkts dropped (rx queue 0 full): 0 Num pkts dropped (bad fcs q 0): 0 Num bytes received on port 0: 0 Num pkts sent from port 0: 10 Num bytes sent from port 0: 1000 Num pkts received on port 1: 0 Num pkts dropped (rx queue 1 full): 0 Num pkts dropped (bad fcs q 1): 0 Num bytes received on port 1: 0 Num pkts sent from port 1: 0 Num bytes sent from port 1: 0 Num pkts received on port 2: 0 Num pkts dropped (rx queue 2 full): 0 Num pkts dropped (bad fcs q 2): 0 Num bytes received on port 2: 0 Num pkts sent from port 2: 0 Num bytes sent from port 2: 0 Num pkts received on port 3: 0 Num pkts dropped (rx queue 3 full): 0 Num pkts dropped (bad fcs q 3): 0 Num bytes received on port 3: 0 Num pkts sent from port 3: 0 Num bytes sent from port 3: 0
The division of the hardware into modules was hinted at in the previous section. Understanding these modules is essential in making the most of the available designs. The distributed projects in the NFP, including the NIC, all follow the same modular structure. This design is a pipeline where each stage is a separate module. A diagram of the pipeline is shown on the right.
The first stage in the pipeline consists of several queues which we call the Rx queues. These queues receive packets from IO ports such as the Ethernet ports and the PCI over DMA and provide a unified interface to the rest of the system. These ports are connected into a wrapper called the User Data Path which contains the processing stages. The current design (version 1.0 Beta) has 4 Ethernet Rx queues and 4 CPU DMA queues. The Ethernet and DMA queues are interleaved so that to the User Data Path, Rx Queue 0 is Ethernet Port 0, Rx Queue 1 is DMA port 0, Rx Queue 2 is Ethernet Port 1, and so on. Packets that arrive into CPU DMA Rx Queue X are packets that have been sent by the software out of interface nf2cX.
In the User Data Path, the first module a packet passes through is the Input Arbiter. The input arbiter decides which Rx queue to service next, and pulls the packet from that Rx queue and hands it to the next module in the pipeline: The output port lookup module. The output port lookup module is responsible for deciding which port a packet goes out of. After that decision is made, the packet is then handed to the output queues module which stores the packet in the output queues corresponding to the output port until the Tx queue is ready to accept the packet for transmission.
The Tx queues are analogous to the Rx queues and they send packets out of the IO ports instead of receiving. Tx queues are also interleaved so that packets sent out of User Data Path port 0 are sent to Ethernet Tx queue 0, and packets sent out of User Data Path port 1 are sent to CPU DMA Tx queue 0, and so on. Packets that are handed to DMA Tx queue X pop out of interface nf2cX. For more information on these modules, you can go here.
For almost each of these modules, there is a set of registers that access status information and set control signals for the module. These registers are described in the Register Map.
The router SCONE (Software Component Of NetFPGA) is a user level router that performs IPv4 forwarding, handles ARPs and various ICMP messages, has telnet (port 23) and web (port 8080) interfaces to handle router control, and also implements a subset of OSPF named PW-OSPF. SCONE mirrors a copy of its MAC addresses, IP addresses, routing table, and ARP table to the NetFPGA card which hardware accelerates the forwarding path.
To modify the way SCONE operates after launch, use the telnet command line interface, or the web interface. The web interface supports all the commands available from the telnet interface, and provides AJAX style capabilities to keep the data current as the router runs. To use either of these interfaces, connect to one of the IP addresses specified in the cpuhw file on port 23 for telnet, or port 8080 for http. To get a list of commands using telnet issue the ? command. A graphical version of this printout is available from the Command List link of the web interface. For help with a command enter the command and put ? for its arguments. IE 'show ip ?'. Many of the commands are documented in the Required Commands section here.
For an incoming packet to take the hardware forwarding path the packet must get a hit on the destination MAC address table, the routing table, and the ARP table for the next hop. SCONE builds and maintains the routing table containing static and dynamic routes, and the ARP table in software. When changes are detected in the software tables, SCONE copies them down into the NetFPGA's hardware tables. If you want packets to be passed up to software that match the IP addresses assigned to the interfaces you must also push these IP addresses down into a hardware table. Following are some code snippets for writing to these various tables. In general to write you first write the registers representing the data for the row, then you write the row number into the corresponding WR register. To read a row, first write the row number you want into the RD register, then read the data from the corresponding registers.
uint8_t mac_addr[6]; unsigned int mac_hi = 0; unsigned int mac_lo = 0; mac_hi |= ((unsigned int)mac_addr[0]) << 8; mac_hi |= ((unsigned int)mac_addr[1]); mac_lo |= ((unsigned int)mac_addr[2]) << 24; mac_lo |= ((unsigned int)mac_addr[3]) << 16; mac_lo |= ((unsigned int)mac_addr[4]) << 8; mac_lo |= ((unsigned int)mac_addr[5]); writeReg(&netfpga, ROUTER_OP_LUT_MAC_0_HI_REG, mac_hi); writeReg(&netfpga, ROUTER_OP_LUT_MAC_0_LO_REG, mac_lo); // 1,2,3 can be substituted in for 0 to set the MAC for the other ports
NOTE: One confusing aspect of using a user level process to control the router is that the MAC addresses of the router will not match the kernel MAC addresses. That is, the MAC addresses of the nf2cX interfaces do not necessarily reflect those that are actually in the hardware. The nf2cX interfaces are software independent entities that do not necessarily reflect the state of the hardware. In the next section we will describe the Router Kit which reflects the Linux's view of the network to HCORR (Hardware Component of Reference Router). This includes the ports' MAC addresses, IP addresses, the routing table, and the ARP cache.
The HCORR has a table that stores IP addresses called the destination IP filter table. Packets whose destination IP address matches an entry in this table will be sent to the hardware. The number of entries in this table can be found in the Register Map. We use this table to filter out the IP addresses of the router and the IP addresses of PW-OSPF multicast packets. We write into the table as follows:
struct in_addr ip; writeReg(&rs->netfpga, ROUTER_OP_LUT_DST_IP_FILTER_IP_REG, ntohl(ip.s_addr)); // i is the row to write the IP address to writeReg(&rs->netfpga, ROUTER_OP_LUT_DST_IP_FILTER_WR_ADDR_REG, i);
Writing to the Routing table is similar to writing to the Destination IP filter table. Note that the output port(s) is specified in one-hot-encoded format corresponding to the output port of the User Data Path to the Tx Queues. The function getOneHotPortNumber returns the values (in decimal): 1 , 4, 16, 64 which correspond to physical output ports 0-3 on the NetFPGA card. You can also tell a route to specifically send to software by specifying 2, 8, 32, 128 corresponding to the nf2c0,1,2,3 interfaces. To send out of MAC ports 0 and 1, write 1+4=5 as the output port entry.
struct in_addr ip, mask, gw; int i; char* iface; /* write the ip */ writeReg(&netfpga, ROUTER_OP_LUT_RT_IP_REG, ntohl(ip.s_addr)); /* write the mask */ writeReg(&netfpga, ROUTER_OP_LUT_RT_MASK_REG, ntohl(mask.s_addr)); /* write the next hop */ writeReg(&netfpga, ROUTER_OP_LUT_RT_NEXT_HOP_IP_REG, ntohl(gw.s_addr)); /* write the port */ writeReg(&netfpga, ROUTER_OP_LUT_RT_OUTPUT_PORT_REG, getOneHotPortNumber(iface)); /* write the row number */ writeReg(&netfpga, ROUTER_OP_LUT_RT_LUT_WR_ADDR_REG, i);
uint8_t mac_addr[6]; unsigned int mac_hi = 0; unsigned int mac_lo = 0; struct in_addr ip; int i; mac_hi |= ((unsigned int)mac_addr[0]) << 8; mac_hi |= ((unsigned int)mac_addr[1]); mac_lo |= ((unsigned int)mac_addr[2]) << 24; mac_lo |= ((unsigned int)mac_addr[3]) << 16; mac_lo |= ((unsigned int)mac_addr[4]) << 8; mac_lo |= ((unsigned int)mac_addr[5]); writeReg(&netfpga, ROUTER_OP_LUT_ARP_MAC_HI_REG, mac_hi); writeReg(&netfpga, ROUTER_OP_LUT_ARP_MAC_LO_REG, mac_lo); /* write the next hop ip data */ writeReg(&netfpga, ROUTER_OP_LUT_ARP_NEXT_HOP_IP_REG, ntohl(ip.s_addr)); /* set the row */ writeReg(&netfpga, ROUTER_OP_LUT_ARP_LUT_WR_ADDR_REG, i);
Router Kit is a simple approach to providing hardware acceleration to an unmodified Linux system. It is comprised of a single program, rkd (Router Kit Daemon), which monitors the Linux routing table and ARP cache and mirrors it down to the NetFPGA IPv4 reference router implementation.
rkd [-h|--help} [-d|--daemon} [-i|--interval] <in_ms>
rkd should work from the command line without any external configuration options. Simply run (./rkd). To run the process in the background use -d. You may specify the polling time in milliseconds using the -i option.
Router Kit is only useful on a Linux host with NetFPGA2 installed, and the ipv4 reference router bitfile loaded. Given this setup each port on the NetFPGA2 card is available to Linux via a nfc* interface (i.e. nfc0, nfc1, nfc2, and nfc3 assuming a single card is installed).
rkd will attempt to mirror all ARP cache and routing table entries associate with a NetFPGA interface into hardware. This provides a very simple (and familiar) method of of adding entries to the hardware. For example, to add a static ARP entry, simply use the arp(8) command. The following command will add a static ARP entry.
arp -s 1.2.3.4 ca:fe:de:ad:d0:d0 -i nfc0
To add an entry into the routing table, use route(8) (or ip(8)). For example, adding a default entry with a next hop of 10.0.0.1 out of the first port would look something like:
route add default gw 10.0.0.1 dev nfc0
Router kit is not limited to manual manipulation from the command line. All state (including dynamic state) is mirrored. To wit, running rkd alongside a standard routing daemon, such as XoRP, or Zebra/Quagga, should provide hardware acceleration of the forwarding table without any further configuration (provided the routing software is using the NetFPGA interfaces for forwarding).
rkd continuously polls the routing table and ARP cache state from /proc/net/route and /proc/net/arp respectively. When a change in state is detected, ./rkd writes the updated state to the NetFPGA through the register interface. All traffic not handled by the hardware is DMA'd to software where it is processed by the Linux kernel.
In this section we will go through the available tools to communicate with the Hardware Component of the Reference Router (HardCORR) and go through the process of modifying the design, simulating it, and finally implementing it. The tools that we will go over quickly are a Java GUI and a Standalone Command Line Interface (CLI).
The Java GUI allows the user to change entries in the Routing Table and ARP cache as well as the router's MAC and IP addresses. It also provides updates on counter values and graphs of throughput and much more. The GUI has a part that is written in C that provides the interface between the Java binaries and the driver. This native library is compiled from the nf2util.c file that contains the readReg and writeReg utilities used in the previous section. The library connects to the GUI using the Java Native Access (jna) library.
To build the GUI, first make sure that you have Sun's Java Development Kit (version >=1.6.0) installed and make sure the java, javac, and jar binaries are in your path (otherwise, you can edit the Makefile under lib/java/gui to reflect the actual locations). Then cd into NF2/lib/java/gui and type make clean. Then type make. You should get output similar to below:
make[1]: Entering directory `/home/jnaous/NF2/lib/C/common' gcc -fpic -c nf2util.c gcc -shared nf2util.o -o libnf2.so make[1]: Leaving directory `/home/jnaous/NF2/lib/C/common' Building java... Done Writing router gui manifest... Building router jar... Writing script to start router gui... Writing event capturing router gui manifest... Building event capturing router jar... Writing script to start event capturing router gui...
To run the GUI for the router, cd into NF2/lib/java/gui and type ./router.sh. The GUI should pop-up. The GUI constantly polls the data it reads from the hardware. To make updates faster, you can change the update rate under the File menu at the top left.
The Quickstart Panel provides a summary of things that can be done with the hardware. There is also a tab for viewing statistics and a tab for details. The details page will show the data path pipeline we saw earlier under the Reference NIC Walkthrough with clickable buttons. Clicking on those buttons will open up new buttons with more details and configuration settings.
A standalone small CLI that allows you to change routing table entries, ARP cache entries, and other settings is also provided in case the user doesn't want to run SCONE. The CLI is under NF2/lib/C/router. To build it, type make in that directory. You should see output similar to the following:
gcc -g -c -o cli.o cli.c gcc -g -c -o ../common/util.o ../common/util.c gcc -lncurses cli.o ../common/nf2util.o ../common/util.o ../common/reg_defines.h -o cli gcc -g -c -o regdump.o regdump.c gcc -lncurses regdump.o ../common/nf2util.o ../common/reg_defines.h -o regdump gcc -g -c -o show_stats.o show_stats.c gcc -lncurses show_stats.o ../common/nf2util.o ../common/util.o ../common/reg_defines.h -o show_stats
For help on using the CLI, start it by typing ./cli and then type help in the CLI.
We invite you to extend this CLI and any of our software tools and contribute them back so we can expand our library and make it easier for anybody to use NetFPGA.
This section will guide you through the process of creating your own project based on the reference router and adding a library module in the data path that would limit the rate at which the NetFPGA output packets on a particular port. We will first go through an overview of the router design and the interface between modules. Then we will explain how to add a library module into the design and put it in the data path. Following that, we will go through verifying the design by simulation and finally implementing it as a downloadable bitfile.
Hopefully you still remember the reference pipeline and what each major module in the pipeline does. We show it again here for reference. Please go over it again if you need to know what each module does.
The user data path is 64-bits wide running at 125MHz. This would give the data path a peak bandwidth of 8Gbps. Packets are passed between modules using a fifo-like simple push interface with four signals: WR, RDY, DATA, and CTRL. The WR and RDY signals are each one bit wide while the CTRL is 8 bits wide and the DATA is 64 bits wide. Say module i wants to push a packet to module i+1. When i+1 is ready to accept some data, it will assert the RDY signal. Module i then puts the packet data on the DATA bus, sets the CTRL bus according to the data it is transmitting (the values of the CTRL bus are discussed later), and raises the WR signal whenever it is ready to send. Module i+1 should be ready to latch the data transmitted the next clock cycle after asserting the RDY signal. If module i+1 cannot accept anymore data, it will deassert the RDY signal at least one clock cycle before the data should stop arriving. The figure below demonstrates a valid transaction.
