ICING

ICING is an architecture that upholds consent-to-connectivity. This is a how-to on using the NetFPGA hardware for it.

Show Contents...Hide Contents...

Introduction to NetFPGA

NetFPGA is a PCI card that has an FPGA (programmable hardware) and 4 GigE ports. There are two ways the software communicates with the hardware:

1. DMA
2. Register access

The NetFPGA driver creates four nf2cX Ethernet interfaces. Packets can be DMA'd into and out of these interfaces. The image below shows what happens to the packets.

The logic cloud in the picture can be anything the user decides. There are plenty of library modules that can be interconnected in a fashion similar to Click to create useful devices.

The user can interact with the logic on the FPGA using registers too. Each module in the hardware can have an arbitrary number of registers that are exposed to the software. The NetFPGA kernel maps these registers into kernel space. Userspace programs can read and write to these register using ioctl calls. There are wrapper functions around the icotl calls to make this easier for users. The registers can implement counters, status registers, control registers, table entries, and so on.

To fill the logic cloud, you download a bitfile to the FPGA. This is done using a tool called nf2_download. You just do:

 nf2_download 

Getting the Tree for ICING

The NetFPGA tree is pretty hefty (maybe around 1Gig). You don't really need it all, but you can get it if you have the bandwidth and don't want to follow the instructions below. I recommend setting up ssh keys on nity.stanford.edu and using ssh-agent. I've created the icing user account that both Mike and David can use.

Getting the Whole Tree

Type the following command:

 svn co svn+ssh://icing@nity.stanford.edu/hpn/home/svn/nf2/NF2/new_tree NF2

Getting the Necessary Parts

Type the following:

 svn co --depth immediates svn+ssh://icing@nity.stanford.edu/hpn/home/svn/nf2/NF2/new_tree NF2
 cd NF2/lib
 svn co svn+ssh://icing@nity.stanford.edu/hpn/home/svn/nf2/NF2/new_tree/lib/C
 cd ../projects
 svn co svn+ssh://icing@nity.stanford.edu/hpn/home/svn/nf2/NF2/new_tree/projects/icing_forwarder

That's it.

Network Setup

Currently we will run out of a VM on Jad's machine (beirut.stanford.edu). When we want to expand and use multiple NetFPGAs, we can use the NetFPGA cluster which has about 40 NetFPGAs.

The setup in the VM is shown below:

• eth6 <--> nf2c0
• eth7 <--> nf2c1
• eth8 <--> nf2c2
• eth9 <--> nf2c3

eth1 is used for host-only communication and is used to send/rcv registers requests and not for packets. eth0 is natted and is connected to the internet.

If the VM does not have the nf2cX interfaces, the run

 sudo /sbin/ifrename

This will rename the interfaces to the correct names. The network will come up with none of the interfaces brought up. You will have to bring them up according to what you would like (using ifconfig). I suggest creating a script that will bring the network up as you would like it to be.

Building and Running

First download the bitfile. You need to do this on every reboot.

 nf2_download NF2/projects/icing_forwarder/synth/nf2_top_par.bit

I have written a cli to read and write entries into the wildcard table. Later I'll add exact table support.

 cd NF2/projects/icing_forwarder/sw/cli
 make
 ./cli

The file nf_icing.c has all the functions to read and write entries.

Features and Design

The NetFPGA implementation is design as a set of pipeline stages. As a packet passes through each stage, the stage can modify the packet and/or prepend new information to the beginning of the packet. The following stages are present in the ICING implementation:

1. Rx Queues: Accept packets from Ethernet and CPU
2. Input Arbiter: Decide which Rx Queue to service
3. Matcher: See if a packet matches a description and prepend a module header with the result.
4. Output Port Lookup: Check Matcher result, lookup in Flow tables, modify MAC hdr, decide where to send packet.
5. Output Queues: store packet until output port free
6. Tx Queue: transfer packet from Output Queue to Ethernet or CPU.

The relevant ones to ICING are the Matcher stage and the Output Port Lookup stage.

Matcher

The matcher stage allows the user to match the first X bytes of a packet (where X is determined at synthesis time) with the possibility of using a bitmask. If the packet matches, then a Module Header is added with the least significant bit set to 1. Otherwise, it is set to 0. This allows checking for example if a packet is an IP packet of a certain type or an ICING packet of a certain type, ...

Output Port Lookup

The hardware tables are of two types: Exact and wildcard.

The exact tables require a full entry. It will match every bit of the entry with the packet. The wildcard table allows the use of a bitmask. The wildcard table should be used for default routes and don't cares while the exact table (which can fit more than 65000 entries) should be used for individual flows. Entries that miss in both tables are dropped by default, so the user will need to add entries to specify sending packets to software or sending packets from software to ports.

For example, if we would like all packets sent from the CPU nf2cX to go out on Ethernet port X, then we need to add 4 entries in the wildcard table. Each entry will have the src port set, and the output ports set to match the source port. The MAC headers should also be set correspondingly. The exact table has precedence over the wildcard table. So if we would like all flow packets to be dropped if they don't match in the Exact table, then we can add an entry in the wildcard table to that effect.

Note that the output ports field of an entry is a bitmask, allowing a packet to be multicast or dropped. One confusing aspect of this is that the Ethernet and CPU ports alternate:

7	6	5	4	3	2	1	0
CPU 3	MAC 3	CPU 2	MAC 2	CPU 1	MAC 1	CPU 0	MAC 0

So to send a packet to nf2c3 and out of Ethernet port 0, we would set the output ports field as 0x81.

On the other hand, the source port is binary encoded with the values above. So if a packet has source port 2, then it came in on Ethernet port 1.

Testing Environment

I've setup a virtual machine that can communicate with the NetFPGA on my machine. I wil be running the virtual machine continuously. Everybody should use the same username icing. All passwords are set to the same thing (including root and passphrases). Call or msg me to get the passwd. To ssh into the machine, first ssh into my machine (beirut) then ssh to IP address 192.168.0.1 using username icing. The NetFPGA tree is already setup there. We should also setup the icing git tree there as well.

Let me know if there are any problems.