NetFlowProbe

This project provides a simple NetFlow v5 exporting tool. NetFlow as a protocol for flow monitoring, first implemented in Cisco routers, is the most popular measurement solution of network traffic. Statistics on IP traffic flows provide information about who communicates with whom, how long, how often, using what protocol and service and also how much data was transfered. NetFlow data might be utilize in many applications such as routing optimization, application troubleshooting, traffic mix monitoring, accounting and billing, anomaly tracking and others.

The metering and exporting processes are completely implemented on the NetFPGA card whereas control, configuration and collecting processes are implemented as a user software running in the host computer.

The characteristics of the NetFlow design are as follows:

1. Measurement of four 1 Gbps interfaces at the line rate
2. Measurement of IPv4 flows only
3. Memory for up to 4000 concurrent flows.
4. Timestamp with millisecond resolution synchronized to host clock
5. Export of flowrecords using NetFlow v5 export protocol

The project contains an initialization script, configuration script, set of regression tests. Java GUI is under construction and is expected to be finished by March 2009.

Show Contents...Hide Contents...

Project summary

Status

:

; Version:

Author

Martin Zadnik:

Download

Install from NetFPGA Yum repository

1. Install the NetFPGA Base Package
2. Install the netflow probe

 yum install netfpga-netflow

Obtain Tarball from NetFPGA Distributions

Login account for the NetFPGA required for download. For a new account go to: http://netfpga.org/netfpgawiki/index.php?title=Special:UserLogin&type=signup

Download NetFlow tarball from NetFPGA beta distributions website (http://www.liberouter.org/~xzadni00/netfpga.netflow.1_0.tar.gz) into you root home directory (/root/). If you want to preserve your previous /root/NF2 directory please duplicate it:

 [root@nf-test ~]# cp -r NF2 NF2-backup

Unpack netfpga.netflow.1_0.tar.gz:

 [root@nf-test ~]# tar -xvf netfpga.netflow.1_0.tar.gz

Regression Tests

The regression tests verify the functionality of the metering and exporting process. In order to run the tests, you need to have the machine connected for the regression tests as shown in Run Regression Tests section of the Guide.

After connecting the cables check that the PERL5LIB and NF2_DESIGN_DIR variables are set correctly:

 [root@nf-test ~]# env | egrep "NF2_DESIGN_DIR|PERL5LIB"

The output should be:

 PERL5LIB=/root/NF2/projects/netflow/lib/Perl5:/root/NF2/lib/Perl5:
 NF2_DESIGN_DIR=/root/NF2/projects/netflow

If this is not the case then include setup_env_variable:

 [root@nf-test ~]# . NF2/projects/netflow/sw/setup_env_variable

Run the following command to run the regression tests.

 nf21_regress_test.pl --project netflow

Regression Tests

Test 1: Filter packets

Name

test_netflow_filtering

Description

Test verifies that only (TCP, UDP, ICMP)/IPv4 packets are selected for metering process. Other packets are discarded.
# Initialize netfpga hardware
# Send 3 packets of each type (TCP, UDP, ICMP, other) on eth1.
# After each round counters are checked to verify number of accepted packets and discarded packets. $ Location

projects/netflow/regress/test_netflow_filtering $ Output

 SUCCESS!

Test 2: Timestamp

Name

test_netflow_timestamp

Description

Speed up and slow down timestamp counter. Verify that the timestamp counter is managable.
# Initialize netfpga hardware
# Speed up the timestamp counter twice the original value.
# Verify the speed up took place.
# Slow down the timestamp counter twice the original value.
# Verify the slow down took place. $ Location

projects/netflow/regress/test_netflow_timestamp $ Output

 SUCCESS!

Test 3: Capacity

Name

test_netflow_capacity

Description

Send 4000 flows from eth1 to nf2c0. Verify the number of accepted flows.
# Initialize netfpga hardware
# Send 4000 flows, each flow has port number increased by one to the previous one.
# Check counter of new flows $ Location

projects/netflow/regress/test_netflow_capacity $ Output

 SUCCESS!

Test 4: Active timeout

Name

test_netflow_acttimeout

Description

Send N packets with interval 1 s between each packet from eth1 to nf2c0 while setting the active timeout. Verify the number of flows created.
# Initialize netfpga hardware
# Send 10 packets with interval 1 s between each packet from eth1 to nf2c0 while setting the active timeout to 1500 ms.
# Verify the number of flows created, expected number is 5.
# Send 10 packets with interval 1 s between each packet from eth1 to nf2c0 while setting the active timeout to 4500 ms.
# Verify the number of flows created, expected number is 2. $ Location

projects/netflow/regress/test_netflow_acttimeout $ Output

 SUCCESS!

Test 5: Inactive timeout

Name

test_netflow_inacttimeout

Description

Send N packets with interval 1 s between each packet from eth1 to nf2c0 while setting the inactive timeout. Verify the number of flows created.
# Initialize netfpga hardware
# Send 10 packets with interval 1 s between each packet from eth1 to nf2c0 while setting the inactive timeout to 2000 ms.
# Verify the number of flows created and the number of updates, expected number of created flows is 1 and expected number of updates is 9.
# Send 10 packets with interval 1 s between each packet from eth1 to nf2c0 while setting the active timeout to 0 ms.
# Verify the number of flows created and the number of updates, expected number of created flows is 10 and expected number of updates is 0. $ Location

projects/netflow/regress/test_netflow_inacttimeout $ Output

 SUCCESS!

Test 6: Flow record

Name

test_netflow_record

Description

Send N packets from eth1 to nf2c0. Verify that the generated NetFlow packet contains correct information elements.
# Initialize netfpga hardware
# Send 10 packets of the same flow from eth1 to nf2c0.
# Wait for an inactive timeout.
# Read the generated NetFlow packet.
# Verify that its source and destination IP addresses are the same as the addresses entered during initialization.
# Verify source and destination IP addresses in the flow record.
# Verify source and destination ports in the flow record.
# Verify protocol in the flow record.
# Verify number of bytes and packets reported in the flow record. $ Location

projects/netflow/regress/test_netflow_record $ Output

 SUCCESS!

Usage

Preparation

Run the following command:

 cd NF2/projects/netflow/sw
 . setup_env_variable
 ./netflow_setup.pl

NetFlow metering

The NetFlow metering process is controlled via the netflow.pl command:

   netflow.pl
   [-a ] (ms)
        [-i ] (ms)
        [-c :]
        [-e :]
        [-p <8-bit output port mask>]
        [-stat]
        [-conf]
        [-s]
        [-f]
        [-d]

   netflow.pl --help  - show detailed help

Options

-a :

Specify the active timeout of the flow in milliseconds. When the active timeout is exceeded the flow is released from the flow cache.

-i :

Specify the inactive timeout of the flow in milliseconds. When the inactive timeout is exceeded the flow is released from the flow cache.

-c ::

Specify the IP address and port of the collector, i.e., the destination IP address of the NetFlow packet and the destination port number.

-e ::

Specify the IP address and port of the exporter, i.e., the source IP address of the NetFlow packet and the source port number.

-p <8-bit output port mask in hexadecimal>:

Specify the output ports to send the NetFlow packets from. The output port is one-hot encoded and it is possible to assert more bits, i.e., port is one-hot encoded and it is possible to assert more bits, i.e., to duplicate NetFlow packets to multiple physical output ports. There are 8 ports (4 physical and 4 software ports). These ports are numbered according to table below. Please note that packets are always forwarded to SW0 = nf2c0 other software ports are disabled. For example if the mask is 0x3 then packets are forward not only to SW0 but also to PHY0.

Number
Port Name
Description
0
PHY0
Physical port 0 of NetFPGA card
1
SW0
Software port 0 seen as nf2c0 in SW
2
PHY1
Physical port 1 of NetFPGA card
3
SW1
Software port 1 seen as nf2c1 in SW
4
PHY2
Physical port 2 of NetFPGA card
5
SW2
Software port 2 seen as nf2c2 in SW
6
PHY3
Physical port 3 of NetFPGA card
7
SW3
Software port 3 seen as nf2c3 in SW

-stat

Display statistics about collecting NetFlow data.

-conf

Display configuration of NetFlow metering process.

-s

Start NetFlow metering process.

-f

Finish NetFlow metering process and clean up flow cache.

-d

Start NetFlow with default parameters.

Examples

Start the NetFlow metering process with default parameters specified in netflow.pl:

 netflow.pl -d

Finish NetFlow metering process and clean flow cache:

 netflow.pl -f

Set the inactive timeout to 10 s, the active timeout to 30 s. The NetFlow packets will be send out of physical port 3 with the destination IP address 192.168.1.2 and destionation port 9996 (typical for NetFlow), with source IP address 192.168.0.1 and source port 1234. Parameter -s starts metering process after the configuration has been finished.

 netflow.pl -i 10000 -a 30000 -c 192.168.1.2:9996 -e 192.168.0.1:1234 -p 0x10 -s

Print out statistics about current configuration of NetFlow metering process:

 netflow.pl -conf

Print out statistics about collecting of NetFlow data:

 netflow.pl -stat

How to make your first test

1. Connect network interface port eth1 with netfpga network interface port nf2c0.
2. Run:
   1. cd NF2/projects/netflow/sw
   2. . setup_env_variable
   3. ./netflow_setup.pl
   4. ./netflow.pl -d; ./netflow.pl -conf -stat
3. Open new terminal and run: tcpdump -i eth1 -nn -s 0 dst port 9996
4. Open another terminal and run: tcpreplay -i eth1 <file.dump>
5. NetFPGA cards emits NetFlow packets out its physical network interface nf2c0 to eth1 where you dump them.

Output

NetFlow metering process emits NetFlow packets on one or more of its output interfaces (if enabled and configured correctly, see parameter -p ). These packets could be sent to distant collector or dumped by tcpdump on local NetFPGA network port (nf2c0-nf2c3), for example:

 tcpdump -i nf2c0 -w 

Wireshark

Wireshark provides a dissector to parse NetFlow v5 data. Wireshark recognizes NetFlow v5 packets automatically if the destination port number is 9996. Otherwise you have to select a CFLOW dissector.

Collector

There are plenty of NetFlow v5 free collectors available: Collectors. You can install collector directly to NetFPGA host machine or you can use output ports of NetFPGA card to emit NetFlow packets to distant collector.

Statistics of NetFlow metering process

Command netflow.pl -stat allows to dump important data about NetFlow monitoring process.

Packets total:

All packets that have been seen by the NetFlow metering process.

Packets accepted:

Packets that have been selected as valid for NetFlow monitoring. These packets are IPv4/TCP/UDP/ICMP.

Packets discarded:

Packets that have been dropped as non-valid for NetFlow monitoring. E.g., IPv6, ARP, VLAN, ...

Packets rejected because of no space available:

The NetFlow metering process has limited capacity of its flow cache due to the limited resources on NetFPGA a the indexing hierarchy. The index memory is based on bucket structure which means that certain bucket might overflow earlier than the whole flow cache and therefore some flows might not make it in the flow cache despite there is still space available. All such packets are also accumulated in this counter.

Number of active flows:

Current occupation of the flow cache.

Total number of flows seen:

Accumulator of created flows since last upload of firmware.

Total number of updates:

Accumulator of updated flows since last upload of firmware.

Total number of released flows:

Accumulator of release flows since last upload of firmware.

Timestamping

Every packet submitted for NetFlow metering process is assigned a 32-bit timestamp with millisecond resolution. By default the timestamp counter (sys_up_time) that generates timestamps is not synchronized with host clock which means that after a certain interval there might be a skew between host clocks and sys_up_time. You might prevent this by using script time.pl:

 cd NF2/projects/netflow/sw
 . setup_env_variable
 ./netflow_setup.pl

Then open a new terminal and run:

 ./time.pl

The output tells you what is the current skew between host time and the NetFlow metering sys_up_time, how many hardware clock cycles are in one millisecond (i.e., the length of millisecond in clock cycles (1 clock cycle = 8 ns)) and what is the increment to this value to compensate the skew. The algorithm used in time.pl is ad hoc and is able to maintain synchronization within +-2 milliseconds. Its convergence is poor so please feel free to come up with your own.

Project documentation

The documentation to project (description of firmware processing pipeline) could be downloaded from NetFlow Probe Documentation.

Original Source

This project provides a simple NetFlow v5 exporting tool. NetFlow as a protocol for flow monitoring, first implemented in Cisco routers, is
the most popular measurement solution of network traffic. Statistics on IP traffic flows provide information about who communicates with whom, how long,
how often, using what protocol and service and also how much data was transfered. NetFlow data might be utilize in many applications such as routing optimization, application troubleshooting, traffic mix monitoring, accounting and billing, anomaly tracking and others.


The metering and exporting processes are completely implemented on the NetFPGA card whereas control, configuration and collecting processes are implemented as a user software running in the host computer.

The characteristics of the NetFlow design are as follows:
#Measurement of four 1 Gbps interfaces at the line rate
#Measurement of IPv4 flows only
#Memory for up to 4000 concurrent flows.
#Timestamp with millisecond resolution synchronized to host clock
#Export of flowrecords using NetFlow v5 export protocol

The project contains an initialization script, configuration script, set of regression tests. Java GUI is under construction and is expected to be finished by March 2009.

== Project summary ==

; Status: 
; Version: 
; Author: Martin Zadnik

== Download ==

=== Install from NetFPGA Yum repository ===
#Install the [http://netfpga.org/netfpgawiki/index.php/Guide#Install_NetFPGA_Base_Package NetFPGA Base Package]
#Install the netflow probe
 yum install netfpga-netflow

=== Obtain Tarball from NetFPGA Distributions === 

Login account for the NetFPGA required for download.  For a new account go to: [http://netfpga.org/netfpgawiki/index.php?title=Special:UserLogin&type=signup http://netfpga.org/netfpgawiki/index.php?title=Special:UserLogin&type=signup]

Download NetFlow tarball from NetFPGA beta distributions website (http://www.liberouter.org/~xzadni00/netfpga.netflow.1_0.tar.gz) into you root home directory (/root/). If you want to preserve your previous /root/NF2 directory please duplicate it:
 [root@nf-test ~]# cp -r NF2 NF2-backup
Unpack netfpga.netflow.1_0.tar.gz:
 [root@nf-test ~]# tar -xvf netfpga.netflow.1_0.tar.gz

== Regression Tests ==

The regression tests verify the functionality of the metering and exporting process. In order to run the tests, you need to have the machine connected for the regression tests as shown in [http://netfpga.org/netfpgawiki/index.php/Guide#Run_Regression_Tests Run Regression Tests] section of the Guide.

After connecting the cables check that the PERL5LIB and NF2_DESIGN_DIR variables are set correctly:

 [root@nf-test ~]# env | egrep "NF2_DESIGN_DIR|PERL5LIB"

The output should be:

 PERL5LIB=/root/NF2/projects/netflow/lib/Perl5:/root/NF2/lib/Perl5:
 NF2_DESIGN_DIR=/root/NF2/projects/netflow

If this is not the case then include setup_env_variable:

 [root@nf-test ~]# . NF2/projects/netflow/sw/setup_env_variable

Run the following command to run the regression tests.

 nf21_regress_test.pl --project netflow

=== Regression Tests === 

====Test 1: Filter packets ====
:; Name: test_netflow_filtering
:; Description: Test verifies that only (TCP, UDP, ICMP)/IPv4 packets are selected for metering process. Other packets are discarded.
::# Initialize netfpga hardware
::# Send 3 packets of each type (TCP, UDP, ICMP, other) on eth1.
::# After each round counters are checked to verify number of accepted packets and discarded packets.
:; Location
projects/netflow/regress/test_netflow_filtering
:; Output
 SUCCESS!

====Test 2: Timestamp ====
:; Name: test_netflow_timestamp
:; Description: Speed up and slow down timestamp counter. Verify that the timestamp counter is managable.
::# Initialize netfpga hardware
::# Speed up the timestamp counter twice the original value.
::# Verify the speed up took place.
::# Slow down the timestamp counter twice the original value.
::# Verify the slow down took place.
:; Location
projects/netflow/regress/test_netflow_timestamp
:; Output
 SUCCESS!

====Test 3: Capacity ====
:; Name: test_netflow_capacity
:; Description: Send 4000 flows from eth1 to nf2c0. Verify the number of accepted flows.
::# Initialize netfpga hardware
::# Send 4000 flows, each flow has port number increased by one to the previous one.
::# Check counter of new flows
:; Location
projects/netflow/regress/test_netflow_capacity
:; Output
 SUCCESS!

====Test 4: Active timeout====
:; Name: test_netflow_acttimeout
:; Description: Send N packets with interval 1 s between each packet from eth1 to nf2c0 while setting the active timeout. Verify the number of flows created. 
::# Initialize netfpga hardware
::# Send 10 packets with interval 1 s between each packet from eth1 to nf2c0 while setting the active timeout to 1500 ms. 
::# Verify the number of flows created, expected number is 5.
::# Send 10 packets with interval 1 s between each packet from eth1 to nf2c0 while setting the active timeout to 4500 ms. 
::# Verify the number of flows created, expected number is 2.
:; Location
projects/netflow/regress/test_netflow_acttimeout
:; Output
 SUCCESS!

====Test 5: Inactive timeout====
:; Name: test_netflow_inacttimeout
:; Description: Send N packets with interval 1 s between each packet from eth1 to nf2c0 while setting the inactive timeout. Verify the number of flows created. 
::# Initialize netfpga hardware
::# Send 10 packets with interval 1 s between each packet from eth1 to nf2c0 while setting the inactive timeout to 2000 ms. 
::# Verify the number of flows created and the number of updates, expected number of created flows is 1 and expected number of updates is 9.
::# Send 10 packets with interval 1 s between each packet from eth1 to nf2c0 while setting the active timeout to 0 ms. 
::# Verify the number of flows created and the number of updates, expected number of created flows is 10 and expected number of updates is 0.
:; Location
projects/netflow/regress/test_netflow_inacttimeout
:; Output
 SUCCESS!

====Test 6: Flow record ====
:; Name: test_netflow_record
:; Description: Send N packets from eth1 to nf2c0. Verify that the generated NetFlow packet contains correct information elements. 
::# Initialize netfpga hardware
::# Send 10 packets of the same flow from eth1 to nf2c0.
::# Wait for an inactive timeout.
::# Read the generated NetFlow packet.
::# Verify that its source and destination IP addresses are the same as the addresses entered during initialization. 
::# Verify source and destination IP addresses in the flow record.
::# Verify source and destination ports in the flow record.
::# Verify protocol in the flow record.
::# Verify number of bytes and packets reported in the flow record.
:; Location
projects/netflow/regress/test_netflow_record
:; Output
 SUCCESS!




== Usage ==

=== Preparation ===

Run the following command:
 cd NF2/projects/netflow/sw
 . setup_env_variable
 ./netflow_setup.pl

=== NetFlow metering ===

The NetFlow metering process is controlled via the <tt>netflow.pl</tt> command:
   netflow.pl
   [-a <active timeout>] (ms)
        [-i <inactive timeout>] (ms)
        [-c <IP address of collector>:<port>]
        [-e <IP address of exporter>:<port>]
        [-p <8-bit output port mask>]
        [-stat]
        [-conf]
        [-s]
        [-f]
        [-d]

   netflow.pl --help  - show detailed help

==== Options ====
;-a <active timeout>
:Specify the active timeout of the flow in milliseconds. When the active timeout is exceeded the flow is released from the flow cache.

;-i <inactive timeout>
:Specify the inactive timeout of the flow in milliseconds. When the inactive timeout is exceeded the flow is released from the flow cache.

;-c <IP address of collector>:<port>
:Specify the IP address and port of the collector, i.e., the destination IP address of the NetFlow packet and the destination port number.

;-e <IP address of exporter>:<port>
:Specify the IP address and port of the exporter, i.e., the source IP address of the NetFlow packet and the source port number.

;-p <8-bit output port mask in hexadecimal>
:Specify the output ports to send the NetFlow packets from. The output port is one-hot encoded and it is possible to assert more bits, i.e., port is one-hot encoded and it is possible to assert more bits, i.e., to duplicate NetFlow packets to multiple physical output ports. There are 8 ports (4 physical and 4 software ports). These ports are numbered according to table below. Please note that packets are always forwarded to SW0 = nf2c0 other software ports are disabled. For example if the mask is 0x3 then packets are forward not only to SW0 but also to PHY0.

{|border="1" cellpadding="5"
|-
!Number
!Port Name
!Description
|-
|0
|PHY0
|Physical port 0 of NetFPGA card
|-
|1
|SW0
|Software port 0 seen as nf2c0 in SW
|-
|2
|PHY1
|Physical port 1 of NetFPGA card
|-
|3
|SW1
|Software port 1 seen as nf2c1 in SW
|-
|4
|PHY2
|Physical port 2 of NetFPGA card
|-
|5
|SW2
|Software port 2 seen as nf2c2 in SW
|-
|6
|PHY3
|Physical port 3 of NetFPGA card
|-
|7
|SW3
|Software port 3 seen as nf2c3 in SW
|}
  

;-stat
:Display statistics about collecting NetFlow data.

;-conf
:Display configuration of NetFlow metering process.

;-s
:Start NetFlow metering process.

;-f
:Finish NetFlow metering process and clean up flow cache.

;-d
:Start NetFlow with default parameters.

==== Examples ====

Start the NetFlow metering process with default parameters specified in netflow.pl:
 netflow.pl -d

Finish NetFlow metering process and clean flow cache:
 netflow.pl -f

Set the inactive timeout to 10 s, the active timeout to 30 s. The NetFlow packets will be send out of physical port 3 with the destination IP address 192.168.1.2 and destionation port 9996 (typical for NetFlow), with source IP address 192.168.0.1 and source port 1234. Parameter -s starts metering process after the configuration has been finished.
 netflow.pl -i 10000 -a 30000 -c 192.168.1.2:9996 -e 192.168.0.1:1234 -p 0x10 -s

Print out statistics about current configuration of NetFlow metering process:
 netflow.pl -conf

Print out statistics about collecting of NetFlow data:
 netflow.pl -stat

==== How to make your first test ====
#Connect network interface port eth1 with netfpga network interface port nf2c0.
#Run:
##cd NF2/projects/netflow/sw
##. setup_env_variable
##./netflow_setup.pl
##./netflow.pl -d; ./netflow.pl -conf -stat
#Open new terminal and run: tcpdump -i eth1 -nn -s 0 dst port 9996
#Open another terminal and run: tcpreplay -i eth1 <file.dump>
#NetFPGA cards emits NetFlow packets out its physical network interface nf2c0 to eth1 where you dump them.

==== Output ====
NetFlow metering process emits NetFlow packets on one or more of its output interfaces (if enabled and configured correctly, see parameter '''-p'''). These packets could be sent to distant collector or dumped by tcpdump on local NetFPGA network port (nf2c0-nf2c3), for example:
 tcpdump -i nf2c0 -w <file.dump>

===== Wireshark =====

Wireshark provides a dissector to parse NetFlow v5 data. Wireshark recognizes NetFlow v5 packets automatically if the destination port number is 9996. Otherwise you have to select a CFLOW dissector.

===== Collector =====
There are plenty of NetFlow v5 free collectors available: [http://www.switch.ch/network/projects/completed/TF-NGN/floma/software.html Collectors]. You can install collector directly to NetFPGA host machine or you can use output ports of NetFPGA card to emit NetFlow packets to distant collector.

===== Statistics of NetFlow metering process =====
Command netflow.pl -stat allows to dump important data about NetFlow monitoring process.

;Packets total
:All packets that have been seen by the NetFlow metering process.

;Packets accepted
:Packets that have been selected as valid for NetFlow monitoring. These packets are IPv4/TCP/UDP/ICMP.

;Packets discarded
:Packets that have been dropped as non-valid for NetFlow monitoring. E.g., IPv6, ARP, VLAN, ...
  
;Packets rejected because of no space available
:The NetFlow metering process has limited capacity of its flow cache due to the limited resources on NetFPGA a the indexing hierarchy. The index memory is based on bucket structure which means that certain bucket might overflow earlier than the whole flow cache and therefore some flows might not make it in the flow cache despite there is still space available. All such packets are also accumulated in this counter.
 
;Number of active flows
:Current occupation of the flow cache.
 
;Total number of flows seen
:Accumulator of created flows since last upload of firmware.

;Total number of updates
:Accumulator of updated flows since last upload of firmware.

;Total number of released flows
:Accumulator of release flows since last upload of firmware.

=== Timestamping ===
Every packet submitted for NetFlow metering process is assigned a 32-bit timestamp with millisecond resolution. By default the timestamp counter (sys_up_time) that generates timestamps is not synchronized with host clock which means that after a certain interval there might be a skew between host clocks and sys_up_time. You might prevent this by using script time.pl:
 cd NF2/projects/netflow/sw
 . setup_env_variable
 ./netflow_setup.pl
Then open a new terminal and run:
 ./time.pl
The output tells you what is the current skew between host time and the NetFlow metering sys_up_time, how many hardware clock cycles are in one millisecond (i.e., the length of millisecond in clock cycles (1 clock cycle = 8 ns)) and what is the increment to this value to compensate the skew. The algorithm used in time.pl is ad hoc and is able to maintain synchronization within +-2 milliseconds. Its convergence is poor so please feel free to come up with your own.

== Project documentation ==
The documentation to project (description of firmware processing pipeline) could be downloaded from [http://www.liberouter.org/~xzadni00/netflowprobedoc.pdf NetFlow Probe Documentation].