ipt_NETFLOW linux 2.6.x-5.x kernel module by abc@openwall.com -- 2008-2019.

High performance NetFlow v5, v9, IPFIX flow data export module for Linux
kernel. Created to be useful for linux routers in high-throughput networks.
It should be used as iptables target.

=========================
= Detailed Feature List =

• High performance and scalability. For highest performance module could be
  run without conntrack being enabled in kernel. Reported to be able to
  handle 10Gbit traffic with more than 1500000 pps with negligible server
  load (on S5500BC).

• NetFlow v5, v9, and IPFIX are fully supported.

  Support of v9/IPFIX is adding flexibility to exporting of flow data
  plus greater visibility of traffic, letting export many additional fields
  besides what was possible in v5 era. Such as

• IPv6 option headers, IPv4 options, TCP options, ethernet type, dot1q
  service and customer VLAN ids, MAC addresses, and

• Full IPv6 support,

• NAT translations events (from conntrack) using NetFlow Event Logging (NEL).
  This is standardized way for v9/IPFIXr, but module export such events even
  for v5 collectors via specially crafted pseudo-records.

• Deterministic (systematic count-based), random and hash Flow Sampling.
  With appropriate differences in support of v5, v9, and IPFIX.

• SNMP agent (for net-snmp) for remote management and monitoring.

• Options Templates (v9/IPFIX) let export useful statistical,
  configurational, and informational records to collector.
  Such as metering, exporting, sampling stat and reliability stat, sampling
  configuration, network devices ifName, ifDescr list.

• Tested to compile and work out of the box on Centos 6, 7, Debian and

• Ubuntu. Many vanilla Linux kernels since 2.6.18 up to the latest (as of

• writing is 3.19) are supported and tested.

• Module load time and run-time (via sysctl) configuration.

• Flexibility in enabling features via ./configure script. This will let you
  disable features you don't need, which increase compatibility with custom
  kernels and performance.

• SNMP-index translation rules, let convert meaningless and unstable
  interface indexes (ifIndex) to more meaningful numbering scheme.

• Easy support for catching mirrored traffic with promisc option. Which is
  also supporting optional MPLS decapsulation and MPLS-aware NetFlow.

============================
= OBTAINING LATEST VERSION =

$ git clone git://github.com/aabc/ipt-netflow.git ipt-netflow
$ cd ipt-netflow

================
= INSTALLATION =

Five easy steps.

** 1. Prepare Kernel source

If you have package system install kernel-devel package, otherwise install
raw kernel source from http://kernel.org matching exactly version of your
installed kernel.

a) What to do for Centos:

  ~# yum install kernel-devel

b) What to do for Debian and Ubuntu:

  ~# apt-get install module-assistant
  ~# m-a prepare

c) Otherwise, if you downloaded raw kernel sources don't forget to create
.config by copying it from your distribution's kernel. Its copy could reside
in /boot or sometimes in /proc, examples:

  kernel-src-dir/# cp /boot/config-`uname -r` .config
or
  kernel-src-dir/# zcat /proc/config.gz > .config

Assuming you unpacked kernel source into `kernel-src-dir/' directory.
Then run:

  kernel-src-dir/# make oldconfig

After that you'll need to prepare kernel for modules build:

  kernel-src-dir/# make prepare modules_prepare

Note: Don't try to `make prepare' in Centos kernel-devel package directory
(which is usually something like /usr/src/kernels/2.6.32-431.el6.x86_64)
as this is wrong and meaningless.

** 2. Prepare Iptables

Before this step it also would be useful to install pkg-config if don't
already have.

If you have package system just install iptables-devel (or on Debian, Ubuntu
and derivatives libxtables-dev if available, otherwise iptables-dev)
package, otherwise install iptables source matching version of your
installation from ftp://ftp.netfilter.org/pub/iptables/

a) What to do for Centos:

  # yum install iptables-devel

b) What to do for Debian or Ubuntu:

  # apt-get install iptables-dev pkg-config

c) Otherwise, for raw iptables source build it and make install.

** 3. Prepare net-snmp (optional)

In case you want to manage or monitor module performance via SNMP you
may install net-snmp. If you want to skip this step run configure
with --disable-snmp-agent option.

a) For Centos:

  # yum install net-snmp net-snmp-devel

b) For Debian or Ubuntu:

  # apt-get install snmpd libsnmp-dev

c) Otherwise install net-snmp from www.net-snmp.org

** 4. Now, to actually build the module run:

  ~/ipt-netflow# ./configure
  ~/ipt-netflow# make all install
  ~/ipt-netflow# depmod

This will install kernel module and iptables specific library.

Troubleshooting:

 a) Sometimes you will want to add CC=gcc-3 to make command.
 Example: make CC=gcc-3.3

 b) Compile module with actual kernel source compiled.
 I.e. first compile kernel and boot into it, and then compile module.
 If you are using kernel-devel package check that its version matches
 your kernel package.

 c) If you have sources in non-standard places or configure isn't able to
 find something run ./configure --help to see how to specify paths manually.

 d) To run irqtop on Debian 8 you may need to install:

   # apt-get install ruby ruby-dev ncurses-dev
   # gem install curses

 z) If all fails create ticket at
      https://github.com/aabc/ipt-netflow/issues

** 5. After this point you should be able to load module and
use -j NETFLOW target in your iptables. See next section.

=====================
= Configure Options =

Configure script allows to enable or disable optional features:

 --enable-natevents
     enables natevents (NEL) support, (this and option will require
 conntrack support to be enabled into kernel and conntack
     module (nf_conntrack) loaded before ipt_NETFLOW. Usually this is
     done automatically because of `depmod', but if you don't do `make
     install' you'll need to load nf_conntrack manually.
     Read below for explanation of natevents.

 --enable-sampler
     enables flow sampler. Read below for explanation of its configuration
 option.

 --enable-sampler=hash
     additionally enables 'hash' sampler.

 --disable-snmp-agent
     disables building net-snmp agent module, which is enabled by default.

 --enable-snmp-rules
     enables SNMP-index conversion rules. Read below for explanation
     of snmp-rules.

 --enable-macaddress
     enables exporting of src and dst MAC addresses for every flow
     in v9/IPFIX. Difference in any of MAC address will be accounted
     as different flow. I.e. MAC addresses will be part of flow key.

 --enable-vlan
     enables exporting of dot1q VLAN Ids and Priorities for every flow
     in v9/IPFIX. It supports outer and second dot1q tags if present.

     Any of two previous options will enable exporting of Ethernet Packet
     Type, ethernetType(256).

 --enable-direction
     enables exporting of flowDirection(61) Element for v9/IPFIX.

     Packets captured in PREROUTING and INPUT chains will be accounted as
     ingress flows(0), in OUTPUT and POSTROUTING as egress flows(1), and
     in FORWARD will have flowDirection set to undefined value 255.

 --enable-aggregation
     enables aggregation rules. Read below for explanation of aggregation.

 --disable-dkms
     disable creating dkms.conf and auto-install module into DKMS tree.

 --disable-dkms-install
     only disable auto-install into DKMS, but still create dkms.conf, in
     case you will want to install it manually.

 --enable-physdev
 Export ingressPhysicalInterface(252) and egressPhysicalInterface(253)
 (relevant for bridges) in V9 and IPFIX. If your collector does not
 support these Elements but you still need physdevs then use
 --enable-physdev-override, in that case physdevs will override normal
 interface numbers ingressInterface(10) and egressInterface(14).

 --enable-promisc
 Enables capturing of promiscuous packets into raw/PREROUTING chain.
 See README.promisc Solution 1 for usage details and example.

 --promisc-mpls
 Enables MPLS label stack decapsulation for promiscuous packets. (For
 IPv4 and IPv6 packets only). This also enables MPLS-aware NetFlow (v9
 and IPFIX), you may wish to specify with --promisc-mpls=n how much MPLS
 labels you want to be recorded and exported (default is 3, maximum is
 10, set to 0 to not report anything).

===========
= RUNNING =

1. You can load module directly by insmod like this:

   insmod ipt_NETFLOW.ko destination=127.0.0.1:2055 debug=1

   Or if properly installed (make install; depmod) by this:

   modprobe ipt_NETFLOW destination=127.0.0.1:2055

   See, you may add options in insmod/modprobe command line, or add
   them in /etc/modprobe.conf or /etc/modprobe.d/ipt_NETFLOW.conf
   like thus:

   options ipt_NETFLOW destination=127.0.0.1:2055 protocol=9 natevents=1

2. Statistics is in /proc/net/stat/ipt_netflow
   Machine readable statistics is in /proc/net/stat/ipt_netflow_snmp
   To view boring slab statistics: grep ipt_netflow /proc/slabinfo
   Dump of all flows is in /proc/net/stat/ipt_netflow_flows

3. You can view parameters and control them via sysctl, example:

   sysctl net.netflow

   sysctl net.netflow.hashsize=32768

   Note: For after-reboot configuration I recommend to store module parameters
   in modprobe configs instead of storing them in /etc/sysctl.conf, as it's
   less clear when init process will apply sysctl.conf, before of after
   module's load.

4. Example of directing all IPv4 traffic into the module:

   iptables -I FORWARD -j NETFLOW

   iptables -I INPUT -j NETFLOW

   iptables -I OUTPUT -j NETFLOW

   Note: It is preferable (because easier to understand) to insert
   NETFLOW target at the top of the chain, otherwise not all traffic may
   reach NETFLOW if your iptables configuration is complicated and some
   other rule inadvertently consume the traffic (dropping or acepting before
   NETFLOW is reached). It's always good to test your configuration.
   Use iptables -L -nvx to check pkts/bytes counters on the rules.

5. If you want to account IPv6 traffic you should use protocol 9 or 10.
   Example of directing all IPv6 traffic into the module:

   sysctl net.netflow.protocol=10

   ip6tables -I FORWARD -j NETFLOW

   ip6tables -I INPUT -j NETFLOW

   ip6tables -I OUTPUT -j NETFLOW

   Note: First enable right version of protocol and after that add ip6tables
   rules, otherwise you will get errors in dmesg.

6. If you want to account NAT events (NEL):

   sysctl net.netflow.natevents=1

   Note that natevents feature is completely independent from traffic accounting
   (it's using so called conntrack events), thus you don't need to set or change
   any iptables rules to use that. You may need to enable kernel config option
   CONFIG_NF_CONNTRACK_EVENTS though (if it isn't already enabled).
   For details on how they are exported for different protocol versions see
   below.

7. For SNMP support you will need to add this command into snmpd.conf to
   enable IPT-NETFLOW-MIB in SNMP agent:

   dlmod netflow /usr/lib/snmp/dlmod/snmp_NETFLOW.so

   Restart snmpd for changes to take effect. Don't forget to properly configure
   access control. Example simplest configuration may looks like (note that this
   is whole /etc/snmp/snmpd.conf):

   rocommunity public 127.0.0.1
   dlmod netflow /usr/lib/snmp/dlmod/snmp_NETFLOW.so

   Note, that this config will also allow full read-only access to the whole
   linux MIB. To install IPT-NETFLOW-MIB locally, copy file IPT-NETFLOW-MIB.my
   into ~/.snmp/mibs/

   • Detailed example of SNMP configuration is there:
   • https://github.com/aabc/ipt-netflow/wiki/Configuring-SNMP-access

   To check that MIB is installed well you may issue:

   $ snmptranslate -m IPT-NETFLOW-MIB -IR -Tp iptNetflowMIB

   This should output IPT-NETFLOW-MIB in tree form.

   To check that snmp agent is working well issue:

   $ snmpwalk -v 1 -c public 127.0.0.1 -m IPT-NETFLOW-MIB iptNetflowMIB

   Should output full MIB. If MIB is not installed try:

   $ snmpget -v 1 -c public 127.0.0.1 .1.3.6.1.4.1.37476.9000.10.1.1.1.1.0

   Which should output STRING: "ipt_NETFLOW".

   MIB provides access to very similar statistics that you have in
   /proc/net/stat/ipt_netflow, you can read description of objects in
   text file IPT-NETFLOW-MIB.my

   If you want to access to SNMP stat in machine readable form for your
   scripts there is file /proc/net/stat/ipt_netflow_snmp

   Note: Using of SNMP v2c or v3 is mandatory for most tables, because
   this MIB uses 64-bit counters (Counter64) which is not supported in old
   SNMP v1. You should understand that 32-bit counter will wrap on 10Gbit
   traffic in just 3.4 seconds! So, always pass option -v2c' or -v3'
   to net-snmp utils. Or, for example, configure option defVersion 2c' in ~/.snmp/snmp.conf You can also have defCommunity public' ov v3
   auth parameters (defSecurityName, defSecurityLevel, defPassphrase)
   set there (man snmp.conf).

   Examples for dumping typical IPT-NETFLOW-MIB objects:

   • Module info (similar to modinfo, SNMPv1 is ok for following two objects):

     $ snmpwalk -v 1 -c public 127.0.0.1 -m IPT-NETFLOW-MIB iptNetflowModule

   • Read-write sysctl-like parameters (yes, they are writable via snmpset, you
     may need to configure write access to snmpd, though):

     $ snmpwalk -v 1 -c public 127.0.0.1 -m IPT-NETFLOW-MIB iptNetflowSysctl

   • Global performance stat of the module (note -v2c, because rest of the
     objects require SNMP v2c or SNMP v3):

     $ snmpwalk -v2c -c public 127.0.0.1 -m IPT-NETFLOW-MIB iptNetflowTotals

   • Per-CPU (metering) and per-socket (exporting) statistics in table format:

     $ snmptable -v2c -c public 127.0.0.1 -m IPT-NETFLOW-MIB iptNetflowCpuTable
     $ snmptable -v2c -c public 127.0.0.1 -m IPT-NETFLOW-MIB iptNetflowSockTable

===========
= OPTIONS =

Options can be passed as parameters to module or changed dynamically
via sysctl net.netflow or IPT-NETFLOW-MIB::iptNetflowSysctl

protocol=5
- what version of NetFlow protocol to use. Default is 5.
You can choose from 5, 9, or 10 (where 10 is IPFIX). If you plan
to account IPv6 traffic you should use protocol 9 or 10 (IPFIX),
because NetFlow v5 isn't compatible with IPv6.

destination=127.0.0.1:2055
- where to export netflow, to this ip address. Port is optional, default
is 2055. You will see this connection in netstat like this:

   udp 0 0 127.0.0.1:32772 127.0.0.1:2055 ESTABLISHED 

destination=[2001:db8::1]:2055
- export target using IPv6 address. Brackets are optional, but otherwise
you should delimit port with 'p' or '#' character.

destination=127.0.0.1:2055,192.0.0.1:2055
- mirror flows to two (can be more) addresses, separate addresses
with comma.

destination=127.0.0.1:2055@127.0.0.2
- bind socket to address (127.0.0.2).

destination=127.0.0.1:2055%eth0
- bind socket to interface (eth0). May be useful for multi-homed boxes.

sampler=deterministic:123
sampler=random:123
sampler=hash:123
- enables Flow Sampling. To disable set to the empty value or to `0'.
Note, that this is flow sampling (as of RFC 7014), not packet
sampling (PSAMP).

   There is three sampling modes:

     deterministic:  select each N-th observed flow; in IPFIX this mode
                     is called Systematic count-based Sampling;
     random:         select randomly one out of N flows.
     hash:           select hash-randomly one out of N flows.

   Number after colon is population size N, with valid values 2-16383.
   (This 16383 limit is for compatibility with NetFlow v5.)
     Using 'deterministic' and 'random' sampling will not reduce resource
   usage caused by the module, because flows are sampled late in exporting
   process. This will reduces amount of flows which go to the collector,
   thus, reducing load on the collector.
     On the other hand, using 'hash' sampling will reduce CPU and memory
   load caused by the module, because flows are discarded early in the
   processing chain. They are discarded almost like in random sampler,
   except that pseudo-random value is depend on the Flow Key hash for each
   packet.
     All required NetFlow/IPFIX information to signal use of sampling is
   also sent to the collector. 'Hash' sampling will be presented as 'random'
   sampling to the collector, because of their similarity.
   Note, that Flow Sampling is compatible with NetFlow v5, v9, and IPFIX.

natevents=1
- Collect and send NAT translation events as NetFlow Event Logging (NEL)
for NetFlow v9/IPFIX, or as dummy flows compatible with NetFlow v5.
Default is 0 (don't send).

   For NetFlow v5 protocol meaning of fields in dummy flows are such:
     Src IP, Src Port  is Pre-nat source address.
     Dst IP, Dst Port  is Post-nat destination address.
       - These two fields made equal to data flows catched in FORWARD chain.
     Nexthop, Src AS  is Post-nat source address for SNAT. Or,
     Nexthop, Dst AS  is Pre-nat destination address for DNAT.
     TCP Flags is SYN+SCK for start event, RST+FIN for stop event.
     Pkt/Traffic size is 0 (zero), so it won't interfere with accounting.

   Natevents are compilation disabled by default, to enable you will need to
   add --enable-natevents option to ./configure script.

   For techical description of NAT Events see:
     http://tools.ietf.org/html/draft-ietf-behave-ipfix-nat-logging-04

inactive_timeout=15
- export flow after it's inactive for 15 seconds. Default value is 15.

active_timeout=1800
- export flow after it's active for 1800 seconds (30 minutes). Default
value is 1800.

refresh-rate=20
- for NetFlow v9 and IPFIX it's rate how frequently to re-send templates
(per packets). You probably don't need to change default (which is 20).

timeout-rate=30
- for NetFlow v9 and IPFIX it's rate when to re-send old templates (in
minutes). No need to change it.

debug=0
- debug level (none).

sndbuf=number
- size of output socket buffer in bytes. I recommend you to put higher
value if you experience netflow packet drops (can be seen in statistics
as 'sock: fail' number.)
Default value is system default.

hashsize=number
- Hash table bucket size. Used for performance tuning.
Abstractly speaking, it should be minimum two times bigger than flows
you usually have, but not need to.
Default is system memory dependent small enough value.

maxflows=2000000
- Maximum number of flows to account. It's here to prevent DOS attacks.
After this limit is reached new flows will not be accounted. Default is
2000000, zero is unlimited.

aggregation=string..
- Few aggregation rules (or some say they are rule.)

   Buffer for aggregation string 1024 bytes, and sysctl limit it
     to ~700 bytes, so don't write there a lot.
   Rules worked in definition order for each packet, so don't
     write them a lot again.
   Rules applied to both directions (dst and src).
   Rules tried until first match, but for netmask and port
      aggregations separately.
   Delimit them with commas.

   Rules are of two kinds: for netmask aggregation
      and port aggregation:

   a) Netmask aggregation example: 192.0.0.0/8=16
   Which mean to strip addresses matching subnet 192.0.0.0/8 to /16.

   b) Port aggregation example: 80-89=80
   Which mean to replace ports from 80 to 89 with 80.

   Full example:
      aggregation=192.0.0.0/8=16,10.0.0.0/8=16,80-89=80,3128=80

   Aggregation rules are enabled by default, if you feel you don't need them
   you may add --disable-aggregation to ./configure script.

snmp-rules=string...
- Few SNMP-index conversion rules similar to fproble-ulog.

   Quoting man fprobe-ulog:

     "Comma separated list of interface name to SNMP-index conversion
     rules.  Each rule consists of interface base name and SNMP-index
     base separated by colon (e.g. ppp:200). Final SNMP-index is  sum
     of corresponding SNMP-index base and interface number.
     In the above example SNMP-index of interface ppp11 is 211.

     If interface  name  did not fit to any of conversion rules then
     SNMP-index will be taken from kernel."

   This implementation isn't optimized for performance (no rule caching
   or hashing), but should be fast if rules list are short.

   Rules are parsed in order from first to last until first match.

   snmp-rules are compilation disabled by default, to enable you will need
   to add --enable-snmp option to ./configure script.

scan-min=1
- Minimal interval between flow export scans. Sometimes could be useful
to reduce load on exporting CPU by increasing this interval. Value are
in kernel jiffies units (which is x/HZ seconds).

promisc=1
- Enables promisc hack. See README.promisc Solution 1 for details.

exportcpu=number
- Lock exporter to single CPU. This may be useful to fine control CPU
load. Common use case: with smp_affinity and RSS you spread packet
processing to all CPUs except one, and lock it to the exporter. While
exporter CPU load generally is not high, for someone it may be not
desirable to combine it with packet processing on very highly loaded
routers.

   This option could be changed at runtime with:

     # echo number > /sys/module/ipt_NETFLOW/parameters/exportcpu

engine_id=number
- Observation Domain ID (on IPFIX, Source Id on NetFlow v9, or Engine Id
on NetFlow v5) value to be exported. This may help your collector to
distinguish between multiple exporters. On Netflow v9 and IPFIX this
value is 32-bit on NetFlow v5 only 8 low bits are significant.
Default value is 0.

   This option could be changed at runtime with:

     # echo number > /sys/module/ipt_NETFLOW/parameters/engine_id

====================
= HOW TO READ STAT =

Statistics is your friend to fine tune and understand netflow module
performance.

To see stat in human readable form:
# cat /proc/net/stat/ipt_netflow

How to interpret the data:

ipt_NETFLOW version v1.8-122-gfae9d59-dirty, srcversion 6141961152BE0DFA6A21EF4; aggr mac vlan

This line helps to identify actual source that your module is build on.
Please always supply it in all bug reports.

v1.8-122: 1.8 is release, 122 is commit number after release;
-gfae9d59: fae9d59 is short git commit id;
-dirty: if present, meaning that git detected that sources are changed since
last git commit, you may wish to do git diff' to view changes; srcversion 6141961152BE0DFA6A21EF4: binary version of module, you can compare this with data from modinfo ./ipt_NETFLOW.ko' to identify
actual binary loaded;
aggr mac vlan: tags to identify compile time options that are enabled.

Protocol version 10 (ipfix), refresh-rate 20, timeout-rate 30, (templates 2, active 2). Timeouts: active 5, inactive 15. Maxflows 2000000

Protocol version currently in use. Refresh-rate and timeout-rate
for v9 and IPFIX. Total templates generated and currently active.
Timeout: active X: how much seconds to wait before exporting active flow.
- same as sysctl net.netflow.active_timeout variable.
inactive X: how much seconds to wait before exporting inactive flow.
- same as sysctl net.netflow.inactive_timeout variable.
Maxflows 2000000: maxflows limit.
- all flows above maxflows limit must be dropped.
- you can control maxflows limit by sysctl net.netflow.maxflows variable.

Promisc hack is disabled (observed 0 packets, discarded 0).

observed n: To see that promisc hack is really working.

Natevents disabled, count start 0, stop 0.

- Natevents mode disabled or enabled, and how much start or stop events
  are reported.

Flows: active 5187 (peak 83905 reached 0d0h1m ago), mem 283K, worker delay 100/1000 (37 ms, 0 us, 4:0 0 [3]).

active X: currently active flows in memory cache.
- for optimum CPU performance it is recommended to set hash table size to
at least twice of average of this value, or higher.
peak X reached Y ago: peak value of active flows.
mem XK: how much kilobytes of memory currently taken by active flows.
- one active flow taking 56 bytes of memory.
- there is system limit on cache size too.
worker delay X/HZ: how frequently exporter scan flows table per second.
Rest is boring debug info.

Hash: size 8192 (mem 32K), metric 1.00, [1.00, 1.00, 1.00]. InHash: 1420 pkt, 364 K, InPDU 28, 6716.

Hash: size X: current hash size/limit.
- you can control this by sysctl net.netflow.hashsize variable.
- increasing this value can significantly reduce CPU load.
- default value is not optimal for performance.
- optimal value is twice of average of active flows.
mem XK: how much memory occupied by hash table.
- hash table is fixed size by nature, taking 4 bytes per entry.
metric X, [X, X, X]: how optimal is your hash table being used.
- lesser value mean more optimal hash table use, min is 1.0.
- last three numbers in squares is moving average (EWMA) of hash table
access divided by match rate (searches / matches) for 4sec, and 1, 5, and
15 minutes. Sort of hash table load average. First value is instantaneous.
You can try to increase hashsize if averages more than 1 (increase
certainly if >= 2).
InHash: X pkt, X K: how much traffic accounted for flows in the hash table.
InPDU X, X: how much traffic in flows preparing to be exported.

Rate: 202448 bits/sec, 83 packets/sec; 1 min: 668463 bps, 930 pps; 5 min: 329039 bps, 483 pps

• Module throughput values for 1 second, 1 minute, and 5 minutes.

cpu# pps; <search found new [metric], trunc frag alloc maxflows>, traffic: <pkt, bytes>, drop: <pkt, bytes>
cpu0 123; 980540 10473 180600 [1.03], 0 0 0 0, traffic: 188765, 14 MB, drop: 27863, 1142 K

cpu#: this is Total and per CPU statistics for:
pps: packets per second on this CPU. It's useful to debug load imbalance.

traffic: <pkt, bytes>: how much traffic is accounted.
pkt, bytes: sum of packets/megabytes accounted by module.
- flows that failed to be exported (on socket error) is accounted here too.

drop: <pkt, bytes>: how much of traffic is not accounted.
pkt, bytes: sum of packets/kilobytes that are dropped by metering process.
- reasons these drops are accounted here:
truncated/fragmented packets,
packet is for new flow but failed to allocate memory for it,
packet is for new flow but maxflows is already reached.
Traffic lost due to socket errors is not accounted here. Look below
about export and socket errors.

Export: Rate 0 bytes/s; Total 2 pkts, 0 MB, 18 flows; Errors 0 pkts; Traffic lost 0 pkts, 0 Kbytes, 0 flows.

Rate X bytes/s: traffic rate generated by exporter itself.
Total X pkts, X MB: total amount of traffic generated by exporter.
X flows: how much data flows are exported.
Errors X pkts: how much packets not sent due to socket errors.
Traffic lost 0 pkts, 0 Kbytes, 0 flows: how much metered traffic is lost
due to socket errors.
Note that cberr' errors are not accounted here due to their asynchronous nature. Read below about cberr' errors.

sock0: 10.0.0.2:2055 unconnected (1 attempts).

If socket is unconnected (for example if module loaded before interfaces is
up) it shows now much connection attempts was failed. It will try to connect
until success.

sock0: 10.0.0.2:2055, sndbuf 106496, filled 0, peak 106848; err: sndbuf reached 928, connect 0, cberr 0, other 0

sockX: per destination stats for:
X.X.X.X:Y: destination ip address and port.
- controlled by sysctl net.netflow.destination variable.
sndbuf X: how much data socket can hold in buffers.
- controlled by sysctl net.netflow.sndbuf variable.
- if you have packet drops due to sndbuf reached (error -11) increase this
value.
filled X: how much data in socket buffers right now.
peak X: peak value of how much data in socket buffers was.
- you will be interested to keep it below sndbuf value.
err: how much packets are dropped due to errors.
- all flows from them will be accounted in drop stat.
sndbuf reached X: how much packets dropped due to sndbuf being too small
(error -11).
connect X: how much connection attempts was failed.
cberr X: how much connection refused ICMP errors we got from export target.
- probably you are not launched collector software on destination,
- or specified wrong destination address.
- flows lost in this fashion is not possible to account in drop stat.
- these are ICMP errors, and would look like this in tcpdump:
05:04:09.281247 IP alice.19440 > bob.2055: UDP, length 120
05:04:09.281405 IP bob > alice: ICMP bob udp port 2055 unreachable, length 156
other X: dropped due to other possible errors.

aggr0: ...
aggrX: aggregation rulesets.
- controlled by sysctl net.netflow.aggregation variable.

==========================
= NetFlow considerations =

List of all IPFIX Elements http://www.iana.org/assignments/ipfix/ipfix.xhtml

Flow Keys are Elements that distinguish flows. Quoting RFC: "If a Flow
Record for a specific Flow Key value already exists, the Flow Record is
updated; otherwise, a new Flow Record is created."

In this implementation following Elements are treated as Flow Keys:

 IPv4 source address:        sourceIPv4Address(8),
 IPv6 source address:        sourceIPv6Address(27),
 IPv4 destination address:   destinationIPv4Address(12),
 IPv6 destination address:   destinationIPv6Address(28),
 TCP/UDP source port:        sourceTransportPort(7),
 TCP/UDP destination port:   destinationTransportPort(11),
 input interface:            ingressInterface(10),
 IP protocol:                protocolIdentifier(4),
 IP TOS:                     ipClassOfService(5),
 and address family (IP or IPv6).

Additional Flow Keys if VLAN exporting is enabled:

 First (outer) dot1q VLAN tag: dot1qVlanId(243) and
                             dot1qPriority(244) for IPFIX,
                             or vlanId(243) for NetFlow v9.
 Second (customer) dot1q VLAN tag: dot1qCustomerVlanId(245)
                             and dot1qCustomerPriority(246).

Additional Flow Keys if MAC address exporting is enabled:

 Destination MAC address:    destinationMacAddress(80),
 Source MAC address:         sourceMacAddress(56).

Additional Flow Keys if MPLS-aware NetFlow is enabled:

 Captured MPLS stack is fully treated as flow key (including TTL values),
 which is Elements from mplsTopLabelStackSection(70) to
 mplsLabelStackSection10(79), and, if present, mplsTopLabelTTL(200).

Other Elements are not Flow Keys. Note that outer interface, which is
egressInterface(14), is not regarded as Flow Key. Quoting RFC 7012: "For
Information Elements ... for which the value may change from packet to packet
within a single Flow, the exported value of an Information Element is by
default determined by the first packet observed for the corresponding Flow".

Note that NetFlow and IPFIX modes of operation may have slightly different
Elements being used and different statistics sent via Options Templates.

=========
= VOILA =