SENIC Documentation

The modified kernel that implements the parallel handoff is here: https://github.com/sivasankariit/linux-rl. You can see the commits from Sivasankar in this page: https://github.com/sivasankariit/linux-rl/commits/rl. These changes implement the parallel packet handoff.

# git clone https://github.com/sivasankariit/linux-rl.git
# cd linux-rl
# make menuconfig

... select your options
... use your favourite installation method.  I prefer make-kpkg.
... http://www.debian.org/releases/stable/i386/ch08s06.html.en
# export CONCURRENCY_LEVEL=$(grep ^processor /proc/cpuinfo | wc -l)
# APPEND_TO_VERSION=-senic

# make-kpkg --initrd --append-to-version $APPEND_TO_VERSION kernel_image
# make-kpkg --append-to-version $APPEND_TO_VERSION kernel_headers

... The above commands will produce two .deb files which you can
... install using dpkg -i /path/to/deb

We built SENIC-s on top of QFQ packet scheduling algorithm. Linux already has QFQ qdisc sch_qfq. The modified qdisc is available here: https://github.com/sivasankariit/qfq-rl.

# git clone https://github.com/sivasankariit/qfq-rl

The following iproute2 package incorporates additional per-class statistics for debugging, such as the class's virtual time, system virtual time, etc.

# git clone https://github.com/sivasankariit/iproute2.git
# cd iproute2
# ./configure
# make -j10

Reserving a CPU

SENIC-s works best if the one CPU core it uses is reserved exclusively for egress scheduling. To do this, you have to isolate one CPU core (by default set to 2 in sch_qfq.c) from the kernel scheduler.

# vi /path/to/grub/menu.lst
.. search for kernel boot options, e.g.:
module          /vmlinuz-x.y.z root=/dev/sda2 ro console=tty0

.. change the above to:
module          /vmlinuz-x.y.z root=/dev/sda2 ro console=tty0 isolcpus=2

If you boot into the kernel and run top or htop, you should not see any activity on that CPU.

Configuring link rate

SENIC-s is by default configured for 10Gb/s NIC. If you want to run it at 1Gb/s, you will have to do the following:

# cd qfq-rl
# vi sch_qfq.c +118
/*
 * Link speed in Mbps. System time V will be incremented at this rate and the
 * rate limits of flows (still using the weight variable) should be also
 * indicated in Mbps.
 *
 * This value should actually be about 9844Mb/s but we leave it at
 * 9800 with the hope of having small queues in the NIC.  The reason
 * is that with a given MTU, each packet has an Ethernet preamble (4B)
 * and the frame check sequence (8B) and a minimum recommended
 * inter-packet gap (0.0096us for 10GbE = 12B).  Thus the max
 * achievable data rate is MTU / (MTU + 24), which is 0.98439 with MTU
 * = 1500B and and 0.99734 with MTU=9000B.
 */
--> #define LINK_SPEED              9800    // 10Gbps link
#define QFQ_DRAIN_RATE          ((u64)LINK_SPEED * 125000 * ONE_FP / NSEC_PER_SEC)

.. edit LINK_SPEED accordingly while reading the instructions

The following commands add the new qfq module and create two rate limiters at 1Gb/s and 2Gb/s. The rate limits are implicit weights. So, if your NIC only supports 1Gb/s, then the capacity will be divided in the ratio 1000:2000 between the two rate limiters.

$ cat tc.sh
#!/bin/bash

dev=eth2
mtu=1500
tc qdisc del dev $dev root
rmmod sch_qfq

cd qfq-rl
make
insmod ./sch_qfq.ko
tc qdisc add dev $dev root handle 1: qfq

# Create rate limiters
tc class add dev $dev parent 1: classid 1:1 qfq weight 1000 maxpkt $mtu
tc class add dev $dev parent 1: classid 1:2 qfq weight 2000 maxpkt $mtu
tc filter add dev $dev parent 1: protocol all prio 1 u32 match ip dport 5001 0xffff flowid 1:1
# This filter matches all pkts
tc filter add dev $dev parent 1: protocol all prio 2 u32 match u32 0 0 flowid 1:2

If you want to add a new rate limiter at 100Mb/s, you need to create two things: (1) a new class, and (2) a new filter.

tc class add dev $dev parent 1: classid 1:$classid qfq weight 100 maxpkt $mtu
tc filter add dev $dev parent 1: ..(filter string).. flowid 1:$classid