EyeQ Documentation

I maintain two repositories for EyeQ. One is for bleeding edge development, and the other is for releases. The bleeding edge contains all branches with untested and unfinished code. The release branch is more stable.

The stable repository is here: http://github.com/jvimal/eyeq. The bleeding edge repository is under an older project name: http://github.com/jvimal/perfiso_10g.

You can clone the github repositories, which has the source code to compile the Linux kernel module:

$ git clone https://github.com/jvimal/eyeq.git

I have tested EyeQ with Linux 3.0 and above. EyeQ is a kernel module that implements a queueing discpline. So you need the kernel headers to compile EyeQ. You can find available headers on your system in /lib/modules.

$ cd eyeq

$ ls /lib/modules/
3.2.0-23-generic

$ make

This will produce the performance isolation kernel module called perfiso.ko for your running kernel. If you want to compile for a different kernel, please edit the Makefile and change the version number accordingly.

EyeQ hooks into the transmit path by acting as a queueing discipline. To avoid dependency on iproute, EyeQ replaces the hierarchical token bucket (htb) module so you can use your existing tc tool to install EyeQ. So you need to remove htb before you can use EyeQ. EyeQ hooks into the receive path using netdev_rx_handler_register callback.

To install EyeQ on a device:

$ rmmod sch_htb

$ insmod ./perfiso.ko

$ tc qdisc show
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap...

$ tc qdisc add dev eth0 root handle 1: htb

$ dmesg | tail
perfiso: Init TX path for eth0
perfiso: Init RX path for eth0

$ tc qdisc show
qdisc htb 1: dev eth0 root

We provide a separate python configuration tool to work with EyeQ, as it does not use the standard tc interface. EyeQ kernel module exports a sysfs file which it uses for all configuration mechanisms.

Mininet is a collection of useful scripts to configure features such as network namespaces, containers and Linux Traffic Control, to create lightweight virtual networks on a single machine. I am one of the authors of the second release of Mininet, with Bob Lantz, Brandon Heller and Nikhil Handigol.

Mininet uses tc to configure htb inorder to emulate links with of some capacity. So you can use Mininet to test EyeQ as well. Unfortunately, one of Mininet's limitations (as of 2.0) is that you are constrained to resources that one server can offer especially CPU. One CPU core amounts to about 2–3Gb/s of aggregate switching capacity. So you can try EyeQ safely with few (10) links operating at slower link speeds, say 10–100Mb/s.

The test repository has scripts to create a single-switch topology in Mininet. To test it on Mininet, just do the following steps:

1. Download and install Mininet-2.0 on Ubuntu 12.10+ by following Option 2 on the Mininet Setup Page.

   Alternatively, you can boot the AMI id ami-7eab204e in US-Oregon region in Amazon AWS which has Mininet-2.0 preinstalled.

2. Install paramiko and termcolor dependencies. On Ubuntu, just run:

   $ sudo apt-get install python-paramiko
   $ sudo easy_install termcolor
   

   Disable DNS resolution in ssh. Add "UseDNS no" to /etc/ssh/sshd_config. Then, run sudo reload ssh for changes to take effect.

3. Clone the tests repository.

   $ cd ~/eyeq
   $ git clone https://github.com/jvimal/eyeq-tests.git tests
   $ cd tests
   

4. Ensure correct configuration:
   • SSH settings in config/mininet.py
   • Edit host.py to import config.mininet instead of config.packard.
5. Run the Mininet script sudo python mininet/simple.py and wait for the CLI.
6. Ensure you can log in from h1 to h2 without passwords.
   • In Mininet, type xterm h1
   • You should be able to type ssh 10.0.0.2 and log in without any hassles. If you get "permission denied (public key)" or some such error, please add /root/.ssh/id_rsa.pub to /root/.ssh/authorized_keys.
7. Create tenants on all hosts.

   By default, the Mininet script does not create tenants. You can create tenants by doing the following:

   • In Mininet, type xterm h1
   • Inside the terminal, type the following, which creates 3 tenants on each of the 3 virtual hosts (h1, h2 and h3): python tenant.py -m 3 -T 3
   • By default, the script will create tenants with IP addresses 11.0.T.M where T is the tenant ID and M is the machine number (1 for h1, etc.). Routing tables are set correctly so that packets destined to a tenant pick the corresponding source IP address.
   Alternatively, once you've set up ssh successfully, you can create tenants on all machines simply by passing --conf as shown below:

   $ python mininet/simple.py --conf --num-hosts 3 --num-tenants 3
   

8. Make a few parameter changes so things work at 100Mb/s instead of 10Gb/s.

   $ cd ~/eyeq/
   
   # Set the interface speed to 100Mb/s
   $ tools/pi.py --set 18,100
   Setting ISO_MAX_TX_RATE = 100
   
   # Set the receive speed to 100Mb/s
   $ tools/pi.py --set 5,100
   Setting ISO_VQ_DRAIN_RATE_MBPS = 100
   
   # Set the tokenbucket timeout value to 1ms
   $ tools/pi.py --set 3,1000000
   Setting ISO_TOKENBUCKET_TIMEOUT_NS = 1000000
   
   # Set the rate metering interval to 1ms
   $ tools/pi.py --set 13,10000
   Setting ISO_VQ_UPDATE_INTERVAL_US = 10000
   

   The above scripts are available in tests/mininet/100mbps.sh.

   If you write to the sysfs files directly to change ISO_MAX_TX_RATE and ISO_VQ_DRAIN_RATE_MBPS, you need to force EyeQ to recompute tenant rx and tx rate guarantees. You can do it by running:

   $ echo dev eth0 > /sys/module/perfiso/parameters/recompute_dev
   

9. Check TX fairness.
   • Run iperf servers (iperf -s) on all hosts.
   • Run iperf client from h1:1 (tenant 1) to h2:1 (tenant 1)
     • h1 terminal: iperf -c 11.0.1.2 -t 100 -i 1
   • Run 4 flows from h1:2 to h3:2
     • h1 terminal: iperf -c 11.0.2.3 -t 100 -i 1 -P4
   You should see each iperf get 50Mb/s.
10. The stats command all work in Mininet as well.

    $ tools/pi.py --stats --dev h1-eth0
    

Since ptb is a drop-in replacement for tbf, you will have to remove tbf from a running kernel before you can use ptb. The module is for Linux kernels 3.7 onwards, as the qdisc API datastructures changed a bit. However, the API change is simple so you can easily port it to older kernels.

$ rmmod sch_tbf
$ git clone https://github.com/jvimal/ptb.git
$ cd ptb
$ make
$ insmod ./sch_ptb.ko

There is a sample script with default options so you can test PTB out. Just like Linux's default mq or mqprio qdisc, ptb must be the root qdisc. It cannot be the child of any other qdisc.

$ cat tc.sh
#!/bin/bash

dev=eth2
tc qdisc del dev $dev root
rmmod sch_ptb
make
insmod ./sch_ptb.ko
tc qdisc add dev $dev root handle 1: tbf limit 100000 burst 1000 rate 3Gbit