15.8. The Ultimate Traffic Conditioner: Low Latency, Fast Up & Downloads

15.8.1. Why it doesn't work well by default

ISPs know that they are benchmarked solely on how fast people can download. Besides available bandwidth, download speed is influenced heavily by packet loss, which seriously hampers TCP/IP performance. Large queues can help prevent packet loss, and speed up downloads. So ISPs configure large queues.

These large queues however damage interactivity. A keystroke must first travel the upstream queue, which may be seconds (!) long and go to your remote host. It is then displayed, which leads to a packet coming back, which must then traverse the downstream queue, located at your ISP, before it appears on your screen.

This HOWTO teaches you how to mangle and process the queue in many ways, but sadly, not all queues are accessible to us. The queue over at the ISP is completely off-limits, whereas the upstream queue probably lives inside your cable modem or DSL device. You may or may not be able to configure it. Most probably not.

So, what next? As we can't control either of those queues, they must be eliminated, and moved to your Linux router. Luckily this is possible.

Now, once we have done this, we have eliminated the downstream queue totally (except for short bursts), and gain the ability to manage the upstream queue with all the power Linux offers.

What remains to be done is to make sure interactive traffic jumps to the front of the upstream queue. To make sure that uploads don't hurt downloads, we also move ACK packets to the front of the queue. This is what normally causes the huge slowdown observed when generating bulk traffic both ways. The ACKnowledgements for downstream traffic must compete with upstream traffic, and get delayed in the process.

If we do all this we get the following measurements using an excellent ADSL connection from xs4all in the Netherlands:

Baseline latency: round-trip min/avg/max = 14.4/17.1/21.7 ms Without traffic conditioner, while downloading: round-trip min/avg/max = 560.9/573.6/586.4 ms Without traffic conditioner, while uploading: round-trip min/avg/max = 2041.4/2332.1/2427.6 ms With conditioner, during 220kbit/s upload: round-trip min/avg/max = 15.7/51.8/79.9 ms With conditioner, during 850kbit/s download: round-trip min/avg/max = 20.4/46.9/74.0 ms When uploading, downloads proceed at ~80% of the available speed. Uploads at around 90%. Latency then jumps to 850 ms, still figuring out why.

What you can expect from this script depends a lot on your actual uplink speed. When uploading at full speed, there will always be a single packet ahead of your keystroke. That is the lower limit to the latency you can achieve - divide your MTU by your upstream speed to calculate. Typical values will be somewhat higher than that. Lower your MTU for better effects!

Next, two versions of this script, one with Devik's excellent HTB, the other with CBQ which is in each Linux kernel, unlike HTB. Both are tested and work well.

15.8.2. The actual script (CBQ)

Works on all kernels. Within the CBQ qdisc we place two Stochastic Fairness Queues that make sure that multiple bulk streams don't drown each other out.

Downstream traffic is policed using a tc filter containing a Token Bucket Filter.

You might improve on this script by adding 'bounded' to the line that starts with 'tc class add .. classid 1:20'. If you lowered your MTU, also lower the allot & avpkt numbers!

#!/bin/bash # The Ultimate Setup For Your Internet Connection At Home # # # Set the following values to somewhat less than your actual download # and uplink speed. In kilobits DOWNLINK=800 UPLINK=220 DEV=ppp0 # clean existing down- and uplink qdiscs, hide errors tc qdisc del dev $DEV root 2> /dev/null > /dev/null tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null ###### uplink # install root CBQ tc qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth 10mbit # shape everything at $UPLINK speed - this prevents huge queues in your # DSL modem which destroy latency: # main class tc class add dev $DEV parent 1: classid 1:1 cbq rate ${UPLINK}kbit \ allot 1500 prio 5 bounded isolated # high prio class 1:10: tc class add dev $DEV parent 1:1 classid 1:10 cbq rate ${UPLINK}kbit \ allot 1600 prio 1 avpkt 1000 # bulk and default class 1:20 - gets slightly less traffic, # and a lower priority: tc class add dev $DEV parent 1:1 classid 1:20 cbq rate $[9*$UPLINK/10]kbit \ allot 1600 prio 2 avpkt 1000 # both get Stochastic Fairness: tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10 tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10 # start filters # TOS Minimum Delay (ssh, NOT scp) in 1:10: tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \ match ip tos 0x10 0xff flowid 1:10 # ICMP (ip protocol 1) in the interactive class 1:10 so we # can do measurements & impress our friends: tc filter add dev $DEV parent 1:0 protocol ip prio 11 u32 \ match ip protocol 1 0xff flowid 1:10 # To speed up downloads while an upload is going on, put ACK packets in # the interactive class: tc filter add dev $DEV parent 1: protocol ip prio 12 u32 \ match ip protocol 6 0xff \ match u8 0x05 0x0f at 0 \ match u16 0x0000 0xffc0 at 2 \ match u8 0x10 0xff at 33 \ flowid 1:10 # rest is 'non-interactive' ie 'bulk' and ends up in 1:20 tc filter add dev $DEV parent 1: protocol ip prio 13 u32 \ match ip dst 0.0.0.0/0 flowid 1:20 ########## downlink ############# # slow downloads down to somewhat less than the real speed to prevent # queuing at our ISP. Tune to see how high you can set it. # ISPs tend to have *huge* queues to make sure big downloads are fast # # attach ingress policer: tc qdisc add dev $DEV handle ffff: ingress # filter *everything* to it (0.0.0.0/0), drop everything that's # coming in too fast: tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \ 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1

If you want this script to be run by ppp on connect, copy it to /etc/ppp/ip-up.d.

If the last two lines give an error, update your tc tool to a newer version!

15.8.3. The actual script (HTB)

The following script achieves all goals using the wonderful HTB queue, see the relevant chapter. Well worth patching your kernel for!

#!/bin/bash # The Ultimate Setup For Your Internet Connection At Home # # # Set the following values to somewhat less than your actual download # and uplink speed. In kilobits DOWNLINK=800 UPLINK=220 DEV=ppp0 # clean existing down- and uplink qdiscs, hide errors tc qdisc del dev $DEV root 2> /dev/null > /dev/null tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null ###### uplink # install root HTB, point default traffic to 1:20: tc qdisc add dev $DEV root handle 1: htb default 20 # shape everything at $UPLINK speed - this prevents huge queues in your # DSL modem which destroy latency: tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit burst 6k # high prio class 1:10: tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${UPLINK}kbit \ burst 6k prio 1 # bulk & default class 1:20 - gets slightly less traffic, # and a lower priority: tc class add dev $DEV parent 1:1 classid 1:20 htb rate $[9*$UPLINK/10]kbit \ burst 6k prio 2 # both get Stochastic Fairness: tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10 tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10 # TOS Minimum Delay (ssh, NOT scp) in 1:10: tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \ match ip tos 0x10 0xff flowid 1:10 # ICMP (ip protocol 1) in the interactive class 1:10 so we # can do measurements & impress our friends: tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \ match ip protocol 1 0xff flowid 1:10 # To speed up downloads while an upload is going on, put ACK packets in # the interactive class: tc filter add dev $DEV parent 1: protocol ip prio 10 u32 \ match ip protocol 6 0xff \ match u8 0x05 0x0f at 0 \ match u16 0x0000 0xffc0 at 2 \ match u8 0x10 0xff at 33 \ flowid 1:10 # rest is 'non-interactive' ie 'bulk' and ends up in 1:20 ########## downlink ############# # slow downloads down to somewhat less than the real speed to prevent # queuing at our ISP. Tune to see how high you can set it. # ISPs tend to have *huge* queues to make sure big downloads are fast # # attach ingress policer: tc qdisc add dev $DEV handle ffff: ingress # filter *everything* to it (0.0.0.0/0), drop everything that's # coming in too fast: tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \ 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1

If you want this script to be run by ppp on connect, copy it to /etc/ppp/ip-up.d.

If the last two lines give an error, update your tc tool to a newer version!