Recently I had the opportunity to work with a customer that needed some help with traffic shaping and policing on their network. I had poked around in the past with this, trying to get guaranteed bandwidth for my VoIP phone, but the last time I checked, that setup no longer worked, so it was shelved until further notice. I just had to take care that when I was on the phone, I could not do any large downloads that would rob the bandwidth from my voice packets.
The customer gave me impetus to re-learn Linux Traffic Control. The main tool offered to us is called tc, meaning traffic control. You can learn all about tc at the Linux Advanced Routing and Traffic Control website. I spent several hours there trying to remember all I had forgotten. I also looked around at several other websites with howtos on the matter, but it seemed that they were all pointing back to lartc.org anyway. I poked around long enough to understand the recipes in their cookbook and then wrote up a script of my own.
I wanted to have about 90kb/s of guaranteed bandwidth for VoIP and then some other high priority bandwidth for things like ICMP packets, TCP ACK packets, and other low-latency stuff (things that mark the TOS field in the IP header.) In other words, I wanted to be able to:
- Make sure my VoIP traffic gets through so I don’t have choppy phone calls
- Perform uploads without killing my downloads (let the ACK packets through)
- Be able to type in an SSH session while doing a large download
- Not starve my VPN to work when the network is busy (no more 3-12 second latencies, please.)
- Have fast ping times so I can brag to all my friends
Sounds like I am hoping for a miracle, right? Well, not really. Simply dividing the traffic into several classes and then giving each one a slice of the pie will do a lot on my quest for the Well Tempered Network. I know the VoIP bandwidth, so that is easy. Then the rest, I decided to split into quarters — high priority gets at least 1/4 of the remaining bandwidth, medium priority gets the same, while bulk transfers and the rest of the stuff get anything that is left over (a little less than 1/2 the pipe).
Without this QoS script, I am unable to do a large download (or upload) without killing my VoIP call, uploads kill downloads, ssh is very non-interactive, and pings range in the 400-1100ms range. With this QoS script, I can do simultaneous large downloads and large uploads without hurting my VoIP call quality AND at the same time, ssh interactivity goes up (to the same as with no other traffic) and ping times range in the 80-200ms range. VPN traffic seems to be better too, though sometimes it suffers from latencies beyond my control. I think this means I reached all my goals. I was very happy with it and thought it might be nice to share.
Here is my script inline, or you can download it.
#!/bin/bash -e
# Copyright © Vernon Mauery 2007-2009. All Rights Reserved.
# Author: Vernon Mauery
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
# user settings
RATE_UP=1850
RATE_DOWN=4750
VOIP_RATE=90
VOIP_IP=69.59.224.0/19
VPN_IP=204.146.0.0/16
WAN_IF=ppp0
#
# End Configuration Options
#
LOCK_FILE=/var/run/qos_started
stop_shaper() {
DEV=$1
# Reset everything to a known state (cleared)
tc qdisc del dev $DEV root >& /dev/null || :
tc qdisc del dev $DEV ingress >& /dev/null || :
echo "Shaping removed on $DEV"
}
start_shaper() {
DEV=$1
RATE_DOWN=$2
RATE_UP=$3
# Remove all qdiscs from the device to reset
tc qdisc del dev $DEV root >& /dev/null || :
tc qdisc del dev $DEV ingress >& /dev/null || :
# set queue size to give latency of about 2 seconds on low-prio packets
ip link set dev $DEV qlen 30
#############################
# egress (outbound traffic) #
#############################
# create root HTB qdisc, and point unclassified traffic to 1:40
tc qdisc add dev $DEV root handle 1: htb default 40
# shape all traffic with a rate limit of $RATE_UP which should
# prevent queueing at the ISP, allowing us to control the queue length
# and priority of packets that get sent
tc class add dev $DEV parent 1: classid 1:1 htb
rate ${RATE_UP}kbit burst 6k
# VoIP class (we want guaranteed priority on this one)
tc class add dev $DEV parent 1:1 classid 1:10 htb
rate ${VOIP_RATE}kbit prio 0
# high prio class 1:20 for low-latency packets
tc class add dev $DEV parent 1:1 classid 1:20 htb
rate $[($RATE_UP-$VOIP_RATE)/4]kbit
ceil ${RATE_UP}kbit burst 6k prio 1
# medium priority class 1:30 for small packets and the like
tc class add dev $DEV parent 1:1 classid 1:30 htb
rate $[($RATE_UP-$VOIP_RATE)/4]kbit
ceil ${RATE_UP}kbit burst 6k prio 2
# bulk prio class 1:40
tc class add dev $DEV parent 1:1 classid 1:40 htb
rate $[($RATE_UP-$VOIP_RATE)/2]kbit
ceil ${RATE_UP}kbit prio 3
# SFQ is mostly to make multiple streams share the bandwidth, so in the
# case of our VoIP class, it is really only one stream and pfifo_fast
# would be a better choice. Not many packets should end up in the high
# prio class (1:20), so we just let them go in fifo order as well.
# But the two lowest prio classes (which likely have more traffic)
# will likely benefit from SFQ.
tc qdisc add dev $DEV parent 1:30 handle 30: sfq perturb 10
tc qdisc add dev $DEV parent 1:40 handle 40: sfq perturb 10
# filter VoIP stuff
tc filter add dev $DEV parent 1:0 protocol ip prio 50 u32
match ip protocol 17 0xff
match ip dst $VOIP_IP
flowid 1:10
# TOS Minimum Delay (ssh, NOT scp) in 1:20:
tc filter add dev $DEV parent 1:0 protocol ip prio 20 u32
match ip tos 0x10 0xff flowid 1:20
# ICMP (ip protocol 1) in the interactive class 1:10 so we
# can do measurements & impress our friends:
tc filter add dev $DEV parent 1:0 protocol ip prio 20 u32
match ip protocol 1 0xff flowid 1:20
# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class:
tc filter add dev $DEV parent 1: protocol ip prio 20 u32
match ip protocol 6 0xff
match u8 0x05 0x0f at 0
match u16 0x0000 0xffc0 at 2
match u8 0x10 0xff at 33
flowid 1:20
# put the vpn in 1:30 so it doesn't get hurt by downloads
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32
match ip protocol 6 0xff
match ip dst $VPN_IP
match ip dport 443 0xffff
flowid 1:30
# all other outbound traffic that doesn't match one of these
# filters gets put in the default class, 1:40
#############################
# ingress (inbound traffic) #
#############################
# It is not possible to *shape* inbound traffic (I don't understand
# reasoning behind this), but we are able to limit it. The ingress
# qdisc does not have classes or anything fancy, but we can add
# filters to implement policing. This means we can drop inbound
# packets which will effectively limit our download speeds, which
# makes it so our packets are not getting hung up in the HUGE
# queues at the ISP.
# I also tried to make sure that we didn't drop any VoIP packets
# in the policing effort. You might think of this as profiling. :)
tc qdisc add dev $DEV handle ffff: ingress
# filter VoIP traffic and enqueue it
tc filter add dev $DEV parent ffff: protocol ip prio 50 u32
match ip src $VOIP_IP
flowid :1
# filter *everything* to it (0.0.0.0/0), drop everything that's
# coming in too fast:
tc filter add dev $DEV parent ffff: protocol ip prio 20 u32
match ip src 0.0.0.0/0
police rate $[($RATE_DOWN-$VOIP_RATE)]kbit burst 10k drop
flowid :1
echo "Added shaping for $DEV limited at $RATE_DOWN/$RATE_UP kb/s"
}
start_qos() {
[ -e $LOCK_FILE ] && return 1
start_shaper "$WAN_IF" "$RATE_DOWN" "$RATE_UP"
touch $LOCK_FILE
}
stop_qos() {
stop_shaper "$WAN_IF"
rm -f $LOCK_FILE >& /dev/null
}
qos_status() {
echo "[qdisc]"
tc -s qdisc show dev $WAN_IF
echo "[class]"
tc -s class show dev $WAN_IF
echo "[filter]"
tc -s filter show dev $WAN_IF
}
case $1 in
start)
start_qos
;;
stop)
stop_qos
;;
restart)
stop_qos
start_qos
;;
status)
qos_status
;;
*)
echo "usage: $0 {start|stop|restart|status}"
exit 1
;;
esac
exit 0