After some tests with our tunnel setup, I still want to give a deeper insight into one issue.
In my first post, I wrote:
torsti76 wrote:
As the experts among you might have noticed, doing this kind of "dirty trick" leads to packet fragmentation, as soon as packets larger than 1488 bytes in size (1500 - L2TPv3 header) arrive at one of the bridge interfaces. Packet fragmentation in turn
significantly decreases tunnel throughput.
So, why did I suggest to "artificially" raise the l2tpeth0's MTU in the first place?
Usually, a mechanism called path MTU discovery (PMTU) takes care of finding the largest possible MTU for each TCP connection (cf.
http://www.netheaven.com/pmtu.html). However, this doesn't seem to work for my setup. In fact, if you stick to the default values of all involved MTUs, the following will happen:
- eth0 and eth1 will both have the default Ethernet MTU of 1500 bytes
- l2tpeth0 will have (Ethernet MTU - L2TPv3 header) = 1488 bytes
- the bridge interfaces will inherit the smallest MTU involved, i.e. the one from l2tpeth0 = 1488 bytes
If, in this setup, a client sends a packet through the tunnel, which exceeds 1488 bytes in size, it will never receive an answer. This happens because the PMTU of 1488 bytes (for some mysterious reason) will never become negotiated as it should be per PMTU discovery.
Now, the most obvious solution would be to lower the MTU of each attached client to 1488 bytes, but for a heterogenous network of about 250 client machines this is impractical, to say the least.
Raising the MTU of l2tpeth0 to 1500 bytes of course also circumvents the problem, but leaves a fragment for every packet > 1488 bytes that has to pass the tunnel. These fragments again need to be encapsulated for L2TPv3, are then sent through the tunnel and acknowledged by the server. In the (very common) case of network transfers involving many full-sized packets, this can easily lead to a performance decrease of 90% (!!!). I will give some empirical evidence on that later on.
After giving it some thought, however, I came up with a much more elegant solution.
Instead of raising the MTU of l2tpeth0 to 1500 bytes, I lowered the MTU of eth1 to 1488 bytes:
Code:
ip link set eth1 mtu 1488
Afterwards, I needed to make sure that both servers and clients on either end of the tunnel use the smaller packet size. To accomplish that, the bridges have to propagate the smaller MTU in a proper way to all machines involved in a TCP session.
This can be done by adjusting the maximum allowed
payload (maximum segment size - MSS) of all forwarded packets with an
iptables one-liner:
Code:
iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1448:1536 -j TCPMSS --set-mss 1448
The MSS usually equals to MTU minus 40 bytes, so for this setup it comes to 1448 bytes. The parameter --mss 1448:1536 leaves packets smaller than 1448 and larger than 1536 bytes untouched to avoid unnecessary interactions.
Note however, that you need the appropriate Linux kernel modules for the above command to be accepted. If in doubt about the correct configuration, read the iptables documentation.
As for the promised empirical figure, copying a large file from a Windows server to a client machine through the tunnel yielded about 3 MBytes per second with fragmentation and 35 MBytes per second on the same physical link while using the iptables-approach.
Quite an improvement, isn't it?
