openl2tp.org

Imagine the following:

You are facing the relocation of your employer from site A to site B. But since it's a large institution, you can't just take all the client machines, servers, switches, network wires, etc. and move them from A to B. Instead, the relocation should take place stepwise.

Also, the infrastructure of both locations has to support VLANs as well as routing between VLANs, while both locations are connected to the Internet, but do not have a dedicated connection to each other.

Of course, you could go ahead, duplicate all the essential internal services, like DHCP, DNS, SMTP, Domain Controller, Routing (Core) Switch, etc., from A to B, and then start moving the client computers from A to B. If you choose disjunct networks on both ends, you could even get away connecting both sites by something like an OpenVPN site-to-site tunnel.

Aside from having to have full service redundancy this solution is somewhat complicated, since it involes changing default routes, DNS addresses, etc. on each and every of the moved machine. Given, you can cope with that by means of, e.g., DHCP, there's another serious problem:
Local file services in general, and Windows roaming profiles in particular.

Thus, it would be nice to connect A and B in a way, that's transparent to all network devices (like a "simple" network cable is), such that A and B essentially become one joint location.


With some friendly hints of James Chapman I was able to do exactly that by using the L2TPv3 support, which is part of the Linux kernel since 2.6.38.

I followed these simple steps to achieve my goal (s.b. for individual results):

• Set up two Linux machines with kernel >=2.6.38 (I use 2.6.39), one for each location. Each machine has to have (at least) two physical network interfaces, one connected to the internet, the other to a switch in the internal network of the respective site. The switch-ports have to be tagged with all the VLANs you want to tunnel. On a side note, CPU power of the machines is only important, if you want/have to do IPsec (which I won't cover here), so the machines can be relatively low end.
• Install the "iproute2", "vconfig", "bridge-utils" and James' "l2tpv3tun" utilities (names of actual packages may vary, depending on your preferred distribution - I use Gentoo.)
• Set up the outbound interfaces (the ones connected to the Internet) on both machines, as you usually would with your distribution. For simplicity's sake let's assume, that site A has got the outbound IP-Address 1.2.3.4 and B, 5.6.7.8. Make sure, A can reach B properly before continuing.
• Use the l2tpv3tun utility to create a L2TPv3 tunnel between A and B, like so:
  
  Code:
  site-A:# l2tpv3tun add tunnel tunnel_id 3000 peer_tunnel_id 4000 encap udp \
             local 1.2.3.4 remote 5.6.7.8 udp_sport 5000 udp_dport 6000
  site-A:# l2tpv3tun add session tunnel_id 3000 session_id 1000 peer_session_id 2000
  
  site-B:# l2tpv3tun add tunnel tunnel_id 4000 peer_tunnel_id 3000 encap udp \
             local 5.6.7.8 remote 1.2.3.4 udp_sport 6000 udp_dport 5000
  site-B:# l2tpv3tun add session tunnel_id 4000 session_id 2000 peer_session_id 1000
  
• On both sites, you should now see an interface l2tpeth0. If not, double check, whether the value pairs you chose for remote and local, udp_sport and udp_dport, tunnel_id and remote_tunnel_id, as well as session_id and remote_session_id all compliment each other for sites A and B. (In other words: Site B has to be like site A, only "backwards".)
• Set the link of the l2tpeth0 interfaces on both sites to up state and change their MTU to 1500 (this is important!) like so:
  
  Code:
  ip link set l2tpeth0 up mtu 1500
• Now set up your VLANs by tagging them (a) to the internal interfaces of A and B (let's assume they are both named eth1, you just have to determine the N of your internal ethN by yourself) and (b) to the l2tpeth0 on both ends. Also, don't forget to bring up the links. So, repeat for each VLAN 'VLAN_ID':
  
  Code:
  vconfig add eth1 VLAN_ID
  ip link set eth1.VLAN_ID up
  vconfig add l2tpeth0 VLAN_ID
  ip link set l2tpeth0.VLAN_ID up   # MTU will be inherited from l2tpeth0
  
• Now, for each VLAN 'VLAN_ID', create a bridge between the l2thpeth0.VLAN_ID and eth1.VLAN_ID. Then bring up the bridge in promiscuous mode (It makes sense to name the bridges like the respective VLAN_IDs, too - so for VLAN 5, the bridge interface would be named br5. Anyway, that's a deliberate choice, and you can call your bridge interfaces whatever you want.)
  
  Code:
  brctrl addbr brVLAN_ID
  brctl addif brVLAN_ID eth1.VLAN_ID
  brctl addif brVLAN_ID l2tpeth0.VLAN_ID
  ip link set brVLAN_ID up promisc on
  
  
  And that's it! Now, each and every IP packet in all tunneled VLANs is seen on both sites. Of course, you should keep in mind, that the tunnel represents a bottleneck, especially if slow Internet connections are involved. But deploying QoS or rate-limiting techniques to that setup would be another tutorial.

So, after applying the above setup, everything should work, as if sites A and B would be connected by a wire between their Switches. You can easily test that...

• ... by letting a client computer at site B get its DHCP lease from a DHCP server at site A.
• ... on a client machine at site B, connected to VLAN 8, by logging in a Windows user with a roaming profile, hosted on a machine in VLAN 2 on site A, with VLAN routing done through a routing switch on site A. Then the same, if the routing switch is in location B.
• ... by countless other (IP based!) network access scenarios, you would usually have to happen on your local network.

The bottomline is: Everything IP based you usually do in your routed VLAN setup should now work transparently across the tunnel!

If you have any further questions, inquiries or suggestions, feel free to contact me.

Have a nice weekend,
Torsten

After some tests with our tunnel setup, I still want to give a deeper insight into one issue.

In my first post, I wrote:



As the experts among you might have noticed, doing this kind of "dirty trick" leads to packet fragmentation, as soon as packets larger than 1488 bytes in size (1500 - L2TPv3 header) arrive at one of the bridge interfaces. Packet fragmentation in turn significantly decreases tunnel throughput.

So, why did I suggest to "artificially" raise the l2tpeth0's MTU in the first place?

Usually, a mechanism called path MTU discovery (PMTU) takes care of finding the largest possible MTU for each TCP connection (cf. http://www.netheaven.com/pmtu.html). However, this doesn't seem to work for my setup. In fact, if you stick to the default values of all involved MTUs, the following will happen:

• eth0 and eth1 will both have the default Ethernet MTU of 1500 bytes
• l2tpeth0 will have (Ethernet MTU - L2TPv3 header) = 1488 bytes
• the bridge interfaces will inherit the smallest MTU involved, i.e. the one from l2tpeth0 = 1488 bytes

If, in this setup, a client sends a packet through the tunnel, which exceeds 1488 bytes in size, it will never receive an answer. This happens because the PMTU of 1488 bytes (for some mysterious reason) will never become negotiated as it should be per PMTU discovery.
Now, the most obvious solution would be to lower the MTU of each attached client to 1488 bytes, but for a heterogenous network of about 250 client machines this is impractical, to say the least.

Raising the MTU of l2tpeth0 to 1500 bytes of course also circumvents the problem, but leaves a fragment for every packet > 1488 bytes that has to pass the tunnel. These fragments again need to be encapsulated for L2TPv3, are then sent through the tunnel and acknowledged by the server. In the (very common) case of network transfers involving many full-sized packets, this can easily lead to a performance decrease of 90% (!!!). I will give some empirical evidence on that later on.

After giving it some thought, however, I came up with a much more elegant solution.

Instead of raising the MTU of l2tpeth0 to 1500 bytes, I lowered the MTU of eth1 to 1488 bytes:



Afterwards, I needed to make sure that both servers and clients on either end of the tunnel use the smaller packet size. To accomplish that, the bridges have to propagate the smaller MTU in a proper way to all machines involved in a TCP session.
This can be done by adjusting the maximum allowed payload (maximum segment size - MSS) of all forwarded packets with an iptables one-liner:



The MSS usually equals to MTU minus 40 bytes, so for this setup it comes to 1448 bytes. The parameter --mss 1448:1536 leaves packets smaller than 1448 and larger than 1536 bytes untouched to avoid unnecessary interactions.

Note however, that you need the appropriate Linux kernel modules for the above command to be accepted. If in doubt about the correct configuration, read the iptables documentation.

As for the promised empirical figure, copying a large file from a Windows server to a client machine through the tunnel yielded about 3 MBytes per second with fragmentation and 35 MBytes per second on the same physical link while using the iptables-approach.
Quite an improvement, isn't it?

Just a quick note about l2tpv3tun. The l2tpv3tun commands have been integrated into the standard Linux ip utility (the iproute2 package) by the iproute2 maintainer, Stephen Hemminger. These are available in the iproute2-3.2 package (for the 3.2 kernel).

To use ip instead of l2tpv3tun, simply replace l2tpv3tun with "ip l2tp" in each command, i.e.


becomes


Use l2tpv3tun only when you have kernel a kernel older than 3.2.

Hi torsti

pls tell me how to configure l2tpv3tun to work with cisco router.

Thanks

I didn't tested yet, but I am going setup today.
If I will join one bridge interface multiply l2tpeth0 and eth0.0 .... that each will represent separate vlan and for each eth interface assign IP to use as gateway from the server behind. That should use multiply vlan through the tunnel ?

torsti, thanks very much for putting this article together. This is helping a great deal, I have had a hard time finding a how to l2tp3v that explain as much as you have.

I have a tunnel up, and a session up. I have a single windows box on the inside of each machine. My 2 windows boxes cannot communicate with each other. WinA has an arp entry for WinB. However WinB does not have an arp entry for WinA.

When I tcpdump on either of the linux boxes all I see is arp requests.

I am doing a proof of concept for a L2 failover mechanism across sites. Your help is greatly appreciated. I am happy to send more info if needed.

Hello Everyone,
This my setup


Desktop int eth1 VLAN 300 -------> Linux Router ( bridge for vlan 200 and bridge for vlan 300 ) <---------> Ipsec Tunnel Openswan <-------> Linux Router ( bridge for vlan 200 and
Desktop int eth1 VLAN 200
bridge for vlan 300 and 200 ) ----> Desktop int eth1 VLAN 300
Desktop int eth1 VLAN 200


I can ping from linux router A linux router B on l2tpv3 ends ips and corresponding bridges.
Example.: ping -I br_vlan 300 ip address on anther end.

But trouble is I can't ping nothing from Desktops behind router for corresponding VLAN and ips.




Any help or ideas welcome.

Thank you in advance.

Chuyasin,

first of all sorry for the late reply.

I don't think you'll be able to connect to a Cisco router using the l2tpv3tun Linux command. Cisco - as always - uses a proprietary protocol to do its 'Pseudo-Wire'. So, if you have big Cisco routers on both ends of your connection, you should use the Pseudowire feature as documented by Cisco. I don't own any Cisco device, so I can't help you here.
If Pseudowire is not available on you devices, you have to use Linux boxes on both ends according to my tutorial. You, however, have to configure your routers/firewalls in such a way, that the necessary UDP ports (as defined by your l2tpv3 configuration) are forwarded to your boxes.

Best,
Torsten

You are welcome!


Although I'm not sure, I think you have some kind of MTU issue. For a test, try to reduce the MTU of both Windows boxes to some very small value (e.g. 1300). This can be done either by tools, or by directly editing the respective registry entries.

Google will help you here.

Best,
Torsten

Hi volga629,

did you take into account that the Ipsec Tunnel further reduces your MTU? You should try to find out how many bytes the packet header for your Openswan connection uses and subtract that number from the MTU values I've given above.

Please, tell me if this works for you.

Best,
Torsten