It is currently Wed Aug 23, 2017 8:32 pm

All times are UTC [ DST ]




Post new topic Reply to topic  [ 2 posts ] 
Author Message
 Post subject: wrong routing leads to cpu lockup
PostPosted: Mon Dec 08, 2008 3:16 am 

Joined: Mon Dec 08, 2008 12:57 am
Posts: 1
Hi.

Suppose there is a typical l2tp connection - generally it involves 4 IP addresses:
local-eth-ip, remote-eth-ip - the outer layer, and
local-tunnel-ip, remote-tunnel-ip - inner addresses of created tunnel (ppp0 for example)

Now, suppose that the following route command was issued after connection is up:
Code:
route add remote-eth-ip dev ppp0


What happens if some packet is trying to go to the tunnel? l2tp subsystem generates IP packet to remote-eth-ip, it is routed to tunnel, which lead to generating another packet to remote-eth-ip, and so infinitly.
Userspace l2tp implementations just take 100% CPU, which is bad, but they can be stopped. In the case of openl2tp the infinite cycle runs entirely in the kernel, so the system with 1 CPU completely hangs(only sysrq works, nothing else - framebuffer cursor stops blinking), multicore systems became unusable.

How insane am I to execute such route command? Well, in my case it is the DEFAULT behaviour. Some ISPs (including corbina.ru, which I use) configured servers in a manner where remote-eth-ip=remote-tunnel-ip, so after tunnel creation pppd issued
Code:
route add remote-tunnel-ip dev ppp0
which leads to cpu lockup I described earlier.

Strictly speaking, it is not openl2tp problem, but kernel ppp/routing subsystem one, but it appears only with kernel IP-over-IP tunnel implementations (so, really, only with openl2tp).

So, there is 2 problems:
1)wrong routing leads to cpu lockup - i don't think that it's normal situation when ANY routing mismatch can cause complete system hang
2)the wrong route is added by default and pppd has no option to disable it

The only workaround I found is to fix routing in ip-up scripts. It's ok solution for home use with small network load, but it is unacceptable for big network loads because system can hang before executing of userspace ip-up scripts.

I'm not sure this forum is right place to post about that problem, but until it's resolved by kernel or pppd developers perhaps you need to add some instructions to documentation - because in my situation it really looked a lot like openl2tp problem (for me as user) - I configured some simple openl2tpd.conf with most defaults, started openl2tpd daemon and suddenly LINUX HANGS...

And, probably, you are in touch with kernel ppp subsystem and has ideas how the problem can be fixed - I looked in kernel source, but I failed to find any architecture solution which allows to check for in-kernel routing infinite loops :(

CPU lockup trace attached:
Attachment:

This trace is from latest 2.6.28-rc and openl2tp 1.6, but I have this problem for a long time, since openl2tp 0.1x versions with pppol2tp-kmod for 2.6.1x kernel.


Top
 Profile  
 
 Post subject: Re: wrong routing leads to cpu lockup
PostPosted: Tue Dec 09, 2008 1:17 pm 
Site Admin

Joined: Sun Jul 27, 2008 1:39 pm
Posts: 122
gpfault wrote:
Hi.

Suppose there is a typical l2tp connection - generally it involves 4 IP addresses:
local-eth-ip, remote-eth-ip - the outer layer, and
local-tunnel-ip, remote-tunnel-ip - inner addresses of created tunnel (ppp0 for example)

Now, suppose that the following route command was issued after connection is up:
Code:
route add remote-eth-ip dev ppp0


What happens if some packet is trying to go to the tunnel? l2tp subsystem generates IP packet to remote-eth-ip, it is routed to tunnel, which lead to generating another packet to remote-eth-ip, and so infinitly.


I think that is a configuration problem. remote-eth-ip is already reachable - there is no need to add a route entry to reach it over ppp0.

However, I do agree that it shouldn't cause the system to lock up.

Does the same thing happen if you do the same route add command with GRE or IPIP tunnels?

I'm tempted to add a check in the pppol2tp driver to check for such conditions. But if GRE/IPIP also have similar problems, perhaps the proper fix is in the IP forwarding part of the kernel.

--
James Chapman
Katalix Systems Ltd
http://www.katalix.com
Catalysts for your Embedded Linux software development


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 2 posts ] 

All times are UTC [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group