It is currently Sun Jan 20, 2019 9:36 am

View unanswered posts | View active topics


Board index » Help

All times are UTC [ DST ]



Post new topic Reply to topic  Page 1 of 2
  [ 17 posts ]  Go to page 1, 2  Next
  Print view Previous topic | Next topic 
Author Message
Orion33
 Post subject: Memory leak on TX
PostPosted: Sun Feb 19, 2012 9:43 am 

Joined: Mon Aug 30, 2010 7:10 pm
Posts: 15
Hello jchapman!

Do you know of a memory leak bug on TX channel created by ppp connection? You can google it by keywords "pppd bug soft lockup cpu # 0 stuck for". This message is issued to the console, and nothing can be done with a PC, just reboot.

However, I have long been using this provider (Beeline Russia) and I don't remember that before this problem I had. I found the old kernel 2.6.32.11 and my config and tried to run with them. Now the system does not hang and I can see what happens.

So, I use latest version openl2tp (1.8) and kernel 2.6.32.11 with my config.

Here are boot logs

Code:
Feb 19 16:12:40 server openl2tpd[959]: Start, trace_flags=00000000
Feb 19 16:12:40 server openl2tpd[959]: OpenL2TP V1.8, (c) Copyright 2004-2010 Katalix Systems Ltd.
Feb 19 16:12:40 server openl2tpd[959]: Loading plugin /usr/lib/openl2tp/ppp_unix.so, version V1.5
Feb 19 16:12:40 server openl2tpd[959]: Using config file: /etc/openl2tpd.conf
Feb 19 16:12:40 server INADYN[965]: INADYN: Started 'INADYN version 1.96.2' - dynamic DNS updater.
Feb 19 16:12:40 server pppd[971]: Plugin pppol2tp.so loaded.
Feb 19 16:12:40 server pppd[971]: Plugin openl2tp.so loaded.
Feb 19 16:12:40 server pppd[971]: pppd 2.4.5 started by root, uid 0
Feb 19 16:12:40 server pppd[971]: Using interface ppp0
Feb 19 16:12:40 server pppd[971]: Connect: ppp0 <-->
Feb 19 16:12:40 server pppd[971]: CHAP authentication succeeded
Feb 19 16:12:40 server pppd[971]: CHAP authentication succeeded
Feb 19 16:12:40 server pppd[971]: not replacing existing default route via 10.75.72.1
Feb 19 16:12:40 server pppd[971]: local  IP address 95.26.45.196
Feb 19 16:12:40 server pppd[971]: remote IP address 83.102.254.195
Feb 19 16:12:41 server dhcpd: Internet Systems Consortium DHCP Server 4.1.1-P1
Feb 19 16:12:41 server dhcpd: Copyright 2004-2010 Internet Systems Consortium.
Feb 19 16:12:41 server dhcpd: All rights reserved.
Feb 19 16:12:41 server dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb 19 16:12:42 server dhcpd: WARNING: Host declarations are global.  They are not limited to the scope you declared them in.
Feb 19 16:12:42 server dhcpd: Not searching LDAP since ldap-server, ldap-port and ldap-base-dn were not specified in the config file
Feb 19 16:12:42 server dhcpd: Wrote 0 deleted host decls to leases file.
Feb 19 16:12:42 server dhcpd: Wrote 0 new dynamic host decls to leases file.
Feb 19 16:12:42 server dhcpd: Wrote 1 leases to leases file.
Feb 19 16:12:42 server dhcpd:
Feb 19 16:12:42 server dhcpd: No subnet declaration for eth1 (10.75.74.115).
Feb 19 16:12:42 server dhcpd: ** Ignoring requests on eth1.  If this is not what
Feb 19 16:12:42 server dhcpd:    you want, please write a subnet declaration
Feb 19 16:12:42 server dhcpd:    in your dhcpd.conf file for the network segment
Feb 19 16:12:42 server dhcpd:    to which interface eth1 is attached. **
Feb 19 16:12:42 server dhcpd:
Feb 19 16:12:42 server dhcpd: Listening on LPF/eth0/00:19:5b:6c:88:9a/192.168.0.0/24
Feb 19 16:12:42 server dhcpd: Sending on   LPF/eth0/00:19:5b:6c:88:9a/192.168.0.0/24
Feb 19 16:12:42 server dhcpd: Sending on   Socket/fallback/fallback-net
Feb 19 16:13:01 server INADYN[965]: W: DYNDNS: Error 'RC_IP_CONNECT_FAILED' (0x13) when talking to IP server
Feb 19 16:13:01 server INADYN[965]: W:'RC_IP_CONNECT_FAILED' (0x13) updating the IPs. (it 0)
Feb 19 16:13:40 server pppd[971]: LCP terminated by peer
Feb 19 16:13:40 server pppd[971]: Connect time 1.0 minutes.
Feb 19 16:13:40 server pppd[971]: Sent 1460861250 bytes, received 18113 bytes.
Feb 19 16:13:40 server pppd[971]: Terminating on signal 15
Feb 19 16:13:43 server pppd[971]: Connection terminated.
Feb 19 16:13:44 server pppd[971]: Modem hangup
Feb 19 16:13:44 server pppd[971]: Exit.
Feb 19 16:13:46 server pppd[1255]: Plugin pppol2tp.so loaded.
Feb 19 16:13:46 server pppd[1255]: Plugin openl2tp.so loaded.
Feb 19 16:13:46 server pppd[1255]: pppd 2.4.5 started by root, uid 0
Feb 19 16:13:46 server pppd[1255]: Using interface ppp0
Feb 19 16:13:46 server pppd[1255]: Connect: ppp0 <-->
Feb 19 16:13:49 server pppd[1255]: CHAP authentication succeeded
Feb 19 16:13:49 server pppd[1255]: CHAP authentication succeeded
Feb 19 16:13:49 server pppd[1255]: not replacing existing default route via 10.75.72.1
Feb 19 16:13:49 server pppd[1255]: local  IP address 95.26.45.196
Feb 19 16:13:49 server pppd[1255]: remote IP address 83.102.254.195


You can see that the first connection is created for one minute and passes on the TX 1.4Gb. With constant running ifconfig shows how the traffic is growing at a rate of about 1 GB per second, up to 4GB, reset, and increases again. It then recalls, and everything is fine. In the first minute of any sites do not ping, although the connection is. On the forums write about loopback traffic, but route to pool is present
Code:
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
85.21.192.3     10.75.72.1      255.255.255.255 UGH   0      0        0 eth1
217.118.84.75   10.75.72.1      255.255.255.255 UGH   0      0        0 eth1
213.234.192.8   10.75.72.1      255.255.255.255 UGH   0      0        0 eth1
83.102.254.0    10.75.72.1      255.255.255.0   UG    0      0        0 eth1
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth1
10.72.0.0       0.0.0.0         255.252.0.0     U     0      0        0 eth1
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0 ppp0

Net 83.102.254.0/24 is l2tp pools.
So I do not know what else to say to you for information, ask me, what other report needed to solve this problem.
PS. Sorry for my bad english, I use google translit :roll:
Top
 Profile  
 
jchapman
 Post subject: Re: Memory leak on TX
PostPosted: Sun Feb 19, 2012 2:02 pm 
Site Admin

Joined: Sun Jul 27, 2008 1:39 pm
Posts: 122
Orion33 wrote:
Do you know of a memory leak bug on TX channel created by ppp connection? You can google it by keywords "pppd bug soft lockup cpu # 0 stuck for". This message is issued to the console, and nothing can be done with a PC, just reboot.

I wasn't aware of this. But googling suggests it might be a ppp problem with the kernel you are using. Have you tried a more recent kernel?
Top
 Profile  
 
Orion33
 Post subject: Re: Memory leak on TX
PostPosted: Sun Feb 19, 2012 4:02 pm 

Joined: Mon Aug 30, 2010 7:10 pm
Posts: 15
jchapman wrote:
I wasn't aware of this. But googling suggests it might be a ppp problem with the kernel you are using. Have you tried a more recent kernel?

More recent - a 3.0 +? No, haven't tried
But the writing on the forums that have the same problem on Ubuntu 11.04 with kernel 3.x
What kernel do you advise me to try?
Top
 Profile  
 
jchapman
 Post subject: Re: Memory leak on TX
PostPosted: Sun Feb 19, 2012 7:04 pm 
Site Admin

Joined: Sun Jul 27, 2008 1:39 pm
Posts: 122
Orion33 wrote:
jchapman wrote:
Have you tried a more recent kernel?

More recent - a 3.0 +? No, haven't tried
But the writing on the forums that have the same problem on Ubuntu 11.04 with kernel 3.x
What kernel do you advise me to try?

It is usually best to try the newest kernel possible as kernel developers are more likely to be interested in solving the problem. If you can build from source, even better is to try the latest development kernel from git://git.kernel.org/pub/scm/linux/kern ... t-next.git, but only try that if you are already familiar with building your own kernel from source.
Is the problem easy to reproduce?
Top
 Profile  
 
Orion33
 Post subject: Re: Memory leak on TX
PostPosted: Mon Feb 20, 2012 11:23 am 

Joined: Mon Aug 30, 2010 7:10 pm
Posts: 15
Ok, I'll try the new kernel, when there will be free time. Not sure what will happen quickly apply my configuration to the new kernel.

Quote:
Is the problem easy to reproduce?

For the provider Beeline Russia - yes. There are at least in two cities. How reproduce this problem for you, I don't really understand. I'll check it is possible to connect to the l2tp server from an external IP. If so, I'll give you a login and password for this test.
Top
 Profile  
 
Orion33
 Post subject: Re: Memory leak on TX
PostPosted: Mon Feb 20, 2012 3:55 pm 

Joined: Mon Aug 30, 2010 7:10 pm
Posts: 15
You can try test login and password **** to server ******. Try "host *****" for more IP address. Service must be started automatically through init.d. If to start daemon manually, it works normal.

[Post edited to remove test account info.]
Top
 Profile  
 
jchapman
 Post subject: Re: Memory leak on TX
PostPosted: Tue Feb 21, 2012 1:26 pm 
Site Admin

Joined: Sun Jul 27, 2008 1:39 pm
Posts: 122
Orion33 wrote:
You can try test login and password **** to server ******. Try "host *****" for more IP address. Service must be started automatically through init.d. If to start daemon manually, it works normal.

Thanks for setting up access to your server.
We have reproduced the problem. It is caused by the server configuring a local IP address for its PPP endpoints which is the same address as the public IP of the L2TP server. For example, your L2TP server has IP address 1.2.3.4 and sets the local address of each PPP interface to 1.2.3.4. L2TP clients need to reach your L2TP server at 1.2.3.4, while one of its PPP sessions gets assigned the same address as the server.
There is a bug in the Linux driver which seems to result in a transmit routing loop in this situation - a packet that is transmitted through the PPP interface is somehow intercepted by the IP stack (because the IP desination address is also to the L2TP server) and the packet ends up looping in the kernel, increasing in size each time it is transmitted.
Please make sure that the PPP interface configured for PPP interfaces is _not_ the same as the server's public IP.
Meanwhile, I will ask the Linux kernel developers for the best way to prevent the routing loop to protect against configurations like this.
Thanks again for posting about this issue.
Top
 Profile  
 
Orion33
 Post subject: Re: Memory leak on TX
PostPosted: Wed Feb 22, 2012 6:07 am 

Joined: Mon Aug 30, 2010 7:10 pm
Posts: 15
Thank you for your reply
How do I configure the ppp interface? Since you have access to this issue, could you help me with this? Here is my configuration

Code:
# system
system modify \
max_sessions=0 \
tunnel_establish_timeout=120 \
session_establish_timeout=60 \
deny_remote_tunnel_creates=no \
tunnel_persist_pend_timeout=10 \
session_persist_pend_timeout=5 \

ppp profile modify profile_name=default \
        default_route=yes \

# locally created tunnels and sessions
tunnel create tunnel_name=corbina \
        dest_ipaddr=83.102.254.195 \
        persist=yes \
        max_retries=9999 \

session create tunnel_name=corbina \
        user_name=<my_login> \
        user_password=<my_pass> \
        interface_name=ppp0 \


Tunnel & ppp interface becomes following settings

Code:
ppp0      Link encap:Point-to-Point Protocol
          inet addr:<public_addr>  P-t-P:83.102.254.195  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1460  Metric:1
          RX packets:841 errors:0 dropped:0 overruns:0 frame:0
          TX packets:908 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:73571 (71.8 KiB)  TX bytes:71361 (69.6 KiB)

Code:
Tunnel 34142, from 10.75.74.115 to 83.102.254.195:-
  state: ESTABLISHED
  created at:  Feb 22 11:50:04 2012
  administrative name: 'corbina'
  created by admin: YES, tunnel mode: LAC, persist: YES
  peer tunnel id: 49334, host name: NOT SET
  UDP ports: local 40234, peer 1701
  authorization mode: NONE, hide AVPs: OFF, allow PPP proxy: OFF
  session limit: 0, session count: 1
  tunnel profile: default, peer profile: default
  session profile: default, ppp profile: default
  hello timeout: 60, retry timeout: 1, idle timeout: 0
  rx window size: 10, tx window size: 10, max retries: 9999
  use udp checksums: ON
  do pmtu discovery: OFF, mtu: 1460
  framing capability: SYNC ASYNC, bearer capability: DIGITAL ANALOG
  use tiebreaker: OFF
  trace flags: NONE
  peer vendor name: Cisco Systems, Inc.
  peer protocol version: 1.0, firmware 4384
  peer framing capability: NONE
  peer bearer capability: NONE
  peer rx window size: 32000
  Transport status:-
    ns/nr: 4/81, peer 4/80
    cwnd: 5, ssthresh: 10, congpkt_acc: 0
  Transport statistics:-
    out-of-sequence control/data discards: 0/0
    zlbs tx/txfail/rx: 80/0/2
    retransmits: 0, duplicate pkt discards: 0, data pkt discards: 0
    hellos tx/txfail/rx: 0/0/79
    control rx packets: 83, rx bytes: 1745
    control tx packets: 84, tx bytes: 1226
    data rx packets: 1411, rx bytes: 103217, rx errors: 0
    data tx packets: 1457, tx bytes: 110986, tx errors: 0
    establish retries: 0


Have you tried the latest kernel? There also exists the problem?

PS. Username and password from previous post can be left because it is not my but a public test access by my provider Beeline Russia, from which is available the home page corbina.net only.
Top
 Profile  
 
jchapman
 Post subject: Re: Memory leak on TX
PostPosted: Wed Feb 22, 2012 11:19 am 
Site Admin

Joined: Sun Jul 27, 2008 1:39 pm
Posts: 122
Orion33 wrote:
How do I configure the ppp interface? Since you have access to this issue, could you help me with this? Here is my configuration

That's the problem - you can't. :( The IP address is assigned by the L2TP server. When the L2TP server assigns you its public IP address for its PPP sessions, it is actually quite dangerous because clients are able to access the server's IP stack using a public test account!

Orion33 wrote:
Have you tried the latest kernel? There also exists the problem?

Yes.

Orion33 wrote:
PS. Username and password from previous post can be left because it is not my but a public test access by my provider Beeline Russia, from which is available the home page corbina.net only.

Ok.

We are working on changes to the L2TP driver that will detect this unusual setup to avoid the packet loop. It isn't an OpenL2TP-specific problem.

When you connect using your actual subscriber account, do you get an IP address for your PPP interface which is different to that used by the L2TP server? If so, does it work as expected?
Top
 Profile  
 
Orion33
 Post subject: Re: Memory leak on TX
PostPosted: Wed Feb 22, 2012 2:07 pm 

Joined: Mon Aug 30, 2010 7:10 pm
Posts: 15
jchapman wrote:
When you connect using your actual subscriber account, do you get an IP address for your PPP interface which is different to that used by the L2TP server? If so, does it work as expected?

While I watched it, yes, each time the addresses were different. But the problem is that the pool has a lot of addresses, possibly some other subscriber is allocated the address to which I am trying to connect. This confirms the fact that the problem occurs not always but often.
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  Page 1 of 2
  [ 17 posts ]  Go to page 1, 2  Next

Board index » Help

All times are UTC [ DST ]

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group