PDA

View Full Version : Memory leak with asterisk 1.4.33.1 and GSM?



mck
01-30-11, 09:00 PM
Hi,
I have experienced a sort of memory leak running a Warp with asterisk 1.4.33.1 and gsm module: after some calls via GSM trunk, asterisk + freepbx stops working saying

WARNING[3066]: res_agi.c:313 launch_script: Failed to fork(): Cannot allocate memory

The Warp runs PADS 2.2.5.6, asterisk was upgraded following this link: http://forum.pikatechnologies.com/showthread.php?527-Procedure-to-update-the-Asterisk-version

This is the free output (Warp with 2 GSM trunk configured, no calls)
total used free shared buffers
Mem: 256220 250116 6104 0 18900
Swap: 0 0 0
Total: 256220 250116 6104

vfadmin
02-04-11, 10:39 AM
In this case what is likely necessary is a reduction of overall RAM usage in your software. Recently there have been code changes in the LCD library to reduce its RAM usage. It's memory consumption has been reduced by 22.5MB. These changes have been submitted in trunk code. If you are interest in incorporating these changes into your software the targetted changes are below.

1) Replace the lcdlib package in PADS with the one below:
http://svn.pikatech.com/pads/distro/branches/2.2/release/package/lcdlib/
2) Replace the S46astmanproxy file in the astmanproxy package in PADS with the one below:
http://svn.pikatech.com/pads/distro/branches/2.2/release/package/astmanproxy/S46astmanproxy

Although this RAM reduction is likely significant enough to address your problem, if it is not you may still need to find ways to reduce RAM usage outside of this LCD change.

mrecoskie
04-13-11, 02:26 PM
Hi mck,
What is the 'free' output of the Warp fresh off reboot?

mck
04-13-11, 05:07 PM
Hi Mark,
this is the "free" output of a warp with 1 GSM trunk configured, no calls; I have applied the patch for lcdlib and astmanproxy:

total used free shared buffers
Mem: 256220 219072 37148 0 2412
Swap: 0 0 0
Total: 256220 219072 37148





Hi mck,
What is the 'free' output of the Warp fresh off reboot?

mrecoskie
06-17-11, 02:52 PM
Hi mck,

It doesn't seem like you are running out of RAM memory. I wondering if you could be exceeding the stack limit. I would suggest increasing the stack limit and see if the problem becomes less frequent.

You can view the stack limit using:


ulimit -a

The default is set to 8192. To double this:


ulimit -s 16384

mck
06-29-11, 11:50 AM
Hi Mark,
I have tested your tips putting "ulimit -s 16384" in freepbx_engine script (just before launch of safe_asterisk).
Then I rebooted the warp and I made a few outgoing calls: about 15 calls, including 1 with a duration of 15 minutes, 2 with a duration of 4:30 minutes 1 with a duration of about 7 minutes.
So after this calls I was able to reproduce the problem: the incoming calls fail and in asterisk CLI I see:

-- Executing [s@macro-record-enable:4] AGI("GSM/1", "recordingcheck|20110629-174536|1309362336.39") in new stack
[Jun 29 17:45:36] WARNING[3524]: res_agi.c:313 launch_script: Failed to fork(): Cannot allocate memory
-- Executing [s@macro-record-enable:5] MacroExit("GSM/1", "") in new stack
-- Executing [s@macro-exten-vm:9] Macro("GSM/1", "dial||tr|100") in new stack
-- Executing [s@macro-dial:1] GotoIf("GSM/1", "1?dial") in new stack
-- Goto (macro-dial,s,3)
-- Executing [s@macro-dial:3] AGI("GSM/1", "dialparties.agi") in new stack
[Jun 29 17:45:36] WARNING[3524]: res_agi.c:313 launch_script: Failed to fork(): Cannot allocate memory
-- Executing [s@macro-dial:4] NoOp("GSM/1", "Returned from dialparties with no extensions to call and DIALSTATUS: ") in new stack
-- Executing [s@macro-exten-vm:10] GotoIf("GSM/1", "0?exit|return") in new stack
-- Executing [s@macro-exten-vm:11] Set("GSM/1", "SV_DIALSTATUS=") in new stack
-- Executing [s@macro-exten-vm:12] GosubIf("GSM/1", "0?docfu|1") in new stack
-- Executing [s@macro-exten-vm:13] GosubIf("GSM/1", "0?docfb|1") in new stack
-- Executing [s@macro-exten-vm:14] Set("GSM/1", "DIALSTATUS=") in new stack
-- Executing [s@macro-exten-vm:15] NoOp("GSM/1", "Voicemail is novm") in new stack
-- Executing [s@macro-exten-vm:16] GotoIf("GSM/1", "1?s-|1") in new stack
-- Goto (macro-exten-vm,s-,1)
-- Executing [100@from-did-direct:2] Goto("GSM/1", "|return|1") in new stack
-- Goto (from-did-direct,return,1)
[Jun 29 17:45:36] WARNING[3524]: pbx.c:2458 __ast_pbx_run: Channel 'GSM/1' sent into invalid extension 'return' in context 'from-did-direct', but no invalid handler

This is the "free" output:

total used free shared buffers
Mem: 256220 251248 4972 0 284
Swap: 0 0 0
Total: 256220 251248 4972

This is the relevant "top" output:

top - 17:51:44 up 4:39, 0 users, load average: 1.02, 1.01, 1.00
Tasks: 55 total, 1 running, 54 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.9% us, 0.5% sy, 0.0% ni, 98.6% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 256220k total, 251248k used, 4972k free, 284k buffers
Swap: 0k total, 0k used, 0k free, 31656k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2042 asterisk 20 0 275m 72m 7900 S 0.0 29.2 1:50.43 asterisk
2035 asterisk 20 0 52812 13m 10m S 0.0 5.2 0:14.62 php-cgi
2034 asterisk 20 0 52512 9924 7756 S 0.0 3.9 0:06.84 php-cgi
1892 mysql 20 0 11076 5252 4136 S 0.0 2.0 0:03.34 mysqld
2015 asterisk 20 0 49356 4136 3048 S 0.0 1.6 0:00.19 php-cgi
2026 root 20 0 3952 3492 2556 S 0.0 1.4 0:00.07 ntpd
1751 root 20 0 8768 2308 1456 S 0.0 0.9 0:05.88 astmanproxy
3538 root 20 0 2664 1196 952 R 1.8 0.5 0:00.69 top
2014 asterisk 20 0 3164 1136 572 S 0.0 0.4 0:02.83 lighttpd
2089 root 20 0 2904 1040 812 S 0.0 0.4 0:00.33 dropbear
2278 root 20 0 2904 1036 812 S 0.0 0.4 0:01.03 dropbear
2284 root 20 0 2808 772 652 S 0.0 0.3 0:01.67 sh
2094 root 20 0 2808 768 652 S 0.0 0.3 0:00.10 sh
1766 root 20 0 2808 756 628 S 0.0 0.3 0:00.48 mysqld_safe

This is meminfo:

/persistent1/var/lib/asterisk/bin # cat /proc/meminfo
MemTotal: 256220 kB
MemFree: 4972 kB
Buffers: 284 kB
Cached: 31664 kB
SwapCached: 0 kB
Active: 53196 kB
Inactive: 53000 kB
Active(anon): 40708 kB
Inactive(anon): 43944 kB
Active(file): 12488 kB
Inactive(file): 9056 kB
Unevictable: 3476 kB
Mlocked: 3476 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 77768 kB
Mapped: 23944 kB
Slab: 7660 kB
SReclaimable: 2532 kB
SUnreclaim: 5128 kB
PageTables: 996 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 128108 kB
Committed_AS: 313120 kB
VmallocTotal: 735224 kB
VmallocUsed: 5908 kB
VmallocChunk: 718820 kB


Any hints?


Regards,
Matteo




Hi mck,

It doesn't seem like you are running out of RAM memory. I wondering if you could be exceeding the stack limit. I would suggest increasing the stack limit and see if the problem becomes less frequent.

You can view the stack limit using:


ulimit -a

The default is set to 8192. To double this:


ulimit -s 16384

mrecoskie
07-01-11, 11:10 AM
Hi mck,

It now appears you are running out of RAM. This makes sense since each process has the potential to consume more memory. I would suggest reverting the stack size back to the previous setting and enabling core files. You can do this with the following two commands.

ulimit -s 8192
ulimit -c unlimited

You can use 'ulimit -a' to check if the values are set properly. Now when you encounter the problem hopefully a core file will be produced.

Another thing I would try as a test is to remove the AGI script being called to try and isolate the problem. AGI can be a large memory consumer in some cases.