Repository navigation
io.js 3.0.0 memory leak #2308
Description
Activity
I've also got an app I'm test running at 3.0.0 and after 4 hours it uses the double amount of memory as it used on 2.5.0. And it seems to be climbing on 3.0.0.
@jbergstroem Sorry, I'm just getting on board to SFO now.
Is there a test-case that be used to reproduce the memory leak?
Not that I can think of, unless you already have a system running PM2. Up until the server machine crashes there are no error messages or logs to track it down with. It is most certainly an issue only on io.js 3.0.0, though. I reverted back to 2.5.0 and memory usage has been stable for almost 10 hours now.
fwiw you can't assume that high memory usage and growing early memory usage are a memory leak, the way V8 handles memory and GC changes over time so you may possibly be dealing with a "normal" memory profile. It'd be good to have data for a longer run.
Based on the original report in Unitech/pm2#1500, it seems that the growth rate is 30MB/min and the app runs out of memory in a couple of hours. This does smell like a memory leak, but It would be very hard to debug this, or to know if the memory leak is actually in io.js, without a test case that shows the problem. At the least, can you grab and compare heap-snapshots and see what kinds of objects are growing?
Well, I changed nothing other than upgrading io.js, and tried out both old, current and pre-release versions of PM2. I even tried out bumping down my dependencies from the last time I upgraded those.
When viewed with htop, I saw 500+ MB (kept growing until the server crashed) for each process with 3.0.0, versus about 40-60MB stable per process with 2.5.0. I wouldn't call a 1000% memory increase with up to 2MB per second continuous growth at idle a "normal" curve.
Perhaps it is completely normal if the bug is in PM2 instead, and previous versions' GC still managed to collect it somehow while 3.0.0 doesn't. My actual application code and processes seems unaffected; only the PM2 processes have such high memory usage. The PM2 processes also handle all the cluster and load balancing stuff, which could be more affected by the new Buffer implementation than my code is. I honestly don't know, but I'm willing to give you any data I can possibly provide.
To people experiencing the same issue: The best way to drive this issue forward is more metrics over time and a reduced proof of concept we all can test.
I'm sorry, but I honestly have no idea how to take a heap snapshot of the individual PM2 daemons.
@novacrazy try importing heapdump and send it a unix signal.
Alright, I injected
require('heapdump');into Common.js so all the daemons should have it. Now I just need to wait a bit for things to accumulate and for my rural internet connection to download the files. I'll post the (probably massive) results in a little while.Before you upload the heapdump, be aware that it contains the full JavaScript heap and may include private information (such as user data, keys, etc.) and the source code of your JavaScript functions.
@ofrobots Very good advice, thank you.
Also, @bnoordhuis, heapdump fails to compile with io.js 3.0.0: http://pastebin.com/RVnY5Mrs
Not sure if that is my fault or part of the last V8 upgrade.
I just closed out a similar issue 30 seconds ago. :-)
I don't think that's an issue with node-heapdump but with V8. It's a C++11 project and you'll need at least gcc 4.8 or clang 3.4. You could get away with older versions until now but no more.
64 remaining items
While the major leak will be fixed in the upcoming 3.1.0, I think there's still a few minor leaks around. Please post test cases if you have any!
Current master (actually https://iojs.org/download/nightly/v3.0.1-nightly201508173645dc62ed/) looks pretty good, the https test is showing a possible very slow leak, it's creeping up really slowly but I'm not confident calling it a leak. Will leave it going and see what happens but the main leak(s) appear to be resolved.
the https test is showing a possible very slow leak
Just in case: could you confirm that there is no mistype? The 3.0.1 memory usage on your graph looks constant with https-ping (the bottom one) and very slightly growing over time with http-ping (the upper one) to me.
@ChALkeR sorry, you're absolutely correct, it's the http-ping that appears to be leaking while https-ping is stable, which is very strange!
http-ping stabilises at around 75k and then gradually, but steadily, adds 10k more by the end of the graph.
https-ping fluctuates between 71k and 72k for the entire length.
it's the http-ping that appears to be leaking while https-ping is stable
http-ping has reached steady state in the last 6 hours.
It's probably just memory arena fragmentation or the new v8 engine performing GC less aggressively in this low memory test. It appears to have levelled off.
@rvagg - can you provide a link to your RSS ping test source code?
On a sidenote, my test case with FormData was just invalidated, the issue seems to have been a unconsumed stream (https://gist.github.com/silverwind/54b3829142f93d1127bc#gistcomment-1552967)
ditto, and see https://github.com/rvagg/node-memtest/tree/master/test directly for source of the tests, rss is collected using
psso it works on osx and linux: https://github.com/rvagg/node-memtest/blob/master/memtest.sh#L50
Memory usage settled right down to stable in 3.0.1 (nightly), so call off the dogs on that one.
I'll close this one per the graph above. We don't have any more indicators for more major leaks like the one in 3.0.0.
See Unitech/pm2#1500 for the bug report filed with PM2 before I realized it was being caused by io.js.
With io.js 2.4.0 and 2.5.0 there were no memory leaks for that module. Perhaps it could have something to do with the new Buffer implementation?