Memory issues - Create a better Ajax poll?

GregP

Hey all,

I'm having memory issues with my application. I use a simple setInterval to poll server-side data with Ajax (really, letting DataTables handle the Ajax when I call fnDraw), and implemented a clearInterval tied to an onClick event in order to create a pause in the polling. Memory usage only climbs when actively polling.

I've checked that my method is not appending elements to the DOM without destroying them, and it is not. More accurately, the fnDraw seems to be destroying all the elements created by DataTables and also from within DataTables callbacks. So the leak isn't in the DOM elements.

I checked for circular references, and other than circular references native to jQuery 1.5.1 (which theoretically should be garbage-collected) I can't find anything. Which leads me to believe that somehow I'm not handling the incoming data properly with my overly simple polling function.

Here's a simplified version of what I'm doing:

[code]
fcmc.taskProgressDraw = function() {
$("#task_progress").dataTable().fnDraw();
}

fcmc.taskPollingInterval = setInterval('fcmc.taskProgressDraw()', 2000);
[/code]

I've also tied it all together in a single statement (instead of breaking it out into a function), but breaking it out into a function was an attempt to close circular references. In any event, either way of setting the interval has the same net effect. I'm not 100% sure it's the cause of the memory leak; just trying to tackle possibilities one at a time, and this is the first suspect.

Can anyone recommend a better way to handle the polling? My concern is that I'm not handling success or failure before polling again. I wonder if unresolved polls are building up a cache of data that's not being dealt with..? Kind of grasping at straws a bit here.

Cheers,
Greg

PS, it occurs to me that Webkit's garbage collection is just broken, as well... but I don't want to just make that assumption without trying to optimize my code first.

allan

Hi Greg,

First thing to check is that you are using the latest version of DataTables (1.7.6). Also, are you using Chrome or Safari - since they use different JS engines? Chrome's memory debugging tools are superb, and might be able to shed some light on what is happening.

The basic idea looks fine to me. Personally I would use setInterval(fcmc.taskProgressDraw, 2000); since it just saves one extra eval, but it won't really make any difference. It is possible that there is a closure in DataTables preventing the memory from being freed, but I've done a bit of work on that and I don't see where there would be one if there is. Any chance of a link?

Allan

GregP

I though I was using the latest, but I am not! I'll update to 1.7.6 and let you know if it helps.

I also got rid of that extra eval.

For the tests, I'm using Chrome in Win7, but the same issue persists in Safari on Mac as well. As it turns out, I thought Fx was safe, but it might be leaking, too. I have a lot of refactoring to do with the code, so it might be worth waiting until I think I have the 'best' version for you to inspect, but I can get you a public link into our test box if you want to have a peek anyhow.

I agree that the Chrome tools have awesome tools for checking if there's an issue. As far as I can tell, yes there's an issue-- I'm adding objects with each pass. I assume they're JavaScript objects (not DOM) because I have a consistent number of DOM elements no matter how many times I redraw the tables. In any event, the problem for me isn't confirming that there's an issue, it's figuring out how to solve it. ;-)

Since it's a commercial and unreleased project we're working on I'd have to give you the link privately. Drop me an email and I'll forward it to you. In the meantime, I'll try out 1.7.6!

Thanks,
Greg

GregP

Alrighty... up to jQuery 1.5.2 and DataTables 1.7.6.

On top of that, I did a bunch of "overkill" stuff such as setting variables and objects to null once I was done with them.

1.7.6 alone seemed to be an improvement for Webkit and IE9. There's a leak somewhere, but it's slower (IE9, for example, took this past hour to climb from ~35 to ~80MB. I believe Chrome is trending at roughly the same speed). Firefox, however, is still leaking like a sieve.

As far as I can tell, I plugged up any leaks on my end. But this is my first major attempt at plugging a memory leak; possibly because it's also my first polling application.

Some Googling reveals that jQuery-based applications tend to be trickier to deal with in terms of leaks (due to jQuery itself either leaking or making it too easy for developers to code with bad practices). Memory Leak Checker in IE6/IE7 (which had awful garbage collection) shows circular references in jQuery itself, confirming in part that it's a possible culprit.

I'm still not convinced it's purely on jQuery, though. Call it low self-confidence, but I keep squinting at my code with a skeptical eye.

What would be interesting to me would be to find out if anyone else out there is doing a polling-based application with DataTables, and have they ever checked for memory growth.

kerauno

I'm doing the same polling every 10 seconds and experiencing memory growth. Using the "sIeve" tool, i see orphan elements for the tr's that have been cleared using fncleartable. These 'tr' orphan elements duplicate each time i do a fncleartable. The only way i've seen to clear them out is to do a page refresh. Then they are released.

I'd love it if someone could help.

Using dataTables 1.8.2 and JQuery 1.7.1

By the way, fantastic easy feature rich product Allan. kudos to you for doing such a great job.

Thanks

GregP

Kerauno,

I admit I haven't monitored memory consumption lately. At one point I implemented an "away" event listener, so if the user leaves the page for X number of minutes, the polling pauses. This allowed "something" to clear memory (my guess: the garbage collector wasn't working for reasons related to the frequency of my poll, but that's as far as I can guess).

So, the first "solution" was to disable polling when the user wasn't active (of course, when they return the polling resumes).

Next, I am pretty sure my implementation was flawed.

I was originally using a basic initialization:

[code]
$('#myTable').dataTable( { params } );
[/code]

Then the poll looked a little something like this (going by memory):

[code]
var poll = setInterval('$("#myTable").dataTable().fnDraw()', 1000);
[/code]

(I can't remember if I needed the '.dataTable()' in there or if $('#myTable') already had the fnDraw function as a property; I can check tomorrow).

As I progressed in DT and JavaScript in general, I realized that this pattern was a lot better:

[code]
var oTable = $('#myTable').dataTable( { params } );

var poll = setInterval(function() { oTable.fnDraw() }, 1000);
[/code]

Did it fix my memory issue? I don't know for sure, because I switched to this around the same time that I implemented the "away". But I do know that with the DataTable object (oTable) properly cached, you are certainly going to use the same references to the same object, which is a good thing for memory usage and garbage collection.

allan

> I'm doing the same polling every 10 seconds and experiencing memory growth. Using the "sIeve" tool, i see orphan elements for the tr's that have been cleared using fncleartable.

Are you attaching events to the elements or doing anything other than showing them? Between Greg and myself we've done a fair bit of work on this and I'm reasonably confident that there are no leaks in DataTables core, but we have found that garbage collectors are not aggressive enough to always clear nodes out as quickly as they are drawn. That and attaching events to the rows or their decedents at all will most certainly cause a leak.

Allan

GregP

Yes, can't believe I forgot to mention that. I was originally also using .click() for binding certain events on each row. I switched to .delegate() (and more recently to .on() to stay on top of things) and that made a big difference.

kerauno

Thanks guys for the response. i've been trying to figure this out for a month. I did disable my events and did the same test and same results.

I'm also using .on for everything now.

I do have links in my data.

After i posted this last night, i think i came up with a solution. Which" is just to update existing rows and then add new ones or delete them. This does not cause any orphan elements to show up in "sIeve" tool or at least on updating the same data.

is there anything i could do to improve performance?

I have not had a chance to thoroughly test, but my initial tests seem to work.

What do you think of the following code?

Thanks

var otable = $('#'+tableid).dataTable();
var oSettings = otable.fnSettings();

var imax_new=0;
var imax_old=0;
var imax_update=0;

if (!$.isEmptyObject(json)) {

imax_old=oSettings.aoData.length;

imax_new = json.aaData.length;
imax_update=imax_new;
if (imax_update > imax_old) imax_update=imax_old;

/*** replace existing rows ***/
for(i = 0; i < imax_update ; ++i)
{
otable.fnUpdate(json.aaData[i] , i, 1, false, false);
}

/*** add new rows ***/
for(i = imax_update; i < imax_new ; ++i)
{
otable.oApi._fnAddData(oSettings, json.aaData[i]);
}
}

if (imax_old > imax_new) {
/*** delete old rows ***/
if (imax_new == 0) {
otable.fnClearTable();
} else {

for(i=imax_new; i< imax_old;i++)
{
otable.fnDeleteRow(i, null, false);
}
}
}
otable.fnDraw();

kerauno

Also as an FYI to you both, when i added the following header, the sieve tool reported a bunch of leaks. Removing it and the leaks are gone.

kerauno

These orphan nodes are not classified as a leak, they do increase the memory usage of the browser. Probably because the garbage collector is not aggressive enough as you stated. I just haven't seen them cleared out unless i force a page refresh. I'm doing a poll every 10 seconds using setTimeout. I clear my timer before the request and reset the timer after i load the table.

When using the code below to update the table every 10 seconds, I see orphan "TR" elements increase by the number of rows i'm displaying. So if i'm displaying 10 rows, there are 60 of these in 1 minute.

otable.fnClearTable();
for (var i=0; i

GregP

You seem well beyond any advice I could offer, but just because I wasn't totally clear (and I like to cross my t's for future readers) when I switched to .on() I was setting the listener as an ancestor of the table (one that is never destroyed during redraw). You're probably already doing the same, but like I said... just like to be clear. ;-)

In theory that wouldn't matter anyhow in terms of leaks, *if* your rows are getting fully destroyed.

Since I don't use fnAddData or fnStandingRedraw or fnClearTable or any other more granular functions (I only use fnDraw to rebuild the table on each poll), I'm a lousy person to advise you on how to ensure that orphaned references are cleared.

Greg

kerauno

Greg, you've done great from what i've seen and your advice and experience is always welcome by everyone. That's how we all learn. From others... and hours and hours of fiddling.

I really appreciate your input and time.

kerauno

Allan, I agree datatables does NOT have a leak . Datatables is by far the best jquery plugin that i have seen for table driven data and with great documentation. Outstanding work you have done.

With that said, here is my end result of refreshing the table without using fnClearTable function. I re-adjusted it based on greg stating he only used fndraw and after reviewing saw that there was no reason to use the plugin fnstandingredraw since i was not clearing the table. Also applied code to check if row data has changed before doing the fnupdate. This really sped things up since i was getting script not responding in ie6 when iterating through 500 rows.

var otable = $('#'+tableid).dataTable();
var oSettings = otable.fnSettings();

var imax_new=0;
var imax_old=0;
var imax_update=0;

var bchanged = false;

if (!$.isEmptyObject(json)) {

imax_old=oSettings.aoData.length;

imax_new = json.aaData.length;
imax_update=imax_new;
if (imax_update > imax_old) imax_update=imax_old;

/*** replace exising rows ***/
for(i = 0; i < imax_update ; ++i)
{
if (json.aaData[i].toString().indexOf(oSettings.aoData[i]._aData.toString()) !== 0) {
bchanged=true;
otable.fnUpdate(json.aaData[i] , i, 1, false, false);
}
}
/*** add new rows ***/
for(i = imax_update; i < imax_new ; ++i)
{
otable.oApi._fnAddData(oSettings, json.aaData[i]);
}
}

if (imax_old > imax_new) {
bchanged=true;
/*** delete rows ***/
if (imax_new == 0) {
otable.fnClearTable();
} else {
for(i=imax_new; i< imax_old;i++)
{
otable.fnDeleteRow(i, null, false);
}
}
}
if (bchanged) otable.fnDraw();