Performance with 267 columns 117 rows

jan_goyvaerts

I'm currently investigating a browser-freezing problem when using a large number of columns. I'm having a bit of trouble where to look because I'm rather new to browser/JS profiling.

However, I *think* what's happening is that the browser chokes on reflow. Because many of the bottlenecks are single statements - they are getting or setting sizes. Which I presume cause a storm of recomputations for something as complex as a table with so much columns. Maybe there's also a possible influence from the Scroller add-on (release 1.2.0) ?

For a smaller number of columns there seems to be no problem.

Is this a problem already familiar to somebody in here ?

Or any hint what alternatives I can try ? Preferably by maintaining that many columns. :-)

Thanks !

Jan

allan

Can you link to the page please so we can take a look. If you remove the computations and just put in dummy data, does that help?

Allan

jan_goyvaerts

Unfortunately I am not at liberty to make it public. But I'll try to make something in Fiddle as a proof of concept.

It would really be convenient if Scroller would also work horizontally. Because that's where the problem lies: too much markup.

jan_goyvaerts

I've prepared an example on JSFiddle: http://jsfiddle.net/DuA8v/4/

First press "Add Big table" and then "Fill table". Wait for a few seconds to see the table appear. The spent time is displayed too. The "render" time is basically the duration of the DT call. The "build" time is the time spent to create the whole thing - apart from calling DT.

The styling of the result is not really what we have here - but the intended result is there: Showing it takes 4/5s to render it. I'm no JS/CSS/... guru; hence my call for help to find out what's going on.

To "simulate" the server, the DT settings object is converted into a string and then back into a JSON object. And only then fed to DT.

Is there something obvious left to do to make this process go faster ?

jan_goyvaerts

Gave the wrong url. The correct is: http://jsfiddle.net/jan_goyvaerts/DuA8v/25/

Sorry about that.

allan

> bPaginate: false,

You are going to get no benefit from deferred rendering since you need all rows up front. So the biggest change I would suggest is enabling paging, or using Scroller.

Makes your table draw in 600mS for me: http://jsfiddle.net/DuA8v/26/

Allan

jan_goyvaerts

I've updated the jsfiddle sample - it is using Scroller and is little more parameterizable.

http://jsfiddle.net/jan_goyvaerts/DuA8v/28/

The total amount of rows is extended to 5,000. As is our use case. Reducing the columns does make it much more responsive.

Maybe this is shedding some light ? I guess I'm making a stupid mistake somewhere... :-/

allan

No stupid mistake, it just that it is taxing both DataTables and the browser!

Using DataTables 1.10 gives a performance boost of about 1.5 seconds. Disabling autoWidth gives about another 1.5 seconds.

Doing a profile on the table creation shows:

- 32% of the time is getting the scroll top position
- 30% of the time is getting the offsetWidth for the scrolling elements
- 17.5% Getting the widest element for sizing (auto width)
- 9% For initialisation

Any suggestions for how to improve any of them are very welcome. At the moment I would suggest using server-side processing. DataTables can't virtualise the columns in the same way as it can for the rows with Scroller, so the less data you ask it to process the better.

Allan

jan_goyvaerts

Doing it with a web service was my next attempt. To check the difference. I'll include the 1.10 and see if autowidth can be disabled for our usages.

I think it might be a solution to actually disable Scroller on a case basis. When the table is too wide, then let's use the classic paging mechanism. With the navigation buttons and one page in memory only. Even if there are thousands of rows.

jan_goyvaerts

Thinking about it: A violent hack that *seems* to be working for me is to run _fnScrollDraw just once. Because the result of some measures was always the same.

[code]
function _fnScrollDraw(o) {
var firstPass = $(o.nTable).data("opt-first-pass") === undefined;
if (firstPass) {
console.log("Running _fnScrollDraw()");
var start = window.performance.now();
$(o.nTable).data("opt-first-pass", false);
_fnScrollDraw_original(o);
var duration = window.performance.now() - start;
console.log("Running _fnScrollDraw() took " + duration + " ms");
}
else {
console.log("Skipping _fnScrollDraw()");
}
}
[/code]

But I'm not suggesting it's useless to do so ! :-) But maybe it means there is an opportunity to win time somewhere ?

allan

Agreed - i'm certain that there optimisations that can go in there.

Allan

jan_goyvaerts

Anothing thing I was wondering about is reusing existing the existing instead of removing and adding them each time all over again. Wouldn't that cut in the cost of rendering ?

jan_goyvaerts

Maybe it's faster also if Scroller only adds the rows to the DOM when they're about to become visible ? Instead of adding the whole prefetch buffer at once ?

Something I hacked - but failed to do it properly - is to put the rendering in a timeout function:

(1) Put the rows into a buffer array.
(2) Schedule a timeout function:
(2.1) Remove the 10 first rows from the buffer
(2.2) Add the removed rows to the DOM
(2.3) If the buffer isn't empty => GOTO (2)

It's actually cool - the browser responds much faster and the rows appear while you at it. But something went wrong somewhere. :-)

I'd like very much to have a try but I'm currently a bit overwhelmed at work.

jan_goyvaerts

As an example of asynchronous processing. It works perfectly when letting it run to its end. Otherwise it's very easy to mess it all up.

Also the effect is lost when Scroller only resets the viewpoint AFTER everything has been drawn. :-)

So, there's a little tinkering left here...

[code]
function _fnDraw() {
......
......

var body = $(oSettings.nTBody);
body.children().detach();

// HACK: Draws the rows individually in background so *seems* less slow. It replaces the commented code underneath

asyncDraw(anRows, oSettings, body);

// body.append( $(anRows) );
// /* Call all required callback functions for the end of a draw */
// _fnCallbackFire( oSettings, 'aoDrawCallback', 'draw', [oSettings] );
// /* Draw is complete, sorting and filtering must be as well */
// oSettings.bSorted = false;
// oSettings.bFiltered = false;
// oSettings.bDrawing = false;

}

/**
* Hack to draw the rows asynchronously. It'll take at least just as much time,
* but the first response comes a lot faster.
*
* @param anRows The rows left to add
* @param oSettings Settings
* @param body Element to append the rows to
*/
function asyncDraw(anRows, oSettings, body) {
if (anRows.length == 0) {
_fnCallbackFire( oSettings, 'aoDrawCallback', 'draw', [oSettings] );
oSettings.bSorted = false;
oSettings.bFiltered = false;
oSettings.bDrawing = false;
}
else {
window.setTimeout(function() {
var page = anRows.splice(0,1);
body.append($(page));
asyncDraw(anRows, oSettings, body);
},0);
}
}

[/code]

allan

Looks like a good idea. There will likely be some quirks due to the async nature of the code, but it might be worth exploring further. Patches to Scroller are very welcome :-)

Allan

jan_goyvaerts

Is there an address I can send the modified .js files to ? I've made a number of other changes but they're not 100% just yet.

I guess won't divulge company secrets by sending them to you. :-)

allan

A pull request on github would be great: https://github.com/DataTables/Scroller . Otherwise allan @ this domain will get to me.

Allan

jan_goyvaerts

Was what I sent any useful ?

allan

I don't think I received anything I'm afraid.

Allan

jan_goyvaerts

Just sent it again. This time the JS source code is in a zip file.