Boost performance on large table by lazy gatherData from HTML without bAutoType

hkdennis2k

I have few large table using datatable with bPaginate=false and bSort=true.

The performance was really WORST in IE8( e.g. 60+ seconds for 1000 rows).
I did a innerHTML hack before and reduce the time to 15 seconds.

I've just apply another two tweak today to reduce by 50%.

a)
In an unpatched datatable, datatable() -> _fnGatherData() which spend a lot of time to gather data from HTML table.
What I did is change data array "oSettings.aoData[row]._aData" to "oSettings.aoData[row]._aDataF" which is a function who return the dataarray.

The function will capture the row data and return it on first run; then I do cache the result by a self-rewrite function.
I've also edit others code that it should use the new aDataF correctly.

In my test it shown me 15s to 8s improvement.

b)
The another performance improvement can be done by a simple fix on function _fnDraw.
Near the line "if (nBody[0]) {"
It appends rows to table row by row that is very slow.

A lot of reflow/redraw time can be saved by using createDocumentFragment(), and collect all anRows[i] before appendChild to table.


I have another question here, how can I submit my patch ??

hkdennis2k

More about patch a)

Because datatable detect column type in _fnGatherData just after read, you will have to set "sType" on all columns inorder to get the lazy initialization.

The actual gatherDataFromRow will actual happen on user first click on the column header to sort the table.

I am not sure why, but the total time is much smaller and it runs much faster to capture data from innerHTML by lazy methods. It takes only 250ms by the lazy function in my same test case.

allan

Hi hkdennis2k,

Sounds interesting! If your patch is small enough you can post it here in the forum, or send it to me directly using http://datatables.net/contact .

Allan

allan

Hi hkdennis2k,

Thanks for sending over your code - I've just been having a look at it. Unfortunately the whole document has been reformatted, so it's a bit harder to pic out your changes than just a diff, but I've been looking at the lazy loading function - a couple of questions: how does this work with filtering? Does the first key stroke effectively load the whole table? How about an initial filter using oSearch? Also, what is the impact on sorting when using multiple columns to start with, or anything but the default?

Thanks,
Allan

hkdennis2k

I've just submitted a patch for bVisible. I know it is ugry patch.

Sorry, I've press the wrong button in visual studio that it formatted the file..
I suggest you that you can format both the original and my file with eclipse(or sth else), and use the compare with each other feature in eclipse to see all the changes.

I've tried myself filtering and sorting and bVisible.
The concept itself is simple, when any one get read "_aData[i]" for data, it should rathern run "_afData()[i]" as ask for the result.

The sorting works very smooth, and even smoother. I am not sure why but in my test cases, _fnGatherData() took a lot of time to read innerHTML, but it just milliseconds if I do I on sorting.

I tried hideing column with bVisible, it will force the afData() happen before it remove the column, so no lazy here.

I've never test a initial filtering, but I suggest you move the initial filtering/sorting to a delayed setTimeout callback.

Sammy83

Hi there,

is there any chance to get that patch?
I've a little project where i use a table with about 2000 rows and many jeditable fields.
And it very slow at the moment and i'm tring to optimize it.

So is there a way to get this patch?

regards
Sammy

hkdennis2k

I've submitted the patch to allan before, but I don't see he is going to intergrate the idea into the next version of datatable.

I may find sometime to redo my patch in a smarter/cleaner way. However, you should know that my patch will broke few plugin compatibility by it's nature.

_andreas_

Hello hkdennis2k,

It would be great if you could upload your patch somewhere. I would love to test it, as we are also experiencing performance problems with > 600 rows.

Many thanks in advance
Andreas

_andreas_

@allen: could you upload the changes sent by hkdennis2k somewhere?
I would like to try if I can fix my performance issues with some lines out of it. I'm sure Sammy83 would also be interessted in having a look at it.

Greetings from Austria
Andreas

rosso

Any chance this patch made it into a release lately. I've found fnAddData to be very slow with very large tables where bPaginate=false

allan

I've uploaded hkdennis2k version to http://datatables.net/dev/dataTables-lazy-loading.js . It's DataTables 1.6.1 so a bit behind the current release, and unfortunately it's reformatted, so it's not easy to do a diff or patch DataTables. I tried it on the basic unit tests but unfortunately it didn't pass - however this file might be useful for anyone looking to be able to do something similar. The thing that makes me nervous about lazy loading is that DataTables actually needs all of the data in order to do the first draw - it can't just load it on demand. Otherwise, sorting filtering etc simply won't work as expected.

What I would suggest is if you are seeing speed issues with large data sets is to look at using server-side processing which will cope with many millions of rows no problem.

@rosso - Have you got the second parameter set to false in fnAddData - http://datatables.net/api#fnAddData ? Otherwise the table will be redrawn every time which is slow.

Allan