Semi threading datatables to stop script duration warnings when server side code is not an option

cdmdotnet

Hi

We have a situation where we can have large amounts of data - in the name of tens of thousands of records - and as to be expected both IE and FF warn that the scripting is taking a long time to process.
Unfortunately we can't introduce server side operations for this ( not for me trying to but I can't convince management the change warrented ).

What I have managed to do is change the code from the 1.6.2 version of datatables to use setTimeout in a few places which doesn't appear to have an side effects - sure the loading takes its time but at least I don't get any warnings. I also added a bool flag so you can turn this feature off.

My problem now is trying to keep our version of datatables current as you progress with it, so the questions I'm actually getting to is: How can I submit the changes I've made so you can assess them and see if they're worth incorporating back into the main stream?

cdmdotnet

PS don't judge me on the fact we have such a high record count. I know this page takes forever to load and has HUGE code size with the bloated html involved. But unfortuntaely The client has demanded that all records are onscreen and can be sorted, and we use datatables in other areas of the site as it is indended to be used, and I don't want to introduce another JQuery plug-in to do the same/similar thing - namely sorting in this case.

allan

Hi cdmdotnet,

If the code changes are fairly small, then you could post a diff in the forum. Otherwise drop it to me using http://datatables.net/contact . I hadn't thought of using setTimeout() to avoid that kind of message before - good thinking :-).

It will be interesting to see how you have done this. I'm not sure that I will include this option in the core distribution, as I'm slightly concerned that it might promote misuse of it, and lead to pages which take a load time to load, where server-side processing would be a better option - but I'll certainly consider it.

Regards,
Allan

cdmdotnet

I'd love to create a diff file but all the whitespace and line endings have been changed so the diff files I'm generating are a mess.

What I have done is : ( line references are at the begining of each line and include the caridge return )

line 829 :
[code]
+ "bThreadSorting": true,
[/code]
line 1955
[code]
- oSettings.fnInitComplete( oSettings, json );
+ setTimeout(function() { oSettings.fnInitComplete(oSettings, json); }, 0);
[/code]
line 1964
[code]
- oSettings.fnInitComplete( oSettings );
+ setTimeout(function() { oSettings.fnInitComplete(oSettings); }, 0);
[/code]
line 1969
[code]
- _fnProcessingDisplay( oSettings, false );
+ setTimeout(function() { _fnProcessingDisplay(oSettings, false); }, 0);
[/code]
line 2199
[code]
- function _fnGatherData( oSettings )
+ function _fnGatherData(oSettings, bInitHandedOff)
[/code]
line 2344
[code]
- }
+ if (oSettings.oFeatures.bThreadSorting)
+ setTimeout(function() { _fnProcessObtainedData(oSettings, bInitHandedOff); }, 0);
+ else
+ _fnProcessObtainedData(oSettings, bInitHandedOff);
+ }
[/code]
line 4949
[code]
- _fnMap( oSettings.oFeatures, oInit, "bAutoWidth" );
+ _fnMap(oSettings.oFeatures, oInit, "bThreadSorting");
+ _fnMap( oSettings.oFeatures, oInit, "bAutoWidth" );
[/code]
line 5142
[code]
- }
+ if (oSettings.oFeatures.bThreadSorting)
+ setTimeout(function() { _fnProcessObtainedData(oSettings, bInitHandedOff); }, 0);
+ else
+ _fnProcessObtainedData(oSettings, bInitHandedOff);
+ }
[/code]
line 5146
[code]
- _fnGatherData( oSettings );
+ if (oSettings.oFeatures.bThreadSorting)
+ setTimeout(function() { _fnGatherData(oSettings, bInitHandedOff); }, 0);
+ else
+ _fnGatherData(oSettings, bInitHandedOff);
[/code]
line 5148
[code]
+ });
[/code]
line 5149
[code]
- /* Copy the data index array */
+ /*
+ * Function: _fnProcessObtainedData
+ * Purpose: Process the obtained data , best done in a async manner via setTimeout
+ * Returns: -
+ * Inputs: object:oSettings - dataTables settings object
+ * bool:bInitHandedOff - ???
+ */
+ function _fnProcessObtainedData(oSettings, bInitHandedOff)
+ {
+ /* Copy the data index array */
[/code]
line 5168
[code]
- });
+ }
[/code]

Hopefully that is of some use to you. Theres really only three or four core changes and the bThreadSorting flag either enables or disables then.

allan

Hi cdmdotnet,

Thanks very much for that. Very interesting indeed. I'll have a go at integrating this into my development version at some point in the near future and see how it performs. This will probably after the 1.7 release, but it's certainly an interesting option if one is willing to allow long execution delays.

Regards,
Allan

bfjf

I also have a large table (100 columns x 500 rows) in a situation where I'd rather not use server-side. The overall appearance and responsiveness once the table has loaded is fantastic. Things just get sluggish when the table first loads and if I remove columns.

The sort event function already seems to have a setTimeout to allow the processing icon to load (this works great). The problem I have is the first initialization which locks up fast browsers and with slower browsers I get the stop-script dialog. The second problem (that seems to be far slower) is using the column hide/show on large tables using oTable.fnSetColumnVis(i,true). This seems to be disproportionately slower than simple re-creating the table.

John Resig, in his "Secrets of the Javascript Ninja" book, has an example of how to create giant tables by drawing/inserting only a subset of the rows using several calls to setTimeout. I looked around the code, but I can't find where the table elements are created.

allan

fnSetColumnVis is probably quite slow with 100 columns due to the need to recalculate the column widths - doing a profile on this action would be quite interesting.

I Haven't read John's book, but it sounds like an interesting approach to rendering a lot of rows. I'm sure I've seen another table plug-in for jQuery which does take that approach, but I can't remember it's name off the top of my head, nor does it provide the feature set that DataTables does (filtering was one as I remember).

It's an interesting idea using setTimeout to effectively create threads, and stop the browser giving warning errors. Perhaps an option for the future, although I am wary of it, since it allows UI's which are very slow to be created... (although the opposing argument is, that is up to the dev).

Interesting... :-)

Allan

bfjf

FYI: I made a few little changes to get this to work (I'm using "bUsePassedData" to provide the matrix to datatables). I made similar changes to _fnDraw; That however tied in with more code and led to unwanted behaviors (e.g. the processing icon need to be driven asynchronously too) that I didn't feel like fixing given that the code below pretty much did what I needed.

original code starting line 6665
[code]
for ( i=0 ; i