Beating the IE Dead Horse

Occam's Chainsaw

Howdy Allan,

Sweet plugin! It works good when you exclude large data arrays and Internet Explorer. I've got an install of DT on my site that is choking on IE.

I installed a fresh install of DT 1.7.6 and started to play with 2 of the .js versions in the zip file:

http://heykoolaid.com/dataTables/examples/data_sources/ajax.html
http://heykoolaid.com/dataTables/examples/data_sources/js_array.html

I throttled each of them to 10k records, and put some loading time stamps in 'em to get a good idea for how long specific processes are taking..

IE9 works sorta' good. IE8 chokes to death. All the others are fine (obviously).

Here is our scenario: we have approximately 10,000 rows, but we only need to display around 25 - 100 of them at a time. Dom manipulation around all 10k before the page is rendered is very expensive in IE.

Would it be possible to build the dom around the first 25 - 100 and keep the rest of them cached. When the user requests a new page, sorting the table, or paginates the records results in the next set of listings populating the table already built in the page from the json list in memory?

allan

I haven't thought of providing an option for doing this before, but I quite like the idea - thanks for suggesting it! I don't think it will be all that hard to put into DataTables as it stands. Let me have a little hack around and I'll get back to you in the next day or two with a build that does this and we can see if it makes a significant difference to IE.

Regards,
Allan

allan

I've just implemented this in a fork of the 1.8 development version of DataTables, which you can get from here: http://datatables.net/dev/jquery.dataTables.delayedRendering.js . I don't have access to an IE machine at the moment, so can't test the impact on that - but there is almost no impact on Safari with 5000 rows... Would be interesting to see what you find.

Regards,
Allan

Occam's Chainsaw

AWESOME!

Clearly this fix couldn't go without reward.

==============
for the IE DEAD HORSE FIX
Item # 01 - Dontation to Allan Jardine - DataTables.net
==============

What you've also done, is created a fix that basically makes your server side solution obsolete. I've throttled my JSON result set to 25 records, and it still loads in less than 3 seconds.

Thanks for this wonderful tool!

Cheers,
Jake

allan

Hi Jake,

Thanks very much for the donation :-). Very much appreciated!

How much of a performance improvement are you seeing? When you say a JSON result set of 25 records in 3 seconds - that's rather slow I would have said...

Allan

benjaminv

Allan,

This is great. It makes datatables very usable for larger datasets using IE. It loads immensely faster in IE than it used to. However, it appears there is an issue with setting column visibility with this forked code. It removes the column from each row but fails to remove the column header and gives the following JS error:
Uncaught TypeError: Cannot read property 'parentNode' of undefined
$.fn.dataTable.fnSetColumnVisjquery.dataTables.delayedRendering.js:2049
ColVis._fnDomColumnButtonColVis.js:465
d.event.handlejquery-1.5.2.min.js:16
d.event.add.m.k.handle.mjquery-1.5.2.min.js:16

I downloaded the 1.8 dev code and loaded my page with that, there were no problems with column visibility. So, it appears to be something specific to this fork. Any help would be greatly appreciated.

Thanks

Ben

allan

I suspect what is happening is that fnSetColumnVis is trying to find the TD element which hasn't yet been created. I think I'll need to get back to you on this one - it looks like that might be a little tricky (or at least require a little thought :-) ).

Allan

benjaminv

Allan,

Thanks, I appreciate it. Also of note: column reordering moves the column headers, but not the data. That produces an error as well:

Uncaught TypeError: Cannot read property 'childNodes' of null
fnDomSwitchColReorder.js:67
$.fn.dataTableExt.oApi.fnColReorderColReorder.js:204
$.fn.dataTable.bDeferjquery.dataTables.delayedRendering.js:2245
ColReorder._fnMouseUpColReorder.js:750
ColReorder._fnMouseDownColReorder.js:668
d.event.handlejquery-1.5.2.min.js:16
d.event.add.m.k.handle.mjquery-1.5.2.min.js:16

Hopefully this will help you track down the problem. If you need any more info, please let me know.

Datatables has been great and is an integral part of our application. I really look forward to this branch working.

Thanks

Ben

Occam's Chainsaw

Hi Allan,

Typo on my part- I meant to say 25k records!

It's EXTREMELY FAST now with IE8 and blazing fast with IE9. This is an incredible improvement!

Regards,
Jake

UPEngineer

Allan or Occam's Chainsaw,

You guys have peaked my curiosity on this deferred loading implementation.

Can you give me an example of how to/how did you implement it?

You mentioned you used JSON for the records.

This sounds like it would be perfect for my needs in our app.

Any helps would be appreciated!!!

Scott

allan

@Ben: Ah - it's ColReorder that is triggering this! Okay that narrows it down - thanks :-)

@Jake: Cool! I'm happy with that!!!

@Scott: All you need to do is grab the forked code from the above link and then use either a JS array or Ajax source for your data source: http://datatables.net/examples/data_sources/ajax.html or http://datatables.net/examples/data_sources/js_array.html . The way it is implemented is that DataTables will only create the TR/TD nodes for the rows which get displayed (a bit like server-side processing). Unlikely server-side processing however, once a node is created, it is retained, so it doesn't need to be recreated if shown again.

The upshot (and what makes me ever so slightly nervous - so I need to think on this a bit) is that not all nodes are available at all times. So functions like fnGetNodes() will return only nodes which have been pre-rendered. If you use fnGetNodes to attach events it simply won't work - live events should be used instead. It's not a problem if you are aware of this, there is nothing that can't be dealt with as a result, I'm just wondering if it might be confusing.

If I implement this in DataTables 1.8 (which if it is coping with 25k rows, then it's very very tempting) then it will almost certainly be with an optional flag to enable this feature. bDeferRender perhaps...

Regards,
Allan

Occam's Chainsaw

I'll leave those links alive on my site if anyone else cares to test.

http://heykoolaid.com/dataTables/examples/data_sources/ajax.html
http://heykoolaid.com/dataTables/examples/data_sources/js_array.html

I've got both of them throttled to +- 20k records.

Please do what you can to leave this in DataTables! It really breaks down all the barriers experienced with Internet Explorer.

Cheers
Jake

UPEngineer

I am going to try to implement this.

One question though, is the delayedrendering automatic? Or is a method we have to call on datatables. In looking at the source code, it appears to be automatic and default. Is this correct?

Thanks,

Scott

mg

@Jake - Thanks for putting up the demo pages.
I've tried testing in both IE8 and IE6, and
both loaded successfully; while IE6 was slower,
it still loaded - which is pretty nice for IE6.

Paging worked fine and # of entries to display worked fine.
Column sorting was a little slow on IE8, and generated
a "script on this page is causing IE to run slowly..." message on IE6.

allan

@Scott: In the development version linked above it is fully automatic. When I integrate it into the trunk, I will probably make it optional using a parameter such as bDeferRender. Would be interested to know how you get on with it.

@Jake: I'll wrap up what I'm currently working on in the DataTables core and then implement this properly into the trunk and make a decision then. If it passes all unit tests and doesn't make anything too complicated, then it will go in. It sounds like it really does give a significant improvement - excellent :-)

@mg: This change doesn't effect the speed of sorting at all. It effects the initialisation time of the table. WHere before it used to create the DOM for all rows and columns during initialisation, it would now create the elements only when needed. The data used for sorting and filtering is still in its internal cache and those algorithms are unchanged.

Allan

UPEngineer

Allan,

When I use the test pages on IE6, it pukes when sorting. Isn't this done from the memory cache or is it rebuilding on the fly?

I still have a lot of users of IE6 and I have been told not to force a switch to another browser :(

Maybe this will make them LOL

allan

@Scott: How many rows are you using? As I mentioned in my last post (directed at mg) this change doesn't effect sorting speed. DataTables does indeed have it's own internal cache for sorting - but there is a limit to how many statements IE will process before throwing up it's slow script warning - 5'000'000 ( http://support.microsoft.com/kb/175500 ). Sounds like a lot, but there is a lot going on with the sorting. It's possible it can be optimised some more, although I'm not sure where at the moment. DataTables 1.7.5 introduced a number of optimisations, so I'd make sure you are using the latest version first.

Allan

UPEngineer

@allan: I am using the delayedRendering.js build using Occam's test data here:

http://heykoolaid.com/dataTables/examples/data_sources/js_array.html


I was just testing for IE6. All other browsers seem to sort very quickly, but IE6 doesn't.

Of course I haven't done anything with it yet, just a quick test. Things such as disabling sort classes, etc.

I will test some more.

Thanks,
Scott

allan

I've now built this into DataTables core, given the benefit in speed that it gives - which is fantastic :-). This delayed rendering of the TR/TD elements can be enabled by setting bDeferRender: true at initialisation time. The reason it isn't on by default is that it changes the old behaviour of DataTables, and I want to retain backwards compatibility with v1.8. Following on from the v1.8 release, I will consider making this on by default (possibly without the option of disabling it) if it proves successful for v2 onwards. I've also set up unit tests for this new option and it passes all tests for all data sources (2500+ tests now :-) ).

For maximum speed you will want to disable the sorting classes which highlight the currently sorting column (bSortClasses). I'll write an article about the performance / features tradeoff for DataTables soon.

I'd be most interested to know how you all get on with the new changes! You can pick up the nightly version with this change from the download page: http://datatables.net/download/ .

Regards,
Allan

UPEngineer

Well I have tried to implement this in my app, and I guess I am doing something wrong.

I am building a JSON return with ajax using defered render.

Have about 29000 records to return. Now I get a script timeout error. I am assuming since it is taking longer than 30 seconds to build the array. I could raise the timeout but I don't want someone waiting forever for it to load.

Anyways, I guess I am doing something wrong obviously :)

I want to get away from server-side processing on one table due to some weird fields that I have to hack the sql stuff to get it to work satisfactorily.

The only thing I havent done is build a js array on page load populating it from mysql, but I don't see how it would be any different than pulling from ajax.

Any help thrown my way would be appreciated.

Scott

allan

Hi Scott,

Just a sanity check - you've got "bDeferRender": true in your code now do you with this new version? I don't think you are "doing something wrong obviously" - 29000 records is a lot of data to process, and it takes time to do so. IE allows 5 million instructions before a timeout is issued, which is 170 odd instructions per record for your instruction set - it might sound like a lot, but function calls etc all stack up quite quickly.

I have been thinking of a way to speed up sorting quite significantly (even in IE), but I think it will need to wait until 2.0 since it will not be backwards compatible with the current sorting plug-ins. Something to watch out for in future :-). For now though, have you got bDeferRender: true and bSortClasses: false?

And yes, there would be no difference to DataTables between getting a JS source or an Ajax source.

Allan

UPEngineer

Allan,

I have pretty much scrapped my experiment :)

Yeah, I have bDeferRender set to true.

I was going off what Occum posted about 25,000 records with JSON in a few seconds and pretty much making server-side obsolete.

Anyways, the 1.8 dev. branch has broken a couple of things, things you may or may not be aware of. So I think I am going to stick with the 1.7.6 for now and tweak it and test the 1.8 branch.

Some things to note for you in case you didn't know.

If you set bDeferRender to true and bSortClasses: false, then the sort icons do not appear. Took me forever to figure that one out :) Set bSortClasses: true & bDeferRender:true, they appear. bDeferRender:false & bSortClasses:false they appear.

Also, using the 1.8 branch breaks TableTools 2.0.1 in my setup. The copy/excel/xls does not copy any rows. Even with bDeferRender:false or true.

DT 1.7.6 restores TableTools functionality.

Anyways, more testing.

Scott

allan

Hi Scott,

Thanks for the heads up about the two issues.

TableTools: It makes reference to aoColumns[i].nTf - which is now incorrect as nTf has been removed from the columns object. So TableTools will need an update for the 1.8 DataTables release. The work around for the moment is to set bFooter:false for the button configuration. I think my "fix" will be to strip out the footer from the TableTools output.

Sort classes and deferred rendering: Good find. Thanks for that. I'll fix that shortly.

Thanks for the testing!

Allan

UPEngineer

Allan,

I guess I didn't have the latest build to fix the ColVis issue. And now I have the Tabletools Issue working too.

Maybe I am dense and just don't get it and maybe I am misunderstanding completely :)

Can you explain in laymen's terms what the deferRender actually does for us? An example of when and why it may be used? Maybe I am trying to get it to do something it is not designed for in the first place.

Thanks for all your help.

Scott

allan

Sure no problem - at some point I'll write an article about the various data source options that DataTables offers - before for now:

By default when DataTables loads the Ajax data source, it will process each row for inclusion in the table. This processing involves creating the TR and child TD nodes for the row, as well as rendering the content (if fnRender is given), type detection and various other things. When done the row is then read for sorting, filter or whatever else DataTables needs to do - most of which does not require the DOM elements (since that's very slow).

What bDeferRender does is to delay the building of the TR / TD elements until they are actually needed (i.e. for display). So, for example, if you are displaying 1000 rows, then it will need to create the elements for 1000 rows. If you are displaying only 10, then it will need to create the elements for only 10 rows - regardless of your data source size. However - the data is still needed for sorting, filtering etc, and these processes are unaffected. When DataTables draws a row on the page, it will check if it has been rendered or not already - if not then it will render it there and then.

Thus, the idea that Jake originally put forward saves a big chunk of time in the creation of the DOM elements, since it is spread out over time, rather than all done at the same time. It will not, however, magically make the Javascript processor in IE8- in any way considered to be "fast" by modern browser standards. The IE6 engine in particular is terribly slow. 29'000 is a lot of data to process, and if your Javascript engine is slow then there is no amount of optimisation that will make it decent. There are things I can do to help it, and things I plan to do, but it is a decade old browser now :-)

Regards,
Allan

thesun

i am trying to pass some html in the jsarray method for some columns, using asp.net mvc, seems like it just couldn't parse the encodede html, any examples you could show?

msab

Hi, I am running 1.8.2 and I see this issue on IE7 which is standard in our company. I have about 1500 rows in the table and it is paginated at 125 per page and it is very slow to unuseable. I do the same thing in Firefox and it renders fast(in a second or so).

I am not a developer but figured out how to use datatables to display my data and I love it.

Here is the config I am using

$(document).ready(function() {
$('#example').dataTable( {
"bPaginate": true,
"sPaginationType": "full_numbers",
"iDisplayLength": 125,
"bLengthChange": false,
"bFilter": true,
"bSort": true,
"bDeferRender": true,
"bInfo": false,
"bAutoWidth": false } );
} );

Any updates on how to fix this would be appreciated.

Thanks,
Mike

allan

bDeferRender is only useful if you are using Ajax sourced data: http://datatables.net/release-datatables/examples/ajax/ajax.html

Allan

compsult

Hi Allan,
[deep bow for creating this tool]. Since I embed data in the tag that I use for later Jquery processing, I am returning the whole from the server. Since the bDeferRender only helps if I am using raw, ajax sourced data, I am out of luck, is that correct?
Best
Mike

allan

Hmmm - possibly... :-). How are you getting the data form your HTML returned table into DataTables? DataTables won't parse the HTML from an Ajax source and if you are just putting the whole table onto the page, then the TR elements are created anywhere so deferred rendering wouldn't have any effect (since they are all rendered already).

Allan

compsult

Thanks for the prompt response. I am putting the whole table on the page :(
My stats are 16 seconds for 290 records.

I could rewrite my code to send this hidden data back as invisible columns, but embedding the hidden data in the element was rather convenient (i.e., I don't have to search all the s for hidden data). That rewrite would be rather painful, though

If you are able to compensate for the inadequacies if IE, I am committing to a $50 donation

Details below
I get the with this call
[code]
$.get("run.php", QueryString, function(data)
{
[...]
$('table.dataTable').InitDataTables(TableName, null, oQueryParams);
[...]
}
$.fn.InitDataTables = function( TableName, SelectAry, oQueryParams ) {
[...]
oGlobalData.oTable = $("#"+TableName).dataTable( {
"aLengthMenu": [[25, 50, 75, -1], [25, 50, 75, "All"]],
"iDisplayLength": 25,
"bJQueryUI": true,
"bAutoWidth": false,
"bDeferRender": true,
"sDom": '<"H"fp<"TableName"><"BTNS">>t<"F"li>'
});
[...]
}
[/code]