columns.visible() change crashes page

djkong7

I deal with genetic data and I often need to display a genetic profile in the UI. Since the profiles I show often have over 1000 markers with 2 data points for each marker, I have tables that get to be over 2000 columns wide. DataTables has been able to handle this without any problems until I started to mess with column visibilities. I sometimes need to hide over 99% of the columns and only show columns that have a specific class. I've done all I can to improve the performance, but it still takes a long time (2-3) minutes if it finishes at all. DataTables seems to consume about 5GB of memory to perform this operation and regularly maxes out the page memory and the page crashes. This massive memory usage, along with most of the time spent, comes from the call to visible. i.e. table.columns([columns to hide]).visible(false, false); Example with 1000 columns at http://live.datatables.net/lusexaho/1/edit . Is there any way to reduce the amount of time and memory it takes to change the visibilities?

allan

I don't understand why hiding columns would make the memory usage jump - we store a reference to the cell nodes regardless of visibility, so all it needs to do is remove them from the document. I would have thought that memory use would be static for that (unless the browser needs more memory internally for some reason). 5GB is certainly nuts!

Your example takes 3 seconds for me, which I'm not too surprised by, since it needs to remove 999*2 (header and the single body row) elements from the document.

On a trace, one thing I do see as a performance issue in DataTables is that it is calling _fnDrawHead on every remove, rather than doing it at the end (we might need to consider that if the false parameter is being passed as the second parameter as you are to batch it), which explains why it is around 3 seconds rather than <1 second. But its not up at 2-3 minutes. Is that how slow that demo is for you? What browser are you using?

Allan

djkong7

Here is an example that shows it a little better http://live.datatables.net/lusexaho/4/edit

This one takes much longer and is going well over 5GB for me. I'm on the latest version of chrome. My page is idling around 1.5GB and the visibility change can take an extra 4-5GB on top of that. This idle on the example seemed very high to me and it looks like this is some sort of artifacting from editing the page. Idle is around 200MB when first opening the page. However, in the example and in my real environment, total usage over 5GB isn't uncommon.

I load this data with ajax and then parse it to create the column and data objects (this is to get a dynamic number of columns) and then initialize the table with the columns option and data option. I also included some other extensions in the example and the ones commented I use but didn't/couldn't reasonably put them in the example.

I would ideally like to be able to get this visibility change down to around 5 seconds if possible. It seems like this might be possible if there is a way to not call _fnDrawHead every time. I don't use a CDN and host my own files for this so if I needed to, I would be able to edit my copy of the DataTables .js file

allan

Thank you - that example takes 17.61 seconds for me on my laptop.

I've just tried a little change in the visibility functions to reduce the calls to the header build function and its now down to 1.1 seconds for me: http://live.datatables.net/lusexaho/5/edit .

The DataTables file can be downloaded from here (the ?1 is cache busting - I uploaded a broken version first!).

There is a change required in your code to make use of this change - after calling columns(...).visible(true|false, false) you must then call column(...).visible(true|false); (i.e. no second parameter). I can improve that I think, but that's a workaround to give an immediate improvement.

Allan

allan

Small change made - no need for that second column().visible() call now. I'm going to run our unit tests on this and if all good commit it in.

Allan

allan

Yup - all good so I've committed that change in. Thanks for flagging this one up!

It will be in the nightly build shortly.

Allan

djkong7

Thank you for the quick responses and the quick update. This is just what I needed. I decided to pull the nightly build into my environment to get this update as this functionality is a large part of the usability of my UI. So far, no issues to report.

allan

Amazing how that one little thing can have such an impact on performance. Good to hear its working well for you.

Allan