Large backbone data set causes massive lag in filtering
Large backbone data set causes massive lag in filtering
Hi Datatables,
I've recently started using your DT plugin to manage our backbone record set that we have.
I followed the Backbone example using the
jQuery.fn.dataTableExt.oApi.fnReloadAjax = function (oSettings, sNewSource, fnCallback, bStandingRedraw) code posted on this site.
We found that it worked perfectly till we started using record sets of about 1000 and up.
The reason for this is because we give the user the option to filter numerical data using a slider (which if moved very fast can fire off up to 10 filters requests in a second).
This means that the table is getting cleared, the information is then re-added based on it's filter status and ordered 10x (when you have 1000-4000 records, that starts to add up).
My backbone model does get notified when the filterStatus is updated, and so I decided to scrap the ajax-fire-hose technique and attempted to manually update the rows (which I also do for another backbone-view and it works perfectly)
My reason for this code was that it would be faster to add/remove a small set of records vs rebuilding the entire table every time a record is updated.
The backbone code is as follows :
[code]
var self = this;
if (self.dataTable != null){
if (!self.get('matchedSearchFilter') && self.dataTableRow != null) {
self.dataTable.fnDeleteRow(self.dataTableRow);
self.dataTableRow = null;
}
else if (self.get('matchedSearchFilter') && self.dataTableRow == null) {
self.dataTableRow = self.dataTable.fnAddData(self.attributes);
}
}
[/code]
But unfortunately this does not work.. in fact, the performance is even worse on 200 records, than the previous example at 4000.
I think the reason for this is because now when the filter suddenly needs to add say 30 records it needs to individual fire off 30 fnAddData requests, and probably hits a bottle neck at the DOM.
The only solution I can come up with is to remove the slider and introduce a different type of filter range that only fires off an update request when the user has actually selected the value they want.
Is there another dataTable function I can use?
I've recently started using your DT plugin to manage our backbone record set that we have.
I followed the Backbone example using the
jQuery.fn.dataTableExt.oApi.fnReloadAjax = function (oSettings, sNewSource, fnCallback, bStandingRedraw) code posted on this site.
We found that it worked perfectly till we started using record sets of about 1000 and up.
The reason for this is because we give the user the option to filter numerical data using a slider (which if moved very fast can fire off up to 10 filters requests in a second).
This means that the table is getting cleared, the information is then re-added based on it's filter status and ordered 10x (when you have 1000-4000 records, that starts to add up).
My backbone model does get notified when the filterStatus is updated, and so I decided to scrap the ajax-fire-hose technique and attempted to manually update the rows (which I also do for another backbone-view and it works perfectly)
My reason for this code was that it would be faster to add/remove a small set of records vs rebuilding the entire table every time a record is updated.
The backbone code is as follows :
[code]
var self = this;
if (self.dataTable != null){
if (!self.get('matchedSearchFilter') && self.dataTableRow != null) {
self.dataTable.fnDeleteRow(self.dataTableRow);
self.dataTableRow = null;
}
else if (self.get('matchedSearchFilter') && self.dataTableRow == null) {
self.dataTableRow = self.dataTable.fnAddData(self.attributes);
}
}
[/code]
But unfortunately this does not work.. in fact, the performance is even worse on 200 records, than the previous example at 4000.
I think the reason for this is because now when the filter suddenly needs to add say 30 records it needs to individual fire off 30 fnAddData requests, and probably hits a bottle neck at the DOM.
The only solution I can come up with is to remove the slider and introduce a different type of filter range that only fires off an update request when the user has actually selected the value they want.
Is there another dataTable function I can use?
This discussion has been closed.
Replies
But the problem still remains that once the filter is called, and we do a complete reload using ajaxLoad, everything stops for about 1sec
I'm just going to give my final solution for anyone who will have this problem in the future..
In my model I have the following :
it is a simple delay setTimeout pattern, it doesn't give real time feedback, but when you are working with 5000 records, it isn't beneficial attempting to filter 5000 records every 5ms on a mobile phone etc.
So i've added the following : (the filterData is what filters the data and then calls this.dataTable.fnReloadAjax();)
[code]
filterBuffer:function(){
this.filterBufferFlag = false;
this.filterTimeout = null;
this.filterViaSearch();
},
filterViaSearch:function () {
var instance = this;
if (!this.filterBufferFlag){
//reset flags
this.filterBufferFlag = true;
instance.filterData(instance.filteredPriceRange);
}else{
//reset the timeOut
if (instance.filterTimeout != null){
window.clearTimeout(this.filterTimeout);
}
instance.filterTimeout = setTimeout(function(){instance.filterBuffer()}, 200);
}
},
[/code]
If you are doing that in a loop, its really bad news as it does a full redraw on every loop. Pass false as a second parameter to stop that and then call fnDraw at the end of your loop.
Allan