DataTable became very slow when large amount of data load by infinite scroll
DataTable became very slow when large amount of data load by infinite scroll
Hi DataTable fellows
I am using DataTable with infinite scroll, it's so awesome for its functionality and simplicity, great thx to Allan creating her.
I just have one problem now, I think it's not the problem of the table but I would like to hear your input.
The problem is during I doing a stress test and keep strolling, the more data I fetch, the slower each response became, my datatable do new POST to fetch data when it scroll to the end of the table . when I fetched about 2000 rows, the response already need 2 or 3 sec for each new POST fetch, which was just 200ms when it starts.
(Screenshot here: http://screencast.com/t/5ygWYMnzTS2n )
The problem is not with the network, which works fine and each POST consume the same time about 200ms , it's about the data already loaded, as it gets more data the more time needed for the table to refresh. I know it's really not the problem of DataTable, but doesn't anyone have a suggestion about this problem.
One solution I think is do pagination rather than infinite scroll, but doesn't pagination still store data in memory? so it will be slow as data grows. I think it's the general problem for all kind of infinite scroll when data grows too much, what is solution you come about ?
Thank you very much!!
Oscar
I am using DataTable with infinite scroll, it's so awesome for its functionality and simplicity, great thx to Allan creating her.
I just have one problem now, I think it's not the problem of the table but I would like to hear your input.
The problem is during I doing a stress test and keep strolling, the more data I fetch, the slower each response became, my datatable do new POST to fetch data when it scroll to the end of the table . when I fetched about 2000 rows, the response already need 2 or 3 sec for each new POST fetch, which was just 200ms when it starts.
(Screenshot here: http://screencast.com/t/5ygWYMnzTS2n )
The problem is not with the network, which works fine and each POST consume the same time about 200ms , it's about the data already loaded, as it gets more data the more time needed for the table to refresh. I know it's really not the problem of DataTable, but doesn't anyone have a suggestion about this problem.
One solution I think is do pagination rather than infinite scroll, but doesn't pagination still store data in memory? so it will be slow as data grows. I think it's the general problem for all kind of infinite scroll when data grows too much, what is solution you come about ?
Thank you very much!!
Oscar
This discussion has been closed.
Replies
I think you are basically correct, yes. There comes a point when the amount of data that you have in the DOM becomes a little too much to cope with and will slow the browser down massively. This is probably not unexpected, the more data, the more processing - etc. You are correct in saying that pagination will help significantly, since the browser won't be rendering such a massive document - it would only be rendering the elements that are needed for each page, rather than the thousands of rows.
However, have you had a look at the Scroller plug-in for DataTables: http://datatables.net/extras/scroller/ ? I've been viewing this plug-in as depreciating infinite scrolling in the core - it is basically infinite scrolling, but it also takes advantage of the pagination controls that DataTables presents. There is a blog post explaining more about it here: http://datatables.net/blog/Introducing_Scroller_-_Virtual_Scrolling_for_DataTables
Regards,
Allan
Thank you very much for your kind help! After investing how to implement the server side processing for some days, I finally managed to realize it in my datatable! Thank you very much for adding this genius rendering methodology, now I can have infinite data on the infinite scroll table :)
here is a screen capture of this:
http://screencast.com/t/ciVqoysyHw1
However, there are some new problem, I will really appreciate if anyone can have some input to this, the problem is:
1 the message in the last column is too long to display, I wish I could limit the wide the table, so that the last column doesn't show all content inside, I will try to make a pop up to show the full content when mouse over
2 when the window width is smaller than the table width, I can scroll x-direction, but the table header doens't follow the content when I scroll, is there solution to this?
Thank in advance, Oscar
1. When you are returning your data from the server-side processing script, just do a check to see if the string is longer than a given number of characters (say 50) and if it is, truncate it to 50 and add an ellipsis (...).
2. Have you got sScrollX set to 100% in your DataTables initialisation?
Allan
Thank you for your continuous help!!
To Answer1 Basically I just need to display some part of a very long message in the last column.
Your 1st recommendation is good but if I make the server side only reply part of the message to fit the column, I will lose the message content to show full message when user mouseover the "td". so DataTable is designed to display all data in td ?? (I see if too long DataTable Warning showing saying it changed the width to minimum width of content)
To Answer2 sScrollX set to 100% is awesome, but it seems I have some problem with the width of column sometimes.
it's very strange that when I first load the data, every thing is fine, the width= the length of column content, like this
http://screencast.com/t/pH4z58pF
But after I scroll to some where else, and come back to the first row, it became like this, the column is somehow unreasonably widened , like this
http://screencast.com/t/N5CLvDwAK
http://screencast.com/t/p8ozPSw4wOkd
I tried many times, but width of column always change like this, I just wonder if it may be caused something by the calculation of DataTable, or I messed up something (I get the project from the mid of somewhere from others)
FYI my config :[code]
oTable = $('#notificationHistory').dataTable( {
"sScrollY": window.innerHeight-400,
"sAjaxSource": "/NotifiWebAdmin/secure/report/datatable/",
"bServerSide": true,
"bProcessing": true,
"sDom": "frtiS",
"sScrollX":"100%",
"bScrollCollapse": true,
"bFilter": true,
"aaSorting": [[4,'desc']],
"aoColumns": [
{ "sClass": "center", "sWidth": "100px"},
{ "sClass": "center", "sWidth": "100px" },
{ "sClass": "center", "sWidth": "100px" },
{ "sClass": "center", "sWidth": "100px" },
{ "sClass": "center", "sWidth": "100px" },
{ "sClass": "center", "sWidth": "100px" },
{ "sClass": "left", "sWidth": "100px" }
],
"fnServerData": function ( sSource, aoData, fnCallback ) {
var moo= $("#reportCommand").serialize()+ "&fromPtr="+ 0;
aoData.push( { "name": "reportCommand", "value": moo } );
$.ajax({
dataType:'json',
type: "POST",
url: '/secure/report/datatable/',
data: aoData,
success: fnCallback });
}, //end "fnServerData"
"oLanguage": {
"sSearch": "Filter the search result:"
}
[/code]
I will really appreciate your help!
Oscar
This is how HTML tables work - yes. There are a couple of options - probably the most attractive is that you could wrap the element in a DIV which has overflow:hidden, and then use a mouseover tooltip a bit like this: http://datatables.net/release-datatables/examples/advanced_init/events_post_init.html .
> But after I scroll to some where else, and come back to the first row, it became like this, the column is somehow unreasonably widene
I'm afraid I've no idea what would cause that - I've not seen that issue before. Can you link to a live page please?
Allan
To enable the "last column displaying partial content and show full content when mouse over", I basically send back one more column of abbreviated content in server side response and don't display the full content column in the table, but save the information of that column in an array of String after POST success.
[code]
"fnServerData": function ( sSource, aoData, fnCallback ) {
$.ajax({
dataType:'json',
type: "POST",
url: some_url
data: aoData,
success:function(result){
//after POST success, save the data as
//array of String before sending back
//to DataTable
fnCallback (result);
}
[/code]
and then use a mouse-over jQuery callback to add the data to the td's " title "
[code]Element.setAttribute("title",actualMessage[index]);[/code]
hope this may help anyone has the same question
Thanks again to allen for the help!
Oscar