Data Load Issue

akbarehsan

Hi,

I am using Perl to read data as JSON format from a Postgres Database. I am also using JQuery UI to format the display. The problem I have the volume of data. I am loading 18,000 records. It loads in about 20 seconds in Firefox but takes up to a minute in IE (my users will run away in that time!) and that too after it has thrown a message about stopping the script because it is taking too long. My application is not in production yet.

I would like to know how best to optimize my application so that it loads faster. I am sure other people have loaded large volume of data. Would be interested in knowing what they did to resolve the issue.? Also, how do I shut IE up so that it does not complain about a script taking too long to run?

Thanks,

Akbar

allan

You could try the deferred rendering in the 1.8 beta: http://datatables.net/beta/1.8/examples/ajax/defer_render.html - which should give a nice boost. However, there isn't going to be a magic fix to get it to go from 1 minute of processing to less than a second, or I would have put it in already! With 18'000 records I would suggest using server-side processing.

Allan

akbarehsan

Allan,
Thanks for such a quick response. I must say that DataTables are great. This is my first time using it.

I am not expecting it to go down to 1 second from 1 minute in IE. In Firefox, it is just 15 seconds. I am hoping to get it down to 15 in IE. I am going to try your suggestion and post the result..

I am alittle new to everything, Perl, JSON, DataTables etc. When you say server-side processing, do you mean create JSON file on the server first (may be use CRON to run your script to create the file) and then feed that file to DataTables script?

Thanks,

Akbar

akbarehsan

Hi Allan,

Using deferRender with DataTables1.8 reduced the time by half in Firefox ( from 15 seconds to 8) and in IE by about 2/3 (from 1 minute to 22 seconds). For now, I can live with that.

However, I do get that irritating message in about a "script running too long and that it should be stopped". Any thoughts on that will be appreciated.

Thanks,

Akbar

barat

because You should use server side processing as someone mentioned above. Use PHP to generate json file. There is even an example in examples section.
http://datatables.net/examples/server_side/server_side.html

allan

Yup the message that IE throws up is there for a reason :-). 22 seconds is a long time for an end user to wait for a page to do something. It might be possible to 'chunk' the processing in DataTables to avoid this message, although I would be very reluctant to put that into the core, since as I say, it's there for a reason.

As barat points out, there is an example of server-side processing on this site which will allow you to deal with millions of rows no problem.

Allan