Thoughts regarding DataTables and 50,000 rows of data...

concise

My dataset has 50,000 rows (5 columns) and we upload a .csv file to our Laravel php framework that then uses DataTables to parse the data. We love using DataTables but the problem is the load time really suffers and it is taking about 4-5 seconds to load just 3,000 rows.

Now, I understand that we can enable server-side processing - https://datatables.net/reference/option/serverSide - my questions are:

1. Can DataTables work for 50,000 in a reasonable load time if we implement server-side processing
2. How long would you imagine a dataset of 50,000 rows taking to load?
3. Will the search function work?

And! Lastly! What would you guys suggest for data format? Maybe JSON? We are using Laravel - not sure how that impacts/ helps the conversation....

I guess I'm not the first to ask such questions and of course - happy to pay the support/ maintenance for help with implementation.

Desconhecido

You have to figure out if its taking that time to get the data or to render it.

If its taking 5s just to render then something is wrong, do you have the "deferRender": true ?

Yes, search here works on a 300k table with 14 fields

allan

Can DataTables work for 50,000 in a reasonable load time if we implement server-side processing

I've seen it use tens of millions of rows, so yes. The key is that the processing is done by the server-side, so you need to optimise at the server.

How long would you imagine a dataset of 50,000 rows taking to load?

The whole point of server-side processing is that you don't load all of the rows at once - just a sub-set. For example the first ten, then the next as the user changes the paging in the table, etc. There is detailed documentation about this.

Will the search function work?

If you implement it on the server-side, yes. See this page for the client / server interaction DataTables uses.

What would you guys suggest for data format?

JSON without question. You should probably use an SQL engine for the data as well rather than trying to parse the full file for sorting, filtering and paging on every request - that would quickly overload your server. SQL databases are obviously optimised for that kind of thing.

Allan

concise

really amazing advice - thanks very much.