Handle millions of records

freddyd

Hey there,

I'm working on a project which handles about 4 million users. I'd like to make a more interactive list and this is why I decided to use DataTables.

The thing is that load 4M of records at once consumes a lot of resources and isn't efficient at all. I was wondering work with pagination but i don't find how to make this approach. This is what i am testing:

$('#apiResults').DataTable({
"processing": true,
"serverSide": true,
"ajax": {
url: "api/search/username",
type: 'POST'
}
});

'api/search/username' returns 4millions of record. I always get errors related to memory because it is a lot of data.
With a small dataset, it works perfectly.

I'm pretty sure I'm missing some parameter to maybe, load X number of first pages and then, load the rest with ajax or something like that.

I hope you can help me.

Thanks in advance.

kthorngren

What are you using for your server script? Does it support the server side protocol and limit the rows returned to the client?

Kevin

freddyd

@kthorngren i’m working with PHP in the backend, specifically with laravel. As i said, with a small dataset it works fine, the thing is with big ones

tangerine

If you are applying DataTables' server-side processing correctly, you should only be seeing one page of data returned at a time, depending on your pageLength setting.

Answering Kevin's questions with better detail would be a good idea.

colin

To re-iterate what everyone else is saying, look at the Ajax tab on this example - you will only see the data being returned relevant to the page being displayed. This would be the case regardless of how large the dataset is.

Colin

freddyd

Go It @colin! Thanks every one!