Handle millions of records

Handle millions of records

freddydfreddyd Posts: 3Questions: 0Answers: 0

Hey there,

I'm working on a project which handles about 4 million users. I'd like to make a more interactive list and this is why I decided to use DataTables.

The thing is that load 4M of records at once consumes a lot of resources and isn't efficient at all. I was wondering work with pagination but i don't find how to make this approach. This is what i am testing:

$('#apiResults').DataTable({
"processing": true,
"serverSide": true,
"ajax": {
url: "api/search/username",
type: 'POST'
}
});

'api/search/username' returns 4millions of record. I always get errors related to memory because it is a lot of data.
With a small dataset, it works perfectly.

I'm pretty sure I'm missing some parameter to maybe, load X number of first pages and then, load the rest with ajax or something like that.

I hope you can help me.

Thanks in advance.

Replies

  • kthorngrenkthorngren Posts: 7,761Questions: 25Answers: 1,726

    What are you using for your server script? Does it support the server side protocol and limit the rows returned to the client?

    Kevin

  • freddydfreddyd Posts: 3Questions: 0Answers: 0

    @kthorngren i’m working with PHP in the backend, specifically with laravel. As i said, with a small dataset it works fine, the thing is with big ones

  • tangerinetangerine Posts: 2,448Questions: 19Answers: 275

    If you are applying DataTables' server-side processing correctly, you should only be seeing one page of data returned at a time, depending on your pageLength setting.

    Answering Kevin's questions with better detail would be a good idea.

  • colincolin Posts: 7,697Questions: 0Answers: 1,297

    To re-iterate what everyone else is saying, look at the Ajax tab on this example - you will only see the data being returned relevant to the page being displayed. This would be the case regardless of how large the dataset is.

    Colin

  • freddydfreddyd Posts: 3Questions: 0Answers: 0

    Go It @colin! Thanks every one!

Sign In or Register to comment.