Can Ajax handle 5GB JSON file which contains 20 millions of records?

jiamaozheng

Any suggestions to handle a JSON file which has 20 millions of rows.

1. Code:
   "ajax": {
   "url": "../static/data/data.json",
   "dataSrc": ''
   }

2. File:
   JSON file is a static file.

3. Outcomes:
   a) Tried to load 200MB JSON file which has half millions of records, and it works.
   b) Tried to load 5GB JSON file which has 20 millions of records, but failed.

allan

Fundamentally there aren't any limits built into DataTables - however, loading a 5GB JSON file is going to bring any computer or browser to its knees, never mind saturating the bandwidth for your client.

I would strongly suggest that you don't load a 5GB file in the browser!

Instead, import it into a database and use server-side processing.

Allan

jiamaozheng

Hi allan, thanks for quick answer, and I am very appreciated it.

I used Scroller https://datatables.net/extensions/scroller/ for loading data. I believe Scroller only draw the current screen, right?

I can do server-side processing, but it also need to do SQL querying, save output into a JSON text file (or alternatively store data in a variable (so data is in the memory), and draw data on the screen using Ajax.

Do you know anyone has successfully accomplished my question?

Thanks,

allan

I believe Scroller only draw the current screen, right?

With a bit of buffering, yes. However, it would still load all 20GB of data up front if you are using client-side processing. If this is over the Internet (as opposed to an Intranet) I can't imagine anyone waiting around for it to load!

I can do server-side processing, but it also need to do SQL querying, save output into a JSON text file

Don't save it to a JSON file - just output JSON as the response to the request.

Allan

jiamaozheng

I feel that there are limits of data we can handle due to the saturation of network Ajax calls.

jiamaozheng

Hi allan, the main thing is that I need to display these 20 millions of row to allow users to search? Do you know anyone has accomplished this using datables? Thanks,

allan

Yes, with server-side processing.

saturation of network Ajax calls.

Compared to loading 5GB data files, an Ajax request for 10 records is nothing.

Allan

jiamaozheng

Hi Allan, thanks for your suggestions and I will try server-side processing to see what's happened.