Can Ajax handle 5GB JSON file which contains 20 millions of records?

Can Ajax handle 5GB JSON file which contains 20 millions of records?

jiamaozhengjiamaozheng Posts: 10Questions: 1Answers: 0
edited September 2017 in Free community support

Any suggestions to handle a JSON file which has 20 millions of rows.

  1. Code:
    "ajax": {
    "url": "../static/data/data.json",
    "dataSrc": ''
    }

  2. File:
    JSON file is a static file.

  3. Outcomes:
    a) Tried to load 200MB JSON file which has half millions of records, and it works.
    b) Tried to load 5GB JSON file which has 20 millions of records, but failed.

This question has an accepted answers - jump to answer

Answers

  • allanallan Posts: 61,446Questions: 1Answers: 10,054 Site admin

    Fundamentally there aren't any limits built into DataTables - however, loading a 5GB JSON file is going to bring any computer or browser to its knees, never mind saturating the bandwidth for your client.

    I would strongly suggest that you don't load a 5GB file in the browser!

    Instead, import it into a database and use server-side processing.

    Allan

  • jiamaozhengjiamaozheng Posts: 10Questions: 1Answers: 0

    Hi allan, thanks for quick answer, and I am very appreciated it.

    I used Scroller https://datatables.net/extensions/scroller/ for loading data. I believe Scroller only draw the current screen, right?

    I can do server-side processing, but it also need to do SQL querying, save output into a JSON text file (or alternatively store data in a variable (so data is in the memory), and draw data on the screen using Ajax.

    Do you know anyone has successfully accomplished my question?

    Thanks,

  • allanallan Posts: 61,446Questions: 1Answers: 10,054 Site admin

    I believe Scroller only draw the current screen, right?

    With a bit of buffering, yes. However, it would still load all 20GB of data up front if you are using client-side processing. If this is over the Internet (as opposed to an Intranet) I can't imagine anyone waiting around for it to load!

    I can do server-side processing, but it also need to do SQL querying, save output into a JSON text file

    Don't save it to a JSON file - just output JSON as the response to the request.

    Allan

  • jiamaozhengjiamaozheng Posts: 10Questions: 1Answers: 0

    I feel that there are limits of data we can handle due to the saturation of network Ajax calls.

  • jiamaozhengjiamaozheng Posts: 10Questions: 1Answers: 0
    edited September 2017

    Hi allan, the main thing is that I need to display these 20 millions of row to allow users to search? Do you know anyone has accomplished this using datables? Thanks,

  • allanallan Posts: 61,446Questions: 1Answers: 10,054 Site admin
    Answer ✓

    Yes, with server-side processing.

    saturation of network Ajax calls.

    Compared to loading 5GB data files, an Ajax request for 10 records is nothing.

    Allan

  • jiamaozhengjiamaozheng Posts: 10Questions: 1Answers: 0

    Hi Allan, thanks for your suggestions and I will try server-side processing to see what's happened.

This discussion has been closed.