How to deal with gigantic data (think gmail with millions of msgs)
How to deal with gigantic data (think gmail with millions of msgs)
Say I have a stream of data, coming in through websockets. DataTables will display no more than 100 or so at a time, the rest will reside on the server. The data is real-time, so updates are coming in every few seconds.
How can I handle such a situation?
I'll only have to display a 100 or so messages, so I don't need (or want) to record any more in any client data structure or disk. When the client clicks 'next,' I need to request the right batch of of records from the server. This is not terribly difficult, in theory. What if the user filters on some words, then starts iterating through the results...a little more complex. Say the user adds a filter, now I need to handle incoming real-time messages and filter them as well.
What can DataTables do to help me in this situation. I can't use the pre-existing ajax functionality. The API looks very interesting. I can obviously add data to the table, is it my responsibility to remove data which can't be seen by the user (or does datatables figure that out?)? Is there a way for me to intercept the request "display records 1000 to 11000" so I can have my code request it from server and push the response into the table? Can I intercept filter or sort requests?
Look forward to any ideas others may have. I am basically trying to avoid building the whole thing on my own.
Thanks
How can I handle such a situation?
I'll only have to display a 100 or so messages, so I don't need (or want) to record any more in any client data structure or disk. When the client clicks 'next,' I need to request the right batch of of records from the server. This is not terribly difficult, in theory. What if the user filters on some words, then starts iterating through the results...a little more complex. Say the user adds a filter, now I need to handle incoming real-time messages and filter them as well.
What can DataTables do to help me in this situation. I can't use the pre-existing ajax functionality. The API looks very interesting. I can obviously add data to the table, is it my responsibility to remove data which can't be seen by the user (or does datatables figure that out?)? Is there a way for me to intercept the request "display records 1000 to 11000" so I can have my code request it from server and push the response into the table? Can I intercept filter or sort requests?
Look forward to any ideas others may have. I am basically trying to avoid building the whole thing on my own.
Thanks
This discussion has been closed.
Replies