Websocket?

GregP

I'm just figuring out if DataTables will meet my needs. It SEEMS to offer all the tools at an abstract level, but the mechanisms themselves might not be compatible with my goal of using websockets to receive a near-realtime series of JSON-formatted updates.

If I'm not mistaken, DataTables wants to operate using request (passing along variables related to pagination, etc), and expects a JSON-formatted reply to that request. Even for automatically-updated tables, I suspect it is meant to operate with polling intervals.

I'd like to do something similar, using the variables DataTables provides (pagination, sort order, etc), but after sending the 'request' (a websocket open and send) the page would then update based on websocket messages. The page wouldn't actively send a request until sort order, pagination, etc., was updated by the end-user.

Is DataTables the right place to start looking, or should I think about rolling my own? I think DataTables is far superior code to what I could feasibly come up with in the short term (or even the long-term! I'm a bit of a fledgling developer!) but if it will take serious amounts of work to use websockets, I might be better off...?

Curious to hear your thoughts on websocket + DataTables! Thanks for your time!

allan

Hi GregP,

Sounds very interesting - I'll certainly try to offer a hand with getting this going, and I believe it will be possible with DataTables :-).

I presume that you are interested in using the server-side processing option of DataTables? I.e. a request is sent to the server for every single page draw (filtering, sorting etc), or are you just looking for a load once, and client-side processing? I'll presume server-side for the moment...

The magic bit of integration with DataTables will need to use the fnServerData callback function ( http://datatables.net/usage/callbacks#fnServerData ). With this you will get to know what DataTables is looking for ( http://datatables.net/usage/server-side ), and be given a callback function to call when the data has been retrieved. So what your fnServerData function would do, I guess, is to write to the socket, and then when data comes back based on the write, it would call the callback function with the needed data.

One thing that the socket might be used for with DataTables would be a continuously updating table (top stock performers or something). To do that, I'd consider using the API methods fnClearTable and fnAddData )or fnUpdate to update what is already there).

Regards,
Allan

GregP

Server-side absolutely. The sorting widgets would just send the info to the server so that it can do the heavy lifting and send back the properly-ordered data.

The project is a combination of both user-requested and continuously-updating table. There are several rows, but for the purposes of illustration, let's say that there is a Task with a particular ID, and a progress for that task, which the server has access to.

The user-directed part comes when a change is requested for pagination or sort order. Maybe I want to see 50 rows instead of 10, or sort by progress instead of ID. At that point a message is sent to the server advising it of the change, and the data that is sent back to through the websocket continues to follow that pattern until the user requests a change.

Where websockets are useful is that there doesn't NEED to be the client-side interaction to retrieve the data. Neither through polling nor through interaction (ie. mouseclick). If the user has requested 50 records, sorted by ID, the server will keep sending data along the websocket without additional requests.

I will need to read the links you gave me before hoping for additional help; what it looks like from your initial response is that the trick would be to use fnServerData to send the request and parameters via the websocket and then trigger fnClearTable and fnAddData from Socket.onmessage.

Thanks for the response! I think that'll get me going on my own for a little while. It's a slow-moving project, but I will keep you udpated!

allan

> what it looks like from your initial response is that the trick would be to use fnServerData to send the request and parameters via the websocket and then trigger fnClearTable and fnAddData from Socket.onmessage

That is one option - although it might lead to a few complications, mixing it with server-side processing. Another would be to simply call "fnDraw" when you get a message from the server saying that the table should be updated. That will then trigger another request to the server for the latest data and redraw it as normal.

Look forward to hearing how you get on.

Regards,
Allan

GregP

Hey Allan,

I think we're on the same page with our concerns; this is part of the reason I wasn't sure about using DataTables with WebSocket.

Here's my assumption, which may not be accurate: The design of DataTables centers around the concept of a client-side request being fulfilled by a response. There's always a 1:1 mapping of requests to responses. Whereas with Websockets, there might be dozens or 'infinite' messages from the server with nary a request from the client.

I like your suggestion, but it's safe to assume that we are always ready for redraw; which is why we decided to use websocket: it can just keep pumping out information without processing and sorting out all the HTTP requests from the client side.

What I'm trying to suss is how to use all the beautiful table enhancements of DataTables (including creation fo widgets, storing the parameters, and handing off server-side processing) while breaking the 1:1 request-to-response paradigm. :D

allan

Yup there are a few issues to be considered - that's for sure. I guess before being able to actually come up with a decent answer, it might be useful to know the parameters you are working with, so the solution can consider that (i.e. a system which needs to cope with millions of rows will be different than one designed for ten rows).

Firstly, how many rows are you expecting to deal with? Also are you going to be using pagination, and how quickly will the data be updated? I presume the updates are on a cell by cell basis.

Basically the trick is that it sounds like you effectively have two data sources - the individual cell updates, and whole table updates (for paging etc). When the whole table is being updated - there might need to be a block on individual cell updates. Or if it's fast enough, just update the whole table whenever only one cell needs done (not optimal - but an option).

Regards,
Allan

GregP

Thanks for your interest, Allan! I hope to not bug you too much, but as a beginner-to-intermediate developer, I'm a little bit on the 'needing my hand held' side still. I tend to be overly wordy in my explanations, so forgive me for not cutting to the point:

The plan is to only have one data source: the whole table update. An overview page will have multiple tables but displaying only a handful of rows (say, 4 tables with 10 rows each), every other page will provide a solitary detailed table with say 25 rows.

We will have pagination requirements, but the server will handle these, generating on server side information like "page 2 of 10"; we display this on the client side but in fact the data object will only contain the records that apply to page 2 of 10. If the user clicks 'next', all that happens is that the server will start sending what it sees as the records that relate to page 3 of ten and the whole table (including "page 3 of 10" label) wil update accordingly.

Ditto for sorting. Literally everything sorting and pagination related is handed off to the server, and we're only expected to send the 'dumb' requests whenever we want the websocket to send out a different page or with different sorting requirements.

The part that's tripping me up is that looking through the DataTables API, functionality is described in terms like this:

"DataTables will send a number of variables to the server to allow it to perform the required processing, and then return the data in the format required by DataTables."

So, the existing mechanism is formed as an HTTP request with POST/GET, expecting a JSON response. But for Websockets, the client could just send a message containing whatever parameters would normally be in the the POST (ie. iSortCol_x); and there's no concept of a response like in Ajax, but rather the Server would just start transmitting a different data set along the WebSocket.

As I'm "thinking out loud", I'm wondering if it's a better bet to just use custom functions for the messaging, leaving DataTables to handle opening the socket in the init phase, and then updating the table(s) using the appropriate function (fnDraw?) from within the WebSocket.onmessage() function?

Tricky also: getting the sort/pagination widgets to send information along the WebSocket messaging mechanism rather than the built-in HTTP mechanism?

Sorry for all the yapping. ;-)

Greg

allan

When when talking about page 2 of 10, the table as a whole might has say 100 records in it? If it was just 25 or so, then I'd say use DataTables in client-side processing mode, and the API functions to update the information. But with 100 rows, then there will be a small overhead, if the cells are being updated often (and if there are a number of different tables). Are you expecting fast updates? (I'm imagining a table a bit like 'top' at the moment - is that about the right idea?).

Is just redrawing each page an option to update it? That way you can use server-side processing entirely, without needing to update individual cells as well. So using your web-socket a bit like an XHR - request / receive - but use it a lot more frequently than you would for an XHR.

You'll almost certainly need to implement some kind of messaging layer between DataTables and the web-socket - as you say, DataTables server-side processing is request / receive based, the problem I'm seeing is how to marry that for displaying each whole page, with updates for individual cells. A messaging protocol which you define between the client and server could be used to do that. For example - client->server request page x, server->client reply page x, server->client update cell m,n. etc.

Allan

GregP

Thanks for your continued input, Allan! Poking around in your forums I see that you're quite busy, so I really do appreciate it.

[edit: just realized after posting that I didn't answer your specific questions! Here goes:
- The entire record on the server-side is unknown, depending on who deploys the app. We have to err on the side of caution and assume that the entire record could be in the thousands, even though it will more often not be.
- The reason we can't hand off sorting and pagination to the client side is that the table will be changing almost constantly. By the time you sort by name with the client-side data, new server-side data is coming in. The back-end team does not want to bother with granular requests (for example, requesting only changes in task progress) so I'm kind of tied to their design.
- Expecting fast updates... sort of. The idea is to create logic such that one of two things always happens: 1) the table won't get sent along the socket unless there has been a change. However, there usually WILL be a change! 2) the front-end will incorporate a 'blocking' mechanism such that it will freeze a data set during render, finish rendering, and then unblock the data so that the websocket can update it. In other words, during heavy activity, the table will update as fast as it is able to.
- Redrawing the entire page from the server is not an option. The back-end team does not want to send tags, only data. But having the client side redraw the entire table each time is indeed an option.]

I think the thing to do is just give it a try at this point. I'm waiting for the back-end dev team to fire up a websocket server so that I can properly prototype, but my current thinking is this (while I'm waiting on them!):

Since this is a web app with certain expectations, I can use DataTables even though it may be overkill for my needs. I would use my own mechanism for getting the data from the server and passing it into a JavaScript array; the function would then hand off rendering to DataTables. The only piece missing would be hooking into the sorting and pagination widgets. I haven't looked at your code yet, but I suspect it could be tuned to send the 'request' message (ie. passing the parameters even over HTTP instead of WS) without expecting a reply.

Cheers,
Greg