dataTable.row.add(json) and dataTable.ajax.reload()

dma_k

Dear community,

I my scenario I have dataTable which is bound to some AJAX source. And it happens that during the execution of dataTable.ajax.reload() the function dataTable.row.add(json) is called. As the result:

1. aoData now contains results from server (N rows) plus one row added by row.add().
2. This additional row (N+1) is not displayed on the screen (which caused real headache in catching the problem).

I think that DataTables are not behaving correctly in this case. If callback(json) is invoked, add data rows should be removed before new are added.

Code in __reload():

_fnClearTable( settings );
var data = _fnAjaxDataSrc( settings, json );
for ( var i=0, ien=data.length ; i<ien ; i++ ) {
    _fnAddData( settings, data[i] );
 }

Suggested:

var data = _fnAjaxDataSrc( settings, json );
_fnClearTable( settings );
for ( var i=0, ien=data.length ; i<ien ; i++ ) {
    _fnAddData( settings, data[i] );
 }

allan

The _fnAjaxDataSrc function should be extremely fast - unless there is an override in ajax.dataSrc the action is not asynchronous, so there should be virtually no difference between the two code suggestions above.

Have you used ajax.dataSrc?

I don't see a problem making the change above, I'm just concerned it won't solve the issue. Are you able to link me to a page showing the problem?

Allan

dma_k

_fnAjaxDataSrc function should be extremely fast

M-m-m. Why? I pass the following option to DataTables:

"ajax" : function(request, callback, settings) { ... do request to backend ...; callback({ 'data' : json }); }

Request to backend may take a while. Even if backend is "extremely fast" and replies in say 50ms, this time is enough to call dataTable.row.add(json) and make a damage.

I'm just concerned it won't solve the issue. Are you able to link me to a page showing the problem?

First of all, the problem cannot be revealed just by showing you a particular page with some request. The problem was revealed under testing of the application under high load, basically, creating dozens of parallel requests to JavaScript application. And it turned out that in majority of cases the whole test suit passed OK, but in some cases it failed. The rate was 1/30 (1 failed vs 30 successful). These kind of problems are difficult to catch, and it took me literally 2 weeks of observations and experiments to find out the problem. Partially the problem is in application (it shouldn't allow table update while it is reloaded), but the component is also misbehaving, as its model and view could be made inconsistent. Namely: two identical records are present in the model, only one row is displayed. Even if I would be able to give you such URL or screenshot, you won't get much as visually it looks OK. The only way to debug the problem is verbose JavaScript logging (that include AXAJ requests), putting high load and having a lot of patience.

allan

_fnAjaxDataSrc function should be extremely fast
M-m-m. Why?

Because all it does is check the settings to see what property should be used and simply return that property. It doesn't make the Ajax call itself.

Request to backend may take a while.

Sure - but that has nothing to do with the _fnAjaxDataSrc function. It is synchronous and only useful once the Ajax call has completed.

The internal _fnBuildAjax function is what makes the Ajax request, and the function passed to it is what is run after the Ajax load has completed.

As I say, I don't have a problem moving the call around - I just don't think it will solve the issue. It might reduce the hit rate from 1 in 30 to 1 in 60, but it is hard to say.

Allan

dma_k

Yes, you're right. _fnAjaxDataSrc is not making AJAX call. I was wrong. There is something different that causing the problem. Perhaps you have an idea? Could it be the code in _fnAddData?

allan

My best guess is that you are running into a timing issue. As I say the change in order of the code you made tipped the scales slightly more in your favour so you just made is less likely to happen.

I think I need a better understanding of what is happening - and I'll probably need to try and set aside a couple of hours sometime to try and reproduce the problem sometime (although I can't guarantee when that would be at the moment).

You said:

This additional row (N+1) is not displayed on the screen

Does the information element below the table show the correct row count?

Does calling draw() solve the issue? Are you calling draw() after your row.add() call (as the documentation notes this is required)?

Allan

dma_k

Yes, I do call draw() function, exactly like this:

dataTable.row.add(json).draw(false);

When table is corrupted, the number of rows in the model is reported to be 10, but only 9 are displayed. Here comes the screenshot. The row with ID 63371 is present twice in the model, but shown only once. The one added to the end of the model is added by row.add(). As sorting is descendant by column ID, there should be two rows.

I have also dumped the whole object as JSON. I have removed most columns from screenshot and data values from JSON, so don't pay attention to the fact that number of columns does not match the model.

I have also to make this note: The problem is happening not exactly with dataTable.ajax.reload() as this topic subject says, but during initial table creation:

var dataTable = $("table#id").DataTable(options);
... table is now waiting for reply from server for initial data...
dataTable.row.add(json).draw(false); // under certain condition creates problem

but I feel this is equivalent to:

dataTable.ajax.reload();
dataTable.row.add(json).draw(false);

allan

Thanks for the extra information. It is almost equivalent, but not quite. The API methods will do a table clear before adding the data, while the initialisation will not. That might well be the problem. It isn't something I've tried before.

Can you try adding the data using initComplete - i.e. after the JSON data has been loaded.

Allan

dma_k

No, you have misunderstood me :) I don't have a specific intention to add one row just after table initialization, and of course initComplete will do it job well.

Let's assume that there is another polling cycle, that listens on incoming updates from the server:

var dataTable = $("table#id").DataTable(options);

serverPoller.startPolling({ onRowAdd : function(json) {
  dataTable.row.add(json).draw(false); // if server sends an update at the moment when dataTable received the thole table, it creates a problem
}});

So this event (table row update) is not under my explicit control. The workaround is that I create a jQuery.deferred that puts all updates in a queue to be executed after dataTable is completely initialized.

The API methods will do a table clear before adding the data, while the initialisation will not.

If you say the current implementation is correct, then I should see two rows in a table. It should not matter that initialisation does not clear the table. Table should be consistent.

allan

Hi,

Sorry I misunderstood. However, could you put your startPolling into initComplete? Or is it not under your control to start the polling?

I suspect you might need a queue in the polling start up to make sure that the table is fully initialised before items can be added.

Regarding the current implementation on startup - I'm not sure if the current implementation is correct - it isn't something I've tested before - what happens to rows added while an Ajax load is occurring. I'll need to spend a little bit of time researching what is happening there, but I'm not sure when I'll get a chance to do so, as things are a bit busy atm!

Allan

dma_k

The poller generally pushes information into other UI elements, not only table. So I cannot put it into initComplete. I have applied workaround for the problem, it's more about the DataTable component. Leave it for future.

I also have some signs that there is tension between dataTable.row(rowNode).remove().draw(false); and dataTable.row(rowNode).data(json).draw(false);, basically the same table row is deleted and updated simultaneously. The side effect is that row is actually not deleted (stays visible), however I would expect that that the row is removed. I don't have much information concerning this scenario, perhaps will take me few weeks to collect it.

allan

Thanks for this - yes I suspect there could be potential for async issues between using the ajax methods to get new data and the client-side methods to manipulate the data. The clear table being immediately before the data add for the ajax reload should mitigate most, but with async processing (both from the polling handler and the ajax) there is likely always to be potential for issues, unless some kind of queuing is implemented.

As a quick example:

1. Call ajax.reload()
2. Request sent off
3. Table cleared
4. Rows start to be added
5. Polling interrupts the row add and does its own add
6. Rest of the rows are added

The fact that there are two data sources is causing the issue. Possibly DataTables should have an internal queue to handle this - although that would add a fair bit of code weight to the core library I think.

I'll need to have a bit more of a think about this.

Allan