State Restore - looping through states and filter data

bg7 · March 9

I'm using State Restore and want to loop through the saved states and collect data from the rows that appear when each state is loaded. The approach I'm currently using works but is complicated. It is roughly as follows:

Save the current state to a temporary state.
Loop through the existing saved states, load one at a time and listen for the draw to complete as each state is applied, then grab the resulting rows and get the data I need and save that data off to a list.
Restore the saved temporary state.
Remove the temporary state.
Switch to another screen to use the collected data (in this case to generate charts based on the user's saved states all at once).

It all happens quickly enough that you can't tell as a user which is good. That said, it is complicated and I was hoping I could simply loop through the saved states and ask for the filtered data for each state without affecting the current table. Is it possible and I've missed a simple way to do this? If I could call something like filter() on the state that would massively simplify things but from what I can tell that's not currently possible.

https://datatables.net/reference/api/filter()

One other thing I wanted to ask is if there's a way to add a confirmation modal to the "Remove All" for states. By default there's a confirmation modal when you try to remove a single state which seems reasonable but there's no confirmation modal if you remove all of the states at once which seems odd given it's potentially far more destructive. A confirmation modal by default would be great. I couldn't figure out a simple way to add one in manually.

https://datatables.net/extensions/staterestore/examples/initialisation/removeAll.html

Thanks.

Ben

kthorngren · March 9

One option might be to use stateSaveParams to save the desired filtered data set with the state. Then you can loop through with either stateRestore.states() or stateRestore.activeStates() and grab the saved filtered data without needing to load the states. Similar to this example. The example doesn't run properly and throws this error:

Uncaught TypeError: minEl.val is not a function

@allan will need to fix the error.

One other thing I wanted to ask is if there's a way to add a confirmation modal to the "Remove All" for states.

The removeAllStates states this:

No confirmation modal is shown as default when using this button.

One option might be to create a custom button and use stateRestore.states().remove() setting the skipModal parameter to true.

Kevin

allan · March 10

Ah sorry, the Vanilla JS example code incorrectly uses a jQuery method, which is what is throwing an error there. Replace:

    stateSaveParams: function (s, data) {
        data.min = minEl.val();
        data.max = maxEl.val();
    },
    stateLoadParams: function (s, data) {
        minEl.val(data.min);
        maxEl.val(data.max);
    }

with:

    stateSaveParams: function (s, data) {
        data.min = minEl.value;
        data.max = maxEl.value;
    },
    stateLoadParams: function (s, data) {
        minEl.value = data.min;
        maxEl.value = data.max;
    }

I'll get the example updated as part of a rewrite process that StateRestore will shortly be undergoing.

Allan

bg7 · March 10

Kevin and Allan,

If I'm understanding the stateSaveParam suggestion correctly I don't think this would fit my use case. The list of rows that match a saved state could change while using the system so saving a snapshot of the results at a given time wouldn't work. Does that sound right? I was hoping there was a way to create something like a filter() from the saved state so that I wouldn't have to loop through the states and load them into the existing table, listen for events, manage temporary states to be able to restore the original state, etc. Is that just not realistic?

I can give the custom button a try. I saw the documentation for the remove all states saying that it doesn't have a confirmation modal. I guess I'm suggesting that perhaps it should have one by default as the single remove has one and removing everything at once is obviously far more dangerous. Just my two cents.

Thanks.

Ben

allan · March 10

Hi Ben,

Very good point, yes, that seems wrong to me. You can workaround that with:

DataTable.ext.buttons.removeAllStates.action = function (e, dt, node) {
    dt.stateRestore.states().remove(false);
    node.blur();
};

The default action is to pass true into stateRestore.states().remove() which skips the modal.

The proper fix should really be to default to showing the confirm modal, and giving the button a configuration option to allow it to be skipped.

Regarding your main question about looping through states to grab data, I think Kevin is on the money with the use of stateRestore.states(). Use that to get the list of states, then create a queue mechanism that will loop over them, draw, grab data and repeat until the queue is empty. No need to use stateSaveParam for that process.

Allan

kthorngren · March 10

The list of rows that match a saved state could change while using the system so saving a snapshot of the results at a given time wouldn't work. Does that sound right?

I guess it depends on your solution. I don't fully understand your solution and might not understand your requirement for this.

I was hoping there was a way to create something like a filter() from the saved state so that I wouldn't have to loop through the states and load them into the existing table

My original idea was to save the filtered rows. You said this might not work with your solution. My updated idea might not work either but it will reduce the amount of row data stored if each row has a unique data column. This test case shows saving the data from the unique data column using stateSaveParams. The button List saved data will iterate each save state and list the filtered rows based the unique data point.
https://live.datatables.net/nipeceto/1/edit

If storing the full row data or the unique data won't work for you then you could iterate the stateRestore.states() states and get the s.savedState.search term and optionally iterate s.savedState.columns to get their search terms. You could then use filter() with these search terms to get the appropriate rows. Depending on the search mode, etc the filter() results may or may not match the saved filtered tows.

Kevin

bg7 · March 11

Allan,

I added the code you posted for the remove all states - thanks. The confirmation modal is showing up now but it's missing the 'x' to close it in the top right corner. Is that easy to add?

Remove:

Remove All:

Also curious if would it make sense to have the default confirmation modals for the create, rename, remove, remove all states also have cancel buttons? It would make the modals more intuitive from a UX perspective. And one small thing - the confirmation modal title for remove all would make more sense as "Remove State(s)" plural.

Thanks.

Ben

allan · March 11

Hi Ben,

You've hit upon a number of reasons why I plan to rewrite StateRestore here... There are just too many noddy little things which shouldn't be there.

There isn't a trivial option to fix that close button - I'll need to make a code change. I'll try and get that done tomorrow.

Regarding the cancel button, I've been in two minds about that. Currently you can click the background to dismiss (cancel) the action and the X icon (if it were there!). A third option as a cancel button could certainly be added, and probably should be, but I've always hesitated about it!

Allan

bg7 · March 11

Allan and Kevin,

Makes sense. Sometimes the best thing is a refactoring house cleaning. We'll manage without the 'x' for the moment. If you're taking votes, I vote cancel button. Now as for which order the cancel and ok buttons go in... no clue what's "right".

I'm pretty sure I see what you guys are suggesting with the example you put together Kevin. If I'm not mistaken the idea is to take some subset of the data when the state was created and save data that to the state itself. Then when you loop through the states you don't have to actually load the state because you can simply grab the data right off it. (Correct me if I'm wrong.)

Unfortunately that won't work for us. Our data is live data from an electrical grid and it's constantly changing (and the table's data is constantly updating to reflect that). So for example, a saved state might be a search of all generators that are currently running. That list might be different from the time you save the state to the time you go back later and loop through the states to, e.g. use that data to generate charts for each generator. As the data is dynamic, saving a snapshot of that data at any given point isn't going to work for us. Hopefully that makes sense.

Kevin, I did actually put together a button that inspected the state restore object and collected all the various search strings, filters, etc. I was then able to use it in conjunction with the filter() method with a custom function to essentially implement that filter on the fly. It worked, was quick and didn't require loading/saving states but it was a lot of code and felt really brittle.

Door number two is the approach I'm using at the moment. Save the current state to a temporary state, loop through the saved states loading each one and then listen for the draw to complete before getting the data from the resulting rows and then load the next state and so on and then restore the original state at the end so the user is none the wiser. It's a bit of code but on the plus side it doesn't require inspecting the state object and pulling it apart. And it's also thankfully quick enough that you can't tell as a user what's happened.

I was hoping there was a way to grab a state object and then do something like call filter() off that to get the table's data with the state's filter applied without actually affecting the data the user is looking at. Sounds like that's not possible but that's why I wanted to ask. If that's a feature you think would be worth adding I'd definitely take advantage of it. Hopefully it makes sense what I'm describing.

One other small issue I ran into was when I was creating the temporary state. I set the creationModal flag on the savedStates button extension config to true:

creationModal: true,

so that users get a modal to name their state immediately upon creation. That's definitely a good thing. The problem is when I'm creating the temporary state that flag is still true but of course I don't want to prompt the user to give it a name. I tried passing that flag into the add method:

            dt.stateRestore.state.add(TEMP_STATE_NAME, { creationModal: false });

Unfortunately it appears the DataTables code stops checking if it's already set to true in the config:

            if (this.c.creationModal || options !== undefined && options.creationModal) {
                this._creationModal(createFunction, identifier, options);
            }

So there doesn't seem to be a way to override it. It'd be nice if you could pass a flag into the add method as a second param like you can for the remove. As a workaround, I ended up turning it on and off:

                                dt.settings()[0]._stateRestore.c.creationModal = false;
                                dt.stateRestore.state.add(TEMP_STATE_NAME);
                                dt.settings()[0]._stateRestore.c.creationModal = true;

Not pretty but it does work. If there's a better way to handle this I'd be happy to hear.

Thanks.

Ben

kthorngren · March 11

when when you loop through the states you don't have to actually load the state because you can simply grab the data right off it. (Correct me if I'm wrong.)

Correct. Sounds like your data changes in such a way that the same search will yield different results depending on when it's performed. In this case this solution won't work.

I was then able to use it in conjunction with the filter() method with a custom function to essentially implement that filter on the fly. It worked, was quick and didn't require loading/saving states but it was a lot of code and felt really brittle.

Yes, depending on how the table is searched this could have different results. For example Smart Search versus Regex, etc. You could post a test case or update mine with your working code. Maybe we can provide suggestions to make it feel less brittle.

I was hoping there was a way to grab a state object and then do something like call filter() off that to get the table's data with the state's filter applied without actually affecting the data the user is looking at.

There is nothing built-in for this. You could create an API plugin that could be called with the stateRestore object as the parameter returning the data from the table. This might help clean up the code and provide one place to make changes.

@allan can comment on adding it as a feature.

Unfortunately it appears the DataTables code stops checking if it's already set to true in the config

Almost all Datatables config options are static and checked only at initialization. To change them requires destroying and reinitializing the Datatable with the desired set of options.

It'd be nice if you could pass a flag into the add method as a second param like you can for the remove.

Again @allan can comment on this.

Not pretty but it does work. If there's a better way to handle this I'd be happy to hear.

Accessing the private Datatables API directly like that is not recommended. However sometimes it is necessary to accomplish what you want. The best way to handle this is by creating an API plugin. This way if the private API changes you can update the API to accommodate the changes.

Here is a simple example that might help you get started with either API code:
https://live.datatables.net/fatuhada/1/edit

Kevin

allan · March 12

I was hoping there was a way to grab a state object and then do something like call filter() off that to get the table's data with the state's filter applied without actually affecting the data the user is looking at.

Sorry no, there is no option for that at the moment. A number of the operations only "commit" when a draw is performed - for example you can apply multiple filters to a table, but the filtering actions will only happen when the table is redrawn. As such there is no way to get the display data from a table "offline". The closest option would be to have a hidden table that you can change the state of.

It'd be nice if you could pass a flag into the add method as a second param like you can for the remove.

Agreed, that is a design flaw in the way StateRestore currently operates. My intention with the rewrite is to separate the actual states from the UI layer. For example, I recently wrote a table to display the states rather than the button list for a client and that worked really well.

I've just made a few commits to StateRestore to address:

The delete all confirm modal will now be shown by default with an option to disable
The delete all title works for plurals now
The delete all modal has a close icon.

The nightly build has those changes if you want to try it out.

The deeper fixes will hopefully drop out of my rewrite, which I am aiming to work on when I get DataTables 3 beta 1 out (that isn't far away).

Allan

bg7 · March 13

Allan,

Makes sense. That's exciting about DataTables 3! Definitely looking forward to it.

I noticed in using the standard ajax configuration for StateRestore here:

https://datatables.net/extensions/staterestore/examples/initialisation/ajax.html

config: {
                    ajax: '../data/stateRestoreSave.php'
                }

that it sends data as this type:

"application/x-www-form-urlencoded"

Is there a way to configure it to send it as json? It would be easier to process on the server. I'm using the custom ajax function instead to do that but thought I'd ask in case I'm missing a simple config flag.

Thanks.

Ben

allan · March 13

Hi Ben,

Another one for the list! Unfortunately no, that parameter takes a string or a function. There isn't currently a way to configure extra parameters for it. Ideally it should be like the DataTables ajax option which can be a string, object or function. I can look at adding that in if you like?

Allan

bg7 · March 15

Allan,

If you added it that would be fantastic and I'd absolutely use it! Is that a simple change I could patch and try or no? It would eliminate a bunch of code on my end.

I suspect it would also help with a weird issue I'm running into with the "Remove All States" button. When I click it, one delete request is sent per state. For some reason each delete request is processed successfully on the server and all of the database records are removed but on the client it always leaves one state. And If I only have one state in the list it refuses to delete it, even though it gets deleted from the server. I checked the network traffic and the "Remove All" requests look identical to the "Remove" requests. At first I thought it was a timing issue but given that it doesn't work for a single state and the "Remove" button for the individual state works fine I'm pretty confused. I have a feeling using the DataTables code for the client would run smoother. Or I'm just completely missing something. And yeah, I know your example on the site works fine and that's why I'm guessing it'd run smoother if I could piggyback off the DataTables code to process the request. As an aside, is there a reason the Remove All isn't a single request to delete everything at once instead of looping through all the records and creating individual requests for each one?

There's something odd going on with the checkmark that shows up next to the current state.

If I create a state the checkmark shows up. If I change the search terms it disappears and if I change them back it shows back up again. All that works fine. But if I reload the page it stops working. I tracked it down to the StateRestore.prototype._deepCompare method. It looks like the comparison is failing on the column the table is ordered by, by default - "datastreamName".

            order: [3, "asc"],

I let Copilot try to track it down (I don't entirely trust that thing as it makes a mess more often than not so feel free to ignore this part) and this is what it had to say:

The failure is happening in StateRestore.prototype._deepCompare at core/uiCoreWebClient/src/main/webapp/js/datatables/datatables.js (around 51387), where it does a strict type check:
typeof states[0][keys[0][i]] !== typeof states[1][keys[1][i]]
What your keys[0][i] === '0' means
That '0' is not a column id/name. It’s the array index while recursively comparing an array entry (e.g., order item like [col, dir]).
So at that level _deepCompare is effectively comparing:
left orderItem[0] (string in one state)
right orderItem[0] (number in the other state)
Why types diverge
This comes from mixed representation of sort column:
one path keeps column token as a name/string (or numeric string),
another path resolves/normalizes to a numeric column index.

I then let it try to "fix" this and predictably it made a mess that didn't help at all.

One other thing I stumbled across is what I think is a typo. There are a bunch of references in the code to "isPreDefined". There's one single reference to "isPredefined". Note the lower case 'd' in the second one. I'm guessing it's supposed to be uppercase. No clue what the consequences might be of the naming mismatch elsewhere in the code.

The two are showing up in the same object which was very confusing at first because my brain didn't initially notice the case difference and I couldn't see how it could have two of the same key, let alone with one set to true and the other false. You can actually see both appearing in some other posts:

https://datatables.net/q/?q=isPreDefined

Thanks.

Ben

bg7 · 3:34PM

Allan,

This morning I let Copilot take another crack at the ordering and state matching issues I've been seeing. This time the patches it generated worked.

Previously if I saved a state, then reloaded the page, then loaded that saved state there were two issues. First the column it should have been sorting on it didn't sort on. Second, even if I sorted on that column manually the checkmark in the saved state dropdown didn't show up to indicate it matched. Copilot gave me two blocks to update and both of those issues are now resolved.

The first block it had me put in StateRestore.prototype._deepCompare to replace this:

// If the keys dont equal, or their corresponding types are different we can return false
            if (keys[0][i] !== keys[1][i] || typeof states[0][keys[0][i]] !== typeof states[1][keys[1][i]]) {
                return false;
            }

With this:

            // If the keys dont equal, or their corresponding types are different we can return false
            var k0 = keys[0][i];
            var k1 = keys[1][i];

            if (k0 !== k1) {
              return false;
            }

            var leftVal = states[0][k0];
            var rightVal = states[1][k1];
            var leftType = typeof leftVal;
            var rightType = typeof rightVal;

            if (leftType !== rightType) {
              var numericLike =
                (leftType === 'string' || leftType === 'number') &&
                (rightType === 'string' || rightType === 'number') &&
                leftVal !== '' && rightVal !== '' &&
                !isNaN(leftVal) && !isNaN(rightVal) &&
                Number(leftVal) === Number(rightVal);

              if (numericLike) {
                continue; // critical: prevents falling through to strict !== compare
              }

              return false;
            }

The second change went into StateRestoreCollection.prototype._fixTypes. This block:

        if (state.order) {
            for (i = 0; i < state.order.length; i++) {
                fixNum(state.order[i], 0);
            }
        }

got replaced with this:

        if (state.order) {
          for (i = 0; i < state.order.length; i++) {
            var entry = state.order[i];

            if (!Array.isArray(entry) || entry.length < 2) {
              continue;
            }

            var v = entry[0];

            // Convert numeric strings only; keep named columns as strings
            if (typeof v === 'string') {
              if (/^\d+$/.test(v)) {
                entry[0] = parseInt(v, 10);
              }
            }
            else {
              fixNum(entry, 0);
            }
          }
        }

It's now ordering properly on selecting a saved state and the matching works to get the checkmark to show up.

Any thoughts on these changes?

Thanks.

Ben

State Restore - looping through states and filter data

State Restore - looping through states and filter data

Answers

Howdy, Stranger!

Categories

DataTables

State Restore - looping through states and filter data

State Restore - looping through states and filter data

Answers

Howdy, Stranger!

Quick Links

Categories

DataTables