DOM Filter not using encoded HTML?

MrRoo

We are using the Datatables plugin with multiple sources (Oracle via ASP, XML and SOAP) and have recently stumbled apon a strange issue with the & char.

If we have the text: Apples&Pears and try to search for s&p it will not find any results. However searching for S&a_m_p;p (remove _) will work. We also find results with &n_b_s_p; (remove _) when searching for &.

We have tried to perform replaces in ASP and JS on the data as it is loaded, however it still appears to find & and not &. We tried to replace &a_m_p; (remove _) with \u0026 - same issue...

We thought of setting the column types to HTML, but that did not help. We are using Datatables as a basic setup with three columns:

[code]
$('#tableid).dataTable({
sPaginationType": "full_numbers", // Set paging style.
"sDom": '<"top"lf<"clear">>rt<"bottom"ip<"clear">>', // Set layout for the various items.
"aoColumns": [ // How to sort?
{ "sType": "html" }, // NO HTML.
{ "sType": "html" }, // NO HTML
{ "sType": "html" } // NO HTML.
]
});
[/code]


Has anyone had this issue before? I would hate to remove the exisiting filter box and repalce it with a custom box that encodes the search query first.

allan

If you search for "s&p" - does that work?

Unfortunately there is no easy way of doing this transform in Javascript (inserting into a node is the best it gets as far as I am aware) - but this certainly warrants further investigation...

Allan

MrRoo

Hi Allan,

Thanks for your fast reply. For some reason no matter how we try to encode the &a_m_p; (_ added else it would show the symbol only) it does not allow us to search with the symbol.

We are thinking of inserting our own filter box, with an encode HTML function (& to &a_m_p;) before it is sent to the Filter API, that would be one solution...

The html space (&n_b_s_p;) is also found when searching for the & symbol.

Edit: When adding the quotes, no results are found, even when providing an exact match in the records.

MrRoo

Hi Allan,

we have been doing some more testing with two scripts we are using. Both pull data from the same source (ASMX webservice) using SOAP. One script uses the jQuery Append to build the table, then run the DataTables on this table. This script does not allow us to search by s&p.

The other, makes use of a js array and the addData API in DataTables, this script allows us to search s&p.

It would appear that the jQuery append is causing the issue of encoding the & symbol?

MrRoo

Hi Allan,

On further investigation if we do not decode the string before inserting it in the datatables using the addData API it still has the issue.

When DataTables builds the filter entry for each row, does it take the HTML text i.e. will see &a_m_p; and not &? If we insert the values into DataTables, then its not an issue because it sees & as a symbol, without the HTML encoding?

We tested datatables on a pure html table and it can only find the row if we enter in &a_m_p;

allan

The strings which are filtered on use whatever innerHTML gives - so this should be an entity for something like &. So I think that is where the problem is coming from. You can dump the fnSettings().asDataSearch array to console.dir (or whatever) to see the strings that are being search on.

One method to overcome this for now, would be to detach the filtering function DataTables puts onto the input box, and then put your own on it (calling fnFilter) and escaping the string with html entities (which I'm not 100% how one would do...).

Allan

MrRoo

Hi Allan,

We will try do some more testing today - thanks for your assistance thus far.

On a side note, FireFox claims that innerHTML will read the symbols with encoding: "If a , or node has a child text node that includes the characters (&), (<) or (>), .innerHTML will return these characters as &a_m_p, &l_t and &g_t (remove _) respectively. Use element.textContent to get a correct copy of these text node's content.". The source: https://developer.mozilla.org/en/DOM:element.innerHTML

Having said that, it appears only to occur with a few containers...

MrRoo

Hi Allan,

We tested by console.dir(fnSettings().asDataSearch) and it shows the filter array with &a_m_p; and not the symbol &. What we did for testing:

1. Download the latest DataTables 1.6.2 and open the zero_config.html file.
2. Do a string replace for " / " and repalce it with the symbol &. (remove the spaces when adding the &) : i.e. Win98+ / OSX becomes Win98+&OSX.
3. Update the javascript that configures DataTables (see below).
4. Run in FireFox with FireBug open (using log for a condensed result set on the forums, .dir shows same results).

[code]

var oTable;

$(document).ready(function() {
oTable = $('#example').dataTable();
var oSettings = oTable.fnSettings();

console.log(oSettings.asDataSearch)
} );

[/code]

Firebug results (a reduced set - remove the _):
[code]
["Gecko Firefox 1.0 Win 98+&a_m_p;OSX.2+ 1.7 A ", "Gecko Firefox 1.5 Win 98+&a_m_p;OSX.2+ 1.8 A ", "Gecko Firefox 2.0 Win 98+&a_m_p;OSX.2+ 1.8 A ", "Gecko Firefox 3.0 Win 2k+&a_m_p;OSX.3+ 1.9 A ", "Gecko Camino 1.0 OSX.2+ 1.8 A ", "Gecko Camino 1.5 OSX.3+ 1.8 A ", "Gecko Netscape 7.2 Win ...p;Mac OS 8.6-9.2 1.7 A ", "Gecko Netscape Browser 8 Win 98SE+ 1.7 A ", "Gecko Netscape Navigato...n 98+&a_m_p;OSX.2+ 1.8 A ", "Gecko Mozilla 1.0 Win 95+&a_m_p;OSX.1+ 1 A ", "Gecko Mozilla 1.1 Win 95+&a_m_p;OSX.1+ 1.1 A ", "Gecko Mozilla 1.2 Win 95+&a_m_p;OSX.1+ 1.2 A ", "Gecko Mozilla 1.3 Win 95+&a_m_p;OSX.1+ 1.3 A ", "Gecko Mozilla 1.4 Win 95+&a_m_p;OSX.1+ 1.4 A ", "Gecko Mozilla 1.5 Win 95+&a_m_p;OSX.1+ 1.5 A ", "Gecko Mozilla 1.6 Win 95+&a_m_p;OSX.1+ 1.6 A ", "Gecko Mozilla 1.7 Win 98+&a_m_p;OSX.1+ 1.7 A ", "Gecko Mozilla 1.8 Win 98+&a_m_p;OSX.1+ 1.8 A "]
[/code]

As you said, we have been investigating a custom filter box to apply to the scripts when we load the library with an existing table.

Note: we ran a similar test for &n_b_s_p;, &l_t; and &g_t; all having the issue of not being encoded for the seraching.

For a temp solution we have overwritten the filter event as follows (remove the _):
[code]
$(".dataTables_filter input").unbind().keyup( function () {
var sUserFilter = this.value;
sUserFilter = sUserFilter.replace(/\&/g,'&a_m_p;').replace(//g,'&g_t;');
oTable.fnFilter(sUserFilter);
});
[/code]

Would you suggest a better method to perform this?

allan

A slightly more generic (temporary) solution would be to put the value into a created element. For example:

[code]
$(".dataTables_filter input").unbind().keyup( function () {
var n = document.createElement('div');
n.innerHTML = this.value;
oTable.fnFilter( n.innerHTML );
});
[/code]
That will take care of other entities like £ etc. But it's still not quite a solution I'll be happy putting into DataTables... Still thinking of a good way to do it.

Allan

MrRoo

Hi Allan,

Thank you that fix is suitable for our needs, however it only appears to filter if the last char is a space (in IE7, FF works fine)...

allan

Hi MrRoo,

Not sure what would be causing the last space issue, but I've got a better fix for this html entity in a filter issue now. I'll include it in the 1.7 beta 3 which I hope to release soon. I've just tried it with IE7 and it doesn't show the space issue, so hopefully this will do the trick for you.

Regards,
Allan