EM Dash causes Invalid JSON Response error

shiftymccool

Hello all,

I have a working report using DataTables but as soon as a piece of data comes through with an EM dash I get the "Invalid JSON Repsonse" alert.

Response from server:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: CFID=26523; Path=/; Secure
Set-Cookie: CFTOKEN=97593167; Path=/; Secure
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Date: Mon, 15 Jan 2018 14:42:29 GMT

Highligted character is causing the error:

Actual text (I'm guessing the character will get stripped but I'll try anyway):
ONeal

I'm not exactly sure what I can do to solve this, any ideas? Let me know if you need more information.

Thanks!

tangerine

Presumably the intention in your data example is O'Neal.
My first and obvious question would be "why use em dash in the data"? Is it really necessary?

shiftymccool

This isn't my data. I can only assume there was a copy/paste from some text editor into the database. That part is out of my hands, I have to find a way to make it work with the data I have.

allan

If you are getting "Invalid JSON Response" then jQuery is failing to parse the JSON (and jQuery uses JSON.parse() from the browser). This suggests that the JSON data really is invalid JSON (I've never seen that error caused by anything else).

My guess is that its a character encoding issue - JSON should be UTF8. If your data set is something else, you'll need to convert it.

Allan

shiftymccool

The comment posting keeps stripping the character on view (still there on edit) so I will use screenshots to illustrate better.

If I do this:

I get this:

But, if I do this:

I get this:

I'm not sure why this is but JSON.parse() doesn't seem to have a problem with the stringified version even though it starts as a string, but it definitely has an effect on the result.

I'm already encoding UTF-8 and this character is a UTF-8 character. The JSON is displaying correctly in the Chrome dev tools so I have no idea what's going on. Any other thoughts?

shiftymccool

I have a correction. What I mistook for an "em dash" is actually an "end of medium" character (http://www.fileformat.info/info/unicode/char/19/index.htm). Would this explain the error? Why would Chrome dev tools display the character without issue while DataTables throws an error?

kthorngren

Interesting problem. Here is what I found. The JSON.parse() docs point to the ECMA 262 JSON.parse spec. In the spec is this:

Parse JText using the grammars in 15.12.1. Throw a SyntaxError exception if JText did not conform to the JSON grammar for the goal symbol JSONText.

15.12.1 has this:

JSONStringCharacter ::
SourceCharacter but not one of " or \ or U+0000 through U+001F

Unicode Character 'END OF MEDIUM' (U+0019) is in the invalid range.

Kevin

shiftymccool

I found something similar on a ColdFusion (yeah, yeah, I know) page but it's still strange that performing a stringify first seems to allow the parse to handle it just fine (see my previous comment).

kthorngren

but it's still strange that performing a stringify first seems to allow the parse to handle it just fine

In your example try this:
console.log(typeof json);

I suspect it will return "string" instead of "object". Basically you are encapsulating everything into a string then JSON.parse() is reverting it back to the original string.

I would suggest either sanitizing your data in your server code or you can use the AJAX "dataFilter" function to sanitize your data, for example:

      "ajax": {
        url: "url",
        dataFilter: function(d) {
          return d.replace(/\u0019/g, '');
          
        }
      },

Kevin

shiftymccool

You are exactly correct and the dataFilter bit works great... Now to add a bunch of other characters to that list. Thanks!

As a side note, since JSON.parse() doesn't handle these characters at all causing DataTables to throw an error, would it be reasonable to add a flag to automatically filter these?

Thanks for your help everyone!

shiftymccool

Another note. This seems to take care of that whole range:

dataFilter: function(d) {
  return d.replace(/[\u0000-\u001F]/g, ''); 
}

Thanks again!

kthorngren

As a side note, since JSON.parse() doesn't handle these characters at all causing DataTables to throw an error, would it be reasonable to add a flag to automatically filter these?

The parsing is executed by ajax before the data gets to Datatables. In the ajax docs under dataType there is this regarding JSON:

"json": Evaluates the response as JSON and returns a JavaScript object.

and

The JSON data is parsed in a strict manner; any malformed JSON is rejected and a parse error is thrown.

This is why you need to sanitize the data before it is parsed by ajax.

Kevin

allan

Awesome detective work Kevin!

Regarding the point about Chrome not throwing the error - it does if you use \u0019 directly:

JSON.parse('{"test":"12\u001934"}')
VM79:1 Uncaught SyntaxError: Unexpected token in JSON at position 11

When you use JSON.stringify (e.g. JSON.stringify({"test":"12�34"})) on the character (rather than its code point), JSON.stringify converts it to the code point:

"{"test":"12\u001934"}"

Thus it works when you JSON.parse it.

Allan