CORS policy issue

rf1234

I know this is not quite a Data Tables question but I guess someone might be able to help me out with this ... e.g. @kthorngren

I do a lot of screen scraping and the like to import all kinds of data using cron jobs. I only do this server side using PHP. Works fine, no issues. Now I have a challenge: I need to do something similar with Java Script at the front end. Why can't I do this server side? The hyperlinks I follow can be intranet links that require the user to have an intranet connection or to be logged in. Hence the server side call won't work. Must be done client side.

This is what I am trying to do:
* After the user successfully saved the data including the hyperlinks using Editor I need to follow those hyperlinks, get the data as text and then send them to the server so that they can be saved in a large text field that is available for full text search.

This is my code so far. As you can see I don't have the server call yet (no problem) but simply log the text extracted to the console. But that doesn't make a difference.

editor
    .on('submitSuccess', function (e, json, data, action) {   
        if ( action !== "remove") {            
            //comma/blank separated string of hyperlinks :
            var linkArr = json.data[0].ctr.links.split(', ');
            $.each(linkArr, function(key, value) {
                $.get(value).then(function(data) {
                    var blb    = new Blob([data], {type: "text/plain"});
                    var reader = new FileReader();

                    // This fires after the blob has been read/loaded.
                    reader.addEventListener('loadend', (e) => {
                        var txt = e.srcElement.result;
                        console.log(txt);
                    });
                    // Start reading the blob as text.
                    reader.readAsText(blb);
                });
            });
        }
    });

If I use hyperlinks from my own domain this works like a charm. But if I use links from other domains I get this error.

I tried many things including this one here in PHP on the respective page. But nothing worked. The error persisted.

header('Access-Control-Allow-Origin: *');

Would anyone have a solution for this using Javascript or jQuery? Thanks for your help.

kthorngren

I have not needed to solve this issue but maybe this AJAX Cross Origin plugin might help or maybe this article will offer some ideas.

Kevin

rf1234

Hey Kevin, I forgot to turn on forum notifications and only just saw this. I will try the plugin. There is a way to turn off CORS by the way but you need to do it in your browser using an extension. That did the trick for me but it won't work for my users.

The other way is using JSONP or a proxy. Proxy won't work because it is another server which means the user's intranet login cannot be used. I tried JSONP already and it gave me a CORB error. CORB is another security mechanism implemented in Chromium browsers.

allan

If I use hyperlinks from my own domain this works like a charm. But if I use links from other domains I get this error.

Its working as intended then. It is the server operator who needs to allow CORS headers on the target URL - in the case above w3schools. That's a server you control, so you can't add that header - thus your only option is to use a proxy really. JSONP isn't suitable unless you control the server as well (or it already knows to response as javascript with a function (which is all JSONP is).

Allan

rf1234

Ok, but if I use s proxy I will not have access to the user's intranet pages?! Or is there a way, Allan?

kthorngren

One option might be to try the URL in Javascript and if it fails send the link to your server PHP script to fetch and return the data. If both fail then, hopefully, its not due to CORS.

Kevin

allan

Only if the proxy was inside their Intranet. Otherwise your Javascript would be able to request their intranet pages and send them back to some other server, which would be a massive security hole! The very one that CORS prevents.

Allan

rf1234

Thanks to both of you!

I will need to ask every client's IT dept to allow access to the respective domain ... That's probably not going to work because it is the public sector ... Will take forever ... Just like vaccination procurement, if you know what I mean ... Invented here but not available here. Stay healthy!

rf1234

@kthorngren I always do this server side using PHP. No CORS, no worries. I only need the client side access due to intranet pages holding the content. Thank you!

allan

If the screen scraping is part of the service you provide your customers, perhaps you could give them an Electron app that they would run internally and would do the scraping and reporting. That would give the admins the ability to review it. I haven’t to say though, it isn’t the kind of thing any of the network admins I’ve worked with in the past would allow - the whole point of the intranet is that is stays private. It really depends on the service you provide your customers and the terms of that service.

Allan