issue cdn not resolved
issue cdn not resolved
AleTeddy79
Posts: 1Questions: 1Answers: 0
hi but this in unresolved too
https://cdn.datatables.net/plug-ins/1.10.20/dataRender/ellipsis.js
not work anymore
Answers
Thank you - I'm looking into that. Use https://datatables-cdn.com/plug-ins/1.10.20/dataRender/ellipsis.js for the moment.
Allan
Hey @allan
Is that datatables-cdn.com url you posted always up and a reliable backup to the cdn.datatables.net?
Just trying to come up with a backup plan if there is ever a CDN issue again like there was this morning.
FYI, I am not upset, just updating our procedure and response plan.
Thanks in advance.
Dave
Hi Dave,
My plan is to maintain it to try and minimise disruption. It won't be the main URL - that will continue to be cdn.datatables.net. The problem at the moment is that the TTL on the cdn subdomain is quite long, so it is going to take a little while for the DNS to clear.
So while the new domain name will stay active, once the "real" CDN is responding normally, I would encourage you to switch back. Just not with any urgency - I'm not intentionally going to rug pull!
Allan
What was the original TTL set to? We would like to understand how long we should wait before switching? We are seeing intermittent caching issues with some of our downstream use cases and want to understand before taking corrective action. @allan
The main one is 5 minutes, but it is now up to individual routers. For example if you set your DNS to resolve with 1.1.1.1 or 8.8.8.8, they are all up to date and will resolve correctly. However, some routers do their own thing (they shouldn't but do - I'm wondering if some might even be smart enough to see the
cdnpart). Unfortunately, there is next to nothing I can do about that. I've got two networks here, the ISP for one is resolving correctly with their DNS servers, while the other isn't. Unfortunately, it will just take a bit of time.Allan
@allan thanks, definitely understand that. We are/were interested in if the value was something else besides 5 minutes (like at the beginning of the incident)?
Not as far as I am aware. It is bothering me that the CDN subdomain isn't refreshing as quickly as I would like. I suspect that is because it is more highly trafficked, so it won't be considered "stale" as quickly.
Allan
Understood @allan . Basically we were trying to determine if we even needed to swap URLs at all if we were approaching the "original" expiration time for most cached items anyway. Thanks again!
Just to add a note of interest - one of our users continued to have unavailability even after we switched to the new workaround domain. This confused us because our pages were loading fine for everybody except him (including myself on multiple devices and connections).
Turns out the user was behind a big corporate network whose Cisco managed firewall had flagged up datatables-cdn.com as a malicious site due to being registered only one day prior. The switch within our pages was immediately flagged as a possible script injection and thus the unavailability ensued. Ended up escalating to a very high up director who could understand the issue and override the block!
Busy day...
@ty_rex
Thanks for sharing!
@allan
As always, thanks for your support and help through this. I am just determining if I want to have that secondary url as a backup that we would automatically fall over to, or if we want to host the files as the fall over.
@afriedrichsen - we're in complete agreement.
The conversation here is really good and helpful with knowing what went on and providing info for the decisions to be made.
Thanks,
Dave
What a tough day! It was difficult to explain that the application is done correctly. I had one PC that worked with the Editor application. On the other PCs it didn't. On the working PC I went into Page Source and copied all the CDN content and then created a local link to it. But it didn't work. How can I avoid these problems in the future? Can I store some CDN content locally?
Thanks.
Sure, you can use the download manager and download pretty much everything.
Yeah - not a day I ever want to repeat this one.
You can store all of it locally if you wish
. The CDN is there as a convenience, you don't need to use it. If you go to the download builder you'll see a "Download" tab at the bottom of the page which will download the selected software.
I have taken steps to try and ensure that this particular attack can't happen again. Unfortunately it involved social engineering against the domain's registrar - DataTables is widely used and a valuble target I guess. Thankfully CloudFlare did an awesome job to help mitigate the problem. It's sickening that someone would attempt to target an open source project like this.
Thank you all for your understanding and support. Not sure about you all, but I'm having a whisky tonight...
Allan
@allan Can you clarify if this malicious action was on the old or new domain?
With the new domain it would make sense to me that security appliances like Cisco would flag it as a potential problem. Just want to understand more...if we have to wait for post incident review that is understandable.
Again, thanks for your prompt responses and hard work. Really appreciate it.
The attack was on datatables.net.
I threw up
datatables-cdn.comas a mirror ofcdn.datatables.netwhile working to resolve the issue, allowing a simple replacement of the domain to let websites continue to use it. Unfortunately for some using Cisco routers, they detected the new domain and didn't allow it. That is something I wasn't expecting and is entirely dependent on what router hardware and configuration a user had in front of them.Allan
Understood @allan. Do we know if this was a DDoS or some other type of attack? Again, if it needs to wait for postmortem understood.
Once again, appreciate the hard work and prompt answers on what was a tough day for you.
Just a point regarding alternatives to Datatables' CDN. Before I knew about the datatables-cdn.com workaround, I had figured out my own solution using the various Datatables libraries hosted at CDNJS.
By handpicking the various plugin JS/CSS sources, I was able to cobble together a replacement for the combined request that I was previously putting through cdn.datatables.net - e.g. BS5, Buttons, HTML, Print etc.
However, I noticed one plug-in that was missing - no CDNs are hosting ColumnControl. It would be fab if this could be published too as a future mitigation measure.
It was domain hijacking with social engineering against the registrar
.
I'll submit it to cdnjs tomorrow.
Allan