Cookie Limit? (Long Cookie Fails)

datastorm

Hello,
I am having this issue:
http://pastie.org/605921

A short cookie works for me, but the cookie for my really big table fails because I think it is hitting the limit.
Is there any way I can specify which columns I want to sort OR maybe there is a way to modify the code and make the long cookie work?

Cheers!

allan

Hi datastorm,

Cookies have a hard limit of 4Kb (from the old Netscape days), and your cookie doesn't look like it's anything near that - just under 1Kb. So I don't think you are having a problem there - sticking some debug into DataTables where it reads the cookie would tell for sure. Beyond that, it's just a case of adding some debug information to figure out why it might be failing (number of columns changing?).

Regards,
Allan

mathie

Not sure how related this is but I got 400 bad request in Firefox (request header exceed server limit from Apache) after several datatables together on the same page.

allan

Hi mathie,

Cookies have a limit of 4Kb in size. I'm not sure if this is all in one cookie, or the sum of multiple cookies, but it sounds like you might be hitting that. One thing you could do is to remove, from DataTables core, the saving of individual column filtering (for example) and anything else that you might not need to save - to save to space.

Also looking at the Apache debug logs might yield some clues.

Regards,
Allan

mathie

It happens again to me. It's multiple cookies, each is small but the sum is about 6Kb and yet Firefox already died. I would suggest a method to encode cookie data to make it smaller since not many people read cookie contents trying to understand it so human-readability has a lower priority. Modifying the core is possible but will need to patch for every new version. If a site uses DataTables extensively, this limit can be reached very easily. Thanks for listening to my requests.

Son

Error message is below:
___________________
400 Bad Request
Your browser sent a request that this server could not understand.
Size of a request header field exceeds server limit.

http://pastie.org/683927 (full cookie string)

allan

Hi Son,

Thanks very much for posting your cookies - that is very helpful and this problem is a little more serious than I had thought. The problem is not multiple tables on each page, but rather multiple pages which are using DataTables and DataTables is storing them as a root cookie. The reason for this is I ran into bugs in IE when trying to set a path of cookies (which would fix this issue). I don't believe encoding the data would help fix the problem, but rather I need to find a way to work around the IE issue. Added to my to-do list...

Regards,
Allan

jperlman

Hi Allan,

I'm running into the same problem as described by mathie, was this ever fixed or is it still on the to-do list?

Thanks.

Jonathan

allan

Yup - still on the to do list I'm afraid. Other stuff keeps over taking it :-(. I'll try to bump it up again!

Allan

vex

allan, how did you set the path? I found this which might be off help: http://stackoverflow.com/questions/2156399/restful-cookie-path-fails-in-ie-without-trailing-slash

Seems like the path has to have a trailing slash for IE to work:
[quote]Also, due to an obscure bug in the underlying WinINET InternetGetCookie implementation, IE’s document.cookie will not return a cookie if it was set with a path attribute containing a filename.

For instance, if a page sets a cookie on itself like so:

Set-Cookie: HTTPSet-PathCookie=PASS;path=/check.htm

…the cookie will be sent with HTTP requests but will not appear in the document.cookie collection.[/quote]

Hope this helps find a solution, since I'm also hitting the cookie limit.

allan

Hi vex,

That's superb info - thanks for that. Bookmarked this page :-)

Regards,
Allan

vex

I've played around with it a bit and unfortunally it doesn't seem to work. I'm using mod_rewrite so never ever have a filename in the URI. Though I want to propose a change anyhow that should ease the current situation a bit. Instead of setting the path to / set it to to the location, but exclude the last part. This should be the same as not specifying the path at all. For example, for the path "datatables.net/this/that/example.html" set the path to "/this/that/". And since we're now storing most of the URI in the path we only need to append the last bit to the cookie name (_example.html instead of _this_that_example.html).

This way the cookie limit will not be reached as quickly as it is now. The modified code:
[code] function _fnCreateCookie ( sName, sValue, iSecs )
{
var date = new Date();
date.setTime( date.getTime()+(iSecs*1000) );

/*
* Shocking but true - it would appear IE has major issues with having the path being
* set to anything but root. We need the cookie to be available based on the path, so we
* have to append the pathname to the cookie name. Appalling.
*/
var parts = window.location.pathname.split('/');
sName += '_' + parts.pop().replace(/[\/:]/g,"").toLowerCase();
document.cookie = sName + "=" + encodeURIComponent(sValue) +
"; expires=" + date.toGMTString() +
"; path=" + parts.join('/') + "/";
}

function _fnReadCookie ( sName )
{
var parts = window.location.pathname.split('/'),
sNameEQ = sName + '_' + parts[parts.length-1].replace(/[\/:]/g,"").toLowerCase() + '=';
var sCookieContents = document.cookie.split(';');

for( var i=0 ; i

vex

Patch for above changes for 1.6.2 incase anyone wants to test it.
[code]--- dataTables-1.6/media/js/jquery.dataTables.js 2010-03-01 08:41:55.000000000 +0100
+++ dataTables-1.6/media/js/jquery.dataTables.js 2010-03-03 09:43:04.611268789 +0100
@@ -4683,10 +4683,17 @@
* set to anything but root. We need the cookie to be available based on the path, so we
* have to append the pathname to the cookie name. Appalling.
*/
+ /*
sName += '_'+window.location.pathname.replace(/[\/:]/g,"").toLowerCase();

document.cookie = sName+"="+encodeURIComponent(sValue)+
"; expires="+date.toGMTString()+"; path=/";
+ */
+ var parts = window.location.pathname.split('/');
+ sName += '_' + parts.pop().replace(/[\/:]/g,"").toLowerCase();
+ document.cookie = sName + "=" + encodeURIComponent(sValue) +
+ "; expires=" + date.toGMTString() +
+ "; path=" + parts.join('/') + "/";
}

/*
@@ -4697,7 +4704,9 @@
*/
function _fnReadCookie ( sName )
{
- var sNameEQ = sName +'_'+ window.location.pathname.replace(/[\/:]/g,"").toLowerCase() + "=";
+ //var sNameEQ = sName +'_'+ window.location.pathname.replace(/[\/:]/g,"").toLowerCase() + "=";
+ var parts = window.location.pathname.split('/'),
+ sNameEQ = sName + '_' + parts[parts.length-1].replace(/[\/:]/g,"").toLowerCase() + '=';
var sCookieContents = document.cookie.split(';');

for( var i=0 ; i

gamluca

I experience the same problem, and for the moment i disabled the SaveState function.

I try to apply this patch to the 1.6.2, but i receive an error, any help ? i'm not very skilled :D

[code]
root@testTT:~# patch dataTables-1.6/media/js/jquery.dataTables.js datatables.patch
patching file dataTables-1.6/media/js/jquery.dataTables.js
Hunk #1 FAILED at 4683.
patch unexpectedly ends in middle of line
Hunk #2 FAILED at 4704.
2 out of 2 hunks FAILED -- saving rejects to file dataTables-1.6/media/js/jquery.dataTables.js.rej

[/code]

bigsilk

Just my two cents:

Have you considered html5? I'm looking into it now. See my discussion. Hasn't gotten going yet, but you might check back.

allan

Hi vex,

Thanks very much for the patch - it will be included in DataTables 1.7. Having spent a bit of time investigating this, I fail to see what could be done that is any better given the IE bug.

Having said that, what I have done is to implement a check to see if we are about to go over the typical 4KiB buffer side for document.cookie (although of course it can chance from browser to browser - IE8 allows 10KiB if I recall correctly) and if so, to delete one of the old DataTables cookies. This delete process will find and delete the oldest DataTables cookie in the path (the create time is stored in the object put into the cookie now, since this isn't available in document.cookie). While far from perfect, it does mean that we shouldn't run into any more 500 error issues. However, it does mean that if a lot of cookies are being saved with the same path, some data will be lost...

What do you all think?

@bigsilk: Yeah, using a local database would be ideal - but not suitable for use in DataTables at this time since it is not widely deployed (i.e IE).

Regards,
Allan

Petah

I'm still running into this problem in firefox.

Petah

Any word on this.
It is happening every time I request a page that has about 20 data tables in.

allan

There is code in DataTables which should handle this already (see _fnCreateCookie) but I only tested this over multiple pages - not on a single page. I think the simple fact of the 4KiB limit of cookies and the amount of data needed to store state for 20 tables being more than 4KiB is bad news - and I don't really see that there is a huge amount that DataTables can do about it - it's simply not possible to hold that much data in a cookie - therefore a different approach would be needed. For example it might be possible to use the cookie callback functions to send the state to the server and store it there in a database - then you have effectively no limit.

Sorry I can't help more than that :-)

Allan

Petah

Ok, I lied. Its not happening every request. Its quite sporadic, it defiantly happens more often when there are a lot of tables on one page, but also sometimes happens with just a few. The number of cookies gets quite high though (maybe 20-30).

Is there any hooks that I could use to be able to use my own load/save methods?

allan

There is fnStateSaveCallback and fnStateLoadCallback ( http://datatables.net/usage/callbacks#fnStateSaveCallback ) which you can link into to modify / copy the state saving DataTables does. It might be worth changing the line "if ( iLength+10 > 4096 )" in DataTables to "if ( iLength+1000 > 4096 )" - just as an experiment to see if it helps.

Allan

Dave177

I'm on datatables 1.7.5 and I just had a user try to send a 8KB cookie to the server. Not sure how that is possible, wasn't datatables going to cut it off in the 1.7.x series?

http://pastie.org/1962694

allan

That's a big cookie :-). Yup DataTables 1.7.x+ has some code in it which should try to mitigate this problem by deleting older cookies. Can you perhaps send me a link to your site which is seeing this issue please?

Thanks,
Allan

Dave177

The site is internal, so you won't be able to access it. What are you looking for?

allan

Why it is happening :-). I don't know why this would be is the thing, since there is code in there which should stop this - so I suspect some trace debugging of the functions involved will be required. Are you doing anything special with the cookies, like changing the name or anything?

Allan

Dave177

I'm also including tabletools 2.0.0 and colvis 1.0.3 on my pages. I don't believe I'm doing anything with cookies. The frustrating thing is I can't reproduce this one demand.

Most of my javascript declarations look something like the following (although I do have one complicated one with ajax backend refreshes http://pastie.org/1965804)

$(document).ready(function() {
$('.datatable').dataTable( {
"sScrollX": "100%",
"sDom": 'CT<"clear">lfrtip',
"oTableTools": {
"sSwfPath": "/swf/copy_cvs_xls_pdf.swf",
"aButtons": [ "copy",
{
"sExtends": "xls",
"sFileName": "Sidekick.csv.csv",
"bFooter": false
}
]
},
"aaSorting": [[1,'asc']],
"sPaginationType": "full_numbers",
"bStateSave": true
} );
} );

PhilWhite

[quote]
DataTables 1.7.x+ has some code in it which should try to mitigate this problem by deleting older cookies
[/quote]

I think there's a bug in this code in 1.8.1 where it tries to calculate the current cookie size - it passes the wrong string to _fnReadCookie(), fails to find an existing cookie and so double-counts it. This causes it to delete older cookies when it hasn't yet reached the 4kb limit.

Fix is to replace '_fnReadCookie ( sNameFile )' with _fnReadCookie (sName)'

allan

Thanks for the heads up! I'll take a look :-)

Allan

Benjamin

Is there a way to delete data table cookies, without deleting non-related cookies. E.g delete cookies on click of a button?

NetDev

Hi

Why use cookie?
Is www.jstorage.info (LocalStore) not the better way?

Or a hybride. When jstorage not work, then use cookie.

What's the easiest way to implement this in the current version?
Which methods or callbacks can i use for this?

Regards

logan

Hello,

I have the same problem with the cookies since I'm using multiple datatables on the same page. After a couple states saved for different tables (like 5 cookies) it gives the bad request error. If there's a new work around for this (such as what NetDev has recommended) I would like to know the updates...

Regards
Logan

jschell12

I second logan on this, I am using 1.9.0 and I had a stakeholder (only one oddly) get this error in IE-8 (even though I use Chrome Frame to try and address compatibility issues):

[quote]HTTP 400 - Bad Request (Request Header too long)[/quote]

Anyways, she was able to fix her problem by not only deleting the cookies but also unchecking the Preserve Favorites box.


P.S. Love this plugin