Large files upload problem

tomekkie

The Upload forms work very strange with large files.

In case of files above 100 MB - it takes ages for the select file button text - to change from "Choose file" to "0" (progress) after the selection is done. (only the title attribute changes almost immediately).

The same happens on my tables as well as on examples on this site.
On 100MB file selection, in Firefox 64 bit, I have to wait minutes for the form to react, while in Firefox 32 bit - just nothing happens.
(except of change of the title attribute)

Then the progress rises slowly to 100% stays 100% for a second, and the button text changes back to "Choose file", no error message show up, despite file validation constraints.

allan

Hi,

The long pause you are seeing before the upload starts is due to the amount of time it takes the browser to read the file. I've added an option into Editor which will show an "Uploading file" message while this is happening.

The second error is being caused by the fact that the upload is larger that PHP is allowing. This can result in either invalid JSON being returned (with a PHP warning) or valid JSON being returned, but with no upload information since the upload wasn't completed. In both cases Editor should show an error.

To fix both, find the Editor.upload static method in the Editor source code and replace with the function shown in the JavaScript pane here: http://live.datatables.net/wafudota/1/edit .

These fixes will be in the 1.5.5 release (which isn't due until the end of January).

Regards,
Allan

tomekkie

Thanks.

I had just to remove the <i> tags from the message, but it works.

Unfortunately, despite changing the server upload_max_filesize to 500M, I am still getting the Internal server error.

I have tried a number of jquery upload plugins on my site and only these supporting chunking worked with 200 MB file, when I set the chunk size below 50 MB. Is there a chance of including the chunking in the table upload?

allan

I am still getting the Internal server error.

What is the actual error? You may need to check your sever's error logs if it isn't shown in the output.

Currently the Editor upload doesn't support chunking, but that is certainly a good option for a future enhancement.

Allan

tomekkie

The files below 100 MB do upload.

With a 150 MB file - the upload reaches 100%, stays 100% for a while and throws the message in the editor:

"A server error occurred while uploading the file".

I couldn't find any related error among the server logs.

Would that be difficult to put there some simple chunking?

allan

I'm not sure that it would be difficult, but it wouldn't be trivial either. The client-side aspect is relatively simple - the split() method can be used to chuck the file and then just upload it in pieces. But the PHP and .NET libraries for Editor would need to be updated to allow for this chunking. That and aligning the client-side correctly (for progress updates, etc) would take a fair bit of time. I will look into adding this for a future version of Editor.

Allan

allan

"A server error occurred while uploading the file".

That means that the server wasn't able to fully upload the file. Probably due to a limit on the file size. That should be shown as a PHP warning (possibly you might need to turn warnings on).

Allan

tomekkie

Would it make sense to adapt a third party upload plugin, which is already known to support chunking and resumable uploads - like blueimp or plupload - and put it into the custom field type template?

I guess it should be easier for a novice and allow to keep the original plugins files intact.

allan

Yes, that is absolutely possible. The documentation for how to create plug-in field types is available in the manual.

Obviously that's only one half of it - the server-side would also need to be considered - many available uploaders probably come with PHP implementations which you could integrate into the Editor libraries.

Allan

tomekkie

Well, I have done it that way.
I have the blueimp plugin inside the custom type editor field and once the file gets uploaded - it returns the file data. The main field data is the file_id. I should collect the file data brought by plugin and append it to the post data to server inside the get function

get: function ( conf ) {
return $('#btn', conf._input).attr('value');
fd.append( 'fdata', '.......');
},

but this doesn't seem to work. Could you give me some hint?

allan

I'm not quite understanding why you would use append() in the get value function? Also, that line will never execute since it is after the return statement.

Allan

tomekkie

Oh, sorry, I wanted to withdraw this comment because I got it all working - simply by putting the field and file data into array, but I could not log in for a while.