Using Scroller with slow response

Bitpunchsoftware

I am currently using Datatables with server-side processing and the scroller plugin.

While the basic example (https://datatables.net/release-datatables/extensions/Scroller/examples/server-side_processing.html) runs fine, as soon as you increase the ajax delay a little bit the scroller plugin seems to not be able to determine the height/count of rows (i assume).
http://live.datatables.net/hupajice/1/

The same happens when using a "real" ajax call:
http://live.datatables.net/xufugiwa/1/edit

Am i missing something?
Or is there a bug in scroller?

(As my server has response times > 50ms the same behaviour can be observed with a "real" ajax option and prohibits Datatables use for me)

Edit:
Now i fear for my sanity. The behaviour is not consistently reproducible. Sometimes it works. Sometimes it doesn't. E.g. when i click on the "run with JS" button in JS Bin multiple times without changing the code.