Limit rendered cells and still keep displayBuffer responsive

sebastianbarth

We have quite big tables with many columns (up to 100) and rows (unlimited). To display them, we added pagination and also use the scroller plugin for virtual scrolling.

On a 1080p screen there is enough space to have about ~37 rows visible at once. With a scroller.displayBuffer set to 4 datatable fetches about 148 rows per page request, leading to 15300 <td>s rendered. The rendering alone takes 1-3 seconds (with freeze of the browser) depending on the user's device.

This is quite a long time. To reduce the rendering effort we want to reduce the value of scroller.displayeBuffer depending on the columns displayed, for instance:
* 10 columns: displayBuffer: 20
* 50 columns: displayBuffer: 5
* 100 columns: displayBuffer: 2

This would increase the frequency for fetching and rendering data when scrolling down, but would vastly improve the performance due to limiting the number of rendered cells to a sensible, manageable count when having many columns.

While this at first sight looks like a simple thing, it fails in the real world where the available vertical screen size of a table differs much and may also change with the resolution/orientation of the device after initial rendering drastically.
For instance a table that renders 100 rows and 100 columns may will render 10k cells. By Increasing the screen size to just 150 percent the scroller plugin will fetch and render 15k cells, which increases the rendertime dramatically.

In our usecases, such high column counts are expected as a short time situation before they are decreased to a more meaningful value by the user. This usually happens in a short time, unless the browser freezes for seconds due to rendering to many cells. Often in those cases the user causes more events due to rigid exploration, leading to further freezes. That is totally frustrating.

So what we aim for is having a configuration that allows to limit the cells rendered to a fixed number, e.g. 10.000, to keep the rendering in an acceptable range.

How could we do this?

The easiest would be if there is a configuration option that allows us to do so, or since we couldn't find such option a way to calculate the displayBuffer based on the number of columns and page size (rows count):

displayBuffer: MAX_CELL_COUNT / NUMBER_OF_COLUMNS / pageSizeInRows;

Problem with this is, besides that we couldn't find a legal way to get the pageSizeInRows, that the equation is hardcoded to a specific page size and does not update with screen/container size changes.

Any options?