Slow table rendering using Django as server-side

AskSmart

Hi.

This is a template of my client-side code empowered by DataTables: http://live.datatables.net/xipileni/13/edit

For now, the data for the table comes from the Django templating engine, such that all the rows are embedded at run-time, into the HTML file (there might be an enormous amount of data).

Of course, this kind of rendering is extremely slowish, and with a database of size greater than 1GB, my application crashes.

This stems from 2 main problems:

1. Client-side: As described above - no matter what, once the table view is requested by a client, all the database's records are rendered at once into the HTML file, via the Django templating engine.

2. Server-side: In order to fetch the database's data for (1), the table view queries the database to provide it with all the records it has. This, without the client-side rendering, is sufficient to slow things down.

Can you recommend me a way to overcome these problems?

Thank you.

kthorngren

Start with this FAQ. Sounds like your best option is to implement server side processing so that only the page being displayed is fetched from the DB and sent to the client. Using server side processing will not use the date range search plugin you have as that is a client side only process. You will need to implement that feature in your server script when querying the DB. You can use ajax.data to send the date range parameters to the server, like this example.

There are some Django Server Side Processing libraries, like this, that might help you implement the server side protocol. I haven't used them so not sure how well they work.

Kevin

AskSmart

@kthorngren
Hi Kevin, and thanks for replying.

Can't it be simplified by tuning the SQL query to receive like a tuple of:

( The page number, The number of entries to be shown (as the client can choose at the bottom 10/25/50/100 entries) )

What do you think of that?

How can I send this tuple to the server view from the client-side?

kthorngren

Just for background start with reading about the differences between the supported processing modes.

Can't it be simplified by tuning the SQL query to receive like a tuple of:

You can create your own paging functionality but there will be complications. If you do this then Datatables will be in client side processing mode. This means all the functions like sorting, searching and paging will be performed with the data in the client. Which might be ok for your solution. You can then create your "Fetch the next set of data" button to load a new set of client side data. For example you can load 1000 records, then use your button to load the next set of 1000 records.

However if you need these features to be performed against the full set of data then you will need to enable server side processing and have a server script that supports the protocol.

Kevin

kthorngren

Another option, if you want to create your own paging, is to turn off the sorting, searching, page length and paging features and create your own inputs for these which you send to the server via jQuery ajax().

Kevin