Increase loading speeds when making an ajax call to an API.

mbrown

I'm making a call to an api to show info about a bunch of datasets in a database. For the organization below there are >700 datasets and the load time is several seconds. Most of the examples seem to use data stored locally in json files and I haven't found many examples calling an api.

Here is the call I'm making, please let me know if there is a better way to do this.

$(document).ready(function() {
    var url = 'http://catalogue.data.gov.bc.ca/api/3/action/package_search?q=sub_org:"e51a8106-11c7-4436-a967-7cee18bfb159"&rows=1000';
        $.ajax({
            url: url,
            success: function(data) {
                $('#example').dataTable({
                    data: data.result.results,
                    columns: [{
                        data,
                        render: function(data,type, row) {
                            urlstring = 'http://catalogue.data.gov.bc.ca/dataset/' + data['name'];
                            hyperstring = "<a href=" + urlstring + ">"+data['title']+"</a>";
                            return hyperstring;
                        }
                    }, {
                        data: 'notes'
                    },
                    {
                     data: "license_title" 
                 }]
             });
            }
});

})

Here is the result:

http://mbrownshoes.github.io/ckan_datatables/

I have several different organizations who's datasets I'd like to show. I've looked at server-side but I won't be able to load a script onto a server. I'm really looking for a way to load each page at a time (search feature doesnt have to work for all datasets), but show the tabs for the correct number of pages after the first load.

Any help/suggestions would be greatly appreciated.

Thanks

tangerine

Calling your url directly - i.e. without DataTables - also takes several seconds.

http://catalogue.data.gov.bc.ca/api/3/action/package_search?q=sub_org:"e51a8106-11c7-4436-a967-7cee18bfb159"&rows=1000

Are you sure you are calling their api correctly?

allan

Thank you for the link - that is a massive help.

The Ajax data being loaded is 4.7MB - that's going to take a fair amount of time on any connection I would think. 8.7 seconds just to request and retrieve the data on my connection - that's not rendering it (although that appears to happen quickly once it has been loaded).

I have three suggestions to help speed things up:

1. There is a lot of data in the JSON feed that is not used in the table. It shows only three columns, but each row has a lot (>50) properties. Stripping it back so you only send the data you actually need will help reduce the bandwidth requirements hugely.
2. Your server doesn't apply to use gzip. That will help reduce the amount of time taken to transmit the data to the client
3. Enable deferRender in DataTables to help improve the draw time. i don't think it will help much, since it isn't the part that is really slow, but it will help some!

Allan

mbrown

Thanks for the responses. Yes Alan, I think I should focus on (1) and try and reduce the amount of information that is sent from the server.

I am able to call fewer rows by setting rows=25 in the api call and I was thinking I could just make a new call when each of the datatables page tabs is selected. For instance for the second tab I could call rows 26-50. I'm also able to request the number of records (729 for this organization), it's just getting datatables to know there are that many records and so it should create this many tabs - in advance of receiving the actual data.

Perhaps this plugin would help set the number of pages from the get go?
http://datatables.net/plug-ins/api/fnPagingInfo

Thanks again for the help!

allan

It sounds like you are describing server-side processing. However, I don't believe that is required here. My point is that each object contains a lot of data that is not being used. I would suggest simply stripping out the data that is not being used - reducing the data payload and making it a whole lot faster.

Server-side processing it typically only needed for tens of thousands of rows.

Of course, if you can't remove the data that isn't being used, thens server-side processing will help, but it is a lot more work than my suggestion.

Allan