Design Suggestion: Paginate over a subset of API data from an API with limited search/filter

sjordan

This question is more a request for a suggested design based on the community's experience.

Essentially I need to paginate over a subset of API data from an API with limited search/filter and potentially large data volume.

Assume there is a service accessed via an API that has hundreds of records. The service returns pages of results 20 at a time (i.e. pagination). On my UI, I wish to display valid records from the records returned. However, not all records are valid and the API has essentially no filter feature.

The UI should always attempt to display 20 records even if another call to the API is needed. Calls to the API pass both a previous and a next page token. Can you suggest an ajax pagination strategy to display 20 valid records at a time, page for the additional records as needed and keep both the API calls and the UI display in sync while navigating to previous and next screens.

I'm considering on initial page load to grab a reasonably large number records from the API (ex. 500 records) on the server, filter server-side and pass valid records to the data option of the DataTables data source.

I'm hoping this not such a rare requirement.

allan

Hi,

If filtering is important to you, and the remote API doesn't really have much in the way of filtering, then I would suggest doing some kind of proxy / cache. Read the remote data and store it in your own database (or even just a JSON file) when DataTables can then read. Of course with caching comes invalidation issues - I don't know what the data is, how often it might change, new record notification, or anything like that, so that would be up to you, but it is worth considering to overcome the limitations of the API.

If that isn't an option, then use ajax pointing to the API URL in question and enable server-side processing. The problem you will hit is that DataTables needs to know how many records there are in the full data set to show its pagination. To overcome that (to some extent) change the pagingType to be simple (previous and next buttons only) and use the xhr event to fake a value for the recordsTotal and recordsFiltered return JSON parameters (or set to a real value if the API returns the number of records in its response).

Are you able to tell me what API it is you are working with? Sounds like the Stripe one, but I'm sure a lot operate like this!

Allan

sjordan

Hi Allan

Thank you for your response.

I am happy to share the API with you. It's the Calendly API and the scheduled events endpoint in particular (see docs https://developer.calendly.com/api-docs/2d5ed9bbd2952-list-events)

Essentially, I'm pulling apppointments and wish to display them on my site.

You'll notice a couple challenges:

• While some sorting has been implemented in the API, the only supported value is start_time. So that means no sorting by appointment owner, location, appointment name, etc.
• Filtering is limited. For example, no option to only return appointments at a certain location

I hear what you say about caching and invalidation. That could be a challenge because I could imagine that the nature of appointments, they change relatively frequently: new appointments, cancellations, reschedules, edits, etc.

Thank you
SJordan

allan

Thank you. Yeah, it does look limited in terms of filtering and sorting. If there was a webhook to give you a notification of a change or addition, then locally cached data might be worth considering (if sorting and filtering are important enough to your application to warrant that work).

Aside from that the API doesn't appear to give the total number of records that are available, so you'll need to fake the recordsTotal and recordsFiltered values, and change language.info to just show the current page number (or disable info).

Allan

sjordan

Just circling back on this.

Thanks to Allan's suggestion to address the remote API limitations, I implemented a webhook (see https://developer.calendly.com/api-docs/c1ddc06ce1f1b-create-webhook-subscription).

It was frustrating to get 90% of the way to my ideal solution with the API approach, only to have to rewrite my code.

This shows the importance of thoroughly understanding requirements and limitations, and doing some spec development before committing to a solution.

But hey, I did learn a strong use case for webhooks: they allow data to be stored locally, which gives more control to the local app. Of course there are tradeoffs, such as now the data is consuming our local resources.

Find the best solution to the problem.

Cheers

allan

Thanks for the update - good to hear you got it working as you need!

Allan