Pagination with json only data

arsalansiddiqui

Hi all, i'm developing logs management system. This will keep logs of network devices, there are few parts to make it.

I'm reading logs via API of that device. So the response is in array form than i json encode it. On front end i have html jquery datatable and it shows the logs.

Now the problem is there are to many logs generated per day and all the logs need to be fetched from network device on realtime, so the json has all the record which i would like to paginate to stop loading all data at once.

Note: I can get count of records in the api, maybe that helps in pagination

My json response look like this:

{"data":[{"time":"19:09:29","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:10:56","message":"dhcp alert on vlan904: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.140.1"},{"time":"19:11:09","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:11:25","message":"dhcp alert on vlan1500: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.114.1"},{"time":"19:12:10","message":"dhcp alert on vlan504: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.4.3"},{"time":"19:12:13","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:13:09","message":"dhcp alert on vlan503: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.3.3"},{"time":"19:13:36","message":"dhcp alert on vlan505: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.5.3"},{"time":"19:13:37","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:15:38","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:15:46","message":"dhcp alert on vlan503: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.3.3"},{"time":"19:17:36","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:18:43","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:19:13","message":"dhcp alert on vlan1500: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.114.1"},{"time":"19:19:18","message":"dhcp alert on vlan1713: discovered unknown dhcp server, mac 78:A3:51:49:EA:48, ip 192.168.0.1"},{"time":"19:20:17","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:22:12","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:23:06","message":"dhcp alert on vlan502: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.2.3"},{"time":"19:23:13","message":"dhcp alert on vlan503: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.3.3"},{"time":"19:23:22","message":"dhcp alert on vlan1625: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.76.1"},{"time":"19:23:27","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:23:34","message":"dhcp alert on vlan505: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.5.3"},{"time":"19:25:26","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:26:28","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:27:01","message":"dhcp alert on vlan1609: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.182.1"},{"time":"19:27:22","message":"dhcp alert on vlan163: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.14.1"},{"time":"19:28:03","message":"dhcp alert on vlan502: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.2.3"},{"time":"19:28:38","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:28:58","message":"dhcp alert on vlan1633: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.24.1"},{"time":"19:29:13","message":"dhcp alert on vlan1624: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.58.1"},{"time":"19:29:27","message":"dhcp alert on vlan503: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.3.3"},{"time":"19:29:55","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:30:59","message":"dhcp alert on vlan1609: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.182.1"},{"time":"19:31:55","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:32:01","message":"dhcp alert on vlan1609: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.182.1"},{"time":"19:32:02","message":"dhcp alert on vlan1714: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.120.1"},{"time":"19:32:13","message":"dhcp alert on vlan1624: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.58.1"},{"time":"19:32:13","message":"dhcp alert on vlan1624: discovered unknown dhcp server, mac C8:3A:35:90:23:38, ip 192.168.1.1"},{"time":"19:33:33","message":"dhcp alert on vlan505: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.5.3"},{"time":"19:33:40","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:33:50","message":"dhcp alert on vlan1500: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.114.1"},{"time":"19:34:25","message":"dhcp alert on vlan1678: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.186.1"},{"time":"19:34:56","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:35:01","message":"dhcp alert on vlan1609: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.182.1"},{"time":"19:35:24","message":"dhcp alert on vlan503: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.3.3"},{"time":"19:36:47","message":"dhcp alert on vlan1609: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.182.1"},{"time":"19:36:49","message":"dhcp alert on vlan904: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.140.1"},{"time":"19:37:17","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:38:36","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:39:59","message":"dhcp alert on vlan503: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.3.3"},{"time":"19:40:19","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:41:47","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:42:15","message":"dhcp alert on vlan904: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.140.1"},{"time":"19:43:35","message":"dhcp alert on vlan502: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.2.3"},{"time":"19:43:37","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:44:04","message":"dhcp alert on vlan504: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.4.3"},{"time":"19:44:36","message":"dhcp alert on vlan1609: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.182.1"},{"time":"19:44:43","message":"dhcp alert on vlan1500: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.114.1"},{"time":"19:45:45","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:46:20","message":"dhcp alert on vlan1609: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.182.1"},{"time":"19:46:59","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:47:07","message":"dhcp alert on vlan502: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.2.3"},{"time":"19:48:24","message":"dhcp alert on vlan1678: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.186.1"},{"time":"19:48:26","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:48:33","message":"dhcp alert on vlan503: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.3.3"},{"time":"19:49:41","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"19:50:37","message":"dhcp alert on vlan1500: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.114.1"},{"time":"19:50:45","message":"dhcp alert on vlan904: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.140.1"},{"time":"19:56:08","message":"dhcp alert on vlan1633: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.20.24.1"},{"time":"19:57:18","message":"dhcp alert on vlan502: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.2.3"},{"time":"19:58:21","message":"dhcp alert on vlan1633: discovered unknown dhcp server, mac 14:CC:20:C7:D4:74, ip 192.168.1.1"},{"time":"20:12:04","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"20:51:29","message":"dhcp alert on vlan1617: discovered unknown dhcp server, mac 18:D6:C7:EB:FD:6E, ip 192.168.0.1"},{"time":"20:52:39","message":"dhcp alert on vlan501: discovered unknown dhcp server, mac 6C:20:56:AC:F9:3F, ip 10.21.1.3"}]}

kthorngren

I would think it would take a lot of logs sent in one message to cause performance issues. But if you want to page them you will need to do it in your server code. Maybe you can add a continuation flag in the response to tell the client to fetch more data right away. Or if you determine a buffer size of 100 log messages and the response length is 100 then have the client perform another fetch right away.

Kevin

arsalansiddiqui

That's the question i dont' know how to paginate server-side php that works with datatable.

kthorngren

Datatables does have a Server Side Processing protocol which may or may not work for your situation. Your API would need to be adjusted to accept the parameters sent by Datatables. Datatables has a Scroller Extension that works in conjunction with server side processing to fetch rows as the user scrolls.

Otherwise you will need to create your own. Without knowing what your API supports and how it works it would be hard to say what to do. Is there a parameter to tell the API how many messages to return? Or do you fetch all of them via PHP and return them to the client?

Now the problem is there are to many logs generated per day and all the logs need to be fetched from network device on realtime

How often are you fetching the logs?

How do you tell the API which message to start with and how many to return?

Generally with Datatables I use jQuery getJSON() to fetch the API then use rows.add() to add the rows to the Datatable. Also, you can use rows().remove() to remove the oldest rows if you want to keep the number of rows below a certain number.

If you still have questions please provide more details of how your API works, if you are accessing it from the client or PHP and how often you want to fetch the data from the client.

Kevin