Paginating search results¶

Our /search endpoints return multiple records per response. How many records are included in a single response (=page size) is determined by an optional search parameter called page_size. The default value for page_size is 20, and its value can be set to any integer between 1 and 200.

To change the number of records returned, change the "page_size" parameter. E.g., specify 200 for a page size of 200 records. Please note that search requests are limited to returning only the first 100,000 records regardless of the pagination parameters. If your search filter matches more than 100,000 records, we recommend modifying your search filter to separate your result set.

When the search criteria match more records than what fits on a single page then you have to paginate the search results using one of the following strategies in order to get all the records that match your search.

Strategy 1: Calculate the number of pages¶

Every response from a /search endpoint has a field in the response header called X-PW-TOTAL. It shows an upper bound of the total number of records that match the search criteria.

To calculate the number of pages to cycle through, divide the total number of records by the page size. For the sake of example, let's assume that the page size is 200 and the total number of records is 775. The number of pages is 775/200 = 3.875 which rounded up to the nearest integer gives us 4 pages.

Now, to cycle through the pages we need to set the page_number parameter in the request to 1 initially, and then send additional requests for the subsequent pages.

The request header will look like this:

For the first page

{
  ...
  "page_size": 200,
  "page_number": 1
  ...
}

For the second page

{
  ...
  "page_size": 200,
  "page_number": 2
  ...
}

and so on.

This strategy has one caveat; if records are added/modified in the system between the time when the header field is evaluated and when the actual pages are requested, then those records may be included or omitted in/from the results incorrectly.

Strategy 2: Count the records on each page¶

In this scenario, we send a search request and specify the first page, by setting page_number to 1. Then use a logic that performs the following evaluation on the response.

Count the number of records in the response
Check if the record count equals the page_size
1. If the record is less than the page size then we are on the last page and we can stop paginating
2. If the record count equals the page size then increase page_number by one, send another request and start over with this evaluation logic.

Using the same example as above, with a page size of 200 and 775 total records the evaluation would go down as follows

Page 1

{
  ...
  "page_size": 200,
  "page_number": 1
  ...
}

The response will have 200 records which equals the page size, so let's continue requesting.

Page 2

{
  ...
  "page_size": 200,
  "page_number": 2
  ...
}

The response will have 200 records which equals the page size, so let's continue requesting. The 3rd page will have identical results.

Page 4 (last page)

{
  ...
  "page_size": 200,
  "page_number": 4
  ...
}

The response will have 175 records, which is LESS than the page size, which tells us we are on the last page and we can stop paginating.

Remember to sort you search results!¶

It is highly recommended that you sort the results you get back from a /search endpoint. This ensures that records are returned in a consistent fashion across requests. One common sorting scenario is to sort records by the date they were last updated, in descending order. For this sorting please add the following parameters to the search request:

{
  ...
  "sort_by": "date_modified",
  "sort_direction": "desc"
  ...
}