How to Use Relative Pagination In Your Application

Loading all the records for a particular model from Shopify is one of the most time consuming operations an application can perform, whether they are orders, products, variants, or something else. This is especially true for large shops that have millions of records.

To help address this problem, we’ve scrapped thepageparameter and added support forrelative cursor-based paginationvia link header for some select endpoints in the2019-07 REST Admin API version. In this article, we look at what this change means for you as a Shopify app developer, explain how to employ two different forms of relative cursor pagination in your application, and go over how to usethe new bulk operations APIto fetch all records.

Relative cursors vs thepageparameter

Before we dive into relative pagination, let’s look at what came before. The classic way to load a shop’s records from Shopify is by using apageparameter, where starting from page one, each subsequent request has apageparameter one higher than the previous. The problem with this approach is that the larger thepagenumber, the larger the offset in the query. The larger the offset, the slower the request is, up until the point that it times out.

There is an alternative the provides better performance no matter how far into the records you go: relative cursors. This basically means remembering the point where the previous page ended and the next page begins. Think of it like using a bookmark: instead of flipping through pages in a book to find the place you last stopped reading, you can jump right to the correct location. More information about the implementation and performance differences between the two can be found on theShopify Engineering Blog.

Introducing relative cursor-based pagination to Shopify

In the2019-07 API version, we removed thepageparameter andadded support for relative cursor-based paginationvia link header on select endpoints. The remaining endpoints will be updated in the2019-10 REST Admin API version. Older API versions that supportpagewill continue to work for nine months from the day of the version release that replaces them. The endpoints updated in the 2019-07 version will continue to work withpageby using an older API version until April 2020, and those updated in the 2019-10 version will continue to work withpagein an older API version until July 2020.

You might also like:Introducing API Versioning at Shopify.

Types of relative pagination

Inside of Shopify, we support two types of relative pagination: using asince_idand following URLs from theLinkheader in the response. In the following sections we’ll give examples of how to use each one and explain the tradeoffs of both.

Pagination withsince_id

This is the simplest form of relative cursor pagination. Each request will include asince_idparameter with the id of the last record from the previous page. This does require the records to be sorted by id ascending, but this happens automatically when asince_idparameter is present. To ensure the first page is sorted by id, you can include asince_idparameter with a value of 0. This form of pagination already existed on all endpoints prior to the 2019-07 API version.

For example, if you’re requesting pages of products from your shop, the request for the first page might look like the following:

https://shop-domain.myshopify.com/admin/products.json?limit=5&fields=id,title&since_id=0

This will come back with the following response:

Taking the id of the last record, the request to get the next page would be:

https://shop-domain.myshopify.com/admin/products.json?limit=5&fields=id,title&since_id=45678

Using asince_idis the fastest solution and should be the first choice for pagination when order doesn’t matter.

Pagination with a link header

What about when order does matter? Maybe you want to load the first 1,000 products with the most inventory, but with the 250 upper bound on limit, this would mean making at least four requests. You also wouldn’t want to usesince_id, because you’d have to fetchallof the products before determining the 1,000 with the highest inventory. Depending on the size of the store, this could mean loading a lot more records from Shopify. Furthermore, the inventory total is not exposed as a single field on products, so it would also mean adding up the inventory quantity of the individual variants and sorting on that.

There is an easier way: using theLinkheader. In the newer API versions where thepageparameter is removed, every request that has multiple pages will include a response header calledLink. This header contains the URLs for the previous and next pages, if they exist, respecting the sort order asked for. All of the data needed to load the next page is embedded in the URL inside of apage_infoparameter.

However, there is a downside: our tests showed that using the URLs from the link header is 10 to 30 percent slower than usingsince_id. But still, they are both faster than using apageparameter, particularly when there are thousands of records or more.

Using the above example of loading products sorted by total inventory quantity descending, the first request would look like:

https://shop-domain.myshopify.com/admin/api/2019-07/products.json?order=inventory_total+desc&limit=250

The response will contain aLinkheader that would look like this if there are more pages:

< https://shop-domain.myshopify.io/admin/api/2019-07/products.json?limit=250&page_info=eyJsYXN0X2lkIjoxOTg1ODQxNzk0LCJsYXN0X3ZhbHVlIjoiQWNxdWlyZWQgQ2FudmFzIFBlbiIsImRpcmVjdGlvbiI6Im5leHQifQ%3D%3D>; rel="next"

And the URL for the next page would look like this:

https://shop-domain.myshopify.io/admin/api/2019-07/products.json?limit=250&page_info=eyJsYXN0X2lkIjoxOTg1ODQxNzk0LCJsYXN0X3ZhbHVlIjoiQWNxdWlyZWQgQ2FudmFzIFBlbiIsImRpcmVjdGlvbiI6Im5leHQifQ%3D%3D

This URL can be parsed out of the header and, after following it, the subsequent request will have another header in the response that could look like the following:

; rel="previous", ; rel="next

Since this request wasn’t for the first page, this time there is a next and a previous page URL in the header. Parsing out the next page URL and following it can be repeated until the desired number of records have been retrieved, or until there is no next page URL in the header, at which point all the matching records have been retrieved.

Note that after the first request, thelimitparameter is the only parameter in the URL besidespage_info. This is because it is safe to modify thelimitin between requests. Parameters that could break the request if they were to change, such as sort order and filters, are embedded directly inside ofpage_info. Trying to add or modify filters or order parameters on URLs from the link header will cause the request to fail. If a different sort order or filtering is needed, then you must restart on the first page. It’s also important to note that the link header URLs are temporary and should not be saved for later use.

You might also like:The Shopify GraphQL Learning Kit.

Migrating your app to use relative pagination

Based on our API release schedule, support will be removed in April 2020 for the 2019-04 API version, which is the last version that will support thepageparameter on those endpoints updated in the 2019-07 version. Similarly, support will be removed in July 2020 for the 2019-07 API version, at which point no endpoints will support thepageparameter. It is important to have all applications migrated over to using relative pagination by then, or they will stop working past the first page.

To make it easier to update, theShopify API gemnow supports relative pagination. Using it to get the first 1,000 products sorted by inventory total descending is fairly simple:

For applications not written in Ruby, the change that added support to the Shopify API gem can be used as a blueprint. It’s availableon GitHub.

还有一个大好处this switch: speed! Both methods of relative pagination are faster than using apageparameter, particularly with large page numbers. In our experiments on a large test shop, loading the 100,000th product with a relative cursor was hundreds of times faster than using apageparameter.

One common case for usingpageright now is to make concurrent requests on a series of page numbers all at once. With relative cursors this will no longer be possible, as it will require getting one page to know where the next page starts. Relative cursor-based pagination is faster for getting all the records on large shops, and this increase in speed will make up for the lack of concurrent requests on these shops.

Using bulk operations in the Admin GraphQL API

AtShopify Unite 2019, we introduced批量操作in GraphQL, which allow apps to fetch large amounts of data in a completely new way. Instead of making a series of requests for successive pages of records, the app makes a single request to kick off an operation which will run asynchronously. The app periodically polls the status of the operation and, once it’s complete, will get a link to download a file containing the full set of results.

批量操作使用相对游标nternally, and so has the same performance benefits as using relative cursor-based pagination. It is a good choice to replace concurrent requests using apageparameter.

Bulk operations are now available for our Admin GraphQL API as of 2019-10. Examples of how they can be used can be foundin our docs.

Migrate your app for better functionality

All apps will eventually have to move away from pagination using thepageparameter, but between the increased speed of using relative cursors and the simplicity of using the bulk operations API, these changes will help your app have a faster overall performance, particularly when dealing with shops with large amounts of data. If you have questions about pagination or the new bulk operations API, head overto our forums.

Did you miss the announcements from Shopify Unite?Check outour new APIsorwatch the track sessions on YouTube.