From 532f049b4f024323cca901121fc978415486845a Mon Sep 17 00:00:00 2001 From: Sunandadadi Date: Mon, 24 Jul 2023 16:42:08 -0400 Subject: [PATCH 1/4] Quay API v2 --- enhancements/api-v2.md | 121 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 enhancements/api-v2.md diff --git a/enhancements/api-v2.md b/enhancements/api-v2.md new file mode 100644 index 0000000..26145c5 --- /dev/null +++ b/enhancements/api-v2.md @@ -0,0 +1,121 @@ +--- +title: Quay API V2 design +authors: + - "@sdadi" +reviewers: + - "@syahmed" + - "@sleesinc" + - "@obulatov" + - "@bcaton" +approvers: + - "@syahmed" + - "@sleesinc" + - "@obulatov" + - "@bcaton" +creation-date: 2023-07-24 +last-updated: yyyy-mm-dd +status: implementable + +# Quay API v2 + +This is a proposal to design a generic strategy to improve Quay API's to support pagination, filtering and sorting. + +## Release Signoff Checklist + +- [x] Enhancement is `implementable` +- [x] Design details are appropriately documented from clear requirements +- [ ] Test plan is defined +- [ ] Graduation criteria for dev preview, tech preview, GA + +## Summary + +Currently, only a few APIs have the option to paginate results on the backend. The goal of the proposal is to have a re-usable system to perform actions on the backend and send the curated result on the frontend. Ideally frontend should only display the results and allow the backend to perform +all the heavy lifting. + +## Motivation + +In the quay ui, filtering, sorting and occasionally paginating the results happen on the frontend. +This leads to high rendering time on the frontend and thereby a poor user experience. + +### Goals + +Implement a generic system that is consistent across different endpoints to support the following operations on Quay endpoints: +- pagination +- filtering +- sorting + +## Design Details + +### API Design + +- Proposed path of the endpoints: In an attempt to not change the existing endpoints, new APIs will be under `/api/v2_draft/`. + `v2_draft` will be replaced with `v2` when ready to release. +- Existing decorators like the scope of API, page parameters will be reused where ever required. + +### Pagination + +It is difficult to say that one pagination strategy suits all needs. After understanding the pros and cons of various pagination strategies, +cusor-based pagination seems to work best for our current needs. + +The following will be expected request parameters for an API that is to be paginated: +1. limit: number of items to be returned for the page, max limit set to 100 items per page (based from current Quay API guidelines). +2. prev_cursor/next_cursor + + a. prev_cursor: encrypted string fetch previous page results + + b. next_cursor: encrypted string to fetch next page results + +Response body: +``` +{ + results: [items per page], + total_count: count of all items on all pages, + page_info: { + next_cursor: encrypted metadata for next page + prev_cursor: encrypted metadata for prev page + } +} +``` + +Cursor metadata should return a unique sequential column to base cursors on along with additional data when required. +(Eg: sorting results on non-unique keys, like datetime) + +### Filtering + +Typical syntax of an API filter will look like: `key[operation]=` where: +- key: is the field on which the search will be performed (Eg: repo name, org name, etc) +- operation: can be different based on the datatype of the key. (Eg: for integers, `lt|lte|gt|gte`, for strings: `like|eq|regex`) +- value: is what the query will filter for + +Eg: (GET `/tags?tag_name[like]=quay`, GET `/logs?created_at[lte]=2023-07-20 19:56`) + +### Sorting + +Sort order will be represented by the query parameter `sort`, ascending order by `+` and descending order by `-`. +Multiple sorts can be passed in the url by adding comma separated keys. +Eg: (GET `/tags?sort=+created_at`, GET `/users?sort=+creation_date,-last_accessed`) + + +### Notes/Constraints + +The major caveat of cursor-based pagination is that we cannot fetch results from page number. User cannot directly navigate to the nth page. +They'd need to request pages from 1 - n as next page results are derived from the previous page's cursor. + +### Graduation Criteria + +##### Dev Preview -> Tech Preview + +- Ability to utilize the enhancement end to end +- Sufficient test coverage +- End user documentation, relative API stability + +##### Tech Preview -> GA + +- More testing (on scale) +- Sufficient time for feedback +- Available by default + +## Open questions + +> 1. Do we want to set expiration time on cursors. If yes, how to choose this time? + From 163ecb44feadf5401a54296d1da97d94221a0806 Mon Sep 17 00:00:00 2001 From: Sunandadadi Date: Wed, 26 Jul 2023 11:18:41 -0400 Subject: [PATCH 2/4] added formating for multi-filtering + minor fixes --- enhancements/api-v2.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/enhancements/api-v2.md b/enhancements/api-v2.md index 26145c5..7cdaa8b 100644 --- a/enhancements/api-v2.md +++ b/enhancements/api-v2.md @@ -57,11 +57,14 @@ Implement a generic system that is consistent across different endpoints to supp It is difficult to say that one pagination strategy suits all needs. After understanding the pros and cons of various pagination strategies, cusor-based pagination seems to work best for our current needs. +Cursor will contain a unique sequential db column to base cursors on along with additional data when required. +(Eg: sorting results on non-unique keys, like datetime) + The following will be expected request parameters for an API that is to be paginated: 1. limit: number of items to be returned for the page, max limit set to 100 items per page (based from current Quay API guidelines). 2. prev_cursor/next_cursor - a. prev_cursor: encrypted string fetch previous page results + a. prev_cursor: encrypted string to fetch previous page results b. next_cursor: encrypted string to fetch next page results @@ -77,15 +80,13 @@ Response body: } ``` -Cursor metadata should return a unique sequential column to base cursors on along with additional data when required. -(Eg: sorting results on non-unique keys, like datetime) - ### Filtering -Typical syntax of an API filter will look like: `key[operation]=` where: +Typical syntax of an API filter will look like: `key1[operation1]=&key2[operation2]=` where: - key: is the field on which the search will be performed (Eg: repo name, org name, etc) - operation: can be different based on the datatype of the key. (Eg: for integers, `lt|lte|gt|gte`, for strings: `like|eq|regex`) - value: is what the query will filter for +- multiple supported filters can be added to the query parameters using `&` Eg: (GET `/tags?tag_name[like]=quay`, GET `/logs?created_at[lte]=2023-07-20 19:56`) @@ -117,5 +118,5 @@ They'd need to request pages from 1 - n as next page results are derived from th ## Open questions -> 1. Do we want to set expiration time on cursors. If yes, how to choose this time? +> 1. Do we want to set expiration time on cursors? If yes, how to choose this time? From 33a258aa0b79b81a78fd45d4e323632093be5b3a Mon Sep 17 00:00:00 2001 From: Sunandadadi Date: Tue, 15 Aug 2023 22:45:43 -0400 Subject: [PATCH 3/4] changes based on review --- enhancements/api-v2.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/enhancements/api-v2.md b/enhancements/api-v2.md index 7cdaa8b..6d38a62 100644 --- a/enhancements/api-v2.md +++ b/enhancements/api-v2.md @@ -1,5 +1,6 @@ --- title: Quay API V2 design +--- authors: - "@sdadi" reviewers: @@ -58,7 +59,14 @@ It is difficult to say that one pagination strategy suits all needs. After under cusor-based pagination seems to work best for our current needs. Cursor will contain a unique sequential db column to base cursors on along with additional data when required. -(Eg: sorting results on non-unique keys, like datetime) +For example, sorting results on non-unique keys, like datetime (currently used in the UI for sorting usage logs) + +Cursor data is extensible and can look like: +``` +{"id": 123} +{"id": "2023-08-02T12:34:56Z", "offset": 20} +{"id": "2023-08-02T12:34:56Z", "some_other_key": val1, "another_key": val2} +``` The following will be expected request parameters for an API that is to be paginated: 1. limit: number of items to be returned for the page, max limit set to 100 items per page (based from current Quay API guidelines). @@ -96,6 +104,16 @@ Sort order will be represented by the query parameter `sort`, ascending order by Multiple sorts can be passed in the url by adding comma separated keys. Eg: (GET `/tags?sort=+created_at`, GET `/users?sort=+creation_date,-last_accessed`) +Sorting will be supported only on select keys. Every API endpoint will maintain a list of supported sortable keys. If a user +makes a request with an unsupported key, 400 Bad Request response will be sent along with a list of supported sortable keys. +Example of 400 response: +``` +{ + "title": "Unsupported sort key", + "detail": "The request is sortable only on one of these keys: [a, b, c]", + "status": 400, +} +``` ### Notes/Constraints From ee16c7e61b8fbfaa09c7231b740951ff6e31dbb6 Mon Sep 17 00:00:00 2001 From: Sunandadadi Date: Wed, 16 Aug 2023 09:53:28 -0400 Subject: [PATCH 4/4] fixing header --- enhancements/api-v2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enhancements/api-v2.md b/enhancements/api-v2.md index 6d38a62..184ac17 100644 --- a/enhancements/api-v2.md +++ b/enhancements/api-v2.md @@ -1,6 +1,5 @@ --- title: Quay API V2 design ---- authors: - "@sdadi" reviewers: @@ -16,6 +15,7 @@ approvers: creation-date: 2023-07-24 last-updated: yyyy-mm-dd status: implementable +--- # Quay API v2