Merging to release-5.7: [TT-13761] add batch request to the latest open api specs (#6797) #6807

buger · 2024-12-22T08:29:59Z

User description

TT-13761 add batch request to the latest open api specs (#6797)

Summary	add batch request to the latest open api specs
Type	Story
Status	In Code Review
Points	N/A
Labels	-

When the new Gateway Open Api spec was created. The Batch request
endpoint was left out .This should be added to the gateway OAS.

This pr also fixes an issue where the external OAS Url we were using now
return error 404. This pr changes that to use a local copy of the
external oas . I.e we have changed from :
https://raw.githubusercontent.com/OAI/OpenAPI-Specification/main/schemas/v3.0/schema.json
to
https://raw.githubusercontent.com/TykTechnologies/tyk/refs/heads/master/apidef/oas/schema/3.0.json

Link: https://tyk.io/docs/5.5/tyk-gateway-api/

TT-13761

Co-authored-by: itachi sasuke [email protected]

PR Type

Enhancement, Bug fix

Description

Added support for batch requests in the OpenAPI specification. This includes a new endpoint /{listen_path}/tyk/batch and detailed documentation on its usage, request structure, and response format.
Introduced new schemas (BatchRequestStructure, BatchReplyUnit, and RequestDefinition) to support batch request functionality.
Updated references to the OpenAPI schema URL to use a local copy instead of an external URL, addressing a 404 error issue.

Changes walkthrough 📝

Relevant files

Bug fix

.redocly.lint-ignore.yaml `Update OpenAPI schema URL to local reference` .redocly.lint-ignore.yaml Updated the reference to the OpenAPI schema URL to use a local copy instead of an external URL.	+1/-1

Enhancement

swagger.yml `Add batch request support and update OpenAPI schema references` swagger.yml Added a new `Batch requests` section with detailed descriptions and examples. Introduced a new endpoint `/{listen_path}/tyk/batch` for batch requests. Defined new schemas for `BatchRequestStructure`, `BatchReplyUnit`, and `RequestDefinition`. Updated references to the OpenAPI schema URL to use a local copy.	+295/-6

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

<details open> <summary><a href="https://tyktech.atlassian.net/browse/TT-13761" title="TT-13761" target="_blank">TT-13761</a></summary> <br /> <table> <tr> <th>Summary</th> <td>add batch request to the latest open api specs</td> </tr> <tr> <th>Type</th> <td> <img alt="Story" src="https://tyktech.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10315?size=medium" /> Story </td> </tr> <tr> <th>Status</th> <td>In Code Review</td> </tr> <tr> <th>Points</th> <td>N/A</td> </tr> <tr> <th>Labels</th> <td>-</td> </tr> </table> </details>  --- When the new Gateway Open Api spec was created. The Batch request endpoint was left out .This should be added to the gateway OAS. This pr also fixes an issue where the external OAS Url we were using now return error 404. This pr changes that to use a local copy of the external oas . I.e we have changed from : https://raw.githubusercontent.com/OAI/OpenAPI-Specification/main/schemas/v3.0/schema.json to https://raw.githubusercontent.com/TykTechnologies/tyk/refs/heads/master/apidef/oas/schema/3.0.json Link: https://tyk.io/docs/5.5/tyk-gateway-api/ [TT-13761] [TT-13761]: https://tyktech.atlassian.net/browse/TT-13761?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: itachi sasuke <[email protected]> (cherry picked from commit 0276c06)

github-actions · 2024-12-22T08:30:27Z

Swagger Changes






  
  
  
  
  
  
  
  
  
  
  
  
    
    
    
    
    
    
    
    
    
    
     _        __  __
    /{listen_path}/tyk/batch:
    BatchReplyUnit:
    BatchRequestStructure:
    RequestDefinition:
   _| |_   _ / _|/ _|  between swagger-prev.yml
  + one list entry added:
  + one list entry added:
  + one list entry added:
  + one list entry added:
  + one list entry added:
  + one map entry added:
  + three map entries added:
  - one list entry removed:
  - one list entry removed:
  - one list entry removed:
  - one list entry removed:
  ± value change
  ± value change
 / _' | | | | |_| |_       and swagger-current.yml
 \__,_|\__, |_| |_|   returned nine differences
components.schemas
paths
paths./tyk/apis/oas.get.responses.200.content.application/json.schema.items.allOf
paths./tyk/apis/oas.post.requestBody.content.application/json.schema.allOf
paths./tyk/apis/oas/import.post.requestBody.content.application/json.schema.$ref
paths./tyk/apis/oas/{apiID}.get.responses.200.content.application/json.schema.allOf
paths./tyk/apis/oas/{apiID}.patch.requestBody.content.application/json.schema.$ref
paths./tyk/apis/oas/{apiID}.put.requestBody.content.application/json.schema.allOf
tags
| (_| | |_| |  _|  _|

github-actions · 2024-12-22T08:30:47Z

API Changes

no api changes detected

github-actions · 2024-12-22T08:31:01Z

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis ✅ 6797 - Fully compliant Fully compliant requirements: Add the batch request endpoint to the latest Gateway Open API Specification (OAS). Replace the external OAS URL with a local copy due to the external URL returning a 404 error. Not compliant requirements: None
⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review Documentation Clarity Ensure that the added documentation for the batch request endpoint is clear, accurate, and provides sufficient examples for users to understand its usage. - description: \| Tyk supports batch requests, so a client makes a single request to the API but gets a compound response object back. This is especially handy if clients have complex requests that have multiple synchronous dependencies and do not wish to have the entire request / response cycle running for each event. To enable batch request support, set the `enable_batch_request_support` value to `true` Batch requests that come into Tyk are run through the whole Tyk machinery and use a relative path to prevent spamming. This means that a batch request to Tyk for three resources with the same API key will have three requests applied to their session quota and request limiting could become active if they are being throttled. Tyk reconstructs the API request based on the data in the batch request. This is to ensure that Tyk is not being used to proxy requests to other hosts outside of the upstream API being accessed. Batch requests are created by POSTING to the `/{listen_path}/tyk/batch/` endpoint. These requests do not require a valid key, but their request list does. <h3>Sample Request</h3> ```{json} { "requests": [ { "method": "GET", "headers": { "x-tyk-test": "1", "x-tyk-version": "1.2", "authorization": "1dbc83b9c431649d7698faa9797e2900f" }, "body": "", "relative_url": "get" }, { "method": "GET", "headers": { "x-tyk-test": "2", "x-tyk-version": "1.2", "authorization": "1dbc83b9c431649d7698faa9797e2900f" }, "body": "", "relative_url": "get" } ], "suppress_parallel_execution": false } ``` The response will be a structured reply that encapsulates the responses for each of the outbound requests. If `suppress_parallel_execution` is set to `true`, requests will be made synchronously. If set to `false` then they will run in parallel and the response order is not guaranteed. <h3>Sample Response</h3> ``` [ { "relative_url": "get", "code": 200, "headers": { "Access-Control-Allow-Credentials": [ "true" ], "Access-Control-Allow-Origin": [ "" ], "Content-Length": [ "497" ], "Content-Type": [ "application/json" ], "Date": [ "Wed, 12 Nov 2014 15:32:43 GMT" ], "Server": [ "gunicorn/18.0" ], "Via": [ "1.1 vegur" ] }, "body": "{ "args": {}, "headers": { "Accept-Encoding": "gzip", "Authorization": "1dbc83b9c431649d7698faa9797e2900f", "Connect-Time": "2", "Connection": "close", "Host": "httpbin.org", "Total-Route-Time": "0", "User-Agent": "Go 1.1 package http", "Via": "1.1 vegur", "X-Request-Id": "6a22499a-2776-4aa1-80c0-686581a8be4d", "X-Tyk-Test": "2", "X-Tyk-Version": "1.2" }, "origin": "127.0.0.1, 62.232.114.250", "url": "http://httpbin.org/get" }" }, { "relative_url": "get", "code": 200, "headers": { "Access-Control-Allow-Credentials": [ "true" ], "Access-Control-Allow-Origin": [ "" ], "Content-Length": [ "497" ], "Content-Type": [ "application/json" ], "Date": [ "Wed, 12 Nov 2014 15:32:43 GMT" ], "Server": [ "gunicorn/18.0" ], "Via": [ "1.1 vegur" ] }, "body": "{ "args": {}, "headers": { "Accept-Encoding": "gzip", "Authorization": "1dbc83b9c431649d7698faa9797e2900f", "Connect-Time": "7", "Connection": "close", "Host": "httpbin.org", "Total-Route-Time": "0", "User-Agent": "Go 1.1 package http", "Via": "1.1 vegur", "X-Request-Id": "1ab61f50-51ff-4828-a7e2-17240385a6d2", "X-Tyk-Test": "1", "X-Tyk-Version": "1.2" }, "origin": "127.0.0.1, 62.232.114.250", "url": "http://httpbin.org/get" }" } ] ``` With the body for each request string encoded in the `body` field. * `expire_analytics_after`: If you are running a busy API, you may want to ensure that your MongoDB database does not overflow with old data. Set the `expire_analytics_after` value to the number of seconds you would like the data to last for. Setting this flag to anything above `0` will set an `expireAt` field for each record that is written to the database. Important: Tyk will not create the expiry index for you. In order to implement data expiry for your analytics data, ensure that the index is created This is easily achieved using the [MongoDB command line interface](https://docs.mongodb.com/getting-started/shell/client/). * `dont_set_quota_on_create`: This setting defaults to `false`, but if set to `true`, when the API is used to edit, create or add keys, the quota cache in Redis will not be re-set. By default, all updates or creates to Keys that have Quotas set will re-set the quota (This has been the default behaviour since 1.0). This behaviour can be bypassed on a case-by-case basis by using the `suppress_reset` parameter when making a REST API request. This is the advised mode of operation as it allows for manual, granular control over key quotas and reset timings. * `cache_options`: This section enables you to configure the caching behaviour of Tyk and to enable or disable the caching middleware for your API. * `cache_options.enable_cache`: Set this value to `true` if the cache should be enabled for this endpoint, setting it to false will stop all caching behaviour. * `cache_options.cache_timeout`: The amount of time, in seconds, to keep cached objects, defaults to `60` seconds. * `cache_options.cache_all_safe_requests`: Set this to `true` if you want all safe requests (GET, HEAD, OPTIONS) to be cached. This is a blanket setting for APIs where caching is required but you don't want to set individual paths up in the definition. * `cache_options.enable_upstream_cache_control`: Set this to `true` if you want your application to control the cache options for Tyk (TTL and whether to cache or not). See [Caching](/docs/basic-config-and-security/reduce-latency/caching/) for more details. * `response_processors`: Response processors need to be specifically defined so they are loaded on API creation, otherwise the middleware will not fire. In order to have the two main response middleware components fire, the following configuration object should be supplied. ```{json} "response_processors": [ { "name": "header_injector", "options": { "add_headers": {"name": "value"}, "remove_headers": ["name"] } }, { "name": "response_body_transform", "options": {} } ] ``` The options for the `header_injector` are global, and will apply to all outbound requests. name: "Batch requests" Schema Reference Update Verify that all instances of the external OAS URL have been correctly replaced with the local copy and that the new references are functional. examples: oasExampleList: $ref: '#/components/examples/oasExampleList' schema: items: allOf: - $ref: https://raw.githubusercontent.com/TykTechnologies/tyk/refs/heads/master/apidef/oas/schema/3.0.json - $ref: '#/components/schemas/XTykAPIGateway' type: array description: List of API definitions in Tyk OAS format. "403": content: application/json: example: message: Attempted administrative access with invalid or missing key! status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: Forbidden summary: List all APIs in Tyk OAS API format. tags: - Tyk OAS APIs post: description: Create an API with Tyk OAS API format on the Tyk Gateway. operationId: createApiOAS parameters: - description: The base API which the new version will be linked to. example: 663a4ed9b6be920001b191ae in: query name: base_api_id required: false schema: type: string - description: The version name of the base API while creating the first version. This doesn't have to be sent for the next versions but if it is set, it will override base API version name. example: Default in: query name: base_api_version_name required: false schema: type: string - description: The version name of the created version. example: v2 in: query name: new_version_name required: false schema: type: string - description: If true, the new version is set as default version. example: true in: query name: set_default required: false schema: type: boolean requestBody: content: application/json: example: components: securitySchemes: bearerAuth: description: The API Access Credentials scheme: bearer type: http info: description: This is a sample OAS. title: OAS Sample version: 1.0.0 openapi: 3.0.3 paths: /api/sample/users: get: operationId: getUsers responses: "200": content: application/json: schema: items: properties: name: type: string type: object type: array description: fetched users summary: Get users tags: - users security: - bearerAuth: [] servers: - url: https://localhost:8080 x-tyk-api-gateway: info: name: user state: active: true server: listenPath: strip: true value: /user-test/ upstream: url: https://localhost:8080 schema: allOf: - $ref: https://raw.githubusercontent.com/TykTechnologies/tyk/refs/heads/master/apidef/oas/schema/3.0.json - $ref: '#/components/schemas/XTykAPIGateway' responses: "200": content: application/json: example: action: added key: e30bee13ad4248c3b529a4c58bb7be4e status: ok schema: $ref: '#/components/schemas/ApiModifyKeySuccess' description: API created. "400": content: application/json: example: message: the payload should contain x-tyk-api-gateway status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: Bad Request "403": content: application/json: example: message: Attempted administrative access with invalid or missing key! status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: Forbidden "500": content: application/json: example: message: file object creation failed, write error status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: Internal server error. summary: Create an API with Tyk OAS format. tags: - Tyk OAS APIs /tyk/apis/oas/{apiID}: delete: description: Deleting an API definition will remove the file from the file store, the API definition will not be unloaded, a separate reload request will need to be made to disable the API endpoint. operationId: deleteOASApi parameters: - description: The API ID. example: 1bd5c61b0e694082902cf15ddcc9e6a7 in: path name: apiID required: true schema: type: string responses: "200": content: application/json: example: action: deleted key: 1bd5c61b0e694082902cf15ddcc9e6a7 status: ok schema: $ref: '#/components/schemas/ApiModifyKeySuccess' description: API deleted "400": content: application/json: example: message: Must specify an apiID to delete status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: Bad Request "403": content: application/json: example: message: Attempted administrative access with invalid or missing key! status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: Forbidden "404": content: application/json: example: message: API not found status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: API not found. "500": content: application/json: example: message: Delete failed status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: Internal server error. summary: Deleting a Tyk OAS API. tags: - Tyk OAS APIs get: description: Get Tyk OAS API definition using an API ID. operationId: getOASApi parameters: - description: "By default mode is empty which means it will return the Tyk API OAS spec including the x-tyk-api-gateway part. \n When mode=public, the Tyk OAS API spec will exclude the x-tyk-api-gateway part in the response." example: public in: query name: mode required: false schema: enum: - public type: string - description: ID of the API you want to fetch example: 4c1c0d8fc885401053ddac4e39ef676b in: path name: apiID required: true schema: type: string responses: "200": content: application/json: examples: oasExample: $ref: '#/components/examples/oasExample' schema: allOf: - $ref: https://raw.githubusercontent.com/TykTechnologies/tyk/refs/heads/master/apidef/oas/schema/3.0.json - $ref: '#/components/schemas/XTykAPIGateway' description: OK headers: x-tyk-base-api-id: description: ID of the base API if the requested API is a version. schema: type: string style: simple "400": content: application/json: example: message: the requested API definition is in Tyk classic format, please use old API endpoint status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: Bad Request "403": content: application/json: example: message: Attempted administrative access with invalid or missing key! status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: Forbidden "404": content: application/json: example: message: API not found status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: API not found. summary: Get a Tyk OAS API definition. tags: - Tyk OAS APIs patch: description: \|- You can use this endpoint to update Tyk OAS part of the Tyk API definition. This endpoint allows you to configure Tyk OAS extension based on query params provided(similar to import). operationId: patchApiOAS parameters: - description: ID of the API you want to fetch. example: 4c1c0d8fc885401053ddac4e39ef676b in: path name: apiID required: true schema: type: string - $ref: '#/components/parameters/UpstreamURL' - $ref: '#/components/parameters/ListenPath' - $ref: '#/components/parameters/CustomDomain' - $ref: '#/components/parameters/AllowList' - $ref: '#/components/parameters/ValidateRequest' - $ref: '#/components/parameters/MockResponse' - $ref: '#/components/parameters/Authentication' requestBody: content: application/json: example: components: securitySchemes: bearerAuth: description: The API Access Credentials scheme: bearer type: http info: description: This is a sample OAS. title: OAS Sample version: 1.0.0 openapi: 3.0.3 paths: /api/sample/users: get: operationId: getUsers responses: "200": content: application/json: schema: items: properties: name: type: string type: object type: array description: fetched users summary: Get users tags: - users security: - bearerAuth: [] servers: - url: https://localhost:8080 x-tyk-api-gateway: info: name: user state: active: true server: listenPath: strip: true value: /user-test/ upstream: url: https://localhost:8080 schema: $ref: https://raw.githubusercontent.com/TykTechnologies/tyk/refs/heads/master/apidef/oas/schema/3.0.json responses: "200": content: application/json: schema: $ref: '#/components/schemas/ApiModifyKeySuccess' description: API patched. "400": content: application/json: example: message: Must specify an apiID to patch status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: Bad Request "403": content: application/json: example: message: Attempted administrative access with invalid or missing key! status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: Forbidden "404": content: application/json: example: message: API not found status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: API not found. "500": content: application/json: example: message: file object creation failed, write error status: error schema: $ref: '#/components/schemas/ApiStatusMessage' description: Internal server error. summary: Patch API in Tyk OAS format. tags: - Tyk OAS APIs put: description: \|- Updating an API definition uses the same signature an object as a `POST`, however it will first ensure that the API ID that is being updated is the same as the one in the object being `PUT`. Updating will completely replace the file descriptor and will not change an API Definition that has already been loaded, the hot-reload endpoint will need to be called to push the new definition to live. operationId: updateApiOAS parameters: - description: ID of the API you want to fetch example: 4c1c0d8fc885401053ddac4e39ef676b in: path name: apiID required: true schema: type: string requestBody: content: application/json: example: components: securitySchemes: bearerAuth: description: The API Access Credentials scheme: bearer type: http info: description: This is a sample OAS. title: OAS Sample version: 1.0.0 openapi: 3.0.3 paths: /api/sample/users: get: operationId: getUsers responses: "200": content: application/json: schema: items: properties: name: type: string type: object type: array description: fetched users summary: Get users tags: - users security: - bearerAuth: [] servers: - url: https://localhost:8080 x-tyk-api-gateway: info: name: user state: active: true server: listenPath: strip: true value: /user-test/ upstream: url: https://localhost:8080 schema: allOf: - $ref: https://raw.githubusercontent.com/TykTechnologies/tyk/refs/heads/master/apidef/oas/schema/3.0.json - $ref: '#/components/schemas/XTykAPIGateway'

github-actions · 2024-12-22T08:31:12Z

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Score
Security	Limit the number of batch requests to prevent resource exhaustion Add a maximum limit to the number of `requests` in the `BatchRequestStructure` schema to prevent excessive resource consumption. swagger.yml [5554-5559] requests: items: $ref: '#/components/schemas/RequestDefinition' nullable: true type: array + maxItems: 50 Suggestion importance[1-10]: 10 Why: Setting a maximum limit on the number of batch requests is a crucial security measure to prevent excessive resource consumption, which could lead to denial-of-service attacks or performance degradation.	10
	Add validation to the `relative_url` field to restrict its format and prevent misuse Ensure that the `relative_url` field in the `BatchRequestStructure` schema is validated to prevent potential misuse or abuse by restricting it to a specific format or domain. swagger.yml [5573-5574] relative_url: type: string + pattern: "^/valid-path-prefix/.*$" Suggestion importance[1-10]: 9 Why: Adding a pattern validation to the `relative_url` field enhances security by preventing misuse or abuse, such as accessing unintended paths. This is a critical improvement for ensuring the integrity of the API.	9
	Replace sensitive authorization tokens in examples with placeholders Ensure that the `authorization` header in the `BatchRequestStructure` example is replaced with a placeholder or anonymized value to avoid exposing sensitive information. swagger.yml [4434] -authorization: 1dbc83b9c431649d7698faa9797e2900f +authorization: <your-auth-token> Suggestion importance[1-10]: 8 Why: Replacing sensitive authorization tokens in examples with placeholders is a good practice to avoid accidental exposure of sensitive information, improving the security and professionalism of the documentation.	8
General	Restrict the `method` field to valid HTTP methods to ensure proper usage Add a description or validation to clarify the expected format of the `method` field in the `RequestDefinition` schema to ensure only valid HTTP methods are allowed. swagger.yml [5572-5573] method: type: string + enum: + - GET + - POST + - PUT + - DELETE + - PATCH Suggestion importance[1-10]: 7 Why: Restricting the `method` field to valid HTTP methods improves the schema's robustness and ensures that only appropriate values are used, reducing the risk of errors or misuse.	7

sonarqubecloud · 2024-12-22T08:47:38Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

buger added Story Tyk Product Development labels Dec 22, 2024

github-actions bot added the Review effort [1-5]: 4 label Dec 22, 2024

lghiur merged commit 04631ba into release-5.7 Dec 22, 2024
40 of 41 checks passed

lghiur deleted the merge/release-5.7/0276c06f5a1c28fbf23f61d15fea07509c3413b5 branch December 22, 2024 08:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merging to release-5.7: [TT-13761] add batch request to the latest open api specs (#6797) #6807

Merging to release-5.7: [TT-13761] add batch request to the latest open api specs (#6797) #6807

buger commented Dec 22, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Dec 22, 2024

github-actions bot commented Dec 22, 2024

github-actions bot commented Dec 22, 2024

github-actions bot commented Dec 22, 2024

sonarqubecloud bot commented Dec 22, 2024

Merging to release-5.7: [TT-13761] add batch request to the latest open api specs (#6797) #6807

Merging to release-5.7: [TT-13761] add batch request to the latest open api specs (#6797) #6807

Conversation

buger commented Dec 22, 2024 • edited by github-actions bot Loading

User description

PR Type

Description

Changes walkthrough 📝

github-actions bot commented Dec 22, 2024

github-actions bot commented Dec 22, 2024

github-actions bot commented Dec 22, 2024

PR Reviewer Guide 🔍

github-actions bot commented Dec 22, 2024

PR Code Suggestions ✨

sonarqubecloud bot commented Dec 22, 2024

Quality Gate failed

buger commented Dec 22, 2024 •

edited by github-actions bot

Loading