diff --git a/README.md b/README.md index 640134a..68f2317 100644 --- a/README.md +++ b/README.md @@ -72,6 +72,11 @@ Options: --tokenUrl The tokenUrl for openIdConnect -> oauth2 transformation -d, --delete-examples-with-id If set, delete any JSON Schema examples that have an `id` property --oidc-to-oath2 Convert openIdConnect security to oath2. + --convertJsonComments If used, convert `$comment` in JSON schemas + to `x-comment`. If omitted, delete + all `$comment` in JSON schemas. + (Use `--verbose` to log deletion + to stdout) -s, --scopes If set, this JSON/YAML file describes the OpenID scopes. This is an alias for --oidc-to-oath2 -v, --verbose Verbose output @@ -79,7 +84,13 @@ Options: -h, --help display help for command ``` -The verbose mode logs the changes to standard error output stream. +The verbose mode logs all changes to standard error output stream. + +The tool returns a 0 status code upon success or a non-zero status code +if it finds constructs that cannot be down-converted, such as +using `contentMediaType: application/octet-stream` with a `format` +other than `binary`, or if a schema has `contentEncoding: base64` +and has an existing `format` that is not already `base64`. The tool only supports local file-based documents, not URLs. Download such files to convert: @@ -170,8 +181,8 @@ mySchema: This also applies to the schema used in parameters or in `requestBody` objects and in responses. -**Note** This transformation is disabled by default because it breaks `openapi-generator` 5.4 in -cases where the referenced schema is an array. +**Note** This transformation is disabled by default because it breaks +`openapi-generator` 5.4 in cases where the referenced schema is an array. It generates Typescript types for such cases as ```typescript @@ -223,7 +234,7 @@ components: const: '1.0.0' ``` - becomes +becomes ```yaml components: @@ -235,7 +246,7 @@ components: - '1.0.0' ``` -### Convert type arrays to nullable +### ⤓ Convert type arrays to nullable If a schema has a type array of exactly two values, and one of them is the string `'null'`, the type is converted to the non-null string item, @@ -302,11 +313,11 @@ be possible (`properties`, `allOf` etc.) (Contributions welcome.) -### Remove `unevaluatedProperties` +### ⤓ Remove `unevaluatedProperties` The tool removes the `unevaluatedProperties` value, introduced in later versions of JSON Schema, -as this is not supported in OAS 3.0 JSON Schema Draft 4 +as this is not supported in JSON Schema Draft 4 used in OAS 3.0. ```yaml @@ -331,14 +342,27 @@ becomes ... ``` -### Rename `$comment` as `x-comment` +The tool removes any `$id` or `$schema` keywords that may appear +inside schema objects. + +### ⤓ Convert `$comment` to `x-comment` + +JSON Schema introduced `$comment` in schemas in 2020-12. +Since OAS 3.0 uses JSON Schema Draft 4, and some tools +will flag `$comment` as invalid, this tool removes these comments. -The tool renames the `$comment` keyword in schema objects as `x-comment` -as `$comment` is not supported in OAS 3.0 JSON Schema Draft 4 -used in OAS 3.0. and can cause problems with some tools. -`x-comment` is more easily ignored since it does not start with `$`. +An earlier version of the tool converted `$comment` to `x-comment` +However, other tools which do not allow `$comment` may not not support +`x-comment` either. + +Use the `--convert-schema-comments` CLI option or set +`convertSchemaComments` to `true` +in the `Converter` constructor options +to requst conversion of +`$comment` to `x-comment` rather than deleting `$comment`. + +For example, -For exmample, ```yaml Problems: title: Problems @@ -365,16 +389,70 @@ becomes $ref: '#/components/schemas/apiProblem' ``` -### Remove schema `$id` and `$schema` +### Convert `contentEncoding: base64` to `format: byte` + +JSON Schema Draft 7 and later uses `contentEncoding` to specify +[the encoding of non-JSON string content] +(https://json-schema.org/understanding-json-schema/reference/non_json_data). +Draft 4 supports `format: byte` for `Base64` encoded strings. + +This tool converts `type: string` schemas as follows: + + + + + + + + +2 + + + + + + + + + +
OAS 3.1 schemaOAS 3.0 schema
+
+type: string
+contentEncoding: base64
+
+		 +
+type: string
+format: byte
+
+
+
+type: string
+contentMediaType: 'application/octet-stream'
+
+		 +
+type: string
+format: binary
+
+
-The tool removes any `$id` or `$schema` keywords that may appear -inside schema objects. ## Unsupported down conversions +Currently, the tool does not support the following situations. +Contributions welcome! + * `openapi-down-convert` does not convert `exclusiveMinimum` and `exclusiveMaximum` -as defined in JSON Schema 2012-12; these handled differently in JSON Schema Draft 4 -used in OAS 3.0. Contributions welcome! -* Webhooks are not addressed. Contributions welcome! + as defined in JSON Schema 2012-12; these handled differently in JSON Schema Draft 4 + used in OAS 3.0. +* Webhooks are not addressed. * The tool only supports self-contained documents. It does not follow or resolve -external `$ref` documents embedded in the source document. + external `$ref` documents embedded in the source document. +* Request body and response body `content` object transformations, such as + reversing `content: { 'application/octet-stream': {} }` as + described in [Migrating from OpenAPI 3.0 to 3.1.0](https://www.openapis.org/blog/2021/02/16/migrating-from-openapi-3-0-to-3-1-0) +* Converting other `contentEncoding` values (`7bit`, `8bit`, `binary`, + `quoted-printable`, `base16`, `base32`) (Note: `contentEncoding: base64` is supported by + converting to `format: byte` as listed above.) +* Converting `contentMediaType: 'type/subtype` to `media: { type: 'type/subtype'}` for non-JSON data. diff --git a/package-lock.json b/package-lock.json index de0dd4d..0049b85 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@apiture/openapi-down-convert", - "version": "0.9.0", + "version": "0.10.0", "lockfileVersion": 2, "requires": true, "packages": { "": { "name": "@apiture/openapi-down-convert", - "version": "0.9.0", + "version": "0.10.0", "license": "ISC", "dependencies": { "commander": "^9.4.1", diff --git a/package.json b/package.json index 31cc22e..cde1fa7 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@apiture/openapi-down-convert", - "version": "0.9.0", + "version": "0.10.0", "description": "Tool to down convert OpenAPI 3.1 to OpenAPI 3.0", "main": "lib/src/index.js", "bin": { diff --git a/src/RefVisitor.ts b/src/RefVisitor.ts index be891a0..f600b5a 100644 --- a/src/RefVisitor.ts +++ b/src/RefVisitor.ts @@ -62,7 +62,8 @@ export function visitSchemaObjects(node: any, schemaCallback: SchemaVisitor): an if (schemas != null && typeof schemas === 'object') { for (const schemaName in schemas) { const schema = schemas[schemaName]; - schemas[schemaName] = schemaCallback(schema); + const newSchema = schemaCallback(schema); + schemas[schemaName] = newSchema; } } } diff --git a/src/cli.ts b/src/cli.ts index 68967f8..bf3c2fe 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -25,6 +25,7 @@ async function main(args: string[] = process.argv) { .option('--oidc-to-oauth2 ', 'Convert openIdConnect security to oauth2 to allow scope definition') .option('-s, --scopes ', 'Alias for --oidc-to-oauth2') .option('-v, --verbose', 'Verbose output') + .option('--convert-schema-comments', 'Convert $comment to x-comment instead of deleting $comment values') .parse(args); const opts = cli.opts(); const sourceFileName: string = opts.input || 'openapi.yaml'; @@ -37,6 +38,7 @@ async function main(args: string[] = process.argv) { authorizationUrl: opts.authorizationUrl, tokenUrl: opts.tokenUrl, scopeDescriptionFile: opts.scopes, + convertSchemaComments: opts.convertSchemaComments, }; const converter = new Converter(source, cOpts); try { diff --git a/src/converter.ts b/src/converter.ts index 94597f1..959d66d 100644 --- a/src/converter.ts +++ b/src/converter.ts @@ -45,6 +45,12 @@ export interface ConverterOptions { * `{ scope1: "description of scope1", ... }` */ scopeDescriptionFile?: string; + /** Earlier versions of the tool converted $comment to x-comment + * in JSON Schemas. The tool now deletes $comment values by default. + * Use this option to preserve the conversion and not delete + * comments. + */ + convertSchemaComments?: boolean; } export class Converter { @@ -56,6 +62,8 @@ export class Converter { /** The tokenUrl for openIdConnect -> oauth2 transformation */ private tokenUrl: string; private scopeDescriptions = undefined; + private convertSchemaComments = false; + private returnCode = 0; /** * Construct a new Converter @@ -69,6 +77,7 @@ export class Converter { this.authorizationUrl = options?.authorizationUrl || 'https://www.example.com/oauth2/authorize'; this.tokenUrl = options?.tokenUrl || 'https://www.example.com/oauth2/token'; this.loadScopeDescriptions(options?.scopeDescriptionFile); + this.convertSchemaComments = options?.convertSchemaComments; } /** Load the scopes.yaml file and save in this.scopeDescriptions @@ -103,6 +112,20 @@ export class Converter { console.warn(...message); } + /** + * Log an error message to `console.error` stream. Prefix the message string with `Error: ` + * if it does not already start with `'Error'`. Increments the `returnCode`, causing + * the CLI to throw an Error when done. + * @param message parameters for `console.error` + */ + private error(...message) { + if (!message[0].startsWith('Error')) { + message[0] = `Error: ${message[0]}`; + } + this.returnCode++; + console.error(...message); + } + /** * Convert the OpenAPI document to 3.0 * @returns the converted document. The input is not modified. @@ -117,11 +140,19 @@ export class Converter { this.convertSecuritySchemes(); } this.convertJsonSchemaExamples(); - this.convertJsonSchemaComments(); + this.convertJsonSchemaContentEncoding(); + this.convertJsonSchemaContentMediaType(); this.convertConstToEnum(); this.convertNullableTypeArray(); this.removeUnsupportedSchemaKeywords(); - this.renameSchema$comment(); + if (this.convertSchemaComments) { + this.renameSchema$comment(); + } else { + this.deleteSchema$comment(); + } + if (this.returnCode > 0) { + throw new Error('Cannot down convert this OpenAPI definition.'); + } return this.openapi30; } @@ -164,25 +195,6 @@ export class Converter { visitSchemaObjects(this.openapi30, schemaVisitor); } - /** - * OpenAPI 3.1 uses JSON Schema 2020-12 which allows schema `$comment`; - * OpenAPI 3.0 uses JSON Scheme Draft 7 does not allow it. - * Replace all `$comment` with `x-comment` - */ - convertJsonSchemaComments() { - const schemaVisitor: SchemaVisitor = - (schema: SchemaObject): SchemaObject => - { - if (schema.hasOwnProperty('$comment')) { - schema['x-comment'] = schema['$comment']; - delete schema['$comment']; - this.log(`schema $comment renamed to x-comment`); - } - return this.walkNestedSchemaObjects(schema, schemaVisitor); - }; - visitSchemaObjects(this.openapi30, schemaVisitor); - } - private walkNestedSchemaObjects(schema, schemaVisitor) { for (const key in schema) { const subSchema = schema[key]; @@ -246,9 +258,7 @@ export class Converter { } renameSchema$comment() { - const schemaVisitor: SchemaVisitor = - (schema: SchemaObject): SchemaObject => - { + const schemaVisitor: SchemaVisitor = (schema: SchemaObject): SchemaObject => { if (schema.hasOwnProperty('$comment')) { schema['x-comment'] = schema['$comment']; delete schema['$comment']; @@ -259,6 +269,96 @@ export class Converter { visitSchemaObjects(this.openapi30, schemaVisitor); } + private deleteSchema$comment() { + const schemaVisitor: SchemaVisitor = (schema: SchemaObject): SchemaObject => { + if (schema.hasOwnProperty('$comment')) { + const comment = schema['$comment']; + delete schema['$comment']; + this.log(`schema $comment deleted: ${comment}`); + } + return this.walkNestedSchemaObjects(schema, schemaVisitor); + }; + visitSchemaObjects(this.openapi30, schemaVisitor); + } + + /** + * Convert + * ``` + * contentMediaType: 'application/octet-stream' + * ``` + * to + * ``` + * format: binary + * ``` + * in `type: string` schemas. + * Warn if schema has a `format` already and it is not `binary`. + */ + convertJsonSchemaContentMediaType() { + const schemaVisitor: SchemaVisitor = (schema: SchemaObject): SchemaObject => { + if ( + schema.hasOwnProperty('type') && + schema['type'] === 'string' && + schema.hasOwnProperty('contentMediaType') && + schema['contentMediaType'] === 'application/octet-stream' + ) { + if (schema.hasOwnProperty('format')) { + if (schema['format'] === 'binary') { + this.log(`Deleted schema contentMediaType: application/octet-stream (leaving format: binary)`); + delete schema['contentMediaType']; + } else { + this.error( + `Unable to down-convert schema with contentMediaType: application/octet-stream to format: binary because the schema already has a format (${schema['format']})`, + ); + } + } else { + delete schema['contentMediaType']; + schema['format'] = 'binary'; + this.log(`Converted schema contentMediaType: application/octet-stream to format: binary`); + } + } + return this.walkNestedSchemaObjects(schema, schemaVisitor); + }; + visitSchemaObjects(this.openapi30, schemaVisitor); + } + + /** + * Convert + * ``` + * contentEncoding: base64 + * ``` + * to + * ``` + * format: byte + * ``` + * in `type: string` schemas. It is an error if the schema has a `format` already + * and it is not `byte`. + */ + convertJsonSchemaContentEncoding() { + const schemaVisitor: SchemaVisitor = (schema: SchemaObject): SchemaObject => { + if (schema.hasOwnProperty('type') && schema['type'] === 'string' && schema.hasOwnProperty('contentEncoding')) { + if (schema['contentEncoding'] === 'base64') { + if (schema.hasOwnProperty('format')) { + if (schema['format'] === 'byte') { + this.log(`Deleted schema contentEncoding: base64 (leaving format: byte)`); + delete schema['contentEncoding']; + } else { + this.error( + `Unable to down-convert schema contentEncoding: base64 to format: byte because the schema already has a format (${schema['format']})`, + ); + } + } else { + delete schema['contentEncoding']; + schema['format'] = 'byte'; + this.log(`Converted schema: 'contentEncoding: base64' to 'format: byte'`); + } + } else { + this.error(`Unable to down-convert contentEncoding: ${schema['contentEncoding']}`); + } + } + return this.walkNestedSchemaObjects(schema, schemaVisitor); + }; + visitSchemaObjects(this.openapi30, schemaVisitor); + } private json(x) { return JSON.stringify(x, null, 2); diff --git a/test/converter.spec.ts b/test/converter.spec.ts index 0946d08..ccf0ae7 100644 --- a/test/converter.spec.ts +++ b/test/converter.spec.ts @@ -1,4 +1,4 @@ -/* eslint-disable prefer-destructuring */ +/*eslint-disable prefer-destructuring */ /* eslint-disable @typescript-eslint/no-explicit-any */ import { describe, expect, test } from '@jest/globals'; @@ -291,11 +291,11 @@ describe('resolver test suite', () => { a: { type: 'string', $comment: 'This is a comment.', - } + }, }, }, }; - const converter = new Converter(input, {}); + const converter = new Converter(input, { convertSchemaComments: true }); const converted: any = converter.convert(); const a = converted.components.schemas.a; @@ -355,89 +355,139 @@ describe('resolver test suite', () => { done(); }); - test('Remove $id and $schema keywords', (done) => { - // const sourceFileName = path.join(__dirname, 'data/root.yaml'); // __dirname is the test dir - const input = { - openapi: '3.1.0', - components: { - schemas: { - a: { - $id: 'http://www.example.com/schemas/a', - $schema: 'https://json-schema.org/draft/2020-12/schema', - type: 'string', - }, - }, - }, - }; - const expected = { - openapi: '3.0.3', - components: { - schemas: { - a: { - type: 'string', - }, - }, - }, - }; - const converter = new Converter(input, { verbose: true }); - const converted: any = converter.convert(); - expect(converted).toEqual(expected); - done(); - }); + test('Remove $id and $schema keywords', (done) => { + // const sourceFileName = path.join(__dirname, 'data/root.yaml'); // __dirname is the test dir + const input = { + openapi: '3.1.0', + components: { + schemas: { + a: { + $id: 'http://www.example.com/schemas/a', + $schema: 'https://json-schema.org/draft/2020-12/schema', + type: 'string', + }, + }, + }, + }; + const expected = { + openapi: '3.0.3', + components: { + schemas: { + a: { + type: 'string', + }, + }, + }, + }; + const converter = new Converter(input, { verbose: true }); + const converted: any = converter.convert(); + expect(converted).toEqual(expected); + done(); + }); - test('Rename $comment to x-comment', (done) => { - const input = { - openapi: '3.1.0', - components: { - schemas: { - a: { - type: 'object', - $comment: 'a comment on schema a', - properties: { - b: { - type: 'object', - $comment: 'A comment on a.b', - properties: { - s: { - type: 'string', - $comment: 'A comment on a.b.s', - }, - }, - }, - }, - }, - }, - }, - }; - const expected = { - openapi: '3.0.3', - components: { - schemas: { - a: { - type: 'object', - 'x-comment': 'a comment on schema a', - properties: { - b: { - type: 'object', + test('Rename $comment to x-comment', (done) => { + const input = { + openapi: '3.1.0', + components: { + schemas: { + a: { + type: 'object', + $comment: 'a comment on schema a', + properties: { + b: { + type: 'object', + $comment: 'A comment on a.b', + properties: { + s: { + type: 'string', + $comment: 'A comment on a.b.s', + }, + }, + }, + }, + }, + }, + }, + }; + const expected = { + openapi: '3.0.3', + components: { + schemas: { + a: { + type: 'object', + 'x-comment': 'a comment on schema a', + properties: { + b: { + type: 'object', - 'x-comment': 'A comment on a.b', - properties: { - s: { - type: 'string', - 'x-comment': 'A comment on a.b.s', - }, - }, - }, - }, - }, - }, - }, - }; - const converter = new Converter(input, { verbose: true }); - const converted: any = converter.convert(); - expect(converted).toEqual(expected); - done(); - }); + 'x-comment': 'A comment on a.b', + properties: { + s: { + type: 'string', + 'x-comment': 'A comment on a.b.s', + }, + }, + }, + }, + }, + }, + }, + }; + const converter = new Converter(input, { verbose: true, convertSchemaComments: true }); + const converted: any = converter.convert(); + expect(converted).toEqual(expected); + done(); + }); + + test('Delete $comment (not convert to x-comment)', (done) => { + const input = { + openapi: '3.1.0', + components: { + schemas: { + a: { + type: 'object', + $comment: 'a comment on schema a', + properties: { + b: { + type: 'object', + $comment: 'A comment on a.b', + properties: { + s: { + type: 'string', + $comment: 'A comment on a.b.s', + }, + }, + }, + }, + }, + }, + }, + }; + const expected = { + openapi: '3.0.3', + components: { + schemas: { + a: { + type: 'object', + properties: { + b: { + type: 'object', + properties: { + s: { + type: 'string', + }, + }, + }, + }, + }, + }, + }, + }; + const converter = new Converter(input, { verbose: true }); + const converted: any = converter.convert(); + expect(converted).toEqual(expected); + done(); + }); test('Convert nullable type array', (done) => { // const sourceFileName = path.join(__dirname, 'data/root.yaml'); // __dirname is the test dir @@ -558,19 +608,19 @@ describe('resolver test suite', () => { test('Remove info.license.identifier', (done) => { // const sourceFileName = path.join(__dirname, 'data/root.yaml'); // __dirname is the test dir const input = { - openapi: "3.1.0", + openapi: '3.1.0', info: { license: { name: 'MIT', - identifier: 'MIT' + identifier: 'MIT', }, }, }; const expected = { - openapi: "3.0.3", + openapi: '3.0.3', info: { license: { - name: 'MIT' + name: 'MIT', }, }, }; @@ -599,3 +649,178 @@ describe('resolver test suite', () => { done(); }); }); + +test('binary encoded data with existing binary format', (done) => { + const input = { + openapi: '3.1.0', + components: { + schemas: { + binaryEncodedDataWithExistingBinaryFormat: { + type: 'string', + format: 'binary', + contentEncoding: 'base64', + }, + }, + }, + }; + const converter = new Converter(input); + let caught = false; + try { + converter.convert(); + } catch (e) { + caught = true; + } + expect(caught).toBeTruthy() + // TODO how to check that Converter logged a specific note? + done(); +}); + +test('binary encoded data with byte format', (done) => { + // const sourceFileName = path.join(__dirname, 'data/root.yaml'); // __dirname is the test dir + const input = { + openapi: '3.1.0', + components: { + schemas: { + binaryEncodedDataWithByteFormat: { + type: 'string', + format: 'byte', + contentEncoding: 'base64', + }, + }, + }, + }; + const expected = { + openapi: '3.0.3', + components: { + schemas: { + binaryEncodedDataWithByteFormat: { + type: 'string', + format: 'byte', + }, + }, + }, + }; + const converter = new Converter(input); + const converted: any = converter.convert(); + expect(converted).toEqual(expected); + done(); +}); + +test('binary encoded data with no existing format', (done) => { + // const sourceFileName = path.join(__dirname, 'data/root.yaml'); // __dirname is the test dir + const input = { + openapi: '3.1.0', + components: { + schemas: { + binaryEncodedDataWithNoFormat: { + type: 'string', + contentEncoding: 'base64', + }, + }, + }, + }; + const expected = { + openapi: '3.0.3', + components: { + schemas: { + binaryEncodedDataWithNoFormat: { + type: 'string', + format: 'byte', + }, + }, + }, + }; + const converter = new Converter(input); + const converted: any = converter.convert(); + expect(converted).toEqual(expected); + done(); +}); + +test('contentMediaType with existing binary format', (done) => { + const input = { + openapi: '3.1.0', + components: { + schemas: { + binaryEncodedDataWithExistingBinaryFormat: { + type: 'string', + contentMediaType: 'application/octet-stream', + format: 'binary' + }, + }, + }, + }; + const expected = { + openapi: '3.0.3', + components: { + schemas: { + binaryEncodedDataWithExistingBinaryFormat: { + type: 'string', + format: 'binary', + }, + }, + }, + }; + const converter = new Converter(input); + const converted: any = converter.convert(); + expect(converted).toEqual(expected); + // TODO how to check that Converter logged to console.warn ? + done(); +}); + + +test('contentMediaType with no existing format', (done) => { + const input = { + openapi: '3.1.0', + components: { + schemas: { + binaryEncodedDataWithExistingBinaryFormat: { + type: 'string', + contentMediaType: 'application/octet-stream', + }, + }, + }, + }; + const expected = { + openapi: '3.0.3', + components: { + schemas: { + binaryEncodedDataWithExistingBinaryFormat: { + type: 'string', + format: 'binary', + }, + }, + }, + }; + const converter = new Converter(input); + const converted: any = converter.convert(); + expect(converted).toEqual(expected); + // TODO how to check that Converter logged to console.warn ? + done(); +}); + +test('contentMediaType with existing unexpected format', (done) => { + const input = { + openapi: '3.1.0', + components: { + schemas: { + binaryEncodedDataWithExistingBinaryFormat: { + type: 'string', + contentMediaType: 'application/octet-stream', + format: 'byte' + }, + }, + }, + }; + + const converter = new Converter(input); + let caught = false; + try { + converter.convert(); + } catch (e) { + caught = true; + } + expect(caught).toBeTruthy(); + // TODO how to check that Converter logged to console.warn ? + done(); +}); + diff --git a/test/data/openapi.yaml b/test/data/openapi.yaml index 913a772..8a4fd62 100644 --- a/test/data/openapi.yaml +++ b/test/data/openapi.yaml @@ -427,12 +427,8 @@ components: format: date-time readOnly: true example: '2021-10-30T19:06:04.250Z' - - resourceTitle: - title: Resource Title - description: A Title for a business object - type: string - maxLength: 80 - x-comment: >- - this maxLength must match the maxLength of - `title` in the `resourcePatch` schema. + $comment: >- + minlength is for no milliseconds, such as + '2021-10-30T19:06:00Z' + maxLength is for '.' plus up to 9 digits for milliseconds, + such as '2021-10-30T19:06:04.999000999Z' \ No newline at end of file