OpenAPI generator upgrade (7.0.0) + Python `lakefs-sdk` #6562

Jonathan-Rosenberg · 2023-09-07T14:26:32Z

Change Description

This PR doesn't actually close the issue, becuase I couldn't find true forward compatibility issues with the current SDK, but it's preferred to upgrade the OpenAPI generator as it's pretty old (Oct 24, 2021)...
This will not break the current behavior and examples from the docs because it's not published as the lakefs-client project but as a new lakefs-sdk project.

arielshaqed

Thanks!

I would like to share a common prefix on all Makefile targets of SDKs / generated API
I prefer to name the existing client targets "legacy"
I think we need a check that the Python SDK is checked-in, like we already have for the "legacy" Python SDK
Once ll this is done, please also check in the generated Python client.

arielshaqed · 2023-10-02T10:43:54Z

.github/workflows/python-api-client.yaml

@@ -28,14 +28,22 @@ jobs:
    - name: Python build and make package
      run: make package-python PACKAGE_VERSION=${{ steps.version.outputs.tag }}

-    - name: Python publish package
+    - name: Python client publish package


Suggested change

- name: Python client publish package

- name: Python SDK publish package

?

SDK legacy

arielshaqed · 2023-10-02T10:45:14Z

Makefile

+OPENAPI_GENERATOR_VERSION_ADVANCED=v7.0.0
+OPENAPI_GENERATOR_IMAGE_ADVANCED=openapitools/openapi-generator-cli:$(OPENAPI_GENERATOR_VERSION_ADVANCED)
+OPENAPI_GENERATOR_ADVANCED=$(DOCKER) run --user $(UID_GID) --rm -v $(shell pwd):/mnt $(OPENAPI_GENERATOR_IMAGE_ADVANCED)


The more I think about these names the less I am a fan of them. Perhaps call them "OPENAPI_GENERATOR_SERVER" and "OPENAPI_GENERATOR_CLIENT_SDK"? (Extremely unimportant, any which way you decide)

the only difference between them is the openAPI version they use. This is the reason that I've added the ADVANCED part to the newer version.
They both generate the client's SDK

arielshaqed · 2023-10-02T10:46:02Z

Makefile


-client-python: api/swagger.yml  ## Generate SDK for Python client
+python-client: api/swagger.yml  ## Generate SDK for Python client - openapi generator version 5.3.0


Prefer to keep a common prefix for all SDKs, like we have common prefixes for other things here.

Suggested change

python-client: api/swagger.yml ## Generate SDK for Python client - openapi generator version 5.3.0

sdk-python-legacy: api/swagger.yml ## Generate SDK for Python client - openapi generator version 5.3.0

arielshaqed

Comments about the template files, too!

arielshaqed · 2023-10-02T10:49:13Z

clients/python/sdk/.openapi-generator-ignore

+Gemfile.lock
+Gemfile
+_config.yml
+templates/


This file looks wrong. Can we get a better file, maybe a "Python gitignore" will do better?

arielshaqed · 2023-10-02T10:49:58Z

clients/python/templates/sdk/client.mustache

+
+    @property
+    def actions(self):
+        warnings.warn("Deprecated property. Use actions_api instead.", DeprecationWarning)


It's a new SDK. Let's not do this!

arielshaqed · 2023-10-02T10:50:16Z

clients/python/templates/sdk/setup.mustache

+  "aiohttp >= 3.0.0",
+{{/asyncio}}
+{{#tornado}}
+  "tornado>=4.2,<5",


We need this?

arielshaqed · 2023-10-02T10:51:06Z

clients/python/templates/sdk/setup.mustache

+    author="{{infoName}}{{^infoName}}OpenAPI Generator community{{/infoName}}",
+    author_email="{{infoEmail}}{{^infoEmail}}[email protected]{{/infoEmail}}",
+    url="{{packageUrl}}",
+    keywords=["OpenAPI", "OpenAPI-Generator", "{{{appName}}}"],


Suggested change

keywords=["OpenAPI", "OpenAPI-Generator", "{{{appName}}}"],

keywords=["OpenAPI", "auto-generated", "{{{appName}}}"],

It's not an OpenAPI generator. We can add something longer that says it was auto-generated in the description, though.

# Conflicts: # Makefile # clients/python/docs/ObjectsApi.md

arielshaqed

Do we need swagger2.yml? (Because it might be folded in the default GitHub view...)

Other than that -- small comments only. Very very neat, thanks!

arielshaqed · 2023-10-04T11:21:38Z

.github/workflows/python-api-client-test-pypi.yaml

+        description: Tag/version to publish
+        required: true
+      project_name:
+        description: The Pypi (test) package name


Suggested change

description: The Pypi (test) package name

description: The PyPI (test) package name

arielshaqed · 2023-10-04T11:22:36Z

.github/workflows/python-api-client.yaml

+#    - name: Build release as latest
+#      working-directory: clients/python
+#      run: bundle exec jekyll build -d _site -b /
+
+#    - name: Build release ${{ steps.docver.outputs.tag }}
+#      working-directory: clients/python
+#      run: bundle exec jekyll build -d _site/${{ steps.docver.outputs.tag }} -b /${{ steps.docver.outputs.tag }}
+
+#    - name: Publish to docs repository
+#      uses: dmnemec/[email protected]
+#      env:
+#        API_TOKEN_GITHUB: ${{ secrets.PERSONAL_TOKEN }}
+#      with:
+#        source_file: clients/python/_site/.
+#        destination_repo: treeverse/docs-lakefs-sdk-python
+#        destination_folder: /
+#        user_email: '[email protected]'
+#        user_name: 'python-docs-action'


Disabled the publish of the new SDK until after I test it over TestPypi

arielshaqed · 2023-10-04T11:24:20Z

api/swagger2.yml

@@ -0,0 +1,4631 @@
+openapi: "3.0.0"


Do we need this file?

Good catch.
Was checked in by mistake

# Conflicts: # clients/python-legacy/lakefs_client/api/retention_api.py # clients/python/.openapi-generator/FILES # clients/python/docs/RetentionApi.md # clients/python/test/test_retention_api.py

add support for Python SDK generated using openapi generator 7.0.0

8dd57eb

Jonathan-Rosenberg requested a review from arielshaqed September 7, 2023 14:26

This was referenced Sep 18, 2023

Bump activesupport from 7.0.5 to 7.0.7.2 in /clients/python #6475

Closed

Bump commonmarker from 0.23.9 to 0.23.10 in /clients/python #6352

Merged

arielshaqed requested changes Oct 2, 2023

View reviewed changes

arielshaqed reviewed Oct 2, 2023

View reviewed changes

Jonathan-Rosenberg marked this pull request as ready for review October 2, 2023 12:40

Jonathan-Rosenberg added 5 commits October 2, 2023 17:57

check in the python sdk code

2a11525

Merge branch 'master' into feature/upgrade-opengenerator-version

01379c1

generate new python sdk

59ebe2d

add makefile changes and re-run them

5020e2a

update github workflow. fix makefile

028a6b9

Jonathan-Rosenberg requested a review from arielshaqed October 3, 2023 13:16

Jonathan-Rosenberg added 3 commits October 4, 2023 12:23

add TestPypi release workflow

31e8826

Merge branch 'master' into feature/upgrade-opengenerator-version

b0a815d

# Conflicts: # Makefile # clients/python/docs/ObjectsApi.md

make client-python

2771351

Jonathan-Rosenberg added the include-changelog PR description should be included in next release changelog label Oct 4, 2023

Jonathan-Rosenberg added 2 commits October 4, 2023 13:42

comment out publishing new python sdk

351b58a

add python-legacy to gitattributes

fc78edc

arielshaqed approved these changes Oct 4, 2023

View reviewed changes

Jonathan-Rosenberg added 3 commits October 4, 2023 14:38

PR fixes

c0e098f

Merge branch 'master' into feature/upgrade-opengenerator-version

7aba733

# Conflicts: # clients/python-legacy/lakefs_client/api/retention_api.py # clients/python/.openapi-generator/FILES # clients/python/docs/RetentionApi.md # clients/python/test/test_retention_api.py

make gen

c6044c8

Jonathan-Rosenberg merged commit 621b8c9 into master Oct 4, 2023
32 checks passed

Jonathan-Rosenberg deleted the feature/upgrade-opengenerator-version branch October 4, 2023 13:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenAPI generator upgrade (7.0.0) + Python `lakefs-sdk` #6562

OpenAPI generator upgrade (7.0.0) + Python `lakefs-sdk` #6562

Jonathan-Rosenberg commented Sep 7, 2023

arielshaqed left a comment

arielshaqed Oct 2, 2023

Jonathan-Rosenberg Oct 2, 2023

arielshaqed Oct 2, 2023

Jonathan-Rosenberg Oct 2, 2023

arielshaqed Oct 2, 2023

arielshaqed left a comment

arielshaqed Oct 2, 2023

arielshaqed Oct 2, 2023

arielshaqed Oct 2, 2023

arielshaqed Oct 2, 2023

arielshaqed left a comment

arielshaqed Oct 4, 2023

arielshaqed Oct 4, 2023

Jonathan-Rosenberg Oct 4, 2023 •

edited

Loading

arielshaqed Oct 4, 2023

Jonathan-Rosenberg Oct 4, 2023

	- name: Python client publish package
	- name: Python SDK publish package


		client-python: api/swagger.yml ## Generate SDK for Python client
		python-client: api/swagger.yml ## Generate SDK for Python client - openapi generator version 5.3.0

	python-client: api/swagger.yml ## Generate SDK for Python client - openapi generator version 5.3.0
	sdk-python-legacy: api/swagger.yml ## Generate SDK for Python client - openapi generator version 5.3.0

	keywords=["OpenAPI", "OpenAPI-Generator", "{{{appName}}}"],
	keywords=["OpenAPI", "auto-generated", "{{{appName}}}"],

	description: The Pypi (test) package name
	description: The PyPI (test) package name

OpenAPI generator upgrade (7.0.0) + Python lakefs-sdk #6562

OpenAPI generator upgrade (7.0.0) + Python lakefs-sdk #6562

Conversation

Jonathan-Rosenberg commented Sep 7, 2023

Change Description

arielshaqed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arielshaqed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arielshaqed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jonathan-Rosenberg Oct 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OpenAPI generator upgrade (7.0.0) + Python `lakefs-sdk` #6562

OpenAPI generator upgrade (7.0.0) + Python `lakefs-sdk` #6562

Jonathan-Rosenberg Oct 4, 2023 •

edited

Loading