Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

⬆️ Upgrade libraries (pydantic v2) #6366

Conversation

giancarloromeo
Copy link
Contributor

@giancarloromeo giancarloromeo commented Sep 16, 2024

What do these changes do?

Upgrade libraries (inside packages folder) to Pydantic v2.

  • aws-library
  • dask-task-models-library
  • models-library
  • notifications-library
  • postgres-database
  • pytest-simcore
  • service-integration
  • service-library
  • settings-library
  • simcore-sdk

Note: the models-library has a dedicated PR (see #6333)

Related issue/s

How to test

Dev-ops checklist

Copy link

codecov bot commented Sep 16, 2024

Codecov Report

Attention: Patch coverage is 88.24503% with 71 lines in your changes missing coverage. Please review.

Please upload report for BASE (pydantic_v2_migration@19e3923). Learn more about missing BASE report.

Files with missing lines Patch % Lines
...ackages/aws-library/src/aws_library/ec2/_models.py 60.8% 9 Missing ⚠️
...icelib/rabbitmq/rpc_interfaces/catalog/services.py 0.0% 8 Missing ⚠️
...brary/src/common_library/pydantic_type_adapters.py 0.0% 5 Missing ⚠️
...odels-library/src/models_library/projects_nodes.py 16.6% 5 Missing ⚠️
...ages/settings-library/src/settings_library/base.py 88.2% 4 Missing ⚠️
.../settings-library/src/settings_library/postgres.py 70.0% 2 Missing and 1 partial ⚠️
...common-library/src/common_library/serialization.py 33.3% 1 Missing and 1 partial ⚠️
...rary/src/servicelib/aiohttp/requests_validation.py 75.0% 2 Missing ⚠️
...abbitmq/rpc_interfaces/dynamic_scheduler/errors.py 0.0% 2 Missing ⚠️
...bitmq/rpc_interfaces/dynamic_scheduler/services.py 0.0% 2 Missing ⚠️
... and 22 more
Additional details and impacted files

Impacted file tree graph

@@                   Coverage Diff                   @@
##             pydantic_v2_migration   #6366   +/-   ##
=======================================================
  Coverage                         ?   80.2%           
=======================================================
  Files                            ?     543           
  Lines                            ?   19325           
  Branches                         ?    2073           
=======================================================
  Hits                             ?   15499           
  Misses                           ?    3519           
  Partials                         ?     307           
Flag Coverage Δ
unittests 80.2% <88.2%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...ckages/aws-library/src/aws_library/ec2/__init__.py 100.0% <ø> (ø)
...ackages/aws-library/src/aws_library/ec2/_errors.py 100.0% <100.0%> (ø)
packages/aws-library/src/aws_library/s3/_client.py 95.3% <100.0%> (ø)
...kages/aws-library/src/aws_library/s3/_constants.py 100.0% <100.0%> (ø)
packages/aws-library/src/aws_library/s3/_errors.py 100.0% <100.0%> (ø)
packages/aws-library/src/aws_library/s3/_utils.py 100.0% <100.0%> (ø)
...ackages/aws-library/src/aws_library/ssm/_errors.py 100.0% <100.0%> (ø)
...ommon-library/src/common_library/errors_classes.py 100.0% <100.0%> (ø)
.../src/common_library/pydantic_networks_extension.py 100.0% <100.0%> (ø)
...dask_task_models_library/container_tasks/docker.py 100.0% <100.0%> (ø)
... and 126 more

@giancarloromeo giancarloromeo changed the title ⬆️ WIP: Upgrade libraries ⬆️ WIP: Upgrade libraries (pydantic v2) Sep 16, 2024
@giancarloromeo giancarloromeo self-assigned this Sep 16, 2024
@giancarloromeo giancarloromeo added the t:maintenance Some planned maintenance work label Sep 16, 2024
@giancarloromeo giancarloromeo added this to the Doppelbock milestone Sep 16, 2024
@giancarloromeo giancarloromeo linked an issue Sep 18, 2024 that may be closed by this pull request
@pcrespov pcrespov requested a review from GitHK October 8, 2024 16:59
Copy link

sonarqubecloud bot commented Oct 9, 2024

@giancarloromeo giancarloromeo enabled auto-merge (squash) October 9, 2024 09:49
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to merge after you have addressed my comments. Looks good to me.

  1. please refactor by moving to TypeAdapterSomeClass all the most used functions and import them from common_library. I have left a few comments regarding these
  2. there are a few checks that I as you to run if all are OK, go ahead an merge

Thanks a lot for the effort. It's looking good.

Comment on lines +91 to +92
min_length=0,
max_length=256,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe use these constants as well here

Comment on lines +71 to +72
AWS_TAG_KEY_MIN_LENGTH: Final[int] = 1
AWS_TAG_KEY_MAX_LENGTH: Final[int] = 128
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would dlclare them at the top of the file as it is common convention in python

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, forgot there :)

Comment on lines +84 to +85
AWS_TAG_VALUE_MIN_LENGTH: Final[int] = 0
AWS_TAG_VALUE_MAX_LENGTH: Final[int] = 256
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

declare on the top of the file

@@ -49,6 +49,8 @@
from types_aiobotocore_s3 import S3Client
from types_aiobotocore_s3.literals import BucketLocationConstraintType

_BYTE_SIZE_ADAPTER: Final[TypeAdapter[ByteSize]] = TypeAdapter(ByteSize)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use the lowercase way of declaring it TypeAdapterByteSize or import it from a common library since this one is used quite a lot in the entire code base

@@ -67,7 +69,9 @@ async def simcore_s3_api(
@pytest.fixture
def bucket_name(faker: Faker) -> S3BucketName:
# NOTE: no faker here as we need some specific namings
return parse_obj_as(S3BucketName, faker.pystr().replace("_", "-").lower())
return TypeAdapter(S3BucketName).validate_python(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would declare this one in the context of the file, in Lowe case like TypeAdapterS3BucketName



class TwilioSettings(BaseCustomSettings):
TWILIO_ACCOUNT_SID: str = Field(..., description="Twilio account String Identifier")
TWILIO_AUTH_TOKEN: str = Field(..., description="API tokens")
TWILIO_COUNTRY_CODES_W_ALPHANUMERIC_SID_SUPPORT: list[CountryCodeStr] = Field(
default=parse_obj_as(
list[CountryCodeStr],
default=TypeAdapter(list[CountryCodeStr]).validate_python(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extract this in this module at the top for performance reasons

Comment on lines +91 to +92
base_url = TypeAdapter(AnyHttpUrl).validate_python(settings_with_defaults.base_url)
api_base_url = TypeAdapter(AnyHttpUrl).validate_python(settings_with_defaults.api_base_url)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e from common can be used

@@ -218,8 +217,8 @@ def e_tag_fixture() -> str:
async def mock_filemanager(mocker: MockerFixture, e_tag: str, faker: Faker) -> None:
mocker.patch(
"simcore_sdk.node_ports_common.filemanager._get_file_meta_data",
return_value=parse_obj_as(
FileMetaDataGet, FileMetaDataGet.Config.schema_extra["examples"][0]
return_value=TypeAdapter(FileMetaDataGet).validate_python(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the common one can be used

Comment on lines +236 to +237
abort_upload=TypeAdapter(AnyUrl).validate_python(faker.uri()),
complete_upload=TypeAdapter(AnyUrl).validate_python(faker.uri()),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use the common one

Comment on lines +249 to +250
TypeAdapter(ByteSize).validate_python(21800510238),
TypeAdapter(ByteSize).validate_python(10485760),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e you might have a common one

@giancarloromeo giancarloromeo merged commit 1d92881 into ITISFoundation:pydantic_v2_migration Oct 9, 2024
41 of 63 checks passed
@giancarloromeo giancarloromeo deleted the is4481/upgrade-libs branch November 21, 2024 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
t:maintenance Some planned maintenance work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Major upgrade to pydantic 2.0
4 participants