Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: introduce pydantic-settings for config definition and validation #5202

Merged
merged 1 commit into from
Jun 19, 2024
Merged

feat: introduce pydantic-settings for config definition and validation #5202

merged 1 commit into from
Jun 19, 2024

Conversation

bowenliang123
Copy link
Contributor

@bowenliang123 bowenliang123 commented Jun 14, 2024 •
edited
Loading

Description

  • introduce pydantic-settings for configs parsing and validation
  • it benefits the config entry's in the following aspects:
    • entry definition for each config item
    • data-type validation
    • default value
    • proper descriptions
    • grouping
    • deprecation
  • no impact on the souce config file .env, all the configs is still read via dotenv inside pydantic-settings
  • no impact on configs reading from Flask's app.config['XXXX']

Type of Change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update, included: Dify Document
  • Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
  • Dependency upgrade

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • TODO

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
  • optional I have made corresponding changes to the documentation
  • optional I have added tests that prove my fix is effective or that my feature works
  • optional New and existing unit tests pass locally with my changes

@bowenliang123 bowenliang123 force-pushed the pydantic-setting branch 3 times, most recently from 7e15429 to e584d37 Compare June 15, 2024 05:49
@bowenliang123 bowenliang123 changed the title improve: introduce pydantic-settings for configs parsing and validation refactor: introduce pydantic-settings for configs parsing and validation Jun 15, 2024
@bowenliang123 bowenliang123 marked this pull request as ready for review June 15, 2024 09:55
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. 🐍 python labels Jun 15, 2024
@bowenliang123 bowenliang123 force-pushed the pydantic-setting branch 4 times, most recently from 006dc60 to 82c08a2 Compare June 15, 2024 15:21
@bowenliang123 bowenliang123 marked this pull request as draft June 16, 2024 00:55
@bowenliang123 bowenliang123 marked this pull request as ready for review June 16, 2024 01:06
@dosubot dosubot bot added the 💪 enhancement New feature or request label Jun 16, 2024
@bowenliang123 bowenliang123 force-pushed the pydantic-setting branch 2 times, most recently from a71e2ad to a562f51 Compare June 17, 2024 09:10
@takatost takatost requested a review from laipz8200 June 18, 2024 05:27
@bowenliang123 bowenliang123 force-pushed the pydantic-setting branch 2 times, most recently from 18648f7 to 69fe305 Compare June 18, 2024 06:41
laipz8200
laipz8200 requested changes Jun 18, 2024
View reviewed changes
Copy link
Member

@laipz8200 laipz8200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution! This PR will help us migrate to a more type-safe system. However, there are still some issues. Would you mind addressing them?

api/app.py Outdated Show resolved Hide resolved
api/tests/unit_tests/settings/test_app_settings.py Outdated Show resolved Hide resolved
@bowenliang123 bowenliang123 changed the title refactor: introduce pydantic-settings for configs parsing and validation feat: introduce pydantic-settings for config definition and validation Jun 18, 2024
@bowenliang123 bowenliang123 marked this pull request as draft June 18, 2024 09:55
@bowenliang123 bowenliang123 marked this pull request as ready for review June 19, 2024 01:49
@bowenliang123
Copy link
Contributor Author

@laipz8200 Hi, I have resolved all the problems you listed. Please have a check.

@laipz8200 laipz8200 self-requested a review June 19, 2024 04:58
laipz8200
laipz8200 previously approved these changes Jun 19, 2024
View reviewed changes
Copy link
Member

@laipz8200 laipz8200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Thanks for your hard work.

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Jun 19, 2024
@bowenliang123 bowenliang123 marked this pull request as draft June 19, 2024 05:27
@bowenliang123 bowenliang123 marked this pull request as draft June 19, 2024 05:27
@bowenliang123
Copy link
Contributor Author

Please avoid changing unrelated lines in poetry lockfile. Use dev/sync-poetry script to refresh the poetry lockfile for good.
The --no-update option in poetry lock -C api --no-update helps in this case.

@bowenliang123 bowenliang123 marked this pull request as ready for review June 19, 2024 05:32
@bowenliang123 bowenliang123 requested a review from laipz8200 June 19, 2024 05:32
@laipz8200
Copy link
Member

Please avoid changing unrelated lines in poetry lockfile. Use dev/sync-poetry script to refresh the poetry lockfile for good. The --no-update option in poetry lock -C api --no-update helps in this case.

You're right.

@bowenliang123
Copy link
Contributor Author

I've done it~ Thanks for reviewing.

@laipz8200 laipz8200 merged commit 3cc6093 into langgenius:main Jun 19, 2024
8 checks passed
@bowenliang123 bowenliang123 deleted the pydantic-setting branch June 19, 2024 05:42
ZhouhaoJiang added a commit that referenced this pull request Jun 20, 2024
@ZhouhaoJiang
Merge branch 'refs/heads/main' into feat/llm-ops-tracing
f9c9efe 
* refs/heads/main: (21 commits)
  fix: sentry config float type err (#5416)
  fix: prompt editor insert cursor position (#5415)
  fix: Revert "feat: initial support for Milvus 2.4.x (#3795)" downgrading to 2.3.x for Linux arm64 installation failure (#5414)
  fix: optional parameter missing default value None in http request node (#5413)
  feat: new icons (#5412)
  fix bug: tencent vdb #5378 (#5408)
  Corrected an error in the APi docs (#5398)
  feat: update template (#5395)
  fix: unnecessory data fetch when swithing apps category on explore page (#5155)
  chore: extract retrival method literal values into enum (#5060)
  feat: add log date timezone (#4623)
  docs(api/README): Remove unnecessary `=` (#5380)
  Fix: use new button (#5384)
  refactor: refactor the button component using `forwardRef` (#4379)
  feat: initial support for Milvus 2.4.x (#3795)
  feat: introduce pydantic-settings for config definition and validation (#5202)
  feat: support opensearch approximate k-NN (#5322)
  Add sample environment variables for Aliyun OSS (#5366)
  Fix: multi image preview sign (#5376)
  feat: default timezone to user's local timezone in activate form (#5374)
  ...

# Conflicts:
#	api/config.py
#	api/requirements.txt
@takatost takatost mentioned this pull request Jun 28, 2024
Merged
HuberyHuV1 pushed a commit to HuberyHuV1/dify that referenced this pull request Jul 22, 2024
@bowenliang123 @laipz8200
feat: introduce pydantic-settings for config definition and validation (
95a1d65 
langgenius#5202)

Co-authored-by: -LAN- <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@laipz8200 laipz8200 laipz8200 approved these changes

Assignees
No one assigned
Labels
💪 enhancement New feature or request lgtm This PR has been approved by a maintainer size:XL This PR changes 500-999 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bowenliang123 @laipz8200