Merge branch 'refs/heads/main' into feat/knowledge-admin-role

* refs/heads/main: (36 commits) feat: add claude3 function calling (#5889) refactor: Create a `dify_config` with Pydantic. (#5938) fix: zhipuai pytest correction (#5934) docs(api/core/tools/docs/en_US/tool_scale_out.md): Format by markdownlint. (#5903) chore: remove dify SaaS URL in default configs (#5888) feat: pr template (#5886) fix bug : TencentVectorDBConfig Add TENCENT_VECTOR_DB_DATABASE (#5879) refactor(api/core/app/apps/base_app_generator.py): improve input validation and sanitization in BaseAppGenerator (#5866) chore: click area that trigger showing tracing config is too large (#5878) Fix/remove tsne position test (#5858) Fix/docker env namings (#5857) doc: docker-compose won't start due to wrong README (#5859) fix:retieval setting document link 404 (#5861) chore:remove .env.example duplicate key (#5853) fix: not show opening question if the opening message is empty (#5856) fix: react.js error 185 maximum update depth exceeded in streaming responses during conversation (#5849) Ensure *.sh are LF-style, so that they can be used directly by Docker for Windows (#5793) feat: add export permission (#5841) feat: update LangfuseConfig host config (#5846) fix:unable to select workplace at the bottom (#5785) ...
langgenius · Jul 4, 2024 · 82f5778 · 82f5778
2 parents eae3621 + aecdfa2
commit 82f5778
Show file tree

Hide file tree

Showing 91 changed files with 1,406 additions and 379 deletions.
diff --git a/.devcontainer/post_create_command.sh b/.devcontainer/post_create_command.sh
@@ -3,7 +3,7 @@
 cd web && npm install
 
 echo 'alias start-api="cd /workspaces/dify/api && flask run --host 0.0.0.0 --port=5001 --debug"' >> ~/.bashrc
-echo 'alias start-worker="cd /workspaces/dify/api && celery -A app.celery worker -P gevent -c 1 --loglevel INFO -Q dataset,generation,mail,ops_trace"' >> ~/.bashrc
+echo 'alias start-worker="cd /workspaces/dify/api && celery -A app.celery worker -P gevent -c 1 --loglevel INFO -Q dataset,generation,mail,ops_trace,app_deletion"' >> ~/.bashrc
 echo 'alias start-web="cd /workspaces/dify/web && npm run dev"' >> ~/.bashrc
 echo 'alias start-containers="cd /workspaces/dify/docker && docker-compose -f docker-compose.middleware.yaml -p dify up -d"' >> ~/.bashrc
 

diff --git a/.gitattributes b/.gitattributes
@@ -0,0 +1,7 @@
+# Ensure that .sh scripts use LF as line separator, even if they are checked out
+# to Windows(NTFS) file-system, by a user of Docker for Window. 
+# These .sh scripts will be run from the Container after `docker compose up -d`.
+# If they appear to be CRLF style, Dash from the Container will fail to execute
+# them.
+
+*.sh      text eol=lf
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -1,32 +1,34 @@
+# Checklist:
+
+> [!IMPORTANT]  
+> Please review the checklist below before submitting your pull request.
+
+- [ ] Please open an issue before creating a PR or link to an existing issue
+- [ ] I have performed a self-review of my own code
+- [ ] I have commented my code, particularly in hard-to-understand areas
+- [ ] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods
+
 # Description
 
-Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
+Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. Close issue syntax: `Fixes #<issue number>`, see [documentation](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) for more details.
 
-Fixes # (issue)
+Fixes 
 
 ## Type of Change
 
-Please delete options that are not relevant.
-
 - [ ] Bug fix (non-breaking change which fixes an issue)
 - [ ] New feature (non-breaking change which adds functionality)
 - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
 - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs)
 - [ ] Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
 - [ ] Dependency upgrade
 
-# How Has This Been Tested?
+# Testing Instructions
 
 Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
 
-- [ ] TODO
+- [ ] Test A
+- [ ] Test B
+
 
-# Suggested Checklist:
 
-- [ ] I have performed a self-review of my own code
-- [ ] I have commented my code, particularly in hard-to-understand areas
-- [ ] My changes generate no new warnings
-- [ ] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods
-- [ ] `optional` I have made corresponding changes to the documentation 
-- [ ] `optional` I have added tests that prove my fix is effective or that my feature works
-- [ ] `optional` New and existing unit tests pass locally with my changes
diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -48,7 +48,7 @@
                 "--loglevel",
                 "info",
                 "-Q",
-                "dataset,generation,mail,ops_trace"
+                "dataset,generation,mail,ops_trace,app_deletion"
             ]
         },
     ]

diff --git a/README.md b/README.md
@@ -192,6 +192,11 @@ If you'd like to configure a highly-available setup, there are community-contrib
 - [Helm Chart by @BorisPolonsky](https://github.com/BorisPolonsky/dify-helm)
 - [YAML file by @Winson-030](https://github.com/Winson-030/dify-kubernetes)
 
+#### Using Terraform for Deployment
+
+##### Azure Global
+Deploy Dify to Azure with a single click using [terraform](https://www.terraform.io/).
+- [Azure Terraform by @nikawang](https://github.com/nikawang/dify-azure-terraform)
 
 ## Contributing
 

diff --git a/README_AR.md b/README_AR.md
@@ -175,6 +175,12 @@ docker compose up -d
 - [رسم بياني Helm من قبل @BorisPolonsky](https://github.com/BorisPolonsky/dify-helm)
 - [ملف YAML من قبل @Winson-030](https://github.com/Winson-030/dify-kubernetes)
 
+#### استخدام Terraform للتوزيع
+
+##### Azure Global
+استخدم [terraform](https://www.terraform.io/) لنشر Dify على Azure بنقرة واحدة.
+- [Azure Terraform بواسطة @nikawang](https://github.com/nikawang/dify-azure-terraform)
+
 
 ## المساهمة
 

diff --git a/README_CN.md b/README_CN.md
@@ -197,6 +197,12 @@ docker compose up -d
 - [Helm Chart by @BorisPolonsky](https://github.com/BorisPolonsky/dify-helm)
 - [YAML 文件 by @Winson-030](https://github.com/Winson-030/dify-kubernetes)
 
+#### 使用 Terraform 部署
+
+##### Azure Global
+使用 [terraform](https://www.terraform.io/) 一键部署 Dify 到 Azure。
+- [Azure Terraform by @nikawang](https://github.com/nikawang/dify-azure-terraform)
+
 ## Star History
 
 [![Star History Chart](https://api.star-history.com/svg?repos=langgenius/dify&type=Date)](https://star-history.com/#langgenius/dify&Date)

diff --git a/README_ES.md b/README_ES.md
@@ -199,6 +199,12 @@ Si desea configurar una configuración de alta disponibilidad, la comunidad prop
 - [Gráfico Helm por @BorisPolonsky](https://github.com/BorisPolonsky/dify-helm)
 - [Ficheros YAML por @Winson-030](https://github.com/Winson-030/dify-kubernetes)
 
+#### Uso de Terraform para el despliegue
+
+##### Azure Global
+Utiliza [terraform](https://www.terraform.io/) para desplegar Dify en Azure con un solo clic.
+- [Azure Terraform por @nikawang](https://github.com/nikawang/dify-azure-terraform)
+
 
 ## Contribuir
 

diff --git a/README_FR.md b/README_FR.md
@@ -197,6 +197,12 @@ Si vous souhaitez configurer une configuration haute disponibilité, la communau
 - [Helm Chart par @BorisPolonsky](https://github.com/BorisPolonsky/dify-helm)
 - [Fichier YAML par @Winson-030](https://github.com/Winson-030/dify-kubernetes)
 
+#### Utilisation de Terraform pour le déploiement
+
+##### Azure Global
+Utilisez [terraform](https://www.terraform.io/) pour déployer Dify sur Azure en un clic.
+- [Azure Terraform par @nikawang](https://github.com/nikawang/dify-azure-terraform)
+
 
 ## Contribuer
 

diff --git a/README_JA.md b/README_JA.md
@@ -196,6 +196,12 @@ docker compose up -d
 - [Helm Chart by @BorisPolonsky](https://github.com/BorisPolonsky/dify-helm)
 - [YAML file by @Winson-030](https://github.com/Winson-030/dify-kubernetes)
 
+#### Terraformを使用したデプロイ
+
+##### Azure Global
+[terraform](https://www.terraform.io/) を使用して、AzureにDifyをワンクリックでデプロイします。
+- [nikawangのAzure Terraform](https://github.com/nikawang/dify-azure-terraform)
+
 
 ## 貢献
 

diff --git a/README_KL.md b/README_KL.md
@@ -197,6 +197,13 @@ If you'd like to configure a highly-available setup, there are community-contrib
 - [Helm Chart by @BorisPolonsky](https://github.com/BorisPolonsky/dify-helm)
 - [YAML file by @Winson-030](https://github.com/Winson-030/dify-kubernetes)
 
+#### Terraform atorlugu pilersitsineq
+
+##### Azure Global
+Atoruk [terraform](https://www.terraform.io/) Dify-mik Azure-mut ataatsikkut ikkussuilluarlugu.
+- [Azure Terraform atorlugu @nikawang](https://github.com/nikawang/dify-azure-terraform)
+
+
 ## Contributing
 
 For those who'd like to contribute code, see our [Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md). 

diff --git a/README_KR.md b/README_KR.md
@@ -190,6 +190,12 @@ Dify를 Kubernetes에 배포하고 프리미엄 스케일링 설정을 구성했
 - [Helm Chart by @BorisPolonsky](https://github.com/BorisPolonsky/dify-helm)
 - [YAML file by @Winson-030](https://github.com/Winson-030/dify-kubernetes)
 
+#### Terraform을 사용한 배포
+
+##### Azure Global
+[terraform](https://www.terraform.io/)을 사용하여 Azure에 Dify를 원클릭으로 배포하세요.
+- [nikawang의 Azure Terraform](https://github.com/nikawang/dify-azure-terraform)
+
 ## 기여
 
 코드에 기여하고 싶은 분들은 [기여 가이드](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md)를 참조하세요.

diff --git a/api/.env.example b/api/.env.example
@@ -72,6 +72,13 @@ TENCENT_COS_SECRET_ID=your-secret-id
 TENCENT_COS_REGION=your-region
 TENCENT_COS_SCHEME=your-scheme
 
+# OCI Storage configuration
+OCI_ENDPOINT=your-endpoint
+OCI_BUCKET_NAME=your-bucket-name
+OCI_ACCESS_KEY=your-access-key
+OCI_SECRET_KEY=your-secret-key
+OCI_REGION=your-region
+
 # CORS configuration
 WEB_API_CORS_ALLOW_ORIGINS=http://127.0.0.1:3000,*
 CONSOLE_CORS_ALLOW_ORIGINS=http://127.0.0.1:3000,*

diff --git a/api/README.md b/api/README.md
@@ -11,6 +11,7 @@
 
    ```bash
    cd ../docker
+   cp .middleware.env.example .middleware.env
    docker compose -f docker-compose.middleware.yaml -p dify up -d
    cd ../api
    ```
@@ -66,7 +67,7 @@
 10. If you need to debug local async processing, please start the worker service.
 
    ```bash
-   poetry run python -m celery -A app.celery worker -P gevent -c 1 --loglevel INFO -Q dataset,generation,mail,ops_trace
+   poetry run python -m celery -A app.celery worker -P gevent -c 1 --loglevel INFO -Q dataset,generation,mail,ops_trace,app_deletion
    ```
 
    The started celery app handles the async tasks, e.g. dataset importing and documents indexing.

diff --git a/api/app.py b/api/app.py
@@ -1,6 +1,6 @@
 import os
 
-from configs.app_config import DifyConfig
+from configs import dify_config
 
 if not os.environ.get("DEBUG") or os.environ.get("DEBUG", "false").lower() != 'true':
     from gevent import monkey
@@ -81,7 +81,7 @@ def create_flask_app_with_configs() -> Flask:
     with configs loaded from .env file
     """
     dify_app = DifyApp(__name__)
-    dify_app.config.from_mapping(DifyConfig().model_dump())
+    dify_app.config.from_mapping(dify_config.model_dump())
 
     # populate configs into system environment variables
     for key, value in dify_app.config.items():

diff --git a/api/commands.py b/api/commands.py
@@ -8,6 +8,7 @@
 from flask import current_app
 from werkzeug.exceptions import NotFound
 
+from configs import dify_config
 from constants.languages import languages
 from core.rag.datasource.vdb.vector_factory import Vector
 from core.rag.datasource.vdb.vector_type import VectorType
@@ -112,7 +113,7 @@ def reset_encrypt_key_pair():
     After the reset, all LLM credentials will become invalid, requiring re-entry.
     Only support SELF_HOSTED mode.
     """
-    if current_app.config['EDITION'] != 'SELF_HOSTED':
+    if dify_config.EDITION != 'SELF_HOSTED':
         click.echo(click.style('Sorry, only support SELF_HOSTED mode.', fg='red'))
         return
 

diff --git a/api/configs/__init__.py b/api/configs/__init__.py
@@ -0,0 +1,4 @@
+
+from .app_config import DifyConfig
+
+dify_config = DifyConfig()
diff --git a/api/configs/app_config.py b/api/configs/app_config.py
@@ -8,6 +8,9 @@
 from configs.packaging import PackagingInfo
 
 
+# TODO: Both `BaseModel` and `BaseSettings` has `model_config` attribute but they are in different types.
+# This inheritance is depends on the order of the classes.
+# It is better to use `BaseSettings` as the base class.
 class DifyConfig(
     # based on pydantic-settings
     BaseSettings,

diff --git a/api/configs/feature/__init__.py b/api/configs/feature/__init__.py
@@ -50,25 +50,25 @@ class EndpointConfig(BaseModel):
     CONSOLE_API_URL: str = Field(
         description='The backend URL prefix of the console API.'
                     'used to concatenate the login authorization callback or notion integration callback.',
-        default='https://cloud.dify.ai',
+        default='',
     )
 
     CONSOLE_WEB_URL: str = Field(
         description='The front-end URL prefix of the console web.'
                     'used to concatenate some front-end addresses and for CORS configuration use.',
-        default='https://cloud.dify.ai',
+        default='',
     )
 
     SERVICE_API_URL: str = Field(
         description='Service API Url prefix.'
                     'used to display Service API Base Url to the front-end.',
-        default='https://api.dify.ai',
+        default='',
     )
 
     APP_WEB_URL: str = Field(
         description='WebApp Url prefix.'
                     'used to display WebAPP API Base Url to the front-end.',
-        default='https://udify.app',
+        default='',
     )
 
 
@@ -82,7 +82,7 @@ class FileAccessConfig(BaseModel):
                     'Url is signed and has expiration time.',
         validation_alias=AliasChoices('FILES_URL', 'CONSOLE_API_URL'),
         alias_priority=1,
-        default='https://cloud.dify.ai',
+        default='',
     )
 
     FILES_ACCESS_TIMEOUT: int = Field(
@@ -136,7 +136,7 @@ class HttpConfig(BaseModel):
     def CONSOLE_CORS_ALLOW_ORIGINS(self) -> list[str]:
         return self.inner_CONSOLE_CORS_ALLOW_ORIGINS.split(',')
 
-    inner_WEB_API_CORS_ALLOW_ORIGINS: Optional[str] = Field(
+    inner_WEB_API_CORS_ALLOW_ORIGINS: str = Field(
         description='',
         validation_alias=AliasChoices('WEB_API_CORS_ALLOW_ORIGINS'),
         default='*',

diff --git a/api/configs/middleware/__init__.py b/api/configs/middleware/__init__.py
@@ -7,6 +7,7 @@
 from configs.middleware.storage.amazon_s3_storage_config import S3StorageConfig
 from configs.middleware.storage.azure_blob_storage_config import AzureBlobStorageConfig
 from configs.middleware.storage.google_cloud_storage_config import GoogleCloudStorageConfig
+from configs.middleware.storage.oci_storage_config import OCIStorageConfig
 from configs.middleware.storage.tencent_cos_storage_config import TencentCloudCOSStorageConfig
 from configs.middleware.vdb.chroma_config import ChromaConfig
 from configs.middleware.vdb.milvus_config import MilvusConfig
@@ -143,7 +144,7 @@ class CeleryConfig(DatabaseConfig):
 
     @computed_field
     @property
-    def CELERY_RESULT_BACKEND(self) -> str:
+    def CELERY_RESULT_BACKEND(self) -> str | None:
         return 'db+{}'.format(self.SQLALCHEMY_DATABASE_URI) \
             if self.CELERY_BACKEND == 'database' else self.CELERY_BROKER_URL
 
@@ -167,6 +168,7 @@ class MiddlewareConfig(
     GoogleCloudStorageConfig,
     TencentCloudCOSStorageConfig,
     S3StorageConfig,
+    OCIStorageConfig,
 
     # configs of vdb and vdb providers
     VectorStoreConfig,

diff --git a/api/configs/middleware/storage/oci_storage_config.py b/api/configs/middleware/storage/oci_storage_config.py
@@ -0,0 +1,35 @@
+from typing import Optional
+
+from pydantic import BaseModel, Field
+
+
+class OCIStorageConfig(BaseModel):
+    """
+    OCI storage configs
+    """
+
+    OCI_ENDPOINT: Optional[str] = Field(
+        description='OCI storage endpoint',
+        default=None,
+    )
+
+    OCI_REGION: Optional[str] = Field(
+        description='OCI storage region',
+        default=None,
+    )
+
+    OCI_BUCKET_NAME: Optional[str] = Field(
+        description='OCI storage bucket name',
+        default=None,
+    )
+
+    OCI_ACCESS_KEY: Optional[str] = Field(
+        description='OCI storage access key',
+        default=None,
+    )
+
+    OCI_SECRET_KEY: Optional[str] = Field(
+        description='OCI storage secret key',
+        default=None,
+    )
+
diff --git a/api/configs/middleware/vdb/tencent_vector_config.py b/api/configs/middleware/vdb/tencent_vector_config.py
@@ -42,3 +42,8 @@ class TencentVectorDBConfig(BaseModel):
         description='Tencent Vector replicas',
         default=2,
     )
+
+    TENCENT_VECTOR_DB_DATABASE: Optional[str] = Field(
+        description='Tencent Vector Database',
+        default=None,
+    )
diff --git a/api/controllers/console/app/app.py b/api/controllers/console/app/app.py
@@ -190,6 +190,10 @@ class AppExportApi(Resource):
     @get_app_model
     def get(self, app_model):
         """Export app"""
+        # The role of the current user in the ta table must be admin, owner, or editor
+        if not current_user.is_editor:
+            raise Forbidden()
+
         app_service = AppService()
 
         return {
-Original file line number
+Diff line change
@@ Expand Up / @@ -48,7 +48,7 @@ @@
                     "--loglevel",
                     "info",
                     "-Q",
-                    "dataset,generation,mail,ops_trace"
+                    "dataset,generation,mail,ops_trace,app_deletion"
                 ]
             },
         ]
@@ Expand Down @@
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,4 @@

		from .app_config import DifyConfig

		dify_config = DifyConfig()