-
Notifications
You must be signed in to change notification settings - Fork 77
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[DONE] Fix remote encoding and other stuff (#1056)
* add test for video search utils * test_index_and_delete_es * replace USE_DISTANT_ENCODING_TRANSCODING by USE_REMOTE_ENCODING_TRANSCODING - use store_remote_encoded_video to store video data at the end of encoding - fix discipline filter - change pod/video_encode_transcript/encoding_tasks.py to call store_remote_encoded_video in rest api * remove import dressing in encode module to use it without django * remove unused import * remove unused import - fix pep8 compliance * move json calling to prevent none objeect * fix pep8 * create rest view to importing transcript - call it from task - remove old files * fix transcritping * fix import transcript error * add new configuration * fix access token link * fix download file in main view * add test to remote encoding * update docker * add wait docker and show running container * add env file * split env and start container * start container on one line * start container on one line remove bracket * split build and run container * add env to up container * add elasticsearch verion * export elasticsearch version * move env * use make to try * add setting local * remove touch * fix file * fix setting local * remove unused dot * remove brace * use settings_local_docker_full_test file * add detach to docker and add setting to xapi * add docker exec to test * remove -p and add detach * remove unused option for docker up * replace healthy by wait * add test_encode_transcode command, use form to fix download view and prevent injection * simple test to try * fix test for download file, add encode video * fix pep8 and increase delay to encode * fix unit test and improve code after review * change test remote encode in unit test * change command to run test, add print * call encode to prevent thread - check if run test and pod back run on the same db * work on local - try on GH * Add token and specify url to call back at the end of encoding * change ES version and fix pep8 * fix token * add staff status for user * Add some translations * use covergae to run test * add transcripting in test - change setting to do it - improve transcripting model and task * change transcode by transcript ! * fix pep8 in test settings --------- Co-authored-by: Aymeric Jakobowski <[email protected]>
- Loading branch information
1 parent
ab939b8
commit 935faac
Showing
31 changed files
with
718 additions
and
221 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| name: Pod Encoding Full Docker | ||
| run-name: ${{ github.actor }} is testing Pod encoding 🚀 | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| - master | ||
| - develop | ||
| - features/** | ||
| - dependabot/** | ||
| pull_request: | ||
| branches: | ||
| - main | ||
| - master | ||
| - develop | ||
| env: | ||
| DJANGO_SUPERUSER_USERNAME: "admin" | ||
| DJANGO_SUPERUSER_PASSWORD: "passwd" | ||
| DJANGO_SUPERUSER_EMAIL: "[email protected]" | ||
| ELASTICSEARCH_TAG: "elasticsearch:7.17.18" | ||
| ELASTICSEARCH_VERION: "elasticsearch:7.17.18" | ||
| NODE_TAG: "node:19" | ||
| PYTHON_TAG: "python:3.9-buster" | ||
| REDIS_TAG: "redis:alpine3.16" | ||
| DOCKER_ENV: "full" | ||
| GECKODRIVER_VER: "v0.29.0" | ||
| FIREFOX_VER: "87.0" | ||
|
|
||
| jobs: | ||
| Pod-Docker-Encoding-Actions: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - run: echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event." | ||
| - run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by GitHub!" | ||
| - run: echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}." | ||
| - name: Check out repository code | ||
| uses: actions/checkout@v4 | ||
| - run: echo "💡 The ${{ github.repository }} repository has been cloned to the runner." | ||
| - run: echo "🖥️ The workflow is now ready to test your code on the runner." | ||
| - name: List files in the repository | ||
| run: | | ||
| ls ${{ github.workspace }} | ||
| - run: echo "🍏 This job's status is ${{ job.status }}." | ||
| - name: Create settings local file | ||
| run: | | ||
| mv pod/custom/settings_local_docker_full_test.py pod/custom/settings_local.py | ||
| - name: cat settings local | ||
| run: cat pod/custom/settings_local.py | ||
| - name: Create env file containers | ||
| run: | | ||
| touch .env.dev | ||
| echo DJANGO_SUPERUSER_USERNAME=$DJANGO_SUPERUSER_USERNAME >> .env.dev | ||
| echo DJANGO_SUPERUSER_PASSWORD=$DJANGO_SUPERUSER_PASSWORD >> .env.dev | ||
| echo DJANGO_SUPERUSER_EMAIL=$DJANGO_SUPERUSER_EMAIL >> .env.dev | ||
| echo ELASTICSEARCH_TAG=$ELASTICSEARCH_TAG >> .env.dev | ||
| echo ELASTICSEARCH_VERSION=$ELASTICSEARCH_TAG >> .env.dev | ||
| echo NODE_TAG=$NODE_TAG >> .env.dev | ||
| echo PYTHON_TAG=$PYTHON_TAG >> .env.dev | ||
| echo REDIS_TAG=$REDIS_TAG >> .env.dev | ||
| echo DOCKER_ENV=full >> .env.dev | ||
| echo GECKODRIVER_VER=v0.29.0 | ||
| echo FIREFOX_VER=87.0 | ||
| - name: cat env | ||
| run: cat .env.dev | ||
| - name: make Build container | ||
| run: | | ||
| sudo rm -rf ./pod/log | ||
| sudo rm -rf ./pod/static | ||
| sudo rm -rf ./pod/node_modules | ||
| docker-compose -f ./docker-compose-full-dev-with-volumes.yml -p esup-pod build --build-arg ELASTICSEARCH_VERSION=$ELASTICSEARCH_TAG --build-arg NODE_VERSION=$NODE_TAG --build-arg PYTHON_VERSION=$PYTHON_TAG --no-cache | ||
| docker-compose -f ./docker-compose-full-dev-with-volumes.yml up --detach --force-recreate --always-recreate-deps | ||
| - name: Sleep for 60 seconds to wait run server on pod back | ||
| uses: jakejarvis/wait-action@master | ||
| with: | ||
| time: '60s' | ||
| - name: show running container | ||
| run: docker ps | ||
| - run: | | ||
| echo "🍏 Docker is UP ${{ job.status }}." | ||
| docker exec pod-back-with-volumes ps auxf | ||
| - name: run test in docker | ||
| run: docker exec pod-back-with-volumes coverage run --source='.' manage.py test_encode_transcript | ||
| - name: Stop containers | ||
| if: always() | ||
| run: docker-compose -f ./docker-compose-full-dev-with-volumes.yml down | ||
| - run: echo "END" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| DEBUG = True | ||
|
|
||
| TEST_REMOTE_ENCODE = True | ||
| TEST_SETTINGS = True | ||
|
|
||
| ALLOWED_HOSTS = ["*"] | ||
|
|
||
| ADMINS = ( | ||
| ('Nicolas', '[email protected]'), | ||
| ) | ||
|
|
||
| USE_PODFILE = True | ||
| USE_NOTIFICATIONS = False | ||
| EMAIL_ON_ENCODING_COMPLETION = False | ||
| SECRET_KEY = 'A_CHANGER' | ||
|
|
||
| # We specify here that we're using ES version 7\n | ||
| ES_VERSION = 7 | ||
| ES_URL = ['http://elasticsearch:9200/'] | ||
| CACHES = { | ||
| 'default': { | ||
| 'BACKEND': 'django_redis.cache.RedisCache', | ||
| 'LOCATION': 'redis://redis:6379/3', | ||
| 'OPTIONS': { | ||
| 'CLIENT_CLASS': 'django_redis.client.DefaultClient', | ||
| }, | ||
| 'KEY_PREFIX': 'pod' | ||
| }, | ||
| 'select2': { | ||
| 'BACKEND': 'django_redis.cache.RedisCache', | ||
| 'LOCATION': 'redis://redis:6379/2', | ||
| 'OPTIONS': { | ||
| 'CLIENT_CLASS': 'django_redis.client.DefaultClient', | ||
| }, | ||
| }, | ||
| } | ||
| SESSION_ENGINE = 'redis_sessions.session' | ||
| SESSION_REDIS = { | ||
| 'host': 'redis', | ||
| 'port': 6379, | ||
| 'db': 4, | ||
| 'prefix': 'session', | ||
| 'socket_timeout': 1, | ||
| 'retry_on_timeout': False, | ||
| } | ||
|
|
||
| # Only in containerized environments | ||
| MIGRATION_MODULES = {'flatpages': 'pod.db_migrations'} | ||
|
|
||
| # If DOCKER_ENV = full: activate encoding, transcription and remote xapi | ||
| USE_REMOTE_ENCODING_TRANSCODING = True | ||
| ENCODING_TRANSCODING_CELERY_BROKER_URL = 'redis://redis:6379/7' | ||
| POD_API_URL = "http://pod-back:8080/rest" | ||
| POD_API_TOKEN = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" | ||
|
|
||
| USE_TRANSCRIPTION = True | ||
| TRANSCRIPTION_TYPE = "WHISPER" | ||
| TRANSCRIPTION_MODEL_PARAM = { | ||
| 'WHISPER': { | ||
| 'fr': { | ||
| 'model': "small", | ||
| 'download_root': "/usr/src/app/transcription/whisper/", | ||
| }, | ||
| 'en': { | ||
| 'model': "small", | ||
| 'download_root': "/usr/src/app/transcription/whisper/", | ||
| } | ||
| } | ||
| } | ||
|
|
||
| USE_XAPI_VIDEO = False | ||
| XAPI_CELERY_BROKER_URL = "redis://redis:6379/6" | ||
|
|
||
| # for maximum console logging\n | ||
| LOGGING = {} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.