Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Support versioning of user project #37

Closed
wants to merge 149 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
149 commits
Select commit Hold shift + click to select a range
802e659
MAJOR CHANGE: Refactor repository
Ramimashkouk Mar 18, 2024
6aae431
feat: Add CLI `init` command
Ramimashkouk Mar 18, 2024
faa0cd0
fix: Add deps to .toml file
Ramimashkouk Mar 18, 2024
8c34afa
chore: Divide processes into build and run ones
Ramimashkouk Mar 19, 2024
c331a61
feat: Add `/bot/builds` endpoint
Ramimashkouk Mar 19, 2024
7856b69
chore: Add CORS mechanism
Ramimashkouk Mar 20, 2024
8744344
refactor: Modify Preset body
Ramimashkouk Mar 20, 2024
bb81ecb
feat: Return info with `/bot/builds` & `/bot/runs`
Ramimashkouk Mar 21, 2024
7f6a0ab
chore: Update possible statuses of a process
Ramimashkouk Mar 21, 2024
75ecebc
fix: Handle error to return `Failed` properly
Ramimashkouk Mar 21, 2024
82b0239
chore: Fix "recording each log twice"
Ramimashkouk Mar 21, 2024
43a84aa
refactor: Unify executing subprocess.run
Ramimashkouk Mar 21, 2024
d6bbfc7
fix: Give up on bot logging to let the IO flow
Ramimashkouk Mar 22, 2024
e517c6c
feat: Add DB
Ramimashkouk Mar 22, 2024
fe77ba4
chore: Define literal types to preset end_status
Ramimashkouk Mar 22, 2024
9f6936d
refactor: refactor ProcessManager
Ramimashkouk Mar 23, 2024
ca2a690
feat: Update DB after each process start and stop
Ramimashkouk Mar 25, 2024
ee5d986
fix: Check process statuses periodically
Ramimashkouk Mar 26, 2024
64785f6
feat: Add `/flows` endpoint
Ramimashkouk Mar 27, 2024
ea818a5
fix: Check run_id in build's runs before saving
Ramimashkouk Mar 27, 2024
6d9df9f
feat: Introduce Docker implementation for backend
Ramimashkouk Mar 27, 2024
ddb3f81
chore: Fix the used dff version
Ramimashkouk Mar 27, 2024
5842942
feat: Provide logging system
Ramimashkouk Mar 28, 2024
c18b59d
fix: Avoid logging out of proj dir
Ramimashkouk Mar 29, 2024
514997e
feat: Serve static files
Ramimashkouk Mar 29, 2024
8d390c4
fix: Correct config server to reload
Ramimashkouk Mar 29, 2024
1ec9a6b
init new front
mxerf Mar 30, 2024
c793346
refactor: Make proper use of pydantic for config
Ramimashkouk Mar 30, 2024
9b79e93
fix: Change `/` to `/home` so that front see it
Ramimashkouk Apr 1, 2024
744880f
feat: Add Dockerfile putting front&end together
Ramimashkouk Apr 1, 2024
2245539
Merge branch 'MAJOR_CHANGE/new_structure' into MAJOR_CHANGE/new_front
Ramimashkouk Apr 1, 2024
359fbbd
Merge pull request #12 from deeppavlov/MAJOR_CHANGE/new_front
Ramimashkouk Apr 1, 2024
fa212c2
nodes layout ui
mxerf Apr 2, 2024
f1de041
fix: Integrate front in back then build the wheel
Ramimashkouk Apr 3, 2024
6805855
chore: Delete old frontend file
Ramimashkouk Apr 3, 2024
edbe185
redirect /app
mxerf Apr 4, 2024
b4c44a5
Merge remote-tracking branch 'origin/MAJOR_CHANGE/new_front' into MAJ…
Ramimashkouk Apr 4, 2024
376029f
fix: Fix static files routing
Ramimashkouk Apr 4, 2024
56a6b74
fix: Cancel pending tasks of websocket properly
Ramimashkouk Apr 4, 2024
d7a7bdb
feat: Add `build_scenario` cli command
Ramimashkouk Apr 5, 2024
79507a1
chore: Enhance retrieval from `/builds` & `/runs`
Ramimashkouk Apr 5, 2024
978969a
feat: Add pagination to `/builds` and `/runs`
Ramimashkouk Apr 7, 2024
bdc8700
feat: Add endpoint to get logs
Ramimashkouk Apr 7, 2024
ff83ef1
fix: Raise error when process not found
Ramimashkouk Apr 7, 2024
019ab66
feat: Connect `build_scenario` command to ui
Ramimashkouk Apr 8, 2024
f1879f1
refactor: Move processes and mangers to services
Ramimashkouk Apr 8, 2024
b23de8f
chore: Update storage logic
Ramimashkouk Apr 8, 2024
e6edc70
critical fixes
mxerf Apr 9, 2024
33b70cc
nodes-layout-ui
mxerf Apr 2, 2024
3e268c8
fix multiplie transitions bug
mxerf Apr 9, 2024
9f99870
fix types bug
mxerf Apr 10, 2024
2657583
Merge branch 'MAJOR_CHANGE/new_front' into MAJOR_CHANGE/new_structure
Ramimashkouk Apr 10, 2024
c9b6813
refactor: Rename pid to run_id/build_id
Ramimashkouk Apr 10, 2024
584c497
fix: Handle process expected errors
Ramimashkouk Apr 10, 2024
c72d7cc
feat: Include versioning in endpoint paths
Ramimashkouk Apr 10, 2024
874f012
chore: Access db asynchronously in `build_scenario`
Ramimashkouk Apr 10, 2024
1597639
start node fix + python conditions + new conditions structure
mxerf Apr 11, 2024
f060c8e
Merge branch 'MAJOR_CHANGE/new_structure' into MAJOR_CHANGE/new_front
mxerf Apr 11, 2024
92b53eb
delete old nodes
mxerf Apr 11, 2024
754f8e6
Merge remote-tracking branch 'origin/MAJOR_CHANGE/new_structure' into…
mxerf Apr 11, 2024
7a8bde6
new api adaptation
mxerf Apr 11, 2024
ebb2055
upd str
mxerf Apr 23, 2024
f8acc79
node instr header fix
mxerf Apr 23, 2024
54233f7
init e2e front test
mxerf May 7, 2024
743e176
start node fix + python conditions + new conditions structure
mxerf Apr 11, 2024
5dfc850
new api adaptation
mxerf Apr 11, 2024
57a33f3
chore: Adapt to new frontend json format
Ramimashkouk Apr 12, 2024
1a9c01d
style: black backend up
Ramimashkouk Apr 12, 2024
f646e50
style: isort things up
Ramimashkouk Apr 12, 2024
1b29062
ci: Add github workflows
Ramimashkouk Apr 12, 2024
0ab147f
fix: Maintain data across Docker restarts
Ramimashkouk Apr 12, 2024
dec213d
ci: fix workflows
Ramimashkouk Apr 14, 2024
8936933
refactor: Move get_last_id to parent class
Ramimashkouk Apr 17, 2024
d863b4d
fix: Update front lock
Ramimashkouk May 7, 2024
4b0ba66
fix: Correct docker internal paths
Ramimashkouk May 7, 2024
9726e07
test: Add unit tests for services
Ramimashkouk Apr 17, 2024
f5b88bc
test: Add unit tests for endpoints
Ramimashkouk Apr 17, 2024
722ddaf
style: black up tests
Ramimashkouk Apr 17, 2024
8e4b27e
style: isort tests
Ramimashkouk Apr 17, 2024
a1c3b5e
test: Add websocket_manager unit test
Ramimashkouk Apr 17, 2024
eb98dee
fix: Unify code to use async processes
Ramimashkouk Apr 20, 2024
c5b2ebf
chore: Create log path in get_logger
Ramimashkouk Apr 20, 2024
056dbe0
test: Add unit `test_start` for run
Ramimashkouk Apr 20, 2024
63ce01f
test: Add endpoints integration test
Ramimashkouk Apr 20, 2024
438b00c
intgr-test: Add websocket
Ramimashkouk Apr 22, 2024
1d6e6ee
refactor: Clean test_api_integration up
Ramimashkouk Apr 22, 2024
290a7ec
style: Format and lint code
Ramimashkouk Apr 22, 2024
8d8d680
ci: Add tests to workflow
Ramimashkouk Apr 22, 2024
f3c6779
test: Fix communicating indefinetly
Ramimashkouk Apr 23, 2024
be99bba
fix: Avoid creating 2 event_loop in same thread
Ramimashkouk Apr 23, 2024
45daeb6
chore: Catch internal process logs
Ramimashkouk Apr 25, 2024
a5a8398
style: black up
Ramimashkouk Apr 25, 2024
e9f8110
test: Fix test_check_status
Ramimashkouk Apr 25, 2024
8728055
feat: Add `alive` status to run processes
Ramimashkouk Apr 26, 2024
555cb39
refactor: Move fixtures and use consts
Ramimashkouk Apr 26, 2024
9df8a0d
unit-test: Correct intg-tests to actual unit tests
Ramimashkouk Apr 26, 2024
63fc789
test: Add backend e2e
Ramimashkouk Apr 26, 2024
b7477cf
feat: Add indexing for dff services
Ramimashkouk May 3, 2024
78276c7
refactor: json_translator
Ramimashkouk May 6, 2024
26d644f
fix: Adapt index linenos to front changes
Ramimashkouk May 6, 2024
3a9e740
Merge feat/support_indexed_search into MAJOR_CHANGE/new_front
Ramimashkouk May 7, 2024
6ae28f8
style: black up
Ramimashkouk May 8, 2024
3a2a1db
Merge branch 'MAJOR_CHANGE/new_front' into MAJOR_CHANGE/tests_new_front
Ramimashkouk May 8, 2024
453e677
test: Fix e2e test making it work with one client
Ramimashkouk May 8, 2024
2663307
chore: Rename e2e test
Ramimashkouk May 8, 2024
7bf772a
Merge branch 'test/add_tests' into feat/support_indexed_search
Ramimashkouk May 8, 2024
a89de98
fix: Write conditions ending with one \n
Ramimashkouk May 8, 2024
9ea1714
Merge branch 'feat/support_indexed_search' into MAJOR_CHANGE/new_front
Ramimashkouk May 8, 2024
e541131
Merge branch 'MAJOR_CHANGE/new_front' into MAJOR_CHANGE/tests_new_front
Ramimashkouk May 8, 2024
05d51ac
port 5173 to 8000 change
mxerf May 13, 2024
95ff800
test: Add e2e test workflow
Ramimashkouk May 14, 2024
a494df5
chore: Push bun.lockb
Ramimashkouk May 14, 2024
d7a2925
feat: Add make file
Ramimashkouk May 15, 2024
9c9a963
test: Fix unit test test_start
Ramimashkouk May 15, 2024
ec931cb
refactor: Use high-level pathlib
Ramimashkouk May 15, 2024
53cc260
refactor: Return process id by manager.start
Ramimashkouk May 15, 2024
94a14af
fix: Avoid accidental additions to process attrs
Ramimashkouk May 15, 2024
6a030bf
refactor: Convert process status to enum
Ramimashkouk May 15, 2024
2e2ad08
style: black up
Ramimashkouk May 15, 2024
88294ce
chore: Fix check_status test
Ramimashkouk May 15, 2024
d46604d
chore: Raise NotImplementedError instead of pass
Ramimashkouk May 16, 2024
45d8076
chore: Add df_designer_project to .gitignore
Ramimashkouk May 16, 2024
2871081
test: Assert ws connection step by step
Ramimashkouk May 16, 2024
7ec1941
fix: Avoid storing all run info in build yml
Ramimashkouk May 16, 2024
18c5ef6
chore: Return status.value and fix `is_alive`
Ramimashkouk May 16, 2024
79c60c8
style: black up
Ramimashkouk May 16, 2024
78bd34a
chore: Check path before passing to cmd
Ramimashkouk May 16, 2024
7b41da3
test: Fix waiting for a process
Ramimashkouk May 17, 2024
287e4c2
refactor: Simplify things in logging system
Ramimashkouk May 20, 2024
ce40fad
ci: Remove repetitions among jobs
Ramimashkouk May 20, 2024
a2a9301
chore: Check uniqueness of start_node in graph
Ramimashkouk May 20, 2024
3478157
chore: Do various tests chores
Ramimashkouk May 22, 2024
6705950
fixes
Ramimashkouk May 27, 2024
d4b95de
Merge branch 'fix/e2e-front' into fix/review_fixes
Ramimashkouk May 27, 2024
40a1ec4
fix: Correct start node translation
Ramimashkouk May 27, 2024
09e2ea5
merge
mxerf May 24, 2024
57310be
init
mxerf May 28, 2024
db78ade
Merge branch 'fix-docker' into fix/review_fixes
Ramimashkouk May 28, 2024
7d790b6
ci: Add publish and release workflow
Ramimashkouk May 28, 2024
363c329
nodes layout fix
mxerf May 28, 2024
6d046bf
Merge branch 'fixes/front' into ci/publish2pypi
Ramimashkouk Jun 2, 2024
b61e370
feat: Version each of bot & df_designer user dirs
Ramimashkouk Jun 2, 2024
5650c1b
chore: Checkout commits copying the file
Ramimashkouk Jun 2, 2024
65fb2c5
fix: Sync between the two gits
Ramimashkouk Jun 2, 2024
09ebf48
chore: Tag build_id without prefix
Ramimashkouk Jun 3, 2024
515c6e3
style: Black up
Ramimashkouk Jun 3, 2024
6aa4fce
test: Fix tests
Ramimashkouk Jun 3, 2024
975a5fc
chore: Include git in docker
Ramimashkouk Jun 3, 2024
55a2b5f
chore: Reorder docker
Ramimashkouk Jun 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .github/workflows/backend_check.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
name: check backend

on:
push:
branches:
- dev
- master
pull_request:
branches:
- dev

jobs:
build_backend_and_check_codestyle:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: set up python 3.10
uses: actions/setup-python@v5
with:
python-version: '3.10'

- name: setup poetry and install dependencies
run: |
python -m pip install --upgrade pip poetry
python -m poetry lock --no-update
python -m poetry install --with lint --no-interaction
working-directory: backend/df_designer

- name: run black codestyle
run: |
python -m poetry run black --line-length=120 --check .
working-directory: backend/df_designer

- name: run flake8 codestyle
run: |
python -m poetry run flake8 --max-line-length 120 --ignore=E203 .
working-directory: backend/df_designer

- name: run isort codestyle
run: |
python -m poetry run isort --line-length=120 --diff .
working-directory: backend/df_designer

test_backend:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: set up python 3.10
uses: actions/setup-python@v5
with:
python-version: '3.10'

- name: setup poetry and install dependencies
run: |
python -m pip install --upgrade pip poetry
python -m poetry lock --no-update
python -m poetry install --with lint --no-interaction
working-directory: backend/df_designer

- name: build wheel
run: |
python -m poetry build
working-directory: backend/df_designer

- name: Create new project
run: |
python -m poetry run dflowd init --destination ../../ --no-input --overwrite-if-exists
working-directory: backend/df_designer

- name: run tests
run: |
python -m poetry install
python -m poetry run pytest ../backend/df_designer/app/tests/ --verbose
working-directory: df_designer_project
45 changes: 45 additions & 0 deletions .github/workflows/build_and_upload_release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: build_and_upload_release

on:
release:
types:
- published

jobs:
build:
name: build and upload release
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

- name: set up python
uses: actions/setup-python@v5

- name: setup poetry
run: |
python -m pip install --upgrade pip poetry

- name: build wheels and test uploading to pypi
if: startsWith(github.ref, 'refs/tags/v') != true
run: |
python -m poetry --build publish --dry-run
working-directory: backend/df_designer

- name: build wheels and upload to pypi
if: startsWith(github.ref, 'refs/tags/v')
env:
POETRY_PYPI_TOKEN_PYPI: ${{ secrets.PYPI_TOKEN }}
run: |
python -m poetry --build publish
working-directory: backend/df_designer

- name: upload binaries into release
if: startsWith(github.ref, 'refs/tags/v')
uses: svenstaro/upload-release-action@v2
with:
repo_token: ${{ secrets.GITHUB_TOKEN }}
file: backend/df_designer/dist/*
tag: ${{ github.ref }}
overwrite: true
file_glob: true
28 changes: 0 additions & 28 deletions .github/workflows/ci.yml

This file was deleted.

33 changes: 33 additions & 0 deletions .github/workflows/docker_check.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: Build Docker Images

on:
push:
branches:
- dev
- master
pull_request:
branches:
- dev

jobs:
build_images:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: setup poetry and install dependencies
run: |
python -m pip install --upgrade pip poetry
python -m poetry lock --no-update
python -m poetry install --with lint --no-ansi --no-interaction
working-directory: backend/df_designer

- name: Create new project
run: python -m poetry run dflowd init --destination ../../ --no-input --overwrite-if-exists
working-directory: backend/df_designer

- name: Build Frontend
run: docker build -f Dockerfile --build-arg PROJECT_DIR=df_designer_project --target=frontend-builder .

- name: Build backend & run app
run: docker build -f Dockerfile --build-arg PROJECT_DIR=df_designer_project --target=runtime .
140 changes: 140 additions & 0 deletions .github/workflows/e2e_test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
name: test app

on:
push:
branches:
- dev
- master
pull_request:
branches:
- dev

jobs:

build_frontend:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

- name: Install bun
run: npm install -g bun

- name: Install frontend dependencies
run: bun install
working-directory: frontend

- name: Build frontend
run: bun run build
working-directory: frontend

- run: touch file1.txt

- name: Archive frontend dist
uses: actions/upload-artifact@v4
with:
name: frontend-dist
path: frontend/dist


build_backend:
runs-on: ubuntu-latest
needs: build_frontend
steps:
- uses: actions/checkout@v4

- name: Create dist directory
run: mkdir -p frontend/dist

- name: Download frontend dist
uses: actions/download-artifact@v4
with:
name: frontend-dist
path: frontend/dist

- name: copy static files
run: |
cp -r frontend/dist/. backend/df_designer/app/static/

- name: set up python 3.10
uses: actions/setup-python@v5
with:
python-version: '3.10'

- name: setup poetry and install dependencies
run: |
python -m pip install --upgrade pip poetry
python -m poetry lock --no-update
python -m poetry install --with lint --no-interaction
working-directory: backend/df_designer

- name: build wheel
run: python -m poetry build
working-directory: backend/df_designer

- name: Archive backend dist
uses: actions/upload-artifact@v4
with:
name: backend-dist
path: backend/df_designer/dist


run_app:
runs-on: ubuntu-latest
needs: build_backend
steps:
- uses: actions/checkout@v4

- name: set up python 3.10
uses: actions/setup-python@v5
with:
python-version: '3.10'

- name: setup dflowd poetry and install dependencies
run: |
python -m pip install --upgrade pip poetry
python -m poetry lock --no-update
python -m poetry install --with lint --no-interaction
working-directory: backend/df_designer

- name: Create new project
run: |
python -m poetry run dflowd init --destination ../../ --no-input --overwrite-if-exists
working-directory: backend/df_designer

- name: Create dist directory
run: mkdir -p backend/df_designer/dist

- name: Download backend dist
uses: actions/download-artifact@v4
with:
name: backend-dist
path: backend/df_designer/dist

- name: setup project poetry and install dependencies
run: |
python -m pip install --upgrade pip poetry
python -m poetry lock --no-update
python -m poetry install --no-interaction
working-directory: df_designer_project

- name: Run back & front
run: |
python -m poetry run dflowd run_backend &
sleep 10
working-directory: df_designer_project

- name: Install bun
run: npm install -g bun

- name: Install frontend dependencies
run: bun install
working-directory: frontend

- name: Cypress run
uses: cypress-io/github-action@v6
with:
install-command: bun add cypress
command: bun run e2e:chrome
working-directory: frontend
browser: chrome
16 changes: 16 additions & 0 deletions .github/workflows/frontend_check.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# name: front check

# on: [push, pull_request]

# jobs:
# build_and_check_frontend:
# runs-on: ubuntu-latest
# steps:
# - uses: actions/checkout@v4

# - name: Install bun
# run: npm install -g bun

# - name: Install dependencies
# run: bun install
# working-directory: frontend
7 changes: 6 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -251,4 +251,9 @@ cython_debug/

./flows.json
*.sqlite
my_project
my_project
/test-results/
/playwright-report/
/blob-report/
/playwright/.cache/
df_designer_project
Loading
Loading