Are the benchmarks fair by not including gevent?

[vecchp]

Hi I recently have been looking at various Django Frameworks and noticed django-ninja. I discovered your benchmarking suite which is great! I did notice however that gevent was left out which basically kneecaps any wsgi server as you aren't allowed to leverage green-threads. I modified your docker-compose commands as such to add the following.

version: '3.7'

x-common: &common
  build: .
  ports:
      - "8000:8000"


# Services not meant to be started all at once
# see the run_test.py for details

services:
  flask_marshmallow_gunicorn:
    <<: *common
    working_dir: /app_flask_marshmallow
    command: gunicorn main:app --bind 0.0.0.0:8000 --workers ${WORKERS}
  
  drf_gunicorn:
    <<: *common
    working_dir: /app_drf
    command: gunicorn drf.wsgi:application --bind 0.0.0.0:8000 --workers ${WORKERS}

  drf_gunicorn_gevent:
    <<: *common
    working_dir: /app_drf
    command: gunicorn drf.wsgi:application --worker-class gevent --bind 0.0.0.0:8000 --workers ${WORKERS}

  ninja_gunicorn:
    <<: *common
    working_dir: /app_ninja
    command: gunicorn djninja.wsgi:application --bind 0.0.0.0:8000 --workers ${WORKERS}

  ninja_gunicorn_gevent:
    <<: *common
    working_dir: /app_ninja
    command: gunicorn djninja.wsgi:application --worker-class gevent --bind 0.0.0.0:8000 --workers ${WORKERS}

  ninja_gunicorn_uvcorn:
    <<: *common
    working_dir: /app_ninja
    command: gunicorn djninja.asgi:application --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000 --workers ${WORKERS}

  ninja_uvicorn:
    <<: *common
    working_dir: /app_ninja
    command: uvicorn djninja.asgi:application --host 0.0.0.0 --workers ${WORKERS}

  fastapi_gunicorn:
    <<: *common
    working_dir: /app_fastapi
    command: gunicorn main:app --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000 --workers ${WORKERS}

  fastapi_uvicorn:
    <<: *common
    working_dir: /app_fastapi
    command: uvicorn main:app --host 0.0.0.0 --workers ${WORKERS}

  network_service:
    build: .
    command: python network_service.py > /dev/null

When running the benchmarks it appears that the gaps aren't as big after. Perhaps I missed something here so maybe you can share some thoughts!

Async IO Job

x86-64 Python 3.11

Framework	1	2	4	8
drf_gunicorn	9.74	19.34	37.50	72.01
drf_gunicorn_gevent	297.32	340.59	337.24	334.75
ninja_gunicorn	9.73	19.33	37.48	71.71
ninja_gunicorn_gevent	171.20	276.39	412.06	452.38
ninja_gunicorn_uvcorn	296.92	319.25	338.25	328.51
ninja_uvicorn	301.76	322.52	348.39	323.84

x86-64 PyPy 3.10

Framework	1	2	4	8
drf_gunicorn	9.18	18.41	35.32	65.46
drf_gunicorn_gevent	156.33	240.95	242.46	266.09
ninja_gunicorn	9.24	18.01	35.27	65.19
ninja_gunicorn_gevent	96.36	161.58	193.13	213.53
ninja_gunicorn_uvcorn	126.48	190.78	241.23	268.06
ninja_uvicorn	133.44	190.50	269.07	261.84

Sync IO Job - monkeypatched

Replaces Ninja's async view with

@api.get("/iojob")
def iojob(request):  # <-- Remove the "async" keyword
    # No need for an async context manager, just use requests.get
    response = requests.get('http://network_service:8000/job')
    data = response.text
    return {"success": True}

x86-64 Python 3.11

Framework	1	2	4	8
drf_gunicorn	9.74	19.31	38.31	72.09
drf_gunicorn_gevent	270.34	345.49	335.87	328.36
ninja_gunicorn	9.77	19.31	38.32	72.47
ninja_gunicorn_gevent	293.26	345.83	350.65	344.91
ninja_gunicorn_uvcorn	300.17	312.18	334.53	336.92
ninja_uvicorn	282.68	344.62	336.81	326.91

Modified Test Suite: https://github.com/vecchp/django-ninja-benchmarks

[vitalik]

Hi @vecchp

well I followed the recomened by docs setups (which seems 90% of what users do by default)

gevent indeed can bring boost, but the last time I used it on a project (to be fair it was 3-5 years ago) I experienced weird django database "connection gone" issue when running for long time (seems due to some inconsistency in threadlocals in asgiref and gevent)

[vecchp]

Not certain but you may have run into an issue with psycopg2 not being compatible with gevent out of the box. It needed to be patched with psycogreen. I'm sure there may have been a similar issues with async at the time since you had to either use aiopg (a wrapper around psycopg2 async mode) or asyncpg.

Psycopg3 now resolves this out of the box with gevent and asyncio. Might be worth a shot including gevent in these benchmarks with that in mind. Shouldn't be too hard to reuse the work in my branch once you are done with the pydantic upgrade you mentioned in the other issue.