-
Notifications
You must be signed in to change notification settings - Fork 6
Contributing
We have adopted Black as our code formatting tool. Line length is 120.
The easiest way to use is is to install the pre-commit hook:
- Install pre-commit: pip install pre-commit
- Execute pre-commit install to install git hooks in your .git/ directory.
Another good way to have it working is to set it up in your code editor. Pycharm, for example, has good support for this.
The pre-commit is not mandatory but Continuous Integration will check if the formatting is respected!
For the Python backend, we use the Django builtin test framework. Tests can be executed with
docker-compose exec iaso ./manage.py test
The few translation for the Django side (login and reset password email etc..) are separated from the test. We only translate the template for now not the python code (string on model or admin).
When modifying or adding new strings to translate, use the following command to regenerate the translations:
manage.py makemessages --locale=fr --extension txt --extension html
This will update hat/locale/fr/LC_MESSAGES/django.po with the new strings to
translate.
After updating it with the translation you need to following command to have them reflected in the interface:
manage.py compilemessages
In development the servers will reload when they detect a file change, either in Python or Javascript. If you need reloading for the bluesquare-components code, see the "Live Bluesquare Components" section.
If you need to restart everything
docker-compose stop && docker-compose start
If you encounter problems, you can try to rebuild everything from scratch.
# kill containers
docker-compose kill
# remove `iaso` container
docker-compose rm -f iaso
# build containers
docker-compose build
# start-up containers
docker-compose up
To run a Jupyter Notebook, just copy the env variable from runaisasdev.sh, activate the virtualenv and run
python manage.py shell_plus --notebook
See also How to deploy wiki
Django 3 requires version 2+ of the gdal library. Sadly, Beanstalk is based on Amazon Linux that can only install gdal 1 from the epel repository. To be able to use gdal 2, first identify the AMI of the Elastic Beanstalk EC2 server. In EC2, launch a new instance based on that AMI. In the instance, run (based on https://stackoverflow.com/questions/49637407/deploying-a-geodjango-application-on-aws-elastic-beanstalk and adapted to use /usr instead of /usr/local): (For Amazon Linux 2, use geos-3.5.2)
wget http://download.osgeo.org/geos/geos-3.4.2.tar.bz2 tar xjf geos-3.4.2.tar.bz2 cd geos-3.4.2 ./configure --prefix=/usr make sudo make install cd ..
wget http://download.osgeo.org/proj/proj-4.9.1.tar.gz wget http://download.osgeo.org/proj/proj-datumgrid-1.5.tar.gz tar xzf proj-4.9.1.tar.gz cd proj-4.9.1/nad tar xzf ../../proj-datumgrid-1.5.tar.gz cd .. ./configure --prefix=/usr make sudo make install cd ..
sudo yum-config-manager --enable epel sudo yum -y update
sudo yum install make automake gcc gcc-c++ libcurl-devel proj-devel geos-devel autoconf automake gdal cd /tmp
curl -L http://download.osgeo.org/gdal/2.2.3/gdal-2.2.3.tar.gz | tar zxf -cd gdal-2.2.3/ ./configure --prefix=/usr --without-python
make -j4 sudo make install
sudo ldconfig
Then go to Actions -> Image -> Create Image When it's ready, go to the Beanstalk Instance Settings and specify the AMI reference of the image we just created.
If you need to test s3 storage in development, you have to:
- Set the AWS_STORAGE_BUCKET_NAME env variable to a bucket created for such tests
- Set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env variables appropriately
- Set the USE_S3 env variable to 'true'
These are actually exactly the same steps we use on AWS.
During local development, by default, the Javascript and CSS will be loaded from
a webpack server with live reloading of the code. To locally test the compiled
version as it is in production ( minified and with the same compilation option).
You can launch docker-compose with the TEST_PROD=true environment variable
set.
e.g TEST_PROD=true docker-compose up
This can be useful to reproduce production only bugs. Please also test with this configuration whenever you modify webpack.prod.js to validate your changes.
Alternatively this can be done outside of docker by running:
-
npm run webpack-prodto do the build - Launching the django server with
TEST_PRODe.g.TEST_PROD=true python manage.py runserver.
Iaso queue certains functions (task) for later execution, so they can run outside an HTTP request. This is used for functions that take a long time to execute so they don't canceled in the middle by a timeout of a connection closed. e.g: bulk import, modifications or export of OrgUnits. Theses are the functions marked by the decorator @task_decorator, when called they get added to a Queue and get executed by a worker.
The logic is based on a fork of the library django-beanstalk-worker from tolomea, please consult it's doc for reference.
In production on AWS, we use Elastic Beanstalk workers which use a SQS queue.
In local development, you can run a worker by using the command:
docker-compose run iaso manage tasks_worker
Alternatively, you can call the url tasks/run_all which will run all the pending tasks in queue.
If you want to develop a new background task, the endpoint /api/copy_version/
is a good example of how to create a task and to plug it to the api.
To call a function with the @task decorator, you need to pass it a User objects, in addition to the other function's arguments, this arg represent which user is launching the task. At execution time the task will receive a iaso.models.Task instance in argument that should be used to report progress. It's mandatory for the function, at the end of a successful execution to call task.report_success() to mark its proper completion.