Revamp worker server architecture

[cysjonathan]

Coursemology will enqueue reevaluation jobs as a lower priority, but since the concurrent jobs are limited, it's still possible that these 1000 jobs hog all "available seats" and take up to 5 minutes to terminate, causing other higher priority jobs that are enqueued after all seats are taken to wait for the workers to be free

.e.g. a clubhouse restaurant has 30 tables and serves club members first. But if there is no queue and 1000 non-members arrive, the first 30 are seated and will take 5 minutes to clear. During this 5 minutes, if a club member arrives, they will go to the front of the queue, but still have to wait for the seats to empty.

Possible solutions

1. Have some reserved workers only for high priority jobs, so high priority jobs can always run if there are high-priority workers available.
   Note: this introduces complexity into the worker configuration
2. Once we split frontend and backend, we can refactor jobs to be spun up on demand using container services (e.g. AWS ECS). This allows us to scale concurrency to any arbitrary value we want.