Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: rewrite cron to use async system #2407

Merged
merged 15 commits into from
Aug 23, 2024
Merged

feat: rewrite cron to use async system #2407

merged 15 commits into from
Aug 23, 2024

Conversation

safeer
Copy link
Contributor

@safeer safeer commented Aug 16, 2024

This refactor removes the cron job system's state management, cron execution, and hashring management, in favor of the async call system.

Data tables changed as follows:

  • cron_jobs is still used to maintain the job list
    • Removes state, adds last_execution::timestampz and last_async_call_id::bigint
  • async_calls is inserted to by the cron system
    • cron_jobs.last_async_call_id = async_calls.id
    • Adds the notion of a cron origin in async_calls.origin

After a deployment, all valid unscheduled cron jobs are scheduled; a row is added to async_calls with a pending state and scheduled_at is set to the job's next execution time. The corresponding row in cron_jobs is also updated with the scheduled async call, the computed next execution time, and the inserted async call ID.

On completion of a cron async call, the next execution of that job is scheduled. Effectively, every cron job will have exactly one scheduled execution.

Closes #2197

@ftl-robot ftl-robot mentioned this pull request Aug 16, 2024
@safeer safeer changed the title feat: cron using async feat: rewrite cron to use async system Aug 20, 2024
Justfile Outdated Show resolved Hide resolved
@safeer safeer added the run-all A PR with this label will run the full set of CI jobs in the PR rather than in the merge queue label Aug 21, 2024
@safeer safeer force-pushed the saf/cron-async branch 2 times, most recently from 4a36b47 to 27c5c20 Compare August 22, 2024 19:18
@safeer safeer marked this pull request as ready for review August 22, 2024 19:25
@safeer safeer requested review from a team, jonathanj-square and matt2e and removed request for a team August 22, 2024 19:25
@safeer safeer force-pushed the saf/cron-async branch 2 times, most recently from c78af2e to 7810fc8 Compare August 22, 2024 23:04
@safeer safeer requested a review from wesbillman August 23, 2024 17:08
Copy link
Collaborator

@wesbillman wesbillman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code LGTM but I don't have a ton of context in this area so it's probably best to get another set of eyes.

backend/controller/cronjobs/cronjobs_integration_test.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@alecthomas alecthomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work @safeer , it feels really good to get this done.

backend/controller/cronjobs/cronjobs.go Outdated Show resolved Hide resolved
backend/controller/cronjobs/cronjobs.go Show resolved Hide resolved
backend/controller/cronjobs/cronjobs.go Outdated Show resolved Hide resolved
FROM cron_jobs j
INNER JOIN deployments d on j.deployment_id = d.id
WHERE j.key = sqlc.arg('key')::cron_job_key
FOR UPDATE SKIP LOCKED;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want FOR UPDATE SKIP LOCKED here? Above it makes sense to me, but in this case it will not return a cron job even if it exists, in the case when it is locked.

Copy link
Contributor Author

@safeer safeer Aug 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetCronJobByKey is used to schedule the next execution of a completed job. Theoretically, there could be a race condition with another controller that's scheduling all cron jobs (via GetUnscheduledCronJobs), and two dupe jobs will be scheduled, so the lock is needed here to deconflict

backend/controller/cronjobs/cronjobs.go Outdated Show resolved Hide resolved
backend/controller/cronjobs/sql/queries.sql Show resolved Hide resolved
backend/controller/sql/async_queries.sql Show resolved Hide resolved
sqlc.yaml Outdated Show resolved Hide resolved
@safeer safeer force-pushed the saf/cron-async branch 2 times, most recently from fafcba4 to cfe16aa Compare August 23, 2024 21:02
@safeer safeer added this pull request to the merge queue Aug 23, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 23, 2024
@safeer safeer force-pushed the saf/cron-async branch 2 times, most recently from 4d38d19 to fe429b8 Compare August 23, 2024 21:45
also made AsyncOriginCron type safe
@safeer safeer added this pull request to the merge queue Aug 23, 2024
Merged via the queue into main with commit 8ce9faf Aug 23, 2024
74 checks passed
@safeer safeer deleted the saf/cron-async branch August 23, 2024 23:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run-all A PR with this label will run the full set of CI jobs in the PR rather than in the merge queue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Convert cron jobs to async calls
3 participants