Add IP ratelimit middleware to JobMgr submit endpoint

[bodom0015]

Problem

We would like to ratelimit the JobMgr submit endpoint, while allowing unrestricted access to the other endpoints

Approach

• split /api/v1/job/submit into its own ingress rule
• apply ip-ratelimit middleware to job submit endpoint (prod + staging)

How to Test

1. Navigate to https://clean.frontend.staging.mmli1.ncsa.illinois.edu/configuration
2. Login using the link at the top-right (to avoid hcaptcha)
3. Paste the following example input and submit a new CLEAN job:

>WP_063460136
MAIPPYPDFRSAAFLRQHLRATMAFYDPVATDASGGQFHFFLDDGTVYNTHTRHLVSATRFVVTHAMLYRTTGEARYQVGMRHALEFLRTAFLDPATGGYAWLIDWQDGRATVQDTTRHCYGMAFVMLAYARAYEAGVPEARVWLAEAFDTAEQHFWQPAAGLYADEASPDWQLTSYRGQNANMHACEAMISAFRATGERRYIERAEQLAQGICQRQAALSDRTHAPAAEGWVWEHFHADWSVDWDYNRHDRSNIFRPWGYQVGHQTEWAKLLLQLDALLPADWHLPCAQRLFDTAVERGWDAEHGGLYYGMAPDGSICDDGKYHWVQAESMAAAAVLAVRTGDARYWQWYDRIWAYCWAHFVDHEHGAWFRILHRDNRNTTREKSNAGKVDYHNMGACYDVLLWALDAPGFSKESRSAALGRP

4. Click the browser's Back button and attempt to submit the same job again (within ~30s of the first submission)
   • You should see that Traefik rejects this request and returns HTTP 429: Too Many Requests
   • You should see that a response header is returned telling the user when they can retry the request again

[bodom0015]

This is the part we would need to adjust if we want to raise or lower the allowed rate of requests 👍

rate = average / period = 4 / 120s = 1 / 30s
which means we currently allow a single request every 30s

See https://doc.traefik.io/traefik/middlewares/http/ratelimit/#configuration-options for further explanation

[bodom0015]

For reference I think that the last time we had an incident, we had ~1200 jobs over 24 hours - this was too many

Some ideas:
24 hours * 60 minutes * 60 seconds = 86400 seconds - maybe this should be our period?

86400 / 1200 = ~72 requests per second - maybe this should be our absolute maximum for rate? But this is the maximum across all users (not per-user, like our limits)

Would 1 request per 120s be an acceptable baseline for the initial release? We can experiment with a longer period and/or changing the number of requests over time as our users' needs become better defined.

We can also add burst to this as well, if we would like to allow a brief (but not sustained) temporary push beyond our limits

[bodom0015]

No need to apply this to JobMgr, as this was only requested by ChemScraper

Will revisit if we see a need in the future 👍