aoc runner sends requests to advent of code even if solution file does not exist

[i13e]

What I think is happening is even if there is a non-answer provided, the library is still trying to fetch the answer from AOC. However, if the problem isn't solved yet then the runner has nothing to cache, so this behavior repeats. Additionally, if several solutions are missing, this fires off the rate limiter, which quickly maxes out. I believe this behavior shouldn't be the default. Maybe missing solutions, which are already well-handled, should trigger a skipped or error state, and continue instead of trying to fetch a solution. Below is my entry point and console logs:

def mysolve(year: str, day: str, data: str) -> tuple[str, str]:
    ans_1, ans_2 = "", ""
    try:
        mod_name = f"aoc_i13e.aoc{year}.q{day:02d}"
        mod = importlib.import_module(name=mod_name)

        part_1 = getattr(mod, "part_1")
        ans_1 = str(part_1(data))

        part_2 = getattr(mod, "part_2")
        ans_2 = str(part_2(data))
    except (ModuleNotFoundError, AttributeError):
        pass

    return ans_1, ans_2

WARNING:aocd.runner:error submitting - cowardly refusing to submit non-answer: ''
   0.20s   2024/11 - Plutonian Pebbles                          i13e/default   ✔ part a: 194557                             ? part b:  (correct answer unknown)
WARNING:aocd.runner:error submitting - cowardly refusing to submit non-answer: ''
WARNING:aocd.utils:you're being rate-limited - slow down on the requests! see https://github.com/wimglenn/advent-of-code-data/issues/59 (delay=0.16s)
WARNING:aocd.utils:you're being rate-limited - slow down on the requests! see https://github.com/wimglenn/advent-of-code-data/issues/59 (delay=0.32s)
   0.10s   2024/12 - Garden Groups                              i13e/default   ? part a:  (correct answer unknown)          ? part b:  (correct answer unknown)
WARNING:aocd.utils:you're being rate-limited - slow down on the requests! see https://github.com/wimglenn/advent-of-code-data/issues/59 (delay=0.64s)
WARNING:aocd.runner:error submitting - cowardly refusing to submit non-answer: ''
WARNING:aocd.utils:you're being rate-limited - slow down on the requests! see https://github.com/wimglenn/advent-of-code-data/issues/59 (delay=1.28s)
WARNING:aocd.utils:you're being rate-limited - slow down on the requests! see https://github.com/wimglenn/advent-of-code-data/issues/59 (delay=2.56s)
   0.10s   2024/13 - Claw Contraption                           i13e/default   ? part a:  (correct answer unknown)          ? part b:  (correct answer unknown)
WARNING:aocd.runner:error submitting - cowardly refusing to submit non-answer: ''
WARNING:aocd.utils:you're being rate-limited - slow down on the requests! see https://github.com/wimglenn/advent-of-code-data/issues/59 (delay=5.12s)
   0.10s   2024/14 - Restroom Redoubt                           i13e/default   ? part a:  (correct answer unknown)          ? part b:  (correct answer unknown)

[wimglenn]

You're right about the underlying cause of the rate-limiter tripping.

The problem is this plugin is indicating that it is succeeding (by returning a value at all) and the runner will assume you are returning at least one valid answer.

You have an easy fix available which is to just remove the try/except from the entry point. The runner already has special handling for when the entry point raises exception, and it won't go looking for pre-existing answers in that case.

So I'd recommend to change your function like this:

def mysolve(year: str, day: str, data: str) -> tuple[str, str]:
    mod_name = f"aoc_i13e.aoc{year}.q{day:02d}"
    mod = importlib.import_module(name=mod_name)

    part_1 = getattr(mod, "part_1")
    ans_1 = str(part_1(data))

    part_2 = getattr(mod, "part_2", lambda data: "")
    ans_2 = str(part_2(data))

    return ans_1, ans_2

There will still be a WARNING about part B being unfinished when you're part-way through a puzzle (2014/11 here), but you shouldn't trip the rate limiter for the subsequent unsolved puzzles (2014/12+). It only goes looking for answers on the web page if it thinks it has some answer to compare with.

Still, the suggestion to avoid hitting the server by skipping when an entry-point is "successfully" returning non-answers is a good one, that has been implemented in #157. I won't release this until January, but feel free to install directly from main branch and let me know if that works as expected!

[i13e]

Thanks for your suggestion! Makes a lot of sense why it works that way. Love your project btw, I think it abstracts all the right parts of AOC so people can just focus on writing code.

[wimglenn]

I appreciate the kind words.