Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change: Fix endless loop in NVDApi #1073

Merged
merged 2 commits into from
Dec 12, 2024
Merged

Change: Fix endless loop in NVDApi #1073

merged 2 commits into from
Dec 12, 2024

Conversation

n-thumann
Copy link
Member

@n-thumann n-thumann commented Dec 10, 2024

What

This PR fixes the endless loop of requests.
Note: The diff looks more complicated than it is. It's actually just changing indentation + inverting an if-clause + the actual fix. I recommend checking the commits separately.

Why

Because we should stop requesting results when there aren't any more available.

You can reproduce the issue with:

import logging
from pontos.nvd.cve import CVEApi
import asyncio
from datetime import datetime

logging.basicConfig(level=logging.INFO)


async def foo():
    async with CVEApi() as api:
        cves = await api.cves(
            last_modified_start_date=datetime(2024, 11, 19),
            last_modified_end_date=datetime(2024, 12, 9),
            start_index=244000,
        )

        async for _ in cves:
            pass


asyncio.run(foo())

It is expected that this will send two requests: One returning 2.000 results and another one returning 1.625 results.

Pontos however keeps sending requests, because self._downloaded_result (2.000 + 1.625) < self._current_request_results (247.625), even though the API doesn't return any more:

➜  pontos git:(main) ✗ poetry run python foo.py
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=244000&resultsPerPage=2000 "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=246000&resultsPerPage=2000 "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=247435&resultsPerPage=1435 "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=247435&resultsPerPage=0 "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=247435&resultsPerPage=0 "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=247435&resultsPerPage=0 "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=247435&resultsPerPage=0 "HTTP/1.1 200 OK"
[...]

How

Tested using the Python snippet above and using CVE DB Updater of vt-cve-library.

References

Jira: https://jira.greenbone.net/browse/VTA-630

Checklist

  • Tests

Copy link

Conventional Commits Report

Type Number
Changed 2

🚀 Conventional commits found.

Copy link

codecov bot commented Dec 10, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.91%. Comparing base (54d814c) to head (32553f4).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1073   +/-   ##
=======================================
  Coverage   89.91%   89.91%           
=======================================
  Files         108      108           
  Lines        7176     7178    +2     
  Branches      811      812    +1     
=======================================
+ Hits         6452     6454    +2     
  Misses        521      521           
  Partials      203      203           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@n-thumann n-thumann changed the title Fix endless loop nvd api Change: Fix endless loop in NVDApi Dec 10, 2024
@n-thumann n-thumann marked this pull request as ready for review December 11, 2024 15:58
@n-thumann n-thumann requested a review from a team as a code owner December 11, 2024 15:58
@n-thumann n-thumann added the make release To trigger GitHub release action. label Dec 11, 2024
@y0urself y0urself enabled auto-merge (squash) December 12, 2024 09:44
@y0urself y0urself force-pushed the fix_endless_loop_nvd_api branch from b4b06f3 to 32553f4 Compare December 12, 2024 09:44
@y0urself y0urself merged commit 7631819 into main Dec 12, 2024
18 checks passed
@y0urself y0urself deleted the fix_endless_loop_nvd_api branch December 12, 2024 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
make release To trigger GitHub release action.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants