Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change: Fix endless loop in NVDApi #1073

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Conversation

n-thumann
Copy link
Member

@n-thumann n-thumann commented Dec 10, 2024

What

This PR fixes the endless loop of requests.
Note: The diff looks more complicated than it is. It's actually just changing indentation + inverting an if-clause + the actual fix. I recommend checking the commits separately.

Why

Because we should stop requesting results when there aren't any more available.

You can reproduce the issue with:

import logging
from pontos.nvd.cve import CVEApi
import asyncio
from datetime import datetime

logging.basicConfig(level=logging.INFO)


async def foo():
    async with CVEApi() as api:
        cves = await api.cves(
            last_modified_start_date=datetime(2024, 11, 19),
            last_modified_end_date=datetime(2024, 12, 9),
            start_index=244000,
        )

        async for _ in cves:
            pass


asyncio.run(foo())

It is expected that this will send two requests: One returning 2.000 results and another one returning 1.625 results.

Pontos however keeps sending requests, because self._downloaded_result (2.000 + 1.625) < self._current_request_results (247.625), even though the API doesn't return any more:

➜  pontos git:(main) ✗ poetry run python foo.py
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=244000&resultsPerPage=2000 "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=246000&resultsPerPage=2000 "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=247435&resultsPerPage=1435 "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=247435&resultsPerPage=0 "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=247435&resultsPerPage=0 "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=247435&resultsPerPage=0 "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: GET https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-11-19T00%3A00%3A00&lastModEndDate=2024-12-09T00%3A00%3A00&startIndex=247435&resultsPerPage=0 "HTTP/1.1 200 OK"
[...]

How

Tested using the Python snippet above and using CVE DB Updater of vt-cve-library.

References

Checklist

  • Tests

Copy link

Conventional Commits Report

Type Number
Changed 2

🚀 Conventional commits found.

Copy link

codecov bot commented Dec 10, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.91%. Comparing base (c638f14) to head (b4b06f3).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1073   +/-   ##
=======================================
  Coverage   89.91%   89.91%           
=======================================
  Files         108      108           
  Lines        7176     7178    +2     
  Branches      811      812    +1     
=======================================
+ Hits         6452     6454    +2     
  Misses        521      521           
  Partials      203      203           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@n-thumann n-thumann changed the title Fix endless loop nvd api Change: Fix endless loop in NVDApi Dec 10, 2024
@n-thumann n-thumann marked this pull request as ready for review December 11, 2024 15:58
@n-thumann n-thumann requested a review from a team as a code owner December 11, 2024 15:58
@n-thumann n-thumann added the make release To trigger GitHub release action. label Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
make release To trigger GitHub release action.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant