Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mongodb backend can't handle collections with alot of documents #173

Open
Halk0 opened this issue Jan 13, 2023 · 1 comment · May be fixed by #174
Open

Mongodb backend can't handle collections with alot of documents #173

Halk0 opened this issue Jan 13, 2023 · 1 comment · May be fixed by #174

Comments

@Halk0
Copy link

Halk0 commented Jan 13, 2023

I encountered a bug where querying all documents from collections with lots of documents.
I identified that the problem was how mongodb_filte.py handled filtering. Currently filtering
is done using multiple aggregate queries to mongodb using document ID's that matched
previous filters, this can lead to a situation where a database contains lots of documents
the hard limit of 16 megabyte queries is exceeded and mongodb cannot process the filter. My
changes make it so that instead of multiple filters and queries a single filter is generated
based on the request made and that is passed to mongodb. This way the 16 megabyte query
size limit should never be triggered, and mongodb can process the filter internally.

@Halk0 Halk0 linked a pull request Jan 13, 2023 that will close this issue
@Halk0
Copy link
Author

Halk0 commented Jan 13, 2023

To reproduce this bug you would need a mongodb containing documents, which IDs size alone when combined
into a single filter would amount to larger than a 16 megabyte query. This happens due to generating the filter based
on IDs received from previous queries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant