Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Qdrant tests on CI #1337

Open
burnash opened this issue May 7, 2024 · 2 comments · Fixed by #1526
Open

Fix Qdrant tests on CI #1337

burnash opened this issue May 7, 2024 · 2 comments · Fixed by #1526
Assignees
Labels
bug Something isn't working

Comments

@burnash
Copy link
Collaborator

burnash commented May 7, 2024

Background

#998 makes qdrant working again but leaves a few things to be fixed. Qdrant is querying state in a wrong way (relies on order which is not there). Some tests are incorrect. Also we should test embedded qdrant.

Note we use following hack to go around ordering:

    def _create_point_no_vector(self, obj: Dict[str, Any], collection_name: str) -> None:
        """Inserts a point into a Qdrant collection without a vector.

        Args:
            obj (Dict[str, Any]): The arbitrary data to be inserted as payload.
            collection_name (str): The name of the collection to insert the point into.
        """
        # we want decreased ids because the point scroll functions orders by id ASC
        # so we want newest first
        id_ = 2**64 - int(precise_time() * 10**6)
        self.db_client.upsert(
            collection_name,
            points=[
                models.PointStruct(
                    id=id_,
                    payload=obj,
                    vector={},
                )
            ],
        )

Task

    • decide if the keep the hack above or to enable ordering of records by created_at. ordering needs additional indexes, and indexes work only on server version of qdrant. so not sure it is worth the effort.
    • add qdrant to all the tests where weaviate is added
    • write additional tests that make sure we get the newest schema and right state when we have state that is not committed (no corresponding entry in the _dlt_loads collection)
    • fix test_pipeline_replace which is loading same data twice
    • enable local qdrant tests on CI (we have workflow with local tests)
    • enable server qdrant tests
@burnash burnash added the bug Something isn't working label May 7, 2024
@sh-rp
Copy link
Collaborator

sh-rp commented May 8, 2024

NOTE: To re-enable the failing tests remove them from SKIPPED_TESTS in docs/tools/prepare_examples_tests.py

@rudolfix rudolfix moved this from Todo to Planned in dlt core library Jun 2, 2024
@rudolfix rudolfix moved this from Planned to In Progress in dlt core library Jun 3, 2024
@steinitzu steinitzu assigned steinitzu and unassigned sh-rp Jun 27, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in dlt core library Jul 4, 2024
@rudolfix rudolfix reopened this Jul 10, 2024
@rudolfix
Copy link
Collaborator

rudolfix commented Jul 10, 2024

@steinitzu #1545 is fixing some of issues above. IMO one still remain:

  • test_pipeline_replace looks like loading same data twice so is not testing replace disposition
    pls do another PR for those two

@rudolfix rudolfix moved this from Done to In Progress in dlt core library Jul 10, 2024
@rudolfix rudolfix moved this from In Progress to Planned in dlt core library Jul 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Todo
Development

Successfully merging a pull request may close this issue.

4 participants