-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
check in Qdrant to registry #61
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,150 @@ | ||
## Qdrant Connector Overview | ||
|
||
The Qdrant Data Connector allows for connecting to a Qdrant instance giving you an instant GraphQL API that supports querying on top of your data. This uses the [Typescript Data Connector SDK](https://github.com/hasura/ndc-sdk-typescript) and implements the [Data Connector Spec](https://github.com/hasura/ndc-spec). | ||
|
||
In order to use this connector you will need a Qdrant database setup. This connector currently only supports querying. | ||
|
||
## Before you get started | ||
|
||
It is recommended that you: | ||
|
||
* Setup a [Qdrant Database instance](https://qdrant.tech/) | ||
* Install the [Hasura3 CLI](https://github.com/hasura/v3-cli#hasura-v3-cli) | ||
* Log in via the CLI | ||
* Install the [connector plugin](https://hasura.io/docs/latest/hasura-cli/connector-plugin/) | ||
* Install [VSCode](https://code.visualstudio.com) | ||
* Install the [Hasura VSCode Extension](https://marketplace.visualstudio.com/items?itemName=HasuraHQ.hasura) | ||
|
||
## Deployment For Hasura Users | ||
|
||
To deploy a connector and use it in a Hasura V3 project, follow these steps: | ||
|
||
1. Create a Hasura V3 project (or use an existing project) | ||
|
||
2. Generate a configuration file for your Qdrant Database, there are 2 ways to get the configuration file. | ||
|
||
First you'll need to clone [this repo](https://github.com/hasura/ndc-qdrant), and run ```npm install``` | ||
i. The easiest way to generate a configuration file is to run the generate-config script using ts-node. | ||
|
||
When running this script specify: | ||
|
||
--url The URL where Qdrant is hosted | ||
|
||
--key The API key for connecting to the Qdrant Client. | ||
|
||
--output The name of the file to store the configuration in | ||
|
||
Example Usage: | ||
|
||
```ts-node generate-config --url https://qdrant-url --key QdrantApiKey --output config.json``` | ||
|
||
ii. You can also run the connector in configuration mode and generate the config file using CURL. | ||
|
||
```ts-node ./src/index.ts configuration serve``` | ||
|
||
You can then send a CURL request specifying the qdrant_url and qdrant_api_key to get the configuration file. | ||
|
||
Example: | ||
|
||
```curl -X POST -H "Content-Type: application/json" -d '{"qdrant_url": "https://link-to-qdrant.cloud.qdrant.io", "qdrant_api_key": "QdrantApiKey"}' http://localhost:9100 > config.json``` | ||
|
||
3. Once you have a configuration file, you can deploy the connector onto Hasura Cloud | ||
|
||
Ensure you are logged in to Hasura CLI | ||
|
||
```hasura3 cloud login --pat 'YOUR-HASURA-TOKEN'``` | ||
|
||
From there, you can deploy the connector: | ||
|
||
```hasura3 connector create qdrant:v1 --github-repo-url https://github.com/hasura/ndc-qdrant/tree/main --config-file ./config.json``` | ||
|
||
## Usage | ||
|
||
Once your connector is deployed, you can get the URL of the connector using: | ||
```hasura3 connector list``` | ||
|
||
``` | ||
my-cool-connector:v1 https://connector-9XXX7-hyc5v23h6a-ue.a.run.app active | ||
``` | ||
|
||
In order to use the connector once deployed you will first want to reference the connector in your project metadata: | ||
|
||
```yaml | ||
kind: "AuthConfig" | ||
allowRoleEmulationFor: "admin" | ||
webhook: | ||
mode: "POST" | ||
webhookUrl: "https://auth.pro.hasura.io/webhook/ddn?role=admin" | ||
--- | ||
kind: DataConnector | ||
version: v1 | ||
definition: | ||
name: my_connector | ||
url: | ||
singleUrl: 'https://connector-9XXX7-hyc5v23h6a-ue.a.run.app' | ||
``` | ||
|
||
If you have the [Hasura VSCode Extension](https://marketplace.visualstudio.com/items?itemName=HasuraHQ.hasura) installed | ||
you can run the following code actions: | ||
|
||
* `Hasura: Refresh data source` | ||
* `Hasura: Track all collections / functions ...` | ||
|
||
This will integrate your connector into your Hasura project which can then be deployed or updated using the Hasura3 CLI: | ||
|
||
``` | ||
hasura3 cloud build create --project-id my-project-id --metadata-file metadata.hml | ||
``` | ||
|
||
## Service Authentication | ||
|
||
If you don't wish to have your connector publically accessible then you must set a service token by specifying the `SERVICE_TOKEN_SECRET` environment variable when creating your connector: | ||
|
||
* `--env SERVICE_TOKEN_SECRET=SUPER_SECRET_TOKEN_XXX123` | ||
|
||
Your Hasura project metadata must then set a matching bearer token: | ||
|
||
```yaml | ||
kind: DataConnector | ||
version: v1 | ||
definition: | ||
name: my_connector | ||
url: | ||
singleUrl: 'https://connector-9XXX7-hyc5v23h6a-ue.a.run.app' | ||
headers: | ||
Authorization: | ||
value: "Bearer SUPER_SECRET_TOKEN_XXX123" | ||
``` | ||
|
||
While you can specify the token inline as above, it is recommended to use the Hasura secrets functionality for this purpose: | ||
|
||
```yaml | ||
kind: DataConnector | ||
version: v1 | ||
definition: | ||
name: my_connector | ||
url: | ||
singleUrl: 'https://connector-9XXX7-hyc5v23h6a-ue.a.run.app' | ||
headers: | ||
Authorization: | ||
valueFromSecret: BEARER_TOKEN_SECRET | ||
``` | ||
|
||
NOTE: This secret should contain the `Bearer ` prefix. | ||
|
||
|
||
## Default Collection Parameters: | ||
|
||
You'll find that each collection on your graph is parameterized, and that you have the ability to pass in the following parameters as collection arguments: | ||
|
||
vector | ||
positive | ||
negative | ||
|
||
These will allow you to perform vector searches, or to get recommendations. | ||
|
||
You can pass in a search vector to the vector parameter, which is a flat list of floats. This will typically be the output from some embedding model, and it will return results ordered by closest match. You'll likely want to ensure that you are passing a limit on all your queries. | ||
|
||
You can also pass in an array of ID's to the positive and negative parameters to provide example data-points. This is an easy way to get recommendations without having to manage or deal with passing around entire vectors. If you know the ID of some positive and negative data-points, you can simply pass the ID's. You must provide at least 1 positive example when using this. You can provide a list of positive examples, a list of positive and a list of negative, but you cannot provide only a list of negative examples. | ||
|
||
You can read more about these parameters [here](https://qdrant.tech/documentation/concepts/search/) |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
{ | ||
"overview":{ | ||
"description":"The Qdrant Data Connector allows for connecting to a Qdrant instance giving you an instant GraphQL API on top of your Qdrant Vector Data.", | ||
"title":"Qdrant Data Connector", | ||
"logo":"logo.png", | ||
"tags":["database"], | ||
"latest_version":"v0.1" | ||
}, | ||
"author":{ | ||
"support_email":"[email protected]", | ||
"homepage":"https://hasura.io", | ||
"name":"Hasura" | ||
}, | ||
"is_verified":true, | ||
"is_hosted_by_hasura":false, | ||
"source_code":{ | ||
"is_open_source":true, | ||
"repository":"https://github.com/hasura/ndc-qdrant", | ||
"version":[ | ||
{ | ||
"tag":"v0.1", | ||
"hash":"eec3339a83a9cdea6ff9f4a76f2ed38849a1bff4", | ||
"is_verified": true | ||
} | ||
] | ||
} | ||
} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@daniel-chambers what do you reckon about this workflow instead of running the config server? What's your perspective from having worked with the Dynamo config?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the configuration server is necessary from a tooling perspective. My understanding was that our tooling is envisioned to use configuration servers to do things like:
However, I'm not sure how exactly the tooling is supposed to run a configuration server etc... that's why I assumed in my DynamoDB connector docker container that the config server port would be exposed and that you passed the standard command line args to the docker container to start either the main server or the config server.
I'm not against having some other way like this
generate-config
tool as well, though.