Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update photo's caption #24

Merged
merged 10 commits into from
Oct 29, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,14 @@ save_it-*.tar
# Temporary files, for example, from tests.
/tmp/
.DS_Store
data
dev.sh
start.sh
run.sh
nohup.out

# data
_local
data

# scripts
_local*
_dev*
_stag*
_prod*
nohup.out
44 changes: 38 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

A telegram bot can Save photos and Search photos

## Features

- [x] Save photos via a link
- [x] Search photos using semantic search
- [x] Find similar photos by photo
Expand All @@ -27,13 +29,8 @@ messages:

```
/search cat

/search dog

/search girl

/similar photo

/similar photo
```

Expand All @@ -58,10 +55,45 @@ mix deps.get
```

```sh
# run
# start typesense
docker compose up
```

```sh
# modify env
export TELEGRAM_BOT_TOKEN=
export TYPESENSE_URL=
export TYPESENSE_API_KEY=

iex -S mix run --no-halt
```

Pro Tips: create shell script for fast run app

1. touch start.sh

```sh
#!/bin/sh

export TELEGRAM_BOT_TOKEN=<YOUR_TELEGRAM_BOT_TOKEN>

export TYPESENSE_URL=<YOUR_TYPESENSE_URL>
export TYPESENSE_API_KEY=<YOUR_TYPESENSE_API_KEY>

export GOOGLE_OAUTH_CLIENT_ID=<YOUR_GOOGLE_OAUTH_CLIENT_ID>
export GOOGLE_OAUTH_CLIENT_SECRET=<YOUR_GOOGLE_OAUTH_CLIENT_SECRET>

iex -S mix run --no-halt
```
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Enhance security guidance for configuration management.

The current setup instructions expose sensitive configuration details. Consider these security improvements:

  1. Add a warning about keeping start.sh private and not committing it to version control
  2. Recommend using a .env file instead of direct exports
  3. Add guidance about secure token management

Here's a suggested improvement:

 Pro Tips: create shell script for fast run app

+⚠️ Security Notice:
+- Never commit the following files to version control
+- Store these files securely and restrict access
+
+1. Create a .env file
+
+```env
+TELEGRAM_BOT_TOKEN=your_token_here
+TYPESENSE_URL=your_url_here
+TYPESENSE_API_KEY=your_key_here
+GOOGLE_OAUTH_CLIENT_ID=your_client_id_here
+GOOGLE_OAUTH_CLIENT_SECRET=your_client_secret_here
+```
+
+2. Create start.sh
+
 ```sh
 #!/bin/sh

-export TELEGRAM_BOT_TOKEN=<YOUR_TELEGRAM_BOT_TOKEN>
-export TYPESENSE_URL=<YOUR_TYPESENSE_URL>
-export TYPESENSE_API_KEY=<YOUR_TYPESENSE_API_KEY>
-export GOOGLE_OAUTH_CLIENT_ID=<YOUR_GOOGLE_OAUTH_CLIENT_ID>
-export GOOGLE_OAUTH_CLIENT_SECRET=<YOUR_GOOGLE_OAUTH_CLIENT_SECRET>
+# Load environment variables from .env file
+source .env

 iex -S mix run --no-halt

<!-- This is an auto-generated comment by CodeRabbit -->


2. execute permission

```sh
chmod +x start.sh
```

3. run

```sh
./start.sh
```
2 changes: 0 additions & 2 deletions config/runtime.exs
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,3 @@ config :save_it, :typesense_api_key, System.get_env("TYPESENSE_API_KEY", "xyz")
# optional
config :save_it, :google_oauth_client_id, System.get_env("GOOGLE_OAUTH_CLIENT_ID")
config :save_it, :google_oauth_client_secret, System.get_env("GOOGLE_OAUTH_CLIENT_SECRET")

config :save_it, :web_url, System.get_env("WEB_URL", "http://localhost:4000")
File renamed without changes.
9 changes: 9 additions & 0 deletions docs/dev-logs/2024-10-25.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# 2024-10-25

## Req call typesense API alway :timeout, but typesense was updated.

```elixir
** (MatchError) no match of right hand side value: {:error, %Req.TransportError{reason: :timeout}}
(save_it 0.2.0-rc.1) lib/migration/typesense.ex:11: Migration.Typesense.create_collection!/1
priv/typesense/reset.exs:3: (file)
```
Empty file added docs/dev/readme.md
Empty file.
14 changes: 14 additions & 0 deletions docs/dev/typesense.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Typesense

## Typesense API Errors

```
# 400 Bad Request - The request could not be understood due to malformed syntax.
# 401 Unauthorized - Your API key is wrong.
# 404 Not Found - The requested resource is not found.
# 409 Conflict - When a resource already exists.
# 422 Unprocessable Entity - Request is well-formed, but cannot be processed.
# 503 Service Unavailable - We’re temporarily offline. Please try again later.
```

docs: https://typesense.org/docs/27.1/api/api-errors.html#api-errors
42 changes: 42 additions & 0 deletions lib/migration/typesense.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
defmodule Migration.Typesense do
def create_collection!(schema) do
req = build_request("/collections")
{:ok, res} = Req.post(req, json: schema)

res.body
end

def delete_collection!(collection_name) do
req = build_request("/collections/#{collection_name}")
{:ok, res} = Req.delete(req)

res.body
end

def list_collections() do
req = build_request("/collections")
{:ok, res} = Req.get(req)

res.body
end
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Improve error handling and add documentation for public API functions.

The public functions need several improvements for production readiness:

  1. Add proper error handling for HTTP requests
  2. Add response validation
  3. Add documentation and type specs

Here's a suggested implementation for better error handling and documentation:

+  @doc """
+  Creates a new collection in Typesense with the given schema.
+
+  ## Parameters
+    - schema: Map containing the collection schema
+
+  ## Returns
+    - {:ok, response_body} on success
+    - {:error, reason} on failure
+
+  ## Raises
+    - RuntimeError if the API request fails
+  """
+  @spec create_collection!(map()) :: map() | no_return()
   def create_collection!(schema) do
     req = build_request("/collections")
-    {:ok, res} = Req.post(req, json: schema)
-
-    res.body
+    case Req.post(req, json: schema) do
+      {:ok, %{status: status} = res} when status in 200..299 ->
+        res.body
+      {:ok, %{status: status, body: body}} ->
+        raise "Failed to create collection. Status: #{status}, Body: #{inspect(body)}"
+      {:error, error} ->
+        raise "Failed to create collection: #{inspect(error)}"
+    end
   end

Similar improvements should be applied to delete_collection!/1 and list_collections/0.

Committable suggestion was skipped due to low confidence.


defp get_env() do
url = Application.fetch_env!(:save_it, :typesense_url)
api_key = Application.fetch_env!(:save_it, :typesense_api_key)

{url, api_key}
end

defp build_request(path) do
{url, api_key} = get_env()

Req.new(
base_url: url,
url: path,
headers: [
{"Content-Type", "application/json"},
{"X-TYPESENSE-API-KEY", api_key}
]
)
end
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Enhance configuration handling and request building.

Consider the following improvements for the helper functions:

  1. Add URL validation
  2. Use module attributes for constants
  3. Add documentation for maintainability

Here's a suggested implementation:

+  @content_type "application/json"
+  @typesense_config_keys [:typesense_url, :typesense_api_key]
+
+  @doc false
   defp get_env() do
-    url = Application.fetch_env!(:save_it, :typesense_url)
-    api_key = Application.fetch_env!(:save_it, :typesense_api_key)
+    url = 
+      :save_it
+      |> Application.fetch_env!(:typesense_url)
+      |> validate_url!()
+    
+    api_key = Application.fetch_env!(:save_it, :typesense_api_key)
 
     {url, api_key}
   end
 
+  @doc false
   defp build_request(path) do
     {url, api_key} = get_env()
 
     Req.new(
       base_url: url,
       url: path,
       headers: [
-        {"Content-Type", "application/json"},
+        {"Content-Type", @content_type},
         {"X-TYPESENSE-API-KEY", api_key}
       ]
     )
   end
+
+  defp validate_url!(url) do
+    uri = URI.parse(url)
+    unless uri.scheme in ["http", "https"] and uri.host do
+      raise ArgumentError, "Invalid Typesense URL: #{url}"
+    end
+    url
+  end

Committable suggestion was skipped due to low confidence.

end
35 changes: 35 additions & 0 deletions lib/migration/typesense/photo.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
defmodule Migration.Typesense.Photo do
alias Migration.Typesense

@photos_schema %{
"name" => "photos",
"fields" => [
# image: base64 encoded string
%{"name" => "image", "type" => "image", "store" => false},
%{
"name" => "image_embedding",
"type" => "float[]",
"embed" => %{
"from" => ["image"],
"model_config" => %{
"model_name" => "ts/clip-vit-b-p32"
}
}
},
%{"name" => "caption", "type" => "string", "optional" => true},
%{"name" => "file_id", "type" => "string"},
%{"name" => "belongs_to_id", "type" => "string"},
%{"name" => "inserted_at", "type" => "int64"}
],
"default_sorting_field" => "inserted_at"
}

def create_collection!() do
Typesense.create_collection!(@photos_schema)
end

def reset!() do
Typesense.delete_collection!(@photos_schema["name"])
Typesense.create_collection!(@photos_schema)
end
end
Loading
Loading