Skip to content

Commit

Permalink
[pre-commit.ci] Add auto fixes from pre-commit.com hooks
Browse files Browse the repository at this point in the history
for more information, see https://pre-commit.ci
  • Loading branch information
pre-commit-ci[bot] committed Oct 30, 2024
1 parent a8b2baf commit 867c9f4
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 7 deletions.
6 changes: 3 additions & 3 deletions docs/source/safety_and_evaluation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Synthetic Dataset Generation
The evaluation utilizes a synthetic dataset of 1000 queries, carefully structured to test different aspects of the system:

- **Regular Situations (80%)**: Divided equally among three levels of detail:

- Low: 1-2 sentences, brief queries with minimal context
- Medium: 2-3 sentences with moderate background information
- High: Detailed paragraph with extensive context
Expand Down Expand Up @@ -150,7 +150,7 @@ Performance Metrics
RAGAS Metrics By Category
^^^^^^^^^^^^^^^^^^^^^^^^^^

Note: These metrics were obtained using a synthetic dataset specifically generated from services in the Greater Toronto Area (GTA).
Note: These metrics were obtained using a synthetic dataset specifically generated from services in the Greater Toronto Area (GTA).
The RAG system evaluated used specialized prompts that differ marginally from the current system.

.. list-table::
Expand Down Expand Up @@ -264,4 +264,4 @@ To enable re-ranking in your API calls, simply set the `rerank` parameter to `tr
"longitude": -79.3832,
"radius": 5000,
"rerank": true
}
}
11 changes: 7 additions & 4 deletions eval/collect_rag_outputs.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
import asyncio
import json
import logging
from typing import Dict, List, Any
from typing import Dict, Any

import aiohttp
from tqdm.asyncio import tqdm_asyncio
Expand All @@ -27,7 +27,8 @@ async def fetch_recommendation(
"query": query["query"],
"answer": result["message"],
"context": [
str(service["id"]) for service in (result.get("services", []) or [])
str(service["id"])
for service in (result.get("services", []) or [])
],
"ground_truth": query["context"],
}
Expand All @@ -39,7 +40,9 @@ async def fetch_recommendation(
return None


async def process_samples(samples_file: str, output_file: str, batch_size: int = 5) -> None:
async def process_samples(
samples_file: str, output_file: str, batch_size: int = 5
) -> None:
"""Process samples in batches and save results."""
# Load samples
with open(samples_file, "r") as f:
Expand Down Expand Up @@ -90,4 +93,4 @@ def main() -> None:


if __name__ == "__main__":
main()
main()

0 comments on commit 867c9f4

Please sign in to comment.