v1.12.5

What's Changed

• Fix/lang code nlu evals by @kmaphoenix in #255

Full Changelog: 1.12.4...1.12.5

v1.12.4

What's Changed

• Fix/tool call quality bug by @kmaphoenix in #253

Full Changelog: 1.12.3...1.12.4

v1.12.3

What's Changed

• Feature/conversation rebase by @kmaphoenix in #240
• Fix/default creds inheritance by @kmaphoenix in #246
• Prevent IndexError in collect_playbook_responses when not in playbook by @SeanScripts in #244
• feat: add support for flow invoke; clean up creds passing in evals by @kmaphoenix in #248
• Fix/optional tool call metrics evals by @kmaphoenix in #250
• Fix/support lang code conversation by @kmaphoenix in #251

Full Changelog: 1.12.2...1.12.3

v1.12.2

Enhancements

• Added support for language_code on all applicable methods in Flows class
• Added support for parameters when using the Datastore Evaluations class and notebook
• Added support for Playbook Versions
• New notebook to check status of datastores and search for Datastore IDs, Doc IDs, and URLs
• Added helper methods for Search to make listing urls / doc ids / documents much easier users

Bug Fix

• Fixed bug in CopyUtil class that was causing the create_entity_type method to fail
• Fixed a bug in Dataframe Functions which was causing scopes to not be inherited properly
• Fixed new Vertex Agents Evals notebook links for Github and GCP workbench launching to point to correct location

What's Changed

• fix: add support for language_code on applicable methods by @kmaphoenix in #222
• fix: update copy_util to resolve bug issue 192 by @my3sons in #205
• Feat/parameter support datastore evals by @kmaphoenix in #225
• feat: add support for playbook versions by @kmaphoenix in #226
• Fix/scopes dataframe functions by @kmaphoenix in #228
• Update vertex_agents_evals.ipynb by @YuncongZhou in #231
• Feature/datastoreindexurls by @agutta in #235
• Feat/add vais search methods by @kmaphoenix in #237
• chore: update notebook to use latest scrapi code by @kmaphoenix in #238

New Contributors

• @my3sons made their first contribution in #205
• @YuncongZhou made their first contribution in #231
• @agutta made their first contribution in #235

Full Changelog: 1.12.1...1.12.2

v1.12.1

Bug

• Patch to require google-cloud-aiplatform as part of the setuptools
• The lack of google-cloud-aiplatform in setuptools was causing import errors in some classes that rely on vertexai as an import

Full Changelog: 1.12.0...1.12.1

v1.12.0

New Features

Evaluations are here! 🎉

What are Evaluations? 📐 📈

We know that building an Agent is only part of the journey.
Understanding how that Agent responds to real-world queries is a key indicator of how it will perform in Production.
Running evaluations, or "evals", allows Agent developers to quickly identify "losses", or areas of opportunities for improving Agent design.

Evals can provide answers to questions like:

• What is the current performance baseline for my Agent?
• How is my Agent performing after the most recent changes?
• If I switch to a new LLM, how does that change my Agent's performance?

Evaluation Toolsets in SCRAPI 🛠️🐍

For this latest release, we have included 2 specific Eval setups for developers to use with Agent Builder and Dialogflow CX Agents.

1. DataStore Evaluations
2. Multi-turn, Multi-Agent w/ Tool Calling Evaluations

These are offered as two distinct evaluations toolsets because of a few reasons:

• They support different build architectures in DFCX vs. Agent Builder
• They support different metrics based on the task you are trying to evaluate
• They support different tool calling setups: Native DataStores vs. arbitrary custom tools

Metrics by Toolset. 📏

The following metrics are currently supported for each toolset.
Additional metrics will be added over time to support various other evaluation needs.

• DataStore Evaluations
  • Url Match
  • Context Recall
  • Faithfulness
  • Answer Correctness
  • RougeL
• Multi-Turn, Multi-Agent w/ Tool Callling Evaluations
  • Semantic Similarity
  • Exact Match Tool Quality

Getting Started with Evaluations 🏁

1. Start by choosing your Eval toolset based on the Agent architecture you are evaluating
   • DataStore Evaluations
   • Multi-turn, Multi-Agent w/ Tool Calling Evaluations
2. Build an Evaluation Dataset. You can find detailed information about the dataset formats in each of the toolset instructions
3. Run your evals!

Example Eval Setup for Multi-Turn, Mutli-Agent w/ Tools

import pandas as pd
from dfcx_scrapi.tools.evaluations import Evaluations
from dfcx_scrapi.tools.evaluations import DataLoader

data = DataLoader()

INPUT_SCHEMA_REQUIRED_COLUMNS = ['eval_id', 'action_id', 'action_type', 'action_input', 'action_input_parameters', 'tool_action', 'notes']

sample_df = pd.DataFrame(columns=INPUT_SCHEMA_REQUIRED_COLUMNS)

sample_df.loc[0] = ["travel-ai-001", 1, "User Utterance", "Paris", "", "", ""]
sample_df.loc[1] = ["travel-ai-001", 2, "Playbook Invocation", "Travel Inspiration", "", "", ""]
sample_df.loc[2] = ["travel-ai-001", 3, "Agent Response", "Paris is a beautiful city! Here are a few things you might enjoy doing there:\n\nVisit the Eiffel Tower\nTake a walk along the Champs-Élysées\nVisit the Louvre Museum\nSee the Arc de Triomphe\nTake a boat ride on the Seine River", "", "", ""]

sample_df = data.from_dataframe(sample_df)
agent_id = "projects/your-project/locations/us-central1/agents/11111-2222-33333-44444" # Example Agent
evals = Evaluations(agent_id, metrics=["response_similarity", "tool_call_quality"])
eval_results = evals.run_query_and_eval(sample_df.head(10))

What's Changed

• Feat/evaluations by @kmaphoenix in #217
• Feat/evals notebook by @kmaphoenix in #218

Full Changelog: 1.11.2...1.12.0

v1.11.2

What's Changed

• fix: improve markdown line handling by @kmaphoenix in #213
• Add build_search_engine_proto() to Engines by @rantman in #215

New Contributors

• @rantman made their first contribution in #215

Full Changelog: 1.11.1...1.11.2

v1.11.1

What's Changed

• FR - allow users to pass endUserMetadata as an optional in detect_intent and autoeval colab by @jkshj21 in #210
• FR-186 - Export results into multiple mode by @Naveenkm13 in #209
• Add creds to the constructors by @MRyderOC in #204
• Feat/playbook instructions parsing by @kmaphoenix in #212

New Contributors

• @Naveenkm13 made their first contribution in #209

Full Changelog: 1.11.0...1.11.1

v1.11.0

New Features

Agent Builder SDK is here! 🎉

The long awaited Agent Builder SDK is now live in SCRAPI, so you can now build your Agents, Tools, and Examples directly from code without interacting with the UI.

In just a few lines of code, you can have an Agent up and running!

from dfcx_scrapi.core.agents import Agents
from dfcx_scrapi.core.playbooks import Playbooks

a = Agents()
agent = a.create_agent(
    project_id="my-project",
    display_name="SCRAPI PLAYBOOKS",
    gcp_region="us-east1",
    playbook_agent=True
)

p = Playbooks(agent.name)
playbooks_map = p.get_playbooks_map(agent.name, reverse=True)
p.update_playbook(
    playbooks_map["Default Generative Agent"],
    goal="You are a friendly Tesla service center agent.\nYour job is to help users book appointments and answer any questions they have.",
    instructions=["Greet the user.", "Answer any questions the have to the best of your ability."]
    )

We've released a GenAI Agents 101 Notebook to help you get started with using the new API endpoints.

As usual, please file bugs/FRs as you see them!

Enhancements

• New Vertex AI Conversation evaluation notebook added for evaluating your Vertex RAG setups
• New Vertex Search notebook added for quickly building and testing your Vertex Data Stores
• Added a new flag in Sessions.detect_intent to enable data store connection signals

Bug Fix

• Fixed a bug in FulfillmentBuilder that was causing bool values not to be accepted
• Pinned linter.yml file and requirements to specific pylint version

What's Changed

• Bool flag to Sessions.detect_intent to enbale data store connection signals by @Mrpatekful in #177
• Feature/test cases util by @jkshj21 in #164
• feat: Vertex Search config and example notebook by @kmaphoenix in #180
• feat: Add 1-click buttons to colabs by @kmaphoenix in #181
• Vertex AI Conversation automatic evaluation tooling by @Mrpatekful in #183
• Update linter.yml with certain pylint version by @totemws in #191
• Create README.md by @totemws in #189
• Bug/fulfillment builder param preset bug by @MRyderOC in #188
• Feature/generative playbooks by @kmaphoenix in #198
• feat: add simply playbook agent creation; update tests by @kmaphoenix in #199
• fix: tools imports now that rollout is complete by @kmaphoenix in #200
• Fix/update playbook mask by @kmaphoenix in #201
• Feat/simple tool creation by @kmaphoenix in #202
• feat: add sample genai_agent notebook and tool by @kmaphoenix in #203

New Contributors

• @totemws made their first contribution in #191

Full Changelog: 1.10.5...1.11.0

v1.10.5

What's Changed

• Fixed duplicated detect intent call if parameters are provided. by @Mrpatekful in #175
• BUGFIX: Set Region When ID is Missing by @kmaphoenix in #176

New Contributors

• @Mrpatekful made their first contribution in #175

Full Changelog: 1.10.4...1.10.5