Can't seem to be able to call actions

[maxesse]

Hi there, I have an OpenAPI spec for Open meteo which i used successfully alongside other tools in the past as an openai assistant, it would retrieve the weather forecast and provide it to the user 100% of the time.

With agents, i tried in every possible way, from explaining how to call the tool, to being descriptive, but it just refuses to call that API. This happens with GPT-4o models. I am really not sure what else can I do to make it realise it's got that action available?
Replies range from 'give me the data and i'll give you the forecast', or if i am more descriptive on how to call the tool, it'll construct the URL so i can go fetch the forecast myself (lol).

This is the spec I'm passing to the actions:

openapi: 3.0.0
servers:
  - url: 'https://api.open-meteo.com'
info:
  title: Open-Meteo APIs
  description: >-
    Open-Meteo offers free weather forecast APIs for open-source developers and
    non-commercial use. No API key is required.
  version: '1.0'
  contact:
    name: Open-Meteo
    url: 'https://open-meteo.com'
  termsOfService: 'https://open-meteo.com/en/features#terms'
paths:
  /v1/forecast:
    get:
      tags:
        - Weather Forecast APIs
      summary: 7 day weather forecast for coordinates
      description: 7 day weather forecast for given latitude and longitude coordinates
      parameters:
        - name: hourly
          in: query
          explode: false
          schema:
            type: array
            items:
              type: string
              enum:
                - temperature_2m
                - relative_humidity_2m
                - precipitation
                - cloud_cover
                - wind_speed_10m
                - wind_direction_10m
                - wind_gusts_10m
                - weather_code
        - name: daily
          in: query
          schema:
            type: array
            items:
              type: string
              enum:
                - temperature_2m_max
                - temperature_2m_min
                - precipitation_sum
                - weather_code
                - sunrise
                - sunset
                - wind_speed_10m_max
                - wind_gusts_10m_max
        - name: latitude
          in: query
          required: true
          description: WGS84 coordinate
          schema:
            type: number
            format: float
        - name: longitude
          in: query
          required: true
          description: WGS84 coordinate
          schema:
            type: number
            format: float
        - name: current_weather
          in: query
          schema:
            type: boolean
        - name: temperature_unit
          in: query
          schema:
            type: string
            default: celsius
            enum:
              - celsius
              - fahrenheit
        - name: timezone
          in: query
          description: Timezone for the response data
          schema:
            type: string
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                type: object
                properties:
                  latitude:
                    type: number
                    example: 52.52
                  longitude:
                    type: number
                    example: 13.419
                  timezone:
                    type: string
                    example: "Europe/Berlin"
                  hourly:
                    $ref: '#/components/schemas/HourlyResponse'
                  daily:
                    $ref: '#/components/schemas/DailyResponse'
                  current_weather:
                    $ref: '#/components/schemas/CurrentWeather'
        '400':
          description: Bad Request
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: boolean
                  reason:
                    type: string
                    example: 'Latitude must be in range of -90 to 90°'
components:
  schemas:
    HourlyResponse:
      type: object
      required:
        - time
      properties:
        time:
          type: array
          items:
            type: string
        temperature_2m:
          type: array
          items:
            type: number
        relative_humidity_2m:
          type: array
          items:
            type: number
        precipitation:
          type: array
          items:
            type: number
        cloud_cover:
          type: array
          items:
            type: number
        wind_speed_10m:
          type: array
          items:
            type: number
        wind_direction_10m:
          type: array
          items:
            type: number
        wind_gusts_10m:
          type: array
          items:
            type: number
        weather_code:
          type: array
          items:
            type: number
    DailyResponse:
      type: object
      required:
        - time
      properties:
        time:
          type: array
          items:
            type: string
        temperature_2m_max:
          type: array
          items:
            type: number
        temperature_2m_min:
          type: array
          items:
            type: number
        precipitation_sum:
          type: array
          items:
            type: number
        weather_code:
          type: array
          items:
            type: number
        sunrise:
          type: array
          items:
            type: string
        sunset:
          type: array
          items:
            type: string
        wind_speed_10m_max:
          type: array
          items:
            type: number
        wind_gusts_10m_max:
          type: array
          items:
            type: number
    CurrentWeather:
      type: object
      required:
        - time
        - temperature
        - wind_speed
        - wind_direction
        - weather_code
      properties:
        time:
          type: string
        temperature:
          type: number
        wind_speed:
          type: number
        wind_direction:
          type: number
        weather_code:
          type: integer

[acehoss]

@maxesse I've had some weird issues with this as well. Do you see Available Actions below the schema box after entering the schema?

I've notice then that if I click Update in the action screen, the action works, but then if I save the agent, it stops working. Then if I just open the action and click Update again but don't save the agent the action still works.

For debugging I just ask the agent "I'm doing some testing, can you make a markdown table with the tools you have available?" and it'll print everything out. I have a few different actions configured, and that helped me notice that saving the agent sometimes wipes out actions. I'd save and ask this and some of the actions would be missing, then go into that action, click update, and ask again, and the action would be back in the list.

[acehoss]

Related to #4990

[maxesse]

Just tried what you said, and indeed it seems to lose the action when saving.

This is before doing anything, then clicking update:

and this is after saving:

[acehoss]

@danny-avila I think this is probably related to my issue #4990

1. create an agent and save it
2. create a custom action, add the OpenAPI schema , click update on the action but don't save the agent
3. ask the agent to make a markdown table of its tools (works well with Claude 3.5 Sonnet) - the new action is shown in the list
4. open the agent builder and save the agent
5. ask again for the tools table, the custom action is no longer present
6. open the agent builder, open the custom action, click update on the action but do not save the agent
7. ask the agent again for the tools table, the custom action is present again

[maxesse]

And indeed if I try a query after updating it works, then I save the agent and it forgets how to call the action.

[danny-avila]

Hi @maxesse following the thread, can you confirm the exact build you're on as well as provider/model combination?

No amount/order of saving the Agent creates the issue for me:

[danny-avila]

Actually I think I just recreated it in the exact order @acehoss gave. thank you for your concise instructions and thorough debugging. I'll patch this up momentarily.

[maxesse]

Ah awesome! I am running the latest main branch right now, if it helps.

[danny-avila]

Closed by #5020