Extracting actions corresponding to screenshots

[piperwolters]

Hi, thank you for this great work!

I am trying to iterate over each of the demos and extract pairs of screenshots with the corresponding action (if any). I am using your weblinx package, but struggling to understand how to use the Demonstration and Replay objects to do this.

Here is my code, if you have any advice I would appreciate it.

import os
import json
import glob
import weblinx as wl


data_root = './weblinx/'
demos = os.path.join(data_root, 'demonstrations')

for demo in os.listdir(demos):
    print("Processing...", demo)

    demo_data = wl.Demonstration(demo, base_dir=demos)
    task = demo_data.form['description']
    replay = wl.Replay.from_demonstration(demo_data)

    # Filter the turns to keep only the ones that are relevant for the task
    turns = replay.filter_by_intents(
        "click", "textInput", "load", "say", "submit"
    )

    # Only keep the turns that have a good screenshot (i.e., the screenshot is not empty)
    turns = wl.filter_turns(
        turns, lambda t: t.has_screenshot() and t.get_screenshot_status() == "good"
    )

    # Remove chat turns where the speaker is not the navigator (e.g. if you want to train a model to
    # predict the next action)
    turns = wl.filter_turns(
        turns,
        lambda turn: not (
            turn.type == "chat" and turn.get("speaker") != "navigator"
        ),
    )

    actions = replay.list_intents()  # or turns
    screenshots = replay.list_screenshots()

    # TODO: len(actions) != len(screenshots), which makes sense but how to know which 2 screenshots
    # will show the action being taken 

    break

[xhluca]

An action is associated with a turn, you can get the args of the turn via: https://mcgill-nlp.github.io/weblinx/docs/core/#turnargs

You can get the relative path to the screenshot with this: https://mcgill-nlp.github.io/weblinx/docs/core/#turnget_screenshot_path