Zero Shot Instance Segmentation #57
Conversation
There was a problem hiding this comment.
Overall looks good. Looks like this is an example showing how to sample vids, run models, and add preds to video frames. Consider using our video utils in fiftyone.utils.video if possible/appropriate.
This may be out of scope of the example but it felt like a tiny wrapup (however brief) assessment at the end touching on "how things turned out" might be nice. But that may be out of scope here.
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "dataset.ensure_frames()\n", |
There was a problem hiding this comment.
It looks like this ensure_frames() call isn't necessary until later; conceptually seems a bit out of place here as you are only sampling the vids here
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Download the dataset from [this zip file](https://drive.google.com/file/d/18okj067dIOXgeHu47bsKAtaq56jRWisQ/view) to a folder `taranaki` and unzip it." |
There was a problem hiding this comment.
Clicking this said I need to request permission, maybe note it if expected
| "source": [ | ||
| "We will use the following libraries:\n", | ||
| "- [FiftyOne](https://github.com/voxel51/fiftyone) to organize our dataset and visualize the results\n", | ||
| "- [transformers](https://huggingface.co/docs/transformers/index) from Hugging Face to load and run inference with [OWL-ViT](https://huggingface.co/docs/transformers/model_doc/owlvit) for zero-shot object detection\n", |
There was a problem hiding this comment.
Probably capitalize 'transformers'
| "mp4_files = glob(\"taranaki/data/*\")\n", | ||
| "\n", | ||
| "### Create PNGs for each frame\n", | ||
| "for mf in mp4_files:\n", |
There was a problem hiding this comment.
Any particular reason not to use fouv.sample_video(s), fouv.transform_video(s), etc here?
There was a problem hiding this comment.
Maybe too heavy weight, but maybe another option, use to_frames() and convert/analyze video-as-frames dataset (optionally going back to video)
| "outputs": [], | ||
| "source": [ | ||
| "def add_sam_tracks(dataset):\n", | ||
| " for sample in dataset[100:].iter_samples(autosave=True, progress=True):\n", |
Shows how to