How to improve false positives of background images?

[bw4sz]

The tree and bird backbones are trained on a broad array of images. However, there is no way to anticipate all the varied backgrounds that exist in new imagery. It is very common that we see a single type of background object that consistently appears as a predicted detection. In our everglades bird surveys, we had this issue with lily pads.

This discussion thread is for users to share how they improved the models in this case.

@ethanwhite can you share some general guidelines of images and training approach in this case?

The general idea is to

1. Make predictions on a large set of imagery to capture a range of backgrounds.
2. Optimally find frames with both foreground detections of interest and the nuisance backgrounds
3. Annotate the foreground detections
4. Hold a small number of annotations as an evaluation dataset. This could include completely empty frames.
5. Retrain the baseline model with new sets of annotations, often with a low learning rate
6. Go back to step 1 and repeat until you are satisfied with model performance.

[mdhamer]

Hi all,

I am a waterfowl biologist with the Washington Dept. of Fish and Wildlife. We are hoping to transition from low-level observational aerial surveys of waterfowl and sea birds to aerial image surveys. We have begun to collect imagery and have started to use the Bird Detector model.

The model is predicting false detections to some aquatic vegetation in the background of our images. Below are a few examples.

For reference, the GSD of the imagery is ~1.1 cm and the model was run with default parameters and a score threshold of 0.50.

Thank you,
Matt Hamer

[bw4sz]

Thanks Matt. I made a dummy script here to give you a sense of what happens. Obviously don't annotate images with the boxes overlaid on them, it was just what I had.

>>> annotations.head()
                                          image_path  xmin  ymin  xmax  ymax label
0  2022_03_10_2022ImagingRnD_14_14_42_07336.JPG_S...  1977   776  2002   815  Bird

from deepforest import main
from deepforest import utilities
import os

# Load annotations made in QGIS and finetune the bird model
rgb_file="/Users/benweinstein/Downloads/2022_03_10_2022ImagingRnD_14_14_42_07336.JPG_ST_4.JPG"
shapefile="/Users/benweinstein/Downloads/2022_03_10_2022ImagingRnD_14_14_42_07336.shp"
annotations = utilities.shapefile_to_annotations(rgb=rgb_file, shapefile=shapefile)
annotation_path = "/Users/benweinstein/Downloads/2022_03_10_2022ImagingRnD_14_14_42_07336.csv"
annotations.to_csv(annotation_path)
m = main.deepforest(label_dict={"Bird":0})
m.use_bird_release()
m.config["train"]["root_dir"] = os.path.dirname(annotation_path)
m.config["train"]["csv_file"] = annotation_path
m.config["train"]["epochs"] = 2
m.create_trainer()
m.trainer.fit(m)

# Run some evaluate, repeat above with seperate annotations
#m.config["eval"]["csv_file"] = <file>
#m.evaluate()

# Save model for later
m.save_model("finetuned_model")

# Later, we can load that model and we can use it.
new_model = main.deepforest.load_from_checkpoint("finetuned_model.pt")
new_model.predict_tile(raster_path=rgb_file)

You can then use this model for predictions. Annotate perhaps 10 images and let's see the results.

[ethanwhite]

Updating with info on what we've been doing for the Everglades work. It looks like the current model was built by adding 1000 randomly selected empty crops from old digitized hand counts of birds. We have also in the past included hand selected empty images that included the type of background that we were having issues with, but it looks like those haven't been used since @bw4sz added the randomly selected empty crops.

I've added an issue to try to redo some runs with the hand selected empty crops weecology/EvergladesSpeciesModel#42.

Regardless of whether you're using hand selected or random empty crops the method is to add the empty images to the training dataset:

https://github.com/weecology/EvergladesSpeciesModel/blob/5fb97f9d1811a76c614a646ad1c1e0508c7a0251/everglades_species.py#L97-L107

The label file (inferred_empty_annotations.csv in the linked code) contains 1 row per image with xmin, xmax, ymin, and ymax all set to 0. The label doesn't matter in this case. E.g.,

,image_path,xmin,ymin,xmax,ymax,label,Unnamed: 0
0,3BRamp080_03_09_2022_0.png,0.0,0.0,0.0,0.0,Great Egret,
1,3BRamp080_03_09_2022_1.png,0.0,0.0,0.0,0.0,Great Egret,
2,3BRamp080_03_09_2022_2.png,0.0,0.0,0.0,0.0,Great Egret,
3,3BRamp080_03_09_2022_3.png,0.0,0.0,0.0,0.0,Great Egret,

[mdhamer]

Thank you for documenting the retraining process above.

I was able to retrain using images with background types that were causing false detections, this greatly reduced the number of false detections from sea grass.

Here are a few examples before and after retraining:

[bw4sz]

Great! Any advice for future users, anything about the workflow and docs that could have helped you get to this result more easily. No thing is too small to help reduce user friction.