Multi-class training

[bw4sz]

Originally an issue.

"""
Dear all,

This is not an issue reporting but rather a request for guidance pertaining to image sizes, number of classes, data quantity and training on a new multi class model training.

The image files I am using for training are quite large in size ranging from 12563512 pixels (4516 × 2782) to 83678686 (12523 × 6682). Several such image files are used with a single csv file for annotations. There are a total of 2903 annotations across 13 classes.

In general, the code works for me but the predictions are very poor even on the file used for training. The scores for predictions are very low ~0.01 and the bounding boxes don't look right. I am presently unable to train the model for larger epochs due to hardware limitations. Any suggestion this would also be welcome. There is a chance that we might be able to get access to google cola pro or azure sagemaker and that may solve some of these problems.

In general, looking to get an advice on the approach and the architecture. Here is the code.

Dictionary labels are taken from data frame but here is standalone creation
dict_labels = {'Sitaphal': 0, 'Other': 1, 'Mango': 2, 'Lemon': 3, 'Tree': 4, 'Chinch': 5, 'Coconut': 6, 'Jambhul': 7, 'tree': 8, 'Coconut tree': 9, 'chinch': 10, 'coconut': 11, 'w': 12, 'Orange': 13}

Initializing a multi class model
multi_class_m = main.deepforest(num_classes=len(dict_labels), label_dict=dict_labels)
deepforest_release_model = main.deepforest()
deepforest_release_model.use_release()
multi_class_m.model.backbone.load_state_dict(deepforest_release_model.model.backbone.state_dict())
multi_class_m.model.head.regression_head.load_state_dict(deepforest_release_model.model.head.regression_head.state_dict())

Setting configuration for the multi class model
multi_class_m.config["train"]["csv_file"] = get_data("automated_all_annot.csv")
multi_class_m.config["train"]["root_dir"] = os.path.dirname(get_data("automated_all_annot.csv"))
multi_class_m.config["train"]["fast_dev_run"] = False
multi_class_m.config["batch_size"] = 5
multi_class_m.config["train"]['epochs'] = 10
multi_class_m.config["workers"] = 2

Training
multi_class_m.create_trainer(log_every_n_steps=1)
multi_class_m.trainer.fit(multi_class_m)
assert multi_class_m.num_classes == len(dict_labels)

Prediction: score_threshold is kept 0 to see whatever is predicted
test_predict = get_data("AMD-SRD-123456002-PTV-EP01.jpg")
multi_class_m.config["score_thresh"] = 0.0
boxes = multi_class_m.predict_image(path=test_predict, return_plot = True, thickness=10, color = (0, 165, 255))
predictions_df = multi_class_m.predict_image(path=test_predict, return_plot = False, thickness=10, color = (0, 165, 255))

Here is a google drive link to download the folder containing all image files and the annotation csv.

Thank you Ben and all the development team for making this available for us! Much much appreciated. Looking forward to your comments and suggestions.
"""

[bw4sz]

Let's break this down into pieces so I can understand better. Here is a screenshot of an image

1. Use split_raster to cut your raster and annotations into smaller pieces. This will help hardware constraints.
2. Take a single train image and use it both as train and test for a few epochs. Paste results here. This confirms that the model can overfit and learn correctly. It tests the formatting and other issues.

[macdatar]

Hello,
Thank you Ben!
Yes. I had missed split_raster function and was going to try out this exact thing tomorrow. I was wondering how split raster handles a CSV file with multiple images referenced in it but as suggested, for now, I will work with a single image and see if I can overfit it to the data.
Deepforest uses 10 cm resolution with 400x400 px images in training. Our data is at about 2.5 cm resolution so to try and cover approx similar 40x40 m area per split, I will start off with a crop size of 1500 to 1600 px.
Will report the findings tomorrow!

[jyo94]

I have performed both the steps mentioned by @bw4sz , split raster into smaller tiles. Prepared train and test. Trained for few epochs.
##Performed training as mentioned in the documentation,
#initial the model and change the corresponding config file
m = main.deepforest()
m.config['gpus'] = '-1' #move to GPU and use all the GPU resources
m.config["train"]["csv_file"] = train_file_mod
m.config["train"]["root_dir"] = os.path.dirname(train_file_mod)
m.config["score_thresh"] = 0.4
m.config["train"]['epochs'] = 10
m.config["validation"]["csv_file"] = validation_file_mod
m.config["validation"]["root_dir"] = os.path.dirname(validation_file_mod)
#create a pytorch lighting trainer
m.create_trainer()
#load the lastest release model
m.use_release()
m.trainer.fit(m)
#save the prediction result to a prediction folder
save_dir = os.path.join(os.getcwd(),'pred_result')
try:
os.mkdir(save_dir)
except FileExistsError:
pass
results = m.evaluate(train_file_mod, os.path.dirname(train_file_mod), iou_threshold = 0.4, savedir= save_dir)

Now my question is ,

1. How to modify it for a multi-class problem?
2. How to include multiple images (as mentioned by @macdatar ) for training rather than using single image for training.
3. And most important how to save the model trained on our custom data?
4. If saved or checkpoints created, can we use that model and train with new set of images to imporve model performance?

[macdatar]

Hello @jyo94

Please go through the documentation and try to struggle your way through as much as possible before reading below!! :)
Regardless of whether that solves your question, you will find many side things that you were not looking for but are glad that you found. Here are my answers to your questions.

• 1. For this, you can do something like this:

#a dictionary of labels
dict_labels = {'Tree1': 0, 'Tree2': 1, 'Tree3': 2, 'Tree4': 3, 'Tree5': 4, 'Tree6': 5} #just a dictionary of labels
multi_class_m = main.deepforest(num_classes=len(dict_labels), label_dict=dict_labels)
deepforest_release_model = main.deepforest()
deepforest_release_model.use_release()
multi_class_m.model.backbone.load_state_dict(deepforest_release_model.model.backbone.state_dict())
multi_class_m.model.head.regression_head.load_state_dict(deepforest_release_model.model.head.regression_head.state_dict())

• 2. For the second part, do something like below

START A FOR LOOK OVER ALL YOUR XML ANNOTATIONS. INSIDE THE FOR LOOP, DO THE FOLLOWING

#xml file is the file for annotations of one image
single_annotation = utilities.xml_to_annotations(xml)
single_annotation["image_path"] = image_name

single_annotation.to_csv("{}/example.csv".format(basedir), index=False)
single_split_annotation = preprocess.split_raster(path_to_raster=image_file,
annotations_file="{}/example.csv".format(basedir),
base_dir=basedir,
patch_size=1500,
patch_overlap=0.05)

#join individual annotation dfs
combined_split_annotations = pd.concat([combined_split_annotations, single_split_annotation])

END THE FOR LOOP

#write the dataframe of combined (split multiple times on an image) annotations to a csv
combined_split_annotations.to_csv(os.path.join(basedir, "combined_split_annotations.csv"), index=False)

#specify this combined csv file that you wrote as the csv file for training
multi_class_m.config["train"]["csv_file"] = get_data(os.path.join(basedir, "combined_split_annotations.csv"))

• 3. For this, do the following. Use the native function to save the model. Dont use something like pickle to dump the model. It didnt work for me (showing both saving and loading)

multi_class_m.save_model('filename')
loaded_model = main.deepforest.load_from_checkpoint('filename')

• 4. Yes I think you can load a model and train it again.

@bw4sz Please feel free to mention if any of this is not optimal and possible better methods that might be available.
@jyo94 All the best!!!

[jyo94]

Thank you so much @macdatar.