int8 model, how to rescale the predicted bounding boxes

[hu8813]

hello,
i had a yolov8 model (.pt) which i wanted to convert to .onnx in int8

since yolo's export function lacks support of onnx int8, i exported it to .tflite with int8 parameter, also using my data.yaml as parameter
then i had a int8.tflite model in saved_model

using tf2onnx.convert i converted .tflite to .onnx (int8)

the problem during Inference is, the bounding boxes need to be rescaled, but i don't know how to find out those x_scale and y_scale parameters

following code works for rescaling, but bounding boxes are not exactly same as the float model,

i manually made out those scaling factors below (i.e. *= 2.65 or *= 3.25) by training many different values, and seeing which seems best.

my question is. how can i find out those values? i resize images to 320x320, original frame size is 2880x1860, pls see following file to see preprocess and postprocess;
https://gist.github.com/hu8813/7699b346e02951dff871083cf6248232

the code i used for Inference:

import cv2
from prepare import YOLOv8

model_path = "./best_int8.onnx"
image_path = "./test.jpg"

frame = cv2.imread(image_path)

detector = YOLOv8(model_path, conf_thres=0.05, iou_thres=0.1)
boxes, scores, class_ids = detector(frame)

for box, score, class_id in zip(boxes, scores, class_ids):
    x_min, y_min, x_max, y_max = box

    x_min *= 2.65
    y_min *= 310
    x_max *= 3.25
    y_max *= 330
    width = (x_max - x_min) / 2
    height = (y_max - y_min) / 2
    x_center = (x_min + x_max) / 2
    y_center = (y_min + y_max) / 2

Here are x_min, y_min, x_max, y_max Values in Float and int8 models:

Float Model:
Frame 1
384.78772 2.7558293 405.63232 2.8917012
413.13806 2.801322 432.97253 2.9235559
356.4293 2.7064931 376.5004 2.8407447

Frame 2
448.05814 2.7975142 509.1641 3.0998619

Int8 Model:
Frame 1
1154.5808 882.2938 1213.4985 925.41516
1241.0344 897.523 1291.4307 936.5822
1068.7329 866.2054 1128.1746 908.5512

Frame 2
1345.1605 893.75183 1526.5665 992.9845

[fatcat-z]

Could you please to take a look at our test file to see if it helps?