The confidence of boxes decreases after the custom dataset is added into training datasets

[zhuitaiyang]

I added private datasets (include person and car) to the training set of YOLO-World, hoping that YOLO-World can maintain zero-shot ability while adapting to private datasets. Through testing on lvis datasets, I found that the index only dropped a little. But when I used model detecting fire just now, I found that, i need to set the confidence level very low（0.00005） to detect it, and the model without private datasets can be detected despite using a higher confidence threshold（0.01）, what is the problem?
with private datasets

without private datasets

[nightwishhhhh]

same here