Regarding the evaluation results of synapse not reaching the level of the paper, #78
Comments
|
I am experiencing the same issue. I was able to get results consistent with the paper for the ACDC dataset, but not for the Synapse dataset using the provided weights. Were you able to resolve the issue? I got an average Dice score of 0.8687, with the spleen Dice score dropping by 5.8 (95.77 to 89.98) compared to the results in the paper. Mean_Dice dsc:nan img0001.nii.gz img0002.nii.gz img0003.nii.gz img0004.nii.gz img0008.nii.gz img0022.nii.gz img0025.nii.gz img0029.nii.gz img0032.nii.gz img0035.nii.gz img0036.nii.gz img0038.nii.gz Mean_Dice dsc:0.8686665385765965 |
|
This is the output from running the evaluation script as specified in the README. /home/sntanzi/miniconda3/envs/unetr_pp/lib/python3.8/site-packages/scipy/init.py:138: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3) stage: 0 stage: 1 I am using stage 1 from these plans I am using data from this folder: /home/sntanzi/lustre/unetr_plus_plus/DATASET_Synapse/unetr_pp_raw/unetr_pp_raw_data/Task02_Synapse/Task002_Synapse/unetr_pp_Data_plans_v2.1 |
|
I meet the same question |
Hello, can I ask you which code you got this result by running? I am running the evaluation code given by the author and do not have such detailed results |
Hello, I run the unetr_pp/inference/predict_simple.py and unetr_pp/inference_synapse.py files. For your information, you can look at the evaluation script of the BraTs dataset 'run_evaluation_tumor.sh', it's evaluated in the same manner. |
|
@mathntanzi0 Thank you very much for your reply. I have finished this work some time ago and got a result similar to yours. |
First of all, thank you for your excellent work. I have followed your steps to train the model on the synapse dataset and then evaluated it, but the result I obtained was only 0.8638526351861252. I understand that there might be some discrepancy, but the result is significantly lower than expected. Are there some detailed settings that have not been made public? If possible, please inform me. Below is part of the content from my evaluation file
"mean": {
"1": {
"Accuracy": 0.998657093644419,
"Dice": 0.8479975310456979,
"False Discovery Rate": 0.20889159480622957,
"False Negative Rate": 0.03930279119994839,
"False Omission Rate": 0.00013919367103899154,
"False Positive Rate": 0.0012081377903464345,
"Jaccard": 0.7627434971988629,
"Negative Predictive Value": 0.999860806328961,
"Precision": 0.7911084051937705,
"Recall": 0.9606972088000516,
"Total Positives Reference": 115561.58333333333,
"Total Positives Test": 158912.5,
"True Negative Rate": 0.9987918622096537
},
"10": {
"Accuracy": 0.9998240917377723,
"Dice": 0.7663670871785314,
"False Discovery Rate": 0.18122477756846053,
"False Negative Rate": 0.24555561145875057,
"False Omission Rate": 9.999149942430817e-05,
"False Positive Rate": 7.599327323995486e-05,
"Jaccard": 0.6353516399531766,
"Negative Predictive Value": 0.9999000085005757,
"Precision": 0.8187752224315394,
"Recall": 0.7544443885412496,
"Total Positives Reference": 15261.75,
"Total Positives Test": 14635.25,
"True Negative Rate": 0.9999240067267597
},
"11": {
"Accuracy": 0.9995659818911203,
"Dice": 0.8200419504751024,
"False Discovery Rate": 0.13332194919158954,
"False Negative Rate": 0.21263713167420595,
"False Omission Rate": 0.00029348285479180886,
"False Positive Rate": 0.0001410687372232211,
"Jaccard": 0.6991175380144169,
"Negative Predictive Value": 0.9997065171452082,
"Precision": 0.8666780508084103,
"Recall": 0.7873628683257939,
"Total Positives Reference": 42044.916666666664,
"Total Positives Test": 36767.416666666664,
"True Negative Rate": 0.9998589312627768
},
"12": {
"Accuracy": 0.9999655788657217,
"Dice": 0.7341160324680612,
"False Discovery Rate": 0.25930794548195624,
"False Negative Rate": 0.2562280618432938,
"False Omission Rate": 1.5434259001815032e-05,
"False Positive Rate": 1.8989249479221353e-05,
"Jaccard": 0.5834289373180755,
"Negative Predictive Value": 0.9999845657409981,
"Precision": 0.7406920545180439,
"Recall": 0.7437719381567063,
"Total Positives Reference": 2093.3333333333335,
"Total Positives Test": 2165.3333333333335,
"True Negative Rate": 0.9999810107505209
},
"13": {
"Accuracy": 0.999962717385909,
"Dice": 0.7390265292858933,
"False Discovery Rate": 0.2627450982806246,
"False Negative Rate": 0.2436348630660118,
"False Omission Rate": 1.7573534143480063e-05,
"False Positive Rate": 1.97118810078926e-05,
"Jaccard": 0.5929833058276595,
"Negative Predictive Value": 0.9999824264658566,
"Precision": 0.7372549017193754,
"Recall": 0.7563651369339883,
"Total Positives Reference": 2450.75,
"Total Positives Test": 2519.1666666666665,
"True Negative Rate": 0.9999802881189921
},
"2": {
"Accuracy": 0.9997822312377777,
"Dice": 0.8733454879439214,
"False Discovery Rate": 0.11384628133799028,
"False Negative Rate": 0.13691050112803535,
"False Omission Rate": 0.00014495952815063974,
"False Positive Rate": 7.329979330860288e-05,
"Jaccard": 0.8322771568821644,
"Negative Predictive Value": 0.9998550404718495,
"Precision": 0.8861537186620096,
"Recall": 0.8630894988719647,
"Total Positives Reference": 73265.0,
"Total Positives Test": 70832.75,
"True Negative Rate": 0.9999267002066913
},
"3": {
"Accuracy": 0.9997748789639517,
"Dice": 0.8753217646049097,
"False Discovery Rate": 0.11798308329616229,
"False Negative Rate": 0.13106948876535726,
"False Omission Rate": 0.00012613355501851127,
"False Positive Rate": 9.943509030216346e-05,
"Jaccard": 0.8376290873872634,
"Negative Predictive Value": 0.9998738664449814,
"Precision": 0.8820169167038375,
"Recall": 0.8689305112346428,
"Total Positives Reference": 73942.5,
"Total Positives Test": 73059.66666666667,
"True Negative Rate": 0.9999005649096979
},
"4": {
"Accuracy": 0.9999027573103243,
"Dice": 0.7019076972177962,
"False Discovery Rate": 0.2391666702619987,
"False Negative Rate": 0.2607527924828959,
"False Omission Rate": 5.433529634845218e-05,
"False Positive Rate": 4.2933821660271855e-05,
"Jaccard": 0.5923179343938021,
"Negative Predictive Value": 0.9999456647036515,
"Precision": 0.7608333297380012,
"Recall": 0.7392472075171039,
"Total Positives Reference": 9870.416666666666,
"Total Positives Test": 9144.75,
"True Negative Rate": 0.9999570661783398
},
"5": {
"Accuracy": 0.9999284802484277,
"Dice": 0.7859994912096306,
"False Discovery Rate": 0.19657391831445872,
"False Negative Rate": 0.19981177576084005,
"False Omission Rate": 3.762295571920989e-05,
"False Positive Rate": 3.390873478323342e-05,
"Jaccard": 0.6532811515462691,
"Negative Predictive Value": 0.9999623770442807,
"Precision": 0.8034260816855413,
"Recall": 0.8001882242391599,
"Total Positives Reference": 6148.5,
"Total Positives Test": 6028.5,
"True Negative Rate": 0.9999660912652168
},
"6": {
"Accuracy": 0.998165835752708,
"Dice": 0.9606366802950771,
"False Discovery Rate": 0.03209041883876928,
"False Negative Rate": 0.045915344085172394,
"False Omission Rate": 0.0011099276976853196,
"False Positive Rate": 0.0007701325262892874,
"Jaccard": 0.9247390511599275,
"Negative Predictive Value": 0.9988900723023146,
"Precision": 0.9679095811612308,
"Recall": 0.9540846559148276,
"Total Positives Reference": 813503.4166666666,
"Total Positives Test": 796908.4166666666,
"True Negative Rate": 0.9992298674737107
},
"7": {
"Accuracy": 0.9978665676434897,
"Dice": 0.8213783935282041,
"False Discovery Rate": 0.07830232643992856,
"False Negative Rate": 0.23743359521999277,
"False Omission Rate": 0.0018321548191929991,
"False Positive Rate": 0.0003127595002620313,
"Jaccard": 0.7279004760411034,
"Negative Predictive Value": 0.998167845180807,
"Precision": 0.9216976735600716,
"Recall": 0.7625664047800073,
"Total Positives Reference": 220097.5,
"Total Positives Test": 165129.41666666666,
"True Negative Rate": 0.999687240499738
},
"8": {
"Accuracy": 0.9998454878927551,
"Dice": 0.9260722616470306,
"False Discovery Rate": 0.05992965043327159,
"False Negative Rate": 0.08569103776204541,
"False Omission Rate": 8.78194983011542e-05,
"False Positive Rate": 6.686607078647729e-05,
"Jaccard": 0.8628836471486713,
"Negative Predictive Value": 0.9999121805016987,
"Precision": 0.9400703495667283,
"Recall": 0.9143089622379544,
"Total Positives Reference": 36828.0,
"Total Positives Test": 36154.583333333336,
"True Negative Rate": 0.9999331339292136
},
"9": {
"Accuracy": 0.9997237096245818,
"Dice": 0.8893857244491098,
"False Discovery Rate": 0.0882442086885407,
"False Negative Rate": 0.13016862494041498,
"False Omission Rate": 0.0001706856755630291,
"False Positive Rate": 0.00010597618760058158,
"Jaccard": 0.8023176955212987,
"Negative Predictive Value": 0.9998293143244371,
"Precision": 0.9117557913114593,
"Recall": 0.8698313750595851,
"Total Positives Reference": 46719.333333333336,
"Total Positives Test": 44452.083333333336,
"True Negative Rate": 0.9998940238123993
}
}
The text was updated successfully, but these errors were encountered: