Stress Test Contradictions

[jbabcanec]

I have done some stress testing with the engine and wanted to reveal a couple things. I first constructed an artificial bene generator which pulls from a list of CMS's ICD10 list. This was refined to use combinations of ICDs from this whitepaper:

https://pubmed.ncbi.nlm.nih.gov/35601156/

I used most frequent ICD chapters and combos in generating artificial benes with varying amounts of ICD10s and demographic factors which worked decently well. I was able to test on the order of ~100k examples using your engine to produce risk scores. I set up macros in SAS to do the same type of calculations and produced risk scores and then used proc compare to analyze the results.

When testing this many cases I had around 5% error where scores did not match. Upon investigating this seemed to be due to the way that your engine and CMS's model handles demographic contradictions. (example: male with ovarian cancer, female with testicular cancer). I modified my artificial bene generator to properly bin age/sex ICD10s to appropriate benes which then cleared with no error at the order of ~100k examples.

In conclusion, I have yet to see a realistic example fail however I noticed the presence of demographic contradictions producing different scores.

[yubin-park]

@jbabcanec, thanks so much for testing! I am, in fact, in the process of rewriting the entire module, and I would love to hear your thoughts for the new version. Please DM me on LinkedIn https://www.linkedin.com/in/yubin-park-phd/ if that's okay for you. Thanks so much!