Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Likely bug in fix_deid_tokens regexp #71

Open
SphtKr opened this issue Mar 8, 2023 · 0 comments
Open

Likely bug in fix_deid_tokens regexp #71

SphtKr opened this issue Mar 8, 2023 · 0 comments

Comments

@SphtKr
Copy link

SphtKr commented Mar 8, 2023

I noticed this bit of code is probably not performing as intended:

    def fix_deid_tokens(text, processed_text):
        deid_regex  = r"\[\*\*.{0,15}.*?\*\*\]" 
        indexes = [m.span() for m in re.finditer(deid_regex,text,flags=re.IGNORECASE)]

Take for example this string with two de-ID'd portions:

[**9-17**] constant pressure/pain over L chest.  MD [**First Name (Titles) 14**]

The regex matches the whole string, rather than matching the two de-ID'd tokens separately--and therefore merges the whole string as one token in the spaCy Document.

The problem is that the .{0,15} is greedy matching (no ?) and therefore consumes the first **] if the first tag is short enough and if there is a later **] that the pattern can match. One fix for this would be to add a ?, like so:

deid_regex  = r"\[\*\*.{0,15}?.*?\*\*\]"

But it seems that dropping the .{0,15} altogether makes more sense, since the .*? follows it...

deid_regex  = r"\[\*\*.*?\*\*\]"

though there may be a reason for that block that I'm unaware of. Nevertheless, it's not working entirely as intended either.

Working on getting the pipeline to work with modern Pandas and spaCy... so far pretty good... will submit a PR if I can get it to work, but this abuts a block that needs changes and affects the original code. Can submit a separate PR just for this, once I'm confident the fix is the right one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant