The word split is kinda too aggressive

[shushi2016]

Thanks for this great work. I tried it out and found that the split is sometimes too aggressive to me, for example, the 'occupational' is split into 'occ', 'u', 'p', 'a', 't' and ional', and 'particulate' into 'part', 'icu', 'late'. Strangely it's not always like this - sometimes I can get 'occupational' and 'particulate' correctly. Any thoughts about why this happens?

[keredson]

Hey Shi,

Not really sure. For those words I'm seeing:

>>> import wordninja
>>> wordninja.split('occupational')
['occupational']
>>> wordninja.split('particulate')
['particulate']

Are those the exact strings you're trying?

The algo is deterministic. It shouldn't be giving you different outcomes for the same strings multiple calls.

Derek

[SpongebobSquamirez]

I don't remember correctly anymore, but I also ran into this issue a while back (if I recall I solved the problem). I think it had something to do with either ligatures (like the combined fi character), trying to split really really long strings (which gave a bug for some reason), or with one word having a typo/one letter being missing at the beginning or end of the string. See if maybe one of those is the cause of your problem. NLP is not fun.

Using something like hunspell or some other correction with levenshtein distance might help fix any rogue character issues.