Skip to content

Releases: xujiajun/gotokenizer

v1.1.0

17 Oct 02:12
572fbff
Compare
Choose a tag to compare
  • Add Support Custom word Filter (MM RMM BMM)
  • Add Support filter stop tokens (MM RMM BMM)
  • Add tests
  • Fixed test error
  • Update ReadMe

v1.0.0 release

15 Oct 09:41
Compare
Choose a tag to compare
  • Support Maximum Matching Method
  • Support Minimum Matching Method
  • Support Reverse Maximum Matching
  • Support Reverse Minimum Matching
  • Support Bidirectional Maximum Matching
  • Support Bidirectional Minimum Matching
  • Support using Stop Tokens