Releases: xujiajun/gotokenizer
Releases · xujiajun/gotokenizer
v1.1.0
v1.0.0 release
- Support Maximum Matching Method
- Support Minimum Matching Method
- Support Reverse Maximum Matching
- Support Reverse Minimum Matching
- Support Bidirectional Maximum Matching
- Support Bidirectional Minimum Matching
- Support using Stop Tokens