This is a ngrams package in C++, which can be used for character or word ngram analysis. It uses Ternary Search Tree instead of hashing table for faster ngram frequency counting. Words are converted to unique IDs and encoded to more compact base 256 integers.
It is a simplified implementation of Dr. Vlado Keselj's Text-Ngrams 1.6, which is a very flexible Ngram package in perl. See more information at http://users.cs.dal.ca/~vlado/srcperl/Ngrams/Ngrams.html
- download and save the source code.
$> make$> ngrams --type=word --n=3 --in= sample.txtor$> ngrams --type=character -n=3 --in= sample.txt
That's it. If you found any bug or have any suggestion, please kindly send me email [email protected]. Thanks.
Zheyuan Yu. Feb 18,2006
- Mar 28, 2005. Zheyuan Yu Modify ternary search tree to improve performance, save pointer of TstItems into vector, instead of saving TstItems. It will save some time when vector grows.
- Feb 18, 2006. Zheyuan Yu Initial implemenation