1 unstable release
0.1.0 | Nov 18, 2023 |
---|
#736 in Machine learning
59 downloads per month
8KB
60 lines
sentencepiece-model
SentencePiece model parser generated from the SentencePiece protobuf definition.
use sentencepiece_model::SentencePieceModel;
let model = SentencePieceModel::from_file("tests/t5-spiece.model")?;
assert_eq!(model.pieces.len(), 32000);
assert_eq!(model.trainer()?.unk_id(), 2);
lib.rs
:
SentencePiece model parser generated from the SentencePiece protobuf definition.
See SentencePieceModel
for the entry point for parsing and accessing sentencepiece models.
Dependencies
~0.2–3.5MB
~28K SLoC