Tokenizers
为研究和生产而优化的快速先进的分词器。「💥 Fast State-of-the-Art Tokenizers optimized for Research and Production」
Main metrics
Overview
Name With Owner | huggingface/tokenizers |
---|---|
Primary Language | Rust |
Program language | Rust (Language Count: 8) |
Platform | |
License: | Apache License 2.0 |
Release Count | 150 |
Last Release Name | v0.22.1 (Posted on ) |
First Release Name | v0.0.3 (Posted on ) |
Created At | 2019-11-01 17:52:20 |
Pushed At | 2025-09-19 09:46:10 |
Last Commit At | |
Stargazers Count | 10112 |
Watchers Count | 124 |
Fork Count | 971 |
Commits Count | 1903 |
Has Issues Enabled | |
Issues Count | 1096 |
Issue Open Count | 85 |
Pull Requests Count | 565 |
Pull Requests Open Count | 27 |
Pull Requests Close Count | 168 |
Has Wiki Enabled | |
Is Archived | |
Is Fork | |
Is Locked | |
Is Mirror | |
Is Private |