Tokenizers
为研究和生产而优化的快速先进的分词器。「💥 Fast State-of-the-Art Tokenizers optimized for Research and Production」
Main metrics
Overview
| Name With Owner | huggingface/tokenizers |
|---|---|
| Primary Language | Rust |
| Program language | Rust (Language Count: 8) |
| Platform | |
| License: | Apache License 2.0 |
| Release Count | 146 |
| Last Release Name | v0.21.4 (Posted on ) |
| First Release Name | v0.0.3 (Posted on ) |
| Created At | 2019-11-01 17:52:20 |
| Pushed At | 2025-07-29 13:32:04 |
| Last Commit At | |
| Stargazers Count | 9976 |
| Watchers Count | 125 |
| Fork Count | 951 |
| Commits Count | 1886 |
| Has Issues Enabled | |
| Issues Count | 1086 |
| Issue Open Count | 79 |
| Pull Requests Count | 554 |
| Pull Requests Open Count | 27 |
| Pull Requests Close Count | 160 |
| Has Wiki Enabled | |
| Is Archived | |
| Is Fork | |
| Is Locked | |
| Is Mirror | |
| Is Private |
