Tokenizers

为研究和生产而优化的快速先进的分词器。「💥 Fast State-of-the-Art Tokenizers optimized for Research and Production」

Main metrics

Overview

Name With Ownerhuggingface/tokenizers
Primary LanguageRust
Program languageRust (Language Count: 8)
Platform
License:Apache License 2.0
Release Count150
Last Release Namev0.22.1 (Posted on )
First Release Namev0.0.3 (Posted on )
Created At2019-11-01 17:52:20
Pushed At2025-09-19 09:46:10
Last Commit At
Stargazers Count10112
Watchers Count124
Fork Count971
Commits Count1903
Has Issues Enabled
Issues Count1096
Issue Open Count85
Pull Requests Count565
Pull Requests Open Count27
Pull Requests Close Count168
Has Wiki Enabled
Is Archived
Is Fork
Is Locked
Is Mirror
Is Private
To the top