Crates / tokenizers
Rust crate · real downloads · data Oct 27, 2026
tokenizers downloads, and should you depend on it?
Fastest implementation of the most used tokenizers in AI. crates.io · docs · repository
alternatives · is it still maintained?
Widely used and gaining.
tokenizers: 1,028 crates depend on it; downloads jump on release days (1.3x the baseline).
How this verdict is decided (4 rules)
- Usage: crates.io reverse dependencies (crates that list it as a dependency): 10,000+ = foundational, 1,000+ = widely used, 100+ = established, under 100 = few direct dependents.
- Direction: the median quiet day in the last 30 days of the window against the first 30, read against the median tracked crate (the whole field moved +39% from late summer to October, so a raw trend says nothing). 10%+ ahead of the field = gaining; 10%+ behind = losing ground; otherwise steady. Steady with no release in 2+ years = quiet for years.
- Release spikes: the median release-window day against the quiet-day baseline: under 1.15x = no spike, 1.15x to 1.3x = small bumps, 1.3x or more = jumps on release days.
- Release recency: days since the newest version that was not yanked, from crates.io publish times.
release window (release day + next day)quiet dayquiet-day median · download the 89 days as CSV
The numbers in one paragraphtokenizers got 17,806,041 crates.io downloads from Jul 30, 2026 to Oct 26, 2026. 8% of them landed within 2 days of one of its 3 releases. On the other 83 quiet days it got a median of 183,031 a day, about 1,281,217 a week. Capping release days at that baseline leaves 17,452,944 release-adjusted downloads for the window.
What this means if you're choosing a dependency
- Well trodden. 1,028 crates depend on it, so common problems tend to have public answers and breaking changes get noticed fast.
- Actively released. 1.0.0-rc.3 shipped 8 days ago (Oct 19, 2026).
- Fit check: license Apache-2.0, edition 2024 (from the latest version's manifest).
- Compare it by the baseline, not the raw count. 8% of its 17,806,041 downloads in the window landed within 2 days of a release; the steady number is 183,031 a day.
Alternatives that do a similar job
Crates that share its crates.io category or keywords, most similar first. Each line is the crate's own description, then how it compares on the same data.
Its place in the dependency graph
Built on tokenizers
2 of the 1243 crates we track depend on it directly (biggest first).
tokenizers builds on
Tracked crates its latest version (1.0.0-rc.3) depends on (optional ones marked).
What the chart says
- Weekend quiet days ran at 65% of weekday quiet days (137,260 vs 210,776). A dip that deep is the signature of work-week CI pipelines and developers building on weekdays.
- Biggest day: 346,282 on Oct 6, 2026, a quiet day with no release nearby.
Every number
| All-time downloads | 42,423,190 (outside the crates.io top 1,000 by all-time downloads) |
| Downloads in the window | 17,806,041 (Jul 30, 2026 to Oct 26, 2026, 89 days) |
| Within 2 days of a release | 1,451,283 (8.2%) |
| Quiet-day median (baseline) | 183,031 a day; weekdays 210,776, weekends 137,260 |
| Release-adjusted downloads | 17,452,944 (release-window days capped at the baseline) |
| Baseline rank | #978 of the 1243 crates we track |
| Quiet-day trend | +66% raw, +20% against the median tracked crate (+39%) |
| Releases in the window | 3 |
| Versions published, ever | 43 |
| Latest version | 1.0.0-rc.3, published Oct 19, 2026 (license Apache-2.0, edition 2024) |
| First published | Aug 8, 2019 |
| Reverse dependencies | 1,028 |
Releases in the window
Each one opens a 2-day release window on the chart.
| Version | Published | Downloads that version got (lifetime) |
|---|---|---|
| 1.0.0-rc.3 | Oct 19, 2026 | 824 |
| 1.0.0-rc.2 | Sep 21, 2026 | 9,165 |
| 0.23.2 | Sep 3, 2026 | 1,974,551 |
How we compute this
crates.io counts every download, including CI runners, mirrors and bots that fetch each new version, so raw counts run high (why, in The Manual). We split the last 89 days into release windows (the day a version of tokenizers was published, plus the next day) and quiet days, and use the median quiet day as the baseline: an upper bound on steady use, never a head count. The method and data come from our study of 90 days of crates.io data from 69 crates, featured in This Week in Rust #674; check any crate live with the real-downloads checker.
Badge for your README
Maintainers can show the baseline in a README. It is optional, updates when we refresh the data, and links back here. Preview, then copy:
README, light
README, dark
Publishing a crate of your own? How to publish a Rust crate to crates.io, step by step.
Categories
Sources
- Daily downloads: crates.io API, /crates/tokenizers/downloads (last 90 days, all versions), fetched Oct 27, 2026.
- Versions and publish times: crates.io API, /crates/tokenizers/versions, same fetch.
- Reverse dependencies: crates.io reverse dependencies, same fetch. Dependency edges between tracked crates: each crate's default version, crates.io API.
- Fetched politely (one request every 2 seconds, identified User-Agent) per the crates.io data access policy. Data is crates.io's; the release-window split, baseline and verdict rules are ours, CC BY 4.0.
Up next
Next by real downloads: The next crate down the real-downloads ranking: 181,762 a day. Established and steady.handlebarsRust crate. Handlebars templating implemented in Rust. Real use: 182K downloads a day on quiet days; 829 crates depend on it.Read it next →Also worth your time
Art: Nicholas Roerich, Nag Lake (1925). Public domain, via Wikimedia Commons.
.jpg){kind=link}