notbugAs an Amazon Associate I earn from qualifying purchases.
Want a good read? Try FreeBSD Mastery: Jails (IT Mastery Book 15)Want a good monitor light? See my photosAll times are UTC		 Ukraine

Bot filter coming soon

To deter bots pegging the database CPU to 100%, a bot testing filter to be added to the website. This should not affect newsfeeds etc. Anubis seems light-weight - it is already in use within the FreeBSD Project. This notice is just a heads up in case you see something odd. This notice will be updated after Anubis is installed.

Port details on branch 2024Q3
py-tiktoken Fast BPE tokeniser for use with OpenAI's models
0.9.0_4 textproc on this many watch lists=2 search for ports that depend on this port Find issues related to this port Report an issue related to this port View this port on Repology. pkg-fallout 0.9.0_3Version of this port present on the latest quarterly branch.
Maintainer: sunpoet@FreeBSD.org search for ports maintained by this maintainer
Port Added: 2023-08-21 17:15:36
Last Update: 2025-07-03 08:46:01
Commit Hash: f3f7e55
People watching this port, also watch:: jdictionary, py311-Automat, py311-python-gdsii, py311-PyOpenGL, p5-Sane
Also Listed In: python
License: MIT
WWW:
https://github.com/openai/tiktoken
Description:
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Homepage    cgit ¦ Codeberg ¦ GitHub ¦ GitLab ¦ SVNWeb - no subversion history for this port

Manual pages:
FreshPorts has no man page information for this port.
pkg-plist: as obtained via: make generate-plist
There is no configure plist information for this port.
Dependency lines:
  • ${PYTHON_PKGNAMEPREFIX}tiktoken>0:textproc/py-tiktoken@${PY_FLAVOR}
To install the port:
cd /usr/ports/textproc/py-tiktoken/ && make install clean
To add the package, run one of these commands:
  • pkg install textproc/py-tiktoken
  • pkg install py311-tiktoken
NOTE: If this package has multiple flavors (see below), then use one of them instead of the name specified above.
NOTE: This is a Python port. Instead of py311-tiktoken listed in the above command, you can pick from the names under the Packages section.
PKGNAME: py311-tiktoken
Package flavors (<flavor>: <package>)
  • py311: py311-tiktoken
distinfo:
TIMESTAMP = 1739930208 SHA256 (tiktoken-0.9.0.tar.gz) = d02a5ca6a938e0490e1ff957bc48c8b078c88cb83977be1625b1fd8aac792c5d SIZE (tiktoken-0.9.0.tar.gz) = 35991

Expand this list (62 items)

Collapse this list.

SHA256 (rust/crates/aho-corasick-1.1.3.crate) = 8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916 SIZE (rust/crates/aho-corasick-1.1.3.crate) = 183311 SHA256 (rust/crates/autocfg-1.4.0.crate) = ace50bade8e6234aa140d9a2f552bbee1db4d353f69b8217bc503490fc1a9f26 SIZE (rust/crates/autocfg-1.4.0.crate) = 17712 SHA256 (rust/crates/bit-set-0.5.3.crate) = 0700ddab506f33b20a03b13996eccd309a48e5ff77d0d95926aa0210fb4e95f1 SIZE (rust/crates/bit-set-0.5.3.crate) = 14470 SHA256 (rust/crates/bit-vec-0.6.3.crate) = 349f9b6a179ed607305526ca489b34ad0a41aed5f7980fa90eb03160b69598fb SIZE (rust/crates/bit-vec-0.6.3.crate) = 19927 SHA256 (rust/crates/bstr-1.11.3.crate) = 531a9155a481e2ee699d4f98f43c0ca4ff8ee1bfd55c31e9e98fb29d2b176fe0 SIZE (rust/crates/bstr-1.11.3.crate) = 351536 SHA256 (rust/crates/cfg-if-1.0.0.crate) = baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd SIZE (rust/crates/cfg-if-1.0.0.crate) = 7934 SHA256 (rust/crates/fancy-regex-0.13.0.crate) = 531e46835a22af56d1e3b66f04844bed63158bc094a628bec1d321d9b4c44bf2 SIZE (rust/crates/fancy-regex-0.13.0.crate) = 85930 SHA256 (rust/crates/heck-0.5.0.crate) = 2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea SIZE (rust/crates/heck-0.5.0.crate) = 11517 SHA256 (rust/crates/indoc-2.0.5.crate) = b248f5224d1d606005e02c97f5aa4e88eeb230488bcc03bc9ca4d7991399f2b5 SIZE (rust/crates/indoc-2.0.5.crate) = 14396 SHA256 (rust/crates/libc-0.2.169.crate) = b5aba8db14291edd000dfcc4d620c7ebfb122c613afb886ca8803fa4e128a20a SIZE (rust/crates/libc-0.2.169.crate) = 757901 SHA256 (rust/crates/memchr-2.7.4.crate) = 78ca9ab1a0babb1e7d5695e3530886289c18cf2f87ec19a575a0abdce112e3a3 SIZE (rust/crates/memchr-2.7.4.crate) = 96670 SHA256 (rust/crates/memoffset-0.9.1.crate) = 488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a SIZE (rust/crates/memoffset-0.9.1.crate) = 9032 SHA256 (rust/crates/once_cell-1.20.3.crate) = 945462a4b81e43c4e3ba96bd7b49d834c6f61198356aa858733bc4acf3cbe62e SIZE (rust/crates/once_cell-1.20.3.crate) = 33456 SHA256 (rust/crates/portable-atomic-1.10.0.crate) = 280dc24453071f1b63954171985a0b0d30058d287960968b9b2aca264c8d4ee6 SIZE (rust/crates/portable-atomic-1.10.0.crate) = 174760 SHA256 (rust/crates/proc-macro2-1.0.93.crate) = 60946a68e5f9d28b0dc1c21bb8a97ee7d018a8b322fa57838ba31cc878e22d99 SIZE (rust/crates/proc-macro2-1.0.93.crate) = 52388 SHA256 (rust/crates/pyo3-0.22.6.crate) = f402062616ab18202ae8319da13fa4279883a2b8a9d9f83f20dbade813ce1884 SIZE (rust/crates/pyo3-0.22.6.crate) = 546746 SHA256 (rust/crates/pyo3-build-config-0.22.6.crate) = b14b5775b5ff446dd1056212d778012cbe8a0fbffd368029fd9e25b514479c38 SIZE (rust/crates/pyo3-build-config-0.22.6.crate) = 31369 SHA256 (rust/crates/pyo3-ffi-0.22.6.crate) = 9ab5bcf04a2cdcbb50c7d6105de943f543f9ed92af55818fd17b660390fc8636 SIZE (rust/crates/pyo3-ffi-0.22.6.crate) = 69352 SHA256 (rust/crates/pyo3-macros-0.22.6.crate) = 0fd24d897903a9e6d80b968368a34e1525aeb719d568dba8b3d4bfa5dc67d453 SIZE (rust/crates/pyo3-macros-0.22.6.crate) = 8182 SHA256 (rust/crates/pyo3-macros-backend-0.22.6.crate) = 36c011a03ba1e50152b4b394b479826cad97e7a21eb52df179cd91ac411cbfbe SIZE (rust/crates/pyo3-macros-backend-0.22.6.crate) = 66312 SHA256 (rust/crates/quote-1.0.38.crate) = 0e4dccaaaf89514f546c693ddc140f729f958c247918a13380cccc6078391acc SIZE (rust/crates/quote-1.0.38.crate) = 31252 SHA256 (rust/crates/regex-1.11.1.crate) = b544ef1b4eac5dc2db33ea63606ae9ffcfac26c1416a2806ae0bf5f56b201191 SIZE (rust/crates/regex-1.11.1.crate) = 254170 SHA256 (rust/crates/regex-automata-0.4.9.crate) = 809e8dc61f6de73b46c85f4c96486310fe304c434cfa43669d7b40f711150908 SIZE (rust/crates/regex-automata-0.4.9.crate) = 618525 SHA256 (rust/crates/regex-syntax-0.8.5.crate) = 2b15c43186be67a4fd63bee50d0303afffcef381492ebe2c5d87f324e1b8815c SIZE (rust/crates/regex-syntax-0.8.5.crate) = 357541 SHA256 (rust/crates/rustc-hash-1.1.0.crate) = 08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2 SIZE (rust/crates/rustc-hash-1.1.0.crate) = 9331 SHA256 (rust/crates/serde-1.0.217.crate) = 02fc4265df13d6fa1d00ecff087228cc0a2b5f3c0e87e258d8b94a156e984c70 SIZE (rust/crates/serde-1.0.217.crate) = 79019 SHA256 (rust/crates/serde_derive-1.0.217.crate) = 5a9bf7cf98d04a2b28aead066b7496853d4779c9cc183c440dbac457641e19a0 SIZE (rust/crates/serde_derive-1.0.217.crate) = 57749 SHA256 (rust/crates/syn-2.0.98.crate) = 36147f1a48ae0ec2b5b3bc5b537d267457555a10dc06f3dbc8cb11ba3006d3b1 SIZE (rust/crates/syn-2.0.98.crate) = 297807 SHA256 (rust/crates/target-lexicon-0.12.16.crate) = 61c41af27dd6d1e27b1b16b489db798443478cef1f06a660c96db617ba5de3b1 SIZE (rust/crates/target-lexicon-0.12.16.crate) = 26488 SHA256 (rust/crates/unicode-ident-1.0.16.crate) = a210d160f08b701c8721ba1c726c11662f877ea6b7094007e1ca9a1041945034 SIZE (rust/crates/unicode-ident-1.0.16.crate) = 47684 SHA256 (rust/crates/unindent-0.2.3.crate) = c7de7d73e1754487cb58364ee906a499937a0dfabd86bcb980fa99ec8c8fa2ce SIZE (rust/crates/unindent-0.2.3.crate) = 7306

Collapse this list.

Packages (timestamps in pop-ups are UTC):
py39-tiktoken
ABIaarch64amd64armv6armv7i386powerpcpowerpc64powerpc64le
FreeBSD:13:latest--------
FreeBSD:13:quarterly------0.5.20.5.2
FreeBSD:14:latest--------
FreeBSD:14:quarterly------0.5.20.5.2
FreeBSD:15:latest--n/a-n/a-0.5.2_1-
 

py311-tiktoken
ABIaarch64amd64armv6armv7i386powerpcpowerpc64powerpc64le
FreeBSD:13:latest0.9.0_40.9.0_4-0.9.0_40.9.0_4---
FreeBSD:13:quarterly0.9.0_30.9.0_3-0.9.0_10.9.0_3---
FreeBSD:14:latest0.9.0_30.9.0_4-0.9.0_30.9.0_4---
FreeBSD:14:quarterly0.9.0_30.9.0_3-0.9.0_30.9.0_3---
FreeBSD:15:latest0.9.0_30.9.0_4n/a0.9.0_3n/a---
 

Dependencies
NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered.
Build dependencies:
  1. py311-setuptools>=62.4 : devel/py-setuptools@py311
  2. py311-setuptools-rust>=1.5.2 : devel/py-setuptools-rust@py311
  3. py311-wheel>=0 : devel/py-wheel@py311
  4. rust>=1.88.0 : lang/rust
  5. python3.11 : lang/python311
  6. py311-build>=0 : devel/py-build@py311
  7. py311-installer>=0 : devel/py-installer@py311
Test dependencies:
  1. python3.11 : lang/python311
Runtime dependencies:
  1. py311-regex>=2022.1.18 : textproc/py-regex@py311
  2. py311-requests>=2.26.0 : www/py-requests@py311
  3. python3.11 : lang/python311
This port is required by:
for Run
  1. devel/py-ddtrace
  2. misc/py-aider-chat
  3. misc/py-langchain-openai
  4. misc/py-litellm
  5. misc/py-openai-whisper
Configuration Options:
===> The following configuration options are available for py311-tiktoken-0.9.0_4: BLOBFILE=off: Blob file support ===> Use 'make config' to modify these settings
Options name:
textproc_py-tiktoken
USES:
cargo python:3.9+
FreshPorts was unable to extract/find any pkg message
Master Sites:
Expand this list (2 items)
Collapse this list.
  1. https://files.pythonhosted.org/packages/source/t/tiktoken/
  2. https://pypi.org/packages/source/t/tiktoken/
Collapse this list.

There are no commits on branch 2024Q3 for this port
Login
User Login
Create account

Servers and bandwidth provided by
New York Internet, iXsystems, and RootBSD

This site
What is FreshPorts?
About the authors
Issues
FAQ
How big is it?
Security Policy
Privacy
Blog
Contact

Search
Enter Keywords:
 
more...

Latest Vulnerabilities
ruby32Jul 25
ruby33Jul 25
ruby34Jul 25
ruby35Jul 25
rubygem-resolvJul 25
firefoxJul 24
firefoxJul 24
firefoxJul 24
firefoxJul 24
firefoxJul 24
firefoxJul 24
firefoxJul 24
firefoxJul 24
firefoxJul 24
firefoxJul 24

28 vulnerabilities affecting 252 ports have been reported in the past 14 days

* - modified, not new

All vulnerabilities

Last processed:
2025-07-27 12:33:51 UTC


Ports
Home
Categories
Deleted ports
Sanity Test Failures
Newsfeeds

Statistics
Graphs
NEW Graphs (Javascript)

Calculated hourly:
Port count 33528
Broken 128
Deprecated 238
Ignore 245
Forbidden 0
Restricted 2
No CDROM 1
Vulnerable 37
Expired 20
Set to expire 187
Interactive 0
new 24 hours 1
new 48 hours1
new 7 days34
new fortnight83
new month215
Servers and bandwidth provided by
New York Internet, iXsystems, and RootBSD 		Valid HTML, CSS, and RSS. Copyright © 2000-2025 Dan Langille. All rights reserved.