notbugAs an Amazon Associate I earn from qualifying purchases.
Want a good read? Try FreeBSD Mastery: Jails (IT Mastery Book 15)
Want a good monitor light? See my photosAll times are UTC
Ukraine

Bot filter coming soon

To deter bots pegging the database CPU to 100%, a bot testing filter to be added to the website. This should not affect newsfeeds etc. Anubis seems light-weight - it is already in use within the FreeBSD Project. This notice is just a heads up in case you see something odd. This notice will be updated after Anubis is installed.

Port details on branch 2024Q3
py-tiktoken Fast BPE tokeniser for use with OpenAI's models
0.9.0_4 textproc on this many watch lists=2 search for ports that depend on this port Find issues related to this port Report an issue related to this port View this port on Repology. pkg-fallout 0.9.0_3Version of this port present on the latest quarterly branch.
Maintainer: sunpoet@FreeBSD.org search for ports maintained by this maintainer
Port Added: 2023-08-21 17:15:36
Last Update: 2025-07-03 08:46:01
Commit Hash: f3f7e55
People watching this port, also watch:: jdictionary, py311-Automat, py311-python-gdsii, py311-PyOpenGL, p5-Sane
Also Listed In: python
License: MIT
WWW:
https://github.com/openai/tiktoken
Description:
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Homepage    cgit ¦ Codeberg ¦ GitHub ¦ GitLab ¦ SVNWeb - no subversion history for this port

Manual pages:
FreshPorts has no man page information for this port.
pkg-plist: as obtained via: make generate-plist
There is no configure plist information for this port.
Dependency lines:
  • ${PYTHON_PKGNAMEPREFIX}tiktoken>0:textproc/py-tiktoken@${PY_FLAVOR}
To install the port:
cd /usr/ports/textproc/py-tiktoken/ && make install clean
To add the package, run one of these commands:
  • pkg install textproc/py-tiktoken
  • pkg install py311-tiktoken
NOTE: If this package has multiple flavors (see below), then use one of them instead of the name specified above.
NOTE: This is a Python port. Instead of py311-tiktoken listed in the above command, you can pick from the names under the Packages section.
PKGNAME: py311-tiktoken
Package flavors (<flavor>: <package>)
  • py311: py311-tiktoken
distinfo:
TIMESTAMP = 1739930208 SHA256 (tiktoken-0.9.0.tar.gz) = d02a5ca6a938e0490e1ff957bc48c8b078c88cb83977be1625b1fd8aac792c5d SIZE (tiktoken-0.9.0.tar.gz) = 35991

Expand this list (62 items)

Collapse this list.

SHA256 (rust/crates/aho-corasick-1.1.3.crate) = 8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916 SIZE (rust/crates/aho-corasick-1.1.3.crate) = 183311 SHA256 (rust/crates/autocfg-1.4.0.crate) = ace50bade8e6234aa140d9a2f552bbee1db4d353f69b8217bc503490fc1a9f26 SIZE (rust/crates/autocfg-1.4.0.crate) = 17712 SHA256 (rust/crates/bit-set-0.5.3.crate) = 0700ddab506f33b20a03b13996eccd309a48e5ff77d0d95926aa0210fb4e95f1 SIZE (rust/crates/bit-set-0.5.3.crate) = 14470 SHA256 (rust/crates/bit-vec-0.6.3.crate) = 349f9b6a179ed607305526ca489b34ad0a41aed5f7980fa90eb03160b69598fb SIZE (rust/crates/bit-vec-0.6.3.crate) = 19927 SHA256 (rust/crates/bstr-1.11.3.crate) = 531a9155a481e2ee699d4f98f43c0ca4ff8ee1bfd55c31e9e98fb29d2b176fe0 SIZE (rust/crates/bstr-1.11.3.crate) = 351536 SHA256 (rust/crates/cfg-if-1.0.0.crate) = baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd SIZE (rust/crates/cfg-if-1.0.0.crate) = 7934 SHA256 (rust/crates/fancy-regex-0.13.0.crate) = 531e46835a22af56d1e3b66f04844bed63158bc094a628bec1d321d9b4c44bf2 SIZE (rust/crates/fancy-regex-0.13.0.crate) = 85930 SHA256 (rust/crates/heck-0.5.0.crate) = 2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea SIZE (rust/crates/heck-0.5.0.crate) = 11517 SHA256 (rust/crates/indoc-2.0.5.crate) = b248f5224d1d606005e02c97f5aa4e88eeb230488bcc03bc9ca4d7991399f2b5 SIZE (rust/crates/indoc-2.0.5.crate) = 14396 SHA256 (rust/crates/libc-0.2.169.crate) = b5aba8db14291edd000dfcc4d620c7ebfb122c613afb886ca8803fa4e128a20a SIZE (rust/crates/libc-0.2.169.crate) = 757901 SHA256 (rust/crates/memchr-2.7.4.crate) = 78ca9ab1a0babb1e7d5695e3530886289c18cf2f87ec19a575a0abdce112e3a3 SIZE (rust/crates/memchr-2.7.4.crate) = 96670 SHA256 (rust/crates/memoffset-0.9.1.crate) = 488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a SIZE (rust/crates/memoffset-0.9.1.crate) = 9032 SHA256 (rust/crates/once_cell-1.20.3.crate) = 945462a4b81e43c4e3ba96bd7b49d834c6f61198356aa858733bc4acf3cbe62e SIZE (rust/crates/once_cell-1.20.3.crate) = 33456 SHA256 (rust/crates/portable-atomic-1.10.0.crate) = 280dc24453071f1b63954171985a0b0d30058d287960968b9b2aca264c8d4ee6 SIZE (rust/crates/portable-atomic-1.10.0.crate) = 174760 SHA256 (rust/crates/proc-macro2-1.0.93.crate) = 60946a68e5f9d28b0dc1c21bb8a97ee7d018a8b322fa57838ba31cc878e22d99 SIZE (rust/crates/proc-macro2-1.0.93.crate) = 52388 SHA256 (rust/crates/pyo3-0.22.6.crate) = f402062616ab18202ae8319da13fa4279883a2b8a9d9f83f20dbade813ce1884 SIZE (rust/crates/pyo3-0.22.6.crate) = 546746 SHA256 (rust/crates/pyo3-build-config-0.22.6.crate) = b14b5775b5ff446dd1056212d778012cbe8a0fbffd368029fd9e25b514479c38 SIZE (rust/crates/pyo3-build-config-0.22.6.crate) = 31369 SHA256 (rust/crates/pyo3-ffi-0.22.6.crate) = 9ab5bcf04a2cdcbb50c7d6105de943f543f9ed92af55818fd17b660390fc8636 SIZE (rust/crates/pyo3-ffi-0.22.6.crate) = 69352 SHA256 (rust/crates/pyo3-macros-0.22.6.crate) = 0fd24d897903a9e6d80b968368a34e1525aeb719d568dba8b3d4bfa5dc67d453 SIZE (rust/crates/pyo3-macros-0.22.6.crate) = 8182 SHA256 (rust/crates/pyo3-macros-backend-0.22.6.crate) = 36c011a03ba1e50152b4b394b479826cad97e7a21eb52df179cd91ac411cbfbe SIZE (rust/crates/pyo3-macros-backend-0.22.6.crate) = 66312 SHA256 (rust/crates/quote-1.0.38.crate) = 0e4dccaaaf89514f546c693ddc140f729f958c247918a13380cccc6078391acc SIZE (rust/crates/quote-1.0.38.crate) = 31252 SHA256 (rust/crates/regex-1.11.1.crate) = b544ef1b4eac5dc2db33ea63606ae9ffcfac26c1416a2806ae0bf5f56b201191 SIZE (rust/crates/regex-1.11.1.crate) = 254170 SHA256 (rust/crates/regex-automata-0.4.9.crate) = 809e8dc61f6de73b46c85f4c96486310fe304c434cfa43669d7b40f711150908 SIZE (rust/crates/regex-automata-0.4.9.crate) = 618525 SHA256 (rust/crates/regex-syntax-0.8.5.crate) = 2b15c43186be67a4fd63bee50d0303afffcef381492ebe2c5d87f324e1b8815c SIZE (rust/crates/regex-syntax-0.8.5.crate) = 357541 SHA256 (rust/crates/rustc-hash-1.1.0.crate) = 08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2 SIZE (rust/crates/rustc-hash-1.1.0.crate) = 9331 SHA256 (rust/crates/serde-1.0.217.crate) = 02fc4265df13d6fa1d00ecff087228cc0a2b5f3c0e87e258d8b94a156e984c70 SIZE (rust/crates/serde-1.0.217.crate) = 79019 SHA256 (rust/crates/serde_derive-1.0.217.crate) = 5a9bf7cf98d04a2b28aead066b7496853d4779c9cc183c440dbac457641e19a0 SIZE (rust/crates/serde_derive-1.0.217.crate) = 57749 SHA256 (rust/crates/syn-2.0.98.crate) = 36147f1a48ae0ec2b5b3bc5b537d267457555a10dc06f3dbc8cb11ba3006d3b1 SIZE (rust/crates/syn-2.0.98.crate) = 297807 SHA256 (rust/crates/target-lexicon-0.12.16.crate) = 61c41af27dd6d1e27b1b16b489db798443478cef1f06a660c96db617ba5de3b1 SIZE (rust/crates/target-lexicon-0.12.16.crate) = 26488 SHA256 (rust/crates/unicode-ident-1.0.16.crate) = a210d160f08b701c8721ba1c726c11662f877ea6b7094007e1ca9a1041945034 SIZE (rust/crates/unicode-ident-1.0.16.crate) = 47684 SHA256 (rust/crates/unindent-0.2.3.crate) = c7de7d73e1754487cb58364ee906a499937a0dfabd86bcb980fa99ec8c8fa2ce SIZE (rust/crates/unindent-0.2.3.crate) = 7306

Collapse this list.


Packages (timestamps in pop-ups are UTC):
py39-tiktoken
ABIaarch64amd64armv6armv7i386powerpcpowerpc64powerpc64le
FreeBSD:13:latest--------
FreeBSD:13:quarterly------0.5.20.5.2
FreeBSD:14:latest--------
FreeBSD:14:quarterly------0.5.20.5.2
FreeBSD:15:latest--n/a-n/a-0.5.2_1-
 

py311-tiktoken
ABIaarch64amd64armv6armv7i386powerpcpowerpc64powerpc64le
FreeBSD:13:latest0.9.0_40.9.0_4-0.9.0_40.9.0_4---
FreeBSD:13:quarterly0.9.0_10.9.0_3-0.9.0_10.9.0_3---
FreeBSD:14:latest0.9.0_30.9.0_4-0.9.0_30.9.0_4---
FreeBSD:14:quarterly0.9.0_30.9.0_3-0.9.0_30.9.0_3---
FreeBSD:15:latest0.9.0_30.9.0_4n/a0.9.0_3n/a---
 

Dependencies
NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered.
Build dependencies:
  1. py311-setuptools>=62.4 : devel/py-setuptools@py311
  2. py311-setuptools-rust>=1.5.2 : devel/py-setuptools-rust@py311
  3. py311-wheel>=0 : devel/py-wheel@py311
  4. rust>=1.88.0 : lang/rust
  5. python3.11 : lang/python311
  6. py311-build>=0 : devel/py-build@py311
  7. py311-installer>=0 : devel/py-installer@py311
Test dependencies:
  1. python3.11 : lang/python311
Runtime dependencies:
  1. py311-regex>=2022.1.18 : textproc/py-regex@py311
  2. py311-requests>=2.26.0 : www/py-requests@py311
  3. python3.11 : lang/python311
This port is required by:
for Run
  1. devel/py-ddtrace
  2. misc/py-aider-chat
  3. misc/py-langchain-openai
  4. misc/py-litellm
  5. misc/py-openai-whisper

Configuration Options:
===> The following configuration options are available for py311-tiktoken-0.9.0_4: BLOBFILE=off: Blob file support ===> Use 'make config' to modify these settings
Options name:
textproc_py-tiktoken
USES:
cargo python:3.9+
FreshPorts was unable to extract/find any pkg message
Master Sites:
Expand this list (2 items)
Collapse this list.
  1. https://files.pythonhosted.org/packages/source/t/tiktoken/
  2. https://pypi.org/packages/source/t/tiktoken/
Collapse this list.

There are no commits on branch 2024Q3 for this port