notbugAs an Amazon Associate I earn from qualifying purchases.
Want a good read? Try FreeBSD Mastery: Jails (IT Mastery Book 15)
Want a good monitor light? See my photosAll times are UTC
Ukraine
Port details
py-pytokens Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons
0.4.1 devel on this many watch lists=0 search for ports that depend on this port Find issues related to this port Report an issue related to this port View this port on Repology. pkg-fallout 0.3.0Version of this port present on the latest quarterly branch.
Maintainer: sunpoet@FreeBSD.org search for ports maintained by this maintainer
Port Added: 2025-11-30 07:21:05
Last Update: 2026-02-03 05:09:45
Commit Hash: 772a759
Also Listed In: python
License: MIT
WWW:
https://github.com/tusharsadhwani/pytokens
Description:
pytokens provides a fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
Homepage    cgit ¦ Codeberg ¦ GitHub ¦ GitLab ¦ SVNWeb - no subversion history for this port

Manual pages:
FreshPorts has no man page information for this port.
pkg-plist: as obtained via: make generate-plist
There is no configure plist information for this port.
USE_RC_SUBR (Service Scripts)
  • no SUBR information found for this port
Dependency lines:
  • ${PYTHON_PKGNAMEPREFIX}pytokens>0:devel/py-pytokens@${PY_FLAVOR}
To install the port:
cd /usr/ports/devel/py-pytokens/ && make install clean
To add the package, run one of these commands:
  • pkg install devel/py-pytokens
  • pkg install py311-pytokens
NOTE: If this package has multiple flavors (see below), then use one of them instead of the name specified above.
NOTE: This is a Python port. Instead of py311-pytokens listed in the above command, you can pick from the names under the Packages section.
PKGNAME: py311-pytokens
Package flavors (<flavor>: <package>)
  • py311: py311-pytokens
distinfo:
TIMESTAMP = 1769874368 SHA256 (pytokens-0.4.1.tar.gz) = 292052fe80923aae2260c073f822ceba21f3872ced9a68bb7953b348e561179a SIZE (pytokens-0.4.1.tar.gz) = 23015

Packages (timestamps in pop-ups are UTC):
py311-pytokens
ABIaarch64amd64armv6armv7i386powerpcpowerpc64powerpc64le
FreeBSD:13:latest0.4.10.4.1--0.4.1n/an/an/a
FreeBSD:13:quarterly0.3.00.3.0--0.3.0n/an/an/a
FreeBSD:14:latest0.4.10.4.1--0.4.1---
FreeBSD:14:quarterly0.3.00.3.0--0.3.0---
FreeBSD:15:latest0.4.10.4.1n/a-n/an/a--
FreeBSD:15:quarterly0.3.00.3.0n/a-n/an/a--
FreeBSD:16:latest0.4.10.4.1n/a-n/an/a--
Dependencies
NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered.
Build dependencies:
  1. py311-mypy>=0 : devel/py-mypy@py311
  2. py311-setuptools>=61 : devel/py-setuptools@py311
  3. py311-wheel>=0 : devel/py-wheel@py311
  4. python3.11 : lang/python311
  5. py311-build>=0 : devel/py-build@py311
  6. py311-installer>=0 : devel/py-installer@py311
Test dependencies:
  1. python3.11 : lang/python311
Runtime dependencies:
  1. python3.11 : lang/python311
This port is required by:
for Run
  1. devel/py-black

Configuration Options:
No options to configure
Options name:
devel_py-pytokens
USES:
python
FreshPorts was unable to extract/find any pkg message
Master Sites:
Expand this list (2 items)
Collapse this list.
  1. https://files.pythonhosted.org/packages/source/p/pytokens/
  2. https://pypi.org/packages/source/p/pytokens/
Collapse this list.

Number of commits found: 2

Commit History - (may be incomplete: for full details, see links to repositories near top of page)
CommitCreditsLog message
0.4.1
03 Feb 2026 05:09:45
commit hash: 772a759e8fbcf6657c1b546c92d951158f681542commit hash: 772a759e8fbcf6657c1b546c92d951158f681542commit hash: 772a759e8fbcf6657c1b546c92d951158f681542commit hash: 772a759e8fbcf6657c1b546c92d951158f681542 files touched by this commit
Po-Chuan Hsieh (sunpoet) search for other commits by this committer
devel/py-pytokens: Update to 0.4.1

Changes:	https://github.com/tusharsadhwani/pytokens/commits/main
0.3.0
30 Nov 2025 07:16:57
commit hash: c543b0e18486c8d1f2b4f2f0e347715a9c2cf877commit hash: c543b0e18486c8d1f2b4f2f0e347715a9c2cf877commit hash: c543b0e18486c8d1f2b4f2f0e347715a9c2cf877commit hash: c543b0e18486c8d1f2b4f2f0e347715a9c2cf877 files touched by this commit
Po-Chuan Hsieh (sunpoet) search for other commits by this committer
devel/py-pytokens: Add py-pytokens 0.3.0

pytokens provides a fast, spec compliant Python 3.14+ tokenizer that runs on
older Pythons.

Number of commits found: 2