notbugAs an Amazon Associate I earn from qualifying purchases.
Want a good read? Try FreeBSD Mastery: Jails (IT Mastery Book 15)
Want a good monitor light? See my photosAll times are UTC
Ukraine
This referral link gives you 10% off a Fastmail.com account and gives me a discount on my Fastmail account.
New feature planned: get notified when the package is available. Now is the time to contribute ideas/suggestions.
Port details
py-tokenizers Fast state-of-the-art tokenizers optimized for research and production
0.15.2_1 textproc on this many watch lists=0 search for ports that depend on this port Find issues related to this port Report an issue related to this port View this port on Repology. pkg-fallout Package not present on quarterly.This port was created during this quarter. It will be in the next quarterly branch but not the current one.
Maintainer: tagattie@FreeBSD.org search for ports maintained by this maintainer
Port Added: 2024-02-12 08:36:07
Last Update: 2024-02-19 11:59:23
Commit Hash: 7f5a7b0
Also Listed In: python
License: APACHE20
WWW:
https://github.com/huggingface/tokenizers
Description:
Provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Main features: - Train new vocabularies and tokenize, using today's most used tokenizers. - Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU. - Easy to use, but also extremely versatile. - Designed for research and production. - Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token. - Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.
Homepage    cgit ¦ Codeberg ¦ GitHub ¦ GitLab ¦ SVNWeb - no subversion history for this port

Manual pages:
FreshPorts has no man page information for this port.
pkg-plist: as obtained via: make generate-plist
There is no configure plist information for this port.
Dependency lines:
  • ${PYTHON_PKGNAMEPREFIX}tokenizers>0:textproc/py-tokenizers@${PY_FLAVOR}
To install the port:
cd /usr/ports/textproc/py-tokenizers/ && make install clean
To add the package, run one of these commands:
  • pkg install textproc/py-tokenizers
  • pkg install py39-tokenizers
NOTE: If this package has multiple flavors (see below), then use one of them instead of the name specified above.
NOTE: This is a Python port. Instead of py39-tokenizers listed in the above command, you can pick from the names under the Packages section.
PKGNAME: py39-tokenizers
Package flavors (<flavor>: <package>)
  • py39: py39-tokenizers
distinfo:
TIMESTAMP = 1707900121 SHA256 (tokenizers-0.15.2.tar.gz) = e6e9c6e019dd5484be5beafc775ae6c925f4c69a3487040ed09b45e13df2cb91 SIZE (tokenizers-0.15.2.tar.gz) = 320256

Expand this list (280 items)

Collapse this list.

SHA256 (rust/crates/aho-corasick-1.1.2.crate) = b2969dcb958b36655471fc61f7e416fa76033bdd4bfed0678d8fee1e2d07a1f0 SIZE (rust/crates/aho-corasick-1.1.2.crate) = 183136 SHA256 (rust/crates/anstream-0.6.11.crate) = 6e2e1ebcb11de5c03c67de28a7df593d32191b44939c482e97702baaaa6ab6a5 SIZE (rust/crates/anstream-0.6.11.crate) = 30239 SHA256 (rust/crates/anstyle-1.0.6.crate) = 8901269c6307e8d93993578286ac0edf7f195079ffff5ebdeea6a59ffb7e36bc SIZE (rust/crates/anstyle-1.0.6.crate) = 14604 SHA256 (rust/crates/anstyle-parse-0.2.3.crate) = c75ac65da39e5fe5ab759307499ddad880d724eed2f6ce5b5e8a26f4f387928c SIZE (rust/crates/anstyle-parse-0.2.3.crate) = 24699 SHA256 (rust/crates/anstyle-query-1.0.2.crate) = e28923312444cdd728e4738b3f9c9cac739500909bb3d3c94b43551b16517648 SIZE (rust/crates/anstyle-query-1.0.2.crate) = 8739 SHA256 (rust/crates/anstyle-wincon-3.0.2.crate) = 1cd54b81ec8d6180e24654d0b371ad22fc3dd083b6ff8ba325b72e00c87660a7 SIZE (rust/crates/anstyle-wincon-3.0.2.crate) = 11272 SHA256 (rust/crates/autocfg-1.1.0.crate) = d468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa SIZE (rust/crates/autocfg-1.1.0.crate) = 13272 SHA256 (rust/crates/base64-0.13.1.crate) = 9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8 SIZE (rust/crates/base64-0.13.1.crate) = 61002 SHA256 (rust/crates/bitflags-1.3.2.crate) = bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a SIZE (rust/crates/bitflags-1.3.2.crate) = 23021 SHA256 (rust/crates/bitflags-2.4.2.crate) = ed570934406eb16438a4e976b1b4500774099c13b8cb96eec99f620f05090ddf SIZE (rust/crates/bitflags-2.4.2.crate) = 42602 SHA256 (rust/crates/cc-1.0.83.crate) = f1174fb0b6ec23863f8b971027804a42614e347eafb0a95bf0b12cdae21fc4d0 SIZE (rust/crates/cc-1.0.83.crate) = 68343 SHA256 (rust/crates/cfg-if-1.0.0.crate) = baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd SIZE (rust/crates/cfg-if-1.0.0.crate) = 7934 SHA256 (rust/crates/clap-4.5.0.crate) = 80c21025abd42669a92efc996ef13cfb2c5c627858421ea58d5c3b331a6c134f SIZE (rust/crates/clap-4.5.0.crate) = 55379 SHA256 (rust/crates/clap_builder-4.5.0.crate) = 458bf1f341769dfcf849846f65dffdf9146daa56bcd2a47cb4e1de9915567c99 SIZE (rust/crates/clap_builder-4.5.0.crate) = 163521 SHA256 (rust/crates/clap_derive-4.5.0.crate) = 307bc0538d5f0f83b8248db3087aa92fe504e4691294d0c96c0eabc33f47ba47 SIZE (rust/crates/clap_derive-4.5.0.crate) = 29042 SHA256 (rust/crates/clap_lex-0.7.0.crate) = 98cc8fbded0c607b7ba9dd60cd98df59af97e84d24e49c8557331cfc26d301ce SIZE (rust/crates/clap_lex-0.7.0.crate) = 11915 SHA256 (rust/crates/colorchoice-1.0.0.crate) = acbf1af155f9b9ef647e42cdc158db4b64a1b61f743629225fde6f3e0be2a7c7 SIZE (rust/crates/colorchoice-1.0.0.crate) = 6857 SHA256 (rust/crates/console-0.15.8.crate) = 0e1f83fc076bd6dd27517eacdf25fef6c4dfe5f1d7448bafaaf3a26f13b5e4eb SIZE (rust/crates/console-0.15.8.crate) = 36364 SHA256 (rust/crates/crossbeam-deque-0.8.5.crate) = 613f8cc01fe9cf1a3eb3d7f488fd2fa8388403e97039e2f73692932e291a770d SIZE (rust/crates/crossbeam-deque-0.8.5.crate) = 21726 SHA256 (rust/crates/crossbeam-epoch-0.9.18.crate) = 5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e SIZE (rust/crates/crossbeam-epoch-0.9.18.crate) = 46875 SHA256 (rust/crates/crossbeam-utils-0.8.19.crate) = 248e3bacc7dc6baa3b21e405ee045c3047101a49145e7e9eca583ab4c2ca5345 SIZE (rust/crates/crossbeam-utils-0.8.19.crate) = 42328 SHA256 (rust/crates/darling-0.14.4.crate) = 7b750cb3417fd1b327431a470f388520309479ab0bf5e323505daf0290cd3850 SIZE (rust/crates/darling-0.14.4.crate) = 25168 SHA256 (rust/crates/darling_core-0.14.4.crate) = 109c1ca6e6b7f82cc233a97004ea8ed7ca123a9af07a8230878fcfda9b158bf0 SIZE (rust/crates/darling_core-0.14.4.crate) = 57485 SHA256 (rust/crates/darling_macro-0.14.4.crate) = a4aab4dbc9f7611d8b55048a3a16d2d010c2c8334e46304b40ac1cc14bf3b48e SIZE (rust/crates/darling_macro-0.14.4.crate) = 1896 SHA256 (rust/crates/derive_builder-0.12.0.crate) = 8d67778784b508018359cbc8696edb3db78160bab2c2a28ba7f56ef6932997f8 SIZE (rust/crates/derive_builder-0.12.0.crate) = 35456 SHA256 (rust/crates/derive_builder_core-0.12.0.crate) = c11bdc11a0c47bc7d37d582b5285da6849c96681023680b906673c5707af7b0f SIZE (rust/crates/derive_builder_core-0.12.0.crate) = 31438 SHA256 (rust/crates/derive_builder_macro-0.12.0.crate) = ebcda35c7a396850a55ffeac740804b40ffec779b98fffbb1738f4033f0ee79e SIZE (rust/crates/derive_builder_macro-0.12.0.crate) = 6288 SHA256 (rust/crates/either-1.10.0.crate) = 11157ac094ffbdde99aa67b23417ebdd801842852b500e395a45a9c0aac03e4a SIZE (rust/crates/either-1.10.0.crate) = 18334 SHA256 (rust/crates/encode_unicode-0.3.6.crate) = a357d28ed41a50f9c765dbfe56cbc04a64e53e5fc58ba79fbc34c10ef3df831f SIZE (rust/crates/encode_unicode-0.3.6.crate) = 45741 SHA256 (rust/crates/env_logger-0.10.2.crate) = 4cd405aab171cb85d6735e5c8d9db038c17d3ca007a4d2c25f337935c3d90580 SIZE (rust/crates/env_logger-0.10.2.crate) = 36402 SHA256 (rust/crates/errno-0.3.8.crate) = a258e46cdc063eb8519c00b9fc845fc47bcfca4130e2f08e88665ceda8474245 SIZE (rust/crates/errno-0.3.8.crate) = 10645 SHA256 (rust/crates/esaxx-rs-0.1.10.crate) = d817e038c30374a4bcb22f94d0a8a0e216958d4c3dcde369b1439fec4bdda6e6 SIZE (rust/crates/esaxx-rs-0.1.10.crate) = 175210 SHA256 (rust/crates/fastrand-2.0.1.crate) = 25cbce373ec4653f1a01a31e8a5e5ec0c622dc27ff9c4e6606eefef5cbbed4a5 SIZE (rust/crates/fastrand-2.0.1.crate) = 14664 SHA256 (rust/crates/fnv-1.0.7.crate) = 3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1 SIZE (rust/crates/fnv-1.0.7.crate) = 11266 SHA256 (rust/crates/getrandom-0.2.12.crate) = 190092ea657667030ac6a35e305e62fc4dd69fd98ac98631e5d3a2b1575a12b5 SIZE (rust/crates/getrandom-0.2.12.crate) = 36163 SHA256 (rust/crates/heck-0.4.1.crate) = 95505c38b4572b2d910cecb0281560f54b440a19336cbbcb27bf6ce6adc6f5a8 SIZE (rust/crates/heck-0.4.1.crate) = 11567 SHA256 (rust/crates/hermit-abi-0.3.5.crate) = d0c62115964e08cb8039170eb33c1d0e2388a256930279edca206fff675f82c3 SIZE (rust/crates/hermit-abi-0.3.5.crate) = 14437 SHA256 (rust/crates/humantime-2.1.0.crate) = 9a3a5bfb195931eeb336b2a7b4d761daec841b97f947d34394601737a7bba5e4 SIZE (rust/crates/humantime-2.1.0.crate) = 16749 SHA256 (rust/crates/ident_case-1.0.1.crate) = b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39 SIZE (rust/crates/ident_case-1.0.1.crate) = 3492 SHA256 (rust/crates/indicatif-0.17.8.crate) = 763a5a8f45087d6bcea4222e7b72c291a054edf80e4ef6efd2a4979878c7bea3 SIZE (rust/crates/indicatif-0.17.8.crate) = 64869 SHA256 (rust/crates/indoc-2.0.4.crate) = 1e186cfbae8084e513daff4240b4797e342f988cecda4fb6c939150f96315fd8 SIZE (rust/crates/indoc-2.0.4.crate) = 14311 SHA256 (rust/crates/instant-0.1.12.crate) = 7a5bbe824c507c5da5956355e86a746d82e0e1464f65d862cc5e71da70e94b2c SIZE (rust/crates/instant-0.1.12.crate) = 6128 SHA256 (rust/crates/is-terminal-0.4.12.crate) = f23ff5ef2b80d608d61efee834934d862cd92461afc0560dedf493e4c033738b SIZE (rust/crates/is-terminal-0.4.12.crate) = 7470 SHA256 (rust/crates/itertools-0.11.0.crate) = b1c173a5686ce8bfa551b3563d0c2170bf24ca44da99c7ca4bfdab5418c3fe57 SIZE (rust/crates/itertools-0.11.0.crate) = 125074 SHA256 (rust/crates/itertools-0.12.1.crate) = ba291022dbbd398a455acf126c1e341954079855bc60dfdda641363bd6922569 SIZE (rust/crates/itertools-0.12.1.crate) = 137761 SHA256 (rust/crates/itoa-1.0.10.crate) = b1a46d1a171d865aa5f83f92695765caa047a9b4cbae2cbf37dbd613a793fd4c SIZE (rust/crates/itoa-1.0.10.crate) = 10534 SHA256 (rust/crates/lazy_static-1.4.0.crate) = e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646 SIZE (rust/crates/lazy_static-1.4.0.crate) = 10443 SHA256 (rust/crates/libc-0.2.153.crate) = 9c198f91728a82281a64e1f4f9eeb25d82cb32a5de251c6bd1b5154d63a8e7bd SIZE (rust/crates/libc-0.2.153.crate) = 740614 SHA256 (rust/crates/linux-raw-sys-0.4.13.crate) = 01cda141df6706de531b6c46c3a33ecca755538219bd484262fa09410c13539c SIZE (rust/crates/linux-raw-sys-0.4.13.crate) = 1493855 SHA256 (rust/crates/lock_api-0.4.11.crate) = 3c168f8615b12bc01f9c17e2eb0cc07dcae1940121185446edc3744920e8ef45 SIZE (rust/crates/lock_api-0.4.11.crate) = 27487 SHA256 (rust/crates/log-0.4.20.crate) = b5e6163cb8c49088c2c36f57875e58ccd8c87c7427f7fbd50ea6710b2f3f2e8f SIZE (rust/crates/log-0.4.20.crate) = 38307 SHA256 (rust/crates/macro_rules_attribute-0.2.0.crate) = 8a82271f7bc033d84bbca59a3ce3e4159938cb08a9c3aebbe54d215131518a13 SIZE (rust/crates/macro_rules_attribute-0.2.0.crate) = 15408 SHA256 (rust/crates/macro_rules_attribute-proc_macro-0.2.0.crate) = b8dd856d451cc0da70e2ef2ce95a18e39a93b7558bedf10201ad28503f918568 SIZE (rust/crates/macro_rules_attribute-proc_macro-0.2.0.crate) = 8264 SHA256 (rust/crates/matrixmultiply-0.3.8.crate) = 7574c1cf36da4798ab73da5b215bbf444f50718207754cb522201d78d1cd0ff2 SIZE (rust/crates/matrixmultiply-0.3.8.crate) = 57530 SHA256 (rust/crates/memchr-2.7.1.crate) = 523dc4f511e55ab87b694dc30d0f820d60906ef06413f93d4d7a1385599cc149 SIZE (rust/crates/memchr-2.7.1.crate) = 96307 SHA256 (rust/crates/memoffset-0.9.0.crate) = 5a634b1c61a95585bd15607c6ab0c4e5b226e695ff2800ba0cdccddf208c406c SIZE (rust/crates/memoffset-0.9.0.crate) = 9033 SHA256 (rust/crates/minimal-lexical-0.2.1.crate) = 68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a SIZE (rust/crates/minimal-lexical-0.2.1.crate) = 94841 SHA256 (rust/crates/monostate-0.1.11.crate) = 878c2a1f1c70e5724fa28f101ca787b6a7e8ad5c5e4ae4ca3b0fa4a419fa9075 SIZE (rust/crates/monostate-0.1.11.crate) = 14003 SHA256 (rust/crates/monostate-impl-0.1.11.crate) = f686d68a09079e63b1d2c64aa305095887ce50565f00a922ebfaeeee0d9ba6ce SIZE (rust/crates/monostate-impl-0.1.11.crate) = 7187 SHA256 (rust/crates/ndarray-0.15.6.crate) = adb12d4e967ec485a5f71c6311fe28158e9d6f4bc4a447b474184d0f91a8fa32 SIZE (rust/crates/ndarray-0.15.6.crate) = 275225 SHA256 (rust/crates/nom-7.1.3.crate) = d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a SIZE (rust/crates/nom-7.1.3.crate) = 117570 SHA256 (rust/crates/num-complex-0.4.5.crate) = 23c6602fda94a57c990fe0df199a035d83576b496aa29f4e634a8ac6004e68a6 SIZE (rust/crates/num-complex-0.4.5.crate) = 29614 SHA256 (rust/crates/num-integer-0.1.46.crate) = 7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f SIZE (rust/crates/num-integer-0.1.46.crate) = 22331 SHA256 (rust/crates/num-traits-0.2.18.crate) = da0df0e5185db44f69b44f26786fe401b6c293d1907744beaa7fa62b2e5a517a SIZE (rust/crates/num-traits-0.2.18.crate) = 51930 SHA256 (rust/crates/number_prefix-0.4.0.crate) = 830b246a0e5f20af87141b25c173cd1b609bd7779a4617d6ec582abaf90870f3 SIZE (rust/crates/number_prefix-0.4.0.crate) = 6922 SHA256 (rust/crates/numpy-0.20.0.crate) = bef41cbb417ea83b30525259e30ccef6af39b31c240bda578889494c5392d331 SIZE (rust/crates/numpy-0.20.0.crate) = 71258 SHA256 (rust/crates/once_cell-1.19.0.crate) = 3fdb12b2476b595f9358c5161aa467c2438859caa136dec86c26fdd2efe17b92 SIZE (rust/crates/once_cell-1.19.0.crate) = 33046 SHA256 (rust/crates/onig-6.4.0.crate) = 8c4b31c8722ad9171c6d77d3557db078cab2bd50afcc9d09c8b315c59df8ca4f SIZE (rust/crates/onig-6.4.0.crate) = 32616 SHA256 (rust/crates/onig_sys-69.8.1.crate) = 7b829e3d7e9cc74c7e315ee8edb185bf4190da5acde74afd7fc59c35b1f086e7 SIZE (rust/crates/onig_sys-69.8.1.crate) = 638216 SHA256 (rust/crates/parking_lot-0.12.1.crate) = 3742b2c103b9f06bc9fff0a37ff4912935851bee6d36f3c02bcc755bcfec228f SIZE (rust/crates/parking_lot-0.12.1.crate) = 40967 SHA256 (rust/crates/parking_lot_core-0.9.9.crate) = 4c42a9226546d68acdd9c0a280d17ce19bfe27a46bf68784e4066115788d008e SIZE (rust/crates/parking_lot_core-0.9.9.crate) = 32445 SHA256 (rust/crates/paste-1.0.14.crate) = de3145af08024dea9fa9914f381a17b8fc6034dfb00f3a84013f7ff43f29ed4c SIZE (rust/crates/paste-1.0.14.crate) = 18157 SHA256 (rust/crates/pkg-config-0.3.29.crate) = 2900ede94e305130c13ddd391e0ab7cbaeb783945ae07a279c268cb05109c6cb SIZE (rust/crates/pkg-config-0.3.29.crate) = 20563 SHA256 (rust/crates/portable-atomic-1.6.0.crate) = 7170ef9988bc169ba16dd36a7fa041e5c4cbeb6a35b76d4c03daded371eae7c0 SIZE (rust/crates/portable-atomic-1.6.0.crate) = 140689 SHA256 (rust/crates/ppv-lite86-0.2.17.crate) = 5b40af805b3121feab8a3c29f04d8ad262fa8e0561883e7653e024ae4479e6de SIZE (rust/crates/ppv-lite86-0.2.17.crate) = 22242 SHA256 (rust/crates/proc-macro2-1.0.78.crate) = e2422ad645d89c99f8f3e6b88a9fdeca7fabeac836b1002371c4367c8f984aae SIZE (rust/crates/proc-macro2-1.0.78.crate) = 47158 SHA256 (rust/crates/pyo3-0.20.2.crate) = 9a89dc7a5850d0e983be1ec2a463a171d20990487c3cfcd68b5363f1ee3d6fe0 SIZE (rust/crates/pyo3-0.20.2.crate) = 434326 SHA256 (rust/crates/pyo3-build-config-0.20.2.crate) = 07426f0d8fe5a601f26293f300afd1a7b1ed5e78b2a705870c5f30893c5163be SIZE (rust/crates/pyo3-build-config-0.20.2.crate) = 30029 SHA256 (rust/crates/pyo3-ffi-0.20.2.crate) = dbb7dec17e17766b46bca4f1a4215a85006b4c2ecde122076c562dd058da6cf1 SIZE (rust/crates/pyo3-ffi-0.20.2.crate) = 64601 SHA256 (rust/crates/pyo3-macros-0.20.2.crate) = 05f738b4e40d50b5711957f142878cfa0f28e054aa0ebdfc3fd137a843f74ed3 SIZE (rust/crates/pyo3-macros-0.20.2.crate) = 7925 SHA256 (rust/crates/pyo3-macros-backend-0.20.2.crate) = 0fc910d4851847827daf9d6cdd4a823fbdaab5b8818325c5e97a86da79e8881f SIZE (rust/crates/pyo3-macros-backend-0.20.2.crate) = 49710 SHA256 (rust/crates/quote-1.0.35.crate) = 291ec9ab5efd934aaf503a6466c5d5251535d108ee747472c3977cc5acc868ef SIZE (rust/crates/quote-1.0.35.crate) = 28136 SHA256 (rust/crates/rand-0.8.5.crate) = 34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404 SIZE (rust/crates/rand-0.8.5.crate) = 87113 SHA256 (rust/crates/rand_chacha-0.3.1.crate) = e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88 SIZE (rust/crates/rand_chacha-0.3.1.crate) = 15251 SHA256 (rust/crates/rand_core-0.6.4.crate) = ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c SIZE (rust/crates/rand_core-0.6.4.crate) = 22666 SHA256 (rust/crates/rawpointer-0.2.1.crate) = 60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3 SIZE (rust/crates/rawpointer-0.2.1.crate) = 7490 SHA256 (rust/crates/rayon-1.8.1.crate) = fa7237101a77a10773db45d62004a272517633fbcc3df19d96455ede1122e051 SIZE (rust/crates/rayon-1.8.1.crate) = 170448 SHA256 (rust/crates/rayon-cond-0.3.0.crate) = 059f538b55efd2309c9794130bc149c6a553db90e9d99c2030785c82f0bd7df9 SIZE (rust/crates/rayon-cond-0.3.0.crate) = 9913 SHA256 (rust/crates/rayon-core-1.12.1.crate) = 1465873a3dfdaa8ae7cb14b4383657caab0b3e8a0aa9ae8e04b044854c8dfce2 SIZE (rust/crates/rayon-core-1.12.1.crate) = 70701 SHA256 (rust/crates/redox_syscall-0.4.1.crate) = 4722d768eff46b75989dd134e5c353f0d6296e5aaa3132e776cbdb56be7731aa SIZE (rust/crates/redox_syscall-0.4.1.crate) = 24858 SHA256 (rust/crates/regex-1.10.3.crate) = b62dbe01f0b06f9d8dc7d49e05a0785f153b00b2c227856282f671e0318c9b15 SIZE (rust/crates/regex-1.10.3.crate) = 253101 SHA256 (rust/crates/regex-automata-0.4.5.crate) = 5bb987efffd3c6d0d8f5f89510bb458559eab11e4f869acb20bf845e016259cd SIZE (rust/crates/regex-automata-0.4.5.crate) = 617406 SHA256 (rust/crates/regex-syntax-0.8.2.crate) = c08c74e62047bb2de4ff487b251e4a92e24f48745648451635cec7d591162d9f SIZE (rust/crates/regex-syntax-0.8.2.crate) = 347228 SHA256 (rust/crates/rustc-hash-1.1.0.crate) = 08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2 SIZE (rust/crates/rustc-hash-1.1.0.crate) = 9331 SHA256 (rust/crates/rustix-0.38.31.crate) = 6ea3e1a662af26cd7a3ba09c0297a31af215563ecf42817c98df621387f4e949 SIZE (rust/crates/rustix-0.38.31.crate) = 375443 SHA256 (rust/crates/ryu-1.0.16.crate) = f98d2aa92eebf49b69786be48e4477826b256916e84a57ff2a4f21923b48eb4c SIZE (rust/crates/ryu-1.0.16.crate) = 47351 SHA256 (rust/crates/scopeguard-1.2.0.crate) = 94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49 SIZE (rust/crates/scopeguard-1.2.0.crate) = 11619 SHA256 (rust/crates/serde-1.0.196.crate) = 870026e60fa08c69f064aa766c10f10b1d62db9ccd4d0abb206472bee0ce3b32 SIZE (rust/crates/serde-1.0.196.crate) = 77097 SHA256 (rust/crates/serde_derive-1.0.196.crate) = 33c85360c95e7d137454dc81d9a4ed2b8efd8fbe19cee57357b32b9771fccb67 SIZE (rust/crates/serde_derive-1.0.196.crate) = 55775 SHA256 (rust/crates/serde_json-1.0.113.crate) = 69801b70b1c3dac963ecb03a364ba0ceda9cf60c71cfe475e99864759c8b8a79 SIZE (rust/crates/serde_json-1.0.113.crate) = 146970 SHA256 (rust/crates/smallvec-1.13.1.crate) = e6ecd384b10a64542d77071bd64bd7b231f4ed5940fba55e98c3de13824cf3d7 SIZE (rust/crates/smallvec-1.13.1.crate) = 34952 SHA256 (rust/crates/spm_precompiled-0.1.4.crate) = 5851699c4033c63636f7ea4cf7b7c1f1bf06d0cc03cfb42e711de5a5c46cf326 SIZE (rust/crates/spm_precompiled-0.1.4.crate) = 557527 SHA256 (rust/crates/strsim-0.10.0.crate) = 73473c0e59e6d5812c5dfe2a064a6444949f089e20eec9a2e5506596494e4623 SIZE (rust/crates/strsim-0.10.0.crate) = 11355 SHA256 (rust/crates/strsim-0.11.0.crate) = 5ee073c9e4cd00e28217186dbe12796d692868f432bf2e97ee73bed0c56dfa01 SIZE (rust/crates/strsim-0.11.0.crate) = 13710 SHA256 (rust/crates/syn-1.0.109.crate) = 72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237 SIZE (rust/crates/syn-1.0.109.crate) = 237611 SHA256 (rust/crates/syn-2.0.48.crate) = 0f3531638e407dfc0814761abb7c00a5b54992b849452a0646b7f65c9f770f3f SIZE (rust/crates/syn-2.0.48.crate) = 250566 SHA256 (rust/crates/target-lexicon-0.12.13.crate) = 69758bda2e78f098e4ccb393021a0963bb3442eac05f135c30f61b7370bbafae SIZE (rust/crates/target-lexicon-0.12.13.crate) = 25424 SHA256 (rust/crates/tempfile-3.10.0.crate) = a365e8cd18e44762ef95d87f284f4b5cd04107fec2ff3052bd6a3e6069669e67 SIZE (rust/crates/tempfile-3.10.0.crate) = 33376 SHA256 (rust/crates/termcolor-1.4.1.crate) = 06794f8f6c5c898b3275aebefa6b8a1cb24cd2c6c79397ab15774837a0bc5755 SIZE (rust/crates/termcolor-1.4.1.crate) = 18773 SHA256 (rust/crates/thiserror-1.0.56.crate) = d54378c645627613241d077a3a79db965db602882668f9136ac42af9ecb730ad SIZE (rust/crates/thiserror-1.0.56.crate) = 20592 SHA256 (rust/crates/thiserror-impl-1.0.56.crate) = fa0faa943b50f3db30a20aa7e265dbc66076993efed8463e8de414e5d06d3471 SIZE (rust/crates/thiserror-impl-1.0.56.crate) = 15367 SHA256 (rust/crates/unicode-ident-1.0.12.crate) = 3354b9ac3fae1ff6755cb6db53683adb661634f67557942dea4facebec0fee4b SIZE (rust/crates/unicode-ident-1.0.12.crate) = 42168 SHA256 (rust/crates/unicode-normalization-alignments-0.1.12.crate) = 43f613e4fa046e69818dd287fdc4bc78175ff20331479dab6e1b0f98d57062de SIZE (rust/crates/unicode-normalization-alignments-0.1.12.crate) = 91546 SHA256 (rust/crates/unicode-segmentation-1.11.0.crate) = d4c87d22b6e3f4a18d4d40ef354e97c90fcb14dd91d7dc0aa9d8a1172ebf7202 SIZE (rust/crates/unicode-segmentation-1.11.0.crate) = 102740 SHA256 (rust/crates/unicode-width-0.1.11.crate) = e51733f11c9c4f72aa0c160008246859e340b00807569a0da0e7a1079b27ba85 SIZE (rust/crates/unicode-width-0.1.11.crate) = 19187 SHA256 (rust/crates/unicode_categories-0.1.1.crate) = 39ec24b3121d976906ece63c9daad25b85969647682eee313cb5779fdd69e14e SIZE (rust/crates/unicode_categories-0.1.1.crate) = 87298 SHA256 (rust/crates/unindent-0.2.3.crate) = c7de7d73e1754487cb58364ee906a499937a0dfabd86bcb980fa99ec8c8fa2ce SIZE (rust/crates/unindent-0.2.3.crate) = 7306 SHA256 (rust/crates/utf8parse-0.2.1.crate) = 711b9620af191e0cdc7468a8d14e709c3dcdb115b36f838e601583af800a370a SIZE (rust/crates/utf8parse-0.2.1.crate) = 13435 SHA256 (rust/crates/wasi-0.11.0+wasi-snapshot-preview1.crate) = 9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423 SIZE (rust/crates/wasi-0.11.0+wasi-snapshot-preview1.crate) = 28131 SHA256 (rust/crates/winapi-0.3.9.crate) = 5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419 SIZE (rust/crates/winapi-0.3.9.crate) = 1200382 SHA256 (rust/crates/winapi-i686-pc-windows-gnu-0.4.0.crate) = ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6 SIZE (rust/crates/winapi-i686-pc-windows-gnu-0.4.0.crate) = 2918815 SHA256 (rust/crates/winapi-util-0.1.6.crate) = f29e6f9198ba0d26b4c9f07dbe6f9ed633e1f3d5b8b414090084349e46a52596 SIZE (rust/crates/winapi-util-0.1.6.crate) = 12234 SHA256 (rust/crates/winapi-x86_64-pc-windows-gnu-0.4.0.crate) = 712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f SIZE (rust/crates/winapi-x86_64-pc-windows-gnu-0.4.0.crate) = 2947998 SHA256 (rust/crates/windows-sys-0.52.0.crate) = 282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d SIZE (rust/crates/windows-sys-0.52.0.crate) = 2576877 SHA256 (rust/crates/windows-targets-0.48.5.crate) = 9a2fa6e2155d7247be68c096456083145c183cbbbc2764150dda45a87197940c SIZE (rust/crates/windows-targets-0.48.5.crate) = 6904 SHA256 (rust/crates/windows-targets-0.52.0.crate) = 8a18201040b24831fbb9e4eb208f8892e1f50a37feb53cc7ff887feb8f50e7cd SIZE (rust/crates/windows-targets-0.52.0.crate) = 6229 SHA256 (rust/crates/windows_aarch64_gnullvm-0.48.5.crate) = 2b38e32f0abccf9987a4e3079dfb67dcd799fb61361e53e2882c3cbaf0d905d8 SIZE (rust/crates/windows_aarch64_gnullvm-0.48.5.crate) = 418492 SHA256 (rust/crates/windows_aarch64_gnullvm-0.52.0.crate) = cb7764e35d4db8a7921e09562a0304bf2f93e0a51bfccee0bd0bb0b666b015ea SIZE (rust/crates/windows_aarch64_gnullvm-0.52.0.crate) = 430182 SHA256 (rust/crates/windows_aarch64_msvc-0.48.5.crate) = dc35310971f3b2dbbf3f0690a219f40e2d9afcf64f9ab7cc1be722937c26b4bc SIZE (rust/crates/windows_aarch64_msvc-0.48.5.crate) = 798483 SHA256 (rust/crates/windows_aarch64_msvc-0.52.0.crate) = bbaa0368d4f1d2aaefc55b6fcfee13f41544ddf36801e793edbbfd7d7df075ef SIZE (rust/crates/windows_aarch64_msvc-0.52.0.crate) = 821663 SHA256 (rust/crates/windows_i686_gnu-0.48.5.crate) = a75915e7def60c94dcef72200b9a8e58e5091744960da64ec734a6c6e9b3743e SIZE (rust/crates/windows_i686_gnu-0.48.5.crate) = 844891 SHA256 (rust/crates/windows_i686_gnu-0.52.0.crate) = a28637cb1fa3560a16915793afb20081aba2c92ee8af57b4d5f28e4b3e7df313 SIZE (rust/crates/windows_i686_gnu-0.52.0.crate) = 870285 SHA256 (rust/crates/windows_i686_msvc-0.48.5.crate) = 8f55c233f70c4b27f66c523580f78f1004e8b5a8b659e05a4eb49d4166cca406 SIZE (rust/crates/windows_i686_msvc-0.48.5.crate) = 864300 SHA256 (rust/crates/windows_i686_msvc-0.52.0.crate) = ffe5e8e31046ce6230cc7215707b816e339ff4d4d67c65dffa206fd0f7aa7b9a SIZE (rust/crates/windows_i686_msvc-0.52.0.crate) = 888693 SHA256 (rust/crates/windows_x86_64_gnu-0.48.5.crate) = 53d40abd2583d23e4718fddf1ebec84dbff8381c07cae67ff7768bbf19c6718e SIZE (rust/crates/windows_x86_64_gnu-0.48.5.crate) = 801619 SHA256 (rust/crates/windows_x86_64_gnu-0.52.0.crate) = 3d6fa32db2bc4a2f5abeacf2b69f7992cd09dca97498da74a151a3132c26befd SIZE (rust/crates/windows_x86_64_gnu-0.52.0.crate) = 826213 SHA256 (rust/crates/windows_x86_64_gnullvm-0.48.5.crate) = 0b7b52767868a23d5bab768e390dc5f5c55825b6d30b86c844ff2dc7414044cc SIZE (rust/crates/windows_x86_64_gnullvm-0.48.5.crate) = 418486 SHA256 (rust/crates/windows_x86_64_gnullvm-0.52.0.crate) = 1a657e1e9d3f514745a572a6846d3c7aa7dbe1658c056ed9c3344c4109a6949e SIZE (rust/crates/windows_x86_64_gnullvm-0.52.0.crate) = 430165 SHA256 (rust/crates/windows_x86_64_msvc-0.48.5.crate) = ed94fce61571a4006852b7389a063ab983c02eb1bb37b47f8272ce92d06d9538 SIZE (rust/crates/windows_x86_64_msvc-0.48.5.crate) = 798412 SHA256 (rust/crates/windows_x86_64_msvc-0.52.0.crate) = dff9641d1cd4be8d1a070daf9e3773c5f67e78b4d9d42263020c057706765c04 SIZE (rust/crates/windows_x86_64_msvc-0.52.0.crate) = 821600

Collapse this list.


Packages (timestamps in pop-ups are UTC):
py39-tokenizers
ABIlatestquarterly
FreeBSD:13:aarch640.15.1-
FreeBSD:13:amd640.15.2_1-
FreeBSD:13:armv6--
FreeBSD:13:armv70.15.1-
FreeBSD:13:i3860.15.2_1-
FreeBSD:13:powerpc--
FreeBSD:13:powerpc64--
FreeBSD:13:powerpc64le--
FreeBSD:14:aarch640.15.1-
FreeBSD:14:amd640.15.2_1-
FreeBSD:14:armv6--
FreeBSD:14:armv70.15.1-
FreeBSD:14:i3860.15.2_1-
FreeBSD:14:powerpc--
FreeBSD:14:powerpc64--
FreeBSD:14:powerpc64le--
FreeBSD:15:aarch640.15.2_1-
FreeBSD:15:amd640.15.2_1-
FreeBSD:15:armv70.15.1-
FreeBSD:15:powerpc--
FreeBSD:15:powerpc64--
FreeBSD:15:powerpc64le--
 
Dependencies
NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered.
Build dependencies:
  1. py39-maturin>=1.0<2.0 : devel/py-maturin@py39
  2. rust>=1.76.0 : lang/rust
  3. pkgconf>=1.3.0_1 : devel/pkgconf
  4. python3.9 : lang/python39
  5. py39-build>=0 : devel/py-build@py39
  6. py39-installer>=0 : devel/py-installer@py39
Test dependencies:
  1. oniguruma.pc : devel/oniguruma
  2. python3.9 : lang/python39
Runtime dependencies:
  1. py39-huggingface-hub>=0.16.4<1.0 : misc/py-huggingface-hub@py39
  2. python3.9 : lang/python39
This port is required by:
for Run
  1. misc/py-litellm

Configuration Options:
No options to configure
Options name:
textproc_py-tokenizers
USES:
cargo python
FreshPorts was unable to extract/find any pkg message
Master Sites:
Expand this list (2 items)
Collapse this list.
  1. https://files.pythonhosted.org/packages/source/t/tokenizers/
  2. https://pypi.org/packages/source/t/tokenizers/
Collapse this list.

Number of commits found: 3

Commit History - (may be incomplete: for full details, see links to repositories near top of page)
CommitCreditsLog message
0.15.2_1
19 Feb 2024 11:59:23
commit hash: 7f5a7b0e524c8d32727681a4324905e350dc433ccommit hash: 7f5a7b0e524c8d32727681a4324905e350dc433ccommit hash: 7f5a7b0e524c8d32727681a4324905e350dc433ccommit hash: 7f5a7b0e524c8d32727681a4324905e350dc433c files touched by this commit
Mikael Urankar (mikael) search for other commits by this committer
lang/rust: Bump revisions after 1.76.0

PR:		276920
0.15.2
14 Feb 2024 09:17:15
commit hash: f6fa55bd8e81236b4dfc09c6ea2e8931792369b9commit hash: f6fa55bd8e81236b4dfc09c6ea2e8931792369b9commit hash: f6fa55bd8e81236b4dfc09c6ea2e8931792369b9commit hash: f6fa55bd8e81236b4dfc09c6ea2e8931792369b9 files touched by this commit
Hiroki Tagato (tagattie) search for other commits by this committer
textproc/py-tokenizers: update to 0.15.2

While here, enable tests.

Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.15.2

Reported by:	portscout
0.15.1
12 Feb 2024 08:34:14
commit hash: e3dfc2fad4565873d807ba820a24094b97890f98commit hash: e3dfc2fad4565873d807ba820a24094b97890f98commit hash: e3dfc2fad4565873d807ba820a24094b97890f98commit hash: e3dfc2fad4565873d807ba820a24094b97890f98 files touched by this commit
Hiroki Tagato (tagattie) search for other commits by this committer
textproc/py-tokenizers: add port: Fast state-of-the-art tokenizers optimized for
research and production

Provides an implementation of today's most used tokenizers, with a
focus on performance and versatility.

Main features:
- Train new vocabularies and tokenize, using today's most used
  tokenizers.
- Extremely fast (both training and tokenization), thanks to the Rust
  implementation. Takes less than 20 seconds to tokenize a GB of text
  on a server's CPU.
- Easy to use, but also extremely versatile.
- Designed for research and production.
- Normalization comes with alignments tracking. It's always possible
  to get the part of the original sentence that corresponds to a given
  token.
- Does all the pre-processing: Truncate, Pad, add the special tokens
  your model needs.

WWW: https://github.com/huggingface/tokenizers

Number of commits found: 3