notbugAs an Amazon Associate I earn from qualifying purchases.
Want a good read? Try FreeBSD Mastery: Jails (IT Mastery Book 15)
Port details
py-scrapy High level scraping and web crawling framework
2.5.0 www on this many watch lists=1 search for ports that depend on this port Find issues related to this port Report an issue related to this port View this port on Repology. pkg-fallout 2.5.0Version of this port present on the latest quarterly branch.
Maintainer: skreuzer@FreeBSD.org search for ports maintained by this maintainer
Port Added: 2009-12-14 02:58:59
Last Update: 2021-09-21 12:58:08
Commit Hash: 6b9696f
People watching this port, also watch:: logwatch, sshguard, py38-gunicorn
Also Listed In: python
License: BSD3CLAUSE
Description:
SVNWeb : git : Homepage
pkg-plist: as obtained via: make generate-plist
There is no configure plist information for this port.
Dependency lines:
  • ${PYTHON_PKGNAMEPREFIX}Scrapy>0:www/py-scrapy@${PY_FLAVOR}
To install the port: cd /usr/ports/www/py-scrapy/ && make install clean
To add the package, run one of these commands:
  • pkg install www/py-scrapy
  • pkg install py38-Scrapy
NOTE: This is a Python port. Instead of py38-Scrapy listed in the above command, you can pick from the names under the Packages section.
PKGNAME: py38-Scrapy
Package flavors (<flavor>: <package>)
  • py38: py38-Scrapy
distinfo:
Packages (timestamps in pop-ups are UTC):
py27-Scrapy
ABIlatestquarterly
FreeBSD:11:aarch641.3.3_21.6.0_2
FreeBSD:11:amd64--
FreeBSD:11:armv60.22.21.6.0_2
FreeBSD:11:i386--
FreeBSD:11:mips--
FreeBSD:11:mips640.22.2-
FreeBSD:12:aarch641.3.3_2-
FreeBSD:12:amd64--
FreeBSD:12:armv61.3.3_21.6.0_2
FreeBSD:12:armv71.3.3_21.6.0_2
FreeBSD:12:i386--
FreeBSD:12:mips--
FreeBSD:12:mips641.3.3_2-
FreeBSD:12:powerpc64--
FreeBSD:13:aarch64--
FreeBSD:13:amd64--
FreeBSD:13:armv6--
FreeBSD:13:armv7--
FreeBSD:13:i386--
FreeBSD:13:mips--
FreeBSD:13:mips64--
FreeBSD:13:powerpc64--
FreeBSD:14:aarch64--
FreeBSD:14:amd64--
FreeBSD:14:armv6--
FreeBSD:14:armv7--
FreeBSD:14:i386--
FreeBSD:14:mips--
FreeBSD:14:mips64--
FreeBSD:14:powerpc64--
 
py36-Scrapy
ABIlatestquarterly
FreeBSD:11:aarch641.3.3_2-
FreeBSD:11:amd64--
FreeBSD:11:armv6--
FreeBSD:11:i386--
FreeBSD:11:mips--
FreeBSD:11:mips64--
FreeBSD:12:aarch641.3.3_2-
FreeBSD:12:amd64--
FreeBSD:12:armv61.3.3_2-
FreeBSD:12:armv71.3.3_2-
FreeBSD:12:i386--
FreeBSD:12:mips--
FreeBSD:12:mips641.3.3_2-
FreeBSD:12:powerpc64--
FreeBSD:13:aarch64--
FreeBSD:13:amd64--
FreeBSD:13:armv6--
FreeBSD:13:armv7--
FreeBSD:13:i386--
FreeBSD:13:mips--
FreeBSD:13:mips64--
FreeBSD:13:powerpc64--
FreeBSD:14:aarch64--
FreeBSD:14:amd64--
FreeBSD:14:armv6--
FreeBSD:14:armv7--
FreeBSD:14:i386--
FreeBSD:14:mips--
FreeBSD:14:mips64--
FreeBSD:14:powerpc64--
 
py37-Scrapy
ABIlatestquarterly
FreeBSD:11:aarch64-1.6.0_2
FreeBSD:11:amd64--
FreeBSD:11:armv6-1.6.0_2
FreeBSD:11:i386--
FreeBSD:11:mips--
FreeBSD:11:mips64--
FreeBSD:12:aarch64--
FreeBSD:12:amd64--
FreeBSD:12:armv6-1.6.0_2
FreeBSD:12:armv7-1.6.0_2
FreeBSD:12:i386--
FreeBSD:12:mips--
FreeBSD:12:mips64--
FreeBSD:12:powerpc64-1.6.0_2
FreeBSD:13:aarch641.6.0_2-
FreeBSD:13:amd64--
FreeBSD:13:armv61.6.0_21.6.0_2
FreeBSD:13:armv71.6.0_21.6.0_2
FreeBSD:13:i386--
FreeBSD:13:mips--
FreeBSD:13:mips64--
FreeBSD:13:powerpc641.6.0_2-
FreeBSD:14:aarch64--
FreeBSD:14:amd64--
FreeBSD:14:armv61.6.0_2-
FreeBSD:14:armv71.6.0_2-
FreeBSD:14:i386--
FreeBSD:14:mips--
FreeBSD:14:mips64--
FreeBSD:14:powerpc641.6.0_2-
 
py38-Scrapy
ABIlatestquarterly
FreeBSD:11:aarch64--
FreeBSD:11:amd642.5.02.5.0
FreeBSD:11:armv6--
FreeBSD:11:i3862.5.02.5.0
FreeBSD:11:mips--
FreeBSD:11:mips64--
FreeBSD:12:aarch64-2.5.0
FreeBSD:12:amd642.5.02.5.0
FreeBSD:12:armv6--
FreeBSD:12:armv7--
FreeBSD:12:i3862.5.02.5.0
FreeBSD:12:mips--
FreeBSD:12:mips64--
FreeBSD:12:powerpc64--
FreeBSD:13:aarch64-2.5.0
FreeBSD:13:amd642.5.02.5.0
FreeBSD:13:armv6--
FreeBSD:13:armv7--
FreeBSD:13:i3862.5.02.5.0
FreeBSD:13:mips--
FreeBSD:13:mips64--
FreeBSD:13:powerpc64-2.5.0
FreeBSD:14:aarch642.5.0-
FreeBSD:14:amd642.5.0-
FreeBSD:14:armv6--
FreeBSD:14:armv7--
FreeBSD:14:i3862.5.0-
FreeBSD:14:mips--
FreeBSD:14:mips64--
FreeBSD:14:powerpc64--
 

Dependencies
NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered.
Build dependencies:
  1. py38-setuptools>0 : devel/py-setuptools@py38
  2. python3.8 : lang/python38
Runtime dependencies:
  1. py38-cryptography>=2.0 : security/py-cryptography@py38
  2. py38-cssselect>=0.9.1 : www/py-cssselect@py38
  3. py38-itemloaders>=1.0.1 : devel/py-itemloaders@py38
  4. py38-parsel>=1.5 : textproc/py-parsel@py38
  5. py38-openssl>=16.2.0 : security/py-openssl@py38
  6. py38-queuelib>=1.4.2 : sysutils/py-queuelib@py38
  7. py38-service_identity>=16.0.0 : security/py-service_identity@py38
  8. py38-w3lib>=1.17.0 : www/py-w3lib@py38
  9. py38-zope.interface>=4.1.3 : devel/py-zope.interface@py38
  10. py38-protego>=0.1.15 : www/py-protego@py38
  11. py38-itemadapter>=0.1.0 : devel/py-itemadapter@py38
  12. py38-lxml>=3.5.0 : devel/py-lxml@py38
  13. py38-pydispatcher>=2.0.5 : devel/py-pydispatcher@py38
  14. py38-setuptools>0 : devel/py-setuptools@py38
  15. python3.8 : lang/python38
There are no ports dependent upon this port

Configuration Options:
Options name:

USES:

FreshPorts was unable to extract/find any pkg message
Master Sites:
Expand this list (2 items)
Collapse this list.
  1. https://files.pythonhosted.org/packages/source/S/Scrapy/
  2. https://pypi.org/packages/source/S/Scrapy/
Collapse this list.

Number of commits found: 33

Commit History - (may be incomplete: see SVNWeb link above for full details)
DateByDescription
21 Sep 2021 12:58:08
 files touched by this commit commit hash:6b9696f10c0b053b20af479dd7af68fe39bf3e51  2.5.0
decke search for other commits by this committer
www/py-scrapy: Add CPE information

Approved by:    portmgr (blanket)
26 Jun 2021 16:46:42
 files touched by this commit commit hash:a9ed9dd7a32fd90be5a71024854bb5a6197ac837  2.5.0
dbaio search for other commits by this committer
www/py-scrapy: Update to 2.5.0

Changelog:	https://docs.scrapy.org/en/latest/news.html?highlight=news#scrapy-2-5-0-2021-04-06

This also removes/disables the experimental HTTP/2 support, there are
issues with Twisted and h2 at this moment which breaks scrapy.

PR:		256259
Approved by:	maintainer timeout (skreuzer)
06 Apr 2021 14:31:07
 files touched by this commit commit hash:305f148f482daf30dcf728039d03d019f88344eb  1.6.0_2
mat search for other commits by this committer
Remove # $FreeBSD$ from Makefiles.
28 Dec 2020 23:02:15
Original commit files touched by this commit Revision:559531  1.6.0_2
antoine search for other commits by this committer
Drop python 2.7 support from a few ports

With hat:	portmgr
10 Apr 2019 06:18:32
Original commit files touched by this commit Revision:498530  1.6.0_2
antoine search for other commits by this committer
Bump a few PORTREVISIONs after r498529

With hat:	portmgr
23 Mar 2019 01:50:31
Original commit files touched by this commit Revision:496622  1.6.0_1
yuri search for other commits by this committer
textproc/py-scrapy: Fix the dependency version

PR:		235471
Submitted by:	wen
Approved by:	portmgr (port compliance)
16 Mar 2019 12:47:20
Original commit files touched by this commit Revision:495872  1.6.0
skreuzer search for other commits by this committer
Fix typo in RUN_DEPENDS

Require py-parsel >= 1.2.0
09 Mar 2019 00:55:41
Original commit files touched by this commit Revision:495093  1.6.0
skreuzer search for other commits by this committer
Update to 1.6.0

PR:		235471
Submitted by:	yuri@
20 Jun 2018 17:05:44
Original commit files touched by this commit Revision:472884  1.3.3_2
mat search for other commits by this committer
Use PY_FLAVOR for dependencies.

FLAVOR is the current port's flavor, it should not be used outside of
this scope.

Sponsored by:	Absolight
19 Jun 2018 01:41:42
Original commit files touched by this commit Revision:472739  1.3.3_2
yuri search for other commits by this committer
www/py-scrapy: Add USE_PYTHON=concurrent

Approved by:	portmgr
30 Nov 2017 15:50:34
Original commit files touched by this commit Revision:455210  1.3.3_1  Sanity Test Failure
mat search for other commits by this committer
Convert Python ports to FLAVORS.

  Ports using USE_PYTHON=distutils are now flavored.  They will
  automatically get flavors (py27, py34, py35, py36) depending on what
  versions they support.

  There is also a USE_PYTHON=flavors for ports that do not use distutils
  but need FLAVORS to be set.  A USE_PYTHON=noflavors can be set if
  using distutils but flavors are not wanted.

  A new USE_PYTHON=optsuffix that will add PYTHON_PKGNAMESUFFIX has been
  added to cope with Python ports that did not have the Python
  PKGNAMEPREFIX but are flavored.

  USES=python now also exports a PY_FLAVOR variable that contains the
(Only the first 15 lines of the commit message are shown above View all of this commit message)
07 Jun 2017 00:25:33
Original commit files touched by this commit Revision:442812  1.3.3_1
skreuzer search for other commits by this committer
Fix dependencies

PR:		219311
Submitted by:	Eric Camachat <eric@camachat.org>
12 May 2017 01:49:11
Original commit files touched by this commit Revision:440672  1.3.3
skreuzer search for other commits by this committer
Update to 1.3.3

PR:		219212
Submitted by:	Daniel Ylitalo
19 Jul 2016 13:24:52
Original commit files touched by this commit Revision:418782  1.1.1
skreuzer search for other commits by this committer
Update to version 1.1.1
01 Apr 2016 14:33:58
Original commit files touched by this commit Revision:412351  0.22.2
mat search for other commits by this committer
Remove ${PORTSDIR}/ from dependencies, categories v, w, x, y, and z.

With hat:	portmgr
Sponsored by:	Absolight
17 Aug 2015 14:20:41
Original commit files touched by this commit Revision:394508  0.22.2
mat search for other commits by this committer
Remove UNIQUENAME and LATEST_LINK.

UNIQUENAME was never unique, it was only used by USE_LDCONFIG and now,
we won't have conflicts there.

Use PKGBASE instead of LATEST_LINK in PKGLATESTFILE, the *only* consumer
is pkg-devel, and it works just fine without LATEST_LINK as pkg-devel
has the correct PKGNAME anyway.

Now that UNIQUENAME is gone, OPTIONSFILE is too. (it's been called
OPTIONS_FILE now.)

Reviewed by:	antoine, bapt
Exp-run by:	antoine
Sponsored by:	Absolight
Differential Revision:	https://reviews.freebsd.org/D3336
02 Feb 2015 18:24:48
Original commit files touched by this commit Revision:378312  0.22.2
skreuzer search for other commits by this committer
Adopt port
03 Jan 2015 08:41:21
Original commit files touched by this commit Revision:376109  0.22.2
mva search for other commits by this committer
- Convert USE_PYTHON to new USES in www/

With hat:	python@
02 Dec 2014 22:40:57
Original commit files touched by this commit Revision:373790  0.22.2
bapt search for other commits by this committer
Reset maintainership:
- Lots of maintainers timeout
- unresponsive for almost a year

PR:		195059
25 Nov 2014 21:48:20
Original commit files touched by this commit Revision:373434  0.22.2
skreuzer search for other commits by this committer
Update to version 0.22.2

PR:		189719
Submitted by:	amim.knabben@gmail.com
Approved by:	maintainer timeout (clsung, 28 weeks)
29 Jan 2014 13:21:32
Original commit files touched by this commit Revision:341733  0.16.5_1
miwi search for other commits by this committer
- Stage support
- Convert to PYDISTUTILS_AUTOPLIST
- Bump PORTREVISION
13 Jan 2014 21:00:04
Original commit files touched by this commit Revision:339634  0.16.5
rene search for other commits by this committer
Python cleanup:
- USE_PYTHON* = 2.X -> USE_PYTHON* = 2
- USE_PYTHON* = 2.X+ -> USE_PYTHON* = yes
Reviewed by:	python (mva, rm)
Approved by:	portmgr-lurkers (mat)
08 Dec 2013 12:04:08
Original commit files touched by this commit Revision:335877  0.16.5
mva search for other commits by this committer
New USES=twisted, to replace the old USE_TWISTED knob.

twisted can be configured with the arguments run or build to replace
the previous USE_TWISTED_RUN and USE_TWISTED_BUILD knobs. The twisted
components can be added as comma-separated arguments. If you previously
wrote
    USE_TWISTED=        yes
    USE_TWISTED=        conch names
    USE_TWISTED_RUN=    yes
you now would write
    USES=       twisted
    USES=       twisted:conch,names
    USES=       twisted:run
20 Sep 2013 23:36:54
Original commit files touched by this commit Revision:327776  0.16.5
bapt search for other commits by this committer
Add NO_STAGE all over the place in preparation for the staging support (cat:
www)
21 Jun 2013 06:17:24
Original commit files touched by this commit Revision:321444  0.16.5
clsung search for other commits by this committer
- Update to 0.16.5
12 Feb 2013 12:46:23
Original commit files touched by this commit Revision:312109  0.16.4
clsung search for other commits by this committer
- Update to 0.16.4
02 Dec 2012 10:58:49
Original commit files touched by this commit Revision:308097  0.16.2
rm search for other commits by this committer
- update to 0.16.2
- trim Makefile header
- remove indefinite article from COMMENT
- add LICENSE
- remove not needed BUILD_DEPENDS
- depend on TwistedCore via USE_TWISTED_RUN instead of direct dependency on
meta-port
- convert to optionsng
- convert PYTHON_SITELIBDIR usage with PYTHON_PKGNAMEPREFIX

PR:		173661
Submitted by:	rm (myself)
Approved by:	maintainer timeout (2 weeks+)
Feature safe:	yes
18 Nov 2011 02:22:51
Original commit files touched by this commit   0.12.0.2542
clsung search for other commits by this committer
- Unbreak
- take maintainership

PR:             ports/162557
Submitted by:   clsung
Approved by:    maintainer (qingfeng_AT_douban.com)
Feature safe:   yes
11 Jul 2011 20:48:01
Original commit files touched by this commit   0.12.0.2542
pav search for other commits by this committer
- Mark BROKEN: does not compile

Reported by:    pointyhat
28 Jun 2011 04:27:35
Original commit files touched by this commit   0.12.0.2542
delphij search for other commits by this committer
Update to version 0.12.0.2542, change maintainer email address to a new
one.

PR:             ports/158370
Submitted by:   maintainer
19 Mar 2011 12:38:54
Original commit files touched by this commit   0.10.3
miwi search for other commits by this committer
- Get Rid MD5 support
02 Oct 2010 07:55:19
Original commit files touched by this commit   0.10.3
lwhsu search for other commits by this committer
- Update to 0.10.3
- Add sqlite to RUN_DEPENDS
- Add support for SSL and JSON

PR:             ports/149884 (based on)
Submitted by:   lwhsu
Approved by:    maintainer timeout
14 Dec 2009 02:58:44
Original commit files touched by this commit   0.8
wen search for other commits by this committer
Scrapy is a high level scraping and web crawling framework for writing
spiders to crawl and parse web pages for all kinds of purposes, from
information retrieval to monitoring or testing web sites.

WWW: http://scrapy.org/

PR:             ports/136811
Submitted by:   Qing Feng <qingfeng@me.com>

Number of commits found: 33