notbugAs an Amazon Associate I earn from qualifying purchases.
Want a good read? Try FreeBSD Mastery: Jails (IT Mastery Book 15)

Why was the website so slow for so long?

The cause of the slowdown was a change to the ZFS dataset. In conjunction with the database server, very little caching was being done. This combination resulted in increased disk I/O as the system churned through the database. Details in the blog post which outlines the various things which changed. Many graphs.

Port details
p5-Web-Scraper Web Scraping Toolkit inspired by Scrapi
0.38 www on this many watch lists=1 search for ports that depend on this port Find issues related to this port Report an issue related to this port 0.38Version of this port present on the latest quarterly branch.
Maintainer: perl@FreeBSD.org search for ports maintained by this maintainer
Port Added: 2007-12-12 08:15:22
Last Update: 2019-07-05 20:15:24
SVN Revision: 505931

People watching this port, also watch: codespell, xtruco, p5-Test-SharedFork

Also Listed In: perl5
License: ART10 GPLv1+
Description:
SVNWeb : Homepage
pkg-plist: as obtained via: make generate-plist
Expand this list (11 items)
  1. /usr/local/share/licenses/p5-Web-Scraper-0.38/catalog.mk
  2. /usr/local/share/licenses/p5-Web-Scraper-0.38/LICENSE
  3. /usr/local/share/licenses/p5-Web-Scraper-0.38/ART10
  4. /usr/local/share/licenses/p5-Web-Scraper-0.38/GPLv1+
  5. bin/scraper
  6. lib/perl5/site_perl/Web/Scraper.pm
  7. lib/perl5/site_perl/Web/Scraper/Filter.pm
  8. lib/perl5/site_perl/Web/Scraper/LibXML.pm
  9. lib/perl5/site_perl/man/man3/Web::Scraper.3.gz
  10. lib/perl5/site_perl/man/man3/Web::Scraper::Filter.3.gz
  11. lib/perl5/site_perl/man/man3/Web::Scraper::LibXML.3.gz
Collapse this list.
Dependency lines:
  • p5-Web-Scraper>0:www/p5-Web-Scraper
To install the port: cd /usr/ports/www/p5-Web-Scraper/ && make install clean
To add the package: pkg install p5-Web-Scraper
PKGNAME: p5-Web-Scraper
Flavors: there is no flavor information for this port.
distinfo:
Packages:
p5-Web-Scraper
ABIlatestquarterly
FreeBSD:11:aarch640.380.38
FreeBSD:11:amd640.380.38
FreeBSD:11:i3860.380.38
FreeBSD:12:aarch640.380.38
FreeBSD:12:amd640.380.38
FreeBSD:12:i3860.380.38
FreeBSD:13:aarch640.38-
FreeBSD:13:amd640.38-
FreeBSD:13:i3860.38-
 

Dependencies
NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered.
Build dependencies:
  1. p5-HTML-Parser>0 : www/p5-HTML-Parser
  2. p5-HTML-Selector-XPath>=0.03 : www/p5-HTML-Selector-XPath
  3. p5-HTML-Tagset>0 : www/p5-HTML-Tagset
  4. p5-HTML-TreeBuilder-LibXML>=0.04 : www/p5-HTML-TreeBuilder-LibXML
  5. p5-HTML-TreeBuilder-XPath>=0.08 : www/p5-HTML-TreeBuilder-XPath
  6. p5-HTML-Tree>=3.23 : www/p5-HTML-Tree
  7. p5-UNIVERSAL-require>0 : devel/p5-UNIVERSAL-require
  8. p5-URI>0 : net/p5-URI
  9. p5-XML-XPathEngine>=0.08 : textproc/p5-XML-XPathEngine
  10. p5-YAML>0 : textproc/p5-YAML
  11. p5-libwww>=5.827 : www/p5-libwww
  12. perl5>=5.28.r1<5.29 : lang/perl5.28
  13. p5-Module-Build-Tiny>=0.039 : devel/p5-Module-Build-Tiny
  14. perl5>=5.28.r1<5.29 : lang/perl5.28
Runtime dependencies:
  1. p5-HTML-Parser>0 : www/p5-HTML-Parser
  2. p5-HTML-Selector-XPath>=0.03 : www/p5-HTML-Selector-XPath
  3. p5-HTML-Tagset>0 : www/p5-HTML-Tagset
  4. p5-HTML-TreeBuilder-LibXML>=0.04 : www/p5-HTML-TreeBuilder-LibXML
  5. p5-HTML-TreeBuilder-XPath>=0.08 : www/p5-HTML-TreeBuilder-XPath
  6. p5-HTML-Tree>=3.23 : www/p5-HTML-Tree
  7. p5-UNIVERSAL-require>0 : devel/p5-UNIVERSAL-require
  8. p5-URI>0 : net/p5-URI
  9. p5-XML-XPathEngine>=0.08 : textproc/p5-XML-XPathEngine
  10. p5-YAML>0 : textproc/p5-YAML
  11. p5-libwww>=5.827 : www/p5-libwww
  12. perl5>=5.28.r1<5.29 : lang/perl5.28
This port is required by:
for Build
  1. graphics/p5-Gimp
  2. japanese/p5-WWW-MobileCarrierJP
  3. www/p5-Scrappy
  4. www/p5-WWW-HatenaDiary
  5. www/p5-WWW-Mixi-Scraper
  6. www/p5-Web-Scraper-Config
for Run
  1. graphics/p5-Gimp
  2. japanese/p5-WWW-MobileCarrierJP
  3. www/p5-Scrappy
  4. www/p5-WWW-HatenaDiary
  5. www/p5-WWW-Mixi-Scraper
  6. www/p5-Web-Scraper-Config

Configuration Options

USES:

Master Sites:
  1. ftp://ftp.auckland.ac.nz/pub/perl/CPAN/authors/id/M/MI/MIYAGAWA/
  2. ftp://ftp.cpan.org/pub/CPAN/authors/id/M/MI/MIYAGAWA/
  3. ftp://ftp.cpan.org/pub/CPAN/modules/by-module/Web/
  4. ftp://ftp.funet.fi/pub/languages/perl/CPAN/authors/id/M/MI/MIYAGAWA/
  5. ftp://ftp.kddlabs.co.jp/lang/perl/CPAN/authors/id/M/MI/MIYAGAWA/
  6. ftp://ftp.mirrorservice.org/sites/cpan.perl.org/CPAN/authors/id/M/MI/MIYAGAWA/
  7. http://backpan.perl.org/authors/id/M/MI/MIYAGAWA/
  8. http://ftp.jaist.ac.jp/pub/CPAN/authors/id/M/MI/MIYAGAWA/
  9. http://ftp.twaren.net/Unix/Lang/CPAN/authors/id/M/MI/MIYAGAWA/
  10. https://cpan.metacpan.org/authors/id/M/MI/MIYAGAWA/
  11. https://cpan.metacpan.org/modules/by-module/Web/
  12. https://www.cpan.org/authors/id/M/MI/MIYAGAWA/

Number of commits found: 30

Commit History - (may be incomplete: see SVNWeb link above for full details)
DateByDescription
05 Jul 2019 19:15:24
Original commit files touched by this commit  0.38
Revision:505931
sunpoet search for other commits by this committer
Add NO_ARCH
06 Oct 2018 13:06:45
Original commit files touched by this commit  0.38
Revision:481365
des search for other commits by this committer
Fix every instance of RUN_DEPENDS:=${BUILD_DEPENDS} in p5 ports, except
for where it resulted in a change in output from build-depends-list or
run-depends-list.

Approved by:	portmgr (adamw)
27 May 2018 20:15:20
Original commit files touched by this commit  0.38
Revision:470993
sunpoet search for other commits by this committer
Update WWW

search.cpan.org is shutting down.
It will redirect to metacpan.org after June 25, 2018.

With hat:	perl
15 Sep 2017 08:58:50
Original commit files touched by this commit  0.38
Revision:449893
mat search for other commits by this committer
Fix license information for portgs that use "the same license as Perl".

Sponsored by:	Absolight
01 Apr 2016 14:33:58
Original commit files touched by this commit  0.38
Revision:412351
mat search for other commits by this committer
Remove ${PORTSDIR}/ from dependencies, categories v, w, x, y, and z.

With hat:	portmgr
Sponsored by:	Absolight
12 Jan 2015 18:36:59
Original commit files touched by this commit  0.38
Revision:376876
sunpoet search for other commits by this committer
- Update to 0.38
- Sort PLIST

Changes:	http://search.cpan.org/dist/Web-Scraper/Changes
26 Nov 2014 13:08:38
Original commit files touched by this commit  0.37_1
Revision:373448
mat search for other commits by this committer
Change the way Perl modules are installed, update the default Perl to 5.18.

Before, we had:

  site_perl :           lib/perl5/site_perl/5.18
  site_perl/perl_arch : lib/perl5/site_perl/5.18/mach
  perl_man3 :           lib/perl5/5.18/man/man3

Now we have:

  site_perl : lib/perl5/site_perl
  site_arch : lib/perl5/site_perl/mach/5.18
  perl_man3 : lib/perl5/site_perl/man/man3

Modules without any .so will be installed at the same place regardless of the
(Only the first 15 lines of the commit message are shown above View all of this commit message)
07 Nov 2013 16:38:14
Original commit files touched by this commit  0.37
Revision:333137
vanilla search for other commits by this committer
Support STAGEDIR.
20 Sep 2013 23:36:54
Original commit files touched by this commit  0.37
Revision:327776
bapt search for other commits by this committer
Add NO_STAGE all over the place in preparation for the staging support (cat:
www)
29 Aug 2013 04:08:54
Original commit files touched by this commit  0.37
Revision:325559
sunpoet search for other commits by this committer
- Use single space after WWW:
02 Aug 2013 18:52:11
Original commit files touched by this commit  0.37
Revision:324174
mat search for other commits by this committer
- Convert to new perl framework
- Trim Makefile header
- Remove MAKE_JOBS_SAFE=yes, it's the default.
19 Dec 2012 08:48:23
Original commit files touched by this commit  0.37
Revision:309228
jadawin search for other commits by this committer
- Update to 0.37
- Changelog: http://cpansearch.perl.org/src/MIYAGAWA/Web-Scraper-0.37/Changes
21 Nov 2011 14:36:51
Original commit files touched by this commit  0.36

sunpoet search for other commits by this committer
- Update to 0.36
- Add BUILD_DEPENDS and TEST_DEPENDS

Changes:        http://search.cpan.org/dist/Web-Scraper/Changes
Feature safe:   yes
07 Oct 2011 11:22:46
Original commit files touched by this commit  0.35

culot search for other commits by this committer
- Fix LICENSE (Artistic 1 instead of Artistic 2)
- Pet portlint(1)

PR:             ports/161336
Submitted by:   Simon Olofsson <simon@olofsson.de>
28 Sep 2011 14:07:41
Original commit files touched by this commit  0.35

sunpoet search for other commits by this committer
- Update to 0.35

Changes:        http://search.cpan.org/dist/Web-Scraper/Changes
08 Sep 2011 13:20:15
Original commit files touched by this commit  0.34

az search for other commits by this committer
- Replace ../../authors in MASTER_SITE_SUBDIR with CPAN:CPANID macro.
  See http://wiki.freebsd.org/Perl for details.
- Change maintainership from ports@ to perl@ for ports in this changeset.

With perl@ hat
28 Feb 2011 11:13:15
Original commit files touched by this commit  0.34

bapt search for other commits by this committer
- update to 0.34
- add licenses
- changes: http://cpansearch.perl.org/src/MIYAGAWA/Web-Scraper-0.34/Changes
18 Feb 2011 14:56:13
Original commit files touched by this commit  0.33

clsung search for other commits by this committer
- Update to 0.33
24 Sep 2010 02:03:44
Original commit files touched by this commit  0.32

pgollucci search for other commits by this committer
- only 13% of the p5- ports embed @comment $FreeBSD$:
  so standarize and remove it

With Hat:   perl@
04 Feb 2010 09:30:46
Original commit files touched by this commit  0.32

tobez search for other commits by this committer
Update to 0.32.

Changes:        http://search.cpan.org/dist/Web-Scraper/Changes
28 Jul 2009 11:46:59
Original commit files touched by this commit  0.31_2

tobez search for other commits by this committer
Hmm, fun with QAT.
28 Jul 2009 11:30:38
Original commit files touched by this commit  0.31_1

tobez search for other commits by this committer
Pacify QAT.  Not sure how my own tests did not catch this.
28 Jul 2009 11:23:10
Original commit files touched by this commit  0.31

tobez search for other commits by this committer
Update to 0.31.

Changes:        http://search.cpan.org/dist/Web-Scraper/Changes
15 Jul 2009 00:27:18
Original commit files touched by this commit  0.30

pgollucci search for other commits by this committer
- Update to 0.30

PR:             ports/136692
Submitted by:   TERAMOTO Masahiro <markun@onohara.to>
Tested by:      Cezary Morga <cm@therek.net>
02 Apr 2009 13:47:41
Original commit files touched by this commit  0.28

tobez search for other commits by this committer
Update to 0.28.
27 Mar 2009 22:24:47
Original commit files touched by this commit  0.27

miwi search for other commits by this committer
- Update to 0.27
19 Jan 2009 16:54:07
Original commit files touched by this commit  0.26

jadawin search for other commits by this committer
- Update to 0.26
15 Jan 2009 10:13:27
Original commit files touched by this commit  0.25

jadawin search for other commits by this committer
- Update to 0.25
12 Feb 2008 23:11:16
Original commit files touched by this commit  0.24

miwi search for other commits by this committer
- Update to 0.12

PR:             120554
Submitted by:   Masahiro Teramoto <markun@onohara.to> (maintainer)
12 Dec 2007 08:01:53
Original commit files touched by this commit  0.22

clsung search for other commits by this committer
Web::Scraper is a web scraper toolkit, inspired by Ruby's equivalent
Scrapi.

WWW:    http://search.cpan.org/dist/Web-Scraper/

PR:             ports/117554
Submitted by:   Gea-Suan Lin <gslin at gslin.org>

Number of commits found: 30

Login
User Login
Create account

Servers and bandwidth provided by
New York Internet, iXsystems, and RootBSD

This site
What is FreshPorts?
About the authors
Issues
FAQ
How big is it?
The latest upgrade!
Privacy
Blog
Contact

Search
Enter Keywords:
 
more...

Latest Vulnerabilities
chromiumMay 24
piwigoMay 23
drupal7May 22
tomcat-develMay 22
tomcat7May 22
tomcat85May 22
tomcat9May 22
unboundMay 22
zabbix3-proxyMay 20
zabbix3-serverMay 20
rubygem-actionpack52May 19
rubygem-actionpack60May 19
rubygem-actionview52May 19
rubygem-actionview60May 19
rubygem-activestorage52May 19

14 vulnerabilities affecting 165 ports have been reported in the past 14 days

* - modified, not new

All vulnerabilities

Last updated:
2020-05-24 19:56:04


Ports
Home
Categories
Deleted ports
Sanity Test Failures
Newsfeeds

Statistics
Graphs
NEW Graphs (Javascript)
Traffic

Calculated hourly:
Port count 39419
Broken 80
Deprecated 621
Ignore 370
Forbidden 5
Restricted 147
No CDROM 76
Vulnerable 24
Expired 15
Set to expire 576
Interactive 0
new 24 hours 4
new 48 hours9
new 7 days40
new fortnight86
new month249

Servers and bandwidth provided by
New York Internet, iXsystems, and RootBSD
Valid HTML, CSS, and RSS.
Copyright © 2000-2020 Dan Langille. All rights reserved.