FreshPorts -- The Place For Ports If you buy from Amazon USA, please support us by using this link.
Follow us
Blog
Twitter

I am looking for an LTO tape library. Do you have one to spare?
Port details
crawl 0.4_11 www on this many watch lists=2 search for ports that depend on this port
Small, efficient web crawler with advanced features

There is no maintainer for this port.
Any concerns regarding this port should be directed to the FreeBSD Ports mailing list via ports@FreeBSD.org search for ports maintained by this maintainer
Port Added: 23 Jun 2001 13:09:54
License: not specified in port


The crawl utility starts a depth-first traversal of the web at the
specified URLs. It stores all JPEG images that match the configured
constraints.  Crawl is fairly fast and allows for graceful termination.
After terminating crawl, it is possible to restart it at exactly
the same spot where it was terminated. Crawl keeps a persistent
database that allows multiple crawls without revisiting sites.

The main reason for writing crawl was the lack of simple open source
web crawlers. Crawl is only a few thousand lines of code and fairly
easy to debug and customize.

Some of the main features:
 - Saves encountered JPEG images
 - Image selection based on regular expressions and size contrainsts
 - Resume previous crawl after graceful termination
 - Persistent database of visited URLs
 - Very small and efficient code
 - Supports robots.txt

WWW: http://www.monkey.org/~provos/crawl/
SVNWeb : Main Web Site : Distfiles Availability : PortsMon

NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered.

Required To Build:
  1. devel/pkgconf
Required Libraries:
  1. devel/libevent2
  2. databases/db5
There are no ports dependent upon this port

To install the port: cd /usr/ports/www/crawl/ && make install clean
To add the package: pkg install www/crawl


Configuration Options
===> The following configuration options are available for crawl-0.4_11:
     EXAMPLES=on: Build and/or install examples
===> Use 'make config' to modify these settings

Master Sites:
  1. http://distcache.FreeBSD.org/ports-distfiles/
  2. http://www.monkey.org/~provos/

Number of commits found: 34

Commit History - (may be incomplete: see SVNWeb link above for full details)
DateByDescription
21 Aug 2014 22:50:30
Original commit files touched by this commit  0.4_11
Revision:365599
mandree search for other commits by this committer
Berkeley DB cleanup, remove versions 4.0 ... 4.7.
- Mk/bsd.database.mk rewrite, new default to db5.
- db6 is eligible by default only if installed on the system.
- Bump PORTREVISION of all ports that directly depend on BerkeleyDB or
  where USE_BDB is found in the port's directory
- Patch a few ports such that they will pick up or work with newer
  versions.
- Add UPDATING entry
- Drive-by format fix for pks
- Drop BerkeleyDB option from mail/popular for now, requires more work.
- Exp-run logs linked from the PR below.
- Ports that do not build (IGNORE, BROKEN, etc.) have pro-forma changes
  for new Berkeley DB, but are untested.

NOTE: please read UPDATING and the Wiki page before proceeding!

Announcement:	http://lists.freebsd.org/pipermail/freebsd-ports-announce/2014-August/000090.html
Wiki reference:	https://wiki.freebsd.org/Ports/BerkeleyDBCleanup
PR:		192690
Approved by:	portmgr (implicit, PORTREVISION bump on unstaged ports)
29 Jul 2014 17:12:50
Original commit files touched by this commit  0.4_10
Revision:363362
adamw search for other commits by this committer
Rename all patches that contain '::' as a path separator, and use
'__' instead.
24 Jul 2014 13:32:59
Original commit files touched by this commit  0.4_10
Revision:362796  Sanity Test Failure
bapt search for other commits by this committer
Only use libevent2

Remove libevent as libevent2 is providing a good compatibility interface as well
as providing better performances.
Remove custom patches from libevent2 and install libevent2 the regular way
Mark ports abusing private fields of the libevent1 API as broken
Import a patch from fedora to have honeyd working with libevent2
Remove most of the patches necessary to find the custom installation we used to
have for libevent2

With hat:	portmgr
05 Mar 2014 13:21:16
Original commit files touched by this commit  0.4_9
Revision:347110
bapt search for other commits by this committer
Enforce using libevent2 via the compat API (libevent2 performs way better,
libevent1 should die in short term)
20 Sep 2013 23:36:54
Original commit files touched by this commit  0.4_8
Revision:327776
bapt search for other commits by this committer
Add NO_STAGE all over the place in preparation for the staging support (cat:
www)
19 Mar 2011 12:38:54
Original commit files touched by this commit  0.4_8
miwi search for other commits by this committer
- Get Rid MD5 support
25 Jul 2010 15:39:20
Original commit files touched by this commit  0.4_8
mm search for other commits by this committer
Update libevent to 1.4.14b

PR:             ports/147723
Approved by:    maintainer (timeout)
21 Dec 2009 02:19:12
Original commit files touched by this commit  0.4_7
dougb search for other commits by this committer
For ports maintained by ports@FreeBSD.org, remove names and/or
e-mail addresses from the pkg-descr file that could reasonably
be mistaken for maintainer contact information in order to avoid
confusion on the part of users looking for support. As a pleasant
side effect this also avoids confusion and/or frustration for people
who are no longer maintaining those ports.
19 Aug 2008 16:40:17
Original commit files touched by this commit  0.4_7
mnag search for other commits by this committer
- Update libevent dependency and bump PORTREVISION
02 Jan 2008 23:43:03
Original commit files touched by this commit  0.4_6
mnag search for other commits by this committer
- Bump PORTREVISION since devel/libevent are updated.
21 Sep 2007 20:21:30
Original commit files touched by this commit  0.4_5
mnag search for other commits by this committer
- Change libevent lib and bump PORTREVISION since devel/libevent are updated.
06 Apr 2007 18:28:46
Original commit files touched by this commit  0.4_4
mnag search for other commits by this committer
- Bump PORTREVISION and change LIB_DEPENDS to reflect libevent update.
18 Dec 2006 15:15:01
Original commit files touched by this commit  0.4_3
leeym search for other commits by this committer
- utilize USE_BDB
05 Dec 2006 13:29:01
Original commit files touched by this commit  0.4_3
mnag search for other commits by this committer
- Bump PORT_REVISION and change LIB_DEPENDS to reflect update in devel/libevent
- Fix many wrong BUILD_DEPENDS. Thanks to ldd(1)
13 May 2006 04:41:22
Original commit files touched by this commit  0.4_2
edwin search for other commits by this committer
Remove USE_REINPLACE from categories starting with W
14 Apr 2006 20:45:44
Original commit files touched by this commit  0.4_2
linimon search for other commits by this committer
Reset petef due to no response to email.  We hope to see him back sometime.

Hat:    portmgr
24 Jan 2006 03:14:23
Original commit files touched by this commit  0.4_2
edwin search for other commits by this committer
SHA256ify

Approved by: krion@
19 Jul 2005 02:18:32
Original commit files touched by this commit  0.4_2
kris search for other commits by this committer
Now builds on 6.x
01 Jul 2005 22:47:21
Original commit files touched by this commit  0.4_2
jylefort search for other commits by this committer
Chase the libevent update.

Reported by:    pointyhat
02 Jun 2005 20:14:55
Original commit files touched by this commit  0.4_1
ume search for other commits by this committer
Our getaddrinfo(3) never returns EAI_NODATA on 5.2-RELEASE and
later as RFC 3493 deprecated it.  So, we have to see EAI_NONAME
instead.

Approved by:    petef
10 Apr 2005 21:35:28
Original commit files touched by this commit  0.4
kris search for other commits by this committer
BROKEN on FreeBSD 6.0: Does not compile
31 Mar 2004 03:12:58
Original commit files touched by this commit  0.4
trevor search for other commits by this committer
SIZEify (maintainer timeout)
12 Mar 2004 13:43:32
Original commit files touched by this commit  0.4
petef search for other commits by this committer
Unbreak on -current.
06 Feb 2004 08:13:11
Original commit files touched by this commit  0.4
kris search for other commits by this committer
BROKEN on 5.x: does not compile
21 Dec 2003 18:39:12
Original commit files touched by this commit  0.4
petef search for other commits by this committer
Update to 0.4.
05 May 2003 03:16:01
Original commit files touched by this commit  0.3_2
petef search for other commits by this committer
- fix build on -current
- no longer need to depend on libmd5 from libwww, libmd does fine
- install an example crawl.conf

PR:             51696
Submitted by:   Rui Lopes <rui@ruilopes.com>
07 Mar 2003 06:12:57
Original commit files touched by this commit  0.3_1
ade search for other commits by this committer
Clear moonlight beckons.
Requiem mors pacem pkg-comment,
And be calm ports tree.

E Nomini Patri, E Fili, E Spiritu Sancti.
21 Nov 2002 23:56:43
Original commit files touched by this commit  0.3_1
edwin search for other commits by this committer
PERL -> REINPLACE_CMD
Noticed on: bento
22 Aug 2002 19:13:46
Original commit files touched by this commit  0.3_1
ade search for other commits by this committer
BerkeleyDB cleanup - stage 2

Update databases/db3 to 3.3.11, and fix the few ports that need sorting
after the shlib version update, and a slight API change from 3.2.x->3.3.x
04 Feb 2002 00:39:19
files touched by this commit
petef search for other commits by this committer
Update to 0.3    
23 Dec 2001 10:14:50
files touched by this commit
petef search for other commits by this committer
- update to 0.2   - remove redundant GNU_CONFIGURE (this is implied by
USE_AUTOCONF)    
23 Aug 2001 00:07:25
files touched by this commit
petef search for other commits by this committer
Change my email address to petef@FreeBSD.org for the ports I maintain.    
04 Jul 2001 07:23:38
files touched by this commit
dwcjr search for other commits by this committer
Update to 1.0b    
23 Jun 2001 17:09:54
files touched by this commit
dwcjr search for other commits by this committer
Add crawl, a worm that searches for jpegs    

Number of commits found: 34

Login
User Login
Create account

Servers and bandwidth provided by
New York Internet, SuperNews, and RootBSD

This site
What is FreshPorts?
About the authors
FAQ
How big is it?
The latest upgrade!
Privacy
Forums
Blog
Contact

Search
Enter Keywords:
 
more...

Latest Vulnerabilities
flacNov 25
asterisk11Nov 21
asterisk11Nov 21
phpmyadminNov 21
kde4-runtimeNov 20
kwebkitpartNov 20
yiiNov 19
chromiumNov 18
kde4-workspaceNov 17
dbusNov 11
wgetNov 08
konversation-kde4Nov 05
jenkinsOct 31
jenkins-ltsOct 31
twikiOct 31

9 vulnerabilities affecting 11 ports have been reported in the past 14 days

* - modified, not new

All vulnerabilities


Ports
Home
Categories
Deleted ports
Sanity Test Failures
Newsfeeds


Statistics
Graphs
NEW Graphs (Javascript)
Traffic

Calculated hourly:
Port count 24390
Broken 160
Deprecated 79
Ignore 417
Forbidden 1
Restricted 206
No CDROM 93
Vulnerable 18
Expired 12
Set to expire 74
Interactive 0
new 24 hours 8
new 48 hours10
new 7 days51
new fortnight181
new month237

Servers and bandwidth provided by
New York Internet, SuperNews, and RootBSD
Valid HTML, CSS, and RSS.
Copyright © 2000-2014 Dan Langille. All rights reserved.