Database of robots.txt-derived permissions Maintained by:perl@FreeBSD.org Port Added: 14 Apr 2011 13:19:02 Also Listed In:perl5 License: not specified in port
This module parses /robots.txt files which are used to forbid conforming
robots from accessing parts of a web site. The parsed files are kept in
a WWW::RobotRules object, and this object provides methods to check if
access to a given URL is prohibited.
WWW: http://search.cpan.org/dist/WWW-RobotRules/
WWW::RobotRules parses /robots.txt files which are used to forbid conforming
robots from accessing parts of a web site. The parsed files are kept in a
WWW::RobotRules object, and this object provides methods to check if access
to a given URL is prohibited.
WWW: http://search.cpan.org/dist/WWW-RobotRules/
This new port is needed to update www/p5-libwww.
If you buy from Amazon USA, please support us by using this link.
