17:58 koobs 

ports-mgmt/portscout: Fix broken !SiteHandler logic

Fix a logic bug introduced in r401037, which caused ports without a
SiteHandler (GitHub, SourceForge, PyPI) not to be processed through the
fallback FTP or HTTP(S) cases.

I didn't pick this up in testing because only my ports were tested, most
of which use PyPI/GitHub, using the `restrict maintainer` setting in
portscout.conf.

[1] https://svnweb.freebsd.org/changeset/ports/401037

Pointyhat:	koobs
Reported by:	kwm (for mate, libdrm)
MFH:		2016Q1

 

16:50 koobs 

ports-mgmt/portscout: Make GitHub ports work, fix bugs

A number of people reported no updates being detected for ports using
USE_GITHUB=yes, even after originally adding the GitHub site handler in
r401037 [1]. Investigation revealed that the FindNewestFile subroutine
and the vercompare() method assumed (or are designed so) that responses
returned from site handlers will be in a normalised version format. For
site handlers that return 'versions', this works well. For the github
handler, in the fallback use of the API for fetching repository tags, it
does not.

Additionally it turns out, portscout currently only uses/stores a
normalized version ('ver') in its database, in its general design
attempt to be a generic version comparison tool

In particular, portscout does not reference or store PORTVERSION
or DISTVERSION{FULL}, so we have nothing 'canonical' to compare
the responses from Github (tags) against.

This change special-cases Github in the FindNewestFile subroutine, which
was obtained via Portroach [2] (OpenBSD's portscout fork).

Extending this, we also now only match version-esque looking strings
from the tag, in an attempt to normalise, because they come in many
forms, including {foo-}X.Y.Z{-bar}, foo_X_Y_Z, among others.

While I'm here,

* Fix copypasta of $github_client_id, when $github_client_secret was
  intended
* Add code to use authenticated requests for Github project tags in the
  fallback (to /releases) case.
* Add and update some more debug messages to help diagnosis of future
  issues

Special thank you's to:

 * matthew, allanjude, mandree, des, Brendan Molloy for your regex, perl
   help and moral support.

[1] http://svnweb.freebsd.org/changeset/ports/401037
[2] https://github.com/jasperla/portroach

Differential Revision:	D4780

 

07:37 koobs 

ports-mgmt/portscout: Add GitHub and PyPI site handlers & MORE!

Over the past several months portscout.freebsd.org appears to have been
more frequently NOT finding updates, particularly for ports that use
CHEESESHOP (PyPI) as their MASTER_SITES.

Portscout has also never worked for ports using GitHub for distribution
files due to the following:

  a) Portscout, prior to 'guessing', requests a randomly named file
     from the Site and expects a 4xx (404) in response. If it doesn't
     receive a 4xx response, it increments a 'lie counter' and does not
     check the site again in the next run.
  b) The GitHUB handlers (SUBDIR/MASTER_SITES) in bsd.sites.mk
     construct a URL that ends in a a dummy query paramater (for the

(Only the first 15 lines of the commit message are shown above )