16:50 koobs 

ports-mgmt/portscout: Make GitHub ports work, fix bugs

A number of people reported no updates being detected for ports using
USE_GITHUB=yes, even after originally adding the GitHub site handler in
r401037 [1]. Investigation revealed that the FindNewestFile subroutine
and the vercompare() method assumed (or are designed so) that responses
returned from site handlers will be in a normalised version format. For
site handlers that return 'versions', this works well. For the github
handler, in the fallback use of the API for fetching repository tags, it
does not.

Additionally it turns out, portscout currently only uses/stores a
normalized version ('ver') in its database, in its general design
attempt to be a generic version comparison tool

In particular, portscout does not reference or store PORTVERSION
or DISTVERSION{FULL}, so we have nothing 'canonical' to compare
the responses from Github (tags) against.

This change special-cases Github in the FindNewestFile subroutine, which
was obtained via Portroach [2] (OpenBSD's portscout fork).

Extending this, we also now only match version-esque looking strings
from the tag, in an attempt to normalise, because they come in many
forms, including {foo-}X.Y.Z{-bar}, foo_X_Y_Z, among others.

While I'm here,

* Fix copypasta of $github_client_id, when $github_client_secret was
  intended
* Add code to use authenticated requests for Github project tags in the
  fallback (to /releases) case.
* Add and update some more debug messages to help diagnosis of future
  issues

Special thank you's to:

 * matthew, allanjude, mandree, des, Brendan Molloy for your regex, perl
   help and moral support.

[1] http://svnweb.freebsd.org/changeset/ports/401037
[2] https://github.com/jasperla/portroach

Differential Revision:	D4780

 

07:37 koobs 

ports-mgmt/portscout: Add GitHub and PyPI site handlers & MORE!

Over the past several months portscout.freebsd.org appears to have been
more frequently NOT finding updates, particularly for ports that use
CHEESESHOP (PyPI) as their MASTER_SITES.

Portscout has also never worked for ports using GitHub for distribution
files due to the following:

  a) Portscout, prior to 'guessing', requests a randomly named file
     from the Site and expects a 4xx (404) in response. If it doesn't
     receive a 4xx response, it increments a 'lie counter' and does not
     check the site again in the next run.
  b) The GitHUB handlers (SUBDIR/MASTER_SITES) in bsd.sites.mk
     construct a URL that ends in a a dummy query paramater (for the
     filename), so that fetch saves the correct filename to DISTDIR.
     This means for any DISTFILE name provided, a 200 OK response is
     returned

These two factors unfortunately leave us in a position where there is no
good way to workaround this in the ports framework, including overriding
DISTFILES, DISTNAME, FETCH_ARGS, or the SUBDIR URL itself for various
reasons (not matching distinfo, file conflicts in DISTDIR, etc)

Fortunately, the portroach project (OpenBSD's fork of portscout)
contains a site handler for GitHub and PyPI (among others) already [1].

These site handlers use API endpoints at GitHub and PyPI that respond
JSON respectively, providing a faster and more accurate way to determine
the latest version of a package, without having to go through the
'guessing' process.

This commit:

- Adds GitHub and PyPI site handlers, and modifies or extends them to
  accept/match our MASTER_SITES URL's.
- Adds authenticated API request support and two settings for the
  GitHub site handler
- Add p5-JSON to RUN_DEPENDS (needed by new site handlers)
- Add HTTPS option for supporting https:// MASTER_SITES. Currently
  portscout does not check (fails) https:// MASTER_SITES [2]
- Take MAINTAINER'ship
- Adds badly needed logging/debugging messages to key parts of the
  process retaining the conditional logic that ties the verbosity to
  "quiet" or "debug" portscout.conf settings.
- Renables the SQLITE3 option (previously commented out) and renames it
  to SQLITE (the standard, as per bsd.options.desk.mk)
- Creates a DATABASE option group allowing either/or SQLITE or
  POSTGRESQL to be selected
- Switches option conditionals where possible to options helpers
- Backport a fix for maintainer matching/mapping [3]
- Adds LICENSE (BSD2CLAUSE)
- Updates and sorts pkg-plist

This change was tested again ports maintained by me, and resulted in
'new versions' being found and reported for 42 of my ports (of 123).

[1] https://github.com/jasperla/portroach/tree/master/Portroach/SiteHandler
[2] Reported by: truckman
[3]
https://github.com/jasperla/portroach/commit/2f6ee134ddc178f74688b37b986b66ecb0481782

PR:			203996
Approved by:		maintainer (timeout, 15 days)
Differential Revision:	https://reviews.freebsd.org/D3996