14:27 Mathieu Arnold (mat) 

framework: Remove $FreeBSD$

Where appropriate fiddle with a few other things.

    5d33e04

13:29 mat 

Add set pipefail in most framework scripts.

set pipefail changes the pipeline return status from being the return
status of the last command to the last non 0 exit status of any command
in the pipeline.  This is needed to make sure all the commands in a
pipeline did actually return a non 0 status and not only the last one.

PR:		250723
Exp-run by:	antoine
Reviewed by:	bapt
Approved by:	bapt
Differential Revision:	https://reviews.freebsd.org/D27007

 

16:51 mat 

Backout r554139.

 

16:42 mat 

Add set pipefail in most framework scripts.

set pipefail changes the pipeline return status from being the return
status of the last command to the last non 0 exit status of any command
in the pipeline.  This is needed to make sure all the commands in a
pipeline did actually return a non 0 status and not only the last one.

 

20:49 bdrewery 

generate-symbols: Need @dir entries if PREFIX!=LOCALBASE.

- makeplist/check-plist: Don't suggest or complain about @dir entries
  for debug symbols added by this script. Same as is done for the
  debug symbols themselves.

Sponsored by:	DellEMC

 

01:07 bdrewery 

Revert r505129: I still have the wrong version.

 

00:44 bdrewery 

Allow generate-symbols.sh to actually work.

It splits out symbols to PREFIX as well not LOCALBASE. Fix
check-stagedir.sh for that.

 

23:55 gerald 

Add share/info/*/dir to those files in STAGEDIR to ignore when checking
for orphans, i.e., files in STAGEDIR that are not covered by plist.

This is a follow-up to revision 484628 after which texinfo files are
now installed into ${PREFIX}/share/info. (A file "dir" is then created
and maintained by the tooling.)

Approved by:	portmgr (antoine)

 

11:09 mat 

Extract PLIST_SUB substitutions into a sed script.

This fixes PLIST_SUB being too big and PLIST_SUB_SED getting bigger than
_POSIX2_LINE_MAX.

PR:		222355
Reported by:	asomers
Reviewed by:	asomers, mfechner
Sponsored by:	Absolight
Differential Revision:	 https://reviews.freebsd.org/D14014

 

09:26 mat 

SC2162: read without -r will mangle backslashes.

By default, read will interpret backslashes before spaces and line
feeds, and otherwise strip them. This is rarely expected or desired.

Normally you just want to read data, which is what read -r does. You
should always use -r unless you have a good reason not to.

PR:             227109
Submitted by:   mat
Sponsored by:   Absolight

 

09:26 mat 

SC2185: Some finds don't have a default path. Specify '.' explicitly.

(false positive, split flags to avoid triggering it.)

PR:             227109
Submitted by:   mat
Sponsored by:   Absolight

 

22:29 gerald 

Ignore info/*/dir (as used by the lang/gcc* ports, for example) in addition
to */info/dir and info/dir.

PR:		219274
Approved by:	portmgr (mat)

 

13:20 tijl 

The output of tools like awk, date, sort, tr,... depends on the current
locale set by the user.  Add LANG=C and LC_ALL=C at the beginning of
bsd.port.mk and export them so all commands are executed with the C locale.
LC_ALL=C overrides all other LC_* variables.  LANG is used by setlocale(3)
as default value for LC_* variables, so normally it isn't used when LC_ALL
is set, but there's code out there that looks at LANG directly so it's safer
to set it as well.  The only commands not captured by this are !=
assignments before any inclusion of bsd.port.*mk.

Introduce USE_LOCALE=<locale> that adds LANG=<locale> and LC_ALL=<locale> to
CONFIGURE_ENV and MAKE_ENV so upstream build systems can be executed with a
different locale (e.g. USE_LOCALE=en_US.UTF-8).

PR:		215882
Exp-run by:	antoine
Approved by:	portmgr (antoine)

 

19:01 mat 

Ease debugging of Mk/Scripts scripts.

Sponsored by:	Absolight
Differential Revision:	https://reviews.freebsd.org/D6474

 

17:04 bdrewery 

Add some work-in-progress scripts for splitting symbols out into
PREFIX/lib/debug.

This is only missing the bsd.port.mk pieces to hook it up fully.  A blocker
for hooking that up has been sub-packages, even though some implementation
could be made without them.  For now just commit what I have so it is not
forgotten.

Obtained from:	OneFS
Sponsored by:	EMC / Isilon Storage Division
With hat:	portmgr

 

20:08 bapt 

Factorize the function to validate env

Reviewed by:	antoine
Differential Revision:	https://reviews.freebsd.org/D2966

 

23:05 bapt 

Make fonts repecting XDG

Xorg is now looking in ${LOCALBASE}/share/fonts by default
Xorg now accepts symlinks in etc/X11/fontpath.d (as decribed in Xserver(1))
Large cleanup on lots of font ports
All fonts are now properly dynamically generating fonts.dir and fonts.scale
instead of sometime overwriting existing ones)
All fonts are generating fontconfig's cache
Improve consistency in fonts ports

 

21:05 bdrewery 

Don't replace PREFIX anywhere in the plist line, only do it for the beginning.

With hat:	portmgr
Sponsored by:	EMC / Isilon Storage Division

 

04:04 bdrewery 

- Move parse_plist() into a new file, functions.sh.

With hat:	portmgr

 

03:55 bdrewery 

- Rework check on $makeplist in parse_plist(). The variable was/is always 0 as
  the makeplist code does not use parse_plist(). Rename the param to
  'parse_comments' to be more clear on what it is for. This is kept for
  upcoming refactoring.

With hat:	portmgr

 

03:37 bdrewery 

Limit the scope of variables in parse_plist()

With hat:	portmgr

 

01:10 bapt 

Remove now useless gnomehier, pkg handles automatically the directories where
needed making gnomehier entirely useless now.

Differential Revision:	https://reviews.freebsd.org/D1355
Reviewed by:	antoine, kwm
Approved by:	gnome (kwm)

 

13:47 antoine 

- Add three empty dirs that belonged to gnomehier to devel/gconf2,
  this prevents stage violations during build of ports using gconftool-2
- White list etc/gconf/gconf.xml.defaults from stage orphans as gconftool-2
  is unpredictable

With hat:	portmgr

 

07:14 antoine 

- During makeplist, emit only empty directories inside prefix and
  directories outside prefix
- Whitelisting orphan directories not starting with / is no longer necessary
- Adjust missing directories check to use a list of all directories instead
  of the dirs from make-plist

Differential Revision:	https://reviews.freebsd.org/D1101
Reviewed by:	bapt
With hat:	portmgr

 

21:39 ak 

- Accept absolute path for @fc, @fontsdir, @fcfontsdir, @shell keywords

Approved by:	portmgr (antoine, bapt)
Differential Revision:	https://reviews.freebsd.org/D721

 

06:19 antoine 

info keyword now handles full paths

With hat:	portmgr

 

19:17 antoine 

Replace @dirrm/@dirrmtry with @dir in makeplist and leftovers check

Differential Revision:	https://reviews.freebsd.org/D901
Reviewed by:	bapt
With hat:	portmgr

 

18:22 bdrewery 

check-plist no longer needs to check for directories owned by dependencies.

pkg-1.3.8's auto directory handling makes this a non-issue.

We should add a new check to warn of possibly needing @dir for empty dirs
though.

With hat:	portmgr

 

18:14 bdrewery 

Garbage collect dependency mtree check removed in r368803

 

07:53 antoine 

Do not report directories in plist as orphans

With hat:	portmgr

 

06:22 bapt 

adjust check_stagedir.sh to take in account %%KEY%%@dirrm*

 

05:47 bapt 

Update to 1.3.8 Changes:
- Plist parser do not accept modes like o+w g+s u+r
- @owner/@group with no argument reset the default ownership to root/wheel
- Plug regression tests into the release mechanism as mandatory
- Allow url in repository configuration to be overriden by another configuration
  file
- If one of -P, -I or -R is not explicitly given on the command line do not emit
  error messages while searching for a mechanism to find data about the latest
  available ports.
- External keywords can now take arguments
- Directories under PREFIX are automatically handled and removed if needed
- MTREE are not packaged anymore neither extracted on final installation
- OS major checking (to determine if pkg is running on an upgraded base system)
  uses the same mechanism as ABI detection instead of relying on UNAME_r which
  can have false positivie on jails
- Add @dir keyword for plist which is used to package explicitly directories
  (directories with non root/wheel or outside prefix) and empty directories
  (@dirrm and @dirrmtry are now considered as deprecated)
- root/wheel are now considered as the default user/group when creating a
  package except @owner/@group are used to change that behaviour
- Import newer version of libucl which improves error messages in case of
  problem parsing manifest or configuration files
- Fix pkg register so it checks for conflicts before actually copying files on
  to the filesystem
- pkg now support a new no_provide_lib annotation to preventing automatic
  populating of provided libs (useful for packages bundling libraries like wine)
- Improve documentations
- Ability to select usage of IPv4/IPv6 via config, repository config or command
  line
- Lots of bug fixes

Side effects:
- Convert to install-mtree target into a no-op target Add empty directories to
  plist of ports that needs them to run:
- share/aclocal to devel/automake and devel/automake14
- share/applications to devel/desktop-file-utils
- share/xml share/sgml to textproc/xmlcatmgr
- Prevent check-plist to issue warnings about some @dir*
- Adjist check_leftovers.sh to not take in account anything related to mtree
- Bump the default required version of pkg to 1.3.8

 

19:03 antoine 

Fix typo

With hat:	portmgr

 

14:36 antoine 

Make check-stagedir.sh aware of (owner,group,perm) syntax for keywords

Differential Revision:	https://reviews.freebsd.org/D712
Reviewed by:	bapt
With hat:	portmgr

 

17:06 antoine 

- Make it possible to use sample keyword with a full path
- Prefer @dirrmtry over @unexec rmdir in makeplist

Differential Revision:	https://reviews.freebsd.org/D705
Reviewed by:	bapt
With hat:	portmgr

 

13:03 bapt 

Remove support for pkg_install
Merge back bsd.pkgng.mk into bsd.port.mk
Add a note about @stopdaemon not being supported anymore

With hat:	portmgr
Differential Revision:	https://reviews.freebsd.org/D693

 

20:20 bdrewery 

- Fix 'make check-plist' incorrectly considering all installed packages as
  run dependencies if the port being checked has no RUN_DEPENDS. This could
  cause false-positive "Possibly owned by dependency" warnings.

With hat:	portmgr

 

21:03 bdrewery 

- Fix check-plist so that its 'files in plist but missing from STAGEDIR' check
  does not incorrectly ignore plist items in %%DOCSDIR%% and %%EXAMPLESDIR%%.
  It was incorrectly ignoring these items when %%PORTDOCS%% or %%PORTEXAMPLES%%
  was missing. This manifested as a failure in 'make package' later when pkg(8)
  detected the missing files.

With hat:	portmgr

 

22:26 antoine 

Do not report parent directories of LOCALBASE as orphans

Reported and tested by:	gerald
With hat:	portmgr

 

21:07 antoine 

Do not report files @commented by multiple options as orphans

With hat:	portmgr

 

10:30 daichi 

fix a typo

 

17:42 bapt 

Add an @shell keyword to handle adding and remove a shell path in /etc/shell
Bump port revision of all ports that were doing it wrong prior to the keyword

CR:		D208
Reviewed by:	antoine
With hat:	portmgr

 

16:09 antoine 

Add more files to orphans white list:
- .DS_Store files
- some VCS related files

Note: it's still recommended to not extract/install those files if they are
not needed

PR:		ports/189859
Submitted by:	adamw (first version)
Reviewed by:	bapt
With hat:	portmgr

 

01:51 bdrewery 

- %D is based on @cwd, not PREFIX; Fix false-postives when using pkg_install
  and @cwd has changed away from PREFIX:

    plist:
      @cwd /
      @dirrmtry var/spool/dma

    ====> Checking for pkg-plist issues (check-plist)
    ===> Parsing plist
    DIR '/usr/local/etc/dma' CWD: /usr/local
    DIR '/usr/local/var/spool/dma' CWD:
    DIR 'share/licenses/dma-v0.9,1' CWD: /usr/local
    DIR '/usr/local/share/licenses' CWD: /usr/local
    ===> Checking for items in STAGEDIR missing from pkg-plist
    Error: Orphaned: @unexec rmdir "/var/spool/dma" >/dev/null 2>&1 || :
    ===> Checking for directories owned by MTREEs
    ===> Checking for directories handled by dependencies
    ===> Checking for items in pkg-plist which are not in STAGEDIR
    Error: Missing: @dirrmtry var/spool/dma
    ===> Error: Plist issues found.

With hat:	portmgr
Reported by:	erwin

 

17:21 bdrewery 

- Fix check-plist showing @sample dest files as orphans and also showing in
  makeplist.

Reported by:	droso
With hat:	portmgr

 

03:09 bdrewery 

- Ignore MTREE owned PREFIX and PREFIX/info removal for now.
  The removal of PREFIX/info may be a bug in bsd.port.mk and needs
  more testing.

With hat:	portmgr
Reported by:	gerald

 

22:42 bdrewery 

- Fix check-stagedir causing staging violating. Quote all directory names
  when doing plist checks. This could leak out if an unsupported format
  was used such as 'rmdir -p' or 'rmdir dir1 dir2 dir3'

With hat:	portmgr
Reported by:	antoine
Pointyhat to:	bdrewery

 

22:31 bdrewery 

- Fix false-positive orphanned dir when '@unexec rmdir -p' is used.

Reported by:	antoine
With hat:	portmgr

 

22:30 bdrewery 

- Refactor a bit into functions

With hat:	portmgr

 

12:00 bdrewery 

check-plist: Don't error if GNOME_MTREE_FILE is missing, which can happen
in nox11 ports that USE_GNOME.

With hat:	portmgr
Reported by:	swills

 

20:46 antoine 

- Handle plist containing @unexec /bin/rmdir
- Handle directories ending with digits

With hat:	portmgr

 

23:28 bdrewery 

- Fix handling of "@comment @cwd" so it does not reset real @cwd; only
  change the cwd for commented lines (such as %%OPTION%%file). Reset it
  back to the normal on the first non-commented line.
- Sort files after stripping out @comment

With hat:	portmgr

 

14:11 antoine 

Fix directory keywords handling

Reviewed by:	bdrewery
With hat:	portmgr

 

12:56 bdrewery 

- For pkg run-depends dir listing, consider deps recursively.
  pkg_install already did this.
- Speedup the pkg_install run-depends listing by processing unique
  packages.

With hat:	portmgr
Reported by:	antoine

 

22:08 antoine 

Fill orphans white list with files regenerated by installation scripts:
- info directories
- fonts.dir and fonts.scale
- desktop database
- mime database

With hat:	portmgr

 

18:42 bdrewery 

- Split MTREE check into its own check
- Split dependency-owned check its own and make into a non-fatal warning.
  This is not black-and-white. Python/Rubygem ports cleanup the lib dir
  for every installed package, and for python/ruby-gems too. This is fine,
  the extensions should not clean them up then; the error is valid here.
  However, there's other frameworks which don't cleanup in the master package
  and yield unexpected results. An example is that php does not cleanup
  etc/php itself. Only extensions do. So if an extension depends on another
  extension it will complain that the dependent extension owns etc/php,
  so we remove from bsd.php.mk, now it's a leftover. The proper place is in the
  php port itself, but it's unknown how many other cases there are like this.
  So for now make this non-fatal and into a warning.

  This is all mostly moot once pkg auto handles directories anyhow.

With hat:	portmgr
Discussed with:	antoine

 

16:22 bdrewery 

- Sort directories by Lexicographical Depth-First order (find -ds)

With hat:	portmgr

 

06:44 bdrewery 

- Rename check-orphans to check-plist. Keep the old for backwards-compat.
- Bug fixes:
  makeplist/check-plist:
   - Fix showing directories owned by RUN_DEPENDS/LIB_DEPENDS [1]
   - Use proper "rmdir PATH" syntax, not "rmdir >/dev/null... PATH" which
     pkg will not recognize.
   - Never consider base /etc/mtree/BSD.usr.dist or LOCLABASE
     Templates/BSD.local.dist as needing @dirrm handling.
  check-plist:
   - Fix showing PORTDOCS/PORTEXAMPLES files when the OPTIONS are not set.
     makeplist will still suggest them.
   - Fix showing files installed through unselected OPTIONS as orphans,
     by considering "@comment file" to be ignored. [2]
   - Fix @sample, @fc, @fcfontsdir, @fontsdir support
   - Fix return status when orphans are found to be non-zero
   - Add note when PREFIX=!LOCALBASE
   - Be more clear when orphans/no orphans are found.
   - Add a whitelist mechanism for globally approved ignores.
   - Add *.bak/*.orig to orphan whitelist for now.
   - Fix false-positive with dirs installed to /, such as with
     archivers/dpkg creating /var/db/dpkg
   - Fix false-positive with @dirrm ending in /, such as with
     ports-mgmt/poudriere with a @dirrmtry share/zsh/
   - There are likely still some false-positives. I fixed as many as I could
     find. Please let me know of others.
    * One in particular that is not easily fixable is installing a file into
      a directory owned by another port where that other port is not a run-time
      dependency. So the leaf port may create all of the parent dirs and never
      clean them up. Cleaning them up is not proper unless no other package is
      depending on them. This will be addressed by pkg(8) once pkg_install is
      EOL, or sooner.
   - Consider @dirrm of directories owned by run-time dependencies, or
     /etc/mtree/* or Templates/BSD.local.dist (at LOCALBASE) as fatal errors.
     These should not be removed in the plist. @comment lines are not
     considered for this; they will not ignore an error.
        ===> Checking for directories owned by dependencies or MTREEs
        Error: Owned by dependency: @dirrmtry share/locale/af/LC_MESSAGES
        Error: Owned by dependency: @dirrmtry share/locale/af
        Error: Owned by dependency: @dirrmtry %%PERL5_MAN3%%
   - Detect files in plist that do not exist in the stagedir. Pkgng already did
     this, but now we have it unified with this check. @comment lines are
     not considered for this; they will not ignore an error.
        ===> Checking for items in pkg-plist which are not in STAGEDIR
        Error: Missing: foo
        Error: Missing: @dirrmtry bar
   - Change orphaned output due to several new errors introduced:
        ===> Checking for items in STAGEDIR missing from pkg-plist
        Error: Orphaned: foo
   - Send errors to stderr
  check_leftovers.sh:
   - Prefer longer values for PLIST_SUB.
  bsd.options.mk:
   - Fix NOPORTDOCS/WITHOUT_NLS/NOPORTEXAMPLES not unsetting their respective
     OPTIONS.
   - Mark NOPORTDOCS/NOPORTEXAMPLES deprecated and hook them into the
     OPTIONS_WARNING to tell users the new format.
- check-stagedir.sh refactoring:
  - Switch to using new PLIST_SUB_SED
  - Use ! instead of , in sed(1) regex to allow files/dirs with ','
  - Rework PORTEXAMPLES/PORTDOCS handling so it acts on PLIST_SUB_SED'd
    value and not absolutes. Also simplify the regex for these a bit to
    allow reuse.
  - No longer need DOCSDIRS/EXAMPLESDIR in env
  - Wrap long lines
  - Unset some vars in env when they are done being used to free space for
    larger sed vars
  - Cleanup redundant sed regexes
- Add a Scripts/plist_sub_sed_sort.sh to prefer longer values when substituting
  over shorter values.

- To make check-plist ignore a file *as an orphan* do one of the following:
  1. Install it
  2. post-install: ${RM} ${STAGEDIR}file
  3. Put the file behind an OPTION with a PLIST_SUB: %%OPTION%%file
  4. Add to plist as a @comment [2]:
     @comment file
     @comment @dirrmtry dir

Reviewed by:	mat (much earlier version)
Discussed with:	mat, antoine, bapt, swills (various bits)
With hat:	portmgr
PR:		ports/185561 [1]
Reported By:	Alexander Yerenkow <yerenkow@gmail.com> [1]
Tested with:	pkg and pkg_install
Reported by:	many (false-positives)

[2] This is a partial solution, we may still need a plist.ignore too. It
    doesn't make much sense to add files in main pkg-plist we don't care about,
    but maybe it does since you'll see and reconsider them being ignored
    someday. @comment is used as all the OPTION PLIST_SUB deactivations use
    @comment instead of something like @ignore.

 

17:02 bapt 

Allow to properly run make check-orphans and make makeplist for gnome ports

 

16:35 mat 

Remove quotes around values we get in PLIST_SUB. [1]

While there, don't replace 1 char entries in PLIST_SUB.

Noticed by:	amdmi3 [1]
Sponsored by:	Absolight

 

09:19 mat 

Try and force people to check the plist makeplist generates.

With hat:	portmgr
Reviewed by:	bapt
Sponsored by:	Absolight

 

14:16 mat 

Greatly enhance makeplist to replace all it can find in PLIST_SUB.

With hat:	portmgr
Sponsored by:	Absolight

 

10:35 bapt 

Do not show orphans anymore with pkgng for .info files

 

16:08 mat 

Fixup makeplist for directories.

Without, it'll generate :
@dirrmtry %%PORTDOCS%%%%DOCSDIR%%
With :
%%PORTDOCS%%@dirrmtry %%DOCSDIR%%

PR:		ports/184287
Submitted by:	mat
Approved by:	bapt

 

12:23 mandree 

- in qa.sh, properly work with filenames containing blank or tab,
  by setting IFS to just LF

- in check-stagedir.sh, properly detect unset environmental variables,
  and list them all

- in check-stagedir.sh, also list the full prefix as known, not just
  its parents

PR:		184039
Approved by:	portmgr (bapt)

 

19:07 mandree 

bsd.stage.mk: Fewer false positives, much faster, easier maintenance.

Bugfixes:

* @cmd in pkg-plist is now properly handled.
  - It was previously treated the same as though there was a directory
    following it, missing the prefix. (ordering matters in case...esac)

  - Due to the cwd=${PREFIX} inside the while read line loop, state
    tracking was broken and every new line assumed that cwd were the
    prefix.

* stage-qa no longer complains about unstripped binaries if debugging is
  active (WITH_DEBUG set && WITHOUT_DEBUG unset).

(Only the first 15 lines of the commit message are shown above )