MirOS Manual: mirror-maker(7), mirroring-ports(7)

MIRRORING-PORTS(7)           BSD Reference Manual           MIRRORING-PORTS(7)


     mirroring-ports - how to build a mirror for ports distfiles


     The OpenBSD Ports Collection offers some powerful tools to mirror
     software sources. The ports infrastructure provides a mirror-maker target
     that can be used to build a Makefile to facilitate mirroring distfiles.
     This target builds ${DISTDIR}/Makefile, which can be used on almost any
     Unix-like machine to mirror OpenBSD distfiles. The variable
     RECURSIVE_FETCH_LIST can be set to 'Yes' if port dependencies  should
     also be recorded.

     A sample Makefile entry is formatted like this:

     all:: audio/tracker/tracker-5.3
     .PHONY: audio/tracker/tracker-5.3
     audio/tracker/tracker-5.3:  tracker-5.3.tgz

     tracker-5.3.tgz: $F
             @MAINTAINER="espie@openbsd.org" \
              SITES="ftp://ftp.uni-trier.de/pub/unix/audio/tracker/ " \
              CIPHER="sha1" CKSUM="b0973d6a9c363caebd3a71547412f42b0681f323" \
              exec ${FETCH} "$@"

     This Makefile is usually invoked by the user from the directory where
     they wish to mirror distfiles, with variables FETCH and F set on the com-
     mand line, e.g.,

           cd mirror && mmake -k -j 5 -f path_to_makefile FETCH=fetch_script

     Targets are set up so that each port is referenced by its full name, and
     retrieves all its distfiles. The default all target is used to retrieve
     all distfiles. Targets ftp and cdrom can be used to retrieve all dist-
     files necessary to build ftp and cdrom packages, respectively. Actual
     fetching is usually invoked with a parallel-make option, so that several
     retrievals through ftp can proceed simultaneously.

     The F variable can be set to a dummy name, or a recent filename, to force
     re-fetching of anything which is older than the filename. Its intended
     use is to force re-fetching existing files, or to checksum all files.

     The ${FETCH} script should be supplied by the user, and will download and
     verify the archive file. It must obey the following variables:

     MAINTAINER   Port maintainer, used to report errors,

     ERROR        Some ports problems can be detected while building the
                  Makefile, in which case this variable will be set to a prop-
                  er error message.

     DIST_SUBDIR  See ports(7) for more details. The ${FETCH} script is
                  responsible for creating this subdirectory and cd'ing to it
                  before performing the actual fetch.

     SITES        A list of sites to try for fetching the distribution file.

     CIPHER       The checksumming utility to use for verifying the distribu-
                  tion file. It will normally be set to sha1(1) unless you
                  tinker with PREFERRED_CIPHER while building the mirroring

     CKSUM        The corresponding checksum. If neither CIPHER nor CKSUM nor
                  ERROR are set, the distribution file needs not be checked.

     A standard fetch strategy is to try all sites in order: whenever the dis-
     tribution file is found, download it; verify the checksum; erase the file
     and try the next site if it doesn't match.

     Mirroring sites should update their master Makefile fairly often. Activi-
     ties a proper mirror should offer (in order of decreasing importance):

     Mirror new files
                Use a proper fetch script to download missing files,

     Run ${PORTSDIR}/infrastructure/fetch/link-checksums
                This script creates permanent hardlinks that preserve dist-
                files against checksum changes.

     Verify all checksums
                All checksums should be verified from time to time, and main-
                tainers notified of persistent discrepancies,

     Check mastersites liveliness
                Use a tool such as 'mirror' to check that the master sites
                haven't fallen off the Earth. Even though the first site in
                the site list is the most important site, good mirrors will
                scan all sites and report all problems,

     Remove old files
                To gain room this, the mirror should maintain a list of
                'active' files (easy enough, just provide a fetch script that
                just lists the file names), and remove files that are no
                longer active. Since OpenBSD releases happen every six months,
                this delay should be longer than that.
     Sample scripts are provided in the ${PORTSDIR}/infrastructure/fetch


                Main mirroring Makefile
                Sample script usable to retrieve distfiles.
                Sample script to check all distfiles checksums.
                Populating checksums subdirectories with links, to guard
                against shifting checksums.




     This infrastructure was introduced for OpenBSD 2.7 by Marc Espie, with
     feedback from Bob Beck, Todd Fries, Camiel Dobbelaar, and a few other


     Changing checksums is a recurring problem that is outside the direct con-
     trol of the OpenBSD Project. Some software distributors change distribu-
     tion files without warning, without changing the file name proper. Once
     the problem has been identified, the port maintainer should usually con-
     tact the software author to fix the problem, or, if the software author
     is unresponsive, the maintainer should use DIST_SUBDIR to provide some
     state to guard against shifting checksums.

     However, a more robust approach is also needed, so that ports users can
     depend on distfiles mirrors to carry what they need irrespective of those
     synchronization issues. The link-checksums script creates another access
     to the distfiles, indexed through the actual checksums that the files
     should match. Provided mirroring is run sufficiently often, together with
     link-checksums, two versions of the same distfile with respective check-
     sums cksum1 and cksum2 will be available under the names
     ${DISTFILES}/sha1/cksum1/distfile and ${DISTFILES}/sha1/cksum2/distfile.

     Starting revision 1.281, if REFETCH is set to true, bsd.port.mk will try
     to retrieve files under that naming scheme as a last resort.

MirOS BSD #10-current         February 11, 2016                              2

Generated on 2017-04-03 16:26:17 by $MirOS: src/scripts/roff2htm,v 1.88 2017/01/29 00:51:06 tg Exp $

These manual pages and other documentation are copyrighted by their respective writers; their source is available at our CVSweb, AnonCVS, and other mirrors. The rest is Copyright © 2002–2017 The MirOS Project, Germany.
This product includes material provided by mirabilos.

This manual page’s HTML representation is supposed to be valid XHTML/1.1; if not, please send a bug report — diffs preferred.