3 How to Mirror FreeBSD

3.1.1 With FTP mirror

You can use a FTP mirror program to get the files. There are a lot around and widely used, like:

• ftp/mirror

• ftp/ftpmirror

• ftp/emirror

• ftp/spegla

• ftp/omi

• some even use ftp/wget

ftp/mirror was very popular, but seemed to have some drawbacks, as it is written in perl(1), and had real problems with mirroring large directories like a FreeBSD site. There are rumors that the current version has fixed this by allowing a different algorithm for comparing the directory structure to be specified.

In general FTP is not really good for mirroring. It transfers the whole file if it has changed, and does not create a single data stream which would benefit from a large TCP congestion window.

3.1.2 With RSYNC

A better way to mirror the FTP area is rsync. You can install the port net/rsync and then use rsync to sync with your upstream host. rsync is already mentioned in Section 2.4.2. Since rsync access is not required, your preferred upstream site may not allow it. You may need to hunt around a little bit to find a site that allows rsync access.

A command line to mirror FreeBSD could look like that:

% rsync -vaz --delete ftp4.de.FreeBSD.org::FreeBSD/ /pub/FreeBSD/
         

Consult the documentation for rsync, which is also available at http://rsync.samba.org/ about the various options to be used with rsync. If you sync the whole module (unlike subdirectories), be aware that the module-directory (here "FreeBSD") will not be created, so you cannot omit the target directory. Also you might want to set up a script framework that calls such a command via cron(8).

3.1.3 With CVSup

A few sites, including the one-and-only ftp-master.FreeBSD.org even offer CVSup to mirror the contents of the FTP space. You need to install a cvsup client, preferably from the port net/cvsup. (Also reread Section 2.4.4.) A sample supfile suitable for ftp-master.FreeBSD.org looks like this:

          #
          # FreeBSD archive supfile from master server
          #
          *default host=ftp-master.FreeBSD.org
          *default base=/usr
          *default prefix=/pub
          #*default release=all
          *default delete use-rel-suffix
          *default umask=002

          # If your network link is a T1 or faster, comment out the following line.
          #*default compress

          FreeBSD-archive release=all preserve
         

It seems CVSup would be the best way to mirror the archive in terms of efficiency, but it is only available from few sites.

3.2.1 Using CVSup

CVSup was already described to some detail in Section 2.4.4 and Section 3.1.3.

Here we just describe an example to set up the supfile:

          #
          # FreeBSD CVS supfile from master server
          #
          *default host=cvsup-master.FreeBSD.org
          *default base=/usr
          *default prefix=/pub/FreeBSD/development/FreeBSD-CVS
          *default release=cvs
          *default delete use-rel-suffix
          *default umask=002

          # If your network link is a T1 or faster, comment out the following line.
          #*default compress

          cvs-all
         

You should also have a look at /usr/share/examples/cvsup

3.2.2 Using other methods

Using other methods than CVSup is generally not recommended. We describe them in short here anyway. Since most sites offer the CVS repository as part of the FTP fileset under the path /pub/FreeBSD/development/FreeBSD-CVS, the following methods could be used.

• FTP

• RSYNC

• maybe even HTTP

If you find a site that supports it, you could use net/sup. But it is inferior to CVSup and its deficiencies caused John Polstra to develop CVSup in the first place, so it is clearly not recommended.

% cvs -d /home/ncvs co www

        #
        # WWW module supfile for FreeBSD
        #
        *default host=cvsup3.de.FreeBSD.org
        *default base=/usr
        *default prefix=/usr/local
        *default release=cvs tag=.
        *default delete use-rel-suffix

        # If your network link is a T1 or faster, comment out the following line.
        *default compress

        # This collection retrieves the www/ tree of the FreeBSD repository
        www
       

3.3.1 Mirroring the FreeBSD documentation

Since the documentation is referenced a lot from the web pages, it is recommended that you mirror the FreeBSD documentation as well. However, this is not as trivial as the www-pages alone.

First of all, you should get the doc sources, again preferably via CVSup. Here is a corresponding sample supfile:

         #
         # FreeBSD documentation supfile
         #
         *default host=cvsup3.de.FreeBSD.org
         *default base=/usr
         *default prefix=/usr/share
         *default release=cvs tag=.
         *default delete use-rel-suffix

         # If your network link is a T1 or faster, comment out the following line.
         #*default compress

         # This will retrieve the entire doc branch of the FreeBSD repository.
         # This includes the handbook, FAQ, and translations thereof.
         doc-all
        

Then you need to install a couple of ports. You are lucky, there is a meta-port: textproc/docproj to do the work for you. You need to set up some environment variables, like SGML_CATALOG_FILES. Also have a look at your /etc/make.conf (copy /usr/share/examples/etc/make.conf if you do not have one), and look at the DOC_LANG variable. Now you are probably ready to run make in your doc directory (/usr/share/doc by default) and build the documentation. Again you need to make it accessible for your web server and make sure the links point to the right location.