Tech Stuff - FreeBSD Update Survival Guide

This is a survival guide to keeping FreeBSD up-to-date ('ish). This note covers the automated tools (freebsd-update and portsnap) and manual methods for updating FreeBSD. In general, the automated tools should be the preferred method. The manual methods are included for the sake of completeness, because some people prefer them and because sometimes - TINA (there is no alternative). While the FreeBSD Handbook is generally a terrific resource - always check it first - but it can either make assumptions about background knowlege or it keeps things as simple as possible to minimize errors. As we all know FreeBSD, like Perl, provides at least 8,000 ways to do the same thing.

Note on the move from CVS to subversion: The notes on manual methods have been updated to reflect the FreeBSD change from CVS to subversion that from March 2013 covers both source upgrades and the ports system. cvsup and csup are now dead. R.I.P. As a passing observation if you were using entirely automated tools (freebsd-update and portsnap) you would not need be aware of this change - and might not even know about it. Is this an advantage?

Note on Sloth: We generally follow the 'slothful' theory of OS maintenance. We check the security advisories, nod sagely when we read 'em, and using the new automated tools (portsnap and freebsd-update) we keep our systems updated. And occasionally even upgrade the ports we have installed. We do nothing exciting, nothing even remotely interesting and if it smacks of risk, we lie down in a dark room for a couple of hours, then check that we have access to emergency medical assistance, before doing anything. However, fateful days arrive. FreeBSD finally stops maintaining our release and we gotta do something. Always the minimum. A minor version upgrade before a major version. A major version upgrade only when it gets to a minimum of x.1. Hell can freeze over before we break that last rule. This survival guide was written because we suspect we are not alone and besides, the down-side of the 'slothful' maintenance theory is we are not constantly doing stuff and so forget....a lot....frequently.

Note on Extreme Sloth: By extreme application of the 'slothful' method of maintenance you could find yourself more that one major version behind (say, using 6.x when the current version is 9.x). The best advice here is to never try a multi-version upgrade, say, from 6.x to 9.x - always pass through the intervening major version(s) - certainly as far as Step 6 since ABI (Application Binary Interface) changes may cause serious problems. In such cases one of the CD based direct install methods may be quicker and less risky.

Overview All possible update/upgrade methods
Upgrading from source (buildworld) one major release to another, in this case, 8.x to 9.x but the principles are the same for all releases. The old-fashioned manual way.
Upgrading from source (buildworld) Theory of Operation and risk overview.
Upgrading from source (buildworld) Step by Step detail.
Upgrading from source If it all goes horribly wrong. Some, hopefully, helpful notes.
Upgrading Using freebsd-update. Upgrading major or minor versions.
Updates Using freebsd-update. Keeping your system updated.
Ports collection Maintenance overview.
Ports collection manual update. Updating the ports collection from the subversion repository (cvsup or csup are now deprecated and since February 28th 2013 the CVS tree is no longer updated)
Ports collection updating with portsnap Maintenance (including automated maintenance) with portsnap.
Ports vs RubyGems Which way forward for Ruby users?

Upgrading FreeBSD

As with all things FreeBSD there are about 100,000 million ways of doing things which can leave folk confused when they read three different HOWTOs all describing different ways to accomplish the same goal. In many cases it's just simple habit. The author of the HOWTO has always done it that way. There is no attempt to analyze the reasons - it just works. But one is left with that vaguely uncomfortable feeling that if anything goes wrong the only solution will be a panicky google search and a lot of hope.

Possible Upgrade methods:

Here are your real choices for upgrade:

sysinstall from a CD and select Upgrade (does a binary upgrade). All the documentation suggests this route has not been extensively tested and is therefore not to be recommended.
sysinstall from a CD, select new system and put back everything that you backed up. While this may look draconian it has the merit that any cruft is entirely removed and you get a pristine system which may be better in the long run.
Binary upgrade procedure using freebsd-update. The best current route for all upgrades.
Upgrade from source - the process we use here - the so-called buildworld. For dinosaurs, folk who have always done it this way, and masochists. Covers most nearly all of us quite neatly.

Our upgrade probably represents pretty much a worst case scenario since we have on our target system:

No CDROM - we could add one but don't want to. So it's not possible to do any of the CDROM based upgrades. We have to do a source upgrade.
Not enough space in /usr to allow us to do a buildworld - but plenty of space in /var. So we have to use non-standard paths everywhere.

Upgrading from Source (buildworld)

To summarize the upgrade:

From FreeBSD 8.1 to 9.1. To check your current version:
```
uname -a
```
Source upgrade from the current subversion repository using the svn utility (the old cvsup and csup methods are now dead).
Non-standard locations for the source and object trees. By default the source is assumed to be in /usr/src and object files plus build output are placed in /usr/obj. We have to place them in /var/src and /var/obj because of space constraints in a poor installation done many years ago and which we have never found the time to change.

Note: If you are using standard source/object paths we note the normal commands as well.

The handbook says it, every HOWTO in the world says it, so we're gonna say it too. Back up the system before you start. See rsync and backup. As a minimum you should back up /etc and if it's a web server /usr/local/www (or wherever you keep your webs) and then anything else that is unique to this system (check-list).

Theory of Operation

Most folks get kinda nervous when it comes time for an upgrade and postpone the fateful day as long as possible. This is a big mistake since the FreeBSD team have done a great job in making it relatively simple and painless. This section is for those who want to know when they are in danger and when they are not!

Step 1 Sync source. Totally painless using cvsup or csup. No impact on the rest of the system other than using about ~20m of disk space. Production work can run concurrently. Do it as often as you want.
Step 2 make buildworld. Totally painless but can take a long time. Compiles and builds the new infrastructure, libraries, utilities, etc. in the /obj directories but installs NOTHING. You can stop at this point for as long as you want and continue to run production systems. Generally, anything with the word build in front, like buildworld is painless since it only builds and does NOT install anything. If there is no build or it has the word install then it does, surprisingly, install stuff.
Step 3 make kernel. Reasonably painless. Compiles, builds and installs a new kernel - but copies the old one to /boot/kernel.old first. If your system crashes after this step is complete the reboot will load the new kernel by default. You can force it to reload the old one (the process is described in the section). Otherwise you can continue production and stay in this state as long as you want.
Step 4 boot to single user mode. Not risky. The new kernel may not work but since recovery is trivial and nothing has changed there is almost no risk. At this point you are running a new kernel while everything else (utilities etc.) are the old versions. You are in single user mode and cannot run any production tasks. Minimize use of commands to the essentials. Typically top, ps and various other system commands either don't work or give strange results.
Step 5 mergemaster and installworld. Risky. At this point the processes will update/merge configuration files in /etc and install the upgraded utilities and libraries. Crashes here can vary from slightly to extremely painful.
Step 6 boot to normal mode. Happiness and joy. Normally. Depending on the nature of the upgrade you should be back running production systems which may not be optimized for the new version - so continue to the next step and soon as practical.
Step 7 fix ports. Updating the ports is a pretty solid test of the system so if it works you are home free and should have all your snazzy new, optimized applications working.
Step Back if it all goes horribly wrong. Some, non-exhaustive, notes about recovery.

Step 1 Sync Source

In the bright shiny new world of FreeBSD the old CVS repository is no longer supported (so the old cvsup and csup utilities are now obsolete) but has been replaced with a subversion repository (for documentation and background see subversion on-line documentation).

This step involves updating the source to the version you want to install using the svn utility. In our case we will use the RELEASE version of FreeBSD 9.1 (see notes. If you do not have the svn utility (it is not installed by default) then it's available in /usr/ports/devel/subversion.

By default, the build process assumes /usr/src for the source files and places compiled files and applications in /usr/obj. We will be using non-standard paths so need we to do more work. If you are using the standard paths ignore commands marked with NON_STD_PATH_ONLY comment below. Download the source files required:

# the first time this process is performed use
svn co svn://name.of.repository/base/release/x.x.x /usr/src

# NON_STD_PATH_ONLY
svn co svn://name.of.repository/base/release/x.x.x /var/src

# where name.of.repository is the required host or mirror site
# the default is svn.freebsd.org
# but a list of current mirrors is available from this list
# x.x.x is the version in this case it will be 9.1.0
# there is a handy web interface to the subversion system

# to subsequently keep the source up to date for the selected version just use

svn update /usr/src

# NON_STD_PATH_ONLY
svn update /var/src

# if the run was terminated abnormally then use
svn cleanup /usr/src

The full list of BRANCH TAGS is available. The documentation mentions -CURRENT and -RELEASE but rarely mentions -STABLE in relation to the branch types. There are, in fact, three major branches:

-CURRENT	Pure development branch - for hairy chested developers, or if you are really in need of some new feature and are prepared to take some risks. Resides in /head, for example, svn.freebsd.org/base/head. Binary updates (freebsd-update) are NOT supported for this tag.
-STABLE	Development branch for current release and is used as the base from which subsequent releases for each major version are generated. Typically contains the latest security updates at the point at which you download but it is changing all the time. Some level of risk is involved, offset against being udated for the lastest security fixes, and significantly lower risk than using -CURRENT. Resides in the /stable branch under the major version number, for example, svn.freebsd.org/base/stable/8 will provide the latest major version 8 sources (see here). Binary updates (freebsd-update) are NOT supported for this tag. If you use this branch you must use source updating for all subsequent security fixes.
-RELEASE	A specific, frozen-in-time, release of FreeBSD, for example, 9.1. Resides under the /release branch and has a major, minor and third level (normally 0) version number, for example, svn.freebsd.org/base/release/9.1.0 (see here). Binary updates (freebsd-update) are supported for this branch only since this constitutes the so called security branch after installation.

You can find the information about what version you will have by looking at the script /usr/src/sys/conf/newvers.sh (or in our non-standard case /var/src/sys/conf/newvers.sh) once the svn run is complete. Look for the variables REVISION and BRANCH - these will also be output when you load the new kernel and issue either uname -r or uname -a.

When this step is complete you should read the file /usr/src/UPDATING (or in our non-standard case /var/src/UPDATING) for specific notes about the release.

Step 2 buildworld

This step involves building all the various tools/utilities required to run FreeBSD. They are built and saved in the defined directores and are NOT INSTALLED (remember if it says build at the front it does NOT install). This process is carried out in normal production mode and has no impact (other than CPU and disk utilization) on production jobs. We usually run it overnight. This process consumed ~440m of disc space on our system.

By default, the build process assumes /usr/src for the source files and places compiled files and applications in /usr/obj. We will be using non-standard paths so we need to do more work. If you are using the standard paths ignore commands marked with NON_STD_PATH_ONLY comment below:

# check to make sure you have enough space on the target
# partitions
df -m

# NON_STD_PATH_ONLY
# set environmental variable to change OBJ path
MAKEOBJDIRPREFIX=/var/obj
export MAKEOBJDIRPREFIX
# do not issue commands if you are using /usr/obj

# NON_STD_PATH_ONLY
cd /var/src

# if you are using the standard build paths issue
cd /usr/src

# in all cases
make buildworld

# this will run for some time

At the end of this step your base system is unchanged (all the new stuff is in the /src and /obj directories) and you can remain in this state for as long as you want.

Step 3 build and install kernel

This step involves building a basic kernel to run the new version of FreeBSD. You will see two methods of doing this in the literature. The commands below show both and note the differences. This process is carried out in normal production mode and has no impact (other than CPU and disk utilization) on production jobs. This process consumed ~290m on our target system.

We show building a kernel using the GENERIC file which is the least risk path and the only one you can use if you want to use binary security updates. Once the upgrade process is entirely complete you can go back and build a new custom kernel if required.

By default the kernel build process assumes /usr/src for the source files and places compiled files and applications in /usr/obj. We will be using non-standard paths so need to do more work. If you are using the standard paths ignore commands marked with NON_STD_PATH_ONLY comment below:

# check to make sure you 
# have enough space
df -m

# NON_STD_PATH_ONLY
# set environmental variable to change OBJ path
# only if you have reloaded since the previous step
# or confirm with printenv
MAKEOBJDIRPREFIX=/var/obj
export MAKEOBJDIRPREFIX
# do not issue the above line if you are using /usr/obj

# NON_STD_PATH_ONLY
cd /var/src

# if you are using the standard path issue
cd /usr/src

# in all cases
make kernel KERNCONF=GENERIC
# this makes and installs the new kernel
# but copies the current kernel to /boot/kernel.old 
# before doing so

# you will also see the following
make buildkernel KERNCONF=GENERIC
make installkernel KERNCONF=GENERIC
# the above two command have exactly the same effect as 
# make kernel KERNCONF=GENERIC
# if you like typing two lines use the above method

At the end of this step you will have in /boot/kernel the new version of a GENERIC FreeBSD kernel for your selected release and in /boot/kernel.old your previous version (in our case 5.3) of the kernel. If your system crashes or you reboot and do nothing the default will be to load the new version kernel. You can however force load the old kernel at boot time.

Step 4 boot to single user mode

This step involves booting the new kernel since all the subsequent work will be done under the new version of FreeBSD and in single user mode. This is not a risky step since if the new kernel crashes you can revert to the previous one at boot time. Issue the following commands:

# shutdown current system cleanly
shutdown -r now
# OR
reboot

# at the beastie prompt select 4 (boot to single user mode)
# or enter boot -s if its just a prompt
# when the shell prompt appears accept it [ENTER]
# then do
fsck -p
# fsck not strictly necessary if you shut down cleanly 
# but does no harm

# change root from read-only
mount -u /

# mount all fstab entries (load all file systems)
mount -a

# if your BIOS clock is at wall-clock time (local time)
# issue this command, if it is at UTC time do not
# if you have no idea then issue the command anyway
adjkerntz -i

# your system should now be functional and ready for the next step
# confirm the kernel version number
uname -a
# which should reflect the new version you expect
# if not STOP and figure out why before moving on

If the new kernel crashes or fails to load in single user mode then take as much diagnostic information as you can (re-run the boot if necessary) and then load the previous kernel version as follows:

# at the beastie screen select option 6 if present
# then at the prompt enter
boot /boot/kernel.old
# and your previous kernel will load

# to restore your previous kernel as default while you 
# figure out what went wrong
cd /boot
# save broken kernel in case you need more info
mv kernel kernel.bad

# restore old kernel to default
mv kernel.old kernel

If everything has gone well you should now be loaded on a new OS but with all the tools and infrastructure using the old version. Time now to change that.

Step 5 mergemaster and installworld

This is the biggest risk step since it involves upgrading the tools/utilities and configuration files to the new version and can be extremely lengthy. First we run mergemaster to fix up our /etc configuration files using the -p option:


# NON_STD_PATH_ONLY
# you need to define where the source directory is
mergemaster -p -m /var/src
# do not issue the above line if you are using default /usr/src

# if you are using the default (/usr/src) just issue:
mergemaster -p

# you will be prompted to reconcile or ignore changed files
# the following changes are important
# 1. password file 
#    if you are using a dhcp client 6.2 requires
#    a new _dhcp user account
# 2. group file
#    6.x introduces a new audit group
# in most other cases keep your own files (use the d option)
# when merging the group and/or password (or any other files)
# select the m option which displays each line of the file
# with current on left and new on right
# the prompt (%) allows you to select:
# l   - use left version 
# r   - use right version
# e l - edit then use left version
# e r - edit then use right version
# e b - edit then use left and right concatented
# e   - edit new version

At the end of the first mergemaster command we have versions of configuration files that have changed (normally in /var/tmp/temproot/etc) ready to replace all our /etc files. These files have NOT been installed so you can revert to your old system at anytime. Now we upgrade the base system and utilties to the new version by running make installworld which we built light-years ago with the make buildworld.

By default, the build process assumes /usr/src for the source files and places compiled files and applications in /usr/obj. We will be using non-standard paths so need to do more work. If you are using the standard paths ignore commands marked with NON_STD_PATH_ONLY comment below:

# NON_STD_PATH_ONLY
cd /var/src

# if you are using the standard paths then
cd /usr/src

# NON_STD_PATH_ONLY
# we need to define that our obj path is non standard
# edit Makefile and Makefile.inc1 in each file
# find line starting with MAKEOBJDIRPREFIX?= /usr/obj
# change to MAKEOBJDIRPREFIX?= /var/obj

# we expect (but did not try) this to work instead of editing
# the two makefiles
MAKEOBJDIRPREFIX=/var/obj
export MAKEOBJDIRPREFIX

# in all cases 
make installworld

# this runs for 5 - 10 minutes

When we first ran the make installworld it failed with the following message:

touch: not found
*** Error code 127

After some panicky googling - we enjoy the panics, makes us feel alive - we discovered various mail archives that indicated the clock was wrong. Having followed the clock fixing instructions make installworld continued to fail. Our suspicions are now that the clock problem fix is extremely misleading advice - or perhaps fixed another problem entirely. The key point at this stage is that make installworld is trying to use the tools in the /obj directories (the new versions of the tools and utilities) to perform its actions, not those in the normal locations such as /usr/local/sbin etc.. This is done to ensure that the new kernel is matched with the new utilities to minimise any problems and to allow fairly serious changes to take place in userland.

If you get the Error Code 127 message we would recommend that you first confirm that the new version of the utility which has not been found, actually exists (was built correctly). In the case of touch (our problem) you will find this in (default) /usr/obj/var/src/usr.bin/touch (in our non-standard case /var/obj/var/src/usr.bin/touch). This is a directory and should contain at least touch.o and a touch executable. If these are not present, something mis-fired and you should back-track to step 1 and repeat the process checking each step of the way for good results. In our case the 127 error was a consequence of our non-standard paths - fixed using the makefile edits described above.

Finally we run mergemaster again to bring the /etc directories and others up-to-date (again if you are using standard paths ignore the NON_STD_PATH_ONLY commented lines):

# NON_STD_PATH_ONLY
# supply the location of the /obj directory
mergemaster -i -m /var/obj

# if running with standard paths use this line
mergemaster -i

# this can run for a considerable time

At the end of this step you have a system which has all the base libraries, tools and utilities ugraded to the new release (but not the ports). Time to test it.

# you can just do this
exit

# but we would suggest that you do a full reboot
shutdown -r now 
# OR
reboot

# at the beastie prompt hit ENTER (loads default) or wait for 
# timer to expire which then loads the default
# and watch your system spring into life

If you need a custom kernel you can do it anytime from here onwards using your favorite procedure.

Finally in this section we need to do a bit of housekeeping by cleaning the /obj directory (unless you will be building a custom kernel in which case do it afterwards). To recover some of that space you consumed:

# the old classic method
# NON_STD_PATH_ONLY
cd /var/obj
chflags -R noschg *
rm -rf *

# if you are using the standard paths
cd /usr/obj
chflags -R noschg *
rm -rf *

# OR 
# NON_STD_PATH_ONLY
cd /var/src
make cleanworld

# if you are using the standard paths
cd /usr/src
make cleanworld

# in all cases to clean temproot
# used by mergemaster
cd /var/tmp
chflags -R noschg temproot/*
rm -r temproot

Your production work should now be running but perhaps non-optimally since it may (most likely is) running via a compatability layer. Now it's time to fix the ports and bring everything up to date.

Step 6 upgrade ports

This is a low risk task but should be done as soon as practical. The current port tree on your newly upgraded system reflects those of the previous release and may have substantially changed. Certainly the ABI of the installed ports reflects the previous version so they need to be updated in any case. You can either use portsnap or a manual process as described here:

# this command fetches the latest ports
# zaps the ports tree (deleting any special patches you may have installed!)
# and rebuilds it
portsnap fetch extract update

# to subsequently update just use
portsnap fetch update

#  or use a cron job

When this is complete you should read /usr/ports/UPDATING. The endless joys.

Now run the following commands to clean up the ports collection:

# if you don't have portupgrade installed you should have, so
cd /usr/ports/ports-mgmt/portupgrade
make install clean

# update the ports index
cd /usr/ports
make fetchindex

# update the ports database
pkgdb -u 

# fix the ports database looking for stale or duplicate entries
pkgdb -F
# BEWARE: can ask lots of questions


# now upgrade your all your ports (will take a long time)
portupgrade -af

# or you can do them selectively, for example
portupgrade apache13

Welcome to the bright shiny new world. Take a couple of days off.

Step Back If things go wrong

If things appear to have gone wrong then these notes, together with a good dose of panic may help. However if you are reading this section for real and you do not have a backup, start thinking about ritual suicide.

If it is any comfort the first time we ever did this we ran all the step 5 processes at least a couple of times before we were successful and lived to tell the tale.

Up until Step 4 nothing serious has happened - recovery is trivial by restarting the entire process. You do however want to free up the space you may already have consumed by doing the following:

# the old classic method
# NON_STD_PATH_ONLY
cd /var/obj
chflags -R noschg *
rm -rf *

# if you are using the standard paths
cd /usr/obj
chflags -R noschg *
rm -rf *

# OR 
# NON_STD_PATH_ONLY
cd /var/src
make cleanworld

# if you are using the standard paths
cd /usr/src
make cleanworld

Then start the entire process again from Step 1.

If you fail in step 5 doing the first mergemaster run nothing has happened. Boot into your old kernel and try and figure what went wrong. Or you can rerun and watch more closely for clues - the utility provides lots of options for starting again - try them all - but methodically. If you are using non-standard paths make sure the -m option is correct for your /obj directory. You can jump backwards and forwards between the old and new kernels on a reboot as often as you like. You may even become addicted to the process. Look at the various files in /var/tmp/temproot and try and figure how far you got.
If you fail in step 5 doing the make installworld you may end up with a partially updated system, however it typically fails very early in the run when make attempts the first access. In this case nothing has yet happened and you can most likely reboot into the old kernel without any problems. However it's best to stay in the single user mode if you can and continue the debug.

If it fails later in the run try and figure the exact place that it failed and the reasons (use google - from another system, since yours is unusable - snigger, snigger) and in this case stay in the single user new kernel for as long as you can to repair the system. You can rerun the make installworld (well it can't get any worse can it) or use make installworld > /tmp/installworld to capture the output and then use an editor to plow through the file. Again if you are using non-standard paths make sure the MAKEOBJDIRPREFIX variable is set and correct or that you have correctly edited the Makefile and Makefile.inc1 files.

If you have no ideas and it's a re-image and re-install or nothing, then boot into either kernel (try both) using full multi-user mode and see what happens. You may get some more diagnostic ideas. It may even work.
If you fail in step 5 doing the second mergemaster you may end up with a partially updated system - but the commands and base libraries have been updated. You should be able to boot into the new system. However try and figure the exact place that it failed and the reasons and stay in the single user new kernel for as long as you can to repair the system. You can rerun mergemaster using mergemaster -i > /tmp/merge2 to capture the output and then use an editor to plow through the file looking for the errors. Again, if you are using non-standard paths make sure you use the -m option and that it points to the correct path.

Updating FreeBSD

While the utility freebsd-update has apparently been part of the ports collection for a number of years (so we are told) as of 6.2 it is now part of the base system. freebsd-update does a ton of things but has two major functions. First, it is a method of keeping your system updated with kernel patches and security fixes. Second, it can be used to upgrade FreeBSD either from one minor version to another (say, from 8.1 to 8.3) or from one major version to another (say, 8.2 to 9.0). To use the freebsd-update update service to maintain security fixes you must meet the following criteria:

Be on a -RELEASE version of FreeBSD (see notes on subversion branches).

# to check your release 
uname -r
# anything post 6 with -RELEASE is supported

Have a bog-standard GENERIC kernel. Custom kernel users are out of luck.
Your source lives in /usr/src. Non standard source locations are also out of luck.

This is a great service innovation. What was a painful process is now trivial in the extreme. We currently wait for a Security Advisory (freebsd-announce mailing list) and then do a manual update. The author of freebsd-update, Colin Percival, wrote to encourage use of the daily cron service since the "no update" overhead is trivial. Find a way to use this service - it really is that good.

Running freebsd-update for security maintenance

Freebsd-update will take one or more of the following parameters:

# command format
freebsd-update [options]
# where one or more options may be present from below:
# fetch - download (not install) required updates
# install - installed updated downloaded by fetch
# rollback - uninstall the most recent install
# cron - required parameter when running from cron (see below)

# fetch then install updates
freebsd-update fetch install

# running from cron
# daily cron driven automatic update
# add the following line to /etc/crontab
0 3 * * * root /usr/sbin/freebsd-update cron fetch install
# IMPORTANT: If your machine is set to UTC use a random 
# value other than 3 to avoid a deluge of requests to the 
# update servers or use this to run every n days at 4 am or other time
0   4   */n    *   *  /usr/sbin/freebsd-update cron fetch install

Note: Corrected version of this note. For whatever reason during the update procedure we ended up with two versions of freebsd-update and no /etc/freebsd-update.conf. The latest (correct) version is installed in /usr/sbin and assuming a normal PATH variable (usr/sbin is before /usr/local/sbin), this is found and executed before any older freebsd-update (in /usr/local/sbin). The missing freebsd-update.conf may be found at /usr/src/etc/freebsd-update.conf (or in our case /var/src/etc/freebsd-update.conf) and is shown below. Finally there is a version of freebsd-update on the ports tree which cannot be installed under FreeBSD 6.2 and in any case is the OLD version which uses a different parameter set. If you have incorrectly deleted the new version you can restore it from http://www.freebsd.org/cgi/cvsweb.cgi/src/usr.sbin/freebsd-update/freebsd-update.sh download version 1.8, place in /usr/sbin and rename to freebsd-update (remove the .sh suffix) and chown to 0644.

# delete the old version if installed
rm /usr/local/sbin/freebsd-update
# check and delete any freebsd-update.conf[.sample]
# in /usr/local/etc

freebsd-update.conf

As stated earlier after the upgrade we did not have a copy of the configuration file in /etc/freebsd-upgrade.conf after the source upgrade process. You can find it in /usr/src/etc/freebsd-update.conf and copy unmodified to /etc. In case this method fails we reproduce the standard distribution of this file below. Note: The parameter format of this file has changed from previous versions of freebsd-update hence our advice to delete any old freebsd-update.conf files from /usr/local/etc to avoid confusion.

# $FreeBSD: src/etc/freebsd-update.conf,v 1.1.2.1 2006/09/02 11:05:23 cperciva Exp $

# Trusted keyprint.  Changing this is a Bad Idea unless you've received
# a PGP-signed email from <security-officer@FreeBSD.org> telling you to
# change it and explaining why.
KeyPrint 800651ef4b4c71c27e60786d7b487188970f4b4169cc055784e21eb71d410cc5

# Server or server pool from which to fetch updates.  You can change
# this to point at a specific server if you want, but in most cases
# using a "nearby" server won't provide a measurable improvement in
# performance.
ServerName update.FreeBSD.org

# Components of the base system which should be kept updated.
Components src world kernel

# Example for updating the userland and the kernel source code only:
# Components src/base src/sys world

# Paths which start with anything matching an entry in an IgnorePaths
# statement will be ignored.
IgnorePaths

# Paths which start with anything matching an entry in an UpdateIfUnmodified
# statement will only be updated if the contents of the file have not been
# modified by the user.
UpdateIfUnmodified /etc/ /var/

### Default configuration options:

# Directory in which to store downloaded updates and temporary
# files used by FreeBSD Update.
# WorkDir /var/db/freebsd-update

# Destination to send output of "freebsd-update cron" if an error
# occurs or updates have been downloaded.
# MailTo root

# Is FreeBSD Update allowed to create new files?
# AllowAdd yes

# Is FreeBSD Update allowed to delete files?
# AllowDelete yes

# If the user has modified file ownership, permissions, or flags, should
# FreeBSD Update retain this modified metadata when installing a new version
# of that file?
# KeepModifiedMetadata yes

freebsd-update upgrading minor or major versions

All the latest release notes fully document upgrading to a new major or minor version of FreeBSD using freebsd-update. We have used it for minor version upgrading (8.1 to 8.3) and it was a breeze. We used it for a major release upgrade (8.x to 9.x) and it was a disaster. To be fair, the reason it was a disaster was because the new major release would not install on our hardware (we confirmed that by subsequently trying a clean install with 9.x from a CD).

When performing an upgrade the process is relatively trivial and consists of:

# inspects your system and MAY alter some files which 
# you are asked to verify/merge (rarely happens on minor version updates)
# in the example below we show 8.3-RELEASE it could 
# have been any valid RELEASE with a major or minor version change
freebsd-update upgrade -r 8.3-RELEASE

# when this completes then

freebsd-update install

# can out put a humungous amount of data using |more which is a pain
# when this completes it will normally ask you to
# boot to the new system

shutdown -r now

# when the new system has booted
# run freebsd-update again to finish the install

freebsd-update install

# for minor version releases that it usually it
# but for major release updates you will be prompted to 
# rebuild all installed applications (ports) and when that is done
# run freebsd-update yet again

freebsd-update install

# finally boot into the new system again
shutdown -r now

Notes:

When updating to a new release lots of stuff changes (much less so with a minor release). When scripts and configurations files are modified freebsd-update diffs them, loads them into vi and presents them for reconciliation. Since most of the changes are simple version number comment field changes this can be a serious pain (in our case well over 50 files were changed, over 40 were comment only changes). However, be warned - diligence is necessary (no :wq to every file without looking). Further, unless the file is visibly short all the diff marks (:/>>>/) need to be viewed and reconciled. Failure to do this will cause the resulting file to contain the diff indicators and many essential system files will simply croak when loaded (including the password file in our case) resulting in endless hours of fun looking for the files that were modified (guess how we found this out). While there is really no alternative to genuine file changes, having to undergo the edit reconciliation for comment only changes results in a serious shortcoming in the process. Better, perhaps, to move the replaced files into a special location and let the user manually reconcile those of interest. Just how often do you change /etc/periodic/weekly/310.locate?

Ports Collection Updating

P portsnap is the latest addition to the various utilities available to maintain the ports and packages collections on FreeBSD. Before describing portsnap it is worthwhile reviewing all the choices that are available for ports and packages.

Classic Ports: The classic port management system which uses the following commands:
```
cd /usr/ports/port-category/port-name
make install clean
```

Manual update: All ports are now managed in a subversion version control system. To manually update the ports tree do the following:

# clean the ports tree
# you may want to think about saving any special
# or modified files before you delete the directory!
rm -r /usr/ports
# recreate the ports base
mkdir /usr/ports

# download the ports tree
svn co svn://svn.freebsd.org/ports/head /usr/ports

# use this handy svn web viewer
# rebuild the index 
portupgrade -ar

# to subsequently update the ports collection
svn update /usr/ports

portsnap: Preferred method of updating the ports trees and part of the base since 6.x. Configuring and using portsnap.

portupgrade/portinstall: Utilities to manage upgrading installed ports (portupgrade) or the installation of ports/packages (portinstall). Not installed by default but the port may be found in sysutils/portuprade or in its new location ports-mgnt/portupgrade. Portupgrade, in spite of its name, installs both portinstall and portupgrade. The basic command lines are shown below:

portupgrade -R port-name
# where port-name is as much as required to identify the port
# use pkg_info|grep port-name to get full details

portinstall -PR port-name
# where -P uses a package if available and 
# -R update all dependencies
# port-name is as it appears in the ports collection
# or enough of it to be unique
# alternate to make install clean - no need to know the
# collection-name

portsnap

portsnap is now (6.x onward) installed as part of the base and does not appear in the ports collection. portsnap is more secure and faster than using either cvsup or csup. It uses a configuration file (/etc/portsnap.conf) which was not installed as part of our upgrade process outlined above. We reproduce the standard distribution file (which you can use unmodified) in all its glory in case you ever misplace or lose it:

# $FreeBSD: src/etc/portsnap.conf,v 1.1.2.3 2006/01/21 09:58:12 cperciva Exp $

# Default directory where compressed snapshots are stored.
# WORKDIR=/var/db/portsnap

# Default location of the ports tree (target for "update" and "extract").
# PORTSDIR=/usr/ports

# Server or server pool from which to fetch updates.  You can change
# this to point at a specific server if you want, but in most cases
# using a "nearby" server won't provide a measurable improvement in
# performance.
SERVERNAME=portsnap.FreeBSD.org

# Trusted keyprint.  Changing this is a Bad Idea unless you've received
# a PGP-signed email from <security-officer@FreeBSD.org> telling you to
# change it and explaining why.
KEYPRINT=9b5feee6d69f170e3dd0a2c8e469ddbd64f13f978f2f3aede40c98633216c330

# Example of ignoring parts of the ports tree.  If you know that you
# absolutely will not need certain parts of the tree, this will save
# some bandwidth and disk space.  See the manual page for more details.
#
# WARNING: Working with an incomplete ports tree is not supported and
# can cause problems due to missing dependencies.  If you have REFUSE
# directives and experience problems, remove them and update your tree
# before asking for help on the mailing lists.
#
# REFUSE arabic chinese french german hebrew hungarian japanese
# REFUSE korean polish portuguese russian ukrainian vietnamese

Note: The REFUSE parameters in the file above are commented out but you may want to review these parameters depending on what languages and groups you really want since portsnap maintains additional indexes which approach a doubling in space requirements for the ports collection.

The first time you run portsnap requires the following command:

portsnap fetch extract update

Note: The above command zaps and replaces the entire ports collection. If you have special patches or other stuff in there, save it (preferably before running portsnap!) and restore after running. The command downloaded a whopping 49M file and keeps its status files in /var/db/portsnap from which it does the comparisons of what to download.

One you have completed the initialization process you can run a manual or cron driven update a shown:

# manual update (assuming a regular cron job has 
# fetched the updates - see crontab below)
portsnap update

# if no crontab entry then latest updates 
# need to be manually fetched then updated with following command
portsnap fetch update

# daily cron driven update
# add the following line to /etc/crontab
0 3 * * * root /usr/sbin/portsnap cron
# IMPORTANT: If your machine is set to UTC use a random 
# value other than 3 to avoid a deluge of requests to the 
# update servers or use this to run every n days at 4 am
0   4   */n    *   *  /usr/sbin/portsnap cron

Note: If no crontab job is performed on a regular basis then a manual portsnap fetch update can take a seriously long time. Whether portsnap is longer or shorter that issuing a csup/cvsup with ports-all was not measured - it just seemed to take longer! The key advantage of portsnap is that it can be driven off a crontab job (daily, weekly etc) and thus keep the ports system in good shape at all times.

A bare bones 6.3 install and moving files from an existing 5.x server

For a variety of reasons we decided that to upgrade an operational server running 5.4 we would do a clean install of 6.3 and move all the files. We were somewhat suspicious that serious cruft had grown over time on the 5.4 server and wanted no down time on the existing server. So we installed a bare bones 6.3 on a new server and consequently were only under modest time pressure to complete the task. In both the new 6.3 and 5.4 servers we only had remote ssh access which may help explain some of the tasks.

Problems and Issues

This lists the order in which we carried out the tasks and problems we encountered in building a new (6.3) server and moving files from an existing (operational) 5.4 server.

Using sysinstall we built a new server with NO options selected (and excluding the ports collection) and with a single FBSD partition.
Our first problem was that we had no name resolution service so could not access the outside world. Ouch. The nameserver library routines are always installed but we had no /etc/resolv.conf and since IP addresses and other details were configured statically we had a minor problem:
```
# create a bare bones resolv.conf
vi /etc/resolv.conf
domain example.com
# we had to open the DNS caching server
# at this address to enable queries from the server IP
# using an extended allow-recusion {x.x.x.x;}; in
# named.conf at 192.168.1.1
nameserver 192.168.1.1
:wq
```
We ran freebsd-update to bring the base up to the latest standard (a default /etc/freebsd-update.conf was already present):
```
freebsd-update fetch install
```
We ran portsnap to bring the ports to the latest standard (a default /etc/portsnap.conf was already present but we uncommented and edited the REFUSE statements to keep the ports size to a minimum):
```
portsnap fetch extract update
```

Ports vs RubyGems

RubyGems is Ruby's automated installer and package manager (along the lines of CPAN). The problem here is on systems with an established package manager (BSD ports or apt-get and others on some Linux distros) you have make a choice - Gems, your OS's method or work in some kind of mixed state. As an example you can install devel/ruby-gems from the ports collection under FreeBSD and then use:

gem install rails

Or use directly use the the ports collection like this:

cd /usr/ports/www/rubygem-rails
make install clean

# OR
portinstall rubygem-rails

So what's the difference:

There is no visibility in the ports collection of the installs using RubyGems - instead you have to use a separate set of commands (albeit relatively simple ones) to check on what is installed (gem list) rather than pkg_info.
RubyGems is a pretty active community and few (relative to the total of RubyGems) are available using the ports system. However the major infrastructure gems are (depending on your definition of major and infrastructure).
There is a finite delay between updating the ports collection and releasing via RubyGems. If you need a hot-from-the-press version installing via RubyGems will get the very latest - maybe, or maybe not, via the ports collection.
Both methods now install to the same place - there was a time when this was not the case (/usr/local/lib/ruby/gems/x.x/gems/). RubyGem commands are aware of packages installed via ports (which uses a wrapper to invoke gem).
The RubyGems command line is, arguably, simpler for an install.

We chose to go with ports where we could get them and fall back to using the gem install xxx only if necessary. Many others use RubyGems exclusively and for serious ruby users this may be the only route.

RSYNC - clients and servers

Back-up check list

Archiving requirements are unique to all users and the systems they are running. this is not meant to be an exhaustive section on how to backup your server but some notes that may, or may not, help. This section is, quite simply, some thoughts about what (and when) to backup to try and avoid the ghastly intake of breath if the fateful day arrives.

/etc - not big and it contains all that stuff that you took ages to configure all those config files are jammed full of change control notes (aren't they?). Weekly update, shorter if you are busy bees.
/usr/local/etc - most of the application stuff is here, mail configs, web configs etc, etc. Vital stuff. Weekly update unless you are busy bees.
If this is a web server /usr/local/www (or wherever you keep your webs). Frequency depends on what you are prepared to lose. From 1 - 3 days.
DB files. We run mysqldump every day and store the results in /var/archive/sql using a dd-mm-yyyy.sql format which we prune weekly (and keep for two months). Then rsync the /var/archive/sql directory every 3 - 6 days.
Slow moving stuff like custom kernel configurations we copy changed files daily with a trivial script to scan defined locations for todays date and copy to /var/archive/misc and rsync weekly.

One day real soon now™ we'll finish this section.

Problems, comments, suggestions, corrections (including broken links) or something to add? Please take the time from a busy life to 'mail us' (at top of screen), the webmaster (below) or info-support at zytrax. You will have a warm inner glow for the rest of the day.