Managing third-party source code with CVS

Chris Houghten

Abstract

THIS IS INCOMPLETE! DONT READ IT YET!!

This paper continues on where Luke Mewburn leaves off in his paper describing techniques for using revision control and documentation to install and maintain third-party source code.

Introduction

Please see Luke Mewburns paper and then return here to see my clarifications His paper can be found at http://www.mewburn.net/luke/talks/sage-au-99/#updateexisting

I really did not like the outcome of Lukes procedure. What I really wanted was a way to have my code changes isolated. In my opinion, none of the third party code itself is of amy importance to me personally and I just need it checked in my CVS repository so I can build it. In my case, the versions of the third party code were so different that simple merging was not an option and the errors I was getting were so generic that I needed to be able to isolate my code and possible manually merger them in one at a time.

Anyway, my procedure is as follows: 1: import the new package as described by Luke 2: Checkout the module as described by Luke 3: Make your changes as described by Luke 4: Run update as described by Luke 5: Commit your changes as described by Luke 6: TAG ALL THE FILES YOU JUST COMMITTED USING A AG SPECIFICALLY FOR ONLY YOUR CODE 7: Import the new version of the third party source as described by Luke 8: Checkout the module MERGING THE NEW VERSION WITH ONLY YOUR SOURCE 9: Update as described by Luke 10: commit your changes as described by Luke 11: TAG ALL THE FILES YOU JUST COMMITTED USING A NEW TAG SPECIFICALLY FOR ONLY YOUR CODE Theses tags you make for your source code allow you to checkout just your code so you can see what you have changed and manually merge/copy/mangle it to get it to work with the newer version of the third party source It also allows you to be able to completely disregard the older third party source because that source essentially no longer exists and the merging from the old third party version to the new one was already done by that third party This is especially true and useful when skipping versions or when the third party completely changed their code through refactoring or eliminating of some code and merging is more or less impossible.

Importing a new package

To import a new package, we require a fresh copy of the source. Suggested locations to look for software are described below in Finding third-party source.

In this example we will use the Internet Software Consortium's DHCP server - dhcp 2.0b1 pl18.

Create a new entry in the 3RDPARTY file for the package. Complete the sections on the package name and version, and where the package came from. This is useful in case you get sidetracked and forget where you got the distribution from.
Extract the source into a subdirectory. Usually most programs nowadays are packaged so that they extract into a suitably named subdirectory (e.g., dhcp-2.0b1pl18), but there is always one software developer who does not bother, so check first.
Change directory into the top-level of the distribution.
Import the source into the repository. To do this, you will need the following information:
- location in repository (e.g., net/dhcp)
- vendor tag (e.g., ISC)
- release tag (e.g., dhcp-2-0-b1-pl18).
  Note that the tags can not have periods in them, so we change the "." in the release name to "-".
Run cvs import to import the code:
```
	% cvs import -m 'ISC dhcp 2.0b1pl18' net/dhcp ISC dhcp-2-0-b1-pl18
	N net/dhcp/CHANGES
	N net/dhcp/Makefile.conf
		...
	N net/dhcp/server/dhcpd.leases.cat5

	No conflicts created by this import
```
If -m '...' is not supplied an editor will be invoked and you will be prompted for a commit message.
The "N " at the start of the each line is the status character; "N" means new file. Refer to the CVS info documentation [4] node "import output" for the meaning of other status characters.
The package is now in the repository.
Depending upon how the CVSROOT/loginfo file has been configured, you might receive an email log message similar to:
```
	Update of /src/cvsroot/net/dhcp
	In directory wombat.cs.rmit.edu.au:/tmp/dhcp-2.0b1pl18

	Log Message:
	ISC dhcp 2.0b1pl18

	Status:

	Vendor Tag:     ISC
	Release Tags:   dhcp-2-0-b1-pl18

	N net/dhcp/CHANGES
	N net/dhcp/Makefile.conf
		...
	N net/dhcp/server/dhcpd.leases.cat5

	No conflicts created by this import
```

To compile and install the package, a checked out copy is required:

	% cd ~/some-scratch-directory
	% cvs checkout net/dhcp
	% cd net/dhcp

Configure the package for compilation. There should be suitable documentation in the distribution regarding this. A lot of programs use a configure script to automatically determine the settings of the current system. Some, however, will require manual editing of Makefiles and source code.
For this example, I needed to change some of the paths in Makefile.conf and includes/site.h to reflect our site's policy on where we install configuration and run-time files.
After modifying the files, we can see what files changed from the distribution by running:
```
	% cvs update
	M Makefile.conf
	M includes/site.h
```
The actual changes can be shown with cvs diff which will display the diff output. I prefer cvs diff -up, which outputs a unified diff ("-u"), with the C function the change is in preceeding each block ("-p").
Before compilation we can commit our changes back to the repository using:
```
	% cvs commit -m 'update for local paths'
	Checking in Makefile.conf;
	/src/cvsroot/net/dhcp/Makefile.conf,v  <--  Makefile.conf
	new revision: 1.2; previous revision: 1.1
	done
	Checking in includes/site.h;
	/src/cvsroot/net/dhcp/includes/site.h,v  <--  site.h
	new revision: 1.2; previous revision: 1.1
	done
```
Configure the program with:
```
	% ./configure sunos5-cc
```
We can see what ./configure changed by running cvs update again:
```
	% cvs update
	? Makefile
	? client/Makefile
	? common/Makefile
	? relay/Makefile
	? server/Makefile
```
The "?" means that CVS does not know about the file.
Compile and install the program:
```
	% make
	% make install
```
Update the 3RDPARTY file with the relevant information!

If you think you will want to use the cvs history functionality (q.v.), indicate to CVS that you do not need the working copy anymore:

	% cvs release -d net/dhcp
	? Makefile
	? client/Makefile
	? common/Makefile
	? relay/Makefile
	? server/Makefile
	You have [0] altered files in this repository.
	Are you sure you want to release (and delete) directory
	`net/dhcp': y

Updating an existing package

It is rare for third-party software to remain unchanged forever; updates are available (even on a regular basis).

An important part of managing third-party software is upgrading to a newer version in a sane way.

The process to import an updated version of a package is similar to importing the original package. However, because changes in the newer version may conflict with any local changes that you may have made to the product, there are a couple of steps that are different.

In this example, we'll be upgrading dhcp 2.0b1 from pl18 to pl27.

In the directory containing the extracted source import the new version:
```
	% cvs import -m 'ISC dhcp 2.0b1pl27' net/dhcp ISC dhcp-2-0-b1-pl27
	U net/dhcp/CHANGES
	C net/dhcp/Makefile.conf
		...
	U net/dhcp/server/dhcpd.leases.cat5

	1 conflicts created by this import.
	Use the following command to help the merge:

		cvs checkout -jISC:yesterday -jISC net/dhcp
```
As you can see there was a conflict on the import (Makefile.conf; the file with the status character of "C"). This is CVS's way of indicating that those files have been modified from the previous release of the vendor code.
The recommended command to "help the merge";
```
	cvs checkout -jISC:yesterday -jISC net/dhcp
```
should NOT be performed, for the following reasons:
- it only has a granularity of one day; so that it will not work if the previous vendor release was imported today.
- it does not do a great job of merging, adding or removing files that changed between vendor releases.
It ends up creating more work than necessary.
Checkout the updated source, automatically merging in any changes. Whether or not CVS suggests a checkout command, you should use a checkout of the form:
```
	cvs checkout -j old-release-tag -j new-release-tag module
```
old-release-tag is the previous release tag, and new-release-tag is the one just imported.
For this example the cvs command is:
```
	% cvs checkout -j dhcp-2-0-b1-pl18 -j dhcp-2-0-b1-pl27 net/dhcp
	U net/dhcp/CHANGES
	U net/dhcp/Makefile.conf
	RCS file: /src/cvsroot/net/dhcp/Makefile.conf,v
	retrieving revision 1.1.1.1
	retrieving revision 1.1.1.2
	Merging differences between 1.1.1.1 and 1.1.1.2 into Makefile.conf
		...
	U net/dhcp/server/dhcpd.leases.cat5
```
The advantages of using checkout -j old-release-tag -j new-release-tag include:
- files added or deleted by the vendor between releases are handled automatically, and
- merges between changes are cleaner.
Remember; always use cvs checkout -j old-release-tag -j new-release-tag when checking out code for the first time after you have updated a vendor release. It saves a lot of time.
Search for and resolve any conflicts. A couple of CVS commands are useful for this. cvs update will show any files modified, added, or removed by the vendor between releases.
```
	% cvs update
	M Makefile.conf
```
Note that even though include/site.h was locally modified in the previous release, it does not show up as modified here. This is because it was not actually modified by the vendor between the two vendor releases.
cvs diff will display any changes between the last release and this code; changes either from local updates or from the new release. The diff output will also display our changes to include/site.h.
It is probably more useful to use cvs diff -r VENDOR to display changes between the most recent vendor release and this code:
```
	% cvs diff -r ISC
	Index: Makefile.conf
	=======================================================
	RCS file: /src/cvsroot/net/dhcp/Makefile.conf,v
	retrieving revision 1.1.1.2
	diff -r1.1.1.2 Makefile.conf
	133a134,141
	> ## RMITCS overrides
	> #BINDIR = /usr/local/sbin
	> #CLIENTBINDIR = /usr/local/sbin
	> #ADMMANDIR = /usr/local/man/cat1m
	> #FFMANDIR = /usr/local/man/cat4
	> #VARRUN = /var/run
	> #VARDB = /var/run/dhcp
	> #SCRIPT=none
	Index: includes/site.h
	=======================================================
	RCS file: /src/cvsroot/net/dhcp/includes/site.h,v
	retrieving revision 1.1.1.1
	retrieving revision 1.2
	diff -r1.1.1.1 -r1.2
	39a40
	> #define _PATH_DHCPD_PID       "/var/run/dhcpd.pid"
	45a47
	> #define _PATH_DHCPD_DB        "/var/run/dhcpd.leases"
	50a53
	> #define _PATH_DHCPD_CONF      "/usr/local/etc/dhcpd.conf"
```
Peruse the output of the diffs and determine if things look reasonable. In this case I am fairly happy with the changes.
Refer to the CVS info documentation [4] node "Conflicts example" for more information on resolving conflicts.
In Tips and tricks below I will discuss techniques for minimising conflicts when you make local changes.

Commit the changes. This is a similar operation to the commit after the original import:

	% cvs commit -m 'merge new version'
	Checking in Makefile.conf;
	/src/cvsroot/net/dhcp/Makefile.conf,v  <--  Makefile.conf
	new revision: 1.3; previous revision: 1.2
	done

Configure the program. This is where the 3RDPARTY documentation is extremely useful. Generally the installation procedures are similar between releases (especially patch releases). In this case, (after checking the installation documentation of the newer release), we run:
```
	% ./configure sunos5-cc
```
Compile and install the program is as before:
```
	% make
	% make install
```
Update the 3RDPARTY file with the new information (release tag, installation host and date, and any modified install notes).

Advanced CVS operations

CVS has other operations which are useful to know. Whilst these may be more relevant to a software developer, they are still useful for our purposes:

cvs annotate
Displays the specified files with the version information of the last modification of each line preceeding the contents of the line. For example:

	% cvs annotate Makefile.conf
		...
	1.1          (lukem    25-Mar-99): #VARDB = /etc
	1.1          (lukem    25-Mar-99): #SCRIPT=solaris
	1.2          (lukem    26-Mar-99): ## RMITCS overrides
	1.2          (lukem    26-Mar-99): #BINDIR = /usr/local/sbin
		...

cvs tag symbolic_tag
Assigns the symbolic tag symbolic_tag to the to the nearest repository revisions of the files checked out in the current directory.
This is useful to checkpoint software for future reference and/or comparison.
cvs rtag symbolic_tag module
Similar to cvs tag, except that it performs the operation directory on the repository copy of module (I.e., module does not have to be checked out).
cvs history
CVS tracks the history of each checkout, commit, release, rtag, and update. cvs history displays this history. By default, it shows the modules you currently have checked out.
If cvs release is used after you have finished with each checked-out module, then cvs history is a useful indicator of what is still checked-out. Some people do not bother releasing modules, so the history may not be that useful.

Tips and Tricks

CVS is a powerful tool and it has useful functionality that is often only hinted at in the documentation. This sections covers some of that functionality, and other tips and tricks that I felt might be relevant.

cvs checkout -j old-vendor-release -j new-vendor-release module

In my opinion, this is one of the most useful underdocumented operations in CVS, especially for the purpose of maintaining third-party packages (or anything that uses vendor branches).

As shown in the example in Updating an existing package, this operation checks out a copy of module and merges any changes made by the vendor between release tags old-vendor-release and new-vendor-release. This includes files that were added or removed between releases.

Tagging the installed release

It may be useful to tag a package at a known working state after successful installation or prior to the import of a new version. This can simplify determining the local changes made to the previous vendor release.

The sequence of operations would be something like:

Successfully install a package (e.g., net/dhcp)
Tag it with:
```
	% cvs rtag dhcp-local net/dhcp
```

A while later a new version comes out. Import and update it with:

	% cvs import -m 'dhcp 2.0b1pl28' ISC dhcp-2-0-b1-pl28 net/dhcp
	% cvs checkout -j dhcp-2-0-b1-pl27 -j dhcp-2-0-b1-pl28 net/dhcp

Determine the local changes in the previous release:

	% cvs diff -r dhcp-2-0-b1-pl27 -r dhcp-local > diffs.last

Determine the local changes after the checkout:
```
	% cvs diff -r ISC > diffs.now
```
Compare diffs.last and diffs.now to see if the changes look similar.
Install as before.
Update the known working state tag to the current code with:
```
	% cvs rtag dhcp-local net/dhcp
```
(etc.)

Minimising conflicts

It is possible to modify code to minimise conflicts when the vendor changes the same sections of the code. Rather than change a line provide an override line which has the same effect.

In Makefile and shell script assignments add a line after the original. For example, if the original assignments were:
```
	CC=gcc
	CFLAGS=-Wall -Werror -g
```
and cc -O is the preferred compiler, change this to:
```
	CC=gcc
	CFLAGS=-Wall -Werror -g
	CC=cc
	CFLAGS=-O
```
In C source files, use preprocessor directives to comment out or override code. For example, if the original code was:
```
	/* #define _PATH_DHCPD_DB	"/etc/dhcpd.leases" */
```
change this to:
```
	/* #define _PATH_DHCPD_DB	"/etc/dhcpd.leases" */
	#define _PATH_DHCPD_DB		"/var/run/dhcpd.leases"
```
It may be necessary to add #undef item before a local #define item value, in case item was defined elsewhere.

Binary files

Certain packages are distributed with binary files that must be stored unmodified in the repository and be checked out unmodified upon a check-out.

The simplest way to support a package with some binary files is:

After extracting the original source, move the binary files into another directory (in the same structure as the original distribution)

Import the remaining source as per normal:

	% cvs import -m '...' vendor release module

In the directory with the binary files, run
```
	% cvs import -kb -m '...' vendor release module
```
(the "-kb" is the important part).

Symbolic links

CVS ignores symbolic links upon import (the files have a status of "L").

A workaround is to add a local script to module which is run manually after checkout to regenerate the symlinks. I call this fixlinks, and generate it with:

	% find . -type l -print | perl -e 'while (<>) { chomp ; \
	    print "ln -s ", readlink($_), " $_\n"; } ' > fixlinks

An example fixlinks (from our net/netatalk package) is:

	ln -s ../sys/netatalk ./include/netatalk
	ln -s ../codepage.h ./etc/afpd/nls/codepage.h
	ln -s kpatch-4.2 ./sys/ultrix/kpatch-4.4
	ln -s kpatch-4.2 ./sys/ultrix/kpatch-4.3
	ln -s ../../ultrix/sys/cdefs.h ./sys/solaris/sys/cdefs.h

Upon checkout, regenerate the links with:

	% sh ./fixlinks

Automatically generating modules file

The CVSROOT/modules file provides a mapping from a module alias to a directory or directories. I find it useful to have shortcuts to the packages in the repository. For example, this allows me to run cvs checkout dhcp instead of cvs checkout net/dhcp.

I wrote a script a few years ago called genmodules [7] which which parses CVSROOT/commitlog and updates CVSROOT/modules as necessary.

Working with multiple repositories

If you are working with multiple repositories (e.g., a local repository and that of an open source software project), it may help to have shell aliases which do the right thing.

For example, in my .cshrc I have:

	alias   ncvs    'env CVSROOT=cvs.netbsd.org:/cvsroot cvs'
	alias	cscvs	'env CVSROOT=wombat:/src/cvsroot cvs'

Setting defaults for CVS commands

If you are always invoking a CVS command with the same set of options, you can simplify your typing by adding a relevant entry in $HOME/.cvsrc. For example, I have a line of the form:

	update -dP

which means that cvs update ... is run as cvs update -dP ... ("-d"; build directories, "-P"; prune empty directories).

Organising the repository

It is highly recommended to have a sensible organisation for the third-party packages in the repository. For example, we have the following directories; it should be trivial to infer the contents of the directories:

	% ls $CVSROOT
	CVSROOT     devel      lang       rmit       www
	ai          docs       mail       security   x11
	archivers   file       net        text
	audio       graphics   news       utils

Finding third-party source

An element of managing third-party source is finding the source in the first place :-) A few good places to look include:

The SAGE-AU ftp site contains a lot of software relevant to system administrators (of course! :-): ftp://ftp.sage-au.org.au/pub/
The SAGE-AU FTP Update Notification. This is a service which emails you when packages that you have tagged as "packages you are interested in" have been updated.: http://www.sage-au.org.au/resource/ftp_update.html
The AARNET mirror site:: ftp://mirror.aarnet.edu.au/
The web front-end to the AARNET mirror site:: http://mirror.aarnet.edu.au/
The Telstra Internet archive server:: ftp://archie.au/
Walnut Creek's ftp server:: ftp://ftp.cdrom.com/
The package source for NetBSD. package source in NetBSD parlance consists of a Makefile and optional patches. This was derived from similar work in FreeBSD. Running make downloads a fresh copy of the third party source, check for any other packages it depends upon, applies any patchs, compiles and installs the program. The Makefiles are useful to peruse to find distribution sites, and the patches are sometimes useful (for security patches for older packages, etc). (Australian mirror sites):: ftp://ftp.au.netbsd.org/pub/NetBSD/NetBSD-current/pkgsrc/; ftp://ftp.au.freebsd.org/pub/FreeBSD/FreeBSD-current/ports/
Sometimes the use of search engines such as Altavista can be useful. However, in recent years I have noticed you spend a lot of time searching through a list which has hundreds of matches due to hits of ports of the package on various linux mirror sites. Your Mileage May Vary. The search facility on http://www.freshmeat.net/ may be useful to Linux users.

Summary

I believe that CVS is well suited to the task of managing third-party software. I have been using it this role for over five years for three different employers. For over three years I have also been using CVS as one of the many distributed developers of the NetBSD project [8].

Before I started my current position at RMIT Computer Science nearly two years ago, /usr/local/src was a two gigabyte directory with a haphazard structure (I hesistate to call it "organisation"). In many cases there multiple copies of the same product, without any obvious indication of which was the currently installed version (in the case of the Columbia Appletalk Package there were six different versions of the source code). As we rebuilt systems (including rebuilding /usr/local from scratch on new machines), we kept all new products in the CVS repository.

Hand in hand with CVS is the maintenance of relevant documentation. Without documentation a CVS tree is almost as bad as the proverbial unorganised /usr/local/src. NetBSD provided the inspiration for the 3RDPARTY file, although I have expanded upon it since then.

Acknowledgements

Thanks to Matt Green for teaching me the checkout -j old -j new module trick (amongst others); in my opinion it is one of the most useful commands to know when maintaining third party source.

Giles Lean's assistance by reviewing the paper and providing a wealth of feedback was much appreciated.

References

[1]	Cyclic CVS site, http://www.cyclic.com/cvs/info.html
[2]	CVS Overview, http://www.cyclic.com/cyclic-pages/overview.html
[3]	Introduction to CVS, http://www.cyclic.com/cvs/doc-blandy-text.html
[4]	*The CVS info* documentation.** This should be installed as part of the CVS installation.
[5]	CVS Reference Manual, http://www.loria.fr/~molli/cvs/doc/cvs_toc.html
[6]	SSH secure shell, http://www.ssh.fi/
[7]	genmodules, http://www.cs.rmit.edu.au/~lukem/src/genmodules
[8]	The NetBSD Project (Australian mirror), http://www.au.netbsd.org/