The CentOS Rebuild and Release Process

"Just want to point out that CentOS does not use anything from that page 
- and details / scripts on that page have nothing to do with the CentOS 
process."

"The build environment is a staged install of CentOS. And mock.  Every
buildroot is created with a command to mock.

Here is an example scripts that I use to build "staged" extra's stuff
for CentOS5:

http://people.centos.org/hughesjr/buildsystem/

You will notice there is nothing particularly special about any of that.

It just uses a local file:\\ dir in the config, there is a createrepo in
the build script between packages if it builds successfully.

The cfg files are the mock configs, the other files are the build
scripts (the generate script creates some "lock files" in a list
directory so we can have more than one machine build against the same
directory of SRPMS.  It orders them in reverse date order (build oldest
file first).

There is no magic here.

Once built ... we use the tmverifyrpms against it from here:

 http://mirror.centos.org/centos-4/4/build/distro/

The RPM either passes or fails the link test, the files test, and the
size test ... if it fails, we figure out why and fix it.

Not sure what else you are looking for."

1. When will a new CentOS release be issued?

The goal for release of a new point release or update set is four to eight weeks after the release by upstream. It may move + or - about two weeks. A new major version will be expected to take longer than a mid-cycle point release; a mid-cycle point release similarly takes more time than the 'entry into maintenance mode' point release.

There is a QA mailing list and a QA team. People who are interesting in doing QA (and can test the release on real hardware), who have Linux sysadmin backgrounds, and who participate on IRC and/or the forums and/or the mailing lists and are known to the developers can participate. This is not an open SIG type group, for a number of reasons, chief among which is that there has been abuse of such membership to simply be an 'early access' venue, without the work of the group being performed.

There are a number of developers who could do the rebuild work in an emergency.

2. How is CentOS built?

CentOS is based on open source SRPMS provided by a prominent North American Enterprise Linux vendor, henceforth referred to as "upstream". You can get the CentOS versions of SourceRPMs (the thing used to build the RPMS) from http://mirror.centos.org/centos/5/.

Every repo has a SRPMS directory and every RPM has an SRPM. One SRPM may produce multiple RPMs. As CentOS removes certain trademarks of the upstream, new packages added to a series need to be examined, and a patch and specfile patch written; sometimes a change in a 'freshening' will require similar changes.

CentOS uses a mock buildroot environment (a separate chroot for each package), not unlike what is being done for Fedora or Scientific Linux. SL has been reported to use mock directly. Fedora uses mock with a program called koji CentOS uses mock via a program called plague All 3 solutions produce the same type of RPMS as rpmbuild is the underlying builder in each instance.

An example script to compare built RPMS to original RPMS and a script to produce ISOs (given a tree of RPMS) is available at: http://mirror.centos.org/centos/4/build/distro/

The buildsys files to use mock are found at http://dev.centos.org/centos/buildsys/

Each RPM is built with mock, examined using the compare script which hughesjr linked to in a mailing list post. If its comparison does not detect variances, then it is a good candidate package. If there is something different, then basic diagnostic skills are used to determine the cause of the variance.

Summarizing, there is NO MAGIC here. One takes a stack of SRPMS, builds them, compares them, and when they all compare correctly (and not before), you put them in a directory tree and run the ISO builder: buildinstall It is not hard, but it is time consuming.

If something does not compare correctly, developers have to troubleshoot to find out why. What does the upstream have present (or absent) in their buildroot to produce a package, or version of a package, that CentOS does not have? These differences in buildroot package content can cause the autotools to make differing decisions on how and what to build.

CentOS developers debug this by hand looking at the verify script output of libraries linked. In most cases, either upstream has something CentOS doesn't have or CentOS has something upstream doesn't have. The buildroot must be adjusted as necessary (based on the individual issue) and another rebuild attempted. It is not automated and is only time consuming trial and error.

It is not glamorous, nor automated. Build individual SRPMS, one at a time. Compare the binaries, rinse and repeat as necessary. QA, Lather, Rinse, and Repeat until the CentOS developers are satisfied with the product.

When all packages finally compare without exception, put all the right files in a directory and build the trees with buildinstall, then use mkisofs to make iso images. Then functional testing starts, and the QA group looks for unexpected variances here as well. Repeat yet again until the CentOS developers are satisfied with the final product.

It is all in the scripts hughesjr published ... with the exception of mock and when mock is properly configured on your build box machine, that is just a command like:

setarch $ARCH mock -r "<config_file_name>" --arch="$ARCH" <path_to_SRPM>

The mock config file is the one for this build you are doing, the $ARCH is i386, x86_64, i686, etc.

Mock is not required, it is just easier to get a "clean build" using it.

The last aspect of the ISO spin is the comps.xml file ... this file is in the ./repodir/ directory on the ISOs and CentOS trees for CentOS 5 and in the base directory on CentOS-4 and earlier. This file is what produces the Anaconda install groups. This file must be created from several different upstream files (by combining them together), as upstream has install numbers based on prices. We have removed the need for install numbers and entitlements since CentOS provides all RPMS available on all installs.

---

This page created by PhilSchaffner based on posts by JohnnyHughes (hughesjr) on the CentOS fora. Minor edits by RussHerrold. Wiki contributors with edit rights are invited to make updates and corrections.