newsNews: Hotspot #2: the project Test-Autobuild

Show feedback again
Latest News
Gna! closing soon posted by beuc, Tue Jan 31 22:31:06 2017 - 5 replies
mail subsystem upgraded posted by beuc, Sat May 23 18:17:18 2015 - 1 reply
dl/home/cvs/svn/ upgraded to debian8 posted by beuc, Thu May 21 22:12:23 2015 - 2 replies SSL certificate renewed posted by zerodeux, Thu Apr 24 07:13:06 2014 - 4 replies
Heartbleed status and measures posted by beuc, Wed Apr 9 09:47:29 2014 - 2 replies
[125 news in archive]

Hotspot #2: the project Test-Autobuild

Item posted by Mathieu Roy <yeupou> on Fri Nov 5 19:13:10 2004.

On a regular basis, Gna! people pick and interview one vivid and mature project to highlight the software and personality that drive Libre Software community, similarly to project of the month -- yes, we assume the fact that we grab good ideas from others places.

The second hotspot is dedicated to the Test-Autobuild project.

Test-Autobuild Description:
Test-Autobuild check out pristine sources of a software from a version control repository, invoked a script to build the software and install
it in a virtual root. Snapshots of the virtual root are taken before
and after build to identify which files were installed. Snapshots of
any designated 'package' directories are also taken to identify any
RPM, Debian PKG, Tar, ZIP files which were built. Finally a set of
output modules are run to generate HTML status page, copy packages
and build logs to a Web / FTP server, send email notifications of
build status, create ISO images.

Test-Autobuild was registered on Gna! on March 2004 and is leaded by Daniel Berrange and Dennis Gregorovic.

An Interview with Daniel Berrange and Dennis Gregorovic:

| Gna!: Hello,
| Can you give a two line description of Test-Autobuild your grandma(s) could
| understand? What is the intended audience of Test-Autobuild?

Dan: Test-AutoBuild is a tool for automatically running build and test
processes, without need for user interaction. It checks code out of
version control, invokes the build & test processes on each project
and then creates status pages, send email alerts and uploads build
logs & packages to FTP/HTTP download sites.

The target audience for use of Test-AutoBuild is any project with more
than one developer working on the codebase, so pretty much any
non-trivial project! The value it brings to the development process
is to increase awareness of the state of the codebase. At any given
time you should be able to know with confidence that a code tree is
compiling and passing all available test cases. When a developer
checks in a piece of code that breaks the build, there is a direct
impact on the productivity of other team members. At the simplest
level they may be unable to compile their code after pulling down
updates from version control. More seriously, unexpected changes in
the runtime behavior of the program can result in many hours wasted
debugging time.

By automatically running the build processes continuously (24x7),
problems are detected and dealt with as soon as possible, minimizing
the disruption for developers. Even greater benefit is seen when unit
test cases are incorporated in the build cycle, and parallel
Test-AutoBuild instances are setup across different platforms
(GNU/Linux, Solaris, etc), since its not feasible to expect developers
to test their code submissions across all supported configurations.

Dennis: Test-AutoBuild can even be useful for a project with just one
developer. As Dan points out, with multiple Test-AutoBuild instances,
the developer can avoid the laborious task of building and testing the
project code across different platforms and configurations.
Additionally, with Test-AutoBuild you can set up "clean-room" where the
builds and test runs will performed consistently in a controlled
environment. This is often more reliable than using the same
environment where you do your development.

| Gna!: Do you propose/maintain a collection of recipes (in the Gentoo
| or uOS style) to "Test-Autobuild" some popular packages?

Dan: When creating the initial content for our GNA webpages (now
aliased from, I started on a tutorial illustrating
how to create control files for use with common build tools. So far
I've illustrated use of AutoMake/AutoConf in the C/C++ world,
ExtUtils::MakeMaker in the Perl world, and ANT in the Java world. Java
is in an interesting situation because while ANT is the de facto
standard low level build tool, there is yet to arrive any clear
favourite higher level language similar to AutoMake. Although Maven is
starting to become more widespread, there are a lot of projects which
don't want to be constrained by its particular view of how Java should
be built. Thus I think there is still a gap in the Java toolspace for
a flexible AutoMake style layer around ANT.

| Gna!: What is the difference with other popular unattended build packages,
| like eg. Tinderbox?

Dan: Tinderbox has a quite a different design to Test-AutoBuild, with
a client-server architecture. The client component performs the build
of a module and sends the results across to the server when complete.
Multiple clients can be setup to build the software across many
platforms passing results to a single server. As such its probably
more comparable to a combination of Test-AutoBuild with the Bicycle
aggregation service. The client component, however, does not have the
same level of functionality as Test-AutoBuild, with no direct support
for caching builds state of multiple related projects. My preference
is for separate pieces of infrastructure each performing a simple,
well defined task, since this makes it easier to integrate with
existing / legacy software without having to replace an entire system.

The most frequently quoted competitor to Test-AutoBuild is
CruiseControl. At an implementation level, the latter is written in
Java, while Test-AutoBuild is Perl - I'll leave it up to readers to
decide which is preferable ;-) Much of the build functionality is
very similar, both being able to checkout from a variety of source
control repositories, both can send email alerts, generate status
pages, make build logs available, etc. The differentiating factor is
that Test-AutoBuild has direct support for managing multiple projects
in a single build instance, while CruiseControl pushes this knowledge
down to the underlying build tool such as ANT. The advantage for
Test-AutoBuild here is that with this knowledge it can do intelligent
build caching of individual modules which haven't changed, provide
incremental alerts and status pages and grouped package downloads.

Other competitors include LuntBuild ( and
AntHill or their proprietary version AntHill pro. I consider AntHill
Pro to be the biggest competitor in terms of functionality, but then
it is closed source and with the work we're currently doing the gap
will be much less significant. More detaileds comparisons against our
competitors are being gradually put up on ourwebsite.

| Gna!: Aren't you overlapping some distro packaging tools
| functionality (Eg. RedHat and Debian also use procedures to package
| from "pristine sources" and make automation easy; including chroots,
| installed files discovery, autodeps, etc)?

Dan: I consider Test-AutoBuild to be an important value-add on top of
the packaging tools used by the distributions such as Red Hat and
Debian. When building an RPM of their software, a developer still has
the manual task of checking the code out of version control, running
whatever scripts are need to create the source tar.gz distribution,
and then invoking rpmbuild. When building multiple related RPMs, it
quickly becomes a large burden on the developer to repeat these steps
over and over, not to mention the care required to ensure you are
building against the correct pre-requisite RPMs. When building RPMs
for an entire GNU/Linux distribution comprising many hundreds, or even
thousands of RPMs, automation on-top of the basic packaging tools is
absolutely essential. This is where a tool such as Test-AutoBuild
comes into its own. During day-to-day work, the developer only has to
concern themselves with writing code & checking it into version
control. A short while later, after the next build cycle has
completed, binary packages of their software will be uploaded to the
build status pages for anyone to access.

A fundamental rule in writing Test-AutoBuild is that we should not
make any assumptions about how a project's development process may
operate. So we have to be operate with any combination of programming
language, build tool, packaging format, source control repository
within a single Test-AutoBuild instance. So while RPM provides
functionality to detect installed files, we have to cater for projects
who do not use RPM or an equivalent. This is a particularly common
situation in Java projects most of which simply distribute JARs, or
development undertaken by companies for internal use only, for which a
custom deployment solution has been written. That said, one of the
ideas for enhancing Test-AutoBuild still further is to enable
automatic extraction build metadata when already defined in the
packaging tool files. For example, rather than have the administrator
list build dependencies in the Test-AutoBuild configuration file, it
could be determined from 'BuildRequires' in the RPM spec file.

| Gna!: Who are you? How many developers contribute to Test-Autobuild
| regularly?
| How the project is leaded?

Dan: I'm currently working for Red Hat, in London, where I'm a member
of the Global Professional Services team. Prior to that I was in their
Content Management team (CCM), as the technical lead on the APLAWS+
project. Naturally during the course of my work, I've introduced use
of Test-AutoBuild to nearly all the development teams I've been
involved with, since very few have any build automation processes.

At this point in time nearly all development is undertaken by Dennis
and myself. One of our prime motivations for setting up a public
presence on the GNA service was to solicit contributions and feedback
from developers in the Open Source community, as well as spread use of
Test-AutoBuild beyond Red Hat and our own projects. The leadership of
project development is very informal, we try and start off important
design changes with discussion via email (& to a less extent on
IRC). Dennis is doing much of the big work on the development branch,
while I'm doing low risk changes to the stable branch, along with
maintaining the website.

Dennis: I'm in Boston, also working for Red Hat, doing release
management for Red Hat Enterprise Linux. Previously, I was doing
release and build management in the CCM engineering group. I'm a
developer at heart and focus most of my energies on tools that are
SCM-related. I really enjoy doing Test-AutoBuild development because
as it gets better, my life gets easier.

| Gna!: When was started the project and why?

The first version, which was a single Perl script, was written by
Richard Jones ( back in the year 2000 while we were
both working at BiblioTech Ltd ( We had a large
team of up to 18 developers working on numerous projects for our online
community sites, so build and test automation was key to ensuring code
quality was maintained. Richard setup the control files to build the
project, run unit tests, and package it all in RPMs. On the
development servers, Rich had setup a related script that would
periodically grab the latest RPMs from the automated build website &
install them - a task typically performed by APT/YUM these days. So
not only was it providing us with a build, test and packaging system,
it enabled integration of related tools to lower administrative
overhead of keeping all the servers uptodate with the latest RPMs.

| Gna!: How things have gone for the project since then?
| What are your plans? How do you see the future of Test-Autobuild?

Dan: Test-AutoBuild has gone through several iterations since
Richard's original Perl script. When I left BiblioTech to work at
ArsDigita, I started the process of gradually refactoring the Perl
script to allow more general usage. The original version only dealt
with CVS and had a fixed set of tasks it ran at the end of each build
cycle. So I introduced module APIs for obtaining code checkouts from
source control repositories, and for performing post-build output
tasks, such as sending email alerts, generating HTML status pages,
copying packages and build logs to a web directory, etc. I introduced
use of this refactored version to the project I was working on at the
time & after a while our Engineering team decided it would be useful
for their needs too. In the meantime we were acquired by Red Hat, who
were more than happy for us to develop AutoBuild during course of our
day-to-day work. After working on Test-AutoBuild on & off for a couple
of years in Red Hat, we decided it was time to try and engage people
in community, hence our launch on GNA earlier this year.

The current work that's happening on the development branch of
Test-AutoBuild will be a very significant milestone, providing a
completely customizable build work-flow engine. The current stable
release will always perform the build cycle in three stages, first
checkout all the code from source control, then build each project,
then run the output modules to generate HTML pages, send alerts,
etc. With the new customizable work-flow engine it'll be possible to
customize this process in any way whatsoever. One example benefit is
that it will be possible to generate incremental HTML pages after each
project has been built, rather than only at the end of the complete
cycle. When building a large number of projects, this will reduce the
time between code check-in and build feedback still further. It will
also be possible synchronize multiple Test-AutoBuild instances, for
example, one would perform the basic build and unit tests. The second
instance, instead of checking out code from version control, could
download RPMs generated by the first RPM & perform integration tests
on them. This is taking Test-AutoBuild into the realm of being a
general purpose development process automation framework, rather than
being targeted puring to the build stage.

Another very interesting development is the work Red Hat are doing on
the Bicycle project. This is provides a aggregation service for
continuous integration tools such as Test-AutoBuild, CruiseControl and
BugZilla. Amongst other things it will display a historical view of
the build status. It will also be a valuable add-on for situations in
which multiple Test-AutoBuild instances are run against the same code,
but across different hardware architectures and OS platforms.

Dennis: Over the last few years we have seen an impressive growth in
the functionality, maturity, and number of open source version control
systems, such as Subversion and GNU Arch. I expect that in the next
few years we will see a similar growth in the area of open source
process automation tools, and Test-AutoBuild will be right there among
them. The items that Dan mentions for the future of Test-AutoBuild
are exiting, but I think they may just be the tip of the iceberg.

| Gna!: Which license did you choose for Test-Autobuild and why?

Dan: The original version of Richard's autobuild script was released
under the GNU GPL. Since I chose to incrementally refactor this
script, rather than re-write from scratch, the decision to cover
Test-AutoBuild under GNU GPL was made for me.

| Gna!: Can you mention some real world use cases of Test-Autobuild?

Dan: I have an instance of Test-AutoBuild setup to perform nightly
builds of all my random, open source projects, so anyone who may come
across one can always get the latest code I have. It is also a useful
sanity check in case I forget to check one or more files into CVS,
since I'll have an email alert reminding me the next morning.

Within the CCM group at Red Hat, our engineering team had an instance
of Test-AutoBuild building all the applications they maintained. Any
member of the services team, which is spread out across London, Munich
and Boston could easily obtain builds of the latest development branch
and all supported stable branches, without need to direct
communication with the engineers. We also have a nightly process
which pushed the latest builds out to the public
server. Dennis has also been introducing use of Test-AutoBuild into
some of the development groups within Red Hat. I'm also in process of
setting up a continuous builder for a project I'm involved in which
has to build, test and integrate a mixture of Java, Perl and C++

Dennis: In addition to the Test-AutoBuild instances in the CCM group
that Dan mentions, we also had an instance for building documentation
and another for release builds. I point this out because it shows the
versatility of Test-AutoBuild. In the documentation instance, we
produced HTML, PDF, and Postscript output from SGML source; the HTML
output was packaged up, but it was also copied to a directory served
by Apache. This way, the latest CCM documentation was always
available online. The release build instance of Test-AutoBuild was on
a separate machine with limited access to ensure a consistent build
environment. This instance was also configured accordingly. For
example, it never used cached builds, and it would pull from Perforce
using a specific tag or changelist number, instead of just grabbing
the latest code.

| Gna!: Do you have any industrial or institutional support?
| If any, how so?

Dan: Both myself and Dennis are fortunate enough to be employed at Red
Hat who are more than happy for us to further develop Test-AutoBuild
and related tools such as Bicycle. Situations arising within Red Hat
have also provided motivation for developing new capabilities in the
Test-AutoBuild framework, adding support for Perforce source control
system, caching of builds to speed up cycles when no changes have been
made in source control.

| Gna!: Are you looking for contributions?
| If so, what kind of contributions could be of use to the project?

Dan: Our prime motivation for making Test-AutoBuild available on the
GNA project service was to try and encourage involvement of other open
source developers. We've got a fair few ideas floating around that we
so far haven't had time to work on, so we're open to offers from
anyone interested in writing some Perl. One significant development
task I'd like to see some work on at some point is a web based UI for
configuring and managing the builder instance. This would lower the
learning curve for users trying to setup a build instance for their
projects, as well as addressing the most significant feature gap to
the proprietary AntHill Pro tool. Such a configuration tool, wouldn't
even necessarily need to be written in Perl, or directly integrated
in the main codebase, rather it could evolve independently as a layer
on-top of the basic Test-AutoBuild code.

Dennis: The feature that I most look forward to is coordination
between separate Test-AutoBuild instances. For example, you could
have one instance just building and packaging the software, and other
instances grabbing those packages and testing them. Also, with the
archiving and generic work-flow work being done on the development
branch, the web UI is going to need improvements.

So, to answer your question, yes, we're looking for contributions and
help. :)

| Gna!: What tools do you use when working on Test-Autobuild? Why?

Dan: It goes without saying that we have an instance of Test-AutoBuild
building itself ( Personally I don't find a
need for fancy integrated development environments such as Eclipse,
so do nearly all my work in Emacs, or very rarely Vi.

Dennis: Emacs, and occasionally the Perl profiler and debugger

| Gna!: Why did you choose Gna! as host?

Dan: I took a look at most of the open source hosting services available, and
really liked the speed of the GNA website, and the clarity of its layout.
I also think its important not to concentrate too many projects with a
single service, since its creates a large single point of failure. That
pretty much ruled out SourceForge. At the time I was looking the GNU
Savannah site was also still not accepting new project registrations.

| Gna!: Anything else?

Dan: Something I observe time and again when getting involved in new
projects is the lack of good quality development infrastructure.
Build and test automation, just one small part of the solution, is
typically missing altogether. The need for a good version control
system goes without saying & the open source community is spoiled for
choice with the de facto standard of CVS, and now the advanced
capabilities offered by Subversion and GNU Arch. In contrast in the
closed source world, companies are still paying large sums of money
for vastly inferior & unreliable tools, which are killing productivity
of their developers. The tools used for performing formal builds also
play an important role. While many developers will find an IDE great
for day-to-day development work, they should not be used at the
expense of a solid build tool like Make or ANT. There is no way an IDE
can match these when it comes to providing 100% reliable and
repeatable release builds.

| Gna!: Are there links of interest that should come along this

Our website is the first port of call, Since we're
written in Perl, the formal releases are all uploaded to CPAN, which
generates HTML API documentation, so for interested developers this is
worth a visit (
Also worth keeping an eye on in the future is Bicycle, which although
still in an alpha phase of development, is very functional.

Dennis: Dan mentioned the Bicycle project several times. It also came
out of the CCM group at Red Hat at can be found at the Gna! site.
It's a bit young, but well designed and full of potential.

Finally, for those who like to communicate over IRC, I can usually be
found in #autobuild on the Freenode network.



Screenshots and more (posted by Loic Dachary, Wed Nov 10 10:06:09 2004)

Are there screenshots of your software in action ?

More seriously, I'd be extremely interested to run it against poker3d ( The first thing I need to give it a try is a pre-packaged version for Debian GNU/Linux.
I can't find this at ?

The reason for this is that most software providing this kind of service almost work. They work most of the time but they will occasionally fail and when this happens they use most (if not more) of the time they pretended to buy you. An example of this misbehaviour is apt-proxy. An example of software without this flaw is ccache.

Therefore, when I find an attractive software like yours, I need to give it a try in the simplest useful case, let it run for a few weeks and see if it works flawlessly. If it does not I need to discard it and give it a try six month later in the hope it matured a bit. The idea is that if an infrastructure software is not able to work flawlessly in the simplest useful case, there is very little chance that it works better in a more complex case.

I hope you understand the rationale of my request. Thanks a lot for undertanking this project and let me (us ;-) know when a straitforward package is ready to demonstrate it.

[ Reply ]



Start a New Thread:

You could post if you were logged in
Show feedback again

Back to the top

Powered by Savane 3.1-cleanup