LinuxCNC Buildbot: Buildslave Admin's Guide

This document describes how to set up a Buildbot buildslave for LinuxCNC.

Running a buildslave requires only a computer that can check out and compile the linuxcnc git repo, can run twisted python, and can make outgoing TCP connections to the buildmaster computer. A buildslave machine can run comfortably on a 20 GB disk.

Disable the screensaver or set it to only blank the screen (so as not to soak up CPU cycles with useless fancy eye candy).
Make a user to run the buildslave. It is not a good idea to run the buildslave as yourself. I'll refer to the user as just "the buildslave user" in this document. You can name the user anything you want. Give this user a sneaky password that only you know.
Install the following packages:
- 'ntp' (so the time will be right)
- 'lsb-release' (so the buildslave can describe itself)
- 'fakeroot' (so the buildslave can build the debian package)
- 'build-essential' (because it's essential for building)
- 'tree' (used by some build steps to show what's going on)
- 'pbuilder' (used to install build dependencies)
- 'time' (provides detailed info about the time commands take to run)
- 'lintian' (used to validate built debian packages)
- 'aptitude' (used to install build-dependencies)
apt-get install ntp lsb-release fakeroot build-essential tree pbuilder time lintian aptitude
Do an anonymous Git checkout as the buildslave user (in a throw-away directory out of the way) to make sure sure you can, and to make sure you have the ssh hostkey of the Git server machine.
git clone https://github.com/LinuxCNC/linuxcnc.git linuxcnc-dev
Several of the build steps require password-less sudo for the buildslave user.
Add these lines to /etc/sudoers:
buildslave ALL = ALL, NOPASSWD: /usr/bin/apt-get, /usr/lib/pbuilder/pbuilder-satisfydepends, /usr/sbin/pbuilder, /bin/umount, /bin/dmesg -c, /usr/bin/make V=1 setuid
Defaults!/usr/sbin/pbuilder env_keep=DEB_BUILD_OPTIONS
At least on Hardy and Lucid, apt automatically updates the package list using a daily cronjob, /etc/cron.daily/apt. The buildbot also tries to update the package lists when it builds, and this can lead to conflict and spurious build failures. Thus you should disable automatic updating of the package lists, and let the buildbot handle it.
The cronjob is configured by a bunch of APT::Periodic apt-conf variables. A reasonable way to set these variables is to install the update-notifier-common package, then change /etc/apt/apt.conf.d/10periodic so APT::Periodic::Update-Package-Lists is "0".
Install an appropriate version of buildbot. 0.7.10 is the oldest version known to work. Versions before 0.7.10 don't work well with Git. 0.8 works, 0.9 and newer will not work until I upgrade the buildmaster.
The buildbot package in Ubuntu Lucid is new enough that it Just Works:
apt-get install buildbot
Ubuntu Hardy has buildbot 0.7.6, which is too old. If your buildslave is Hardy or older, you need to install buildbot from somewhere other than the default Ubuntu repository. I've had success with Michael Hudson-Doyle's PPA
Debian 10 "Buster" has buildbot 2.0.1, which is too new. The buildbot-slave package from Debian 9 "Stretch" installs and works fine, but make sure you mark it as "hold" so apt won't try to upgrade to the Buster version.
curl -O http://ftp.us.debian.org/debian/pool/main/b/buildbot-slave/buildbot-slave_0.8.12-1_all.deb

sudo apt install ./buildbot-slave_0.8.12-1_all.deb

sudo apt-mark hold buildbot-slave
On Debian Stretch and newer distros (but not on Debian Jessie and older, and not on Ubuntu Precise and older), you need to set a sysctl variable to allow the buildslave user to run dmesg.
Add a file named /etc/sysctl.d/allow-dmesg.conf, containing the line: kernel.dmesg_restrict=0
Reload sysctl, for example by running sudo systemctl restart systemd-sysctl, or by rebooting the machine.
Now we're ready to set up the buildslave itself. You will need two pieces of information from the buildmaster admin: the slave name and the slave password. Ask for this on the emc-devel mailing list.
The slave name is just a unique identifier for your buildslave. It should ideally be descriptive of the environment that you're building on, for example "hardy-rtai-x86".
The slave password is totally unrelated to the Unix login password of the buildslave user.
mkdir ~/BuildBot/slave/$SLAVE_NAME
buildslave create-slave --umask=022 ~/BuildBot/slave/$SLAVE_NAME buildbot.linuxcnc.org:5133 $SLAVE_NAME $SLAVE_PASSWORD
(The buildslave directory above is just a suggestion, you can put it anywhere you like.)
If you're using buildbot version 0.7.9 or older, you need to edit buildbot.tac to set "usepty = 0". 0.7.10 and newer do not need this change.
On Ubuntu releases from Dapper through Lucid, the buildbot package provides an init script that starts the buildslave automatically on system boot. This script sources /etc/default/buildbot, so you need to fill it out with the particulars of your setup.
On other distros, buildbot may be started differently.
Edit the info/admin and info/host files in your buildslave directory (~/BuildBot/slave/$SLAVE_NAME in the example above). admin should have your name and email address, and host should have a brief description of your host. For some examples (minus the email addresses), see the BuildSlave page.
Add the git release tag signing keys to the buildslave's gnupg keyring. (These have to be long-format key IDs because Jeff Epler's key, used for signing 2.4 release tags, has a collision. But anyway, long key IDs are just a good idea in general.)
- v2.4: 25DD353196935D7D
- v2.5: 445B1785BC92B87F
- v2.6: 26ADE41E54621DFA
- v2.7 (and later?): 0A30317D741499B0
gpg --keyserver keys.openpgp.org --recv-keys 25DD353196935D7D 445B1785BC92B87F 26ADE41E54621DFA 0A30317D741499B0
Note: Debian Stretch (and newer distros?) doesn't need this.

If you want your buildslave to run the real-time test suite, you need to allow the buildslave user to lock more memory than the default configuration allows. If you're starting buildbot from the init scripts at system boot, the easiest way to do this is to add this line to /etc/default/buildbot:
ulimit -l unlimited
This step is optional!
If you want your buildslave to build debs, you need to set up a pbuilder chroot:
1. Create the pbuilder chroot:
  1. Get the LinuxCNC debian package archive signing key. Because of widespread disruption of the PKI, the easiest way is to get it from the linuxcnc git archive (branch 2.9 or later).
    Export the key to a new keyring:
    - pre-Buster: gpg --homedir=${LINUXCNC_DEV}/gnupg --armor --export 3CB9FD148F374FEF > linuxcnc-keyring.gpg
    - Buster and newer (pbuilder 0.230.4): gpg --homedir=${LINUXCNC_DEV}/gnupg --export 3CB9FD148F374FEF > linuxcnc-keyring.gpg
  2. Alternatively, you can try to fetch the linuxcnc.org archive signing key from the PKI (keys.opengpg.org, keyserver.ubuntu.com):
    gpg --keyserver keys.openpgp.org --recv-key 3CB9FD148F374FEF
    Export the key to a new keyring:
    - pre-Buster: gpg --armor --export 3CB9FD148F374FEF > linuxcnc-keyring.gpg
    - Buster and newer (pbuilder 0.230.4): gpg --export 3CB9FD148F374FEF > linuxcnc-keyring.gpg
  3. On Raspbian only, override root's pbuilder config: cat > /root/.pbuilderrc
```
                                    DEBOOTSTRAPOPTS=(
                                        '--variant=buildd'
                                        '--keyring' '/etc/apt/trusted.gpg'
                                    )
                            
```
  4. Create the pbuilder tarball: sudo pbuilder --create --basetgz $HOME/pbuilder-$DIST-emc2.tgz --distribution $DIST --components 'main' --othermirror "$LINUXCNC_ARCHIVE" --keyring linuxcnc-keyring.gpg
  LINUXCNC_ARCHIVE is the LinuxCNC debian package archive for your $DIST. For example, for Lucid this is "deb http://www.linuxcnc.org lucid base" and for Hardy it's "deb http://www.linuxcnc.org hardy base"
  The pbuilder chroot tarball needs to be located in the buildslave's home directory, and needs to have the name specified above.
2. Give the buildslave user password-less sudo by adding the following line to /etc/sudoers (already done in the sudo step above):
  buildslave ALL = ALL, NOPASSWD: /usr/bin/apt-get, /usr/sbin/pbuilder, /bin/umount
3. Add the buildmaster's source repo to /etc/apt/sources.list.d/linuxcnc-buildbot.list. The exact form you use here depends on what distribution you're building for, look at the buildbot front page for specifics. For Lucid realtime you'd use this:
```
deb     http://buildbot.linuxcnc.org/ lucid master-rt
deb-src http://buildbot.linuxcnc.org/ lucid master-rt
deb     http://buildbot.linuxcnc.org/ lucid v2.5_branch-rt
deb-src http://buildbot.linuxcnc.org/ lucid v2.5_branch-rt
deb     http://buildbot.linuxcnc.org/ lucid scratch-rt
deb-src http://buildbot.linuxcnc.org/ lucid scratch-rt
```
4. Add the linuxcnc debian archive signing key to apt's keyring.
  - buildbot: EF1B07FEE0EE663E
  - linuxcnc.org: 3CB9FD148F374FEF
  sudo apt-key adv --keyserver keys.openpgp.org --recv-keys EF1B07FEE0EE663E 3CB9FD148F374FEF

Updating pbuilder on a buildslave

Run the same command as when creating the pbuilder tarball, but with instead of --create say --update and --override-config: sudo pbuilder --update --basetgz $HOME/pbuilder-$DIST-emc2.tgz --override-config --distribution $DIST --components 'main universe' --mirror http://archive.debian.org/debian --othermirror "$LINUXCNC_ARCHIVE" --keyring linuxcnc-keyring

If the debian archive signing key has expired

We support really old, End-Of-Lifed releases of Debian. When the archive signing key has expired the only way I know to get apt to run is to turn of key expiration checking:
sudo pbuilder --login --save-after-login --basetgz $HOME/pbuilder-$DIST-emc2.tgz
echo 'APT::Acquire::Check-Valid-Until "false";' >> /etc/apt/apt.conf.d/15pbuilder

That's it! I can't believe it was so easy!