Wash, Rinse, Repeat - Organised Repitition

HEAnet-Debian
apt.heanet.ie
Meta-Packaging
Repeatable Install Methodology
Example: name-server
Example: Cork pop-pc

If you have to do a task more than once, like installing a system, but true, zero intervention, automation (which we'll discuss later) is not desirable (like for example when servers are intended for different tasks) the next best thing is consistent repeatability.

This does not just mean the ability to repeat what you've done before, this means being able to get known hardware into a known-state. This encompasses everything from access-control to the packages that are installed.

If you install your servers to a known (useful!) state, and you can do that repeatably, then you significantly reduce your overhead the next time you have to do it.

HEAnet-Debian

When you buy brand-new hardware, you can't always be guaranteed that your Operating System will support all of it. So you usually have to install the neccessary drivers. This introduces time and complexity into your install phase, why should you have to build and load drivers from a floppy just so you can get your network or RAID controller working?

When we installed our first Dell Poweredge 2650, we (or rather, a co-worker Gráinne Sheerin) documented it thouroughly. We even confirmed it's repeatability by asking another co-worker to perform the install again from the documentation. This worked, but it was a long and involved install procedure.

This arose because the Debian install CD didnt have the neccessary Linux modules needed to make network cards and SCSI controllers in Dell Poweredge 2650 and 1650's work. So after some research, we decided to make our own.

We constructed our own Debian install CD, titled HEAnet-Debian, which worked flawlessly on Dell Poweredge Hardware. Through the linux-poweredge list, the CD image now has a regular user-base and we still get correspondence from admins who use it.

This single step knocked pages off of our documentation, and over an hour off of our installation time.

apt.heanet.ie

Allthough Debian has well over 10,000 Packages it's not always going to have everything you need. There was software, sometimes highly esoteric or specialist, sometimes written by ourselves which we found useful.

Rather than depend upon the unreliable and inconsistent process of building from source on individual servers we decided to implement a central software repository, as recommended by Limoncelli and Hogan, in the Good Book.

Debian uses dpkg and apt-get for package management, these tools make package management substantially easier than any other alternative we've found. We decided to make an aptable respository of software.

This gave us the advantage of being able to install our custom packages directly using the existing Operating System Procedures. Initially we packaged third-party software. The Eric Wassenaar version of "host", and the RIPE version of whois and the irr-toolset, all of which are in common use with ISPs.

We then moved on to packaging Apticron, rcs-report and libpam-auth-tacplus, all of which we'll discuss later. We also packaged our custom kernels and modules aswell as c-hey, an inter-terminal messaging tool we like.

By utilising this methodolgy it was possible for us to apt-get install any of our packaged software, on any of our servers. We no longer had to worry abound conflicting file locations, and upgrades could be automated (more on that later).

Meta-Packaging

If never encountered Debian's apt-get you've been missing out on a lot. apt-get is a tool for installing packages but it's not like rpm, other packaging formats or evev like BSD ports. Unlike rpm you are not going to waste hours trying to figure out the latest comination of mad -q arguments just to figure out an order you can install things without anything breaking.

apt-get is a managed, rational way to handle dependencies. It is a powerful tool that assumes you know what you want, you can apt-get install a package and it will install all of the neccessary dependencies and resolve all of the conflicts for you. This is a great tool than can take the pain out of package management, but as with any powerful tool, you can blow your foot off if you don't wield it correctly. Fortunately apt comes with a -s argument which simulates what would have happened so you can make sure nothing bad is going to happen.

We made use of apt-get's power by creating meta-packages. Meta-packages are packages which don't actually contain anything, they just depend on others. Using this ability, we could create toolkit packages that depended on packages we like. By installing this one package, we could install dozens or hundreds of other packages we like, consistently and repeatably.

We created a number of such packages, starting with heanet-std-toolkit, this is the package we install on to every machine. It contains all of the tools we like to have available everywhere, simple things like ssh, less, vim, wget and debugging tools likes lsof, nmap and strace. We also created meta-packages which contained the neccessary tools to build a kernel, create documentation and other common tasks.

Repeatable Install Methodology

By utilising the HEAnet-Debian CD, apt.heanet.ie and meta-packaging our install methodology became repeatable. Unlike the previous ad-hoc methodology we now had an easy and consistent way to install our servers into a known state before any customisation and configuration for a specific task.

Example: name-server

A good example to cite is the installation of our Primary and Secondary Name-servers. We take our DNS very seriously, HEAnet's Senior Technical Officer is a former hostmaster for IE and HEAnet has a long and proud history of serving DNS well. The IPv4 address of our Primary Name-server hasn't changed in over 10 years, we provide secondary NS for literally hundreds of zones aswell as offering the entire MAPS RBL+ zones for our clients.

As part of the rollout and testing of the HEAnet RBL+ service it was neccessary to upgrade the servers. Drawing on our policy, we decided to implement it on two Dell 1650 Poweredge servers with all of the usual trimmings: Dual system disks, dual PSU, dual network.

When the hardware arrived, due to our investment in repeatability, we were in a position to install it immediately. The machines literally went from cardboard box to production servers in a rack in less than 1 hour.

The install can be summarised as:

Opening the box
Rack-mounting the Server
Inserting the HEAnet-Debian CD
Configuring network details
Configuring RAID
Installing the standard packages using our meta-package
Installing and Configuring bind 9 (including chroot and high-uid)

The most significant factor of delay in the process is choosing a hostname.

Example: Cork pop-pc

Our POP's all had a workstation for local support tasks and these were running Redhat 6.0 with telnetd. There was only one staff member with login privledges on these hosts. These hosts had never been update to deal with security issues.

So bringing these hosts into the family so to speak posed a challenge. It was going to involve either a road trip to four seperate sites or talking remote operators thorugh a process of reinstalling the machines which while doable would have been stressfull.

Given that by now we had developed a good deal of knowledge about our choosen platforms install methodology it was possible to reinstall these systems without phyiscal access or onsite intervention. The Cork POP pc was also one of the first systems to support a serial console.

Basiclly we built a new root filesystem on a seperate portion of the harddrive, dropped to a chroot within this and utilised all of our install tools to bring the system to a known state. At one stage it was possible to ssh into the debian install on this host and never notice that it was running ontop of redhat unless you ran uname -a and spotted that it was a Redhat kernel.

While this example is a touch over the top it is an example of how some of the methods allow you to acomplish taks heretofore impractical.