In the Beginning There was ...

Hetrogenous Network
Problems this caused
Example: Apache/SSH Upgrades/RedHat
Requirements and upcoming Challenges

HEAnet is Ireland's National Education and Research Network, providing high quality Internet Services to over 130,000 students and staff in Irish Universities, Institutes of Technology and other educational and research organisations. HEAnet has an always maintained an amazingly talented staff and provides a high quality research network.

HEAnet also offers services to clients and the wider internet community. These include DNS, Mail, Website Hosting, Software Mirroring, USEnet and Mailing-List hosting.

Hetrogenous Network

When we started the re-organisation the Server Network mainly consisted of Elonex Tower PC's of varying models. We also had several SUN Sparc servers running Solaris, 2 DEC (*cough*, sorry Compaq, no I mean HP) Alpha Servers running Digital Unix 4 and Tru64 and several Cobalts used for network management. There were also two windows NT servers, one for Virus Checking, and one as an authentication server.

The core of the server network was the Elonex Towers, which were running various versions of the RedHat GNU/Linux Distribution, ranging from 6.0 to 7.2.

Systems were accessed on a pro-rata basis, with seperate system accounts held on each server by members of the Network Operations Center.

Problems this Caused

This set-up caused a number of problems. Allthough we had a very satisfactory relationship with our hardware support provider, there were ocasional hardware problems with the Elonex servers.

The hetrogenous nature of the installations meant that things were managed differently on each server. On some machines, SSH was an ssh.com binary, on others OpenSSH installed to /usr/local/, on others it was /usr/, on others it was installed from RPM. As each server was installed on an indivudual basis there was little consistency in the install methodolgy. Before things sounds too bad, it has to be noted that the servers were usable and did provide an excellent level of service.

Besides hardware, software was also broadly hetrogenous. We ran two web-servers within the network for example; Aolserver and Apache. There was also the problem of general overload of some of the Servers.

This diversity of the servers meant that it was hard to keep track of what utilities were available, what the correct procedures were and correct file locations on any given server.

Example: Apache/SSH Upgrades/RedHat

During Spring/Summer 2002 remote exploits were discovered for OpenSSH and Apache, both of which affected us on a large number of our hosts. As such, it was of course neccessary to upgrade the affected hosts. This process revealed itself to be far more complex than first thought, and both upgrades took a full 2 days of human effort. Of course we were never exploited, we made sure of that, but this should not take that much effort!

Generally the problems consisted of library conflicts, problems with LD_LIBRARY_PATH, networking issues and other issues. None of them on their own were utterly unresolvable, they just involved a large ammount of intervention and debugging on the part of the Admin. When repeated accross a large number of hosts, this process becomes rather involved.

Installing Apache involved tracking down previous configuration arguments (which in some cases, involved an educated guess based on the output of ldd, strace and behaviour of the binary) and associated modules (mod_php and mod_perl are both deployed here).

In the case of one RedHat server, the installation some DocBook/SGML tools was completely abandonded as the sheer volume of dependencies quickly became un-manageable. Using RedHat it proved impossible in terms of practicality to upgrade between versions. This was not desirable. Colm likes to call this: RPM meltdown syndrome.

Allthough our team was fully capable of managing these issues, the expenditure of time was inefficient. Although we never allowed issues to become service-affecting, and our level or service towards customers was top-notch, the management side of things was letting us down.

Requirements and upcoming Challenges

During summer 2002, at HEAnet, our focus began to shift slightly. As a body with a heavy emphasis on serving our clients as well as possible, a new impetus was created behind developing further services and expanding the base of our current services.

This Service Development phase was to involve a period of researching adequate hardware, followed by purchasing and set-up in order to offer new services. There would also be a Hiring-Round later, during which the size of the Network Operations Center was to undergo a large increase.

These factors combined to offer a significant additional incentive to consolidate System installation and management procedures, as well as better methods of access.