Administrative Access
Managing Scale
As the number of servers and staff grew we faced a fundamental problem in scaling access to the systems. The system had been that each user was given a login account on each server but as the numbers grew this became harder and harder to manage. We had to ensure that new hires got access to all sytems, remember our password on dozens of systems and finally remove all of a users accounts when the left the company. The latter was prone to problems with various production processes such as reporting cron jobs being run out of users accounts.
Root passwords were stored in a single password book but soon contention and ability to work from remote sites became an issue. Solutions such as users maintaining private databases of root passwords using pwman or gpasman for example were not easy to maintain and posed security risks.
Some systems were even discovered to be running known vunerable telnet daemons and no ssh daemon. The mandating of SSH as a remote access protocol is a core part of the policy that was developed.
Role Account, SSH keys
The existing system had required maintaining m * n accounts where m is the number of staff and n is the number of servers. With the number of sysadmin growing into double digits and the number of servers well on its way to fifty this was a recipe for chaos. Now many systems in out of the way corners of the network tended to only have accounts for a single member of staff, generally the person who installed the system. This as one can imagine posed significant challenges in dealing with problems if that person was not available.
The solution we implemented was to create a role account on each system that would have a copy of each admins ssh public keys. These keys would be stored in a central repository and each machine would update it's copy once a day. Admins also got shell accounts on two systems, one was the email server and the other the cvs and network management system. Users would have needed accounts on these systems in any case so it was the logical choice.
The switch to using ssh keys also provides an incentive to users to setup ssh-agent on their desktops to allow secure passwordless access to all of the servers. The migration to this methodology was accomplished as existing systems were upgraded or as part of our installation process. As a result we were reduced to managing only m*2+n accounts, which given the growth of the network provided a major saving.
SecurID
To solve the challenges of providing root access to servers we utilised the existing secure token authentication system which is based on RSA SecurID tokens and Cisco Secure Server. This system was used to provide secure login services to the routers in the network. A custom PAM module was developed which implements the TACACS+ protocol used by Cisco. This is based on a username / PIN / random number combination, the random number changing every sixty seconds. Each combination can only be used once to prevent replay attacks. This module also set the LOGNAME enviroment variable which is used by various tools for logging purposes which allows tracking of changes back to specific users. In the event of network failure the module falls back to using the local root password configured at install time.
Serial Console
Being an ISP we tend to have servers located in farflung corners of the network and a backup access method is always needed in case of any problems access the host via the network. How many admins have accidently downed the interface they are logged in over or had a system become unresponsive ? Access to a console via the serial port is a lifesaver in these situations and prevent having to have a set of remote hands and eyes reboot the system.
One benefit of choosing server grade hardware is that the BIOS will generally have features like supporting headless booting and redirecting the BIOS console onto the serial port which means you do not need to have a keyboard or monitor plugged into the system. The Dell BIOS even supports features like forceably rebooting the system which were once the preserve of commerical unix vendors.
Demonstration!
This is the part where you need to have been at one of our talks!
© 2003 Colm MacCárthaigh & Colin Whittaker
