Change Control
Change Control is the process by which you go about making changes to systems that are in production. It is the thing that will allow you to always know who did what and why and if neccessary undo things. Much has been written on this subject we will only hit the highlights of things that should be considered.
CVS/RCS
CVS & RCS provide convenient methods for recording changes to config files and other text based files. RCS works locally on the host whereas CVS can be used across the network. We have employed them to manage changes to DNS zonefiles and router configs primarily. CVS is also used to support various software development projects within the admin team.
Both CVS and RCS require you to check-out files in order to make changes, and then check-in when finished, documenting your changes. In fact this document you are reading was managed using CVS.
Reporting
apt-listchanges, a tool which is installed on all of our systems which e-mails the relevant sections of the changelogs to root on the local machine of any packages updated. This means that by reviewing the local mailbox an admin may asertain what changes & upgrades have been made.
RCS-report is a tool orignally developed with the HEAnet community, by DCU's Cillian Sharkey, that we have packaged for debian. It mails a summary of any changes made to a file under RCS managment to a designated email address. In our case the Network Operations list.
CVS commit emails, It is possible to have cvs e-mail a message to a specific address any time a change is committed to a repository. This can be setup globally or repository specific.
Sample Output
From: root <heanet@NS.HEANET.IE>
Subject: RCS Report for shankill
RCS Update report for shankill
==================================================================
Some changes have been made to RCS files on shankill in the
past 24 hours. There follows a summary.
=================================================================
RCS Report
==========
/chroot/bind/var/cache/bind/pz/193.1.199:
CHANGED - 1 revision since 1.1
=====> 1.2 root Tue Aug 13 15:05:22 2002 <=====
updated record for pop workstation in city west
forward and reverse is kasei.hea.net
pc.citywest.hea.net is a CNAME to above
-----> diff to 1.1 <-----
--- /chroot/bind/var/cache/bind/pz/193.1.199 2002/08/08 11:20:51 1.1
+++ /chroot/bind/var/cache/bind/pz/193.1.199 2002/08/13 14:05:22 1.2
@@ -2,7 +2,7 @@
; Data file for reverse address to hostname.
;
@ IN SOA killiney.hea.ie. hostmaster.hea.ie. (
- 2002080100 ; Serial
+ 2002081300 ; Serial
86400 ; Refresh - 24 hours
7200 ; Retry - 2 hours
3600000 ; Expire - 1000 hours
@@ -53,6 +53,7 @@
196 IN PTR Dactyl.hea.net.
197 IN PTR Olympus-f0-0-ipv4.citywest.core.hea.net.
199 IN PTR Alexander.hea.net.
+200 IN PTR kasei.hea.net.
206 IN PTR Citywest.webcam.hea.net.
;207 broadcast address in citywest
;208 network address
/chroot/bind/var/cache/bind/pz/hea.net:
CHANGED - 2 revisions since 1.2
=====> 1.4 heanet Tue Aug 13 15:48:34 2002 <=====
added kasei.hea.net, pc.citywest points at this
-----> diff to 1.3 <-----
--- /chroot/bind/var/cache/bind/pz/hea.net 2002/08/13 10:40:55 1.3
+++ /chroot/bind/var/cache/bind/pz/hea.net 2002/08/13 14:48:34 1.4
@@ -4,7 +4,7 @@
@ IN SOA ns.hea.ie. hostmaster.heanet.ie. (
- 2002081300 ; Serial
+ 2002081302 ; Serial
86400 ; Refresh - 12 hours
7200 ; Retry - 1 hours
3600000 ; Expire - 4 days
@@ -76,7 +76,8 @@
dublin.netflow IN CNAME chomper
circus IN A 193.1.219.222 ; Wireless LAN hub
Citywest.webcam IN A 193.1.199.206 ; axis webcam
-pc.citywest IN A 193.1.199.193
+pc.citywest IN CNAME kasei
+kasei IN A 193.1.199.200
cmod.link IN A 193.1.194.102
colms-2500 IN A 193.1.219.149
Cork-2924 IN A 193.1.199.70 ; Ethernet switch Cisco 2924
=====> 1.3 heanet Tue Aug 13 11:40:55 2002 <=====
added entry for citywest PC
-----> diff to 1.2 <-----
--- /chroot/bind/var/cache/bind/pz/hea.net 2002/08/09 13:49:11 1.2
+++ /chroot/bind/var/cache/bind/pz/hea.net 2002/08/13 10:40:55 1.3
@@ -4,7 +4,7 @@
@ IN SOA ns.hea.ie. hostmaster.heanet.ie. (
- 2002080900 ; Serial
+ 2002081300 ; Serial
86400 ; Refresh - 12 hours
7200 ; Retry - 1 hours
3600000 ; Expire - 4 days
@@ -76,6 +76,7 @@
dublin.netflow IN CNAME chomper
circus IN A 193.1.219.222 ; Wireless LAN hub
Citywest.webcam IN A 193.1.199.206 ; axis webcam
+pc.citywest IN A 193.1.199.193
cmod.link IN A 193.1.194.102
colms-2500 IN A 193.1.219.149
Cork-2924 IN A 193.1.199.70 ; Ethernet switch Cisco 2924
==================================================================
rcs_report running on shankill
Maintainence windows, at-risk periods and quiet-time
Maintainence windows and at-risk periods are the times when you have allocated for making changes to systems that may have an impact on users. This is neccessary because some tasks are going to require that machines be rebooted or software updated and tested which leads to service disruption. Such windows should be frequent enough to ensure that any tasks which have to be done can be done and still have time to back them out in case of problems.
Always give yourself more time than you will need. It is better to finish the task with lots of time to spare than to run over. In the first case people will think you are great when you finish early. Running over the time window you have reserved makes you look bad.
Well-designed hardware and software can minimise the requirement for such interuptions but unless you are prepared to commit large amounts of resources then it can not be eliminated.
Quiet-time is the opposite, it is a period during which no changes which might possibly affect service is performed unless absolutly neccessary. Such example might include end of financal quater or year-end. In our case a good example is CAO offers season as Third level institutions finalise offer details with the CAO.
Thank Crunchie
Never make changes on a Friday. O.K. maybe not never but only when you have to. Not changing things on a Friday means less risk of things breaking over the weekend and your time off being ruined.
© 2003 Colm MacCárthaigh & Colin Whittaker
