Administrating the CIIPS Network - the easy way

Some suggestions to make it easier to maintain and improve the CIIPS network.

To achieve this, I believe the following must be done:

Keep things consistent

Make sure knowledge is generic and not related to the different computers. Make sure every supported program is available on all platforms and operating systems, with the same version and the same configuration. (cfengine, Store)

Automate as much as possible.

By leaving the repetitive tasks to the computer, one assures they are done the same way everywhere, every time. (cfengine)

Make it possible to back out changes.

When problems arise, make sure it is easy to get back to a earlier state when things where working. Use version control systems to keep track of all human-edited configuration files, and make sure it is easy to remove a newly upgraded software package if problems are discovered. (CVS, Store)

Detect problems as early as possible.

If potential problems are detected and fixed before they become problems, less work is required to fix the resulting domino-effect when one system fail. (Example: a full disk can stop the backup system from working properly, or a failing NFS server can hang processes on other servers and eventually bring other servers to their knees).
Make sure to monitor all hosts and services, and warn when something is about to go wrong. Keep statistics to find repeating problems. (Palantir, mon)

Keep the users informed

Make sure the users know where to find information on current problems, and the status of their reported problems. This is easiest solved using a database and web based problem tracking system. (Bugzilla)

Prepare for disaster

Some suggestions:

• Hard-disks fails after a few years. Keep only new HDs in the servers, and reuse the old server HDs in workstations until they fail.
• Back up everything, and back up often. Make it easy to get files back from backup, and let individual users get their files back on their own. It saves support personnel the hassle, and make it easier for the users to recover from their personal disasters.
• Make it easy to replace workstations. A HD mirroring system to make it trivial to reinstall a workstation and easy to replace a broken computer. (Autosetup)

Tools

• cfengine
• Store
• CVS
• palantir
• mon
• Bugzilla
• Autosetup is not publicly available, but I can get it from the author. More expensive alternatives are also available. jacal might be an alternative.