9. Various & Sundry Administrative Tasks

Linux has proven itself to be extremely reliable during the over three years I have had it in service as an Internet server and requires very little hands-on administration to keep it running. Where possible, many repetitive or tedious administrative tasks can and should be automated through crontab entries and script files. However, to ensure that Linux continues to operate in a trouble-free manner, various quick checks can be done from time to time. These include:

9.1 Checking Storage Space

It is important to check from time to time that adequate free space remains on the storage devices. Use the "df" command to get a report of available space. It will look as follows (information shown is from the Internet server at my place of employment):

Filesystem         1024-blocks  Used Available Capacity Mounted on
/dev/sda1            1888052  135908  1654551      8%   /
/dev/sdd1            4299828  100084  3977246      2%   /archive
/dev/hda2            3048303  897858  1992794     31%   /archive2
/dev/hda1              11677    1380     9694     12%   /boot
/dev/sdc1            4299828  350310  3727020      9%   /home
/dev/sdb1            4299828  598504  3478826     15%   /usr
/dev/sda2            1888083  700414  1090075     39%   /var
/dev/scd0             593958  593958        0    100%   /cdrom

These file-systems are pretty stable in that they have a fairly slow growth pattern.

The "/" (aka root) file-system, mounted on /dev/hda1, contains the Linux kernel, device drivers, and other directories. It also is where user mail messages are stored (/var/spool/mail/) as well as log files (/var/adm/) but as mail messages are received and log files are recycled, the available capacity stays fairly stable (an estimated growth of about 1% per month). Log files are rotated and purged automatically on a weekly basis, so you'll always have about a month's worth of log information available to you. Tip: If this file-system is growing rapidly, concentrate your efforts in the /var/spool/mail directory -- look for huge mailboxes (something like ``find /var/spool/mail -size +1000k'' would display a list of mailboxes larger than 1Mb in size). If you find a file much larger than 1,000,000 bytes in size, the user probably isn't retrieving their mail, is on a high-volume mailing list, or their e-mail package isn't configured to remove the mail from the server. Contact the user and/or clear the mail file, using "> mailbox", (eg. ``>smithj'' to clear Joe Smith's mail box). Also check the ``/tmp/'' directory, which may need to be cleaned out on an occasional basis (usually old tin* files left over from aborted newsreader sessions, old print files, etc).

The "/usr/" (aka user) file-system, mounted on /dev/hda2, contains user-installable (user meaning user-installed by system administrator) software, things like your web site pages, etc. This is the largest file-system, and is also fairly slow-growth. The log files for the web pages may also be stored here, and grow in size; check and trim them periodically as needed. On my machines, at the beginning of each month the current web log files are moved to month summary logs (eg. access_log.11 for November's log entries). At the end of the year these logs are all deleted and the cycle starts again (which means each January 1st should see a fair improvement in available space). Tip: If this file-system is growing rapidly, check the /usr/local/etc/httpd/logs and the /usr/local/squid/logs/ directories (if you have them). There may be log file(s) that are getting too large (if, perhaps, the web site received a high number of visits). If, however, the logs are purged automatically on a regular basis as I have them, you shouldn't run into any problems with space here (indeed, as the logs are used for statistical analysis of my site's traffic I'd rather not have to delete them if possible). Another place to check for potentially deletable files is in /usr/tmp/.

The "/home/" (aka user's personal home) file-system, mounted on /dev/hda3, contains all the user directories and personal files. Unless you are giving out shell accounts, most of these will be useless and unaccessible to the user (these directories are created when each users' accounts are created, and can later be used to forward the user's mail, etc.). However shell account users, as well as any non-shell accounts which have web pages (eg. personal web pages) will probably have them stored here. In addition, main server pages are stored here in the /home/httpd directory under Red Hat, while other distributions usually place them in the /usr file system (see the "Web Server Administration" section for more information).

This file-system is probably the slowest growth unless you are offering a lot of shell accounts. Tip: If this file-system suddenly grows in size, it is probably because one of your users is adding web pages or binary files in his/her personal space. Check the ``/var/adm/xferlog.*'' log files for recent activity, which should show you which user has added to their web pages.

I also have an "/archive/" (aka archive files) file-system, mounted on /dev/hdb1, which is a spare 1.02 Gb hard drive that can be used for any purpose (eg. data files, software kits, etc.) I am using a good portion (approximately 70%) of this drive for disk-to-disk full current backups of the system). Generally speaking you can add your own devices and mount them as you wish.

I also have a CD-ROM drive, mounted as "/mnt/cdrom/" on /dev/scd0, which is a 24X-speed SCSI CD-ROM device that can read any ISO9660 formatted CD. It is used primarily for software installation, but DOS/Windows CD's can be mounted and then accessed from Windows 3.x/95/NT network shares as needed via a Samba service (see the "Windows-style File and Print Services with Samba" section for details).

The "rm" command will delete a file. Usage is ``rm filename''. If you want confirmation of deletion, use the "-i" option (eg. ``rm -i *''). You would then be asked to confirm each file before it is deleted. (Note: This is the default for normal shell users, but beware -- the root account will not confirm before deleting files unless you specify the "-i" option!) Be careful you don't make a silly typo with this command -- particularly when logged in as "root" -- because you may end up regretting deleting the wrong file.

9.2 Managing Processes

From time to time you may wish to view processes that are running on Linux. To obtain a list of these processes, type ``ps -aux'', which will look similar to the following:

USER       PID %CPU %MEM  SIZE   RSS TTY STAT START   TIME COMMAND
bin         69  0.0  1.0   788   320  ?  S   Nov 30   0:00 /usr/sbin/rpc.portmap
frampton 10273  0.0  2.1  1136   664  p0 S    14:12   0:00 -bash
frampton 10744  0.0  1.1   820   360  p0 R    17:25   0:00 ps -aux
frampton 10745  0.0  0.8   788   264  p0 S    17:25   0:00 more
nobody   10132  0.0  1.8  1016   588  ?  S    13:36   0:00 httpd
nobody   10133  0.0  1.8   988   568  ?  S    13:36   0:00 httpd
nobody   10413  0.0  1.8  1012   580  ?  S    14:56   0:00 httpd
nobody   10416  0.0  1.8  1012   580  ?  S    14:56   0:00 httpd
nobody   10418  0.0  1.8  1012   588  ?  S    14:57   0:00 httpd
nobody   10488  0.0  1.7   976   556  ?  S    15:34   0:00 httpd
nobody   10564  0.0  1.8   988   564  ?  S    16:06   0:00 httpd
nobody   10600  0.0  1.8   988   564  ?  S    16:15   0:00 httpd
nobody   10670  0.0  1.8   988   568  ?  S    16:45   0:00 httpd
nobody   10704  0.0  1.7   976   552  ?  S    17:03   0:00 httpd
root         1  0.0  1.0   776   312  ?  S   Nov 30   1:13 init [3]
root         2  0.0  0.0     0     0  ?  SW  Nov 30   0:00 (kflushd)
root         3  0.0  0.0     0     0  ?  SW  Nov 30   0:00 (kswapd)

The list shows you the owner of the process ("nobody" for special services such as web servers), the process identification number, the % of CPU time the process is currently using, the % of memory the process is consuming, and other related information, as well as a description of the task itself.

To get more information on a given process, type ``ps pid'' (where "pid" is the process identification number). Looking at our example above, "ps 10704" would display:

10704  ?  S     0:00 /usr/local/etc/httpd/httpd

This would tell you that this particular process is a web server (the Apache web server appears multiple times in the process list; for information on why see the "Web Server Administration" section).

If you happen to notice a service is not operating, you can use the "kill -HUP pid" (where "pid" is the process identification number as shown in the process list produced with "ps"). For example, if Internet services (a process called inetd, process #123 in our example) are not working as they should, a ``kill -HUP 123'' (or even safer, specify the process name: ``kill -HUP inetd'') should restart the process. The -HUP option to the kill command means "hang up"; the process knows that it is supposed to reload itself.

Another thing to try if you are unable to resolve the problem would be to shut the system down and reboot it (see the "System Shutdown & Restart" section for details).

Red Hat provides a better way of starting and stopping some processes, which are covered in the following "Starting and Stopping Processes" section.

9.3 Starting and Stopping Processes

The Red Hat distribution of Linux provides a slightly more organized way of managing processes. Instead of hunting and killing them by finding their process id in the process table, Red Hat provides a collection of scripts in the ``/etc/rc.d/init.d'' directory which will allow you to start and stop processes as desired.

For example, to shut down the ``httpd'' (Apache web server) service, simply run the httpd script, as follows:

/etc/rc.d/init.d/httpd stop

In much the same manner, you can use the ``start'' option to start a service. Or, if you have made changes to a configuration file and wish to restart a service so those changes are recognized, you can use the ``restart'' option.

9.4 Automating Tasks with Cron and Crontab files

Like most Linux users, you may find it necessary to schedule repetitive tasks to be run at a certain time. Such tasks can occur as frequently as once a minute, to as infrequently as once a year. This scheduling can be done by using the ``cron'' facilities.

The cron facilities as implemented in Linux are fairly similar to those available in other Unix implementations. However, Red Hat has adopted a slightly different way of scheduling tasks than is usually done in other distributions of Linux. Just as in other distributions, scheduling information is placed in the system ``crontab'' file (locating in the ``/etc/'' directory), using the following format:

minute hour day month year command

You can specify each time component as an integer number (eg. 1 through 12 for the months January through December), or specify one or more components as ``*'' characters which will be treated as wildcards (eg. * in the month component means the command will run at the given day and time in every month. Here are some examples:

# Mail the system logs at 4:30pm every June 15th.
30 16 15 06 * for x in /var/log/*; do cat ${x} | mail postmaster; done

# Inform the administrator, at midnight, of the changing seasons.
00 00 20 04 * echo 'Woohoo, spring is here!'
00 00 20 06 * echo 'Yeah, summer has arrived, time to hit the beach!'
00 00 20 10 * echo 'Fall has arrived.  Get those jackets out.  :-('
00 00 20 12 * echo 'Time for 5 months of misery.  ;-('

Note that commands which produce output to standard out (ie. a terminal) such as the examples above using ``echo'' will have their output mailed to the ``root'' account. If you want to avoid this, simply pipe the output to the null device as follows:

00 06 * * * echo 'I bug the system administrator daily at 6:00am!' >/dev/null

In addition to the standard ``crontab'' entries, Red Hat adds several directories:

/etc/cron.hourly/
/etc/cron.daily/
/etc/cron.weekly/

As their names suggest, executable files can be placed in any of these directories, and will be executed on an hourly, daily, or weekly basis. This saves a bit of time when setting up frequent tasks; just place the executable script or program (or a symbolic link to one stores elsewhere) in the appropriate directory and forget about it.