1.1. Linux or GNU/Linux, that is the question.

Many people feel that Linux should really be called GNU/Linux. This is because Linux is only the kernel, not the applications that run on it. Most of the basic command line utilities were written by the Free Software Foundation while developing their GNU operating system. Among those utilities are some of the most basic commands like cp, mv, lsof, and dd.

In a nutshell, what happened was, the FSF started developing GNU by writing things like compliers, C libraries, and basic command line utilities before the kernel. Linus Torvalds, started Linux by writing the Linux kernel first and using applications written for GNU.

I do not feel that this is the proper forum to debate what name people should use when referring to Linux. I mention it here, because I feel it is important to understand the relationship between GNU and Linux, and to also explain why some Linux is sometimes referred to as GNU/Linux. The document will be simply referring to it as Linux.

GNU's side of the issue is discussed on their website:

The relationship - http://www.gnu.org/gnu/linux-and-gnu.html

Why Linux should be GNU/Linux - http://www.gnu.org/gnu/why-gnu-linux.html

GNU/Linux FAQ's - http://www.gnu.org/gnu/gnu-linux-faq.html

Here are some Alternate views:

http://librenix.com/?inode=2312

http://www.topology.org/linux/lingl.html

http://atulchitnis.net/writings/gnulinux.php

1.2. Trademarks

Microsoft, Windows, Windows NT, Windows 2000, and Windows XP are trademarks and/or registered trademarks of Microsoft Corporation.

Red Hat is a trademark of Red Hat, Inc., in the United States and other countries.

SuSE is a trademark of Novell.

Linux is a registered trademark of Linus Torvalds.

UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company Ltd.

GNU is a registered trademark of the Free Software Foundation.

Other product names mentioned herein may be trademarks and/or registered trademarks of their respective companies.

2.1. Various parts of an operating system

UNIX and 'UNIX-like' operating systems (such as Linux) consist of a kernel and some system programs. There are also some application programs for doing work. The kernel is the heart of the operating system. In fact, it is often mistakenly considered to be the operating system itself, but it is not. An operating system provides provides many more services than a plain kernel.

It keeps track of files on the disk, starts programs and runs them concurrently, assigns memory and other resources to various processes, receives packets from and sends packets to the network, and so on. The kernel does very little by itself, but it provides tools with which all services can be built. It also prevents anyone from accessing the hardware directly, forcing everyone to use the tools it provides. This way the kernel provides some protection for users from each other. The tools provided by the kernel are used via system calls. See manual page section 2 for more information on these.

The system programs use the tools provided by the kernel to implement the various services required from an operating system. System programs, and all other programs, run `on top of the kernel', in what is called the user mode. The difference between system and application programs is one of intent: applications are intended for getting useful things done (or for playing, if it happens to be a game), whereas system programs are needed to get the system working. A word processor is an application; mount is a system program. The difference is often somewhat blurry, however, and is important only to compulsive categorizers.

An operating system can also contain compilers and their corresponding libraries (GCC and the C library in particular under Linux), although not all programming languages need be part of the operating system. Documentation, and sometimes even games, can also be part of it. Traditionally, the operating system has been defined by the contents of the installation tape or disks; with Linux it is not as clear since it is spread all over the FTP sites of the world.

2.2. Important parts of the kernel

The Linux kernel consists of several important parts: process management, memory management, hardware device drivers, filesystem drivers, network management, and various other bits and pieces. Figure 2-1 shows some of them.

Probably the most important parts of the kernel (nothing else works without them) are memory management and process management. Memory management takes care of assigning memory areas and swap space areas to processes, parts of the kernel, and for the buffer cache. Process management creates processes, and implements multitasking by switching the active process on the processor.

At the lowest level, the kernel contains a hardware device driver for each kind of hardware it supports. Since the world is full of different kinds of hardware, the number of hardware device drivers is large. There are often many otherwise similar pieces of hardware that differ in how they are controlled by software. The similarities make it possible to have general classes of drivers that support similar operations; each member of the class has the same interface to the rest of the kernel but differs in what it needs to do to implement them. For example, all disk drivers look alike to the rest of the kernel, i.e., they all have operations like `initialize the drive', `read sector N', and `write sector N'.

Some software services provided by the kernel itself have similar properties, and can therefore be abstracted into classes. For example, the various network protocols have been abstracted into one programming interface, the BSD socket library. Another example is the virtual filesystem (VFS) layer that abstracts the filesystem operations away from their implementation. Each filesystem type provides an implementation of each filesystem operation. When some entity tries to use a filesystem, the request goes via the VFS, which routes the request to the proper filesystem driver.

A more in-depth discussion of kernel internals can be found at http://www.tldp.org/LDP/lki/index.html. This document was written for the 2.4 kernel. When I find one for the 2.6 kernel, I will list it here.

2.3. Major services in a UNIX system

This section describes some of the more important UNIX services, but without much detail. They are described more thoroughly in later chapters.

3.1. Background

This chapter is loosely based on the Filesystems Hierarchy Standard (FHS). version 2.1, which attempts to set a standard for how the directory tree in a Linux system is organized. Such a standard has the advantage that it will be easier to write or port software for Linux, and to administer Linux machines, since everything should be in standardized places. There is no authority behind the standard that forces anyone to comply with it, but it has gained the support of many Linux distributions. It is not a good idea to break with the FHS without very compelling reasons. The FHS attempts to follow Unix tradition and current trends, making Linux systems familiar to those with experience with other Unix systems, and vice versa.

This chapter is not as detailed as the FHS. A system administrator should also read the full FHS for a complete understanding.

This chapter does not explain all files in detail. The intention is not to describe every file, but to give an overview of the system from a filesystem point of view. Further information on each file is available elsewhere in this manual or in the Linux manual pages.

The full directory tree is intended to be breakable into smaller parts, each capable of being on its own disk or partition, to accommodate to disk size limits and to ease backup and other system administration tasks. The major parts are the root (/ ), /usr , /var , and /home filesystems (see Figure 3-1). Each part has a different purpose. The directory tree has been designed so that it works well in a network of Linux machines which may share some parts of the filesystems over a read-only device (e.g., a CD-ROM), or over the network with NFS.

The roles of the different parts of the directory tree are described below.

• The root filesystem is specific for each machine (it is generally stored on a local disk, although it could be a ramdisk or network drive as well) and contains the files that are necessary for booting the system up, and to bring it up to such a state that the other filesystems may be mounted. The contents of the root filesystem will therefore be sufficient for the single user state. It will also contain tools for fixing a broken system, and for recovering lost files from backups.

• The /usr filesystem contains all commands, libraries, manual pages, and other unchanging files needed during normal operation. No files in /usr should be specific for any given machine, nor should they be modified during normal use. This allows the files to be shared over the network, which can be cost-effective since it saves disk space (there can easily be hundreds of megabytes, increasingly multiple gigabytes in /usr). It can make administration easier (only the master /usr needs to be changed when updating an application, not each machine separately) to have /usr network mounted. Even if the filesystem is on a local disk, it could be mounted read-only, to lessen the chance of filesystem corruption during a crash.

• The /var filesystem contains files that change, such as spool directories (for mail, news, printers, etc), log files, formatted manual pages, and temporary files. Traditionally everything in /var has been somewhere below /usr , but that made it impossible to mount /usr read-only.

• The /home filesystem contains the users' home directories, i.e., all the real data on the system. Separating home directories to their own directory tree or filesystem makes backups easier; the other parts often do not have to be backed up, or at least not as often as they seldom change. A big /home might have to be broken across several filesystems, which requires adding an extra naming level below /home, for example /home/students and /home/staff.

Although the different parts have been called filesystems above, there is no requirement that they actually be on separate filesystems. They could easily be kept in a single one if the system is a small single-user system and the user wants to keep things simple. The directory tree might also be divided into filesystems differently, depending on how large the disks are, and how space is allocated for various purposes. The important part, though, is that all the standard names work; even if, say, /var and /usr are actually on the same partition, the names /usr/lib/libc.a and /var/log/messages must work, for example by moving files below /var into /usr/var, and making /var a symlink to /usr/var.

The Unix filesystem structure groups files according to purpose, i.e., all commands are in one place, all data files in another, documentation in a third, and so on. An alternative would be to group files files according to the program they belong to, i.e., all Emacs files would be in one directory, all TeX in another, and so on. The problem with the latter approach is that it makes it difficult to share files (the program directory often contains both static and sharable and changing and non-sharable files), and sometimes to even find the files (e.g., manual pages in a huge number of places, and making the manual page programs find all of them is a maintenance nightmare).

3.2. The root filesystem

The root filesystem should generally be small, since it contains very critical files and a small, infrequently modified filesystem has a better chance of not getting corrupted. A corrupted root filesystem will generally mean that the system becomes unbootable except with special measures (e.g., from a floppy), so you don't want to risk it.

The root directory generally doesn't contain any files, except perhaps on older systems where the standard boot image for the system, usually called /vmlinuz was kept there. (Most distributions have moved those files into the /boot directory. Otherwise, all files are kept in subdirectories under the root filesystem:

3.3. The /etc directory

The /etc maintains a lot of files. Some of them are described below. For others, you should determine which program they belong to and read the manual page for that program. Many networking configuration files are in /etc as well, and are described in the Networking Administrators' Guide.

3.4. The /dev directory

The /dev directory contains the special device files for all the devices. The device files are created during installation, and later with the /dev/MAKEDEV script. The /dev/MAKEDEV.local is a script written by the system administrator that creates local-only device files or links (i.e. those that are not part of the standard MAKEDEV, such as device files for some non-standard device driver).

This list which follows is by no means exhaustive or as detailed as it could be. Many of these device files will need support compiled into your kernel for the hardware. Read the kernel documentation to find details of any particular device.

If you think there are other devices which should be included here but aren't then let me know. I will try to include them in the next revision.

3.5. The /usr filesystem.

The /usr filesystem is often large, since all programs are installed there. All files in /usr usually come from a Linux distribution; locally installed programs and other stuff goes below /usr/local. This makes it possible to update the system from a new version of the distribution, or even a completely new distribution, without having to install all programs again. Some of the subdirectories of /usr are listed below (some of the less important directories have been dropped; see the FSSTND for more information).

3.6. The /var filesystem

The /var contains data that is changed when the system is running normally. It is specific for each system, i.e., not shared over the network with other computers.

3.7. The /proc filesystem

The /proc filesystem contains a illusionary filesystem. It does not exist on a disk. Instead, the kernel creates it in memory. It is used to provide information about the system (originally about processes, hence the name). Some of the more important files and directories are explained below. The /proc filesystem is described in more detail in the proc manual page.

Note that while the above files tend to be easily readable text files, they can sometimes be formatted in a way that is not easily digestible. There are many commands that do little more than read the above files and format them for easier understanding. For example, the freeprogram reads /proc/meminfo converts the amounts given in bytes to kilobytes (and adds a little more information, as well).

4.1. Hardware Utilities

	# /dev/MAKEDEV -v ttyS0
	create ttyS0   c 4 64 root:dialout 0660
	

	# mknod /dev/ttyS0 c 4 64
	# chown root.dialout /dev/ttyS0
	# chmod 0644 /dev/ttyS0
	# ls -l /dev/ttyS0
		crw-rw----   1 root dialout    4,   64 Oct 23 18:23 /dev/ttyS0
	 
	

4.2. Kernel Modules

This section will discuss kernel modules.

TO BE ADDED

5.1. Two kinds of devices

UNIX, and therefore Linux, recognizes two different kinds of device: random-access block devices (such as disks), and character devices (such as tapes and serial lines) , some of which may be serial, and some random-access. Each supported device is represented in the filesystem as a device file. When you read or write a device file, the data comes from or goes to the device it represents. This way no special programs (and no special application programming methodology, such as catching interrupts or polling a serial port) are necessary to access devices; for example, to send a file to the printer, one could just say

$ cat filename > /dev/lp1
$

Since devices show up as files in the filesystem (in the /dev directory), it is easy to see just what device files exist, using ls or another suitable command. In the output of ls -l, the first column contains the type of the file and its permissions. For example, inspecting a serial device might give

	$ ls -l /dev/ttyS0
		crw-rw-r--    1 root     dialout    4,  64 Aug 19 18:56 /dev/ttyS0
	
	$
	

Note that usually all device files exist even though the device itself might be not be installed. So just because you have a file /dev/sda, it doesn't mean that you really do have an SCSI hard disk. Having all the device files makes the installation programs simpler, and makes it easier to add new hardware (there is no need to find out the correct parameters for and create the device files for the new device).

5.2. Hard disks

This subsection introduces terminology related to hard disks. If you already know the terms and concepts, you can skip this subsection.

See Figure 5-1 for a schematic picture of the important parts in a hard disk. A hard disk consists of one or more circular aluminum platters\ , of which either or both surfaces are coated with a magnetic substance used for recording the data. For each surface, there is a read-write head that examines or alters the recorded data. The platters rotate on a common axis; typical rotation speed is 5400 or 7200 rotations per minute, although high-performance hard disks have higher speeds and older disks may have lower speeds. The heads move along the radius of the platters; this movement combined with the rotation of the platters allows the head to access all parts of the surfaces.

The processor (CPU) and the actual disk communicate through a disk controller . This relieves the rest of the computer from knowing how to use the drive, since the controllers for different types of disks can be made to use the same interface towards the rest of the computer. Therefore, the computer can say just ``hey disk, give me what I want'', instead of a long and complex series of electric signals to move the head to the proper location and waiting for the correct position to come under the head and doing all the other unpleasant stuff necessary. (In reality, the interface to the controller is still complex, but much less so than it would otherwise be.) The controller may also do other things, such as caching, or automatic bad sector replacement.

The above is usually all one needs to understand about the hardware. There are also other things, such as the motor that rotates the platters and moves the heads, and the electronics that control the operation of the mechanical parts, but they are mostly not relevant for understanding the working principles of a hard disk.

The surfaces are usually divided into concentric rings, called tracks, and these in turn are divided into sectors. This division is used to specify locations on the hard disk and to allocate disk space to files. To find a given place on the hard disk, one might say ``surface 3, track 5, sector 7''. Usually the number of sectors is the same for all tracks, but some hard disks put more sectors in outer tracks (all sectors are of the same physical size, so more of them fit in the longer outer tracks). Typically, a sector will hold 512 bytes of data. The disk itself can't handle smaller amounts of data than one sector.

Each surface is divided into tracks (and sectors) in the same way. This means that when the head for one surface is on a track, the heads for the other surfaces are also on the corresponding tracks. All the corresponding tracks taken together are called a cylinder. It takes time to move the heads from one track (cylinder) to another, so by placing the data that is often accessed together (say, a file) so that it is within one cylinder, it is not necessary to move the heads to read all of it. This improves performance. It is not always possible to place files like this; files that are stored in several places on the disk are called fragmented.

The number of surfaces (or heads, which is the same thing), cylinders, and sectors vary a lot; the specification of the number of each is called the geometry of a hard disk. The geometry is usually stored in a special, battery-powered memory location called the CMOS RAM , from where the operating system can fetch it during bootup or driver initialization.

Unfortunately, the BIOS has a design limitation, which makes it impossible to specify a track number that is larger than 1024 in the CMOS RAM, which is too little for a large hard disk. To overcome this, the hard disk controller lies about the geometry, and translates the addresses given by the computer into something that fits reality. For example, a hard disk might have 8 heads, 2048 tracks, and 35 sectors per track. Its controller could lie to the computer and claim that it has 16 heads, 1024 tracks, and 35 sectors per track, thus not exceeding the limit on tracks, and translates the address that the computer gives it by halving the head number, and doubling the track number. The mathematics can be more complicated in reality, because the numbers are not as nice as here (but again, the details are not relevant for understanding the principle). This translation distorts the operating system's view of how the disk is organized, thus making it impractical to use the all-data-on-one-cylinder trick to boost performance.

The translation is only a problem for IDE disks. SCSI disks use a sequential sector number (i.e., the controller translates a sequential sector number to a head, cylinder, and sector triplet), and a completely different method for the CPU to talk with the controller, so they are insulated from the problem. Note, however, that the computer might not know the real geometry of an SCSI disk either.

Since Linux often will not know the real geometry of a disk, its filesystems don't even try to keep files within a single cylinder. Instead, it tries to assign sequentially numbered sectors to files, which almost always gives similar performance. The issue is further complicated by on-controller caches, and automatic prefetches done by the controller.

Each hard disk is represented by a separate device file. There can (usually) be only two or four IDE hard disks. These are known as /dev/hda, /dev/hdb, /dev/hdc, and /dev/hdd, respectively. SCSI hard disks are known as /dev/sda, /dev/sdb, and so on. Similar naming conventions exist for other hard disk types; see Chapter 4 for more information. Note that the device files for the hard disks give access to the entire disk, with no regard to partitions (which will be discussed below), and it's easy to mess up the partitions or the data in them if you aren't careful. The disks' device files are usually used only to get access to the master boot record (which will also be discussed below).

5.3. Storage Area Networks - Draft

A SAN is a dedicated storage network that provides block level access to LUNs. A LUN, or logical unit number, is a virtual disk provided by the SAN. The system administrator the same access and rights to the LUN as if it were a disk directly attached to it. The administrator can partition, and format the disk in any means he or she chooses.

Two networking protocols commonly used in a SAN are fibre channel and iSCSI . A fibre channel network is very fast and is not burdened by the other network traffic in a company's LAN. However, it's very expensive. Fibre channel cards cost around $1000.00 USD each. They also require special fibre channel switches.

iSCSI is a newer technology that sends SCSI commands over a TCP/IP network. While this method may not be as fast as a Fibre Channel network, it does save money by using less expensive network hardware.

More To Be Added

5.4. Network Attached Storage - Draft

A NAS uses your companies existing Ethernet network to allow access to shared disks. This is filesystem level access. The system administrator does not have the ability to partition or format the disks since they are potentially shared by multiple computers. This technology is commonly used to provide multiple workstations access to the same data.

Similar to a SAN, a NAS need to make use of a protocol to allow access to it's disks. With a NAS this is either CIFS/Samba , or NFS.

Traditionally CIFS was used with Microsoft Windows networks, and NFS was used with UNIX & Linux networks. However, with Samba, Linux machines can also make use of CIFS shares.

Does this mean that your Windows 2003 server or your Linux box are NAS servers because they provide access to shared drives over your network? Yes, they are. You could also purchase a NAS device from a number of manufacturers. These devices are specifically designed to provide high speed access to data.

More To Be Added

5.5. Floppies

A floppy disk consists of a flexible membrane covered on one or both sides with similar magnetic substance as a hard disk. The floppy disk itself doesn't have a read-write head, that is included in the drive. A floppy corresponds to one platter in a hard disk, but is removable and one drive can be used to access different floppies, and the same floppy can be read by many drives, whereas the hard disk is one indivisible unit.

Like a hard disk, a floppy is divided into tracks and sectors (and the two corresponding tracks on either side of a floppy form a cylinder), but there are many fewer of them than on a hard disk.

A floppy drive can usually use several different types of disks; for example, a 3.5 inch drive can use both 720 KB and 1.44 MB disks. Since the drive has to operate a bit differently and the operating system must know how big the disk is, there are many device files for floppy drives, one per combination of drive and disk type. Therefore, /dev/fd0H1440 is the first floppy drive (fd0), which must be a 3.5 inch drive, using a 3.5 inch, high density disk (H) of size 1440 KB (1440), i.e., a normal 3.5 inch HD floppy.

The names for floppy drives are complex, however, and Linux therefore has a special floppy device type that automatically detects the type of the disk in the drive. It works by trying to read the first sector of a newly inserted floppy using different floppy types until it finds the correct one. This naturally requires that the floppy is formatted first. The automatic devices are called /dev/fd0, /dev/fd1, and so on.

The parameters the automatic device uses to access a disk can also be set using the program setfdprm . This can be useful if you need to use disks that do not follow any usual floppy sizes, e.g., if they have an unusual number of sectors, or if the autodetecting for some reason fails and the proper device file is missing.

Linux can handle many nonstandard floppy disk formats in addition to all the standard ones. Some of these require using special formatting programs. We'll skip these disk types for now, but in the mean time you can examine the /etc/fdprm file. It specifies the settings that setfdprm recognizes.

The operating system must know when a disk has been changed in a floppy drive, for example, in order to avoid using cached data from the previous disk. Unfortunately, the signal line that is used for this is sometimes broken, and worse, this won't always be noticeable when using the drive from within MS-DOS. If you are experiencing weird problems using floppies, this might be the reason. The only way to correct it is to repair the floppy drive.

5.6. CD-ROMs

A CD-ROM drive uses an optically read, plastic coated disk. The information is recorded on the surface of the disk in small `holes' aligned along a spiral from the center to the edge. The drive directs a laser beam along the spiral to read the disk. When the laser hits a hole, the laser is reflected in one way; when it hits smooth surface, it is reflected in another way. This makes it easy to code bits, and therefore information. The rest is easy, mere mechanics.

CD-ROM drives are slow compared to hard disks. Whereas a typical hard disk will have an average seek time less than 15 milliseconds, a fast CD-ROM drive can use tenths of a second for seeks. The actual data transfer rate is fairly high at hundreds of kilobytes per second. The slowness means that CD-ROM drives are not as pleasant to use as hard disks (some Linux distributions provide `live' filesystems on CD-ROMs, making it unnecessary to copy the files to the hard disk, making installation easier and saving a lot of hard disk space), although it is still possible. For installing new software, CD-ROMs are very good, since maximum speed is not essential during installation.

There are several ways to arrange data on a CD-ROM. The most popular one is specified by the international standard ISO 9660 . This standard specifies a very minimal filesystem, which is even more crude than the one MS-DOS uses. On the other hand, it is so minimal that every operating system should be able to map it to its native system.

For normal UNIX use, the ISO 9660 filesystem is not usable, so an extension to the standard has been developed, called the Rock Ridge extension. Rock Ridge allows longer filenames, symbolic links, and a lot of other goodies, making a CD-ROM look more or less like any contemporary UNIX filesystem. Even better, a Rock Ridge filesystem is still a valid ISO 9660 filesystem, making it usable by non-UNIX systems as well. Linux supports both ISO 9660 and the Rock Ridge extensions; the extensions are recognized and used automatically.

The filesystem is only half the battle, however. Most CD-ROMs contain data that requires a special program to access, and most of these programs do not run under Linux (except, possibly, under dosemu, the Linux MS-DOS emulator, or wine, the Windows emulator.

Ironically perhaps, wine actually stands for ``Wine Is Not an Emulator''. Wine, more strictly, is an API (Application Program Interface) replacement. Please see the wine documentation at http://www.winehq.com for more information.

There is also VMWare, a commercial product, which emulates an entire x86 machine in software. See the VMWare website, http://www.vmware.com for more information.

A CD-ROM drive is accessed via the corresponding device file. There are several ways to connect a CD-ROM drive to the computer: via SCSI, via a sound card, or via EIDE. The hardware hacking needed to do this is outside the scope of this book, but the type of connection decides the device file.

5.7. Tapes

A tape drive uses a tape, similar to cassettes used for music. A tape is serial in nature, which means that in order to get to any given part of it, you first have to go through all the parts in between. A disk can be accessed randomly, i.e., you can jump directly to any place on the disk. The serial access of tapes makes them slow.

On the other hand, tapes are relatively cheap to make, since they do not need to be fast. They can also easily be made quite long, and can therefore contain a large amount of data. This makes tapes very suitable for things like archiving and backups, which do not require large speeds, but benefit from low costs and large storage capacities.

5.8. Formatting

Formatting is the process of writing marks on the magnetic media that are used to mark tracks and sectors. Before a disk is formatted, its magnetic surface is a complete mess of magnetic signals. When it is formatted, some order is brought into the chaos by essentially drawing lines where the tracks go, and where they are divided into sectors. The actual details are not quite exactly like this, but that is irrelevant. What is important is that a disk cannot be used unless it has been formatted.

The terminology is a bit confusing here: in MS-DOS and MS Windows, the word formatting is used to cover also the process of creating a filesystem (which will be discussed below). There, the two processes are often combined, especially for floppies. When the distinction needs to be made, the real formatting is called low-level formatting, while making the filesystem is called high-level formatting . In UNIX circles, the two are called formatting and making a filesystem, so that's what is used in this book as well.

For IDE and some SCSI disks the formatting is actually done at the factory and doesn't need to be repeated; hence most people rarely need to worry about it. In fact, formatting a hard disk can cause it to work less well, for example because a disk might need to be formatted in some very special way to allow automatic bad sector replacement to work.

Disks that need to be or can be formatted often require a special program anyway, because the interface to the formatting logic inside the drive is different from drive to drive. The formatting program is often either on the controller BIOS, or is supplied as an MS-DOS program; neither of these can easily be used from within Linux.

During formatting one might encounter bad spots on the disk, called bad blocks or bad sectors. These are sometimes handled by the drive itself, but even then, if more of them develop, something needs to be done to avoid using those parts of the disk. The logic to do this is built into the filesystem; how to add the information into the filesystem is described below. Alternatively, one might create a small partition that covers just the bad part of the disk; this approach might be a good idea if the bad spot is very large, since filesystems can sometimes have trouble with very large bad areas.

Floppies are formatted with fdformat . The floppy device file to use is given as the parameter. For example, the following command would format a high density, 3.5 inch floppy in the first floppy drive:

	$ fdformat /dev/fd0H1440
	Double-sided, 80 tracks, 18 sec/track. Total capacity 
	1440 kB.
	Formatting ... done
	Verifying ... done
	$
	

	$ setfdprm /dev/fd0 1440/1440
	$ fdformat /dev/fd0
	Double-sided, 80 tracks, 18 sec/track. Total capacity 
	1440 KB.
	Formatting ... done
	Verifying ... done
	$
	

fdformatalso validate the floppy, i.e., check it for bad blocks. It will try a bad block several times (you can usually hear this, the drive noise changes dramatically). If the floppy is only marginally bad (due to dirt on the read/write head, some errors are false signals), fdformat won't complain, but a real error will abort the validation process. The kernel will print log messages for each I/O error it finds; these will go to the console or, if syslog is being used, to the file /var/log/messages. fdformat itself won't tell where the error is (one usually doesn't care, floppies are cheap enough that a bad one is automatically thrown away).

	$ fdformat /dev/fd0H1440
	Double-sided, 80 tracks, 18 sec/track. Total capacity 
	1440 KB.
	Formatting ... done
	Verifying ... read: Unknown error
	$
	

	$ badblocks /dev/fd0H1440 1440
	718
	719
	$
	

Many modern disks automatically notice bad blocks, and attempt to fix them by using a special, reserved good block instead. This is invisible to the operating system. This feature should be documented in the disk's manual, if you're curious if it is happening. Even such disks can fail, if the number of bad blocks grows too large, although chances are that by then the disk will be so rotten as to be unusable.

5.9. Partitions

A hard disk can be divided into several partitions. Each partition functions as if it were a separate hard disk. The idea is that if you have one hard disk, and want to have, say, two operating systems on it, you can divide the disk into two partitions. Each operating system uses its partition as it wishes and doesn't touch the other ones. This way the two operating systems can co-exist peacefully on the same hard disk. Without partitions one would have to buy a hard disk for each operating system.

Floppies are not usually partitioned. There is no technical reason against this, but since they're so small, partitions would be useful only very rarely. CD-ROMs are usually also not partitioned, since it's easier to use them as one big disk, and there is seldom a need to have several operating systems on one.

	$ fdisk -l /dev/hda
	
	Disk /dev/hda: 15 heads, 57 sectors, 790 cylinders
	Units = cylinders of 855 * 512 bytes
	
	   Device Boot  Begin   Start     End  Blocks   Id  System
	/dev/hda1           1       1      24   10231+  82  Linux swap
	/dev/hda2          25      25      48   10260   83  Linux native
	/dev/hda3          49      49     408  153900   83  Linux native
	/dev/hda4         409     409     790  163305    5  Extended
	/dev/hda5         409     409     744  143611+  83  Linux native
	/dev/hda6         745     745     790   19636+  83  Linux native
	$
	

 0  Empty           1c  Hidden Win95 FA 70  DiskSecure Mult bb  Boot Wizard hid
 1  FAT12           1e  Hidden Win95 FA 75  PC/IX           be  Solaris boot
 2  XENIX root      24  NEC DOS         80  Old Minix       c1  DRDOS/sec (FAT-
 3  XENIX usr       39  Plan 9          81  Minix / old Lin c4  DRDOS/sec (FAT-
 4  FAT16 <32M      3c  PartitionMagic  82  Linux swap      c6  DRDOS/sec (FAT-
 5  Extended        40  Venix 80286     83  Linux           c7  Syrinx
 6  FAT16           41  PPC PReP Boot   84  OS/2 hidden C:  da  Non-FS data
 7  HPFS/NTFS       42  SFS             85  Linux extended  db  CP/M / CTOS / .
 8  AIX             4d  QNX4.x          86  NTFS volume set de  Dell Utility
 9  AIX bootable    4e  QNX4.x 2nd part 87  NTFS volume set df  BootIt
 a  OS/2 Boot Manag 4f  QNX4.x 3rd part 8e  Linux LVM       e1  DOS access
 b  Win95 FAT32     50  OnTrack DM      93  Amoeba          e3  DOS R/O
 c  Win95 FAT32 (LB 51  OnTrack DM6 Aux 94  Amoeba BBT      e4  SpeedStor
 e  Win95 FAT16 (LB 52  CP/M            9f  BSD/OS          eb  BeOS fs
 f  Win95 Ext'd (LB 53  OnTrack DM6 Aux a0  IBM Thinkpad hi ee  EFI GPT
10  OPUS            54  OnTrackDM6      a5  FreeBSD         ef  EFI (FAT-12/16/
11  Hidden FAT12    55  EZ-Drive        a6  OpenBSD         f0  Linux/PA-RISC b
12  Compaq diagnost 56  Golden Bow      a7  NeXTSTEP        f1  SpeedStor
14  Hidden FAT16 <3 5c  Priam Edisk     a8  Darwin UFS      f4  SpeedStor
16  Hidden FAT16    61  SpeedStor       a9  NetBSD          f2  DOS secondary
17  Hidden HPFS/NTF 63  GNU HURD or Sys ab  Darwin boot     fd  Linux raid auto
18  AST SmartSleep  64  Novell Netware  b7  BSDI fs         fe  LANstep
1b  Hidden Win95 FA 65  Novell Netware  b8  BSDI swap       ff  BBT

5.10. Filesystems

$ ls -l /proc
total 0
dr-xr-xr-x   4 root     root            0 Jan 31 20:37 1
dr-xr-xr-x   4 liw      users           0 Jan 31 20:37 63
dr-xr-xr-x   4 liw      users           0 Jan 31 20:37 94
dr-xr-xr-x   4 liw      users           0 Jan 31 20:37 95
dr-xr-xr-x   4 root     users           0 Jan 31 20:37 98
dr-xr-xr-x   4 liw      users           0 Jan 31 20:37 99
-r--r--r--   1 root     root            0 Jan 31 20:37 devices
-r--r--r--   1 root     root            0 Jan 31 20:37 dma
-r--r--r--   1 root     root            0 Jan 31 20:37 filesystems
-r--r--r--   1 root     root            0 Jan 31 20:37 interrupts
-r--------   1 root     root      8654848 Jan 31 20:37 kcore
-r--r--r--   1 root     root            0 Jan 31 11:50 kmsg
-r--r--r--   1 root     root            0 Jan 31 20:37 ksyms
-r--r--r--   1 root     root            0 Jan 31 11:51 loadavg
-r--r--r--   1 root     root            0 Jan 31 20:37 meminfo
-r--r--r--   1 root     root            0 Jan 31 20:37 modules
dr-xr-xr-x   2 root     root            0 Jan 31 20:37 net
dr-xr-xr-x   4 root     root            0 Jan 31 20:37 self
-r--r--r--   1 root     root            0 Jan 31 20:37 stat
-r--r--r--   1 root     root            0 Jan 31 20:37 uptime
-r--r--r--   1 root     root            0 Jan 31 20:37 
version
$

$ fdformat -n /dev/fd0H1440
Double-sided, 80 tracks, 18 sec/track. Total capacity 
1440 KB.
Formatting ... done
$ badblocks /dev/fd0H1440 1440 $>$ 
bad-blocks
$ mkfs.ext2 -l bad-blocks 
/dev/fd0H1440
mke2fs 0.5a, 5-Apr-94 for EXT2 FS 0.5, 94/03/10
360 inodes, 1440 blocks
72 blocks (5.00%) reserved for the super user
First data block=1
Block size=1024 (log=0)
Fragment size=1024 (log=0)
1 block group
8192 blocks per group, 8192 fragments per group
360 inodes per group

Writing inode tables: done
Writing superblocks and filesystem accounting information: 
done
$

$ mkfs.ext2 -c 
/dev/fd0H1440
mke2fs 0.5a, 5-Apr-94 for EXT2 FS 0.5, 94/03/10
360 inodes, 1440 blocks
72 blocks (5.00%) reserved for the super user
First data block=1
Block size=1024 (log=0)
Fragment size=1024 (log=0)
1 block group
8192 blocks per group, 8192 fragments per group
360 inodes per group

Checking for bad blocks (read-only test): done
Writing inode tables: done
Writing superblocks and filesystem accounting information: 
done
$

$ mount /dev/hda2 /home
$ mount /dev/hda3 /usr
$

	$ mount -t msdos /dev/fd0 
	/floppy
	$
	

$ umount /dev/hda2
$ umount /usr
$

	/dev/fd0            /floppy      msdos   user,noauto      0     0
	

	$ mount /floppy
	$
	

	/dev/fd0    /mnt/dosfloppy    msdos   user,noauto  0  0
	/dev/fd0    /mnt/ext2floppy   ext2    user,noauto  0  0
	

        /dev/fd0    /mnt/floppy    auto   user,noauto  0  0
        

	$ badblocks /dev/fd0H1440 1440 > 
	bad-blocks
	$ fsck -t ext2 -l bad-blocks 
	/dev/fd0H1440
	Parallelizing fsck version 0.5a (5-Apr-94)
	e2fsck 0.5a, 5-Apr-94 for EXT2 FS 0.5, 94/03/10
	Pass 1: Checking inodes, blocks, and sizes
	Pass 2: Checking directory structure
	Pass 3: Checking directory connectivity
	Pass 4: Check reference counts.
	Pass 5: Checking group summary information.
	
	/dev/fd0H1440: ***** FILE SYSTEM WAS MODIFIED *****
	/dev/fd0H1440: 11/360 files, 63/1440 blocks
	$
	

# dumpe2fs
dumpe2fs 1.32 (09-Nov-2002)
Filesystem volume name:   /
Last mounted on:          not available
Filesystem UUID:          51603f82-68f3-4ae7-a755-b777ff9dc739
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal filetype needs_recovery sparse_super
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              3482976
Block count:              6960153
Reserved block count:     348007
Free blocks:              3873525
Free inodes:              3136573
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16352
Inode blocks per group:   511
Filesystem created:       Tue Aug 26 08:11:55 2003
Last mount time:          Mon Dec 22 08:23:12 2003
Last write time:          Mon Dec 22 08:23:12 2003
Mount count:              3
Maximum mount count:      -1
Last checked:             Mon Nov  3 11:27:38 2003
Check interval:           0 (none)
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal UUID:             none
Journal inode:            8
Journal device:           0x0000
First orphan inode:       655612


Group 0: (Blocks 0-32767)
  Primary superblock at 0, Group descriptors at 1-2
  Block bitmap at 3 (+3), Inode bitmap at 4 (+4)
  Block bitmap at 3 (+3), Inode bitmap at 4 (+4)
  Inode table at 5-515 (+5)
  3734 free blocks, 16338 free inodes, 2 directories

5.11. Disks without filesystems

Not all disks or partitions are used as filesystems. A swap partition, for example, will not have a filesystem on it. Many floppies are used in a tape-drive emulating fashion, so that a tar (tape archive) or other file is written directly on the raw disk, without a filesystem. Linux boot floppies don't contain a filesystem, only the raw kernel.

Avoiding a filesystem has the advantage of making more of the disk usable, since a filesystem always has some bookkeeping overhead. It also makes the disks more easily compatible with other systems: for example, the tar file format is the same on all systems, while filesystems are different on most systems. You will quickly get used to disks without filesystems if you need them. Bootable Linux floppies also do not necessarily have a filesystem, although they may.

One reason to use raw disks is to make image copies of them. For instance, if the disk contains a partially damaged filesystem, it is a good idea to make an exact copy of it before trying to fix it, since then you can start again if your fixing breaks things even more. One way to do this is to use dd:

	$ dd if=/dev/fd0H1440 
	of=floppy-image
	2880+0 records in
	2880+0 records out
	$ dd if=floppy-image 
	of=/dev/fd0H1440
	2880+0 records in
	2880+0 records out
	$
	

5.12. Allocating disk space

Filesystem            Size  Used Avail Use% Mounted on
/dev/hda2             9.7G  1.3G  8.0G  14% /
/dev/hda1             128M   44M   82M  34% /boot
/dev/hda3             4.9G  4.0G  670M  86% /usr
/dev/hda5             4.9G  2.1G  2.5G  46% /var
/dev/hda7              31G   24G  5.6G  81% /home
/dev/hda8             4.9G  2.0G  670M  43% /opt
	

6.1. What is virtual memory?

Linux supports virtual memory, that is, using a disk as an extension of RAM so that the effective size of usable memory grows correspondingly. The kernel will write the contents of a currently unused block of memory to the hard disk so that the memory can be used for another purpose. When the original contents are needed again, they are read back into memory. This is all made completely transparent to the user; programs running under Linux only see the larger amount of memory available and don't notice that parts of them reside on the disk from time to time. Of course, reading and writing the hard disk is slower (on the order of a thousand times slower) than using real memory, so the programs don't run as fast. The part of the hard disk that is used as virtual memory is called the swap space.

Linux can use either a normal file in the filesystem or a separate partition for swap space. A swap partition is faster, but it is easier to change the size of a swap file (there's no need to repartition the whole hard disk, and possibly install everything from scratch). When you know how much swap space you need, you should go for a swap partition, but if you are uncertain, you can use a swap file first, use the system for a while so that you can get a feel for how much swap you need, and then make a swap partition when you're confident about its size.

You should also know that Linux allows one to use several swap partitions and/or swap files at the same time. This means that if you only occasionally need an unusual amount of swap space, you can set up an extra swap file at such times, instead of keeping the whole amount allocated all the time.

A note on operating system terminology: computer science usually distinguishes between swapping (writing the whole process out to swap space) and paging (writing only fixed size parts, usually a few kilobytes, at a time). Paging is usually more efficient, and that's what Linux does, but traditional Linux terminology talks about swapping anyway.

6.2. Creating a swap space

A swap file is an ordinary file; it is in no way special to the kernel. The only thing that matters to the kernel is that it has no holes, and that it is prepared for use with mkswap. It must reside on a local disk, however; it can't reside in a filesystem that has been mounted over NFS due to implementation reasons.

The bit about holes is important. The swap file reserves the disk space so that the kernel can quickly swap out a page without having to go through all the things that are necessary when allocating a disk sector to a file. The kernel merely uses any sectors that have already been allocated to the file. Because a hole in a file means that there are no disk sectors allocated (for that place in the file), it is not good for the kernel to try to use them.

One good way to create the swap file without holes is through the following command:

	$ dd if=/dev/zero of=/extra-swap bs=1024 
	count=1024
	1024+0 records in
	1024+0 records out
	$
	

A swap partition is also not special in any way. You create it just like any other partition; the only difference is that it is used as a raw partition, that is, it will not contain any filesystem at all. It is a good idea to mark swap partitions as type 82 (Linux swap); this will the make partition listings clearer, even though it is not strictly necessary to the kernel.

After you have created a swap file or a swap partition, you need to write a signature to its beginning; this contains some administrative information and is used by the kernel. The command to do this is mkswap, used like this:

	$ mkswap /extra-swap 1024
	Setting up swapspace, size = 1044480 
	bytes
	$
	

You should be very careful when using mkswap, since it does not check that the file or partition isn't used for anything else. You can easily overwrite important files and partitions with mkswap! Fortunately, you should only need to use mkswap when you install your system.

The Linux memory manager limits the size of each swap space to 2 GB. You can, however, use up to 8 swap spaces simultaneously, for a total of 16GB.

6.3. Using a swap space

An initialized swap space is taken into use with swapon. This command tells the kernel that the swap space can be used. The path to the swap space is given as the argument, so to start swapping on a temporary swap file one might use the following command.

$ swapon /extra-swap
$

/dev/hda8        none        swap        sw     0     0
/swapfile        none        swap        sw     0     0

You can monitor the use of swap spaces with free. It will tell the total amount of swap space used.

$ free
             total       used       free     shared   
 buffers
Mem:         15152      14896        256      12404       2528
-/+ buffers:            12368       2784
Swap:        32452       6684      25768
$

That last line (Swap:) shows similar information for the swap spaces. If this line is all zeroes, your swap space is not activated.

The same information is available via top, or using the proc filesystem in file /proc/meminfo. It is currently difficult to get information on the use of a specific swap space.

A swap space can be removed from use with swapoff. It is usually not necessary to do it, except for temporary swap spaces. Any pages in use in the swap space are swapped in first; if there is not sufficient physical memory to hold them, they will then be swapped out (to some other swap space). If there is not enough virtual memory to hold all of the pages Linux will start to thrash; after a long while it should recover, but meanwhile the system is unusable. You should check (e.g., with free) that there is enough free memory before removing a swap space from use.

All the swap spaces that are used automatically with swapon -a can be removed from use with swapoff -a; it looks at the file /etc/fstab to find what to remove. Any manually used swap spaces will remain in use.

Sometimes a lot of swap space can be in use even though there is a lot of free physical memory. This can happen for instance if at one point there is need to swap, but later a big process that occupied much of the physical memory terminates and frees the memory. The swapped-out data is not automatically swapped in until it is needed, so the physical memory may remain free for a long time. There is no need to worry about this, but it can be comforting to know what is happening.

6.4. Sharing swap spaces with other operating systems

Virtual memory is built into many operating systems. Since they each need it only when they are running, i.e., never at the same time, the swap spaces of all but the currently running one are being wasted. It would be more efficient for them to share a single swap space. This is possible, but can require a bit of hacking. The Tips-HOWTO at http://www.tldp.org/HOWTO/Tips-HOWTO.html, which contains some advice on how to implement this.

6.5. Allocating swap space

Some people will tell you that you should allocate twice as much swap space as you have physical memory, but this is a bogus rule. Here's how to do it properly:

• Estimate your total memory needs. This is the largest amount of memory you'll probably need at a time, that is the sum of the memory requirements of all the programs you want to run at the same time. This can be done by running at the same time all the programs you are likely to ever be running at the same time.

  For instance, if you want to run X, you should allocate about 8 MB for it, gcc wants several megabytes (some files need an unusually large amount, up to tens of megabytes, but usually about four should do), and so on. The kernel will use about a megabyte by itself, and the usual shells and other small utilities perhaps a few hundred kilobytes (say a megabyte together). There is no need to try to be exact, rough estimates are fine, but you might want to be on the pessimistic side.

  Remember that if there are going to be several people using the system at the same time, they are all going to consume memory. However, if two people run the same program at the same time, the total memory consumption is usually not double, since code pages and shared libraries exist only once.

  The free and ps commands are useful for estimating the memory needs.

• Add some security to the estimate in step 1. This is because estimates of program sizes will probably be wrong, because you'll probably forget some programs you want to run, and to make certain that you have some extra space just in case. A couple of megabytes should be fine. (It is better to allocate too much than too little swap space, but there's no need to over-do it and allocate the whole disk, since unused swap space is wasted space; see later about adding more swap.) Also, since it is nicer to deal with even numbers, you can round the value up to the next full megabyte.

• Based on the computations above, you know how much memory you'll be needing in total. So, in order to allocate swap space, you just need to subtract the size of your physical memory from the total memory needed, and you know how much swap space you need. (On some versions of UNIX, you need to allocate space for an image of the physical memory as well, so the amount computed in step 2 is what you need and you shouldn't do the subtraction.)

• If your calculated swap space is very much larger than your physical memory (more than a couple times larger), you should probably invest in more physical memory, otherwise performance will be too low.

It's a good idea to have at least some swap space, even if your calculations indicate that you need none. Linux uses swap space somewhat aggressively, so that as much physical memory as possible can be kept free. Linux will swap out memory pages that have not been used, even if the memory is not yet needed for anything. This avoids waiting for swapping when it is needed: the swapping can be done earlier, when the disk is otherwise idle.

Swap space can be divided among several disks. This can sometimes improve performance, depending on the relative speeds of the disks and the access patterns of the disks. You might want to experiment with a few schemes, but be aware that doing the experiments properly is quite difficult. You should not believe claims that any one scheme is superior to any other, since it won't always be true.

6.6. The buffer cache

Reading from a disk is very slow compared to accessing (real) memory. In addition, it is common to read the same part of a disk several times during relatively short periods of time. For example, one might first read an e-mail message, then read the letter into an editor when replying to it, then make the mail program read it again when copying it to a folder. Or, consider how often the command ls might be run on a system with many users. By reading the information from disk only once and then keeping it in memory until no longer needed, one can speed up all but the first read. This is called disk buffering, and the memory used for the purpose is called the buffer cache.

Since memory is, unfortunately, a finite, nay, scarce resource, the buffer cache usually cannot be big enough (it can't hold all the data one ever wants to use). When the cache fills up, the data that has been unused for the longest time is discarded and the memory thus freed is used for the new data.

Disk buffering works for writes as well. On the one hand, data that is written is often soon read again (e.g., a source code file is saved to a file, then read by the compiler), so putting data that is written in the cache is a good idea. On the other hand, by only putting the data into the cache, not writing it to disk at once, the program that writes runs quicker. The writes can then be done in the background, without slowing down the other programs.

Most operating systems have buffer caches (although they might be called something else), but not all of them work according to the above principles. Some are write-through: the data is written to disk at once (it is kept in the cache as well, of course). The cache is called write-back if the writes are done at a later time. Write-back is more efficient than write-through, but also a bit more prone to errors: if the machine crashes, or the power is cut at a bad moment, or the floppy is removed from the disk drive before the data in the cache waiting to be written gets written, the changes in the cache are usually lost. This might even mean that the filesystem (if there is one) is not in full working order, perhaps because the unwritten data held important changes to the bookkeeping information.

Because of this, you should never turn off the power without using a proper shutdown procedure or remove a floppy from the disk drive until it has been unmounted (if it was mounted) or after whatever program is using it has signaled that it is finished and the floppy drive light doesn't shine anymore. The sync command flushes the buffer, i.e., forces all unwritten data to be written to disk, and can be used when one wants to be sure that everything is safely written. In traditional UNIX systems, there is a program called update running in the background which does a sync every 30 seconds, so it is usually not necessary to use sync. Linux has an additional daemon, bdflush, which does a more imperfect sync more frequently to avoid the sudden freeze due to heavy disk I/O that sync sometimes causes.

Under Linux, bdflush is started by update. There is usually no reason to worry about it, but if bdflush happens to die for some reason, the kernel will warn about this, and you should start it by hand (/sbin/update).

The cache does not actually buffer files, but blocks, which are the smallest units of disk I/O (under Linux, they are usually 1 KB). This way, also directories, super blocks, other filesystem bookkeeping data, and non-filesystem disks are cached.

The effectiveness of a cache is primarily decided by its size. A small cache is next to useless: it will hold so little data that all cached data is flushed from the cache before it is reused. The critical size depends on how much data is read and written, and how often the same data is accessed. The only way to know is to experiment.

If the cache is of a fixed size, it is not very good to have it too big, either, because that might make the free memory too small and cause swapping (which is also slow). To make the most efficient use of real memory, Linux automatically uses all free RAM for buffer cache, but also automatically makes the cache smaller when programs need more memory.

Under Linux, you do not need to do anything to make use of the cache, it happens completely automatically. Except for following the proper procedures for shutdown and removing floppies, you do not need to worry about it.

7.1. System Resources

Being able to monitor the performance of your system is essential. If system resources become too low it can cause a lot of problems. System resources can be taken up by individual users, or by services your system may host such as email or web pages. The ability to know what is happening can help determine whether system upgrades are needed, or if some services need to be moved to another machine.

# top
 12:10:49  up 1 day,  3:47,  7 users,  load average: 0.23, 0.19, 0.10
125 processes: 105 sleeping, 2 running, 18 zombie, 0 stopped
CPU states:   5.1% user   1.1% system   0.0% nice   0.0% iowait  93.6% idle
Mem:   512716k av,  506176k used,    6540k free,       0k shrd,   21888k buff
Swap: 1044216k av,  161672k used,  882544k free                  199388k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
 2330 root      15   0  161M  70M  2132 S     4.9 14.0  1000m   0 X
 2605 weeksa    15   0  8240 6340  3804 S     0.3  1.2   1:12   0 kdeinit
 3413 weeksa    15   0  6668 5324  3216 R     0.3  1.0   0:20   0 kdeinit
18734 root      15   0  1192 1192   868 R     0.3  0.2   0:00   0 top
 1619 root      15   0   776  608   504 S     0.1  0.1   0:53   0 dhclient
    1 root      15   0   480  448   424 S     0.0  0.0   0:03   0 init
    2 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 keventd
    3 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kapmd
    4 root      35  19     0    0     0 SWN   0.0  0.0   0:00   0 ksoftirqd_CPU0
    9 root      25   0     0    0     0 SW    0.0  0.0   0:00   0 bdflush
    5 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kswapd
   10 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kupdated
   11 root      25   0     0    0     0 SW    0.0  0.0   0:00   0 mdrecoveryd
   15 root      15   0     0    0     0 SW    0.0  0.0   0:01   0 kjournald
   81 root      25   0     0    0     0 SW    0.0  0.0   0:00   0 khubd
 1188 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kjournald
 1675 root      15   0   604  572   520 S     0.0  0.1   0:00   0 syslogd
 1679 root      15   0   428  376   372 S     0.0  0.0   0:00   0 klogd
 1707 rpc       15   0   516  440   436 S     0.0  0.0   0:00   0 portmap
 1776 root      25   0   476  428   424 S     0.0  0.0   0:00   0 apmd
 1813 root      25   0   752  528   524 S     0.0  0.1   0:00   0 sshd
 1828 root      25   0   704  548   544 S     0.0  0.1   0:00   0 xinetd
 1847 ntp       15   0  2396 2396  2160 S     0.0  0.4   0:00   0 ntpd
 1930 root      24   0    76    4     0 S     0.0  0.0   0:00   0 rpc.rquotad

# iostat
Linux 2.4.20-24.9 (myhost)       12/23/2003

avg-cpu:  %user   %nice    %sys   %idle
          62.09    0.32    2.97   34.62

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
dev3-0            2.22        15.20        47.16    1546846    4799520

# iostat -x
Linux 2.4.20-24.9 (myhost)       12/23/2003

avg-cpu:  %user   %nice    %sys   %idle
          62.01    0.32    2.97   34.71

Device:  rrqm/s wrqm/s r/s  w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/hdc   0.00   0.00 .00 0.00   0.00   0.00  0.00  0.00     0.00     2.35  0.00  0.00 14.71
/dev/hda   1.13   4.50 .81 1.39  15.18  47.14  7.59 23.57    28.24     1.99 63.76 70.48 15.56
/dev/hda1  1.08   3.98 .73 1.27  14.49  42.05  7.25 21.02    28.22     0.44 21.82  4.97  1.00
/dev/hda2  0.00   0.51 .07 0.12   0.55   5.07  0.27  2.54    30.35     0.97 52.67 61.73  2.99
/dev/hda3  0.05   0.01 .02 0.00   0.14   0.02  0.07  0.01     8.51     0.00 12.55  2.95  0.01

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Dec22 ?        00:00:03 init
root         2     1  0 Dec22 ?        00:00:00 [keventd]
root         3     1  0 Dec22 ?        00:00:00 [kapmd]
root         4     1  0 Dec22 ?        00:00:00 [ksoftirqd_CPU0]
root         9     1  0 Dec22 ?        00:00:00 [bdflush]
root         5     1  0 Dec22 ?        00:00:00 [kswapd]
root         6     1  0 Dec22 ?        00:00:00 [kscand/DMA]
root         7     1  0 Dec22 ?        00:01:28 [kscand/Normal]
root         8     1  0 Dec22 ?        00:00:00 [kscand/HighMem]
root        10     1  0 Dec22 ?        00:00:00 [kupdated]
root        11     1  0 Dec22 ?        00:00:00 [mdrecoveryd]
root        15     1  0 Dec22 ?        00:00:01 [kjournald]
root        81     1  0 Dec22 ?        00:00:00 [khubd]
root      1188     1  0 Dec22 ?        00:00:00 [kjournald]
root      1675     1  0 Dec22 ?        00:00:00 syslogd -m 0
root      1679     1  0 Dec22 ?        00:00:00 klogd -x
rpc       1707     1  0 Dec22 ?        00:00:00 portmap
root      1813     1  0 Dec22 ?        00:00:00 /usr/sbin/sshd
ntp       1847     1  0 Dec22 ?        00:00:00 ntpd -U ntp
root      1930     1  0 Dec22 ?        00:00:00 rpc.rquotad
root      1934     1  0 Dec22 ?        00:00:00 [nfsd]
root      1942     1  0 Dec22 ?        00:00:00 [lockd]
root      1943     1  0 Dec22 ?        00:00:00 [rpciod]
root      1949     1  0 Dec22 ?        00:00:00 rpc.mountd
root      1961     1  0 Dec22 ?        00:00:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
root      2057     1  0 Dec22 ?        00:00:00 /usr/bin/spamd -d -c -a
root      2066     1  0 Dec22 ?        00:00:00 gpm -t ps/2 -m /dev/psaux
bin       2076     1  0 Dec22 ?        00:00:00 /usr/sbin/cannaserver -syslog -u bin
root      2087     1  0 Dec22 ?        00:00:00 crond
daemon    2195     1  0 Dec22 ?        00:00:00 /usr/sbin/atd
root      2215     1  0 Dec22 ?        00:00:11 /usr/sbin/rcd
weeksa    3414  3413  0 Dec22 pts/1    00:00:00 /bin/bash
weeksa    4342  3413  0 Dec22 pts/2    00:00:00 /bin/bash
weeksa   19121 18668  0 12:58 pts/2    00:00:00 ps -ef

# vmstat
   procs                      memory      swap          io     system      cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 0  0  0 181604  17000  26296 201120    0    2     8    24  149     9 61  3 36

7.2. Filesystem Usage

Many reports are currently talking about how cheap storage has gotten, but if you're like most of us it isn't cheap enough. Most of us have a limited amount of space, and need to be able to monitor it and control how it's used.

user@server:~> df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda3              5242904    759692   4483212  15% /
tmpfs                   127876         8    127868   1% /dev/shm
/dev/hda1               127351     33047     87729  28% /boot
/dev/hda9             10485816     33508  10452308   1% /home
/dev/hda8              5242904    932468   4310436  18% /srv
/dev/hda7              3145816     32964   3112852   2% /tmp
/dev/hda5              5160416    474336   4423928  10% /usr
/dev/hda6              3145816    412132   2733684  14% /var

user@server:~> df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/hda3                  0       0       0    -  /
tmpfs                  31969       5   31964    1% /dev/shm
/dev/hda1              32912      47   32865    1% /boot
/dev/hda9                  0       0       0    -  /home
/dev/hda8                  0       0       0    -  /srv
/dev/hda7                  0       0       0    -  /tmp
/dev/hda5             656640   26651  629989    5% /usr
/dev/hda6                  0       0       0    -  /var

user@server:~> du file.txt
1300    file.txt

user@server:~> du -h file.txt
1.3M     file.txt

user@server:~> du -h /usr/local
4.0K    /usr/local/games
16K     /usr/local/include/nessus/net
180K    /usr/local/include/nessus
208K    /usr/local/include
62M     /usr/local/lib/nessus/plugins/.desc
97M     /usr/local/lib/nessus/plugins
164K    /usr/local/lib/nessus/plugins_factory
97M     /usr/local/lib/nessus
12K     /usr/local/lib/pkgconfig
2.7M    /usr/local/lib/ladspa
104M    /usr/local/lib
112K    /usr/local/man/man1
4.0K    /usr/local/man/man2
4.0K    /usr/local/man/man3
4.0K    /usr/local/man/man4
16K     /usr/local/man/man5
4.0K    /usr/local/man/man

user@server:~> du -hs /usr/local
210M    /usr/local

7.3. Monitoring Users

Just because you're paranoid doesn't mean they AREN'T out to get you... Source Unknown

From time to time there are going to be occasions where you will want to know exactly what people are doing on your system. Maybe you notice that a lot of RAM is being used, or a lot of CPU activity. You are going to want to see who is on the system, what they are running, and what kind of resources they are using.

user@server:~>  who
bjones   pts/0        May 23 09:33
wally    pts/3        May 20 11:35
aweeks   pts/1        May 22 11:03
aweeks   pts/2        May 23 15:04

user@server:~> ps -u aweeks
	
20876 pts/1    00:00:00 bash
20904 pts/2    00:00:00 bash
20951 pts/2    00:00:00 ssh
21012 pts/1    00:00:00 ps

user@server:~> w
aweeks   :0        09:32   ?xdm?  30:09   0.02s -:0
aweeks   pts/0     09:33    5:49m  0.00s  0.82s kdeinit: kded
aweeks   pts/2     09:35    8.00s  0.55s  0.36s vi sag-0.9.sgml
aweeks   pts/1     15:03   59.00s  0.03s  0.03s /bin/bash

8.1. An overview of boots and shutdowns

The act of turning on a computer system and causing its operating system to be loaded is called booting. The name comes from an image of the computer pulling itself up from its bootstraps, but the act itself slightly more realistic.

During bootstrapping, the computer first loads a small piece of code called the bootstrap loader, which in turn loads and starts the operating system. The bootstrap loader is usually stored in a fixed location on a hard disk or a floppy. The reason for this two step process is that the operating system is big and complicated, but the first piece of code that the computer loads must be very small (a few hundred bytes), to avoid making the firmware unnecessarily complicated.

Different computers do the bootstrapping differently. For PCs, the computer (its BIOS) reads in the first sector (called the boot sector) of a floppy or hard disk. The bootstrap loader is contained within this sector. It loads the operating system from elsewhere on the disk (or from some other place).

After Linux has been loaded, it initializes the hardware and device drivers, and then runs init. init starts other processes to allow users to log in, and do things. The details of this part will be discussed below.

In order to shut down a Linux system, first all processes are told to terminate (this makes them close any files and do other necessary things to keep things tidy), then filesystems and swap areas are unmounted, and finally a message is printed to the console that the power can be turned off. If the proper procedure is not followed, terrible things can and will happen; most importantly, the filesystem buffer cache might not be flushed, which means that all data in it is lost and the filesystem on disk is inconsistent, and therefore possibly unusable.

8.2. The boot process in closer look

When a PC is booted, the BIOS will do various tests to check that everything looks all right, and will then start the actual booting. This process is called the power on self test , or POST for short. It will choose a disk drive (typically the first floppy drive, if there is a floppy inserted, otherwise the first hard disk, if one is installed in the computer; the order might be configurable, however) and will then read its very first sector. This is called the boot sector; for a hard disk, it is also called the master boot record, since a hard disk can contain several partitions, each with their own boot sectors.

The boot sector contains a small program (small enough to fit into one sector) whose responsibility is to read the actual operating system from the disk and start it. When booting Linux from a floppy disk, the boot sector contains code that just reads the first few hundred blocks (depending on the actual kernel size, of course) to a predetermined place in memory. On a Linux boot floppy, there is no filesystem, the kernel is just stored in consecutive sectors, since this simplifies the boot process. It is possible, however, to boot from a floppy with a filesystem, by using LILO, the LInux LOader, or GRUB, the GRand Unifying Bootloader.

When booting from the hard disk, the code in the master boot record will examine the partition table (also in the master boot record), identify the active partition (the partition that is marked to be bootable), read the boot sector from that partition, and then start the code in that boot sector. The code in the partition's boot sector does what a floppy disk's boot sector does: it will read in the kernel from the partition and start it. The details vary, however, since it is generally not useful to have a separate partition for just the kernel image, so the code in the partition's boot sector can't just read the disk in sequential order, it has to find the sectors wherever the filesystem has put them. There are several ways around this problem, but the most common way is to use a boot loader like LILO or GRUB. (The details about how to do this are irrelevant for this discussion, however; see the LILO or GRUB documentation for more information; it is most thorough.)

When booting, the bootloader will normally go right ahead and read in and boot the default kernel. It is also possible to configure the boot loader to be able to boot one of several kernels, or even other operating systems than Linux, and it is possible for the user to choose which kernel or operating system is to be booted at boot time. LILO, for example, can be configured so that if one holds down the alt, shift, or ctrl key at boot time (when LILO is loaded), LILO will ask what is to be booted and not boot the default right away. Alternatively, the bootloader can be configured so that it will always ask, with an optional timeout that will cause the default kernel to be booted.

It is also possible to give a kernel command line argument, after the name of the kernel or operating system. For a list of possible options you can read http://www.tldp.org/HOWTO/BootPrompt-HOWTO.html.

Booting from floppy and from hard disk have both their advantages, but generally booting from the hard disk is nicer, since it avoids the hassle of playing around with floppies. It is also faster. Most Linux distributions will setup the bootloader for you during the install process.

After the Linux kernel has been read into the memory, by whatever means, and is started for real, roughly the following things happen:

• The Linux kernel is installed compressed, so it will first uncompress itself. The beginning of the kernel image contains a small program that does this.

• If you have a super-VGA card that Linux recognizes and that has some special text modes (such as 100 columns by 40 rows), Linux asks you which mode you want to use. During the kernel compilation, it is possible to preset a video mode, so that this is never asked. This can also be done with LILO, GRUB or rdev.

• After this, the kernel checks what other hardware there is (hard disks, floppies, network adapters, etc), and configures some of its device drivers appropriately; while it does this, it outputs messages about its findings. For example, when I boot, I it looks like this:

  LILO boot: Loading linux. Console: colour EGA+ 80x25, 8 virtual consoles Serial driver version 3.94 with no serial options enabled tty00 at 0x03f8 (irq = 4) is a 16450 tty01 at 0x02f8 (irq = 3) is a 16450 lp_init: lp1 exists (0), using polling driver Memory: 7332k/8192k available (300k kernel code, 384k reserved, 176k data) Floppy drive(s): fd0 is 1.44M, fd1 is 1.2M Loopback device init Warning WD8013 board not found at i/o = 280. Math coprocessor using irq13 error reporting. Partition check: hda: hda1 hda2 hda3 VFS: Mounted root (ext filesystem). Linux version 0.99.pl9-1 (root@haven) 05/01/93 14:12:20

  The exact texts are different on different systems, depending on the hardware, the version of Linux being used, and how it has been configured.

• Then the kernel will try to mount the root filesystem. The place is configurable at compilation time, or any time with rdev or the bootloader. The filesystem type is detected automatically. If the mounting of the root filesystem fails, for example because you didn't remember to include the corresponding filesystem driver in the kernel, the kernel panics and halts the system (there isn't much it can do, anyway).

  The root filesystem is usually mounted read-only (this can be set in the same way as the place). This makes it possible to check the filesystem while it is mounted; it is not a good idea to check a filesystem that is mounted read-write.

• After this, the kernel starts the program init (located in /sbin/init) in the background (this will always become process number 1). init does various startup chores. The exact things it does depends on how it is configured; see Section 2.3.1 for more information (not yet written). It will at least start some essential background daemons.

• init then switches to multi-user mode, and starts a getty for virtual consoles and serial lines. getty is the program which lets people log in via virtual consoles and serial terminals. init may also start some other programs, depending on how it is configured.

• After this, the boot is complete, and the system is up and running normally.

8.3. More about shutdowns

It is important to follow the correct procedures when you shut down a Linux system. If you fail do so, your filesystems probably will become trashed and the files probably will become scrambled. This is because Linux has a disk cache that won't write things to disk at once, but only at intervals. This greatly improves performance but also means that if you just turn off the power at a whim the cache may hold a lot of data and that what is on the disk may not be a fully working filesystem (because only some things have been written to the disk).

Another reason against just flipping the power switch is that in a multi-tasking system there can be lots of things going on in the background, and shutting the power can be quite disastrous. By using the proper shutdown sequence, you ensure that all background processes can save their data.

The command for properly shutting down a Linux system is shutdown. It is usually used in one of two ways.

If you are running a system where you are the only user, the usual way of using shutdown is to quit all running programs, log out on all virtual consoles, log in as root on one of them (or stay logged in as root if you already are, but you should change to root's home directory or the root directory, to avoid problems with unmounting), then give the command shutdown -h now (substitute now with a plus sign and a number in minutes if you want a delay, though you usually don't on a single user system).

Alternatively, if your system has many users, use the command shutdown -h +time message, where time is the time in minutes until the system is halted, and message is a short explanation of why the system is shutting down.

# shutdown -h +10 'We will install a new 
disk.  System should
> be back on-line in three hours.'
#

Broadcast message from root (ttyp0) Wed Aug  2 01:03:25 1995...

We will install a new disk.  System should
be back on-line in three hours.
The system is going DOWN for system halt in 10 minutes !!

When the real shutting down starts after any delays, all filesystems (except the root one) are unmounted, user processes (if anybody is still logged in) are killed, daemons are shut down, all filesystem are unmounted, and generally everything settles down. When that is done, init prints out a message that you can power down the machine. Then, and only then, should you move your fingers towards the power switch.

Sometimes, although rarely on any good system, it is impossible to shut down properly. For instance, if the kernel panics and crashes and burns and generally misbehaves, it might be completely impossible to give any new commands, hence shutting down properly is somewhat difficult, and just about everything you can do is hope that nothing has been too severely damaged and turn off the power. If the troubles are a bit less severe (say, somebody hit your keyboard with an axe), and the kernel and the update program still run normally, it is probably a good idea to wait a couple of minutes to give update a chance to flush the buffer cache, and only cut the power after that.

In the old days, some people like to shut down using the command sync three times, waiting for the disk I/O to stop, then turn off the power. If there are no running programs, this is equivalent to using shutdown. However, it does not unmount any filesystems and this can lead to problems with the ext2fs ``clean filesystem'' flag. The triple-sync method is not recommended.

(In case you're wondering: the reason for three syncs is that in the early days of UNIX, when the commands were typed separately, that usually gave sufficient time for most disk I/O to be finished.)

8.4. Rebooting

Rebooting means booting the system again. This can be accomplished by first shutting it down completely, turning power off, and then turning it back on. A simpler way is to ask shutdown to reboot the system, instead of merely halting it. This is accomplished by using the -r option to shutdown, for example, by giving the command shutdown -r now.

Most Linux systems run shutdown -r now when ctrl-alt-del is pressed on the keyboard. This reboots the system. The action on ctrl-alt-del is configurable, however, and it might be better to allow for some delay before the reboot on a multiuser machine. Systems that are physically accessible to anyone might even be configured to do nothing when ctrl-alt-del is pressed.

8.5. Single user mode

The shutdown command can also be used to bring the system down to single user mode, in which no one can log in, but root can use the console. This is useful for system administration tasks that can't be done while the system is running normally.

8.6. Emergency boot floppies

It is not always possible to boot a computer from the hard disk. For example, if you make a mistake in configuring LILO, you might make your system unbootable. For these situations, you need an alternative way of booting that will always work (as long as the hardware works). For typical PCs, this means booting from the floppy drive.

Most Linux distributions allow one to create an emergency boot floppy during installation. It is a good idea to do this. However, some such boot disks contain only the kernel, and assume you will be using the programs on the distribution's installation disks to fix whatever problem you have. Sometimes those programs aren't enough; for example, you might have to restore some files from backups made with software not on the installation disks.

Thus, it might be necessary to create a custom root floppy as well. The Bootdisk HOWTO by Graham Chapman contains instructions for doing this. You can find this HOWTO at http://www.tldp.org/HOWTO/Bootdisk-HOWTO/index.html. You must, of course, remember to keep your emergency boot and root floppies up to date.

You can't use the floppy drive you use to mount the root floppy for anything else. This can be inconvenient if you only have one floppy drive. However, if you have enough memory, you can configure your boot floppy to load the root disk to a ramdisk (the boot floppy's kernel needs to be specially configured for this). Once the root floppy has been loaded into the ramdisk, the floppy drive is free to mount other disks.

9.1. init comes first

init is one of those programs that are absolutely essential to the operation of a Linux system, but that you still can mostly ignore. A good Linux distribution will come with a configuration for init that will work for most systems, and on these systems there is nothing you need to do about init. Usually, you only need to worry about init if you hook up serial terminals, dial-in (not dial-out) modems, or if you want to change the default run level.

When the kernel has started itself (has been loaded into memory, has started running, and has initialized all device drivers and data structures and such), it finishes its own part of the boot process by starting a user level program, init. Thus, init is always the first process (its process number is always 1).

The kernel looks for init in a few locations that have been historically used for it, but the proper location for it (on a Linux system) is /sbin/init. If the kernel can't find init, it tries to run /bin/sh, and if that also fails, the startup of the system fails.

When init starts, it finishes the boot process by doing a number of administrative tasks, such as checking filesystems, cleaning up /tmp, starting various services, and starting a getty for each terminal and virtual console where users should be able to log in (see Chapter 10).

After the system is properly up, init restarts getty for each terminal after a user has logged out (so that the next user can log in). init also adopts orphan processes: when a process starts a child process and dies before its child, the child immediately becomes a child of init. This is important for various technical reasons, but it is good to know it, since it makes it easier to understand process lists and process tree graphs. There are a few variants of init available. Most Linux distributions use sysvinit (written by Miquel van Smoorenburg), which is based on the System V init design. The BSD versions of Unix have a different init. The primary difference is run levels: System V has them, BSD does not (at least traditionally). This difference is not essential. We'll look at sysvinit only.

9.2. Configuring init to start getty: the /etc/inittab file

When it starts up, init reads the /etc/inittab configuration file. While the system is running, it will re-read it, if sent the HUP signal (kill -HUP 1); this feature makes it unnecessary to boot the system to make changes to the init configuration take effect.

The /etc/inittab file is a bit complicated. We'll start with the simple case of configuring getty lines. Lines in /etc/inittab consist of four colon-delimited fields:

id:runlevels:action:process

1:2345:respawn:/sbin/getty 9600 tty1

Different versions of getty are run differently. Consult your manual page, and make sure it is the correct manual page.

If you wanted to add terminals or dial-in modem lines to a system, you'd add more lines to /etc/inittab, one for each terminal or dial-in line. For more details, see the manual pages init, inittab, and getty.

If a command fails when it starts, and init is configured to restart it, it will use a lot of system resources: init starts it, it fails, init starts it, it fails, init starts it, it fails, and so on, ad infinitum. To prevent this, init will keep track of how often it restarts a command, and if the frequency grows to high, it will delay for five minutes before restarting again.

9.3. Run levels

A run level is a state of init and the whole system that defines what system services are operating. Run levels are identified by numbers. Some system administrators use run levels to define which subsystems are working, e.g., whether X is running, whether the network is operational, and so on. Others have all subsystems always running or start and stop them individually, without changing run levels, since run levels are too coarse for controlling their systems. You need to decide for yourself, but it might be easiest to follow the way your Linux distribution does things.

The following table defines how most Linux Distributions define the different run levels. However, run-levels 2 through 5 can be modified to suit your own tastes.

Services that get started at a certain runtime are determined by the contents of the various rcN.d directories. Most distributions locate these directories either at /etc/init.d/rcN.d or /etc/rcN.d. (Replace the N with the run-level number.)

In each run-level you will find a series of if links pointing to start-up scripts located in /etc/init.d. The names of these links all start as either K or S, followed by a number. If the name of the link starts with an S, then that indicates the service will be started when you go into that run level. If the name of the link starts with a K, the service will be killed (if running).

The number following the K or S indicates the order the scripts will be run. Here is a sample of what an /etc/init.d/rc3.d may look like.

# ls -l /etc/init.d/rc3.d
lrwxrwxrwx  1 root root 10 2004-11-29 22:09 K12nfsboot -> ../nfsboot
lrwxrwxrwx  1 root root  6 2005-03-29 13:42 K15xdm -> ../xdm
lrwxrwxrwx  1 root root  9 2004-11-29 22:08 S01pcmcia -> ../pcmcia
lrwxrwxrwx  1 root root  9 2004-11-29 22:06 S01random -> ../random
lrwxrwxrwx  1 root root 11 2005-03-01 11:56 S02firewall -> ../firewall
lrwxrwxrwx  1 root root 10 2004-11-29 22:34 S05network -> ../network
lrwxrwxrwx  1 root root  9 2004-11-29 22:07 S06syslog -> ../syslog
lrwxrwxrwx  1 root root 10 2004-11-29 22:09 S08portmap -> ../portmap
lrwxrwxrwx  1 root root  9 2004-11-29 22:07 S08resmgr -> ../resmgr
lrwxrwxrwx  1 root root  6 2004-11-29 22:09 S10nfs -> ../nfs
lrwxrwxrwx  1 root root 12 2004-11-29 22:40 S12alsasound -> ../alsasound
lrwxrwxrwx  1 root root  8 2004-11-29 22:09 S12fbset -> ../fbset
lrwxrwxrwx  1 root root  7 2004-11-29 22:10 S12sshd -> ../sshd
lrwxrwxrwx  1 root root  8 2005-02-01 09:24 S12xntpd -> ../xntpd
lrwxrwxrwx  1 root root  7 2004-12-02 20:34 S13cups -> ../cups
lrwxrwxrwx  1 root root  6 2004-11-29 22:09 S13kbd -> ../kbd
lrwxrwxrwx  1 root root 13 2004-11-29 22:10 S13powersaved -> ../powersaved
lrwxrwxrwx  1 root root  9 2004-11-29 22:09 S14hwscan -> ../hwscan
lrwxrwxrwx  1 root root  7 2004-11-29 22:10 S14nscd -> ../nscd
lrwxrwxrwx  1 root root 10 2004-11-29 22:10 S14postfix -> ../postfix
lrwxrwxrwx  1 root root  6 2005-02-04 13:27 S14smb -> ../smb
lrwxrwxrwx  1 root root  7 2004-11-29 22:10 S15cron -> ../cron
lrwxrwxrwx  1 root root  8 2004-12-22 20:35 S15smbfs -> ../smbfs

How run levels start are configured in /etc/inittab by lines like the following:

l2:2:wait:/etc/init.d/rc 2

The command in the fourth field does all the hard work of setting up a run level. It starts services that aren't already running, and stops services that shouldn't be running in the new run level any more. Exactly what the command is, and how run levels are configured, depends on the Linux distribution.

When init starts, it looks for a line in /etc/inittab that specifies the default run level:

id:2:initdefault:

While the system is running, the telinit command can change the run level. When the run level is changed, init runs the relevant command from /etc/inittab.

9.4. Special configuration in /etc/inittab

The /etc/inittab has some special features that allow init to react to special circumstances. These special features are marked by special keywords in the third field. Some examples:

9.5. Booting in single user mode

An important run level is single user mode (run level 1), in which only the system administrator is using the machine and as few system services, including logins, as possible are running. Single user mode is necessary for a few administrative tasks, such as running fsck on a /usr partition, since this requires that the partition be unmounted, and that can't happen, unless just about all system services are killed.

A running system can be taken to single user mode by using telinit to request run level 1. At bootup, it can be entered by giving the word single or emergency on the kernel command line: the kernel gives the command line to init as well, and init understands from that word that it shouldn't use the default run level. (The kernel command line is entered in a way that depends on how you boot the system.)

Booting into single user mode is sometimes necessary so that one can run fsck by hand, before anything mounts or otherwise touches a broken /usr partition (any activity on a broken filesystem is likely to break it more, so fsck should be run as soon as possible).

The bootup scripts init runs will automatically enter single user mode, if the automatic fsck at bootup fails. This is an attempt to prevent the system from using a filesystem that is so broken that fsck can't fix it automatically. Such breakage is relatively rare, and usually involves a broken hard disk or an experimental kernel release, but it's good to be prepared.

As a security measure, a properly configured system will ask for the root password before starting the shell in single user mode. Otherwise, it would be simple to just enter a suitable line to LILO to get in as root. (This will break if /etc/passwd has been broken by filesystem problems, of course, and in that case you'd better have a boot floppy handy.)

10.1. Logins via terminals

Section 2.3.2 shows how logins happen via terminals. First, init makes sure there is a getty program for the terminal connection (or console). getty listens at the terminal and waits for the user to notify that he is ready to login in (this usually means that the user must type something). When it notices a user, getty outputs a welcome message (stored in /etc/issue), and prompts for the username, and finally runs the login program. login gets the username as a parameter, and prompts the user for the password. If these match, login starts the shell configured for the user; else it just exits and terminates the process (perhaps after giving the user another chance at entering the username and password). init notices that the process terminated, and starts a new getty for the terminal.

Note that the only new process is the one created by init (using the fork system call); getty and login only replace the program running in the process (using the exec system call).

A separate program, for noticing the user, is needed for serial lines, since it can be (and traditionally was) complicated to notice when a terminal becomes active. getty also adapts to the speed and other settings of the connection, which is important especially for dial-in connections, where these parameters may change from call to call.

There are several versions of getty and init in use, all with their good and bad points. It is a good idea to learn about the versions on your system, and also about the other versions (you could use the Linux Software Map to search them). If you don't have dial-ins, you probably don't have to worry about getty, but init is still important.

10.2. Logins via the network

Two computers in the same network are usually linked via a single physical cable. When they communicate over the network, the programs in each computer that take part in the communication are linked via a virtual connection, a sort of imaginary cable. As far as the programs at either end of the virtual connection are concerned, they have a monopoly on their own cable. However, since the cable is not real, only imaginary, the operating systems of both computers can have several virtual connections share the same physical cable. This way, using just a single cable, several programs can communicate without having to know of or care about the other communications. It is even possible to have several computers use the same cable; the virtual connections exist between two computers, and the other computers ignore those connections that they don't take part in.

That's a complicated and over-abstracted description of the reality. It might, however, be good enough to understand the important reason why network logins are somewhat different from normal logins. The virtual connections are established when there are two programs on different computers that wish to communicate. Since it is in principle possible to login from any computer in a network to any other computer, there is a huge number of potential virtual communications. Because of this, it is not practical to start a getty for each potential login.

There is a single process inetd (corresponding to getty) that handles all network logins. When it notices an incoming network login (i.e., it notices that it gets a new virtual connection to some other computer), it starts a new process to handle that single login. The original process remains and continues to listen for new logins.

To make things a bit more complicated, there is more than one communication protocol for network logins. The two most important ones are telnet and rlogin. In addition to logins, there are many other virtual connections that may be made (for FTP, Gopher, HTTP, and other network services). It would be ineffective to have a separate process listening for a particular type of connection, so instead there is only one listener that can recognize the type of the connection and can start the correct type of program to provide the service. This single listener is called inetd; see the Linux Network Administrators' Guide for more information.

10.3. What login does

The login program takes care of authenticating the user (making sure that the username and password match), and of setting up an initial environment for the user by setting permissions for the serial line and starting the shell.

Part of the initial setup is outputting the contents of the file /etc/motd (short for message of the day) and checking for electronic mail. These can be disabled by creating a file called .hushlogin in the user's home directory.

If the file /etc/nologin exists, logins are disabled. That file is typically created by shutdown and relatives. login checks for this file, and will refuse to accept a login if it exists. If it does exist, login outputs its contents to the terminal before it quits.

login logs all failed login attempts in a system log file (via syslog). It also logs all logins by root. Both of these can be useful when tracking down intruders.

Currently logged in people are listed in /var/run/utmp. This file is valid only until the system is next rebooted or shut down; it is cleared when the system is booted. It lists each user and the terminal (or network connection) he is using, along with some other useful information. The who, w, and other similar commands look in utmp to see who are logged in.

All successful logins are recorded into /var/log/wtmp. This file will grow without limit, so it must be cleaned regularly, for example by having a weekly cron job to clear it. The last command browses wtmp.

Both utmp and wtmp are in a binary format (see the utmp manual page); it is unfortunately not convenient to examine them without special programs.

10.4. X and xdm

XXX X implements logins via xdm; also: xterm -ls

TO BE ADDED

10.5. Access control

The user database is traditionally contained in the /etc/passwd file. Some systems use shadow passwords, and have moved the passwords to /etc/shadow. Sites with many computers that share the accounts use NIS or some other method to store the user database; they might also automatically copy the database from one central location to all other computers.

The user database contains not only the passwords, but also some additional information about the users, such as their real names, home directories, and login shells. This other information needs to be public, so that anyone can read it. Therefore the password is stored encrypted. This does have the drawback that anyone with access to the encrypted password can use various cryptographic methods to guess it, without trying to actually log into the computer. Shadow passwords try to avoid this by moving the password into another file, which only root can read (the password is still stored encrypted). However, installing shadow passwords later onto a system that did not support them can be difficult.

With or without passwords, it is important to make sure that all passwords in a system are good, i.e., not easily guessed. The crack program can be used to crack passwords; any password it can find is by definition not a good one. While crack can be run by intruders, it can also be run by the system administrator to avoid bad passwords. Good passwords can also be enforced by the passwd program; this is in fact more effective in CPU cycles, since cracking passwords requires quite a lot of computation.

The user group database is kept in /etc/group; for systems with shadow passwords, there can be a /etc/shadow.group.

root usually can't login via most terminals or the network, only via terminals listed in the /etc/securetty file. This makes it necessary to get physical access to one of these terminals. It is, however, possible to log in via any terminal as any other user, and use the su command to become root.

10.6. Shell startup

When an interactive login shell starts, it automatically executes one or more pre-defined files. Different shells execute different files; see the documentation of each shell for further information.

Most shells first run some global file, for example, the Bourne shell (/bin/sh) and its derivatives execute /etc/profile; in addition, they execute .profile in the user's home directory. /etc/profile allows the system administrator to have set up a common user environment, especially by setting the PATH to include local command directories in addition to the normal ones. On the other hand, .profile allows the user to customize the environment to his own tastes by overriding, if necessary, the default environment.

11.1. What's an account?

When a computer is used by many people it is usually necessary to differentiate between the users, for example, so that their private files can be kept private. This is important even if the computer can only be used by a single person at a time, as with most microcomputers. Thus, each user is given a unique username, and that name is used to log in.

There's more to a user than just a name, however. An account is all the files, resources, and information belonging to one user. The term hints at banks, and in a commercial system each account usually has some money attached to it, and that money vanishes at different speeds depending on how much the user stresses the system. For example, disk space might have a price per megabyte and day, and processing time might have a price per second.

11.2. Creating a user

The Linux kernel itself treats users are mere numbers. Each user is identified by a unique integer, the user id or uid, because numbers are faster and easier for a computer to process than textual names. A separate database outside the kernel assigns a textual name, the username, to each user id. The database contains additional information as well.

To create a user, you need to add information about the user to the user database, and create a home directory for him. It may also be necessary to educate the user, and set up a suitable initial environment for him.

Most Linux distributions come with a program for creating accounts. There are several such programs available. Two command line alternatives are adduser and useradd; there may be a GUI tool as well. Whatever the program, the result is that there is little if any manual work to be done. Even if the details are many and intricate, these programs make everything seem trivial. However, Section 11.2.4 describes how to do it by hand.

11.3. Changing user properties

There are a few commands for changing various properties of an account (i.e., the relevant field in /etc/passwd):

Other tasks need to be done by hand. For example, to change the username, you need to edit /etc/passwd directly (with vipw, remember). Likewise, to add or remove the user to more groups, you need to edit /etc/group (with vigr). Such tasks tend to be rare, however, and should be done with caution: for example, if you change the username, e-mail will no longer reach the user, unless you also create a mail alias.

11.4. Removing a user

To remove a user, you first remove all his files, mailboxes, mail aliases, print jobs, cron and at jobs, and all other references to the user. Then you remove the relevant lines from /etc/passwd and /etc/group (remember to remove the username from all groups it's been added to). It may be a good idea to first disable the account (see below), before you start removing stuff, to prevent the user from using the account while it is being removed.

Remember that users may have files outside their home directory. The find command can find them:

find / -user username

Some Linux distributions come with special commands to do this; look for deluser or userdel. However, it is easy to do it by hand as well, and the commands might not do everything.

11.5. Disabling a user temporarily

It is sometimes necessary to temporarily disable an account, without removing it. For example, the user might not have paid his fees, or the system administrator may suspect that a cracker has got the password of that account.

The best way to disable an account is to change its shell into a special program that just prints a message. This way, whoever tries to log into the account, will fail, and will know why. The message can tell the user to contact the system administrator so that any problems may be dealt with.

It would also be possible to change the username or password to something else, but then the user won't know what is going on. Confused users mean more work.

A simple way to create the special programs is to write `tail scripts':

#!/usr/bin/tail +2
This account has been closed due to a security breach.
Please call 555-1234 and wait for the men in black to arrive.

If user billg is suspected of a security breach, the system administrator would do something like this:

# chsh -s 
/usr/local/lib/no-login/security billg
# su - tester
This account has been closed due to a security breach.
Please call 555-1234 and wait for the men in black to arrive.
#

Tail scripts should be kept in a separate directory, so that their names don't interfere with normal user commands.

12.1. On the importance of being backed up

Your data is valuable. It will cost you time and effort re-create it, and that costs money or at least personal grief and tears; sometimes it can't even be re-created, e.g., if it is the results of some experiments. Since it is an investment, you should protect it and take steps to avoid losing it.

There are basically four reasons why you might lose data: hardware failures, software bugs, human action, or natural disasters. Although modern hardware tends to be quite reliable, it can still break seemingly spontaneously. The most critical piece of hardware for storing data is the hard disk, which relies on tiny magnetic fields remaining intact in a world filled with electromagnetic noise. Modern software doesn't even tend to be reliable; a rock solid program is an exception, not a rule. Humans are quite unreliable, they will either make a mistake, or they will be malicious and destroy data on purpose. Nature might not be evil, but it can wreak havoc even when being good. All in all, it is a small miracle that anything works at all.

Backups are a way to protect the investment in data. By having several copies of the data, it does not matter as much if one is destroyed (the cost is only that of the restoration of the lost data from the backup).

It is important to do backups properly. Like everything else that is related to the physical world, backups will fail sooner or later. Part of doing backups well is to make sure they work; you don't want to notice that your backups didn't work. Adding insult to injury, you might have a bad crash just as you're making the backup; if you have only one backup medium, it might destroyed as well, leaving you with the smoking ashes of hard work. Or you might notice, when trying to restore, that you forgot to back up something important, like the user database on a 15000 user site. Best of all, all your backups might be working perfectly, but the last known tape drive reading the kind of tapes you used was the one that now has a bucketful of water in it.

When it comes to backups, paranoia is in the job description.

12.2. Selecting the backup medium

The most important decision regarding backups is the choice of backup medium. You need to consider cost, reliability, speed, availability, and usability.

Cost is important, since you should preferably have several times more backup storage than what you need for the data. A cheap medium is usually a must.

Reliability is extremely important, since a broken backup can make a grown man cry. A backup medium must be able to hold data without corruption for years. The way you use the medium affects it reliability as a backup medium. A hard disk is typically very reliable, but as a backup medium it is not very reliable, if it is in the same computer as the disk you are backing up.

Speed is usually not very important, if backups can be done without interaction. It doesn't matter if a backup takes two hours, as long as it needs no supervision. On the other hand, if the backup can't be done when the computer would otherwise be idle, then speed is an issue.

Availability is obviously necessary, since you can't use a backup medium if it doesn't exist. Less obvious is the need for the medium to be available even in the future, and on computers other than your own. Otherwise you may not be able to restore your backups after a disaster.

Usability is a large factor in how often backups are made. The easier it is to make backups, the better. A backup medium mustn't be hard or boring to use.

The typical alternatives are floppies and tapes. Floppies are very cheap, fairly reliable, not very fast, very available, but not very usable for large amounts of data. Tapes are cheap to somewhat expensive, fairly reliable, fairly fast, quite available, and, depending on the size of the tape, quite comfortable.

There are other alternatives. They are usually not very good on availability, but if that is not a problem, they can be better in other ways. For example, magneto-optical disks can have good sides of both floppies (they're random access, making restoration of a single file quick) and tapes (contain a lot of data).

12.3. Selecting the backup tool

There are many tools that can be used to make backups. The traditional UNIX tools used for backups are tar, cpio, and dump. In addition, there are large number of third party packages (both freeware and commercial) that can be used. The choice of backup medium can affect the choice of tool.

tar and cpio are similar, and mostly equivalent from a backup point of view. Both are capable of storing files on tapes, and retrieving files from them. Both are capable of using almost any media, since the kernel device drivers take care of the low level device handling and the devices all tend to look alike to user level programs. Some UNIX versions of tar and cpio may have problems with unusual files (symbolic links, device files, files with very long pathnames, and so on), but the Linux versions should handle all files correctly.

dump is different in that it reads the filesystem directly and not via the filesystem. It is also written specifically for backups; tar and cpio are really for archiving files, although they work for backups as well.

Reading the filesystem directly has some advantages. It makes it possible to back files up without affecting their time stamps; for tar and cpio, you would have to mount the filesystem read-only first. Directly reading the filesystem is also more effective, if everything needs to be backed up, since it can be done with much less disk head movement. The major disadvantage is that it makes the backup program specific to one filesystem type; the Linux dump program understands the ext2 filesystem only.

dump also directly supports backup levels (which we'll be discussing below); with tar and cpio this has to be implemented with other tools.

A comparison of the third party backup tools is beyond the scope of this book. The Linux Software Map lists many of the freeware ones.

12.4. Simple backups

A simple backup scheme is to back up everything once, then back up everything that has been modified since the previous backup. The first backup is called a full backup, the subsequent ones are incremental backups. A full backup is often more laborious than incremental ones, since there is more data to write to the tape and a full backup might not fit onto one tape (or floppy). Restoring from incremental backups can be many times more work than from a full one. Restoration can be optimized so that you always back up everything since the previous full backup; this way, backups are a bit more work, but there should never be a need to restore more than a full backup and an incremental backup.

If you want to make backups every day and have six tapes, you could use tape 1 for the first full backup (say, on a Friday), and tapes 2 to 5 for the incremental backups (Monday through Thursday). Then you make a new full backup on tape 6 (second Friday), and start doing incremental ones with tapes 2 to 5 again. You don't want to overwrite tape 1 until you've got a new full backup, lest something happens while you're making the full backup. After you've made a full backup to tape 6, you want to keep tape 1 somewhere else, so that when your other backup tapes are destroyed in the fire, you still have at least something left. When you need to make the next full backup, you fetch tape 1 and leave tape 6 in its place.

If you have more than six tapes, you can use the extra ones for full backups. Each time you make a full backup, you use the oldest tape. This way you can have full backups from several previous weeks, which is good if you want to find an old, now deleted file, or an old version of a file.

# tar --create --file /dev/ftape 
/usr/src
tar: Removing leading / from absolute path names in 
the archive
#

# tar -cMf /dev/fd0H1440 
/usr/src
tar: Removing leading / from absolute path names in 
the archive
Prepare volume #2 for /dev/fd0H1440 and hit return:
#

# tar --compare --verbose -f 
/dev/ftape
usr/src/
usr/src/linux
usr/src/linux-1.2.10-includes/
....
#

# tar --create --newer '8 Sep 1995' 
--file /dev/ftape /usr/src 
--verbose
tar: Removing leading / from absolute path names in 
the archive
usr/src/
usr/src/linux-1.2.10-includes/
usr/src/linux-1.2.10-includes/include/
usr/src/linux-1.2.10-includes/include/linux/
usr/src/linux-1.2.10-includes/include/linux/modules/
usr/src/linux-1.2.10-includes/include/asm-generic/
usr/src/linux-1.2.10-includes/include/asm-i386/
usr/src/linux-1.2.10-includes/include/asm-mips/
usr/src/linux-1.2.10-includes/include/asm-alpha/
usr/src/linux-1.2.10-includes/include/asm-m68k/
usr/src/linux-1.2.10-includes/include/asm-sparc/
usr/src/patch-1.2.11.gz
#

# tar --extract --same-permissions 
--verbose --file 
/dev/fd0H1440
usr/src/
usr/src/linux
usr/src/linux-1.2.10-includes/
usr/src/linux-1.2.10-includes/include/
usr/src/linux-1.2.10-includes/include/linux/
usr/src/linux-1.2.10-includes/include/linux/hdreg.h
usr/src/linux-1.2.10-includes/include/linux/kernel.h
...
#

# tar xpvf /dev/fd0H1440 
usr/src/linux-1.2.10-includes/include/linux/hdreg.h
usr/src/linux-1.2.10-includes/include/linux/hdreg.h
#

# tar --list --file 
/dev/fd0H1440
usr/src/
usr/src/linux
usr/src/linux-1.2.10-includes/
usr/src/linux-1.2.10-includes/include/
usr/src/linux-1.2.10-includes/include/linux/
usr/src/linux-1.2.10-includes/include/linux/hdreg.h
usr/src/linux-1.2.10-includes/include/linux/kernel.h
...
#

12.5. Multilevel backups

The simple backup method outlined in the previous section is often quite adequate for personal use or small sites. For more heavy duty use, multilevel backups are more appropriate.

The simple method has two backup levels: full and incremental backups. This can be generalized to any number of levels. A full backup would be level 0, and the different levels of incremental backups levels 1, 2, 3, etc. At each incremental backup level you back up everything that has changed since the previous backup at the same or a previous level.

The purpose for doing this is that it allows a longer backup history cheaply. In the example in the previous section, the backup history went back to the previous full backup. This could be extended by having more tapes, but only a week per new tape, which might be too expensive. A longer backup history is useful, since deleted or corrupted files are often not noticed for a long time. Even a version of a file that is not very up to date is better than no file at all.

With multiple levels the backup history can be extended more cheaply. For example, if we buy ten tapes, we could use tapes 1 and 2 for monthly backups (first Friday each month), tapes 3 to 6 for weekly backups (other Fridays; note that there can be five Fridays in one month, so we need four more tapes), and tapes 7 to 10 for daily backups (Monday to Thursday). With only four more tapes, we've been able to extend the backup history from two weeks (after all daily tapes have been used) to two months. It is true that we can't restore every version of each file during those two months, but what we can restore is often good enough.

Figure 12-1 shows which backup level is used each day, and which backups can be restored from at the end of the month.

Backup levels can also be used to keep filesystem restoration time to a minimum. If you have many incremental backups with monotonously growing level numbers, you need to restore all of them if you need to rebuild the whole filesystem. Instead you can use level numbers that aren't monotonous, and keep down the number of backups to restore.

To minimize the number of tapes needed to restore, you could use a smaller level for each incremental tape. However, then the time to make the backups increases (each backup copies everything since the previous full backup). A better scheme is suggested by the dump manual page and described by the table XX (efficient-backup-levels). Use the following succession of backup levels: 3, 2, 5, 4, 7, 6, 9, 8, 9, etc. This keeps both the backup and restore times low. The most you have to backup is two day's worth of work. The number of tapes for a restore depends on how long you keep between full backups, but it is less than in the simple schemes.

A fancy scheme can reduce the amount of labor needed, but it does mean there are more things to keep track of. You must decide if it is worth it.

dump has built-in support for backup levels. For tar and cpio it must be implemented with shell scripts.

12.6. What to back up

You want to back up as much as possible. The major exception is software that can be easily reinstalled, but even they may have configuration files that it is important to back up, lest you need to do all the work to configure them all over again. Another major exception is the /proc filesystem; since that only contains data that the kernel always generates automatically, it is never a good idea to back it up. Especially the /proc/kcore file is unnecessary, since it is just an image of your current physical memory; it's pretty large as well.

Gray areas include the news spool, log files, and many other things in /var. You must decide what you consider important.

The obvious things to back up are user files (/home) and system configuration files (/etc, but possibly other things scattered all over the filesystem).

12.7. Compressed backups

Backups take a lot of space, which can cost quite a lot of money. To reduce the space needed, the backups can be compressed. There are several ways of doing this. Some programs have support for compression built in; for example, the --gzip (-z) option for GNU tar pipes the whole backup through the gzip compression program, before writing it to the backup medium.

Unfortunately, compressed backups can cause trouble. Due to the nature of how compression works, if a single bit is wrong, all the rest of the compressed data will be unusable. Some backup programs have some built in error correction, but no method can handle a large number of errors. This means that if the backup is compressed the way GNU tar does it, with the whole output compressed as a unit, a single error makes all the rest of the backup lost. Backups must be reliable, and this method of compression is not a good idea.

An alternative way is to compress each file separately. This still means that the one file is lost, but all other files are unharmed. The lost file would have been corrupted anyway, so this situation is not much worse than not using compression at all. The afio program (a variant of cpio) can do this.

Compression takes some time, which may make the backup program unable to write data fast enough for a tape drive. This can be avoided by buffering the output (either internally, if the backup program if smart enough, or by using another program), but even that might not work well enough. This should only be a problem on slow computers.

14.1. The concept of localtime

Time measurement is based on mostly regular natural phenomena, such as alternating light and dark periods caused by the rotation of the planet. The total time taken by two successive periods is constant, but the lengths of the light and dark period vary. One simple constant is noon.

Noon is the time of the day when the Sun is at its highest position. Since (according to recent research) the Earth is round, noon happens at different times in different places. This leads to the concept of local time. Humans measure time in many units, most of which are tied to natural phenomena like noon. As long as you stay in the same place, it doesn't matter that local times differ.

As soon as you need to communicate with distant places, you'll notice the need for a common time. In modern times, most of the places in the world communicate with most other places in the world, so a global standard for measuring time has been defined. This time is called universal time (UT or UTC, formerly known as Greenwich Mean Time or GMT, since it used to be local time in Greenwich, England). When people with different local times need to communicate, they can express times in universal time, so that there is no confusion about when things should happen.

Each local time is called a time zone. While geography would allow all places that have noon at the same time have the same time zone, politics makes it difficult. For various reasons, many countries use daylight savings time, that is, they move their clocks to have more natural light while they work, and then move the clocks back during winter. Other countries do not do this. Those that do, do not agree when the clocks should be moved, and they change the rules from year to year. This makes time zone conversions definitely non-trivial.

Time zones are best named by the location or by telling the difference between local and universal time. In the US and some other countries, the local time zones have a name and a three letter abbreviation. The abbreviations are not unique, however, and should not be used unless the country is also named. It is better to talk about the local time in, say, Helsinki, than about East European time, since not all countries in Eastern Europe follow the same rules.

Linux has a time zone package that knows about all existing time zones, and that can easily be updated when the rules change. All the system administrator needs to do is to select the appropriate time zone. Also, each user can set his own time zone; this is important since many people work with computers in different countries over the Internet. When the rules for daylight savings time change in your local time zone, make sure you'll upgrade at least that part of your Linux system. Other than setting the system time zone and upgrading the time zone data files, there is little need to bother about time.

14.2. The hardware and software clocks

A personal computer has a battery driven hardware clock. The battery ensures that the clock will work even if the rest of the computer is without electricity. The hardware clock can be set from the BIOS setup screen or from whatever operating system is running.

The Linux kernel keeps track of time independently from the hardware clock. During the boot, Linux sets its own clock to the same time as the hardware clock. After this, both clocks run independently. Linux maintains its own clock because looking at the hardware is slow and complicated.

The kernel clock always shows universal time. This way, the kernel does not need to know about time zones at all. The simplicity results in higher reliability and makes it easier to update the time zone information. Each process handles time zone conversions itself (using standard tools that are part of the time zone package).

The hardware clock can be in local time or in universal time. It is usually better to have it in universal time, because then you don't need to change the hardware clock when daylight savings time begins or ends (UTC does not have DST). Unfortunately, some PC operating systems, including MS-DOS, Windows, and OS/2, assume the hardware clock shows local time. Linux can handle either, but if the hardware clock shows local time, then it must be modified when daylight savings time begins or ends (otherwise it wouldn't show local time).

14.3. Showing and setting time

In Linux, the system time zone is determined by the symbolic link /etc/localtime. This link points to a time zone data file that describes the local time zone. The time zone data files are located at either /usr/lib/zoneinfo or /usr/share/zoneinfo depending on what distribution of Linux you use.

For example, on a SuSE system located in New Jersey the /etc/localtime link would point to /usr/share/zoneinfo/US/Eastern. On a Debian system the /etc/localtime link would point to /usr/lib/zoneinfo/US/Eastern.

If you fail to find the zoneinfo directory in either the /usr/lib or /usr/share directories, either do a find /usr -print | grep zoneinfo or consult your distribution's documentation.

What happens when you have a users located in a different timezone? A user can change his private time zone by setting the TZ environment variable. If it is unset, the system time zone is assumed. The syntax of the TZ variable is described in the tzset manual page.

The date command shows the current date and time. For example:

$ date
Sun Jul 14 21:53:41 EET DST 1996
$

$ date -u
Sun Jul 14 18:53:42 UTC 1996
$

# date 07142157
Sun Jul 14 21:57:00 EET DST 1996
# date
Sun Jul 14 21:57:02 EET DST 1996
#

Beware of the time command. This is not used to get the system time. Instead it's used to time how long something takes. Refer to the time man page.

date only shows or sets the software clock. The clock commands synchronizes the hardware and software clocks. It is used when the system boots, to read the hardware clock and set the software clock. If you need to set both clocks, you first set the software clock with date, and then the hardware clock with clock -w.

The -u option to clock tells it that the hardware clock is in universal time. You must use the -u option correctly. If you don't, your computer will be quite confused about what the time is.

The clocks should be changed with care. Many parts of a Unix system require the clocks to work correctly. For example, the cron daemon runs commands periodically. If you change the clock, it can be confused of whether it needs to run the commands or not. On one early Unix system, someone set the clock twenty years into the future, and cron wanted to run all the periodic commands for twenty years all at once. Current versions of cron can handle this correctly, but you should still be careful. Big jumps or backward jumps are more dangerous than smaller or forward ones.

14.4. When the clock is wrong

The Linux software clock is not always accurate. It is kept running by a periodic timer interrupt generated by PC hardware. If the system has too many processes running, it may take too long to service the timer interrupt, and the software clock starts slipping behind. The hardware clock runs independently and is usually more accurate. If you boot your computer often (as is the case for most systems that aren't servers), it will usually keep fairly accurate time.

If you need to adjust the hardware clock, it is usually simplest to reboot, go into the BIOS setup screen, and do it from there. This avoids all trouble that changing system time might cause. If doing it via BIOS is not an option, set the new time with date and clock (in that order), but be prepared to reboot, if some part of the system starts acting funny.

Another method would be to use either hwclock -w or hwclock --systohc to sync the hardware clock to the software clock. If you want to sync your software clock to your hardware clock then you would use hwclock -s or hwclock --hwtosys. For more information on this command read man hwclock.

14.5. NTP - Network Time Protocol

A networked computer (even if just over a modem) can check its own clock automatically by comparing it to the time on another computer known to keep accurate time. Network Time Protocol (or NTP) does exactly that. It is a method of verifying and correcting your computer's time by synchronizing it with a another system. With NTP your system's time can be maintained to within milliseconds of Coordinated Universal Time. Visit http://www.time.gov/about.html for more info.

For more casual Linux users, this is just a nice luxury. At my home all our clocks are set based upon what my Linux system says the time is. For larger organizations this "luxury" can become essential. Being able to search log files for events based upon time can make life a lot easier and take a lot of the "guess work" out of debugging.

Another example of how important NTP can be is with a SAN. Some SAN's require NTP be configured and running properly to allow for proper synchronization over filesystem usage, and proper timestamp control. Some SANs (and some applications) can become confused when dealing with files that have timestamps that are in the future.

Most Linux distributions come with a NTP package of some kind, either a .deb or .rpm package. You can use that to install NTP, or you can download the source files from http://www.ntp.org/downloads.html and compile it yourself. Either way, the basic configuration is the same.

14.6. Basic NTP configuration

The NTP program is configured using either the /etc/ntp.conf or /etc/xntp.conf file depending on what distribution of Linux you have. I won't go into too much detail on how to configure NTP. Instead I'll just cover the basics.

An example of a basic ntp.conf file would look like:

# --- GENERAL CONFIGURATION ---
server  aaa.bbb.ccc.ddd
server  127.127.1.0
fudge   127.127.1.0 stratum 10

# Drift file.

driftfile /etc/ntp/drift
	

The most basic ntp.conf file will simply list 2 servers, one that it wishes to synchronize with, and a pseudo IP address for itself (in this case 127.127.1.0). The pseudo IP is used in case of network problems or if the remote NTP server goes down. NTP will synchronize against itself until the it can start synchronizing with the remote server again. It is recommended that you list at least 2 remote servers that you can synchronize against. One will act as a primary server and the other as a backup.

You should also list a location for a drift file. Over time NTP will "learn" the system clock's error rate and automatically adjust for it.

The restrict option can be used to provide better control and security over what NTP can do, and who can effect it. For example:

# Prohibit general access to this service.
restrict default ignore

# Permit systems on this network to synchronize with this
# time service. But not modify our time.
restrict aaa.bbb.ccc.ddd nomodify

# Allow the following unrestricted access to ntpd

restrict aaa.bbb.ccc.ddd
restrict 127.0.0.1

NTP slowly corrects your systems time. Be patient! A simple test is to change your system clock by 10 minutes before you go to bed and then check it when you get up. The time should be correct.

Many people get the idea that instead of running the NTP daemon, they should just setup a cron job job to periodically run the ntpdate command. There are 2 main disadvantages of using this method.

The first is that ntpdate does a "brute force" method of changing the time. So if your computer's time is off my 5 minutes, it immediately corrects it. In some environments, this can cause problems if time drastically changes. For example, if you are using time sensitive security software, you can inadvertently kill someones access. The NTP daemon slowly changes the time to avoid causing this kind of disruption.

The other reason is that the NTP daemon can be configured to try to learn your systems time drift and then automatically adjust for it.

14.7. NTP Toolkit

There are a number of utilities available to check if NTP is doing it's job. The ntpq -p command will print out your system's current time status.

# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*cudns.cit.corne ntp0.usno.navy.  2 u  832 1024  377   43.208    0.361   2.646
 LOCAL(0)        LOCAL(0)        10 l   13   64  377    0.000    0.000   0.008

The ntpdc -c loopinfo will display how far off the system time is in seconds, based upon the last time the remote server was contacted.

# ntpdc -c loopinfo
offset:               -0.004479 s
frequency:            133.625 ppm
poll adjust:          30
watchdog timer:       404 s
	

ntpdc -c kerninfo will display the current remaining correction.

# ntpdc -c kerninfo
pll offset:           -0.003917 s
pll frequency:        133.625 ppm
maximum error:        0.391414 s
estimated error:      0.003676 s
status:               0001  pll
pll time constant:    6
precision:            1e-06 s
frequency tolerance:  512 ppm
pps frequency:        0.000 ppm
pps stability:        512.000 ppm
pps jitter:           0.0002 s
calibration interval: 4 s
calibration cycles:   0
jitter exceeded:      0
stability exceeded:   0
calibration errors:   0
	

A slightly more different version of ntpdc -c kerninfo is ntptime

# ntptime
ntp_gettime() returns code 0 (OK)
  time c35e2cc7.879ba000  Thu, Nov 13 2003 11:16:07.529, (.529718),
  maximum error 425206 us, estimated error 3676 us
ntp_adjtime() returns code 0 (OK)
  modes 0x0 (),
  offset -3854.000 us, frequency 133.625 ppm, interval 4 s,
  maximum error 425206 us, estimated error 3676 us,
  status 0x1 (PLL),
  time constant 6, precision 1.000 us, tolerance 512 ppm,
  pps frequency 0.000 ppm, stability 512.000 ppm, jitter 200.000 us,
  intervals 0, jitter exceeded 0, stability exceeded 0, errors 0.

Yet another way to see how well NTP is working is with the ntpdate -d command. This will contact an NTP server and determine the time difference but not change your system's time.

# ntpdate -d 132.236.56.250
13 Nov 14:43:17 ntpdate[29631]: ntpdate 4.1.1c-rc1@1.836 Thu Feb 13 12:17:20 EST 2003 (1)
transmit(132.236.56.250)
receive(132.236.56.250)
transmit(132.236.56.250)
receive(132.236.56.250)
transmit(132.236.56.250)
receive(132.236.56.250)
transmit(132.236.56.250)
receive(132.236.56.250)
transmit(132.236.56.250)
server 132.236.56.250, port 123
stratum 2, precision -17, leap 00, trust 000
refid [192.5.41.209], delay 0.06372, dispersion 0.00044
transmitted 4, in filter 4
reference time:    c35e5998.4a46cfc8  Thu, Nov 13 2003 14:27:20.290
originate timestamp: c35e5d55.d69a6f82  Thu, Nov 13 2003 14:43:17.838
transmit timestamp:  c35e5d55.d16fc9bc  Thu, Nov 13 2003 14:43:17.818
filter delay:  0.06522  0.06372  0.06442  0.06442
         0.00000  0.00000  0.00000  0.00000
filter offset: 0.000036 0.001020 0.000527 0.000684
         0.000000 0.000000 0.000000 0.000000
delay 0.06372, dispersion 0.00044
offset 0.001020

13 Nov 14:43:17 ntpdate[29631]: adjust time server 132.236.56.250 offset 0.001020 sec

If you want actually watch the system synchronize you can use ntptrace.

# ntptrace 132.236.56.250
cudns.cit.cornell.edu: stratum 2, offset -0.003278, synch distance 0.02779
truetime.ntp.com: stratum 1, offset -0.014363, synch distance 0.00000, refid 'ACTS'

If you need your system time synchronized immediately you can use the ntpdate remote-servername to force a synchronization. No waiting!

# ntpdate 132.236.56.250
13 Nov 14:56:28 ntpdate[29676]: adjust time server 132.236.56.250 offset -0.003151 sec

14.8. Some known NTP servers

A list of public NTP servers can be obtained from: http://www.eecis.udel.edu/~mills/ntp/servers.html. Please read the usage information on the page prior so using a server. Not all servers have the available bandwidth to allow a large number of systems synchronizing against them. Therefore it is a good idea to contact a system's administrator prior to using his/her server for NTP services.

14.9. NTP Links

More detailed information on NTP can be obtained from the NTP homepage:http://www.ntp.org.

Or from http://www.ntp.org/ntpfaq/NTP-a-faq.htm

18.1. Newsgroups and Mailing Lists

This guide cannot teach you everything about Linux. There just isn't enough space. It is almost inevitable that at some point you will find something you need to do, that isn't covered in this (or any other) document at the LDP.

One of the nicest things about Linux is the large number of forums devoted to it. There are forums relating to almost all facets of Linux ranging from newbie FAQs to in depth kernel development issues. To receive the most from them, there are a few things you can do.

18.2. IRC

IRC (Internet Relay Chat) is not covered in the Eric Raymond document, but IRC can also be an excellent way of finding the answers you need. However it does require some practice in asking questions in the right way. Most IRC networks have busy #linux channels and if the answer to your question is contained in the man pages, or in the HOWTOs then expect to be told to go read them. The rule about typing in clear and grammatical English still applies.

Most of what has been said about newsgroups and mailing lists is still relevant for IRC, with a the following additions