Linux Loadable Kernel Module HOWTO

2.1. Terminology

Loadable kernel modules are often called just kernel modules or just modules, but those are rather misleading terms because there are lots of kinds of modules in the world and various pieces built into the base kernel can easily be called modules. We use the term loadable kernel module or LKM for the particular kinds of modules this HOWTO is about.

Some people think of LKMs as outside of the kernel. They speak of LKMs communicating with the kernel. This is a mistake; LKMs (when loaded) are very much part of the kernel. The correct term for the part of the kernel that is bound into the image that you boot, i.e. all of the kernel except the LKMs, is "base kernel." LKMs communicate with the base kernel.

In some other operating systems, the equivalent of a Linux LKM is called a "kernel extension."

Now what is "Linux"? Well, first of all, the name is used for two entirely different things, and only one of them is really relevant here:

1. The kernel and related items distributed as a package by Linus Torvalds.

2. A class of operating systems that traditionally are based on the Linux kernel.

Only the first of these is really useful in discussing LKMs. But even choosing this definition, people are often confused when it comes to LKMs. Is an LKM part of Linux or not? Though an LKM is always part of the kernel, it is part of Linux if it is distributed in the Linux kernel package, and not otherwise. Thus, if you have loaded into your kernel a device driver LKM that came with your device, you can't, strictly speaking, say that your kernel is Linux. Rather, it's a slight extension of Linux. As you might expect, it is commonplace to use the name "Linux" approximately -- Lots of variations on Linux are in use and are widely distributed, and referred to as "Linux." In this document, though, we will stick to the strictest definition in the interest of clarity.

2.2. History of Loadable Kernel Modules

LKMs did not exist in Linux in the beginning. Anything we use an LKM for today was built into the base kernel at kernel build time instead. LKMs have been around at least since Linux 1.2 (1995).

Device drivers and such were always quite modular, though. When LKMs were invented, only a small amount of work was needed on these modules to make them buildable as LKMs. However, it had to be done on each and every one, so it took some time. Since about 2000, virtually everything that makes sense as an LKM has at least had the option of being an LKM.

2.3. The Case For Loadable Kernel Modules

You often have a choice between putting a module into the kernel by loading it as an LKM or binding it into the base kernel. LKMs have a lot of advantages over binding into the base kernel and I recommend them wherever possible.

One advantage is that you don't have to rebuild your kernel as often. This saves you time and spares you the possibility of introducing an error in rebuilding and reinstalling the base kernel. Once you have a working base kernel, it is good to leave it untouched as long as possible.

Another advantage is that LKMs help you diagnose system problems. A bug in a device driver which is bound into the kernel can stop your system from booting at all. And it can be really hard to tell which part of the base kernel caused the trouble. If the same device driver is an LKM, though, the base kernel is up and running before the device driver even gets loaded. If your system dies after the base kernel is up and running, it's an easy matter to track the problem down to the trouble-making device driver and just not load that device driver until you fix the problem.

LKMs can save you memory, because you have to have them loaded only when you're actually using them. All parts of the base kernel stay loaded all the time. And in real storage, not just virtual storage.

LKMs are much faster to maintain and debug. What would require a full reboot to do with a filesystem driver built into the kernel, you can do with a few quick commands with LKMs. You can try out different parameters or even change the code repeatedly in rapid succession, without waiting for a boot.

LKMs are not slower, by the way, than base kernel modules. Calling either one is simply a branch to the memory location where it resides. [1]

Sometimes you have to build something into the base kernel instead of making it an LKM. Anything that is necessary to get the system up far enough to load LKMs must obviously be built into the base kernel. For example, the driver for the disk drive that contains the root filesystem must be built into the base kernel.

2.4. What LKMs Can't Do

There is a tendency to think of LKMs like user space programs. They do share a lot of their properties, but LKMs are definitely not user space programs. They are part of the kernel. As such, they have free run of the system and can easily crash it.

2.5. What LKMs Are Used For

There are six main things LKMs are used for:

• Device drivers. A device driver is designed for a specific piece of hardware. The kernel uses it to communicate with that piece of hardware without having to know any details of how the hardware works. For example, there is a device driver for ATA disk drives. There is one for NE2000 compatible Ethernet cards. To use any device, the kernel must contain a device driver for it.

• Filesystem drivers. A filesystem driver interprets the contents of a filesystem (which is typically the contents of a disk drive) as files and directories and such. There are lots of different ways of storing files and directories and such on disk drives, on network servers, and in other ways. For each way, you need a filesystem driver. For example, there's a filesystem driver for the ext2 filesystem type used almost universally on Linux disk drives. There is one for the MS-DOS filesystem too, and one for NFS.

• System calls. User space programs use system calls to get services from the kernel. For example, there are system calls to read a file, to create a new process, and to shut down the system. Most system calls are integral to the system and very standard, so are always built into the base kernel (no LKM option). But you can invent a system call of your own and install it as an LKM. Or you can decide you don't like the way Linux does something and override an existing system call with an LKM of your own.

• Network drivers. A network driver interprets a network protocol. It feeds and consumes data streams at various layers of the kernel's networking function. For example, if you want an IPX link in your network, you would use the IPX driver.

• TTY line disciplines. These are essentially augmentations of device drivers for terminal devices.

• Executable interpreters. An executable interpreter loads and runs an executable. Linux is designed to be able to run executables in various formats, and each must have its own executable interpreter.

5.1. Could Not Find Kernel Version...

A common error is to try to insert an object file which is not an LKM. For example, you configure your kernel to have the USB core module bound into the base kernel instead of generated as an LKM. In that case, you end up with a file usbcore.o, which looks pretty much the same as the usbcore.o you would get if you built it as an LKM. But you can't insmod that file.

So do you get an error message telling you that you should have configured the kernel to make USB core function an LKM? Of course not. This is Unix, and explanatory error messages are seen as a sign of weakness. The error message is

$ insmod usbcore.o
usbcore.o: couldn't find the kernel version this module was compiled for

What insmod is telling you is that it looked in usbcore.o for a piece of information any legitimate LKM would have -- the kernel version with which the LKM was intended to be used -- and it didn't find it. We know now that the reason it didn't find it is that the file isn't an LKM. See Section 10.2 for information on how you can see what insmod is seeing and confirm that the file is not in fact an LKM.

If this is a module you created yourself with the intention of it being an LKM, the next question you have is: Why isn't an LKM? The most usual cause of this is that you did not include linux/module.h at the top of your source code and/or did not define the MODULE> macro. MODULE is intended to be set via the compile command (-DMODULE) and determine whether the compilation produces an LKM or an object file for binding into the base kernel. If your module is like most modern modules and can be built only as an LKM, then you should just define it in your source code (#define MODULE) before you include include/module.h.

5.2. What Happens When An LKM Loads

So you've successfully loaded an LKM, and verified that via /proc/modules. But how do you know it's working? That's up to the LKM, and varies according to what kind of LKM it is, but here are some of the more common actions of an LKM upon being loaded.

The first thing a device driver LKM does after loading (which is what the module would do at boot time if it were bound into the base kernel) is usually to search the system for a device it knows how to drive. Just how it does this search varies from one driver to the next, and can usually be controlled by module parameters. But in any case, if the driver doesn't find any device it is capable of driving, it causes the load to fail. Otherwise, the driver registers itself as the driver for a particular major number and you can start using the device it found via a device special file that specifies that major number. It may also register itself as the handler for the interrupt level that the device uses. It may also send setup commands to the device, so you may see lights blink or something like that.

You can see that a device driver has registered itself in the file /proc/devices. You can see that the device driver is handling the device's interrupts in /proc/interrupts.

A nice device driver issues kernel messages telling what devices it found and is prepared to drive. (Kernel messages in most systems end up on the console and in the file /var/log/messages. You can also display recent ones with the dmesg program). Some drivers, however, are silent. A nice device driver also gives you (in kernel messages) some details of its search when it fails to find a device, but many just fail the load without explanation, and what you get is a list of guesses from insmod as to what the problem might have been.

A network device (interface) driver works similarly, except that the LKM registers a device name of its choosing (e.g. eth0) rather than a major number. You can see the currently registered network device names in /proc/net/dev

A filesystem driver, upon loading, registers itself as the driver for a filesystem type of a certain name. For example, the msdos driver registers itself as the driver for the filesystem type named msdos. (LKM authors typically name the LKM the same as the filesystem type it will drive).

5.3. Intelligent Loading Of LKMs - Modprobe

Once you have module loading and unloading figured out using insmod and rmmod, you can let the system do more of the work for you by using the higher level program modprobe. See the modprobe man page for details.

The main thing that modprobe does is automatically load the prerequisites of an LKM you request. It does this with the help of a file that you create with depmod and keep on your system.

Example:

modprobe msdos

This performs an insmod of msdos.o, but before that does an insmod of fat.o, since you have to have fat.o loaded before you can load msdos.o.

The other major thing modprobe does for you is to find the object module containing the LKM given just the name of the LKM. For example, modprobe msdos might load /lib/2.4.2-2/fs/msdos.o. In fact, modprobe's argument may be a totally symbolic name that you have associated with some actual module. For example, modprobe eth0 loads the appropriate network device driver to create and drive your eth0 device, assuming you set that up properly in modules.conf. Check out the man pages for modprobe and the configuration file modules.conf (usually /etc/modules.conf) for details on the search rules modprobe uses.

modprobe is especially important because it is by default the program that the kernel module loader uses to load an LKM on demand. So if you use automatic module loading, you will need to set up modules.conf properly or things will not work. See Section 5.4.

depmod scans your LKM object files (typically all the .o files in the appropriate /lib/modules subdirectory) and figures out which LKMs prerequire (refer to symbols in) other LKMs. It generates a dependency file (typically named modules.dep), which you normally keep in /lib/modules for use by modprobe.

You can use modprobe to remove stacks of LKMs as well.

Via the LKM configuration file (typically /etc/modules.conf), you can fine tune the dependencies and do other fancy things to control LKM selections. And you can specify programs to run when you insert and remove LKMs, for example to initialize a device driver.

If you are maintaining one system and memory is not in short supply, it is probably easier to avoid modprobe and the various files and directories it needs, and just do raw insmods in a startup script.

5.4. Automatic LKM Loading and Unloading

# echo "sbin/mymodprobe" >/proc/sys/kernel/modprobe

alias eth0 3c59x

5.5. /proc/modules

To see the presently loaded LKMs, do

cat /proc/modules

You see a line like

serial                   24484   0

The left column is the name of the LKM, which is normally the name of the object file from which you loaded it, minus the ".o" suffix. You can, however, choose any name you like with an option on insmod.

The "24484" is the size in bytes of the LKM in memory.

The "0" is the use count. It tells how many things presently depend on the LKM being loaded. Typical "things" are open devices or mounted fileystems. It is important because you cannot remove an LKM unless the use count is zero. The LKM itself maintains this count, but the module manager uses it to decide whether to permit an unload.

There is an exception to the above description of the use count. You may see -1 in the use count column. What that means is that this LKM does not use counts to determine when it is OK to unload. Instead, the LKM has registered a subroutine that the module manager can call that will return an indication of whether or not it is OK to unload the LKM. In this case, the LKM ought to provide you with some custom interface, and some documentation, to determine when the LKM is free to be unloaded.

Do not confuse use count with "dependencies", which are described below.

Here is another example, with more information:

lp                      5280   0 (unused)
parport_pc              7552   1
parport                 7600   1 [lp parport_pc]

You cannot unload an LKM that has dependencies. But you can remove those dependencies by unloading the dependent LKMs.

The "(unused)" legend means the LKM has never been used, i.e. it has never been in a state where it could not be unloaded. The kernel tracks this information for one simple reason: to assist in automatic LKM unloading policy. In a system where LKMs are loaded and unloaded automatically (see Section 5.4), you don't want to automatically load an LKM and then, before the guy who needed it loaded has a chance to use it, unload it because it is not in use.

Here is something you won't normally see:

mydriver                8154   0 (deleted)

There are similar statuses "initializing" and "uninitialized".

The legend "(autoclean)" refers to the autoclean flag, discussed in Section 5.4.

5.6. Where Are My LKM Files On My System?

The LKM world is flexible enough that the files you need to load could live just about anywhere on your system, but there is a convention that most systems follow: The LKM .o files are in the directory /lib/modules, divided into subdirectories. There is one subdirectory for each version of the kernel, since LKMs are specific to a kernel (see Section 6). Each subdirectory contains a complete set of LKMs.

The subdirectory name is the value you get from the uname --release command, for example 2.2.19. Section 6.3 tells how you control that value.

When you build Linux, a standard make modules and make modules_install should install all the LKMs that are part of Linux in the proper release subdirectory.

If you build a lot of kernels, another organization may be more helpful: keep the LKMs together with the base kernel and other kernel-related files in a subdirectory of /boot. The only drawback of this is that you cannot have /boot reside on a tiny disk partition. In some systems, /boot is on a special tiny "boot partition" and contains only enough files to get the system up to the point that it can mount other filesystems.

6.1. Some LKMs Prerequire Other LKMs

One reason you get this is because you have not loaded another LKM that contains instructions or data that your LKM needs to access. A primary purpose of modprobe is to avoid this failure. See Section 5.3.

6.2. An LKM Must Match The Base Kernel

The designers of loadable kernel modules realized there would be a problem with having the kernel in multiple files, possibly distributed independently of one another. What if the LKM mydriver.o was written and compiled to work with the Linux 1.2.1 base kernel, and then someone tried to load it into a Linux 1.2.2 kernel? What if there was a change between 1.2.1 and 1.2.2 in the way a kernel subroutine that mydriver.o calls works? These are internal kernel subroutines, so what's to stop them from changing from one release to the next? You could end up with a broken kernel.

To address this problem, the creators of LKMs endowed them with a kernel version number. The special .modinfo section of the mydriver.o object file in this example has "1.2.1" in it because it was compiled using header files from Linux 1.2.1. Try to load it into a 1.2.2 kernel and insmod notices the mismatch and fails, telling you that you have a kernel version mismatch.

But wait. What's the chance that there really is an incompatibility between Linux 1.2.1 and 1.2.2 that will affect mydriver.o? mydriver.o only calls a few subroutines and accesses a few data structures. Surely they don't change with every minor release. Must we recompile every LKM against the header files for the particular kernel into which we want to insert it?

To ease this burden, insmod has a -f option that "forces" insmod to ignore the kernel version mismatch and insert the module anyway. Because it is so unusual for there to be a significant difference between any two kernel versions, I recommend you always use -f. You will, however, still get a warning message about the mismatch. There's no way to shut that off.

But LKM designers still wanted to address the problem of incompatible changes that do occasionally happen. So they invented a very clever way to allow the LKM insertion process to be sensitive to the actual content of each kernel subroutine the LKM uses. It's called symbol versioning (or sometimes less clearly, "module versioning."). It's optional, and you select it when you configure the kernel via the "CONFIG_MODVERSIONS" kernel configuration option.

When you build a base kernel or LKM with symbol versioning, the various symbols exported for use by LKMs get defined as macros. The definition of the macro is the same symbol name plus a hexadecimal hash value of the parameter and return value types for the subroutine named by the symbol (based on an analysis by the program genksyms of the source code for the subroutine). So let's look at the register_chrdev subroutine. register_chrdev is a subroutine in the base kernel that device driver LKMs often call. With symbol versioning, there is a C macro definition like

  #define register_chrdev register_chrdev_Rc8dc8350

This macro definition is in effect both in the C source file that defines register_chrdev and in any C source file that refers to register_chrdev, so while your eyes see register_chrdev as you read the code, the C preprocessor knows that the function is really called register_chrdev_Rc8dc8350.

What is the meaning of that garbage suffix? It is a hash of the data types of the parameters and return value of register_chrdev. No two combinations of parameter and return value types have the same hash value.

So let's say someone adds a paramater to register_chrdev between Linux 1.2.1 and Linux 1.2.2. In 1.2.1, register_chrdev is a macro for register_chrdev_Rc8dc8350, but in 1.2.2, it is a macro for register_chrdev_R12f8dc01. In mydriver.o, compiled with Linux 1.2.1 header files, there is an external reference to register_chrdev_Rc8dc8350, but there is no such symbol exported by the 1.2.2 base kernel. Instead, the 1.2.2 base kernel exports a symbol register_chrdev_R12f8dc01.

So if you try to insmod this 1.2.1 mydriver.o into this 1.2.2 base kernel, you will fail. And the error message isn't one about mismatched kernel versions, but simply "unresolved symbol reference."

As clever as this is, it actually works against you sometimes. The way genksyms works, it often generates different hash values for parameter lists that are essentially the same.

And symbol versioning doesn't even guarantee compatibility. It catches only a small subset of the kinds of changes in the definition of a function that can make it not backward compatible. If the way register_chrdev interprets one of its parameters changes in a non-backward-compatible way, its version suffix won't change -- the parameter still has the same C type.

And there's no way an option like -f on insmod can get around this.

So it is generally not wise to use symbol versioning.

Of course, if you have a base kernel that was compiled with symbol versioning, then you must have all your LKMs compiled likewise, and vice versa. Otherwise, you're guaranteed to get those "unresolved symbol reference" errors.

6.3. If You Run Multiple Kernels

Now that we've seen how you often have different versions of an LKM for different base kernels, the question arises as to what to do about a system that has multiple kernel versions (i.e. you can choose a kernel at boot time). You want to make sure that the LKMs built for Kernel A get inserted when you boot Kernel A, but the LKMs built for Kernel B get inserted when you boot Kernel B.

In particular, whenever you upgrade your kernel, if you're smart, you keep both the new kernel and the old kernel on the system until you're sure the new one works.

The most common way to do this is with the LKM-hunting feature of modprobe. modprobe understands the conventional LKM file organization described in Section 5.6 and loads LKMs from the appropriate subdirectory depending on the kernel that is running.

You set the uname --release value, which is the name of the subdirectory in which modprobe looks, by editing the main kernel makefile when you build the kernel and setting the VERSION, PATCHLEVEL, SUBLEVEL, and EXTRAVERSION variables at the top.

6.4. SMP symbols

Besides the checksum mentioned above, the symbol version prefix contains "smp" if the symbol is defined in or referenced by code that was built for symmetric multiprocessing (SMP) machines. That means it was built for use on a system that may have more than one CPU. You choose whether to build in SMP capability or not via the Linux kernel configuration process (make config, etc.), to wit with the CONFIG_SMP configuration option.

So if you use symbol versioning, you will get unresolved symbols if the base kernel was built with SMP capability and the LKM you're inserting was not, or vice versa.

If you don't use symbol versioning, never mind.

Note that there's generally no reason to omit SMP capability from a kernel, even if you have only one CPU. Just because the capability is there doesn't mean you have to have multiple CPUs. However, there are some machines on which the SMP-capable kernel will not boot because it reaches the conclusion that there are zero CPUs!

6.5. You Are Not Licensed To Access The Symbol

The copyright owners of some kernel code license their programs to the public to make and use copies, but only in restricted ways. For example, the license may say you may only call your copy of the program from a program which is similarly licensed to the public.

(Is that confusing? Here's an example: Bob writes an LKM that provides data compression subroutines to other LKMs. He licenses his program to the public under the GNU Public License (GPL). According to some interpretations, that license says if you make a copy of Bob's LKM, you can't allow Mary's LKM to call its compression subroutines if Mary does not supply her source code to the world too. The idea is to encourage Mary to open up her source code).

To support and enforce such a license, the licensor can cause his program to export symbols under a special name that is the real name of the symbol plus the prefix "GPLONLY". A naive loader of a client LKM would not be able to resolve those symbols. Example: Bob's LKM provides the service bobsService() and declares it to be a GPL symbol. The LKM consequently exports bobsService() under the name GPLONLY_bobsService. If Mary's LKM refers to bobsService, the naive loader will not be able to find it, so will fail to load Mary's LKM.

However, a modern version of insmod knows to check for GPLONLY_bobsService if it can't find bobsService. But the modern insmod will refuse to do so unless Mary's LKM declares that it is licensed to the public under GPL.

The purpose of this appears to be to prevent anyone from accidentally violating a license (or from credibly claiming that he accidentally violated the license). It is not difficult to circumvent the restriction if you want to.

If you see this failure, it is probably because you're using an old loader (insmode) that doesn't know about GPLONLY.

The only other cause would be that the LKM author wrote the source code in such a way that it will never load into any Linux kernel, so there would be no point in the author distributing it.

6.6. An LKM Must Match Prerequisite LKMs

The same ways an LKM must be compatible with the base kernel, it must be compatible with any LKMs which it accesses (e.g. the first LKM calls a subroutine in the second). The preceding sections limit their discussions to the base kernel just to keep it simple.

10.1. How They Work

insmod makes an init_module system call to load the LKM into kernel memory. Loading it is the easy part, though. How does the kernel know to use it? The answer is that the init_module system call invokes the LKM's initialization routine right after it loads the LKM. insmod passes to init_module the address of the subroutine in the LKM named init_module as its initialization routine.

(This is confusing -- every LKM has a subroutine named init_module, and the base kernel has a system call by that same name, which is accessible via a subroutine in the standard C library also named init_module).

The LKM author set up init_module to call a kernel function that registers the subroutines that the LKM contains. For example, a character device driver's init_module subroutine might call the kernel's register_chrdev subroutine, passing the major and minor number of the device it intends to drive and the address of its own "open" routine among the arguments. register_chrdev records in base kernel tables that when the kernel wants to open that particular device, it should call the open routine in our LKM.

But the astute reader will now ask how the LKM's init_module subroutine knew the address of the base kernel's register_chrdev subroutine. This is not a system call, but an ordinary subroutine bound into the base kernel. Calling it means branching to its address. So how does our LKM, which was not compiled anywhere near the base kernel, know that address? The answer to this is the relocation that happens at insmod time.

How that relocation happens is fundamentally different between Linux 2.4 and Linux 2.6. In 2.6, insmod pretty much just passes the verbatim contents of the LKM file (.ko file) to the kernel and the kernel does the relocation. In 2.4, insmod does the relocation and passes a fully linked module image, ready to stuff into kernel memory, to the kernel. The description below covers the 2.4 case.

insmod functions as a relocating linker/loader. The LKM object file contains an external reference to the symbol register_chrdev. insmod does a query_module system call to find out the addresses of various symbols that the existing kernel exports. register_chrdev is among these. query_module returns the address for which register_chrdev stands and insmod patches that into the LKM where the LKM refers to register_chrdev.

If you want to see the kind of information insmod can get from a query_module system call, look at the contents of /proc/ksyms.

Note that some LKMs call subroutines in other LKMs. They can do this because of the __ksymtab and .kstrtab sections in the LKM object files. These sections together list the external symbols within the LKM object file that are supposed to be accessible by other LKMs inserted in the future. insmod looks at __ksymtab and .kstrtab and tells the kernel to add those symbols to its exported kernel symbols table.

To see this for yourself, insert the LKM msdos.o and then notice in /proc/ksyms the symbol fat_add_cluster (which is the name of a subroutine in the fat.o LKM). Any subsequently inserted LKM can branch to fat_add_cluster, and in fact msdos.o does just that.

10.2. The .modinfo Section

An ELF object file consists of various named sections. Some of them are basic parts of an object file, for example the .text section contains executable code that a loader loads. But you can make up any section you want and have it used by special programs. For the purposes of Linux LKMs, there is the .modinfo section. An LKM doesn't have to have a section named .modinfo to work, but the macros you're supposed to use to code an LKM cause one to be generated, so they generally do.

To see the sections of an object file, including the .modinfo section if it exists, use the objdump program. For example:

To see all the sections in the object file for the msdos LKM:

objdump msdos.o --section-headers

objdump msdos.o --full-contents --section=.modinfo

You can use the modinfo program to interpret the contents of the .modinfo section.

So what is in the .modinfo section and who uses it? insmod uses the .modinfo section for the following:

• It contains the kernel release number for which the module was built. I.e. of the kernel source tree whose header files were used in compiling the module.

  insmod uses that information as explained in Section 6.

• It describes the form of the LKM's parameters. insmod uses this information to format the parameters you supply on the insmod command line into data structure initial values, which insmod inserts into the LKM as it loads it.

10.3. The __ksymtab And .kstrtab Sections

Two other sections you often find in an LKM object file are named __ksymtab and .kstrtab. Together, they list symbols in the LKM that should be accessible (exported) to other parts of the kernel. A symbol is just a text name for an address in the LKM. LKM A's object file can refer to an address in LKM B by name (say, getBinfo"). When you insert LKM A, after having inserted LKM B, insmod can insert into LKM A the actual address within LKM B where the data/subroutine named getBinfo is loaded.

See Section 10.1 for more mind-numbing details of symbol binding.

10.4. Ksymoops Symbols

insmod adds a bunch of exported symbols to the LKM as it loads it. These symbols are all intended to help ksymoops do its job. ksymoops is a program that interprets and "oops" display. And "oops" display is stuff that the Linux kernel displays when it detects an internal kernel error (and consequently terminates a process). This information contains a bunch of addresses in the kernel, in hexadecimal.

ksymoops looks at the hexadecimal addresses, looks them up in the kernel symbol table (which you see in /proc/ksyms (Linux 2.4) or /proc/kallsyms (Linux 2.6), and translates the addresses in the oops message to symbolic addresses, which you might be able to look up in an assembler listing.

So lets say you have an LKM crash on you. The oops message contains the address of the instruction that choked, and what you want ksymoops to tell you is 1) in what LKM is that instruction, and 2) where is the instruction relative to an assembler listing of that LKM? Similar questions arise for the data addresses in the oops message.

To answer those questions, ksymoops must be able to get the loadpoints and lengths of the various sections of the LKM from the kernel symbol table.

Well, in Linux 2.4, insmod knows those addresses, so it just creates symbols for them and includes them in the symbols it loads with the LKM.

In particular, those symbols are named (and you can see this for yourself by looking at /proc/ksyms):

__insmod_name_Ssectionname_Llength

name is the LKM name (as you would see in /proc/modules.

sectionname is the section name, e.g. .text (don't forget the leading period).

length is the length of the section, in decimal.

The value of the symbol is, of course, the address of the section.

Insmod also adds a pretty useful symbol that tells from what file the LKM was loaded. That symbol's name is

__insmod_name_Ofilespec_Mmtime_Vversion

name is the LKM name, as above.

filespec is the file specification that was used to identify the file containing the LKM when it was loaded. Note that it isn't necessarily still under that name, and there are multiple file specifications that might have been used to refer to the same file. For example, ../dir1/mylkm.o and /lib/dir1/mylkm.o.

mtime is the modification time of that file, in the standard Unix representation (seconds since 1969), in hexadecimal.

version tells the kernel version level for which the LKM was built (same as in the .modinfo section). It is the value of the macro LINUX_VERSION_CODE in Linux's linux/version.h file. For example, 132101.

The value of this symbol is meaningless.

In Linux 2.6, it works differently. (I haven't figured out how yet).

10.5. Other Symbols

insmod adds another symbol, similar to the ksymoops symbols. This one tells where the persistent data lives in the LKM, which rmmod needs to know in order to save the persistent data.

__insmod_name_Plength

10.6. Debugging Symbols

There is another kind of symbol that relates to an LKM: kallsyms symbols. These are not exported symbols; they do not show up in proc/ksyms. They refer to addresses in the kernel that are nobody's business except the module they are in, and are not meant to be referenced by anything except a debugger. Kdb, the kernel debugger that comes with the Linux kernel, is one user of kallsyms symbols.

The kallsyms facility works for both the base kernel and LKMs. For the base kernel, you select it when you build the base kernel, with the CONFIG_KALLSYMS configuration option. When you do that, the kernel contains a kallsyms symbol for all the symbols in the base kernel object files. You know your base kernel is participating in the kallsyms facility if you see the symbol __start___kallsyms in /proc/ksyms.

For an LKM, you decide at load time whether it will contain kallsyms symbols. You include the kallsyms definitions in the data you pass to the init_module system call to load the LKM. insmod does this if either 1) you specify the --kallsyms option, or 2) insmod determines, by looking at /proc/ksyms, that the base kernel is participating in the kallsyms facility. The kallsyms that insmod defines are all the symbols in the LKM object file. To wit, those are the symbols you see when you run nm on the LKM object file.

Each loaded LKM that is participating in kallsyms has its own kallsyms symbol table. When the base kernel is participating in the kallsyms facility, the individual LKM kallysms symbol tables are linked into a master symbol table so that a debugger can look up a symbol anywhere in the kernel. When the base kernel is not participating in kallsyms, a debugger must look explicitly at a particular LKM to find symbols for that LKM. Kdb, for one, cannot do this. So the basic rule is: If you're going to do any kernel debugging, use CONFIG_KALLSYMS.

Note that the __kallsyms section has nothing to do with LKMs. That's a section in the base kernel object module. The base kernel doesn't have the luxury of something as high-level as sophisticated as insmod to load it, so it needs that extra object file section to facilitate its participation in kallsyms.

Similarly, the program kallsyms has nothing to do with LKMs. It is what creates the __kallsyms section.

There is another kind of debugging symbol -- the kind that gcc creates with its -g option. These are unrelated to the kallsyms facility. They do not get loaded into kernel memory. Kdb does not use them. But Kgdb (which gets information both from kernel memory and source and object files) does.

10.7. Memory Allocation For Loading

This section is about how Linux allocates memory in which to load an LKM. It is not about how an LKM dynamically allocates memory, which is the same as for any other part of the kernel.

The memory where an LKM resides is a little different from that where the base kernel resides. The base kernel is always loaded into one big contiguous area of real memory, whose real addresses are equal to its virtual addresses. That's possible because the base kernel is the first thing ever to get loaded (besides the loader) -- it has a wide open empty space in which to load. And since the Linux kernel is not pageable, it stays in its homestead forever.

By the time you load an LKM, real memory is all fragmented -- you can't simply add the LKM to the end of the base kernel. But the LKM needs to be in contiguous virtual memory, so Linux uses vmalloc to allocate a contiguous area of virtual memory (in the kernel address space), which is probably not contiguous in real memory. But the memory is still not pageable. The LKM gets loaded into real page frames from the start, and stays in those real page frames until it gets unloaded.

This design is based on the principle that it's much easier to get a large contiguous area of virtual memory than to get a large contiguous area of real memory because there is orders of magnitude more virtual address space than real memory. In the early days of virtual memory, that was certainly true -- a 32 bit machine had a 4GiB virtual address space and rarely more than 64 MiB of real memory. Now, however, it's quite often just the reverse, with virtual address space being the constrained resource. So this LKM loading strategy may not make sense on your system.

Some CPUs can take advantage of the properties of the base kernel to effect faster access to base kernel memory. For example, on one machine, the entire base kernel is covered by one page table entry and consequently one entry in the translation lookaside buffer (TLB). Naturally, that TLB entry is virtually always present. For LKMs on this machine, there is a page table entry for each memory page into which the LKM is loaded. Much more often, the entry for a page is not in the TLB when the CPU goes to access it, which means a slower access.

This effect is probably trivial.

It is also said that PowerPC Linux does something with its address translation so that transferring between accessing base kernel memory to accessing LKM memory is costly. I don't know anything solid about that.

The base kernel contains within its prized contiguous domain a large expanse of reusable memory -- the kmalloc pool. In some versions of Linux, the module loader tries first to get contiguous memory from that pool into which to load an LKM and only if a large enough space was not available, go to the vmalloc space. Andi Kleen submitted code to do that in Linux 2.5 in October 2002. He claims the difference is in the several per cent range.

10.8. Linux internals

If you're interested in the internal workings of the Linux kernel with respect to LKMs, this section can get you started. You should not need to know any of this in order to develop, build, and use LKMs.

The code to handle LKMs is in the source files kernel/module.c in the Linux source tree.

The kernel module loader (see Section 5.4) lives in kernel/kmod.c.

(Ok, that wasn't much of a start, but at least I have a framework here for adding this information in the future).

11.1. Simpler hello.c

Lkmpg gives an example of the world's simplest LKM, hello-1.c. But it is not as simple as it could be and depends on your having kernel messaging set up a certain way on your system to see it work. Finally, the program requires you to include -D options on your compile command to work, because it does not define some macros in the source code, where the definitions belong.

Here is an improved world's simplest LKM, hello.c.

/* hello.c 
 * 
 * "Hello, world" - the loadable kernel module version. 
 *
 * Compile this with 
 *
 *          gcc -c hello.c -Wall
 */

/* Declare what kind of code we want from the header files */
#define __KERNEL__         /* We're part of the kernel */
#define MODULE             /* Not a permanent part, though. */

/* Standard headers for LKMs */
#include <linux/modversions.h> 
#include <linux/module.h>  

#include <linux/tty.h>      /* console_print() interface */

/* Initialize the LKM */
int init_module()
{
  console_print("Hello, world - this is the kernel speaking\n");
  /* More normal is printk(), but there's less that can go wrong with 
     console_print(), so let's start simple.
  */

  /* If we return a non zero value, it means that 
   * init_module failed and the LKM can't be loaded 
   */
  return 0;
}


/* Cleanup - undo whatever init_module did */
void cleanup_module()
{
  console_print("Short is the life of an LKM\n");
}

Compile this with the simple command

$ gcc -c -Wall -nostdinc -I /usr/src/linux/include hello.c

The -I above assumes that you have the source code from which your base kernel (the base kernel of the kernel into which you hope to load hello.c) was built in the conventional spot, /usr/src/linux. If you're masochistic enough to be using symbol versioning in your base kernel, then you better have run 'make dep' on that kernel source too, because that's what builds the .ver files that change the names of all your symbols.

But note that it's reasonably common not to have the kernel headers installed there, and often, the wrong headers are installed there. When you use a kernel that you loaded from a distribution CD, you often have to separately load the headers for it. To be safe, if you're playing with compiling LKMs, you really should compile your own kernel, so you know exactly what you're working with and can be absolutely sure you're working with matching header files.

The -nostdinc option isn't strictly necessary, but is the right thing to do. It will keep you out of trouble and also remind you that the services of the standard C library, which you may have melded in your mind with C itself, are not available to kernel code. -nostdinc says not to include "standard" directories in the include file search path. This means, most notably, /usr/include.

The -c option says you just want to create an object (.o) file, as opposed to gcc's default which is to create the object file, then link it with a few other standard object files to create something suitable for exec'ing in a user process. As you will not be exec'ing this module but rather adding it to the kernel, that link phase would be entirely inappropriate.

-Wall (which makes the compiler warn you about lots of kinds of questionable code) is obviously not necessary, but this program should not generate any warnings. If it does, you need to fix something.

11.2. Using the Kernel Build System

Lkmpg contains fine instructions for building (compiling) an LKM (except that the __KERNEL__ macro and usually the MODULE macro should be defined in the source code instead of with -D compiler options as Lkmpg suggests). But it deserves mention that some Linux kernel programmers believe that the only right way to build an LKM is to add it to a copy of the complete Linux source tree and build it with the existing Linux make files just like the LKMs that are part of Linux.

There are advantages to this. The biggest one is that when Linux programmers change the way LKMs interface with the rest of the kernel in a way that affects how you build an LKM, you're covered.

On the other hand, you will probably find from a code management point of view that you really have to keep your own code and Linux separate, and from a coding point of view, you really need to understand all the intricacies of how your code gets compiled, especially when it changes.

11.3. Rubini et al: Linux Device Drivers

The most popular book on writing device drivers is O'Reilly's Linux Device Drivers by Alessandro Rubini, Jonathan Corbet, and Greg Kroah-Hartman.

Even if you're writing an LKM that isn't a device driver, you can learn a lot from this book that will help you.

The first edition of this book covers Linux 2.0, with notes about differences in 2.2. The second edition (June 2001) covers Linux 2.4. The third edition (April 2005) covers Linux 2.6. Of course, if you know anything about Linux, you know that a book like this doesn't perfectly cover any release, because Linux changes frequently. Linux 2.6 as was current a month after the Third Edition was released had significant differences from the Linux 2.6 about which the book was written.

The second edition of this book is available under the FDL. You can read it at http://www.xml.com/ldd/chapter/book/. The third edition is available under the terms of the Creative Commons Attribution-ShareAlike license, and you'll find it at http://lwn.net/Kernel/LDD3/.

This book is also available in print in any decent technical book store.

11.4. Module Use Counts

It is essential that the kernel not try to reference the code of a module after it has been unloaded; i.e. you must not unload a module while it is in use. An example of in use is a device driver for which a device special file is open. Because there is an open file descriptor for it, a user might do a read of the device and to execute that read, the kernel would want to call a function that is in the device driver. You can see that there would be a problem if you unloaded that device driver module before the read -- the kernel would reuse the memory that used to contain the read subroutine and there's no telling what instructions the kernel would branch to when it thinks it's calling the read subroutine.

In the original design, the LKM increments and decrements its use count to tell the module manager whether it is OK to unload it. For example, if it's a filesystem driver, it would increment the use count when someone mounts a filesystem of the type it drives, and decrement it at unmount time.

Later a more flexible alternative was added. Your LKM can register a function that the module manager will call whenever it wants to know if it is OK to unload the module. If the function returns a true value, that means the LKM is busy and cannot be unloaded. If it returns a false value, the LKM is idle and can be unloaded. The module manager holds the big kernel lock from before calling the module-busy function until after its cleanup subroutine returns or sleeps, and unless you've done something odd, that should mean that your LKM cannot become busy between the time that you report "not busy" and the time you clean up.

So how do you register the module-busy function? By putting its address in the unfortunately named can_unload field in the module descriptor ("struct module"). The name is truly unfortunate because the boolean value it returns is the exact opposite of what "can unload" means: true if the module manager cannot unload the LKM.

The module manager ensures that it does not attempt to unload the module before its initialization subroutine has returned or sleeps, so you are safe in setting the can_unload field anywhere in the initialization subroutine except after a sleep.

can_unload is little known and rarely used. Starting with Linux 2.6, it no longer exists.

Whether you use traditional use counts of can_unload, there are cases where you cannot be sure that your module doesn't get unloaded while it is still in use. If your LKM creates a kernel thread that executes LKM code, it is just about impossible to be absolutely sure that thread is gone before the LKM gets unloaded. There are various other kernel services that you can give addresses within your LKM that won't properly let you know when they have forgotten them.

The problem used to be worse than it is now. For example, it used to be that if your LKM created a proc filesystem file, you couldn't stop the LKM from getting unloaded while some process was executing your read and write routines for the file. This and other instances of the problem have been fixed by having code outside the LKM understand that the address it's using might be in an LKM, and therefore increment and decrement the use count as necessary. Where this function is implemented, you often see a structure member named "owner" which is a handle for the LKM (i.e. a struct module address).

These problems may be fixed in future version of Linux. Until then, you can just cross your fingers. Some people believe these types of problems are so hard to fix that the proper design for Linux is just to make it impossible ever to unload an LKM. Starting with Linux 2.6, the CONFIG_MODULE_UNLOAD kernel build configuration option determines whether module unloading is allowed or not.

12.1. Linux 2.4 - Linux 2.6

15.1. Executable Interpreters

You must have at least one executable interpreter bound into the base kernel, because in order to load an executable interpreter LKM, you have to run an executable and something has to interpret that executable.

That one bound-in executable interpreter is almost certainly the ELF interpreter, since virtually all executables in a Linux system are ELF.

Historical note: Before ELF existed on Linux (c. 1995), the normal executable format was a.out. For a while, part ELF/part a.out systems were common. Some still exist.

modprobe binfmt_aout

modprobe binfmt_elf

modprobe binfmt_java

15.2. Block Device Drivers

  
  modprobe floppy 'floppy="daring two_fdc 0,thinkpad 0x8,fifo_depth"'

modprobe loop

modprobe linear

modprobe raid0

  modprobe rd

modprobe xd

15.3. SCSI Drivers

Detailed information about SCSI drivers is in SCSI-2.4-HOWTO.

Linux's SCSI function is implemented in three layers, and there are LKMs for all of them.

In the middle is the mid-level driver or SCSI core. This consists of the scsi_mod LKM. It does all those things that are common among SCSI devices regardless of what SCSI adapter you use and what class of device (disk, scanner, CD-ROM drive, etc.) it is.

There is a low-level driver for each kind of SCSI adapter -- typically, a different driver for each brand. For example, the low-level driver for Advansys adapters (made by the company which is now Connect.com) is named advansys. (If you are comparing ATA (aka IDE) and SCSI disk devices, this is a major difference -- ATA is simple and standard enough that one driver works with all adapters from all companies. SCSI is less standard and as a result you should have less confidence in any particular adapter being perfectly compatible with your system).

High-level drivers present to the rest of the kernel an interface appropriate to a certain class of devices. The SCSI high-level driver for tape devices, st, for example, has ioctls to rewind. The high-level SCSI driver for CD-ROM drives, sr, does not.

Note that you rarely need a high-level driver specific to a certain brand of device. At this level, there is little room for one brand to be distinguishable from another.

One SCSI high-level driver that deserves special mention is sg. This driver, called the "SCSI generic" driver, is a fairly thin layer that presents a rather raw representation of the SCSI mid-level driver to the rest of the kernel. User space programs that operate through the SCSI generic driver (because they access device special files whose major number is the one registered by sg (to wit, 21)) have a detailed understanding of SCSI protocols, whereas user space programs that operate through other SCSI high-level drivers typically don't even know what SCSI is. SCSI-Programming-HOWTO has complete documentation of the SCSI generic driver.

The layering order of the SCSI modules belies the way the LKMs depend upon each other and the order in which they must be loaded. You always load the mid-level driver first and unload it last. The low-level and high-level drivers can be loaded and unloaded in any order after that, and they hook themselves into and establish dependency on the mid-level driver at both ends. If you don't have a complete set, you will get a "device not found" error when you try to access a device.

Most SCSI low-level (adapter) drivers don't have LKM parameters; they do generally autoprobe for card settings. If your card responds to some unconventional port address you must bind the driver into the base kernel and use kernel "command line" options. See BootPrompt-HOWTO. Or you can twiddle The Source and recompile.

Many SCSI low-level drivers have documentation in the drivers/scsi directory in the Linux source tree, in files called README.*.

modprobe scsi_mod

modprobe sd_mod

modprobe st

modprobe sr_mod

modprobe sg

modprobe wd7000

modprobe aha152x

modprobe aha1542

modprobe aha1740

modprobe aic7xxx

modprobe advansys asc_iopflag=1 asc_ioport=0x110,0x330 asc_dbglvl=1

modprobe in2000

modprobe BusLogic

modprobe dtc

modprobe eata

modprobe eata_dma

  modprobe eata_pio

modprobe fdomain

modprobe NCR5380 ncr_irq=xx ncr_addr=xx ncr_dma=xx ncr_5380=1   \
                 ncr_53c400=1

modprobe g_NCR5380 ncr_irq=5 ncr_addr=0x350 ncr_5380=1

modprobe g_NCR5380 ncr_irq=255 ncr_addr=0xc8000 ncr_53c400=1

modprobe NCR53c406a

modprobe 53c7,8xx

modprobe ncr53c8xx

modprobe ppa ppa_base=0x378 ppa_nybble=1

modprobe pas16

modprobe qlogicfas

modprobe qlogicisp

  modprobe seagate

  modprobe t128

  modprobe u14-34f

modprobe ultrastor

15.4. Network Device Drivers

modprobe bsd_comp

modprobe slhc

modprobe dummy

modprobe eql

modprobe dlci

modprobe sdla

modprobe plip io=0x378 irq=7

modprobe ppp

modprobe slip slip_maxdev=1

modprobe baycom modem=1 iobase=0x3f8 irq=4 options=1

modprobe strip

modprobe wavelan io=0x390 irq=0

modprobe wic

modprobe scc

 
modprobe 8390

modprobe ne io=0x300 irq=11

modprobe ne io=0x300 irq=11

modprobe 3c503 io=0x300 irq=5 xcvr=0

modprobe 3c503 io=0x300 irq=5 xcvr=0

modprobe 3c503 io=0x300 irq=5 xcvr=0

modprobe 3c503 io=0x300 irq=5 xcvr=0

modprobe 3c59x debug=1 options=0,,12

modprobe wd io=0x300 irq=5 mem=0x0D0000 mem_end=0x0D8000

modprobe smc-ultra io=0x200 irq=5

modprobe smc9194 io=0x200 irq=5 ifport=0

modprobe at1700 io=0x260 irq=5

modprobe e2100 io=0x300 irq=5 mem=0xd0000 xcvr=0

modprobe depca io=0x200 irq=7

modprobe ewrk3 io=0x300 irq=5

modprobe eexpress io=0x300 irq=5

modprobe eepro io=0x200 irq=5

modprobe fmv18x io=0x220 irq=5

modprobe hp-plus io=0x200 irq=5

modprobe hp io=0x300 irq=5

modprobe hp100 hp100_port=0x100

modprobe eth16i io=0x2a0 irq=5

modprobe ni52 io=0x360 irq=9 memstart=0xd0000 memend=0xd4000

modprobe ac3200

modprobe apricot io=0x300 irq=10

 
modprobe de4x5 io=0x000b irq=10 is_not_dec=0

modprobe tulip

modprobe dgrs debug=1 dma=0 spantree=0 hashexpire=300 ipaddr=199,86,8,221
modprobe ipxnet=111

modprobe de600 de600_debug=0

modprobe de620 bnc=0 utp=0 io=0x378 irq=7

modprobe ibmtr io=0xa20 irq=5

modprobe arcnet io=0x300 irq=2 shmem=0xd0000 device=arc1

modprobe isdn

modprobe icn portbase=0x320 membase=0xd0000 icn_id=idstring icn_id2=idstring2

modprobe pcbit mem=0xd0000 irq=5

modprobe teles io=0xd0000,15,0xd80,2 teles_id=idstring

15.5. CDROM Device Drivers

  modprobe aztcd aztcd=0x340

modprobe gscd gscd=0x340

modprobe sbpcd sbpcd=0x340

modprobe mcd mcd=0x300,11,0x304,5

modprobe mcdx mcdx=0x300,11,0x304,5

modprobe optcd optcd=0x340

modprobe cm206 cm206=0x300,11

modprobe sjcd sjcd_base=0x340

modprobe isp16 isp16_cdrom_base=0x340 isp16_cdrom_irq=3
         isp16_cdrom_dma=0 isp16_cdrom_type=Sanyo

  modprobe cdu31a cdu31a_port=0x340 cdu31a_irq=5

modprobe sonycd535 sonycd535=0x340

15.6. Filesystem Drivers

modprobe minix

modprobe ext

modprobe ext2

modprobe xiafs

modprobe fat

modprobe msdos

modprobe vfat

modprobe vfat

modprobe nfs

modprobe smbfs

modprobe ncpfs

modprobe isofs

modprobe hpfs

modprobe sysv

modprobe affs

modprobe ufs

15.7. Miscellaneous Device Driver

modprobe misc

15.8. Serial Device Drivers

modprobe serial

  modprobe cyclades

  modprobe stallion

modprobe istallion

modprobe riscom8 iobase=0xXXX iobase1=0xXXX iobase2=...

15.9. Parallel Device Drivers

  modprobe lp.o io=0x378 irq=0

15.10. Bus Mouse Device Drivers

modprobe atixlmouse

modprobe busmouse

modprobe msbusmouse

modprobe psaux

15.11. Tape Device Drivers

For SCSI tape device drivers, see Section 15.3. There are no LKMs for QIC-02 tape devices, but there is a device driver you can bind into the base kernel.

modprobe ftape tracing=3

15.12. Watchdog Timers

modprobe wdt

  modprobe softdog

modprobe pcwd

15.13. Sound Device Drivers

Configuring sound is a complex task. Read the files in directory Documention/sound in the Linux source tree.

Example:

modprobe sound

Option: dma_buffsize=32768