This guide describes a work in progress, to port Linux to a custom PowerPC-based board. This means making the operating system work on unfamiliar hardware. Anyone who is on the same track might benefit from reading this paper, as it highlights the pitfalls and problematic points along the way.

Before attempting to port Linux, know at least the following: (whenever possible, a link to a proper information source is attached)

This section describes the tools we used during the process. Most are trivial to install and use. When neccesary, consult the appropriate url or manual.

The board is based on PPC750 (PowerPC) processor. It is 6U VME64 standard. The board is designed to host two PCI Mezzanine cards (CCPMC) - Mezzanine cards that comply with Std CCPMC1386 can be attached.

Copyright (c) 2002 Shie Elrich

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in Appendix A, GNU Free Documentation License.

The minimum requirments are obviously a development station and a target. However, the recommended way of working is having a third host which acts as a server. The server runs several services such as ftp, telnet, NFS, tftp (if needed) and CVS. The main role of the server is to run CVS and track version control, however once you can boot the target from network, the server will also hold the target images, and filesystem, which makes development much easier.

Regardless, the first step is to install a tool-chain (compiler, linker etc.) for your target. The HardHat Linux cdrom includes all the needed files, and the installation sequence is documented in the HardHat Linux documentation. During the installation, you must select your LSP (basic software for the selected board), and HardHat will install a set of tools and a kernel source tree matching your LSP.

We had a board that had vxWorks running on it, so we setup the target to boot using the standard vxWorks loader. Once the loader initiated, we used visionICE to take-over the target (so that vxWorks won't load an image file) and load a Linux image into the target. What you need to do at this point is get an ICE, connect it to the network and to the target - through a JTAG connection - and install the ICE software on your host.

What should have been done so far:

If you've installed the Linux kernel that comes with HardHat, then cross-compiling should already be enabled in the kernel Makefile. If your kernel is not from the HardHat CD, you should enable cross-compiling in the Makefile by defining a CROSS_COMPILE entry in the following manner: (a code segment from the main Makefile)

CROSS_COMPILE 	= /opt/hardhat/devkit/ppc/7xx/bin/ppc_7xx-
AS		= $(CROSS_COMPILE)as
LD		= $(CROSS_COMPILE)ld
CC		= $(CROSS_COMPILE)gcc

The Linux kernel is modular, and allows you to configure it and choose which “blocks” should be compiled with the kernel. In order to do this, first cd /usr/src/linux (assuming your kernel source code is installed at /usr/src/linux). Once there, type make xconfig.After saving your options, you should make vmlinux to create a kernel image suitably for using with VisionICE.

We will not go into more details here, as it's outside the scope of this document. For more information, try http://www.tldp.org/HOWTO/Kernel-HOWTO.html

First, configure the terminal program, in our case minicom, the following way: 9600 bps, 8 bits, no parity, 1 stop bit and no flow control of any kind. The serial port in Linux should be /dev/ttyS0 for COM1, /dev/ttyS1 for COM2 etc.

Start the target. You should see the vxWorks bootloader on your terminal screen, and should be able to stop the boot sequence by pressing the space bar.

Once the target is stopped, run the VisionICE software and perform the following steps:

After pressing the Run button, nothing happened. At that moment, and for some time after, it seemed that nothing was happening and the kernel was stuck. We used ICE to step through the initialization code of the kernel and rule out some potential problems, like virtual memory errors, only to finally discover that the problem was simple: the kernel was indeed booting but since the console (tty) driver had problems, we couldn't see anything!

As stated in the previous chapter, the machine starts to boot, but nothing happens. At least, nothing that we can see. The screen is blank and no kernel messages appear. At this point, you have to ask yourself, is it really booting?

Since the console wouldn't start, and ICE died real fast, we had no choice. We had to debug somehow, and the oldest way is good here - printing to the screen. Obviously, we couldn't use printk(), so we wrote a short function which pushes characters straight into the serial port. We used the boot process “map” shown in the previous section, and inserted some prints along the way. This helped us to know at what stage we are completing and where we're dying. The following piece of code prints a single character to the serial port, by polling it and waiting for it to be free.

                                 /* tx holding reg empty or tx    */
#define LSR_THREMPTY 0x20        /* fifo is empty (in fifo mode ) */
#define THR_REG      0x00        /* Transmit holding reg */
#define LSR_REG      0x05        /* Line status reg */
#define COM1_ADDRESS 0xFF600300  /* == replace with your UART address */

void print_char (char ch) {
	volatile  unsigned char status = 0;
	/* wait until txempty */
	while ((status & LSR_THREMPTY ) == 0)
		status = *((volatile unsigned char *)(COM1_ADDRESS + LSR_REG));

	*((volatile unsigned char *)(COM1_ADDRESS + THR_REG)) = ch;
}

Although it is not a porting issue, the way you modify your code matters. It's easier if you do it right the first time. The Linux kernel uses standard configuration flags CONFIG_XXXX (like CONFIG_PPC, CONFIG_ISA etc), which are used to mark a certain machine, architecture or device. We defined ourselves a new flag (let's call it CONFIG_TESTMACH), and surrounded our new/modified code with these flags:

....original code....
#ifdef CONFIG_TESTMACH
....modified code....
#else
....original code....
#endif /* CONFIG_TESTMACH */

To “activate” our code, we added the new flag to the kernel configuration file - .config - by adding CONFIG_TESTMACH=y to it. In the first stage, this solution allows you a quick way to find the code you changed, but later the flag you chose will allow you to add your code into the kernel tree and into the configuration program (make xconfig).

Now that the console was working, we could see the real problems. The system wasn't booting yet. Since we were working with C code, we traced the code, and found that a function called sdram_size() wasn't completing correctly. The function probed a register for the size of the RAM, a register our board doesn't have. We made the function return a given value of 128MB, it's an ugly hack, but our board doesn't have a way of knowing the amount of RAM.

Wwe had the same problems with a bunch of functions called todc_XXXX, mainly todc_get_rtc_time(), todc_set_rtc_time(), and time_init() since we don't have a RTC (real-time clock) chip on our board, and those functions were using it. For the time being, we made the todc_XXX function only set and get a constant date and time, since our board doesn't have a bios battery and so cannot keep time when powered off.

Once all this was done, we found todc_calibrate_descr(), which again uses the RTC chip. We had to replace that function with our own:

void calibrate_decr() {
	int freq, divisor;
	freq = bus_freq();
	divisor = 4;
	tb_ticks_per_jiffy = freq / HZ / divisor;
	tb_to_us = mulhwu_scale_factor(freq / divisor, 1000000);
}

Our board uses an Intel ethernet chip, called i82559er, which has a module called eepro100. After compiling the module and booting, we discovered that the module isn't working, although an ethernet device was found. We guessed that it was an irq problem, and that the devices don't get the IRQs they need. We modified a function called pmppc_map_irq() to map our ethernet devices:

XXXX_map_irq(struct pci_dev *dev, unsigned char idsel, unsigned char pin) {
	static char pci_irq_table[][4] =
	/*
	 *      PCI IDSEL/INTPIN->INTLINE
	 *      A       B       C       D
	 */
	{
		{22,    0,      0,      0},/* IDSEL 3 - Ethernet */
		{0,     0,      0,      0},/* IDSEL 4 - unused   */
		{0,     0,      0,      0},/* IDSEL 5 - unused   */
		{0,     0,      0,      0},/* IDSEL 6 - ????     */
		{0,     0,      0,      0},/* IDSEL 7 - unused   */
		{0,     0,      0,      0},/* IDSEL 8 - unused   */
		{0,     0,      0,      0},/* IDSEL 9 - unused   */
	};

	const long min_idsel = 3, max_idsel = 9, irqs_per_slot = 4;
	return PCI_IRQ_TABLE_LOOKUP;
}

The function maps IRQs according to IDselects, which means in the order on the PCI bus by which the devices are set. This structure is a bit tricky: min_idsel denotes the topleft corner of the array, and max_idsel is the bottomleft corner. irqs_per_slot is the number of IRQs per line. The structure is as follows:

each cell contains (IDSEL, SLOT#, IRQ)
		+----------------------------------------+
		| (3,0,22) | (3,1,0) | (3,2,0) | (3,3,0) |
		+----------------------------------------+
		| (4,0,0)  | (4,1,0) | (4,2,0) | (4,3,0) |
		+----------------------------------------+
							..........
							..........
		+----------------------------------------+
		| (9,0,0)  | (9,1,0) | (9,2,0) | (9,3,0) |
		+----------------------------------------+

As you can see, our i8559er needs IRQ 22, and is seated in IDselect 3. Of course, we didn't know that at the start, so we wrote a small piece of code that read all the vendor IDs in all the IDselects. Once done we compiled, but the ethernet device still didn't work.

The next problem was that the module couldn't decide on a MAC address for the device. The MAC address should be written on an EEPROM chip (connected to the device), but we discovered that the hardware guys decided that i82559 doesn't need the EEPROM, so they removed it. After hardcoding a MAC address inside eepro100.c, the ethernet device finally worked. The final solution was to make the module read the MAC address from NVRAM memory, and if no other choice was available, to fall back to a default MAC address.

We had new problems, some would say good problems. We didn't have a bootloader yet, however we needed to pass a command line to the kernel at boot time. We hard-coded the command line into the kernel inside the parse_options(). After that was finished, we made console_init() and serial_console_setup() work the way they should. They no longer ignored the command line, but still RTS and DTR stay low.

Another important issue was memory mapping. The file arch/ppc/mm/init.c contains a function called MMU_init(). This function is actually a big switch statment, divided by the machine type. Each machine maps its memory using the setbat() and ioremap() functions. The BAT mechanism is a way of translating virtual addresses into physical ones. Thus, setbat() is used by specifying a virtual address, a physical address and a page size. Not every size can be used here; you should use one of the finite set of sizes, ranging from 128KB to 256MB. We mapped our IO memory so that virtual equalled physical.

As mentioned, there is another way of mapping memory - ioremap(). ioremap() is used to map physical addresses into virtual ones, making them available to the kernel. The function does not allocate any memory, simply returns a virtual address by which one can access the memory region. The following is a snippet from MMU_init():

case _MACH_mymachine:
	setbat(0, LOW_IO_VIRT_BASE, LOW_IO_PHYS_BASE, LOW_IO_SIZE, IO_PAGE);
	ioremap(UNIVERSE_BASE,UNIVERSE_SIZE);      /* Universe VME */
	ioremap(EEPRO100_BASE,EEPRO100_SIZE);      /* Ethernet EEPRO100 */
	break;

As you can see, we don't take the return value of ioremap(). We don't need it, since at this stage the kernel maps the addresses so that virtual address == physical address.

The CPC700 has a “feature” which is supposed to make some memory access use 64 bit wide. This is a problem since some test-and-set registers on our board might get set unintentionally, because we were trying to read something 16 bits lower. In order to solve this situation, we set the memory controller to 64 bit wide intervals. If you try to access those areas in another manner (8 or 16 bit access), the CPC700 simply throws them away. We had to be able to read/write those areas, since important “discretes” (controlled by an Altera device) were mapped there.

In order to access those areas, we needed a function that does a 64 bit write. As far as I know, doing a 64 bit write on a PowerPC is possible in two ways: using cache lines and using a floating point register. The floating point register is a 64 bit sized register, so when we write it, the whole 64 bit get written. The problem is that you can't do floating point in the kernel. Since the kernel doesn't save the floating point registers during context switch, it doesn't allow FP, and will throw an exception if done in the kernel.

After messing with cache lines, we decided to go the FP way, and added the following function:

void out64(__u32 addr, long long *pVal) {
	__u32 flags, tmp_msr;

	save_flags(flags);
	cli();
	tmp_msr = __get_MSR();
	tmp_msr |= MSR_FP;
	tmp_msr &= ~(MSR_FE0 | MSR_FE1);
	__put_MSR(tmp_msr);

	sysOut64(addr, pVal);
	__put_MSR(flags & ~(MSR_EE));
	restore_flags(flags);
}

The function adds a floating point to the PowerPC MSR register, and makes sure that no exceptions will be generated as a result of doing FP. Once done, it uses an assembly code, described below in the sysOut64() to do the actual floating-point operation. Note that the function turns off interrupts, but this is acceptable here, since we use the function on rare occasion.

_GLOBAL(sysOut64)
stwu    r1, -DEPTH(r1)
mflr    r0
stw     r31, FP_LOC(r1)
stw     r0,  LR_LOC(r1)
mr              r31, r1
stfd    fr0, FPR_SAVE(r31)      /* save floating point reg contents */

lfd     fr0,0(r4)
stfd  fr0,0(r3)
eieio

lfd     fr0, FPR_SAVE(r31)      /* restore floating point value */
lwz     r4, 0(r1)               /* now restore the stack frame  */
lwz     r0, 4(r4)
mtlr    r0
lwz     r31, -4(r4)
mr      r1, r4
blr

While Linux was booting using an NFS filesystem, this was not enough. For an actual field product, we needed Linux to boot from an independent device, without the need for a network at all. We decided to create a special kind of image, called initrd, which is basically a Linux kernel with a compressed file. The compressed file includes a Linux filesystem. The filesystem is unpacked to a ramdisk on boot, and mounted as the root filesystem.

During the boot process, the bootloader relocated the kernel image to address zero - which was fine, and the initrd part to a higher address. The area to which initrd was relocated was not mapped in our kernel's memory, and all we got was a kernel error (access to bad area). After modifying the bootloader to relocate initrd to a different address, all was fine and Linux booted succesfully.