FreeBSD Handbook
Prev	Chapter 12 Storage	Next

12.4 RAID

12.4.1 Software RAID

12.4.1.1 Concatenated Disk Driver (CCD) Configuration

Original work by Christopher Shumway. Revised by Jim Brown.

When choosing a mass storage solution the most important factors to consider are speed, reliability, and cost. It is rare to have all three in balance; normally a fast, reliable mass storage device is expensive, and to cut back on cost either speed or reliability must be sacrificed.

In designing the system described below, cost was chosen as the most important factor, followed by speed, then reliability. Data transfer speed for this system is ultimately constrained by the network. And while reliability is very important, the CCD drive described below serves online data that is already fully backed up on CD-R's and can easily be replaced.

Defining your own requirements is the first step in choosing a mass storage solution. If your requirements prefer speed or reliability over cost, your solution will differ from the system described in this section.

12.4.1.1.1 Installing the Hardware

In addition to the IDE system disk, three Western Digital 30GB, 5400 RPM IDE disks form the core of the CCD disk described below providing approximately 90GB of online storage. Ideally, each IDE disk would have its own IDE controller and cable, but to minimize cost, additional IDE controllers were not used. Instead the disks were configured with jumpers so that each IDE controller has one master, and one slave.

Upon reboot, the system BIOS was configured to automatically detect the disks attached. More importantly, FreeBSD detected them on reboot:

ad0: 19574MB <WDC WD205BA> [39770/16/63] at ata0-master UDMA33
ad1: 29333MB <WDC WD307AA> [59598/16/63] at ata0-slave UDMA33
ad2: 29333MB <WDC WD307AA> [59598/16/63] at ata1-master UDMA33
ad3: 29333MB <WDC WD307AA> [59598/16/63] at ata1-slave UDMA33

Note: If FreeBSD does not detect all the disks, ensure that you have jumpered them correctly. Most IDE drives also have a ``Cable Select'' jumper. This is not the jumper for the master/slave relationship. Consult the drive documentation for help in identifying the correct jumper.

Next, consider how to attach them as part of the file system. You should research both vinum(8) (Chapter 13) and ccd(4). In this particular configuration, ccd(4) was chosen.

12.4.1.1.2 Setting Up the CCD

The driver ccd(4) allows you to take several identical disks and concatenate them into one logical file system. In order to use ccd(4), you need a kernel with ccd(4) support built in. Add this line to your kernel configuration file, rebuild, and reinstall the kernel:

pseudo-device   ccd     4

On 5.X systems, you have to use instead the following line:

device   ccd

Note: In FreeBSD 5.X, it is not necessary to specify a number of ccd(4) devices, as the ccd(4) device driver is now self-cloning -- new device instances will automatically be created on demand.

The ccd(4) support can also be loaded as a kernel loadable module in FreeBSD 3.0 or later.

To set up ccd(4), you must first use disklabel(8) to label the disks:

disklabel -r -w ad1 auto
disklabel -r -w ad2 auto
disklabel -r -w ad3 auto

This creates a disklabel for ad1c, ad2c and ad3c that spans the entire disk.

Note: Since FreeBSD 5.1-RELEASE, the bsdlabel(8) utility replaces the old disklabel(8) program. With bsdlabel(8) a number of obsolete options and parameters have been retired; in the examples above the option -r should be removed. For more information, please refer to the bsdlabel(8) manual page.

The next step is to change the disk label type. You can use disklabel(8) to edit the disks:

disklabel -e ad1
disklabel -e ad2
disklabel -e ad3

This opens up the current disk label on each disk with the editor specified by the EDITOR environment variable, typically vi(1).

An unmodified disk label will look something like this:

8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  c: 60074784        0    unused        0     0     0   # (Cyl.    0 - 59597)

Add a new e partition for ccd(4) to use. This can usually be copied from the c partition, but the fstype must be 4.2BSD. The disk label should now look something like this:

8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  c: 60074784        0    unused        0     0     0   # (Cyl.    0 - 59597)
  e: 60074784        0    4.2BSD        0     0     0   # (Cyl.    0 - 59597)

12.4.1.1.3 Building the File System

The device node for ccd0c may not exist yet, so to create it, perform the following commands:

cd /dev
sh MAKEDEV ccd0

Note: In FreeBSD 5.0, devfs(5) will automatically manage device nodes in /dev, so use of MAKEDEV is not necessary.

Now that you have all of the disks labeled, you must build the ccd(4). To do that, use ccdconfig(8), with options similar to the following:

ccdconfig ccd0 32 0 /dev/ad1e /dev/ad2e /dev/ad3e

The use and meaning of each option is shown below:

: The first argument is the device to configure, in this case, /dev/ccd0c. The /dev/ portion is optional.
: The interleave for the file system. The interleave defines the size of a stripe in disk blocks, each normally 512 bytes. So, an interleave of 32 would be 16,384 bytes.
: Flags for ccdconfig(8). If you want to enable drive mirroring, you can specify a flag here. This configuration does not provide mirroring for ccd(4), so it is set at 0 (zero).
: The final arguments to ccdconfig(8) are the devices to place into the array. Use the complete pathname for each device.

After running ccdconfig(8) the ccd(4) is configured. A file system can be installed. Refer to newfs(8) for options, or simply run:

newfs /dev/ccd0c

12.4.1.1.4 Making it All Automatic

Generally, you will want to mount the ccd(4) upon each reboot. To do this, you must configure it first. Write out your current configuration to /etc/ccd.conf using the following command:

ccdconfig -g > /etc/ccd.conf

During reboot, the script /etc/rc runs ccdconfig -C if /etc/ccd.conf exists. This automatically configures the ccd(4) so it can be mounted.

Note: If you are booting into single user mode, before you can mount(8) the ccd(4), you need to issue the following command to configure the array:
ccdconfig -C

To automatically mount the ccd(4), place an entry for the ccd(4) in /etc/fstab so it will be mounted at boot time:

/dev/ccd0c              /media       ufs     rw      2       2

12.4.1.2 The Vinum Volume Manager

The Vinum Volume Manager is a block device driver which implements virtual disk drives. It isolates disk hardware from the block device interface and maps data in ways which result in an increase in flexibility, performance and reliability compared to the traditional slice view of disk storage. vinum(8) implements the RAID-0, RAID-1 and RAID-5 models, both individually and in combination.

See Chapter 13 for more information about vinum(8).

12.4.2 Hardware RAID

FreeBSD also supports a variety of hardware RAID controllers. These devices control a RAID subsystem without the need for FreeBSD specific software to manage the array.

Using an on-card BIOS, the card controls most of the disk operations itself. The following is a brief setup description using a Promise IDE RAID controller. When this card is installed and the system is started up, it displays a prompt requesting information. Follow the instructions to enter the card's setup screen. From here, you have the ability to combine all the attached drives. After doing so, the disk(s) will look like a single drive to FreeBSD. Other RAID levels can be set up accordingly.

12.4.3 Rebuilding ATA RAID1 Arrays

FreeBSD allows you to hot-replace a failed disk in an array. This requires that you catch it before you reboot.

You will probably see something like the following in /var/log/messages or in the dmesg(8) output:

ad6 on monster1 suffered a hard error.
ad6: READ command timeout tag=0 serv=0 - resetting
ad6: trying fallback to PIO mode
ata3: resetting devices .. done
ad6: hard error reading fsbn 1116119 of 0-7 (ad6 bn 1116119; cn 1107 tn 4 sn 11) status=59 error=40
ar0: WARNING - mirror lost

Using atacontrol(8), check for further information:

# atacontrol list
ATA channel 0:
    Master:      no device present
    Slave:   acd0 <HL-DT-ST CD-ROM GCR-8520B/1.00> ATA/ATAPI rev 0

ATA channel 1:
    Master:      no device present
    Slave:       no device present

ATA channel 2:
    Master:  ad4 <MAXTOR 6L080J4/A93.0500> ATA/ATAPI rev 5
    Slave:       no device present

ATA channel 3:
    Master:  ad6 <MAXTOR 6L080J4/A93.0500> ATA/ATAPI rev 5
    Slave:       no device present

# atacontrol status ar0
ar0: ATA RAID1 subdisks: ad4 ad6 status: DEGRADED

You will first need to detach the disk from the array so that you can safely remove it:
```
# atacontrol detach 3
```
Replace the disk.

Reattach the disk as a spare:

# atacontrol attach 3
Master:  ad6 <MAXTOR 6L080J4/A93.0500> ATA/ATAPI rev 5
Slave:   no device present

Rebuild the array:
```
# atacontrol rebuild ar0
```

The rebuild command hangs until complete. However, it is possible to open another terminal (using Alt+Fn) and check on the progress by issuing the following command:

# dmesg | tail -10
[output removed]
ad6: removed from configuration
ad6: deleted from ar0 disk1
ad6: inserted into ar0 disk1 as spare

# atacontrol status ar0
ar0: ATA RAID1 subdisks: ad4 ad6 status: REBUILDING 0% completed

Wait until this operation completes.

Prev	Home	Next
Adding Disks	Up	Creating and Using Optical Media (CDs & DVDs)

This, and other documents, can be downloaded from ftp://ftp.FreeBSD.org/pub/FreeBSD/doc/.

For questions about FreeBSD, read the documentation before contacting <questions@FreeBSD.org>.
For questions about this documentation, e-mail <doc@FreeBSD.org>.