I’ve been playing with FreeNAS for a few days and I can’t say that I like it. I can’t put my finger on it, apart from a general feeling of working with over-padded mittens. I’ve therefore decided to go at it from scratch with Ubuntu.
The core of any decent NAS server is a bank of drives configured in a RAID array, so I’m going to start off my NAS building by configuring a level 5 software RAID set.
My scenario is that of an established Ubuntu setup, booting from a single SATA drive that I wish to add a large RAID array to, creating a large storage area that can be mounted anywhere on the filesystem, independent of the main OS and its partitions. For now I’m not intending to mount the OS itself on a RAID device – that’s another topic – but it would really make sense to have the OS sitting on a desperate RAID 1 device, benefiting from the additional redundancy. The nice thing here is that you just need a small drive to hold only the OS. Grab a couple of 5GB drives off Ebay for a fiver and use them!
For now this is all taking place inside a VirtualBox VM, but it should work just as well on a physical system.
So, first of all create a regular VM, with four SATA drives attached to it. The first one of these (on port 0) should be your OS drive. The other three (on ports 1,2 & 3) should be equally sized drives – I used drives of 5 GB for this test, but they can be as large or small as you want, it don’t matter, just as long as they are of equal size (and even then, it doesn’t matter too much as we’re gonna partition them) – these form your RAID array.
Install Ubuntu Server onto the first drive. During the install just look for a device called “sda” and choose “Guided partitioning” for that. If you’re unsure, just connect the first drive by itself, install, shutdown, connect the RAID drives and reboot.
Once the OS is installed you can use
cat /proc/diskstats
to see all the physical drives and partitions on your system. Each drive on your system should show up in /dev/ as something like
/dev/sdx
where x is a letter starting with “a”, onwards.
Your OS drive should show as /dev/sda and you should also see something like /dev/sda1, /dev/sda2, /dev/sda5 and so on. sda represents the physical drive (well, as physical as it can be on a VM) and the numbered “sda”s afterwards represent partitions on the drive. Your OS drive is obviously already partitioned and formatted, but your array drives will likely have nothing to show but the plain unnumbered versions, as they are unpartioned as yet.
To start the process of building the RAID you’ll want to format the three drives of your array. If you’re using the command-line (you Linux warrior, you) then use fdisk to partition and format each drive. Run
sudo fdisk /dev/sdb
which will take you into the interactive fdisk programme. It’s a little confusing so for now just type “n” (for new) and then “enter” to create a new partition. Following this choose “p” for Primary partition and then type “1″ to create partition 1. You’ll be asked what cylinder to start the partition on. I just accept the default (1) and also accept the default for the end cylinder (which will depend on the size of drive that you created). You’ll now need to mark the partition as wanting to be part of a RAID array. Type “t” (for type) and then enter “fd” for, erm, RAID array (it was logical up until then). Then write these changes to the drive with “w” (for write – yay, consistency returns).
You’ll need to repeat this for every physical drive that you want to be part of the array. You’ve already done /dev/sdb, so now get /dev/sdc and /dev/sdd done.
The only problem you might encounter here is if for some reason there is already a partition on the drives. If this is the case then before using fdisk to do any of the above steps, type “d” (for delete) to remove the already existing partition – then carry on as outlined.
Okay, so your drives are all ready to be RAIDed up! Time to install ::drumroll:: mdadm
sudo apt-get install mdadm
This is the package that allows you to create and control software RAID arrays. We’ll use it initially to create your RAID set
sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1
With that command you’ve said “create a RAID disk device at /dev/md0, comprised of three disks, called sdb1, sdc1 and sdd1″. Notice that we specified the partitions to use on your three SATA drives, not the drives themselves. Be sure to get the number of RAID devices correct – it’ll just cause heartache later on if you get it wrong.
Hopefully everything should go okay with that. If it has, then the system should already be syncronising the partitions with each other, in the background. You can watch this process and see when it’s complete with the most fantastic watch command
watch cat /proc/mdstat
You’ll see the percentage compeleted slowly creeping upwards. This could take minutes to hours, depending on the size of the drives and the numbers used. Why not go bake a cake while you’re waiting? Raspberry jam, for preference, as I love raspberries.
After a period of time equal to n the drives should be syncronised and the array built.
You’ll have to tell the mdadm configuration file about the new RAID that you’d like it to manage. The following command will output details about your new array and add it to the config file.
sudo mdadm --detail --scan >> /etc/mdadm/mdadm.confOkay, we’ve got a RAID array! It is currently sitting at /dev/md0, looking like a regular drive device to the OS. However it isn’t formatted or mounted just yet, so lets format it.
sudo mke2fs -j /dev/md0
This formats /dev/md0 with an ext3 filesystem. It’s a nice command as it handles all the settings for you and makes sensible assumptions. BTW, if you want the filesystem as ext2 then just leave off the -j option.
So now we’ve got a formatted device device we’ll have to mount it on the filesystem. You can mount it anywhere you want really, but for now I’m sticking it in /var/, so that it’s accessible to all users. You’ll need to create a mount point, which is just a directory that the to-be-mounted device will take over, like a Goa’uld or something:
sudo mkdir /var/media/
So that our RAID mounts on every boot we’ll need to add it to the fstab file, whereUubuntu keeps all the dirt on what drives it should take care of. Open fstab in nano:
sudo nano /etc/fstab
and add the following line to he bottom of the file:
/dev/md0 /var/media auto defaults 0 3
Now if you want to you can reboot the system and see that the RAID will get mounted on /var/media, but we can use the “Mount all” command to make it happen right now:
mount -aHopefully your RAID device should now be mounted at /var/media. Go on, have fun – copy some files to it. Copy them back. Delete them. Just go crazy.
I’ll hopefully follow this up with configuring alerts for failing drives in your system.
too bad you couldn’t find time or motivation to write your smart-howto.
Do you think you will do so?
Great How-to. It all worked like a charm.
It took about 5-6 Hours to synchronize the disks which was surprising to me. I am using three 1.5TB drives though. Im averaging about 18MB/s R on it which seems a little slow but not bad at all. RAID 5 FTW.
Now, at some point Ill have to figure out what to do when I add another drive
Looks like my RAID lasted a day and now it won’t mount cat /proc/mdstat gives me:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : inactive sdd1[2](S) sda1[0](S) sdc1[1](S)
4395410304 blocks
unused devices:
When I try to mount it I get:
you must specify the filesystem type
Dmesg | tail tells me:
[ 3167.806904] EXT3-fs: unable to read superblock
Oh yeah, here’s an output of mdadm -E on each device http://pastebin.com/m10b91733
Its weird that some report more failures than others…