I’ve been playing with FreeNAS for a few days and I can’t say that I like it. I can’t put my finger on it, apart from a general feeling of working with over-padded mittens. I’ve therefore decided to go at it from scratch with Ubuntu.

The core of any decent NAS server is a bank of drives configured in a RAID array, so I’m going to start off my NAS building by configuring a level 5 software RAID set.

My scenario is that of an established Ubuntu setup, booting from a single SATA drive that I wish to add a large RAID array to, creating a large storage area that can be mounted anywhere on the filesystem, independent of the main OS and its partitions. For now I’m not intending to mount the OS itself on a RAID device – that’s another topic – but it would really make sense to have the OS sitting on a desperate RAID 1 device, benefiting from the additional redundancy. The nice thing here is that you just need a small drive to hold only the OS. Grab a couple of 5GB drives off Ebay for a fiver and use them!

For now this is all taking place inside a VirtualBox VM, but it should work just as well on a physical system.

So, first of all create a regular VM, with four SATA drives attached to it. The first one of these (on port 0) should be your OS drive. The other three (on ports 1,2 & 3) should be equally sized drives – I used drives of 5 GB for this test, but they can be as large or small as you want, it don’t matter, just as long as they are of equal size (and even then, it doesn’t matter too much as we’re gonna partition them) – these form your RAID array.

Install Ubuntu Server onto the first drive. During the install just look for a device called “sda”  and choose “Guided partitioning” for that. If you’re unsure, just connect the first drive by itself, install, shutdown, connect the RAID drives and reboot.

Once the OS is installed you can use

cat /proc/diskstats

to see all the physical drives and partitions on your system. Each drive on your system should show up in /dev/ as something like

/dev/sdx

where x is a letter starting with “a”, onwards.

Your OS drive should show as /dev/sda and you should also see something like /dev/sda1, /dev/sda2, /dev/sda5  and so on. sda represents the physical drive (well, as physical as it can be on a VM) and the numbered “sda”s afterwards represent partitions on the drive. Your OS drive is obviously already partitioned and formatted, but your array drives will likely have nothing to show but the plain unnumbered versions, as they are unpartioned as yet.

To start the process of building the RAID you’ll want to format the three drives of your array. If you’re using the command-line (you Linux warrior, you) then use fdisk to partition and format each drive. Run

sudo fdisk /dev/sdb

which will take you into the interactive fdisk programme. It’s a little confusing so for now just type “n” (for new) and then “enter” to create a new partition. Following this choose “p” for Primary partition and then type “1″ to create partition 1. You’ll be asked what cylinder to start the partition on. I just accept the default (1) and also accept the default for the end cylinder (which will depend on the size of drive that you created). You’ll now need to mark the partition as wanting to be part of a RAID array. Type “t” (for type) and then enter “fd” for, erm, RAID array (it was logical up until then). Then write these changes to the drive with “w” (for write – yay, consistency returns).

You’ll need to repeat this for every physical drive that you want to be part of the array. You’ve already done /dev/sdb, so now get /dev/sdc and /dev/sdd done.

The only problem you might encounter here is if for some reason there is already a partition on the drives. If this is the case then before using fdisk to do any of the above steps, type “d” (for delete) to remove the already existing partition – then carry on as outlined.

Okay, so your drives are all ready to be RAIDed up! Time to install ::drumroll:: mdadm

sudo apt-get install mdadm

This is the package that allows you to create and control software RAID arrays. We’ll use it initially to create your RAID set

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1

With that command you’ve said “create a RAID disk device at /dev/md0, comprised of three disks, called sdb1, sdc1 and sdd1″. Notice that we specified the partitions to use on your three SATA drives, not the drives themselves. Be sure to get the number of RAID devices correct – it’ll just cause heartache later on if you get it wrong.

Hopefully everything should go okay with that. If it has, then the system should already be syncronising the partitions with each other, in the background. You can watch this process and see when it’s complete with the most fantastic watch command

watch cat /proc/mdstat

You’ll see the percentage compeleted slowly creeping upwards. This could take minutes to hours, depending on the size of the drives and the numbers used. Why not go bake a cake while you’re waiting? Raspberry jam, for preference, as I love raspberries.

After a period of time equal to n the drives should be syncronised and the array built.

You’ll have to tell the mdadm configuration file about the new RAID that you’d like it to manage. The following command will output details about your new array and add it to the config file.

sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Okay, we’ve got a RAID array! It is currently sitting at /dev/md0, looking like a regular drive device to the OS. However it isn’t formatted or mounted just yet, so lets format it.

sudo mke2fs -j /dev/md0

This formats /dev/md0 with an ext3 filesystem.  It’s a nice command as it handles all the settings for you and makes sensible assumptions. BTW, if you want the filesystem as ext2 then just leave off the -j option.

So now we’ve got a formatted device device we’ll have to mount it on the filesystem. You can mount it anywhere you want really, but for now I’m sticking it in /var/, so that it’s accessible to all users. You’ll need to create a mount point, which is just a directory that the to-be-mounted device will take over, like a Goa’uld or something:

sudo mkdir /var/media/

So that our RAID mounts on every boot we’ll need to add it to the fstab file, whereUubuntu keeps all the dirt on what drives it should take care of. Open fstab in nano:

sudo nano /etc/fstab

and add the following line to he bottom of the file:

/dev/md0 /var/media auto defaults 0 3

Now if you want to you can reboot the system and see that the RAID will get mounted on /var/media, but we can use the “Mount all” command to make it happen right now:

mount -a

Hopefully your RAID device should now be mounted at /var/media. Go on, have fun – copy some files to it. Copy them back. Delete them. Just go crazy.

I’ll hopefully follow this up with configuring alerts for failing drives in your system.

References

• http://bfish.xaedalus.net/?p=188
• http://mywheel.net/blog/index.php/software-raid-in-ubuntu/
• http://www.review-ninja.com/2009/05/software-raid-raid-arrays-mdadm-on.html

I’ve been wanting a new laptop for a while now, ever since my last one (a Samsung Q25) spat out smoke and started melting any power adapter connected to it. Up until now I’ve always assumed that I’d end up with a Macbook, but today I came across the Dell XPS M1330.

Dear lordie, this is one sexy laptop. Only 13″ across (helloooo? Metric?) it’s mega-portable, but has got enough power to play modern 3D games, thanks to a dedicated graphics card, dual core processor and oodles of memory. Plus, it comes in pink and is sold with Ubuntu pre-installed!

I need it so badly.

I’ve just spent a day learning about GIt – we’re starting on a big project over the summer with additional developers and we’ll need a source control system. It’s been a bit of a steep learning curve as I’ve had to learn about versioning systems from scratch as well and now my brain needs to download.

So here is the fuckwits guide to starting a git-based project.

First of all, install Git. On Ubuntu that’ll be:

sudo apt-get install git-core

I don’t care what it is on other systems, as I don’t use them. Go figure it out yourself.

If you’re in an organisation then you’ll probably want a central repository. Choose the machine the repo will live on, install git-core, create a new user to own your repo (I’ve just called it “git”) and create a directory for you new project to live in (I just create a folder in the git user’s home dir).

sudo adduser git

su git

cd ~/

mkdir projecthome

cd projecthome

You’ll now need to make this folder into a git project:

git init

Fucking chavs are even in SCMs now (“give me yer money, init!?”).

Create a file, just to test things.

touch index.php

You’ll now need to tell git that you’re thinking of commiting this file to the repo.

git add index.php

or to add everything in the dir:

git add .

Once you’ve added any files you’ll need to commit them. This is the big deal and what Git is all about:

git commit -m "This is the first commit of the project"

The message is very important, as it tells other people what you did in this commit.

To see what you just did, type:

git log

Woo, you now have a project. You may now do a small dance.

Now switch to the machine where you will actually do your work on. You’ll need to make a copy of the repo on your repo server (make sure git-core is installed):

git clone git@<yourserveraddress>:/home/git/projecthome/

You’ll see some gumph appear and hopefully if you do an “ls” you’ll see:

index.php

Nice! You’ve got a working copy from a remote repo.

Now, try adding a new file here:

touch another.php

You’ll need to tell Git you want to add this file to your local repo:

git add another.php

And now we’ll commit it to the local repo:

git commit -m "I've just added another file"

And check out the log to see what has been done:

git log

Now, at this point you’ve committed your changes to your local copy (remember the clone?) of the project repo. Nothing wrong with that – you’ve got content versioning going on and can do all kinds of fancy things. However, none of your coworkers can see the changes you’ve made. Not much use really. Therefore you’ll push those changes back to the central repo, with, erm:

git push

Woo! Now everyone else can access the changes you’ve made.

If one of your co-workers makes a change on their copy of the repo and pushes it back to the central repo you’ll need to pull it back down. It’s just as obvious a command again:

git pull

Any changes they’ve made will show up in your directory.

I’m going to brain dump some more here later on. For now I’m off to figure out branching and merging…