Just moozing

Before you can check your notes, you must make them…

Of iSCSI, snapshots, NAS and such

leave a comment »

My work with virtual machines have made me conclude that they should not contain data, only configurations and programs, so the natural questions is “then what?”. As I see it, there are basically three options – NFS, iSCSI and CIFS. This blog entry is about how I decided to do things.

The basics

CIFS is the thing to use in a windows environment. It works for users. As an administrator, you might not have or want access to your users passwords, and for CIFS to work, you need to log in to the server with proper credentials to get access as a certain user. This is good for users, and I use to mount my shares from my NAS. It is not good for administrative things where you need access to system things or others users home directories, like backup.

NFS is the good old way of giving access to files on a file server. No real security is implemented, and you must trust all machine on the local network. besides that I have always had an issue with synchronizing uids and gids, but using an LDAP server this is solved. NFS4 has the possibility of certificates, encryption and other kinds of modern security provided you have a Kerberos system running.

iSCSI is a way of charing block devices. It means that a NAS can serve a 100GB block device, which for most purposes, resembles an extra disk in a computer. You have the possibility of using all your normal disk tools on these block devices – including encryption. Using open-iscsi on my Linux boxes, it is a simple solution that requires minimal effort to set up. Though untested, it seems that Windows supports it well also. It also support multipath, which seems cool.

I have recently acquires a QNAP TS-420 NAS with 4x2TB disks, and it support all the above protocols. My choice is to use iSCSI for data storage in my virtual machines and backups, and CIFS for accessing user files.

Storage layout – client side

For a standard virtual machine running QEMU/KVM on Linux and no graphics, requirements a pretty low. I have servers running using as little as 100 MB of memory and less than 1GB of disk space. My default setup looks like this:

  • Virtual hard disk, 8 GB using qcow2
  • An iSCSI disk for storage of mysql, LDAP, IMAP, files or other data

Using open-iscsi, there really is no big difference between having two disks and the above. There is a joker about backup and resizing.

Normally, LVM would be a good choice on top of some RAID device, but the disk redundancy issue has been moved to the NAS, so the client is simpler. Again to simplify, I use the entire device (say, /etc/sda) for EXT4 without partitions. Also, I do backup from the NAS, not the virtual server itself.

Resizing is possible even on a live system. The QNAP supports resizing (as in “increasing size”) of iSCSI targets with connections, and EXT4 support resize of mounted file systems (using resize2fs). The trick is to tell the iSCSI initiator to check is the target has changed size.

Basic cookbook for resizing a mounted iSCSI partition (running EXT3/4):

  1. Increase size of iSCSI target
  2. Reread the info about iSCSI target: iscsiadm -m node -R
  3. Resize the underlying file system (to fill “partition”): resize2fs /dev/sdX

Always be carefull when working with data, and check your backups. That said, there is a lot of sanity checking and errors before you break anything, so it is fairly safe to do.

Storage layout – server side

Using the QNAP, it is trivial. It simply has a page in its web interface for handling these things.

On a Linux host, you need to consider how to handle the disks. I go for the combination of RAID and LVM as described here. The basic idea is to have the hardware redundancy (for speed and fail-over) using RAID and also have the partition flexibility of LVM. Resizing partitions is a handy option, especially when you have file of mail servers that will have a steadily increasing storage need.

The general layout

  • A server with multiple disks (eg, a HP Proliant server)
  • RAID1 or RAID5 using mdadm
  • LVM volume group on the entire multi device
  • LVM logical volumes using up to around 80-90% of the available space (to enable snapshots)

The really cool thing about LVM is the ability to do snapshots. Backing up several GB of data takes time and with a moderately active server, you are guaranteed to have changed while backing up. This might not be a problem, but some systems will require certain files and/or databases to be synchronized, and your backup might not be up for it. The solution is to do a snapshot. It is instant and takes up no space, before you start to make changes to the original disk. The snapshot is your guarantee that nothing will change on the partition while you backup.

The snapshot appears as any other partition on your LVM except it takes up less space. It may be mounted or used as an iSCSI target. I use it for regular file backup to another location, but it could be used when changing, say, the root partition, and you want a roll-back option. Yes, that is very cool.

Closing comments

  • That was a quick presentation of my storage strategy.
  • This is compatible with puppet and vagrant.
  • iSCSI may be optimized in many ways. I currently do 35-40 MB/s throughput, but I suspect it could get up to 80GB/s when I find the right levers.
  • I am having multiple links to Microsoft in this entry. Maybe I am getting old (or Microsoft have relevant documents on their site).

Written by moozing

February 23, 2014 at 09:00

Posted in Tech

Tagged with , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: