HA on filesystems

The RAID system, beeing hardware or software, will protect your local filesystems against disk failure. But when these filesystems are shared to other system, you need also to protect against failure of the host sharing the filesystem. This can be done by using a SAN and a cluster software that will mount and share again some SAN LUN's if a server failed.

But there are alternative solutions to the costly SAN. The OpenSource community offers you various kind of mechanisms to build highly-available filesystem.

Filesystem replication

This is a first approach where a local filesystem is replicated from one server to another one. You can use file replications mechanisme like rsync wich is a standard commands in all Linux / Unix flavours. But this is implemented at user-level, run at time interval, leaving some delta (even of the size of the minute) between hosts.

You may then consider to use DRBD, the Linux disk blocks replication software. This software allows to create a mirror of any filesystem by replicating blocks across the network as soon as they are written on the primary host.
By implementing various protocols and way of working, you are able to make :
  • asynchronous replication (don't wait that the modified blocks are copied)
  • synchronous replication (wait that the modified blocks are copied, if the copy failed, the primary host filesystem returns an error like if a write errors occured on the primary server to let the system eventually retries the whole write process).
  • three-nodes setup
  • multi-master replication (dual primary support)
  • consistency checks using hasing algorythms
  • (...)
DRBD tightly integrates with the Pacemaker clustering engine to manage automatically the promotion and demotion of secondary copies.

DRBD can be used to synchronize the storage of a server to another one, taking part into high-availability, failover, disaster recovery, ... scenario's.

Clustered filesystems

A clustered filesystem may be or may not be an highly-available. First of all, this is a kind of filesystem that can be accessed simultaneously by more than one host.
A clustere filesystem use a Distributed Lock Manager to manage the filesystem locks between the various hosts that access the filesystem. This is a process that needs to be run on at least one host. Ideally, it should be made higly-available, thus integrated into one cluster software like Pacemaker to be always available. If the DLM is not there, the hosts cannot access the clustered filesystem as the locks cannot be placed anymore.
The clustered filesystem can be created on top of the block devices of replicated filesystem as setup by DRBD as seen above. This will create a redundant clustered filesystem.
To benefit from the clustered filesystem, you need to access it using block device connections like iSCSI or shared disk storage (disk storage boxes directly attached to more than one host).
Once again, by using iSCSI we have a cheapest alternative as there are Open Source implementation of the iSCSI protocol. Shared storage may be significally more costly.

Distributed filesystems

Distributed filesystems must not be confused with clustered filesystems. Clustered filesystems is about giving access to more than one host on a filesystem, wherever it is located.
Distributed filesystem is a filesystem which has part of it located on various system but is seen as a single unique standard filesystem from the host.