I wanted to start writing about GlusterFS, a Clustered/Distributed filesystem using FUSE. I believe Filesystems like this will slowly shrink traditional FC SAN deployments. SAN deployments are costly, using high priced drives that in many cases never get utilized. With the emergence of SATAII/SATAIII, AoE, 10GiB ethernet, and clustered/distributed filesystems any company can build a robust storage environment over ethernet that can replace the following:

  • CIFS -EMC CX4-240 2GB FC disks
  • NFS – EMC CX4-240 2GB FC disks
  • Oracle RMAN backup – iSCSI SATA 3GB, iSCSI 1GB ethernet
  • File backup – Tape no disk, we want to add disk

I am currently working on moving the services mentioned above to GlusterFS. We believe this move will save us money without sacrificing performance.

GlusterFS over FUSE client:

In this post I will describe briefly the requirements of running GLIBC 2.8 and why its import before you start the move to GlusterFS. Also I describe the benefits of running in FUSE mode as compared to CIFS/NFS.

GlusterFSNative Mode using FUSE client

Before you move to GlusterFS you must ask why? Do I need to? In my eyes there are three reasons why you may want to move to GlusterFS

  1. Take cheap commodidty hardware/disks and put them to use.
  2. Throughput – you need to push data at speeds past 125 MB/s without using an expensive SAN/NAS.
  3. Protection – you may need to protect your data with more than just RAID and Tape/D2D backups. GlusterFS will use AFR to create mirrors of your data.

If you want points #2 or #3 you will want to run GlusterFSNative mode. You do not need to for #3, but it is highly recommended if you are going to be pushing more than 25 MB/s.

At my job we need/want all three. We want to use GlusterFSfor three type of data processes:

  1. Oracle RMAN backup – high reads/writes
  2. NFS share for XenServer Virtual Machines – low reads/writes
  3. GlusterFS shared filesystem for random data – medium reads/writes

So I started down the road for Oracle RMAN backups. We are running iSCSI on a 16 drive RAID50 over 1GiB for RMAN backups. We consistently get 118 MB/s throughput. The problem is with iSCSI you can’t get more than that. TCP sesions cannot be shared accross NICs. So GlusterFS would be perfect for us. We have tested the RAID50 and know we can get 250 MB/s in writes and 300+ in reads. So I started to setup GlusterFS and quickly found that GlusterFS Native client is only supported with GLIBC 2.8 or higher. The GlusterFSNFS/CIFS option would not work for this. The reason is Native mode will start connections accross all network adapters, aggregating your data and achieving more than 1GiB. NFS/CIFS can only bind to one adapter therefor limiting you to 1GiB.

So now our Oracle servers need to be upgraded to SLES 11 as well as Oracle 11g.

GlusterFS over CIFS/NFS:

Luckily only our DB servers are running SLES 10. All of our application servers are runnign openSuSE 11.1 as PV virtual machines wich supports GlusterFS 3.xxxx. We plan to use NFS only in one use case, Citrix XenServer shared storage. Citrix XenServer will only use FC(LVM), iSCSI(LVM) or NFS as a shared filesystsm for running VMs in HA mode. Currently for testing we have the following setup for XenServer:

  • 6 x PV Virtual Machines running GlusterServer
  • Each VM is on a seperate physical server
  • Each VM has 1 x 250GB SATA disk
  • Filesystem is mirrored = 672GB useable
  • Exported as NFS for Citrix XenServer

So far I have been running 4 VMs on this storage as well as some logging data. We plan to move our existing CIFS data from EMC FC to GlusterFSas well. Before we do we just need to test how well it works with Windows Server.

In January we will be building out our production GlusterFS filesystem. Once it is built I will be posting performance/problems and TCO for building this out.

  1. live xenserver backup
    May 20, 2011 at 3:51 am

    This is very useful, I think this will come in handy for me at work

  2. April 20, 2012 at 4:34 am

    Hi this is kinda of off topic but I was wanting to know if blogs use WYSIWYG editors or if you have to manually code with HTML. I’m starting a blog soon but have no coding knowledge so I wanted to get guidance from someone with experience. Any help would be enormously appreciated!

  3. May 2, 2012 at 7:18 pm

    Any update on this project?

    • May 3, 2012 at 4:46 pm

      Yes. Had many issues with replication running LAMP stack. JAVA would have files that had wrong permissions and ownership with failed syncs. We abandoned GlusterFS and used NFS with 10G. GlusterFS is still growing and being used successfully. If you plan on useing it just test, test, test. Shutdown nodes, remove network cables, put extra files on a replicated node. Make sure you don’t have any issues before you walk down that path of putting data on the GlusterFS.

      Also if uptime is not a concern, the distributed method works well for the price.

  4. October 26, 2012 at 3:52 pm

    It’s fantastic that you are getting ideas from this article as well as from our discussion made at this place.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: