I wanted to start writing about GlusterFS, a Clustered/Distributed filesystem using FUSE. I believe Filesystems like this will slowly shrink traditional FC SAN deployments. SAN deployments are costly, using high priced drives that in many cases never get utilized. With the emergence of SATAII/SATAIII, AoE, 10GiB ethernet, and clustered/distributed filesystems any company can build a robust storage environment over ethernet that can replace the following:
- CIFS -EMC CX4-240 2GB FC disks
- NFS – EMC CX4-240 2GB FC disks
- Oracle RMAN backup – iSCSI SATA 3GB, iSCSI 1GB ethernet
- File backup – Tape no disk, we want to add disk
I am currently working on moving the services mentioned above to GlusterFS. We believe this move will save us money without sacrificing performance.
GlusterFS over FUSE client:
In this post I will describe briefly the requirements of running GLIBC 2.8 and why its import before you start the move to GlusterFS. Also I describe the benefits of running in FUSE mode as compared to CIFS/NFS.
GlusterFSNative Mode using FUSE client
- Take cheap commodidty hardware/disks and put them to use.
- Throughput – you need to push data at speeds past 125 MB/s without using an expensive SAN/NAS.
- Protection – you may need to protect your data with more than just RAID and Tape/D2D backups. GlusterFS will use AFR to create mirrors of your data.
If you want points #2 or #3 you will want to run GlusterFSNative mode. You do not need to for #3, but it is highly recommended if you are going to be pushing more than 25 MB/s.
At my job we need/want all three. We want to use GlusterFSfor three type of data processes:
- Oracle RMAN backup – high reads/writes
- NFS share for XenServer Virtual Machines – low reads/writes
- GlusterFS shared filesystem for random data – medium reads/writes
So I started down the road for Oracle RMAN backups. We are running iSCSI on a 16 drive RAID50 over 1GiB for RMAN backups. We consistently get 118 MB/s throughput. The problem is with iSCSI you can’t get more than that. TCP sesions cannot be shared accross NICs. So GlusterFS would be perfect for us. We have tested the RAID50 and know we can get 250 MB/s in writes and 300+ in reads. So I started to setup GlusterFS and quickly found that GlusterFS 3.xxx Native client is only supported with GLIBC 2.8 or higher. The GlusterFSNFS/CIFS option would not work for this. The reason is Native mode will start connections accross all network adapters, aggregating your data and achieving more than 1GiB. NFS/CIFS can only bind to one adapter therefor limiting you to 1GiB.
So now our Oracle servers need to be upgraded to SLES 11 as well as Oracle 11g.
GlusterFS over CIFS/NFS:
Luckily only our DB servers are running SLES 10. All of our application servers are runnign openSuSE 11.1 as PV virtual machines wich supports GlusterFS 3.xxxx. We plan to use NFS only in one use case, Citrix XenServer shared storage. Citrix XenServer will only use FC(LVM), iSCSI(LVM) or NFS as a shared filesystsm for running VMs in HA mode. Currently for testing we have the following setup for XenServer:
- 6 x PV Virtual Machines running GlusterServer
- Each VM is on a seperate physical server
- Each VM has 1 x 250GB SATA disk
- Filesystem is mirrored = 672GB useable
- Exported as NFS for Citrix XenServer
So far I have been running 4 VMs on this storage as well as some logging data. We plan to move our existing CIFS data from EMC FC to GlusterFSas well. Before we do we just need to test how well it works with Windows Server.
In January we will be building out our production GlusterFS filesystem. Once it is built I will be posting performance/problems and TCO for building this out.