[gpfsug-discuss] GPFS and both Samba and NFS

Orlando Richards orlando.richards at ed.ac.uk
Fri Dec 13 15:15:03 GMT 2013

On 12/12/13 18:14, Lindsay Todd wrote:
> Hello,
> Since this is my first note to the group, I'll introduce myself first.
>   I am Lindsay Todd, a Systems Programmer at Rensselaer Polytechnic
> Institute's Center for Computational Innovations, where I run a 1.2PiB
> GPFS cluster serving a Blue Gene/Q and a variety of Opteron and Intel
> clients, run an IBM Watson, and serve as an adjunct faculty.  I also do
> some freelance consulting, including GPFS, for several customers.
> One of my customers is needing to serve GPFS storage through both NFS
> and Samba; they have GPFS 3.5 running on RHEL5 (not RHEL6) servers.  I
> did not set this up for them, but was called to help fix it.  Currently
> they export NFS using cNFS; I think we have that straightened out
> server-side now.  Also they run Samba on several of the servers; I'm
> sure the group will not be surprised to hear they experience file
> corruption and other strange problems.
> I've been pushing them to use Samba-CTDB, and it looks like it will
> happen.  Except, I've never used this myself.  So this raises a couple
> questions:
> 1) It looks like RHEL5 bundles in an old version of CTDB. Should that be
> used, or would we be better with a build from the Enterprise Samba site,
> or even a build from source?

Hi Lindsay,

We rebuild ctdb from the (git) source (in the 1.2.40 branch currently), 
after running into performance problems with the sernet bundled version 
(1.0.114). It's easy to build:

git clone git://git.samba.org/ctdb.git ctdb.git
cd ctdb.git
git branch -r
git checkout -b "my_build" origin/1.2.40
cd packaging/RPM/
yum install /root/rpmbuild/RPMS/x86_64/ctdb*.rpm

I then take the Sernet src rpm and rebuild it, using ctdb.h from the 
above rather than the 1.0.114 version they use. This is possibly not 
required, but I thought it best to be sure that the differing headers 
wouldn't cause any problems. I remain, as ever, very grateful to Sernet 
for providing these!

> 2) Given that CTDB can also run NFS, what are people who need both
> finding works best: run both cNFS + Samba-CTDB, or let CTDB run both?
>   It seems to me that if I let CTDB run both, I only need a single
> floating IP address for each server, while if I also use cNFS, I will
> want a floating address for both NFS and Samba, on each server.

We let CTDB run both, but we didn't come to that decision by comparing 
the merits of both options. I think Bristol (Bob Cregan is cc'd, I'm not 
sure he's on this list) run cNFS and CTDB side by side. As you say - 
you'd at least require different IP addresses to do that.

> Thanks for the help!

Best of luck :)

> R. Lindsay Todd, PhD
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

    Dr Orlando Richards
   Information Services
IT Infrastructure Division
        Unix Section
     Tel: 0131 650 4994
   skype: orlando.richards

The University of Edinburgh is a charitable body, registered in 
Scotland, with registration number SC005336.

More information about the gpfsug-discuss mailing list