[gpfsug-discuss] GPFS and Lustre on same node

Sven Oehme oehmes at us.ibm.com
Fri Aug 8 20:09:44 BST 2014


Vic, Sergi,

you can not compare Lustre and GPFS without providing a clear usecase as 
otherwise you compare apple with oranges. 
the reason for this is quite simple, Lustre plays well in pretty much one 
usecase - HPC, GPFS on the other hand is used in many forms of deployments 
from Storage for Virtual Machines, HPC, Scale-Out NAS, Solutions in 
digital media, to hosting some of the biggest, most business critical 
Transactional database installations in the world. you look at 2 products 
with completely different usability spectrum, functions and features 
unless as said above you narrow it down to a very specific usecase with a 
lot of details.
even just HPC has a very large spectrum and not everybody is working in a 
single directory, which is the main scale point for Lustre compared to 
GPFS and the reason is obvious, if you have only 1 active metadata server 
(which is what 99% of all lustre systems run) some operations like single 
directory contention is simpler to make fast, but only up to the limit of 
your one node, but what happens when you need to go beyond that and only a 
real distributed architecture can support your workload ? 
for example look at most chip design workloads, which is a form of HPC, it 
is something thats extremely metadata and small file dominated, you talk 
about 100's of millions (in some cases even billions) of files, majority 
of them <4k, the rest larger files , majority of it with random access 
patterns that benefit from massive client side caching and distributed 
data coherency models supported by GPFS token manager infrastructure 
across 10's or 100's of metadata server and 1000's of compute nodes. 
you also need to look at the rich feature set GPFS provides, which not all 
may be important for some environments but are for others like Snapshot, 
Clones, Hierarchical Storage Management (ILM) , Local Cache acceleration 
(LROC), Global Namespace Wan Integration (AFM), Encryption, etc just to 
name a few. 

Sven

------------------------------------------
Sven Oehme 
Scalable Storage Research 
email: oehmes at us.ibm.com 
Phone: +1 (408) 824-8904 
IBM Almaden Research Lab 
------------------------------------------



From:   Vic Cornell <viccornell at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date:   08/08/2014 10:16 AM
Subject:        Re: [gpfsug-discuss] GPFS and Lustre on same node
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Disclaimers - I work for DDN - we sell lustre and GPFS. I know GPFS much 
better than I know Lustre.

The biggest difference we find between GPFS and Lustre is that GPFS - can 
usually achieve 90% of the bandwidth available to a single client with a 
single thread.

Lustre needs multiple parallel streams to saturate - say an Infiniband 
connection.

Lustre is often faster than GPFS and often has superior metadata 
performance - particularly where lots of files are created in a single 
directory.

GPFS can support Windows - Lustre cannot. I think GPFS is better 
integrated and easier to deploy than Lustre - some people disagree with 
me.

Regards,

Vic


On 8 Aug 2014, at 14:14, Sergi Moré Codina <sergi.more at bsc.es> wrote:

> Hi all,
> 
> About main differences between GPFS and Lustre, here you have some bits 
from our experience:
> 
> -Reliability: GPFS its been proved to be more stable and reliable. Also 
offers more flexibility in terms of fail-over. It have no restriction in 
number of servers. As far as I know, an NSD can have as many secondary 
servers as you want (we are using 8).
> 
> -Metadata: In Lustre each file system is restricted to two servers. No 
restriction in GPFS.
> 
> -Updates: In GPFS you can update the whole storage cluster without 
stopping production, one server at a time.
> 
> -Server/Client role: As Jeremy said, in GPFS every server act as a 
client as well. Useful for administrative tasks.
> 
> -Troubleshooting: Problems with GPFS are easier to track down. Logs are 
more clear, and offers better tools than Lustre.
> 
> -Support: No problems at all with GPFS support. It is true that it could 
take time to go up within all support levels, but we always got a good 
solution. Quite different in terms of hardware. IBM support quality has 
drop a lot since about last year an a half. Really slow and tedious 
process to get replacements. Moreover, we keep receiving bad "certified 
reutilitzed parts" hardware, which slow the whole process even more.
> 
> 
> These are the main differences I would stand out after some years of 
experience with both file systems, but do not take it as a fact.
> 
> PD: Salvatore, I would suggest you to contact Jordi Valls. He joined EBI 
a couple of months ago, and has experience working with both file systems 
here at BSC.
> 
> Best Regards,
> Sergi.
> 
> 
> On 08/08/2014 01:40 PM, Jeremy Robst wrote:
>> On Fri, 8 Aug 2014, Salvatore Di Nardo wrote:
>> 
>>> Now, skipping all this GSS rant, which have nothing to do with the 
file
>>> system anyway  and  going back to my question:
>>> 
>>> Could someone point the main differences between GPFS and Lustre?
>> 
>> I'm looking at making the same decision here - to buy GPFS or to roll
>> our own Lustre configuration. I'm in the process of setting up test
>> systems, and so far the main difference seems to be in the that in GPFS
>> each server sees the full filesystem, and so you can run other
>> applications (e.g backup) on a GPFS server whereas the Luste OSS 
(object
>> storage servers) see only a portion of the storage (the filesystem is
>> striped across the OSSes), so you need a Lustre client to mount the 
full
>> filesystem for things like backup.
>> 
>> However I have very little practical experience of either and would 
also
>> be interested in any comments.
>> 
>> Thanks
>> 
>> Jeremy
>> 
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at gpfsug.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
> 
> 
> -- 
> 
> ------------------------------------------------------------------------
> 
>     Sergi More Codina
>     Barcelona Supercomputing Center
>     Centro Nacional de Supercomputacion
>     WWW: http://www.bsc.es      Tel: +34-93-405 42 27
>     e-mail: sergi.more at bsc.es   Fax: +34-93-413 77 21
> 
> ------------------------------------------------------------------------
> 
> WARNING / LEGAL TEXT: This message is intended only for the use of the
> individual or entity to which it is addressed and may contain
> information which is privileged, confidential, proprietary, or exempt
> from disclosure under applicable law. If you are not the intended
> recipient or the person responsible for delivering the message to the
> intended recipient, you are strictly prohibited from disclosing,
> distributing, copying, or in any way using this message. If you have
> received this communication in error, please notify the sender and
> destroy and delete any copies you may have received.
> 
> http://www.bsc.es/disclaimer.htm
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140808/f4de4ccd/attachment-0003.htm>


More information about the gpfsug-discuss mailing list