[gpfsug-discuss] GPFS and Lustre on same node
Sven Oehme
oehmes at us.ibm.com
Fri Aug 8 20:09:44 BST 2014
Vic, Sergi,
you can not compare Lustre and GPFS without providing a clear usecase as
otherwise you compare apple with oranges.
the reason for this is quite simple, Lustre plays well in pretty much one
usecase - HPC, GPFS on the other hand is used in many forms of deployments
from Storage for Virtual Machines, HPC, Scale-Out NAS, Solutions in
digital media, to hosting some of the biggest, most business critical
Transactional database installations in the world. you look at 2 products
with completely different usability spectrum, functions and features
unless as said above you narrow it down to a very specific usecase with a
lot of details.
even just HPC has a very large spectrum and not everybody is working in a
single directory, which is the main scale point for Lustre compared to
GPFS and the reason is obvious, if you have only 1 active metadata server
(which is what 99% of all lustre systems run) some operations like single
directory contention is simpler to make fast, but only up to the limit of
your one node, but what happens when you need to go beyond that and only a
real distributed architecture can support your workload ?
for example look at most chip design workloads, which is a form of HPC, it
is something thats extremely metadata and small file dominated, you talk
about 100's of millions (in some cases even billions) of files, majority
of them <4k, the rest larger files , majority of it with random access
patterns that benefit from massive client side caching and distributed
data coherency models supported by GPFS token manager infrastructure
across 10's or 100's of metadata server and 1000's of compute nodes.
you also need to look at the rich feature set GPFS provides, which not all
may be important for some environments but are for others like Snapshot,
Clones, Hierarchical Storage Management (ILM) , Local Cache acceleration
(LROC), Global Namespace Wan Integration (AFM), Encryption, etc just to
name a few.
Sven
------------------------------------------
Sven Oehme
Scalable Storage Research
email: oehmes at us.ibm.com
Phone: +1 (408) 824-8904
IBM Almaden Research Lab
------------------------------------------
From: Vic Cornell <viccornell at gmail.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date: 08/08/2014 10:16 AM
Subject: Re: [gpfsug-discuss] GPFS and Lustre on same node
Sent by: gpfsug-discuss-bounces at gpfsug.org
Disclaimers - I work for DDN - we sell lustre and GPFS. I know GPFS much
better than I know Lustre.
The biggest difference we find between GPFS and Lustre is that GPFS - can
usually achieve 90% of the bandwidth available to a single client with a
single thread.
Lustre needs multiple parallel streams to saturate - say an Infiniband
connection.
Lustre is often faster than GPFS and often has superior metadata
performance - particularly where lots of files are created in a single
directory.
GPFS can support Windows - Lustre cannot. I think GPFS is better
integrated and easier to deploy than Lustre - some people disagree with
me.
Regards,
Vic
On 8 Aug 2014, at 14:14, Sergi Moré Codina <sergi.more at bsc.es> wrote:
> Hi all,
>
> About main differences between GPFS and Lustre, here you have some bits
from our experience:
>
> -Reliability: GPFS its been proved to be more stable and reliable. Also
offers more flexibility in terms of fail-over. It have no restriction in
number of servers. As far as I know, an NSD can have as many secondary
servers as you want (we are using 8).
>
> -Metadata: In Lustre each file system is restricted to two servers. No
restriction in GPFS.
>
> -Updates: In GPFS you can update the whole storage cluster without
stopping production, one server at a time.
>
> -Server/Client role: As Jeremy said, in GPFS every server act as a
client as well. Useful for administrative tasks.
>
> -Troubleshooting: Problems with GPFS are easier to track down. Logs are
more clear, and offers better tools than Lustre.
>
> -Support: No problems at all with GPFS support. It is true that it could
take time to go up within all support levels, but we always got a good
solution. Quite different in terms of hardware. IBM support quality has
drop a lot since about last year an a half. Really slow and tedious
process to get replacements. Moreover, we keep receiving bad "certified
reutilitzed parts" hardware, which slow the whole process even more.
>
>
> These are the main differences I would stand out after some years of
experience with both file systems, but do not take it as a fact.
>
> PD: Salvatore, I would suggest you to contact Jordi Valls. He joined EBI
a couple of months ago, and has experience working with both file systems
here at BSC.
>
> Best Regards,
> Sergi.
>
>
> On 08/08/2014 01:40 PM, Jeremy Robst wrote:
>> On Fri, 8 Aug 2014, Salvatore Di Nardo wrote:
>>
>>> Now, skipping all this GSS rant, which have nothing to do with the
file
>>> system anyway and going back to my question:
>>>
>>> Could someone point the main differences between GPFS and Lustre?
>>
>> I'm looking at making the same decision here - to buy GPFS or to roll
>> our own Lustre configuration. I'm in the process of setting up test
>> systems, and so far the main difference seems to be in the that in GPFS
>> each server sees the full filesystem, and so you can run other
>> applications (e.g backup) on a GPFS server whereas the Luste OSS
(object
>> storage servers) see only a portion of the storage (the filesystem is
>> striped across the OSSes), so you need a Lustre client to mount the
full
>> filesystem for things like backup.
>>
>> However I have very little practical experience of either and would
also
>> be interested in any comments.
>>
>> Thanks
>>
>> Jeremy
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at gpfsug.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>
>
> --
>
> ------------------------------------------------------------------------
>
> Sergi More Codina
> Barcelona Supercomputing Center
> Centro Nacional de Supercomputacion
> WWW: http://www.bsc.es Tel: +34-93-405 42 27
> e-mail: sergi.more at bsc.es Fax: +34-93-413 77 21
>
> ------------------------------------------------------------------------
>
> WARNING / LEGAL TEXT: This message is intended only for the use of the
> individual or entity to which it is addressed and may contain
> information which is privileged, confidential, proprietary, or exempt
> from disclosure under applicable law. If you are not the intended
> recipient or the person responsible for delivering the message to the
> intended recipient, you are strictly prohibited from disclosing,
> distributing, copying, or in any way using this message. If you have
> received this communication in error, please notify the sender and
> destroy and delete any copies you may have received.
>
> http://www.bsc.es/disclaimer.htm
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140808/f4de4ccd/attachment-0003.htm>
More information about the gpfsug-discuss
mailing list