[gpfsug-discuss] Online data migration tool
Nikhil Khandelwal
nikhilk at us.ibm.com
Thu Nov 30 00:00:23 GMT 2017
Hi Aaron,
By large block size we are primarily talking about block sizes 4 MB and
greater. You are correct, in my previous message I neglected to mention
the file create performance for small files on these larger block sizes due
to the subblock change. In addition to the added space efficiency, small
file creation (for example 32kB files) on large block size filesystems will
improve.
In the case of a 1 MB block size, there would be no real difference in file
creates. For a 16 MB block size, however there will be a performance
improvement for small file creation as a part of the subblock change for
new filesystems. For users who are upgrading from 4.X.X to 5.0.0, the file
creation speed will remain the same after the upgrade.
I hope that helps, sorry for the confusion.
Thank you,
Nikhil Khandelwal
Spectrum Scale Development
Client Adoption
From: Aaron Knister <aaron.knister at gmail.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 11/29/2017 03:42 PM
Subject: Re: [gpfsug-discuss] Online data migration tool
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Thanks, Nikhil. Most of that was consistent with my understnading, however
I was under the impression that the >32 subblocks code is required to
achieve the touted 50k file creates/second that Sven has talked about a
bunch of times:
http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf
http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_Sven_Oehme_-_News_from_Research.pdf
http://files.gpfsug.org/presentations/2016/SC16/12_-_Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
from those presentations regarding 32 subblocks:
"It has a significant performance penalty for small files in large block
size filesystems"
although I'm not clear on the specific definition of "large". Many
filesystems I encounter only have a 1M block size so it may not matter
there, although that same presentation clearly shows the benefit of larger
block sizes which is yet *another* thing for which a migration tool would
be helpful.
-Aaron
On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com>
wrote:
Hi,
I would like to clarify migration path to 5.0.0 from 4.X.X clusters. For
all Spectrum Scale clusters that are currently at 4.X.X, it is possible
to migrate to 5.0.0 with no offline data migration and no need to move
data. Once these clusters are at 5.0.0, they will benefit from the
performance improvements, new features (such as file audit logging), and
various enhancements that are included in 5.0.0.
That being said, there is one enhancement that will not be applied to
these clusters, and that is the increased number of sub-blocks per block
for small file allocation. This means that for file systems with a large
block size and a lot of small files, the overall space utilization will
be the same it currently is in 4.X.X. Since file systems created at 4.X.X
and earlier used a block size that kept this allocation in mind, there
should be very little impact on existing file systems.
Outside of that one particular function, the remainder of the performance
improvements, metadata improvements, updated compatibility, new
functionality, and all of the other enhancements will be immediately
available to you once you complete the upgrade to 5.0.0 -- with no need
to reformat, move data, or take your data offline.
I hope that clarifies things a little and makes the upgrade path more
accessible.
Please let me know if there are any other questions or concerns.
Thank you,
Nikhil Khandelwal
Spectrum Scale Development
Client Adoption
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=WUJ15T9xHCCIfLm1wqC74jhfu28fXGLotYoHQvJlMCg&m=GNrHjCLvQL1u_WHVimX2lAlYOGPzciCFrYHGlae3h_E&s=VtVgCRl7kxNRgcl5QeHdZJ0Rz6jCA-jfQXyLztbr5TY&e=
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171129/0d5532c0/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171129/0d5532c0/attachment-0002.gif>
More information about the gpfsug-discuss
mailing list